xAI just released the Grok-1 weights. What does that mean, and can you use them?

Yesterday, Sunday, March 17, 2024, Elon Musk's AI company xAI published the weights of Grok-1, the large language model it built for its Grok chatbot. Weights are the billions of numbers a model learns during training; they are, in a practical sense, the model itself. Anyone can now download them, study them and build on them, including for commercial products.

Musk had said on X the week before that xAI would open up Grok during this week, and the company delivered on Sunday with a short blog post and a public code repository. With 314 billion parameters, it is a very large model to hand to the public.

What xAI released, and where

The announcement came in a brief post on xAI's blog titled "Open Release of Grok-1," dated March 17. It points to a GitHub repository, xai-org/grok-1, where the files and instructions live. Here is what the package contains and what it leaves out:

The weights are shared through a torrent, a way of downloading large files in pieces from many computers at once, using a "magnet link" listed in the repository.

TechCrunch notes that the Grok chatbot on X, which launched last year for Premium+ subscribers, can draw on posts from the social network. The open model has none of that: it is the language engine alone.

How Grok-1 works, in plain terms

Three ideas explain most of what is in the release.

Parameters. Grok-1 has 314 billion of them. A parameter is one adjustable number inside the network, tuned during training so the model gets better at predicting the next word. More parameters usually means more capacity to store patterns about language and the world, but also more memory and computing power to run.

Mixture of Experts. Grok-1 does not use all 314 billion parameters for every word. It is a Mixture-of-Experts (MoE) model: parts of the network are split into specialist sub-networks, called experts, and a small routing component picks which ones handle each token (a word or piece of a word). xAI says about 25% of the weights are active for any given token. A good picture is a large hospital: every patient sees a couple of specialists, not the whole staff, which keeps each visit faster than consulting everyone.

Base model. This is the raw checkpoint from the end of pre-training, the first and largest phase of training, in which a model reads enormous amounts of text and learns to continue it. xAI says that phase ended in October 2023 and that the model has not been fine-tuned for any purpose, dialogue included. A base model is good at continuing text, not at following instructions. Ask it a question and it may simply write more questions, because that is a plausible continuation.

You can see the difference in how you would talk to each kind of model:

Chat model style:   "Explain photosynthesis to a 10-year-old."
Base model style:   "Photosynthesis explained for kids, in three short paragraphs.
                    Paragraph 1: Plants are like little kitchens."

The base model works best when you start the text you want and let it carry on.

The license, and why it matters

The weights and code are released under Apache 2.0, a common open-source license. It lets anyone use, change and redistribute the software, including in paid products, as long as they keep the license and copyright notices. TechCrunch points out that this permits commercial use without asking xAI.

Open models are no longer rare. TechCrunch lists Meta's LLaMA, Mistral, Falcon and models from the Allen Institute for AI (AI2), and Google released two open Gemma models in February. Grok-1 joins that group at the large end, with one of the simplest licenses to follow.

Why it matters this week

For researchers, a model of this size with open weights is something to study: how its experts divide the work, how it compares with other open models, and how it responds to further training. Because it is a base model, it is also a starting point. Companies can fine-tune it, meaning train it further on their own examples, to turn it into a chat assistant, a search helper or a specialist tool. TechCrunch reports that Aravind Srinivas, CEO of the search company Perplexity, has already said on X that his team plans to fine-tune Grok for conversational search.

The release also lands in a public argument. Musk sued OpenAI earlier this month, accusing it of abandoning its nonprofit mission, and he has been criticizing the company on X since. Publishing Grok-1's weights lets xAI point to openness of its own. Openness here has limits, as TechCrunch's headline stresses: there is no training code, and no data, so outsiders can use the model but cannot reproduce how it was made.

Can you run Grok-1 yourself, and what would you need?

In principle, yes; in practice, most people cannot on a personal computer. The repository's instructions are short: download the checkpoint with a torrent client, place the folder named ckpt-0 in the checkpoints directory, install the listed Python packages and run a script called run.py, which loads the model and generates text from a test input. The catch is size. The README warns that, with 314 billion parameters, testing the model needs a machine with a lot of GPU memory, meaning the memory on graphics processors, the chips that run AI models. That points to server-class hardware or rented cloud machines, not a laptop.

xAI also says its example code handles the Mixture-of-Experts layer in a simple, inefficient way, chosen to check that the model works correctly rather than to run it fast. Expect community projects to appear that make it lighter or quicker, as has happened with other open models.

If you just want to talk to Grok, the open weights are not the route; the chatbot on X is. If you are a developer or student curious about how large models are built, the repository is a useful place to read real model code, even if you never download the weights.