# What is Mixtral 8x7B, and how does a mixture of experts work?

Date: 2023-12-12 · Author: Robert Scales · https://guidetoartificialintelligence.com/en/learn/what-is-mixtral-8x7b/

**Answer:** Mixtral 8x7B is a language model released on December 11, 2023, by the Paris start-up Mistral AI, with open weights that anyone can download and use under the permissive Apache 2.0 license. It uses a "sparse mixture of experts" design, in which a small router sends each piece of text to two of eight specialist sub-networks, so it has about 46.7 billion parameters in total but uses only about 12.9 billion for each token, and Mistral says it matches or beats GPT-3.5 and Llama 2 70B on most standard tests.

Yesterday Mistral AI, a young French company based in Paris, officially released Mixtral 8x7B, a language model you can download, run on your own hardware and use commercially with very few restrictions. According to Mistral's own tests, it performs about as well as OpenAI's GPT-3.5 on most standard benchmarks.

That claim is getting attention because open models, ones whose files are published for anyone to use, have generally trailed the best closed systems. Developers are already running it on their own computers, and well-known researchers such as Andrej Karpathy have publicly said the results look like a real step forward.

## Who made it, and what was released

Mistral AI was founded in Paris in May. Its three co-founders are Arthur Mensch, its CEO, along with Guillaume Lample and Timothée Lacroix. In September it released Mistral 7B, a smaller model it also gave away as a free download. Yesterday the company also closed a funding round of 385 million euros, about $415 million, led by the venture firm Andreessen Horowitz.

The release has three parts:

- **Mixtral 8x7B**, the base model, trained on text from the open web.
- **Mixtral 8x7B Instruct**, a version tuned to follow instructions, using supervised fine-tuning (training on examples of good answers) and direct preference optimization, or DPO (training on pairs of answers where one is preferred).
- **A paid API**, in beta, where Mixtral runs under the name `mistral-small`, alongside `mistral-tiny` (Mistral 7B) and a stronger `mistral-medium`, which is available only through the API.

The **weights**, the files of numbers that make up a trained neural network, are licensed under Apache 2.0, an open-source license that allows commercial use. They are available on the model-sharing site Hugging Face and through BitTorrent.

## How a mixture of experts works

Most language models are "dense": every part of the network works on every word. Mixtral is **sparse**. Inside each of its layers, one component, called the feedforward block, comes in eight separate versions, the "experts." Before each token (a word or piece of a word) is processed, a small **router** network looks at it and picks two of the eight. Only those two do the work, and their results are added together.

Ars Technica offers a factory analogy: instead of every worker touching every product, a supervisor sends each job to the two workers best suited to it. The factory has many workers, but each job uses only a few.

The name can mislead. "8x7B" suggests eight complete 7-billion-parameter models, or 56 billion parameters. In fact only the feedforward blocks are multiplied by eight; the rest of the network is shared. A **parameter** is one of the adjustable numbers a model learns during training.

| | Mixtral 8x7B |
|---|---|
| Total parameters | 46.7 billion (Mistral's announcement); TechCrunch's quote of Mistral's post gives about 45 billion |
| Parameters used per token | 12.9 billion (announcement); about 12 billion in TechCrunch's quote |
| Experts per layer | 8 |
| Experts chosen per token | 2 |
| Context window | 32,000 tokens |
| Languages | English, French, Italian, German, Spanish |
| License | Apache 2.0 |

The payoff is efficiency. Because only about 13 billion parameters are active for each token, Mixtral runs at roughly the speed and cost of a 13-billion-parameter model while drawing on the knowledge stored in all 46.7 billion. Mistral says it answers about six times faster than Meta's Llama 2 70B while beating it on most benchmarks.

## How well it performs

Mistral reports that Mixtral matches or outperforms both Llama 2 70B and GPT-3.5 on most of the benchmarks it tested, and is strong at writing code. The Instruct version scores 8.30 on MT-Bench, a test of how well chat models handle multi-turn conversations, which Mistral says makes it the best open-source model on that test and comparable to GPT-3.5. On TruthfulQA, a test of whether models repeat common falsehoods, the base model scored 73.9% against Llama 2's 50.2%.

These are the company's own numbers, so independent testing over the coming weeks will matter. Mistral also notes that without a guiding prompt, Mixtral will follow whatever instructions it is given, so builders who need content limits must add them.

> **Try it:** Write one question in English and the same question in French, Spanish, German or Italian. Ask both to Mixtral, through Mistral's platform or a local app such as LM Studio, and compare the answers. Check whether quality holds up across languages, and whether the model answers in the language you used.

## Why it matters

Mixtral narrows the gap between open and closed models. For developers, that means a GPT-3.5-class model they can run on their own servers, inspect, fine-tune and build into products without sending data to an outside company. Early users report running it locally at around 30 tokens per second on an Apple M1 computer.

It also brings mixture-of-experts design into the open. The idea is not new, but Mixtral is a widely available, well-documented example that shows how a model can grow in total knowledge without growing equally in running cost. Mistral has contributed code to vLLM, an open-source serving system, so others can run it efficiently.

Finally, it shows how quickly a newcomer can reach this level. Mistral is about seven months old, and its CEO, Arthur Mensch, describes the goal as building a European leader in generative AI with an open approach.

## How can you try Mixtral yourself?

If you are a developer, the simplest route is Mistral's own platform, which serves Mixtral as `mistral-small`; you can register there for early access to its API.

To run it on your own computer, desktop apps such as LM Studio let you download a version of the model and chat with it offline. Early users report good speeds on recent Apple laptops, though a large download and a well-equipped machine help.

Researchers and companies can download the full weights from Hugging Face and use them under Apache 2.0, including in commercial products. Start with the Instruct version if you want a chat assistant, and add a system-style instruction describing what the model should and should not do.

## Sources

- Mixtral of experts: https://mistral.ai/news/mixtral-of-experts/
- Everybody's talking about Mistral, an upstart French challenger to OpenAI: https://arstechnica.com/information-technology/2023/12/new-french-ai-model-makes-waves-by-matching-gpt-3-5-on-benchmarks/
- Mistral AI, a Paris-based OpenAI rival, closed its $415 million funding round: https://techcrunch.com/2023/12/11/mistral-ai-a-paris-based-openai-rival-closed-its-415-million-funding-round/