What is a mixture-of-experts model?

Most language models are "dense": every part of the network does work for every token, the small piece of text the model handles at each step. A mixture-of-experts model, often shortened to MoE, is "sparse" instead. It contains many parallel sub-networks but only switches on a few of them for any given token. The idea lets a model know more without making every step slower.

The approach is not new. In 2017, a Google Brain team led by Noam Shazeer described a "sparsely-gated" mixture-of-experts layer with up to thousands of expert sub-networks and a trainable gate that picked a few for each input. What has changed is that an openly released model now shows the idea working well at a size many developers can actually use: Mixtral 8x7B, released by Mistral AI in December, with a full technical paper published on January 8.

A restaurant-kitchen analogy

Picture a busy restaurant. In a dense kitchen, every cook touches every order: the pastry chef, the grill cook and the sauce cook all handle your salad. It works, but it is slow and the kitchen can't grow without every order taking longer.

In a mixture-of-experts kitchen, there are eight cooks at each station, and a head chef at the pass glances at each order and calls out the two cooks best placed to handle it. Their work is combined, weighted by how confident the head chef was in each, and the plate moves to the next station, where a different head chef makes a fresh choice. The kitchen employs many cooks, but each order only costs the time of two at each station.

The analogy has one important limit. In Mixtral, the "cooks" are not neatly specialized by topic. The paper's authors checked whether certain experts handled mathematics, biology or philosophy and found no obvious pattern. Routing seemed to follow the structure of the text more than its subject, such as sending code indentation or a word like self in Python to the same experts.

How it works inside a Transformer

A Transformer layer has two main parts: attention, which lets tokens share context, and a feedforward block, a small network that processes each token on its own. In Mixtral, every layer keeps a single shared attention part but replaces the one feedforward block with eight expert blocks.

A router (also called a gating network) is a small learned layer that gives each expert a score for the current token. Mixtral keeps the two highest scores, turns them into weights that add up to one, runs only those two experts and blends their outputs. This happens separately at each of its 32 layers, so a single token can visit different pairs of experts on its way up.

A worked routing table

The table shows how routing might look for one short phrase at one layer. The expert numbers and weights are illustrative, not taken from the real model.

Two things stand out. Each token uses only two of the eight experts, so most of the layer stays idle for that token. And neighboring tokens often reuse the same experts, a pattern the Mixtral paper reports seeing in real text as well.

Why does it make big models cheaper to run?

The cost of producing each token depends mostly on how many parameters (the learned numbers inside the model) are actually used in the calculation. Mixtral has about 47 billion parameters in total, but because only two experts per layer run for each token, only about 13 billion are active at once. Mistral says this lets it run at the speed and cost of a model of roughly that smaller size, while its paper reports results that match or beat Llama 2 70B, a dense model that uses all 70 billion of its parameters for every token.

There is a catch, and the paper is clear about it. Memory does not shrink. All eight experts in every layer must be loaded, so the hardware has to hold the full 47 billion parameters even if each token only uses a part of them. The router also adds some overhead, and the savings are largest when a server handles many requests at once, so the work spreads evenly across experts.

So "cheaper" means cheaper per token in computation, not cheaper to fit on a small machine. For companies serving many users, that trade is attractive: they pay for memory once and save on compute for every token. For someone running a model at home, the memory requirement is usually the limit to check first.