# Mixture-of-experts or dense, what does each design demand from hardware?

Date: 2026-07-14 · Author: Robert Scales · https://guidetoartificialintelligence.com/en/learn/moe-vs-dense-architecture/

**Answer:** A dense model uses all its parameters for every token, while a mixture-of-experts model uses only a small fraction, so it gets more capacity for the same computing per token. The price is memory and data movement, since every expert must stay loaded and be reached quickly, which is why very large MoE models need racks of tightly linked chips.

If you are new to the idea, our [explainer on mixture-of-experts models](/en/learn/mixture-experts-model/) covers how a router sends each token to a few sub-networks. This article picks up from there and asks the practical question: what does each design demand from hardware, and when does each one make sense?

In short, a model has two different sizes. **Total parameters** decide how much memory you need. **Active parameters** (the ones used per token) decide most of the computing per token. In a dense model those two numbers are the same. In a mixture-of-experts (MoE) model they can be far apart.

## How far apart the two numbers get

| Model | Design | Total parameters | Active per token | Share active |
|---|---|---|---|---|
| Qwen3-32B | Dense | 32 billion | 32 billion | 100% |
| Mixtral 8x7B | MoE | About 47 billion | About 13 billion | About 28% |
| Qwen3-235B-A22B | MoE | 235 billion | 22 billion | About 9% |
| DeepSeek-V3 | MoE | 671 billion | 37 billion | About 6% |
| Kimi K2 | MoE | 1 trillion | 32 billion | About 3% |

Look at the first and last rows. Kimi K2 has about 31 times as many parameters as Qwen3-32B, yet does about the same amount of math per token. That is the attraction: more stored knowledge and skill, without a matching rise in the cost of each step.

Parameter count is not a direct measure of quality, and these models differ in many other ways, including training data and methods. The table compares structure, not ability.

## A worked example: the memory bill

Now the catch. Suppose you store each parameter in 8 bits, or one byte, a common compressed format. This simplified arithmetic ignores the extra working memory a real system needs.

| | Dense 32B model | Kimi K2 (1T total, 32B active) |
|---|---|---|
| Memory for the model | About 32 GB | About 1,000 GB |
| Parameters used per token | 32 billion | 32 billion |
| Fits on one 48 GB GPU? | Yes | No |
| Fits on one 300 GB data-centre GPU? | Yes | No, needs at least 4 |

So the MoE model has the same computing bill per token and about 31 times the memory bill. Memory is expensive and [currently scarce](/en/learn/hbm-memory-shortage-explained/), so this is a real trade rather than a free lunch.

## Why MoE strains the connections between chips

Memory size is only the first demand. Nvidia's engineers describe two further problems in a December 2025 post. Since a frontier MoE is too big for one GPU, its experts get spread across many, a method called expert parallelism. Then:

- **Memory bandwidth comes under pressure**, because different experts' parameters must be pulled from fast memory for each token.
- **Chips must talk constantly**, because tokens are sent to experts on other GPUs and the answers must be combined quickly. If that traffic leaves the fast internal links for slower networking, the benefit shrinks.

This is why the design has shaped hardware. Nvidia says that a 72-GPU rack, the GB200 NVL72, runs Kimi K2 Thinking and similar models 10 times faster than its earlier H200 systems, at a tenth of the cost per token. Google, for its part, described its Ironwood TPU as built for workloads that include MoE models. Treat vendor figures with care: they come from the company selling the system, under chosen test conditions.

There is a second hidden cost in training. Experts need to share work evenly, or some become overloaded while others sit idle. The DeepSeek-V3 paper describes a load-balancing method designed to handle this without adding a penalty to training. Dense models simply do not have this problem.

## Which designs are winning, and where

At large sizes, MoE has become the norm. In December 2025, Nvidia noted that the top 10 open models on the Artificial Analysis leaderboard all used it, including DeepSeek-R1, Kimi K2 Thinking, gpt-oss-120B and Mistral Large 3. It also claims nearly all frontier models work this way, but closed labs mostly do not publish their designs, so that broader claim is hard to check.

Dense has not vanished. Qwen3's family includes dense models at 8, 14 and 32 billion parameters alongside its MoE versions, and small dense models remain common. At that size, the savings from routing are smaller and simplicity counts for more: one block per layer, no router and no load balancing.

| If your priority is | Lean toward | Why |
|---|---|---|
| Serving many users at the top end | MoE | Low computing per token, and many requests spread work across experts |
| Fitting the smallest possible memory | Dense | Everything loaded is used, so no memory sits idle for a given token |
| Simple training code | Dense | No router or load balancing |
| High capability from a fixed computing budget per token | MoE | More total parameters for similar math per token |

> **Try it:** Open the model card of any open model you are considering and write down the total and active parameters. Multiply total by the bytes per parameter of the version you plan to download. That is your memory floor, whatever speed the active number suggests.

## Which architecture should you choose if you run models yourself?

Start with memory, because it is the hard limit. Work out the total-parameter size of the download and check that it fits your GPU or unified memory, with room to spare. If a dense model and an MoE model both fit, try both on your own task, since the MoE may run noticeably faster for the same memory than its total size suggests.

If only part of a model fits in fast memory, some tools can leave the rest in slower system memory. MoE models can sometimes do better than dense ones in that situation, because only some experts are active per token. It will still be slower than having everything in fast memory, and results vary by tool and machine, so test before you buy hardware. For a beginner with a typical machine, a small dense or small MoE model from a recent family is the easiest place to start.

## Sources

- Mixture of Experts Powers the Most Intelligent Frontier AI Models, Runs 10x Faster on NVIDIA Blackwell NVL72: https://blogs.nvidia.com/blog/mixture-of-experts-frontier-models/
- DeepSeek-V3 Technical Report: https://arxiv.org/abs/2412.19437v1
- Kimi K2: Open Agentic Intelligence: https://arxiv.org/abs/2507.20534

---

From the Guide to Artificial Intelligence, by Robert Scales, published by Holistic Hybrid Inc. Share and adapt it under CC BY-NC-SA 4.0 (https://creativecommons.org/licenses/by-nc-sa/4.0/): credit the guide, link to https://guidetoartificialintelligence.com/en/learn/moe-vs-dense-architecture/, no commercial use without permission, and keep the same licence.