What is DeepSeek-V3, and how was it trained for so little?

On December 26, 2024, DeepSeek, an AI company in China, published DeepSeek-V3, a new language model whose weights anyone can download. Weights are the billions of numbers a model learns during training; releasing them means you can run, study and adapt the model on your own machines instead of only renting access through a website. Today the company followed up with a detailed technical report on arXiv, the open archive where researchers post papers.

Two things make this release stand out. By DeepSeek's own measurements, V3 is the strongest openly available model so far, close to the best closed systems from OpenAI and Anthropic. And the company says it trained V3 on a budget that most labs would consider tiny.

What DeepSeek released

The package has three parts: the model weights, the paper and an update to DeepSeek's paid API (the service programmers call from their own apps).

A token is a chunk of text, often part of a word; TechCrunch notes that a million tokens is roughly 750,000 words. Until February 8, API customers keep paying the old V2 prices. The license is permissive: developers can download and modify the model and use it in most commercial products.

How it works: a team of specialists

Most chat models are dense: every part of the network works on every word. DeepSeek-V3 is a mixture of experts (MoE). Inside, it holds many smaller sub-networks, the experts, plus a router that picks a few of them for each token. Think of a large hospital: every patient does not see every doctor. A triage nurse sends each case to the right specialists, so the hospital can employ hundreds of experts while each visit stays short.

That is why V3 can be huge yet affordable to run. Its knowledge is spread across 671 billion parameters, but only 37 billion do the work for any given token, which keeps each answer cheap to compute.

The paper describes several refinements on top of that idea:

• Multi-head Latent Attention, carried over from DeepSeek-V2, compresses the notes the model keeps about earlier text, so long conversations need less memory.
• A new load-balancing method keeps the router from sending too much work to a few favorite experts, without the penalty term other MoE models add during training.
• Multi-token prediction trains the model to guess more than one upcoming token at each step, which DeepSeek says improves results.
• Low-precision arithmetic: DeepSeek trained with compact 8-bit numbers (a format called FP8) for much of the math, and says it is the first to show this works at such a large scale.

After this main training, DeepSeek refined the model with supervised fine-tuning (learning from example answers) and reinforcement learning (learning from feedback on its outputs).

The training bill

The headline figure comes from the paper itself. DeepSeek trained V3 on a cluster of 2,048 Nvidia H800 chips. The main training phase took less than two months, and the whole process used 2.788 million GPU hours (one chip working for one hour). At an assumed rental price of $2 per GPU hour, that comes to $5.576 million.

Two cautions matter here. First, DeepSeek says the figure covers only the final official run, not the earlier research and experiments that led to it. Second, TechCrunch points out that H800 chips are among those the US Commerce Department has recently restricted Chinese companies from buying. Building a frontier-class model under those limits is part of what has caught researchers' attention.

How it compares, and its limits

On the MMLU knowledge test, DeepSeek reports a score of 88.5, the best of any open model it compared. It says V3 is roughly level with GPT-4o and Anthropic's Claude 3.5 Sonnet on many tests, beats o1-preview on the MATH-500 math benchmark and leads on competition-style coding tests such as LiveCodeBench. TechCrunch adds that, in DeepSeek's testing, V3 tops Meta's Llama 3.1 405B, Alibaba's Qwen 2.5 72B and GPT-4o on a set of Codeforces programming contests.

These are the company's own numbers, and independent tests will take time. The paper is also candid that V3 trails those two closed models on English factual questions while doing better on Chinese ones. TechCrunch reports that the model declines questions on politically sensitive topics such as Tiananmen Square, as Chinese regulators require. And "open" does not mean "small": running the full model at a usable speed takes a rack of high-end GPUs, not a laptop.

Why it matters this week

For a year, the usual assumption has been that matching the top closed models takes enormous budgets and tens of thousands of chips. DeepSeek-V3 challenges that assumption in public, with a paper others can check. If the efficiency claims hold up, many expect training costs to become a bigger part of the conversation, along with the gap between open and closed models, which DeepSeek says is narrowing.

For builders, the practical news is a strong model with open weights and a low API price. For everyone else, it is a sign that capable AI is no longer coming only from a few American labs.

How can you try DeepSeek-V3 yourself?

The simplest way is DeepSeek's free chat assistant on its website. Create an account, then try a task you know well so you can judge the answer. Avoid pasting private or sensitive information, as you would with any online assistant.

If you write code, DeepSeek's API keeps the same request format as before, so apps already built on its V2 models keep working, and DeepSeek's current prices hold until February 8.

The weights themselves are on GitHub and Hugging Face. Downloading them makes sense for companies and research groups with serious GPU servers; the full model is far too large for a home computer. For most readers, the chat and the API are the practical doors in.