# What do Q4_K_M, GGUF, MXFP4 and FP8 mean, and which one should you download?

Date: 2026-01-11 · Author: Robert Scales · https://guidetoartificialintelligence.com/en/learn/quantization-explained-gguf-mxfp4/

**Answer:** GGUF is a file format, while Q4_K_M, MXFP4 and FP8 are ways of storing each number in fewer bits. For most people, a Q4_K_M GGUF file is the safe default on llama.cpp, Ollama and LM Studio, and a model that was trained for MXFP4, like gpt-oss, should be used in that form.

A model download page can look like a wall of codes: `Q4_K_M`, `UD-Q2_K_XL`, `MXFP4`, `BF16`, `FP8`, `4bit`. Some name the container, the kind of file you are getting. The others name the precision, how many bits store each number.

The [setup guide](/en/learn/run-models-own-computer-2026/) covers the basic idea of quantization and its quality limits. This article decodes the names and explains how the newer formats differ.

## Containers and precision

A **container** is the file format. **GGUF** is the one used by llama.cpp and the tools built on it. Its [specification](https://github.com/ggml-org/ggml/blob/master/docs/gguf.md) aims for fast loading and a single file holding everything needed, including the metadata that describes the model. OpenAI's gpt-oss, like many models on Hugging Face, also comes as **safetensors** files, and `mlx-lm` converts models for Apple's MLX.

**Precision** is how each weight is stored. Training normally uses 16 or 32 bits. Quantization rounds those numbers into fewer bits, and the trick behind every scheme is to keep the rounding from doing too much damage.

## Why plain rounding is not enough

Suppose you had only 16 possible values, which is what 4 bits gives you. Weights differ widely in size, so a single scale for the whole model would flatten most small numbers to zero. The [Register's explainer](https://www.theregister.com/2025/08/10/openai_mxfp4/) shows this with four small numbers: stored directly in 4 bits, 0.0625, 0.375, 0.078125 and 0.25 turn into 0, 0.5, 0 and 0.5.

The fix is to split the weights into small blocks, and give each block its own scale. Small numbers then stay distinct inside their block. The block scale costs a little extra space, and this is why a "4-bit" format is never quite 4 bits per weight.

## The main schemes compared

| Name | Used by | Bits per weight | How it scales |
|---|---|---|---|
| `Q8_0` | llama.cpp | about 8.5 | One scale per block |
| `Q5_K_M` | llama.cpp | about 5.7 | Super-blocks of 8 blocks, 32 weights each |
| `Q4_K_M` | llama.cpp | about 4.9 | Same layout, 4-bit values |
| `Q2_K` | llama.cpp | about 3.2 | Super-blocks of 16 blocks, 16 weights each |
| `MXFP4` | gpt-oss in Ollama and vLLM | 4.25 | 32 weights share one 8-bit scale |
| `FP8` | vLLM and similar servers | 8 | Usually one scale for a whole tensor |
| MLX 4-bit | Apple silicon, through `mlx-lm` | about 4 plus scales | Set by the conversion tool |
| `BF16`, `F16` | Near-original quality | 16 | None to speak of |

The bits per weight for the `Q` formats are the [measurements published by llama.cpp](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md) on Llama 3.1 8B, and the layouts come from [Hugging Face's GGUF table](https://huggingface.co/docs/hub/gguf).

That table puts pure `Q4_K` at 4.5 bits, while the measured `Q4_K_M` file reaches 4.89, because not every part of the model is stored at the lowest precision. Uploaders push this idea further. Files marked `UD-`, from [Unsloth](https://unsloth.ai/docs/basics/unsloth-dynamic-2.0-ggufs), choose layer by layer using a calibration dataset. Unsloth reports better accuracy at the same size. That is the maker's claim, and it also warns that its 1-bit files loop and fail at tool calling.

## What makes MXFP4 different

MXFP4 comes from a [2023 Open Compute Project specification](https://www.opencompute.org/documents/ocp-microscaling-formats-mx-v1-0-spec-final-pdf), whose authors come from AMD, Arm, Intel, Meta, Microsoft and NVIDIA, among others. The arithmetic is easy to check: 32 weights at 4 bits each is 128 bits, plus one 8-bit scale, so 136 bits for 32 weights, or 4.25 bits per weight. That matches the figure in OpenAI's gpt-oss model card.

Two things set it apart from a `Q4` file.

**The numbers are floating point.** Each 4-bit value has a sign, an exponent and a mantissa, like a tiny scientific notation. Some chips handle this directly: NVIDIA lists FP4 support for the [Tensor Cores in DGX Spark](https://docs.nvidia.com/dgx/dgx-spark/hardware.html). Others do not, and The Register notes that the H100 lacks native support yet still runs these models.

**The model can be trained for it.** OpenAI says it post-trained gpt-oss with its mixture-of-experts weights in MXFP4, more than 90% of all parameters, and ran its evaluations in that form. The model has learned to work with that rounding rather than having it imposed afterward. So this is a property of that release, and converting an arbitrary model to MXFP4 does not give you the same benefit by itself.

## A worked example

Take gpt-oss-20b. OpenAI counts 20.9 billion parameters. Using the formula from the [memory calculator](/en/learn/vram-calculator-which-model-fits/), 90% of those at 4.25 bits comes to about 10 GB. The Ollama build is 14 GB, so the small remainder, under 10% of the weights, must account for about 4 GB. Those weights take far more room each.

This also shows why a low label can mislead on a model like this. If most weights stay in MXFP4 whatever the file is called, changing the label on the rest barely moves the size.

> **Try it:** Open a model on Hugging Face that offers several GGUF files. Divide each file's size in GB by the parameter count in billions, then multiply by 8. Compare the bits per weight you get with the table above, and notice which files are higher than the name implies.

## Which file should you pick on a download page?

Start from your engine, because that decides the container:

| Your tool | Look for |
|---|---|
| Ollama, LM Studio, llama.cpp | GGUF, or the tool's default tag |
| Ollama or `mlx-lm` on a Mac | An MLX build, if one is offered |
| vLLM | Safetensors in FP8, AWQ or INT4; [vLLM also lists MXFP4 and GGUF](https://github.com/vllm-project/vllm) |

Then choose the precision. `Q4_K_M` or an MLX 4-bit build is a sensible default for chat and coding. If memory is left over, step up to `Q5_K_M` or `Q6_K` before you spend it elsewhere. If you are short of memory, a larger model at 4 bits usually beats a smaller one at 8, as the setup guide explains. Below about 3 bits, test carefully on your own tasks, especially tool calling, since agents can break before chat seems to suffer.

If the model's makers published and tested a quantized version, start there, because their numbers apply to that exact file. And never quantize a file that is already quantized: llama.cpp warns that it can severely reduce quality, so go back to the 16-bit original instead.

## Sources

- OCP Microscaling Formats (MX) Specification, version 1.0: https://www.opencompute.org/documents/ocp-microscaling-formats-mx-v1-0-spec-final-pdf
- gpt-oss-120b and gpt-oss-20b Model Card: https://cdn.openai.com/pdf/419b6906-9da6-406c-a19d-1bb078ac7637/oai_gpt-oss_model_card.pdf
- GGUF specification: https://github.com/ggml-org/ggml/blob/master/docs/gguf.md

---

From the Guide to Artificial Intelligence, by Robert Scales, published by Holistic Hybrid Inc. Share and adapt it under CC BY-NC-SA 4.0 (https://creativecommons.org/licenses/by-nc-sa/4.0/): credit the guide, link to https://guidetoartificialintelligence.com/en/learn/quantization-explained-gguf-mxfp4/, no commercial use without permission, and keep the same licence.