# What is Mistral 7B, and how can a small model compete with bigger ones?

Date: 2023-09-28 · Author: Robert Scales · https://guidetoartificialintelligence.com/en/learn/what-is-mistral-7b/

**Answer:** Mistral 7B is the first language model from the French startup Mistral AI, released on September 27, 2023: a 7.3-billion-parameter model that anyone can download and use without restrictions under the Apache 2.0 license. Mistral reports that it beats Meta's Llama 2 13B, nearly twice its size, on every benchmark it tested, thanks partly to attention techniques that make it faster and lighter to run.

Yesterday a young French company called **Mistral AI** published its first model, **Mistral 7B**, and made it free for anyone to use for any purpose. It is small by today's standards, about 7.3 billion parameters, yet Mistral says it outperforms models nearly twice as large.

Mistral raised a very large first funding round in June, and according to TechCrunch its founders gained experience building comparable models at Google DeepMind and Meta. The team says the model is the product of about three months of work, including building its training setup and data pipeline from scratch.

## What was released

Mistral offers two models:

- **Mistral 7B**, the base model, trained to continue text.
- **Mistral 7B Instruct**, a version fine-tuned for chat, meaning it was given extra training on examples of instructions and good answers. Mistral used only instruction datasets that are publicly available on Hugging Face, with no private data.

The files can be downloaded in several ways, including a 13.4-gigabyte torrent, a peer-to-peer file-sharing link, which already had a few hundred people sharing it when TechCrunch reported the launch. Mistral also provides reference code for running the model on your own computer and instructions for deploying it on the big cloud platforms (Amazon, Google and Microsoft), along with a GitHub repository and a Discord channel for discussion.

## How it compares

Mistral re-ran every test itself on its own evaluation setup so the comparisons would be fair, a point worth remembering since these are the company's own numbers. Its main claims:

| Compared with | Mistral's result |
|---|---|
| Llama 2 13B (almost twice the size) | Better on all benchmarks tested |
| Llama 1 34B (almost five times the size) | Better on many benchmarks |
| Code Llama 7B (a specialist coding model) | Close on code, while staying good at English tasks |
| Llama 2 13B chat (for the Instruct version) | Better, and the best 7B model on the MT-Bench chat test |

Mistral also offers a helpful way to think about efficiency. On reasoning, comprehension and the broad MMLU knowledge test, it says Mistral 7B performs like a Llama 2 model more than three times its size. That translates into less memory and faster answers for the same quality.

There is an honest exception. On knowledge tests, which ask for facts such as trivia answers, Mistral 7B only matches Llama 2 13B. The company's explanation: a smaller model has fewer parameters in which to store facts.

## How it works: two attention tricks

Like nearly all recent language models, Mistral 7B is a **Transformer**. Its key mechanism, **attention**, lets each word look at earlier words to decide what matters. Mistral changed how that looking happens in two ways.

**Grouped-query attention (GQA).** In standard attention, many separate "heads" each keep their own copy of certain data about earlier words. GQA lets groups of heads share that data, which reduces memory use and speeds up generation with little loss in quality.

**Sliding window attention (SWA).** Each layer of the model looks back over only the most recent 4,096 positions instead of the entire text. That sounds like a limit, but the layers are stacked. Here is the idea, step by step:

1. Layer 1 sees the last 4,096 tokens.
2. Layer 2 looks at layer 1's results, and each of those already summarized 4,096 tokens, so layer 2 can draw on information from about twice as far back.
3. Each higher layer reaches further still.

An analogy: in a long line of people passing messages, each person only talks to the few nearest neighbors, yet news from the far end still arrives after a few rounds. Mistral says this, combined with improvements it contributed to the FlashAttention and xFormers libraries, doubles speed on 16,000-token sequences, and that a fixed window halves the memory cache for 8,192-token inputs without hurting quality.

## Why it matters

The license is the headline. Mistral 7B uses **Apache 2.0**, a standard open-source software license with no limits on use or redistribution beyond giving credit. Meta's Llama 2 and Alibaba's Qwen-7B, by contrast, come with their own licenses that include conditions for very large companies. A hobbyist, a startup or a major corporation can use Mistral 7B on equal terms.

Small, strong models also matter because they are cheap to run. A model of this size can generally run on a single graphics card, so more people can experiment, and companies can keep their data in-house.

There are caveats. The Instruct model has no built-in moderation yet; Mistral says it wants to work with the community on guardrails for settings that need filtered output. And TechCrunch notes that "open" has limits: the model was built privately, and the training data has not been published. Mistral also plans paid commercial products, including hosted services for businesses.

> **Try it:** Pick a question in your field that has a clear right answer. Ask it to a large chatbot and then to a small model such as Mistral 7B Instruct in an online demo. Note where the small model holds its own and where it slips, especially on facts.

## What does the Apache 2.0 license let you do with Mistral 7B?

Apache 2.0 lets you use, copy, modify and distribute the model, including in commercial products, without paying fees or asking permission. You can fine-tune it on your own data and ship the result as part of an app or service.

The main obligations are about credit and notices: when you redistribute it, you include the license and keep the copyright and attribution information. The license also includes a patent grant from contributors, which gives businesses extra legal comfort.

What the license does not do is guarantee anything. The model comes "as is," with no warranty, and responsibility for how it behaves in your product is yours. Since the Instruct version has no moderation built in, add your own safety checks before putting it in front of customers.

## Sources

- Mistral 7B: https://mistral.ai/news/announcing-mistral-7b/
- Mistral AI makes its first large language model free for everyone: https://techcrunch.com/2023/09/27/mistral-ai-makes-its-first-large-language-model-free-for-everyone/