# What does it mean for a language model to be open?

Date: 2022-07-22 · Author: Robert Scales · https://guidetoartificialintelligence.com/en/learn/mean-language-model-open/

**Answer:** A language model is "open" to the degree that outsiders can get its weights (the learned numbers that make up the model), its code and the details of how it was trained, and use them under a license that allows it. BLOOM and OPT are open in this sense, though each license sets rules, while GPT-3 can be used only through OpenAI's paid API.

Two releases this spring and summer have made the word "open" central to conversations about large language models. In May, Meta AI shared OPT-175B, a model with 175 billion parameters, the same size as GPT-3. This month, the BigScience research collaboration released BLOOM, a 176-billion-parameter model trained to write in 46 human languages and 13 programming languages. Both make their weights available to people outside the group that trained them, which GPT-3 does not.

"Open" does not mean one single thing, though. To understand what these releases offer, and what they do not, it helps to look at the pieces that make up a model.

## The parts of a model that can be shared

A trained language model is mostly a very large file of numbers called **weights** or parameters. During training, these numbers are adjusted again and again until the model becomes good at predicting the next word. If you have the weights and the code that runs them, you have the model itself: you can run it on your own computers, study how it behaves inside, and train it further on your own data (this is called fine-tuning).

Around the weights sit other pieces that may or may not be shared:

| Piece | What it tells you or lets you do |
|---|---|
| Weights | Run, inspect and fine-tune the model yourself |
| Code | Reproduce the training or run the model efficiently |
| Training data details | Know what the model learned from, and its likely blind spots |
| Documentation | Understand intended uses, risks and test results |
| License | Know what you are legally allowed to do with all of the above |

GPT-3 sits at the closed end for most people. OpenAI offers it only through its API: you send text and receive text, but the weights stay on OpenAI's servers. When it launched the API in 2020, OpenAI explained the choice: selling access funds its research, the model is too large and expensive for most organizations to run, and an API lets the company cut off misuse.

## How open are OPT and BLOOM?

**OPT-175B**, from Meta AI, comes with the pretrained model and the code used to train and run it, a step Meta presents as new at this scale. Meta also published its development notes and the logbook of the training run. The full 175-billion model, however, is released under a noncommercial license for research, and access is granted on request to academic, government, civil-society and industry research labs. Smaller versions, from 125 million to 30 billion parameters, are fully released for anyone to download.

**BLOOM**, from BigScience, a collaboration funded by Hugging Face, the French government and partner organizations, is available to download from the Hugging Face Hub. It comes with a detailed model card describing its training data (1.6 terabytes of text), its training on the Jean Zay public supercomputer in France, its intended uses and its known risks. Its license, the BigScience RAIL License, grants access but includes a list of use restrictions; RAIL stands for Responsible AI License. Uses such as generating disinformation, harassment or impersonating people without consent are named as misuse.

So both are "open" in the sense that the weights and documentation can be obtained, and both come with conditions. A useful habit is to say exactly what is open (weights, code, data, license terms) rather than relying on the word alone.

## What can you do with BLOOM or OPT?

For most people, the easiest start is to try BLOOM in a browser. Its page on the Hugging Face Hub has a small test box, called an inference widget, where you can type a prompt and see the model continue it.

1. Go to huggingface.co/bigscience/bloom.
2. Read the top of the model card: what BLOOM is, its license and its intended uses.
3. In the widget on the right, try one of the example prompts its authors provide, or type your own. BLOOM continues text rather than following instructions, so start a sentence for it to finish:

```prompt
Il était une fois, dans un petit village au bord d'un lac,
```

4. Click **Compute** and read the continuation. Then try the same opening in another language you know and compare the quality.

Running BLOOM yourself is a different matter. Its weights alone take up 329 gigabytes, and the full model needs powerful graphics processors (GPUs) to run at a reasonable speed. Meta notes that its code can run OPT-175B on 16 NVIDIA V100 GPUs, which is still far beyond a home computer. Most individuals will use the widget, a hosted service, or one of the smaller OPT models, which need much less memory.

> **Try it:** Give BLOOM the same story opening in English, then in French or Spanish. Note where the continuation sounds natural and where it drifts, and check what the model card says about how much of each language was in its training data.

For researchers and builders, open weights make new kinds of work possible: measuring bias directly, studying how the model represents language, fine-tuning a smaller version for a specialized task, or comparing models trained in different ways. BLOOM's model card says its purpose is to make open, public research on large language models possible, and it lists high-stakes uses, such as medical, legal or financial decisions, as out of scope. Open access widens who can do this work; it does not change how carefully the results need to be checked.

## Sources

- BLOOM model card (version 1.3, revision of 21 July 2022): https://huggingface.co/bigscience/bloom/blob/483e9e2e0489e2da4222246222f92a63689c33c9/README.md
- Democratizing access to large-scale language models with OPT-175B: https://ai.facebook.com/blog/democratizing-access-to-large-scale-language-models-with-opt-175b/
- OpenAI API: https://openai.com/index/openai-api/