How do you prepare a dataset to train a language model from scratch?

A pretraining dataset is the raw text a language model reads to learn how language behaves, before anyone teaches it to chat. It is different from the few hundred input-and-output pairs used in fine-tuning: here you need billions of characters, and you will never read most of them. So the work is to choose a good source and then write filters you can trust.

Choosing a source

Do not begin by scraping the Web. Several groups have already done the hard filtering, and their licence terms are documented.

Two warnings hide in that table. ClimbMix is what nanochat downloads by default since March 2026, and its licence bars commercial use. A model trained for fun is one thing; a model you plan to sell is another. And "open" does not mean "free of other people's rights". FineWeb's data comes from crawled web pages, and its licence on the dataset does not mean each page's author agreed to model training. The Common Pile authors took the opposite road, keeping only material whose licence they could trust, and they dropped sources such as YouTube Commons when they could not. Their 7-billion-parameter models came out comparable to budget-matched Llama 1 and 2 models, so the strict path is workable.

For French, FineWeb itself will not help, since it filters out documents that are not confidently English. Use FineWeb2's French subset or Common Corpus.

What cleaning involves

FineWeb's card describes a pipeline worth copying in spirit: filter by URL, detect the language, apply quality rules (for example, rejecting pages that are mostly lists or repeated lines), remove near-duplicates and hide email and IP addresses. It also states plainly that personal information is very likely still in there. No filter is perfect, so if a person's details could be harmful, leave out that source altogether.

Duplicates deserve special attention, because a model repeatedly shown the same text tends to memorize it. Identical copies are easy to catch with a hash, a short fingerprint of the text, as below. Near-copies need a method such as MinHash, which FineWeb uses; the free datatrove library implements it.

A small pipeline you can run

This script reads a JSONL file in which each line has text, source and licence fields. It keeps only licences on your list, hides obvious emails and phone numbers, drops very short documents and exact copies, shuffles, sets aside 2% for validation and writes Parquet files, a compact column format that the training tools read well. Run pip install pyarrow first, and try it on a small sample file.

import hashlib, json, random, re, unicodedata
import pyarrow as pa, pyarrow.parquet as pq

EMAIL = re.compile(r"\b[\w.+-]+@[\w-]+\.[\w.-]+\b")
PHONE = re.compile(r"\b\d{3}[ .-]?\d{3}[ .-]?\d{4}\b")

def clean(text):
    text = unicodedata.normalize("NFC", text)
    text = EMAIL.sub("[email]", text)
    text = PHONE.sub("[phone]", text)
    return re.sub(r"[ \t]+", " ", text).strip()

seen, docs = set(), []
for line in open("raw.jsonl", encoding="utf-8"):
    row = json.loads(line)
    if row["licence"] not in {"cc0-1.0", "cc-by-4.0", "public-domain"}:
        continue
    text = clean(row["text"])
    if len(text) < 200:
        continue
    key = hashlib.sha256(text.lower().encode()).hexdigest()
    if key in seen:
        continue
    seen.add(key)
    docs.append({"text": text, "source": row["source"], "licence": row["licence"]})

random.Random(42).shuffle(docs)
n_val = max(1, len(docs) // 50)
splits = {"val": docs[:n_val], "train": docs[n_val:]}
for name, rows in splits.items():
    pq.write_table(pa.Table.from_pylist(rows), f"{name}.parquet", row_group_size=1024)
    print(name, len(rows), "documents", sum(len(r["text"]) for r in rows), "characters")

The NFC line matters for French. An "é" can be stored as one character or as "e" plus an accent mark, and without normalizing, your tokenizer treats them as two different things. For real volumes, split the output into many files. nanochat uses shards of about 250 million characters, roughly 100 MB compressed each, which lets training download data bit by bit.

Keep the source and licence columns. They cost little, and they are what lets you answer "what did you train this on?" when you publish.

How much text do you need for a model of a given size?

A rough way to compare is tokens per parameter, a parameter being one of the numbers the model learns. Published projects vary widely:

nanochat counts differently, using only the parameters it calls scaling parameters. Its documents put the most efficient point for its small models at about 10 to 12 tokens per such parameter, and its GPT-2-grade speedrun uses 8. In practice that speedrun needs about 150 shards, or some 37 billion characters and about 15 GB compressed. Very small models need far less: the TinyStories authors got fluent children's stories from under 10 million parameters.

So budget usually sets the amount, not a rule. More tokens cost more GPU time, as the cost breakdown shows. Start with what your budget can process, and keep at least a few million tokens of validation text, enough for the score to stop jumping around. Better data can also substitute for more of it: the DCLM authors report matching Llama 3 8B on average across their tasks with 6.6 times less compute by improving the dataset.