# How do you train a small language model on a rented GPU, step by step?

Date: 2026-11-02 · Author: Robert Scales · https://guidetoartificialintelligence.com/en/learn/train-small-llm-rented-gpu/

**Answer:** Rent a machine by the minute, connect over SSH, clone Karpathy's nanochat, run a short test model for about $3, then run the full script for about $64 on eight H100 GPUs. Keep the job in a screen session, copy your checkpoints off the machine, and terminate the instance as soon as you finish, because billing continues until you do.

Our [tiny model guide](/en/learn/train-tiny-language-model-from-scratch/) stays on your laptop. This one moves to real hardware. You will rent a GPU server, train a model whose only ancestor is random numbers, and chat with it. It assumes you can open a terminal and copy commands, and nothing more.

The steps use nanochat, Andrej Karpathy's small training project, and Lambda's on-demand cloud as the example, since nanochat's author uses it. Prices were read on November 2, 2026; any provider with Ubuntu machines and SSH works in much the same way.

## Pick your run

| Run | Machine | Time | Bill | Result |
|---|---|---|---|---|
| Test, 12 layers | 1 H100 ($4.29/h) | About 40 min of training | About $3, plus setup time | Text that shows the method works |
| Full, 24 layers | 8 H100 ($31.92/h) | About 2 h | About $64 | A chat model at GPT-2 grade |
| Full, on one GPU | 1 H100 | About 8 times longer | Similar total, about $57 for training | The same model, overnight |

Always do the test first. The 40 minutes is an estimate from the project's note that a 12-layer run takes about 5 minutes on eight GPUs and that one GPU is about eight times slower. Your first run shows how long setup takes you, and every minute counts.

## Step 1: rent the machine

Create an account with the provider and add a payment method and an SSH key, which is the file that lets your computer log in without a password. In Lambda's console, launch an instance with the GPU count you chose. Billing begins once it passes its health checks. Copy the IP address it shows, then connect:

```bash
ssh ubuntu@YOUR_IP
nvidia-smi
```

`nvidia-smi` lists the GPUs and their memory. Check that you got what you are paying for. nanochat's defaults assume 80 GB per GPU.

## Step 2: install

```bash
git clone https://github.com/karpathy/nanochat.git
cd nanochat
curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env
uv sync --extra gpu
source .venv/bin/activate
```

`uv` is the tool that installs the Python packages the project needs, including PyTorch, the library doing the training. Before you run the `curl` line, you can read the script at that address if you like; it comes from the official uv project.

## Step 3: the test run

Start a **screen** session first. It keeps the job running on the server if your laptop sleeps or your connection drops.

```bash
screen -S test
python -m nanochat.dataset -n 32
python -m scripts.tok_train
python -m scripts.base_train --depth=12 --model-tag=d12 --save-every=1000
```

The first line downloads 32 data files of about 100 MB each. The second trains the tokenizer. The third trains a 12-layer model and saves a checkpoint, a snapshot of the model, every 1,000 steps. Press `Ctrl-a` then `d` to detach; `screen -r test` brings it back.

Watch two things in the log. The **validation bits per byte** should keep falling, and it is explained in [how to evaluate your model](/en/learn/evaluate-your-own-model/). The **text samples**, printed every 2,000 steps, should get less silly. If you run out of graphics memory, cut `--device-batch-size` in half, from 32 to 16 and so on, until it fits.

## Step 4: the full run

On an eight-GPU machine, the whole pipeline is one line:

```bash
screen -L -Logfile runs/speedrun.log -S speedrun bash runs/speedrun.sh
```

The script downloads the data, trains the tokenizer, pretrains a 24-layer model, evaluates it, fine-tunes it for conversation and evaluates it again. Inside, it uses `torchrun`, PyTorch's launcher, which starts one training process per GPU so that the eight share the work. To follow along from another login, run `tail -f runs/speedrun.log`.

When it finishes:

```bash
source .venv/bin/activate
python -m scripts.chat_cli
```

Try a story, a poem, and a question about you. Expect fluent nonsense on the last, since this model knows very little. Its author compares it to talking with a kindergartener.

## Step 5: save it, then stop paying

From your own computer, copy the results, then terminate:

```bash
scp -r ubuntu@YOUR_IP:~/.cache/nanochat/chatsft_checkpoints ./nanochat-model/
scp -r ubuntu@YOUR_IP:~/.cache/nanochat/tokenizer ./nanochat-model/
```

Pretraining checkpoints are in `base_checkpoints` next to those if you want them too. Open a checkpoint folder to confirm that the files arrived, and then terminate the instance in the console. Lambda charges nothing for data going out, though other providers may. Note that nanochat's default dataset is licensed for non-commercial use only, which matters if you plan to [publish the model](/en/learn/publish-licence-your-model/).

> **Try it:** Do the test run with `--depth=12`, note the final validation score, then repeat with `--depth=8`. Compare the two scores, the time and the cost. You will have measured how size buys quality.

## What goes wrong most often, and how do you avoid paying for it?

**The machine keeps billing after training ends.** Lambda bills by the minute from launch until termination, used or not, and other providers work similarly. Note your finish time, and set a phone alarm for it. A forgotten eight-GPU node costs about $766 a day. Check the provider's own rule about what "stop" means, because it may differ from "terminate".

**Storage outlives the machine.** A network filesystem keeps billing until you delete it, even with no instance attached. On Lambda you must attach it when you create the instance, or do without it and copy your results off before terminating.

**A crash loses hours of work.** By default nanochat saves only at the end of pretraining (`--save-every` is -1). For anything long, set a value, and use `--resume-from-step` to restart from the saved step. The full speedrun script does not set this, so add it to the script's training line if you want the insurance.

**A dropped connection kills the job.** Without `screen` or a similar tool, closing the terminal can end the run. Start the session before the job, every time.

**Out-of-memory errors.** Your GPU has less memory than the defaults expect. Lower `--device-batch-size`. nanochat compensates automatically so that training still behaves the same, only more slowly.

**Spot instances vanish.** They are the route to the low figure of about $15 that the project mentions, but providers can take them back. Use them only with regular checkpoints that you copy elsewhere, and treat the saving as uncertain.

**Mixed precision errors.** The speedrun's `--fp8` setting needs an H100 or newer GPU. On older cards, remove the flag and the run uses bf16, a higher-precision format.

## Sources

- nanochat README, speedrun script, CPU script and leaderboard notes: https://github.com/karpathy/nanochat
- Billing: https://docs.lambda.ai/public-cloud/billing/
- torchrun (Elastic Launch): https://docs.pytorch.org/docs/2.15/elastic/run.html

---

From the Guide to Artificial Intelligence, by Robert Scales, published by Holistic Hybrid Inc. Share and adapt it under CC BY-NC-SA 4.0 (https://creativecommons.org/licenses/by-nc-sa/4.0/): credit the guide, link to https://guidetoartificialintelligence.com/en/learn/train-small-llm-rented-gpu/, no commercial use without permission, and keep the same licence.