How do you fine-tune an open-weight model with LoRA?

Fine-tuning means continuing to train a model on your own examples so it picks up a habit: a format, a tone, a type of decision. Retraining every weight of a modern model takes serious hardware, which is why most people now use LoRA, short for Low-Rank Adaptation. Introduced by Microsoft researchers in 2021, it freezes the original model and trains a pair of small extra matrices beside some of its layers. Only those small matrices change. For GPT-3, the authors reported about 10,000 times fewer trainable parameters than full fine-tuning and about a third of the graphics memory, with comparable quality.

The result is an adapter: a file of a few megabytes that sits on top of the base model. You can keep several adapters for one model, swap them, or merge one into the model for faster use. Because the base stays untouched, a bad run costs you nothing but time.

When LoRA is the right tool

LoRA teaches behavior, not facts. It is good at "always answer in this structure", "classify these tickets our way" or "write like our help desk". It is a poor way to make a model remember your price list or policies, which change; for that, give the model the documents at question time instead. A useful test: if a careful prompt with three examples already gets you 90 percent of the way, fine-tuning may only add cost. If the model keeps drifting from a format you need thousands of times a day, an adapter can help, and it lets a small, cheap model do a job that otherwise needs a large one.

A worked run, step by step

The task: turn messy meeting notes into exactly three lines, Decision:, Owner: and Deadline:. The base model is Qwen2.5-0.5B-Instruct, a small open-weight model that runs on a laptop and that Hugging Face uses in its own training examples. Two libraries do the work: PEFT (Parameter-Efficient Fine-Tuning) creates the adapter, and TRL (Transformer Reinforcement Learning) runs supervised fine-tuning, or SFT, which means learning from input-output pairs.

1. Install.

pip install "trl[peft]"

2. Write the data. Put about 300 examples in train.jsonl and 40 more in test.jsonl, one per line, in TRL's conversational prompt-completion format:

{"prompt": [{"role": "user", "content": "Notes: we agreed to move the launch, Sam checks the venue, need answer before Friday 12th"}], "completion": [{"role": "assistant", "content": "Decision: Move the launch date\nOwner: Sam\nDeadline: Friday 12th"}]}

With this format, TRL computes the training loss on the completion only, so the model learns to write the answer rather than to repeat your notes. Write the examples by hand or check every one: the adapter copies your mistakes as faithfully as your good habits.

3. Train.

from datasets import load_dataset
from peft import LoraConfig
from trl import SFTConfig, SFTTrainer

data = load_dataset("json", data_files={"train": "train.jsonl", "test": "test.jsonl"})

peft_config = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, task_type="CAUSAL_LM")

args = SFTConfig(
    output_dir="notes-lora",
    learning_rate=2.0e-4,
    num_train_epochs=3,
    per_device_train_batch_size=4,
    logging_steps=10,
)

trainer = SFTTrainer(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    args=args,
    train_dataset=data["train"],
    eval_dataset=data["test"],
    peft_config=peft_config,
)
trainer.train()
trainer.save_model("notes-lora")

Three settings matter. r is the rank, the width of the small matrices: higher means more capacity and a larger file. lora_alpha scales how strongly the adapter pushes on the model. The learning rate of 2.0e-4 is about ten times what TRL's documentation suggests for full fine-tuning, because so few parameters are being trained. save_model writes only the adapter, not a copy of the model.

How do you know it learned the right thing?

Test it on examples it never saw, and compare it with the model you started from. The training loss going down only tells you the model fits your training file; it does not tell you it will behave on Monday's notes. That is why 40 examples stayed in test.jsonl. Here is a small script that runs the base model and the adapted model on the same test prompts and checks the format:

import json
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

name = "Qwen/Qwen2.5-0.5B-Instruct"
tok = AutoTokenizer.from_pretrained(name)
base = AutoModelForCausalLM.from_pretrained(name)
tuned = PeftModel.from_pretrained(AutoModelForCausalLM.from_pretrained(name), "notes-lora")

def ask(model, messages):
    text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    inputs = tok(text, return_tensors="pt")
    out = model.generate(**inputs, max_new_tokens=80, do_sample=False)
    return tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)

def ok(answer):
    lines = [l for l in answer.strip().splitlines() if l.strip()]
    return [l.split(":")[0] for l in lines] == ["Decision", "Owner", "Deadline"]

tests = [json.loads(line) for line in open("test.jsonl")]
for label, model in [("base", base), ("tuned", tuned)]:
    print(label, sum(ok(ask(model, t["prompt"])) for t in tests), "/", len(tests))

A format check is only half the test. Read at least 20 answers side by side with the expected ones and mark each as right or wrong on content: is the owner the right person, is the deadline the one in the notes? An illustrative outcome might be the base model passing the format check 9 times out of 40 and the adapter 38 times, with three content errors you would then trace back to similar mistakes in your training data.

Also check what you did not mean to change. Ask the adapted model a few ordinary questions outside the task. If it now answers everything in three lines, it learned too narrow a lesson; fewer epochs, a smaller learning rate or a few general examples in the mix usually fix that. Keep the test file and the scores. The next time you add data or change a setting, run the same test, and you will know whether the new adapter is actually better.