What is few-shot prompting?

The term comes from the GPT-3 paper OpenAI researchers published in May. They describe three ways to give the model a task without any extra training. Zero-shot gives only an instruction. One-shot adds a single demonstration. Few-shot adds several. The "shot" is simply one worked example, a pair of input and correct output.

Few-shot prompting has quickly become the everyday working method for people in the API beta. OpenAI's own launch post describes the API as something you program by showing it a few examples of what you want. Even the default question-and-answer prompt that OpenAI suggests is a few-shot prompt: a short list of questions, each followed by a model answer, then a new question left open. Developer Kevin Lacker used exactly that starter prompt in his widely shared July tests of GPT-3.

One task, three prompts

Imagine a small online shop that wants to sort incoming customer emails into four queues: Billing, Shipping, Technical and Other. Here is the same job written three ways.

Zero-shot: instruction only.

Sort the customer email into one category: Billing, Shipping, Technical or Other.

Email: I was charged twice for order 4471. Can you refund one of the payments?
Category:

This often works for an easy case like a double charge, but the model has to guess the answer format. It might write "Billing", "billing issue" or a full sentence explaining its reasoning.

One-shot: instruction plus one example.

Sort the customer email into one category: Billing, Shipping, Technical or Other.

Email: My parcel says delivered but it is not at my door.
Category: Shipping

Email: I was charged twice for order 4471. Can you refund one of the payments?
Category:

One example settles the format: a single word after "Category:". But the model has seen only a Shipping case, so it knows little about where the edges between categories fall.

Few-shot: instruction plus several examples.

Sort the customer email into one category: Billing, Shipping, Technical or Other.

Email: My parcel says delivered but it is not at my door.
Category: Shipping

Email: The app logs me out every time I open the cart page.
Category: Technical

Email: Can I get an invoice with my company name on it?
Category: Billing

Email: Do you have a physical store in Ottawa?
Category: Other

Email: The tracking number you sent me doesn't work on the courier's site.
Category: Shipping

Email: I was charged twice for order 4471. Can you refund one of the payments?
Category:

Now the model has one or more examples for every queue, including a tricky one: a broken tracking number sounds technical but belongs in Shipping. That example teaches a boundary the instruction alone never states.

What makes a good set of examples

The basics of a clear prompt, such as consistent labels and an open slot for the answer, apply here too. Few-shot prompting adds a few questions of its own:

• Does every possible answer appear? If Billing never shows up in your examples, the model is less likely to choose it.
• Do the examples look like real inputs? Use the length, tone and messiness of the emails you actually receive, not tidy textbook sentences.
• Are the hard cases covered? Spend examples on the borderline calls, like the tracking number above, rather than on obvious ones.
• Are the answers in the examples correct? The model copies whatever pattern you show it, mistakes included.

It is worth knowing what few-shot prompting is not. It is not training. The GPT-3 paper is careful on this point: no weights are updated. The model reads your examples fresh on every request and forgets them afterward. OpenAI says the API also allows training on your own data for better results, but that is a separate step.

How many examples should you put in a prompt?

The GPT-3 paper gives a useful frame. For its few-shot tests, the authors usually used between 10 and 100 examples, and the reason was practical: that is how many fit in GPT-3's window of 2,048 tokens, the pieces of text the model reads. They also report that more examples are usually, but not always, better, so they tried several numbers and kept the best. On one demanding benchmark of reading and reasoning tasks called SuperGLUE, they report that fewer than eight examples per task were enough for GPT-3 to beat a fine-tuned BERT-Large model.

For everyday work, three points follow. First, cover the space before adding volume: one example per category plus your hardest cases often matters more than a long list of easy ones. Second, budget your tokens. A realistic customer email of three or four sentences, with its label, takes around 50 tokens, so twenty such examples use about half the window, leaving less room for the email you want sorted and for the answer. Third, test instead of guessing. Keep a small set of real emails with known answers out of the prompt, run them through each version, and add examples only where the model keeps getting it wrong.

The paper also shows that larger models make better use of examples than smaller ones. If you work with one of the smaller GPT-3 models, expect to need more examples, or clearer ones, for the same accuracy.