How do you summarize a long report with GPT-3?

A good summary of a long report saves hours, and GPT-3 can draft one in minutes. The catch is that no GPT-3 model can read a long report in one go: even text-davinci-002, which OpenAI lists as strong at summarizing for a particular audience, handles about 4,000 tokens per request, or roughly 3,000 words. So the work has two halves. First you build the summary from pieces; then you check that the pieces add up to the whole.

To make this concrete, we will use a public report many readers know by name: Language Models are Few-Shot Learners, the paper that introduced GPT-3. It runs to more than 70 pages, far beyond any single request.

The method: summarize the parts, then the whole

OpenAI's alignment team used this strategy last year to summarize entire novels. Their system splits a book into chunks, summarizes each one, and then summarizes the joined summaries, repeating until one summary remains. For a report, one round of combining is usually enough.

For the GPT-3 paper, the main body splits neatly at its own headings: Introduction, Approach, Results, Measuring and Preventing Memorization of Benchmarks, Limitations, Broader Impacts, Related Work and Conclusion. The Results section is long, so give each of its subsections a request of its own. That gives you about fifteen chunks, each comfortably inside the window with room left for the answer.

Step 1: one prompt for every section

Use the same instruction each time so the pieces line up:

You are summarizing one section of a research paper for a manager who is not a specialist.
Section: 5. Limitations

Text:
"""
[paste the section]
"""

Write 4 to 6 bullet points. Keep every number exactly as written. Do not add anything that is not in the text.

Bullets:

Two lines matter most. "Keep every number exactly as written" makes the later check much easier. "Do not add anything that is not in the text" reminds the model to work from the page rather than from what it may already have read about GPT-3 elsewhere.

Step 2: combine the section summaries

Paste the bullets from every section, in order and under their headings, into one final request:

Below are bullet-point summaries of each section of a research paper, in order.
Write a summary of the whole paper in 200 words for a non-specialist manager. Start with what was built, then what it can do, then its limits and risks. Use only the information in the bullets.

[section headings and bullets]

Summary:

The order instruction gives the summary a spine. Without it, combined summaries tend to read like a list of facts in the order they arrived; the OpenAI book study found the same weakness, noting that its summaries often read as a list of events rather than a connected account.

How do you check the summary didn't drop key points?

Build a checklist from the report itself before you read the summary, so the summary cannot shape what you look for. Two lists do most of the work: the headings and the numbers.

Then compare the numbers character by character. A summary that says "300 million tokens" instead of "300 billion" reads smoothly and is badly wrong.

Next, look for what is thinly spread. The book-summarizing researchers point out a real weakness of chunking: a theme that shows up a little in many sections may be too minor to survive in any single section summary, and so disappears from the final one. In a report, that is often a caveat raised in passing in more than one place; the GPT-3 paper, for instance, touches on the expense of running very large models under Limitations and again under Energy Usage. Ask yourself what the authors kept coming back to, and check that it is there.

Finally, test the summary against itself. The same study found that moving the chunk boundaries changed the summaries noticeably. If a point matters, run the process a second time with slightly different splits, or ask GPT-3 directly: "Here is a section and a summary of it. List any important point from the section that the summary leaves out." Treat its answer as a list of leads to check in the original, not as a final verdict.