What does OpenAI's Codex paper reveal about AI that writes code?

Yesterday, 7 July 2021, a team of more than fifty OpenAI researchers, led by Mark Chen, posted a paper to arXiv titled "Evaluating Large Language Models Trained on Code." It is the first detailed public description of Codex, the family of models behind GitHub Copilot, the coding assistant that GitHub and OpenAI previewed late last month.

The paper does three things. It explains how Codex was trained. It introduces a new test, HumanEval, that checks whether AI-written programs actually work. And it spends many pages on what can go wrong, from buggy suggestions to biased code.

What Codex is and how it was trained

Codex starts as a GPT language model, the same kind of model as GPT-3, trained to predict the next word of text. OpenAI then fine-tuned it, meaning it continued training on a narrower body of material, in this case code.

OpenAI collected the training data from 54 million public GitHub repositories in May 2020. OpenAI kept unique Python files under 1 MB, which came to 179 GB, then filtered out files that looked auto-generated or had unusually long lines. The final dataset was 159 GB of Python. The models range in size up to 12 billion parameters, the internal numbers adjusted during training.

The paper describes early research versions. OpenAI says the models that power Copilot are separate production descendants of this work.

How HumanEval tests code

Most ways of grading AI-written text compare it with a reference answer word by word. That fails for code: two programs can look very different and both be correct, or look nearly identical while one is broken. OpenAI's answer is to judge functional correctness: run the code and see if it passes unit tests, small automatic checks that a function gives the right output for given inputs.

HumanEval has 164 problems written by hand for this paper, covering language understanding, simple algorithms and basic math, some at the level of easy job-interview questions. They were written fresh because GitHub already holds solutions to many published puzzles, and Codex has read much of GitHub. Each problem comes with an average of 7.7 tests. Because running untrusted code is risky, OpenAI executed every generated program in a locked-down sandbox that keeps it away from other systems.

Here is the format, shown with an illustrative problem written in the same style. The model sees the function name and the docstring, the description in triple quotes, and must write the body:

def count_long_words(sentence, n):
    """Return how many words in the sentence have more than n letters.
    >>> count_long_words("the quick brown fox", 3)
    2
    """

A correct answer splits the sentence into words and counts the ones longer than n. Hidden tests then try other sentences and values of n.

The results, explained

The paper reports results with a measure called pass@k: the chance that at least one of k attempts passes all the tests. Pass@1 means one try; pass@100 means the model gets 100 tries and you count a success if any of them works.

More tries help a lot. With 100 attempts per problem, Codex finds a working solution for 70.2% of problems, and Codex-S for 77.5%. Since real users rarely have hidden tests, the team also picked the single attempt the model rated most likely; that solved 44.5%.

The limits and risks OpenAI lists

The paper is frank about weaknesses. Codex is not sample efficient: it learned from hundreds of millions of lines of code, far more than any programmer sees in a career, yet the authors expect a strong student who has finished an introductory computer science course to solve more HumanEval problems than Codex-12B. It can produce code that does not run, or call functions and variables that do not exist. Its accuracy falls steeply as instructions get longer: in one test, each extra step chained into a docstring cut the pass rate by a factor of two to three. It also mixes up which operation applies to which variable when a description has several of each.

The broader-impacts section goes further:

• Over-reliance. Codex can suggest code that looks right but does not do what the user meant, a special risk for beginners. OpenAI says human oversight is required.
• Misalignment. When a prompt contains subtle bugs, Codex tends to write worse code than it is capable of, copying the style of what it sees. The authors found this gap grows with model size.
• Bias. Codex can be prompted to produce harmful comments, and code whose structure reflects stereotypes. VentureBeat highlights one example: given a function named for race, Codex assumed a small set of fixed categories.
• Security. It can suggest insecure settings and vulnerable code, so qualified people should review its output before running or trusting it.
• Jobs. The paper expects modest productivity gains for now, since programmers spend much of their day on tasks other than coding, and larger effects later.

Why it matters

HumanEval gives researchers a shared, public yardstick for code-writing models, released on GitHub so anyone can test their own systems. Grading by running tests, instead of comparing text, is likely to become a standard way to measure this kind of AI.

It also shows how much training on code changes things: a general GPT model solves almost none of these problems, while Codex solves over a quarter on the first try. And it puts on record, in OpenAI's own words, the problems that come with such a tool.

What does this mean for people using GitHub Copilot?

Copilot runs on a production version of Codex, not the exact models in the paper, but the lessons carry over. Treat its suggestions as a first draft from a fast, well-read assistant that does not understand your intentions. Read every suggestion, run your tests, and be especially careful with security-sensitive code such as encryption or authentication.

The paper's results also suggest how to get better help. Short, precise descriptions work far better than long chains of instructions, so break a big task into small functions with clear docstrings and examples. Keep your own code clean, too: Codex tends to follow the quality of the code around it.

OpenAI told VentureBeat that Copilot requests are rate-limited to prevent automated abuse, and that it is updating its safety tools as it prepares to offer Codex through its API.