What is OpenAI Codex, and can it really write code?

Yesterday OpenAI released an improved version of Codex, its system for turning natural language into code, to developers through its API. It is also the engine behind GitHub Copilot, the AI pair programmer GitHub previewed about six weeks ago. Until now you could only meet it inside Copilot; with the API, developers can build their own tools on top of it. Access starts as a private beta with a waitlist, and OpenAI says the model is free to use during this first period.

Codex grew out of GPT-3. It keeps much of GPT-3's grasp of English, but its training data adds billions of lines of source code from public sources, including GitHub repositories. The result is a model that reads a request such as "make the background blue" and writes the instructions a computer needs to do it.

What Codex is good at

According to OpenAI, Codex is most capable in Python and is also proficient in more than a dozen other languages, among them JavaScript, Go, PHP, Ruby, Swift, TypeScript and even Shell, the command language of a terminal. It can hold 14KB of Python code in memory while it works, compared with 4KB for GPT-3, so it can take more than three times as much surrounding code into account.

OpenAI describes programming as two activities: breaking a problem into smaller pieces, and then connecting those pieces to code that already exists, such as libraries and functions. Codex is strongest at the second part, which is often the most tedious. OpenAI says it has also used Codex to translate code between languages, explain code and refactor (restructure) it. Its demos include building a simple space game and converting Python to Ruby.

Can it really write code? The numbers

A paper OpenAI posted in July, Evaluating Large Language Models Trained on Code, gives a careful answer. The team built a test set called HumanEval: programming problems, each described by a docstring (a short description at the top of a function), and each checked by running unit tests. Only code that actually works counts.

Model	HumanEval problems solved
GPT-3	0%
GPT-J (an open model)	11.4%
Codex, one try	28.8%
Codex, best of 100 tries	70.2%

Two lessons stand out. First, training on code makes a huge difference: GPT-3 solves none of these problems. Second, a single attempt from Codex fails more often than it succeeds, but generating many attempts and keeping one that passes the tests works surprisingly well. The paper also notes where Codex struggles, such as long chains of steps in one description and keeping track of which operation applies to which variable. The paper tested an earlier Codex; OpenAI says the API version is improved, but it has not published new scores for it.

Try it: Write a one-sentence description of a function you could test, such as "return True if a word reads the same backwards." Before using any AI, write three test cases for it. If you have Codex or Copilot access, compare the generated code against your tests.

A practical way to prompt Codex

Codex continues text, like GPT-3, so the best prompt looks like the start of a well-written program. Name the language, describe the goal in a comment, and give a clear function name:

Prompt

# Python 3
# Return the most common word in a text, ignoring case and punctuation.
# If two words tie, return the one that appears first.
def most_common_word(text):

Notice that the comment settles the tricky choices (case, punctuation, ties) up front. Ambiguity in the request becomes bugs in the code.

How should a beginner use AI to learn programming?

Use it as a tutor, not as a shortcut. The skill you are building is understanding, and a tool that hands you finished code can skip that step if you let it. A good routine is: try the problem yourself first, then ask the AI, then compare. Where the two versions differ, work out why. Ask it to explain a line you don't recognize, and check that explanation against the official documentation, because the model can be confidently wrong.

Testing is the habit that protects you. HumanEval exists because generated code that looks right often isn't. Write your tests before you ask for code, run every suggestion, and treat a failing test as a lesson rather than a nuisance. Small steps also help: ask for one function at a time, not a whole program, so you can follow each piece.

Finally, keep the parts that need judgment for yourself. Deciding what a program should do, splitting a problem into pieces and choosing what to test are the parts Codex is weakest at, and they are also what makes someone a programmer. With those in your hands, an AI that fills in the routine parts can make learning faster and less frustrating.