Will AI that writes code replace programmers?

Two announcements in under three weeks have put this question on many developers' minds. On June 29, GitHub opened a limited technical preview of Copilot, a tool that proposes lines and whole functions inside the Visual Studio Code editor. A week later, OpenAI posted a research paper describing Codex, the model family behind it. Watching a comment turn into a working function is striking, and it is natural to wonder what happens to people who write code for a living.

A fair answer starts with the measurements rather than the demos. They describe a tool that is often useful and often wrong, which is a very different thing from a replacement. To see why, it helps to look at what was tested and at what a programmer's working day actually contains.

What the test results really say

The Codex paper introduces a benchmark called HumanEval. Each problem is a short Python function described only by its docstring, the plain-language note at the top of a function that says what it should do. The model writes the body, and hidden unit tests (small automatic checks) decide whether the code works. Only code that runs and passes counts.

Two things stand out. Training on public code makes an enormous difference: GPT-3, a general text model, solves none of these problems. And a single answer from Codex fails more often than it succeeds. The 70.2% figure needs a hundred samples plus some way to pick the one that works, which in practice means tests that a person has written.

GitHub reports a similar picture from its own benchmark. On Python functions whose bodies had been blanked out, Copilot's first suggestion was right 43% of the time, rising to 57% when it was allowed ten tries. Its FAQ adds that Copilot does not run or test what it proposes, and that it may reach for outdated library calls.

These are also small problems, a few lines each. The paper reports that Codex has trouble when a docstring describes a long chain of operations, and when it must keep track of which operation applies to which variable. Most real software is made of exactly those long chains.

What programmers actually do

Typing new lines is only one part of the job. A working programmer also:

• works out with other people what a program should do, often before anyone is sure;
• splits a big problem into pieces and decides how those pieces fit together;
• reads existing code, which may be years old and written by someone else;
• tracks down bugs, including ones that only appear with real data;
• reviews colleagues' changes and keeps the system secure and maintainable.

Copilot sees only the file open in the editor, and long files are cut down to the few hundred lines before the cursor. It knows nothing about the meeting where the requirements changed, the customer who reported the bug or the database three services away. GitHub frames the tool around a developer who stays in charge and stays responsible for the code, a design that matches the numbers above.

A worked example: sorting tasks by exposure

A practical way to think about the question is to go task by task and ask two things: how much of this could a model like Codex draft today, and how easy is it for a person to check the result?

This table is a reasoned judgment, not a measurement. Its point is that "programming" is not a single task, and that a model's strengths line up with the parts that are already the most routine.

Which programming tasks change first?

The first changes are likely in routine work that is quick to verify. Boilerplate, small helper functions and the unit tests that go with them all resemble the public code Codex learned from, and a person can confirm the result in seconds. GitHub's product page highlights these same uses: turning a comment into code, filling in repetitive patterns, proposing tests and showing alternative solutions to compare.

Working in an unfamiliar language or library is another early candidate. Looking up the right function name and its arguments takes real time, and even an imperfect suggestion can shorten that search. The catch is that a newcomer to a language is also the person least able to spot a subtle mistake, so the time saved on typing has to be spent partly on checking.

Work that depends on context outside the file, or on deciding what should be built in the first place, is likely to change slowly. If tools like Copilot spread, many expect the balance of the job to tilt toward reading, reviewing and testing code rather than typing it. The paper's authors also raise open questions about security and the economics of code generation, and those deserve close attention. For now, the skills that protect a programmer are the same ones that make the tool safe to use: understanding the problem, writing good tests and reading code with a critical eye.