What is OpenAI's Assistants API, and how do builders use it?

Yesterday, at its first developer conference, DevDay, in San Francisco, OpenAI released the Assistants API in beta. It is a set of building blocks for adding an AI helper to an app: something that follows a role you define, remembers the conversation, reads the documents you give it, and can run code or trigger actions in your software.

OpenAI describes it as its first step toward helping developers create "agent-like" experiences, meaning software that can work toward a goal and use tools, not only reply to one message at a time. It is available to all developers starting today.

What problem it solves

Until now, a developer who wanted a helpful assistant inside an app had to assemble most of it by hand. A language model on its own is stateless: each request starts from zero. So the developer had to save every message, send the right part of the history back with each new request, trim it when it grew too long for the model's context window (the amount of text it can consider at once), split documents into pieces, store them in a searchable form, and write code to decide which pieces to show the model.

The Assistants API takes over much of that work. In OpenAI's words, it handles "a lot of the heavy lifting" so builders can focus on what their app does.

The building blocks

OpenAI defines an assistant as a purpose-built AI with specific instructions that can draw on extra knowledge and call models and tools. Four parts make that work:

• Instructions. A standing description of the assistant's role, tone and limits, written once rather than pasted into every request.
• Threads. A thread is a stored conversation. You add each new message to it, and OpenAI keeps track of the history and manages what fits in the model's window. OpenAI says threads can be persistent and, in its words, "infinitely long."
• Tools. Three come built in. Code Interpreter writes and runs Python in a sandbox, a sealed-off space where code cannot touch anything else, so the assistant can do math, analyze data files and draw charts. Retrieval lets the assistant search documents you or your users upload, without you having to build the search yourself. Function calling lets the assistant ask your app to run a function you have described, such as checking stock levels, and then use the result in its reply.
• Models. An assistant runs on one of OpenAI's models, such as the new GPT-4 Turbo. Usage is billed at that model's normal per-token rates, with tool pricing listed on OpenAI's pricing page.

According to OpenAI, assistants share their foundation with GPTs, the custom versions of ChatGPT it also announced yesterday. The difference is who builds them and where they live: a GPT is made inside ChatGPT without code, while an assistant is built by a developer and lives inside the developer's own product.

A worked example: a homework helper for a math website

Imagine a small education site that wants a tutor for its algebra pages. Here is how a builder might set it up with the Assistants API:

1. Create the assistant. Give it instructions, for example:

``prompt You are a patient algebra tutor for students aged 13 to 16. Never give the final answer straight away. Ask one guiding question at a time, check the student's reasoning, and use the course notes when explaining a rule. ``

1. Attach knowledge. Upload the site's course notes and turn on Retrieval, so the tutor explains rules the way the course does.
2. Turn on Code Interpreter. When a student asks for a graph of a line or wants a calculation checked, the assistant can compute it and return a chart.
3. Describe one function. For example, save_progress(student_id, topic), so the tutor can ask the site to record that a student finished a lesson. The site's own code does the saving.
4. Start a thread for each student. Each time the student writes, add the message to their thread and ask the assistant to respond. The history stays with OpenAI, so the next session picks up where the last ended.

The builder never writes code to split the notes into chunks, search them or trim old messages. That is the plumbing the API now handles.

Why it matters for builders

The biggest change is time. A prototype that once took days of setup, storage and search code can now start with instructions, a file and a few settings. Small teams and solo developers gain access to tools, like a code sandbox, that were hard to build and run safely.

It also changes where data lives. Conversations and files are stored with OpenAI rather than in your own database. OpenAI states that data and files sent through its API are not used to train its models and that developers can delete them when they choose. Whether that arrangement suits a given project, especially one handling sensitive information, is a decision each team will need to make.

Finally, OpenAI says it plans to let developers plug in their own tools alongside the three built-in ones. If that arrives, assistants could connect to far more services than they can today.

What should builders watch out for while the Assistants API is in beta?

Beta means unfinished. OpenAI has asked developers to share feedback and says it will keep building over the coming weeks, so names, behaviors and limits may change. It is wise to keep your own code loosely tied to the API's details, for example behind a small layer of your own functions, so updates are easier to absorb.

Keep an eye on cost. Because the API manages long threads and pulls in document passages for you, you see less of what is sent to the model on each turn, and every token is billed at the model's rate. Test with realistic conversations and check your usage dashboard before opening an assistant to many users.

Finally, test the answers, not just the setup. Retrieval helps an assistant stay close to your documents, but it can still misread them or fill gaps with guesses. A short list of questions with known correct answers, run every time you change instructions or files, is a simple way to catch problems before your users do.