How does prompt caching cut API costs?

Most AI applications send the same text over and over. A customer-help bot sends its full instructions and a page of policies with every single question. A tool that answers questions about a contract sends the whole contract each time. You pay for those input tokens (the small chunks of text that APIs count and bill) on every call, even though nothing in them has changed.

Prompt caching fixes that waste. The provider keeps the processed version of the beginning of your prompt for a few minutes. When your next request starts with exactly the same text, it reuses that work instead of doing it again, and charges you less for those tokens. Responses also tend to come back faster. The answers themselves are not affected: caching changes the bill and the speed, not what the model writes.

Two providers, two approaches

Anthropic opened prompt caching on its API as a public beta on August 14, 2024, covering two models: Claude 3.5 Sonnet and the smaller Claude 3 Haiku. It is opt-in: you mark where the reusable part of your prompt ends with a cache_control setting. The first call that stores this part (a "cache write") costs 25% more than normal input. Every later call that reuses it (a "cache read") costs only 10% of the normal input price. The cache lasts about five minutes, and each hit resets that timer.

OpenAI followed on October 1, 2024. Its version is automatic and needs no code change. On recent versions of GPT-4o and GPT-4o mini, as well as the o1-preview and o1-mini reasoning models, any prompt longer than 1,024 tokens can be cached, and repeated opening tokens are billed at half price. Caches usually clear after 5 to 10 minutes without use and always within an hour.

A worked cost example

Picture a help desk bot for a small software company. Its system prompt (the fixed instructions that set the model's role and rules) runs to 3,000 tokens: tone guidelines, product facts and a set of example answers. Each customer question adds about 200 tokens. On a busy day the bot answers 1,000 questions, close enough together that the cache stays warm. The figures below are illustrative and count input only, since output prices do not change.

Claude 3.5 Sonnet without caching: 1,000 calls × 3,200 tokens = 3.2 million tokens at $3.00 = $9.60.

Claude 3.5 Sonnet with caching:

• One cache write: 3,000 tokens at $3.75 per million = about $0.01
• 999 cache reads: 2,997,000 tokens at $0.30 per million = about $0.90
• The changing questions: 200,000 tokens at $3.00 per million = $0.60
• Total: about $1.51, roughly 84% less.

GPT-4o with automatic caching: OpenAI caches in steps of 128 tokens, so about 2,944 of the 3,000 fixed tokens count as cached on each repeat. That gives about 2.94 million cached tokens at $1.25 (about $3.68) and 0.26 million full-price tokens at $2.50 (about $0.65), for roughly $4.32 instead of $8.00.

The pattern holds in general: the longer and more stable the fixed part, and the more often it repeats within a few minutes, the bigger the saving. A short prompt under 1,024 tokens gets no discount at all.

Here is the Claude version in Python, using the anthropic library's beta helper that sends the required beta header for you. The help desk text is a placeholder.

import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from the environment
HELP_DESK_RULES = open("help_desk_rules.txt").read()  # the 3,000-token fixed part

response = client.beta.prompt_caching.messages.create(
    model="claude-3-5-sonnet-20240620",
    max_tokens=500,
    system=[
        {
            "type": "text",
            "text": HELP_DESK_RULES,
            "cache_control": {"type": "ephemeral"},  # cache everything up to here
        }
    ],
    messages=[{"role": "user", "content": "How do I reset my password?"}],
)
print(response.usage)

The usage printout tells you what happened. On the first call, cache_creation_input_tokens shows the tokens written to the cache. On the next call within five minutes, cache_read_input_tokens shows the tokens served from it. With OpenAI, look for cached_tokens inside prompt_tokens_details in the usage field.

How do you structure prompts to get cache hits?

Put what never changes first and what always changes last. Both providers cache from the very start of the prompt, so a cache hit needs the opening to match exactly, character for character. A good order is: tool definitions, then the system prompt, then long reference documents and examples, then the conversation, and finally the new question. Anthropic's documentation describes the same order of tools, system, then messages.

Keep anything that varies out of the fixed part. A common mistake is starting the system prompt with today's date, the user's name or a request number. That one changing line at the top breaks the match for everything after it. Move such details to the end, next to the user's question.

Send related requests close together. A cache that sits unused for more than about five minutes is gone, and the next call pays full price (or, with Claude, the slightly higher write price). Steady traffic keeps it warm; a request every hour will not benefit.

Finally, check rather than assume. Watch the cache fields in the usage data for a day. If the read counts stay at zero, something in your opening text is changing between calls, and comparing two raw requests side by side usually reveals it.