Should you still write prompts by hand?

Writing prompts by hand used to be the only option. Now there are three. You can write the prompt yourself; you can ask a model to write or rewrite it, as tools like Anthropic's prompt improver do; or you can use an optimizer, a program that tries many prompt variations against examples and keeps the one that scores best. Research has backed the machine approaches for a while: the 2022 Automatic Prompt Engineer paper found that instructions picked this way matched or beat human-written ones on 19 of 24 language tasks.

That does not make hand-writing obsolete. It changes when it is worth the effort. The clearest way to see the trade-off is to do one task all three ways and score the results.

The task: sorting customer emails

A small online shop wants each incoming email labeled as one of four categories: order status, return or refund, product question or other. They have 60 past emails that a staff member has already labeled correctly. They use 20 to develop prompts and keep 40 aside as a test set: examples the prompts never see while being written, used only to score them.

Way 1: write it by hand

You sort customer emails for an online shop.
Reply with exactly one label: order_status, return_refund,
product_question, or other.
- order_status: where is my order, delivery dates, tracking.
- return_refund: sending something back, refunds, exchanges, damaged items.
- product_question: sizes, materials, compatibility, stock, before buying.
- other: anything else, including complaints about the website.
If an email fits two labels, choose the one the customer needs answered first.

Email:
{email}

Writing it takes ten minutes, and the tie-breaking rule in the last line comes from knowing the business: a damaged item is a refund problem even if the customer also asks when the replacement ships.

Way 2: let a model write or improve it

Paste the hand-written prompt into a prompt improver, or ask any capable chat model: "Improve this prompt for accuracy; keep the four labels unchanged." Anthropic says its improver adds a step-by-step reasoning section, puts examples in a consistent format and tightens the structure, and that in its own tests an improved prompt raised accuracy by 30% on a classification task. You then review the rewrite. Remove anything that changes the labels or adds rules you do not want.

Way 3: run an optimizer

DSPy, an open-source framework from Stanford, lets you describe the task and a scoring rule, then compiles a prompt automatically. Following its documentation:

from typing import Literal
import dspy

dspy.configure(lm=dspy.LM("openai/gpt-4o-mini"))

class SortEmail(dspy.Signature):
    """Label a customer email for an online shop."""
    email: str = dspy.InputField()
    label: Literal["order_status", "return_refund",
                   "product_question", "other"] = dspy.OutputField()

def exact_label(example, prediction, trace=None):
    return example.label == prediction.label

# dev_emails: your 20 development examples as (text, label) pairs
trainset = [dspy.Example(email=e, label=l).with_inputs("email")
            for e, l in dev_emails]

optimizer = dspy.MIPROv2(metric=exact_label, auto="light", num_threads=8)
sorter = optimizer.compile(dspy.Predict(SortEmail), trainset=trainset)

The MIPROv2 optimizer proposes new instructions and picks examples to include, keeping whatever scores best on your metric. DSPy's documentation says it works best with tens to hundreds of examples, so 20 is the low end. The docs estimate a typical light run at about two dollars and twenty minutes.

Scoring all three

Run each version on the 40 held-out emails and count correct labels. An illustrative result:

The numbers are made up to show the method, but the pattern is typical: machine help gives a modest gain on a well-defined task, and a single wrong label in the test set can flip the ranking. The test set is what makes any of this meaningful. Without it, you are comparing prompts by how convincing they sound.

Should you still write prompts by hand?

Usually, you write the first version, and machines help with the next ones. The hand-written draft is where your knowledge of the task goes in: the categories, the exceptions, the rule for ties. Improvers and optimizers are good at wording and structure, but they cannot invent business rules they were never told. Anthropic presents its improver in the same spirit: you give feedback on what is and isn't working, and it revises again.

Which prompts are worth crafting yourself?

Write by hand any one-off request, because describing what you want is faster than setting up tests. Also write by hand any prompt where the rules carry judgment, such as how to respond to an upset customer or what counts as confidential, because those rules come from you, and you need to be able to read and explain them later.

Prompts that run many times on similar inputs are the best candidates for machine help: sorting, extraction, summaries in a fixed format, the instructions inside an agent. There, a few percentage points of accuracy add up, and you can build the test set that tells you whether the change was real. Keep your hand-written version as the baseline, and accept a machine-written prompt only when it beats that baseline on examples it has never seen.