How do you ask ChatGPT about a photo?

Since late September, ChatGPT Plus and Enterprise users have been getting the ability to show ChatGPT pictures: a photo from the phone, a screenshot, a scanned page. The model that reads them is GPT-4 with vision, which OpenAI calls GPT-4V in its technical documents. You can add several images to one message and, in the mobile app, draw on a photo to point at the part you care about.

Getting a good answer works much like a good text prompt. The picture supplies the facts; your words supply the goal. A photo with "What do you think?" gets a general description. A photo with a clear question, some context and a request to show its reading gets something you can use and check.

The basic steps

1. In the app or on the website, tap the photo button (on mobile, the plus sign first) and take or choose a picture. You can add more than one.
2. If only part of the image matters, use the drawing tool in the mobile app to circle it.
3. Write your question in the same message: what you want, for whom, in what form.
4. Ask it to list what it sees first. This one habit turns a hidden reading into something you can verify.

Three worked examples

A fridge, for dinner ideas. OpenAI's own release notes mention this use. The key is asking for the inventory before the recipes:

[Photo of the inside of a fridge]
First, list every food you can identify in this photo, and mark anything you're unsure about with a question mark. Then suggest two vegetarian dinners for two people that use mostly those items and take under 40 minutes. I also have rice, pasta, onions and basic spices.

If the list says "yogurt?" where you see sour cream, correct it in one line and the recipes adjust. Without the list, you might not notice the mix-up until you are cooking.

A chart, for work. Screenshots of graphs are another use OpenAI names. Charts invite confident misreading, so ask for the raw reading separately from the interpretation:

[Screenshot of a bar chart of monthly website visits]
1. Read the value of each bar as precisely as you can and put them in a table. Say if any value is an estimate.
2. Then describe the trend in two sentences.
3. Tell me what this chart cannot show on its own.

Compare the table with the chart's labels before you trust the trend. If your chart has a data table behind it, paste the numbers instead; text is more reliable than pixels.

A sign in French. For a visitor, or a student of French, a photo of a parking or trail sign is a practical test:

[Photo of a municipal parking sign in French]
Transcribe the French text exactly as written, including accents and times. Then translate it into English and explain in plain words when I'm allowed to park here.

Asking for the exact transcription first lets you compare it with the sign letter by letter. If it got "lundi au vendredi" right but misread an hour, you will catch it before you get a ticket.

What not to show it

Think before you upload. Crop out faces, addresses, account numbers and documents you would not hand to a stranger. OpenAI has also limited what ChatGPT will say about people in images: GPT-4V is designed to refuse to identify who someone is or to guess sensitive traits such as age or race from a photo.

What can it misread?

More than its fluent answers suggest. OpenAI's system card for GPT-4V, the safety report it published with the launch, lists the problems its testers found, and they are worth knowing before you rely on a reading:

• Text and symbols. It can miss letters or characters, overlook mathematical symbols, and merge two separate labels that sit close together into one invented term. In a chart or diagram, that can change the meaning.
• Positions and colors. Testers found it can fail to work out where things are in an image and how colors map to values, which is exactly what a chart legend relies on.
• Confident errors. It can hallucinate, meaning describe things that are not there, in an authoritative tone. One blind tester of the Be My Eyes app, which used GPT-4V early, said it "very confidently told me there was an item on a menu that was in fact not there."
• Other languages. OpenAI says it reads English text well but does poorly with some other languages, especially those not written in the Roman alphabet.

Some uses are off the table for now. Be My Eyes tells its users not to rely on the tool for safety or health tasks, such as reading a prescription, scanning an ingredient list for allergens, or deciding when to cross a street, and OpenAI states that GPT-4V is not fit for any medical function and is unreliable for identifying dangerous substances or poisonous foods, toxic mushrooms among them. For anything where a misreading could hurt someone, use the photo to get oriented, then confirm with the original document, the label or a qualified person.