How can ChatGPT now see, hear and speak, and what can you do with it?

Yesterday OpenAI announced that ChatGPT can "see, hear and speak." Over the next two weeks, paying subscribers will be able to talk to it out loud and hear it answer, and to show it pictures, from a photo of a broken appliance to a screenshot of a chart, and ask questions about them.

Until now, ChatGPT has mostly been a typing experience. These additions make it multimodal, a word for AI systems that handle more than one kind of input, here text, speech and images. OpenAI presents it as a more natural way to work with the assistant: say what you mean, or simply show it.

What OpenAI announced, and who gets it

The rollout covers ChatGPT Plus and Enterprise users, starting September 25 and spread over two weeks.

OpenAI says it plans to extend these abilities beyond subscribers, developers among them, shortly after.

How voice works

A spoken conversation chains three steps together:

1. Speech to text. Your words are transcribed by Whisper, OpenAI's open-source speech recognition system, already used in the ChatGPT iPhone app since May.
2. Thinking. ChatGPT reads the transcription and writes a reply, as it does with typed messages.
3. Text to speech. A new text-to-speech model reads the answer aloud.

That last part is the novelty. OpenAI says the model can produce human-like audio from text plus just a few seconds of sample speech. It worked with professional voice actors to create five voices, named Juniper, Sky, Cove, Ember and Breeze, and you pick the one you like.

OpenAI's suggestions are everyday ones: talking while you are on the go, asking for a bedtime story, or settling a debate at the dinner table.

How image input works

You can upload one or more images and ask about them. On the phone, a drawing tool lets you circle the part you care about. Behind the scenes, OpenAI says, image understanding is powered by multimodal versions of GPT-3.5 and GPT-4, which apply their language skills to photographs, screenshots and documents mixing text and pictures.

OpenAI has not explained exactly how its image-reading model, which Ars Technica refers to as GPT-4V, works inside. Ars notes that such systems typically convert images and text into a shared internal representation so one network can reason about both, while stressing that this is general research knowledge, not a description from OpenAI.

A worked example, based on scenarios OpenAI describes:

[Photo of an open refrigerator and a pantry shelf]
Here's what I have at home tonight. Suggest two dinners I can make in under 30 minutes, using mostly these ingredients. Then give me step-by-step instructions for the first one.

A good answer names the ingredients it recognized, so you can spot anything it misread, proposes two meals and lists clear steps. Other examples from OpenAI include talking about a landmark you photograph while traveling, figuring out why a grill will not start, analyzing a work graph and helping a child with a math problem by snapping and circling it. Its demo video shows someone getting help raising a bicycle seat from photos of the bike, the manual and a toolbox.

The risks OpenAI is naming

OpenAI is unusually specific about what could go wrong.

Voice cloning. A model that can imitate a voice from a few seconds of speech could help someone impersonate a public figure or commit fraud. That is why OpenAI limits it to one use, voice chat, with voices made from actors it worked with directly. It also mentions one partner: Spotify is piloting a feature that translates podcasts into other languages in the podcaster's own voice.

Mistakes about images. The model can hallucinate, meaning state things that are not there, including about people. OpenAI warns against relying on its reading of images in high-stakes fields and says the model reads English text well but does poorly with some other languages, especially those not written in the Roman alphabet.

Privacy. OpenAI says it has taken technical steps that greatly restrict how much ChatGPT will analyze, or comment directly on, the people who appear in images. Its approach was shaped by work with Be My Eyes, a free app for blind and low-vision people, whose users said it is valuable to discuss images that happen to include people in the background. Before launch, outside red teamers tested the model for risks in areas such as extremism and scientific knowledge, alongside a varied group of early testers.

Why it matters

Voice makes ChatGPT usable when your hands are busy or typing is hard, and it brings the assistant closer to a conversation partner. Image input removes one of the biggest frictions of all: describing in words something that is easier to show. For people with low vision, the Be My Eyes work hints at real help with daily tasks.

The rollout is gradual on purpose. OpenAI says releasing tools step by step lets it improve safeguards over time, which matters more with voice and vision than with text alone.

What should you keep in mind before showing ChatGPT a photo?

Think about what is in the frame. Crop out faces, addresses, account numbers and anything confidential before uploading, and use the drawing tool to focus attention on the part that matters.

Treat the answer as a helpful first reading, not a verdict. Ask the assistant to list what it sees before drawing conclusions, so you can catch misreadings early. For anything medical, legal, financial or safety-related, use it to prepare questions for a qualified person, not to replace them.

Finally, remember its known weak spots: text in languages that do not use the Roman alphabet, small print, and judgments about people. Within those limits, a quick photo and a clear question can save a great deal of typing.