# How can AI help people with disabilities? Date: 2024-05-28 · Author: Robert Scales · https://guidetoartificialintelligence.com/en/learn/ai-help-people-disabilities/ **Answer:** AI can describe photos and read text aloud for blind and low-vision people, and it can turn speech into live captions for people who are deaf or hard of hearing. The most useful tools let the person stay in charge, ask follow-up questions and switch to a human helper when the AI is unsure. For many disabled people, the promise of AI is practical rather than futuristic: reading the label on a can, knowing what a letter in the mailbox says, following a video call in a language you don't speak. Two kinds of AI make the biggest difference today. Vision models can look at a photo and describe it in words. Speech models can listen to audio and turn it into text on a screen. The examples below come from people and companies who build these tools with disabled users. They show what works well, and they also show where the tools still need a human check. ## Describing the world: Be My AI Be My Eyes is a free app, founded in 2015, that connects blind and low-vision people with sighted volunteers through a live video call. In March 2023 it announced an AI helper powered by OpenAI's GPT-4, first called Virtual Volunteer and now named Be My AI. You take a photo with your phone, and the AI describes it and answers questions about it. What makes it different from older image-to-text apps is the conversation. Be My Eyes gave the example of photographing the inside of a fridge: the AI can say what is there, then suggest what you could cook with those ingredients. If the AI cannot help, the app offers to connect you with a sighted volunteer instead. Be My Eyes tested the tool carefully before a wide release. According to OpenAI's system card for GPT-4 with vision, the pilot began with close to 200 blind and low-vision testers. By September 2023, 16,000 testers were requesting about 25,000 descriptions a day. ## Hearing in text: live captions For people who are deaf or hard of hearing, captions turn sound into something you can read. Microsoft's new Copilot+ PCs, which go on sale on June 18, add live translation to Windows Live Captions: any audio playing on the computer, from a video call to a recorded lecture, is shown as English captions in real time, translated from more than 40 languages. Microsoft says it runs on the laptop's own AI chip and works even offline, so the sound does not need to be sent anywhere to be transcribed. ## A worked example: reading a letter with Be My AI Here is how a blind user might handle an unknown envelope, step by step. The descriptions are illustrative. 1. **Take the photo.** Hold the phone above the letter in good light. Be My AI describes what it sees: a one-page letter from a city's water department with a table near the bottom. 2. **Ask a focused question.** "What is this letter asking me to do, and is there a deadline?" The AI answers that the city will replace water meters on the street and asks residents to book a visit before a date. 3. **Ask for the detail that matters.** "Read me the phone number and the date exactly, digit by digit." 4. **Check what carries a risk.** For anything involving money, health or a legal deadline, confirm the key details: call the number, or tap the option to reach a sighted volunteer and ask them to read the same part. The same pattern works in a general chatbot that accepts photos, such as ChatGPT: ```prompt I am blind. Describe this photo for me in this order: what kind of document it is, who sent it, what it asks me to do, and any dates, amounts or phone numbers. Read numbers digit by digit. If any part is blurry or cut off, tell me which part instead of guessing. ``` The last sentence matters most. It invites the model to say "I can't read that" rather than fill the gap. > **Try it:** Close your eyes, or ask a friend to hand you an object you haven't seen, and use a photo-description tool to identify it. Then ask two follow-up questions, such as its colour and any text on it, and check the answers when you look. ## What are its limits for accessibility? The first limit is accuracy. During the Be My AI pilot, testers reported that the model sometimes made basic errors while sounding completely sure, such as naming a menu item that was not on the menu. For that reason, Be My Eyes tells its users not to rely on the AI for safety and health tasks such as reading a prescription, checking a food label for allergens or deciding when to cross a street. It also says AI should never take the place of a white cane or a guide dog. Red-team testers in the same report found that the model could miss text, merge nearby words, or misjudge where objects are and what colour they are. The second limit is about people. Blind testers told Be My Eyes they wanted descriptions of faces and the people around them, the kind of information a sighted person gets just by looking. But face analysis raises privacy and bias concerns, so the model is designed not to identify people by name. Be My Eyes and OpenAI are working on ways to describe people's visible features without naming them. The third limit is fit. No single tool serves every disability, and the people who live with a disability are the best judges of what helps. The tools that work best are the ones built with their users, as the Be My AI beta was, and that leave the final decision with the person using them. A good accessibility tool offers choice: AI when it is quick and good enough, and a human when the stakes or the doubts are high. ## Sources - Introducing Be My Eyes Virtual Volunteer, powered by OpenAI's GPT-4: https://www.bemyeyes.com/blog/introducing-be-my-eyes-virtual-volunteer - GPT-4V(ision) System Card: https://cdn.openai.com/papers/GPTV_System_Card.pdf - Introducing Copilot+ PCs: https://blogs.microsoft.com/blog/2024/05/20/introducing-copilot-pcs/