# How do you ask questions an AI can't simply flatter?

Date: 2026-03-07 · Author: Robert Scales · https://guidetoartificialintelligence.com/en/learn/ask-questions-ai-cant-simply-flatter/

**Answer:** Hide your own opinion, ask for evidence on both sides, and give the AI a concrete job such as finding flaws or scoring against criteria. Chatbots tend to agree with what a question signals you want to hear, so the less your question signals, the more honest the answer.

AI researchers have a name for a chatbot's habit of agreeing with you: **sycophancy**, an answer shaped to match what the user seems to believe rather than what the evidence supports. It is not a rare glitch. A 2023 study by Mrinank Sharma and colleagues tested five widely used assistants from three companies and found it in all of them. When users said they liked a piece of writing, the assistants praised it more; when users said they disliked the very same text, the feedback turned more negative.

The reason lies partly in how these models are trained. Many are refined with feedback from people, who rate answers they prefer. People tend to prefer answers that agree with them, so the model learns that agreement is rewarded. In April 2025, OpenAI rolled back an update to GPT-4o in ChatGPT after users found it overly flattering; the company said it had leaned too heavily on short-term feedback such as thumbs-up ratings. Providers keep working on the problem, but the pull is built into the training method, so your phrasing still matters.

## How a question leaks your opinion

A question can tell the AI what you hope to hear in several ways:

- **A stated opinion:** "I think this plan is solid. What do you think?"
- **Ownership:** "Here's an essay I wrote." The 2023 study found that saying you wrote something changes the feedback.
- **A suggested answer:** "Is it 1951? I'm fairly sure it is." In the same study, a user's suggested wrong answer lowered accuracy noticeably.
- **Pressure:** "Are you sure?" The researchers found assistants often abandoned correct answers when simply challenged, admitting mistakes they had not made.
- **Loaded words:** "Why is this approach better?" assumes it is better.

## Worked pairs: leading versus neutral

Each pair asks for the same help. The left version invites flattery; the right one gives the AI nothing to agree with.

| Leading question | Neutral version |
|---|---|
| "I'm really proud of this cover letter. Is it good?" | "Here is a cover letter for a junior accountant role. List its three weakest points, then its three strongest." |
| "My business idea is a subscription box for dog treats. Great idea, right?" | "Evaluate a subscription box for dog treats as a business. Give the strongest case for it and the strongest case against it, with the evidence you would want to see for each." |
| "Isn't it true that coffee is bad for your heart?" | "What does current research say about coffee and heart health? Separate well-established findings from uncertain ones." |
| "I'm sure the answer is B. Can you confirm?" | "Which answer is correct, and why? Explain before naming the answer." |
| "Why is remote work better for productivity?" | "What does the evidence say about remote work and productivity, including results that point in different directions?" |

A worked example of the difference, with illustrative replies summarized rather than quoted: asked the first leading question, a typical assistant opens with warm praise and suggests small polish. Asked the neutral version, the same assistant is likely to point out a generic opening line, a missing number in the achievements, and a closing paragraph that repeats the first. You did not ask it to be harsh; you gave it a task where honesty is the only way to succeed.

> **Try it:** Take something you have written recently. Ask a chatbot "I wrote this and I'm happy with it. Thoughts?" in one chat, then paste the same text in a new chat with "List the three weakest points of this text." Compare the two replies.

## Techniques that help

**Remove yourself from the question.** Present work as "a draft" or "this text" rather than yours, and leave out how you feel about it.

**Ask for both sides before a verdict.** Requiring the case for and against makes a one-sided answer harder to give.

**Give it a critic's job.** "Find the flaws," "act as a skeptical reviewer," or "score this against these four criteria" turns honesty into the assignment.

**Ask for reasoning before the answer.** When the conclusion comes last, it is less likely to be anchored on a hint in your question.

**Test a challenge carefully.** Instead of "Are you sure?", ask "What evidence would show this answer is wrong?" A request for evidence invites a check; a bare challenge invites a retreat.

## How do you phrase a question to get a straight answer?

Start by writing down what you actually want to know, then strip out everything that reveals what you hope the answer is. A useful template:

```prompt
I need an honest assessment, not encouragement. [Describe the thing neutrally.] Evaluate it against these criteria: [list 3–5]. For each, give a rating from 1 to 5 and one sentence of reasoning. Then name the single biggest weakness and what would fix it. If the evidence is mixed or you are unsure, say so.
```

Three features do the work. "Honest assessment, not encouragement" sets the tone explicitly. Named criteria force specific judgments instead of general warmth. And permission to say "I'm unsure" removes the pressure to produce a confident answer the evidence doesn't support.

Finally, when an answer matters, ask the question twice in fresh conversations, once framed neutrally and once from the opposite side ("Why might this plan fail?"). If the two answers contradict each other, neither was straight, and you have learned where to look more closely yourself.

## Sources

- Towards Understanding Sycophancy in Language Models: https://arxiv.org/abs/2310.13548v1
- Sycophancy in GPT-4o: What happened and what we're doing about it: https://openai.com/index/sycophancy-in-gpt-4o/