Most of what customers tell companies is true. The difficulty is the rest, which is polite.
Anyone who has demonstrated their own work to a customer knows the sound of it. “Yeah, it’s great. Really clean.” The customer means it, more or less, in the way one means it about a friend’s cooking. Three weeks later the account is cancelled, and the exit form says they could never find anything in the new dashboard.
Nobody lied. A person who had plainly worked hard asked a question, and another person gave the kind answer. It is one of the more attractive human habits, and it makes customer research very difficult. Which raises a question that would have sounded eccentric ten years ago and is now a purchasing decision. If people soften the truth for a human being, would they do better with something that is not one?
Two kinds of fear
The softening has been studied for decades, and it turns out to grow from two fears that are easily confused.
One is the fear of being judged by whoever is listening. The other is the fear of what happens afterwards, when somebody else reads the answer. A review of the research found both at work, pushing people towards whatever sounds best. They need to be kept apart, because one of them has a cure and the other does not.
The first reaches further than embarrassment. Pew once put the same questions to 3,000 Americans, some by telephone and some alone on a website. Speaking to an interviewer, 62% described themselves as “very satisfied” with their family life. Typing alone, 44% did. Most questions hardly moved. The gap opened wherever one answer simply sounds nicer than the other, and satisfaction, the thing product teams ask about most, is that sort of question.
It is worse when the questioner made the thing. In a study in India, people were shown two versions of a piece of technology and asked which they preferred. They were about two and a half times more likely to choose the one they believed their interviewer had built, though the two were identical. That is the founder’s screen share, with a number attached.
Remove the listener
If the listener is the problem, the experiment suggests itself. One study carried it out with unusual rigour, because the researchers were able to check the answers.
They asked university graduates about their grades and then looked up the records. Of those who really had a D or an F on file, a third concealed it from a telephone interviewer. On a web form, with nobody listening, a fifth did.
A machine earns some of the same candour, on one condition: people have to believe it is a machine. In an experiment that deserves to be better known, everyone spoke to the same interviewer on a screen. Half were told it was fully automated. Half were told a person was operating it. Those who believed it was only a computer were less afraid to open up, and observers rated them as more forthcoming. The interviewer was identical throughout. The only thing that differed was who the participants imagined on the other side.
Ordinary survey questions show the same thing. Among 634 iPhone users interviewed either by a person or by an automated system, people admitted more unflattering behaviour to the automated one, and more in text than aloud.
They seem to know it themselves. After one AI-led interview, 43% of participants said they would choose the AI next time, against 19% who wanted a human. One explained: “I felt I could be honest without being judged.”
The form got there first
Here the story ought to end with the machine triumphant. It doesn’t, and the grades study is the reason.
The most candid ordinary instrument ever devised is a form on a screen. Nobody listens to a form. And even there, a fifth of the graduates still hid their bad mark. Taking away the listener shrinks the polite lie. It does not abolish it.
Nobody has yet shown, by checking answers against records, that an AI interviewer extracts more truth than a form does. The nearest thing to a test points the other way. In a laboratory experiment, chat agents with better conversational skills drew more polite, face-saving answers to sensitive questions than clumsier ones did. The more a machine resembles a person, the more people extend it the courtesies they extend to people, evasion included.
An AI interviewer therefore sits somewhere between the form and the human, and its design determines where. A plain one probably lands near the form. One with a first name, a warm manner and a habit of saying “Great!” is on its way back to the founder’s screen share.
Most of this evidence concerns students and paid panel members talking about grades, health and drinking. Nobody has tested whether a business customer criticises a product more truthfully to a machine. The direction is clear enough to act on. The size of the effect, among your own customers, is likely to be modest.
What the machine is for
Why bother, then? Because honesty was never where the form fell short. A form’s weakness is that it cannot ask what you meant.
In one test, about 600 gamers reviewed game trailers through either a chatbot or a conventional survey. The chatbot’s answers averaged 90 words to the survey’s 64. They were more relevant, and far fewer were gibberish.
The difference is the follow-up. A form accepts “too expensive” and moves on. An interviewer asks, “Compared with what?” and discovers that the customer is paying for twenty seats while six people log in. The first answer is a grumble about price. The second is a problem with a solution.
A longer answer is not a truer one. But a specific answer can be checked against what the customer did, and that remains the best defence against the gap between what people say and what they do.
There is a bill for all this. It is easier to walk out on a machine: in the iPhone study, 13% to 14% of people abandoned the automated interviews, against 3% to 7% with human ones. Persistent questioning wears on people, and one participant complained that the AI “felt like an interrogation”. And the machine does nothing whatever for the second fear. A customer who suspects that their account manager will read the transcript is not reassured to learn that software took the notes.
The practical conclusions are few, and none is technical. Say that it is an AI, since the candour depends on people believing it, a matter we take up in AI interviewer disclosure. Say who will read the answers, and if the account manager won’t, say that. Promise only the privacy you can deliver, and for the tenderest subjects, pricing or cancellation or complaints about your own staff, consider a link that carries no name. Keep the interviewer plain: no human name, no chat, no praise for agreeable answers, which only teaches people what you like to hear. Let people type as well as speak. And ask about what happened, not for a verdict. “How satisfied are you with onboarding?” solicits the nice answer. “What was the hardest part of getting started?” makes a complaint the ordinary thing to say. Our list of leading questions is a useful last check.
The founder’s mistake, back at the screen share, was not asking. It was asking in the arrangement most certain to produce kindness: the maker, in person, watching the customer’s face. An AI interviewer rearranges the room. It offers the candour of a form and the depth of a conversation, a combination that no form and few calls achieve. It will not get more truth than a form, and the more it pretends to be one of us, the less it gets. Its chief virtue is that nobody feels the need to spare its feelings.
How Diaform helps
Diaform runs the interview as a chat with an AI interviewer, so no person is listening while your customer answers. We have no research of our own on honesty, and the limits above apply to us. You choose how far it follows up on each question, respondents type or speak with no account needed, and you can share a plain link or, on Pro plans and up, personal invite links when you need to know who answered. There is a 14-day free trial.
