An AI summary of sixty customer interviews takes about forty seconds to produce and, if you are unlucky, about a day to stop believing.
The disillusion usually arrives by way of a quote. It is a wonderful quote. It sits under the first theme, saying exactly what the theme says, in the voice of a customer: “We nearly cancelled because onboarding took three weeks.” It goes onto a slide. Then somebody asks which customer said it, and a search of the transcripts turns up nobody. One customer said onboarding “dragged on”. Another, in a different conversation, mentioned that her team had “talked about cancelling” over a billing problem. The sentence on the slide was spoken by no one. A machine had made it out of two people.
After that, everything in the summary looks different. Are there really five themes? Did “most customers” ask for a Slack integration, or did three? The summary has not changed. It has merely stopped being furniture and become a claim.
Twenty minutes, nine hours
It would be a mistake to respond by throwing the thing away, because what it is good at, it is very good at.
In one study, researchers gave the same set of text messages to two human teams and to an AI model, and asked each to work out the main themes. The people took more than nine hours. The AI took twenty minutes, and found five of the seven themes the people found.
That is the case for the machine, and it is a strong one. Before this, the realistic alternative to an AI summary was not a careful human reading. It was no reading at all. Sixty interviews went into a folder and three of them were remembered. A fast, decent map of what customers are talking about is a real gain.
The trouble is the second finding in the same study, which is quoted rather less often. When the researchers looked at which message the AI had filed under which theme, it agreed with the people less than half the time. The headings were right. What had been sorted beneath them frequently was not.
That split, between the outline and the detail, turns up wherever anyone has looked. In another test, an AI model sorted questions into categories defined by experts. The two experts agreed with each other almost perfectly. The AI agreed with them well on one kind of category and badly on another, in the same exercise. Elsewhere it beat people at labelling news topics and was close to guessing on a few labels in a different set. This is how the claim that AI is “as accurate as humans” manages to be both true and useless. It is true on average. Nobody decides anything on the average. They decide on one theme, one figure, one quote, and there is no telling from the outside which of them is the weak one.
Errors that read well
A summary that failed noisily would be no danger. These fail the way a good liar does, fluently.
There is the quote, to begin with. Researchers who checked the quotations an AI model drew from focus groups found words altered inside them, and separate quotes fused into one. The themes were sound. It was the quotation marks that could not be trusted.
There is inflation. Two administrators struggle with single sign-on and the summary reports that “users find setup confusing”. A study of 4,900 AI-written summaries found them far more likely than human-written ones to claim more than their source did. Anyone hoping that newer models would grow out of it will be disappointed. Some were worse.
There is the dissenter who disappears. A summary preserves the pattern and discards the exception, and because nothing it says is false, the gap is invisible. Readers who checked AI summaries of books they knew well found that more than half had left out something important. Length compounds it. These models attend to the beginning and the end of a long text and neglect the middle, so that in a batch of eighty interviews the fortieth has the worst seat in the house. In customer research this matters more than it sounds. A single sentence from your largest account can outweigh a theme drawn from forty small ones.
And there is inconstancy. One researcher asked an AI model to sort the same 234 texts ten times over, wording the instruction slightly differently each time. The ten results agreed with one another poorly, well short of what researchers accept as reliable. Even the identical instruction did not always return the identical answer.
A caveat is owed here. None of these studies used the kind of material a product team works with. They used tweets, news reports, health interviews and novels, often with models that have since been replaced. They cannot say how often your summaries are wrong. They can say where to look.
Half an hour
Since the error rate cannot be known in advance, it has to be measured, once, and the measuring takes about thirty minutes. One rule governs the whole exercise: read the customer before you read the machine.
Take thirty conversations at random. Not the first thirty, and not the interesting ones. Add any that stand behind a quote or a number you mean to show to someone. For each, read what the customer said and write a line of your own. Only then look at the AI’s version. Do it the other way round and you will agree with it, as one does with a confident colleague.
Then test the three things a summary is worst at. Search for every quote in its transcript: it is there, word for word, or it is not. Take the boldest number in the report, perhaps “most customers mentioned pricing”, and count it by hand in your thirty. And read five conversations the report never mentions, looking for the person who disagreed.
If the AI’s summaries match yours in 27 or more of the 30, let them steer your reading, and check again when you change your questions or your tool. Fewer than 24 and the arrangement is not ready. If a single quote cannot be found, check them all before the report leaves the building. These thresholds are a rule of thumb, to be tightened as the stakes rise.
There are ways to get better summaries in the first place. Define the themes yourself, since a model sorts answers into your categories far better than it invents its own, as our guide to analyzing open-ended responses explains. Summarise one conversation at a time, so that none is lost in the middle. Ask where each quote came from. Ask for the views held by only one or two people. Above all, begin with better answers: a vague reply makes a vague summary, which is why a good follow-up question does more for accuracy than any ingenuity in the prompt.
The quote on the slide was not exactly a lie. Two real customers said two real things, and something that had read all sixty interviews, and remembered them loosely, pressed the two into a sentence better than either. That is a fair description of what an AI summary is. Believe its themes. Check its quotes, its counts and its exceptions. And keep the transcripts, which are the only reason any of this can be checked at all.
How Diaform helps
Diaform’s summaries are written by AI, so all of this applies to them, and we have published no accuracy tests of our own. What Diaform does is make the checking easy. Every conversation keeps its full transcript beside the summary, with word-for-word quotes you can search for, and AI reports show the quotes behind each theme and list the outliers. There is a 14-day free trial.
