Asked how much more they would pay for toothpaste with fluoride, a group of shoppers who do not exist arrived at a figure of $3.40. A study with real people had found $3.27.
On results like that one, an industry has been founded. Its products go by several names, synthetic users, synthetic respondents, silicon samples, and they all work the same way. An AI model is handed a profile and asked to answer as that person would. Describe your customer, say an operations manager at a 200-person logistics company who still does the scheduling in spreadsheets, and you may interview a hundred of her before the kettle has boiled. The answers are articulate and specific. There are quotes for the slide. Eighty-two of the hundred would welcome your new feature.
To anyone with a decision due on Monday and no customers booked, the appeal needs no explaining. It looks like research and it reads like research. Whether it is research has now been tested by people in a position to know.
The vanishing minority
The idea was never absurd, and the toothpaste is not its only success. In another study an AI model was asked to predict the outcomes of 70 survey experiments run in the United States, and did about as well as human forecasters, though it tended to expect larger effects than appeared.
But look at what the successes share. They concern averages. They concern Americans. And they concern matters the internet has discussed at exhausting length, such as toothpaste and politics. A scheduling tool for mid-sized logistics firms is none of these, and an average is seldom what a product team is after. You want to know who is stuck, who is about to leave, and why.
The most revealing test came from the Pew Research Center, which is unusually well placed to run one. Pew keeps a large panel of real people whose answers to hundreds of questions it already knows. Its researchers asked an AI model to stand in for individual panel members, one at a time, on nearly 300 questions, and compared the results.
The AI caught the general drift of many questions. What it lost was the disagreement. On nearly half of the questions, at least one answer option was never chosen by any AI respondent. On one, 11% of real people picked an answer that not a single synthetic respondent selected. A whole constituency had been deleted, and nothing in the output showed that anyone was missing.
The machine was also much too sure of itself. Real people said “not sure” about four times as often as their stand-ins. And asked to speak for particular groups, it reached for a caricature. The AI respondents had 97% of Hispanic adults at least somewhat likely to follow the World Cup. Among the real panel members the figure was 43%. Other researchers, testing four models against 3,200 people, traced the habit to the text the models learn from. Much of what is written about any group is written by people outside it, and so the model acquires the outsider’s view.
Each of these failures lands somewhere tender in product work. The customers who hit a blocking problem during onboarding are usually a minority, and minorities are what gets flattened. A respondent who is never unsure makes every question look settled. And a comparison of segments built on stereotype will confirm whatever you already supposed about your segments.
Too good a customer
There is a quieter flaw, and it is the one likeliest to mislead a team in a hurry.
Researchers at Nielsen Norman Group ran synthetic interviews beside real ones on the subject of online courses. One real learner admitted to finishing three of the seven courses he had started. The synthetic user reported finishing all of them. Synthetic users, the researchers concluded, “seem to care about everything”.
Real customers do not. They forget things, skip steps, lose interest, and fail to care about the feature you have spent a quarter building. A good deal of what a real interview teaches comes from that indifference. A respondent who is conscientious, attentive and interested in everything will make almost any idea look good. Those 82 enthusiastic operations managers were always going to be enthusiastic.
Were a synthetic study at least consistent, you could learn its biases and allow for them. It is not. One team ran the identical prompt three times, in April, June and July. By July the results had shifted substantially, a change the authors connect to an update of the model. Another, working across 43 models, found the answers strongly shaped by the order and labelling of the options. The same hundred “customers”, asked again next month or shown the choices in a different order, may tell you something else.
What it has never read
Those are faults of degree. The last is a fault of kind, and for product research it settles the matter.
A model knows what was in the text it learned from, and nothing else. Pew’s test caught this neatly. A quarter of real people said they had heard a lot about a topic recently in the news. Among the AI respondents it was 3%. The news had happened after the model stopped reading.
Market researchers met the same wall from the other side. Models could estimate fairly well what shoppers would pay for features that already existed. For new ones, they had to be trained on results from real people first.
Your onboarding flow is in no model’s reading. Nor is your pricing page, or the reason a customer cancelled last Tuesday. Ask a synthetic user about them and it will answer all the same, fluently and in detail, because fluent answers are what it produces. It is guessing what a person like that might say about a product like yours.
The most serious attempt to get round this is instructive. Researchers interviewed 1,052 people for two hours each and built a separate AI agent from each transcript. On survey questions, those agents matched their person’s answers 83% as well as the person matched themselves two weeks later. It is an impressive result, and it shows what the best synthetic person requires: a long interview with the real one. That is a way of reusing research you have done. It is no way of skipping it.
Almost none of this evidence comes from product research itself. It comes from opinion polls, laboratory experiments and consumer goods, mostly in the United States, and newer models may do better. But the Pew test used a recent one, and it still missed by 12 percentage points on average.
Rehearsal
None of which makes synthetic users useless. It makes them an instrument of preparation, and one rule separates the good uses from the bad. If a wrong answer would cost you an afternoon, go ahead. If it would change what you build, what you charge or what you tell your board, you need people.
On the safe side of that line there is plenty to do. An AI persona can read your survey and point out a confusing question. It can supply a first list of objections, tell you whether an interview guide runs too long, or let a nervous colleague rehearse. It can help you find your feet in a field you do not know. What it cannot do is tell you how many customers have a problem, discover a problem you have never heard of, size the demand for something new, or rank what matters to people. For those the answer remains customer discovery interviews and concept testing with real prospects.
Whenever synthetic output appears in a document, label it. That one habit stops a guess from becoming a finding three slides later.
And if a colleague wishes to rely on a synthetic study, there is a simple way to find out whether they can. Write down what it concluded. Then put the same questions, in the same words, to fifteen or thirty real customers from the segment the personas were playing; our guide to how many interviews are enough explains why that number will do. Do not merely compare the most popular answer. Look at the spread: the answers real people chose that the AI never did, how many said “not sure” or “this doesn’t matter to me”, and what they raised that the synthetic interviews never mentioned. That last list is the price of having skipped the real thing.
So is it research? No. A hundred synthetic interviews are one well-read colleague’s guess, repeated a hundred times in slightly different words. The colleague has read a very great deal about logistics companies and has never met one of your customers. A guess of that kind is worth having before you write your questions. It is not something to put in front of a board.
One distinction deserves to be kept clear, because the two ideas are constantly confused. A synthetic user is AI playing the customer. An AI interviewer is AI asking the questions while real customers answer. Only the second can tell you something about your product that nobody has yet written down.
How Diaform helps
Diaform is on the AI interviewer side of that line. It asks your questions one at a time in a chat with real customers and follows up on vague answers, and each conversation gets a summary with word-for-word quotes, plus the full transcript. It does not generate respondents and it will not make research instant: you still need real people to answer. If you drafted your questions with synthetic users, start a 14-day free trial and put them to the real ones.
