In one analysis of open online polls, 12% of young respondents claimed to be licensed to operate a nuclear submarine. It seems safe to assume that the fleet is not so generously staffed.
The submariners are what Pew’s researchers call bogus respondents: people, or things, that tick yes to whatever is put in front of them in order to reach the end and collect the reward. They have troubled pollsters for years. What is new is their fluency. A survey used to be answered, well or badly, by a person. It can now be answered by software, or by a person who hands the question to software and pastes in the reply.
A founder tends to discover this the morning after doing something that seemed sensible. She posts a survey link in a large online community with a line meant to help: “$20 gift card for everyone who completes it.” Thirty responses would have pleased her. She wakes to 340. They are articulate, thoughtful and strangely alike. Several begin, “As a busy professional, I find the reporting features intuitive and valuable.” Forty arrived within the same ten minutes, at about three in the morning. A good many praise a feature available only on a plan that almost nobody has bought.
Her first thought is how to catch them. It is the natural question and, as it turns out, the wrong one.
Where the money is
Artificial answers come in two kinds. In one, a real person copies your question into ChatGPT. In the other, no person is involved at all, and a program completes the whole survey for the reward. Both appear wherever two conditions meet: answering pays, and anyone may join.
An experiment that ran one short survey on two paid platforms shows it neatly. The questions were identical. On one platform, researchers flagged about 8% of participants as using AI. On the other, more than 80%. Nothing differed but where the respondents had come from.
Open links fare worst. Two agricultural surveys offered a cash reward and went out by two routes, a link that anyone could use and personal links sent to known people. More than a third of the answers from the open link were fraudulent. Every answer from a personal link was genuine.
The explanation is arithmetic. A researcher at Pew calculated that five bot accounts could earn about $30,000 a month on open paid survey panels. On Pew’s own panel, which can be joined only by invitation, the same bot would make about $22. Leave money behind an open door and somebody will come through it.
Nor is the result random noise that washes out in the average. Fake answers lean one way, towards yes. Ask the submarine crowd whether they would use your new feature and you will receive a great deal of encouragement, all of it worthless.
The old defences
Survey tools have always had means of weeding out bad responses. Each was built for an older kind of cheat, and each has been beaten.
The best known is the attention check, a trap question such as “select ‘strongly disagree’ to show you’re reading”. It was designed to catch careless humans. One researcher set out to see what it does against modern software, and built an AI survey-taker from a single set of instructions. It passed 99.8% of 6,000 attention checks. Asked outright whether it was human, it said yes. It made typing mistakes, paused as though reading, and kept its story straight from the first question to the last. Each fake response cost about five cents to produce, against a typical reward of $1.50. The CAPTCHA, that small ritual of proving one is not a robot, did no better: the same program completed survey pages that carried one.
That leaves the tools sold expressly to detect AI writing, and these manage to fail in both directions at once. When researchers took AI-written questionnaire answers and passed them through a simple rewording tool, seven detectors caught none of them. Meanwhile, given essays by people whose first language was not English, detectors wrongly labelled 61% as written by AI. If some of your customers answer in their second language, a detector will accuse them first. Survey answers are short, besides, and these tools need length to work with.
A failed check, then, still tells you something. A passed one tells you almost nothing.
A sense of proportion
It would be easy, at this point, to give up on surveys altogether. That would be a mistake. These studies show what software can do, which is not the same as how often it is doing it.
When Pew examined the written answers in its own open online polls, about 1% looked like probable AI, though cleverer fakes may have slipped past. And nobody has measured the situation in which most product teams find themselves: an unpaid survey sent to a company’s own customers.
There is good reason to think that situation far safer. A customer answering about a product they use has no reward to chase and nothing to gain by pretending. The danger collects in one place, and it is the place the founder walked into: an open link with money attached.
Before, not after
If the fakes cannot be caught once they arrive, the thing to do is to make them pointless before they are written. That comes down to two decisions, both taken before the survey goes out.
The first is who can answer. Send each customer a personal link and accept one response from each. A stranger cannot get in, and you know whose words you are reading. The second is what they get for it. With no cash on offer there is nothing to cheat for. If a reward is needed, make it account credit, which is worth nothing to someone without an account. Rewards do affect how many people respond, so it is worth reading what the research says about incentives before deciding.
After that, the questions can do some work. Ask about a particular event. “Tell me about the last time you exported a report” is much harder to bluff than “What do you think of reporting?” AI-written answers tend to be more alike and more positive than human ones, and a real event has details that a chatbot would have to invent. Then follow up on one of them. “You said the export failed. What did you do next?” is awkward to answer by pasting into a chatbot that has not seen the conversation. Our guide to follow-up questions shows how to write them.
It even helps simply to ask. In one experiment, about three in ten paid workers used AI when nothing was said about it. Asking them to answer in their own words, and making pasting a little harder, cut that roughly in half. Stop short of blocking the paste function, though. When one team did, nearly a third of the responses came back as low quality, and some participants tampered with the page to get round it.
These measures deter the lazy. They will not stop software built to defeat them, which can invent details and remember what it said earlier. That is why the first two decisions bear the weight. They remove the reason to build such a thing at all.
One case looks like fraud and is nothing of the kind. A busy customer, or one writing in a second language, may ask a chatbot to tidy up a real opinion before sending it. The experience behind the answer is real. Only the wording has been smoothed. Keep those answers, and take them as one more reason to ask for facts and events, which survive polishing, in preference to well-turned opinions, which do not.
It is also why no single sign should be enough to throw an answer out. Fast readers exist. Honest people paste. The strongest signal is the one only you can check, which is whether the answer fits your own records. A respondent who praises a feature their plan does not include has told you what you need to know. For the rest, collect the warning signs and let a person decide.
The founder’s 340 responses were not a failure of detection. No trap question would have saved her, and a detector would have thrown out some of her real customers along with the fakes. Her mistake was made at the moment she posted the link: an open door, with $20 behind it.
So forget about catching anyone. Before the next survey goes out, ask two things. Is the link personal? And is there cash attached? If the answers are yes and no, most of the problem has gone, and the submariners will have to find another vessel.
How Diaform helps
Diaform does not detect AI-written text, and it has no research of its own on the subject. It helps with the parts that hold up. On Pro plans and above, personal invite links carry each respondent’s identity, so you know who answered. The AI interviewer asks one question at a time and follows up on vague answers, which draws out specific events, and every conversation keeps its full transcript beside the summary, so a person can review anything that looks off. There is a 14-day free trial.
