Show one group of people the statement “The onboarding was easy” and another group “The onboarding was difficult”, and ask each whether they agree. If everybody were reporting their experience, the share agreeing with the first would match the share disagreeing with the second. It never quite does. A slice of each group simply agrees with whatever has been put in front of it.
Researchers have a name for the habit, acquiescence, and after decades of experiments of this kind they have a size for it too. One review puts it at “averaging about 10%”.
It is not a large number. It is quite large enough to explain a certain kind of surprise. A team asks new customers to respond to “The onboarding was easy”, and 84% choose “Agree” or “Strongly agree”. The figure goes into the monthly update with a green arrow beside it. A few weeks later somebody sits in on calls with accounts that have gone quiet and hears that setup took days, that two customers gave up on the data import and typed everything in by hand. Several of them had filled in the survey. They had agreed that onboarding was easy.
The convenience of it
It is tempting to blame the customers. The fault lies in the shape of the question, which happens to be the commonest shape in all of survey writing.
Its popularity is no mystery. You write a statement, any statement, and beneath it you put the same row of options, from “Strongly disagree” to “Strongly agree”. One row fits every subject. Twenty statements stack into a tidy grid, and onboarding, pricing, support and reliability are dealt with on a single screen.
All of that convenience belongs to the person writing the survey. The person answering gets the difficulties, of which acquiescence is only the first. It is not dishonesty. Agreeing is the polite thing to do and also the least trouble, which is why it grows when a question is hard and when people are tired. Expect more empty agreement on the eighteenth statement than on the second, one of the ways in which survey fatigue does its damage.
The second difficulty afflicts everybody, including people who would never agree merely to be agreeable. To answer “Setup was easy”, a customer must first decide how easy setup was and then convert that judgment into a quantity of agreement. The conversion can go wrong without anyone noticing. Take “I can usually find the feature I need.” One customer disagrees because she always finds it, and “usually” undersells her. Another disagrees because he rarely finds anything. They have had opposite experiences and given the same answer.
The third is the grid. When every row offers the same options, ticking one column all the way down becomes the line of least resistance. The evidence that the statement format alone causes this is thin. But it is the format that makes a twenty-row grid possible in the first place.
Two repairs
Survey writers have known about yes-saying for decades, and two repairs have become standard. Neither does what was hoped.
The first is to word half the statements negatively, so that anybody who agrees with everything cancels themselves out. This conceals the bias in the totals without removing it from the answers. The yes-sayers land in the middle of the scale, where most of them do not belong. Negative statements are harder to read, besides: to disagree with “The dashboard is rarely clear” is to perform a double negative, and people trip. Then somebody must remember to reverse the scores afterwards. When two researchers examined 27 sets of results from the System Usability Scale, a widely used questionnaire that alternates positive and negative statements, three had been scored wrongly.
The second repair is a longer scale, on the theory that seven or eleven points must be more precise than five. For agreement scales the reverse proved true. In tests across European countries, five points produced better data than seven or eleven, though the five-point versions were also the only ones with a label on every point, which may have helped.
Just ask
The alternative is almost embarrassingly simple. If you want to know how easy setup was, ask how easy setup was. Name the quality in the question and again in the answers: “How easy or difficult was it to get set up?”, with options running from “Very difficult” to “Very easy”.
The strongest test came from four experiments built into a large European survey, in which the same topics were asked both ways. The direct questions gave more valid answers in every one of the four, often by a wide margin.
The wider picture is less dramatic, and it would be misleading to hide that. When reviewers gathered all twenty experiments that had compared the two formats, not one showed the agreement format to be more valid. Some favoured direct questions and many found no clear difference. The test nearest to product work found nothing at all: two hundred employees rated an insurance website in one format or the other, and the scores differed by half a percent.
There is a likely reason. Most of the research put questions about politics, trust and health to the general public, subjects on which many people have no settled view, and yes-saying is strongest among those who know little about the topic. Your weekly users are the opposite. It grows, too, when an interviewer is present, which in most product surveys nobody is.
So a change of format may not transform a survey of engaged customers. What the evidence does say is that direct questions are sometimes much better and almost never worse. A known risk removed for nothing is a good bargain.
The rewrite
The rewriting follows the same three steps each time. Work out which quality the statement was really about: ease, frequency, speed, trust, value. If you cannot name one, the statement was too vague to keep. Turn it into a question, mentioning both ends where the quality has an opposite, as in “easy or difficult”. Then write five answers that use the same word, with a label on every one.
“The product meets my needs” becomes “How well does the product meet your needs?”, from “Not at all” to “Extremely well”. “The price is fair” becomes “How would you describe the price for what you get?”, from “Much too high” to “A very good deal”. The more interesting cases are those in which the adjective turns into an event. “The product is reliable” becomes “In the past month, how often did something not work as expected?” “Support resolved my issue quickly” becomes “How long did it take to get your issue resolved?”, answered against what the customer expected. Wherever you can ask about something that happened, do. The same habit keeps you clear of leading questions.
A few details matter in the answers. Run every scale in the same direction, low to high, so that nobody has to read twice. Make the steps even. And give people who have not used a feature a way of saying so, or they will pile on to the middle option and blur the result.
One consequence should be expected, and welcomed. Since each question now has its own answers, the long grid stops working, and you are obliged to keep the five or six questions you would really act on.
There is one case in which rewriting does more harm than good. Some questionnaires are standards. The System Usability Scale is the best known, and some versions of the Customer Effort Score are another. Their value lies in benchmarks built up over years of asking the question in exactly those words. Alter the wording and your score no longer compares with anything. The same is true of a long trend of your own: if you have asked one statement every quarter for three years, a better question breaks the line. Should you switch, run both versions side by side once, to see how the old numbers map on to the new.
The deepest weakness of the agreement grid, though, was never the bias. It is that the grid must guess in advance what might have gone wrong. Speed, clarity, price, support: each gets its row, and each customer dutifully rates them all, though most do not apply. A direct rating followed by one open question does the work better. “How easy or difficult was setup?” and then “What made it difficult?” The customer tells you which of your guesses mattered, in their own words, and occasionally names something you never thought to list. An open question, moreover, contains no statement to agree with.
You give up a tidy average for every attribute. If you report eight of them to a board each quarter, keep a short direct rating for each. If what you need is to know what to fix, the follow-up will tell you more, and our guides to follow-up questions and analyzing open-ended responses cover how to ask it and what to do with the answers.
That 84% was not exactly wrong. It measured two things at once: what customers thought of setup, and how willing they were to say yes to a pleasant statement from a company they had just joined. The team read it as the first alone. Ask the question directly and the number will be smaller than 84. It will also be about onboarding.
How Diaform helps
Diaform asks your questions one at a time in a chat, so there is no grid to tick down. You write each direct question as a single choice with your own labelled options, and when an answer is vague the AI interviewer follows up, as many as one, three or five times per question. Every conversation gets a summary per question with word-for-word quotes, so the reason sits next to the rating. Diaform will not fix your wording for you. Start a 14-day free trial and try your rewritten questions.
