Fifty-five per cent of the people who answered one American telephone survey said they had done volunteer work in the past year. The true figure, according to a large government survey used as the benchmark, was 27%.
The explanation is almost funny once you see it. Answering a pollster’s questions is itself a small act of helping a stranger, and people who give up their time for others are the sort who do that. The survey was full of helpers. When it asked about helping, it got a wildly flattering reply. On most other subjects the same respondents were perfectly representative.
Hold that in mind during the next meeting about a customer survey. The survey went to 4,000 customers and 400 answered, of whom 70% said they were satisfied. Somebody worries that a tenth is too few to trust. Somebody else points out that polls call elections on a thousand interviews. They argue, politely, about whether ten per cent is enough, and neither raises the matter that would settle it, which is not how many people answered. It is which ones.
The wrong gauge
The response rate is the first thing anyone checks on a survey, and for most of the last century researchers treated it as the measure of quality. Then rates began to fall everywhere, as people stopped answering their telephones, and the profession was obliged to find out whether the old assumption was true.
Pew Research ran one of the clearest tests, and it is the survey with the volunteers. Pew conducted the same poll twice: once in the ordinary way, and once with weeks of additional effort to reach people who were hard to get. The effort raised the response rate from 9% to 22%. Across forty questions, no answer moved by more than four points.
A wider review arrived at the same place by another road. It gathered thirty studies in which survey answers could be checked against real records and found the response rate to be a poor guide to how far off a result was. Most of the variation in accuracy lay between different questions inside one survey, not between surveys with high and low rates.
So the worrier is watching the wrong gauge. That does not make the optimist right. A later look at the same data found that a higher rate does help a little, on average. And all of this research concerns household and political surveys with rates between roughly 9% and 60%. A customer email answered by 3% of its recipients lies outside anything that was tested. The studies show that the rate is a poor gauge. They do not show that a poorly answered survey is accurate.
Who stays silent
What the volunteers demonstrate is the mechanism. Silence does damage when the reason for it is connected with the thing being asked. The technical name is nonresponse bias, and it fits on one line.
Take the survey from the meeting. Ninety per cent of customers stayed silent, and 70% of those who answered were satisfied. If the silent majority would have come in at 68%, the result is out by nine tenths of two points, which is less than two. If they would have come in at 50%, it is out by eighteen. The response rate is identical in both cases. It cannot tell them apart, because the one number that matters, what the silent people think, is precisely the number the survey failed to collect.
A political poll can be lucky here, since the reasons people ignore pollsters have mostly little to do with how they vote. Customer surveys seldom enjoy that luck. A satisfaction survey asks about the relationship, and it is the relationship that decides whether the email is opened. The customer who has drifted away, who no longer logs in and deletes your messages unread, is the one whose opinion you most need and are least likely to get. A survey about cancelling has it worse, since former customers owe you nothing. Even a harmless question about how often people use a feature attracts the heavy users, who are keenest to talk. The risk falls only when the reason for silence is unrelated to the subject. A busy person may skip a survey about which integrations to build, but being busy says little about which tools they use.
To see where pure self-selection ends, look at online reviews, where nobody is invited and everybody chooses whether to speak. An analysis of more than 280 million of them found most piled at the top of the scale, with a smaller pile at the bottom and very little between. The delighted and the furious write. The lukewarm stay at home. An invited survey is milder than an open review box, but the pull is the same, and if your open-text answers are all delight and fury you may suspect that the middle is missing.
More of the same
At about this point the optimist proposes a remedy: more reminders, more responses, and the problem will shrink. It is the natural instinct, and it is wrong in an instructive way.
More responses shrink the margin of error, the uncertainty that comes of having asked only some people. They do nothing about bias, which comes of having heard from the wrong ones. In 2021 an American survey collecting about 250,000 responses a week overstated how many people had been vaccinated by 17 percentage points. A carefully recruited panel of about a thousand came close to the truth. The enormous survey reported a tiny margin of error around the wrong figure.
If 2,000 customers answer in place of 400, and they are the same kind of customer, the result is exactly as skewed as before. It merely looks more certain.
What pollsters would envy
Here the story turns in your favour. A pollster knows next to nothing about the people who did not pick up the telephone. You know a great deal about the customers who did not answer: their plan, how long they have been with you, how often they log in, what they pay, how many tickets they have raised. The silence can be examined directly.
It begins with how you send. Invite a random sample from a list, keep the list, and use links that tie each answer to a customer record. An anonymous link dropped into the app leaves nothing to compare. Then set those who answered beside those you invited. In the survey from the meeting, grouping customers by recent usage gives this:
| Segment by 30-day usage | Share of customers | Share of answers | Satisfied |
|---|---|---|---|
| Weekly active | 40% | 60% | 80% |
| Occasional | 35% | 30% | 60% |
| Dormant | 25% | 10% | 35% |
The story is in the two middle columns. Weekly users are 40% of the customer base and 60% of the answers. Dormant accounts are a quarter of the base and a tenth of the answers. The survey has chiefly heard from people who like the product enough to use it every week.
Add up the answers as they arrived and about 70% are satisfied. Count each segment according to its true share of customers and the figure falls to about 62%. That eight-point drop is larger than any margin of error that would have been quoted.
Weighting of this kind is a repair, and its limits should be stated. Each dormant answer now counts for two and a half, so that forty people are speaking for a thousand accounts. It also assumes that the dormant customers who replied resemble those who did not. If the unhappiest kept silent, even 62% is too high.
Two further checks bear on that last doubt. Compare the people who answered your first invitation with those who needed a reminder. If the late ones score lower, the silent may score lower still. In mail surveys the trick pointed in the right direction, though checked against real records it missed the true size of the error, so treat it as a hint. Better, pick thirty to fifty of the silent customers at random, not the friendly ones, and put your main question to them some other way: a personal email from a named person, a telephone call, a single tap inside the product. If they answer like everyone else, you may relax. If they do not, you have learned which way your headline leans.
None of this makes the response rate irrelevant. A higher rate fills the thin segments, so the effort is worth making where the gaps are, and our guide on how to increase survey response rate covers the methods. Reminders aimed at people who already answer readily only add to survey fatigue. And when a group is all but absent, with fewer than about thirty answers, do not weight it up. Say plainly that the result does not cover them, and reach them with a handful of interviews.
The worrier and the optimist were both arguing about ten per cent, and ten per cent was never the point. A tenth of your customers can speak for the rest, if they are like the rest. Four hundred weekly users cannot speak for a thousand dormant accounts, however many reminders go out.
The survey that found a nation of volunteers did not lack respondents. It had the wrong ones for that question, and nothing in its response rate said so. Yours will not say so either. But you, unlike Pew, have the list.
How Diaform helps
Diaform does not solve nonresponse bias, and an AI interviewer cannot tell you what silent customers think. It can make the checks easier. On Pro plans and up, personal invite links carry each respondent’s identity, so every conversation joins back to your customer data, and CSV export gives you the respondent list to set beside the invited one. For the follow-up sample, one question that people can answer by typing or speaking is a small thing to ask. Try it with a 14-day free trial.
