Do survey incentives work, and how much should you offer?

Offered $25 in cash for completing a survey, 34% of the doctors in one experiment obliged. Offered the same $25 as a cheque, 20% did.

The sum was identical. A cheque, however, has to be taken to a bank, and anything that takes effort to collect is worth less to people than the figure printed on it. It is a small result with a large moral, which is that the manner of a reward matters at least as much as its size. Most arguments about survey incentives are conducted in ignorance of this, and go as follows.

A head of product sends four questions to the 600 customers who cancelled in the past year. A fortnight later, eleven have answered. At the team meeting somebody proposes a raffle: everyone who replies goes into a draw for an iPad. Somebody else objects. Pay people, and you will hear only from those who want the iPad, saying whatever gets them to it fastest.

Both positions have the ring of common sense. Researchers have been running experiments on this very quarrel for more than half a century, and the results favour neither side cleanly. They do point to a particular way of spending the money, and it is not an iPad.

Three points

That rewards raise response is beyond dispute. It is among the most often repeated findings in the whole of survey research.

The surprise is how little they raise it online. When researchers pooled 32 experiments on web surveys, the average incentive lifted the response rate by about three percentage points. Anyone who merely wants a bigger number will find this an expensive way of getting one.

The better reason to pay appears when you look at who responds. In one student survey, the group offered nothing was lopsided, with too many women and too many students with high grades. The group offered five dollars looked much more like the student body as a whole. The money had not simply added people. It had added the people who would otherwise have had no reason to bother.

A customer list behaves in the same way. Ask for nothing in return and you hear from your admirers and from the people who are angry with you. A modest reward can bring in some of the indifferent middle, and some of the customers who have already gone. That matters more than the rate, because the real hazard in any survey is the difference between those who answer and those who don’t. It does not always come off. In one test by Gallup, the paid and unpaid groups turned out to contain the same mix of people. So the thing to check afterwards is who answered, and not only how many.

The trouble with raffles

Now for the iPad. Prize draws are popular for an obvious reason: one prize costs far less than a small payment to everybody. Unfortunately it also does far less.

A Canadian experiment with about 3,000 households compared three kinds of reward directly. A few dollars in cash raised the response. A donation to charity of the same value did not. Nor did entry into a draw with $200 in prizes. One experiment with students did find that a draw helped, so it would be too strong to say that raffles never work. But a small certainty beats a slim chance of something large.

There is one piece of good news for anyone working by email. In postal surveys, money enclosed with the questionnaire reliably beats money promised for later. Online the difference seems to vanish. When Gallup tested it, a five-dollar gift card promised on completion did as well as one sent with the invitation: 20% and 19%, against 13% for no reward at all. And the promised version cost about a fifth as much, since only those who finished were paid.

As to how much, the experiments agree on the shape of the answer, if not on a figure. The effect flattens quickly. When researchers combined 55 experiments, the first dollar enclosed with a mailed survey added about six points to the response rate. Going from five dollars to ten added about six more. Five times the money, that is, for the gain the first dollar had achieved alone.

There is a floor as well. In the student survey, a promised two-dollar credit made no difference whatever. A promised five-dollar credit raised the response from about 30% to 42%. A reward can be too small to register, and beyond a modest sum you are mostly paying more for answers you would have had anyway.

What no study will supply is the right amount for your customers. Nearly all of this research was carried out on households, students and doctors, by governments and universities, mostly by post and telephone. Almost none of it involved a company asking its own customers, and nothing in it says what half an hour of a customer’s time is worth. It tells you what kind of reward to offer. The amount you will have to test.

The sceptic’s case

That leaves the objection, which is the one that stops many teams from offering anything. Will paid people not rush, or flatter?

On rushing the evidence is reassuring. In an American telephone survey, some respondents were sent five dollars in advance and others nothing. The researchers then compared eleven different signs of effort, and only two differed. The paid group skipped fewer questions and answered slightly faster. They were not careless.

On flattery the honest answer is that nobody knows. A review for the US National Academies concluded that paid respondents “may give somewhat more optimistic responses”, and called for more testing. That is an argument for consistency. Offer the same reward to everyone, in every round. Pay in March and not in June, and a dip in satisfaction may be nothing more than the missing gift card.

The sceptic was right to be uneasy, though, about something else. The danger that can wreck a paid survey has little to do with customers shading their answers. It is people who are not customers at all. One research team advertised a survey on social media and offered $15 for each response. Within seven hours they had 271 completed surveys. On inspection they judged 94.5% of them to be fraudulent.

Money on a link that anyone can open attracts people, and software, with no interest in the question. The cure is simple and complete. Pay only those you have invited yourself, from your own customer list, through personal links, with one reward apiece. The detection side of the matter is the subject of AI-generated survey responses.

Where the money belongs

Put the findings together and a rule emerges. A reward is worth paying when the person has little reason to help you, or when you are asking a lot. It is wasted when they already have a stake and you are asking for two minutes.

Churned customers are the clearest case for paying. They owe you nothing and hold exactly the information you lack. A long interview is another, for there you are buying someone’s time, and everyone should get the same amount for it. For a short survey shown inside the product to active customers, a reward adds cost and risk for a few points of response, and the better investment is in timing and length.

Senior buyers are a special case, and the one in which money helps least. A review of 231 surveys of executives found that the usual methods of raising response didn’t work on them. What did work was a topic they cared about and a sender from their own network. For a vice-president, an invitation from the account manager she knows will do more than a ten-dollar gift card. Rules may also intervene. Many employers and public bodies cap or forbid gifts from suppliers, and prize draws and taxable rewards carry rules of their own, which differ from country to country. None of this is legal advice, so consult your legal or finance team before sending anything of value.

Before paying anyone, use what costs nothing. In the broadest review of response experiments, a shorter questionnaire, a more interesting topic and an offer to share the results all raised online response. Our guide to increasing survey response rate covers these, and closing the feedback loop makes your next request easier to accept.

When you do pay, say why you are asking before you mention the reward, and mention it once.

Then split the list at random and offer two different amounts. Compare how many people answered, who they were and how much they wrote. That small test will tell you more about your own customers than any published study can.

So who was right in the meeting? Each of them, about half. A reward would help to reach the customers who cancelled, since they have no other reason to reply. But a raffle is the feeblest form of reward and would have done little. And the fear that paid people answer carelessly does not survive the evidence, though the suspicion of money was sound in one respect: put it on an open link and the wrong people come.

The version that works is modest. Send each former customer a personal link, promise everyone who finishes the same small gift card, and leave the iPad in the shop. It is the cheque again. People respond to what is certain and easy to collect, and not to the size of the number.

How Diaform helps

Diaform does not pay respondents for you, and a chat interview asks for more effort than a one-tap rating, so all of this still applies. It does make a paid study easier to run cleanly. On Pro plans and up, personal invite links carry each respondent’s identity, so you can invite real customers only and match one reward to one person, and every conversation has a summary and the full transcript, which makes review before payment quick. Try it with a 14-day free trial.