Surveys are everywhere: political campaigns, consumer confidence, purchaser managers index, sell-side analysts consensus, customer or employee satisfaction. Their results affect politics as well as policies in many domains, from macroeconomics to corporate management.
Yet the sad reality is that there is a shocking discrepancy between the importance given to surveys when making decisions and their intrinsic quality. Eliciting private information from respondents is particularly challenging when objective truth is inaccessible. In ‘A Bet On Europe’ (Dec 2016), it was observed that the quality of polls on voting intentions has been on a declining trend. This was attributed to a number of factors, including social (media) pressure since respondents tend to hesitate to express unconventional views. Everyone knows it, but most prefer to ignore it. Surveys are irresistible as they blissfully provide arguments for action.
In a paper entitled ‘A Bayesian Truth Serum for Subjective Data’, Drazen Prelec, Professor of Economics at MIT’s Sloan School of Management, devised a simple system – thoroughly tested since then – which helps to obtain an honest answer from respondents. I had a call with him this week to discuss his approach which is based upon the following empirical truth: If people truly hold a particular belief or preference, they are more likely to think that others share that belief or preference with them.
The BTS (short for Bayesian Truth Serum) method relies on asking questions in pairs and analyzing the relationship between the answers. If the first question is, for example, ‘Do you like red wine?’, the second question asks about the individual’s estimate of how many other respondents would answer the same way. Now:
If a person does like red wine, his or her best estimate of the share of respondents who like red wine will be high, at, say 50%, whilst that of an individual who does not like red wine will be low (e.g. 30%)
The average predicted share in favor of red wine will logically and mathematically fall between these estimates
Therefore, those who truly like red wine will see the expected popularity of their own honest opinion to be underestimated (check video, starting 26’)
Put differently, a ‘surprisingly popular’ answer has a higher probability of being true and honest. The formula then assigns high scores to such answer when assessing the overall results of the survey, instead of weighing each answer equally. The BTS approach is simple yet effective.
Polling agencies are finding it hard to change their approach to surveys, according to Mr. Prelec. As a breakthrough, he and his team have been running a survey related to next week’s US elections using innovative techniques as outlined in ‘How Social Science Tools Can Better Predict Election Results’ published this Friday. Their results suggest a much smaller difference between Democratic and Republican candidates than the single-question methodology from the same poll.
The above may represent some food for thoughts for customer and employee surveys as management tools. More generally, the world of surveys would benefit from a total revamp.
Kudos to the pioneers.