While digital communication has made a lot of things easier—like video calling someone on the other side of the world—it has made collecting public opinion and behavior data more challenging.
Government agencies rely on that data from censuses, public opinion, and behavior surveys to make extensive policy and financial decisions that impact quality of life, such as healthcare measures that curb smoking.
Don Dillman, a Washington State University Regents Professor in sociology and internationally renowned survey methodologist, has dedicated his career to improving the design of surveys to collect that information.
When he started his career in the 1970s, he had to worry about improving methods in only three ways—through in-person interviews, postal mail surveys, or telephone surveys. While each had their challenges, household addresses and phone numbers provided a reliable pool to select a representative sample. Phone surveys quickly became the industry standard—in part thanks to a still commonly referenced book written by Dillman—due to their savings in cost and time.
The advent of caller ID, cell phones, and the internet in the 1990s introduced many new ways to communicate, but also made people harder to reach and more likely to ignore or miss surveys.
“In the 1970s you could depend on people being home at a certain time,” Dillman says. “Now people are on the go more, and even if you can reach them, they are more likely not to answer.”
By the early 2000s, response rates to telephone surveys had fallen to single digits, rendering them less reliable. Survey methodologists needed a new option, and Dillman turned to a surprisingly old-school method: postal mail.
From 2007 to 2014 Dillman and WSU sociology doctoral students Jolene Smyth ’07 PhD, Leah Christian ’07 PhD, Michelle Edwards ’13 PhD, Benjamin Messer ’12 PhD, and Morgan Miller—all part of WSU’s Social and Economic Sciences Research Center, one of the only university-based survey labs—developed the web-push methodology that is now used in censuses and other public policy surveys around the world.
The methodology drives people to take an online survey by sending a request through the mail, and has surpassed the response rate of phone surveys by a factor of five.
For their ideas, Dillman and his former graduate students received the Warren J. Mitofsky Innovators Award from the American Association for Public Opinion Research in May, an award last granted in 2015 to Nate Silver, creator of statistics news website FiveThirtyEight.
When phone survey responses declined, online surveys introduced a time- and cost-saving option. However, generating a representative sample from email addresses proved challenging. Individuals often have several emails, and the non-uniform format of addresses makes it difficult to generate a random sample.
Without phone or email options, Dillman returned to postal mail. The USPS still provided access to 98 percent of household addresses.
“Given cybersecurity concerns of today, mail also feels more secure to people,” Dillman says. “It is likely to stay universal.”
In 2007, funding from the National Science Foundation allowed Smyth, Christian, and Dillman’s first design. It involved sending a postal request and $2–4 incentive to take an online survey. The researchers followed up with a reminder, as well as a paper mail-in option. It received a 58 percent response rate, 44 percent online and 14 percent by mail.
“We were so shocked by the response rate over the web,” Smyth says, “It was the first time a web-based survey of households received that high of a response.”
The subsequent nine tests by Dillman and three other graduate students resulted in similar numbers. Over seven years they perfected the methodology and published a book in 2014. In the meantime, countries and organizations began adopting and evolving the method, and are already seeing promising results.
Australia, Canada, and Japan have used the method in their censuses, and the United States will also use it for the Decennial Census in 2020.
“Perhaps the biggest success so far is Canada’s 2016 census,” Dillman says. “Sixty-eight percent of households responded online, and most of the rest responded via mail. That is a huge cost- and time-saver.”
Continually changing technologies and communication social trends will keep survey methodologists busy for years. Dillman’s former students are already leading the survey evolution: Christian is vice president of data science at Nielson, Messer works for Research-in-Action, and the other three work at universities.