As online surveys become more prevalent, researchers face many challenges associated with this shift to increased web-based interviewing. We are often asked by clients how, if at all, data collected online differs from a more traditional telephone survey. Last month, Pew Research Center released the results of an experiment comparing data from two replicate national samples, with one collected over the telephone and the other self-administered on the web (link to the full report here).
In particular, the Pew report examines mode differences, that is differences in responses attributable to the mode in which the question is administered (in this case, telephone versus web). The highlights of this study and conclusions drawn by the Pew team provide valuable insights for researchers as they compare the relative benefits and drawbacks of these two dominant opinion research methodologies.
In general, the Pew team found that differences in responses by survey mode are common, but typically not large. In fact, many of the variations between the phone and web samples were not statistically significant and fell within the study’s margin of error.
However, on three distinct types of questions sizable mode effects were observed:
Political Measures
The experiment found that ratings of political figures were significantly more negative among the web-based sample. As the table below shows, web respondents were much more likely to give political figures a “very unfavorable”
The differences grew even larger when respondents rated political figures from the opposite party. Among Republicans, 53{09f965da52dc6ab4c1643a77bd40d1f729d807040cd8db540234bb981a782222} described Clinton as “very unfavorable” on the web versus 36{09f965da52dc6ab4c1643a77bd40d1f729d807040cd8db540234bb981a782222} on the phone. Similarly, on the web-based survey 63{09f965da52dc6ab4c1643a77bd40d1f729d807040cd8db540234bb981a782222} of Democrats rated Palin as “very unfavorable,” compared to 44{09f965da52dc6ab4c1643a77bd40d1f729d807040cd8db540234bb981a782222} in the phone sample.
A similar pattern emerged on questions that asked survey participants to rate the political parties using a feeling thermometer ranging from cold/negative to warm/favorable.
Mode differences also occurred on image questions for lesser-known political figures. The Pew team found statistically significant differences in the number of respondents who said they had “never heard of” a figure.
The research team hypothesized that the presence of a “never heard of” option on the self-administered web survey caused this particular mode difference. While the interviewers on the phone study were instructed to accept “never heard of” as a response, since it was not offered as an explicit choice to phone respondents, they were less likely to volunteer it as an answer.
Discrimination Measures
The Pew experiment uncovered significant mode differences on questions related to discrimination against different groups. Respondents in the phone sample were more likely to agree that groups faced “a lot” of discrimination than were people interviewed via the web-based survey.
Happiness and Life Satisfaction Measures
The third area where the research team found significant mode differences was on questions related to happiness and life satisfaction. In general, respondents being interviewed over the phone were more likely to give a positive/favorable response to these types of questions.
Similar trends emerged on questions related to the community where a respondent lives, with those interviewed over the phone more likely to give a positive/favorable response on issues ranging from traffic conditions to the area being a good place to live.
The researchers found some mode differences on questions asking respondents to describe their own community involvement and engagement, with telephone respondents reporting higher participation in several of these metrics.
On issues related to personal finances, web respondents were more likely to reveal that they have faced financial challenges in the past year compared to the phone sample.
The mode differences on these financial questions became more significant when looking at subgroups including low-income respondents and non-white respondents.
Key Takeaways
In general, the mode differences found by the Pew team follow the theory that some telephone respondents will give responses to make themselves look more favorable or responses that will be less likely to produce an uncomfortable interaction with the interviewer. Using this logic it makes sense that differences emerged on questions related to divisive political figures, queries about discrimination and questions about sensitive personal issues. However, the Pew team stressed that while there are some significant differences by mode on a handful of questions, an equal number have small or non-existent differences.
So… what exactly does this information mean for researchers?
There are strengths and weaknesses to both phone and web methodologies. While internet-based surveys might elicit some more candid responses from interviewees, phone-based studies continue to provide access to a more representative sample. It is up to researchers to examine the goals of each project and determine which methodology best suits the client’s goals and needs. Moving forward, the Pew research team hypothesized that mixed-methodology surveys may become more prevalent as researchers strive to find the balance between cost-effective interviewing, encouraging honest and thorough responses, and achieving a representative sample.
