No Diving: Research in the Shallow-End of the Public’s Policy Knowledge

The other day, I found myself watching the 2000 Vice Presidential debate between Dick Cheney and Joe Lieberman on YouTube. At one point, Lieberman laid out the highlights of the Democratic ticket’s proposed tax reform, which included a number of specific credits and changes he said would help the middle class and spur growth. Though I had my qualms, I felt Lieberman clearly and succinctly communicated what he felt were the benefits of the proposal. But after he was done, Cheney dropped a rhetorical hammer saying essentially you’d have to have a PhD in math to understand how that complicated policy would benefit you and your family. He said, what Americans need is simple: a tax cut for everyone.

Aside from the crystal clear distinction Cheney skillfully provided voters with a five second sound bite, this exchange got me thinking about the broader disconnect between the “public” and “policy” in public policy debates. That is, the depth of understanding among policy professionals who work on these issues every day compared to the nominal level of knowledge the average American can be expected to have on a policy issue as expansive and complicated as the US tax code. This exchange also made me reflect on my job as a researcher, clarifying my thinking about why simpler research can work to produce clearer insights on complicated policy research.

From my experience working on and helping craft research on complicated policy issues, I’ve realized something that at first glance may seem counter-intuitive: The simpler the subject, the more deep and complicated the research can be. More complicated subjects on the other hand are best served by simpler research designs, the shallow-end of the public’s policy knowledge. Here’s why:

When exploring opinion around subjects that are part of a respondent’s daily life, there are myriad ways opinions are formed and connected to other opinions, making it possible to dive deeply and explore attitudes, beliefs, expectations, hesitations, and concerns.

For instance, a survey on economic policy might evaluate views from personal experiences, such as job loss, or how respondents see the role of government in spurring economic growth. The survey may include whether respondents feel the economy has been getting better or worse over time and how optimistic they are about the economy for future generations. All those types of questions could provide needed context around responses to the economic policy the survey is focused on. As a result, research documents (i.e. surveys, focus group guides, one-on-one interview scripts) can be longer and more robust because there are multiple angles from which to look at the issue and different perspectives from which to frame a policy position.

On the other hand, measuring opinion on complicated policy issues, such as the effect legislation would have on how an industry or business operates, is harder because these issues are disconnected from most peoples’ lives. As a result, the research documents that provide the most useful context and direction are ones that wade in with broad and simple designs.

These simpler research designs often employ a few broad measures of underlying attitudes about the subject, a brief and accurate synopsis of the policy issue, simple straight-forward measures of support or opposition, and two to four simple messages from each side (if needed).

Even this simpler research can provide the essence of what good policy research should be: a common set of accurate; easy to understand conventions about a given subject that helps guide communications and/or actions. But, to do it right three elements are essential to the strategic advice we provide our clients.

  1. The most important element of any opinion research effort is to make sure you are talking to the right audience. Many complicated policy issues may only be relevant to business leaders in a single industry, others may be subjects that highly informed and engaged voters would be interested in and hearing about. But, importantly, research is an iterative process and usually the first step is to quantify what information those most engaged in and affected by a given policy understand and respond to.
  1. The second element is the generous use of open-end questions we use to “listen” to how, if at all, respondents tie complicated policy back to their daily lives or overall world view. As a firm, we pride ourselves on the high quality and richness of the verbatim responses we collect and give these qualitative data appropriate weight in our analysis.
  1. And, third, provide respondents who don’t have enough information or don’t feel strongly enough to choose one of the options provided the opportunity to “opt out.” Forcing respondents into a four-point scale (e.g. Strongly support, somewhat support, somewhat oppose, strongly oppose) on an issue they don’t really understand makes data murky. Telling respondents it’s okay to say “I don’t really know enough” or “I don’t have strong feelings one way or the other” makes the data among those who do choose an option much clearer.

    On many issues, more Americans are in the shallow-end of policy than you may think. In the NBC-WSJ August poll, for instance, we found 40% of Americans didn’t know enough about “the rise of the terrorist group ISIS in Iraq,” 42% didn’t know enough about the Syrian civil war, and about a third said they didn’t know enough about the conflicts in Gaza (35%) and in Ukraine (32%) to say if they were satisfied or dissatisfied with how the US was handing these situations.

    Even 20% said they didn’t know enough about the immigration problem with children from Central America illegally crossing the border to have an opinion, which is a comparatively smaller percent but still a considerable chunk of the electorate. The point is we wouldn’t know what share of Americans weren’t informed enough to offer an answer if we didn’t provide respondents the option to say so.

Still, opinion polling is as much an art as a science. So, when in doubt and when the budget allows, pollsters should field a survey among a very small sample and do post-test interviews among respondents and interviewers (when applicable) to identify questions or language that is confusing.

Bottom Line

In much of the policy work we do we can delve deeply. But, the secret to clear, accurate, and useful research on complicated policy issues is to stay in the shallow end of the policy pool, let respondents say if they just don’t know, and listen using open-ended questions.

Research designs on complicated policy issues that don’t identify the right audience, assume too deep of a knowledge base, and/or force respondents into pre-determined responses can produce fuzzy data that doesn’t hold up in real life.  And, even with all these elements in place, pre-testing is advisable when the budget allows.

Similar Articles

  • Walmart Moms Focus Group Video & Memo
    read more

  • Walmart Mom Focus Groups
    read more

  • Exploring the effects of interview mode on public opinion data
    read more

  • Smartphones during the 2014 Campaigns
    read more

  • Cell Phones and the 2016 Elections
    read more

Public Opinion Strategies helped us to clarify what we wanted to learn and then conducted research and analysis that shed light even beyond the questions we set out to ask. They were very receptive to our suggestions, responsive to our queries, and flexible when we needed them to be.

Public Opinion Strategies has consistently offered unparalleled advice and spot-on polling that has shaped how and where we spent money and deploy key resources. Additionally, they have always been an excellent steward of limited campaign resources, ensuring we spend wisely and not a dollar more than necessary in order to get the information we need.

Robert Blizzard and Public Opinion Strategies did a great job for us throughout our successful campaign for Congress. Robert gave us accurate data, spot on analysis, and professional advice, all of which were essential to our victory.

Public Opinion Strategies is one of our go-to pollsters when it comes to testing public support for bond ballot measures and other initiative proposals. They are available to provide ongoing consultation with regard to crafting of ballot questions, public outreach messaging, and related efforts.

ACLI has worked with Public Opinion Strategies for decades, through several tough industry battles—often ones in which public opinion does not naturally fall on the side of insurers. Yet Bill and his team consistently provide invaluable strategic advice by refining our messages and helping us frame our issues in a way that makes them understandable and persuasive.

The data from Public Opinion Strategies provided important insight and informed our public awareness campaign. We sincerely appreciate their professionalism and expertise in this arena.

Nicole McCleskey and the team at Public Opinion Strategies have been invaluable to me, both during my campaigns and as Governor of the State of New Mexico. It’s not just the accuracy of their numbers, but guiding the overall strategy that makes them so valuable.

Public Opinion Strategies has been a part of our team in Missouri for more than a decade. With their data and guidance, Republicans here were able to attain a majority in the House in 2002 for the first time in fifty years, and we have been able to grow that majority to the point that we now have a record, veto-proof majority.

In my tenure at two leading business associations, facing huge and complex consumer issues, I have benefitted enormously from the objective advisory skills of Bill and his team. They do their homework, they are rigorous, dispassionate and thoughtful. Turning questions into answers is a clever tag, but it’s also an apt description of the professional talents of the firm.

I consider Public Opinion Strategies to be a part of our team. That is the way we have always worked. They have helped us to understand our needs and fashioned research solutions to meet those needs. They have helped us to meet killer deadlines by being flexible, executing rapidly, and insuring quality. Teamwork is the best way to describe it.

Accuracy, speed, and deep knowledge of key issues and public sentiment are the hallmarks of quality opinion research, and on these measures Public Opinion Strategies consistently delivers. I have had the pleasure of working with Public Opinion Strategies for more than 15 years on dozens of issues, and they are undoubtedly the gold standard.

Public Opinion Strategies’ track record of success and wealth of experience in political campaigns and issue advocacy are why they are one of the most trusted and well respected public opinion firms in Washington, D.C. Their insights and perspectives have helped to inform a wide array of public affairs activities across multiple industries.