PRESIDENTIAL POLLING
Public confidence in the accuracy of election polls suffered because of well-publicized errors in public polls in both the 2016 and 2020 presidential elections. The main issue in both contests was an apparent underestimation of support for Donald Trump. While the accuracy of polls was much better during the 2018 and 2022 off-year elections than it had been in the previous presidential elections, some wonder if the previous problems will return this year when Trump is once again on the ballot. Experts are split whether the challenge is the difficulty is measuring vote preference when Trump is involved or whether the issue is broader, reflecting increasing obstacles for all polls.
To be sure, not all polls were wrong even in the last two presidential elections. Many national polls showed Hillary Clinton and Joe Biden winning more votes than Trump which is precisely what happened in the national popular vote. But just as successful airplane flights receive less media and public attention than crashes, accurate polls tend to slip through without much notice. Moreover, the fact that several polls produced estimates at odds with the election results – especially on the state level – served to confirm a widespread suspicion that polls do not and cannot accurately measure public sentiment.
Three major factors stand out in these mistaken poll results: the number of firms conducting and reporting “polls,” the methods they use to gather data, and the way they present the results. The number of firms reporting results of national elections polls has more than doubled in just the last four years. Most of the new entrants into the polling universe use different – and more questionable methods – than more established and experienced firms. Many of the polls from new firms are conducted by people with less training in statistics and opinion research science than is true of the more established firms. Many of the new firms come to their task with obvious political biases. And many report national poll results to market their services and cut methodological corners to reduce costs.
The biggest change in the way polls are conducted is the number of polls that completely or largely use “opt-in” methods. In these polls, potential respondents decide whether or not to participate in the poll, often in response to invitations on social media, email, or on websites. These opt-in polls have proven to have more than twice the errors associated with traditional random sample. One of the main ways these polls differ from reliable polls is that they include a disproportionate number of respondents with strong opinions, thereby giving their preferences much more influence in the poll results than they have in reality.
This is not to say that the traditional methods for conducting polls are were without error. In fact, almost all traditional polling firms have adjusted their methods of data collection in part because of the bad results in the 2016 and 2020 elections. In particular, well-established polling firms supplement interviews conducted by live telephone interviewers using random-digit-dialing methods for sampling with on-line or text-based sampling. These methods have proven quite successful in avoiding the errors most often associated with previous mistakes, i.e., not including a sufficient number of interviews with white voters without college degrees or people who vote but as not particularly interested in politics.
The mistaken results of the two most recent presidential elections also forced pollsters to acknowledge and account a phenomenon some had known and made allowance for in previous years. All polls are “weighted” to account for differences between what is known about a universe and the sample of those people who participate in polls. Traditionally, this meant mathematically increasing the results from the types of people who did not have access to a telephone or a computer, usually men, people with lower incomes, and voters in non-white racial and ethnic groups.
What pollsters learned the hard way was that just using simple demographic indicators like age and race was insufficient to correct the problem. The people who did participate in polls were different from people who did not even if they had the same demographic profile. To illustrate, white voters between the ages of 40 and 59 who answer polling questions have significantly higher levels of formal education and more progressive policy views than all white voters in that age group. To “correct” this problem, more sophisticated pollsters now “weight” their results using five to eight variables rather than simple the two or three variables they formally used to adjust results.
Reporting errors often occur when the commentary says that a candidate is “ahead” without noting that the gap between the candidates’ support is smaller than the error of interviewing a sample rather than the entire electorate. Even when polls report sampling error, they typically ignore the fact that sampling error refers to the estimation for each candidate’s vote rather than the margin between them.
To illustrate, let’s take a poll with a margin of error of 3.0 points showing Trump at 45% and Harris at 42%. In a strictly statistical sense one cannot say that certain that Trump is ahead. The poll results actually say that that there is a 95% probability that Trump’s vote support is between 42% and 48%, and Harris’s support is between 39% and 45%. This is certainly more complicated than saying Trump is “ahead,” but it more accurately shows that the reliable range of estimates for the two candidates’ support overlaps.
Finally, one should also remember the cliché that polls are like a “snapshot” in time. Just because one candidate has more support than the other in a poll does not mean the leading candidate will maintain that position when votes are cast and counted. Things change in the final days of campaigns in ways that no poll can predict – including the fact that sometimes voters change their minds.
Pollsters face other challenges, including estimating which voters will show up and cast ballots, and the fact that Presidents are elected according to the Electoral College model rather than in a national popular vote. Most well-established polls do everything they can to create accurate estimates, but they cannot correct everything that interferes with generate accurate results.
If ever there was a time to cautious, it is with interpreting election polls, especially given the challenges and cost pollsters face even if they are trying to do everything possible to “get it right.”
✍︎ ✍︎ ✍︎
Featuring guest blogger, Harrison Hickman, President & Founder of Hickman Analytics, Inc.