There are a number of problems with polling. Using alleged Russian support for Vladimir Putin’s war on Ukraine, a New Yorker article does a great job covering a bunch of them,[1] but this is by no means solely a Russian problem, as we see in the U.S. presidential election yet again in 2024.[2] A major problem that doesn’t get nearly enough attention are the dismal response rates.
I remember in the first research methods class I took, I think in 2003, my professor, Valerie Sue, said that a proper response rate is 90 percent. It had declined even then (I think she said it had dropped to 70 or 80 percent, a level she clearly disapproved of) and has declined—a lot more[3]—since to something like six or seven percent in 2019,[4] perhaps even less now. I gather that pollsters have been tweaking their models to account for this drop but the simple fact is that with response rates this low, they are effectively left with self-selecting samples, eviscerating any notion that these are random samples. Without random samples, you don’t have representative samples.
When pollsters tweak their models to try to get those samples back, they are inevitably speculating about how non-respondents would respond—somehow extrapolating from self-selecting samples. Which means all we’re really seeing with these supposedly ‘scientific’ polls are pollsters’ guesses.
It’s been a garbage methodology for a while now. It lives because politicians and pundits feel they have no alternative.[5] But because it is so egregiously flawed, I don’t rely on it. I prefer to look at “fundamentals,” meaning I look at how issues are playing (including yard signs and other mindless demonstrations of support) and use that to inform my speculation—yes, speculation—about how elections will turn out.
Late in 2015, when I defending my dissertation, I predicted Donald Trump’s victory because I saw the authoritarian populist anger. I later backtracked from that because Trump’s remarks kept getting more and more awful—I couldn’t imagine that people would actually vote for that. But they did, I believe, largely because of anger at the Obama administration’s neoliberal policies that bailed out banks while leaving underwater homeowners and the unemployed to twist in the wind and then pursued the TransPacific Partnership that would have exported even more jobs overseas.
Similarly, this year, I resisted believing that people would vote for an apparently demented Trump who couldn’t even string together coherent sentences at his rallies. At the end, however, I was compelled to a conclusion he would likely win. And he did.
It’s not much of a methodology. But right now, it’s better than polling.
[1] Joshua Yaffa, “Why Do So Many Russians Say They Support the War in Ukraine?” New Yorker, March 29, 2022, https://www.newyorker.com/news/news-desk/why-do-so-many-russians-say-they-support-the-war-in-ukraine
[2] Scott Clement and Praveena Somasundaram, “How did 2024 election polls fare? We talked to experts,” Washington Post, November 14, 2024, https://www.washingtonpost.com/politics/2024/11/14/elections-polls-trump-harris/
[3] Steven Shepard, “Report: Phone polls aren’t dead yet,” Politico, May 15, 2017, https://www.politico.com/story/2017/05/15/pollsters-phone-polls-238409
[4] Courtney Kennedy and Hannah Hartig, “Response rates in telephone surveys have resumed their decline,” Pew Research Center, February 27, 2019, https://www.pewresearch.org/fact-tank/2019/02/27/response-rates-in-telephone-surveys-have-resumed-their-decline/
[5] Dan Balz, “2020 presidential polls suffered worst performance in decades, report says,” Washington Post, July 18, 2021, https://www.washingtonpost.com/politics/2020-poll-errors/2021/07/18/8d6a9838-e7df-11eb-ba5d-55d3b5ffcaf1_story.html; David Byler, “Polling is broken. No one knows how to fix it,” Washington Post, July 22, 2021, https://www.washingtonpost.com/opinions/2021/07/22/polling-is-broken-no-one-knows-how-fix-it/; Mona Chalabi, “The pollsters were wrong – again. Here's what we know so far,” Guardian, November 4, 2020, https://www.theguardian.com/us-news/commentisfree/2020/nov/04/the-pollsters-were-wrong-again-heres-what-we-know-so-far; David A. Graham, “The Polling Crisis Is a Catastrophe for American Democracy,” Atlantic, November 4, 2020, https://www.theatlantic.com/ideas/archive/2020/11/polling-catastrophe/616986/
Steven Shepard, “Dem pollsters acknowledge ‘major errors’ in 2020 polling,” Politico, April 13, 2021, https://www.politico.com/news/2021/04/13/dems-polling-failure-481044