The price of bad polling

Donald Trump gives the thumbs up to supporters as he leaves his golf club in Sterling, Virginia, on Monday. Picture: AFP

The 2020 polling autopsies are still being written, and there are many unanswered questions.

How did Republicans win 27 of 27 House of Representatives districts identified as toss-ups by the Cook Political Report?

How did Republican senator Susan Collins win Maine by nine points when she didn’t lead in a single major poll for months?

How are Donald Trump and Joe Biden separated by a little under four percentage points in the national popular vote when Nate Silver’s FiveThirtyEight average showed an 8.4-point Biden lead on election day?

New York University economist Arpit Gupta compared the FiveThirtyEight presidential forecast to that of online betting markets, where people trade contracts reflecting the likelihood a candidate will win.

He found that the “markets are pretty well calibrated — state markets that have an estimate of 50 per cent are, in fact, toss-ups in the election. 538 is at least 20 points off — if 538 says that a state has a 74 per cent chance of going for Democrats, it really is a toss-up”.

Prediction markets aren’t perfect, and some are so thinly traded that they can move with a few bets. But at least everyone knows they’re bets. The polling industry and statistical models make claims to objective expertise, yet they again clearly missed Republican strength.

Combined with media conformity, this can lead pundits and the public astray. The prognosticators make projections — shaped by assumptions about demographics, turnout and response rates — which influence the press. These views are then reinforced on social media, especially Twitter. Everyone runs in the same direction, and the press sets the conversation in a way that can further influence pollsters and campaign decisions.

Consider the impact. Radical Democratic Party ideas like US Supreme Court-packing may not have received as much backing from party leadership if forecasts had been more realistic about Democrat odds of winning Senate seats in Iowa and North Carolina, as well as House seats in Florida, California and Texas. That warped discourse may have increased polarisation.

The predictions that Democrats would gain 10-15 seats or more in the House were so wrong that they may have hurt Republican chances to take House control in what was a very close result.

The Cook Political Report, which is taken as gospel by the media, was notably wrong about Republican prospects. Charlie Cook says he and his team were misled by the polls.

“In my judgment, there was a blue wave building, a pretty big one, then something happened, like a fish getting spooked before taking a bite out of a lure,” Cook explained. “Too many of the most experienced political operatives in both parties could see it coming. My guess is that while a majority, albeit a small one, wanted to unseat Trump, they got skittish about giving Democrats unified control.”

Something happened all right. But if he’s right that polls misled him, then the Cook Political Report has no special claim to expertise. Yet its estimates influence the flow of campaign cash. The Chamber of Commerce endorsed 23 House Democrats this year, in part because it anticipated having to work with a larger Democratic Party majority, and it may have saved Nancy Pelosi’s speakership.

The GOP was heavily outspent, yet it may gain as many as 12 net seats in the House. What might have happened if Cook and his House analyst, David Wasserman, had been more accurate and said the battle for House control would be close?

And who knows how the perception that states like Ohio were close at the presidential level (Trump won by eight points) affected voter turnout. It could have mobilised Trump voters, or demoralised them.

While Biden can tout a convincing popular-vote victory, Trump would have tied the Electoral College and thrown the election to the House by closing a gap under one point in Georgia, Arizona and Wisconsin (where a Washington Post-ABC poll released on October 28 had Biden up by 17).

Pollsters are working on better models, but it would be healthier if politics and media narratives were shaped less by instant snapshots of quantified opinion. Public figures could make decisions based more on what they think is right. Journalists and opinion leaders could develop insights on popular sentiment the old-fashioned way — by actual reporting. Those insights might also be shaped by bias, but at least they wouldn’t be presented with an imprimatur of scientific authority.

The Wall Street Journal