The puzzle

Probability samples are the gold standard of survey research, yet when the task is forecasting an election result they do not reliably beat non-probability samples. This post asks why.

The gold-standard reputation is well earned. When samples are compared against known facts about the population, probability samples are consistently more accurate than non-probability panels:

  • Pew Research Center (2023) compared three address-recruited probability panels with three online opt-in samples across 28 benchmarks drawn from government statistics. Average error was 2.6 points for the probability panels and 5.8 points for the opt-in samples. Among 18- to 29-year-olds and Hispanic adults, opt-in error rose to around 11 points, against 3.6 for the probability panels (Pew, 2023).
  • Earlier studies reached the same conclusion, including Pew's own 2016 evaluation of online non-probability samples and a replication and extension published in Public Opinion Quarterly by MacInnis and colleagues (Pew, 2016; MacInnis et al., 2018).
  • Data quality reinforces the gap. Pew's 2026 study of an 11,000-person opt-in sample found that 18% of respondents claimed implausible experiences, such as having visited the International Space Station (Pew, 2026).

The expectation is clear: if sample quality drives accuracy, probability polls should predict election results more accurately than non-probability samples. Pew itself notes, however, that recent election polls are an exception to that expectation.

How I looked at it

The evidence I want to share comes from two investigations.

  1. Published studies. What have researchers found when they compared the election accuracy of probability and non-probability polls? The main sources are the AAPOR task force reports on the 2020 and 2024 US elections and Pew's methodological work.
  2. Recent elections. How did final polls perform, by method, in Germany (February 2025) and Canada (April 2025)? Sweden (September 2026)? These elections provide a separate example of shared errors, with shorter notes on the US and UK in 2024.

For Germany and Canada, the selected pollsters' final published polls are scored against the result using average absolute error per category, along with the error in the gap between the two leading parties. The calculations and their limitations are described at the end.

First investigation: what the studies say

The published studies agree that the sampling frame has little consistent relationship with election accuracy. Probability or non-probability does not seem to be a deciding factor.

AAPOR Task Force on 2024 Pre-Election Polling (October 2025). This review covered 611 polls of presidential, senatorial, gubernatorial and congressional contests from the final two weeks of the US 2024 campaign. Average absolute error on the two-party margin was 3.3 points, down from 5.3 in 2020. The task force concluded that no single methodological recipe guaranteed accuracy and that most major methodological choices showed little relationship with error. Polls from higher-volume firms, those weighting on party identification and those using detailed likely-voter models were slightly more accurate. The differences were small and could reflect other features of those firms rather than the methods themselves (AAPOR, 2025). Executive summary

AAPOR Task Force on 2020 Pre-Election Polling. In the cycle with the largest polling error in 40 years, the task force found that no mode of interviewing was unambiguously more accurate (AAPOR, 2021).

Pew's benchmark study, read for its political measures. The same 2023 study that favours probability panels overall tells a different story on politics. All three probability panels overstated 2020 turnout by 8 to 9 points, even after weighting for volunteering and registration, while two of the three opt-in samples came within a point of the true figure (Pew, 2023). Enns and colleagues, reading the study's appendix, report that the two sample types were about equally accurate on 2020 vote choice.

Enns, Goranson, Rothschild and Streett (May 2025). Using FiveThirtyEight's poll database, they report that probability-based presidential polls in 2020 averaged about 6 points of error, roughly double the non-probability average. In 2024 the gap reversed but narrowed: 3.3 points for probability polls against 4.1 for non-probability, a difference that was not statistically significant. Non-probability polls from firms in AAPOR's Transparency Initiative or depositing data with the Roper Center averaged 2.3 points, and the probability-based American National Election Studies pre-election survey missed by 5.7. The authors work for Verasight, a non-probability provider, so this is an interested contribution, but it is based on published data and released with replication code (Enns et al., 2025).

Pew on bogus respondents (August 2026). This study adds an important caution. An unscreened opt-in sample of 11,114 US adults surveyed just after the 2024 election showed Trump and Harris level at 48% each, close to the result. Removing suspected bogus respondents moved the estimate to a Harris lead of between 3 and 7 points, depending on the screening method. The bogus cases leaned heavily towards the winner, offsetting a genuine skew among the remaining respondents (Pew, 2026). An accurate topline, in other words, is weak evidence of a sound sample. Although this result could also result from failures in the algorithm to identify genuinely bogus interviews in some cases.

Read together, the studies support the proposition for elections specifically, while giving no reason to think the finding transfers to research in general.

Second investigation: recent elections

In Germany and Canada, telephone and online polls were interleaved in the accuracy rankings. Sweden illustrates a different point: randomly recruited panels can share an election-polling error.

Germany, February 2025

The result was CDU/CSU 28.5%, AfD 20.8%, SPD 16.4%, Greens 11.6%, Linke 8.8%, BSW 5.0% and FDP 4.3%. Ranked by average error per party in the final poll:

  • YouGov, online opt-in panel: 0.6 points
  • INSA, online opt-in panel: 0.7
  • Ipsos, online panel: 0.7
  • Forschungsgruppe Wahlen, telephone and online interviews with randomly selected voters: 0.7
  • Wahlkreisprognose, online: 0.9
  • Forsa, telephone interviews with randomly selected respondents: 1.1
  • Allensbach, face-to-face quota sample: 1.3
  • Infratest dimap, telephone plus online panel (final poll 11 days out): 1.5
  • GMS, telephone plus online: 1.6

Forsa finished sixth among these nine firms, behind three online panels. Its final election poll was a telephone survey; the recruitment method of its separate omninet panel does not establish how this poll was conducted. Forschungsgruppe Wahlen used telephone and online interviews with randomly selected voters (Wahlrecht; ZDF). Most of the selected pollsters put the Union slightly high and Linke slightly low. Wahlrecht ZDF

Canada, April 2025

The result was Liberals 43.8%, Conservatives 41.3%, NDP 6.3%, Bloc Québécois 6.3%, Greens 1.2% and PPC 0.7%. Examples from the more and less accurate final polls were:

  • Mainstreet, automated telephone (IVR) with random dialling: 0.4 points
  • Angus Reid, online opt-in panel: 0.5
  • Liaison, IVR with random dialling: 0.5
  • Nanos, live telephone with random dialling: 0.8
  • …
  • EKOS, telephone plus online; panel status not verified for the scored poll: 1.9
  • YouGov MRP, online opt-in panel: 2.0
  • Pollara, telephone plus online: 2.1
  • Innovative, online panel: 2.5

Telephone polls and online panels appeared at both the more and less accurate ends of the comparison. The EKOS estimate scored here put the Liberals on 43.6% and Conservatives on 37.5%, overstating the Liberal lead by 3.6 points. The compilation describes this estimate as telephone/online; it should not be treated as evidence about EKOS's probability panel without confirming the survey's sample source. Most polls overstated the NDP and understated the Conservatives. Late tactical voting is one possible explanation, but the final-poll comparison cannot establish the cause. Final-poll compilation

Sweden, September 2026

Sweden provides an example of shared error among polls that include randomly recruited panels. It does not provide a verified comparison of probability and non-probability samples. The methods of all the firms in the election compilations were not independently established for this piece.

Verian (formerly Sifo) and Novus describe their panels as randomly recruited (Kantar Sifo; Novus). In 2015, Sveriges Radio reported that Ekot and SVT did not cover polls from self-recruited panels. These sources establish the role of random recruitment in Swedish polling, but do not establish that the entire 2026 market used probability samples. Kantar Sifo panel book Novus methods statement Sveriges Radio

The clearest finding concerns the Moderates. Both linked compilations show that all the final pre-election polls they list put the party below its reported result of 19.9%. Their lists of firms and some individual figures differ, so I have not combined them into an accuracy ranking. Wikipedia compilation Valresultatet.se

  • The shared miss included the randomly recruited panels. In the Wikipedia compilation, Novus put the Moderates on 18.1% and Verian on 15.8%, against 19.9% in the result. Random recruitment did not prevent either poll from underestimating the party. Poll figures
  • Both exit polls also put the Moderates low, at 17.5% (Novus) and 18.1% (SVT/Valu). This weakens an explanation based solely on movement between pre-election fieldwork and polling day. Participation, coverage, weighting and measurement problems are possible explanations; these figures do not establish which was responsible. Exit polls

The Swedish example therefore supports a limited conclusion: random recruitment alone did not prevent a shared miss. It cannot establish the relative accuracy of probability and non-probability sampling. Sources: Wikipedia compilation and valresultatet.se.

The US and UK in 2024

  • United States. Beyond the AAPOR findings, the probability-based ANES pre-election survey missed the presidential margin by 5.7 points, worse than most published polls.
  • United Kingdom. The polls collectively overstated Labour (British Polling Council). Ipsos conducted a pre-election test on its probability-based KnowledgePanel, replicating its telephone methodology in large parts. The panel overestimated Labour and underestimated Reform UK, while the telephone quota poll estimated Reform's share correctly (Ipsos UK). The change also involved a different interview mode, so this was not a controlled test of sampling design. It does show that probability recruitment did not eliminate the error. British Polling Council Ipsos UK

Why probability samples don't win elections

An election forecast is the output of a model, and the sample is only one of its inputs. The poll result that gets published has passed through weighting, a turnout model and, increasingly, calibration to how people voted last time. Seen that way, the puzzle becomes less puzzling.

Elections offer an unusually powerful calibration variable. Most people vote the same way as they did last time. The AAPOR task force found that past voting behaviour can substantially improve estimates, but its usefulness depends on how it is measured and how pollsters account for changes in turnout and the electorate (AAPOR, 2025). This provides a plausible explanation for why modelling can reduce differences between sample types. Past vote is a valuable predictor, although recall errors and uncertainty about who will vote limit its use. AAPOR

Random recruitment cannot prevent non-response bias. AP VoteCast reported an unweighted response rate of 2.4% for its probability sample in 2024 (AP VoteCast). Random recruitment does not ensure that respondents remain representative when participation is selective. Both probability and non-probability polls then rely on adjustments to account for who is missing. Probability sampling retains a known recruitment design, but the accuracy of the achieved sample also depends on response patterns and the effectiveness of those adjustments. AP VoteCast methodology

The best sample for benchmarks is not necessarily the best sample for elections. Probability panels over-represent the politically engaged. Pew's panels overstated 2020 turnout by 8 to 9 points even after weighting, and engagement is correlated with how people vote.

Important errors can be shared. The final polls in Germany and Canada often missed the same parties in the same direction, and the Swedish polls shared an underestimate of the Moderates. AAPOR identified difficulties representing Republican voters in Republican-leaning areas and people who voted in 2024 but not in 2020. These findings show that a recruitment design alone cannot explain election-polling accuracy; they do not establish a single cause across countries. AAPOR

Firm practice also matters. AAPOR found small associations between accuracy and polling volume, party weighting and detailed turnout modelling, while most major methodological choices showed little relationship with error. AAPOR

This suggests a more nuanced reading. The election comparisons set one firm's sample and model against another firm's sample and model, so the two are confounded. The firms with probability frames are not necessarily the firms with the strongest election models. It is plausible that the best modellers, working with the best probability samples, would usually come out ahead. The evidence reviewed here does not isolate that effect. The Ipsos comparison also changed interview mode, so it cannot settle the question.

If a probability advantage does exist, one place to look for it is in disruption elections, where last time's vote is a weak guide: surges of new voters, new or fast-rising parties, or realignments. In those elections, there is less for the model to calibrate against, and the sample may have to carry more of the weight. A stronger test would use the same firm, questions, fieldwork dates, interview mode and modelling approach on both sample types across several elections. I have not identified a published comparison at that scale among the sources reviewed here.

Notes on method and sources

How the election figures were calculated. The selected firms' final published polls were compared with the election results. Average error is the mean absolute difference across eight categories in Germany and six in Canada. Germany uses the seven named parties or alliances plus a combined category for all other parties, including Free Voters. Canada uses the Liberals, Conservatives, NDP, Bloc Québécois and Greens plus a combined category for the PPC and all other parties. Missing minor-party shares are included in the combined category rather than treated as separate zero estimates. Figures are rounded to one decimal place. Poll figures come from the linked Wikipedia compilations; German methods were checked against Wahlrecht and the ZDF release. Sweden is discussed separately and is not included in the chart or numerical rankings because the linked compilations do not reconcile.

Limitations.

  • Two numerical election comparisons and one qualitative Swedish example are a small evidence base. Individual polls also carry sampling uncertainty, which varies by sample size, party share, weighting and design. No significance tests were applied to the rankings.
  • Method classifications rely on published descriptions. Several firms use mixed or evolving designs, and some were not independently verified.
  • Average error per party weights small and large parties equally; a margin-based or seat-based measure would rank some firms differently.
  • Only final polls are scored, which disadvantages firms whose last poll came earlier.

Sources

This piece was researched and drafted with AI assistance and reviewed and edited by the author.