Even before the shots were fired, Tuesday's election in Quebec was remarkable. Pauline Marois won her minority government, but with fewer seats than expected. It was a night of surprising outcomes because the results differed substantially from what was predicted by pollsters.
It shouldn't be this way. With good polls, we should know the result of the election before the votes are cast.
How wrong were the pollsters? Eric Grenier, who runs the forecasting site ThreeHundredEight.com, collected data from every poll conducted in the election campaign. Campaigns change vote intentions, so it's not fair to compare early campaign polls against election results. But even the last three polls conducted got it wrong. On average, the Parti Québécois vote was overestimated by 3.1 percentage points. Given the large sample sizes in each of these polls, that's well outside the margin of error. The CAQ vote was overestimated by 1.3 points, an acceptable mistake. The Liberal vote? The average poll underestimated the Liberal vote by 4.8 percentage points. In a first-past-the-post system, that's the difference between a strong second place and a crashing third.
The polls in the most recent Alberta election are even more instructive. The last three soundings were fielded three days, two days and one day before the election. On average, they missed the mark for the governing Tories by nearly 11 percentage points. The Wildrose vote was overestimated by almost six percentage points. The Liberal and NDP vote share estimates, on the other hand, were within the margin of error.
Of course, these errors were explained away by talk of a late shift, especially because of massive strategic voting for the Tories by traditional NDP and Liberal voters. But do the numbers bear this out? Not really. Or at least not in a way that redeems the pollsters.
There are only two possibilities: There was no massive late shift, and the polls simply got the front-runners' vote shares wrong. Or, there was a late shift, in which case the polls didn't have the vote shares for the Liberals and NDP correct.
You can take your pick, but neither scenario inspires confidence.
In theory, polling is simple and very elegant. Only two things are required. First, you require a sample of respondents (say, 1,000 people) from a population (say, the adult population of Quebec) who are willing to give you a (relatively) honest answer. Second, you have to know the probability that each subject would be in your sample. Random sampling is the best way to do this, but it is not required. With these two things, we can appropriately weight responses and can know the opinion of an entire population, within some margin of error.
The problem is not with the first part. Pollsters have developed clever ways to obtain people's honest opinions. They can use live people on the phone. They can find respondents over the Internet. Data collection is more affordable than ever, and text messaging and smartphone apps are promising even swifter data collection.
The problem is the second part. Pollsters simply do not know enough about who responds to polls via some media, who replies through others, and what kinds of people ignore polling requests entirely.
The problem isn't getting sample, it's getting good sample. Simply knowing a respondent's demographic information is not enough to correct for bad sampling. The result is that we cannot extrapolate with sufficient accuracy from our samples to the whole population.
We cannot, in other words, know with much confidence the likely outcome of an election before the votes are cast.
The response of pollsters to this can be anticipated. Election polls are not paid for, so they are often done quickly and on the cheap. Technology is changing quickly, and with it best practices. More people are getting cellphones, and fewer of those with land lines are responding to polls. Why expect them to be accurate?
Well, we should expect them to be accurate, not least because the same firms that perform these polls also perform well-paid work for businesses and government. Presumably those clients want to be sure that they are receiving accurate advice.
There are practical fixes to this problem. First, media outlets should regularly pay for polls. This is the norm in the rest of the democratic world, not the exception. With payment comes more care and more resources for pollsters to conduct better polls.
Second, we need to be much more certain about the accuracy of new technologies before they are widely and uncritically reported on. If firms are not performing the basic research required to prove the accuracy of their tools, is it responsible to report their findings on vote intention in the midst of an election campaign?
Third, firms should release much more information on their polls. The details of what should be reported do not make for exciting reading, but full transparency on weighting and estimation procedures should be the norm.
Democratic elections and statistical science are among the greatest achievements of the last 200 years.
It is good to know what other citizens think, to know how they intend to vote, and to know what issues are important to them. Polling makes it possible to know these things. Statistical science has much to offer democratic elections, but only when it's conducted with the same care with which we conduct our elections.
Peter Loewen is assistant professor in the department of political science at the University of Toronto. He wrote this for the Ottawa Citizen.
