
How does that old saw go — something about lies, damned lies, and political polling? Let me just say up front that I like the relative certainty of most statistics as much as the next guy. I mean, who doesn't appreciate Ol’ Gauss or marvel at the elegance of the Law Of Large Numbers? But the fact of the matter is that the polling process is inherently imprecise — susceptible to (statistical and political) bias and sample errors, and providing at this stage in an election limited predictive value. Polling is not like taking samples of bacteria in a lake; it’s an attempt to gauge human intentions, which resist quantification and change over time. Not to mention that the very measurement process — people asking questions and formulating answers — is itself notoriously unscientific.
The ambiguity of polls is something not easily understood by the public. Or, apparently, by most political reporters, who throw the numbers around as if they’re solid quantitative records of political realities rather than ranges of possibility that are easily misinterpreted. And that’s what makes them so dangerous.
Case in point is the release of those two new polls late Thursday night, each showing a vastly different result. Gallup gave Bush a 13% advantage. The Pew Research Poll completed that same night showed Bush with a 1% lead. A friend wrote me a quick note showing his frustration:
God I hate polling. And you know the orgs, who just today wrote a story on the Bush bounce’s evaporation (AP), will jump all over this one.
Indeed, the next morning the Gallup Poll, which is jointly commissioned by CNN and USA Today, became the day’s big headline. Lots of talk about Bush having the Big Mo’. But this was incredibly irresponsible. There were significant methodological problems with the Gallup poll, the main one being that there was a basic sample in how the party identification of the likely voter turnout was figured.
To wit (via The Left Coaster):
Likely Voter Sample Party IDs
Poll of September 13-15
Reflected Bush Winning by 55%-42%
Total Sample: 767
GOP: 305 (40%)
Dem: 253 (33%)
Ind: 208 (28%)
The flaw here lies in the fact that Gallup’s result assumes that on election day, 40% of the turnout will be Republican, 33% Democrat, and 28% Independent. Actual turnout has never looked like this. In 2000, the proportional turnout was 34% GOP, 34% Democrat, and 33% Independent. In 1996, Democratic turnout was in fact much higher than among Republicans. Re-weighting the sample to reflect a more likely turnout in 2004 would reduce Bush’s lead to a few points.
But the problem was not just the poll itself. It was also how it was reported. News stories about polls almost never question — or even describe — the methodology. Nor do they put them into proper perspective by noting that the reporting news organization’s poll is just one of many. And herein lies the systemic problem with election polling. Each network or paper commissions its own survey and then acts like they got some kind scoop. They want to get their numbers out first, and tout them as news, and so they choose not to mention that the other guys’ numbers say something different. The competition between proprietary polls forces the media, when reporting on the extremely sensitive and important state of our national elections, to be un-journalistic: they use only a single source — their own poll — while ignoring other, potentially conflicting information.
I was talking about this recently with Lynn Vavreck, a political science professor at UCLA who specializes in quantitative methods, and this is how she described the problem: “It’s what I call incentive incompatibility. The candidates, the public, and the media all want different things. USA Today and CNN and Newsweek — their incentive is to sell magazines and newspapers. What voters want is information. And candidates want coverage. And those things often don’t line up. Civic journalism would be nice. But the media want to sell their products, and that’s why USA Today would splash a big headline about that Gallup poll without mentioning the other polls or dealing with the problems with their poll.”
On top of incentive incompatibility, there’s also the general indifference to nuance that characterizes most front page news stories, which is what we see in the USA Today article about that new Gallup Poll:
BUSH CLEAR LEADER IN POLL
By Susan Page
WASHINGTON — President Bush has surged to a 13-point lead over Sen. John Kerry among likely voters, a new Gallup Poll shows. The 55%-42% match-up is the first statistically significant edge either candidate has held this year.
Among registered voters, Bush is ahead 52%-44%.
[That is a very bold — and by extension, not entirely accurate — interpretation of Gallup's numbers. And the second line almost proves itself wrong: THE RESULT IS THE ONLY STATISTICALLY SIGNIFICANT EDGE BECAUSE IT A) SAMPLE ERROR OR B) AN OUTLIER.]
The boost Bush received from the Republican convention has increased rather than dissipated, reshaping a race that for months has been nearly tied. Kerry is facing warnings from Democrats that his campaign is seriously off-track.
With 46 days until the election, analysts say the proposed presidential debates offer Kerry his best chance to change the race.
[Again, it's not really fair to say that the Gallup poll, or any single poll for that matter, reshapes the entire campaign.]
"It doesn't look like the new consultants and strategies of attacks are the right ones" for Kerry, says Matthew Dowd, chief strategist for the Bush campaign. Kerry in recent weeks added veterans of the Clinton White House to his team and began criticizing Bush more sharply on Iraq and other issues.
Dowd says Kerry at this point would "have to defy history" to defeat a sitting president.
[This is not true. Bush himself was lagging — farther and with much more statististical certainty — behind Gore at this stage of the game last time around. And yet Dowd's comments remain unchallenged.]
"We have seen some bouncing around in the numbers," says Mike McCurry, a top Kerry adviser, "but it is our sense that the race is moving back to a much closer race."
[Only in the 8th graf do we find out about that other poll, the one that confirmed what the rest of the polls say, which is that the race is extremely tight:]
A Pew Research Center poll released Thursday shows a tighter contest. The survey, taken Saturday through Tuesday, gives Bush a statistically insignificant lead of 47%-46% among likely voters.
CNN’s item about the same Gallup poll managed to give the story a little more balance. They lead with the variance, and added a sub-hed noting both polls:
LATEST PRESIDENTIAL POLLS VARY WIDELY
Gallup, Pew Research polls get different results
Friday, September 17, 2004
As both camps' campaign schedules intensify, two key polls indicate different standings for President Bush after the convention.
(CNN) — Two new national polls are giving widely divergent views as to whether President Bush's post-convention bounce has solidified or evaporated.
A Gallup Poll released Friday showed the president widening his lead over Kerry among likely voters to 13 percentage points, with 55 percent for Bush and 42 percent for Kerry.
The poll had a margin of error of plus or minus 4 percentage points. (Special report: America Votes 2004, poll tracker)
The race narrowed to 8 points among registered voters, with 52 percent choosing Bush and 44 percent picking Kerry.
The level of support for the president was the highest he has seen in the Gallup Poll since January, when he was enjoying a bump from the capture of Saddam Hussein.
Bush's job approval rating also rose to its highest level since April, with 52 percent saying they approved of his performance and 45 percent saying they did not.
However, a poll by the Pew Research Center for the People and the Press, released Thursday, showed the race between Bush and Kerry now a dead heat . . .
That addressed the nuance better. But it still fails to question the basic credibility of the poll itself. It had to be left to The Left Coast and others to look at the fine print and discover the sample error. But the damage from poll reporting can be hard to undo: detailed fact-checking, like we’re talking about here, can’t compete with the general impression formed by a bunch of Bush at 55, Kerry at 42 headlines. Ever since similarly suspect Newsweek numbers came out just as the RNC closed, there’s been a perception that Bush is steamrolling Kerry. It's like there's a Heisenberg effect in polling, where measurement can change reality. And hidden down in the details of the Gallup poll was another question, with a result that was statistically significant, that confirms such an effect:
Q: Who do you think is most likely to win in November?
September, 2004
Bush: 60
Kerry: 22
August, 2004
Bush: 44
Kerry: 37
That’s Newsweek covers and USA today headlines talking. It’s sure as hell not the actual numbers, which do show a little Bush boost, but have never once suggested some kind of double-digit, insurmountable lead. Aggregating all the polls, best estimates show a very tight race, as it has always been.
As it also was in 2000. That’s when Karl Sigman, an applied mathematician at Columbia, used state-by-state polling to create a Monte Carlo simulation that assigned an 84% probability for Gore to win. Without trying to describe it in detail, I’ll just say that Sigman came up with a clever way to use fairly basic statistics to predict the electoral college outcome. And he came closer than any of the pundits or other political “professionals.” But even with state-by-state polls made available the day before the election, the model was slightly off in the end. That was because, still at that late date, people changed their minds. Or didn’t vote. Or some groups came out in force, while others stayed home.
So even the day before the election, Polls have limited analytical value. And in terms of democracy, they have almost no value, because they divert the public from the actual issues at stake in elections and may even distort the outcome. This is what Canada realized in the late 90’s, when they enacted a polling ban. It actually became illegal to report polling data in the run-up to Canada’s national elections. It was a short-lived experiment (Canada’s Supreme Court overturned the law under the grounds that it restricted freedom of expression), but not a bad idea.