Polinaut: January 21, 2006 Archive

The survey says...... or does it?

Posted at 8:49 AM on January 21, 2006 by Bob Collins

Poking around some more, looking back at the last few polls of the 2004 presidential campaign in Minnesota.

Just for kicks, here are the final polls for that race.

Zogby (November 1)
Bush 47.0%
Kerry 49.0%
(Margin of error: +- 4.1%)

Star Tribune (10/26-10/29)
Bush 41%
Kerry 49%
(Margin of error: unavailable)

MPR/Pioneer Press (10/27 -10/29)
Bush 48%
Kerry 47%
(Margin of error: +-4)

Looks like two out of three were within the margin of error. I don't have the Strib's MOE, but I'm guessing it would've been around 4, so that's the only poll that could've reliably (as reliable as polls can be) been said to indicate Kerry with a lead outside the MOE.

Final results:

Kerry 51.1%
Bush 47.6

A few weeks before, Gary Gilson of the Minnesota News Council invited Strib pollsters and bosses, a few political scribes (we ended up just being there for props), and opponents of the Strib polling methodology, to tape a show to be broadcast on the Minneapolis cable TV system.

The critics provided results of Strib polling and then compared it with election results and said, "see, the polls were wrong." It was not exactly a high moment for the science of statistics and the teaching thereof.

Many of the complaints about the methodology mirrored some of the complaints about this week's Zogby poll. That is, since they were "wrong" about the results of the race, they were, well, wrong.

I don't know whether the Zogby poll this week is right or wrong because there's no possible way to verify it one way or the other other than examining the methodology and finding its weakness. But nobody did that this week, rather the results of previous polls were judged based on the final results of the race -- a judgement that only seems to work if you don't know the methodology nor the limitations of polling data in the first place.

I'll admit that a poll close to an election, is likely to be a better indication of what might happen in the election than a poll one month out, or two months, or seven months. But that doesn't really change the science much. It's still a bad idea to ask a poll to do something it's not designed to do. If it's cloudy today, will it be cloudy Tuesday? You know, I'm not convinced that's the best application of available science.

John Zogby is a weird guy. His explanation for the disconnect between the survey in the days prior to the Mondale-Coleman election were simply bizarre. But let's face it: with a candidate dying days before the election, and the pep rally/memorial service a week earlier, the electorate was simply too volatile to get an accurate reading, and every pollster should've realized that. The story that should have come out from polling data in that race (and did) was that anything could happen on that Tuesday.

But I also have to admit it bothers me -- a lot -- that people who appear to criticize poll methodology, do so by criticizing poll results vs. election results as if the polls represent a snapshot of the future rather than a snapshot of the survey date.

One could just as easily point out the accuracy of Zogby by noting that in Zogby's last polls in the 14 battleground states in 2004, Zogby's results matched the Election Day presidential results in 12 of the 14 states (he missed Iowa and Florida, with only Iowa falling outside the margin of error).

But that -- as the man once said -- would be wrong (too).

I think every decent pollster I know of, outlines his/her poll methodology. That's fair game in analyzing a poll's data. But nobody took that on this past week. Why not?

Again, I'm not defending Zogby's poll numbers, but the methodology is sitting out there to be examined and if the poll is flawed, drop a line in the comments section and point out the methodology flaw, will ya?

But before we go too much further in methodology vs. predictability, here's an oldie-but-a-goodie from Midmorning from September 16, 2004. Understanding election polls. (RealAudio required).

The best sermon this weekend

Posted at 3:04 PM on January 21, 2006 by Bob Collins

The best sermon this weekend comes from Charlie Cook. There's a little blame for everybody, but he pretty much gives the Internet a well-deserved shot:

The Internet has simply taken the hostilities to new heights. Despite being one of the most amazing technological developments of the past 100 years, it is also an electronic version of the inside door of a public bathroom stall. Libelous accusations can be posted anonymously. And information that is inaccurate or taken totally out of context can get widely disseminated instantaneously.