At some point, someone has to ask: What purpose do political polls serve other than getting the name of the sponsoring news organization onto competitors' pages or newscasts?
Over the weekend, of course, two polls for the U.S. Senate race showed Al Franken well in front in one race, well behind in another. Net effect? Useless information.
Today, Minnesota Public Radio is out with a poll showing Barack Obama "surging ahead" of John McCain in Minnesota. It mirrors to a degree the Star Tribune poll two days ago that shows roughly the same thing.
"As they tune in and as they are concerned about the economy, they're asking the question 'Which candidate is better able to handle the economy going forward and to get us out of this economic mess?'" Obama Minnesota campaign boss Jeff Blodgett said. "And overwhelmingly, we are finding that voters are coming toward Barack Obama."
Meanwhile, a National Journal poll shows McCain closing the gap nationally and pulling within the margin of error, apparently because of the economy. On the question of who is best to handle the economy, the poll said, Obama's lead has evaporated.
Two other polls show Obama widening his lead over McCain nationally.
So let's sort this out. Obama is extending his lead, McCain is closing the gap, the economy is responsible for the shift to Obama, and the economy is responsible for the shift to McCain.
We've got plenty of polls; we have precious little clarity.
Maybe if enough polls are done, some statistician can poll the polling data and come up with a good guess for what is actually happening.
Drama Bob, Drama. We all - myself included - just love drama and speculation. I need some reason to bite my nails and stir it up with my conservative friends. Gotta love drama.
The question is whether all polls should be treated equally.
For instance, the Survey USA & Strib polls from last week have drastically different weights for Repubs vs Dems. Working from memory, the Strib poll has 42% dems, Survey USA has 33% Dems. I don't recall breakouts for Independants or Repubs. So, which poll is closer to reality in reflecting the makeup of the electorate? Self-identified Dems probably fall somewhere in the upper 30s, so Strib appears to be over-sampling Dems, while Survey USA does the opposite. Result: Franken v Coleman is a tighter race than either poll indicates - which other polls have subsequently reported.
This stuff isn't quite as confusing as the opening post claims - once you dig beneath the surface veneer.
FiveThirtyEight is about the best site for making any sense of the overall trends of polls, with proper weights for more accurate polls and simulations for Election Day.
The numbers do have something to say, especially with the biggest elephant in the room: The Electoral College.
Polls of 350 and 400? How much can you extrapolate for the entire state? The MPR poll must have a pretty wide margin of error.
Net effect? Useless information.
Not useless, Bob. Looky here:
Matt mentioned it before, and I've mentioned it above, but FiveThirtyEight.com is doing a greatly job of breaking down the numbers, explaining polling trends, and answering riddles like you pose above.
In fact, I commented on your blog a couple of weeks ago about FiveThirtyEight.com, so you already knew this. Why keep your readers in the dark just to post a story about how those darn polls are just so confusing?
First of all, there are many different kinds of poll. A poll of "Likely Voters" pre-screens people based on their past tendency to vote (which is publicly available. A poll of Registered Voters will go after everyone. Some pollsters use only people to make calls, while some are "Robo-polls". Some as the ballot question first, and others ask it after gauging the mood of the voter.
All of these things influence the result. In addition, a margin of error of 3.7 points (the Survey USA that showed Coleman ahead has this) for two polls means that they could be 7 point apart and both might be "right".
What I think we can say about these polls is that when a race is in flux, things will start to get weird before you know what's going on. That's my take on the Senate race - it's Barkley's surge that's doing it.
But aside from that, we should NEVER report polls as "fact" and NEVER claim anything inside of the margin of error is important. Since polls are very often inside the margin of error, they are lousy nooze tools.
In many ways, the individual characteristics of the poll are irrelevant since media "bottom lines" the polls.
Useless? nah. That's why http://fivethirtyeight.com is so helpful. It is very easy to see when there's a bad poll that doesn't go along with all the others, when you lay all the information out in an easy to visualize format.
"Polls of 350 and 400? How much can you extrapolate for the entire state? The MPR poll must have a pretty wide margin of error."
Actually, once you approach 400 or so people, you're going to be about as accurate as if you poll 1000 or more. When you do see larger polls - in the 1000+ respondent range, it is usually because they want to analyze subgroups within the poll; i.e. what do women vs. men think?
The real mistakes get reported when you start with a sample in the 300-400 range and try to claim that you have statistically meaningful info about segments within the sample. i.e. "non-white women with college educations in the $50k-100k income braket think XYZ." Breaking into those segments splits the sample into groups that are far too small to extrapolate info with any reasonable degree of accuracy.
Being able to understand a poll and having a poll be useful to a voter are two different questions.
Let's focus on the 2nd one. What is the usefulness of a poll?
"Let's focus on the 2nd one. What is the usefulness of a poll?"
right now they're useful for backing up premature statements of "I told you so".
On a more serious note, they are useful to news consumers who are trying to understand the people around them.
Polls are not useful to voters for deciding how to vote. (Why should they be?) While I'm sure there is a segment of voters the simply votes with the perceived majority, most voters typically vote based on other things: the candidate's political party, the candidate's position on issues (or an issue), the candidates character, the way the candidate makes decisions, etc.
Polls are useful to understand the way the political race is proceeding, and the opinions of voters. The give the media something to report on, and people are quite simply interested in the way a political race is going. This makes polls news.
Of course, polls are also of great interest to campaign strategists.
Besides that, polls are academically interesting, and they give statisticians something to do.
"Actually, once you approach 400 or so people, you're going to be about as accurate as if you poll 1000 or more. When you do see larger polls - in the 1000+ respondent range, it is usually because they want to analyze subgroups within the poll; i.e. what do women vs. men think?"
This is one of the strange results of statstics. I've had it proven to me, but it is counterintuitive. It relies on a truly random sample population. That is a hard thing to be sure you are getting.
Even with perfect samples, with all the polls flying around any of them could be the 5 out of 100 where the actual numbers are outside the margin of error.
I think the problem of arguing about fivethirtyeight.com is that the vast majority of people won't go there or think about the polls. They just take them at face value. If you aren't willing to put the time in to understand everything behind a poll I think you should just ignore them all. I know I am.
It would be better if they just weren't reported at all. People that want to see and understand them can look them up. It is dangerous for people to "know" things that are actually wrong.
First, the motivation for the plethora of polls is the desire to frame (or see it framed, or both) the whole election idea as a horserace. Consumers of news like that. Polls such as these do have value, but to take advantage of that one needs to know the subtleties of how they were conducted.
I agree with many posters that it it surprisingly easy to reduce sampling variability by getting a large enough sample: only several hundred in principle for all but very close races. This issue quickly becomes swamped however by the issue of, is one asking the right subset of people? and (the more insidious one IMO) are the pollees telling the truth?
Others have mentioned the importance of "likely voters". But in a close race, a small mistake in quantifying these can make a big difference in prediction. The telling the truth one is the 500 pound gorilla here, however.