News Cut: November 11, 2010 Archive

Polling problems

Posted at 10:28 AM on November 11, 2010 by Bob Collins (2 Comments)
Filed under: Media, Politics

Ouch, that one hurt. But the tweet, posted hours before the hapless Timberwolves were to take on the world champion Los Angeles Lakers this week, is an example of what happens when a definitive poll turns out to be not so definitive.

That's the U of M's Larry Jacobs' problem, which he shares with the Humphrey Institute and Minnesota Public Radio after gubernatorial polls released just before the election appear to be inaccurate -- and not by a little.

MPR issued this news release today:

(St. Paul, Minn.) November 11, 2010--Minnesota Public Radio and the University of Minnesota's Humphrey Institute of Public Affairs announced today that they will undertake a thorough review of the methodology used in polls conducted during the 2010 election season. The process will include an internal review of the poll by the Humphrey Institute and an independent audit that will be made public. The independent audit will be conducted by Frank Newport, the editor and (sic) chief of Gallup.

MPR and the Humphrey Institute partnered this year to conduct four polls leading up to Election Day. The final poll, based on interviewing begun nearly two weeks before Election Day, showed results significantly different from the final election tally. This issue will be examined along with the raw data from other polls to determine whether there is a methodological reason for the difference, or whether external events account for the difference.

"We are committed to a transparent review of our polling methodology because we value the importance of continuous improvement in our efforts," said Professor Larry Jacobs, director of the Humphrey Institute's Center for the Study of Politics and Government. "If a shortcoming is identified, we will fix it. If not, we will have third-party verification that our methods are sound."

"The review of polling methodology is a necessary step in continuing to provide Minnesotans with the unbiased information they need to make informed decisions," said Chris Worthington, MPR's managing director of News.

Dean Brian Atwood of the Humphrey Institute added, "I welcome the opportunity to conduct this self analysis and peer review, a regular process for any academic institution. Professor Jacobs is an internationally recognized expert in this field. He is a professional who looks critically at his own work, as well as at polls conducted by others. We are committed to maintaining a very high standard."

I have not talked to anybody at MPR involved in the polling situation, but one doesn't need to to know that MPR has a problem going into the 2012 campaign. Even if MPR and the Humphrey Institute get the methodology fixed (assuming it's broken) before the first poll of the 2012 campaign comes out, few of them will be have any credibility until Election Day, because there's really no other way to prove their reliability.

On his media-watchdog blog, David Brauer has found a Carleton College expert who may (or may not) be involved in the poll introspection.

While (Steven)Schier won't divulge conversations with MPR, he is willing to critique HHH's methodology. "What I can tell you is that the poll problems may lie in two places -- the likely voter screen and the attempt to factor in cell phone use."

As I noted this summer, HHH does not survey cell-phone-only voters, or CPOs. However, it tries to simulate that 25 percent of households by giving additional weight to land line respondents who also have cell phones.

To be sure, it's comforting that MPR is taking the possible poll problems seriously. Of course, any hits to a news organization's credibility is an assault on its vital organs.

But there's another reason why accurate polls matter: They may influence the outcome of elections. Sen. Kathy Saltzman, a moderate DFLer who lost to a Republican last week, told the Woodbury Bulletin that she thinks Democratic legislators may have suffered defeat because voters saw the pre-election polls showing DFLer Mark Dayton leading in the governor's race.

"I think that people were concerned that a (Democrat-controlled) Legislature would be a rubber stamp for some of the policies that he campaigned on," she said.

Maybe she's trying to come up with ways to make her loss sting less, or maybe she's right. If it's the latter, perhaps a larger discussion is in play for news organizations: If polls influence the outcome of elections, what's the value in doing them?