Saturday, February 16, 2008

All Hell Will Break Loose

On the eve of the voting in Pakistan, someone responded on Facebook to my post Pakistan Election Rigging 101 as follows: "All Hell Will Break Loose" tomorrow. As much as I would like to disagree with this comment, the history of elections in Pakistan makes me pause and ponder on it. Regardless of whether the elections are free and fair, there is usually trouble in the aftermath. The general elections of 1970, held under the military regime of General Yahya Khan, were widely believed as free and fair. However, there was serious trouble and Pakistan lost its eastern wing resulting in the creation of Bangladesh. The elections organized by Zulfikar Ali Bhutto in 1977 were generally accepted as flawed and caused widespread rioting leading to military takeover by General Ziaul Haq who executed Zulfikar Ali Bhutto. So the lesson is that Pakistan has a checkered history of general elections. And there is mounting concern, given the opposition leaders' threats of mass protests if the results are not favorable to them. Such protests invariably descend into major rioting, widespread chaos and violence on the streets. Recently, I wrote a blog post titled "What If Pakistan Opinion Polls Are Wrong?". This post outlined the possibility that the opinion polls like the IRI's may be wrong based on the limitations and the history of such polls. But the key reason was to warn my readers and Pakistanis at large to use caution in assuming that the results of the general elections would just exactly reflect the findings of the latest IRI poll. I felt it is particularly important to point to the flaws because the potential for widespread violence and mass casualties in the aftermath of the February 18 elections is so great that anything anyone can do to minimize it is worthwhile. Since I wrote this piece, Wall Street Journal has published the details of the recent flawed polls in the US Presidential primaries. I reproduce this Wall Street Journal article by Carl Bialik as follows:

In the month leading up to last week's delegate-rich California primary, at least a dozen polling firms canvassed the state, collectively calling tens of thousands of households.

Political junkies tracking television, newspaper and online coverage of the voting also heard the names of two main providers of polling data that didn't place a single call: Real Clear Politics and Both are mashing up surveys from various sources this election year to produce composite numbers meant to smooth out aberrant results. Their methods are criticized by statisticians, but their numbers are embraced by news organizations eager for a way to make sense of conflicting polls.

Numbers from Real Clear Politics, which has been averaging polls since the 2002 congressional races, are used regularly on Fox News, MSNBC's "Hardball," and the Web sites of CBS News and the Washington Post. Pollster, which started combining polls in 2006 and attempts a more complicated mix than a straight average, is featured on Slate and the political Web site, Talking Points Memo.

Stirring disparate pollsters in one pot has its critics. "That's dangerous." says Michael Traugott, professor at the University of Michigan, and author of a recent guide to election polls. "I don't believe in this technique."

Among the pitfalls: Polls have different sample sizes, yet in the composite, those with more respondents are weighted the same. They are fielded at different times, some before respondents have absorbed the results from other states' primaries. They cover different populations, especially during primaries when turnout is traditionally lower. It's expensive to reach the target number of likely voters, so some pollsters apply looser screens. Also, pollsters apply different weights to adjust for voters they've missed. And wording of questions can differ, which makes it especially tricky to count undecided voters. Even identifying these differences isn't easy, as some of the included polls aren't adequately footnoted.

Mark Blumenthal, a former Democratic pollster and co-founder of, admits that combining different polls violates "a rule we were taught at pollster school." The site attempts to address concerns over using older polls by giving more weight to the latest ones.

John McIntyre, co-founder of Real Clear Politics, says that averaging polls is better than cherry-picking individual ones, which is what campaigns might do to highlight numbers favoring their candidate, or journalists might do to create the impression of a close race. He requires at least three polls before producing an average. His site's numbers, he says, provide "a clearer picture of where things truly stand."

Combining polls isn't perfect, adds Mr. Blumenthal, but "let's hope that by combining them we're getting some better version of the truth."

A case in point is the Reuters/Zogby poll completed a day before last week's California primary showing Sen. Barack Obama leading Sen. Hillary Clinton in the state, 49% to 36%. That day, SurveyUSA had Sen. Clinton ahead by 10 points -- her eventual winning margin. (Zogby blames an underestimate of the Hispanic vote and an overestimate of the African-American vote.) Still, the composite numbers produced by Pollster and Real Clear Politics showed a close race where one didn't exist.

But then, SurveyUSA had Sen. Clinton ahead by 11 points in Missouri, a state Sen. Obama narrowly won. Zogby had Sen. Obama up by three points. Noting that the Associated Press initially called Missouri for Sen. Clinton as election results came in, SurveyUSA President Jay Leve says, "Missouri is a tough state." The Pollster and Real Clear Politics combined polls did correctly depict a close race.

Jeff Jones, managing editor of the Gallup Poll, complains that some of his competitors use more-careful methodologies than others. "So, the firms that attempt to do things the right way are treated as no better than the ones that use less-accepted methods," he says.

The problem is identifying that best poll. Pollsters tend to do well here, badly there. "There are 50 almost uniquely different state systems," says Mack Shelley, an Iowa State University political scientist. "You almost have to have a poll that's right for each particular state or region."

"There's no single best pollster who always gets all the races right," says Charles Franklin, professor at the University of Wisconsin and co-developer of He says even the best poll occasionally misses the result by a few points or more, thanks to statistical sampling error. Pollster and Real Clear Politics could bolster their case by comparing their numbers directly against those from individual polling firms in terms of election accuracy, a step neither has taken.

Scott Keeter, director of survey research at the Pew Research Center, embodies the conflict many pollsters feel about the averages. "The average of good and bad polls could be worse than just looking at the best available single poll," he says. However, Mr. Keeter adds, "I confess to looking at the averages myself."

Mr. Blumenthal says visitors to his site don't necessarily share the concerns of statisticians. "Asking political junkies to stop paying attention to a horse race is like asking them to stop blinking," he says.

1 comment:

Riaz Haq said...

Early results seem to indicate the elections in Pakistan have been largely free and fair and generally peaceful. Many well known top leaders of PML (Q) are reported to have lost with PPP and PML (N) doing a nearly clean sweep in parts of Sind and Punjab. Religious parties are also suffering major setbacks. Looks like my hopes of ISI political cell of at least temp deactivation were real. Musharraf and Kayani seem to be delivering on their promises. Let's hope the winners now show grace in their victory and try to reconcile and unify the nation to build durable democratic institutions that serve the average people in Pakistan.
Riaz Haq, PakAlumni Worldwide