On The Media

Published by

On The Media
# Marginal Understanding

*Audio not yet available*

Summary

Transcript

BROOKE GLADSTONE: This is On the Media. I'm Brooke Gladstone. Judging from the now unceasing deluge of tracking polls, which candidate leads by 3 points is the most important question facing America. Nearly every day a new poll by Gallup or Zogby, CNN or Time suggests who would win if the election were today, and the pundits and politicos scramble to interpret the polls for those of us who don't know a margin of error from a confidence interval. Often though, the interpreters themselves are misled by the findings, so with the blind leading the blind, we find ourselves adrift in a fog of figures. The Columbia Journalism Review's CampaignDesk.org explored the poll problem in a recent article, and we're joined now by the author, Thomas Lang. Welcome to the show.

THOMAS LANG: Thanks for having me.

BROOKE GLADSTONE: So you give the example of a Newsweek poll that was reported on CNN back in July. The poll found that 51 percent of voters supported Kerry-Edwards, and 45 percent supported Bush-Cheney. That's a 6 point gap. The margin of error was plus or minus 4 percent, and so the CNN analyst reported that the margin of error had been exceeded. What's wrong with this picture?

THOMAS LANG: Well the poll wasn't outside the margin of error. You could take Kerry's number, 51, and either add 4 points to it or subtract 4 points to it, and it would still be within the margin of error. Now, if we took both numbers, for instance, and we took Kerry and subtracted 4 points, we'd have 47. And if we took Bush and added 4 points, we'd have 49. Bush 49 - Kerry 47 -uh-oh - who's in the lead? [LAUGHTER] We don't really know. Now statistically, the probability of that actually being the result is pretty low, but it still is definitely within the margin of error.

BROOKE GLADSTONE: So what you're saying is even though Kerry was polling at 51 percent and Bush at 45 percent, that the candidates could have been tied.

THOMAS LANG: It's possible.

BROOKE GLADSTONE: Does that mean that there really isn't any value in reporting poll numbers that fall inside the margin of error?

THOMAS LANG: Well, there's a value. The most important thing to look for in polling is not necessarily the head to head number, but the internal numbers. For instance, after the Democratic National Convention, polling numbers didn't show a bounce that some people expected from Kerry, but if you went and looked inside the numbers, people were more confident that he could handle Iraq or more confident that he could handle the economy. A consultant who wrote a op-ed for the Boston Globe once compared the head to head numbers, taking the national number -- like waking up in the morning, turning on the TV and there being a national temperature -- and then getting dressed according to that national temperature. The national vote isn't what decides the election. So I encourage people to go out and look for -Zogby has a battleground poll, and also just the state to state polls - what's going on in Florida, what's going on in Michigan - can really possibly tell you more about the race than the head to head poll.

BROOKE GLADSTONE: Now, beside the margin of error, there's another technical aspect of polls that's rarely mentioned in news reports, but you mention it in your piece, and that's called the confidence interval.

THOMAS LANG: The confidence interval indicates that if several samplings of the same randomly-selected pool of voters were taken, then the true number for all voters would lie within the range set by the margin of error for 95 percent of the samples. It's usually 95 percent in the United States.

BROOKE GLADSTONE: You mean if you did the poll a hundred times, it would come up right 95 percent of the time?

THOMAS LANG: Well, come up right -- that's, that's a good way to put it. So let's look at an analogy here. Take a roulette wheel. This is not your average roulette wheel. It has 95 green spots on it, and 5 red spots. The problem with polling is, is you can't see whether or not it lands in red or green. You don't know if the poll that you just came up with is an outlier.

BROOKE GLADSTONE: In other words, any given poll that is reported by the media is probably mostly right but could be completely wrong.

THOMAS LANG: Right. That's, that's a good way of saying it.

BROOKE GLADSTONE: You offer a kind of solution in your piece, and naturally, not to sound like Michael Moore, it seems to lie in Canada, [LAUGHTER] so tell us how it's done by our brethren in the North.

THOMAS LANG: Canada has laws -- the Canadian Election act -- with require its media to report the methodology - the question order - the sponsors of the polls - the margin of error and the confidence interval. The writers there learned about polling, and therefore when two polls come out that are entirely different, can explain that to the readers. Let me read you something that was in one of Vaughn Palmer's article -- a political writer for the Vancouver Sun. "Pollsters, mindful of the principles of random sampling, always record that their results will fall within the margin of error 19 times out of 20. Perhaps this was the 20th time, when the true picture was somewhere outside the box."

BROOKE GLADSTONE: Okay. Well, that's Canada. But would Americans, do you think, accept such elaborate technical explanations?

THOMAS LANG: Well the culture in the United States is to get the information out as quickly as possible, even if it is careless.

BROOKE GLADSTONE: So basically what you're describing is a caveat emptor situation for the news consumer. They can see the poll, but if they don't have the complete story, then they simply don't know whether to trust it.

THOMAS LANG: That's exactly what it is.

BROOKE GLADSTONE: All right. Thank you very much.

THOMAS LANG: Thanks for having me.

BROOKE GLADSTONE: Thomas Lang reports for Columbia Journalism Review's CampaignDesk.org.

THOMAS LANG: Thanks for having me.

BROOKE GLADSTONE: So you give the example of a Newsweek poll that was reported on CNN back in July. The poll found that 51 percent of voters supported Kerry-Edwards, and 45 percent supported Bush-Cheney. That's a 6 point gap. The margin of error was plus or minus 4 percent, and so the CNN analyst reported that the margin of error had been exceeded. What's wrong with this picture?

THOMAS LANG: Well the poll wasn't outside the margin of error. You could take Kerry's number, 51, and either add 4 points to it or subtract 4 points to it, and it would still be within the margin of error. Now, if we took both numbers, for instance, and we took Kerry and subtracted 4 points, we'd have 47. And if we took Bush and added 4 points, we'd have 49. Bush 49 - Kerry 47 -uh-oh - who's in the lead? [LAUGHTER] We don't really know. Now statistically, the probability of that actually being the result is pretty low, but it still is definitely within the margin of error.

BROOKE GLADSTONE: So what you're saying is even though Kerry was polling at 51 percent and Bush at 45 percent, that the candidates could have been tied.

THOMAS LANG: It's possible.

BROOKE GLADSTONE: Does that mean that there really isn't any value in reporting poll numbers that fall inside the margin of error?

THOMAS LANG: Well, there's a value. The most important thing to look for in polling is not necessarily the head to head number, but the internal numbers. For instance, after the Democratic National Convention, polling numbers didn't show a bounce that some people expected from Kerry, but if you went and looked inside the numbers, people were more confident that he could handle Iraq or more confident that he could handle the economy. A consultant who wrote a op-ed for the Boston Globe once compared the head to head numbers, taking the national number -- like waking up in the morning, turning on the TV and there being a national temperature -- and then getting dressed according to that national temperature. The national vote isn't what decides the election. So I encourage people to go out and look for -Zogby has a battleground poll, and also just the state to state polls - what's going on in Florida, what's going on in Michigan - can really possibly tell you more about the race than the head to head poll.

BROOKE GLADSTONE: Now, beside the margin of error, there's another technical aspect of polls that's rarely mentioned in news reports, but you mention it in your piece, and that's called the confidence interval.

THOMAS LANG: The confidence interval indicates that if several samplings of the same randomly-selected pool of voters were taken, then the true number for all voters would lie within the range set by the margin of error for 95 percent of the samples. It's usually 95 percent in the United States.

BROOKE GLADSTONE: You mean if you did the poll a hundred times, it would come up right 95 percent of the time?

THOMAS LANG: Well, come up right -- that's, that's a good way to put it. So let's look at an analogy here. Take a roulette wheel. This is not your average roulette wheel. It has 95 green spots on it, and 5 red spots. The problem with polling is, is you can't see whether or not it lands in red or green. You don't know if the poll that you just came up with is an outlier.

BROOKE GLADSTONE: In other words, any given poll that is reported by the media is probably mostly right but could be completely wrong.

THOMAS LANG: Right. That's, that's a good way of saying it.

BROOKE GLADSTONE: You offer a kind of solution in your piece, and naturally, not to sound like Michael Moore, it seems to lie in Canada, [LAUGHTER] so tell us how it's done by our brethren in the North.

THOMAS LANG: Canada has laws -- the Canadian Election act -- with require its media to report the methodology - the question order - the sponsors of the polls - the margin of error and the confidence interval. The writers there learned about polling, and therefore when two polls come out that are entirely different, can explain that to the readers. Let me read you something that was in one of Vaughn Palmer's article -- a political writer for the Vancouver Sun. "Pollsters, mindful of the principles of random sampling, always record that their results will fall within the margin of error 19 times out of 20. Perhaps this was the 20th time, when the true picture was somewhere outside the box."

BROOKE GLADSTONE: Okay. Well, that's Canada. But would Americans, do you think, accept such elaborate technical explanations?

THOMAS LANG: Well the culture in the United States is to get the information out as quickly as possible, even if it is careless.

BROOKE GLADSTONE: So basically what you're describing is a caveat emptor situation for the news consumer. They can see the poll, but if they don't have the complete story, then they simply don't know whether to trust it.

THOMAS LANG: That's exactly what it is.

BROOKE GLADSTONE: All right. Thank you very much.

THOMAS LANG: Thanks for having me.

BROOKE GLADSTONE: Thomas Lang reports for Columbia Journalism Review's CampaignDesk.org.