In opinion polls, what does “margin of error” mean?
By Murray Bourne, 14 Jan 2008
Don't ever believe surveys. They can never be 'accurate', even to within a margin of error of ±3% (as many of them claim).
Most importantly, people tend to tell the pollster what they think the pollster wants to hear. Or those that have something to hide, will deliberately distort their view.
And then there is the convenience issue. Face it, how many of us are irritated when some pollster asks us for "just a few minutes" of our time (and we know that it will take way more time than that...)?
Anyway, it is always better to observe what people do in a certain situation, rather than ask them what they might do. However, that's not always possible or practical. So we are stuck with opinion polls for a while yet.
Here's an article from the highly experienced Harris Poll people titled "Margin of Error", When Used by Pollsters, Is Widely Misunderstood and Confuses Most People.
After outlining the various sources of polling error, the article reveals the large number of misconceptions that people have with the idea of "margin of error". Only a small percentage of people got the following one right:
Only a very small 12 percent of the public agrees that the words "margin of error" should only address one specific source of error, sampling error - as they almost always do
Under the question "So what?" the article points out:
- The use of words such as "margin of error" is controversial because they are often used when reporting telephone polls even though it is not possible to calculate a real margin of error.
- Pollsters need to do a much better job of explaining all the possible sources of error in their polls not just a theoretical sampling error, which does not take into account of other, potentially substantial, sources of
error;- The accuracy of opinion polls should be judged empirically by the accuracy and reliability of their findings, not on a theoretical basis when there is no way to calculate a real margin of error;
- Pre-election polls should continue to be trusted only so long as their final forecasts are reasonably accurate, not because they are theoretically "scientific" (since there is no means to establish that they are);
- The words "margin of error" should probably not be used at all in conjunction with polling results.
And how about the Harris Poll? What do they do with the phrase "margin of error"?
The Harris Poll has not used the phrase "margin of error" for many years.
People should understand how polls work, and how to interpret the findings, especially as we head into another US election.
See the 4 Comments below.
15 Jan 2008 at 11:38 am [Comment permalink]
Hi Zac,
I think your first paragraph overstates the case a bit. A 'margin of error' statement *can* be made in a way that is meaningful and accurate. Although " the percentage of people who believe Y is x% with a margin of error z%" is something that cannot be established by polling (or any other currently known technology), the statement that "the percentage of those selected at random by procedure p who will say that they believe Y when asked in script S is x% with a margin of error of z% at the c% confidence level" is a meaningful and provable statement. It means that if you repeatedly select a bunches of people by procedure p (which probably includes the specific number selected), and present them all with script S, then at the point where you ask whether or not they believe Y, the percentage who say yes will be between (x-z)% and (x+z)% in on average c% of the bunches. The fact that 90% of humanity with a margin of error of 20% don't have anything like the capacity for parsing the previous sentence is one of the (several) reasons why our species is doomed.
15 Jan 2008 at 12:17 pm [Comment permalink]
Hi Alan and thanks for your input. Good to hear from you again.
Something that I meant to mention in my tirade against opinion polls: How many people really understand the question they are being asked?
I see this all the time in education-based polling. In a field that is multi-variable, it is difficult, if not impossible, to determine what actually resulted in an increase (or decrease) in learning.
But when it comes to political polling, it is more straightforward.
Disclaimer: I also conduct polls, but I regard the results (like all educational polling) as having a margin of error of ±50%. They give a general indication of the lay of the land, and that's all. I am yet to see an educational poll that I am satisfied is "scientifically accurate".
27 Jan 2009 at 1:05 pm [Comment permalink]
Actually Alan,
The continued survival of the species certainly does not require that we all understand the intricacies of of the mathematics or the sciences.
It merely requires that SOME who find it interesting do. I have taken a survey and have found that to a +/- 3% margin of error, with a 95% level of certainty. you are an **** [edited].
Cheers.
27 Jan 2009 at 3:59 pm [Comment permalink]
Well, Erik, I didn't say "doomed to extinction" so "survival" is not necessarily the issue. And the capacity that I actually claimed was lacking was not a scientific one but rather one of perseverance at a challenging *linguistic* task.
Although it may well be true that our needs for invention and scientific progress can be met by devoting only a small fraction of the population to those tasks, it is also true that in a democracy the ultimate responsibility for political decisions generally rests with the population as a whole.
And to make those political decisions in a way that will not doom us to something very nasty requires an ability and willingness to analyse complicated propositions (and also indeed to understand quantitative statements - but I didn't say that in the comment you were complaining about).
cheers,
Alan