Do you find surveys and polls in the media to be misleading?

There are differences in different ways of conducting surveys and polls.
There are those where participants beforehand acknowledge that they will participate in them (which are, apparently, more reliable).
And there are internet polls where people decide if they join yes or no when they are confronted with them.
These have the disadvantage (or ‘danger’ of being less trustworthy) that not everybody has access to internet (not all groups of a society are represented in them then since you lose the participation of (some or more) elderly people, (some or more) less educated people, and (some or more) financialy less fortunate people).
This is what i more or less remember from hearing some statistics people talk about, so it not at all sure if this makes any sense, but it sounds quite logic to me.

As long as the sample of people is truly random, it can give a pretty good cross-section of genuine public opinion. The problems creep in, like @rebbel says, when the samples are not truly random. For example, “Are you for or against gay marriage?” would give very different results if it were hosted on the American Family Association’s website, versus the same poll on Out Magazine’s site. Yet, organizations of every stripe would present their poll data as being “the voice of the people.”

Polling organizations like Gallop and Roper, though, seem to cast a wider net when it comes to who they poll. They’re less likely to get, say, the entire congregation from one church and try to show that “100% of Americans go to church.” ;)

A bigger problem with polls that I can see is how they are interpreted. When the Health Reform bill passed, many people on the Right claimed that most Americans disliked the bill and it should never have been passed. What they chose not to play up was how many disliked the reform bill because it didn’t go far enough in a liberal direction (for example, having a public option, and being single-payer). If you tallied up the number of people who were happy with the bill or preferred more reform, they outnumbered the people who thought that reform was a bad idea, but you wouldn’t know that listening to the Right’s media.

And don’t get me started on push polls…

The polls are absolutely misleading and usually intentionally so, but not in the way that you’re thinking. Usually the sample of people polled follows a standard formula that can be verified. What cannot be standardized is the question.

“How upbeat are you about Obama’s future success?”
“How well do you think President Obama has delivered on his promises?”
“Do you think President Obama will ever deliver on his promises to the American people?”

I learned as a student and refined as a teacher the power of the question. Given time, skill and a certain ‘feel’ for testing (polling), you can ask a question in such a manner as to elicit an answer that will match your opinion.

“Did the kite fly high?”
“How high do you think the kite flew?”
Do you believe the kite could have flown higher?”

All inquire about how high in the sky the kite got, all work to the prejudice of the questioner.

Polls are largely without value on important issues in America today. There are people out there getting paid to poll and re-poll test groups that mirror the intended test group. Find a news source that you believe it fairly independent (NOT reflecting YOUR views) and make informed decisions. Barring that, read both sides and the polls they polled and see who is casting the biggest ‘spin’ shadow.

As a rule of thumb, my journalism instructor said that in order for a poll to qualify as an accurate reflection of a given population a minimum of 20% of the group should be polled. Ever since he gave us this number, I have become a skeptic of surveys and polls. I sincerely doubt their accuracy.

1400 respondents for a national poll is pretty standard. You need about a thousand to achieve statistical significance. Of course, few reporters tell us what kind of confidence we should have in the estimate.

It is getting more difficult to get a good enough response rate for your results to be reliable. Usually what happens is that a couple of thousand phone numbers are selected, and then they are called and called until someone answers. If you get a good response rate, the results of your poll have a very good chance of being an accurate reflection of the entire population.

It also turns out that internet polling is not nearly as unreliable as one might think. I’m not sure what their method is, but I do know they are getting results that are very consistent with polls taken using traditional sampling methods.

The problem is not have only 1400 respondents. The problem is that the media rarely gives us the information we need to evaluate the reliability of the poll. How was the sample selected? What kind of sample was it? What was the response rate? What were the margins of error in the analysis of the results? How was the poll conducted? Were professionals used or amateurs? And so on.

Another problem is that people don’t even know the right questions to ask about poll results and they don’t really understand enough about statistics to be able to evaluate the reliability of a poll. Thus we get questions like this one.

Common sense suggests that 1400 respondents is ridiculous. Yet math tells us it is a perfectly adequate number. People usually go with common sense over science, and that is part of the reason why we don’t solve our problems as efficiently as we could. That’s why we can joke that in America, we try every other solution to a problem first before we try the one that works. Scientists are often screaming from the sidelines, “go this way, go this way!” But politicians will do the common sensical thing because it gets them votes. Eventually they get around to doing what the evidence suggested years before,

I don’t mean to exaggerate, though. There are plenty of times when science is not clear. There is research going both ways, or in multiple ways. But there are a significant number of times where our advice is ignored because people don’t understand how you only need to survey 1000 people in order to be able to make accurate generalizations about what Americans think.

If you don’t believe me, go back to the last election and look at the number of people surveyed in each poll, and the accuracy of those polls. We knew who was going to win long before the election. We knew that because the polls told us so and because the polls were right. In the past, they have been incorrect, but we’ve figured out why that was, and corrected the mistakes. Polls are getting more and more accurate.

I think what makes polling unreliable is how questions are asked. How something is worded can make all the diffence in the world, even if the sample is large enough and diverse enough.

Thanks everyone, this certainly makes re-think the way I see polls and surveys.