Pre-election confessions of a political hack (Part III)
by David Benzion · 10/29/2008 7:00 amPREVIOUSLY–On early primary contenders & They really, really hate us.
Reminder: While I in fact work as a Senior Research Analyst for a Republican political polling firm whose clients this cycle include a variety of Senate, Congressional, statehouse and ballot-initiative campaigns in Colorado, Michigan, Florida and Texas, the opinions expressed here are my own, and do not necessarily reflect those of my employer or clients.
——————
Today I ease up on my conservative brethren and focus instead more on process; to wit, a question posed earlier this week by Wagonburner–
Do pollsters really bias their published polls toward what they think their sponsors want to hear?
The simple answer to that question is “No, they don’t.”
But reality is obviously much more complicated than that, so let’s dive right in.
DON’T MISTAKE CRAP FOR MALICE
Polls are high-risk, high-reward endeavors, offering the promise of secret insight into the future while simultaneously threatening to deceive and mislead.
Folks who make a living in polling have their disagreements about proper sampling/turn-out models, questionnaire wording/crafting, as well as simply fundamental assumptions about what to ask and what conclusions to then draw from the data collected. It is little surprise that to the average outsider it might look like we are just making it all up as we go along.
Well, for starters I can speak for myself and the shop I work in.
We defiantly don’t make things up. In fact, as my boss likes to say…
“The pollster is the only person in any campaign whose job it is is to tell the truth.”
Think about it. Candidates have spouses, press flacks, media gurus, their parents, drivers, lobbyists, old college roommates, interns and a motley collection of other peripheral weasels to offer flattery and make themselves feel good. The pollster has to be the one guy who can slap ‘em upside the head and ground the whole circus in reality.
And high-quality, PRIVATE pollsters do exactly that.
But there’s the rub.
Let’s do a little common-sense thought experiment.
Consider for a moment that polls are routinely and constantly being conducted on behalf of the smartest, most successful and powerful individuals, corporations and special-interests in our society.
These things don’t come cheap–a typical statewide “benchmark” poll for a campaign can routinely cost somewhere in the range of $30-35k.
I can promise you that smart, successful, powerful people don’t cut $35k checks so they can be handed an ambiguous and unreliable pile of crap that doesn’t accurately reflect reality and offer actionable insights into what to do in response.
Feel-good flattery is nice, but there are lots of cheaper ways to get it.
Unfortunately (for you), the public almost never sees or even hear about these private polls. After all, a big part of the $35k value for the sponsor is knowing something that everybody else doesn’t.
The public is instead fed something far less impressive.
- Robo-polls by polling-start ups trying to draw attention to themselves as a form of marketing;
- “Media-sponsored” polls paid for by a newspaper or TV station that wants to generate “news” it can then report;
- Polls conducted for mass-profit, as a means for the pollster to sell “subscriptions” to frightened little old ladies in Omaha;
- Polls conducted by university academics in tweed jackets with elbow pads who never have to step up to the plate and play with the big boys (present company excluded, of course)
.
It’s not that these polls are worthless; some are perfectly well done.
But you get what you pay for. I’ve seen, time and again, the type of quality polling done at shop where I work (and at other private firms, both Republican and Democrat) and trust me, it is a whole different species than the types of polls you are being hit with right now.
To take one simple example–if you owned a Lexus dealership, and wanted to know what types of sales to expect next quarter and who to target with your limited advertising budget, would you spend money researching the opinions of… ?
- Anyone who breathes
- Anyone who has ever seen a car
- Anyone who owns a car
- Anyone who owns or has ever owned a luxury automobile
- Anyone who owns or has ever owned a luxury automobile and expresses an interest in purchasing a new car in the reasonably near future
- Anyone who owns or has ever owned a luxury automobile and expresses an interest in purchasing a new car in the reasonably near future, with an over-sample of past and current Lexus owners
A pollster might have a legitimate reason for choosing option 4 over option 5 versus option 6… but on what planet do options 1, 2, or 3 make any sense? I’m trying to sell a very expensive automobile, and a particular brand at that… what do I care what all these other people think about my business?
Rule of thumb–any public political poll of REGISTERED voters is about as useful as a local tourism board’s restaurant guide. Yes, there is some information there, but it doesn’t actually tell me much.
Look for surveys that make at least some effort to focus in on LIKELY voters (of course, it’s good to know how they define likely voters; good luck with that).
But asking only likely voters costs more money… so now we are back to the “private” vs. “public” polling problem.
I will say this–I don’t know that I’ve ever seen an example where a public polling outfit is intentionally skewing results. It’s just that you get what you pay for.
In other words, don’t mistake crap for malice.
Mind you, perhaps the biggest problem with the public’s perceptions of polls is the fault of the media–those whose responsibility it to try to understand what a survey says and what it means for the future. Do you think the media does a good job interpreting reality in every other aspect of life and presenting it to you in a coherent fashion?
Well, OK then.
So what’s the average Joe/Jane supposed to do?
Ironically enough, the Houston Chronicle’s Richard Dunham had a pretty good article yesterday offering some tips I can whole-heartedly endorse. Other good resources include websites like RealClearPolitics.com, MysteryPollster and 538 (read this New York Times article for some important background). Locally, keep an eye on Professors Taylor and Carnahan over at the University of St. Thomas. Michael Barone, Jeff Birnbaum and Stuart Taylor are straight-shooters. Gallup is a gold-standard. While well to the Left, Democracy Corps is a very well-run outfit that needs to be taken seriously by those who lean-Right. Pew is a sober-minded place.
And, as always, trust that what I say has been handed down like the 10 Commandments to Moses on Mt. Sinai.
That, and $35k, will getcha something.
TOMORROW–MAYBE I WRITE SOMETHING IF I HAVE TIME
Filed Under Front Page · · · · 
Print This Post
··
Comments
21 Responses to “Pre-election confessions of a political hack (Part III)”Search
Our Sponsors
Poll
Loading ...
David, Thanks for the insight, these are great.
Quite often you’ll see the MSM touting their own poll or cherry-picking another poll to buttress their own pre-conceived narrative. It’s like quoting a politician out of context and most people don’t have the time or inclination to search for the poll online to find that it didn’t really say what it purported or that the methodology was shoddy.
The other tip about “retail” polls is when you hear reports about a candidate acting contrary to what the latest generic public polls are indicating. It often, but not always, indicates the candidate’s internal high quality polling reflects substantially different results. Example: “Candidate X today redoubled efforts in this Midwestern state even though polls have shown him to be leading by double digit margins.” His internal polling data may very well show him to actually be in a dead heat or the marginal supporters to be very soft.
BTW; The PDF file @ “needs to be taken seriously” is VERY interesting.
David, I’m certainly no expert in polling, and would not want to criticize your profession (as I wouldn’t want want you to criticize how I serve fries - yuk yuk) but as someone who was called by your company during the Van Arsdale-Fletcher campaign, some of the questions (in my opinion) seemed to be a little leading.
To use your car sales analogy, I was asked questions such as was I aware that Volvo often does not show up for its monthly maintenance inspections, but that Lexus will.
What I wasn’t asked was whether I knew that Volvo was given a full inspection during regular oil changes at the dealership.
But hey, it was effective and that’s politics.
#3 SuperDave
Yeah, you are right. Any conservative reading this blog who is genuinely interested in winning elections should read that poll file.
The most interesting thing (to me) about the .pdf SuperDave mentions is on pages 20 & 21. Note how many people think that economic conservatives have too little influence in the Republican party and note how many people think that social conservatives have too much influence in the Republican party.
Let me start this by saying that I personally oppose abortion. At the same time, I think it’s an issue that has done more damage to the electability of Republicans than any other. Bottom line, most people don’t want the federal government involved in abortion legislation. It’s one of those things that crosses into the arena of governmental involvement in religion, which makes people deeply uneasy. I don’t say this to offend any pro-lifers. I’m one of you. I say this merely from the perspective of getting Republicans elected.
digitaldon37– great question, good for you; I promise to respond when I have time.
So, to paraphrase Hanlon’s razor, “Never attribute to malice that which is adequately explained by
stupiditylaziness/being a cheapskate”?#6 dude
The sad part is Roe v. Wade injected the Federal Gov’t into the debate, where it didn’t belong. The issue should be placed back into the hands of the individual states (as with many, many other things which have been usurped by the Feds).
The entire Greenberg/Quinlan/Rosner survey is a good read.
It’s points out that many republicans think their party isn’t “conservative enough” to win, but that independents think it is “too conservative” to vote for.
When I recall the parts of Reagan’s message that were most appealing to democrats and independents, it’s quite a contrast to more recent candidates.
9. WB, there’s been legislation introduced in multiple sessions of congress to remove the issue from federal jurisdiction. It has never made it out of committee.
What I find interesting is the weighting of the polls. I also find interesting that the after voting polling always seems to favor Democrats. This is often explained by pollsters saying that Conservatives do not either want to or have the time to chat with them. I would think that would also apply to telephone polling. Also, the polling always seems to show the Dems way ahead right up until the final weeks of the election cycle. The polls always seem to tighten to within the statistical margin of error. Maybe conservatives are just then having the time as a group to spend on the election? I do not know as I am not a pollster, just a Texan who spends a lot of time paying attention to politics.
IowaHawk has a great post about this topic. Using the old statistical standby tools balls and an urn.
Iowahawk on target and funny while doing it as usual.
My husband is one of the VMG. He doesn’t pay attention because he believes that most politicians are crooks. Not pestering him this time to actually pay attention because he had already decided to vote for McCain during the primaries (driving me nuts as I felt anyone BUT McCain during the primaries based on his actions since he lost the nomination to Bush). He simply said to me that as far as he is concerned McCain is a genuine war hero and deserves his vote for that reason alone. Simplistic but when the choice is Obams, an easy choice for me to make as well.
David,
Can you touch on how you account for the problems iowahawk touches on in my comment above?
Pumpkin:
I think you’re referring to Exit Polls (as opposed to “after voting polls”).
Here’s how that works most of the time,
VNS (Voter News Services, or whatever name they’re using THIS year)
has gazillions of precincts to poll, and not enough time, people, money, etc., to poll them.
Say they have 100 precincts. In the past, they might pick 20 out of the 100 to poll in. They would cherry-pick 5 that are strongly for Candidate A, 5 that are expected to be strongly for Candidate B, and 10 that are tossups.
So, you’re already down to 20% of the total. (In reality, the number they pick to poll is much smaller.)
Then you have the response rate. Even if 50% of the people talk to the poll taker, you’ve now dropped by half–reality is fewer than that talk to them. Ever sit in your car and watch a poll taker stop people? Most people walk right on by.
So, now you’re at a fraction of a fraction of a fraction, etc., of the voting public.
50 people might represent 50,000 voters, or some such number.
That is why in 2000 (among other reasons) Florida was such a mess. The Exit Polls showed one thing, and the actual numbers were different.
VNS had egg on their faces, and blew up their machinery entirely after that election. They tried again in ‘04, and the system blew up again.
Incidentally, this outfit is where ABC, CBS, NBC & FOX all get their exit polls from.
The AP conducts it’s own exit polls. Some of the media outlets may do their own stuff now, but nobody can afford their own entire operation. There’s just too many voting places in the U.S.
And that’s the reason why the exit polls aren’t conducted everywhere.
You might find a Houston Chronicle operation running in the City, when there’s something like another stadium referendum running, but that’s all their asking about.
Oh yes, one more area why Exit Polls are error-prone, people lie!
If a black person is taking the info down, and a white person is the voter,
how likely might it be the white guy say he voted for Obama to the black poll-taker? Much more likely than a white/white pairing.
You also mentioned Weighting:
All polling has some bias.
That’s what weighting is for.
To smooth out the bias(es) that polling has.
Here’s a very simple bias that can be introduced, that pollsters have to account for…on Monday nights, during Football season, who is more likely to answer the telephone for a poll? The 40-something wife, or the 40-something husband who’s watching his beloved Green Bay Packers whomp on the Cowboys?
No brainer, right?
So, when polling is done for the night, instead of an intended split of 50-50 for gender, you have 48(male)-52(female).
Using your tabulation program, that gets evened back out.
Other stuff shifts too, by necessity, of course.
Weighting is not lying, fudging or changing the data.
It is adjusting things to reflect known, accepted, provable statistical reality.
A recent poll in a midwestern state for their Senate race was shown to be loaded with over 50 white Evangelicals, and not weighted back to something closer to reality. The result? Their poll showed a much higher total for the GOP candidate. Adjusting that through weighting would ease that back to reality. It would depress the GOP vote total, sure, but it wouldn’t give that candidate a false sense of security.
Polling is all about “getting it right”.
Win or lose, if you’re accuracy isn’t sufficient, you’re not doing your job right.
David Benzion’s statement about telling the truth is 100% dead on.
You hope for the random polling to get it right for you, but you weight to adjust for things like Monday Night Football.
Polls Smolls…. The only poll that counts will be taken on November 4….
Simple
18 SS
And the $35,000 ones.
:>)
So if the polls are showing BHO to lead by as much as 15 points in some polls why is BHO still humping it in the red states? One would think since they are so confident BHO has won, and they keep telling us that, he could sit back in his high rise in Chicago and sip expensive adult beverages. Maybe it isn’t in the bag like the BHO camp and the MSM would like for us to think.
This reminds me of my statistics classes in graduate school.
So the $35,000 polls “at the end of the day”, say the same thing the K-Mart Special polls do, so what do you get for 35K, more details?
I’m not a professional pollster, but I do have a sense of smell and it tells me the polls are wrong. If they are, there’s some serious ’splainin to do, if not I will humbly apologize.