(This article first appeared in Broad + Liberty.)
The news headline was a campaign manager’s dream — at the beginning of October no less — just as ballots were beginning to be mailed.
“As Democrats fight to keep control of the Pa. House, poll shows their agenda is popular,” the Spotlight PA headline declared on Tuesday.
In the story, the reader is told “Under Democratic control, the Pennsylvania House has advanced policies expanding universal background checks for guns, providing discrimination protections to people based on sexual orientation, and raising the minimum wage to $15. Using these examples, the poll asked 800 likely voters if they approved or disapproved of this policy agenda.
“Of those surveyed, 63 percent said they strongly or somewhat supported the agenda, while 32 percent somewhat or strongly disagreed. Five percent said they were undecided,” the story said.
Readers might also be interested to learn how popular or not the House Republican agenda is. Or, given that the question dealt with the majority party in the Pennsylvania House, it might seem a logical question to ask about the agenda of the Pennsylvania Senate, led by Republicans.
But the poll failed to ask either of those questions. There was no curiosity about how popular the GOP legislative agenda is.
That problem and others have Republicans grumbling.
Mark Harris, the co-founder of Coldspark, a Republican-oriented political consulting firm in Pittsburgh that does its fair share of polling, said the question has all kinds of problems.
“This is completely loading the gun on describing their policies. In issue-advocacy polling the wording is a huge way to bias the responses and it’s pretty clear they got the biased outcome they wanted,” Harris said.
The question also presents an issue of what pollsters call the “double-barrel bias,” which occurs when you ask someone about multiple issues, but only give them the ability to provide a single answer.
“The key is to ask about individual aspects,” Harris added. “Rather than saying the ‘Democratic agenda of X’ they should test each individual issue separately and without the partisanship.”
Furthermore, poll respondents are notorious for providing answers that can seem contradictory. For example, it’s completely within the realm of possibility for a majority of respondents to approve of one party’s agenda while also approving of another — even if the two parties are fighting against each other.
Some of these contradictions have become so persistent over the decades they’re studied in political science. Fenno’s Paradox notes the contradiction that poll respondents usually disapprove of Congress, but will also approve of their own congressional representative. Or, respondents will express general negative views about the state of public education, yet simultaneously give their own neighborhood school high marks.
Respondents will say they value compromise, but also say their leaders have compromised enough. They’ll say they want political experience in candidates, but that they also love “outsiders.”
All these examples illustrate why knowing the approval numbers for the Democratic caucus and its agenda is mostly unhelpful unless contextualized by a similar question focused on the Republican agenda.
The House GOP caucus had both barrels blazing when asked about the poll.
“This was a Democrat push poll with questions clearly designed to gain a favorable outcome for Democrats,” Jason Gottesman, the spokesperson for the House GOP caucus, told Broad + Liberty. “The agenda and platform laid out by Pennsylvania House Republicans was not polled, no context or details were provided about what House Democrats’ policies would do, questions about anything Republicans would support were framed in the least favorable light, and our caucus was never provided the opportunity to provide a different perspective for the story written about the poll. It appears this is another sad example of the media acting as a one-sided political advocate and contributes to the public’s growing distrust of mainstream media outlets.”
The term “push poll” has become a pejorative to describe any poll someone finds unlikeable. By definition, push polls are not concerned about publishing results as opposed to manipulating the respondent’s opinion so they might possibly change their mind when filling out their ballot — it’s campaign telemarketing disguised as a poll. Additionally, push polls don’t identify who is calling. Gottesman’s characterization is inapt, in those regards.
Yet the poll still contains other problems. In one instance, the “question-order bias” shows itself.
The online survey company Survey Monkey illustrates question-order bias with this example: “[I]f you ask students to answer a very difficult math problem first, then ask how much they enjoy math, they may be tempted to rate their interest lower if they struggled to solve the math problem.”
In the Spotlight PA poll, it asks if a respondent agrees with the U.S. Supreme Court’s overturning of Roe v. Wade, to which 54 percent said they disagreed. Then it asked if the “Supreme Court ruling that Presidents have immunity from prosecution for official acts they commit as President was the right decision or the wrong decision,” to which 53 percent again said they disagreed.
Now that a respondent has disagreed with the U.S. Supreme Court two questions in a row, the poll then asked, “Would you support or oppose a proposal to remove lifetime appointments for Supreme Court justices and replace them with a single eighteen-year term?” Forty-eight percent said they supported the idea.
The next question asked, “Would you support or oppose creating a binding code of conduct for Supreme Court justices?” Fifty-nine said they supported.
The SCOTUS decision to overturn Roe v. Wade is more than two years old, and the pollsters certainly knew by this time that public opinion was tilted against the decision. By asking that question before it asked about term limits for justices, it could have biased the response. The poll did not ask respondents how they felt about less controversial rulings from the Supreme Court.
Another element of the poll received its own story on Monday, saying a majority (53 percent) of Pennsylvania voters disapproved of a pilot program spending $100 million on school vouchers. The policy was supported by Democratic Gov. Josh Shapiro last year.
The conservative political nonprofit Commonwealth Partners took aim at the story in its daily newsletter.
“Spotlight PA has a new poll purporting to show that most Pennsylvanians oppose school vouchers. The story on the poll even takes aim at other polls showing support for school choice by criticizing how the polls frame the poll question,” Commonwealth Partners said in its News and Brews email.
“But Spotlight’s incomplete framing of the question is also questionable. ‘A proposal in Pennsylvania would use $100 million in taxpayer money to establish a voucher program that sends public school students to private or religious schools. Would you approve or disapprove of such a proposal.’ First, the “program” wouldn’t send any students anywhere. Parents and families would. Second, vouchers, as debated in Pa., would also be able to be used for special education services fees in addition to tuition at non-public schools. Third, the proposed vouchers would be available to students trapped by zip code in the worst-performing schools in the commonwealth, a fact Spotlight conveniently omits.”
When contacted for comment, Spotlight PA sidestepped several pointed questions about the polling methods and question construction.
“Spotlight PA was invited to contribute a handful of questions to a poll of several states conducted by the highly rated MassINC Polling Group,” said Spotlight PA deputy editor Sarah Anne Hughes. “The question aimed to gauge whether voters approve or disapprove of the Democratic majority’s actions.”
Spotlight PA did not respond to a follow-up question asking whether the outlet failed to reach out to the House GOP caucus, as Gottesman claims.
Similarly, MassINC, the polling company, did not answer specific questions about the construction of the poll questions and potential biases. The polling firm conducted similar polls in collaboration with a handful of other newsrooms, and it did provide a link to all of the polling results (including crosstabs) as well as resulting news stories, which have been aggregated into this one webpage.
The polling indicates the project was “made possible through a grant from the Knight Election Hub.”
