The unserious criticisms of PFB research

[Bdub]

Bias in PolitiFact’s Ratings: Pants on Fire vs. False

The self-styled "liberal counterpart" to my criticisms of PolitiFact delivered a critique approximately as inept as expected. Karen Street apparently failed to understand the study topic and ignored many of the key descriptions in its ill-informed and summary dismissal of the study.

Street:

It added up all the Falses and Pants on Fire from PolitiFact’s beginning through the end of 2011, then took a percentage of the Pants on Fire of the False + Pants on Fire, then compared that percentage between Democrats and Republicans as a way to show how many more Pants on Fire are given to Republicans.

The above is sort of accurate, though I described a number of refinements of the data that Street fails to mention (I did not count obvious party on party claims, for example) and the study shows the greater likelihood of Republicans receiving a specific subjectively-determined rating from PolitiFact, not the mere difference in the number of "Pants on Fire" ratings.

A serious review does not miss those types of points. They are very important.

It appears that PolitiFact Bias’ Bryan White may have gotten the idea from Eric Ostermeier’s much lauded study where he also took a look at a similar percentage

Ostermeier's study is quite different from mine. His looks at the possibility of selection bias based on PolitiFact's descriptions of its process and the resulting numbers for a set period in time. My study looks at the bias in the application of a rating according to PolitiFact's descriptions to see if the resulting subjectivity affects one party more adversely than the other. I did not get the idea for my study approach from Dr. Ostermeier. It's as likely that I got the idea for my study by translating a set of golden plates revealed by an angel. Selection bias isn't really much of a factor in my study, the exception occurring if PolitiFact selects more ridiculous stories from one party specifically to try to appear more fair.

I’m not sure how he could place such confidence or reliance in such a number as being “a clear bias against Republicans” or how in the world could he expect that “neutral judgment ought to result in approximately equal proportions of unfair Pants on Fire ratings.”

Street's confusion appears to result directly from her failure to appreciate the core proposition of the study: All "Pants on Fire" ratings are subjective and unfair. It directly follows that any substantial difference in application of a subjective standard between one group and another (any two groups) indicates a bias.

Street does not accept the obvious conclusion:

(I)t could be liberal bias, it could be Republicans lie more, it could be the rise of Fox News or the Tea Party, it could be a lot of things—but I don’t think just adding up the Pants on Fire as a percent of all claims found false validates any conclusion of liberal bias. There's no PolitiFact "Achilles' Heel" here as White wants his readers to believe.

It could be liberal bias, indeed. But the data are not explained by "Republicans lie more" because lying is not a subjective thing. The "Pants on Fire" rating is explicitly defined in terms of ridiculousness, and the study takes pains to make that clear, though apparently it wasn't clear enough for Street. As for "Fox News or the Tea Party," neither is in charge of determining which statements PolitiFact finds ridiculous" are in addition to merely false. PolitiFact is apparently solely responsible. It's nonsensical to place the blame elsewhere. Suggesting that the "Pants on Fire" is actually an objective determination (contrary to PolitiFact's descriptions) represents the sole reasonable explanation of the data apart from the one suggested in the study. Arguing for that explanation, of course, obligates the critic to suggest the nature of the objective criteria and somehow excuse PolitiFact for describing its criterion as though it was purely subjective.

All he has found is that when PolitiFact National makes a False ruling on a Republican, they are 74% more likely to make the False Pants on Fire, than they are for a Democrat. This doesn’t mean they’re doing it because they are liberally biased.

The study doesn't say the results mean the PolitiFact National staff is liberally biased. The study says the results show a liberal bias. This one: When PolitiFact National makes a ruling on a Republican, they are 74 percent more likely to make the False "Pants on Fire" than for a Democrat. That bias, in turn, contributes to an inference toward the best explanation: That the staff of PolitiFact National is biased toward the liberal side. That inference is suggested in the study but left to the reader to make.

Street's criticisms represent a parade of her own mistakes. She makes no relevant criticism of the study, and that's why I don't clutter my regular blogs by responding to this type of nonsense.

[Bdub]

Up next, "Content in Reality" by (if we can trust blabbermouth Karen Street) Eric Levine. Levine posts under the pseudonym KnocksvilleE.

Levine:

On his website, Bryan describes this study as the "highest priority on the research list" and argues the study tells a "story of a strong anti-GOP bias at PolitiFact's national operation." I was intrigued.

A minor point: I mentioned the first study as the "highest priority"simply because of the abundance of data, not for any other reason. Levine's presentation suggests I made it the highest priority because of its compelling nature.

Eventually I moved the "Pants on Fire" study to the highest priority on the research list. I did so for the simple reason that PolitiFact has provided a good amount of data for this study.

It doesn't get much clearer than that.

Levine:

The study sets up a false dichotomy of two possibilities to explain the data. Either Republicans lie more often or Politifact is biased.

Levine's just wrong on the facts. I suggested only one explanation for the data we record: Bias against Republicans. I explained in the study why we offer no consideration for the "Republicans lie more" idea. Lacking an objective standard Republicans lying more shouldn't make any difference in how the numbers turn out. There's no false dichotomy set up in the study, contrary to Levine's claim. Based on a subjective standard the options are neutrality, bias against Republicans, or bias against Democrats.

As with Karen Street, we're just not getting the stuff of a serious critique.

Levine:

(Y)ou can only apply conclusions to the the few "newsworthy" statements PolitiFact has encountered. Republicans telling more ridiculous lies than Democrats could explain the data, but it is a fallacy to say it therefore must be THE explanation.

Levine had just finished quoting me accurately "It's not an easy statistic to explain without liberal bias."

So, maybe I'm not committing the fallacy he implies I'm committing?

Maybe Levine should just show how easy it is to explain the data without liberal bias.

Other explanations could also explain the data. For instance, it could be the case that the loudest and most media-covered Republicans are more likely to not only lie, but tell ridiculous lies, than the loudest and most media-covered Democrats.

Explanations such as the above could work if an objective standard accounts for the "Pants on Fire" rating. I emphasized in the study that the conclusion is based on treating PolitiFact's "Pants on Fire" rating according to the subjective definition it receives from PolitiFact. A careful critique would acknowledge that.

If Bryan wants us to accept his bias explanation by process of elimination, he needs to actually eliminate all other possible explanations first.

If I'd intended to use the process of elimination I probably would have mentioned it. The study implicitly invites the reader to use an inference toward the best explanation. If "ridiculous" claims are those that are absurd or made repeatedly despite correction then why doesn't PolitiFact define the rating that way? Is that the best explanation under those circumstances? Levine finds that I have failed to obtain an objective I did not have in mind. Color me unconcerned.

His study further attempts to prove that PolitiFact's justifications for assigning "False" verses "Pants on Fire" ratings are completely subjective.

Levine misspeaks again.

There's no attempt to prove that the "Pants on Fire" rating is completely subjective. Rather, I made the attempt to find an objective standard regardless of the lack in PolitiFact's definition of "Pants on Fire." While I did not find plausible evidence of any such standard, nor did I conclude that one did not exist. Instead, I used the lack of evidence of an objective standard to strengthen the inference that no such standard exists. If Levine would like to suppose otherwise, let him quote me.

While I will admit there can be some subjectivity in what should be considered ridiculous, there are other claims where the possibility of subjectivity is trivial. If practically everyone who knows a claim is false finds it ridiculous, does subjectivity even matter? However, none of this matters if PolitiFact is actually using some kind of objective criteria to justify which statements get a "False" or "Pants on Fire" rating.

Determining what is ridiculous "could" entail "some subjectivity." No kidding.
This question from Levine is key: "If practically everyone who knows a claim is false finds it ridiculous, does subjectivity even matter?"

Yes, absolutely subjectivity matters if everyone finds a claim ridiculous if the criteria for finding it ridiculous are subjective. Under that condition, the finding by almost everyone is fundamentally a subjective finding. Obviously.

What is Levine trying to argue? That the appeal to popularity isn't a fallacy if something is popular enough?

Levine goes on to follow a rabbit-trail regarding my search for objective criteria in PolitiFact's "Pants on Fire" ratings. One of the most common threads I found in the "Pants on Fire" ratings was PolitiFact's condemnation of fear-mongering. I accepted fairly minimal evidence of special appeals since finding them expressed plainly produced, as one might expect, a spare set of results. In the study I used PolitiFact's condemnation of Sarah Palin's "death panel" comment as an example of employing the criterion, the example intended to solidify the concept for the reader.

Levine:

However, nowhere in Politifact's justification of Sarah Palin's "Death Panel" claim did the word "fear" come up. In fact, nowhere can I find the word "fear" in any definition of the word "ridiculous." I'm not sure why he used that as a criteria [sic].

I counted fear mongering as a potential criterion because I could find signs of the justification in PolitiFact's stated reasoning. Restricting my search to particular words as Levine did would surely have resulted in much less significant patterns of justification. I worked to give PolitiFact the benefit of the doubt. I used Palin as an example because it was well known. It was, in fact, not one of the stories in the sample I studied. But PolitiFact did implicitly suggest that the "death panel" claim played on people's fear:

Palin's statement sounds more like a science fiction movie ( Soylent Green , anyone?) than part of an actual bill before Congress. We rate her statement Pants on Fire!

If I'm reading too much into it, fine. That just makes it harder to find signs of an objective standard in PolitiFact's "Pants on Fire" rulings.

(N)owhere does he explain why he chose it. Likely, this means that Bryan merely guessed at possible random criteria and, when he could not find one, he assumed none existed.

As I described in the study, I started with PolitiFact's definition and found nothing objective and measurable in the term "ridiculous." Wanting to give PolitiFact the benefit of the doubt, I studied a randomized sample of PolitiFact stories with either a "False" or "Pants on Fire" rating. I looked at each story (carefully) and examined the reasoning behind the ratings. Based on my observations, I grouped the the similar ones together and calculated their percentage of the sample set. I could have figured out a much easier way to assume that no objective criteria existed.

Levine finally gets around to the central conclusion of the study and fails to offer a clear presentation of it. He quotes me accurately: "If no such criteria exist then the results reflect bias in the PolitiFact organization."

And he replies to that one line:

If you don't think the conclusion follows from the premise, you are right.¹ As I noted before, there are obvious examples of things that are ridiculous. It could be the case that "the loudest and most media-covered Republicans" are saying more of these obviously ridiculous falsehoods than "the loudest and most media-covered Democrats".

Is Levine new to the distinction between "subjective" and "objective"? His answer suggests as much.

If no objective criteria exist for a "Pants on Fire" ruling then the criteria are necessarily subjective. A set of subjective preferences favoring Democrats over Republicans shows an anti-Republican bias. The conclusion follows from the premise in the context of the study's argument. I can arrange the premises and conclusion in the form of a deductive syllogism for Levine if it still isn't obvious.

Levine:

Note: Karen Street also has a great review of Bryan's study. It is worth reading.

Neither review is serious.

The link to Levine's review:
contentinreality.blogspot.com/2012/08/politifact-bias-still-making-same.html