Three Things to Remember About That Study of Fox Viewers Who Watched CNN

A report published in April 2022 showing short-term effects (thus far) in viewers of the right-wing Fox network as greeted with widespread coverage and, in some cases, familiar distortions of its findings.

In the study by David E. Broockman and Joshua L. Kalla, Fox viewers were tasked with watching CNN news broadcasts on weekdays in late August 2020 and September 2020. According to the researchers:

Fact Check

Claim: Fox viewers transform after watching CNN

Description: A study conducted in 2020 had Fox viewers watch CNN. After exposure to CNN, Fox viewers were less likely to hold certain beliefs. However, there was no significant increase in CNN viewership among Fox viewers.

Rating: Partially True

Rating Explanation: The study’s findings indicate that viewers can be influenced by exposure to different news networks, but the claim that Fox viewers ‘transformed’ after watching CNN is overstated.

As might be expected from a sample of Fox News viewers, relative to the average American, the individuals we selected for the study were, on average, older (average age of 54), whiter (95 percent), more Republican (92 percent self-identified as Republican), more active voters (90 percent voted in the 2016 general election), and more frequent Fox News viewers (self-reported watching an average of 840 minutes per week).

Out of 763 individuals who took part, 304 of them were paid $15 per hour to watch CNN during the testing period. Following that period, 727 participants also took part in a survey conducted in late November 2020.

“Had there been a Democratic incumbent president at the time, we would have used the opposite design since we would still expect both CNN and Fox News to engage in partisan coverage filtering,” the researchers wrote. “We do not assume that either network is more objective, simply that they have different partisan slants.”

A December 2021 poll conducted by the Washington Post and the University of Maryland found that the Fox network made up a bigger part of viewers’ media diet than CNN:

Among Democrats and Democrat-leaning independents, a variety of sources — CNN, CBS, NBC, MSNBC, NPR, the Times, The Post — were identified as a main source of news by at least 3 in 10. Among Republicans, though, only two were: local television and Fox News.

According to Broockman and Kallas’ study, CNN devoted more coverage to issues related to the COVID-19 pandemic and election integrity during the monitoring period, while Fox focused more on stories related to ethnicity and protests:

CNN spent 21,244 words discussing “Trump’s failures to protect US & his supporters from COVID-19,” while Fox News spent a relatively scant 2,086 words on this topic. Likewise, CNN spent 10,251 words discussing the severity of COVID19 (e.g., the threat posed by long COVID), while Fox News devoted only 709 words to information describing the virus’ severity. By contrast, Fox News devoted a great deal of airtime to downplaying the severity of COVID-19 and the efforts Donald Trump had undertaken to protect Americans from the virus.

“In other words, partisan media does not simply remind people of certain beliefs they already hold (priming),” Kallas and Broockman wrote. “It leads viewers to learn about a biased set of facts, which appears to have consequences for their attitudes.”

The study found that after exposure to CNN broadcasts, Fox viewers — at least temporarily — were less likely to believe that supporters of Joe Biden, then the Democratic Party’s presidential nominee, would be happy to hear that police officers got shot, and conversely more willing to believe that other countries were more successful in containing the pandemic at the time.

But in seeing blogs and news outlets rush to cover the study, we found three important issues to note for readers:

1. It Is a Pre-Print

The study is posted right now on OSFPreprints, one of several sites devoted to hosting studies that have not undergone the peer review process. Kalla confirmed to us via email that the study is currently under review.

That is not, in and of itself, a critique of Broockman and Pallas’ work, but it is a distinction that is often ignored in press coverage of scientific study. As we have noted in the past, sites that host pre-prints have policies — particularly regarding research on COVID-19 — advising news outlets on how to approach them:

These are preliminary reports that have not been peer-reviewed. They should not be regarded as conclusive, guide clinical practice/health-related behavior, or be reported in news media as established information.

But as so often happens, several news outlets — whether posting “analysis” pieces or aggregated recaps — failed to make that clear to readers.

2. Beware Overstatements in Headlines

One of the outlets that failed to tell readers about the study’s pre-print status, the Independent, promoted its coverage by saying that the Fox viewers were “transformed”:

However Kalla and Broockman themselves wrote that they found “no significant increase in favorability towards CNN.”

They further found that when measuring regular television viewership (as opposed to viewership data reported during the survey), there was “no statistically significant increases in CNN viewership” in the period following subjects’ participation in the study.

“We also see little sign of long-run changes in Fox News viewership,” the researchers added. “Overall, then, there is some evidence that some participants kept watching CNN after the treatment ended, but we do not find evidence of large long-run change.”

3. The Study’s Limitations

In a section that will likely go ignored as their work is further summarized and spread online, Broockman and Kallas noted that their study had limitations.

“The effects we observe may not be generalizable beyond the sample of participants willing to be paid to watch a different news network,” they wrote, adding, “Our experiment was not well-positioned to measure broader impacts of partisan media, such as for outcomes including what other media sources cover, donation behavior, or elite behavior, themes investigated in other research.”

The researchers also pointed out that it is possible that paid participants in the study watched CNN as per their agreement while also recording Fox to watch later.

“We did not explicitly instruct treatment group subjects to refrain from watching Fox during the incentivized period,” they said.

This study in particular also covered only television viewing habits; whether that affected their online reading or viewing habits was not covered.

“Our experiment was not well-positioned to measure broader impacts of partisan media, such as for outcomes including what other media sources cover, donation behavior, or elite behavior, themes investigated in other research,” Kallas and Broockman wrote.