By Kate Peglow and Laila Tabatabai

As a part of our internships this summer, we explored how scientific evidence is used in the media. As we read countless articles and press releases, we encountered multiple claims purportedly supported by science that misinterpreted, oversimplified, or otherwise misused science and were consequently misleading to readers.

Many readers may not understand that few safeguards are in place to make sure that science is correctly interpreted and used once it is published. This prompted us to help consumers be better equipped to critique the claims made in the media.

Below, we list 10 common red flags in media stories that could tip you off to poor science usage, along with examples, so that readers can be on the lookout.

1.    The article doesn’t explain that animal or in vitro studies do not necessarily predict human outcomes.

• 7 Foods that Fight Back: Immune System Boosters | Everyday Health – This article claims that quercetin in onions can act as an antiviral to boost the immune system, but the referenced study was an in vitro study (i.e., carried out in a test tube rather than in a human or animal) of quercetin’s effect on laboratory-grown cells, which the article failed to mention. Animal and other experimental evidence can be very useful when first examining the usefulness of a therapeutic, but there’s generally a long way to go before one can make clinical recommendations.

2.    The article doesn’t provide links or citations to the studies referenced to support its claims.

• Feeling 'Hangry' may be Rooted in Science and not all in our Heads: New Study | Fox News – This article reports recent findings of a study that explored how emotions, including irritability and pleasure, change when one is hungry. The article does go into detail about the study’s methodology and limitations. However, while it mentions the journal’s title, the article does not provide a full citation or link for the reader to refer back to the study, something that should be standard practice these days. Why put the burden on the interested reader to sift through the scientific literature?

3.    The article doesn’t present statistics to support its claims.

• Worst Foods and Drinks for Brain Health, According to Nutrition Experts | Fox News – This article provides a list of the worst foods for brain health. One such food, it says, is fruit juices and other sugary beverages. The article quotes a dietitian who states that “the consumption of sugary beverages is blamed for lowering total brain volume and hippocampal volume.” However, the article does not explain by how much brain volume is reduced and if this value is clinically significant.  

4.    The article gives numbers that aren’t statistically significant, and that’s not explained.

• Eat Dinner Earlier, Dodge Cancer? | ABC News – The article states that the risk of cancer changes depending on the amount of time between dinner and bedtime, and that breast and prostate cancer are “more likely in people who eat just before bedtime, compared with those who leave at least two hours between dinner and going to sleep.” The referenced study did find a reduction in risk for prostate and breast cancer when more than two hours elapsed between dinner and going to sleep compared to when one hour or less elapsed. The association, however, was not statistically significant for breast cancer, which the article fails to mention. Such statistical testing is necessary to distinguish between findings that are likely true and those that are more likely due to chance. If the results are non-significant, then this should be explicitly stated in the article to prevent the reader from being misled. 

5.    The article reports relative differences when absolute differences are small.

• Evusheld to Prevent Covid-19: There Won't be Nearly Enough for Americans Who are Eligible | CNN – This article reports on a new drug that helps protect those who are immunocompromised from COVID-19 infection. The article states that those who received the two doses of EvusheldTM “saw a 77% reduced risk of developing COVID-19 compared to those who received a placebo.” While this claim is in fact true, more context is needed to understand whether it is actually important clinically. The referenced study reveals that the risks over about 3 months of acquiring COVID-19 for the experimental and placebo groups are 0.2% and 1.0%, respectively. (Dividing 1.0 into 0.2 yields the 77% reduction, with rounding error.) But this also means the absolute risk reduction (the difference between the two infection rates) between the experimental and placebo groups is only 0.8%, which the article fails to mention. This in turn means that only 1 of 125 people who receive the drug will benefit from it. Consumers need this information in order to decide whether to take the drug.

6.    The article makes a big deal out of small differences.

• Foods to Eat that Help Your Metabolism | WebMD – This article claims that eating ginger can help boost your metabolism, citing findings from a study showing that “a hot ginger drink with breakfast lowered feelings of hunger and had a strong thermogenic (calorie-burning) effect.” The referenced study did show a thermogenic effect of ginger, but this effect only increased calorie burning by about 43 calories per day. While this increase in burning effect is statistically significant, even if confirmed in other studies, 43 calories is less than 3% of one’s daily caloric intake (based on a 2,000-calorie diet).

7.    The article uses misleading graphs.

• The WHO Estimated COVID-19 Mortality at 3.4%. That Doesn't Tell the Whole Story | Time Magazine – This article makes the claim that COVID-19 is more comparable to the seasonal flu than to previous coronavirus outbreaks by including pie charts that depict the case fatality rate of COVID-19, the rate of death among people who contract COVID-19, next to those of the seasonal flu, Severe Acute Respiratory Syndrome (SARS), and Middle East Respiratory Syndrome (MERS). By focusing only on case fatality rates, this article doesn’t tell the whole story. The charts do not account for the relative severity of these infections, their long-term effects, how common each of the infections has been, and disproportionate impacts on certain populations. 

8.    The article doesn’t explain that the findings relate to some indirect measure, and not the true outcome of interest.

• New Phase 3 Data Show Positive Correlation Between ADUHELM™ Treatment Effect on Biomarkers and Reduction in Clinical Decline in Alzheimer’s Disease | Biogen – In a press release about the new Alzheimer’s disease drug AduhelmTM, Biogen, the drug’s manufacturer, claims a reduction in the rate of clinical decline in Alzheimer’s disease but downplays the fact that their efficacy studies depended on an indirect measure of clinical decline. The company looked at a surrogate biomarker (amyloid beta plaques) rather than the true outcome of interest (cognitive decline). The evidence for actual clinical benefit, despite the impact on the surrogate biomarker, was scant. 

9.    The article doesn’t sufficiently explain potential confounders that could explain the findings of observational studies.

• New Study Suggests Red Wine Reduces COVID Infection Rates | Wine Spectator – This article claims that red wine reduces COVID-19 infection rates but downplays the fact that “confounding factors” could be causing this observed relationship. Confounders are factors that might differ in prevalence between red wine drinkers and non-drinkers and that could in turn affect whether they contract COVID-19. Sometimes those factors can explain any differences between the groups described. The analysis adjusted for demographic characteristics, such as self-rated health, usual walking pace, and chronic diseases, but failed to account for other important confounders. For example, people who drink wine may have lower risks of contracting COVID-19 due to other factors, such as higher socioeconomic status, which was not fully accounted for in the study. Controlling for such confounders is notoriously difficult; the best way to do that is in a randomized experiment, but such experiments are not always feasible or ethical. As a result, no claims that red wine caused a reduction in COVID-19 infection should be made from only this evidence.  

10.    The article doesn’t mention potential conflicts of interest.

• Avocado and Gut Health: Study Finds Positive Effects on Microbiome — Good Gut Feelings – This blog post claims that daily avocado consumption may have a positive effect on microbial diversity in the gut and promotes the consumption of avocados for beneficial health outcomes. However, support and funding for the referenced research was provided in part by the Hass Avocado Board, a fact that the post failed to disclose. The disclosure could, justifiably, increase readers’ skepticism regarding the post’s recommendations. In this case, the study design increased the likelihood that differences would be found between the avocado diet group and the control group, as the researchers looked at a plethora of different gut contents and bacteria rather than having a very focused research goal. That increased the likelihood that some differences between the groups were due to chance and not due just to eating avocados. Furthermore, the clinical relevance of some differences was not clear. Readers have a right to have this important contextual funding information. 

Newspapers, magazines, and websites provide a valuable service when they translate important scientific findings from dry scientific terminology into lay language. But often their efforts leave much to be desired. We hope this list will help you separate the wheat from the chaff.