The COVID-19 pandemic has seen its fair share of boom-and-bust cycles: drugs that appear (at least to some) to be effective, only to be debunked or downgraded as stronger evidence arises. We’re not just talking about those driven by politically tainted hype, like hydroxychloroquine or ivermectin, but also more mundane fare like remdesivir and convalescent plasma. 

It seems like an age ago, but back in the halcyon early days when US deaths were numbered in thousands (over 1 million now!), attention turned to an improbable potential approach to the disease: the century-old bacilli Calmette-Guerin (BCG) vaccine, known for its use worldwide to prevent tuberculosis. When TB rates plummeted in the US, the vaccine fell into disuse, but many countries with higher disease rates continued to recommend it. It’s long been held that the more general immune-enhancing characteristics of the vaccine might convey protection against diseases other than TB. Why not COVID-19? 

In March 2020, a widely discussed preprint on the topic appeared on the scene. By correlating information on BCG vaccination policies from around the world with disease rates in those countries, the authors found that countries that never had a universal BCG policy experienced higher rates of COVID-19 infections and mortality compared to those that didn’t currently have such policies. Moreover, the older the universal BCG policy (and thus presumably the larger the proportion of the population that had been vaccinated), the lower the national COVID-19 mortality rate.  

This finding, which was never published in a medical journal, captured the imaginations of many observers who were desperate for effective interventions against the virus at a time when there was none. Others pointed out that the study had a design known within the ivory tower as “ecological.” This means that data are available at an aggregated level (in this case, at the country level), rather than at the individual level. Thus, there were no data on whether particular people in countries with universal BCG policies actually found themselves on the receiving end of a BCG needle or whether those who had were actually the individuals who came down with COVID-19 or died from it. Still, the finding was intriguing. 

Some colleagues (one, actually, also a brother) and I decided to assess whether the passage of time had strengthened or undermined this provocative finding. Or was the finding a mere accident of timing in which the initial spread of the disease happened to affect non-BCG countries first? We decided to replicate the authors’ methods for the nine months following the initial March 2020 assessment. 

It gets a little technical from here, but for simplicity I’ve summarized our findings in the graph below. In March, six of the nine statistical tests performed by the authors yielded statistically significant results favoring the vaccine, a fairly striking finding. However, this soon evaporated and after May no more than two tests (and often only one) of the nine were significant in any month. Most observers would agree that that could be due to chance.  The size of the reported effect, statistically significant or not, also generally took a hit as the pandemic groaned on. Our findings were published today in the peer-reviewed journal PLOS One.

The study we critiqued was part of the reason that well over a dozen randomized, controlled trials (RCTs) of BCG, in which the vaccine was randomly offered to some but not to others, were initiated. These, at least, offered the prospect of rigorous, individual-level data that would not be an artifact of the crests and troughs of the pandemic. But some trials appear to have been abandoned as actually effective vaccines were identified, and there is to date no convincing published RCT showing BCG to be effective. It’s a fair question whether devoting so many resources to a vaccine with such scant supporting data was a wise investment.