Whether or not you have any interest in bamlanivimab, you should read this post as an amusing example of shoddy statistics being published in top journals.
background & general landscape of the two trials
The BLAZE-1 trial involved randomizing patients within three days of testing positive for COVID-19 to one of four arms: placebo, 700 mg bamlanivimab, 2800 mg bamlanivimab, or 7000 mg bamlanivimab. Some amazing statistical machinations were used to twist this data into a “positive” result.1
Nonetheless, Eli Lilly realized that bamlanivimab didn’t really work, so they switched the treatment arm to a two-antibody cocktail (bamlanivimab plus etesevimab). A new publication now reports all of the data involving placebo, bamlanivimab monotherapy, and the two-antibody cocktail.2
The pair of studies is hilarious because it utilizes the same exact data regarding bamlanivimab (the same 309 patients). In the first publication, this data was manipulated to make bamlanivimab appear effective. However, in the second publication, bamlanivimab was found to be ineffective! These flip-flopping results demonstrate how dubious the statistics were all along.
The statistical manipulations involved in the original publication were obvious (I pointed them out in a prior blog here). What is unique is that reanalysis of the dataset with a different statistical analysis strategy confirms how fragile the original statistical analysis really was.
primary endpoint: Viral load change after 11 days
The first publication compared the change in viral load in each group to the change in the placebo group (table above).1 In most groups, the change was not statistically significant. However, in the intermediate (2800-mg) group, the reduction in viral load was greater in patients receiving bamlanivimab (p = 0.02).
The problem with this analysis is that three comparisons are being made. A Bonferroni correction would suggest that when performing three comparisons, the appropriate p-value cutoff would be 0.05/3 or 0.016 (which would make this result statistically insignificant). There are different ways to correct for multiple comparisons, so it’s debatable whether 0.02 crosses the line – but regardless, it should be clear that this is a borderline result that is not statistically robust. Furthermore, it’s biologically implausible that the intermediate 2800-mg dose should be more effective than a higher 7000-mg dose – which should cast further shade over this result. Nonetheless, the abstract struggled to tout this result as statistically significant:
The second publication included the same 309 patients in the bamlanivimab group, 112 additional patients in a new combination therapy group, and only a mere 13 additional patients in the placebo group (and, yes, it's shady that 90% of the new patients were “randomized” into the combination therapy group).2
The second publication used the same primary endpoint, change in viral load over 11 days. The placebo shifted slightly, with a greater mean drop in viral titers (-3.47 in the first study versus roughly -3.7 in the second study). This small shift in the placebo group erased the statistical significance of the bamlanivimab monotherapy groups:
This illustrates how statistically fragile the original result was. Ironically, the new manuscript concludes that bamlanivimab monotherapy is ineffective (despite analyzing data from the same exact 309 patients):
secondary endpoint: “Hospitalization”
In order to manufacture a positive clinical endpoint, the first publication did the following two things:
- They combined all three bamlanivimab groups together (the 700-mg, 2800-mg, and 7000-mg groups).
- They defined “hospitalization” to mean either hospital admission or emergency department visit.
Using these maneuvers, they limped their way to a statistically significant reduction in “hospitalization.” As shown below, this was 5/309 in the pooled bamlanivimab groups vs. 9/143 in the placebo group (p = 0.015)
In the second publication, the bamlanivimab groups were analyzed separately. As expected, any statistical benefit from bamlanivimab immediately evaporates:
bamlanivimab never actually worked
The first BLAZE-1 publication massaged the data just right, to make it appear that bamlanivimab was effective. The second study added a bit more data and reanalyzed the data, causing bamlanivimab’s apparent efficacy to disappear. This is more damning than failing a separate validation trial: with a minor tweak in the data and statistical reanalysis, the efficacy has disappeared. This strongly suggests that bamlanivimab never worked in the first place.
the bamlanivimab/etesevimab cocktail probably doesn’t work either
Well, version 1.0 was a bust, but surely the upgraded version will work better? Perhaps… but available evidence isn’t looking favorable:
- The authors of the second study evaluated 21 secondary endpoints (for a total of 84 comparisons!). Among these endpoints, there was no signal that bamlanivimab/etesevimab was clinically beneficial.
- Monoclonal antibodies may be even more ineffective against mutant strains of COVID that will likely become dominant over the next couple of months.
- Administration of monoclonal antibodies could make the COVID vaccine less effective, by causing rapid clearance of spike protein. Studies may soon be emerging regarding prophylactic use of monoclonal antibodies following high-risk exposure to COVID. This application should be approached with extreme caution, because it could interfere with the efficacy of the COVID vaccine (which is undoubtedly the most effective therapeutic against COVID).
- The original BLAZE-1 publication was touted as showing that bamlanivimab caused a reduction in viral titers and hospitalization. Statistical analysis in this study was deeply flawed and profoundly fragile.
- A newer, follow-up study now finds that bamlanivimab affects neither viral titer nor hospitalization. The collapse of previously “significant” results was highly predictable, based on the statistical chicanery in the original BLAZE-1 publication.
- Currently there continues to be no persuasive evidence that monoclonal antibodies improve patient-centered clinical endpoints.
- The use of bamlanivimab should be restricted to the context of randomized controlled trials. As it becomes clearer that bamlanivimab monotherapy is ineffective, ongoing use is increasingly difficult to justify.
going further
- I'm so confused about bamlanivimab (BLAZE-1, PulmCrit)
- BLAZE-1: COVID-19 Neutralizing Antibody (RebelEM, Salim Rezaie)
- RCTs don't justify using convalescent plasma or antibody cocktails (PulmCrit)
Opening image credits: Photo by Alex Iby on Unsplash
references
- 1.Chen P, Nirula A, Heller B, et al. SARS-CoV-2 Neutralizing Antibody LY-CoV555 in Outpatients with Covid-19. N Engl J Med. 2021;384(3):229-237. doi:10.1056/NEJMoa2029849
- 2.Gottlieb RL, Nirula A, Chen P, et al. Effect of Bamlanivimab as Monotherapy or in Combination With Etesevimab on Viral Load in Patients With Mild to Moderate COVID-19. JAMA. Published online January 21, 2021. doi:10.1001/jama.2021.0202
- PulmCrit Blogitorial – Use of ECGs for management of (sub)massive PE - March 24, 2024
- PulmCrit Wee: Propofol induced eyelid opening apraxia – the struggle is real - March 20, 2024
- PulmCrit wee: Why I like central lines for GI bleed resuscitation - March 13, 2024
Glad somebody pointing out the flaws in this blaze 1 trial. Can’t even comprehend why we are using 700mg instead of the 2800mg. All so shady. Just promotes a way to eat up more ER resources. Thank you for your continuous promotion of interesting reads !
Thank you for your analysis! This article is great. I would love to know your thoughts on the stats they give for the differences in rates of hospitalization/ED visits (pg 641). I am no stats wizard, but I can’t make any sense of these P values/confidence intervals… “The difference vs placebo was –4.8% (95% CI, –8.9% to –0.6%; P = .09) for the 700 mg group […] and –4.9% (95% CI, –8.9% to –0.8%; P = .049) for the combination group….” How is the P value for the 700 mg group >0.05 if the confidence interval doesn’t cross “0”? What… Read more »
I took the BAM treatment in April 2021. Seems that I’m having a lot of side effects from either the virus OR the treatment. Are there any studies on long/short term effects of BAM? I cannot find anything, which is really scary.
Thoughts on Effectiveness of Severe Acute Respiratory Syndrome Coronavirus 2 Monoclonal Antibody Infusions in High-Risk Outpatients Nicholas Piccicacco et al. Open Forum Infect Dis. 2021. NNT seems much better. Will there be an update since the IDSA is now recommending?