A recent post on the EXTEND Trial stimulated a passionate debate regarding the use of adjusted analysis in RCTs. A number of of those partaking in this discussion argued feverishly for the use of adjusted analysis in RCTs, and made very strong arguments in favor of such methods. I argued that while adjusted analyses may be more statistically precise, they cannot fully control for random error and have the potential for finding a positive result where one does not exist. The recent publication of the ROSE Trial in the NEJM (1), illustrates how a decade of clinical practice can be influenced by the improper interpretation of an adjusted analysis of RCT data.
The current practice of paralysis for severe ARDS was based of the results of the ACURASYS Study (2), published in the NEJM in 2010 by Papazian et al. The authors enrolled 340 patients in 20 ICUs throughout France who were admitted with ARDS and P/F ratios less than 150. Patients were randomized to receive a cisatracurium or placebo infusion for the first 48-hours following randomization. Ventilator strategies and sedation were managed using a formalized protocol which remained consistent between the two groups.
The authors reported a statistically significant improvement in their primary outcome, 90-day mortality citing a hazard ratio of 0.68 (95% CI 0.48 to 0.98; P=0.04). The crude 90-day mortality was 31.6% in the cisatracurium group and 40.7% in the placebo group. This 9.1% absolute difference was not statistically significant when an unadjusted analysis was performed, (P=0.08). It was only after the authors controlled for baseline PaO2:FiO2, SAPS II, and plateau pressure that the results met the threshold for significance.
This small trial which barely crossed the threshold for significance, and only did so with the assistance of an adjusted analysis, is the basis for our practice of paralysis in ARDS. And once again rather than acknowledge that these results were extremely fragile, and should be validated before adopting into clinical practice, we accepted these results as truth. But adjusted analysis or not, no trial of this size should single-handedly change practice.
It should come as no surprise that when the results of the larger, more statistically robust, ROSE Trial were published, their findings were in direct conflict with the ACURASYS trial. In this case, the authors randomized patients presenting with ARDS and a P/F ratio less than 150 to 48-hours of either cisatracurium or usual care. Like ACURASYS, all patients were treated with a low-tidal volume ventilatory strategy. Unlike ACURASYS which mandated both groups be maintained in deep sedation, patients in the control group of the ROSE Trial were maintained in a light state of sedation, striving for a Richmond Agitation–Sedation Scale of 0 or −1. The decision of whether to utilize prone position was left to the discretion of the clinician, but was used infrequently in both groups (only 15.8% of the entire cohort).
From January 2016 through April 2018 the authors enrolled 1006 patients. This was 402 patients shy of their 1408 predetermined sample size, stopping early due to futility. 488 patients (97.4%) of the 501 patients randomized to the cisatracurium group received the drug, with only 17% of the control group receiving a neuromuscular blocking agent during the first 48-hours following enrollment. The authors reported no difference in their primary outcome, 90-day mortality (42.5% in the intervention group VS 42.8% in the control group, P=0.93). Nor did they report a difference in any of their secondary outcomes, in hospital mortality, days free of ventilation, days out of the ICU, or days out of the hospital. The authors reported no difference in the rate of adverse events between groups, but not surprisingly the control group had higher mean levels of physical activity up to day 6 post-randomization and less incidents of acquired ICU weakness at 28-days (46.8% vs 27.5%).
In their discussion section, the authors suggest a number of reasons why their results are so different from the ACURASYS Trial. The ROSE Trial employed a higher PEEP strategy than what was used in the ACURASYS Trial. the ROSE Trial utilized a lighter sedation strategy in the control group, while the ACURASYS Trial employed deep sedation in both the intervention and placebo groups. And while these might represent the diverging results observed in these two trials, I suspect the reasons are far simpler.
Randomization is a tool we employ to control for non-random error or biases. The concept is that any confounders that may influence the results of a trial will be equally distributed between groups, leaving intervention in question as the lone distinction between the active and control groups. Any differences observed can then be thought to be due to the intervention in question. What randomization does not control for is random error, or baseline imbalances in the two groups simply due to deviations in random sampling (4). This type of error is best controlled by increasing the size of the sample in question. As the sample population grows larger it begins to look more and more like the general population from which it was sampled. Because of this, smaller trials are far more vulnerable to random error. It is not uncommon to see the effect size of any intervention oscillate wildly early in a trial’s recruitment. As the trial’s sample grows larger these observed vacillations grow smaller in amplitude, eventually coalescing around the true effect size. Early trials examining the use of neuromuscular blockade in ARDS were small and demonstrated positive results in favor of neuromuscular blockade (90-day mortality was 46.4% vs 71.4%)(3). ACURASYS, while positive, was only barely so. Finally with the publication of ROSE the largest and most statistically robust of the cohorts examining neuromuscular blockade in ARDS we observe the return to an effect size far more likely to represent the underlying reality.
The traditional statistical methodology we employ quantifies the risk for random error, but does not attempt to control it. The confidence interval which surrounds the 9.1% absolute difference in mortality (-1.1127% to 19.1067%) observed in the ACURASYS Trial reflects this uncertainty. An adjusted analysis such as that used in the ACURASYS Trial does more than just quantify the risk of random error influencing the results, it attempts to limit this risk by controlling for potential influential baseline characteristics. But such analyses can only do so much. In this case, such an analysis offered a degree of statistical legitimacy that likely wasn't deserved. No amount of statistical adjustment can completely control for random error. ACURASYS was a small trial, and thus prone to sampling error. In addition there was minimal prior evidence to support its findings. Whether you examined the adjusted or unadjusted analysis, the proper interpretation of this trial is that its results require further validation before they being incorporated into clinical practice. In fact, the authors of ACURASYS conclude,
“The administration of a neuromuscular blocking agent early in the course of severe ARDS managed with low-tidal-volume ventilation may improve outcomes. Future studies are needed to replicate and expand these findings before they can be widely adopted in clinical practice.”
But that is not what is remembered. No one recalls, the size of the study, or the fact that the positive results were only observed when an adjusted analysis was employed. All that is recalled is that this was a statistically significant study.
I am sure most of us have witnessed the results of the ACURASYS Trial used to justify the liberal application of neuromuscular blocking agents. But these rationalizations do not mention the fragility of this trials results. The ROSE Trial challenged these assertions, calling into question the efficacy of the widespread practice of neuromuscular blockade in patients with severe ARDS. Finally, ROSE reminds us of the fragility of science, offering a cautionary tale of relying on statistical adjustments in place of randomization and replication.
- Moss M, Huang DT, Brower RG, et al. Early Neuromuscular Blockade in the Acute Respiratory Distress Syndrome. N Engl J Med. 2019;
- Papazian L, Forel JM, Gacouin A, et al. Neuromuscular blockers in early acute respiratory distress syndrome. N Engl J Med. 2010;363(12):1107-16.
- Gainnier M, Roch A, Forel JM, et al. Effect of neuromuscular blocking agents on gas exchange in patients presenting with acute respiratory distress syndrome. Crit Care Med2004;32:113-119
- Altman DG, Bland JM. Uncertainty and sampling error. BMJ2014;349:g7064.