PulmCrit: ADAPT and SCREEN trials are full of sound and fury, signifying little

I think ICU docs may have developed a bit of an RCT problem; not an addiction, but perhaps a dependency. It all started off fine at first. RCTs are the apex trial design, the only trial capable of proving causality. But we've taken it too far. Currently, a cluster-randomized trial is underway to study whether saying hello can reduce burnout (I'm not making this up – it's the HELLO trial). Things have gotten out of hand.

What's the problem? Well, RCTs cannot answer some questions.

A major limitation of RCTs is that they lack external validity (ie generalizability) for context-dependent interventions. For example, let's take the cases of sepsis alerts:

In a hospital that often misdiagnoses sepsis and has a lot of extra resources, instituting a sepsis alert could be beneficial.
In a different hospital that usually diagnoses sepsis promptly and has few extra resources, instituting a sepsis alert could cause harm (by promoting over-diagnosis and diverting resources to the wrong patients).

So, the benefit of a sepsis alert is highly context-dependent. This isn't like comparing two different drugs. The intervention itself will depend strongly on the usual practice patterns. Even an amazing RCT performed in one health system may not generalize very well to another health system.

Likewise, let's consider the ADAPT trial investigating daily procalcitonin values to tailor antibiotic treatment duration:

For a hospital where physicians are very stingy with antibiotics, using a procalcitonin-guided algorithm could theoretically increase antibiotic usage. Alternatively, for a hospital where physicians are grossly over-utilizing antibiotics, using a procalcitonin-guided algorithm could reduce antibiotic usage. For a hospital somewhere in the middle, using procalcitonin may have little effect.
The details of any specific procalcitonin protocol are hugely impactful. For example, the ADAPT trial prevented physicians from seeing the actual procalcitonin values. High procalcitonin values tend to scare clinicians, so the outcome of the trial could have been entirely different if clinicians had access to the actual procalcitonin values. In real life the physicians will undoubtedly be able to see the actual laboratory value – so it's unclear whether the protocol employed in the ADAPT trial would generalize to real-world practice.

Both ADAPT and SCREEN are large, multi-center RCTs. Regardless of how much internal validity they have, they lack generalizability. It's an open question how these interventions would function in a real-world setting in other hospitals around the world.

One bedrock principle of science is replicability. An experiment must be replicable to be scientific and valid. If we attempted to replicate these trials in a different set of hospitals or with a slightly different protocol, would they replicate? I strongly doubt it. The results of these trials were neutral to mildly positive, so I'd bet that results would swing in either direction depending on the clinical context and random chance.

So these are beautiful trials, but they're testing interventions that are nearly impossible to study in a replicable and generalizable fashion. Performing these trials is worthwhile, but we need to be extremely cautious when interpreting the results. To be honest, I fear that these shiny new RCTs will seduce us into inappropriate generalization within evidence-based guidelines that will saddle our practice with an even higher burden of alerts (sepsis alert! rising procalcitonin! run for the hills!).

Critical care can be frustrating, as we strive to provide the best possible care in evidence-free grey zones. So it's tempting to grasp at whatever shred of evidence might present itself. But we need to exercise restraint and recognize that some questions cannot be answered – even with the best RCTs.

(Further musings about unanswerable questions in a prior blog here).