The Checklist that Saved an Aircraft

On October 30, 1935 Boeing engineers stood proudly by as their newest technological wonder, the Model 299 was about to demonstrate its capabilities to members of the  top brass of the Army Air Corps.  At the controls of the aircraft was the Army’s chief test pilot, Major Ployer P. Hill, copilot 1^st Lt. Donald L. Putt, and three other seasoned crew members.  The aircraft was cleared for take-off, accelerated down the runway, and began to climb into the Ohio sky.  It rose too steeply, and at approximately 300 feet above the ground it stalled, turned sharply, and plummeted to earth. The crash cost Maj. Hill his life and cost Boeing a sizeable contract with the Army.¹

It was concluded that the cause of the crash was “pilot error”.  After all, Major Hill, had not unlocked the elevator and rudder controls.  This omission caused the aircraft to climb too sharply and stalled the engines.  Ironically, this locking mechanism had been installed as a safety measure.  The development of such a complex flying machine engineered to be implemented in such a high-stakes environment was, perhaps, too complicated to be practical.  After all, the Boeing team had witness catastrophic results during a routine demonstration flight, circumstances far less stressful than a combat mission.  As one personally memorably commented, the Model 299 was simply “too much plane for one many to fly.

After careful analysis, Boeing and the Army developed an operational checklist that helped capable pilots safely fly the Model 299.  Eventually, the Army purchased a number of the aircraft.  They were incorporated as the B-17 in 1936.  Dubbed the “Flying Foretress”, the aircraft went on to play a critical role in defeating the Axis powers and bringing an end to World War II.²

Years later, organizations still rely upon machines that are impressive, but eye-wateringly complex.  The issues is what can be done to stop these same things from becoming too complicated to be practical in high-stress, high stakes environments.

What a Checklist is and is not…

Atul Gawande’s celebrated book The Checklist Manifesto stands on the shoulders of a growing body of medical literature (as well as robust evidence in the world of Human Factors Engineering) and brings the concept of the checklist into the limelight.  Subsequently, many organizations have implemented checklists to improve patient safety in a number of clinical domains.  Whatever good intensions clinicians, administrators, and institutions like the World Health Organization may have, it is important to understand what a checklist really is, what it should do, and when it is (and, equally as important, is not) the right tool for the job.

A good checklist fulfills several important objectives:^4,5,6,7

• Provides a standard foundation for verifying device or equipment setup to promote human psychological and physical function
• Provides a sequence for motor movements, sensory input, and cognitive focus along device controls or different pieces of equipment
• Enforces mutual supervision or cross checking, and shared mental models
• Enhances team commitment, ownership, and involvement in preparation for a procedure
• Improves coordination and communication in stressful or high-workload conditions
• Reduces cognitive load
• Reduces practice variability among practitioners, or conversely, guides best practices among practitioners.

Importantly, a checklist is not an instruction manual.  It is not designed to teach or instruct various procedures.  A provider has to know what he or she is doing.  Having large, branched decision-making algorithms as part of a “checklist” is not effective (especially during a crisis).   Likewise, a checklist is not a repository of information.  It’s not a cheat-sheet with patient weights, drug doses or equipment sizes.  This type of cognitive aid is a completely different thing with a completely different purpose and design recommendations.  Finally, it is not a substitute for clinical cognition: if you need to use a checklist to help you make immediate critical decisions or choreograph key steps of a critical procedure in real time…you’re wrong.  Checklists are cognitive aids, not cognitive crutches.  They are not a substitute for you, the clinician.

Briefings are not the same as checklists

Often, people conflate checklists with briefings.  Operational briefings occur before a procedure or event.  Their purpose is distinct from that of a checklist.  Briefings are for discussing the steps of an activity, anticipating error-likely or dangerous situations, foreseeing potential consequences, and assessing team preparedness.  So, line items like “Difficult airway anticipated?” or “Need for DSI?” do not belong on a checklist.  No pilot or military operational team gets in the air or gets to the target and then discusses the plan: that is done ahead of time.  In short, the checklist is completed immediately before the task is executed.  The goal of the checklist is to accomplish what was discussed in the briefing.

The primary benefit of the briefing is enhancing team efficiency, confidence, and safety by sharing a mental model.⁸   Shared mental models are critical to team performance.  They are knowledge structures held by members of a team that enable them to form accurate explanations for the task, coordinate their actions, and adapt their behaviors to meet task demands.⁹  It allows everyone involved to have the same understanding of the dynamics in a complex process.¹⁰  Clearly communicating with a team and briefing improves adaptability, reflexivity, and anticipation.^11,12

Some important aspects of a briefing have been described in the Human Factors Engineering literature:^12,13

• PLAN – What is going to be done
• PURPOSE – Why it is being done and why certain steps or sequences are important
• DATA – The facts: who is involved, what equipment is needed, how you want things set up, where the task will be accomplished (if not otherwise evident).

Checklist Structure and Design

Daniel Boorman, flight deck design expert and Production Test Pilot at Boeing, best summarizes what makes a checklist laudible or lamentable:¹⁴

Bad checklists are vague and imprecise.  They are too long; they are hard to use; they are impractical.  They are made by desk jockeys with no awareness of the situations in which they are to be deployed.  They treat the people using the tools as dumb and try to spell out every single step.  They turn people’s brains off rather than turn them on.  

Good checklists, on the other hand, are precise.  They are efficient, to the point, and easy-to-use even in the most difficulty situations.  They do not try to spell out everything – a checklist cannot fly a plane.  Instead, they provide reminders of only the most critical and important steps – the ones that even the highly skilled professionals using them could miss.  Good checklists are, above all, practical.

Boorman’s observations are bolstered by the human factors literature.  While some exceptions exist, the core concepts (simplicity, readability, efficiency, logical sequence and easy integration into workflow) are globally accepted.⁵

Anyone who has tried constructing a checklist and getting traction on its clinical implementation understands that this is a monumental challenge.  It seems deceptively simple: make a chronological list of important stuff you won’t want people to forget before placing a central line, starting a surgical procedure, or initiating RSI.  In reality, it is much more complex.  This phenomenon is complicated by the lack of high quality evidence clearly delineating optimal checklist structure in the clinical environment.  That said, there are some general guidelines that seem to maintain validity as we apply evidence from other fields.

Often, in emergency medicine and critical care settings parallels are drawn between RSI checklists and the “Take Off” checklist run by airline pilots.  While this may be appropriate in the operating theatre for scheduled surgical cases with otherwise stable patients, the analogy is both theoretically and practically flawed when applied to critically injured or ill, unstable patients.  A more appropriate analogy, perhaps, is the checklist used for a forced landing in a small, single engine aircraft.  Referring to the operations manual for a Cessna 172N Skyhawk is likely a much better example:¹⁵

This more closely approximates the situations in the emergency department and ICU, as well as emergency cases in the OR.  The current situation (unprotected airway or flight with an impending problem) is metastable, unsustainable, or currently failing.  Intervention (intubation or landing) is required urgently or emergently.  Notice several things about the checklist:

1. It is short and to the point.  The longer of the two checklists only has 11 items.  This limits “head down” time, allows the operator to focus more on actually flying the plane, and optimizes both operational efficiency and global situation awareness.
2. Only the critical items are listed.  Notes, discussion, and more detailed instruction are located elsewhere in the document as this information is not necessary at the time of checklist execution.
3. Notice that there are specific responses to the primer or challenge item on the checklist.  The response is not simply “check” or “good”.  It is explicit.  The response is a specific setting such as “65 KIAS” (knots-indicated air speed) or the position of a particular switch “OFF”.
4. Text is easy to read. 
5. The sequence is logical and flows naturally.

Again, while no concrete evidence exists that demonstrates benefit with regard to patient oriented outcomes for emergency checklists, there is some suggestion in the literature that these elements are beneficial in development of operational as well as clinical checklists.^{16,17,18,19,20}

Optimizing Checklist Efficiency

Checklists have been criticized for their lack of proven utility and for wasting valuable time.  There are very reasonable counter arguments that suggest checklists take a short period of time before the procedures starts, but save time in the long run by ensuring all equipment is prepared and available, as well as preparing for dangerous contingencies in advance.  Practically though, medical checklists have been designed that do take an unnecessary amount of time.  There is a fine line between helping and hindering with cognitive offloading tools in the emergency setting.  That said, there are some effective ways to help shorten a checklist.

One suggestion is coming up with standard operating procedures (SOPs).  SOPs are established or prescribed methods to be followed routinely for the performance of designated operations or in designated situations.  They generally incorporate best practices and/or best scientific evidence, are adapted to the specific capabilities of an organization, and are universally understood.  Implemented correctly, they can increase efficiency, ensure uniformity, and eliminate miscommunication.  

The development of a standard airway template or plan obviates the need for in-depth discussion or lengthly checklist review.  This is because it is simply “the way” it is done and the team practices.  For example, if an institution were to adopt the Vortex Approach to general airway management, extensive briefing and discussion are not needed in the event you are unable to place an endrotracheal tube.   Some institutions have adopted the standard of taking the first look with a standard geometry blade and video laryngoscopy system.  Others encourage all learners to make their first intubation attempt with a bougie.  Both standards preclude the need to discuss what the principle technique will be.

Likewise, if there is a standard tray of equipment (e.g. with outlines of where everything goes) that includes backup airways, OPAs, NPAs, and surgical airway equipment, it is reasonable to confirm everything is on the tray and simply have a checklist line that says “Equipment…OUT AND AVAILABLE”.   The same concept may apply to the individual roles and responsibilities.  For example, if the standard arrangement of personnel around a patient dictates that the individual on the patient's right pushes medications/monitors vital signs and the individual on the patient's left assists with BVM/airway equipment there is no need to brief this or have checklist items to review it.  

SOPs create a shared mental model and structured approach for a team.²¹  This unified understanding of critical actions under stress reduces errors²² , improves performance, and enhances team efficiency.²³  When everyone understands the plan, the overall cognitive load and mental resources required to think about what to do next is reduced.²⁴  These are not only the putative benefits of SOPs, but, moreover, demonstrate why these items can be left off a checklist.

While the vast majority of emergency procedures may be performed using an SOP model or standardized approach, some exceptions do exist.  In fact, it is both sensible and safe to diverge from standard practice when atypical clinical circumstances dictate.  Not a problem.  Simply brief the change and share an updated mental model.  It may be counter intuitive, but SOPs are actually helpful, even in unique settings.  When you draw the team’s attention to the fact that something is NOT standard, it can even heighten situational awareness.⁸

Regardless of how long a checklist might be, its organization and structure have perhaps the greatest impact the efficiency of checklist implementation.   “Chunking” is a term used in cognitive psychology to refer to elementary units of information that are grouped together in a specific way (usually by logical associations) to form a more meaningful whole.^25,26   A common example in medicine may be any number of mnemonics we use to remember disease symptoms or treatments. While the intention and evidence for this concept is generally applied to enhance learning and memory, it is reasonable and prudent to apply the concept to other forms of information processing like checklists.^27,28  While it may not incorporate some previously discussed aspects of optimal checklist design, the SOAP ME checklist structure is a good example of chunking in a checklist:²⁹

How a checklist is implemented can also enhance efficiency.  For example, more novice clinicians may choose to perform a checklist in “Read-Execute” fashion.  That is, they read an item off and perform the task.  More experienced clinicians have likely developed a task flow of their own.  In this case it may be very reasonable to use the checklist in a “Crosscheck” manner: execute your normal task flow setting everything up, but use the checklist to ensure nothing was overlooked.¹⁸

Other Important Aspects of Implementation

When are checklists helpful and who should you use them?  While ostensibly simple, this is actually difficult question to answer.  In order to creat a checklist that helps more than it hurts consider four factors: the complexity of the procedure, the risk involved to the patient, the familiarity of the provider, and the reactionary gap of the provider.  “Reactionary gap” is a tactical term adapted to the clinical environment.  It refers to the time available for a provider’s brain to process the situation, decide on a course of action, and respond effectively.

Applying these principles to emergency airway management in the ED or ICU illustrates a checklist's usefulness.  The procedure complexity is unknown (a difficult airway may exist), the risk involved is significant (failure to manage the airway could result in death), the frequency is moderate to low (anatomically and physiologically difficult airways do not occur every shift), and the permissible reactionary gap is low (the patient can deteriorate very quickly under certain circumstances).

Contrast this to operating room airway management in scheduled anesthesia cases.  While routine, pre-surgical checklists are often used, airway checklists prior to induction are rare.  This is quite reasonable.  Checklists in this setting may be of limited utility.  Experienced anesthesiologists have managed thousands or tens of thousands of airways in the very familiar OR setting (they have developed “flows” and routines of safe practice).  While the practice of anesthesia is complex requiring years of training, deliberate practice, and intense study, placement of endotracheal tubes or other airway devices is somewhat less complex after one has gained expertise.  Additionally, being able to talk to the patient, examine them, and review their medical history eliminates some airway management complexity by knowing what lies ahead.  The risk, albeit, is still high and anesthesiologists carry immense responsibility given that airway misadventures in the OR can also be fatal.  However, there is somewhat more of a reactionary gap with a stable ASA 1 and 2 patients that can be properly preoxygenated and denitrogenated.  These patients can have safe apneic times of 6-10 min or even longer depending on the situation.

To be clear: it is not about one’s career, it is about one’s circumstance.  The need for a checklist is not a reflection on one's specialty, but rather the patient needs.  It may be very reasonable for an anesthesiologist on an institution’s “Difficult Airway Response Team” to use a checklist.  Likewise, when outside of the OR with critically ill patients in other areas of the hospital, or outside the hospital in the field (many prehospital physicians do use checklists), it may be very reasonable to use a checklist because the circumstances change. The provider may be outside his or her usual environment, limited tools may be available, normal workflow may be interrupted, and the reactionary gap may be significantly lower.

To reemphasize: the intent of the above comparison is not to dictate who should and who should not use a checklist.  It is meant to highlight some aspects regarding the human factors science that pertain to different situations of airway management and which situations might derive more or less benefit from checklist use.

Emergency vs Normal Checklists

While this review focused on emergency checklists, other types of checklists exist.  For example, checklists for routine procedures, equipment maintenance, or quality/safety checks.  These are referred to as “normal checklists”.¹⁸  The equivalent in the clinical environment would be a checklist to prevent catheter-rated blood stream infections (CLABSIs), a daily checklist to ensure proper stocking and equipment function on a code cart, or before routine surgical procedures such as the World Health Organization Surgical Safety Checklist.  These checklists have, in fact, demonstrated clinical benefit when combined with other measures.^20,30,31  The structure, design, and function of these checklists, although equally important, is somewhat different than emergency checklists.  Their implementation has been discussed by other authors and is best discussed in a separate post.³

Conclusion

The rising popularity and increased implementation of checklists in EDs and ICUs is certainly laudable.  Indeed, more institutions should endeavor to incorporate them. The danger, as outlined above is if “checklists” balloon into complicated, unwieldy, multi-page monstrosities that consume pages, use up valuable minutes, and increase rather than decrease confusion.  All sorts of “helpful” information, pictographs, algorithms, and other cognitive aids have infiltrated “checklists”, mutating them into confusing and unhelpful tools.  The creation of these human factor Frankensteins is usually a function of misunderstanding of what checklists truly are and not carefully reviewing the literature from other occupations regarding the topic.  While we applaud the zeal of clinicians that have embraced the concept and dutifully endeavored to make them a standard of care, we would implore everyone to carefully consider the structure and function of these tools.  Even the safest, most effective instruments can fail if they are designed poorly or are used improperly.

Key Points

• Checklists should be concise, easy to read, flow logically, have specific responses, and only include critical items needed for safe and effective execution of a procedure
• Checklists should be void of extraneous information including tables, drug dosages, or other diagrams.  These cognitive off-loading tools are helpful, but should not be located in the checklist.
• Briefings and checklists are separate entities in the realm of crisis communication.  Conflation of the two makes them both less effective.
• When considering if a checklist may be helpful consider the circumstances: task complexity, patient risk, provider familiarity/expertise, and reactionary gap.

Acknowledgements

Special thanks to Peter Brindley, MD (@docpgb) and Joe Novak, MD for their time, kind feedback, and editorial assistance with this post.  Their significant contributions are greatly appreciated.

Related Links

Human Factors in Airway Management

Resuscitation Communication

References

1. Meilinger PS. When the Fortress Went Down. Air Force Magazine. 2004 October: 78-82.
2. Yarrish G. B-17: The Story Behind Boeing’s First Flying Fortress. Flight Journal.  Available at https://www.flightjournal.com/boeings-first-flying-fortress/. 30 November 2017.  Accessed 10 November 2018.  
3. Clay-Williams R, Colligan L. Back to basics: checklists in aviation and healthcare. BMJ Qual Saf. 2015; 24(7): 428-31.
4. Degani A, Wiener EL. Human factors of flight-deck checklists: the normal checklist. NASA Report NCC2-377. Moffett Field, CA: NASA Ames Research Center; 1990.
5. Degani A, Wiener EL. Cockpit checklists: Concepts, design, and use. Hum Factors. 1993; 35(2): 345-59.
6. Hales BM, Pronovost PJ. The checklist – a tool for error management and performance improvement. J Crit Care. 2006; 21: 231-235
7. Helmreich RL. On error management: Lessons learned from commercial aviation. BMJ. 2000; 320: 781-785.
8. Dekker S. Patient Safety: A Human Factors Approach. Boca Raton, FL: CRC Press; 2011: 177-183.
9. Cannon-Bowers JA, Salas E, Converse S. Shared mental models in expert team decision making.  In: Castellan J, ed. Individual and Group Decision Making: Current Issues. Hillsdale, NJ: LEA; 1993: 221-246.
10. Klein G. Cognitive task analysis of teams. In: Schraagen JM, Chipman SF, Shalin VL, eds. Cognitive Task Analysis. Mahwah, NJ: LEA; 2000.
11. West M. Reflexivity, revolution, and innovation in work teams. In: Beyerlein D, Johnson D, Beyerlein S, eds. Product Development Teams. Stamford, CT: JAI Press; 2000: 1-29.
12. West M. Effective Teamwork: Practical Lessons from Organizational Research. 2^nd Ed. Leicester, UK: BPS Blackwell; 2004.
13. Endsley M, Jones DG. A model of inter- and intrateam situational awareness: Implications for design, training, and measurement.  In: McNeese M, Salas E, Endsley M, eds. New Trends in Cooperative Activities: Understanding System Dynamics in Complex Environments. Santa Monica, CA: Human Factors and Ergonomics Society; 2001: 46-67.
14. Gawande A. The Checklist Manifesto: How to Get Things Right. New York, NY: Metropolitan Books; 2009: 120.
15. Pilot’s Operating Handbook: Cessna 1978 Skyhawk Model 172N. Wichita, KS: Cessna Aircraft Company; 1977: Section 3 (Emergency Procedures) pp 4-5.
16. Verdaasdonk EG, Stassen LB, Wdhiasmara PP, Dankelman J. Requirements for the design and implementation of checklists for surgical processes. Surg Endosc. 2009; 23(4): 715-26.
17. Civil Aviation Authority. Guidance on the Design, Presentation, and Use of Emergency and Abnormal Checklists. Civil Aviation Authority; 2006.
18. Burian BK. Design Guidance for Emergency and Abnormal Checklists in Aviation. Proceedings of the Human Factors and Ergonomics Society Annual Meeting.  1 October 2006.
19. Turner JW, Huntley MS. The Use and Design of Flightcrew Checklists and Manuals. US Department of Transportation, Federal Aviation Administration. Cambridge, MA: National Transportation Systems Center; 1991.
20. Thomassen O, Espeland A, Softeland E, Lossius HM, Heltne JK, Brattebo G. Implementation of checklists in health care; learning from high-reliability organizations. Scand J Trauma Resus Emerg Med. 2011; 19: 53.
21. Cooper JB, Cullen DJ, Eichorn JH, Philip JH, Holzman RS. Administrative Guidelines for response to an adverse anesthesia event. J Clin Anesth. 1993; 5: 79-84.
22. Wilson K, Salas E, Priest H, Andrews D. Errors in the Heat of Battle: Takeing a Closer Look at Shared Cognition Breakdowns through Teamwork.  Hum Factors. 2007; 49: 243-259.
23. Serfaty D, Entin EE, Johnston JH. Team coordination training. In: Cannon-Bowers J, Salas E, eds. Making Decisions Under Stress: Implications for Individual and Team Training. Washington, DC: American Psychological Association; 1998.
24. Stout RJ, Cannon-Bowers JA, Salas E, Milanovich DM. Planning, Shared Mental Models, and Coordinated Performance: An Empirical Link is Established. Hum Factors. 1999; 41: 61-71.
25. Neath I, Surprenant AM. Human memory. 2^nd ed. Belmont, CA: Wadsworth; 2003.
26. Tulving E, Craik FIM. The Oxford handbook of memory. Oxford, UK: Oxford University Press; 2000.
27. Lindley RH. Recording as a function of chunking and meaningfulness. Psychol Sci. 1966; 6: 393-394.
28. Miller GA. The Magical Number Seven, Plus or Minus Two: Some Limits on our Capacity for Processing Information. Psychol Rev. 1956; 63: 81-97
29. Greenwood J. Back 2 Basics Series: Your Simple RSI Checklist – SOAP ME.  Available at https://em.umaryland.edu/educational_pearls/2577/.  4 November 2014.  Accessed 29 November 2018.  
30. LevyMM, Pronovost PJ, Dellinger RP, et al. Sepsis change bundles: converting guidelines into meaningful change in behavior and clinical outcome. Crit Care Med. 2004; 32: S595–S97.
31. Haugen AS, Søfteland E, Eide GE, et al. Impact of the World Health Organization’s Surgical Safety Checklist on safety culture in the operating theatre: a controlled intervention study. Br J Anaesth. 2013; 110: 807–15.

• Author
• Recent Posts

Socialize

Mike Lauria

Regular Contributor at EMCrit.org

Emergency Medicine Resident
University of New Mexico Health Sciences Center
Former US Air Force Pararescue and Critical Care/Flight Paramedic

Socialize

Latest posts by Mike Lauria (see all)

• Invictus: Survival, Sustainment, and Service in the Era of COVID-19 - April 25, 2020
• Emergency Reflex Action Drills - April 23, 2019
• COMM CHECK: On Checklists - January 3, 2019