Assessing Competence in Psychomotor Skills: Do We “Know it When we See it”?
Aditee P Ambardekar, MD, MSEd, Justin L Lockman, MD, MSEd, Alan Jay Schwartz, MD, MSEd
As pediatric anesthesiologists there are many technical skills like laryngoscopy, tracheal intubation, peripheral and central line insertion that we have to master in order to do our jobs. Over the past 10-15 years there has been an explosion of new hands on technical skills like video laryngoscopy, ultrasonography guided peripheral and central line cannulation, ultrasonography guided neural blockade, Point of Care UltraSonography to assess heart and lung function, and EEG guidance of the depth of anesthesia to name just a few. For both new learners as well as long established clinicians and educators how do we assess initial and ongoing competency in these new 21st century skills?
I’ve been thinking about these issues for a long time and have used my perch as the lead writer/reviewer for Ron Litman’s Pediatric Anesthesia Article of the Day to showcase articles that deal with these and other educational issues. Indeed, we have 3 reviewers, Aditee P Ambardekar, MD, MSEd, Justin L Lockman, MD, MSEd, Alan Jay Schwartz, MD, MSEd who have dedicated their professional careers to education and fortunately for us, all 3 are on the PAAD’s executive council. In that capacity, they regularly review educational issues, including today’s PAAD1 and its accompanying editorial2.
Demonstrating initial and ongoing competency has more importance today than perhaps at any other time in history. It’s not just the pace of the introduction of new techniques and technology into clinical practice but the changing lifestyles and work life balance needs of our colleagues that guides much of how we train new residents, fellows, and student nurse anesthetists and AAs. Over the past few weeks we’ve discussed many issues with the supply demand imbalance affecting the practice of pediatric anesthesiology. Demonstrating competency is central to many of these issues. Thus, as an example, does an anesthesia residency really need to be 3 continuous years? Can it be shortened or lengthened based on demonstrated competency in technical skills and knowledge? Further, how do physicians out of training maintain the skills and knowledge they gained during training or develop the new skills that have emerged after their training? Today’s PAAD is another salvo in this discussion. Myron Yaster MD
Original Article:
de Oliveira Filho GR, Soares Garcia JH. The Accuracy of the Learning-Curve Cumulative Sum Method in Assessing Brachial Plexus Block Competency. Anesth Analg. 2024 Aug 1;139(2):281-290. doi: 10.1213/ANE.0000000000006928. Epub 2024 Jun 11. PMID: 38861983.
Editorial
Huffmyer JL, Estes H, Dexter F. Evaluating Competence of Anesthesiology Residents Including Procedures Performed Successfully and Entrustment Scales. Anesth Analg. 2024 Aug 1;139(2):278-280. doi: 10.1213/ANE.0000000000007021. Epub 2024 Jul 15. PMID: 39008866.
How many repetitions of a procedure does it take to demonstrate both the knowledge and the skills to competently perform it on patients unsupervised?
Residents and Fellows in training ask this important question and we, as teachers of anesthesiology trainees, strive to know the answer. Frankly, we should all be wondering this about ourselves as we continue our professional development journeys. Our specialty accrediting and certifying organizations, the Accreditation Council for Graduate Medical Education (ACGME) and The American Board of Anesthesiology (ABA), share responsibility for defining the minimum necessary curriculum and for designing valid assessment tools to answer the question. But is that enough?
In the August 2024 issue of Anesthesia & Analgesia, Getúlio Rodrigues de Oliveira Filho and Jorge Hamilton Soares Garcia reported their investigation of assessing competency of a single type of procedure: brachial plexus block.1 de Oliveira Filho has studied the concept of learning curves for more than 20 years.3 In the recent publication, the team evaluated the learning curve – cumulative sum (LC-CUSUM) method and its risk-adjusted form (RA-LC-CUSUM) to detect psychomotor skill competency for brachial plexus block.
What is LC-CUSUM?
“The LC-CUSUM test [a statistical methodology] was developed to determine when a trainee has reached a predefined level of performance. The LC-CUSUM sequentially tests the null hypothesis that “performance is inadequate” against the alternative that “performance is adequate.” The LC-CUSUM test computes the […] score from successive outcomes; a success increases the score, and a failure decreases it.”4
de Oliveira Filho and colleagues used historical data from 1,713 brachial plexus blocks performed by 32 anesthesia residents to develop learning curves.1 Based upon the LC-CUSUM method, 85% attained competency after a median of 18.5 blocks. Additionally, the authors used the RA-LC-CUSUM method to determine that 77% attained competency after a median of 17.5 blocks.1 Importantly, both methods were associated with substantial false-positive and false-negative rates. Both methods may be inaccurate for high-stakes decisions about resident competency at brachial plexus blocks.1
Huffmyer, Estes and Dexter2 in the accompanying editorial, unveiled the challenges that educators, students and accrediting and certifying organizations face when trying to answer the question, “When are individuals procedurally competent?”
It’s helpful to consider the evolution of how the competency question has been historically answered – and why current investigations like that of de Oliveira Filho and colleagues are still relevant and imperative today. “Competence – the ability to do something well and Competency – an important skill needed to do a job”2 have historically been determined using the surrogate of case logs to quantify completed procedures. “While counts help detail resident experience in certain techniques and index cases, a simple count does not predict or evaluate competence…”.2 It should also be noted that the concept of case logs came out of a necessity to ensure a training program had sufficient procedural volume to support a program of its designated size. While minimum requirements were never meant to imply baseline for competency, without more robust evidence it is all that program directors and educators have as a ruler for minimum resident experience.
Alongside case logs, residents receive “acceptable” evaluations by faculty members who observe their performance. Yet, most faculty have not received formal training in how to know when a procedure is done well. Instead, we often rely on outcomes (e.g., intubation success), which may occur independent of technical competence. Thus, “[Formative] Feedback - transmitting evaluative or corrective information about an action, event, or process to the original or controlling source and [Summative] Assessment – the evaluation or estimation of the nature, quality, or ability of someone or something”2 are both randomly applied and lacking in standardization. The LC-CUSUM technique may offer an additional evaluative resource to guide educators.
To effectively evaluate the necessary large (and ever-increasing) number of developing skills against a predetermined set of competencies needed to document a trainee’s growth and development over time is a nearly insurmountable task. Baker et al.5 and Wanderer et al.6 have investigated and demonstrated valid and reliable evaluation methodologies that can be used in educational research, yet recognized how impractical it would be to implement in real world settings. Too many evaluation data points would have to be collected and an inordinate amount of faculty time invested to acquire sufficient information to make valid decisions about trainee competence. And even that assumes standardization of faculty data collection.
Today’s PAAD by de Oliveira Filho and colleagues aims to develop learning curves that define the median number (count) of procedures associated with success (outcome). While we applaud this large step in the right direction, we are concerned about the high rate of false negatives and false positives. Are these findings related to the large numbers of minute variables/variability in technique, anatomy, equipment, and more for this single block? Perhaps – but that is the reality every day in anesthesia practice!
For that reason, among others, LC-CUSUM alone cannot be the ultimate answer to the competence question. Nonetheless, it may provide evidence-based guidance for revisions to case log requirements in the future. And competency aside, studies such as this may allow us to hone numbers of specific anesthesia procedures at the programmatic level to ensure we have sufficient procedural volume for the size of our training programs.
Huffmyer, Estes and Dexter’s editorial concludes, “This well-designed study scientifically shows a failure of the [LC-CUSUM] method for practical ability to detect competence. We need to use this result! What it means is that the number of procedures completed successfully by a resident should not be the determining factor for competence […] the results of de Oliveira Filho et al suggest that how the block is performed is the key to evaluating competence […] simple success or failure of technical skills does not imply competence. If not done in a rigorous and validated fashion, the evaluation of residents by faculty is random and prone to the leniency/severity bias of the rater.”2
In other words, adding a complex statistical technique does not change the underlying problem: attendings think they know competence when they see it. So where does this leave us? Are they right? How do we know? And importantly, how do we avoid bias in these and other similarly subjective evaluations?
The competence question remains elusive and unanswered. How do you evaluate residents and fellows to assure they are competent? How do you assure the public that our trainees can safely and effectively move into unsupervised clinical practice? How do we apply these same questions and answers to continuing education and recertification? Can we be sure, for example, that a resident who achieves competence performing brachial plexus blocks during training maintains that skill 5 years later if they only perform 10 such blocks each year? Who will assess their competence in an ongoing basis?
These are the questions that keep us up at night as professional educators. There is a great deal of work left to do in this area, and we welcome your input and perspectives about how to make it better! Send Myron Yaster your thoughts at myasterster@gmail.com and he’ll post in a Friday Reader Response.
References
1. de Oliveira Filho GR, Soares Garcia JH. The Accuracy of the Learning-Curve Cumulative Sum Method in Assessing Brachial Plexus Block Competency. Anesthesia and analgesia 2024;139(2):281-290. (In eng). DOI: 10.1213/ane.0000000000006928.
2. Huffmyer JL, Estes H, Dexter F. Evaluating Competence of Anesthesiology Residents Including Procedures Performed Successfully and Entrustment Scales. Anesthesia and analgesia 2024;139(2):278-280. (In eng). DOI: 10.1213/ane.0000000000007021.
3. de Oliveira Filho GR. The construction of learning curves for basic skills in anesthetic procedures: an application for the cumulative sum method. Anesthesia and analgesia 2002;95(2):411-6, table of contents. (In eng). DOI: 10.1097/00000539-200208000-00033.
4. Weil G, Motamed C, Biau DJ, Guye ML. Learning curves for three specific procedures by anesthesiology residents using the learning curve cumulative sum (LC-CUSUM) test. Korean J Anesthesiol 2017;70(2):196-202. (In eng). DOI: 10.4097/kjae.2017.70.2.196.
5. Baker K, Sun H, Harman A, Poon KT, Rathmell JP. Clinical Performance Scores Are Independently Associated with the American Board of Anesthesiology Certification Examination Scores. Anesthesia and analgesia 2016;122(6):1992-9. (In eng). DOI: 10.1213/ane.0000000000001288.
6. Wanderer JP, de Oliveira Filho GR, Rothman BS, Sandberg WS, McEvoy MD. Implementation and Evaluation of the Z-Score System for Normalizing Residency Evaluations. Anesthesiology 2018;128(1):144-158. (In eng). DOI: 10.1097/aln.0000000000001919.