| Article Table of Contents | ||||||
|---|---|---|---|---|---|---|
| Abstract | Introduction | Methods | Results | Discussion | Limitations | Conclusion |
Abstract
Education in point of care ultrasound for entering emergency medicine residents has traditionally been delivered over a 4-week block rotation which we have associated with several drawbacks. A longitudinal postgraduate year-1 (PGY-1) point of care ultrasound (POCUS) rotation was developed to address these deficiencies and compared to our block rotation. The primary objective is to observe the effect that a longitudinal ultrasound curriculum has on knowledge acquisition and practical ultrasound scanning skills. The secondary outcome is resident stated preference for a curriculum format. This is a prospective case control study. From July 2016 to July 2018 30 PGY-1 Emergency Medicine residents participated. Residents entering in 2016 (n=15) participated in our traditional block 4-week rotation and residents entering in 2017 (n=15) participated in a 28-week longitudinal rotation during Emergency Department rotations. In January of the PGY-1 and July of the PGY-2 year the faculty performed direct observation testing (DOT) of echocardiogram (ECHO) and Focused Assessment of Sonography in Trauma (i.e., FAST) examinations. Residents also completed a demographic and self-assessment survey. Residents in the longitudinal rotation had higher DOT scores at both assessment periods versus the block rotation (June: 13.4 vs 16.7, p= 0.04; July: 17.8 vs 20.7, p=0.02). Knowledge based quiz scores were not significantly different. Eighty-nine percent of all surveyed residents preferred a longitudinal curriculum. Our pilot data suggests a longitudinal POCUS rotation may be superior to the traditional block rotation in developing POCUS skills as illustrated by higher DOT scores for ECHO. This longitudinal approach provided a more consistent POCUS education experience that was preferred by the residents and should be considered when planning a residency curriculum.
Introduction
Formal training in Point of Care Ultrasound (POCUS) has been a Residency Review Committee requirement for graduates of allopathic emergency medicine (EM) training programs since 2016.1 POCUS as a component of EM residency training has been increasingly incorporated into residency programs over the past few decades. Published descriptions of these EM ultrasound curricula are few and serve primarily as guidelines. In 1998 Cook and Roepke published a report of the prevalence of established EM residency ultrasound curricula. At that time only one-half of the responding ninety programs had ultrasound curricula.6 The Council of Emergency Medicine Residency Directors established guidelines in 2008 in order to provide minimum education standards for EM residency ultrasound training.7 Presently, all EM residency programs must incorporate POCUS into the curriculum and the Accreditation Council for Graduate Medical Education (ACGME) has defined a sub competency milestone related to EM residency ultrasound training.1,8 Despite the wide implementation of resident training in ultrasound very little has been written about how that education curriculum should be delivered.
Anecdotal evidence suggests that the primary structure for this training is a single 4-week rotation during intern year. This traditional anesthesia-ultrasound rotation combines a morning experience with anesthesia staff practicing airway management and an afternoon session with EM ultrasound faculty completing didactic and bedside instruction in POCUS. Since 2004, our PGY-1 ultrasound instruction was delivered within this anesthesia-ultrasound block model. Annual feedback from our EM residents has shown recurrent challenges with block curriculum contributing a negative impact on their ultrasound experience. A longitudinal PGY-1 ultrasound rotation was developed to address these deficiencies and compared to our block rotation.
To our knowledge, there is no published data regarding longitudinal-based ultrasound rotations for PGY-1 EM residents. Numerous studies in the medical school literature support longitudinal ultrasound curriculum, however it has yet to be studied in residency programs. 3-5 Boulger et. al.2 have recently described an advanced track curriculum for EM house staff with a special interest in ultrasound delivered over the course of three years. There have been few studies performed regarding optimal curricula designs. Cartier et al.9 surveyed residents participating in one particular POCUS course and found residents favored small group format with live demonstrations and, most importantly, hands-on scanning however no details were provided on scheduling.
With residency growth from 10 residents per class in 2006 to 15 residents per class in 2014, our residents and ultrasound faculty have found this block scheduling to be unsatisfactory. The most troublesome aspect of block scheduling has been that at least half of the class receives less structured ultrasound education until the second half of the academic year. Affected residents universally described this as a negative aspect of their training. Residents were also unequally affected by seasonal scheduling factors that reduced the number of supervised bedside scanning shifts such as the December holiday schedule or the summer vacation season. Our resident complement directly contributes to an unequal teaching environment as several blocks must accommodate two residents for the entire rotation. Ultrasound faculty found the block schedule problematic as residents that require additional ultrasound instruction or remediation sometimes are not identified until later in the academic year, when opportunities to schedule additional instructional time are limited. Furthermore, in the block model it is difficult to ensure that residents progress efficiently through the curriculum when each learner may be at a different point.
To address these challenges, we developed a longitudinal ultrasound rotation for our PGY-1 residents. We hypothesized that this longitudinal rotation would optimize content delivery by maintaining instruction content and timing uniform throughout the academic year. Our residents would participate in bedside teaching and complete our didactic curriculum continuously over their entire PGY-1 year. To accomplish this, we assigned shifts during every ED block rotation for the PGY-1 residents to participate in bedside ultrasound instruction with faculty in groups of 2 residents. This allowed for at least the same amount of bedside instruction as the block rotation it replaced. Residents were able to complete our standard online didactic and knowledge assessment evaluations at a self-directed pace (Figure 1). In order to reduce the clinical shift load during the EM block the PGY-1 resident is required to work clinical shifts in the ED during the anesthesia rotation.
Methods
The data presented are the results from a case-control observational study based on our internal QI data collected to assess the efficacy of the longitudinal rotation. Data was collected prospectively from July 2016 to July 2018. All PGY-1 residents were eligible to participate. Participants were informed of the goals of the assessments and general expectations for the rotation at the beginning of each academic year. The study was reviewed and approved by our institutional review board and determined to be exempt.
The primary emergency department setting for our training program is a 120,000 annual visit ED with an on-campus Pediatric ED and an annual US volume of >6000 educational, diagnostic, and procedural studies. Additionally, >95% of faculty are credentialed to perform POCUS examinations. PGY-1 residents entering training in July of 2016 and completing our traditional block rotation serve as the control group. These residents completed 8 three-hour supervised bedside scanning shifts as well as 8 three-hour unsupervised scanning shifts over a 4-week rotation. They were required to complete an online curriculum of lectures and quizzes during the block. PGY-1 residents entering training in July of 2017 and completing the longitudinal ultrasound rotation and comprise the intervention group. The longitudinal rotation included two three-hour supervised ultrasound shifts during each of seven 4-week EM blocks. In quarterly intervals they completed the same assigned ultrasound lectures and quizzes as the control group. Both classes participated in an Introduction to Ultrasound boot camp that took place during the first week of residency with a focus on core examinations including abdominal aorta, cardiac, Focused Assessment with Sonography of Trauma (FAST), biliary ultrasound, and ultrasound guided IV placement.
At the conclusion of each intern ultrasound rotation residents completed a voluntary survey about their ultrasound curriculum experience. The intern Ultrasound Curriculum Evaluation Survey (Appendix 1) was completed by both groups. The survey questions are developed annually by our ultrasound faculty to obtain feedback to prepare for the following year. The goal is to collect data on previous ultrasound training, curriculum preferences, and self- reported knowledge and performance of core EM ultrasound modalities.
To compare the longitudinal curriculum to the block curriculum the groups completed direct-observation testing in January of the PGY-1 and July of the PGY-2 year. During the testing, participants were asked to complete a limited bedside echocardiogram and a FAST examination on the same adult male patient from our standardized patient pool. These modalities were chosen because they are recognized as core scans in the clinical training and practice of EM, encompass thoracic, abdominal and pelvic scanning techniques, and require different probe options. The decision of which ultrasound modalities would be tested was not shared with the residents prior to the testing. It was strictly communicated that they be prepared for testing on any one of the core EM ultrasound modalities.
The resident performance was scored on a standardized direct observation score sheet by two ultrasound faculty with the average score utilized (Appendix 2). The standardized scoring rubric has not been published previously and was initially developed and tested on residents from a different EM residency program in 2014. Feedback from this experience allowed for adjustments to provide an objective scoring assessment of image acquisition. Scores for each standard view ranged from 0 to 3. An image score of 3 was awarded if the learner could demonstrate to the examiner the correct target anatomy (1 point), use the correct axis with complete dimensions of the anatomical structure (1 point), and use of adequate depth, resolution and gain settings (1 point). For example, a score of 0 was given if no identifiable image of target anatomy was obtained such as bladder not visualized in the pelvic FAST view. A score of 1 was given if image showed incorrect axis or did not show entire target region such as inability to visualize left atrium in subxyphoid view. A score of 2 was provided if entire target region visualized with poor resolution and gain or adjustment required, but not performed; for example, an apical four chamber view if all four chambers visualized with septum in vertical axis, but the image was significantly hypoechoic. A score of 3 was given if excellent view of entire target region with no adjustments in gain or depth required.
The data was analyzed using SAS 9.4 (Cary NC). Comparisons of instructor score ranks were made using Wilcoxon Rank Sum tests. Alpha of < 0.05 was used for statistical significance.
Results
All residents successfully completed the PGY-1 academic year. Twenty-eight residents completed the curriculum survey (13 control, 15 intervention) with a survey response rate of 93.3%. Ultrasound training prior to residency was similar between the groups with 40% and 33% of the control and intervention groups respectively reporting no US education prior to residency and 26% and 20% reporting >20 hours of US education. On a 5-point Likert scale there were no significant differences in self-reported knowledge retention, skill retention, or application specific skills (eFAST, Cardiac, Aorta, US guided needle placement, first trimester ultrasound, and renal ultrasound). The intervention group reported significantly increased skill progress (3.3 v. 4.4, p = 0.01) with the longitudinal curriculum and self-reported modality specific skill improvement with the gallbladder examination (2.7 v 3.4, p = 0.02).
Direct observation ultrasound skills were better among the intervention group than the control group in both January and July (Table 1). In January the mean DOT scores for the intervention group were 8.2 and 8.5 for ECHO and FAST respectively compared with 6.1 and 7.3 for the control group. On repeat testing in July the DOT scores for the intervention group were 10.6 and 10.1 for ECHO and FAST respectively compared with 8.4 and 9.4 for the control group. Both the ECHO (P = 0.04, P < 0.01) and total (P = 0.03, P < 0.02) scores for the intervention group were statistically higher. One resident in the control group (N=15) did not complete the standardized DOT during the July session, therefore data for that resident was not included in the direct observation data analysis.
Performance among applications was unequal, with more improvement seen in cardiac ultrasound skills than in the FAST examination assessment. Although both groups saw similar overall improvement during the year, the higher achievement level among the intervention group persisted in July. Mean quiz scores were similar between the groups (89% v. 87%, p = 0.13).
Discussion
Emergency physician use of POCUS in current emergent care is ubiquitous and it is considered a core skill. Presently, all EM residency programs must incorporate POCUS into the curriculum and the ACGME has defined a sub competency milestone related to EM residency ultrasound training.1,8 Despite the wide implementation of resident training in ultrasound very little has been written about how that education curriculum might be delivered.
There is an emerging body of literature describing medical school longitudinal ultrasound education with hands-on simulation training on models and standardized patients. Several institutions have described positive feedback from medical students with regards to a progressive POCUS curriculum through 4 years of training. Typically, these include web-based lectures, hands on peer instruction and practice, instructor-led demonstrations, and skill assessments.3-5 The emerging success of these curricula continues to be studied, but suggests similar adoption in EM training programs, where residents would potentially benefit from longitudinal rather than traditional block ultrasound rotations.
We found that converting the resident experience to a year- long format allowed us to deliver more hours of bedside instruction with the same amount of faculty time. A number of competing factors may affect a program’s ability to create similar curricula such as ED clinical responsibilities, required off-service rotations, and the availability of teaching staff. Our rotation structure is innovative because it allows teachers to evaluate the skills of an entire class over the course of a year as a cohesive whole. Any member of the PGY-1 class can easily be compared to the progress of their peers or prior classes to see if they are falling behind in skill acquisition or didactic completion. As a result, poor performers are identified early and have opportunities for remediation. When compared to the traditional block schedule, all residents have equal access to the training needed to participate in POCUS examinations when off-service and on clinical shifts.
Our trainees in the longitudinal rotation equaled or exceeded the performance of their peers over the academic year. We also discovered that the longitudinal rotation facilitates more meaningful supervised scanning time. The importance of supervised, regular scanning sessions cannot be overstated. As discussed by Noble et al.10, there was greater improvement in POCUS knowledge when participating in a proctored ultrasound training rotation, rather than relying on independent scanning for the majority of study acquisition. This curricular intervention promoted frequent, repetitive exposure to core didactic content and continuous skills training. There is considerable literature describing the benefits of spaced learning theory in a wide range of educational applications. Curricular design should utilize spacing and encourage learners to distribute their study time.11 The deployment of this learning strategy in medical education has also been described in recent work. The spacing effect, or "distributed practice" promotes more durable learning with repeated exposure to material over time.12,13 Dolan et al.14 demonstrated better knowledge retention and improved care quality at 10 months among residents randomized to a curricular intervention with repeated practice. These findings are consistent with our results in which the intervention group reported significantly increased skill progress and self-reported skill with the gallbladder exam. Observed ultrasound skills were also better among the intervention group than the control group.
Test-enhanced learning theory suggests that tests given often and spaced out in time require effortful recall which promotes retention of clinical knowledge.15 Distributive spacing and interval testing is associated with better knowledge retention especially when assessments are conducted in retrieval format (ie., short answer) and not recognition format (i.e., multiple choice).16 Available data to inform optimal spacing intervals is limited. Cepeda et al.17 suggest optimal timing to be 5-10% of the duration the information is desired to be retained (i.e., monthly testing for a 1-yr retention goal). Our learners underwent direct observational summative testing at regular 6-month intervals. More frequent ultrasound knowledge testing and skills assessment is a focus for future study.
Limitations
This is a small pilot study at a single institution. We would have liked to be able to randomize assignment to the block schedule or the longitudinal schedule but the logistics of scheduling our house staff prevented this. Neither group nor the instructors were blinded to the intervention at any point, potentially introducing bias. Our sample sizes were small and limited to two classes of residents; it is possible that unique individual characteristics had a cofounding effect on the results. Interestingly the control group included two residents who graduated from medical schools offering an integrated ultrasound curriculum. We were unable to perform a power calculation and sample size estimation as there was no prior research in this area on which to base the calculations. It would be ideal to repeat this study as a multi-center study, with the current study serving as the pilot.
The observed clinical examination was performed in a setting that is unlike the general ED setting for POCUS studies and it is possible that the test environment and nature of the test affected the results. We attempted to control for variability in faculty assessment of skills by using two assessors. The unique direct observation scoring rubric utilized has not been published previously. In addition, our results do not address how trainees may use ultrasound after the PGY-1 year or completion of their residency training, two outcomes which most ultrasound instructors would find important.
A key outcome of the study is the rotation format preference among residents and some may find this measure too subjective. While resident preference by itself should not be the cornerstone of any training program, we believe it is an important consideration when designing curricula. Incorporating resident feedback can help both motivate the resident for the experience and be the stimulus for making change. In our experience, it was the resident dissatisfaction with the block rotation that was an important consideration when we were designing our curricular modification.
Conclusion
Pilot data on the longitudinal POCUS rotation suggest it may be superior to the traditional block rotation in developing and improving ultrasound skill set throughout EM PGY-1 year. This was demonstrated by higher DOT scores for ECHO and combination scores, along with resident preference for the longitudinal curriculum. Although the longitudinal curriculum is associated with scheduling challenges, it provides a consistent POCUS education experience that has the potential to significantly improve the effectiveness of ultrasound training for resident learners.
Appendix 1. Intern Ultrasound Curriculum Evaluation Survey
What resident graduation year are you?
What ultrasound training experience did you have prior to starting residency?
Throughout your intern EM year how well did you feel that you retained POCUS knowledge?
Throughout intern year how well did you feel you retained POCUS skill set?
Overall, how well did your POCUS education progress throughout intern year?
Based on your experience would you prefer a single block of dedicated ultrasound experience or a longitudinal experience spread throughout intern year?
How strongly does the POCUS education lecture series enhance your POCUS education?
How strongly does the bedside teaching sessions enhance your education?
How would you rate the ultrasound scanning shift scheduling throughout intern year?
Do you feel the biannual direct observation testing motivated you to maintain your POCUS knowledge and skill-set?
Rate your own E-FAST exam performance:
Rate your own cardiac echo exam performance:
Rate your own aorta ultrasound exam performance:
Rate your own ultrasound needle guidance performance:
Rate your own first trimester pregnancy ultrasound performance:
Rate your own gallbladder ultrasound performance:
Rate your own renal ultrasound performance:
Please provide further feedback on your intern POCUS experience:
Appendix 2. Direct Observation Testing Scoring
Rubric Cardiac
*For each core view a score of 0 to 3 was given based upon if the learner could demonstrate: the correct target anatomy (1 point), use the correct axis with complete dimensions of the anatomical structure (1 point), and use of adequate depth, resolution and gain settings (1 point).
Patient position adjustments only if needed to improve views.
FAST exam
*For each core view a score of 0 to 3 was given based upon if the learner could demonstrate: the correct target anatomy (1 point), use the correct axis with complete dimensions of the anatomical structure (1 point), and use of adequate depth, resolution and gain settings (1 point).
Patient position adjustments only if needed to improve views.
Disclosures
This work was presented at the World Congress of Ultrasound in Medical Education in September 2018 and 2019. No financial support given. Authors report no conflicts of interest.
- ACGME Program Requirements for Graduate Medical Education in Emergency Medicine. Accreditation Council for Graduate Medical Education website. http://www.acgme.org/Portals/0/PFAssets/ ProgramRequirements/110_emergency_medicine_2017-07-01.pdf. Published Sept. 30, 2012. Updated July 1, 2017. Accessed January 25, 2019.
- Boulger C, Adams DZ, Hughes D, Bahner DP, King A. Longitudinal Ultrasound Education Track Curriculum Implemented Within an Emergency Medicine Residency Program. J Ultrasound Med. 2017 Jun;36(6):1245-1250.
- Bahner DP, Adkins EJ, Hughes D, Barrie M, Boulger CT, Royall NA. Integrated medical school ultrasound: development of an ultrasound vertical curriculum. Crit Ultrasound J. 2013;5(1):6.
- Fox JC, Schlang JR, Maldonado G, Lotfipour S, Clayman RV. Proactive medicine: the "UCI 30," an ultrasound-based clinical initiative from the University of California, Irvine. Acad Med. 2014; 89:984–989.
- Hoppmann R, Cook T, Hunt P, et al. Ultrasound in medical education: A vertical curriculum at the University of South Carolina School of Medicine. J S C Med Assoc. 2006; 102:330–334.
- Cook T, Roepke T. Prevalence and structure of ultrasound curricula in emergency medicine residencies. J Emerg Med. 1998 July- Aug;16(4):655-7.
- Akhtar S, Theodoro D, Gaspari R, et al. Resident training in emergency ultrasound: consensus recommendations from the 2008 Council of Emergency Medicine Residency Directors Conference. Acad Emerg Med. 2009; 16 (2): S32–S36.
- American College of Emergency Medicine. Ultrasound Guidelines: Emergency, Point-of-Care and Clinical Ultrasound Guidelines in Medicine. Ann Emerg Med. 2017;6995: e27-e54.
- Cartier RA 3rd, Skinner C, Laselle B. Perceived effectiveness of teaching methods for point of care ultrasound. J Emerg Med. 2014 Jul;47(1):86- 91.
- Noble V, Nelson B, Sutingco AN, Marill KA, Cranmer H. Assessment of knowledge retention and the value of proctored ultrasound exams after the introduction of an emergency ultrasound curriculum. BMC Med Education. 2007; 7(1):40.
- H.C. Gooing, K Mann, E Armstrong. Twelve tips for applying the science of learning to health professions education. Medical Teacher. 2017; 39:1, 26-31.
- Dunlosky J, Rawson KA, Marsh EJ, Nathan MJ, Willingham DT. Improving students' learning with effective learning techniques: promising directions from cognitive and educational psychology. Psychol Sci Public Interest. 2013;14:4–58.
- Augustin M. How to learn effectively in medical school: test yourself, learn actively, and repeat in intervals. Yale J Biol Med. 2014 Jun 6;87(2):207-12.
- Dolan BM, Yialamas MA, McMahon GT. A randomized educational intervention trial to determine the effect of online education on the quality of resident-delivered care. J Grad Med Educ. 2015;7(3):376–381.
- Larsen DP, Butler AC, Roediger HL 3rd. Test-enhanced learning in medical education. Med Educ. 2008 Oct;42(10):959-66.
- Yeh DD, Park YS. Improving Learning Efficiency of Factual Knowledge in Medical Education. J Surg Educ. 2015 Sep-Oct;72(5):882-9.
- Cepeda NJ, Vul E, Rohrer D, Wixted JT, Pashler H. Spacing effects in learning: a temporal ridgeline of optimal retention. Psychol Sci. 2008; 19:1095–1102.