Background

In considering how we will train the next generation of future physicians, some have called for the need for “tectonic shifts” in graduate medical education to ensure that trainees are prepared to practice in tomorrow’s healthcare environment.Reference Phillips and Bitton ¹ Perhaps the greatest opportunity to shift how we train the next generation lies in the opportunity to develop new graduate medical education programmes. During a recent national meeting, Thomas Nasca, MD, President and CEO of the Accreditation Council on Graduate Medical Education (ACGME), parallelled the building of a new residency or fellowship programme in the era of the Next Accreditation System to the building of a new medical school. His perspective reflects the growing fiscal, administrative, and curricular demands that physicians, leaders, and educators now encounter in inaugurating new graduate medical education programmes.

In addition to a realisation that graduate medical education frameworks need to catch up to healthcare trends to best prepare our trainees to practice in a quickly changing environment, there is also the stark recognition that we have reached a “closed system” where residency applications continue to grow as a result of increasing medical school graduates, yet the number of residency slots has remained flat over the past 2–5 years.Reference Holt, Miller, Philibert and Nasca ² Thus, in the wake of the Affordable Care Act where physician shortages pose the greatest threat to healthcare access, the impact of developing new graduate medical education programmes, including residency and fellowship programmes, will be significant. Despite this need, a paucity of guidelines exists on best practices related to programme building. The opportunity to inform physicians, educators, and trainees on how to evaluate and implement training programme arose within the Johns Hopkins All Children’s Hospital International Symposium on Postgraduate Education in Pediatric and Congenital Cardiac Care. During this discussion, the following objectives were outlined: to overview the Next Accreditation System as a framework with the ACGME, discuss the opportunities and challenges that may exist in developing new programmes, and discuss future directions in assessment of training competency.

The Next Accreditation System: comparing old and new

In March, 2012, the ACGME introduced a new accreditation framework within graduate medical education, “The Next Accreditation System”. Nasca et alReference Nasca, Philibert, Brigham and Flynn ³ identified that, although the successes of the past accreditation system included improved traineeperformance on certifying exams, enhanced patient complexity and diversity offered to residents, and developing of educator career pathways for programme directors, the cost of this framework included the observation that “program requirements have become prescriptive, and opportunities for innovation have progressively disappeared”. Importantly, this sentinel paper validated what many educators within graduate medical education experienced as an overwhelming administrative burden related to obliging the ACGME reporting expectations in order to remain in compliance. This administrative burden comes as the cost of having committed educators reduce the time that they spend with individual trainees in supervisory or mentoring roles as well as incurring a substantial risk for programme director burnout – a well-known phenomenon in graduate medical education.

In comparing the “Next” (now current) accreditation system with the “Old”, many of the proposed changes have excited those in graduate medical education as steps in the right direction to address the aforementioned barriers. Some programmatic changes include that existing graduate medical education programmes will no longer be visited by the ACGME on an annual basis for programme assessment of regulatory compliance. Instead, existing programmes have been moved into a system by which specific outcome data will be reported electronically and continuously to the ACGME in an electronic dashboard (Web Annual Data System), where trends in data versus single data point will be the focus. In combination with continuous data reporting, programmes will be visited in 10-year cycles called “Self-Study Visits”, where a focus on resident and faculty performance, graduate performance, programme quality, and efforts towards yearly programme improvements will be the focus. The Next Accreditation System touts that “Gone are the days of PIFs (program information files)”, which consumed weeks to months of programme director’s administrative efforts in compiling pages and pages of programme detail for annual site visits. Instead, programmes will internally prepare Annual Program Evaluation where the cumulative summary of the past year’s feedback will be needed to outline goals for improvement within the following academic year. The move away from performing annual “biopsies” on programmes where assessments, often biased or limited by the circumstances of the site visit, has be lauded by many in graduate medical education as being a much more just reasonable method by which to allow programmes to stay focussed on programme improvement over time while still prioritising key annual programme metrics. Examples of the data that will now be reported on an annual or semi-annual basis to the ACGME include the following:

• ∙ Resident Survey

• ∙ Faculty Survey

• ∙ Trainee Milestones, which is defined in a later section

• ∙ Certification Exam Performance

• ∙ Case Log Data/Clinical Experience

• ∙ Hospital Accreditation Data

• ∙ Faculty Member Scholarly Activity

• ∙ Residency Scholarly Activity

Significant concerns exist over the increased emphasis on faculty scholarly activity within the Next Accreditation System. This is due to the fact that, in the past, defining scholarly activity was left to the programme director such that educational efforts including giving didactics, participating in local conferences, and/or community outreach might have broadly been considered faculty “scholarly” work. Interpreting scholarly activity is no longer an option for programme directors in that they must now report PubMed ID numbers as a discrete metric of scholarly activity. Although there are still options to describe grants, conferences, and didactic sessions, the WebADs dashboard calls immediate attention to the publications of the faculty within the programme. The emphasis is such that many have called concern to this new focus in perpetuating false publications in an effort to reduce the risk of citations from the ACGME.Reference Philibert, Lieh-Lai, Miller, Potts, Brigham and Nasca ⁴ This concern this has been by educational leaders given the general Next Accreditation System’s shifts towards promoting positive change in programmes versus penalising programmes for singular issues. Still, new and existing programmes should be prepared to face a new challenge where the efforts of faculty will need to more directly be tied to scholarly work.

Finally, the Next Accreditation System explicitly calls for educators to innovate. This call to action is perhaps the most exciting aspect of a new training framework in outlining that our trainees cannot and should not solely be trained with rubric that reflects medical knowledge, technology, and guidelines that are no longer current. Advances in precision medicine, genomics, and drug discovery obligate us to expose trainees to what is most current; however, past frameworks limited how this could be done robustly while still meeting curricular requirements. The next Accreditation System offers a greater degree of flexibility in addressing requirements where the hope is that graduate medical education leaders can innovate within their institutions and with faculty to optimise exposure to the most current tools and practices within the field.

New graduate medical education programme development: opportunities and challenges within the Next Accreditation System

New categorical and fellowship programme development has been largely encouraged by the ACGME. Thus, the opportunity to introduce new programmes for accreditation is significant across specialties and across our county. Support for new programme applications has been operationalised by streamlining the programme application process. Institutional leaders along with their programme directors now complete electronic templates (Specialty Program Applications) versus paper applications. In addition, new programme site visits are scheduled with much greater notice to allow faculty, institutional, and programme leadership greater time for preparation. In addition to some of these pragmatic changes related to new programme applications, site visitors have been tasked with relaying greater transparency at the end of site visits where preliminary concerns and strengths should be discussed with the programme director before receiving formal notification from the ACGME review committee.

Although a majority of existing programmes now benefit from an expanded window of time between ormal ACGME on-site reviews, new programmes must live between new and old frameworks in trying to achieve full accreditation status. Since implementing Next Accreditation System, the ACGME reports that ~75–80% of programmes were placed into “Continued Accreditation” or will comply with the 10-year self-study framework, 10–15% of programmes were categorised into “Accreditation with Warning” or “Probationary Accreditation” where citations and/or observed trends put the programme out of compliance in some manner, and 2–4% of programmes were approved for “New Program Accreditation”. ⁵ Fewer than 1% of programmes had withdrawal of accreditation since Next Accreditation System has been implemented, which reflects the ACGME’s commitment to supporting graduate medical education programmes versus limiting the number of training opportunities nationally.

Although existing programmes benefit from greater windows of time between accreditation on-site reviews, new programmes may be expected to complete at least one to two on-site visits within the time of their initial application before moving towards continued accreditation. Within our own experience as a newly established training institution at Johns Hopkins All Children’s Hospital, a new cate gorical paediatric residency programme was submitted for review in addition to a new general paediatric surgical fellowship immediately following the implementation of Next Accreditation System in 2012. Some of the challenges we observed in our experiences towards new programme accreditation included that the continual clinical and trainee reporting expectations required a robust information technology system as well as electronic medical record. Further, Our experiences with the ACGME and Next Accreditation System framework led us to recognise that existing programmes are encouraged to innovate only upon having achieved “continued accreditation” status where immediate outcomes and programme trends have been observed to be favourable. Thus, new graduate medical education programmes must comply more tightly with specialty requirements before innovating or changing “standard” training experiences. Institutions considering the establishments of new residency or fellowship programmes must be prepared to address the demands related to the need for continuous data monitoring and reporting necessary for accreditation. In fact, at a recent paediatric programme director’s grassroots discussion, the need to have data on resident’s clinical practices was felt to be so significant that many programme directors felt that the ACGME should consider including an health informatician and data analyst as required programme staff as Next Accreditation System moves forward. Given the growing awareness that discrete metrics including number of inpatient encounters, number of procedures, number of continuity clinics completed, and total diagnoses observed (just to name a few core metrics), the development of each new graduate medical education programme will require robust electronic medical record systems as well as data reporting frameworks to comply with Next Accreditation System requirements. The opportunity for establishing this type of data reporting system includes being able to respond to gaps in each training programme’s observed metrics – for example, low volumes and absent patient populations. These data have aided our new programmes’ ability to revise, supplement, and adjust experiences if needed in a more immediate manner than previous mechanisms would have provided.

Establishment of new programmes also offers the challenge of recruiting the right type of applicant. “What makes your program different from others?” was a question that was heard in the early days of our growth and remains the top “Frequently Asked Question” regarding our new residency. Regardless of whether a graduate medical education training programme is reputed or highly functioning, we noted that most students are heavily influenced by the historic roots of an institution or programme as one of the most appealing reasons for them to apply to train at any particular residency programme. Therefore, the concept of a new programme is not something that is easily messaged to many aspiring medical students who are heavily influenced by the traditions and history that define many United States graduate medical education programmes. It is therefore critical in the establishment of new graduate medical education programmes to develop, promote, and communicate what makes the programme distinctive from other, longer-standing residency or fellowship programmes.

A realisation of the distinct needs and desires of the millennial generations has been the source of much discussion among Undergraduate Medical Education and graduate medical education leadership.Reference Cahill and Sedrak ⁶ Building a new programme requires acknowledging the unique needs of “generation Y” while also distinctively seeking a phenotype in personality, goals, and overall resilience that not all millennial medical students have. The vision for our new residency programme hinged heavily around the goal to develop physician leaders with a unique set of skills necessary to advocate, build, and inform the future of paediatric clinical practice. During our recruitment process, we were very transparent with all applicants when asked the student standard question “What are you looking for in residency?”. Throughout our past two interview seasons, we have messaged the need to include residents with a desire to build new experiences and to recruit those with an interest to grow their leadership skills. Within the first few years of a new programme, it is inevitable that many clinical experiences may be “firsts” for not only the resident as well as the faculty and other team members. Therefore, screening for prospective trainees with experiences in educational programme building, entrepreneurship, and other traits reflective of professional maturity were key important characteristics felt to align with the profile of our new programme. It was also our programmatic goal to be very transparent to interested applicants regarding the “startup” efforts that were ongoing from an institutional perspective. This was critical to maintaining engagement from our residents upon starting their training. Open communication streams were offered to our residents with programme leadership as well as hospital leadership. The value of our resident’s feedback and insights in their early experiences has promoted a deep sense of commitment to our programme and institution and has facilitated communication and leadership skills for our first interns during this unique growth period. Transparency and open access to programme leadership among a group of pioneering residents remain hallmarks of our graduate medical education programmes.

Future directions in evaluation and assessment of training competencies

A major tenet of the Next Accreditation System centres on the introduction of a new framework of evaluation to determine trainee competency. “Milestones” reflect the natural progression of trainee development through core competencies where the aim is to create a more objective, logical trajectory towards professional development. Milestones have explicitly been developed for each specialty with the input of programme directors, trainees, and specialty organisations – for example, American Board of Pediatrics – to effectively articulate in narrative form what trainees should be doing relative to knowledge, attitudes, and skills as they progress from novice towards expert. ⁷ Milestones are a deliberate move away from the traditional Likert scale assessment framework. For example, in this past assessment framework, an intern might have received a “level 5” on a scale of 1 thru 5 for a competency such as patient care. Without a sense of why they had reached this level or how to continue progression towards mastery, this framework limits trainees from understanding the behaviors needed for each competency as well as prevents and understanding of what behaviors they should be seeking to implement in order to progress. Milestones have now been established for each specialty and within the Next Accreditation System, and will be reported by each training programme on a bi-annual basis as part of the annual data summary. At their core, milestones provide the opportunity to more deliberately describe skills and behaviours that ground the more abstract language of competencies, that is, systems-based practice and professionalism, while also informing learners about the granular behaviours we, as supervising faculty, seek to observe at each training level.

Conversely, the debate surrounding milestones includes the observation that the effectiveness of a new evaluation framework depends upon faculty understanding of the framework – that is, it remains a challenge for new and existing graduate medical education programmes to indicate to faculty that trainees demonstrating high proficiency in a respective competency should not translate to an assessment of “expert” if they assess their performance level as favourable or adequate. In fact, the deliberate act of considering where on a spectrum of milestones each trainee’s skill level falls is added work and consideration for teaching faculty that will take time to implement consistently across training programmes. Trainees must also adjust to a perspective that a high degree of competency in patient care does not translate to being considered an “expert” but instead reflects a competency consistent with a 1st year resident. This shift in evaluation language and framework is still a work in progress across the nation; however, for new programmes, there must be a deliberate introduction of milestones such that faculty and trainees can align with the new ACGME evaluative process.

In addition to milestones, the concept of entrustable professional activities was introduced as another conceptual model by which trainees can more effectively be assessed. Although many agree that the concept of competencies, progressive development, and milestones will be challenging to integrate into the rubric of supervising faculty, graduate medical education leaders similarly agree that the concept of “entrustment” is inherently intuitive within our training rubric – that is, whether a resident has the ability to perform an activity at a desired level of performance without direct supervision is a much clearer threshold by which to evaluate performance and skill. Further, the concept that a trainee can be “trusted” to perform a particular activity is anchored on being directly observed to successfully perform this activity consistently over time. Said simply, it is much easier for a faculty member to answer “Do you trust this person to provide care for a patient referred for an acute cardiac problem without direct supervision?” than it is to answer “Is this person competent in Practice Based Learning, Systems Based Practice and Professionalism?”. Thus, the real-world applicability of entrustable professional activities provides a new horizon by which trainees and faculty may increase the effectiveness by which evaluations of performance are communicated. At present, 17 entrustable professional activities have been developed within paediatrics, for example, care of the well newborn and providing recommended health screening, although each specialty has generated its own entrustable professional activities relative to what each subspecialist is expected to do in everyday practice. Thus, within the Next Accreditation System, milestones address the gap in formative feedback needed to provide a very granular approach to assess and give formative feedback to trainees towards progression. Holistically, entrustable professional activities provide a broader approach to assessment such that trainees are more robustly considered within a clinical context where clusters of behaviours are outlined as metrics by which trainees may be given autonomy and trusted to care for patients (Fig 1).

Figure 1

Entrustable professional activities (EPAs) and milestones, putting it all together (adapted from American Board of Pediatrics). ⁸