Outcome Measures Are Foundation of Revised Accreditation System
Abstract
A new accreditation system for residency programs emphasizes outcome-based measurements and a revamped system for assessing program performance.
The so-called “Next Accreditation System” envisioned by the Accreditation Council for Graduate Medical Education (ACGME) is one that will move accreditation of residency training to an “outcome-based” system providing continuous program assessment.
Chris Thomas, M.D., told AADPRT members that residency programs can use “milestones” to identify program strengths and opportunities for improvement and to benchmark each program against the performance of other programs.
In an address to educators at the annual meeting of the American Association of Directors of Psychiatric Residency Training (AADPRT), Chris Thomas, M.D., the chair of the ACGME’s Psychiatry Residency Review Committee, outlined major features of the Next Accreditation System, as the ACGME calls it. The system, articulated by Thomas Nasca, M.D., in a 2012 article in the New England Journal of Medicine, is a comprehensive restructuring of accreditation based on the assessment of residents’ achievement of specific educational outcomes—codified in “milestones” developed by each specialty—that are linked to the six core physician competencies the ACGME formulated in 1999.
The new system also involves a transformation of program assessment, replacing site visits with “continuous observation” (see box).
At the AADPRT meeting, Thomas said the system will be a major departure from a longstanding system of periodic site visits and what he called a “process-based” system of resident assessment that was not linked to acquisition of specific physician skills.
“Under our previous system, trainees were required to do so many months of internal medicine, so many months of neurology, with the expectation that just by attending and showing up, the trainee would somehow become a competent physician,” he said.
Key to this transformation is the introduction of the Milestone Project, in which psychiatry and all other specialties are developing specific goals, or milestones, for assessing resident progression toward competency. A set of 22 milestones for psychiatry in the six domains of physician competency—medical knowledge, patient care, practice-based learning and improvement, systems-based practice, professionalism, and interpersonal and communication skills—were finalized last year and will be implemented for residency programs in July (Psychiatric News, August 28, 2013).
“The purpose of a revamped site-visit system is to fully implement outcome-based accreditation, provide continuous program assessment, free good programs to innovate, assist poor programs to improve, reduce the burden of accreditation, and provide accountability to the public,” explained Chris Thomas, M.D., chair of the ACGME’s Psychiatry Residency Review Committee, to educators at the annual meeting of the American Association of Directors of Psychiatric Residency Training.
Under the new system, site visits will fall into one of three categories: a site visit by a team of two from the ACGME every 10 years as part of the annual self-study, conducted by the program itself; focused site visits; and full site visits. Thomas said that “program information forms” will no longer be required.
A “focused site visit” will be conducted with minimal notification and will require minimal document preparation, when the review committee has specific questions about specific problems or about how the program is conducted in specific areas.
A “full site visit” will be conducted with 60-days’ notice when a new core program has been created, a program’s initial accreditation ends, or the review committee identifies broad issues or major concerns, Thomas said.
Citations will be reviewed annually and can be removed quickly when a program submits a progress report, when new information becomes available about the program performance as indicated by resident or faculty surveys and milestone data, or during findings from a focused or full site visit.
In addition to performance against the milestones (see story above), accreditation will be based on the Annual Accreditation Data System, which is updated to record program changes, core faculty, and residents; program characteristics; faculty and resident scholarly activity; board pass rate (based on a rolling average); and surveys of residents and faculty.
Only physicians count as core faculty, with “core” defined by time devoted to residency program (including clinical, didactic, research and administration). For general psychiatry, this means at least 15 hours per week (for subspecialties, at least 10 hours per week).
“The milestones move accreditation from structure- and process-based to outcomes-based assessment,” Thomas said. “They are intended to follow the trainee’s progression from entrance into residency to practice as an independent clinician. They should help to articulate shared expectations across residency programs, establish aspirational goals of excellence, and provide a framework and a language for common discussion.”
Thomas said residency review committees will receive only aggregate program data, not individual reports on individual residents. Individual residents will be assessed by a Clinical Competency Committee within each program; the task of the committee is to review all resident evaluations semiannually, prepare milestone ratings of each resident, and advise the program director on resident promotion, remediation, probation, or dismissal.
The project is a major change in the way programs operate, and at the AADPRT meeting there was at least some degree of concern—ranging from uncertainty about how it would work to skepticism that it would work at all—but Thomas urged program directors “not to panic.”
In a separate presentation at the AADPRT meeting by program directors from other specialties, Conrad Clemens, M.D., M.P.H., associate dean for graduate medical education at the University of Arizona, backed Thomas’s call for calm in the face of the changes. Clemens said using the milestones at his own institution was easier—or not as difficult—as many had expected. “Some faculty took me aside and said, ‘You know, it’s really not as bad as we thought it would be,’ ” he said.
And Thomas urged directors not to discard the assessment tools they are familiar with and have been using. In fact, the milestones are not, he said, an assessment tool per se; rather, they are a map charting developmental goals in the progression toward independent practice—the project preserves the independence of programs to use their own methods for assessing the achievement of those goals.
He said programs can use the milestones to provide feedback to residents, benchmark residents to the mean overall performance of the program, identify program strengths and opportunities for improvement, and help benchmark each program against the performance of other programs around the country. Residents, in turn, can use the milestones to garner feedback about specific tasks and skills, determine their own strengths and training goals, and better learn the objectives of each rotation.
The first reports on milestones for general psychiatry programs are expected to be filed between November 1 and December 31. Milestones for psychiatry subspecialties are expected to be completed this December for use in July 2015. ■
