Test Successfully Assesses Psychotherapy Competency

A paper-and-pencil exam testing the competency of psychiatry residents in psychodynamic psychotherapy appears to be the first reliable and valid tool for measuring intellectual knowledge in an area for which there has been no gold standard of measurement.

More than five years in the making, the test—developed by psychiatrists and psychoanalysts at the Center for Psychoanalytic Training and Research at Columbia University College of Physicians and Surgeons and the New York State Psychiatric Institute—was used this year by nearly 60 residency programs to test the skills of more than 1,000 residents.

In a paper in the August American Journal of Psychiatry, Linda Mullen, M.D., and colleagues reported that an assessment of the test found it to be a valid and reliable predictor of competency.

The two-and-a-half-hour test was given to 36 psychoanalytic experts and 206 PGY-2 to PGY-4 psychiatry residents from 10 U.S. programs. Program directors provided information such as hours of psychodynamic didactic teaching and supervision and resident-conducted psychodynamic psychotherapy, and rated psychodynamic psychotherapy skills of PGY-4 residents based on cumulative supervisor reports.

The researchers found significant differences in test performance between residents and faculty experts, and between PGY-2 and PGY-4 residents, with more advanced residents and experts scoring progressively better.

The mean PGY-4 scores of different programs varied significantly, whereas PGY-2 scores did not. Higher test scores were positively associated with both the number of hours of resident-conducted psychotherapy and of supervision. Test scores also correlated significantly with program-director evaluations of PGY-4 residents in the sample.

“This is a test of competency that has been validated, and it is also a teaching tool that can be used in a number of ways to chart progress and guide the teaching of residents,” Mullen told Psychiatric News.“ It is easy to administer.. .and you get back a lot of information for the investment of that time.”

She is director of the psychotherapy competency test program at Columbia University College of Physicians and Surgeons and an assistant clinical professor of psychiatry.

Test Based on Case Vignettes

The test consists of case vignettes, drawn from clinical practice and disguised to protect patient confidentiality. Background information about the patient is followed by detailed descriptions of individual psychotherapy sessions. For each session, a multiple-choice question is presented that tests the resident's understanding of the dynamics of the session and the kinds of interventions that the therapist should pursue.

Ronald Rieder, M.D., director of psychiatric residency training at Columbia and one of the report's authors, said the test can be used to compare the skills of residents across a program, but also to compare overall program results with those from other residencies around the country.

“If your residents are doing poorly, program directors will know they have a job on their hands,” Rieder told Psychiatric News.

David Goldberg, M.D., president of the American Association of Directors of Psychiatric Residency Training, said the competency test is part of an effort among psychiatric educators to reinvigorate the teaching of psychotherapy in the context of an evolving—and sometimes contentious—debate about what the competencies of a psychiatrist should be.

“There has been a gradual shift in psychiatry that has been going for 20 years now toward pharmacology and diagnosis as the core of the field,” Goldberg told Psychiatric News. “But there continues to be a debate about where psychiatry should be heading. In the midst of all the advances in psychopharmacology, psychotherapy has become less and less an area that every training program puts a lot of training and time into. Educators and people in practice began to sense that psychiatry was losing its rigor and expertise incorporating psychotherapy and psychosocial dimensions in clinical work.”

`Deepening Marginalization' Cited

Nearly seven years ago, Goldberg initiated a task force of educators to look at psychotherapy training. Surveys of training directors confirmed what Goldberg called a “deepening marginalization” of psychotherapy in residencies.

One outcome of that focus was a change in training requirements calling for every psychiatry resident in the country to demonstrate competency in five modalities: psychodynamic psychotherapy, cognitive-behavioral therapy, supportive psychotherapy, combined psychotherapy with medication, and brief psychotherapy.

But to teach something is one thing; to measure it is another. “This is where the test comes in,” Goldberg said. “How do you measure competency?”

He believes that the proven validity of the test and its growing acceptance among residency programs across the country is an indication of success.“ This is truly a pioneering step, proving that it can be done,” Goldberg said. “It is the only test of its kind in psychiatry.... My residents love it. They think it's an interesting, relevant test, and it has spurred them to want to learn more about this area.”

But whether it is really possible to test how future physicians will perform in the “awesome now” of a clinical encounter with a patient—and in an area as deeply dependent as psychotherapy is on the individual skills of empathy and intuition—remains controversial. And the psychotherapy competency test is not without critics who say that a paper-and-pencil exam can't take the place of supervision of actual clinical experience.

“My response is that you don't take the exam as an overall measure of whether someone can practice,” Goldberg said. “But it does give you a lot of information that can be used in conjunction with a supervisory report, giving you a much more three-dimensional picture of what people know.”

Goldberg said that supervision alone can also be a misleading indicator of real clinical skill. “Under supervision, the resident can have the right emotional tone, but may not know that much technically,” he said.“ It's possible to cover that up. But because the competency test is so case-based, it takes different points in the therapy and asks questions about what residents would do in certain situations, testing whether they understand technically what is going on.”

And Rieder believes the test can serve as a model for measuring other modalities. “We are not only developing a specific test, we are testing a way of making tests,” he said. “There is no reason you couldn't do this for testing interpersonal psychotherapy.”

Residency directors interested in using the test can contact Mullen by e-mail at [email protected].▪