Housing Programs Effective, but Studies Plagued by Limitations

Sam Tsemberis, Ph.D.

Evidence for three program types aimed at reducing homelessness among patients with mental illness was rated as “moderate” in papers published as part of the “Assessing the Evidence Base” series in the March Psychiatric Services.

Supported housing, recovery housing, and residential treatment for people with substance use disorders were each rated as having a moderate level of evidence for effectiveness.

In each case, authors reviewing the evidence for the program types indicated that the programs should receive support from policymakers but are challenged by a lack of a clear definition of service methods, treatment duration, and treatment standards. For instance, regarding recovery housing—which provides supervised short-term housing to help stabilize people with substance use disorders so that they can take their first steps toward recovery in a supportive environment—reviewers stated “studies consistently showed positive outcomes, but the results were tempered by research design limitations, such as lack of consistency in defining the program elements and outcome measures, small samples, and single-site evaluations and by the limited number of studies.”

Similarly, regarding residential treatment for people with substance use disorders, reviewers stated that “a number of randomized, controlled trials were identified, but various methodological weaknesses in study designs—primarily the appropriateness of the samples and equivalence of comparison groups—decreased the level of evidence.”

Even with regard to supported permanent housing—which consumers consistently rated more positively than other housing models—“methodological flaws limited the ability to draw firm conclusions.”

In an editorial accompanying the reviews, Sam Tsemberis, Ph.D, said that the lack of clear program definition applies to “the innumerable group homes, board-and-care homes, adult homes, and other forms of not-well-defined or regulated congregate housing used in most states as permanent housing for people with mental illness.”

Tsemberis is the founder of Pathways to Housing and the “housing-first” model, a supported housing program that began in New York City in 1992 and has been replicated in cities around the country. The program provides “housing first”—without the precondition that consumers participate in mental health or substance use treatment—but also provides supportive-treatment services in the areas of mental and physical health, substance abuse, education, and employment. Since its founding, Pathways has housed more than 600 people in New York alone, and the program maintains an 85 percent retention rate even among individuals not considered “housing ready” by other programs, according to the Pathways website at http://pathwaystohousing.org/our-model/.

In an interview with Psychiatric News, Tsemberis said more research, development of “program fidelity,” and an increased focus on program evaluation are crucial for the great variety of housing models. “There is a great need to be clear about the kind of programs we are operating,” he said. “What kind of programs are helping what kind of person under what kind of circumstances?”

Housing Models Require Better Fidelity

The survey of evidence for housing programs is the second in a series of 14 articles to appear in Psychiatric Services; the series will review evidence for 14 commonly used behavioral health services for people with serious mental illness. Articles in the series began appearing online in Psychiatric Services in Advance in October 2013; all online articles in the series can be accessed free.

In the article on supported permanent housing, Debra Rog, Ph.D., and colleagues at Westat concluded that “the moderate level of evidence indicates that permanent supportive housing is promising, but research is needed to clarify the model and determine the most effective elements for various subpopulations.” They added, “Policymakers should consider including permanent supportive housing as a covered service for individuals with mental and substance use disorders.”

In the review of recovery housing, Sharon Reif, Ph.D., of the Institute for Behavioral Health at the Heller School for Social Policy and Management at Brandeis University and colleagues concluded that “recovery housing appears to be an important component in the continuum of care for some individuals. However, replication of study findings with greater specificity and in more settings is needed.”

Tsemberis, in an interview with Psychiatric News, said he believes the evidence on permanent supported housing is better than moderate, but acknowledged the evidence for effectiveness of other models is weaker due to lack of “program fidelity,” which is the ability to reliably define programs according to established standards and is necessary to improve the evidence base.

Variation Complicates Evaluation

“One of the difficulties in doing studies with all these different housing programs is that there is really no one fidelity scale being applied across all programs,” he said. “There is great diversity across programs, and no one coherent or uniform approach. Programs vary in their mission, in the way they are staffed, and in the intensity and duration of treatment provided. With so much variation, it is difficult to evaluate the effectiveness of any one particular housing model and to be certain we are talking about the same dimensions.”

Tsemberis said the Pathways programs draws on the Substance Abuse and Mental Health Service Administration’s permanent supported housing “fidelity toolkit” (posted here).

Tsemberis said more rigorous evaluation of housing programs is important. He said evaluation needs to include not only the objective measures typically employed—abstinence measures, service utilization—but also more subjective measures that speak to consumers’ own perceptions of quality of life.

“Recovery has typically been defined by clinicians, measuring a reduction in psychiatric symptoms or reduced use of emergency room services, measures that are observable from the system and clinician perspective,” he told Psychiatric News. “But recovery can also be defined according to the consumer’s perspective—working, maintaining a household, or developing relationships. These are the things that matter to consumers, so we need a mix of the objective and more qualitative measures.” ■