Report PII Deliverable 3a

Mental Health Service Act Community Services and Supports (CSS) EVALUATION BRIEF Summary and Synthesis of Findings on CSS Consumer Outcomes Submitted by: UCLA Center for Healthier Children, Youth and Families EMT Associates, Inc. May 2, 2011 This report was made possible through a contract with the Mental Health Services Oversight & Accountability Commission (MHSOAC) Executive Summary Introduction One of the first tasks of the statewide evaluation of the Mental Health Services Act (MHSA) was to review, summarize, and synthesize existing evaluations and studies of Community Services and Supports (CSS) in terms of their impact on consumer outcomes.i A summary and synthesis about the impact of CSS programs is timely because there has not been a statewide evaluation of the effort to date. This evaluation brief begins with a summary of findings on seven key consumer outcomes. Findings are followed by recommendations to guide next steps in evaluation efforts. Summary of Findings Based on the strongest evidence available from the review of evaluation/study reports and other documents, a summary of findings is provided in seven domains of consumer outcomes.ii (See the methods section for more details on the review process.) These domains were the most common across all the reports and documents reviewed:  Homelessness/living situation;  Acute psychiatric hospitalization;  Arrest/incarceration;  Physical health emergency;  Education;  Mental health functioning/quality of life; and,  Employment. In order to generate this summary, researchers categorized the findings into two tiers of evidence. Tier 1 evidence encompasses a small number of studies from four counties. This tier represents high-utility studies with the strongest body of evidence provided by counties; therefore, the findings are based on Tier 1 evidence. Tier 2 evidence includes reports (often in the form of data tables) from a larger number of counties. This tier represents a larger body of evidence from a larger pool of counties but with lower utility for the purpose of this summary. A synopsis of the main findings in each of the domains follows:  Participation in CSS programs is strongly associated with reductions in homelessness. The number of days spent homeless decreased for transition age youth (TAY) and adults. These reductions in homelessness are accompanied by additional improvements in residential outcomes. The number of consumers and number of days spent in more restrictive settings (e.g., residential treatment, emergency shelters) decreased overall for consumers in all age groups. At the same time, the number of consumers and number of days spent in independent or residential living situations increased, particularly for children, TAY, and adults. CSS Consumer Outcomes i  There is a strong association between CSS program participation and reductions in acute psychiatric hospitalizations. The number of hospital episodes for mental health emergencies decreased across all age groups, and estimates of reduction in episodes are especially high for TAY and older adults.  Participation in CSS programs is associated with reductions in arrests, particularly for TAY, adults, and older adults. The number of incarcerations and number of consumers incarcerated also decreased for TAY, adults, and older adults; however, the range of reductions across studies is wide, and reductions in time spent in criminal justice settings are mixed for children and TAY (i.e., sometimes reductions and other times increases were reported).  There is an overall trend of reduced physical health emergencies during CSS program participation for all age groups. However, because the amount of supporting evidence is limited, an association between CSS program participation and reduced physical health emergencies cannot be asserted at this time.  Positive trends in education outcomes exist in terms of school discipline events and improved academic performance for children and youth participating in CSS programs. However, because the amount of supporting evidence is limited, an association between CSS participation and improvements in education cannot be asserted at this time.  There is an overall trend toward improved mental health functioning and quality of life for adults and older adults who participate in CSS programs. However, because the amount of supporting evidence is limited, an association between CSS program participation and improvements in functioning and quality of life cannot be asserted at this time.  There appears to be little to no change in employment outcomes for TAY, adults, and older adults participating in CSS programs for one or more years. However, because the amount of supporting evidence is limited, the association between CSS program participation and employment is tentative. Recommendations Based on the findings, recommendations are provided to guide next steps in evaluation efforts to better track and understand consumer outcomes of MHSA: 1. Define a small standardized set of outcome indicators for data collection across counties to facilitate consistent reporting and aggregation. As part of this standardization, give counties flexibility to collect additional data that are pertinent to their communities. The selection of this set of outcome indicators should be guided by a framework that clearly defines priority indicators for each age group. Note: As part of the Phase II MHSA Statewide Evaluation contracted with the MHSOAC, the UCLA/EMT evaluation team will be developing a standardized template of priority indicators based on the Matrix of California’s Public Mental Health System Prioritized Performance Indicators. CSS Consumer Outcomes ii 2. Develop guidelines for analyzing and reporting consumer outcomes by age group, race/ethnicity, gender, and other important demographics to more fully understand differential outcomes in an effort to address disparities. Note: The standardized template of priority indicators to be developed for the Phase II MHSA Statewide Evaluation will support the development of such guidelines. 3. Dedicate resources to providing counties technical assistance on how best to design evaluation studies; collect and analyze data; and report, disseminate, and utilize findings. The technical assistance should be tailored to the existing capacity of counties so that smaller counties, for example, receive technical assistance that is customized to their needs. 4. Direct more resources to the rigorous evaluation of consumer outcomes across counties in the domains for which the amount of supporting evidence is limited (e.g., physical health emergencies, education, mental health functioning and quality of life, and employment), assuming that they are among the priority indicators ultimately defined. Ideally, there would be a synergistic relationship between building the evaluation capacity of counties and building a stronger evidence base on consumer outcomes. CSS Consumer Outcomes iii Evaluation Brief One of the first tasks of the statewide evaluation of the Mental Health Services Act (MHSA) was to review, summarize, and synthesize existing evaluations and studies of Community Services and Supports (CSS) in terms of their impact on consumer outcomes. A summary and synthesis about the impact of CSS programs is timely because there has not been a statewide evaluation of the effort to date. A description of the methods for collecting, reviewing, and analyzing county-level information on CSS consumer outcomes precedes a more detailed reporting of findings on the impact of CSS programs on consumer outcomes. The evaluation brief ends with a discussion of the review process and findings, and offers supporting recommendations. Methods Data Collection Procedures In March 2011, the evaluation team (via the California Mental Health Directors Association) sent an e-mail to the MHSA Coordinator in every county introducing the evaluation team and explaining the first evaluation deliverable. Counties were asked to submit “existing evaluation/study reports and other documents” that describe the impact of CSS programs on consumer outcomes.iii The request did not ask counties to produce any new information for this purpose. Counties were given approximately three weeks to respond to the request. While waiting for counties to submit documents per the request, the evaluation team performed an exhaustive search of county websites for relevant information. A wide net was cast by searching each site to uncover reports and documents on mental health services that might reflect or encompass MHSA components. At minimum, an Annual Update (FY09-10 or later) was reviewed for each county. Response and Sample Twenty (20) of 58 counties responded to the request for evaluation documentation. Four (4) of those counties reported that they had no CSS-specific evaluation or outcome information to submit. Among the other 16 counties, 361 documents were received and included in the analysis (see Appendix for a table displaying all documents reviewed for each county). The website extraction procedure yielded documents for all 58 counties. Including 181 documents obtained through the website search, a combined total of 542 documents were included in the content analysis of consumer outcomes. In addition, other reports/articles pertaining to the evaluation of MHSA by external evaluators were reviewed. One particular study by the Petris Center (Scheffler et al., 2010) iv was included in the content analysis. This study is an important point of reference for the summary because it is a comprehensive study on Full Service Partnership (FSP) representing a large portion of counties.v CSS Consumer Outcomes 1 Content Analysis To develop a document review framework, researchers first reviewed content from a sample of three counties and then compared the extracted content. This process established consistency in the review across the researchers. A coding scheme was then developed to rate the utility of the information presented (see Appendix for the coding scheme). For the purpose of this review, the utility of a report typically hinged on the extent to which the data source was clear, samples and/or methods were described, and contextual information (including how the data were analyzed and interpreted) was provided. Once all content was extracted and coded, researchers categorized the findings into two tiers of evidence. Tier 1 evidence encompasses a small number of studies from the following counties: Contra Costa, Los Angeles, San Diego, and San Francisco. This tier represents high-utility studies with the strongest body of evidence provided by counties; therefore, the summary of findings is based on Tier 1 evidence. Tier 2 evidence includes reports (often in the form of data tables) from a larger number of counties, but smaller counties are under-represented. (See Appendix for all documents reviewed for each county, including the identification of counties whose documents were included in Tier 1 and Tier 2). This tier represents a larger body of evidence from a larger pool of counties but with lower utility for the purpose of this summary. Because there is limited information on the validity of Tier 2 evidence, data in this tier are presented to describe whether Tier 2 reports converge with or diverge from Tier 1 findings. Seven domains of consumer outcomes are summarized in this evaluation brief because they were the most common across all the documents reviewed.vi Each summary addresses the following:  The overall conclusion of Tier 1 studies for the consumer outcome, including whether the consumer outcome is positive or negative, and the strength of the association between CSS program participation and consumer outcomes;  The different ways that the consumer outcome is measured;  The typical measurement points (e.g., 12 months pre-enrollment and 12 months post- enrollment);  For Tier 1 studies, the range of findings (in terms of reduction or increase) for each age group (children, TAY, adult, and older adult), if available;  Comparison of Tier 1 findings compared to the Petris Center (2010) study if the study reported on the same consumer outcome;  The extent to which Tier 2 evidence converges with or diverges from Tier 1 findings;  For Tier 2 reports, the overall finding (in terms of reduction or increase) for each age group, if available. This summary covers CSS programs; however, it is based largely on FSP studies and reports. Unless studies or reports refer specifically to FSP, the findings are presented generally as findings on CSS programs. CSS Consumer Outcomes 2 Findings Homelessness/Living Situation According to results from Tier 1 studies, participation in CSS programs is associated with reduced homelessness rates. These studies showed substantial decreases in “days spent homeless” from 12 months pre-enrollment to the first 12 months of FSP participation. One study reported an 82 percent reduction for TAY, and reported reductions for adults ranged from 59 to 67 percent across Tier 1 studies. These numbers were lower than the 100 percent reduction in homelessness reported in the Petris Center (2010) study. These significant reductions in homelessness generally appear to be accompanied by additional improvements in residential outcomes. The following results were reported in two Tier 1 studies. One study reported a 66 percent decline in the number of days that children spent in residential treatment, including a corresponding 23 percent increase in placement with family. In the same study, TAY participants experienced a 74 percent increase in shelter/temporary housing and more than a 2000 percent increase in days in Single Room Occupancy (SRO) with a lease. These increases were interpreted in the report as a positive outcome. Additional findings on TAY suggest a small but positive shift (17 percent change) from emergency shelters and residential treatment to independent living (e.g., living in an apartment alone). For adults, a 99 percent increase in the number of days spent in either independent or residential living situations was reported. A 27 percent reduction in shelter/temporary housing (in comparison, the Petris Center [2010] study found a 64 percent reduction in shelter use) and associated 30 percent increase in MHSA Stabilization and SRO with a lease was reported for adults. Similar findings were reported for older adults, though a simultaneous reduction of 26 percent was reported for general living (e.g., own housing independent of system support). Authors of this same report stated that a 52 percent increase in residential treatment and 21 percent increase in supervised placements for adults is difficult to interpret, but may be positive if it is an alternative to an unstable living situation. Likewise, the reported 157 percent increase in supervised placement for TAY may be positive or negative, depending on whether a previously unmet need is being met via the placement. The Petris Center (2010) study reported a 35 percent increase in supervised placements for TAY, adults, and older adults. Data from 12 counties provided Tier 2 evidence on living situation that converge with the positive outcomes suggested by Tier 1 studies. Reductions in homelessness rates ranged from 23 to 100 percent for different CSS programs across counties and age groups. In addition, these data suggested improved residential outcomes for children and TAY participants, such as greater numbers living at home with family or in independent living arrangements. Acute Psychiatric Hospitalizations The evidence from Tier 1 studies suggests that CSS programs reduce the number of acute psychiatric hospitalizations (also defined as “mental health emergencies”). Estimates of reduction were typically presented as a percentage reduction in hospital episodes from 12 months pre-enrollment compared to 12 months post-enrollment. Based on Tier 1 studies, a CSS Consumer Outcomes 3 reduction in episodes ranged from 32 to 87 percent for adults. One estimate for children/youth was a 65 percent reduction; another assessment showed that 40 percent fewer children required hospitalization. Estimates of reduction in episodes were particularly high for TAY (86 percent) and older adults (90 percent). In comparison to these figures, the Petris Center (2010) study reported that at 12 months of treatment, the odds of FSP adult participants using mental health-related emergency services were 67 percent lower than those receiving usual care. All in all, these estimates suggest that acute psychiatric hospitalizations are circumvented during participation in CSS programs for all age groups. Additional evidence from Tier 2 reports shows similar ranges and estimates in the number of consumers who experienced hospitalization, number of hospital episodes, and number of days hospitalized. Eleven (11) counties reported substantial reductions in acute psychiatric hospitalizations for all age groups. For example, two separate reports presented similar findings for children: 82 and 88 percent reductions in hospital episodes for consumers enrolled for 12 months or more. Based on three reports, there was a 23, 48, and 51 percent reduction in hospital days for adults. Overall, the evidence from Tier 2 reports converge with the findings from Tier 1 studies that there is a strong association between CSS program participation and reductions in the frequency of acute psychiatric hospitalizations. Arrest/Incarceration On the whole, participation in CSS programs is associated with reductions in incarcerations and arrests. Results from Tier 1 studies were stated as: reductions in arrests, increases or decreases in time spent in criminal justice settings, and reduced use of justice system services. Reductions in arrests reported across age groups spanned a range of 74 percent for adults, to 78 percent for TAY, and 98 percent for older adults. These results appear high in comparison with the Petris Center (2010) study finding of 56 percent reduction in arrests across counties. With respect to time spent in criminal justice settings, one Tier 1 study demonstrated that FSP substantially reduced the use of justice system services by 17 percent. Another found that TAY participants experienced a 71 percent reduction in time spent in criminal justice settings, while children experienced a 38 percent increase. The study authors suggested that this finding is due possibly to the fact that as children age, exposure to and engagement in risky behavior increases. Reports from 10 counties provided Tier 2 evidence on incarcerations and/or arrests. Reductions in reported number of incarcerations ranged from 15 to 100 percent, while reported reductions in the number of incarceration days spanned between 21 and 90 percent across counties and age groups. Reported decreases in the number of arrests ranged from 50 to 94 percent. These data generally converge with the positive outcomes suggested by the Tier 1 studies above. Physical Health Emergencies Available evidence points to an overall trend of reduced physical health emergencies during CSS program participation. However, the body of evidence on physical health emergencies is relatively small; therefore, a strong association between CSS program participation and reduced CSS Consumer Outcomes 4 physical health emergencies cannot be made at this time. Only one Tier 1 study reported on physical health emergencies. Based on this study, there were substantial reductions in physical health emergencies for all age groups. The number of emergency events during FSP participation (calculated as an “event rate” based on the number of emergencies per person per year for one or more years of FSP participation) declined for children (80 percent), TAY (86 percent), adults (93 percent), and older adults (79 percent). These declines are most pronounced among adults and older adults because they averaged a higher event rate prior to FSP participation. That is, physical health emergencies dropped from approximately 1.5 events (pre-enrollment) to less than .5 events (post-enrollment). This finding is particularly important for older adults because they are a medically vulnerable population. Tier 2 reports from two separate counties converge with these findings. In these reports, reductions in physical health emergencies averaged approximately 50 percent for all age groups combined. However, in some estimates based on specific programs, particularly for TAY, emergencies increased. For example, one report indicated a 12 percent increase in emergencies for TAY consumers who were in an FSP program for more than one year. Another report showed a higher increase of 56 percent for TAY in an FSP program for at least one year. This increase was explained in the report as resulting from a disproportionately high usage of emergency room contacts by a small number of youth. Education Overall, the available evidence points to a positive trend in education outcomes in terms of school discipline events and improved academic performance for children and youth participating in CSS programs. However, evidence in this domain of consumer outcomes is limited. In the two Tier 1 studies reviewed for education, different outcomes were reported: one study focused on self-reports of school attendance and self-reports of grades; another presented findings on school suspensions and school expulsions. According to the first study, there was a 13 percent increase from pre- to post-enrollment in school attendance for children and a 16 percent increase in “very good” and “good” grades; both self-reported outcomes were statistically significant. The second study found a 50 percent reduction in school suspensions among child participants as well as a 91 percent decrease in school expulsions. This same study reported a 75 percent decrease in school suspensions and a 100 percent decrease in school expulsions for TAY participants, but the authors caution that this outcome is based on a very small sample. The Petris Center (2010) study did not report on these same indicators (since the study does not focus on children); instead, the study reported that adult and TAY consumers were 30 percent more likely to begin an education program after 12 months of FSP participation, so evidence from the Tier 1 studies is not directly comparable. Evidence from Tier 2 reports was available from eight counties, and these data generally converge with the outcomes described in the Tier 1 studies above. Reported improvement in school grades ranged from 43 to 48 percent, while reported increases in attendance ranged from 50 to 80 percent. One county reported a 91 percent decrease in school suspensions for children, while another reported that 82 percent of participants had remained the same or CSS Consumer Outcomes 5 decreased suspensions and expulsions over time. Others measured this outcome as increases in the number of consumers in school, which reportedly ranged from 5.5 to 200 percent. Mental Health Functioning and Quality of Life The direction of the findings reported on functioning and quality of life is generally positive, but the evidence supporting this trend is not abundant. A single Tier 1 study compared FSP consumers with homeless clients who were receiving outpatient services on living situation, safety, daily activities, leisure, health, general life, social relationships, and family relationships using 21 items from a biannual consumer survey. Based on these self-report data, the study concluded that FSP consumers scored more favorably on every indicator, and the results were statistically significant. The Petris Center (2010) study compared FSP participants to those receiving usual care and found that consumer functioning (including reduced psychiatric symptoms, improved ability to take care of one’s needs, and dealing with problems more effectively) improved significantly by 27 percent for FSP participants. Tier 2 reports on functioning and quality of life from eight counties measured outcomes in this domain in several different ways, including improved symptomology, decreased service needs, improved ability to handle daily life and problems, achievement of individual recovery/discovery goals, and increased social support. A few reports used the Milestones of Recovery Scale (MORS) to indicate where an individual is in the process of recovery from severe and persistent mental illness. According to these reports, between 62 and 88 percent of consumers agree that they can more effectively handle daily life as a result of participation in CSS programs. As well, between 40 and 100 percent of consumers across age groups reportedly experienced improved symptoms and/or stabilized their functioning. This second tier of evidence generally converges with the study findings described above. Employment The Tier 1 evidence suggests little to no change in employment outcomes as a result of CSS program participation. Although the Petris Center (2010) study indicates a 25 percent increase in employment after 12 months of participation in FSP, only one Tier 1 study assessed employment outcomes and found no statistically significantly positive gains in the year after FSP enrollment as compared to the 12 months prior. Authors of the study postulated that employment changes may be expected to take longer than a year to be affected. Employment outcomes were measured in various ways in Tier 2 reports from eight counties, including number of participants working, number of days working, and number of clients who had wages as a financial support source. However, estimates were typically reported as a percentage increase or reduction in the number of days working. These reports on employment provide mixed evidence with respect to employment outcomes, with a range varying considerably from an 86 percent reduction in the number of days worked to an 84 percent increase in the number of days worked across age groups. Altogether, the amount of evidence on employment is limited, and the evidence that does exist is mixed. CSS Consumer Outcomes 6 Discussion and Recommendations Based on the findings and review process for this summary and synthesis, a discussion and supporting recommendations are provided to guide next steps in evaluation efforts to better track and understand consumer outcomes of MHSA. Discussion The abundance of documents reviewed for this summary and synthesis indicates that, on the whole, counties are collecting data on consumer outcomes. However, the counties do not consistently report on the same indicators for each domain of consumer outcomes. They do not always report the findings by age group or other important consumer demographics (e.g., race/ethnicity and gender). Counties do not always provide information on the data source (e.g., self-report, clinician rating, etc.). Oftentimes they do not provide specific timeframes for pre- and post-measurements. Sample sizes are not consistently provided for the analyses, and often there is no indication of whether the analyses include duplicated or unduplicated counts of consumers. Finally, the analytic methods used are not always clear in the presented results, and the reports are not always accompanied by a narrative explaining and interpreting the findings. Altogether, these limitations hamper the ability to summarize and more fully understand the impact of CSS programs on consumer outcomes across counties. The quality of information analyzed to generate this summary and synthesis dictated the types of conclusions that were drawn about the associations between CSS program participation and the seven domains of consumer outcomes. For example, although many documents were reviewed for this summary, the methods of reporting for most documents make it difficult to clearly interpret the impact of CSS programs on consumer outcomes. Because the integrity of the conclusions hinged upon the integrity of the documents, only a handful of studies (Tier 1) were relied upon to drive the summary. Other documents (Tier 2) were carefully considered in the summary, but their limited information provided supplemental evidence rather than primary evidence. Other factors also influenced the summary. Because less than a quarter of the counties responded to the request for information on consumer outcomes, this review relied heavily on data from those counties in addition to what researchers were able to extract from county websites. This may raise questions about the degree to which the summary represents the entire body of existing evidence. In particular, few of the smaller counties were represented. Based on this summary of the available evidence, CSS program participation is strongly associated with positive consumer outcomes in the domains of homelessness/living situation and acute psychiatric hospitalization. Also, CSS program participation is associated with reductions in arrest/incarceration, but some mixed findings for children and TAY suggest that positive outcomes are possibly not being achieved consistently across all age groups. Consumer outcomes in the domains of physical health emergencies, education, and mental health functioning and quality of life also appear generally positive. However, without more abundant and robust evidence, the association between CSS program participation and these outcomes is CSS Consumer Outcomes 7 tentative. The one exception to these positive trends in consumer outcomes is the domain of employment. The available evidence suggests little to no change in employment for CSS program participants; however, the ability to draw conclusions about the association between CSS program participation and changes in employment is hindered by the lack of more abundant and robust evidence on employment. Recommendations 1. Define a small standardized set of outcome indicators for data collection across counties to facilitate consistent reporting and aggregation. As part of this standardization, give counties flexibility to collect additional data that are pertinent to their communities. The selection of this set of outcome indicators should be guided by a framework that clearly defines priority indicators for each age group. Note: As part of the Phase II MHSA Statewide Evaluation contracted with the MHSOAC, the UCLA/EMT evaluation team will be developing a standardized template of priority indicators based on the Matrix of California’s Public Mental Health System Prioritized Performance Indicators. 2. Develop guidelines for analyzing and reporting consumer outcomes by age group, race/ethnicity, gender, and other important demographics to more fully understand differential outcomes in an effort to address disparities. Note: The standardized template of priority indicators to be developed for the Phase II MHSA Statewide Evaluation will support the development of such guidelines. 3. Dedicate resources to providing counties technical assistance on how best to design evaluation studies; collect and analyze data; and report, disseminate, and utilize findings. The technical assistance should be tailored to the existing capacity of counties so that smaller counties, for example, receive technical assistance that is customized to their needs. 4. Direct more resources to the rigorous evaluation of consumer outcomes across counties in the domains for which the amount of supporting evidence is limited (e.g., physical health emergencies, education, mental health functioning and quality of life, and employment). Ideally, there would be a synergistic relationship between building the evaluation capacity of counties and building a stronger evidence base on consumer outcomes. CSS Consumer Outcomes 8 End Notes i A similar summary of the impact of CSS programs on MHSA values will be available at the end of the year. ii These consumer outcomes are identified in the Performance Indicators for Evaluating the Mental Health System developed by the California Mental Health Planning Council, and most of them have been identified as priority indicators in the Matrix of California’s Public Mental Health System Prioritized Performance Indicators. iii The full request was to submit information on the impact of CSS and Prevention and Early Intervention (PEI) on consumer outcomes and MHSA values. This evaluation brief reports only on CSS and consumer outcomes. iv Scheffler, R., M., Felton, M., Brown, T. T., Chung, J., & Choi, S. (May 2010). Evidence on the effectiveness of Full Service Partnership programs in California’s public mental health system. Berkeley, CA: Nicholas C. Petris Center on Health Care Markets and Consumer Welfare, School of Public Health, University of California, Berkeley. v The Petris Center (2010) study focused on TAY, adults, and older adults. In contrast, this current summary covers all age groups and contributes to the understanding of child and youth outcomes associated with CSS program participation. For the purpose of this summary, the Petris Center (2010) study is used as a descriptive reference and not as a standard by which the findings from county documents should be compared or judged. vi These consumer outcomes are identified in the Performance Indicators for Evaluating the Mental Health System developed by the California Mental Health Planning Council, and most of them have been identified as priority indicators in the Matrix of California’s Public Mental Health System Prioritized Performance Indicators. CSS Consumer Outcomes 9