Mhsoac PEI Report 2011

Mental Health Services Act Prevention and Early Intervention (PEI) EVALUATION REPORT: Summary and Synthesis of PEI Evaluations and Data Elements Submitted by: UCLA Center for Healthier Children, Families & Communities August 31, 2011 This report was made possible through a contract with the Mental Health Services Oversight & Accountability Commission (MHSOAC) Executive Summary Summary and Synthesis of PEI Evaluations and Data Elements Purpose The purpose of this report on Prevention and Early Intervention (PEI) evaluations is to summarize and synthesize what is known about proposed or intended outcomes, reported data elements, and reported outcomes from existing PEI local evaluation plans and reports. Findings and recommendations from this report can inform counties and strengthen future PEI evaluations and reports. Summary of Findings, Discussion, and Recommendations In light of the early stage of PEI implementation, this report examines the local evaluation plans submitted by counties as well as their reports of PEI outcomes to date. We reviewed all available local evaluation plans within the 3-Year PEI Plans, Annual Updates (FY10-11 and FY11- 12), and PEI documents submitted by counties. Each summary of findings below begins with a description of the specific data source(s) and sample(s) used for the analysis. The recommendations are presented within their respective summary sections; however, they are broad recommendations that apply to improving PEI evaluation efforts overall. PEI Intended Outcomes The data source for this analysis was the local evaluation plans in the 3-Year PEI Plans. The sample for this analysis was comprised of 37 counties that included a local evaluation plan (see Appendix A for a list of these counties). The intended outcomes proposed in the local evaluation plans are appropriate and meaningful to the PEI priority populations for which the prevention and early intervention efforts are geared. For example, the intended outcomes for children and youth projects represent typical constructs – such as risk and protective factors – that are measured in prevention and early intervention programs. Intended outcomes proposed in the local evaluation plans cover a wide range of constructs for both individual/family and program/system levels within and across the i five Key Community Mental Health Needs (also referred to as “key areas of need”). Although the local evaluation plan asked counties to specify measurement tools, many intended outcomes were not tied explicitly to actual measurement tools. Therefore, in the future, it will be challenging to synthesize PEI outcomes across counties and/or programs in a coherent and succinct way. (See the full report for a list of the five Key Community Mental Health Needs of the PEI Framework, as well as tables summarizing the intended outcomes proposed by the 37 counties in their local evaluation plans.) Recommendations 1. In order to optimize what could be known about PEI impact, develop a small set of priority indicators and/or measures within each Key Community Mental Health Need and across target populations that counties should collect and report as part of an ongoing effort to evaluate PEI. This may necessitate revisiting PEI evaluation requirements so that there is ongoing reporting of PEI outcomes for local and statewide analysis of PEI efforts. Because effects of prevention efforts on system and community levels are typically detected in the longer term, a short-term evaluation strategy may fall short of capturing change at these levels. Include all levels of PEI outcomes (i.e., individual/family and program/system levels) in the set of priority indicators. This would be in addition to what counties have already selected as their local outcomes of interest. It will be important to educate counties about the value of an agreed-upon set of indicators as a way to garner buy-in for ongoing evaluation efforts. For example, if all counties use the same set of indicators, cross-county comparisons will be possible for gauging success. Content and Quality of Local Evaluation Plans The data source for this analysis was the local evaluation plans. The sample included 37 counties that submitted a local evaluation plan. The purpose of this analysis was to assess the quality of data that could potentially be derived from the local evaluations. While some counties presented solid evaluation plans, there was inconsistency overall across counties in terms of content and quality of the plans. There was often a lack of clarity in presenting, for example, the intended outcomes, design, and measures in many local evaluation plans. In particular, there was a lack of specificity around identifying measures of program/system level outcomes. These gaps are an indication that some counties do not have the internal capacity and/or may not have the appropriate guidance around PEI evaluation goals and expectations to develop (and possibly, to implement) evaluations of high quality. ii Recommendations 2. Establish overall evaluation goals for PEI. Provide clear expectations and guidance to counties so that they can help meet those goals. Developing a set of priority indicators across counties is an example of establishing goals and providing clear expectations and guidance for evaluating PEI in the future. 3. Provide counties with support and technical assistance on designing evaluation studies; collecting and analyzing data; assessing the extent to which programs are implemented as intended (fidelity monitoring); and reporting, disseminating, and utilizing findings. Support to counties might come in the form of county-to-county peer learning collaboratives. The technical assistance should be tailored to the existing capacity of counties so that smaller counties, for example, receive technical assistance that is customized to their needs. Leverage existing resources (e.g., university-based workshops, online evaluation trainings, and evaluation toolkits) to supplement intensive and/or direct technical assistance. 4. Provide counties with guidance to identify and collect outcome data on the family, program, and system levels to ensure that all levels (and not just the individual level) are adequately included in their PEI evaluations. PEI Data Elements The data source for this analysis was the Annual Updates for FY11-12, which were reviewed to assess what data elements have most recently been tracked for PEI. A summary of these types of data elements is based on 30 counties. In addition to the outcome data reported on PEI to date, the data elements represent participant demographics (e.g., participant race/ethnicity), participant characteristics (e.g., risk factors), service provision and utilization (e.g., number of participants served), and program outputs (i.e., program activities such as the completion of staff training and number of referrals). It is important to note the distinction between outcome and process. Outcome questions examine program results or impact on PEI participants. Outcome questions ask: What was the impact of the program? What were the short-term and long-term outcomes? Process questions ask: Whom is the program serving? What are the program outputs? Process-oriented data elements are also referred to as “process variables” throughout this report. Generally, the data elements that counties have reported thus far are appropriate and meaningful and capture a variety of process variables that can potentially help explain PEI outcomes. That is, they are typically collected in evaluation studies, including those that iii examine whom the program serves (demographics and characteristics), what the program provides (service provision and utilization), and what the program activities are (program outputs). Collecting adequate process-oriented data is necessary for counties to soundly interpret outcomes. However, they are not sufficient for assessing outcomes. This is an important distinction for counties to understand. The types of process-oriented data that are collected by counties have to be comprehensive, covering a full range of variables that may help explain PEI outcomes. At the same time, process variables should not be a substitute for outcome data in examining program effectiveness. Recommendations 5. Ensure that counties understand how to use process-oriented data to help interpret program outcomes. This could be accomplished through a combination of technical assistance and reporting structures. Reporting structures might: (1) delineate between outcome and process-oriented data, and (2) require an analysis of how process-oriented data help to explain outcome findings. Individuals Served in PEI The data sources for this analysis were the Annual Updates for FY10-11 and FY11-12. The FY10- 11 Annual Updates reflect implementation for FY08-09 and the FY11-12 Annual Updates reflect implementation for FY09-10. Therefore, for this analysis, the findings are presented by the implementation periods of FY08-09 and FY09-10. Only eight counties reported the number of individuals served for implementation in FY08-09. The estimated number of individuals served was 55,525. For implementation in FY09-10, 30 counties reported these figures with an estimate of 447,634 individuals. For both fiscal years, the reported number of individuals served was presented by age group, race/ethnicity, primary language, and/or cultural group. These numbers should be analyzed and/or compared across counties with caution because they are relative to county size, target population, and project scope. In addition, the validity of numbers is in question, warranting further caution in interpreting the numbers. For example, the numbers reported by counties often do not match up across the groupings of age, race/ethnicity, language, and culture. Furthermore, while it is useful to report numbers of individuals served by race/ethnicity, this information alone is not as meaningful as contextualizing the numbers based on the racial/ethnic makeup of the target community. The number of individuals served broken down specifically by prevention or early intervention iv programs would also be helpful in further understanding the type of program received by PEI participants. Recommendations 6. Have counties report separately: (1) actual number of individuals served across prevention programs, (2) actual number of individuals served for early intervention programs, and (3) estimated number of individuals served in prevention programs. Change reporting format/structure accordingly to aid counties in providing more accurate counts of individuals served. 7. Provide guidance to counties on how to report the number of individuals served across PEI programs, including how to use and report the data in order to describe populations served by PEI. For example, assist counties to utilize the data for examining racial/ethnic disparities in access and outcomes. Help them to contextualize information on race/ethnicity so that it can be compared to the racial/ethnic distribution of individuals across communities and used to examine disproportionality in PEI outcomes across racial/ethnic groups. PEI Reported Outcomes The data sources for this analysis were the FY10-11 Annual Updates, FY11-12 Annual Updates, and documents on PEI outcomes submitted by counties. The sample for this analysis includes 22 counties that had outcome data from one or more of these data sources.i The small number of counties reporting outcome data, as well as the relatively low utility of these data at this time is, in part, reflective of where counties are in the developmental stage of rolling out PEI projects. A sufficient quantity of evaluation data is not available; therefore, we are limited in making statements about what is and is not working with respect to PEI projects in and across counties. Overall, the presentation of findings below is meant to be a descriptive summary of what has been reported on PEI outcomes to date. The summary itself should be reviewed in that context and in no way interpreted as a commentary or criticism of county PEI efforts. The greatest amount of data, as well as the greatest amount of high utility data, was reported in the key area of need for Emotional and Behavioral Health Problems among At-Risk Children, Youth, and Young Adult Populations. (Data were categorized as “high utility” if data sources were clear, samples and study methods were described, and there was contextual information on how the data were analyzed and interpreted.) High utility data from five counties begin to suggest the following with respect to individual/family level outcomes in this key area of need. Although relatively speaking the strongest evidence was presented in this key area of need, the findings still must be interpreted with caution given the small quantity of data: v  PEI program participants are demonstrating decreased behavior problems (e.g., aggression, impulsivity) and improved social competence and skills for children, youth, and transition age youth (TAY);  Programs for the TAY population may have a positive influence on employment and homelessness outcomes, as well as reductions in “legal involvement”, including arrests;  Parent-focused programs may be resulting in improved parenting knowledge, skills, and self-efficacy; decreased parental depression, stress, and anxiety; and improved family functioning. Few counties possessed actual evaluation reports structured in a manner that clearly articulates evaluation questions, methods, and findings; and a majority of reported outcomes consisted of decontextualized fragments of data that were difficult to interpret meaningfully. Moreover, of all the documents reviewed for this summary of PEI reported outcomes, there was no reporting of neutral or negative findings. One of the goals of evaluation is to judge the worth of a program. In order to achieve this, we need to know both what works and what does not. Therefore, it is equally important to report positive findings, as well as neutral or negative findings. The current structure (or lack thereof) provides little guidance to counties on reporting evaluation findings, including the reporting of neutral or negative findings for PEI. Some counties presented only benchmarks for performance in their reports. For example, if their outcome was to decrease parental stress, they reported that 70 percent of participants met their benchmark. However, they did not report the actual outcome of how participants scored on the stress index, indicating the degree of improvement in stress level. A focus on meeting performance benchmarks without providing the underlying measure on which the benchmark is judged may be helpful for quality assurance or program improvement purposes but not for synthesizing findings on PEI impact across counties. Lastly, what has been reported in terms of outcomes up to this point focuses much more on individual/family level outcomes than on program/system level outcomes. This may be a result of the fact that local evaluation plans typically lacked specified measures for intended program/system level outcomes – in particular, system level outcomes such as collaboration and community capacity. It likely reflects the reality that system level evaluation is complex and that there is generally less capacity to conduct this type of evaluation. This is a concern given the context of this initiative, where system and community change are certainly being targeted and are critical cornerstones of prevention and early intervention. vi Recommendations 8. Provide counties with resources, guidance, and technical assistance to report the specific contextual information (e.g., design, methods, sample size, measurement tools) required for interpreting the validity and strength of local findings. Resources might include a budget to support one point person in each county who is responsible for summarizing, synthesizing, and reporting all local evaluation findings at the PEI project level. There might also be a support person or team at the State level responsible for providing the guidance suggested above to the county. 9. Develop a reporting format for PEI evaluation findings such that outcome data are submitted to the State in a manner that facilitates an effective process of summarizing and synthesizing outcomes across counties. Include in the reporting format required content such as evaluation questions, study design, samples, measurement instruments and timeframes for data collection, data analysis, and interpretation of findings. Specifically request that counties report null and negative findings and require that they go beyond reporting performance benchmarks. 10. In order to enhance what can be known about PEI impact, statewide analyses on PEI should group counties based on the type of projects they have chosen for their local evaluation. 11. Help counties identify appropriate program/system level indicators so that they capture the full spectrum of potential effects of prevention and early intervention. vii Evaluation Report: Summary and Synthesis of PEI Evaluations and Data Elements Introduction Prevention and Early Intervention (PEI) is one of five components of the Mental Health Services Act (MHSA). It is intended to help transform the mental health system in California into one that will promote mental health awareness broadly across all communities and facilitate early access to mental health services in natural community settings such as schools and other community organizations. An important aspect of this transformation is the emphasis on intended outcomes for individuals and families, programs and systems, and communities. Within the PEI Guidelines, set forth by the State Department of Mental Health to inform county planning, is the PEI Framework.ii The PEI Framework is comprised of Key Community Mental Health Needs (referred to throughout this report as “key areas of need”) and Priority Populations. This framework was used as a tool to guide the review, analysis, and presentation of findings that follow. The areas of need and the priority populations are outlined here briefly: Five Key Community Mental Health Needs  Disparities in access to mental health services  Psychosocial impact of trauma  Emotional and behavioral health problems among at-risk children, youth, and young adult populations  Stigma and discrimination  Suicide risk Six Priority Populations  Underserved cultural populations  Individuals experiencing onset of serious psychiatric illness  Children/youth in stressed families  Trauma exposed individuals  Children/youth at risk for school failure  Children/youth at risk of or experiencing juvenile justice involvement. 1 Report Purpose and Overview Our charge for this report was to collect and review existing documents on PEI and to summarize and synthesize what is known about: (1) proposed or intended outcomes, (2) reported data elements, and (3) reported findings from existing evaluations, with a focus on outcomes. The report begins by presenting an overview of the data collection procedures, samples of counties represented in various data sources, and county documents used in our analyses. The following three sections describe our analysis of the PEI intended outcomes, overall quality of the local evaluation plans, and PEI data elements. Subsequent sections of our analysis present the number of individuals served by PEI, as well as the outcomes that have been reported. The report concludes with a discussion of the state of future knowledge about PEI based on these analyses. Associated recommendations and strategies for supporting counties in their PEI evaluation efforts are proffered. Methods for Obtaining PEI Documents In March 2011, the evaluation team (via the California Mental Health Directors Association) sent an e-mail to the MHSA Coordinator in every county introducing the evaluation team and describing evaluation deliverables. Counties were asked to submit “existing evaluation/study reports and other documents” that describe the impact of PEI programs on consumer outcomes.iii The request did not ask counties to produce any new information for this purpose. Counties were given approximately three weeks to respond to the request. Twenty (20) of 59 counties responded to this request for documentation. Of these, only six counties initially submitted documents pertaining to PEI. In an attempt to supplement this information and to ensure that our search had been thorough, the evaluation team contacted a purposeful sample of six key informants from both large and small counties (as well as other mental health organizations) who were identified as potentially having important knowledge about existing PEI evaluation efforts throughout the State. Our initial pool of key informants led to a snowball sample of additional informants. We pursued this strategy until the names of no additional key informants were identified. Altogether, we contacted 14 key informants and received a response from each. As a result of this follow-up process, we uncovered one additional report from one county describing its PEI outcomes. There were a total of seven counties that submitted information through these processes. In addition, the evaluation team searched county and other MHSA-related websites for relevant information on PEI, including counties’ FY10-11 Annual Updates, their FY11-12 Annual Updates, 2 and their 3-Year PEI Plans. Annual Updates from FY10-11 that were available online for 55 counties were reviewed. Annual Updates from FY11-12 were available for 37 counties through the Mental Health Services Oversight and Accountability Commission (MHSOAC). All but one county’s 3-Year PEI Plan (for a total of 58) were available for review. The evaluation team also performed a web-based literature search for reports or articles produced by government entities, universities, foundations, or other organizations regarding PEI evaluation. No external PEI evaluations were found. A description of the specific data sources (i.e., PEI documents submitted by counties, Annual Updates, 3-Year PEI Plans, and/or PEI documents retrieved through a web-based search) for each analysis included this report is presented alongside its respective findings. (See Appendix A for a list of PEI documents included in this review as organized by county.) Findings on PEI Intended Outcomes Data Source and Analysis The data source for this analysis was the local evaluation plans in the 3-Year PEI Plans. The sample for this analysis was comprised of 37 counties that included a local evaluation plan (see Appendix A for a list of these counties).iv Twenty-two (22) counties, considered “very small counties” (i.e., counties with a population less than 100,000), are not required to conduct a local evaluation. However, of these very small counties, one opted to conduct a local evaluation. These plans include intended outcomes, or measurable effects of an intervention, for the PEI project(s) that counties proposed to evaluate. All 37 counties with a local evaluation plan proposed to evaluate one PEI project each. Each project potentially includes more than one program. For instance, one county plans to evaluate one project, but the project contains 10 separate programs. Intended Outcomes for Target Populations and Key Community Mental Health Needs Each local evaluation plan for PEI was reviewed to identify the target population(s) (i.e., child and youth, transition age youth [TAY], adults, and older adults) for whom the intended outcomes are proposed. In the findings below, we provide a count of how many counties proposed intended outcomes for each of the target populations for the PEI projects in their local evaluation plans. It is important to note that these counts are not mutually exclusive – that is, each evaluation plan could identify intended outcomes for more than one target population. In fact, most evaluation plans proposed to track indicators for more than one target population because many of these projects target multiple populations simultaneously. 3 Counties that submitted a local evaluation plan were asked to identify intended outcomes for the PEI project(s) they plan to evaluate. Two levels of intended outcomes were to be identified in the plan: individual/family level outcomes (e.g., reduce depression, increase family functioning) and program/system level outcomes (e.g., increase number of referrals, increase collaboration with community-based agencies). The intended outcomes proposed in the local evaluation plans represent the type of information on PEI impact that could potentially be collected through the counties’ local evaluation efforts.v We analyzed the intended outcomes proposed by each county to summarize the breadth and depth of information on individual/family and program/system level outcomes that could be available on PEI in the future. We identified all intended outcomes proposed in the evaluation plans for each project. For the purpose of our analysis, we structured these outcomes based on the PEI Framework regarding key areas of need and priority populations listed in the introduction of this report. We content analyzed all of the intended outcomes by outcome level (individual/family and program/system) for each of the PEI key areas of need. It was necessary to be selective about which intended outcomes to report, thus only those that were clear and appropriate for the outcome level were reported for our purpose (see Tables 1 through 5). If an intended outcome was not an outcome per se (but rather a service utilization variable, for example), it was not reported below. If an intended outcome was inaccurately categorized as an individual/family level outcome rather than a program/system level outcome, it was re-categorized for this report (e.g., increasing referrals and screenings is a program/system level outcome rather than an individual/family level outcome). All in all, most of the intended outcomes proposed in the local evaluation plans were included in this analysis. Furthermore, since some outcomes overlap across the key areas of need (e.g., reducing depression applies to psychosocial impact of trauma and suicide risk), some outcomes were reported in multiple areas of need. Because the key areas of need apply to all six priority populations, the intended outcomes are not separated by priority populations, although some intended outcomes are specific to certain populations and the organizations/systems that serve them. The intended outcomes that are identified for each key area of need were further analyzed for larger themes. The purpose of this thematic analysis is to summarize the body of knowledge that could potentially be realized for each key area of need if counties follow through on collecting data on these intended outcomes. This analysis also could help determine whether there are any gaps to be addressed in evaluating priority outcomes for PEI. 4 Summary of Findings on PEI Intended Outcomes Intended Outcomes for Target Populations Altogether, the PEI local evaluation plans include intended outcomes for all four target populations: children and youth, TAY, adults, and older adults. Of the 37 counties with a local evaluation plan, 27 included intended outcomes for children and youth; 21 plans included intended outcomes for TAY; 20 plans included intended outcomes for adults; and 15 plans included intended outcomes for older adults. Most of the PEI projects that counties have selected for their local evaluations target children and youth. Therefore, more evaluation plans identified intended outcomes for children and youth. Only two plans proposed to evaluate projects exclusively for older adults; however, many more than two plans included intended outcomes for older adults because older adults are served by a number of PEI programs alongside other target populations. For example, a suicide prevention program offered in one county targets TAY, adults, and older adults who speak Spanish as their primary language. Therefore, intended outcomes for this program would apply to all three target populations. Intended Outcomes for Key Community Mental Health Needs Below are summaries of the intended outcomes for each key area of need. Disparities in Access to Mental Health Services Presented in Table 1 are the intended outcomes proposed in 21 local evaluation plans to address disparities in access. Given the program/system emphasis inherent in this area of need, the greater number of intended outcomes on the program/system level is expected. The local evaluation plans propose to measure change in capacity on several levels to reduce disparities in access to mental health services (i.e., access in general and disparities in access for unserved/underserved cultural groups). Starting with individual/family level outcomes, the intended outcomes capture the extent to which perspective (e.g., personal stigma) and knowledge about one’s own mental health and mental health in general are improved to make people more open to seeking and receiving mental health services. At the program/system level, there are discrete outcomes for PEI impact on workforce (e.g., program staff, teachers, primary care providers). Intended outcomes in this area focus on knowledge and skill development within the system as a way to reach a wider population to reduce disparities (e.g., greater knowledge about mental health among primary care providers to properly refer patients to mental health care). Other program level outcomes describe the effects of strategies to make mental health services more inviting, more readily available, and more appropriate especially to unserved/ underserved cultural populations. Finally, 5 intended outcomes for systems and communities describe the effects of greater collaboration and coordination across service systems and community partners who, by virtue of existing relationships with their respective communities, could improve access to mental health services for unserved/underserved cultural populations. Table 1. Intended Outcomes for Disparities in Access to Mental Health Services (21 Plans) Individual/Family Level Outcomes Program/System Level Outcomes Impact on individuals and families Impact on human resource capacity: who could benefit from greater  Improve knowledge of mental health and conditions for access to mental health services: referral among program staff, primary care providers, etc.  Increase knowledge of mental  Improve knowledge of child social-emotional health and availability of development and availability of services among teachers, services child care providers, and school administrators to  Improve ability to recognize risk increase referrals factors for mental health problems Impact on program capacity:  Reduce negative reaction to  Improve access to mental health services and resources stigma of help-seeking (e.g., improve program location convenience and  Improve ability to navigate operating hours) mental health system  Improve time it takes to get access to mental health  Satisfaction from underserved services cultural groups due to culturally  Increase linkages to community services competent services  Improve successful follow through on referrals  Increase in number of individuals and families identified (and served) as needing early intervention services  Increase mental health assessments and early intervention by primary care providers  Increase in culturally appropriate services Impact on system and community capacity:  Increase appropriate referrals to public mental health system (e.g., Full Service Partnership)  Expand use of evidenced based practices in community settings and among organizations (particularly those that service underserved cultural populations and rural communities)  Increase mental health services in communities, schools, and juvenile justice settings  Improve integration of mental health and substance abuse prevention efforts  Increase collaboration amongst community health/mental health providers and other settings  Increase quantity and quality of relationships with other organizations and systems  Increase use of ethnic/cultural community partners 6 Psychosocial Impact of Trauma Presented in Table 2 are the intended outcomes proposed in eight local evaluation plans to address the psychosocial impact of trauma. For individuals, the local evaluation plans propose to measure individual behavior change (e.g., behavioral problems, negative attributes, engagement in social activities), knowledge and skill improvement (e.g., stress management, coping, social competence), and reduction of symptomology (e.g., depression, anxiety for individuals suffering from the impact of trauma). For families, the local evaluation plans propose to measure general family wellness and family/caregiver stress as a result of a family member who is experiencing or has experienced trauma. The intended outcomes for the program/system level reflect capacity change in programs, systems, and within and across communities. For the program level, the local evaluation plans propose to measure change in capacity to provide appropriate services for individuals and families dealing with trauma. For the system and community levels, the intended outcomes reflect changes in the quantity and quality of services and supports through better coordination of services in various service settings to address trauma (e.g., schools, juvenile camps, agency settings other than mental health). Table 2. Intended Outcomes for Psychosocial Impact of Trauma (8 Plans) Individual/Family Level Outcomes Program/System Level Outcomes Impact on individuals who experience trauma: Impact on program capacity:  Increase social activities  Increase appropriate services for  Increase self-sufficiency children and youth who have  Improve social adjustment/competence experienced trauma  Improve feelings of support and wellness  Improve appropriate and early  Improve knowledge of and skills in stress identification of mental health and management, cognitive coping, etc. substance abuse issues related to  Reduce isolation (including feeling of isolation impact of trauma and loneliness) (continued)  Reduce or eliminate symptoms of depression  Reduce symptoms of anxiety  Reduce symptoms of trauma or Post-Traumatic Stress Disorder (PTSD)  No successful suicide  Reduce number of emergency room visits  Reduce incidence of teen-dating violence (continued) 7 Table 2 (Continued). Intended Outcomes for Psychosocial Impact of Trauma (8 Plans) Individual/Family Level Outcomes Program/System Level Outcomes Impact on individuals who experience trauma Impact on system and community capacity: (continued):  Increase mental health services for  Reduce child behavior problems psychosocial trauma in schools and juvenile  Reduce defiant and oppositional behaviors justice settings  Reduce negative attributes (e.g., self-  Expand use of evidenced based practices for blame) of the traumatic event psychosocial trauma in community settings  Satisfaction with services rendered and among organizations (including cultural sensitivity)  Expand training of evidenced based practices for psychosocial trauma to community Impact on the family: providers  Improve family wellness  Improve coordinated services with other  Reduce family stress related to caregiving county departments or agencies of someone who has experienced trauma  Increase community capacity to ameliorate negative impact of trauma exposure Emotional and Behavioral Health Problems among At-Risk Children, Youth, and Young Adult Populationsvi Twenty-five (25) local evaluation plans addressed the emotional and behavioral health problems among at-risk children, youth, and young adult populations (or TAY). The intended outcomes for this key area of need are presented in Table 3. These intended outcomes cover a large spectrum of constructs for individual/family impact. Because of the preventive nature of PEI programs, the intended outcomes for this key area of need typically are not traditional mental health outcomes. On the individual/family level, they are reflective of the different developmental needs of children, youth, and TAY. For example, the intended outcomes address child/youth functioning (e.g., improving coping skills, social skills, mood, behavior), family and parental functioning (e.g., improve family safety, reduce parental stress), school functioning (e.g., improve attendance rates, reduce number of school discipline referrals), and delinquent behaviors (e.g., improve knowledge of anger management, reduce high-risk behaviors such as substance abuse). The intended outcomes for the program/system level are similar in theme to the previous key areas of need; that is, capacity building is the primary objective of these intended outcomes. For human resources, the aim of PEI is to improve the competence level of the workforce, whereas for program capacity, the aim is to improve the ability to engage children, youth, TAY, and their parents and families in appropriate services. The intended outcomes for systems and communities address the quantity and quality 8 of services and supports through greater sharing of resources, greater collaboration across service sectors, and better coordination of services. Table 3. Intended Outcomes for Emotional and Behavioral Health Problems among At- Risk Children, Youth, and Young Adult Populations (25 Plans) Individual/Family Level Outcomes Program/System Level Outcomes Impact on individuals (children, youth and TAY): Impact on human resource capacity:  Increase resiliency and protective factors  Improve competence of staff on  Increase participation in positive activities that promote therapeutic child care mental health  Improve service providers’  Improve individual functioning knowledge of infant and early  Improve socio-emotional wellbeing childhood development  Increase social support and connection  Improve cultural competency  Improve formation of positive relationships across workforce  Improve developmentally appropriate parent  Improve ability of school attachment personnel to recognize signs and  Improve responsiveness to parental direction symptoms associated with  Reduce risk factors that lead to mental health problems emotional disturbance, mental disorder, substance abuse, or  Reduce isolation suicide risk of children and youth  Reduce incidence of suicide attempts  Reduce suicide ideation Impact on program capacity:  Improve time and (Outcomes related to youth and/or TAY) appropriateness of response to  Improve successful management of symptoms referrals of TAY at risk of  Improve coping skills psychosis  Improve regulation of temperament  Improve engagement of TAY in  Improve social skills recovery activities  Improve mood and behavior  Improve engagement of families  Improve youth self-efficacy in treatment  Increase knowledge of and skill use in anger  Improve services and supports management and conflict resolution for first-time employment  Increase skills in leadership and community engagement among program participants  Increase utilization of community support systems  Increase linkages to community  Increase financial stability services  Meet personal recovery goals  Increase number of students  Maintain healthy stable living environment who will more readily utilize  Reduce acuity of distress mental health and other services  Reduce prolonged suffering because of reduction in stigma  Reduce youth aggression and social problems  Increase number of referrals to  Reduce hospitalizations community resources  Reduce high-risk behaviors (e.g., substance abuse, (continued) violence or sexual activity)  Diversion from incarceration  Reduce homelessness (continued) 9 Table 3 (Continued). Intended Outcomes for Emotional and Behavioral Health Problems among At-Risk Children, Youth, and Young Adult Populations (25 Plans) Individual/Family Level Outcomes Program/System Level Outcomes (School-related outcomes for children, youth, and TAY) Impact on system and community  Enter and complete Kindergarten capacity:  Increase school involvement  Expand use of evidenced based  Improve attendance rates practices for depressed TAY in  Improve literacy community settings and among  Improve academic performance (e.g., credit accrual organizations and Grade Point Average)  Expand training of evidenced based  Improve ability to function independently with a practices for depressed TAY to focus on education and vocational goal community providers  Improve student self-esteem  Improve inter-agency cooperation  Avoidance of school failure and care management (e.g., between social services, behavioral health, and  Reduce personal stigma (of mental health) to access education) school-based services  Improve inter-agency collaboration to  Reduce number of suspensions address other basic needs of target  Reduce number of school discipline referrals populations  Reduce recidivism (youth parolees)  Improve ability of school districts to respond to critical incidents and acts Impact on family unit: of violence  Improve family functioning  Improve access to housing supports  Improve family safety  Increase funding through in-kind  Improve family communication contributions and partner funds  Improve relationships between child, youth,  Increase mental health services for parents, and other adults (e.g., teachers) depressed TAY in juvenile justice  Increase utilization of community support systems settings  Reduce family conflict  Increase opportunities for children  Reduce homelessness and youth to participate in safe,  Reduce incidence of reoccurring reports of child culturally-appropriate afterschool abuse and neglect activities  Reduce incidence of child removal from home  Increase exposure of community members to cultural activities and Impact on parents and caregivers: traditional healing experiences  Increase peer/social support among parents  Improve parents’ knowledge of infant and early childhood development  Improve positive parenting skills  Improve competence and confidence in parenting  Improve parent/caregiver mental health  Reduce parent or caregiver stress 10 Stigma and Discrimination Seven (7) local evaluation plans identified intended outcomes for mental health stigma and discrimination against persons with mental illness (see Table 4). The intended outcomes for the individual/family level represent changes in individuals (e.g., increase understanding of mental well-being and illness, reduce personal stigma). Therefore, the proposed outcomes target changes in attitude, knowledge, and behavior on an individual level that ultimately impacts families, communities, and society in reducing mental health stigma and discrimination. The intended outcomes in the program/system level describe changes in the formal structures within and across programs and communities that facilitate these changes for individuals. The core theme for program/system level is increasing capacity of various entities to work together to reduce stigma and discrimination for certain populations where these issues are especially persistent due to cultural beliefs and attitudes about mental illness. Table 4. Intended Outcomes on Stigma and Discrimination (7 Plans) Individual/Family Level Outcomes Program/System Level Outcomes Impact on individuals: Impact on program capacity:  Increase understanding of mental well-  Increase school-based assessment and response being and illness systems to improve access to mental health  Increase receptivity to mental health services (as a way to normalize mental health services intervention and reduce stigma)  Reduce stigma (personal bias against people with mental health issues) Impact on community capacity:  Reduce fear toward people who struggle  Increase use of ethnic/cultural community with mental and emotional health issues partners  Increase community knowledge of early signs of mental illness  Increase collaboration between primary care providers and mental health providers  Increase collaboration with existing partners (e.g., schools, faith-based organizations, social service agencies, and law enforcement)  Reduce stigma amongst specific populations (e.g., older adults and other ethnic and cultural groups) 11 Suicide Risk Presented in Table 5 are the intended outcomes that are proposed in five local evaluation plans to address suicide risk. All five plans address the individual level outcome of preventing successful suicide (not only among those with suicide ideation but among those at risk in the general public). Only one program’s plan proposed to measure family level outcomes. The intended outcomes for this program, which targets the Lesbian, Gay, Bisexual, Transgendered, and Questioning (LGBTQ) community, emphasize the importance of families and communities in supporting individuals to reduce risk factors for suicide ideation, suicide attempt, and successful suicide. On the program/system level, the intended outcomes are about the potential impact of PEI on building capacity of service providers to be more responsive to individuals who are at risk of suicide and of programs to effectively respond to crises in order to prevent suicide. Table 5. Intended Outcomes for Suicide Risk (5 Plans) Individual/Family Level Outcomes Program/System Level Outcomes Impact on individuals: Impact on program capacity:  No successful suicide  Increase work hours for Spanish-speaking service  Increase number of survivors providers to answer calls related to suicide (medium to high-risk callers into  Increase trained multilingual/multicultural crisis line crisis line) volunteers  Reduce isolation  Improve identification of persons suffering from  Reduce depression depressive, anxiety, co-occurring, and other disorders  Reduce suicide attempts  Improve access to mental health counseling services and  Reduce suicidal ideation referrals to public and private mental health services  Reduce hospitalizations for  Improve response times to calls about suicide physical or emotional disorders  Reduce abandonment rates during calls about suicide  Improve time to offer and receive mental health services Impact on families: and supports  Improve communication and  Increase service to diverse populations support among LGBTQ families  Increase family acceptance for Impact on community capacity: LGBTQ youth  Expand range of community building activities and social  Increase social support among support groups LGBTQ family members  Strengthen sense of community affiliation in the LGBTQ community to reduce suicide Impact on society:  Reduce health care costs associated with mental health crises 12 Findings on Content and Quality of Evaluation Plans Data Source and Analysis As in the analysis above on intended outcomes, the data source for this analysis was the local evaluation plans. The sample included the 37 counties that submitted a local evaluation plan. The purpose of this analysis was to assess the quality of data that could potentially be derived from the local evaluations. We developed a rating system to systematically assess the quality of four evaluation components: outcomes, outcome measures, study design, and fidelity monitoring. Each component was primarily assessed for the extent to which the description and information provided was accurate and consistent with the project goals and objectives. We rated the four evaluation components using specific criteria developed for each (see below for criteria). Summary of Findings on Content and Quality The set of criteria for rating the content and quality of the four evaluation components is presented, followed by an analysis of the local evaluation plans using those criteria. Intended Outcomes 0 There are no identified outcomes proposed in the evaluation plan. Proposed outcomes are not consistently clear, measurable, or appropriate. They are co- mingled with process variables that measure program implementation, or they are 1 misidentified as individual/family outcomes rather than program/system outcomes (or vice versa). Proposed outcomes are clear, measurable, and appropriate. They are not co-mingled with 2 process variables. All counties identified intended outcomes for the PEI project(s) they plan to evaluate. Most evaluation plans discerned between individual/family outcomes and program/system outcomes per instructions included in the local evaluation plan template. More than half of the evaluation plans (21 or 57 percent) are rated as “2” meaning that the proposed outcomes are clear and measurable. The remaining plans (16 or 43 percent) do not have this level of clarity with respect to identifying intended outcomes. They either propose outcomes that are not clear or measurable, or the outcomes technically represent process variables that measure program implementation (see “Findings on PEI Data Elements” for a discussion of process-oriented data elements). 13 Outcome Measures There are no clearly identified outcome measures. Although outcomes are identified, 0 there is no information about indicators or data collection instruments to measure the outcomes. Indicators or data collection instruments are identified but they are incomplete or do not 1 consistently reflect the proposed outcomes for individuals/families and programs/systems. Indicators or data collection instruments are clearly identified and reflect the proposed 2 outcomes for individuals/families and programs/systems. Although all counties identified their intended outcomes in the local evaluation plans, not all identified the outcome measures or data collection instruments associated with the intended outcomes. Four counties (11 percent) did not describe measures at all. Close to half of the counties (46 percent) described indicators or data collection instruments to measure the intended outcomes; however, they are either incomplete (e.g., only one measure is identified for five separate outcomes) or they do not reflect the proposed outcomes. The remaining one- third of counties (35 percent) clearly identified the indicators or data collection instruments, and the measures matched the proposed outcomes. Counties that proposed to implement an evidence-based practice typically proposed using an instrument or set of instruments available from or recommended by the developers of the practice. Others proposed using standardized measures, for example, the Patient Health Questionnaire (PHQ-9) for depression. Most counties that proposed outcome measures for their local evaluation identified logging tools or satisfaction questionnaires that they either developed or planned to develop for their local evaluation. Study Design There is no description of the study design in terms of when, how, and for whom data on 0 proposed outcomes will be collected. Description of the study design is incomplete and/or does not consistently match the 1 proposed outcomes. Description of the study design is clear, accurate, and appropriate for the proposed 2 outcomes. Study design is the procedure under which the study is carried out. Typically, study designs are described as non-experimental, quasi-experimental, or experimental. While the local evaluation plan did not explicitly call for a description of study designs in this way, it did ask counties to describe how data will be collected and analyzed. Two counties (5 percent) did not provide any 14 description of data collection and analysis. More than half of the counties (21 or 57 percent) provided a description, but the description is either incomplete or the study design is not consistent with the proposed outcomes (e.g., there is a description of a service tracking log, but the primary outcome is “decreasing social isolation”). In 14 PEI plans (38 percent), the study design is clearly explained, and the design is accurate and appropriate for the proposed outcomes (e.g., pre-test and post-test measurements of parenting skills and knowledge). Fidelity 0 There is no discernable plan for fidelity monitoring. Proposed monitoring of fidelity is limited to general administrative oversight or a generic 1 training protocol. 2 Proposed monitoring of fidelity clearly details when, how, and who will monitor fidelity. Fidelity monitoring is intended to assess the extent to which a program is implemented as intended. This concept can be applied to any program model and not just evidence-based programs. Almost one-third of the counties (12 plans) did not describe in concrete terms the procedures for fidelity monitoring. Another 16 counties (44 percent) proposed a plan for fidelity monitoring; however, the plan is either limited to general administrative oversight (e.g., program manager will monitor fidelity of the program) or a generic training protocol (e.g., staff will be trained on the model). The remaining counties (9 or 24 percent) described the procedures for fidelity monitoring by explaining when monitoring is going to take place (e.g., at different time intervals), how (e.g., observations, data collection), and by whom (e.g., supervisor, trainers, administrators). Findings on PEI Data Elements Data Source and Analysis The data source for this analysis was the Annual Updates for FY11-12, which were reviewed to assess what data elements other than outcomes have most recently been tracked for PEI. In this section of the report, we provide a summary of these types of data elements that were reported by 30 counties. (Table 6 lists typical data elements that have been reported by 30 counties for PEI in their FY11-12 Annual Updates.) These data elements could fall within four categories of data: participant demographics, participant characteristics, service provision and utilization, and program outputs. Some of these data elements could represent outcomes as 15 well, but they are included in this summary because the context in which the data elements were reported suggest they are process rather than outcome variables. It is important to note the distinction between outcome and process. Outcome questions ask about program effects. For example, outcome questions examine program results or impact on PEI participants. Outcome questions ask: What was the impact of the program? What were the short-term and long-term outcomes? Process questions ask about program operations. For example, process questions examine whom the program is serving (e.g., participant demographics and characteristics), what the program activities are (e.g., mental health screening and assessment), and what the program outputs are (e.g., number of program referrals). Process-oriented data elements are referred to as “process variables” throughout this report. Summary of PEI Data Elements Most counties reported the number of individuals or groups participating in a PEI program in the Annual Updates. Most of these counties also reported the race or ethnicity of participants. It has been common for counties to collect data on participant characteristics as a baseline for the purpose of tracking outcomes over time. For example, counties want to know participants’ risk factors such as being homeless or using drugs. The data elements on participant characteristics, however, were also reported to show the types of participants a PEI program is reaching out to and whether the target population is truly being served. This information is being used by some counties to understand barriers to service access. Many of the data elements being reported by counties for PEI represent service provision/utilization and program outputs. The most common data element under service provision/utilization is the number of participants a PEI program has served. As presented earlier in this report, counties are reporting the number of individuals served by age group, race/ethnicity, language, and cultural group in the Annual Updates. They also are reporting the types of services offered (e.g., case management, medication management), as well as the enrollment status of participants (e.g., currently enrolled or terminated). The final category of data elements is program outputs, which are program activities or counts of program activities. Within this category, counties are mainly reporting data elements that speak to the achievements of program outputs such as the hiring and training of staff, development of products (e.g., training materials, resource directories), and number of referrals. These data elements are commonly reported for auditing contractual agreements, although some of these data elements are relevant to tracking PEI outcomes. For example, the rate of referrals between primary care and behavioral health is an important piece of information for assessing 16 the extent to which access is being widened from various points of contact for individuals who need behavioral health services. Table 6. Typical Data Elements Reported for PEI Participant Demographics  Gender  Age  Race  Ethnicity  Place of residence  Primary language  Socio-economic status Participant Characteristics  Risk factors (data typically collected at intake). Examples: o Homeless o Drug use o Inability to access services due to rural location and language o Complications during pregnancy o Need for social/emotional support Services Provision and Utilization  Number of participants served/receiving services  Number of participants enrolled, graduated, terminated, or completed  Number of sessions (e.g., therapy) completed  Location of service  Number of prevention or early intervention services received  Type of staff person by which participant was served  Type of service offered and received. Examples: o Case management o Medication management o Multifamily groups o Home visits o Call-in to hotlines or warm lines (continued) 17 Table 6 (Continued). Typical Data Elements Reported for PEI Program Outputs  Number of project components implemented by program staff and others. Examples: o Trainings, accreditations, workshops, classes, and conferences o Screenings, assessments, and referrals  Achievement of program outputs. Examples: o Implementation of an intervention o Development of resource directory, brochures, and flyers for print and distribution o Creation of committees, teams, task forces o Development of partnerships and partnership MOUs o Number of staff, bilingual staff, volunteers, and interns hired o Number of community events held, meetings attended o Number of people outreached, educated, and trained o Number of participants contacted o Number of stories aired/reported in media o Rate of call abandonment (callers hanging up before call is answered) o Number of new referrals, referral timeline, referrals by language o Number of linkages made to primary care and outpatient mental health services Findings on Individuals Served Across PEI Programs Data Source and Analysis Beginning with this section, the report transitions into a presentation of actual data reported by counties. In the following analysis, we present a summary of findings on PEI participant demographics, which is a process-oriented data element. The data sources for this analysis were information available in the Annual Updates for FY10-11 and FY11-12. The FY10-11 Annual Updates reflect implementation for FY08-09 and the FY11-12 Annual Updates reflect implementation for FY09-10. Therefore, for this analysis, the findings are presented by the implementation periods of FY08-09 and FY09-10. The information that was analyzed from these data sources was the number of individuals served across all PEI programs and estimated number of individuals served for prevention programs. The sample from implementation period FY08-09 included eight counties (or less than 15 percent of 59 counties). The sample from implementation period FY09-10 included 30 counties (or approximately 50 percent of 59 18 counties). The small sample sizes reflect the early stage of PEI implementation for many counties. For the purpose of summarizing these figures for this report, the tables below present the number of individuals by age group, race/ethnicity, and culture.vii Summary of Findings on Individuals Served across PEI Programs As shown in Tables 7 through 9, the range of individuals served is wide, as some counties served small numbers and others served very large numbers of individuals. Very large numbers are mostly explained by PEI media campaign projects that target large populations through telephone, radio, or print advertisement on mental health stigma and discrimination. In FY09- 10, more adults were served by PEI (270,892) than children (77,923), TAY (59,169), or older adults (30,317) combined (see Table 7). In the same year of reporting, the racial groups most served in PEI were Caucasians (71,169) and Hispanics (70,695) (see Table 8). These figures for African Americans, Native Americans, Asian Americans, and Pacific Islanders were comparatively lower. A small group of counties reported on the number of LGBTQ individuals served in PEI (see Table 9). In FY09-10, estimates were provided for additional cultural groups such as veterans and people with disabilities. Table 7. Number of Individuals Served in PEI by Target Population Age Group # of Counties Reporting by Range Sum Implementation Period FY08-09 FY09-10 FY08-09 FY09-10 FY08-09 FY09-10 Children 7 29 24 – 1156 5 – 21,544 3,722 77,923 TAY 5 28 63 – 7183 2 – 34,707 7,987 59,169 Adult 7 29 24 – 4258 10 – 20,2584 8,426 270,892 Older Adult 4 26 10 – 273 0 – 15,606 509 30,317 Unknown 1 6 (N/A) 1 – 7,075 34,581 9,333 TOTAL* 55,525 447,634 19 Table 8. Number of Individuals Served in PEI by Race/Ethnicity Race/Ethnicity # of Counties Reporting Range Sum by Implementation Period FY08-09 FY09-10 FY08-09 FY09-10 FY08-09 FY09-10 African American 5 25 1 – 323 3 – 4,389 555 16,741 Asian American 5 24 34 – 308 2 – 2,409 399 8,601 Caucasian 6 30 16 - 2,540 16 – 13,979 3,413 71,169 Hispanic 7 29 6 - 6,469 3 – 22,489 6,463 70,695 Native American 5 26 2 - 1,191 1 -2,198 1,299 6,479 Pacific Islander 1 14 (N/A) 1 - 373 8 1,129 Multi-Racial 4 19 11 – 97 1 – 1,058 165 3,843 Other race 4 21 2 – 31,112 1 – 946 31,200 3,916 Unknown race 1 21 (N/A) 2 – 247,884 4,270 36,2215 TOTAL* 47,772 544,788 Table 9. Number of Individuals Served in PEI by Culture Culture # of Counties Reporting by Range Sum Implementation Period FY08-09 FY09-10 FY08-09 FY09-10 FY08-09 FY09-10 LGBTQ 2 11 12 – 34 5 – 17,070 46 17,883 Other Group** 0 8 (N/A) 20 – 7,510 (N/A) 8,533 TOTAL* 46 26,416 * The totals do not match across the groupings of age, race, and culture. Because counties could estimate the number of individuals served for prevention efforts under PEI, it is unlikely that the totals will add up perfectly. ** Other cultural groups include veterans, people with HIV/AIDS, and people with disabilities. Findings on PEI Reported Outcomes Data Source and Analysis The data sources for this analysis were the FY10-11 Annual Updates, FY11-12 Annual Updates, and documents on PEI outcomes submitted by seven counties (as described earlier in the “Methods for Obtaining PEI Documents” section). The combined sample for this analysis includes 22 counties (including six very small counties) that had outcome data from one or more of these data sources. (See Appendix A for a list of counties in this particular sample.)viii We established a method for systematically reviewing these documents to understand what is known about PEI outcomes to date. First, all relevant data were extracted from the documents 20 that counties submitted on PEI, as well as from both sets of Annual Updates. The data extraction procedure was guided by a matrix that defined relevant content as: (1) reported process-oriented data elements (e.g., participant demographics and service utilization variables) and (2) reported outcome data. Next, the information extracted on only reported outcomes was categorized according to utility (participant satisfaction data were not included in our analysis of individual outcomes). (See below for the coding scheme used to categorize the outcome data). Low There was insufficient information to interpret the outcome data presented. Utility Medium Data sources, samples, and/or study methods were described, and/or there was some Utility contextual information on how the data were analyzed and interpreted. High Data sources were clear, samples and study methods were described, and there was Utility contextual information on how the data were analyzed and interpreted. Once the information from all reviewed documents had been extracted and categorized into low, medium, and high levels of utility, the data were further classified according to which of the key areas of need they addressed. Within each of these areas, the reported outcomes were clustered according to whether they addressed individual/family level outcomes or program/system level outcomes, as these were the two levels of outcomes that counties were asked to identify in their local evaluation plans. Two evaluation team members separately extracted and categorized content from a sample of three counties and compared results to establish consistency in the review process. Summary of Findings on PEI Reported Outcomes Given the early stage of PEI implementation for most counties, we did not anticipate a great deal of outcome data on PEI programs for the review period. A majority of what was reported included process-oriented data such as participant demographics (in particular, age group served and racial/ethnic background) and service outputs (e.g., the number of clients served or the number of trainings offered). These were discussed earlier in the report under “Findings on PEI Data Elements”. The available outcome information is summarized below. At this point in time, the majority of the counties reporting on their PEI outcomes are not able to provide high quality or utility outcome information based on the use of, for example, validated measurement instruments, large sample sizes, pre- and post-test measurements, comparison groups, and/or statistical tests of significance. The majority of outcome data reported by counties include little or no accompanying information to help assess the validity of 21 the reported findings or to interpret their meaning. Altogether, only five counties submitted data categorized as high utility for the purpose of this review. Thirteen (13) counties submitted medium utility data, while 18 counties submitted low utility data. These counts are not mutually exclusive. For example, counties that submitted high or medium utility data may also have submitted low utility data for different sources and/or projects. (See Table 10 for the number of counties reporting PEI outcome data in each key area of need.) It cannot be emphasized enough that counties are in the early stages of implementing their PEI projects. The small number of counties reporting outcome data, as well as the relatively low utility of these data at this time is, in part, reflective of where counties are in the developmental stage of rolling out PEI projects. A sufficient quantity of evaluation data is not available; therefore, we are limited in making statements about what is and is not working with respect to PEI projects in and across counties. Overall, the presentation of findings below is meant to be a descriptive summary of what has been reported on PEI outcomes to date. The summary itself should be reviewed in that context and in no way interpreted as a commentary or criticism of county PEI efforts. In the remainder of this section, we offer a summary of the PEI outcomes reported by counties to date that is organized by the key areas of need. Table 10. Number of Counties Reporting Outcomes in Each Key Area of Need Key Area of Need # of Counties # of Counties Reporting Reporting High Utility Data Disparities in Access 8 0 Psychosocial Impact of Trauma 3 0 Emotional and Behavioral Health 16 5 Problems among At-Risk Children, Youth, and Young Adult Populations Stigma and Discrimination 5 1 Suicide Risk 5 0 Disparities in Access to Mental Health Services In the area of access disparities, eight counties reported outcomes from their prevention efforts. Counties’ findings included outcomes reflecting the impact on individuals and families; however, unlike the available data from most of the other areas of need, a good proportion of the information tracked and reported in this area also included program/system level outcomes. Still, none of the reported outcome information from either level could be categorized as high utility at this time. 22 Impact on the Individual and/or Family Evaluations of PEI programs in this area of need are reporting positive outcomes in terms of participants’ awareness of their eligibility to receive help, their comfort with seeking help for mental health issues, their knowledge about where to receive help, and how to navigate the mental health system once they access it. Impact on Program and/or System Capacity At the program/system level, counties reported knowledge gain among service and health providers, as well as community “gatekeepers”, who received training. Program capacity to conduct targeted outreach to unserved/underserved groups and more behavioral health consultations reportedly increased as a result of PEI efforts. Among the reporting counties, some claimed this capacity improvement has “reduced cultural and ethnic disparities in our mental health system” and has “improved access”. One county reported “stronger and enhanced coordination of efforts”. Like the other low- level utility data in this area of need, this claim was not backed by specific evidence. Psychosocial Impact of Trauma Three counties reported outcomes for programs targeting the psychosocial impact of trauma; none of the data were categorized as high utility. The available information in this area was clustered according to the impact on the individual who experienced trauma, as well as the impact on that individual’s family. The very small body of information related to the program service impact on individuals who experienced trauma described increased coping skills for approximately half (51 percent) of participants. With no context or data provided to back up the statement, one county reported “significant healing of individuals and families” as a result of the early intervention services they provided. Results that were specific to families included a high level of awareness and knowledge about where to get help (76 percent of participants) and high level of comfort seeking help if needed (76 percent of participants). Emotional and Behavioral Health Problems among At-Risk Children, Youth, and Young Adult Populations The greatest amount of information reported in any one area of need to date was in the area of emotional and behavioral health problems for at-risk children, youth, and young adult populations (or TAY). Sixteen (16) counties reported data in this key area of need spanning all levels of impact and utility (specifically, five counties reported high utility data). 23 Impact on the Individual and/or Family Reported outcomes at the individual level were further classified into outcomes on child, youth, and TAY well-being; school-related outcomes; juvenile justice-related outcomes; and parent/family well-being. In terms of general well-being for children, youth, and TAY, high utility outcomes suggest that PEI program participants have demonstrated decreased behavior problems (e.g., aggression, impulsivity) and improved social competence and skills (as reported generically). Lower utility findings also favor these outcomes in social emotional behavior and competence. A PEI program for TAY in one county also reported high utility data suggesting a 23 percent increase in employment and a 3 percent decrease in homelessness for these participants. In addition, a small amount of lower utility evidence points to influences of PEI programming on mood/depression and resiliency for children, youth, and TAY. There was less high utility evidence provided by counties in the area of school-related outcomes. The one county that reported higher utility data showed that 77 percent of participants in one program demonstrated improvements in school performance, attendance, and disciplinary referrals. Lower utility data in this category support those outcomes. In addition, lower utility data described improvements in high school exit exams and graduation as well as avoidance of expulsion from school or the classroom for high-risk youth. With respect to juvenile justice-related outcomes, high utility data were scarce. One county reported program findings indicating a 12 percent decline in “legal involvement” and an 8 percent reduction in arrests for TAY participants. Lower utility data also suggest reductions in incarceration, recidivism, and participation in risky behaviors (i.e., substance use and gang involvement). Parent and family well-being appeared to be the focus of many PEI efforts for counties that had outcome data to report. A handful of high utility findings indicate that these programs have resulted in improved parenting knowledge, skills, and self-efficacy. A relatively large number of lower utility findings support this notion. Parent depression, stress, and anxiety also reportedly have been relieved for some PEI participants. For example, one county reported a statistically significant reduction in scores on the Patient Health Questionnaire (PHQ-9) for 73 percent of parent participants. Higher utility evidence from two counties points to improved family functioning for program participants (e.g., increased family safety, improved parent-child relationship). Lower utility data reported by a handful of counties appears to support the idea of improvements in these areas. 24 Improved Program and/or System Capacity There was little evidence – high utility or otherwise – concerning human resource and program capacity outcomes. One county reported that increases in mean ratings of self- efficacy among trained service providers were statistically significant. Lower utility data provided by a small number of counties describe program capacity improvements such as increases in service linkage and utilization, as well as faster times to assessment. Very little was reported in this key area of need concerning improved system capacity, such as collaboration outcomes. One county reported alignment of referral, billing, and data sharing procedures across PEI partners. A second county reported on a child welfare-related PEI program that demonstrated reduced rates of re-referrals for child abuse and neglect over time, as well as a decline in the number of re-referrals that were ultimately substantiated. The latter report was based on a relatively strong evaluation design. Stigma and Discrimination Altogether, there was little reported in this key area of need. Five counties reported outcome data from programs targeting stigma and discrimination, and there was only one finding contextualized with adequate information about measurement to be categorized as high utility. The county reporting the high utility data described a 20-27 percent improvement (depending on specific area of knowledge surveyed) concerning facts about mental illness. These improvements were statistically significant. Other counties reported knowledge gain ranging from 29 percent of participants knowing “a lot more” about people living with mental health to 96 percent reporting increased awareness of protective strategies and skills for those suffering from mental illness. At the community level, the increase in service availability was credited by one county as having a positive impact in reducing stigma and discrimination among unserved/underserved cultural populations. Suicide Risk Five counties reported outcome data for programs targeted at suicide risk. No high utility data were reported in this key area of need. The lower utility data that were available are described below. Impact on Individual and/or Family In terms of reduced suicide risk, one county reported number of persons with high levels of risk who contacted the crisis line as well as observations by one juvenile justice center that cited a “significant decrease” in reported suicide attempts. Data to support 25 those statements were not reported, thus the low-level utility categorization. One county reported performance measurement information only. They met their program target concerning the number of high-risk hotline callers that were alive after one month. Another county reported “positive changes in mood and behavior” as a result of its suicide prevention program but provided no accompanying information about measurement. The few counties with data in this key area of need reported increased sense of support and decreased feelings of loneliness or isolation for program participants. Impact on Program and/or System Capacity Scant information was reported in this domain. One county reported that 97 percent of individuals who received training and outreach demonstrated “an increase in awareness about the signs/issues related to suicide”. Others reported that increased program capacity to provide mental health screenings or to make Spanish-speaking counselors available had resulted in comprehensive assessments for more untreated and undiagnosed clients. Summary of Findings, Discussion, and Recommendations The findings described throughout this report on the various data sources (i.e., local evaluation plans within the 3-Year PEI Plans, Annual Updates (FY10-11 and FY11-12), and PEI documents submitted by counties for our analyses are summarized and discussed in terms of their strengths and areas for improvement. Supporting recommendations are provided to guide next steps in both county and statewide PEI evaluation efforts. The recommendations are presented within their respective summary sections; however, they are broad recommendations that apply to improving PEI evaluation efforts overall. PEI Intended Outcomes The data source for this analysis was the local evaluation plans in the 3-Year PEI Plans. The sample for this analysis was comprised of 37 counties that included a local evaluation plan. Overall, the intended outcomes proposed in the local evaluation plans represent appropriate constructs for measuring PEI impact in the key areas of need. In the area of Disparities in Access to Mental Health Services, intended outcomes are appropriately focused on the program/system level. For example, they describe the effects of program strategies to make mental health services more inviting and of system enhancements to improve access to mental 26 health services through better coordination. There is a greater emphasis on individual/family level outcomes in the area of Psychosocial Impact of Trauma. The intended outcomes for individuals, for example, typically measure individual behavior change, knowledge and skill improvement, and reduction of symptomology. Reflective of the larger number of local evaluation plans targeting this key area of need, the number of intended outcomes proposed was most abundant for Emotional and Behavioral Health Problems among At-Risk Children, Youth, and Young Adult Populations. The intended outcomes for children and youth projects are common constructs measured in prevention and early intervention programs. Their focus on risk and protective factors is appropriate for both individual and family levels of outcomes. The program/system level is also addressed by intended outcomes on capacity building within and across programs and service delivery systems. The intended outcomes proposed for Stigma and Discrimination target changes in attitude, knowledge, and behavior on an individual level that ultimately impacts families, communities, and society in reducing stigma and discrimination. Finally, the intended outcomes proposed across the PEI projects for Suicide Risk focus largely on the individual level outcome of preventing successful suicide. There were many intended outcomes proposed in the local evaluation plans that cover a wide range of constructs within and across the key areas of need. Many of these intended outcomes were not tied explicitly to actual measurement tools. Therefore, in the future, it will be challenging to synthesize PEI outcomes across counties and/or programs in a coherent and succinct way. Recommendations 1. In order to optimize what could be known about PEI impact, develop a small set of priority indicators and/or measures within each Key Community Mental Health Need and across target populations that counties should collect and report as part of an ongoing effort to evaluate PEI. This may necessitate revisiting PEI evaluation requirements so that there is ongoing reporting of PEI outcomes for local and statewide analysis of PEI efforts. Because effects of prevention efforts on system and community levels are typically detected in the longer term, a short-term evaluation strategy may fall short of capturing change at these levels. Include all levels of PEI outcomes (i.e., individual/family and program/system levels) in the set of priority indicators. This would be in addition to what counties have already selected as their local outcomes of interest. It will be important to educate counties about the value of an agreed-upon set of indicators as a way to garner buy-in for ongoing evaluation efforts. For example, if all counties use the same set of indicators, cross-county comparisons will be possible for gauging success. 27 Content and Quality of Local Evaluation Plans The purpose of this analysis was to assess the quality of data that could potentially be derived from the local evaluations. The data source for this analysis was the local evaluation plans and the sample included the 37 counties that submitted a local evaluation plan. The submission of a local evaluation plan offers an opportunity for counties to think through their evaluation of PEI. The questions in the plan also help guide counties in thinking through important aspects of evaluation: intended outcomes, measures, design, samples, data collection procedures, analysis, and fidelity monitoring. While some counties presented solid evaluation plans, there was inconsistency overall across counties in terms of content and quality of the plans. This variation across all plans is explained by a number of factors – arguably the most critical of which is the capacity of counties to develop evaluation plans that have integrity and rigor. The lack of clarity in presenting, for example, the intended outcomes, design, and measures in many local evaluation plans is a strong indication that some counties do not have the internal capacity to develop (and possibly, to implement) evaluations of high quality. For instance, while the focus on either individual/family or program/system levels is generally appropriate for the corresponding key area of need, there is a lack of specificity around measures of program/system level outcomes. We surmise that this lack of specificity is due largely to unfamiliarity and challenge of measuring program/system level outcomes in county mental health systems and in the field in general. It may also be due to lack of clarity and guidance around PEI evaluation goals and expectations. Recommendations 2. Establish overall evaluation goals for PEI. Provide clear expectations and guidance to counties so that they can help meet those goals. Developing a set of priority indicators across counties is an example of establishing goals and providing clear expectations and guidance for evaluating PEI in the future. 3. Provide counties with support and technical assistance on designing evaluation studies; collecting and analyzing data; assessing the extent to which programs are implemented as intended (fidelity monitoring); and reporting, disseminating, and utilizing findings. Support to counties might come in the form of county-to-county peer learning collaboratives. The technical assistance should be tailored to the existing capacity of counties so that smaller counties, for example, receive technical assistance that is customized to their needs. Leverage existing resources (e.g., university-based workshops, online evaluation trainings, and evaluation toolkits) to supplement intensive and/or direct technical assistance. 28 4. Provide counties with guidance to identify and collect outcome data on the family, program, and system levels to ensure that all levels (and not just the individual level) are adequately included in their PEI evaluations. PEI Data Elements The data source for this analysis was the Annual Updates for FY11-12, which were reviewed to assess what data elements have been most recently reported for PEI. A summary of these types of data elements that were reported is based on 30 counties. In addition to the outcome data reported on PEI to date, the data elements represent participant demographics (e.g., participant race and/or ethnicity), participant characteristics (e.g., risk factors), service provision and utilization (e.g., number of participants served), and program outputs (i.e., program activities such as the completion of staff training and number of referrals). Generally, the data elements that counties have reported thus far are appropriate and meaningful and capture a variety of process variables that are important to track for other reasons such as helping to explain PEI outcomes. That is, they are typically collected in evaluation studies, including process evaluations that examine who the program serves (characteristics and demographics), what the program provides (service provision and utilization), and what the program activities are (program outputs). Collecting adequate process-oriented data is necessary for counties to soundly interpret outcomes. However, they are not sufficient for assessing outcomes. This is an important distinction for counties to understand. The types of process-oriented data that are collected by counties have to be comprehensive, covering a full range of variables that may help explain PEI outcomes. At the same time, process variables should not be a substitute for outcome data in examining program effectiveness. Recommendations 5. Ensure that counties understand how to use process-oriented data to help interpret program outcomes. This could be accomplished through a combination of technical assistance and reporting structures. Reporting structures might: (1) delineate between outcome and process-oriented data, and (2) require an analysis of how process-oriented data help to explain outcome findings. 29 Individuals Served in PEI The data sources for this analysis included the Annual Updates for FY10-11 and FY11-12. The FY10-11 Annual Updates reflect implementation for FY08-09 and the FY11-12 Annual Updates reflect implementation for FY09-10. Therefore, for this analysis, the findings are presented by the implementation periods of FY08-09 and FY09-10. Counties reported the number of individuals served across all PEI programs and estimated the number of individuals served for prevention programs. Only eight counties reported the number of individuals served for implementation in FY08-09. The estimated number of individuals served was 55,525. For implementation in FY09-10, 30 counties reported these figures with an estimate of 447,634 individuals. For both fiscal years, the reported number of individuals served was presented by age group, race/ethnicity, primary language, and/or cultural group. Reports of how many individuals are being served by PEI are useful for understanding whether or not PEI implementation has begun for each county, as well as the scale of PEI programs. Also, with basic demographic information being reported by counties, these reports are useful for assessing disparities in access to services. Despite these uses, the numbers reported by counties should be analyzed and/or compared across counties with caution because they are relative to county size, target population, and project scope. For example, media campaigns typically target very large numbers of individuals; therefore, the numbers for this type of PEI project would not be compared to the numbers for an early intervention project that targets much smaller groups of individuals. Even the validity of numbers is in question, warranting further caution in interpreting the numbers. For example, the numbers reported by counties often do not match up across the groupings of age, race/ethnicity, language, and culture. Furthermore, while it is useful to report numbers of individuals served by race/ethnicity, this information alone is not as meaningful as contextualizing the numbers based on the racial/ethnic makeup of the target community. Recommendations 6. Have counties report separately: (1) actual number of individuals served across prevention programs, (2) actual number of individuals served for early intervention programs, and (3) estimated number of individuals served in prevention programs. Change reporting format/structure accordingly to aid counties in providing more accurate counts of individuals served. 7. Provide guidance to counties on how to report the number of individuals served across PEI programs, including how to use and report the data in order to describe populations served 30 by PEI. For example, assist counties to utilize the data for examining racial/ethnic disparities in access and outcomes. Help them to contextualize information on race/ethnicity so that it can be compared to the racial/ethnic distribution of individuals across communities and used to examine disproportionality in PEI outcomes across racial/ethnic groups. PEI Reported Outcomes The data sources for this analysis were the FY10-11 Annual Updates, FY11-12 Annual Updates, and documents on PEI outcomes submitted by counties. The sample for this analysis includes 22 counties that had outcome data from one or more of these data sources.ix The small number of counties reporting outcome data, as well as the relatively low utility of these data at this time is, in part, reflective of where counties are in the developmental stage of rolling out PEI projects. A sufficient quantity of evaluation data is not available; therefore, we are limited in making statements about what is and is not working with respect to PEI projects in and across counties. Overall, the presentation of findings below is meant to be a descriptive summary of what has been reported on PEI outcomes to date. The summary itself should be reviewed in that context and in no way interpreted as a commentary or criticism of county PEI efforts. The greatest amount of data, as well as the greatest amount of high utility data, was reported in the key area of need for Emotional and Behavioral Health Problems among At-Risk Children, Youth, and Young Adult Populations (or TAY). (Data were categorized as “high utility” if data sources were clear, samples and study methods were described, and there was contextual information on how the data were analyzed and interpreted.) High utility data from five counties begin to suggest the following with respect to individual/family level outcomes in this key area of need. Although relatively speaking the strongest evidence was presented in this key area of need, the findings still must be interpreted with caution given the small quantity of data:  PEI program participants are demonstrating decreased behavior problems (e.g., aggression, impulsivity) and improved social competence and skills for children, youth, and TAY;  Programs for the TAY population may have a positive influence on employment and homelessness outcomes, as well as reductions in “legal involvement”, including arrests;  Parent-focused programs may be resulting in improved parenting knowledge, skills, and self-efficacy; decreased parental depression, stress, and anxiety; and improved family functioning. 31 Few counties possessed actual evaluation reports structured in a manner that clearly articulates evaluation questions, methods, and findings; and a majority of reported outcomes consisted of decontextualized fragments of data that were difficult to interpret meaningfully. Moreover, of all the documents reviewed for this summary of PEI reported outcomes, there was no reporting of neutral or negative findings. One of the goals of evaluation is to judge the worth of a program. In order to achieve this, we need to know both what works and what does not. Therefore, it is equally important to report positive findings, as well as neutral or negative findings. The current structure (or lack thereof) provides little guidance to counties on reporting evaluation findings, including the reporting of neutral or negative findings for PEI. Some counties presented only benchmarks for performance in their reports. For example, if their outcome was to decrease parental stress, they reported that 70 percent of participants met their benchmark. However, they did not report the actual outcome of how participants scored on the stress index, indicating the degree of improvement in stress level. A focus on meeting performance benchmarks without providing the underlying measure on which the benchmark is judged may be helpful for quality assurance or program improvement purposes but not for synthesizing findings on PEI impact across counties. Lastly, what has been reported in terms of outcomes up to this point focuses much more on individual/family level outcomes than on program/system level outcomes. This may be a result of the fact that local evaluation plans typically lacked specified measures for intended program/system level outcomes – in particular, system level outcomes such as collaboration and community capacity. It likely reflects the reality that system level evaluation is complex and that there is generally less capacity to conduct this type of evaluation. This is a concern given the context of this initiative, where system and community change are certainly being targeted and are critical cornerstones of prevention and early intervention. Recommendations 8. Provide counties with resources, guidance, and technical assistance to report the specific contextual information (e.g., design, methods, sample size, measurement tools) required for interpreting the validity and strength of local findings. Resources might include a budget to support one point person in each county who is responsible for summarizing, synthesizing, and reporting all local evaluation findings at the PEI project level. There might also be a support person or team at the State level responsible for providing the guidance suggested above to the county. 9. Develop a reporting format for PEI evaluation findings such that outcome data are submitted to the State in a manner that facilitates an effective process of summarizing and synthesizing outcomes across counties. Include in the reporting format required content such as evaluation questions, study design, samples, measurement instruments and 32 timeframes for data collection, data analysis, and interpretation of findings. Specifically request that counties report null and negative findings and require that they go beyond reporting performance benchmarks. 10. In order to enhance what can be known about PEI impact, statewide analyses on PEI should group counties based on the type of projects they have chosen for their local evaluation. 11. Help counties identify appropriate program/system level indicators so that they capture the full spectrum of potential effects of prevention and early intervention. End Notes i Of these 22 counties, six very small counties reported outcomes. This number does not include the one very small county that submitted a local evaluation plan because that county did not report any outcomes to date. ii California Department of Mental Health. (2008). Mental Health Services Act: Proposed Guidelines: Prevention and Early Intervention Component of the Three-Year Program and Expenditure Plan. http://www.dmh.ca.gov/Prop_63/MHSA/Prevention_and_Early_Intervention/docs/Rev_PEI_Guidelines_Referenci ng_RM.pdf iii The full request was to submit information on the impact of Community Services and Supports (CSS) and PEI on consumer outcomes and MHSA values. This evaluation report addresses only PEI. iv Some counties may be conducting other PEI evaluations outside the scope of the local evaluation plan submitted to the State. v Because the intended outcomes are presented as part of the local evaluation plan, we do not know if all these outcome data will actually be collected. Also, there is no information at this point to indicate the extent to which counties that submitted a local evaluation plan are implementing their plan. vi To be consistent with the PEI Guidelines, outcomes for children, youth, and TAY are reported together under this key area of need. It is recognized, however, that outcomes for these groups could be significantly different. We differentiate outcomes for these groups to the extent possible given the information provided in the data sources. vii Information on primary language is also included in the reporting of PEI. However, because there often are large discrepancies in these estimates, only the estimates for age group, race/ethnicity, and culture are provided. viii The number of programs/projects for the PEI reported outcomes cannot be estimated because outcomes that were reported were not consistently tied to a specific program/project. ix Of these 22 counties, six very small counties reported outcomes. This number does not include the one very small county that submitted a local evaluation plan because that county did not report any outcomes to date. 33