BHSOAC

Deliverable 2a 1

Behavioral Health Services Oversight & Accountability Commission · eval-deliverable_2a-1 · Evaluation

Read the report at Behavioral Health Services Oversight & Accountability Commission ↗

DRAFT REPORT FOR STAKEHOLDER REVIEW. RESPONSE DEADLINE: AUGUST 31, 2011 Mental Health Services Act Evaluation: Templates for Reporting Priority Indicators Contract Deliverable 2A UCLA Center for Healthier Children, Youth and Families EMT Associates, Inc. June 30, 2011 The following report was funded by the Mental Health Services Oversight and Accountability Commission. 1 Focus The Mental Health Services Act (MHSA) evaluation team was charged with developing templates and reports on statewide and county specific data that would improve understanding of how the MHSA impacted consumers. More specifically, and per contract language, the team will: Design and complete statistical analyses and reports that measure impact of MHSA at individual and system levels on indicators specified in the Matrix of California’s Public Mental Health System Prioritized Performance Indicators at the state and county levels. Draft templates, documentation of analysis, and initial statewide reports will be circulated to key stakeholders and made available to the public for input by posting on the web and making a hard copy available upon request. Individual client outcomes for full service partnerships (FSPs) by age group must be addressed for each domain (education/employment, homelessness/housing and justice involvement) as specified. Note: this impact analysis at the individual level is limited to available data (i.e., a small segment of public mental health clients, and full services partners, is reflected in this data.) Mental Health system performance must address family/client/youth perception of well-being, demographics of FSP population, FSP access to primary care, penetration rate and changes in admissions for the entire public community mental health population, involuntary care, and annual numbers served through [Community Services and Supports programs] CSS. The evaluation team submits the following report in fulfillment of this charge. We do so acknowledging that this report is not final until key stakeholders have reviewed and provided their insights about issues related to measuring the impact of MHSA. Stakeholder Feedback As noted in the contract language, input from key stakeholders and mental health service advocates is key to developing a final report. To reflect input from a range of stakeholder groups in the report’s development, the evaluation team will enlist feedback from existing groups (e.g., FSP Advisory Committee, Equality California, Racial and Ethnic Mental Health Disparities Coalition, California Mental Health Directors Association, National Alliance on Mental Illness, California Mental Health Planning Council, the California Network of Mental Health Clients, United Advocates for Children and Families, and other providers and representatives of un- served and underserved populations) over a one-month period. The evaluation team will avoid imposing additional work on these groups and instead will allow groups to rely on their existing internal processes for reviewing and responding to mental health-related reports. The evaluation team will only provide a set of questions tailored to each group’s expertise to maximize the amount and quality of feedback gained about target issues in this report. Thus, the following report is not a final product. Instead, it is a starting-point from which stakeholders can begin a conversation about measuring mental health impact since the MHSA’s initiation. 2 Feedback Process This report constitutes the beginning steps in a process designed to solicit feedback for numerous consumers and stakeholders. As such, it should be viewed as a draft. The final report, which is due on 9/30/2011, will incorporate the feedback we receive (see Figure 1: Steps Leading to Statewide and County Specific Data Reports on the following page). While we welcome feedback on all aspects of the report, along with this report we have provided a brief “guidance” document. The goal of this guidance is to provide everyone, who so chooses to comment, suggestions regarding the aspects of the report where we would like feedback. Given the timeframe for our contract we would like to receive feedback anytime between 7/29/11 and 8/31/11. After this period we will compile all the feedback, identify common themes and concerns, and revise the reports accordingly. We expect some recommendations from different individuals or organizations to be at odds with each other. We will negotiate these differences by incorporating into the report as many recommendations or alternatives views as make sense given the context. Format of feedback With the exception of general comments, feedback, whether to our guidance questions or your own suggestions, should make reference to a specific page(s) in the document so the evaluation team can most appropriately address the suggestion or concern. Comments can be emailed to the addresses below. Starting July 29th, you can download the documents from the following websites should you need them again, along with the guidance questions. MHSA Website http://www.mhsoac.gov/Announcements/announcements.aspx UCLA http://healthychild.ucla.edu/mhsa_evaluation Email Ashaki Jackson: ashakijackson@mednet.ucla.edu Rob Blagg: rblagg@emt.org OR MHSAevaluation@gmail.com 3 Steps Leading to Statewide and County Specific Data Reports Initial Statewide Evaluation 12/31/12 9/30/12 6/30/12 County reports incorporate county context Three written County specific and statewide reports, on all priority indicators Step 7 (demographics, funding, etc) Revised written report from STEP 5 in response to stakeholder input Step 6 3/31/12 Initial draft written report submitted including data for all priority indicators at the statewide level for the most recent one year period available Step 5 12/31/11 Revised deliverable from STEP 2 in response to stakeholder input obtained. Step 4 s p e 9/30/11 St Revised deliverable from STEP 1 in response to stakeholder input Step 3 9/30/11 Draft written documentation of the process for compiling the data to produce the reports for all priority indicators Step 2 Stakeholder/Consumer Feedback 6/30/11 Draft proposed standardized template for reporting all priority indicators Step 1 Currently Completed 6/30/11 Due Dates 4 Overview The following draft report outlines a strategy to assess the Mental Health Services Act (MHSA) impact throughout California. The strategy would be used to create county-level and state-level reports on outcomes related to Community Services and Supports (CSS) program outcomes using data that all counties collect regularly. The report summarizes how impact would be measured using “priority indicators” that reflect target domains in which MHSA impact should be evident. Thus, the report does not include data analysis; rather it explains data that exists. The report begins with a brief history of priority indicators and their intended use. After explaining select terms that the evaluation team will use throughout this and other reports, we more fully describe priority indicators, including the criteria used for review and the data that could be used to create priority indicators. We then describe the indicators in detail, including their relevant measures, and data sources. Detailed justification of each priority indicator follows. The report concludes with possible, practical ways the indicator set can improve what we know about MHSA impact. Background: Priority Indicators To capture how the MHSA impacts consumers throughout the state, the California Mental Health Planning Council proposed a set of performance outcomes for CSS programs. The CSS outcomes were re-conceptualized as indicators for mental health activities and services throughout California. These priority indicators are broadly defined as key measures of MHSA impact – the reduction of negative outcomes or increase in positive outcomes at the individual (consumer outcomes), system (county mental health system performance), and community levels. For example, rates of consumer homelessness and incarceration should decrease under the MHSA while client satisfaction with services and mental health promotion throughout communities should increase. The set of priority indicators came from discussions involving the Planning Council and mental health service stakeholders with the goal of streamlining the MHSA’s monitoring and planning activities. The need for such indicators was also discussed in the report Evaluation Brief: Summary and Synthesis of Findings on CSS Consumer Outcomes, submitted in preparation for the MHSA evaluation. The Planning Council decided to create priority indicators using data that was already collected across counties, reflected current statues related to the Act, was included in the federal data reporting system, and seemed intuitive to mental health service consumers and other stakeholders. The current indicator set – ultimately adopted by the Mental Health Services Oversight and Accountability Commission – is illustrated in Appendix 1 and has since garnered attention as a way to monitor quality improvement. The Council sees the benefit of these indicators stating, 5 Tracking one’s performance on key indicators over time and/or across programs and/or against other comparable counties can provide useful information to those planning, operating, and monitoring services.1 This report further hones the indicator set. Through data sorting and verification, the indicator set is revised to give a fuller picture of how MHSA contributes to consumers’ lives and shapes mental health service system performance. Objectives The objectives of this report are two-fold. The evaluation team was charged with examining 1) if data already collected by county agencies could sufficiently measure individual-level and system-level priority indicators, identifying gaps and redundancies among indicators (if any). Using these findings, and with stakeholder input, the evaluation team would 2) create data table templates for reporting priority indicators. Creating a Measurement Framework Defining Terms The evaluation team aims to increase understanding around this and other deliverables related to the MHSA evaluation. To ensure that language is clear and accessible to readers throughout the report, we include a glossary for reference (Appendix 2). When possible, our team shares how we interpret terms to include the reader and aid understanding especially where concepts become more complex. Conceptualizing “Individual-level” and “System-level” Indicators The performance indicator framework developed by CMHPC distinguishes between “individual- level indicators” and “system-level indicators.” These terms are widely used in performance monitoring systems, but it is important to clarify the terms to ensure understanding of their relevance, relationship, and priority in the measurement system described here. The individual receiving mental health services is the consumer whether child, transition age youth (TAY; 16-24 years of age), adult, or older adult. A review of technical papers and tables indicates that measuring individuals’ mental health varies and can include indicators such as the fixed attributes a consumer brings to services (i.e., demographics, education level); internal attributes (i.e., psychological and social development); behavior (i.e., the extent to which one exercises self-restraint); or one’s perception (i.e., assessment of personal growth), among others. Each measurement is a mental health indicator that is specific to and bound by the consumer. To achieve a broader understanding of the consumer, individual-level indicators can be derived from the target person or others in immediate contact with the consumer; a parent or teacher might 1 From the report “Performance Indicators for Evaluating the Mental Health System” published by the California Mental Health Council (January 2010; p.3). 6 provide responses about a target child, for example, who might also provide feedback about his or her behavior. Based on these points, we define individual-level indicators as measurement associated with mental health and cues of mental health service impact on a consumer. If mental health agencies are the contexts in which individuals operate, then systems can be explained as the overall context (procedures and policies) within which mental health service agencies operate. Systems encompass all agencies, their operations (service delivery, budgeting, administration, client and staff satisfaction, etc.) as well as the resources and policy supports required to maintain these systems. Researchers have offered a handful of indicators that better craft what is meant by “system-level.” These include, “formal commitments to a [mental health services] approach, sustainability of an initiative or policy agenda, incentives to encourage incorporation of [mental health] principles at the [agency] level, opportunities for [stakeholder engagement] in governance and policy making, and accountability for positive [consumer] development outcomes and provision of essential supports at system and [agency] levels.”2 In sum, we define system-level indicators as those related to the aggregation of activities among all agencies and the structures that maintain the system. We recognize that overlap exists between levels when responses are aggregated. For example, individuals’ self-reported rates of well-being – an individual-level indicator – can provide an assessment of system-level performance when combined as a group response. Also, the number of mental health consumers served – and agency-level indicator – can provide a system-level count when combined. These relationships are not bi-directional; system-level indicators cannot be distilled to agency- or consumer-level data although the opposite relationship might exist. Thus, we add a caveat to our distinctions: individual-level and system level indicators address separate entities, but aggregation of consumer responses can provide additional insight of system functioning. Reviewing Priority Performance Indicators, Measures and Data Sources The UCLA/EMT team considered several performance measurement criteria (outlined below) when evaluating the quality and utility of existing County Mental Health System Performance Indicators (i.e., consumer and system level indicators; see Appendix 1). To ensure consistency with, and build upon, previous work to develop a comprehensive performance measurement framework, we reviewed the criteria used by the California Department of Mental Health’s Quality Improvement Committee3 (QIC) and the California Mental Health Planning Council4 2 John W. Gardner Center for Youth and Their Communities. (2009, October). Positive youth development: Individual, setting and system level indicators (Issue Brief). Stanford, CA: Dukakis, K., London, R., McLaughlin, M., & Williamson, D.; p. 5. 3Chapter 93, Statutes of 2000, an omnibus Health Trailer Bill to the Budget Act of2000, recognized the Quality Improvement Committee (QIC) in law. 4California Mental Health Planning Council’s Performance Indicators for Evaluating the Mental Health System, January 2010. 7 (CMHPC) to establish indicators for the MHSA performance measurement system. The team also conducted a thorough review of literature regarding mental health service performance measurement. The review provided necessary background to evaluate the criteria used by the QIC and CMHPC as well as identify gaps and redundancies among the performance indicators those criteria were used to develop. In this manner, a wide set of mental health consumer and system level measurement domains and relevant indicators were cast. The quality and utility of measures and data sources that could potentially be used to operationalize indicators was reviewed according to several criteria outlined below. The data quality/utility review revealed that there are key elements of service delivery and outcomes for which data sources do not contain adequate measurement properties or are not readily available. In such cases additional data collection options are presented. These methods and criteria guide systematic evaluation of consumer and system level indicators and their underlying measures, to ensure all relevant domains of County Mental Health Systems are captured in the most rigorous and comprehensive manner possible to ultimately produce meaningful and actionable results for users (e.g., consumers/families, policymakers, and providers) who strive to improve the quality of mental health services. Specific criteria used to evaluate each indicator, measure and data source are detailed below. Criteria Used to Assess Performance Indicators Performance Indicator Coverage. To ensure all actionable points in the process of MHSA implementation are assessed, measures of County Mental Health System Performance should pertain to one of the following domains:5 5 Lutterman, T., Ganju, V., Schacht, L., Shaw, R., Monihan, K. (2003). Sixteen state study on mental health performance measures. DHHS Publication No. (SMA) 03-3835. Rockville, MD: Center for Mental Health Services, Substance Abuse and Mental Health Services Administration. 8 Performance Indicator Quality. County Mental Health System Performance measures were evaluated for:  Quality, or the extent to which they are meaningful, unambiguous and widely understood, so that they may speak to all stakeholders and drive improvement;  Support in the research base, suggesting the indicator had been informative and useful across different mental health systems;  Ability to be operationalized using data accessible/obtainable by the evaluation team, such as existing databases, or additional primary data collection, so as not to add significant burden to the measurement framework;  Based on a high level of data integrity (i.e., data collection is embedded within the normal procedures of County MHSA program, collected with fidelity, reliability), so as not to impress undue burden on the evaluation resources of counties;  Linked to critical goals and key drivers of MHSA (i.e., core values), such that the measurement framework is reflective of the overall orientation of the MHSA initiative. Performance Indicator Practicality. Measures were also assessed for their practicality, or the extent to which they are useful for evaluation purposes and statistical analysis. These relate largely to the degree to which indicator concepts are clearly defined, evidence-based, feasible in terms of data availability, understandable and actionable. Indicators must provide information to help improve and maintain MHSA services as well as provide statistical indication of change in services and their impact in consumers and families. Among evaluative needs, indicators should reflect the following criteria:  Able to drive improvement (e.g., produce actionable results)  Useful for identifying opportunities for improvement (e.g., gaps or redundancies in services)  Useful for tracking and comparing performance against both internal (e.g., organizational goals) and external standards (e.g., national benchmarks) Data Quality. Data used to represent priority performance indicators must also be evaluated for quality and utility. Specifically, data must be consistent, trustworthy, and hold properties which allow for the creation of each indicator and robust statistical analysis. Criteria used to evaluate measures and data sources of each measure include:  Adequate base rate (i.e., how often an event occurs, or level at which a scaled response is given on average, must not be so low as to make the indicator useless or meaningless)  Adequate variance (i.e., values of a given measure must be sufficiently distributed about the mean such that statistical analysis can be conducted; values cannot all be clustered at the same point)  Validity o The measure is face valid, can conceptually and logically be said to measure what it was intended to; o The performance measure is internally valid and can logically be tied to a particular program intervention or outcome; 9 o The indicator is externally valid and can logically be generalized to other populations or programs.  Reliability (i.e., the indicator is consistent across time and cases)  Availability and completeness (i.e., indicator relevant data must be obtainable and complete for populations of interest for the period of time under study) Review of Performance Indicators A systematic review of existing public mental health system prioritized performance indicators (see Appendix 1) was conducted utilizing the indicator coverage, quality, practicality, and data quality criteria specified above. This review yielded several distinct areas of measurement (e.g., education, housing, justice involvement, service access and performance), as well as gaps and redundancies in the existing measurement framework. The review of consumer and system level indicators and proposed additional areas of measurement is summarized below. Consumer-level Indicators Consumer outcomes identified by the Planning Council reflect three broad, accessible indicator categories of desired mental health intervention outcomes to be examined primarily across Full Service Partnership consumers. The categories (i.e., Education / Employment, Homelessness / Housing, and Justice Involvement) stem from previous studies and policies (e.g., Assembly Bill 2034) and were informed by indicators already in place for children’s systems of care (later applied to systems for TAY, adults, and older adults). That is, consumer outcomes were grounded in the premise that children should have stable homes, be in school, and stay out of trouble. Similarly, TAY and adults should have stable homes, be employed, and stay out of trouble. The Planning Council further limited indicators to those for which data was already systematically collected across counties. We suggest adding one indicator to the three proposed consumer outcome categories (home, school/employment, justice involvement). Averting psychiatric hospitalizations is a factor to consider when describing desired outcomes for mental health consumers. We are particularly interested in consumers’ visits to and reliance on emergency facilities like hospitals and psychiatric centers to manage their mental health – arguably the point at which management has failed. Thus, we note that Emergency Care (e.g., the reduction of visits to related centers) should be considered as an individual-level (consumer) outcome. System-level Indicators A review of existing county mental health system performance indicators across all mental health service consumers (see Appendix 1) using the criteria specified above yielded three domains of measurement, including system Access, Performance and Structure. Within each domain the existing indicators are focused on measurement of system processes (e.g., services administered, consumers reached) or system outcomes (e.g., consumer/family satisfaction, penetration rate). These domains and levels of measurement are in line with previous 10 evaluations of mental health systems.6,7 However, a review of the mental health system measurement literature revealed several gaps or redundancies among existing system performance indicators. To reduce conceptual and measurement redundancy, and address gaps in system performance measurement, additional or revised indicators proposed, include: All proposed Priority System Performance Indicators are outlined in Table 2 and detailed below. Orientation to Templates The subsequent templates represent a framework of proposed Priority Consumer and System Performance Indicators. The templates are intended to represent a menu of indicators, selected through a detailed review of existing and potential indicators against the indicator coverage, quality, practicality, and data quality criteria specified above. The columns from left to right detail the measurement domains, performance indicators, potential measures, and potential or proposed data sources. Proposed indictors, which are revisions of or complements to the performance indicators established by the California Mental Health Planning Council are highlighted. Proposed external data sources (e.g., California Health Interview Survey) or new primary data collection are identified where necessary. 6 Lutterman, T., Ganju, V., Schacht, L., Shaw, R., Monihan, K. (2003). Sixteen state study on mental health performance measures. DHHS Publication No. (SMA) 03-3835. Rockville, MD: Center for Mental Health Services, Substance Abuse and Mental Health Services Administration. 7Hermann, R.C., Mattke, S., Somekh, D., Silfverhielm, H., Goldner, E., Glover, G. Pirkis, J., Mainz, J., Chan, J.A. (2006). Quality indicators for international benchmarking of mental health care. International Journal for Quality in Health Care, September 2006, 31-38. 11 Two templates, reflecting consumer level priority indicators (Template 1) and mental health system level indicators (Template 2) are presented. Each template includes 1) the domain in which MHSA impact should be evident (e.g., education and employment), 2) priority indicators that reflect said domains, 3) possible ways to measure the priority indicator, and 4) every potential data source that includes at least one variable deemed useful in calculating the priority indicator. Although the Planning Council envisioned consumer outcomes to be measured across Full Service Partnership consumers and system outcomes to be measured across all mental health service consumers (see Appendix 1), data sources in both tables reflect possibilities for outcome calculations across all mental health services consumers (via the Consumer Services and Information [CSI] system) as well as persons enrolled in Full Service Partnerships (via the Data Collection and Reporting [DCR] system). Templates, developed using a thorough review of all data dictionaries related to mental health services, should be read as an inventory of available information. Identifying this information is one step in a process toward sorting and selecting variables, verifying data collection associated with selected variables, and testing data fidelity and reliability in an evaluation of MHSA impact. Template 1. Initial Proposed Template for Reporting Core Priority Indicators: Individual- level (Consumer) Outcomes for Full Service Partnerships8 Domain Indicator Potential Measure(s) Potential/Proposed Data Source(s) 1. 1.1 Average attendance –  Number (increase) of days at school  Data Collection and Education/ score per year annually Reporting (DCR) System9 Employment  Consumer Services and Information (CSI) system  Youth Services Survey for Families (YSS-F) 1.2 Proportion participating  Number (increase) of the consumers  DCR in paid and unpaid participating in paid and unpaid  CSI employment employment 2. 2.1 Housing situation/Index  Number (increase) of days that children or  DCR Homelessness/ – score TAY (younger than 18 years) live in the  CSI Housing family home or a foster home;  Youth Services Survey  Number (increase) of TAY or adults with (YSS) independent residential statuses  Number (increase) of older adults with housing 3. Justice 3.1 Justice involvement  Number (decrease) of consumer arrests  DCR Involvement  CSI  YSS-F  MHSIP-Adult  MHSIP-Older Adult 8 Data sources that reflect all mental health service consumers have been added in the event that knowledge broader than what is learned about Full Service Partnership Consumers is sought. 9 The Data Collection and Reporting (DCR) system collects data for consumers who are enrolled in Full Service Partnerships only. 12 4. Emergency 4.1 Emergency  Number (decrease) of consumer visits to  DCR Care hospitalization for mental the hospital or psychiatric health facility  CSI health episodes for mental health episodes annually Template 2. Initial Proposed Template for Reporting Comprehensive Priority Indicators: System-level Outcomes Domain Indicator Potential Measure(s) Potential/Proposed Data Source(s) 5. Access 5.1 Demographic  Age, gender, race/ethnicity, language  Consumer Services and Profile of Consumers spoken of consumer population compared Information (CSI) system Served to demographic profiles of individuals  Data Collection and Reporting living below the poverty line and the (DCR) system; homeless 5.2 New Consumers  Age, gender, race/ethnicity of new  DCR; by Demographic consumer population in comparison to  CSI Profile those receiving services for more than 6 months 5.3 High Need  Homeless (past 12 days and past  DCR; Consumers Served 12months);  CSI  Unemployment (past 12 days and past 12months) 5.4 Access to Primary  Consumers who have a primary care  DCR; Care Physician physician currently  CSI  Consumers who have had a primary care physician for the past 12 months 5.5 Consumer / Family  Perceived access to services  YSS; Perceptions of Access  YSS-F; to Services  MHSIP-Adult;  MHSIP-Older Adult;  Primary data collection (e.g., surveys, interviews, or focus groups; proposed additional data collection) 6. 6.1 Consumers Served  Ratio – Numerator: CSS consumers  Quarterly Progress Reports10 Performance Annually through CSS* targeted in county plan / Denominator:  Annual Updates consumers served 6.2 Involuntary Care*  Ratio – Numerator: seclusions /  Annual Report on Involuntary Denominator consumers served Detentions;  Ratio – Numerator: restraints /  CSI Denominator consumers served 6.3 24-hour Care*  Ratio – Numerator: utilization of MHRC,  DCR; SNF, SH / Denominator: TAY, Adult,  CSI Older-adult populations;  Ratio – Numerator: utilization of CTF, RCL 14, MHRC / Denominator: Child population; 10 Key informants strongly suggest replacing data collected for quarterly reports (CSS Exhibit 6) with annual updates, which were not a part of the initial data dictionary review and might face a shift in standards in light of Assembly Bill 100. The evaluation team will explore the differences in the reports’ data quality and regularity in future publications.  Asterisks refer to indicators that are processes (not outcomes). 13 Domain Indicator Potential Measure(s) Potential/Proposed Data Source(s)  Consumers in IMD, MHRC, SNF, SH by race/ethnicity;  Readmission to acute care facility within 30/180 days 6.4 Appropriateness of  Treatment protocols for co-morbidity;  DCR; Care*  Hospital readmission rate;  CSI;  Average length of stay in acute care;  YSS;  Consumer/family perceptions of  YSS-F; appropriateness of care  MHSIP-Adult;  MHSIP-Older Adult;  Primary data collection (e.g., surveys, interviews, or focus groups; proposed additional data collection) 6.5 Continuity of  ER use;  DCR; Care*  Reintroduction into community  CSI;  Discharge plans  Primary data collection (e.g., surveys, interviews, or focus groups; proposed data additional collection) 6.6 Penetration Rate  Ratio – Numerator: FSP consumers /  DCR; Denominator: individuals eligible for  CSI; services among targeted populations;  Annual Updates;  Ratio – Numerator: CSS consumers /  California Health Interview Denominator: high need populations Survey (CHIS; proposed external data source) 6.7 Consumer  Client/family perception of improvement  YSS; Wellbeing in functioning (current/over time);  YSS-F;  Client/family perception of quality of life  MHSIP-Adult; (current/over time)  MHSIP-Older Adult;  Primary data collection (e.g., surveys, interviews, or focus groups; proposed additional data collection) 6.8 Satisfaction  Consumer/family satisfaction with the care  YSS; or service  YSS-F;  MHSIP-Adult;  MHSIP-Older Adult 7. Structure 7.1 Workforce  Demographic profile comparison of  Cultural Competence Plans; Composition* workforce to consumer population  WET Plans;  Ratio – Numerator: Staff / Denominator:  Primary data collection (e.g., consumers surveys, interviews, or focus  Consumer/family member employment in groups; proposed additional data the mental health system (i.e., number, collection) FTE, % of workforce) 7.2 Evidence-  Existence of best practice core programs;  Primary data collection (e.g., Based/Best Practice  Fidelity of best practices to established surveys, interviews, or focus Programs and Services* models; groups; proposed additional data  Receipt of best practices services/supports collection) among consumers/families 7.3 Cultural  Client and family perceptions of cultural  YSS; Appropriateness of appropriateness  YSS-F; Services  MHSIP-Adult;  MHSIP-Older Adult * Asterisks refer to indicators that are processes (not outcomes). 14 Domain Indicator Potential Measure(s) Potential/Proposed Data Source(s) 7.4 Recovery,  Consumer/family member/staff  Recovery Oriented Systems Wellness, and perceptions of recovery orientation of Indicators Measure (ROSI; Resilience Orientation system and services proposed additional data collection)  Developing Recovery Enhancing Environments Measure (DREEM; proposed additional data collection) Mental Health Outcomes Indicator Detail To clarify the meaning, importance and potential utility of each domain and indicator, this section provides detailed descriptions of the indicators summarized in the tables. This discussion is based on research and professional literature, research briefs, and technical reports. Individual-level (Consumer) Outcomes for Full Service Partnerships Indicator Detail 1. Domain: Education/Employment This domain encompasses indicators of education for children and Transitional Age Youth (TAY) younger than 18 years of age as well as employment indicators among TAY who are 18 and older, adults and older adults. 1.1 Indicator: Average attendance – score per year11,12 (CMHPC Indicators #2 and #8) Population: Children and TAY Rationale for Inclusion: The number of days a youth attends school during a school year has been used as an indicator of healthy development during adolescence. School attendance has been associated with academic functioning, subjective well-being, and life satisfaction. Youth who are more vulnerable to negative mental health outcomes such as low self-concept and limited sense of social support have been linked with poor academic success including lower assessments of school importance to achieve goals and limited motivation to self-regulate learning behaviors. Further, mental health distress outside of the school environment has been thought to redirect youths’ attentions away from attending school. This indicator will help identify the extent to which MHSA programs bolster youths’ school attendance. Measure: Number (increase) of days at school annually 11 Suldo, S., & Shaffer, E., (2008). Looking beyond psychopathology: The dual-factor model of mental health in youth. School Psychology Review, 37, 52-68. 12 Kearney, C., (2008). School absenteeism and school refusal behavior in youth: A contemporary review. Clinical Psychology Review, 28, 451-471. 15 Data Source(s): Data Collection and Reporting (DCR) System; Consumer Services and Information (CSI) system; Consumer Satisfaction Surveys – Youth Services Survey for Families (YSS-F) version 1.2 Indicator: Proportion participating in paid and unpaid employment13,14 (CMHPC Indicators # 8 and #13) Population: TAY, Adults, and Older Adults Rationale for Inclusion: Unemployment has been identified as a negative outcome of untreated mental illness. Successful employment has been linked to social networks, life stability, and stamina. Some research has shown that vocational training, in combination with mental health services, has been associated with positive employment outcomes such as higher likelihood of being hired in competitive work and having an opportunity to work full-time. A count of all consumers who engage in employment will help identify the amount of consumers employed over time and account for the effectiveness of employment programs for consumers. Measure: Number (increase) of the consumers participating in paid and unpaid employment Data Source(s): Data Collection and Reporting (DCR) System; Consumer Services and Information (CSI) system 2. Domain: Homelessness/Housing This domain encompasses indicators of homelessness and the variety of housing situations among all consumers (children, TAY, adults, and older adults). 2.1 Indicator: Housing Situation/Index – Score15 (CMHPC Indicators #1, #7, #12, #17) Population: Children, TAY, Adults, and Older Adults Rationale for Inclusion: Untreated mental illness has been linked to homelessness and the ability to live independently. Supportive housing provided through MHSA programs is designed to give independent living opportunities to “low-income adults, or older adults with serious mental illness, and children with severe emotional disorders and their families who, at the time of assessment for housing services, meet the criteria for MHSA services in their county of residence and are homeless or at risk for homelessness.” Further, housing provisions might curb homelessness, which will subsequently decrease consumers’ vulnerability to justice involvement. Identifying consumers’ housing situations will improve understanding of access to housing and the range of living situations currently used. 13 Secker, J. & Membery, H. (2003). Promoting mental health through employment and developing healthy workplaces: The potential of natural supports at work. Health Education Research, 18, 207-215. 14 Cook, J., Lehman, A., Drake, R., McFarlane, W., Gold, P., Leff, H., Blyler, C., Toprac, M., Razzano, L., Burke- Miller, J., Blankertz, L., Shafer, M., Pickett-Schenk, S., & Grey, D. (2005). Integration of psychiatric and vocational services: A multisite randomized, controlled trial of supported employment. American Journal of Psychiatry, 162, 1948-1956. 15 MHSA housing program: Background, and information about the application, commitment, and funding processes. (2001, February 1). Retrieved May 20, 2011 from http://www.calhfa.ca.gov/multifamily/mhsa/process/MHSABackground.pdf 16 Measure(s): Number (increase) of days that children and TAY live in the family home or a foster home; number (increase) of TAY, adults, within dependent residential statuses; number (increase) of older adults with housing Data Source(s): Data Collection and Reporting (DCR) System, Consumer Services and Information (CSI) system, Consumer Satisfaction Surveys – Youth Services Survey (YSS) version 3. Domain: Justice Involvement 3.1 Indicator: Justice Involvement16, 17 (CMHPC Indicators #1, #7, #12, #17) Population: Children, TAY, Adults, and Older Adults Rationale for Inclusion: Research has shown that a percentage of former inmates who became mental health service consumers had been arrested previously for behaviors stemming from preexisting disorders. That is, an episode left these consumers vulnerable to arrest and incarceration. Among youth, some studies have found significantly higher occurrence of externalizing behaviors, attention deficit, and defiance among those who had been arrested compared to those who had not. This indicator will follow consumers’ interactions with the justice system to explore how participation in MHSA programs shapes number of arrests. Measure: Number (decrease) of consumer arrests Data Source(s): Data Collection and Reporting (DCR) System, Consumer Services and Information (CSI) system, Consumer Satisfaction Surveys – Youth Services Survey for Families (YSS-F) version, Adult version, and Older Adult version 4. Domain: Emergency Care 4.1 Indicator: Emergency hospitalizations related to mental health episodes (Proposed Indicator) Population: Children, TAY, Adults, and Older Adults Rationale for Inclusion: Hospital stays can indicate poor or lack of mental health management. Mental health services and related supports might curb the need for hospitalization related to mental health episodes. This indicator can account for consumers’ hospitalizations and provide trends of reliance on hospitals for mental health management. Measure: Number (decrease) of consumer visits to the hospital or psychiatric facility for mental health episodes annually 16 Daly, R., (2011, January 7). Study examines relationship of arrests, mental illness. Psychiatric News, 46, 9-10. 17 Center for Behavioral Health Services & Criminal Justice Research. (2009, September). The effects of mental health problems on juvenile arrests (Research Brief). New Brunswick, NJ: Hirschfield, P., Maschi, T., Raskin White, H., & Goldman Traub, L. 17 Data Source(s): Data Collection and Reporting (DCR) System; Consumer Services and Information (CSI) system Mental Health System-level Outcomes for all Consumers Indicator Detail 5. Domain: Access This domain encompasses indicators of consumers’ and families’ ability to obtain timely and convenient care or service based on needs. Processes 5.1 Indicator: Demographic Profile of Consumers Served (CMHPC Indicator #3018) Rationale for Inclusion: Demographic description of those receiving FSP services within and across counties, and in comparison to populations in need (e.g., overall MH service population), will provide a better understanding of who is accessing services. Such information may provide insight into ways to improve FSP service outreach and implementation. Potential Measure(s): Age, gender, race/ethnicity of FSP population compared to demographic profiles of individuals living below the poverty line and the homeless. Potential Data Source(s): Data Collection and Reporting (DCR) System; Consumer Services and Information (CSI) system. 5.2 Indicator: New Consumers by Demographic Profile (CMHPC Indicator #3419) Rationale for Inclusion: Demographic description of all new consumers (i.e., those not receiving services for prior 6 months) within and across counties, and in comparison to the existing service population, will provide description of how access of services may be changing. Specifically, this indicator may serve as a gauge of the penetration of outreach and engagement services, including what has been done to engage underserved populations. Potential Measure(s): Age, gender, race/ethnicity of new consumer population in comparison to those receiving services for more than 6 months. Potential Data Source(s): Data Collection and Reporting (DCR) System; Consumer Services and Information (CSI) system. 5.3 Indicator: High Need Consumers Served20 (Proposed Indicator) Rationale for Inclusion: Previous studies have indicated a high occurrence of mental illness amongst the homeless, those who are unemployed21, and those in poverty. Homeless individuals 18, 19California Mental Health Planning Council’s Performance Indicators for Evaluating the Mental Health System, January 2010. 20Health Care for the Homeless Clinicians’ Network. 2000. “Mental Illness, Chronic Homelessness: An American Disgrace.” Healing Hands 4:1-6. 21Linn, M., Sadifer, R., and Stein, S. (1985). Effects of unemployment on mental and physical health. American Journal of Public Health, 75, 502-506. 18 and those in poverty tend not to seek necessary supportive services. Thus, connecting these groups with appropriate services becomes difficult and requires extensive outreach and engagement to do so. Profiling service to these groups will provide greater understanding of access to services among these high need groups, within counties and across the state. Potential Measure(s): Homeless (past 12 days and past12months); Unemployment (past 12 days and past 12 months) Potential Data Source(s): Data Collection and Reporting (DCR) System Outcomes 5.4 Indicator: Access to Primary Care Physician22, 23 (CMHPC Indicator #3124) Rationale for Inclusion: Individuals with mental illness tend to experience poor health, as compared to the general population. The medical needs of those with mental illness are often not met due to poor access to general health care. This indicator will provide indication of the extent to which FSP services have been successful in connecting consumer with regular sources of primary health care. Potential Measure(s): Consumers who have a primary care physician currently/past 12 months Potential Data Source(s): Data Collection and Reporting (DCR) System 5.5 Indicator: Consumer / Family Perceptions of Access to Services25, 26 (Proposed Indicator) Rationale for Inclusion: Subjective evaluations of services can provide indications that barriers may exist to accessing care or service. Additional qualitative data collection can provide indications of the specific problems that may hinder access to care. Potential Measure(s): Perceived access to services Potential Data Source(s): Consumer Satisfaction Surveys – YSS, YSS-F, Adult, and Older Adult; primary data collection (e.g., surveys, interviews, or focus groups; proposed additional data collection) 22Druss, B.G. and Rosenheck, R.A. (1998). Mental Disorders and Access to Medical Care in the United States. American Journal of Psychiatry, 155(12), 1775-1777. 23 Statutory outcome: Improve health and mental health (WIC 5801(d)(2), WIC 5806(a), WIC 5840(a), WIC 5840(c)) 24California Mental Health Planning Council’s Performance Indicators for Evaluating the Mental Health System, January 2010. 25Onken, S., Dumont , J., Ridgeway, P., Dornan, D., and Ralph, R. (2002). Mental Health Recovery: What Helps and What Hinders? NASMHPD and NTAC. 26 Statutory outcome: Reduce disparities in access (MHSA Section 3(d), WIC 5878.3(b), (WIC 5813.5(d), WIC 5840(a), WIC 5830(a)(1)) 19 6. Domain: Performance This domain includes indicators of the extent to which county mental health system processes met the values and expectations of consumers and families, communities, providers and the MHSA initiative overall. Processes 6.1 Indicator: Consumers Served Annually through CSS27 (FSP, GSD, Outreach & Engagement; CMHPC Indicator #4328) Rationale for Inclusion: Tracking the number of individuals targeted and served though CSS services will provide a snapshot of system implementation and highlight progress toward achieving service goals. Potential Measure(s): CSS consumers targeted compared to those who were served. Potential Data Source(s): Quarterly Progress Reports (i.e., CSS Exhibit 6); Annual Updates. 6.2 Indicator: Involuntary Care29 (CMHPC Indicators #35 & #3630) Rationale for Inclusion: Tracking the number of consumers requiring therapeutic seclusion or restraint, as compared to populations served (e.g., age groups) will provide indication of the extent to which mental health systems employ these therapeutic strategies. Potential Measure(s): Ratio of seclusions/restraints, compared to consumers served and various population (e.g., Adult, Child, TAY, minority, homeless, etc). Potential Data Source(s):Annual Report on Involuntary Detentions; Quarterly Progress Reports (i.e., CSS Exhibit 6); Annual Updates; CSI 6.3 Indicator: 24-hour Care31 (Revision of CMHPC Indicators #37-4132) Rationale for Inclusion: The use of Long-Term Strategies for Community Placement and Alternatives to Institutions for Mental Diseases (IMDs) among various populations/groups (e.g., age, race/ethnicity) can provide a picture of how consumers are cared for within county mental health systems and across the state. Potential Measure(s): Utilization of Institutions for Mental Disease (IMD)/Mental Health Rehabilitation Centers (MHRC)/Specialized Nursing Facilities (SNF)/State Hospitals (SH) 27Statutory outcome: Implement MHSA county plans(WIC 5847(b)) 28 California Mental Health Planning Council’s Performance Indicators for Evaluating the Mental Health System, January 2010. 29 Statutory outcome: Implement Recovery Vision (WIC 5813.5(d)) 30California Mental Health Planning Council’s Performance Indicators for Evaluating the Mental Health System, January 2010. 31 Lutterman, T., Ganju, V., Schacht, L., Shaw, R., Monihan, K. (2003). Sixteen state study on mental health performance measures. DHHS Publication No. (SMA) 03-3835. Rockville, MD: Center for Mental Health Services, Substance Abuse and Mental Health Services Administration. 32 California Mental Health Planning Council’s Performance Indicators for Evaluating the Mental Health System, January 2010. 20 compared to Child/TAY/Adult/Older-adult populations; Consumers in IMD/MHRC/SNF/SH by race/ethnicity; Readmission to acute care facility within 30/180 days. Potential Data Source(s): Data Collection and Reporting (DCR) System; Consumer Services and Information (CSI) system 6.4 Indicator: Appropriateness of Care 33 (Proposed Indicator/Revision of CMHPC Indicator #2434) Rationale for Inclusion: This indicator will focus on the extent to which care or service is relevant to consumer/family needs. Several factors may provide evidence of the appropriateness of care or service, including: 1) the existence of treatment protocols for co-morbidity, as serious mental illness often co-occurs with substance use disorders; 2) high rate of hospital readmission within a relatively short period may indicate poor quality of care; 3) greater length of stay in acute care facilities may indicate inadequate services or supports; 4) consume/family perceptions of the appropriateness of care they receive, including involvement and a sense of empowerment in the treatment decision making process, will provide a key reflection of services as received. Potential Measure(s): Treatment protocols for co-morbidity; Hospital readmission rate; Average length of stay in acute care; Consumer/family perception of appropriateness of care Potential Data Source(s): Data Collection and Reporting (DCR) System; Consumer Services and Information (CSI) system; Consumer Satisfaction Surveys – YSS, YSS-F, Adult, and Older Adult; Quarterly Progress Reports; Annual Updates 6.5 Indicator: Continuity of Care35 (Proposed Indicator) Rationale for Inclusion: This indicator will center on the extent to which county mental health systems provide uninterrupted, coordinated care and services across programs, providers, organizations, and levels of care/service. Potential Measure(s): ER use; Reintroduction into the community; Single care/service point of contact/accountability; Physician reimbursement mechanisms; Documented discharge plans Potential Data Source(s): Data Collection and Reporting (DCR) System; Consumer Services and Information (CSI) system; Consumer Satisfaction Surveys – YSS, YSS-F, Adult, and Older Adult; Quarterly Progress Reports; Annual Updates 33McEwan, K., and Goldner, E. (2001). Accountability and performance indicators for mental health services and supports: A resource kit. Department of Psychiatry, University of British Columbia. Retrieved March 16, 2011 (http://www.hc-sc.gc.ca/hppb/mentalhealth/service) 34 California Mental Health Planning Council’s Performance Indicators for Evaluating the Mental Health System, January 2010. 35Hermann, R.C., Mattke, S., Somekh, D., Silfverhielm, H., Goldner, E., Glover, G. Pirkis, J., Mainz, J., Chan, J.A. (2006). Quality indicators for international benchmarking of mental health care. International Journal for Quality in Health Care, September 2006, 31-38. 21 Outcomes 6.6 Indicator: Penetration Rate36 (Revision of CMHPC Indicator #3337) Rationale for Inclusion: The number of consumers served in relation to those eligible or in need of services among various groups (e.g., age, gender, race/ethnicity, individuals in poverty), will provide a snapshot of the extent to which CSS services are reaching targeted groups. Potential Measure(s): Ratio of FSP consumers serve, compared to eligible for services among targeted populations; Ratio of CSS clients served, as compared to high need populations. Potential Data Source(s): Data Collection and Reporting (DCR) System; Consumer Services and Information (CSI) system; California Health Interview Survey (CHIS; proposed external data source) 6.7 Indicator: Wellbeing38 (Revision of CMHPC Indicator #3339) Rationale for Inclusion: Perceptions of improvements in functioning, the appropriateness of care they receive, participation in treatment, quality of life, and satisfaction with services among consumer groups (e.g., age, gender, race/ethnicity, individuals in poverty), can provide indications of the quality of service within county mental health systems and across the state. Potential Measure(s): Improvement in functioning (current/over time); Quality of life (current/over time) Potential Data Source(s): Consumer Satisfaction Surveys – YSS, YSS-F, Adult, and Older Adult; Primary data collection (e.g., surveys, interviews, or focus groups; proposed additional data collection) 6.8 Indicator: Consumer Satisfaction (Revision of CMHPC Indicator #2540) Rationale for Inclusion: Consumer/family satisfaction with the care and service they receive will provide an important reflection of the ability of county mental health systems to achieve stated values and goals. Potential Measure(s): Consumer/family satisfaction with the care or service Potential Data Source(s): Consumer Satisfaction Surveys – YSS, YSS-F, Adult, and Older Adult 36 Statutory outcome: Increase number of individuals receiving public mental health services (MHSA Section 3(d), WIC 5813.5(a), WIC 5830(a)(4)) 37California Mental Health Planning Council’s Performance Indicators for Evaluating the Mental Health System, January 2010. 38 Statutory outcome: Improve health and mental health (WIC 5801(d)(2), WIC 5806(a), WIC 5840(a), WIC 5840(c)) 39, 40 California Mental Health Planning Council’s Performance Indicators for Evaluating the Mental Health System, January 2010. 22 7. Domain: System Structure This domain includes indicators of the mental health system workforce, and the type and quality of programs/services. Processes 7.1 Indicator: Workforce Composition (Revision of CMHPC Indicator #4541) Rationale for Inclusion: This indicator addresses the extent to which the mental health system workforce is appropriately configure to serve the diverse populations of county mental health systems. Potential Measure(s): Demographic profile; Staff to Consumer ratio; Consumer/family member employment (i.e., number, FTE, % of workforce) Potential Data Source(s): Cultural competence plans; Primary data collection (e.g., surveys, interviews, or focus groups; proposed additional data collection) 7.2 Indicator: Evidence-Based/Best Practice Programs and Services42 (Proposed Indicator) Rationale for Inclusion: Care or services that are implemented based on the best available evidence will lead to improved client outcomes. This indicator will center on whether county/regional/statewide mental health services and supports adhere to best practice criteria established through scientific evidence and/or expert consensus. Potential Measure(s): Existence of best practice core programs; Fidelity of best practices to established models; Receipt of best practices services/supports among consumers/families Potential Data Source(s): Primary data collection (e.g., surveys, interviews, or focus groups; proposed additional data collection) Outcomes 7.3 Indicator: Cultural Appropriateness of Services43 (CMHPC Indicator #2344) Rationale for Inclusion: This indicator addresses the extent to which the care or service is configured to best address the diverse cultures served by county mental health systems. Potential Measure(s): Client and family perceptions of cultural appropriateness Potential Data Source(s): Consumer Satisfaction Surveys – YSS, YSS-F, Adult, and Older Adult 41 California Mental Health Planning Council’s Performance Indicators for Evaluating the Mental Health System, January 2010. 42Anthony, W., Rogers, E., Farkas, M. (2003). Research on evidence-based practices: future directions in an era of recovery. Community Mental Health Journal, 39, 101-114 43Thomas, D. (2002). Evaluating the cultural appropriateness of service delivery in multi-ethnic communities. Evaluation Journal of Australasia, 2, 50-56. 44California Mental Health Planning Council’s Performance Indicators for Evaluating the Mental Health System, January 2010. 23 7.4 Indicator: Recovery, Wellness, and Resilience Orientation45,46 (Proposed Indicator) Rationale for Inclusion: Recovery, wellness, and resilience orientation is a core value of the MHSA initiative. The recovery process generally includes learning ways to manage mental illness, but also involves learning attitudes and skills about living, learning, working, having meaningful relationships, a place in the community and connection to the world. This indicator is focused on the extent to which county mental health systems are structured to provide guidance and support to consumers and families in their transition from living with mental illness as the most important part of their lives to being only a part of who they are. Potential Measure(s): Consumer/family member/staff perceptions of recovery orientation of system and services Potential Data Source(s): Recovery Oriented Systems Indicators Measure (ROSI; proposed additional data collection); Developing Recovery Enhancing Environments Measure (DREEM; proposed additional data collection) Potential Implications of Indicators for MHSOAC, Counties, Consumers & Families The revised indicator set presented in this report is tentative. Using this tentative set, we offer some conversation topics that might be fueled by the indicator set or ways that indicators might be brought into ongoing conversations about the MHSA. In other words, we propose a handful of ways the priority indicators can work for the greater MHSA community, whether in programming, service, or planning efforts. The final set will depend on data quality and reliability, which will be examined in subsequent reports (e.g., Deliverable 2C). Should existing data (already collected across counties) be complete and appropriate, we expect that the indicators detailed in this report will hold several implications for MHSOAC, counties, consumers and families, including the following topics. The questions included are by no means exhaustive but rather are examples of those that might be answered using priority indicators. Evaluating Mental Health System Processes and Outcomes Given the multiple ways that MHSA is designed to shape the mental health system (through consumer access, performance, and structure), system-level indicators reflect diverse points of MHSA impact. The indicators can offer broad insights into system processes (e.g., appropriateness of care) and relevant outcomes (e.g., consumers’ aggregated experiences). As illustration, process questions – those about how consumers receive care and how care is implemented –might address the following points. Also, when considered in light of outcome indicators, such process questions may suggest links between county mental health system processes and consumer or system level outcomes. 45 Statutory outcome: Implement Recovery Vision (WIC 5813.5(d)) 46Anthony, W. (2000). A recovery-oriented service system: setting some system level standards. Psychiatry Rehabilitation, 24, 159-169. 24 Discussion Points Relevant Indicators Processes  To what extent have MHSA interventions addressed demographic (gender, language, income, age) disparities to consumers’ MHSA 5.1, 5.2, 7.1 service use?  What have been the rates of hospital visits, involuntary care, and 24- 6.2, 6.3, 6.4, 6.5 hour care since the MHSA was established?  What best practices for consumer care are most prevalent across counties, and to what extent are supports available to sustain such 7.1 practices? Outcomes  What is the quality of care and services according to consumers and 6.8 their families?  How can various cultural groups be most appropriately served? 7.3 Processes and outcomes  How long do consumers use supports after a mental health episode or 5.4, 5.5, 6.5, 6.8, 7.3 psychiatric hospitalization? Improved monitoring, effectiveness, transparency Among consumer outcomes, the indicator set might more fully explain how consumers navigate available mental health interventions, the provisions made for consumers, and service quality. Example questions to begin conversation about monitoring at the consumer level include the following: Discussion Points Relevant Indicators  To what extent do consumers have access to independent living or 2.1 supported living opportunities?  For children and TAY, what roles do schools play in mental health 1.1 support? 25 Broader questions that can inform what we know about system monitoring, effectiveness, and transparency at the system level are as follows: Discussion Points Relevant Indicators  Are mental health services being accessed as needed, within and across 5.1, 5.2, 5.3, 5.4 target consumer populations or county mental health systems?  To what extent are county mental health systems meeting the values and expectations of consumers and families, communities, providers and the 5.5, 6.4, 6.8, 7.3, 7.4 MHSA initiative overall?  What is that makeup of the mental health system workforce, and the 7.1, 7.2 type and quality of programs/services they provide? Reduce burden/cost We propose that the indicator set will add positive perspectives to the ongoing discussion of mental health’s social costs. Some questions are beyond the indicator set, suggesting that indicators evoke peripheral discussions that are important to maintaining MHSA performance and quality. Particularly, the indicators might inform the following:  What is the cost saved/absorbed by hospitals that consumers use for emergency mental health intervention?  What are the social ramifications of reducing homelessness through mental health intervention? At the system-level, it is equally important to consider such questions as:  Are services reaching those most in need?  Are recovery, wellness, and resilience being promoted? Decision making and feedback loop for continuous improvement Priority indicator findings will drive important decisions made about the MHSA, its consumers, and its systems. At the consumer level, priority indicators can help make the case for more, fewer, or different types of programs in particular domains, for example. At the system-level, the MHSA administration might use findings formatively, meaning to shape existing practices in the mental health system based on what information priority indicators provide. This could be achieved by redistributing funds to areas that require more support, facilitating the revision of programs and supports that fail to meet expected performance levels, or modifying models to capitalize on a set of best practices that have been shown to consistently produce desired results. At both levels, any information gained from priority indicators is an ongoing assessment of the state of mental health and related services to maintain the highest quality of life possible for consumers and sound system performance. 26 Conclusions Per our objectives, the evaluation team inventoried proposed indicators, provided fuller explanations of these indicators, and determined if surveys currently used across counties could provide variables to represent the indicators. The team also proposed what indicators were needed to capture more areas of MHSA impact. All indicators, their domains and data sources were displayed in a template that could prove useful in future reports. Cumulatively, this report was a step toward refining the priority indicator set. The team must assess if the data related to target variables is complete and consistent enough to develop priority indicators. Although this report reflects one framework – what we believe to be an important blueprint to navigate data in search of appropriate variables – any findings that suggest data is substandard will return us to the search. In a forthcoming report, we document the results of our data review and provide systematic methods to calculate and display priority indicators for standardized reports. Next Steps Moving forward, the evaluation team will request access to data associated with each of the data sources featured in the tables. With appropriate state and institutional approvals, the evaluation team will review databases with an eye to data completeness and the regularity with which it is collected. This will inform which data sources are stable and useful in the process of creating priority indicators. The team will invite feedback from key stakeholders, including those within the field whose expertise in research, statistics, and programming can clarify our process toward developing viable measures and indicators. The feedback process, described early in this report, will simultaneously involve consumers, their families, and any interested person with comments on the utility of priority indicators in evaluating MHSA impact. Through this process, experts in the field and key stakeholders will contribute to the development of a plan for appropriate and rigorous analysis of all priority indicators, including the examination of MHSA impact on specific populations (e.g., age groups, race/ethnicity, economic/living situation, language, etc) and in the context of each counties’ unique characteristics (e.g., demographics, funding, economic factors, etc). The feedback process, which is still being developed in collaboration with mental health organization leaders, will be detailed in a subsequent report within the process description. 27 Appendix 1 28 Appendix 2 Glossary Criteria A set of standards on which decisions are made Domain An overarching category within which related items are grouped Indicator A gauge or measure of a particular trend or condition Outcome Change brought about by a guiding course of action Process The breadth of actions taken to achieve an outcome or set of outcomes 29 Appendix 3 Data Sources Reviewed Data Collection & Reporting System for FSP (DCR)  Key Event Tracking (KET)  Partnership Assessment Form (PAF)  Quarterly Assessment Forms (3M) Performance Outcomes & Quality Improvement (POQI)  Youth Services Survey (YSS)  Youth Services Survey for Families (YSS-F)  Adult Survey  Older Adult Survey Client Services and Information System (CSI) County Reports  Revenue and Expenditure Reports (R&E)  Annual Updates  Quarterly Progress Goals and Report (includes CSS Exhibit 6) Annual Report on Involuntary Detentions Cultural Competence Plans 30