Mhsa Contract Deliverable 2d

Mental Health Services Act Evaluation: Compiling Data to Produce All Priority Indicators Contract Deliverable 2D, Phase II UCLA Center for Healthier Children, Youth and Families EMT Associates, Inc. Initially submitted on September 30, 2011 Revised version submitted on October 20, 2011 Final version submitted on November 2, 2011 The following report was funded by the Mental Health Services Oversight and Accountability Commission. Focus The Mental Health Services Act (MHSA) evaluation team was charged with developing templates and reports on statewide and county specific data that would improve understanding of how the MHSA impacted consumers. Per contract language, the team will: Design and complete statistical analyses and reports that measure impact of MHSA at individual and system levels on indicators specified in the Matrix of the California Public Mental Health System Prioritized Performance Indicators at the state and county levels. Draft templates, documentation of analysis, and initial statewide reports will be circulated to key stakeholders and made available to the public for input by posting on the web and making a hard copy available upon request. Individual client outcomes for full service partnerships (FSPs) by age group must be addressed for each domain (education/employment, homelessness/ housing and justice involvement) as specified. Note: this impact analysis at the individual level is limited to available data (i.e., a small segment of public mental health clients, full services partners, is reflected in this data). Mental Health system performance must address family/client/youth perception of well- being, demographics of FSP population, FSP access to primary care, penetration rate and changes in admissions for the entire public community mental health population, involuntary care, and annual numbers served through [Community Services and Supports programs] CSS. The evaluation team submits the following report, which incorporates stakeholder insights, in fulfillment of this charge. Stakeholder Feedback As noted in the contract language, input from key stakeholders and mental health service advocates is key to developing final reports. To this end, all reports (i.e., contract deliverables 2A and 2C) that the evaluation team submits to the Mental Health Services Oversight and Accountability Committee (MHSOAC) are considered drafts until such input is received and incorporated, to a reasonable extent, into subsequent reports. The evaluation team enlisted feedback from a wide range of stakeholders from July 29, 2011 through August 31, 2011 to create a more comprehensive, accurate report. Stakeholders received an e-mail announcing the availability of deliverables on the MHSOAC1 and UCLA2 web sites. A call for feedback and an illustration of how the evaluation team would develop deliverables using stakeholder input were embedded within report introductions to clarify stakeholders’ roles in report creation and increase transparency about this process (see Illustration 1). 1 http://www.mhsoac.ca.gov/Announcements/announcements.aspx 2 http://healthychild.ucla.edu/MHSA_evaluation.asp 1 Illustration 1 Steps Leading to Statewide and County Specific Data Reports Initial Statewide Evaluation 12/31/12 9/30/12 6/30/12 County reports incorporate county context Three written County specific and statewide reports, on all priority indicators Step 7 (demographics, funding, etc) Revised written report from STEP 5 in response to stakeholder input Step 6 3/31/12 Initial draft written report submitted including data for all priority indicators at the statewide level for the most recent one year period available Step 5 12/31/11 Revised deliverable from STEP 2 in response to stakeholder input obtained. Step 4 s p e 9/30/11 St Revised deliverable from STEP 1 in response to stakeholder input Step 3 9/30/11 Draft written documentation of the process for compiling the data to produce the reports for all priority indicators Step 2 Stakeholder/Consumer Feedback 6/30/11 Draft proposed standardized template for reporting all priority indicators Step 1 Currently Completed 6/30/11 Due Dates 2 A complete account of organizations whose representatives responded to correspondence is included at the end of this report (see Appendix A). Webinars were also conducted for MHSA stakeholders and the California Mental Health Directors Association (CMHDA) Indicators, Data, Evaluation and Accountability (IDEA) Ad- Hoc Committee. Webinars described the reports’ purposes, input needs, and feedback process. Webinars were not designed to collect feedback; rather they were intended to provide a synopsis of deliverables. Given the number of webinar participants, it was not feasible to account for all stakeholders’ comments. The evaluation team requested that all feedback was ultimately written and shared with the evaluation team through e-mail correspondence. The evaluation team encouraged all stakeholders to respond to points of interest in the reports as well as use the accompanying guidance document to think critically about questions posed by the evaluation team. Groups were invited to use their existing internal processes for reviewing and responding to mental health-related reports; the evaluation team did not impose any review protocol. The team requested that feedback be specific (e.g., noting page numbers, specific priority indicators, or specific measures), rather than a set of general comments, to optimize feedback use. Stakeholders largely obliged this request. The nature of responses ranged greatly. A table illustrating the types of feedback garnered by each indicator and measure is located in Appendix B. Among stakeholder input were direct responses to guidance questions, feedback that provided historical context for data issues, concern about the accuracy of particular indicators given data quality, and potential ramifications of a university group assessing mental health service consumer outcomes instead of the consumers themselves. The evaluation team anticipated feedback diversity – particularly recommendations that would be at odds with each other – and devoted ample time to negotiating what could and could not be addressed given the available time and data with which we were provided to conduct the evaluation. The team articulated this point early in the reports in the service of transparency and to ground expectations. Overview This draft report proposes processes for creating useful measures of priority indicators that can be used to monitor how the Mental Health Services Act (MHSA) impacts consumer outcomes and mental health service system performance statewide and at the county level. The priority indicators were proposed in a preceding report (Templates for Reporting Priority Indicators, Deliverable 2A). These indicators were intended to identify measurable Community Services and Supports (CSS) program outcomes, including consumer outcomes and measurable characteristics of mental health service system capacity and performance. This report details how each priority indicator can be represented using survey and service description (e.g., outputs) data already collected within each county. Guided by stakeholder insights, the report identifies data sources and items (variables) relevant to the evaluation; data limitations per stakeholder feedback; and methods of combining data into more adequate indicators where appropriate. Criteria for testing individual or multiple-item measures of indicators are also identified. These quality tests will be applied to cull and refine proposed 3 measures once access to the necessary data is acquired. No analyses are included in this report; rather data is organized in preparation for analyses that will take place subsequent to this report. The report is organized by the following topics. Brief discussion of indicator development prior to this project Profile of the data sources used for this project – mental health-related surveys and reports that are regularly submitted by California counties Discussion of the criteria used to select, review, and refine measures Explication of the calculations proposed to create priority indicator measures Notes on examining data for quality and completeness Description of consumer stakeholder group roles in refining measures and calculations Two templates are presented that summarize MHSA domains, priority indicators (i.e., consumer and system level), measures, relevant data sources and items, and necessary calculations. Where data quality concerns can be anticipated, or gaps are evident, recommendations for new data collection to attain measurement goals are included. Stakeholder feedback about data quality and measurement feasibility is incorporated throughout. The report concludes with next steps in refining the measure and indicators to ensure accurate and comprehensive monitoring of consumer outcomes and mental health service system performance. Background The California Mental Health Planning Council (referred to throughout as The Planning Council) proposed a set of priority indicators to assess the impact of the MHSA on consumers and county service systems statewide. Council members designed individual-level priority indicators to create greater clarity about consumers’ dispositions (e.g., employment, education, housing, justice involvement) following interventions coordinated through the MHSA. Similarly, council members proposed that system-level priority indicators (related to consumer access, agency performance, agency structure) would explain how operations changed or were enhanced (if at all) by the Act. These indicators were ultimately adopted by the MHSOAC. As part of ongoing efforts to define priority indicators and identify how these indicators might be measured using data currently collected by counties, the evaluation team reviewed existing datasets to identify appropriate data sources and items. This process began with the set of indicators identified in Templates for Reporting Priority Indicators, Deliverable 2A. This document defines and provides the rationale for indicators recommended for monitoring county use of MHSA funds, and performance of MHSA initiatives. In this report, the evaluation team outlines a data extraction and measurement process to operationalize conceptually complete indicators of consumer outcomes and system performance related to the MHSA. 4 Objectives The evaluation team conducted a search of available data with two goals: to 1) locate variables, relevant to each priority indicator, that are regularly collected; and 2) outline short protocols for converting existing data into priority indicators. The present report briefly documents this process, in which we provide guidelines about how to create relevant measures using existing data to the extent possible. Throughout the report we note challenges in calculation, and areas in which new data might need to be collected. Process for Reviewing Available Data We reviewed several data dictionaries and instruments (e.g., surveys, forms) associated with their respective existing datasets or reports. The datasets and reports, listed below, reflect information that is regularly collected across counties at present. In the absence of access to raw data, we closely examined the qualities (e.g., item wording, response options, intended response population) of each item (variable), sorting which would be most appropriate to represent each priority indicator. Client & Service Information (CSI) The CSI system is a repository of county, client (e.g., age, gender, preferred language, education, employment status, living arrangement, etc.), and service information (number and length of service contact). The data is collected from all consumers who receive mental health services, including consumers involved in Full Service Partnerships. Data Collection and Reporting (DCR) System The DCR system houses data for consumers who are served through Full Service Partnership programs. Data from assessments – the Partnership Assessment Form (PAF), Key Event Tracking (KET), and Quarterly Assessment (3M) – are collected for consumers in specific age categories. The PAF reflects consumer history and baseline information, including consumer education and/or employment, housing situation, legal issues, health status, and substance use. The KET reflects any important changes in the consumer’s life such as housing, education and/or employment, and legal issues during service receipt. The 3M collects follow-up information on key areas such as education, health status, substance use, and legal issues each quarter. Consumer Perception Surveys (YSS for youth responses, YSS-F for family responses) 5 These consumer surveys are instrument sets customized for consumer groups (e.g., youth, adults, and older adults). Instruments are composed of widely validated tools such as the Child Behavior Checklist, Youth Self Report, and Restrictiveness of Living Environment Scale for youth assessment; the Global Assessment of Functioning, Behavior and Symptom Identification Scale, and the California Quality of Life for adults; and the Brief Symptom Inventory, Senior Outcomes Checklist 10, and Index of Independent Activities of Daily Living for older adults. The data, designed to inform treatment planning and service management, is collected from individuals with “serious, persistent” mental illness, have received services for 60 days or more, and are not categorized as “medication only.” At minimum, data is customarily collected at intake, annually, and at discharge, however this schedule has changed in recent years. Findings are reported to the Department of Mental Health semi-annually. Mental Health Statistics Improvement Program (MHSIP for adult responses and MHSIP for older adult responses) The MHSIP consumer surveys are designed to assess client satisfaction, service accessibility, quality, and outcomes. Adult consumers and older adult consumers respond to the survey questions using a 5-point scale of agreement (e.g., 5 = strongly agree and 1 = strongly disagree).3 Quarterly Reports (Exhibit 6)4 Quarterly reports, including Exhibit 6, reflect consumer counts—the number of people who were targeted and receive MHSA services. Counts are aggregated from different consumer pools, including Outreach and Engagement, and Community Services and Supports (CSS), among others. Service types and demographics are not included in reports. Data is reported quarterly then compiled into annual reports. Involuntary Services (Jail Services, Conservatorships, Involuntary Detentions, Seclusion and Restraint) The California Department of Mental Health tracks several involuntary services by County and across the State, including: 72-hour evaluation and treatment (Adults and Children), 14-day intensive treatment (including suicidal treatment), 30-day intensive treatment, 180-day post certification treatment, temporary conservatorships, and permanent conservatorships. Calculation of Measures Meaningful and useful measures must be carefully conceptualized, designed, and constructed. Doing so facilitates how variables are combined or calculated then interpreted. The indicators included in the measurement template proposed in Templates for Reporting Priority Indicators, Deliverable 2A were expressed as absolute figures (e.g., counts, frequencies) or ratios (e.g., 3 Key informants have informed the evaluation team that the MHSIP surveys are sometimes also referred to as “POQI-Adult” and “POQI-Older Adult.” 4 In the previous deliverable (2A), we noted that key informants preferred the use of Annual Updates instead of Quarterly Reports although the state of Annual Updates was uncertain due to Assembly Bill 100. At the time of this report, the data dictionary associated with Annual Updates was unavailable, limiting what we are able to report here. The recommendation is described more fully in the discussion. 6 normalized data). These two types of indicators have distinct requirements for the data used to operationalize them, and implications for stakeholders who would use indicators to drive quality improvement. The following table outlines the implications for these two types of indicators. Absolute Indicators Counties often report data in terms of absolute figures, which might be expressed as the frequency (count) of a specific event or an indication of consumer status (e.g., attendance in school, employment status, housing status, receipt of service, type of service, etc.). Absolute figures can provide 1) a description of a services or outcomes at points in time, and 2) the basis for critical analyses of differences across consumer groups, across time, and attribution of impact. Absolute figures are essential to any assessment of the parameters (e.g., carrying capacity, limits, or sustainability) of mental health systems and services, and the impact on consumers and families. For example, at the mental health system level, the total number of consumers receiving 24-hour care provides the possibility to consider service levels relative to a county’s overall resource capacity. Similarly, absolute measures of consumer outcomes, such as the number of consumers attending school, will allow for comparison to other populations or mental health systems. Absolute figures can provide an important perspective of the capacity, performance, and impact of mental health systems. When information regarding local MHSA context is lacking, absolute indicators can also be useful for stakeholders trying to understand the relative magnitude of county services and impacts, or reasons for prioritizing efforts. For example, identifying the 10 counties with fewest consumer arrests would require absolute figures, whereas ratios (e.g., normalized data) are more useful when making comparisons between counties. Absolute measures of MHSA performance and impact provide for: consistent tracking; data aggregation to key levels (e.g., county, state); and ability to form additional ratios other than those included in the priority indicator template. Ratio Indicators Ratios relate two absolute figures to each other and provide context to both. For example, the efficiency of Community Services and Supports (CSS) can be expressed in terms of the number of consumers served through CSS relative to those who were targeted for service. Alternatively, to shift focus to the impact of the CSS program, the number of consumers served through CSS could be compared to all consumers receiving mental health services. Ratio indicators serve to: relate two absolute figures to each other; 7 make relationships visible and interpretable by a broad audience; and, provide for comparison of different scales of operation relative to a specific service (e.g., number of incarcerated consumers per individual served). Ratios may also be particularly useful for comparing counties or regions. Absolute figures sometimes do not provide the context in which performance or impact may be best understood. This may be particularly true among the diverse counties and regions of California. For example, the magnitude of a service will not always correlate with the size of the county in which it was administered. As illustration, it may be factually correct that county A served twice as many consumers as county B, but this would be misleading if county B were a quarter that size and twice as efficient in the administration of their services. For some indicators, an absolute figure may be the most meaningful piece of information, but for others additional context is needed to accurately understand the implications of figures. Single vs. Multiple Item Measures To produce adequately robust measures, we applied a process to construct single or multiple- item measures as appropriate. In some cases a single data item is adequate to capture an indicator, such as when its meaning is clear and it has adequate variance and precision. For example, the rating of children or TAY consumers’ school attendance in the past year may only require a single item (e.g., ATTENDANCEPAST12 – PAF). But often, single items are not adequate, or can be improved by combining several data items that express different empirical facets of an indicator. For example, to construct a robust and accurate measure of consumer or family perceptions of improvement in functioning, multiple survey (e.g., MHSIP) items are necessary to ensure several facets of this measure are assessed. When carefully constructed, multi-item indicators can improve the reliability, validity and variance characteristics of a resulting measure. Orientation to the Templates The subsequent templates detail how the proposed priority indicators could be constructed. The templates are divided at the individual (Template 1) and system (Template 2) levels, and are intended to present options for constructing measures using existing data or, where existing data was not sufficient, options for future data collection are proposed.5 Although the Planning Council and MHSOAC envisioned consumer outcomes to be measured across Full Service Partnership consumers and system outcomes to be measured across all mental health service consumers (see Appendix C), data sources in both templates reflect possibilities for outcome calculations across all mental health services consumers (via the Consumer Services and 5 Contractually, the evaluation team is responsible for coordinating data collection related to the Participatory Research component of this project (Phase 3, Deliverables 2.a.1 and 2.b.2.). Beyond this requirement, the team only offers suggestions about where new data collection can create more complete mental health service datasets. 8 Information [CSI] system) as well as persons enrolled in Full Service Partnerships (via the Data Collection and Reporting [DCR] system). The columns from left to right detail measurement domains, performance indicators, possible calculation of measures based on what is available (recommended and alternative), the databases or reports from which relevant items can be drawn, and the specific items within each dataset or report which may be used to construct measures. Not all data sources provide accurate indicator measurement; however, we list these sources and their associated variables to generate discussion about how existing data might be re-envisioned or manipulated to represent a priority indicator. How to Read the Templates. The Measure Calculation(s), Data Source(s), and Dimension columns list all proposed methods for calculating or estimating a measure. No one measure has been selected as of this report. Only after the evaluation team gains access to and reviews existing data will measures be selected. These columns should be considered an inventory of possible measures. In the absence of appropriate data sources, or as supplement to existing sources, we recommend new data collection. For example, in Template 1 the Average Attendance – score per year indicator could be measured using the Recommended Ratio, Alternative Estimate 1, or Alternative Estimate 2. The information that the evaluation team needs to calculate any of these can be located in the DCR, YSS, YSS-F, may require new data collection, or a combination thereof. The templates do not reflect a one-to-one relationship between measure calculations and data sources. To make the most efficient use of existing and proposed additional data and data collection processes and provide flexible performance measurement options at the state and county levels, we present “recommended” and “alternative” measures of priority indicators in the Measure Calculation(s) column. Recommended measures are those that would most accurately reflect indicators, while attempting to take advantage of existing data systems. We consider these calculations optimal. Alternative calculations, considered as substitutes or supplements to the recommended calculation, are based on variables that currently exist within at least one of the Department of Mental Health datasets, and with manipulation can provide an approximate measure of the desired performance indicator. In the event that existing data can only provide an approximate measure of an indicator, additional data collection is proposed. The following templates detail how measures of each priority indicator can be constructed from existing or proposed additional data collection. To the extent possible, we have highlighted stakeholder input (blue fields) throughout the templates. More detailed revisions, such as suggested revisions to ratio denominators, have not been highlighted but are described in stakeholder feedback in the report’s Measurement Detail (pp. 23-32). 9 Template 1. Process for Compiling Data and Calculating Priority Indicators: Individual-level (Consumer) Outcomes for Full Service Partnerships6 Blue fields indicate stakeholder feedback. Domain Indicator Measure Calculation(s) Data Source(s) Dimension – Potential Items 1. Education/ 1.1 Average Recommended Ratio Proposed new data Children and TAY Employment attendance – score per Numerator: Number of days collection Number of days absent year at school during a Total number of school year days at consumers’ school consumer’s school year* / YSS Children (Youth Report) (Children, TAY) Denominator: Number of Current and previous expulsions – LES12EXPSUS, days during a consumer’s LES12PSTEXPSUS, MOR12EXPSUS, MOR12PSTEXPSUS school year YSS-F Children (Parent or Guardian Report) Alternative Estimate 1 Current and previous expulsions – LES12EXPSUS, Number of days during LES12PSTEXPSUS, MOR12EXPSUS, MOR12PSTEXPSUS consumer’s school year* – DCR Children Number of expulsions or Attendance Rate Estimate – ATTENDANCECURR (PAF) (3M) suspensions during the year ATTENDANCEPAST12 (PAF) TAY Alternative Estimate 2 Attendance Rate Estimate – ATTENDANCECURR (PAF) (3M) Attendance rate estimate for ATTENDANCEPAST12 (PAF) three quarters 1.2 Proportion Recommended Ratio 1 DCR TAY participating in paid Numerator: Number of Paid employment – Current_In-HouseAvgHrWeek (PAF), and unpaid employed consumers Current_OtherEmploymentAvgHrWeek (PAF), employment (reporting work hours) / Current_SupportedAvgHrWeek (PAF), Denominator: Total number Current_TransitionalAvgHrWeek (PAF), Past12_Competitive (PAF), (TAY over 18, adults, of consumers eligible for Past12_In-House (PAF), Past12_In-HouseAvgHrWeek (PAF), and older adults) employment (over 18 years Past12_OtherEmployment (PAF), old) Past12_OtherEmploymentAvgHrWeek (PAF), Past12_Supported (PAF), Past12_SupportedAvgHrWeek (PAF), Past12_Transitional Recommended Ratio 2 (PAF), Past12_TransitionslavgHrWeek (PAF) Numerator: Number of Unpaid employment – Current_Non-paidAvgHrWeek (PAF), employed consumers Past12_Non-paid (PAF), Past 12_Non-paidAvgHrWeek (PAF) 6 Data sources that reflect all mental health service consumers (e.g., CSI) have been added in the event that broader information than what is learned about Full Service Partnership consumers is sought. *Asterisk indicates information from new data collection or a data source not yet identified. 10 Domain Indicator Measure Calculation(s) Data Source(s) Dimension – Potential Items receiving pay for work / Adults Denominator: Total number Paid employment – Current_In-HouseAvgHrWeek (PAF), of consumers eligible for Current_OtherEmploymentAvgHrWeek (PAF), employment (over 18 years Current_SupportedAvgHrWeek (PAF), old) Current_TransitionalAvgHrWeek (PAF), Past12_Competitive (PAF), Past12_In-House (PAF), Past12_In-HouseAvgHrWeek (PAF), Recommended Ratio 3 Past12_OtherEmployment (PAF), Numerator: Number of Past12_OtherEmploymentAvgHrWeek (PAF), Past12_Supported employed consumers not (PAF), Past12_SupportedAvgHrWeek (PAF), Past12_Transitional receiving pay for work / (PAF), Past12_TransitionslavgHrWeek (PAF) Denominator: Total number Unpaid employment – Current_Non-paidAvgHrWeek (PAF), of consumers eligible for Past12_Non-paid (PAF), Past 12_Non-paidAvgHrWeek (PAF) employment (over 18 years old) Recommended Count 1 Older Adults Number of days employed Paid employment – Current_In-HouseAvgHrWeek (PAF), Current_OtherEmploymentAvgHrWeek (PAF), Current_SupportedAvgHrWeek (PAF), Current_TransitionalAvgHrWeek (PAF), Past12_Competitive (PAF), Past12_In-House (PAF), Past12_In-HouseAvgHrWeek (PAF), Past12_OtherEmployment (PAF), Past12_OtherEmploymentAvgHrWeek (PAF), Past12_Supported (PAF), Past12_SupportedAvgHrWeek (PAF), Past12_Transitional (PAF), Past12_TransitionslavgHrWeek (PAF) Unpaid employment – Current_Non-paidAvgHrWeek (PAF), Past12_Non-paid (PAF), Past 12_Non-paidAvgHrWeek (PAF) 2. Homelessness/ 2.1 Housing situation/ Recommended Ratios 1 DCR Children Housing Index- score Numerator: Number of days Current housing situation – CURRENT (PAF) (KET) that children or TAY (under Previous housing situations (week, month) – (Children, TAY, 18) live in a family home EMERGENCYSHELTER-PASTTWELVEDAYS, adults, and older annually/ Denominator: 365 EMERGENCYSHELTER_PASTTWELVEOCCURENCES, adults) days EMERGENCYSHELTER_PRIORTWELVE (PAF), HOMELESS_PASTTWELVEDAYS (PAF), Recommended Ratios 2 HOMELESS_PASTTWELVEOCCURENCES (PAF), Numerator: Number of days YESTERDAY (PAF) that children or TAY (under DCR TAY 18) live in a foster home Current housing situation – CURRENT (PAF) (KET) annually/ Denominator: 365 Previous housing situations (week, month) – days EMERGENCYSHELTER-PASTTWELVEDAYS, EMERGENCYSHELTER_PASTTWELVEOCCURENCES, EMERGENCYSHELTER_PRIORTWELVE (PAF), HOMELESS_PASTTWELVEDAYS (PAF), 11 Domain Indicator Measure Calculation(s) Data Source(s) Dimension – Potential Items Recommended Ratios 3 HOMELESS_PASTTWELVEOCCURENCES (PAF), Numerator: Number of days YESTERDAY (PAF) TAY, adults, or older adults are homeless/ Denominator: 365 days Recommended Ratios 4 Adults Numerator: Number of TAY Current housing situation – CURRENT (PAF) (KET) or adults with independent Previous housing situations (week, month) – residential statuses/ EMERGENCYSHELTER-PASTTWELVEDAYS, Denominator: Total number EMERGENCYSHELTER_PASTTWELVEOCCURENCES, of FSP, TAY, and FSP adults EMERGENCYSHELTER_PRIORTWELVE (PAF), HOMELESS_PASTTWELVEDAYS (PAF), HOMELESS_PASTTWELVEOCCURENCES (PAF), Recommended Ratios 5 YESTERDAY (PAF) Number of TAY, adults, and 2. Homelessness/ 2.1 Housing situation/ older adults who are not Older Adults Housing (cont’d) Index- score homeless/ Denominator: Current housing situation – CURRENT (PAF) (KET) Total number of FSP TAY, Previous housing situations (week, month) – (Children, TAY, FSP adults, or FSP older EMERGENCYSHELTER-PASTTWELVEDAYS, adults, and older adults EMERGENCYSHELTER_PASTTWELVEOCCURENCES, adults) EMERGENCYSHELTER_PRIORTWELVE (PAF), HOMELESS_PASTTWELVEDAYS (PAF), HOMELESS_PASTTWELVEOCCURENCES (PAF), YESTERDAY (PAF) Recommended Count 1 YSS Children Number of days in housing Specific housing7 – FAMILYMEM, FOSTERHM, THERAPEUTIC, SHELTER, HOMESHELT, GROUPHM, RESIDENTX, HOSPITAL, JAIL, CORRECTIONS, HOMELESS, LIVEOTHER, *WHERE (follow-up to LIVEOTHER) YSS TAY Specific housing – FAMILYMEM, FOSTERHM, THERAPEUTIC, SHELTER, HOMESHELT, GROUPHM, RESIDENTX, HOSPITAL, JAIL, CORRECTIONS, HOMELESS, LIVEOTHER, *WHERE (follow-up to LIVEOTHER) 7 Stakeholders shared that without verification of parents’ housing statuses, children who are identified as living with parents could be homeless if parents are of that status. For this reason, the PARENT variable has been removed from potential items that might be used to assess Homelessness/Housing. 12 Domain Indicator Measure Calculation(s) Data Source(s) Dimension – Potential Items CSI8 TAY PATIENT STATUS CODE (consumer’s housing if recently discharged) Adult PATIENT STATUS CODE (consumer’s housing if recently discharged) Older Adult PATIENT STATUS CODE (consumer’s housing if recently discharged) 3. Justice 3.1 Justice Recommended Ratio 1 DCR Children Involvement Involvement Numerator: Number of Recent arrest – DATEARRESTED (PAF) consumer arrests annually / TAY (Children, TAY, Denominator: 365 days Previous arrests (year) – ARRESTPRIOR12 (PAF adults, and older Older Adults adults) Recommended Ratio 2 – by Recent arrest – DATEARRESTED (PAF) age group YSS TAY Numerator: Number of child, Previous arrests (year) – MOR12AREST, MOR12PSTAREST TAY, adult, or older adult arrests/ Denominator: Total YSS-F Children (Parent or guardian response) number of FSP children, FSP Previous arrests (year) – LES12AREST, LES12PSTAREST TAY, FSP adults, or FSP CSI TAY older adults P-08.0 CONSERVATORSHIP/ COURT STATUS (if consumer is a ward of the court) Recommended Ratio 3 – by age group Older Adults Numerator: Number of child, S-20.0 LEGAL CLASS ADMISSION (if consumer has been TAY, adult, or older adult admitted to acute 24-hour mental health services) arrests / Denominator: County estimate of all MHSIP Adult Adults children, all TAY, all adults, Recent arrests – ARREST or all older adults Previous arrests – LES12AREST, LES12PSTAREST Recommended Count 1 MHSIP Older adult Older Adults Number of incarcerations* Recent arrests – ARREST Previous arrests – LES12AREST, LES12PSTAREST Proposed new data Children, TAY, Adults, Older Adults collection Number of incarcerations 8 Key informants have expressed concerns that CSI data designed to capture all mental health service consumers is of questionable quality. 13 Domain Indicator Measure Calculation(s) Data Source(s) Dimension – Potential Items 4. Emergency 4.1 Emergency Recommended Ratio 1 CSI Children Care intervention for Numerator: Number of S-06.0 SERVICE FUNCTION (Identifies the specific type of mental health mental health episode- service received by the client within 24 Hour, Day, and/or episodes related hospitalizations Outpatient mode of service) annually/ Denominator: TAY (Children, TAY, Number of consumer visits S-06.0 SERVICE FUNCTION (Identifies the specific type of service adults, and older to the hospital for any reason received by the client within 24 Hour, Day, and/or Outpatient mode adults) annually of service) Adults Recommended Ratio 2 S-06.0 SERVICE FUNCTION (Identifies the specific type of service Numerator: Number of received by the client within 24 Hour, Day, and/or Outpatient mode emergency psychiatric of service) interventions / Denominator: Older Adults Number of consumer visits S-06.0 SERVICE FUNCTION (Identifies the specific type of service to a non-hospital intervention received by the client within 24 Hour, Day, and/or Outpatient mode center annually of service) Proposed new data Children, TAY, Adults, Older Adults collection Number of non-psychiatric hospital visits Number of visits to a non-hospital facility for mental health interventions 4.2 Emergency Recommended Ratio 1 CSI Children intervention for co- Numerator: Number of S-06.0 SERVICE FUNCTION (Identifies the specific type of service occurring physical consumer visits to the received by the client within 24 Hour, Day, and/or Outpatient mode injury hospital for physical injuries of service) or physical health disorders TAY that co-occur with mental S-06.0 SERVICE FUNCTION (Identifies the specific type of service health episodes/ received by the client within 24 Hour, Day, and/or Outpatient mode Denominator: Number of of service) consumer visits to the Adults hospital for any reason S-06.0 SERVICE FUNCTION (Identifies the specific type of service annually received by the client within 24 Hour, Day, and/or Outpatient mode of service) Older Adults S-06.0 SERVICE FUNCTION (Identifies the specific type of service received by the client within 24 Hour, Day, and/or Outpatient mode of service) 5. Social 5.1 Proportion who Recommended Ratio 1 Proposed new data Children, TAY, Adults, Older Adults Connections identify family Numerator: Number of collection Number of persons who are related to the consumer who the support9 family members the consumer identifies as supportive consumer identifies as 9 “Family” may or may not include caregivers depending on each consumer’s designation. 14 Domain Indicator Measure Calculation(s) Data Source(s) Dimension – Potential Items reliable supporters, or persons who are consistently present for the consumer*/ Denominator: Number of consumers 5.2 Proportion who Recommended Ratio 1 Proposed new data Children, TAY, Adults, Older Adults identify community Number of community (non- collection Number of persons who are not related to the consumer who the support family) members that the consumer identifies as supportive consumer identifies as Number of organizations that the consumer visits voluntarily and reliable supporters*/ regularly that the consumer identifies as providing appropriate and Denominator: Number of high quality services consumers Recommended Count 2 Number of mental health service organizations and other support services the consumer identifies as being a core resource when needed*/ Denominator: Number of consumers Template 2. Process for Compiling Data and Calculating Priority Indicators: System-Level Outcomes for All Mental Health Consumers10 Domain Indicator Calculation of Measure(s) Data Source(s) Dimension – Potential Items 6. Access 6.1 Demographic Recommended Descriptives CSI Age – C-03.0 Date of Birth Profile of Mean/median, mode, range, and Gender – C-05.0 Gender Consumers Served change over time for age, gender, Race/ethnicity – C-09.0 Ethnicity; C-10.0 Race language, race/ethnicity of consumer DCR Age – Date of Birth population (overall and FSP), in Gender – Gender comparison to State and County Race/ethnicity – Ethnicity_A; Ethnicity_B population demographics Alternate Descriptives 1 CSI Employment Status – P-03.0 Employment Status Mean, mode, range, and change over time for age, gender, race/ethnicity of DCR Income – Wages-Curr; Wages_Past12 individuals living below the poverty line or unemployed Alternate Descriptives 2 CSI Homelessness – P-09.0 Living Arrangement 10 Data sources that reflect Full Service Partnership consumers have been added in the event that specific knowledge about systems from this population is sought. 15 Domain Indicator Calculation of Measure(s) Data Source(s) Dimension – Potential Items Mean, mode, range, and change over DCR Homelessness – Homeless Yesterday11; Homeless_PastTwelveDays; time for age, gender, race/ethnicity of Homeless_PastTwelveOccurences; Homeless_PriorTwelve homeless 6.2 New Recommended Descriptives DCR Age – Age_Group Consumers by Mean, mode, range, and change over Gender – Gender Demographic time for age, gender, race/ethnicity of Race/ethnicity – Ethnicity_A; Ethnicity_B Profile new consumers (< 6 months) Length of Service – PartnershipDate Additional Descriptives CSI Age – Age_Group Mean, mode, range, and change over Gender – Gender time for age, gender, race/ethnicity of Race/ethnicity – Ethnicity_A; Ethnicity_B existing consumers (> 6 months, 1 Length of Service – S-15.0 Admission Date; S-16.0 From/Entry Date; S- year, > 1 year) 17.0 Through/Exit Date; S-18.0 Discharge Date DCR Age – Age_Group Gender – Gender Race/ethnicity – Ethnicity_A; Ethnicity_B Length of Service – PartnershipDate 6.3 High Need Recommended Count 1 DCR Homeless –Homeless Yesterday6; Homeless_PastTwelveDays; Consumers Served Total homeless - FSP consumers Homeless_PastTwelveOccurences; Homeless_PriorTwelve served12 Recommended Count 2 CSI Homeless – P-09.0 LIVING ARRANGEMENT Total homeless - all consumers served Recommended Count 3 DCR Unemployed – Current_Unemployed Total unemployment - FSP consumers served Recommended Count 4 CSI Unemployed – P-03.0 EMPLOYMENT STATUS Total unemployment - all consumers served Recommended Count 5 DCR Arrests – ArrestPast12 Total consumers with justice involvement served Recommended Count 6 CSI Hospitalization – S-06.0 SERVICE FUNCTION Total consumers with multiple psychiatric hospitalizations served 6.4 Access to Recommended Ratio 1 DCR Primary Care Physician – PhysicianCurr; PhysicianPast12 Primary Care Numerator: FSP consumers who have Physician a primary care physician currently and over the past 12 months/Denominator: Total number of FSP consumers 11 Item collected with Partnership Assessment Form (PAF), but not included in DCR data dictionary. 12 Homelessness has customarily been a challenge to measure, particularly beyond the mental health service consumer population. Should housing information about mental health service consumers remain of questionable quality, new data collection strategies (e.g., new surveys of literal and functional homelessness or shelter counts) could be suggested. 16 Domain Indicator Calculation of Measure(s) Data Source(s) Dimension – Potential Items Recommended Ratio 2 Additional Data An item to collect data regarding all mental health consumers’ access to Numerator: Consumers who have a Collection a primary care physician may be added to the CSI or incorporated into primary care physician currently and another data collection mechanism over the past 12 months/Denominator: Total number of consumers 6.5 Consumer / Recommended Rating MHSIP surveys Access to Services – LOCATION; TIMEGOOD; HELPWANT; Family Perceptions Average items to create an aggregate HELPNEED of Access to measure of Perceived Access to Services Services Alternative Description Proposed new data e.g., surveys, interviews, focus groups; proposed data collection Qualitative and quantitative analysis collection of several dimensions of access to services 7. 7.1 Consumers Recommended Ratio Quarterly Progress CSS exhibit 6 Performance Served Annually Numerator: CSS/FSP consumers Reports; Annual through CSS served / Denominator: CSS/FSP Updates consumers targeted DCR FSP Identifier - GlobalID 7.2 Involuntary Recommended Ratio Annual Report on 72 hr Evaluation and Treatment (Adults, Children) Care Numerator: Involuntary Services / Involuntary 14 & 30-day Intensive Treatment Denominator: Consumers served Detentions 180-day Post Certification Treatment Temporary & Permanent Conservatorships Seclusion and Restraint CSI County Client Number (CCN) Alternate Ratio Annual Report on 72 hr Evaluation and Treatment (Adults, Children) Numerator: Involuntary Services / Involuntary 14 & 30-day Intensive Treatment Denominator: Population (County and Detentions 180-day Post Certification Treatment State) Temporary & Permanent Conservatorships Seclusion and Restraint Census Population Total Statewide Population (projected) Projection Data 7.3 24-hour Care Recommended Ratios 1 CSI Residential Information - Hospital, PHF, and SNF (S-20.0 – S-22.0) Numerator: Utilization of MHRC, Age -Date of Birth (C-03.0) SNF, SH among TAY, Adults, or Older Adults/ Denominator: DCR Residential Information – Long-TermCare_PastTwelveOccurences; State/county population, FSP, TAY, Long-TermCare_PriorTwelve; NursingPhysical_PastTwelveDays; Adult, Older Adults NursingPhysical_PastTwelveOccurences; NursingPhysical_PriorTwelve; Yesterday; Current; Recommended Ratio 2 PsychiatricHospital_PastTwelveDays; Numerator: Utilization of CTF, RCL PsychiatricHospital_PastTwelveOccurences; 14, MHRC / Denominator: Total FSP PsychiatricHospital_PriorTwelve children or total county child Age – Age_Group population 17 Domain Indicator Calculation of Measure(s) Data Source(s) Dimension – Potential Items Alternate Count 1 DCR Residential Information – See above Consumers in IMD, MHRC, SNF, SH Race/Ethnicity – CSIRace1-5 by race/ethnicity CSI Residential Information – See above Race/Ethnicity – Race(C-10.0) Alternate Counts 2 DCR Acute Care – MedicalHospital_PastTwelveDays; Readmission to acute care facility MedicalHospital_PastTwelveOccurences; within 30 and 180 days (Overall and PsychiatricHospital_PastTwelveDays; FSP) PsychiatricHospital_PastTwelveOccurences; PsychiatricHospital_PriorTwelve CSI Acute Care – 24 Hour Mode of Service (S-15.0 – S-19.0) 7.4 Recommended Rating MHSIP surveys Appropriateness of Care – RESPECT; RELIGION; UNDRSTD; Appropriateness of Consumer/family perceptions of CULTURE Care appropriateness of care Alternative Rate 1 CSI Hospital, PHF, SNF, and 24 hr Care (S-15 – S-22) Number and percent of consumers DCR Hospital Admission - MedicalHospital_PastTwelveDays; (Overall and FSP) readmitted to a MedicalHospital_PastTwelveOccurences; hospital PsychiatricHospital_PastTwelveDays; PsychiatricHospital_PastTwelveOccurences Alternate Descriptive 2 DCR Acute Care – MedicalHospital_PastTwelveDays; Average length of stay in acute care MedicalHospital_PastTwelveOccurences (Overall and FSP) CSI Acute Care – 24 Hour Mode of Service (S-15.0 – S-19.0) Alternate Count 3 Proposed new data e.g., document review, interviews; proposed data collection Treatment protocols for co-morbidity collection 7.5 Continuity of Recommended Count 1 DCR Emergency Care – MenRelated; PhyRelated; ReferredBy; Care Emergency Care (Overall and FSP) CSI Emergency Care – Acute 24-hour mental health services (S-20.0, S- 21.0, S-22.0) Recommended Count 2 DCR Residential Information –Yesterday; Current; ApartmentAlone; Services provided in community AssistedLiving; CommunityCare; CongregatePlacement; settings FosterHomeNon-relative; GroupHome; IndividualPlacement; ResidentialTreatment Proposed new data e.g., surveys, interviews, or focus groups; proposed data collection collection Recommended Count 3 Proposed new data e.g., surveys, interviews, or focus groups; proposed data collection Documented discharge plans collection 7.6 Penetration Recommended Ratio Quarterly Progress CSS exhibit 6 Rate Numerator: All Consumers / Reports; Annual Denominator: Consumers targeted for Updates service or populations of interest (e.g., California Health Demographic Information age, gender, race/ethnicity, Interview Survey socioeconomic status) (CHIS; proposed external data source) 18 Domain Indicator Calculation of Measure(s) Data Source(s) Dimension – Potential Items Alternate Ratio DCR CSS exhibit 6 Numerator: Consumers / Census Data Holzer Targets - estimates of the prevalence of serious mental Denominator: Holzer Targets illness/serious emotional disturbance in California 7.7 Consumer Recommended Rating MHSIP surveys Functioning – DAILYPRB; CONTROL; CRISIS; BETTRFAM; Wellbeing Client/family perception of BETTRSCH; MEANINGFUL; BETTRNEED; BETTRHANDLE; improvement in functioning (current, DOWANTS; HAPYFREND; DOTHINGS; BELONG; SUPPORT over time, among high need groups) Primary data e.g., surveys, interviews, or focus groups; proposed data collection collection Recommended Rating MHSIP surveys Quality of Life – LIFESAT; LIVRANG; PRIVACYL STAYLONG; Client/family perceptions of quality of SPARETIM; ENJOY; FUN; RELAX; SEEFAMLY; FAMCT; life (current, over time, among high FAMGEN; VISIT; TIMERND; DOPEOPLE; TIMEPEOP; SEEPEOP; need groups) AMTFREND 7.8 Satisfaction Recommended Rating MHSIP surveys Satisfaction – LIKESVCS; CHOICES; RECOMMEND; STAFWILL; Consumer/family satisfaction with COMFQUEST; COMPLAIN; care or service 8. Structure 8.1 Workforce Recommended Ratio Numerator: Proposed new data e.g., surveys, interviews, or focus groups; proposed data collection Composition Number of staff / Denominator: collection Number of consumers Alternate Ratio 1 Cultural Competence Demographic Profile of Workforce – Document review Compare demographic composition of Plans MH workforce to that of the consumer population Alternate Count 2 WET Plans Consumer/family member employment – Document review Consumer/family member employment (i.e., number, FTE, % of workforce) 8.2 Evidence- Recommended Count CSI Best Practices – S-25.0 Evidence-Based Practices / Service Based/Best Practice Use of evidence-based practices Strategies Programs and Recommended Additional Data Proposed new data e.g., surveys, interviews, or focus groups; proposed data collection Services Collection collection Fidelity of best practices to established models Alternate Additional Data Proposed new data e.g., surveys, interviews, or focus groups; proposed data collection Collection collection Receipt and experience of best practice services/supports among consumers/families 8.3 Cultural Recommended Rating MHSIP surveys Cultural Appropriateness – CULTURE Appropriateness of Client and family perceptions of Services cultural appropriateness Proposed new data e.g., surveys, interviews, or focus groups; proposed data collection collection 19 Domain Indicator Calculation of Measure(s) Data Source(s) Dimension – Potential Items 8.4 Recovery, Recommended Additional Data Recovery Oriented Recovery Orientation Wellness, and Collection Systems Indicators Resilience Consumer, family member, and staff Measure (ROSI; Orientation perceptions of recovery orientation of proposed data system and services collection) Developing Recovery Recovery Orientation Enhancing Environments Measure (DREEM; proposed data collection) 20 Mental Health Indicator Measurement Detail To clarify the rationale and potential utility of measures for each indicator, this section provides detailed descriptions of the calculations summarized in the templates. This discussion is based on a thorough review of all relevant existing data and, where appropriate, some alternative data sources. Stakeholder Informed Challenges and Limitations, enclosed in blue text boxes, guided many of the revisions seen in Templates 1 and 2. Individual-level (Consumer) Outcomes for Full Service Partnerships Measurement Detail Consumer indicators are individual-level priority indicators designed to create greater clarity about consumers’ dispositions (e.g., employment, education, housing, justice involvement) following interventions coordinated through the MHSA.13 Domain 1: Education/ Employment 1.1 Indicator: Average Attendance – Score Per Year Rationale for measure: Dividing the number of consumers’ days at school during a 9-month school year (numerator) by the total number of days during consumers’ school year (denominator) will yield attendance rates for child consumers and TAY consumers 18 and younger within each county. The rates will then be averaged across all counties to identify statewide average attendance rates for each age group. Stakeholder Informed Challenges & Limitations: Stakeholders reported that accurate school data was difficult to access and normalize due to the types of programs in which consumers were enrolled. For example, alternative education and home schooling would have different attendance requirements. A ratio of school days attended to total school days would address this issue; however, no strategy exists yet to collect the total number of school days from each school district in the state. 1.2 Indicator: Proportion Participating in Paid and Unpaid Employment Rationale for measures: Employment is measured in three ways: 1) Dividing the number of employed consumers over 18 years old (numerator) by the total number of all consumers over 18 years old who are eligible for employment (denominator) will provide the statewide proportion 13 In early planning, consumer indicators were designed for FSP clients only. However, RFP language for this project suggests that community activities around consumer mental health, which extend beyond FSP, are equally important to consider in MHSA reach and impact. Thus, consumer indicators may be relevant to FSP clients and non-FSP clients in the current report. 21 of eligible consumers who are employed at the time of data collection. 2) Dividing the number of consumers over 18 years old who are employed for pay (numerator) by total number of consumers who are eligible for employment (denominator) will provide statewide employment- for-pay proportions. 3) Dividing the number of consumers over 18 years who are employed without pay (numerator) by total number of consumers who are eligible for employment (denominator) will provide statewide employment-without-pay proportions for TAY (18 years and older) and adult groups. Stakeholder Informed Challenges and Limitations: Stakeholders suggested adding the number of days or period of time during which a consumer was employed. Stakeholders also noted that employment information in the DCR system was very limited, unlike the CSI system, which accounts for all types of employment (part-time, volunteer, etc.). The evaluation team would only have a robust understanding of consumer employment for persons who were not enrolled in FSPs – those who are more likely to be employed in part-time and volunteer work. Lastly, stakeholders offered that employment rates, like the economy, occur in cycles that could be overlooked in analyses if unaccounted for. Domain 2: Homelessness/Housing 2.1 Indicator: Housing Situation/ Index-Score Rationale for measures: To capture the variety of consumers’ housing situations, five counts should be conducted. Among these, we recommend a count of days that 1) child consumers and TAY consumers under the age of 18 (considered herein as dependent youth) live in a family home annually; 2) child consumers and TAY consumers under the age of 18 live in a foster home annually; 3) TAY over 18 (legally considered adults), adult consumers, and older adults are homeless. Further, we recommend 4) a count of TAY over 18 and adults with independent residential statuses as well as 5) a count of TAY over 18, adults, and older adults who are not homeless (have any type of housing). Counts are not summative; rather, they provide statewide statuses of the housing types being used by consumers, to what extent, and the level of need (homelessness). 22 Stakeholder Informed Challenges & Limitations: Stakeholders noted three challenges: 1) both homelessness and housing can change often among the group, yielding complex data. Stakeholders suggested becoming familiar with this particular data before conducting analyses because of its ever-changing nature. 2) Housing data from the CSI system is not updated as regularly as data from the DCR system, and lag might create inaccuracies in the evaluation. 3) Children who are homeless by way of homeless parents are not captured by the DCR. That is, a child who is with one or both parents can be categorized as “housed” without any indication or measure of his or her parents’ living situation. Thus, housing information in the DCR is inaccurate. Beyond data challenges, stakeholders offered that adding the number of days a consumer is housed would be useful to understand housing stability. Domain 3: Justice Involvement 3.1 Indicator: Justice Involvement Rationale for measure: Number of consumer arrests within 12 months will be collected to track statewide rates that may or may not be related to consumers’ mental health episodes. Stakeholder Informed Challenges & Limitations: Stakeholders described different ways to conceptualize “arrests” to create a more specific indicator definition. Some suggested counting incarcerations instead of arrests given that intercession by FSP teams sometimes prevents incarceration. Others suggested counting new arrests given that consumers might be re-arrested for the same offense due to probation violations. Also, the term “episode” was contested as it did not account for chronic mental health issues. In sum, further revision of what is meant by Justice Involvement is needed. In addition to clarifying the indicator, stakeholders suggest adding the number of days a consumer is held to create a rate that can be followed over time. Domain 4: Emergency Care 4.1 Indicator: Emergency Interventions for Mental Health Episodes Rationale for measures: Dividing the number of mental health-related hospitalizations (numerator) by the number of consumers’ hospital visits within 12 months will give an indication of episode severity, crisis, and rate of acute hospitalization for mental health management/intervention. Indirectly, the ratio will give an indication of consumers’ quality of life related to mental health. 23 Stakeholder Informed Challenges & Limitations: Stakeholders suggested that creating an appropriate Emergency Care indicator could be difficult given that 1) visits to the emergency room and hospitalization are distinct in the DCR system, and 2) the CSI is not regularly updated with visits and releases from medical facilities. Among additions to the proposed priority indicator, stakeholders proposed adding the number of days a consumer spends in emergency care given that FSP interventions might yield a shorter psychiatric stay. 4.2 Indicator: Emergency Interventions for Co-occurring Physical Injury Rationale for Inclusion: Dividing the number of physical injury or physical health-related hospitalizations (numerator) by the number of consumers’ hospital visits within 12 months will give an indirect measure of mental health crisis given that mental and physical health are often interrelated (per stakeholder feedback). The ratio is a secondary measure of consumers’ quality of life related to mental health. Domain 5: Social Connections 5.1 Indicator: Proportion Who Identify Family Support Rationale for measures: Dividing the number of family members the consumer identifies as reliable supporters, or persons who are consistently present for the consumer, by the total number of mental health consumers annually will help identify the breadth of a consumer’s local (family) network – one that optimally provides ongoing and immediate support even in times of mental health distress. 5.2 Indicator: Proportion Who Identify Community Support Rationale for measures: Dividing the number of non-family members the consumer identifies as reliable supporters by the total number of consumers, and the number of organizations from which the consumer receives regular, voluntary, and quality services by the total number of consumers, will provide a measure of consumers’ social network. The social network is one that should provide consumers steadfast support toward sustaining mental health. Mental Health System-level Outcomes for All Consumers Measurement Detail System-level priority indicators (related to consumer access, agency performance, and agency structure) explain how operations changed or may have been enhanced by the MHSA. Domain 6: Access 6.1 Indicator: Demographic Profile of Consumers Served Rationale for measure: Descriptive statistics of the age, gender, and race/ethnicity of the service population (overall and specific to FSP consumers) will provide demographic description of 24 those receiving services within and across counties, and allow for examination among populations in need (e.g., homeless, unemployed). Stakeholder Informed Challenges & Limitations: Demographic information (i.e., ethnicity, gender) contained in the DCR is imported from the CSI database. Thus, any FSP consumer not registered in the CSI system will have incomplete demographic information in the DCR database. 6.2 Indicator: New Consumers by Demographic Profile Rationale for measure: Descriptive statistics concerning the age, gender, race/ethnicity of new consumers (i.e., less than 6 months of service receipt) will provide an understanding of who new consumers are and some indication of the populations which are being reached. Additionally, demographic descriptions of existing consumers (i.e., more than 6 months of service receipt) will provide context for evaluating the makeup of new consumers and might give indication that historically underrepresented groups are seeking and/or receiving services. 6.3 Indicator: High Need Consumers Served Rationale for measure: Accurate counts of homeless and unemployed consumers served through the FSP program can provide understanding of the extent to which these high need consumer groups are being served. Alternately, the numbers of homeless and unemployed among all mental health consumers will provide evidence of service to these groups overall, and provide a relative basis with which to evaluate the extent of service to high need consumer groups through FSP. Stakeholder Informed Challenges & Limitations: Stakeholders expressed concerns that variables included in the CSI database, such as Living Situation and Employment, may not be updated regularly for non-FSP consumers. If these items are found to be unreliable, high need consumer served through non-FSP programs may not be accurately described. 6.4 Indicator: Access to Primary Care Physician Rationale for measure: Tracking the number of FSP consumers with access to a primary care physician will provide evidence of the extent to which FSP services may be helping to connect consumers with the health care they need. 25 Stakeholder Informed Challenges & Limitations: Stakeholders expressed concerns that CSI data regarding access to a primary care physician may not be reliable. If this information is not reliably reported, access to a physician among all mental health consumers cannot be accurately assessed and comparisons to FSP consumers cannot be made. 6.5 Indicator: Consumer / Family Perceptions of Access to Services Rationale for measure: Aggregate ratings of consumer and family perceptions of the extent to which they are able to connect with the services they need will provide evidence of the accessibility of MHSA services from the perspective of the consumer. Domain 7: Performance 7.1 Indicator: Consumers Served Annually through CSS Rationale for measure: The number of consumers served annually through CSS (i.e., consumers overall, FSP consumers) relative to those who were targeted for service will allow for CSS service rates to be understood in the context (e.g., type and extent of need among various consumer populations) of the county in which the services were provided. In this case, grounding service rates in county context will provide a more accurate account of service levels/performance than a simple count of consumers. Stakeholder Informed Challenges & Limitations: Stakeholders expressed concerns that Exhibit 6 is not a reliable source for information regarding consumers targeted or served through CSS programs due to the various ways in which counties define these categories. If data regarding consumers targeted or served is found unreliable across counties, the ratio of those served to targeted consumers statewide cannot be accurately estimated. 7.2 Indicator: Involuntary Care Rationale for measure: The ratio of those who received Involuntary Services (e.g., Evaluation and Treatment, Temporary or Permanent Conservatorships, Seclusion and Restraint) to all consumers served will allow for greater understanding of this service relative to the consumer population. Such measures provide for more accurate evaluation of services within and between counties as well as statewide. Alternatively, the ratio of those receiving Involuntary Services to populations of interest (i.e., adults, homeless, unemployed) will allow for evaluation of the performance of these services within and between consumer groups. 7.3 Indicator: 24-hour Care 26 Rationale for measure: The ratio of 24-hour care to consumer populations (i.e., Statewide, by County, TAY, Adult, Older-adult populations – MHRC, SNF, SH; Child populations – CTF, RCL 14, MHRC) will provide an accurate assessment of the performance/extent of these services relative to the size of population for which they were intended. As an alternative, demographic profiles of consumers (consumers overall and FSP consumers specifically) receiving these services may provide useful information regarding the consumer groups who utilize such intense services most. Another measurement option would involve counts of readmissions to acute care facilities (among consumers overall and FSP consumers specifically), which can provide indication of how often consumers require this type of care. 7.4 Indicator: Appropriateness of Care Rationale for measure: Aggregate consumer and family ratings of appropriateness of care will provide an understanding of how services are perceived on average. As an alternative, average length of stay in acute care (among consumers overall and FSP consumers specifically) can provide evidence of the extent to which such intensive services are utilized, which may be more or less appropriate for different consumer groups. Another alternative would be the existence of standard protocols for treating co-morbidity. Issues such as substance abuse often co-occur with mental health issues, thus the existence of treatment protocols for co-morbidity will provide evidence of the existence of appropriate care for such consumers. Stakeholder Informed Challenges & Limitations: Assessing treatment protocols for co- morbidity would require extensive document review to establish their existence in each county, and additional data collection (e.g., surveys or interviews) to assess whether they are appropriately applied. 7.5 Indicator: Continuity of Care Rationale for measure: Use of emergency services among consumers (consumers overall and FSP consumers specifically) may provide evidence of the connection of such services with those they have previously received or are currently receiving. A measure of services provided in community settings may be created from data regarding residential status and living situation. The existence of discharge plans may provide evidence to the continuity of consumers’ paths to recovery. However, the later measures may require additional data collection. Stakeholder Informed Challenges & Limitations: Comprehensively assessing continuity of care may require extensive document review or additional data collection (e.g., surveys or interviews). 27 7.6 Indicator: Penetration Rate Rationale for measure: The ratio of consumers who receive mental health services to the number of persons considered “high need” (e.g., homeless, unemployed) will present the reach of CSS programs into various populations within each county. Alternately, the ratio of mental health consumers served to Holzer Targets in each county would provide indication of the extent to which CSS services are reaching those with serious mental illness/serious emotional disturbance. Stakeholder Informed Challenges & Limitations: Stakeholders have expressed concern about the reliability of Exhibit 6 data. If such data proves unreliable, the penetration of CSS services among populations in need cannot accurately be assessed. 7.7 Indicator: Consumer Wellbeing Rationale for measure: Consumer and family member aggregate ratings of improvement in functioning and quality of life will provide important measures of the perceived impact of services on average from the consumer perspective. As these measures only tap two elements of wellbeing, additional qualitative primary data collection may supplement these ratings by providing more rich understanding of how services impact consumers’ wellbeing. Stakeholder Informed Challenges & Limitations: Different sampling methods for consumer perception survey respondents have been employed over time, which may have implications for analysis and any conclusions drawn. Additional qualitative and quantitative data collection may be needed to create an indicator of wellbeing, which is sensitive to county context and the backgrounds of consumers and their families. 7.8 Indicator: Satisfaction Rationale for measure: Aggregate consumer and family member ratings of satisfaction with care or service will provide an indication of consumers’ perceived services on average. Domain 8: Structure 8.1 Indicator: Workforce Composition Rationale for measure: The ratio of staff to consumers will generate a measure of the size of the workforce relative to the consumer population in each county. As an alternative, comparison of the demographic makeup of the workforce and consumer populations will provide insight into how well the workforce reflects those they serve. Another option would be to consider consumer and family member employment in the mental health system (i.e., number, FTE, percent of workforce), which would provide evidence of the extent to which consumers have been integrated into the service process. 28 Stakeholder Informed Challenges & Limitations: Stakeholders expressed concern about the reliability of workforce data in Cultural Competence Plans and WET Plans over time at the program level. Without a valid source of data regarding workforce makeup, the appropriateness of the workforce for serving the current consumer population cannot be assessed. 8.2 Indicator: Evidence-Based/Best Practice Programs and Services Rationale for measure: The type and number of evidence based or best practice programs implemented in each county would provide indication of the extent to which established high quality programs are being implemented within counties and across the state. Additionally, the extent to which evidence based or best practice programs are being implemented with fidelity would provide indication of the quality of these programs as implemented. Alternatively, the frequency and experience of evidence based or best practice services would provide important evidence of the usefulness of these programs from the consumer perspective. However, the latter measures would require additional data collection. Stakeholder Informed Challenges & Limitations: Stakeholders expressed concern that the fidelity with which evidence-based services are implemented is not captured by the relevant CSI item. Thus, it is recommended that additional data collection (e.g., surveys, interviews, or focus groups) be conducted to assess the fidelity of reported evidence-based program implementation. 8.3 Indicator: Cultural Appropriateness of Services Rationale for measure: Consumer and family member aggregate ratings of cultural appropriateness of services will provide an important measure of the perceived adequacy of services with regard to consumers’ cultural needs. However, only a single survey item directly taps cultural appropriateness of services, thus it will be necessary to augment existing data collection or consider additional data collection in order to create an adequately robust measure. 8.4 Indicator: Recovery, Wellness, and Resilience Orientation Rationale for measure: Measurement of recovery, wellness, and resilience orientation may provide evidence of the extent to which county mental health systems and the state overall are adhering to and achieving stated values and goals. No comprehensive measure of the recovery, wellness, and resilience orientation is currently collected; however, options for established measures exist (e.g., Recovery Oriented Systems Indicators Measure, Developing Recovery Enhancing Environments Measure). 29 Stakeholder Informed Challenges & Limitations: Any measure of recovery orientation must be sensitive to, and informed by, the unique needs and circumstances of diverse cultural, ethnic, regional, and age groups across the state. Stakeholder Feedback Through the feedback process detailed earlier in this report, stakeholders across the state provided unique and well informed perspectives, thoughtful reaction and insight regarding the proposed data sources and methods for calculating all priority indicators. Stakeholders’ specific concerns regarding limitations or challenges of individual indicators can be found in the Mental Health Indicator Measurement Detail section. A table of categorized stakeholder feedback is located in Appendix B. Feedback regarding proposed data sources and methods for calculating all priority indicators presented to stakeholders in the initial draft of this report fell largely into the two domains below: Types of Stakeholder Feedback and Corresponding Revisions Data source availability and quality. Possibly the most common feedback theme was centered on the availability, accuracy, and reliability of data to inform the proposed indicators. Repeatedly, the evaluation team was warned about the completeness and accuracy of existing data (e.g., DCR, CSI, Exhibit 6, and client perception surveys). While many stakeholder concerns in this regard will be verified by a thorough review of existing data, the unique insights and historical knowledge of data integrity within specific counties and throughout the state was central to our revisions of the proposed sources of data to be used to compose priority indictors. As an example, stakeholders expressed concern that many indicators could require additional primary data collection in order to comprehensively capture factors such as Workforce Composition, as they suggested existing sources were incomplete or unreliable. o Corresponding revisions. Based upon feedback regarding data concerns, several indicators were revised to include alternative existing data sources, potential external data sources (e.g., census data), or proposals for additional primary data collection. Additionally, a data quality review process, outlined in the next section, will provide more detailed knowledge of where the inconsistencies or irregularities may be among existing data sources and present logical and practical possibilities for rectifying them. Appropriate indicator calculations. The indicator calculations proposed in the initial draft of this report were largely supported; however, alternative methods of calculation were suggested for a handful of indicators. For example, Holzer Targets were suggested as accurate estimates of the prevalence of serious mental illness or serious emotional disturbance in California, for use in calculating penetration rate. 30 o Corresponding revisions. Based upon feedback regarding alternative calculations, several indicator calculations proposed in this report reflect revisions to ensure they are most appropriate for providing accurate estimation of their respective element of the MHSA system or consumer experience. One such revision was including Holzer Targets as part of an alternative method for calculating penetration rate. Many such revisions to incorporate stakeholder feedback regarding alternative indicator calculation or data sources have been integrated throughout this report. Conclusions Per our objectives, the evaluation team located survey items (variables) that could be used to construct priority indicators and outlined protocols and rationale for calculating each measure. Also, where existing data was not sufficient, measures and indicators for which additional data collection may be helpful (i.e., supplementary) or necessary were noted. This information was guided by stakeholders’ historical knowledge of data sources. All measurement domains, priority performance indicators, calculation of measures (recommended and alternative), the databases or reports from which items can be drawn, and the specific items within each dataset or report were displayed in a series of templates. Overall, this report was an important step in defining and refining the priority performance indicators to the very practical item level. While this framework for constructing indicators is comprehensive of all priority performance indicators, flexibility exists with regard to how each measure may be constructed, which is reflected in the alternate measures and methods of calculation highlighted throughout the templates. To refine the measure of each indicator and solidify the methods of calculation for each measure, the UCLA/EMT team must conduct a thorough data quality review. Next Steps Data Quality Review As of this report, access to most data sources listed in the report templates has not been granted. Once data associated with each item (variable) can be reviewed, we will systematically determine data quality and completeness as well as item appropriateness for each measure/indicator using the following criteria. The data quality review will also take into account input from experts in the field who hold expertise regarding data collection and analysis generally and specific to the data sources specified in this report. This process will drive further development of the indicator template and recommendations regarding existing and additional data collection. The criteria, also outlined in the report Templates for Reporting Priority Indicators, Deliverable 2A, must include: 31 Adequate base rate (i.e., the rate at which an event occurs or level at which a scaled response is given on average, must not be so low as to make the indicator useless or meaningless) Adequate variance (i.e., values of a given measure must be sufficiently distributed through the range of the measure to support analysis) Validity o The measure is face valid, can conceptually and logically be said to measure what it was intended to o The performance measure is internally valid and can logically be tied to a particular program intervention or outcome o The indicator is externally valid and can logically be generalized to other populations or programs Reliability (i.e., the indicator is consistent over time and cases) Availability and completeness of indicator relevant data which is obtainable and complete for populations of interest (e.g., age groups, gender, race/ethnicity, socioeconomic status) for the period of time under study Ability to be aggregated to county and state levels Further, templates are not analysis plans that describe what can be learned for specific mental health consumer groups (e.g., women, Native Americans, "high need," etc.). Rather, they present priority indicators and their proposed measurement across four prescribed age groups – children, TAY, adults, and older adults. Attention to consumers by demographic group exists within system level indicators (refer to Template 2), but exactly what information indicators can provide cannot be stated without a full review of mental health service data. Following review of the data, the evaluation team can make more definitive statements about which demographic groups (age groups excepted) can be described. Initial Reporting of Results for Priority Indicators & Stakeholder Input This report and its companion document (“Templates for Reporting Priority Indicators”, deliverable 2B) will be the foundation for a forthcoming report (deliverable 2E), which details results for all priority indicators at the statewide level and three subsequent county level reports. For each of these subsequent reports, stakeholder input will continue to be a vital part of the report development and revision process, so as to ensure appropriate and accurate monitoring of MHSA consumer outcomes and mental health system processes. 32 Appendix A Participating Organizations and Agencies (In Alphabetical Order) Individuals and groups from the following entities received an e-mail announcing the availability of MHSA Evaluation Team contract deliverables 2A and 2C. It is possible that more persons than are listed received the call through message forwarding. Thus, the following list was created to the best of the team’s knowledge and e-mail verification. APS Healthcare Association of Community Human Services Agencies Bonita House California Association of Social Rehabilitation Agencies California Community Colleges Chancellor's Office California Department of Aging California Department of Mental Health California Institute for Mental Health California Mental Health Director’s Association California Mental Health Planning Council California Network of Mental Health Clients Contra Costa County Health Services Department California Council of Community Mental Health Agencies EMQ Families First Humboldt County California Mental Health Directors Association Indicators, Data, Evaluation Accountability (IDEA) Committee Los Angeles County Department of Mental Health Mental Health America of California Mental Health America of Los Angeles MHSA Partners Mental Health Services Oversight and Accountability Commission Monterey County National Alliance on Mental Illness – California Nevada County Orange County Behavioral Health Services San Bernardino County Department of Behavioral Health San Diego County San Francisco Department of Public Health San Joaquin County Mental Health Board Seeds of Hope Shasta County Health and Human Services Turning Point Community Programs United Advocates for Children and Families 33 Appendix B Table 1: Types of Stakeholder Feedback Received (Summary) Domain Domains Domains Alternative Alternative Challenges with incomplete incomplete Request for domain/measures data sources associated data (changes (additions clarification suggested suggested noted suggested) suggested) • • • • 1. Employment/ Education 2. Homelessness/ Housing • • • Situation • • • 3. Justice Involvement • • 4. Emergency Care • • 5. Access • 6. Performance • • • • • • 7. Structure Additional domains suggested: Additional data sources: Alcohol and Other Drug Use (AOD) Claiming process Co-occurring physical health disorders Recovery Oriented Systems Indicators Measure (ROSI) / Developing Social connections Recovery Enhancing Environments Measure (DREEM) 34 Table 2: Types of Stakeholder Feedback Received (Detailed) Indicator Indicator Indicator Alternative Alternative Challenges with incomplete incomplete Request for domain/measures data sources associated data (changes (additions clarification suggested suggested noted suggested) suggested) • 1. Employment/ Education • • 1.1 Education • • • 1.2 Employment 2. Homelessness/ Housing • • • Situation • • 3. Justice Involvement • • • 4. Emergency Care 5. Access 5.1 Demographic • Profile 5.2 New Consumers by • • Demographic Profile 35 Table 2: Types of Stakeholder Feedback Received (Detailed) Indicator Indicator Indicator Alternative Alternative Challenges with incomplete incomplete Request for domain/measures data sources associated data (changes (additions clarification suggested suggested noted suggested) suggested) 5.3 High Needs • • • Consumers Served 5.4 Access to Primary • • • Care Physicians 5.5 Consumer/ Family Perceptions of Access to Services 6. Performance 6.1 Consumers Served • Annually Through CSS 6.2 Involuntary Care 6.3 24-Hour Care 6.4 Appropriateness of • • Care • • 6.5 Continuity of Care 36 Table 2: Types of Stakeholder Feedback Received (Detailed) Indicator Indicator Indicator Alternative Alternative Challenges with incomplete incomplete Request for domain/measures data sources associated data (changes (additions clarification suggested suggested noted suggested) suggested) • • 6.6 Penetration Rate 6.7 Consumer • Wellbeing 6.8 Satisfaction 7. Structure 7.1 Workforce • • • Composition 7.2 Evidence Based/Best • • • Practice Programs and Services 7.3 Cultural • • • Appropriateness of Services 7.4 Recovery, Wellness, • • and Resilience Orientation 37 Appendix C 38