Mhsa Priority Indicators: Santa Clara County

Mental Health Services Act Evaluation: Report on Prioritized Indicators Contract Deliverable 2F, Phase II Santa Clara County UCLA Center for Healthier Children, Youth and Families EMT Associates, Inc. Submitted for review on November 30, 2012 The following report was funded by the Mental Health Services Oversight and Accountability Commission. The following report was revised in partnership with stakeholders who provided important historical context, data consultation, and revisions to ensure this report is accurate and accessible to the broadest audience possible. Feedback, collected prior to, during, and following report development, was crucial to developing this report. The UCLA-EMT Evaluation Team would like to express sincere appreciation to the research analysts, advocates, consumers and family members, agency representatives, service providers, and MHSOAC representatives who contributed invaluable insights to this document and previous versions. 2 Table of Contents Purpose ........................................................................................................................................................................................ 4 Background ................................................................................................................................................................................ 4 Review of Existing Data ........................................................................................................................................................ 6 Data Sources ......................................................................................................................................................................... 6 Priority Indicators Evaluated .................................................................................................................................. 10 Priority Indicators ............................................................................................................................................................ 11 Report Organization ................................................................................................................................................... 12 Priority Indicator 1: Attendance ..................................................................................................................................... 13 1.1 Expulsions and Suspensions Per Year (CPS) ........................................................................................... 13 1.2 Average School Attendance Per Year (FSP) ............................................................................................. 15 Priority Indicator 2: Employment ................................................................................................................................. 17 Priority Indicator 3: Homelessness and Housing Rates ....................................................................................... 21 Priority Indicator 4: Arrest Rates .................................................................................................................................. 24 Priority Indicator 5: Demographic Profile of Consumers Served ..................................................................... 30 Priority Indicator 6: Demographic Profile of New Consumers .......................................................................... 34 Priority Indicator 7: Penetration of Mental Health Services .............................................................................. 39 Priority Indicator 8: Access to a Primary Care Physician .................................................................................... 42 Priority Indicator 9: Perceptions of Access to Services ........................................................................................ 45 Priority Indicator 10: Involuntary Status ................................................................................................................... 48 Priority Indicator 11: Consumer Perceptions of Improvement in Well-Being as a Result of Services ....................................................................................................................................................................................................... 50 Priority Indicator 12: Satisfaction with Services ..................................................................................................... 53 Appendix A – Priority Indicator Matrix ........................................................................................................................ 55 Appendix B – Preceding Reports ..................................................................................................................................... 56 Appendix C – Recoding Pre-DIG Race Data to Post-DIG Format ........................................................................ 57 3 Purpose The Mental Health Services Act Oversight and Accountability Commission (MHSOAC) charged the UCLA-EMT Evaluation Team with tracking the Mental Health Services Act’s (MHSA) impact on mental health service consumers and the community mental health service system. The current report details the initial effort toward this goal at the county level. Here, we provide a snapshot of consumer outcomes and community mental health system performance across counties. This is a first step toward developing a set of indicators that can help stakeholders with ongoing quality improvement. This report is the first in a series designed to update stakeholders about mental health consumer outcomes and service system progress. Tracking performance on particular outcomes over time, across programs, and within counties can provide useful information to those planning, operating and monitoring services. Indicators are intended to be used for planning, quality improvement, and other applications that stakeholders deem important. In this way, among many others, stakeholders can play a vital role in a continuous quality improvement process. Background What are Priority Indicators and what are they intended to do? Two concerns of public mental health system stakeholders are accountability and the ability to conduct continuous quality improvement activities. One strategy is to use a set of indicators to measure performance. The California Mental Health Planning Council proposed and defined a set of performance indicators, referred to as priority indicators, designed to assess how the MHSA has impacted mental health consumers and the mental health system in target areas that should be most changed through MHSA implementation. Indicators will help track progress among consumers and across community mental health systems. At the consumer level, outcomes such as education and employment will be followed, while outcomes including mental health service penetration and consumer demographics are examined at the broader system level. This report examines the core set of priority indicators vetted by the MHSOAC. Previous work of the UCLA-EMT Evaluation Team leading to this report The evaluation team began its work using the California Mental Health Planning Council’s definitions –its collective vision of how indicators might best be measured. (These fundamental definitions are located in Appendix A and are discussed in preceding reports available at http://healthychild.ucla.edu/MHSA_evaluation.asp.) Priority indicator development was a joint effort among the MHSOAC, stakeholders, and the evaluation team. The evaluation team facilitated discussions among interested stakeholders to create the strongest, most comprehensive representations of priority indicators that aligned with both early conceptualizations and feedback using the data that were already collected across the state with some regularity. Where gaps existed, the evaluation team proposed new data collection that will improve future evaluation but is beyond the team’s current scope of work. The evaluation team adapted advice from stakeholders, and this report examines whether these adapted indicators provide meaningful information. Although stakeholders proposed additional indicators, these measures have not yet been vetted by the MHSOAC to determine whether they add useful and crucial information that aligns with the need. The MHSOAC has yet to decide whether to change the previously approved priority indicators. Thus, the evaluation team explored the first proposed priority indicators in this report, which serves as a fundamental step in the ongoing 4 process to refine and expand priority indicators that are not only measurable but also useful to the range of stakeholders invested in this work. The evaluation team completed extensive groundwork before arriving at the conclusions contained in this report. To date, the team has documented evaluation planning in four reports: Report title: Defining Priority Indicators Report version: Draft for stakeholder review Here, the evaluation team began to refine the core set of priority indicators proposed by the California Mental Health Planning Council to assess target outcomes of mental health consumers and the performance of the mental health system. The evaluation team and the MHSOAC made this report version available to the public through mass e-mail announcements and online at UCLA and MHSOAC websites. A guidance document that included specific questions regarding the initial report’s content and accessibility was also included with the report to aid review. The evaluation team requested that readers alert their peers and clients to the report to broaden the diversity of feedback. The team also hosted two webinars, or online orientations to the report, with stakeholder groups that provided an overview of the report’s purpose and the type of feedback sought. The call for feedback was open for one month. Report title: Defining Priority Indicators Report version: Final, revised with stakeholder input In the revised report, the evaluation team illustrated how stakeholder feedback was integral to indicator development. This report incorporated changes driven by stakeholders’ comments about the comprehensiveness and appropriateness of the indicators. Report title: Compiling Data to Produce All Priority Indicators Report version: Draft for stakeholder review In this report, the evaluation team proposed how priority indicators could be calculated using existing statewide data. The report also detailed all possible data sources and specific variables or data fields that might be used to build comprehensive priority indicators. The evaluation team made this draft widely available for feedback using a strategy similar to that of the Defining Priority Indicators-Draft report; the report’s availability and a call for feedback were announced online. Readers could download the report and an accompanying guidance document from the UCLA or MHSOAC websites and respond with comments within the month-long feedback period. Report title: Compiling Data to Produce All Priority Indicators Report version: Final, revised with stakeholder input The initial report was revised to include information regarding measurement methods and the adequacy of existing data sources, gathered through a stakeholder feedback process similar to that used for the final Defining Priority Indicators report. This report is the next step in documenting priority indicator development. The evaluation team reviewed data from 2005 through 2011 in search of one fiscal year in which data cells were largely filled where expected. Two such fiscal years were identified – FYs 2008-09 and 2009-10. Through analysis, some proposed data sources or methods of indicator calculation, suggested in previous reports by stakeholders and the evaluation team, were found to not be possible or to not produce meaningful outcomes due to data limitations. Decisions made about previously proposed indicators, based on data limitations, are summarized in Appendix C. 5 Review of Existing Data As directed by the MHSOAC, priority indicators were built upon existing data sources that are systematically collected by California counties and reported to the California Department of Health Care Services (DHCS).1 To accomplish this, existing data systems were reviewed to assess their suitability for supporting outcome and performance monitoring through priority indicators. Several criteria were used to evaluate the suitability of existing data sources, including: Available – Data accessible in an analyzable format Complete – Levels of missing information within key data fields did not prevent meaningful analysis and interpretation Sustained – Data sources is likely to continue to exist in the foreseeable future Relevant – Data relevant to populations of interest (e.g., all mental health consumers and Full Service Partnerships Longitudinal – Data available for multiple service years Multilevel – Data can be analyzed at multiple levels (e.g., state, county, and individual) The application of these criteria to each key data source and important considerations and limitations regarding each data source overall are summarized in the Data Sources table below. These criteria were also applied to the specific data fields used to build each priority indicator. Review of indicator-specific data fields is summarized within the tables that introduce the analysis and findings of each priority indicator (see Priority Indicator Analysis and Findings section below). Data Sources Client & Service Information (CSI) Summary: The CSI system is a repository of county, client (e.g., age, gender, preferred language, education, employment status, living arrangement, etc.), and service information (number and length of service contact). The data are collected from all consumers who receive mental health services, including consumers involved in the Full Service Partnership. Review Findings: Available  Complete  Sustained  Relevant  Longitudinal  Multilevel  Considerations and Limitations: Stakeholder feedback to previous evaluation team reports suggested that inconsistency and potential inaccuracy among race and ethnicity data fields may be due in part to changes in the format of these fields in the CSI data system (see DMH Information Notice: 06-02). For details regarding the Race and Ethnicity data field changes and procedures employed by the evaluation team to improve data quality, see Appendix C. Additionally, the completeness of data fields used to calculate indicators varies greatly across fiscal years and among counties (e.g., greater than 50% in some cases). Thus the representativeness and interpretability of such data fields is in doubt. Proportions of missing or unknown information are noted within each indicator section throughout the report. Data Collection and Reporting (DCR) System 1 Previously the Department of Mental Health (DMH); The DHCS abbreviation will be used to refer to work completed by DMH. 6 Summary: The DCR system houses data for consumers who are served through Full Service Partnership programs. Data from assessments – the Partnership Assessment Form (PAF), Key Event Tracking (KET), and Quarterly Assessment (3M) – are collected for consumers in specific age categories. The PAF reflects consumer history and baseline information, including consumer education and/or employment, housing situation, legal issues, health status, and substance use. The KET reflects any important changes in the consumer’s life, such as housing, education and/or employment, and legal issues during FSP. The 3M is used to collect information on a quarterly basis, regarding key areas such as education, health status, substance use, and legal issues. Review Findings: Available  Complete  Sustained  Relevant  Longitudinal  Multilevel  Considerations and Limitations: Race and ethnicity information in the DCR system is imported from the CSI system by DHCS. As such, the limitations of this information noted for the CSI system also apply here. Specifically, stakeholder feedback to previous evaluation team reports suggested that inconsistency and potential inaccuracy among race and ethnicity data fields may be due in part to changes in the format of these fields in the DCR data system (see DMH Information Notice: 06-02 ). For details regarding race and ethnicity data field changes and procedures employed by the evaluation team to improve data quality, see Appendix C. Additionally, representatives from seven counties or municipalities that currently do not have data contained in the DCR database for FYs 2008-09 or 2009-10 were given the opportunity to provide data to the evaluation team for DCR fields used to calculate indicators. Of the counties not captured in the DCR database for various reasons (e.g., county data incompatibly formatted DHCS database), four representatives provided data within eight weeks of receiving the data quality assurance report. This information was considered in analyses and preparation of this report. The DCR data that other county representatives provided or may provide directly to the evaluation team subsequent to June 8, 2012, will be considered for future reports. Performance Outcomes and Quality Improvement (POQI) – Consumer Perception Surveys (CPS) Summary: These consumer surveys are customized for consumer groups (e.g., family members/caregivers, youth, adults, and older adults) receiving mental health services. Instruments are composed of widely validated measures such as the Child Behavior Checklist, Youth Self Report, and Restrictiveness of Living Environment Scale for youth; the Global Assessment of Functioning, Behavior and Symptom Identification Scale, and the California Quality of Life for adults; and the Brief Symptom Inventory, Senior Outcomes Checklist 10, and Index of Independent Activities of Daily Living for older adults. The data, designed to inform treatment planning and service management, are collected from a sample of individuals with “serious, persistent” mental illness who have received services for 60 days or more and are not categorized as “medication only.” Review Findings: Available  Complete  Sustained  Relevant  Longitudinal  Considerations and Limitations: For FY 2008‐09 and prior years, a convenience sampling approach was used in which county-level mental health service providers administered surveys twice a year for a two‐week period, in early May and November. Investigation of the convenience sampling methodology revealed the resulting information was not representative of the larger mental health service population.2 Beginning with FY 2009-10, a random sampling methodology was employed to produce data that are more representative of the perceptions of the mental health service population. As such, comparisons involving CPS data collected in FY 2008-09 and FY 2009-10 cannot be made. Note – The smaller sample generated by the random sampling method employed in FY 2009-10 does not allow for consumer perception analyses at the county levels for this fiscal year. 2 Cowles, E. L., Harris, K., Larsen, C., and Prince, A. (2010). Assessing Representativeness of the Mental Health Services Consumer Perception Survey. 7 Note – The sampling methods that have been employed to date do not capture specific mental health service populations, such as those in institutions for mental disease or prison. Other Sources Estimates of Need for Mental Health Services To achieve a standardized rate for penetration of services across all counties, the evaluation team contracted with Dr. Charles Holzer for statewide and county mental health service need estimates. Dr. Holzer previously developed penetration rate estimates for the California DHCS. He estimated the proportion of persons with serious mental illness among those whose income falls within 200% of the federal poverty level, using data from the most up-to- date National Comorbidity Survey Replication and generated prevalence estimates for several Census years. (For additional information regarding prevalence estimate methodology, see Dr. Holzer’s website at http://66.140.7.155/estimation/3_Synthetic/synthetic.htm). Review Findings: Available  Complete  Relevant  Longitudinal  Involuntary Status Involuntary status information (FY 2008-09) was provided by DHCS for the following service categories: 72 hour Evaluation and Treatment (adults, children); 14- and 30-day Intensive Treatment; 180-day Post Certification Treatment; and Temporary and Permanent Conservatorships. Involuntary status data for FY 2009-10 were not available from DHCS as this report was being prepared. Review Findings: Available  Complete  Sustained  Relevant  Longitudinal  Procedures for handling missing / unknown data The quantity of missing or unknown data (e.g., values) was found to vary considerably across data sources, data fields, and fiscal years. For data fields determined to be necessary for the construction of priority performance indicators (detailed in priority indicator summary tables, see Priority Indicator Analysis and Findings section below), if the amount of missing or unknown data was substantial (i.e., greater than 10% of cases), the evaluation team communicated with DHCS analysts and requested input from counties via a data verification process (detailed below) regarding context and interpretation of such data fields. Where adequate information was received to interpret missing or unknown values (i.e., see Appendix C), the evaluation team was able to analyze and interpret such data fields according to current protocols specified in the data dictionaries relevant to each data system. The proportion of missing and unknown information relevant to each priority indicator is noted throughout the report, in footnotes immediately below the relevant table or figure. Accordingly, the frequencies and percentages included in all data displays do not include missing cases. This report provides insights about counties with data to support priority indicators. Where data is absent, data cells and columns are blacked out. Summary of data “verification” process In a first attempt to calculate priority indicators, the evaluation team asked county representatives to weigh in on the quality of select data. The evaluation team narrowed a pool of possible calculations to one practical calculation for each priority indicator. The selection was based on predetermined criteria (see Compiling Data to Produce All Priority Indicators, November 2, 2011), an extensive review of the available data, and discussions within the evaluation team about 8 whether proposed calculations could be meaningfully extrapolated to mental health consumer populations. This process revealed the need for a more thorough data quality review. Closer examination of the data needed for each calculation revealed that substantial variation (values and reporting patterns) existed among counties/municipalities, within CSI and DCR data fields identified for constructing priority indicators, during FYs 2008-09 and 2009-10. The variation, in addition to stakeholder feedback to our previous report, demonstrated a need for county representatives to indicate the quality of key data and contextual information needed for analysis, interpretation, and decisions based on this data. At the direction of the MHSOAC ad-hoc committee, the evaluation team provided representatives from all counties/municipalities the opportunity to review and comment on their data quality. The team sought feedback from county MHSA coordinators and mental health service directors who were most familiar with local mental health data about the accuracy of particular data (i.e., if the demographic distribution pulled from the state datasets for their particular county seemed correct). The evaluation team developed an outcomes report for targeted data that was distributed to representatives in each county. The committee revised the document for brevity such that representatives would note only whether data were “accurate” or “inaccurate.” A text field was included for any explanation of why data was deemed “inaccurate.” County representatives were asked to respond to non-missing data; “unknown,” “missing,” and blank fields were grouped into one category.3 Recruitment County representatives received an e-mail alert about the incoming report and the evaluation team’s goals. Subsequently, the team distributed county-specific reports via e-mail with an invitation to complete the review by May 4, 2012. The evaluation team distributed .pdf versions of the reports for representatives to review with their data teams but asked representatives to enter their final responses online at a link provided in the report. Counties that were not enrolled in statewide reporting were asked to provide a download of specific data for the years specified. Although the importance of county-level feedback was stressed, neither the MHSOAC ad-hoc committee nor the evaluation team mandated participation. Instead, the consequences of spotty participation from county representatives were noted in the invitation. The text included in the report introduction is as follows: We hope to get responses from all counties. At the very least we hope to get responses from a sufficient number and variety of counties so that the data in the statewide report is representative. To determine that a sufficient representation is achieved, participating counties must commit to participation by April 16, 2012. If we do not get a representative sample of counties, we have been asked to move forward with the statewide report using all the data available from both CSI and DCR (confirmed and unconfirmed). We will also be producing county level reports and will use available confirmed and unconfirmed DCR and CSI data. Again we are hoping every county participates and returns this profile indicating the quality of the data they submitted to the state system. Data Quality Assurance Report Outcomes 3 Cowles, E. L., Harris, K., Larsen, C., and Prince, A. (2010). Assessing Representativeness of the Mental Health Services Consumer Perception Survey. 9 Twenty-eight of 59 total counties and municipalities provided responses within six weeks of receiving their Data Quality Assurance Report. Responding counties represented a broad cross- section of the state population, accounted for substantial proportions of most MHSA regions, and represented the state’s racial and ethnic diversity. (For descriptive analysis of counties/municipalities represented in the quality assurance exercise, see Appendix C.) Stakeholder feedback to previous reports identifying data sources for the statewide MHSA evaluation and feedback to the county-specific Data Quality Assurance Reports were generally consistent. Responses across counties indicated that the majority of fields were accurate. However, a few fields, such as race and ethnicity, received much more inconsistent evaluations of accuracy. Feedback about data quality was a factor in final decisions about what to use from state databases. Strategy Assessment The data quality assurance exercise was an effort by the ad-hoc committee and the evaluation team to identify data that could provide accurate insight about priority indicators. The process was important as the group attempted the first round of calculations; data deemed “accurate” by county representatives, and ultimately used in the draft priority indicators report, was vital to refining calculations. Although the method was well-intentioned, it could not be fully realized because of limited participation from counties; counties that did not participate could not be authentically represented. As a result, the MHSOAC ad-hoc committee redirected the evaluation team to incorporate all data in state databases needed to calculate priority indicators based on the participation rate of counties during the data quality assurance exercise. Without means to ensure that all counties participate, this particular exercise will not be involved in future report development. Priority Indicators Evaluated4 The set of priority performance indicators evaluated in this report were arrived at through the following processes: The careful consideration of the California Mental Health Planning Council and approval of the MHSOAC;5 Consideration of the MHSOAC goal of developing a comprehensive outcome and performance monitoring system built upon existing data systems; Consideration of consumer feedback to previous evaluation team reports regarding proposed priority indicators (e.g., “Defining Priority Indicators”); Review of existing data sources to assess their suitability for supporting outcome and performance monitoring through priority indicators (see Review of Existing Data, above); and County feedback regarding the quality and completeness of key data fields necessary to calculate priority performance indicators (see Summary of data “verification” process, above). Through these evaluation processes and careful deliberation of the MHSOAC in collaboration with the evaluation team, a set of 12 priority performance indicators was developed. These indicators can be categorized as those intended to provide insight into the outcomes of mental health 4 Although we received strong indicator suggestions from stakeholders, this report helps vet the appropriateness of the original set proposed by the California Mental Health Planning Council. If the MHSOAC chooses, it may vet additional indicators, particularly those proposed by stakeholders, when revising the pool. 5 California Mental Health Planning Council (January, 2010). Performance Indicators for Evaluating the Mental Health System. 10 consumers (“Consumer Indicators”) and those intended for monitoring the performance of the community mental health system more broadly (“System Indicators”). Consumer and system indicators, and the consumer groups they assess, are summarized in the table below. Priority Indicators CONSUMERS EVALUATED SERVICE OLDER CHILDREN TAY ADULTS POPULATION ADULTS CONSUMER INDICATORS Indicator 1 – Average School Attendance Per Year All/FSP Consumers x x Indicator 2 – Employed Consumers All/FSP Consumers x x x Indicator 3 – Homelessness and Housing Rates All/FSP Consumers x x x x Indicator 4 – Arrest Rate All/FSP Consumers x x x x SYSTEM INDICATORS Indicator 5 – Demographic Profile of Consumers Served All/FSP Consumers x x x x Indicator 6 – Demographic Profile of New Consumers All/FSP Consumers x x x x Indicator 7 – Penetration of Mental Health Services All Consumers x x x x Indicator 8 – Access to a Primary Care Physician FSP Consumers x x x x Indicator 9 – Perceptions of Access to Services All Consumers x x x x Indicator 10 – Involuntary Status All Consumers x x x x Indicator 11 – Consumer Well-Being All Consumers x x x x Indicator 12 – Satisfaction All Consumers x x x x Criteria used to evaluate priority indicators Specific criteria, developed in collaboration with the MHSOAC, were established to evaluate priority performance indicators. These criteria, outlined for consumer and system indicators below, reflect the goals of the MHSOAC for monitoring consumer outcomes and community mental health system performance at multiple levels (i.e., state and county) for the purposes of planning and quality improvement. These criteria may include: Consumer Indicator Evaluation Criteria: Indicator can describe changes in consumer outcomes (e.g., change since initiation of services) or describe the current status of consumers. Indicator can provide meaningful and relevant insight into the outcomes of service populations of interest (e.g., all mental health consumers, FSP consumers, and demographic groups). Indicator can provide meaningful and relevant insight into the outcomes of consumers statewide and at the county level. Indicator provides “actionable” insight, which stakeholders can use to identify areas for service improvement. System Indicator Evaluation Criteria: Indicator can describe meaningful changes in system performance over time. Indicator can provide meaningful and relevant insight regarding the extent and quality of services provided to populations of interest (e.g., all mental health consumers, FSP consumers, and demographic groups). Indicator can provide meaningful and relevant insight into the performance of the community mental health system at the statewide and county levels. Indicator provides “actionable” insight, which stakeholders can use to identify areas for improving the performance of the mental health system. 11 The application of consumer and system indicator evaluation criteria to each priority indicator is detailed in the Priority Indicator Analysis and Findings section below. Report Organization The remainder of the report summarizes each indicator and its outcomes, calculated using select statewide data from FYs 2008-09 and 2009-10. First, the evaluation team presents individual-level priority indicators (those specific to consumers), followed by a discussion and summary of these indicators. The team then does the same for system-level priority indicators (pertaining to community mental health systems throughout the state). A summary table precedes each indicator and its outcomes to orient the reader to what the indicator measures, how it was calculated, and its usefulness. Following a review of all indicators, the evaluation team describes stakeholder feedback and considerations. The report ends with an outline of the team’s next steps in the evaluation. Indicator summary tables Each priority indicator is introduced and summarized in a concise and organized table in the Priority Indicator Analysis and Findings section below. Indicator summary tables are organized into the following sections: Indicator Summary – Provides a brief definition of the indicator Indicator Calculation – Details the computation used to produce the indicator Data Sources – Specifies the data sources and relevant data fields (variables) used to compute the indicator Review of Existing Data – Review of data quality criteria (specified in the Review of Existing Data section above) as applied to indicator-specific data fields Analytic Potential of Indicator – Review of indicator evaluation criteria (specified in the section on Criteria used to evaluate priority indicators above) Note regarding indicator data displays Each indicator is presented through one or more graphical displays of information. These displays include figures (e.g., bar graphs) and tables of frequencies and percentages. For ease of viewing and interpreting data displays specific to a service population, figures that display indicator information relevant to all mental health consumers are presented in blue, and those relevant to FSP consumers are presented in green. Consistent with reporting practices associated with this work, counties with populations smaller than 200,000 residents are grouped in outcomes displays. This group is referred to as “small counties.” County-specific reports will be created for “small counties” on request. Analyses are conducted where data is present, even if data is minimal. Data cells are blacked out where absolutely no information is available within a county to support a calculation for the priority indicator. 12 Priority Indicators Analysis and Findings: Consumer Indicators Priority Indicator 1: Attendance 1.1 Expulsions and Suspensions Per Year (CPS) Indicator Summary This indicator provides descriptive information regarding the number of youth (children and TAY) expelled and suspended from school during the 2008-09 fiscal year. This indicator illustrates a subset of mental health service consumers who responded to consumer perceptions surveys. Indicator Calculation The number children and TAY who had been enrolled for at least six months and reported being expelled or suspended from school since beginning mental health services or 12 months prior to beginning such services. Note: Data is reported from Youth Satisfaction Surveys (YSS) Data Sources Consumer Perception Surveys (CPS) Data Fields: HowLong, LES12AREST, LES12PSTAREST Review of Existing Data Data sources likely to be sustained Data relevant to populations of interest Amount of missing data for child age group is approximately 16% Amount of missing data for TAY age group is approximately 11% Analytic Potential of Indicator Analysis across time possible with the exception of 2009-10 data Analysis among specific service populations possible State- and county-level analysis possible with the exception of 2009-10 data 13 Figure 1.1-1. Proportion of children who were suspended or expelled prior to beginning services and after receiving services for a period of 6 – 12 months during FY 2008-09 100% 90% 80% 73.7% 71.9% 70% 60% 50% No 40% Yes 28.1% 26.3% 30% 20% 10% 0% Exp/Sus 12 mon prior to Exp/Sus since beginning services services (n = 118) (n = 121) Unknown/missing for FY 2008-09: Sus/Exp prior to services = 16.3% (n =23) Sus/Exp since receiving services = 14.2% (n =20) Figure 1.1-2. Proportion of TAY who were suspended or expelled prior to beginning services and after receiving services for a period of 6 – 12 months during FY 2008-09 100% 90% 84.9% 80% 70.8% 70% 60% 50% No 40% Yes 29.2% 30% 20% 15.1% 10% 0% Exp/Sus 12 mon prior to Exp/Sus since beginning services services (n = 72) (n = 73) Unknown/missing for FY 2009-10: Sus/Exp prior to services = 11% (n =9) Sus/Exp since receiving services = 9.9% (n =8) 14 1.2 Average School Attendance Per Year (FSP) Indicator Summary This indicator provides descriptive information regarding the frequency for which Full Service Partnership consumers (children and TAY) attended school during the 2008-09 and 2009-10 fiscal years. Indicator Calculation The number children and TAY who attended school always, mostly, sometimes, infrequently, and never divided by the number of children for which there were data. The number of TAY who attended school always, mostly, sometimes, infrequently, and never divided by the number of TAY for which there were data. Note: Age groupings were revised such that: Child ages = 1-15 (same as previously) TAY ages = 16-18 (16-25 previously) The TAY age group was revised because education variables would be less clear for clients older than 18. Data Sources DCR (PAF - NONRES) Data Field: AttendanceCurr Review of Existing Data Data sources likely to be sustained Data relevant to populations of interest (FSPs) Amount of missing data for child age group is approximately 2% Amount of missing data for TAY age group is approximately 22% Analytic Potential of Indicator Analysis across time possible Analysis among specific service populations possible State- and county-level analysis possible 15 Figure 1.2-1 –The frequency with which children and TAY attended school during FY 2008- 09 (DCR) 100% 90% 80% 70% 60% 50% Child (n = 64) 40% 28.1% TAY (n = 49) 30% 26.5% 25.0% 28.6% 21.9% 20.4% 20.3% 20% 14.3% 10.2% 10% 4.7% 0% Always Attends Sometimes Infrequently Never attends school most attends attends attends school of the time school school school Unknown/missing values for FY 2008-09: Child = 1.5% (n =1), TAY = 22.2% (n =14) Figure 1.2-2 –The frequency with which children and TAY attended school during FY 2009- 10 (DCR) 100% 90% 80% 70% 60% 50% Child (n = 38) 40% 36.0% 27.9% TAY (n = 41) 30% 28.1% 22.8% 19.3% 20% 15.8% 16.3% 16.3% 14.0% 10% 3.5% 0% Always Attends Sometimes Infrequently Never attends school most attends attends attends school of the time school school school Unknown/missing values for FY 2009-10: Child = 1.1% (n =1), TAY = 32.1% (n =27) 16 Priority Indicator 2: Employment Indicator Summary This indicator provides the proportion of TAY, adults and older adults who are employed (paid and non-paid) and not employed as recorded during the most recent update (second date of service). This indicator provides descriptive information regarding clients’ employment status during their first date of service. Indicator Calculation Client & Service Information (CSI) The number of paid employed clients divided by the total number of TAY, adults, and older adults for whom there were employment data. The number of nonpaid employed clients divided by the total number of TAY, adults, and older adults for whom there were employment data. The number of paid nonemployed clients divided by the total number of TAY, adults, and older adults for whom there were employment data. Note: There were multiple periodic updates for clients within each fiscal year. These ratios provide information for those who had a second periodic update within a given fiscal year (or a second date of service). Additionally, the age groupings were revised to capture those truly eligible for employment. Those who indicated they were retired or incarcerated were excluded from calculations. Data Collection and Reporting (DCR) The number of TAY, adults, and older adults who reported paid employment divided by the total number of TAY, adults, and older adults. The number of TAY, adults, and older adults who reported nonpaid employment divided by the total number of TAY, adults, and older adults. The number of TAY, adults, and older adults who did not report any employment divided by the total number of TAY, adults, and older adults. Note for CSI and FSP data: Age groupings were revised such that TAY ages = 18-25 (previously 16-25) Older adults = 60-65 (60 and up previously) Data Sources CSI Periodic Post-dig, Data Field: Employment Status DCR (PAF - NONRES) Data Fields: Current_CompetitiveAvgHrWeek, Current_SupportedAvgHrWeek, Current_TransitionalAvgHrWeek, Current_In-HouseAvgHrWeek, Current_OtherEmploymentAvgHrWeek, Current_Non-paidAvgHrWeek Review of Existing Data Client & Service Information (CSI) Data sources likely to be sustained Data relevant to populations of interest (all consumers) Data available across multiple service years Amount of missing data for TAY age group is approximately 38% 17 Amount of missing data for Adult age group is approximately 48% Amount of missing data for Older Adult age group is approximately 57% Data Collection and Reporting (DCR) Data sources likely to be sustained Data relevant to populations of interest (FSPs) The amount of missing data for these ratios is unknown given how the employment data are collected. There is no data code option for “missing;” as a consequence, blank responses are either missing or not applicable. Analytic Potential of Indicator Client & Service Information (CSI) Analysis across time possible but very difficult Analysis among specific service populations not possible State- and county-level analysis possible Data Collection and Reporting (DCR) Analysis across time possible Analysis among specific service populations possible State- and county-level analysis possible 18 Figure 2.1 - Proportion of clients who were employed and not employed as reported during their second service date for FY 2008-09 (CSI) Unknown/Missing for FY 2008-09: TAY = 38.4% (n =58), Adult = 48% (n =248), Older adult = 57% (n =16) Figure 2.2 – Proportion of clients who were employed and not employed as reported during their second service date for FY 2009–10 (CSI) 100.0% 90.0% 80.0% 70.0% 60.0% 93.6% Not employed 50.0% 100.0% 100.0% Non-paid 40.0% Paid 30.0% 20.0% 0.5% 10.0% 0.0% 0.0% 0.0% 5.9% 0.0% 0.0% TAY (n = 73) Adult (n = 220) Older adult (n = 9 ) Unknown/Missing for FY 2009-10: TAY = 47% (n =65), Adult = 43.6% (n =170), Older adult = 47% (n =8) 19 Figure 2.3 –The proportion of FSPs who were employed during FY 2008–09 (DCR) Unknown/Missing for FY 2008-09: Unknown Figure 2.4 –The proportion of FSPs who were employed during FY 2009–10 (DCR) Unknown/Missing for FY 2009-10: Unknown 20 Priority Indicator 3: Homelessness and Housing Rates Indicator Summary This indicator summarizes the housing status of all mental health consumers and FSPs served during FYs 2008-09 and 2009-10. There are two parts: (a) a breakdown by most recently available housing status and (b) the percentage of consumers experiencing homelessness at any point during the year. Indicator Calculation Frequencies of the most recent housing statuses were computed for mental health and FSP consumers served in FYs 2008-09 and 2009-10. This calculation excludes consumers with no housing data within the given FYs or consumers whose most recent status was homeless. The percentages of mental health and FSP consumers who experienced homelessness at any point during the given FY were also computed. Note that a consumer who was most recently homeless would not be included in the first indicator for most recent housing status, whereas a consumer who was previously homeless and more recently reported as not homeless would be included. Data Sources Client & Service Information (CSI): H-01.0 County / City / Mental Health Plan Submitting Record; H-02.0 County Client Number; C-03.0 Date of Birth; P-01.0 Date Completed; P-09.0 Living Arrangement Data Collection and Reporting (DCR) Key Event Tracking (KET): 1.01 Global ID; 1.02 Assessment ID; 1.07 Age Group; 3.01 CountyID; 3.06 Assessment Date; 5.01 DateResidentialChange; 5.02 Current Review of Existing Data These data were taken from the Key Event Tracking (KET) updates for FSP consumers and the periodic updates for all mental health consumers, limited to the given fiscal year. Any consumer who did not have an update available during the year was not included. Data sources are likely to be sustained in the foreseeable future, providing a consistent source for tracking system performance moving forward. Taking a conservative approach, we considered cases without valid data “missing.” It should be noted that the data reporting and collection practices currently in place do not allow for a distinction between missing data from unreported changes in housing status and blank values from standard data entry practices. This is especially notable in the KET updates for FSP consumers, leading to large percentages of “missing” data. These results should be interpreted cautiously. In particular, there is the risk of systematic bias in underreporting certain housing statuses. Analytic Potential of Indicator Data across service years support analysis of the distribution and change of housing statuses, including homelessness, among consumers. Indicator Displays The first set of charts displays the most recently reported non-homeless housing statuses of consumers, by percentage, during each fiscal year. The second set displays the percentages of consumers who were reported as experiencing homelessness at any time during the fiscal year. 21 Figure 3.1 – Most recent housing status excluding homelessness, all consumers (CSI) 100% 13.0% 10.8% 90% 0.0% 26.4% 30.0% 25.8% 28.2% 0.0% 80% 41.4% 37.7% 0.0% 70% 0.0% 16.6% 60% 16.3% 10.9% 7.8% 50% 87.0% 89.2% 40% 74.2% 71.8% 30% 57.1% 53.8% 50.7% 51.4% 20% 10% 0% Child Child TAY 08-TAY 09- Adult Adult Older Older 08-09 09-10 09 10 08-09 09-10 Adult Adult 08-09 09-10 House or Apartment Foster Care Group Setting Unknown/Missing for FY 2008-09 = 4.1% (n =7) for children; 9.1% (n =27) for TAY; 19.8% (n =124) for adults; and 13.6% (n =9) for older adults Unknown/Missing for FY 2009-10 = 5.9% (n =5) for children; 9.8% (n =24) for TAY; 22.3% (n =108) for adults; and 10.9% (n =5) for older adults Figure 3.2 – Most recent housing status excluding homelessness, FSP consumers only (DCR) 100% 90% 80% 44.4% 50.0% 70% 59.0% 67.2% 66.7% 73.1% 60% 85.7% 87.5% 5.6% 50% 0.0% 0.0% 40% 0.0% 14.5% 30% 10.3% 0.0% 50.0% 50.0% 20% 26.4% 18.3% 10% 22.4% 26.5% 0.0% 14.3% 12.5% 6.9% 8.7% 0% 0.0% 0.0% Child Child TAY 08-TAY 09- Adult Adult Older Older 08-09 09-10 09 10 08-09 09-10 Adult Adult 08-09 09-10 With Family Independent Foster Care Group Setting Unknown/Missing for FY 2008-09 = 66.7% (n =28) for children; 41.2% (n =42) for TAY; 26.5% (n =67) for adults; and 42.9% (n =6) for older adults Unknown/Missing for FY 2009-10 = 52.5% (n =21) for children; 34.1% (n =47) for TAY; 31.8% (n =104) for adults; and 41.4% (n =12) for older adults 22 Figure 3.3 – Experienced homelessness at any point during the year, all consumers (CSI) 30% 25% 20% 15% FY 2008-2009 FY 2009-2010 9.8% 9.0% 10% 7.0% 5.3% 5% 1.1% 0.9% 0.0% 0.0% 0% Child TAY Adult Older Adult Unknown/Missing for FY 2008-09 = 4.1% (n =7) for children; 9.1% (n =27) for TAY; 19.8% (n =124) for adults; and 13.6% (n =9) for older adults Unknown/Missing for FY 2009-10 = 5.9% (n =5) for children; 9.8% (n =24) for TAY; 22.3% (n =108) for adults; and 10.9% (n =5) for older adults Figure 3.4 – Experienced homelessness at any point during the year, FSP consumers only (DCR) 30% 25.0% 25% 19.8% 19.3% 20% 18.3% 16.7% 15.8% 15% FY 2008-2009 FY 2009-2010 10% 7.1% 5.9% 5% 0% Child TAY Adult Older Adult Unknown/Missing for FY 2008-09 = 66.7% (n =28) for children; 41.2% (n =42) for TAY; 26.5% (n =67) for adults; and 42.9% (n =6) for older adults Unknown/Missing for FY 2009-10 = 52.5% (n =21) for children; 34.1% (n =47) for TAY; 31.8% (n =104) for adults; and 41.4% (n =12) for older adults 23 Priority Indicator 4: Arrest Rates Indicator Summary This indicator provides the proportion of youth, adults, and older adults who reported being arrested 12 months prior to receiving services and the proportion of youth, adults, and older adults who reported being arrested since beginning services. For calculations involving consumer perception surveys, this indicator includes only youth, adults, and older adults who reported receiving services for 6 to 12 months. This indicator provides information regarding whether the proportion of arrested clients has increased or decreased after 6 to 12 months of service. For calculations involving Full Service Partnership consumers, this indicator tracks arrests prior to enrollment using intake data. This indicator accounts for consumers enrolled during the target fiscal years for which PAF surveys are available. Indicator Calculation Consumer Perception Surveys (CPS) The number of reported arrest 12 months prior to services divided by the total number of youth, adults, and older adults for who there was data The number of reported arrest since beginning services divided by the total number of youth, adults, and older adults for who there was data Note: Clients were surveyed multiple times during the 2008 – 2009 fiscal year. However, only one survey administration was used to get both the proportion of clients who reported being arrested 12 months prior to beginning services and since receiving services. Age groupings are as follows: Youth, 1 – 25 years Adult, 26 – 59 years Older adult, 60 and above Data Collection and Reporting (DCR) The number of youth (children and TAY), adults, and older adults reporting arrests 12 months prior to enrollment divided by the total number of unique clients for who there was data The number of youth (children and TAY), adults, and older adults reporting arrests 12 months prior to the past 12 months divided by the total number of unique clients for who there was data Note: In rare cases where two surveys were entered for one client, only the earliest entry was used in calculations. Data Sources Consumer Perception Survey (CPS) for Youth, Adults, and Older Adults Data Fields: HowLong, les12arest, les12pstarest Data Collection and Reporting (DCR PAF NONRES): Age_Group, ArrestPast12, ArrestPrior12 Review of Existing Data Consumer Perception Surveys (CPS) Data sources likely to be sustained Data relevant to populations of interest Approximately 11% missing or unknown values for youth Approximately 9.4% missing or unknown values for adult Approximately 14.3% missing or unknown values for older adult 24 Data Collection and Reporting (DCR) Data sources likely to be sustained Data relevant to populations of interest On average, 1.1% missing or unknown values for children On average, 0.7% missing or unknown values for TAY On average, 4.5% missing or unknown values for adults On average, 0.0% missing or unknown values for older adults Analytic Potential of Existing Data For both data sources Analysis across time possible Analysis among specific service populations not possible State and county individual level analysis possible Note: As of the submission of this report, a new calculation has been proposed to examine arrest rates. The proposed calculation would use FSP-DCR data during consumers’ enrollment (not intake as it is presented here). An updated indicator will be available shortly. 25 Figure 4.1 – Proportion of youth who were arrested prior to beginning services and since receiving services Missing/unknown for Arrested 12 months prior to services = 11.4% (n =26) Missing/unknown for Arrested since receiving services = 10.5% (n =24) Figure 4.2 – Proportion of adults who were arrested prior to beginning services and since receiving services Missing/unknown for Arrested 12 months prior to services = 9.4% (n =14) Missing/unknown for Arrested since receiving services = 9.4% (n =14) 26 Figure 4.3 – Proportion of older adults who were arrested prior to beginning services and since receiving services Missing/unknown for Arrested 12 months prior to services = 14.3% (n =3) Missing/unknown for Arrested since receiving services = 14.3% (n =3) Figure 4.4 - Proportion of children who were arrested within the past 12 months (DCR) Unknown/Missing for FY 2008-09 = 2.2% (n=1) Unknown/Missing for FY 2009-10 = 0.0% (n=0) 27 Figure 4.4 - Proportion of TAY who were arrested within the past 12 months (DCR) Unknown/Missing for FY 2008-09 = 1.4% (n=1) Unknown/Missing for FY 2009-10 = 0.0% (n=0) Figure 4.5 – Proportion of adults who were arrested within the past 12 months (DCR) Unknown/Missing for FY 2008-09 = 0.7% (n=1) Unknown/Missing for FY 2009-10 = 8.2% (n=13) 28 Figure 4.6 Proportion of older adults who were arrested within the past 12 months (DCR) Unknown/Missing for FY 2008-09 = 0.0% (n=0) Unknown/Missing for FY 2009-10 = 0.0% (n=0) 29 Priority Indicators Analysis and Findings: Community Mental Health System Indicators Priority Indicator 5: Demographic Profile of Consumers Served Indicator Summary This indicator profiles the demographics (race/ethnicity, age, and gender) of all mental health consumers and Full Service Partnership consumers served during FYs 2008-09 and 2009-10, in Santa Clara County. Indicator Calculation The frequencies of all mental health consumers and FSP consumers served in FYs 2008-09 and 2009-10 were calculated overall. Additionally, the proportion of consumers represented by race/ethnicity, age, and gender categories was calculated by dividing the number of consumers within each demographic category by all consumers served. Proportions were calculated for both service populations (all consumers and FSPs) and both fiscal years examined (see Figures 5.1-5.6 below). Data Sources Client & Service Information (CSI) Data Fields: H-01.0 County / City / Mental Health Plan Submitting Record; H- 02.0 County Client Number; C-05.0 Gender; C-09.0 Ethnicity; C-10.0 Race; S-05.0 Mode of Service; S-16.0 From / Entry Date; S-17.0 Through / Exit Date; S-23.0 Date of Service. Data Collection and Reporting (DCR) Data Fields: 1.01 Global ID; 1.02 Assessment ID; 1.04 Date Partnership Status Change; 1.05 Partnership Status; 1.07 Age Group; 1.08 Assessment Type; 2.01 CSI Date of Birth; 2.02 Gender; 2.03 CSIRace1; 2.04 CSIRace2; 2.05 CSIRace3; 2.06 CSIRace4; 2.07 CSIRace5; 2.10 CSI Hispanic; 3.01 County ID; 3.05 Partnership Date; 3.06 Assessment Date. Review of Existing Data Data sources likely to be sustained Data relevant to populations of interest (all consumers and FSPs) Data available across multiple service years More than 10% missing or unknown values among demographic variables Analytic Potential of Indicator Analysis across time possible Analysis among specific service populations possible (e.g., all consumers, FSPs, demographic groups) 30 Figure 5.1. Race/ethnicity of mental health consumers 100.0% 90.0% 80.0% 70.0% 60.0% 50.0% 39.8% 39.0% 40.0% 24.6% 30.0% 24.0% 19.0% 17.5% 20.0% 10.0% 6.7% 6.8% 5.3% 5.1% 5.2% 5.1% 0.2% 0.2% 0.8% 0.7% 0.0% White Hispanic/Latino Asian Pacific Islander Black American Multirace Other Indian FY 2008-09 (8,695) FY 2009-10 (7,085) FY 2008-09 Unknown/Missing = 6.3% (n =583); FY 2009-10 Unknown/Missing = 6.3% (n =475) Figure 5.2. Race/ethnicity of FSP consumers 100.0% 90.0% 80.0% 70.0% 60.0% 50.0% 38.5% 36.8% 40.0% 29.6% 29.3% 30.0% 20.0% 7.5% 9.9% 10.9% 10.9% 8.0% 8.5% 10.0% 0.3% 0.3% 2.3% 1.6% 2.9% 2.7% 0.0% White Hispanic/Latino Asian Pacific Islander Black American Multirace Other Indian FY 2008-09 (348) FY 2009-10 (375) FY 2008-09 Unknown/Missing =38.9% (n =222); FY 2009-10 Unknown/Missing = 47.8% (n =343) 31 Figure 5.3. Mental health consumers by age group 100.0% 90.0% 80.0% 70.0% 62.6% 63.8% 60.0% 50.0% 40.0% 30.0% 14.5% 15.6% 20.0% 10.8% 8.8% 12.1% 11.8% 10.0% 0.0% Children TAY Adults Older Adults FY 2008-09 (9,278) FY 2009-10 (7,560) Figure 5.4. FSP consumers by age group 100.0% 90.0% 80.0% 70.0% 60.0% 58.5% 60.0% 50.0% 40.0% 25.1% 30.0% 23.3% 20.0% 11.8% 12.4% 4.9% 4.0% 10.0% 0.0% Children TAY Adult Older Adult FY 2008-09 (570) FY 2009-10 (710) 32 Figure 5.5. Mental health consumers by gender 100.0% 90.0% 80.0% 70.0% 60.0% 49.3% 48.3% 50.7% 51.7% 50.0% 40.0% 30.0% 20.0% 10.0% 0.0% Female Male FY 2008-09 (9,278) FY 2009-10 (7,559) Figure 5.6. FSP consumers by gender 100.0% 90.0% 80.0% 70.0% 58.9% 61.7% 60.0% 50.0% 41.1% 38.3% 40.0% 30.0% 20.0% 10.0% 0.0% Female Male FY 2008-09 (399) FY 2009-10 (420) FY 2008-09 Unknown/Missing = 30.0% (n =171); FY 2009-10 Unknown/Missing = 41.5% (n =298) 33 Priority Indicator 6: Demographic Profile of New Consumers Indicator Summary This indicator profiles new mental health consumers (i.e., served during FY, without service for prior six months) overall and full service partners (FSPs) served during FYs 2008-09 and 2009-10. Indicator Calculation For all mental health consumers, CSI data support calculation of new (i.e., did not receive services for 6 months prior to given FY) versus past consumers (i.e., initial services received prior to the given FY) overall and within race/ethnicity, age, and gender categories. The frequency of new consumers served was divided by all previous consumers served, in each fiscal year, to calculate the proportion of new consumers served. This same calculation was conducted within each demographic category (race/ethnicity, age, and gender), in each FY (see Figures 6.1 – 6.4 below). For FSPs, DCR data support calculation of new (i.e., did not receive services for 6 months prior to given FY) versus existing (i.e., current Full Service Partners) overall and within race/ethnicity, age, and gender categories. The frequency of new consumers served was divided by all existing consumers, in each fiscal year, to calculate the proportion of new consumers served. This same calculation was conducted within each demographic category (race/ethnicity, age, and gender), in each FY (see Figures 6.5 – 6.8, below). Data Sources Client & Service Information (CSI) Data Fields: H-01.0 County / City / Mental Health Plan Submitting Record; H- 02.0 County Client Number; C-05.0 Gender; C-09.0 Ethnicity; C-10.0 Race; S-05.0 Mode of Service; S-16.0 From / Entry Date; S-17.0 Through / Exit Date; S-23.0 Date of Service. Data Collection and Reporting (DCR) Data Fields: 1.01 Global ID; 1.02 Assessment ID; 1.04 Date Partnership Status Change; 1.05 Partnership Status; 1.07 Age Group; 1.08 Assessment Type; 2.01 CSI Date of Birth; 2.02 Gender; 2.03 CSIRace1; 2.04 CSIRace2; 2.05 CSIRace3; 2.06 CSIRace4; 2.07 CSIRace5; 2.10 CSI Hispanic; 3.01 County ID; 3.05 Partnership Date; 3.06 Assessment Date. Review of Existing Data Data sources likely to be sustained Data relevant to populations of interest (all consumers and FSPs) Data available across multiple service years More than 10% missing or unknown values among some demographic fields Analytic Potential of Indicator Analysis across time possible Analysis among specific service populations possible (e.g., all consumers, FSPs, demographic groups) 34 All Consumers – Data Source: Client & Service Information (CSI) Figure 6.1. New and continuing mental health consumers 96.1% 100.0% 94.0% 90.0% 80.0% 70.0% 60.0% 50.0% 40.0% 30.0% 20.0% 6.0% 3.9% 10.0% 0.0% FY 2008-09 (15,886) FY 2009-10 (16,257) Previous Consumers New Consumers Figure 6.2. Race/ethnicity of new mental health consumers 100.0% 90.0% 80.0% 70.0% 60.0% 50.0% 39.3% 39.8% 40.0% 30.7% 32.7% 30.0% 20.0% 6.5% 7.6% 10.3% 9.2% 10.3% 8.7% 10.0% 0.3% 0.2% 0.9% 0.8% 1.6% 1.0% 0.0% White Asian Black Multirace FY 2008-09 (911) FY 2009-10 (596) FY 2008-09 Unknown/Missing = 4.6% (n =44); FY 2009-10 Unknown/Missing = 4.0% (n =25) 35 Figure 6.3. New mental health consumers by age group 100.0% 90.0% 80.0% 70.0% 60.0% 49.2% 54.2% 50.0% 40.0% 30.0% 26.0% 19.5% 17.9% 23.0% 20.0% 10.0% 5.3% 5.0% 0.0% Children TAY Adults Older Adults FY 2008-09 (955) FY 2009-10 (626) Figure 6.4. New mental health consumers by gender 100.0% 90.0% 80.0% 70.0% 60.0% 48.3% 49.0% 51.7% 51.0% 50.0% 40.0% 30.0% 20.0% 10.0% 0.0% Female Male FY 2008-09 (955) FY 2009-10 (626) FY 2008-09 Unknown/Missing = 0.0% (n =0); FY 2009-10 Unknown/Missing = 0.2% (n =1) 36 FSP Consumers – Data Source: Data Collection and Reporting (DCR) Figure 6.5. New and continuing FSP consumers 100.0% 90.0% 80.0% 70.0% 58.8% 60.0% 52.6% 47.4% 50.0% 41.2% 40.0% 30.0% 20.0% 10.0% 0.0% FY 2008-09 (570) FY 2009-10 (718) Previous Consumers New Consumers Figure 6.6. Race/ethnicity of new FSP consumers 100.0% 90.0% 80.0% 70.0% 60.0% 50.0% 38.5% 36.8% 40.0% 29.6% 29.3% 30.0% 20.0% 7.5% 9.9% 10.9% 10.9% 8.0% 8.5% 10.0% 0.3% 0.3% 2.3% 1.6% 2.9% 2.7% 0.0% White Asian Black Multirace FY 2008-09 (348) FY 2009-10 (375) FY 2008-09 Unknown/Missing = 38.9% (n =222); FY 2009-10 Unknown/Missing = 47.8% (n =343) 37 Figure 6.7. New FSP consumers by age group 100.0% 90.0% 80.0% 70.0% 60.0% 58.5% 60.0% 50.0% 40.0% 25.1% 30.0% 23.3% 20.0% 11.8% 12.4% 4.9% 4.0% 10.0% 0.0% Children TAY Adults Older Adults FY 2008-09 (570) FY 2009-10 (718) Figure 6.8. New FSP consumers by gender 100.0% 90.0% 80.0% 61.7% 70.0% 58.9% 60.0% 50.0% 41.1% 38.3% 40.0% 30.0% 20.0% 10.0% 0.0% Female Male FY 2008-09 (399) FY 2009-10 (420) 38 Priority Indicator 7: Penetration of Mental Health Services Indicator Summary This indicator details rates of service access relative to estimates of need for service among residents of Santa Clara County earning less than 200% of the federal poverty income level. This metric is intended to show the extent to which service access is in line with the level of need for services. Indicator Calculation The number of all mental health consumers served (i.e., at least one service received during FY) was divided by estimates of need for service (Holzer Targets) among residents of Santa Clara County earning less than 200% of the federal poverty income level and among demographic category (i.e., race/ethnicity, age, and gender). (See Figures 7.1-7.4 below). Data Sources Client & Service Information (CSI) Data Fields: H-01.0 County/City/Mental Health Plan Submitting Record; H- 02.0 County Client Number; C-05.0 Gender; C-09.0 Ethnicity; C-10.0 Race; S-05.0 Mode of Service; S-16.0 From / Entry Date; S-17.0 Through / Exit Date; S-23.0 Date of Service. Estimates of need for mental health services (Holzer Targets) among Californians earning less than 200% of the federal poverty income level. Review of Existing Data Data sources likely to be sustained Data appropriate for analysis of all mental health consumers. The estimates of need for service (Holzer Targets) used are not appropriate points of comparison for FSP service levels. Data available across multiple service years Less than 10% missing or unknown values (see Appendix C for details of recoding race/ ethnicity data fields) Analytic Potential of Indicator Analysis across time possible Analysis among specific service populations possible (e.g., all consumers, FSPs, demographic groups) 39 Figure 7.1. Penetration of mental health services 100.0% 80.0% 60.0% 39.3% 31.8% 40.0% 20.0% 0.0% FY 2008-09 FY 2009-10 Consumers Served/Holzer Target FY 2008-09 (9,278/23,637) FY 2009-10 (7,559/23,760) Figure 7.2. Penetration of services by race/ethnicity 300.0% 272.7% 250.0% 200.0% 150.0% 108.1% 100.0% 79.3% 57.5% 60.7% 40.6% 50.0% 29.5% 9.6% 18.0% 6. 9% 10.0% 20.7% 2. 9% 21.4% 9.3% 14.2% 0.0% White Hispanic/Latino Asian Pacific Islander Black Amerian Indian Multirace Other FY 2008-09 FY 2009-10 FY 2008-09 Unknown/Missing = 6.3% (n =583); FY 2009-10 Unknown/Missing = 6.3% (n =475) FY 2008-09 FY 2009-10 White (3,458/6,012) (2,766/28,745) Hispanic/ Latino (2,135/11,894) (1,703/24,622) Black (581/733) (482/2,250) Asian (1,521/3,750) (1,348/13,484) Pacific Islander (19/92) (13/443) American Indian (66/224) (48/517) Multirace (465/766) (363/2,563) Other (450/165) (362/335) 40 Figure 7.3. Penetration of mental health services by age 90.0% 77.4% 80.0% 70.0% 53.3% 60.0% 50.0% 36.7% 40.0% 30.0% 24.4% 20.0% 12.6% 10.6% 15.1% 10.0% 2.4% 0.0% Children TAY Adults Older Adults FY 2008-09 FY 2009-10 FY 2008-09 FY 2009-10 Children (1,001/7,937) (665/27,790) TAY (1,125/3,063) (894/8,444) Adult (5,806/10,899) (4,824/31,906) Older Adult (1,346/1,738) (1,177/4,818) Figure 7.4. Penetration of mental health services by gender 50.0% 46.5% 45.0% 40.0% 33.8% 35.0% 30.0% 25.0% 20.0% 15.0% 11.7% 9.2% 10.0% 5.0% 0.0% FY 2008-09 FY 2009-10 FY 2008-09 FY 2009-10 Female Male FY 2008-09 FY 2009-10 Female (4,573/13,528) (3,651/39,490) Male (4,705/10,109) (3,908/33,468) 41 Priority Indicator 8: Access to a Primary Care Physician Indicator Summary This indicator details the level of access to a primary care physician reported among FSP consumers, during FYs 2008-09 and 2009-10, in Santa Clara County. Indicator Calculation The ratio of FSP consumers indicating access to a primary care physician at any point during a fiscal year to all FSP consumers served during a fiscal year was calculated (see Figure 8.1). This ratio was also calculated within demographic categories (i.e., race/ethnicity, age, and gender) for each FY (see Figures 8.2-8.4 below). Data Sources Data Collection and Reporting (DCR) Data Fields: 1.01 Global ID; 1.02 Assessment ID; 1.04 Date Partnership Status Change; 1.05 Partnership Status; 1.07 Age Group; 1.08 Assessment Type; 2.01 CSI Date of Birth; 2.02 Gender; 2.03 CSIRace1; 2.04 CSIRace2; 2.05 CSIRace3; 2.06 CSIRace4; 2.07 CSIRace5; 2.10 CSI Hispanic; 3.01 County ID; 3.05 Partnership Date; 3.06 Assessment Date; 11.01 PhysicianCurr. Review of Existing Data Data source likely to be sustained Data relevant to population of interest (FSPs). Relevant data not available to assess primary care access among all mental health consumers (e.g., CSI). Data available across multiple service years More than 10% missing or unknown values within “PhysicianCurr” and demographic fields (see Appendix C for details of recoding race/ethnicity data fields) Analytic Potential of Indicator Analysis across time possible Analysis among specific service populations possible (e.g., all consumers, FSPs, demographic groups) 42 Figure 8.1. FSP access to a primary care physician 100.0% 80.0% 68.4% 56.4% 60.0% 40.0% 20.0% 0.0% FY 2008-09 (265) FY 2009-10 (346) Figure 8.2. FSP access to a primary care physician by race/ethnicity 100.0% 90.0% 80.0% 70.0% 60.0% 50.0% 40.0% 38.5% 36.9% 30.0% 28.7% 30.8% 20.0% 10.0% 9.2% 13.1% 9.2% 13.1% 10.3% 2.9% 1.5% 5.7% 6.1% 4.0% 3.0% 0.0% 0.0% White Hispanic/Latino Asian Pacific Islander Black Amerian Indian Multirace Other FY 2008-09 (174) FY 2009-10 (198) FY 2008-09 Unknown/Missing = 34.3% (n =91); FY 2009-10 Unknown/Missing = 42.8% (n =148) 43 Figure 8.3. FSP access to a primary care physician by age group 100.0% 90.0% 80.0% 70.0% 60.0% 55.1% 50.0% 50.0% 40.0% 30.0% 24.3% 20.0% 19.1% 19.2% 17.0% 8.7% 6.6% 10.0% 0.0% Children TAY Adults Older Adults FY 2008-09 (52) FY 2009-10 (53) Figure 8.4. FSP access to a primary care physician by gender 100.0% 90.0% 80.0% 70.0% 60.0% 57.0% 50.0% 49.5% 50.5% 43.0% 40.0% 30.0% 20.0% 10.0% 0.0% Female Male FY 2008-09 (196) FY 2009-10 (221) FY 2008-09 Unknown/Missing = 26.0% (n =69); FY 2009-10 Unknown/Missing = 36.1% (n =125) 44 Priority Indicator 9: Perceptions of Access to Services Indicator Summary This indicator provides insight into consumer and family perceptions of access to mental health services, among a sample of those currently accessing the Santa Clara County mental health system. Indicator Calculation Family members/caregivers and TAY respondents’ ratings (1–Strongly Disagree to 5–Strongly Agree) of two self-report items (specified in the Data Sources section below) were averaged to calculate aggregate ratings of perceptions of access to mental health services (see Figures 9.1-9.2 and Tables 9.1-9.2 below). Aggregate ratings were calculated for each fiscal year. Ratings of 3.5 or greater generally indicate positive perceptions. This calculation method is in line with previous DHCS practices. Adult and Older Adult respondents’ ratings (1-Strongly Disagree to 5-Strongly Agree) of 14 self-report items (specified under the Data Sources section below) were averaged to calculate aggregate ratings of perceptions of access to mental health services (see Figures 9.1-9.2 and Tables 9.1-9.2 below). Aggregate ratings were calculated for each fiscal year. Ratings of 3.5 or greater generally indicate positive perceptions. This calculation method is in line with previous DHCS practices. Data Sources Consumer Perception Surveys Family members/caregivers and TAY self-report items analyzed (YSS/YSS-F): o The location of services was convenient for us. o Services were available at times that were convenient for us. Adult and older adult self-report items analyzed (MHSIP): o The location of services was convenient (parking, public transportation, distance, etc.). o Staff were willing to see me as often as I felt it was necessary. o Staff returned my call in 24 hours. o Services were available at times that were good for me. o I was able to get all the services I thought I needed. o I was able to see a psychiatrist when I wanted to. Note: Data collected in FYs 2008-09 and 2009-10 must be interpreted separately because a convenience sampling method was employed to gather FY 2008-09 data and a random sampling method employed to gather data in FY 2009-10. 6 Review of Existing Data Data source likely to be sustained Data relevant to population of interest (i.e., convenience or random sample of all mental health consumers) Data available across multiple service years More than 10% missing or unknown values among demographic groups Analytic Potential of Indicator Analysis across time will be possible if the sampling methodology and instrument used is employed in a consistent manner each year Analysis among specific service populations possible (e.g., all consumers, demographic groups) 6 Cowles, E. L., Harris, K., Larsen, C., and Prince, A. (2010). Assessing Representativeness of the Mental Health Services Consumer Perception Survey. 45 County analysis possible for FY 2008-09 (convenience sample), but data is not available at the county level for FY 2009-10 46 Figure 9.1 – Perceptions of access to services, FY 2008-09 Older Adults 4.27 (n=291) Adults 4.19 (n=2,701) TAY 4.05 (n=1,348) Children 4.46 (n=2,079) 1.00 2.00 3.00 4.00 5.00 Table 9.1. Perceptions of access to services by race/ethnicity, FY 2008-09 Family Member/ TAY Adult Older Adult Caregiver White 4.40 (n=682) 4.03 (n=355) 4.21 (n=1,222) 4.23 (n=170) Hispanic / Latino 4.49 (n=1,209) 4.08 (n=751) 4.26 (n=651) 4.47 (n=48) Asian 4.34 (n=144) 3.93 (n=117) 4.21 (n=292) 4.32 (n=34) Pacific Islander 4.17 (n=38) 4.01 (n=40) 4.29 (n=54) 4.75 (n=2) Black 4.44 (n=223) 4.11 (n=167) 4.22 (n=218) 4.27 (n=14) American Indian 4.45 (n=112) 4.00 (n=114) 4.15 (n=156) 3.98 (n=7) Other 4.49 (n=553) 4.04 (n=430) 4.24 (n=379) 4.54 (n=23) Unknown/Missing Values: Family Member/Caregiver = 9.3% (n =304), TAY = 11.2% (n =221), Adult = 20.3% (n =603), Older Adult = 18.4% (n =67) Table 9.2. New mental health consumers by gender, FY 2008-09 Family Member/ TAY Adult Older Adult Caregiver Female 4.44 (n=735) 4.16 (n=523) 4.27 (n=1,245) 4.24 (n=161) Male 4.47 (n=1,213) 4.00 (n=713) 4.18 (n=1,158) 4.36 (n=93) Unknown/Missing Values: Family Member/Caregiver = 6.7% (n =140), TAY = 9.2% (n =125), Adult = 0.0% (n =13), Older Adult = 12.7% (n =37) 47 Priority Indicator 10: Involuntary Status Indicator Summary This indicator provides insight into the rates of involuntary status among all mental health consumers during FY 2008-09. Involuntary status refers to a legal designation that can be applied to individuals who are found to be a danger to themselves and/or others, and/or gravely disabled. Indicator Calculation The California Department of Health Care Services (DHCS) reports incidents of involuntary status per 10,000 consumers. Such rates are reported here (see Figure 10.1, below). Data Sources The California Department of Health Care Services provides reports of incidents of involuntary status (see http://www.dmh.ca.gov/statistics_and_data_analysis/Involuntary_Detention.asp) Review of Existing Data Data source likely to be sustained Data relevant to population of interest (all mental health consumers). Relevant data are not available to specifically assess involuntary status among FSP consumers. Data available across multiple service years Analytic Potential of Indicator Analysis across time will be possible as information from additional fiscal years becomes available from DHCS Aggregate data do not allow for analysis among specific (e.g., demographic) service populations 48 Table 10.1. Involuntary status per 10,000 consumers, FY 2008-09 (NOTE: horizontal scale reduced for ease of viewing) Additional 14-day Intesive (suicidal) 0.0 14-day Intensive Treatment 8.8 72-Hour Evaluation and Treatment - Child 0.0 72 Hour Evaluation and Treament - Adult 0.0 0 10 20 30 40 50 60 70 80 90 100 49 Priority Indicator 11: Consumer Perceptions of Improvement in Well- Being as a Result of Services Indicator Summary This indicator provides insight into consumer and family perceptions of well-being (i.e., outcomes, functioning, and social connectedness) as a result of mental health services. Indicator Calculation Family members/caregivers and TAY respondents’ ratings (1–Strongly Disagree to 5–Strongly Agree) of 11 self- report items (specified in the Data Sources section below) were averaged to calculate aggregate ratings of perceptions of well-being as a result of mental health services (see Figures 11.1-11.2 and Tables 11.1-11.2 below). Aggregate ratings were calculated for each fiscal year. Ratings of 3.5 or greater generally indicate positive perceptions. This calculation was developed to approximate domains of well-being many respondents noted in their feedback to our initial reports. Adult and older adult respondents’ ratings (1–Strongly Disagree to 5–Strongly Agree) of 14 self-report items (specified in the Data Sources section below) were averaged to calculate aggregate ratings of perceptions of well-being as a result of mental health services (see Figures 11.1-11.2 and Tables 11.1-11.2 below). Aggregate ratings were calculated for each fiscal year. Ratings of 3.5 or greater generally indicate positive perceptions. This calculation was developed to approximate domains of well-being many respondents noted in their feedback to our initial reports. Data Sources Consumer Perception Surveys Family members/caregivers and TAY self-report items analyzed (YSS/YSS-F): o My child is better at handling daily life. o My child gets along better with family members. o My child gets along better with friends and other people. o My child is doing better in school and/or work. o My child is better able to cope when things go wrong. o I am satisfied with our family life right now. o My child is better able to do things he or she wants to do. o I know people who will listen and understand me when I need to talk. o I have people that I am comfortable talking with about my child's problems. o In a crisis, I would have the support I need from family or friends. o I have people with whom I can do enjoyable things. Adult and older adult self-report items analyzed (MHSIP): o I deal more effectively with daily problems. o I am better able to control my life. o I am better able to deal with crisis. o I am getting along better with my family. o I do better in social situations. o I do better in school and/or work. o I do things that are more meaningful to me. o I am better able to take care of my needs. o I am better able to handle things when they go wrong. o I am better able to do things that I want to do. o I am happy with the friendships I have. o I have people with whom I can do enjoyable things. 50 o I feel I belong in my community. o In a crisis, I would have the support I need from family or friends. Note: Data collected in FYs 2008-09 and 2009-10 must be interpreted separately because a convenience sampling method was employed to gather FY 2008-09 data and a random sampling method employed to gather data in FY 2009-10.7 Review of Existing Data Data source likely to be sustained (i.e., most items analyzed for this indicator are included in the August 2012 survey administration) Data relevant to population of interest (i.e., convenience or random sample of all mental health consumers) Data available across multiple service years More than 10% missing or unknown values among demographic groups Analytic Potential of Indicator Analysis across time will be possible if the sampling methodology and instrument used is employed in a consistent manner each year Analysis among specific service populations possible (e.g., all consumers, demographic groups) County analysis possible for FY 2008-09 (convenience sample), but data is not available at the county level for FY 2009-10 7 Cowles, E. L., Harris, K., Larsen, C., and Prince, A. (2010). Assessing Representativeness of the Mental Health Services Consumer Perception Survey. 51 Figure 11.1. Perceptions of improvement in well-being as a result of services, FY 2008-09 Older Adults 3.94 (n=281) Adults 3.92 (n=2,659) TAY 3.96 (n=1,358) Children 4.01 (n=2,074) 1.00 2.00 3.00 4.00 5.00 Table 11.1. Perceptions of improvement in well-being as a result of services by race/ethnicity, FY 2008-09 Family Member/ TAY Adult Older Adult Caregiver White 3.95 (n=679) 3.94 (n=357) 3.92 (n=1,217) 3.95 (n=167) Hispanic / Latino 4.06 (n=1,207) 3.99 (n=754) 4.01 (n=647) 4.13 (n=48) Asian 4.02 (n=145) 3.84 (n=115) 3.95 (n=282) 4.02 (n=34) Pacific Islander 3.89 (n=38) 4.00 (n=40) 4.15 (n=54) 3.88 (n=2) Black 4.04 (n=222) 4.05 (n=167) 4.00 (n=218) 4.02 (n=14) American Indian 4.00 (n=110) 4.07 (n=115) 3.91 (n=155) 3.18 (n=7) Other 4.04 (n=553) 3.98 (n=432) 3.99 (n=380) 4.23 (n=23) Unknown/Missing Values: Family Member/Caregiver = 9.0% (n =291), TAY = 9.8% (n =214), Adult = 16.4% (n =579), Older Adult = 16.7% (n =59) Table 11.2. Perceptions of improvement in well-being as a result of services by gender, FY 2008-09 Family Member/ TAY Adult Older Adult Caregiver Female 4.02 (n=732) 3.96 (n=522) 3.94 (n=1,239) 3.91 (n=159) Male 4.04 (n=1,213) 3.99 (n=715) 3.96 (n=1,152) 4.08 (n=93) Unknown/Missing Values: Family Member/Caregiver = 6.2% (n =129), TAY = 8.9% (n =121), Adult = 10.1% (n =268), Older Adult = 10.3% (n =29) 52 Priority Indicator 12: Satisfaction with Services Indicator Summary This indicator provides insight into consumer and family perceptions of satisfaction with mental health services. Indicator Calculation Family members/caregivers and TAY respondents’ ratings (1–Strongly Disagree to 5–Strongly Agree) of two self- report items (specified in the Data Sources section below) were averaged to calculate aggregate ratings of perceptions of access to mental health services (see Figures 12.1-12.2 and Tables 12.1-12.2 below). Aggregate ratings were calculated for each fiscal year. Ratings of 3.5 or greater generally indicate positive perceptions. This calculation method is in line with previous DHCS practices. Adult and older adult respondents’ ratings (1–Strongly Disagree to 5–Strongly Agree) of 14 self-report items (specified in the Data Sources section below) were averaged to calculate aggregate ratings of perceptions of access to mental health services (see Figures 12.1-12.2 and Tables 12.1-12.2 below). Aggregate ratings were calculated for each fiscal year. Ratings of 3.5 or greater generally indicate positive perceptions. This calculation method is in line with previous DHCS practices. Data Sources Consumer Perception Surveys Family members/caregivers and TAY self-report items analyzed (YSS/YSS-F): o Overall, I am satisfied with the services my child received. o The people helping my child stuck with us no matter what. o I felt my child had someone to talk to when he/she was troubled. o The services my child and/or family received were right for us. o My family got the help we wanted for my child. o My family got as much help as we needed for my child. Adult and older adult self-report items analyzed (MHSIP): o I like the services that I received here. o If I had other choices, I would still get services from this agency. o I would recommend this agency to a friend or family member. Note: Data collected in FY 2008-09 and 2009-10 must be interpreted separately because a convenience sampling method was used to gather FY 2008-09 data and random sampling was used to gather data in FY 2009-10.8 Review of Existing Data Data source likely to be sustained Data relevant to population of interest (i.e., convenience or random sample of all mental health consumers) Data available across multiple service years More than 10% missing or unknown values among demographic groups Analytic Potential of Indicator Analysis across time possible if the sampling methodology and instrument used is consistent each year Analysis among specific service populations possible (e.g., all consumers, demographic groups) 8 Cowles, E. L., Harris, K., Larsen, C., and Prince, A. (2010). Assessing Representativeness of the Mental Health Services Consumer Perception Survey. 53 Figure 12.1. Satisfaction with services, FY 2008-09 Older Adults 4.40 (n=291) Adults 4.32 (n=2,691) TAY 4.10 (n=1,367) Children 4.39 (n=2,093) 1.00 2.00 3.00 4.00 5.00 Table 12.1. Satisfaction with services by race/ethnicity, FY 2008-09 Family Member/ TAY Adult Older Adult Caregiver White 4.36 (n=684) 4.08 (n=358) 4.35 (n=1,222) 4.42 (n=170) Hispanic / Latino 4.41 (n=1,213) 4.14 (n=755) 4.37 (n=651) 4.56 (n=48) Asian 4.28 (n=145) 3.96 (n=118) 4.31 (n=291) 4.35 (n=34) Pacific Islander 4.20 (n=39) 3.95 (n=40) 4.23 (n=54) 5.00 (n=2) Black 4.40 (n=222) 4.20 (n=167) 4.33 (n=219) 4.18 (n=14) American Indian 4.31 (n=111) 4.19 (n=115) 4.34 (n=156) 4.18 (n=14) Other 4.41 (n=555) 4.12 (n=434) 4.35 (n=378) 4.55 (n=23) Unknown/Missing Values: Family Member/Caregiver = 9.3% (n =304), TAY = 10.0% (n =221), Adult = 23.3% (n =603), Older Adult = 18.0% (n =67) Table 12.2. Satisfaction with services by gender, FY 2008-09 Family Member/ TAY Adult Older Adult Caregiver Female 4.38 (n=734) 4.20 (n=526) 4.40 (n=1,247) 4.41 (n=161) Male 4.39 (n=1,219) 4.07 (n=716) 4.29 (n=1,156) 4.41 (n=93) Unknown/Missing Values: Family Member/Caregiver = 6.7% (n =140), TAY = 9.1% (n =125), Adult = 0.0% (n =13), Older Adult = 12.7% (n =37) 54 Appendix A – Priority Indicator Matrix 55 Appendix B – Preceding Reports The evaluation team completed extensive groundwork before arriving at the conclusions contained in this report. To date, the team has documented evaluation planning in six reports based on statewide data made available online at UCLA and MHSOAC websites. Stakeholders were invited to provide feedback on draft reports by responding to questions in an accompanying guidance document. The invitation was shared online and through mass e-mail. Small stakeholder groups participated in one of two webinars, or online orientations to the report, that provided an overview to the report’s purpose and the types of feedback sought. Report title: Defining Priority Indicators Report version: Draft for stakeholder review Here, the evaluation team began to refine the core set of priority indicators proposed by the California Mental Health Planning Council to assess target outcomes of mental health consumers and the performance of the mental health system. Report title: Defining Priority Indicators Report version: Final, revised with stakeholder input In the revised report, the evaluation team illustrated how stakeholder feedback was integral to indicator development. This report incorporated changes driven by stakeholders’ comments about the comprehensiveness and appropriateness of the indicators. Report title: Compiling Data to Produce All Priority Indicators Report version: Draft for stakeholder review In this report, the evaluation team proposed how priority indicators could be calculated using existing statewide data. The report also detailed all possible data sources and specific variables or data fields that might be used to build comprehensive priority indicators. Report title: Compiling Data to Produce All Priority Indicators Report version: Final, revised with stakeholder input The initial report was revised to include information regarding measurement methods and the adequacy of existing data sources, gathered through a stakeholder feedback process similar to that used for the final Defining Priority Indicators report. Report title: Initial Statewide Priority Indicator Report Report version: Draft for stakeholder review The evaluation team used select data from fiscal years 2008-09 and 2009-109 to calculate priority indicators as outlined in the previous report (Compiling Data to Produce All Priority Indicators). Report title: Initial Statewide Priority Indicator Report Report version: Final, revised with stakeholder input The revised report incorporated several calculation updates guided by the MHSOAC. Updated calculations largely focused on outcome data collected after enrollment. The report also reflected input from consumer stakeholders about the use and accessibility of report features (illustrations, descriptions, etc.) and language. 9 The team sorted through datasets from 2005 through 2011 in search of one or more fiscal years (FY) in which data cells were largely filled where expected. Data from FY 2008-09 and 2009-10 fit this requirement. 56 Appendix C – Recoding Pre-DIG Race Data to Post-DIG Format Stakeholder feedback to previous evaluation team reports suggested inconsistency and potential inaccuracy among Race and Ethnicity data fields may be due in part to changes in the format of these fields in the CSI and DCR data systems. In 2006, DMH implemented changes to the Race and Ethnicity fields due to Uniform Data System/Data Infrastructure Grant (DIG) requirements from the federal government (see DMH Information Notice: 06-02). Although DMH provided training about these changes, Race and Ethnicity information seems to be reported inconsistently across counties. Because demographic information in the CSI system is transferred to corresponding fields in the DCR system, Race and Ethnicity information in both systems was analyzed but interpreted with caution. To ameliorate potential shortcomings of this change, the evaluation team used pre-DIG information to fill gaps in missing post-DIG Race and Ethnicity fields for analyses involving demographic information. The table below details the recoding process. Before Recode After Recode (if Post-DIG field empty) Data Post-DIG Data Pre-DIG Field Definition Definition Value Field Value Empty formerly White 1 Race White or Caucasian 1 Ethnicity / Race Empty formerly Hispanic 2 Ethnicity Yes (Hispanic or Latino) Y Ethnicity / Race Empty formerly Black or African Black 3 Race 3 Ethnicity / Race American Empty formerly American Indian or American Native 5 Race 5 Ethnicity / Race Alaska Native Empty formerly Amerasian A Race Other Asian O Ethnicity / Race Empty formerly Hawaiian Native P Race Native Hawaiian P Ethnicity / Race Empty formerly Multiple X Race Multiracial Multiracial Ethnicity / Race Empty formerly Other Asian or Pacific 4 Race Other Asian O Ethnicity / Race Islander 57