BHSOAC
Mhsa Priority Indicators: Santa Cruz County
Read the report at Santa Cruz County ↗
Mental Health Services Act Evaluation:
Report on Prioritized Indicators
Contract Deliverable 2F, Phase II
Santa Cruz County
UCLA Center for Healthier Children, Youth and Families
EMT Associates, Inc.
Submitted for review on November 30, 2012
The following report was funded by the
Mental Health Services Oversight and Accountability Commission.
The following report was revised in partnership with stakeholders who provided important
historical context, data consultation, and revisions to ensure this report is accurate and accessible
to the broadest audience possible. Feedback, collected prior to, during, and following report
development, was crucial to developing this report. The UCLA-EMT Evaluation Team would like to
express sincere appreciation to the research analysts, advocates, consumers and family members,
agency representatives, service providers, and MHSOAC representatives who contributed
invaluable insights to this document and previous versions.
2
Table of Contents
Purpose ..................................................................................................................................................................... 4
Background ............................................................................................................................................................. 4
Review of Existing Statewide Data Systems ............................................................................................... 5
Data Sources ....................................................................................................................................................... 7
Priority Indicators Evaluated ................................................................................................................. 9
Priority Indicators ......................................................................................................................................... 10
Report Organization ................................................................................................................................ 11
Priority Indicator 1: Attendance ................................................................................................................. 12
1.1 Expulsions and Suspensions Per Year (CPS) ..................................................................... 12
1.2 Average School Attendance Per Year (FSP) ....................................................................... 14
Priority Indicator 2: Employment .............................................................................................................. 16
Priority Indicator 3: Homelessness and Housing Rates ..................................................................... 20
Priority Indicator 4: Arrest Rates ............................................................................................................... 22
Priority Indicator 5: Demographic Profile of Consumers Served .................................................. 27
Priority Indicator 6: Demographic Profile of New Consumers ....................................................... 31
Priority Indicator 7: Penetration of Mental Health Services ............................................................ 36
Priority Indicator 8: Access to a Primary Care Physician .................................................................. 39
Priority Indicator 9: Perceptions of Access to Services ..................................................................... 42
Priority Indicator 10: Involuntary Status ................................................................................................ 44
Priority Indicator 12: Satisfaction with Services .................................................................................. 49
Appendix A – Priority Indicator Matrix ..................................................................................................... 51
Appendix B – Preceding Reports ................................................................................................................. 52
Appendix C – Recoding Pre-DIG Race Data to Post-DIG Format ...................................................... 53
3
Purpose
The Mental Health Services Act Oversight and Accountability Commission (MHSOAC)
charged the UCLA-EMT Evaluation Team with tracking the Mental Health Services Act’s
(MHSA) impact on mental health service consumers and the community mental health
service system using a set of performance indicators. The current report – a companion to
the statewide report published during November 2012 – details the initial effort toward
this goal at the county level.
Here, we provide a snapshot of consumer outcomes and community mental health system
performance within an individual county. This is a first step toward developing a set of
indicators that can help stakeholders with ongoing quality improvement. This report is the
first in a series designed to update stakeholders about mental health consumer outcomes
and service system progress. Tracking performance on particular outcomes over time,
across programs, and within counties can provide useful information to those planning,
operating and monitoring services. Indicators were designed for planning, quality
improvement, and other applications that stakeholders deem important. In this way,
among many others, stakeholders can play a vital role in a continuous quality improvement
process.
Background
What are Priority Indicators and what are they intended to do?
Two concerns of public mental health system stakeholders are accountability and the
ability to conduct continuous quality improvement activities. One strategy is to use a set of
indicators to measure performance. The California Mental Health Planning Council
proposed and defined a set of performance indicators, referred to as priority indicators,
designed to assess how the MHSA has impacted mental health consumers and the mental
health system in target areas that should be most changed through MHSA implementation.
Indicators will help track progress among consumers and across community mental health
systems. At the consumer level, outcomes such as education and employment will be
followed, while outcomes including mental health service penetration and consumer
demographics are examined at the broader system level. This report examines the core set
of priority indicators vetted by the MHSOAC within counties.
Previous work of the UCLA-EMT Evaluation Team leading to the statewide and present county reports
The evaluation team began its work using the California Mental Health Planning Council’s
definitions – its collective vision of how indicators might best be measured. (These
fundamental definitions are located in Appendix A and are discussed in preceding reports
available at http://healthychild.ucla.edu/MHSA_evaluation.asp.) Priority indicator
development was a joint effort among the MHSOAC, stakeholders, and the evaluation team.
The evaluation team facilitated discussions among interested stakeholders to create the
strongest, most comprehensive representations of priority indicators that aligned with
both early conceptualizations and feedback using the data that were already collected
across the state with some regularity. Where gaps existed, the evaluation team proposed
4
new data collection that will improve future evaluation but is beyond the team’s current
scope of work.
The evaluation team adapted advice from stakeholders, and this report examines whether
these adapted indicators provide meaningful information. Although stakeholders proposed
additional indicators, these measures have not yet been vetted by the MHSOAC to
determine whether they add useful and crucial information that aligns with the need. The
MHSOAC has yet to decide whether to change the previously approved priority indicators.
Thus, the evaluation team explored the first proposed priority indicators in this report,
which serves as a fundamental step in the ongoing process to refine and expand priority
indicators that are not only measurable but also useful to the range of stakeholders
invested in this work.
Appendix B summarizes the MHSOAC-approved reports that 1) document the available
data and proposed calculations, and 2) lead to the initial statewide report on which this
document is based. Included in this summary is the stakeholder review process that took
place throughout the development of each report. This information can also be reviewed in
the report “Initial Statewide Priority Indicator Report,” available on the UCLA and MHSOAC
websites.
The following report stems from the statewide report. In it, we document priority indicator
development at the county level during two target fiscal years – 2008-09 and 2009-10. 1
Review of Existing Statewide Data Systems
As directed by the MHSOAC, priority indicators were built upon existing data sources that
are systematically collected by California counties and reported to the California
Department of Health Care Services (DHCS).2 To build indicators for use at the state and
county levels, existing data systems were reviewed to assess their suitability for supporting
outcome and performance monitoring. Several criteria were used to evaluate the suitability
of existing data sources, including:
Available – Data accessible in an analyzable format
Complete – Levels of missing information within key data fields did not prevent meaningful
analysis and interpretation
Sustained – Data sources is likely to continue to exist in the foreseeable future
Relevant – Data relevant to populations of interest (e.g., all mental health consumers and
Full Service Partnerships
1 The evaluation team reviewed data from 2005 through 2011 in search of one fiscal year in which data cells
were largely filled where expected. Two such fiscal years were identified – FYs 2008-09 and 2009-10.
Through analysis, some proposed data sources or methods of indicator calculation, suggested in previous
reports by stakeholders and the evaluation team, were found to not be possible or to not produce meaningful
outcomes due to data limitations. Decisions made about previously proposed indicators, based on data
limitations, are summarized in the statewide report appendix.
2 Previously the Department of Mental Health (DMH); The DHCS abbreviation will be used to refer to work
completed by DMH.
5
Longitudinal – Data available for multiple service years
Multilevel – Data can be analyzed at multiple levels (e.g., state, county, and individual)
The application of these criteria to each key data source and important considerations and
limitations regarding each data source overall are summarized in the Data Sources table
below. These criteria were also applied to the specific data fields used to build each priority
indicator that would be used to monitor performance statewide and at the county level.
Review of indicator-specific data fields is summarized within the tables that introduce the
analysis and findings of each priority indicator (see Priority Indicator Analysis and Findings
section below).
6
Data Sources
Client & Service Information (CSI)
Summary: The CSI system is a repository of county, client (e.g., age, gender, preferred language, education,
employment status, living arrangement, etc.), and service information (number and length of service contact).
The data are collected from all consumers who receive mental health services, including consumers involved
in the Full Service Partnership.
Review Findings:
Available Complete Sustained Relevant Longitudinal Multilevel
Considerations and Limitations: Stakeholder feedback to previous evaluation team reports suggested that
inconsistency and potential inaccuracy among race and ethnicity data fields may be due in part to changes in
the format of these fields in the CSI data system (see DMH Information Notice: 06-02). For details regarding
the Race and Ethnicity data field changes and procedures employed by the evaluation team to improve data
quality, see Appendix C.
Additionally, the completeness of data fields used to calculate indicators varies greatly across fiscal years and
among counties (e.g., greater than 50% in some cases). Thus the representativeness and interpretability of
such data fields is in doubt. Proportions of missing or unknown information are noted within each indicator
section throughout the report.
Data Collection and Reporting (DCR) System
Summary: The DCR system houses data for consumers who are served through Full Service Partnership
programs. Data from assessments – the Partnership Assessment Form (PAF), Key Event Tracking (KET), and
Quarterly Assessment (3M) – are collected for consumers in specific age categories. The PAF reflects
consumer history and baseline information, including consumer education and/or employment, housing
situation, legal issues, health status, and substance use. The KET reflects any important changes in the
consumer’s life, such as housing, education and/or employment, and legal issues during FSP. The 3M is used
to collect information on a quarterly basis, regarding key areas such as education, health status, substance
use, and legal issues.
Review Findings:
Available Complete Sustained Relevant Longitudinal Multilevel
Considerations and Limitations: Race and ethnicity information in the DCR system is imported from the CSI
system by DHCS. As such, the limitations of this information noted for the CSI system also apply here.
Specifically, stakeholder feedback to previous evaluation team reports suggested that inconsistency and
potential inaccuracy among race and ethnicity data fields may be due in part to changes in the format of these
fields in the DCR data system (see DMH Information Notice: 06-02). For details regarding race and ethnicity
data field changes and procedures employed by the evaluation team to improve data quality, see Appendix C.
Additionally, representatives from seven counties or municipalities that currently do not have data contained
in the DCR database in FYs 2008-09 or 2009-10, for several reasons (e.g., county data incompatibly formatted
DHCS database). Moving forward it will be imperative to indicator tracking of FSPs for all counties to be
included in the DCR database
Performance Outcomes and Quality Improvement (POQI) – Consumer Perception
Surveys (CPS)
Summary: These consumer surveys are customized for consumer groups (e.g., family members/caregivers,
youth, adults, and older adults) receiving mental health services. Instruments are composed of widely
validated measures such as the Child Behavior Checklist, Youth Self Report, and Restrictiveness of Living
Environment Scale for youth; the Global Assessment of Functioning, Behavior and Symptom Identification
7
Scale, and the California Quality of Life for adults; and the Brief Symptom Inventory, Senior Outcomes
Checklist 10, and Index of Independent Activities of Daily Living for older adults. The data, designed to inform
treatment planning and service management, are collected from a sample of individuals with “serious,
persistent” mental illness who have received services for 60 days or more and are not categorized as
“medication only.”
Review Findings:
Available Complete Sustained Relevant Longitudinal
Considerations and Limitations: For FY 2008‐09 and prior years, a convenience sampling approach was used
in which county-level mental health service providers administered surveys twice a year for a two‐week
period, in early May and November. Investigation of the convenience sampling methodology revealed the
resulting information was not representative of the larger mental health service population.3 Beginning with
FY 2009-10, a random sampling methodology was employed to produce data that are more representative of
the perceptions of the mental health service population. As such, comparisons involving CPS data collected in
FY 2008-09 and FY 2009-10 cannot be made.
Note – The smaller sample generated by the random sampling method employed in FY 2009-10 does not
allow for consumer perception analyses at the county levels for this fiscal year.
Note – The sampling methods that have been employed to date do not capture specific mental health service
populations, such as those in institutions for mental disease or prison.
Other Sources
Estimates of Need for Mental Health Services
To achieve a standardized rate for penetration of services across all counties, the evaluation team contracted
with Dr. Charles Holzer for statewide and county mental health service need estimates. Dr. Holzer previously
developed penetration rate estimates for the California DHCS. He estimated the proportion of persons with
serious mental illness among those whose income falls within 200% of the federal poverty level, using data
from the most up-to-date National Comorbidity Survey Replication and generated prevalence estimates for
several Census years. (For additional information regarding prevalence estimate methodology, see Dr.
Holzer’s website at http://66.140.7.155/estimation/3_Synthetic/synthetic.htm).
Review Findings:
Available Complete Relevant Longitudinal
Involuntary Status
Involuntary status information (FY 2008-09) was provided by DHCS for the following service categories: 72
hour Evaluation and Treatment (adults, children); 14- and 30-day Intensive Treatment; 180-day Post
Certification Treatment; and Temporary and Permanent Conservatorships. Involuntary status data for FY
2009-10 were not available from DHCS as this report was being prepared.
Review Findings:
Available Complete Sustained Relevant Longitudinal
Procedures for handling missing / unknown data
The quantity of missing or unknown data (e.g., values) was found to vary considerably
across data sources, data fields, and fiscal years. For data fields determined to be necessary
for the construction of priority performance indicators (detailed in priority indicator
summary tables, see Priority Indicator Analysis and Findings section below), if the amount
3 Cowles, E. L., Harris, K., Larsen, C., and Prince, A. (2010). Assessing Representativeness of the Mental Health
Services Consumer Perception Survey.
8
of missing or unknown data was substantial (i.e., greater than 10% of cases), the evaluation
team communicated with DHCS analysts and requested input from counties via a data
verification process (detailed below) regarding context and interpretation of such data
fields. Where adequate information was received to interpret missing or unknown values
(i.e., see Appendix C), the evaluation team was able to analyze and interpret such data fields
according to current protocols specified in the data dictionaries relevant to each data
system. The proportion of missing and unknown information relevant to each priority
indicator within each county is noted throughout the report in footnotes immediately
below the relevant table or figure. Accordingly, the frequencies and percentages included in
all data displays do not include missing cases. The label “data unavailable” is inserted
where data is unavailable (data fields are blank).
Priority Indicators Evaluated4
The set of priority performance indicators evaluated in this report were arrived at through
the following processes:
The careful consideration of the California Mental Health Planning Council and approval of
the MHSOAC;5
Consideration of the MHSOAC goal of developing a comprehensive outcome and
performance monitoring system built upon existing data systems;
Consideration of consumer feedback to previous evaluation team reports regarding
proposed priority indicators (see report, “Defining Priority Indicators”);
Review of existing data sources to assess their suitability for supporting outcome and
performance monitoring through priority indicators (see Review of Existing Data, above);
and
County feedback regarding the quality and completeness of key data fields necessary to
calculate priority performance indicators
Through these evaluation processes and careful deliberation of the MHSOAC in
collaboration with the evaluation team, a set of 12 priority performance indicators was
developed. These indicators can be categorized as those intended to provide insight into
the outcomes of mental health consumers (“Consumer Indicators”) and those intended for
monitoring the performance of the community mental health system more broadly
(“System Indicators”). Consumer and system indicators, and the consumer groups they
assess, are summarized in the table below.
4 Although we received strong indicator suggestions from stakeholders, this report helps vet the
appropriateness of the original set proposed by the California Mental Health Planning Council. If the MHSOAC
chooses, it may vet additional indicators, particularly those proposed by stakeholders, when revising the pool.
5 California Mental Health Planning Council (January, 2010). Performance Indicators for Evaluating the Mental
Health System.
9
Priority Indicators
CONSUMERS EVALUATED
SERVICE OLDER
CHILDREN TAY ADULTS
POPULATION ADULTS
CONSUMER INDICATORS
Indicator 1 – Average School Attendance Per Year All/FSP Consumers x x
Indicator 2 – Employed Consumers All/FSP Consumers x x x
Indicator 3 – Homelessness and Housing Rates All/FSP Consumers x x x x
Indicator 4 – Arrest Rate All/FSP Consumers x x x x
SYSTEM INDICATORS
Indicator 5 – Demographic Profile of Consumers Served All/FSP Consumers x x x x
Indicator 6 – Demographic Profile of New Consumers All/FSP Consumers x x x x
Indicator 7 – Penetration of Mental Health Services All Consumers x x x x
Indicator 8 – Access to a Primary Care Physician FSP Consumers x x x x
Indicator 9 – Perceptions of Access to Services All Consumers x x x x
Indicator 10 – Involuntary Status All Consumers x x x x
Indicator 11 – Consumer Well-Being All Consumers x x x x
Indicator 12 – Satisfaction All Consumers x x x x
Criteria used to evaluate priority indicators
Specific criteria, developed in collaboration with the MHSOAC, were established to evaluate
priority performance indicators. These criteria, outlined for consumer and system
indicators below, reflect the goals of the MHSOAC for monitoring consumer outcomes and
community mental health system performance at multiple levels (i.e., state and county) for
the purposes of planning and quality improvement. These criteria may include:
Consumer Indicator Evaluation Criteria:
Indicator can describe changes in consumer outcomes (e.g., change since initiation of
services) or describe the current status of consumers.
Indicator can provide meaningful and relevant insight into the outcomes of service
populations of interest (e.g., all mental health consumers, FSP consumers, and demographic
groups).
Indicator can provide meaningful and relevant insight into the outcomes of consumers
statewide and at the county level.
Indicator provides “actionable” insight, which stakeholders can use to identify areas for
service improvement.
System Indicator Evaluation Criteria:
Indicator can describe meaningful changes in system performance over time.
Indicator can provide meaningful and relevant insight regarding the extent and quality of
services provided to populations of interest (e.g., all mental health consumers, FSP
consumers, and demographic groups).
Indicator can provide meaningful and relevant insight into the performance of the
community mental health system at the statewide and county levels.
Indicator provides “actionable” insight, which stakeholders can use to identify areas for
improving the performance of the mental health system.
10
The application of consumer and system indicator evaluation criteria to each priority
indicator is detailed in the Priority Indicator Analysis and Findings section below.
Report Organization
The remainder of the report summarizes each indicator and its county-specific outcomes,
calculated using select data from FYs 2008-09 and 2009-10. The evaluation team presents
individual-level priority indicators (those specific to consumers), then does the same for
system-level priority indicators (pertaining to community mental health systems
throughout the state). A summary table precedes each indicator and its outcomes to orient
the reader to what the indicator measures, how it was calculated, and its usefulness.
Detailed stakeholder feedback, integral to shaping this and earlier reports, can be reviewed
in “Initial Statewide Priority Indicator Report.” 6
Indicator summary tables
Each priority indicator is introduced and summarized in a concise and organized table in
the Priority Indicator Analysis and Findings section below. Indicator summary tables are
organized into the following sections:
Indicator Summary – Provides a brief definition of the indicator
Indicator Calculation – Details the computation used to produce the indicator
Data Sources – Specifies the data sources and relevant data fields (variables) used to
compute the indicator
Review of Existing Data – Review of data quality criteria (specified in the Review of Existing
Data section above) as applied to indicator-specific data fields
Analytic Potential of Indicator – Review of indicator evaluation criteria (specified in the
section on Criteria used to evaluate priority indicators above)
Note regarding indicator data displays
Each indicator is presented through one or more graphical displays of information. These
displays include bar graphs and tables of frequencies and percentages. For ease of viewing
and interpreting data displays specific to a service population, figures that display indicator
information relevant to all mental health consumers are presented in blue, and those
relevant to FSP consumers are presented in green.
Consistent with reporting practices associated with this work, counties with populations
smaller than 200,000 residents are grouped in outcomes displays. This group is referred to
as “Small Counties.” County-specific reports will be created for “small counties” on request.
Analyses are conducted where data is present, even if data is minimal. We include the label
“data unavailable” where absolutely no information is available within a county’s data field
to support a calculation for the priority indicator.
6 Stakeholder feedback was not collected for each of the county reports developed, thus is not included in the
current report. A full account of stakeholder feedback is available online at MHSOAC and UCLA sites, within
“Initial Statewide Priority Indicator Report.”
11
Priority Indicators Analysis and Findings: Consumer Indicators
Priority Indicator 1: Attendance
1.1 Expulsions and Suspensions Per Year (CPS)
Indicator Summary
This indicator provides descriptive information regarding the number of youth (children and TAY) expelled and
suspended from school during the 2008-09 fiscal year. This indicator illustrates a subset of mental health service
consumers who responded to consumer perceptions surveys.
Indicator Calculation
The number children and TAY who had been enrolled for at least six months and reported being expelled or
suspended from school since beginning mental health services or 12 months prior to beginning such services.
Note: Data is reported from Youth Satisfaction Surveys (YSS)
Data Sources
Consumer Perception Surveys (CPS) Data Fields: HowLong, LES12AREST, LES12PSTAREST
Review of Existing Data
Data sources likely to be sustained
Data relevant to populations of interest
Amount of missing data for child age group is approximately 6%
Amount of missing data for TAY age group is approximately 46%
Analytic Potential of Indicator
Analysis across time possible with the exception of 2009-10 data
Analysis among specific service populations possible
State- and county-level analysis possible with the exception of 2009-10 data
12
Figure 1.1-1. Proportion of children who were suspended or expelled prior to beginning
services and after receiving services for a period of 6 – 12 months during FY 2008-09
100%
90%
76.2%
80% 73.9%
70%
60%
50%
No
40%
Yes
26.1%
30% 23.8%
20%
10%
0%
Exp/Sus 12 mon prior to servicesExp/Sus since beginning services
(n = 21) (n =23)
Unknown/Missing for FY 2008-09: Exp/Sus prior to services = 19% (n = 5), Exp/Sus since receiving services = 11.5%
(n = 3)
Figure 1.2-2. Proportion of TAY who were suspended or expelled prior to beginning services
and after receiving services for a period of 6 – 12 months during FY 2008-09
100%
90% 85.7%
80%
71.4%
70%
60%
50%
No
40%
Yes
28.6%
30%
20% 14.3%
10%
0%
Exp/Sus 12 mon prior to servicesExp/Sus since beginning services
(n = 14) (n = 14)
Unknown/Missing for FY 2008-09: Exp/Sus prior to services = 6.7% (n = 1), Exp/Sus since receiving services = 6.7%
(n = 1)
13
1.2 Average School Attendance Per Year (FSP)
Indicator Summary
This indicator provides descriptive information regarding the frequency for which Full Service Partnership
consumers (children and TAY) attended school during the 2008-09 and 2009-10 fiscal years.
Indicator Calculation
The number children and TAY who attended school always, mostly, sometimes, infrequently, and never
divided by the number of children for which there were data.
The number of TAY who attended school always, mostly, sometimes, infrequently, and never divided by
the number of TAY for which there were data.
Note: Age groupings were revised such that:
Child ages = 1-15 (same as previously)
TAY ages = 16-18 (16-25 previously)
The TAY age group was revised because education variables would be less clear for clients older than 18.
Data Sources
DCR (PAF - NONRES) Data Field: AttendanceCurr
Review of Existing Data
Data sources likely to be sustained
Data relevant to populations of interest (FSPs)
Data fields necessary to construct this indicator regarding FSP consumers (i.e., DCR) were not available
as of the preparation of this report.
Analytic Potential of Indicator
Given the absence of DCR data for FSP consumers:
Analysis across time not currently possible
Analysis among specific service populations not currently possible
State- and county-level analysis not currently possible
14
Figure 1.2-1 –The frequency with which children and TAY attended school during FY 2008-
09 (DCR)
DATA UNAVAILABLE
Figure 1.2-2 –The frequency with which children and TAY attended school during FY 2009-
10 (DCR)
DATA UNAVAILABLE
15
Priority Indicator 2: Employment
Indicator Summary
This indicator provides the proportion of TAY, adults and older adults who are employed (paid and non-paid) and
not employed as recorded during the most recent update (second date of service). This indicator provides
descriptive information regarding clients’ employment status during their first date of service.
Indicator Calculation
Client & Service Information (CSI)
The number of paid employed clients divided by the total number of TAY, adults, and older adults for whom
there were employment data.
The number of nonpaid employed clients divided by the total number of TAY, adults, and older adults for
whom there were employment data.
The number of paid nonemployed clients divided by the total number of TAY, adults, and older adults for
whom there were employment data.
Note: There were multiple periodic updates for clients within each fiscal year. These ratios provide
information for those who had a second periodic update within a given fiscal year (or a second date of
service). Additionally, the age groupings were revised to capture those truly eligible for employment.
Those who indicated they were retired or incarcerated were excluded from calculations.
Data Collection and Reporting (DCR)
The number of TAY, adults, and older adults who reported paid employment divided by the total number of
TAY, adults, and older adults.
The number of TAY, adults, and older adults who reported nonpaid employment divided by the total number
of TAY, adults, and older adults.
The number of TAY, adults, and older adults who did not report any employment divided by the total number
of TAY, adults, and older adults.
Note for CSI and FSP data:
Age groupings were revised such that
TAY ages = 18-25 (previously 16-25)
Older adults = 60-65 (60 and up previously)
Data Sources
CSI Periodic Post-dig, Data Field: Employment Status
DCR (PAF - NONRES) Data Fields: Current_CompetitiveAvgHrWeek, Current_SupportedAvgHrWeek,
Current_TransitionalAvgHrWeek, Current_In-HouseAvgHrWeek, Current_OtherEmploymentAvgHrWeek,
Current_Non-paidAvgHrWeek
Review of Existing Data
Client & Service Information (CSI)
Data sources likely to be sustained
Data relevant to populations of interest (all consumers)
Data available across multiple service years
Amount of missing data for TAY age group is approximately 96%
Amount of missing data for Adult age group is approximately 95%
Amount of missing data for Older Adult age group is approximately 100%
Data Collection and Reporting (DCR)
16
Data sources likely to be sustained
Data relevant to populations of interest (FSPs)
The amount of missing data for these ratios is unknown given how the employment data are collected. There is no
data code option for “missing;” as a consequence, blank responses are either missing or not applicable.
Analytic Potential of Indicator
Client & Service Information (CSI)
Analysis across time possible but very difficult
Analysis among specific service populations not possible
State- and county-level analysis possible
Data Collection and Reporting (DCR)
Analysis across time possible
Analysis among specific service populations not possible
State- and county-level analysis possible
17
Figure 2.1 - Proportion of clients who were employed and not employed as reported during
their second service date for FY 2008-09 (CSI)
Unknown/Missing for FY 2008-09: TAY = 96.4% (n = 27), Adult = 94.7% (n = 18),
Older adults = 100% (n = 2)
Figure 2.2 – Proportion of clients who were employed and not employed as reported during
their second service date for FY 2009–10 (CSI)
Unknown/Missing for FY 2008-09: TAY = 91.3% (n = 21), Adult = 100% (n = 25),
Older adults = 100% (n = 4)
18
Figure 2.3 –The proportion of FSPs who were employed during FY 2008–09 (DCR)
Unknown/Missing for FY 2008-09: Unknown
Figure 2.4 –The proportion of FSPs who were employed during FY 2009–10 (DCR)
Unknown/Missing for FY 2009-10: Unknown
19
Priority Indicator 3: Homelessness and Housing Rates
Indicator Summary
This indicator summarizes the housing status of all mental health consumers and FSPs served during FYs 2008-09
and 2009-10. There are two parts: (a) a breakdown by most recently available housing status and (b) the
percentage of consumers experiencing homelessness at any point during the year.
Indicator Calculation
Frequencies of the most recent housing statuses were computed for mental health and FSP consumers served in
FYs 2008-09 and 2009-10. This calculation excludes consumers with no housing data within the given FYs or
consumers whose most recent status was homeless. The percentages of mental health and FSP consumers who
experienced homelessness at any point during the given FY were also computed.
Note that a consumer who was most recently homeless would not be included in the first indicator for most recent
housing status, whereas a consumer who was previously homeless and more recently reported as not homeless
would be included.
Data Sources
Client & Service Information (CSI): H-01.0 County / City / Mental Health Plan Submitting Record; H-02.0 County
Client Number; C-03.0 Date of Birth; P-01.0 Date Completed; P-09.0 Living Arrangement
Data Collection and Reporting (DCR) Key Event Tracking (KET): 1.01 Global ID; 1.02 Assessment ID; 1.07
Age Group; 3.01 CountyID; 3.06 Assessment Date; 5.01 DateResidentialChange; 5.02 Current
Review of Existing Data
These data were taken from the Key Event Tracking (KET) updates for FSP consumers and the periodic updates for
all mental health consumers, limited to the given fiscal year. Any consumer who did not have an update available
during the year was not included. Data sources are likely to be sustained in the foreseeable future, providing a
consistent source for tracking system performance moving forward. Taking a conservative approach, we
considered cases without valid data “missing.” It should be noted that the data reporting and collection practices
currently in place do not allow for a distinction between missing data from unreported changes in housing status
and blank values from standard data entry practices. This is especially notable in the KET updates for FSP
consumers, leading to large percentages of “missing” data. These results should be interpreted cautiously. In
particular, there is the risk of systematic bias in underreporting certain housing statuses.
Data fields necessary to construct this indicator regarding FSP consumers (i.e., DCR) were not available
as of the preparation of this report.
Data fields necessary to construct this indicator regarding all consumers (i.e., CSI) were not available as
of the preparation of this report.
Analytic Potential of Indicator
Data across service years support analysis of the distribution and change of housing statuses, including
homelessness, among consumers.
Indicator Displays
The first set of charts displays the most recently reported non-homeless housing statuses of consumers, by
percentage, during each fiscal year. The second set displays the percentages of consumers who were reported as
experiencing homelessness at any time during the fiscal year.
20
Figure 3.1 – Most recent housing status excluding homelessness, all consumers (CSI)
DATA UNAVAILABLE
Figure 3.2 – Most recent housing status excluding homelessness, FSP consumers only (DCR)
DATA UNAVAILABLE
Figure 3.3 – Experienced homelessness at any point during the year, all consumers (CSI)
DATA UNAVAILABLE
Figure 3.4 – Experienced homelessness at any point during the year, FSP consumers only
(DCR)
DATA UNAVAILABLE
21
Priority Indicator 4: Arrest Rates
Indicator Summary
This indicator provides the proportion of youth, adults, and older adults who reported being arrested 12 months
prior to receiving services and the proportion of youth, adults, and older adults who reported being arrested since
beginning services.
For calculations involving consumer perception surveys, this indicator includes only youth, adults, and older adults
who reported receiving services for 6 to 12 months. This indicator provides information regarding whether the
proportion of arrested clients has increased or decreased after 6 to 12 months of service.
For calculations involving Full Service Partnership consumers, this indicator tracks arrests prior to enrollment using
intake data. This indicator accounts for consumers enrolled during the target fiscal years for which PAF surveys are
available.
Indicator Calculation
Consumer Perception Surveys (CPS)
The number of reported arrest 12 months prior to services divided by the total number of youth, adults, and
older adults for who there was data
The number of reported arrest since beginning services divided by the total number of youth, adults, and older
adults for who there was data
Note: Clients were surveyed multiple times during the 2008 – 2009 fiscal year. However, only one survey
administration was used to get both the proportion of clients who reported being arrested 12 months prior to
beginning services and since receiving services.
Age groupings are as follows:
Youth, 1 – 25 years
Adult, 26 – 59 years
Older adult, 60 and above
Data Collection and Reporting (DCR)
The number of youth (children and TAY), adults, and older adults reporting arrests 12 months prior to
enrollment divided by the total number of unique clients for who there was data
The number of youth (children and TAY), adults, and older adults reporting arrests 12 months prior to the
past 12 months divided by the total number of unique clients for who there was data
Note: In rare cases where two surveys were entered for one client, only the earliest entry was used in calculations.
Data Sources
Consumer Perception Survey (CPS) for Youth, Adults, and Older Adults Data Fields: HowLong, les12arest,
les12pstarest
Data Collection and Reporting (DCR PAF NONRES): Age_Group, ArrestPast12, ArrestPrior12
Review of Existing Data
Consumer Perception Surveys (CPS)
Data sources likely to be sustained
Data relevant to populations of interest
Approximately 6.1% missing or unknown values for youth
Approximately 10.3% missing or unknown values for adult
Approximately 100% missing or unknown values for older adult
22
Data Collection and Reporting (DCR)
Data sources likely to be sustained
Data relevant to populations of interest
Data for children was unavailable at the time of this report
On average, 16.7% missing or unknown values for adults
On average, 2.9% missing or unknown values for adults
On average, 0.0% missing or unknown values for older adults
Analytic Potential of Existing Data
For both data sources
Analysis across time possible
Analysis among specific service populations not possible
State and county individual level analysis possible
Note: As of the submission of this report, a new calculation has been proposed to examine
arrest rates. The proposed calculation would use FSP-DCR data during consumers’
enrollment (not intake as it is presented here). An updated indicator will be available shortly.
23
Figure 4.1 – Proportion of youth who were arrested prior to beginning services and since
receiving services
Missing/unknown for Arrested 12 months prior to services = 7.3% (n = 3)
Missing/unknown for Arrested since receiving services = 4.9% (n = 2)
Figure 4.2 – Proportion of adults who were arrested prior to beginning services and since
receiving services
Missing/unknown for Arrested 12 months prior to services = 10.3% (n = 3)
Missing/unknown for Arrested since receiving services = 10.3% (n = 3)
24
Figure 4.3 – Proportion of older adults who were arrested prior to beginning services and
since receiving services (CPS)
DATA UNAVAILABLE
Figure 4.4 - Proportion of children who were arrested within the past 12 months (DCR)
DATA UNAVAILABLE
Figure 4.5 - Proportion of TAY who were arrested within the past 12 months (DCR)
Missing/Unknown for FY 2008-09 = 16.7% (n=1)
Figure 4.6 – Proportion of adults who were arrested within the past 12 months (DCR)
Missing/Unknown for FY 2008-09 = 2.9% (n=1)
25
Figure 4.7 Proportion of older adults who were arrested within the past 12 months (DCR)
Missing/Unknown for FY 2008-09 = 0.0% (n=0)
26
Priority Indicators Analysis and Findings: Community Mental
Health System Indicators
Priority Indicator 5: Demographic Profile of Consumers Served
Indicator Summary
This indicator profiles the demographics (race/ethnicity, age, and gender) of all mental health consumers and Full
Service Partnership consumers served during FYs 2008-09 and 2009-10. It summarizes levels of service to
California’s diverse population supported by the community mental health system.
Indicator Calculation
The frequencies of all Santa Cruz County mental health consumers and FSP consumers served in FY 2008-09 and
2009-10 were calculated overall.
Additionally, the proportion of consumers represented by race/ethnicity, age, and gender categories was
calculated by dividing the number of consumers within each demographic category by all consumers served within
Contra Costa County. Proportions were calculated for both service populations (all consumers and FSPs) and both
fiscal years examined (see Figures 5.1-5.6 below).
Data Sources
Client & Service Information (CSI) Data Fields: H-01.0 County / City / Mental Health Plan Submitting Record; H-02.0
County Client Number; C-05.0 Gender; C-09.0 Ethnicity; C-10.0 Race; S-05.0 Mode of Service; S-16.0 From / Entry
Date; S-17.0 Through / Exit Date; S-23.0 Date of Service.
Data Collection and Reporting (DCR) Data Fields: 1.01 Global ID; 1.02 Assessment ID; 1.04 Date Partnership Status
Change; 1.05 Partnership Status; 1.07 Age Group; 1.08 Assessment Type; 2.01 CSI Date of Birth; 2.02 Gender; 2.03
CSIRace1; 2.04 CSIRace2; 2.05 CSIRace3; 2.06 CSIRace4; 2.07 CSIRace5; 2.10 CSI Hispanic; 3.01 County ID; 3.05
Partnership Date; 3.06 Assessment Date.
Review of Existing Data
Data sources likely to be sustained
Data relevant to populations of interest (all consumers & FSPs)
Data available across multiple service years
Less than 10% missing or unknown values for CSI and DCR data (see Appendix C for details of recoding race/
ethnicity data fields)
Analytic Potential of Indicator
Analysis across time possible
Analysis among specific service populations possible
27
Figure 5.1. Race/ethnicity of mental health consumers
100.0%
90.0%
80.0%
70.0%
53.7%
60.0%
52.3%
50.0%
40.0%
32.5% 33.0%
30.0%
20.0%
8.5% 9.5%
10.0%
0.8% 1.0% 0.0% 0.0% 2.5% 2.5% 0.6% 0.5% 1.3% 1.1%
0.0%
White Hispanic/Latino Asian Pacific Islander Black American Multirace Other
Indian
FY 2008-09 (4,173) FY 2009-10 (3,977)
FY 2008-09 Unknown/Missing = 2.5% (n = 106); FY 2009-10 Unknown/Missing = 2.8% (n = 119)
Figure 5.2. Race/ethnicity of FSP consumers
100.0%
90.0%
80.0%
67.7% 67.3%
70.0%
60.0%
50.0%
40.0%
30.0% 24.8%
23.5%
20.0%
10.0% 4.1% 3.1%
0.0% 0.0% 0.0% 0.0% 3.0% 2.3% 2.0% 1.5% 0.8% 0.0%
0.0%
White Hispanic/Latino Asian Pacific Islander Black American Multirace Other
Indian
FY 2008-09 (133) FY 2009-10 (98)
FY 2008-09 Unknown/Missing = 6.3% (n = 9); FY 2009-10 Unknown/Missing = 3.9% (n = 4)
28
Figure 5.3. Mental health consumers by age group
100.0%
90.0%
80.0%
70.0%
60.0%
50.0%
42.5% 42.2%
40.0%
31.1% 31.9%
30.0%
22.7%
21.7%
20.0%
3.7% 4.2%
10.0%
0.0%
Children TAY Adults Older Adults
FY 2008-09 (4,282) FY 2009-10 (4,093)
FY 2008-09 Unknown/Missing = 0.0% (n = 0); FY 2009-10 Unknown/Missing = 0.0% (n = 0)
Figure 5.4. FSP consumers by age group
100.0%
90.0%
80.0%
70.0%
60.0%
43.4%
50.0%
40.0% 35.3% 35.3%
28.3% 29.4%
30.0% 28.3%
20.0%
10.0%
0.0% 0.0%
0.0%
Children TAY Adult Older Adult
FY 2008-09 (145) FY 2009-10 (102)
FY 2008-09 Unknown/Missing = 0.0% (n = 0); FY 2009-10 Unknown/Missing = 0.0% (n = 0)
29
Figure 5.5. Mental health consumers by gender
100.0%
90.0%
80.0%
70.0%
58.6% 59.2%
60.0%
50.0%
41.4% 40.8%
40.0%
30.0%
20.0%
10.0%
0.0%
Female Male
FY 2008-09 (4,282) FY 2009-10 (4,092)
FY 2008-09 Unknown/Missing = 0.0% (n = 0); FY 2009-10 Unknown/Missing = 0.0% (n = 1)
Figure 5.6. FSP consumers by gender
100.0%
90.0%
80.0%
70.0%
57.6%
56.5%
60.0%
50.0%
43.5% 42.4%
40.0%
30.0%
20.0%
10.0%
0.0%
Female Male
FY 2008-09 (138) FY 2009-10 (99)
FY 2008-09 Unknown/Missing = 4.8% (n = 7); FY 2009-10 Unknown/Missing = 2.9% (n = 3)
30
Priority Indicator 6: Demographic Profile of New Consumers
Indicator Summary
This indicator profiles new mental health consumers (i.e., served during FY, without service for prior six months)
overall and full service partners (FSPs) served during FYs 2008-09 and 2009-10.
Indicator Calculation
For all Contra Costa County mental health consumers, CSI data supports calculation of new versus past (i.e., initial
services received prior to the given FY) consumers overall and within race/ethnicity, age, and gender categories.
The frequency of new consumers served was divided by all previous consumers served, in each fiscal year, to
calculate the proportion of new consumer served. This same calculation was conducted within each demographic
category (race/ethnicity, age, and gender), in each FY (see Figures 6.1 – 6.4, below).
For FSPs, Santa Cruz County DCR data supports calculation of new versus existing (i.e., current full service partners)
overall and within race/ethnicity, age, and gender categories. The frequency of new consumers served was divided
by all existing consumers, in each fiscal year, to calculate the proportion of new consumers served. This same
calculation was conducted within each demographic category (race/ethnicity, age, and gender), in each FY (see
Figures 6.5 – 6.8, below).
Data Sources
Client & Service Information (CSI) Data Fields: H-01.0 County / City / Mental Health Plan Submitting Record; H-02.0
County Client Number; C-05.0 Gender; C-09.0 Ethnicity; C-10.0 Race; S-05.0 Mode of Service; S-16.0 From / Entry
Date; S-17.0 Through / Exit Date; S-23.0 Date of Service.
Data Collection and Reporting (DCR) Data Fields: 1.01 Global ID; 1.02 Assessment ID; 1.04 Date Partnership Status
Change; 1.05 Partnership Status; 1.07 Age Group; 1.08 Assessment Type; 2.01 CSI Date of Birth; 2.02 Gender; 2.03
CSIRace1; 2.04 CSIRace2; 2.05 CSIRace3; 2.06 CSIRace4; 2.07 CSIRace5; 2.10 CSI Hispanic; 3.01 County ID; 3.05
Partnership Date; 3.06 Assessment Date.
Review of Existing Data
Data sources likely to be sustained
Data relevant to populations of interest (all consumers & FSPs)
Data available across multiple service years
More than 10% missing or unknown values for DCR Data and FY 2008-09 CSI data; less than 10% missing
or unknown values for FY 2009-10 CSI data (see Appendix C for details of recoding race/ ethnicity data
fields)
Analytic Potential of Indicator
Analysis across time possible
Analysis among specific service populations possible
31
All Consumers – Data Source: Client & Service Information (CSI)
Figure 6.1. New and continuing mental health consumers
100.0%
90.0%
80.0% 74.9%
66.8%
70.0%
60.0%
50.0%
40.0% 33.2%
30.0% 25.1%
20.0%
10.0%
0.0%
FY 2008-09 (7,698) FY 2009-10 (9,483)
Previous Consumers New Consumers
FY 2008-09 Unknown/Missing = 16.0%; (n = 1,515); FY 2009-10 Unknown/Missing = 0.0% (n = 0)
Figure 6.2. Race/ethnicity of new mental health consumers
100.0%
90.0%
80.0%
70.0%
60.0%
51.8% 51.3%
50.0%
40.0%
33.3% 32.6%
30.0%
20.0%
9.5% 10.6%
10.0%
0.7% 0.9% 0.0% 0.0% 2.7% 2.7% 0.7% 0.6% 1.4% 1.3%
0.0%
White Hispanic/Latino Asian Pacific Islander Black American Multirace Other
Indian
FY 2008-09 (2,590) FY 2009-10 (2,307)
FY 2008-09 Unknown/Missing = 2.2%; (n = 59); FY 2009-10 Unknown/Missing = 3.0% (n = 71)
32
Figure 6.3. New mental health consumers by age group
100.0%
90.0%
80.0%
70.0%
60.0%
50.0%
41.8% 41.9%
40.0%
32.7% 32.6%
30.0%
22.6% 21.9%
20.0%
3.5%
10.0%
3.0%
0.0%
Children TAY Adults Older Adults
FY 2008-09 (2,649) FY 2009-10 (2,378)
FY 2008-09 Unknown/Missing 0.0% (n = 0); FY 2009-10 Unknown/Missing 0.0% (n = 0)
Figure 6.4. New mental health consumers by gender
100.0%
90.0%
80.0%
70.0%
57.2% 55.8%
60.0%
50.0% 42.8% 44.2%
40.0%
30.0%
20.0%
10.0%
0.0%
Female Male
FY 2008-09 (2,649) FY 2009-10 (2,377)
FY 2008-09 Unknown/Missing 0.0% (n = 0); FY 2009-10 Unknown/Missing 0.0% (n = 1)
33
FSP Consumers – Data Source: Data Collection and Reporting (DCR)
Figure 6.5. New and continuing FSP consumers
100.0%
100.0%
90.0%
80.0%
70.3%
70.0%
60.0%
50.0%
40.0%
29.7%
30.0%
20.0%
10.0%
0.0%
0.0%
FY 2008-09 (145) FY 2009-10 (102)
Previous Consumers New Consumers
FY 2008-09 Unknown/Missing 0.0% (n = 0); FY 2009-10 Unknown/Missing 0.0% (n = 0)
Figure 6.6. Race/ethnicity of new FSP consumers
100.0%
90.0%
80.0%
67.2% 67.3%
70.0%
60.0%
50.0%
40.0%
30.0% 24.6%
23.5%
20.0%
4.1%
10.0% 0.0% 0.0% 0.0% 0.0% 3.0% 2.2% 2.0% 3.0% 3.1% 0.0% 0.0%
0.0%
White Hispanic/Latino Asian Pacific Islander Black American Multirace Other
Indian
FY 2008-09 (134) FY 2009-10 (98)
FY 2008-09 Unknown/Missing 6.3% (n = 9); FY 2009-10 Unknown/Missing 3.9% (n =4)
34
Figure 6.7. New FSP consumers by age group
100.0%
90.0%
80.0%
70.0%
60.0%
50.0%
43.4%
40.0% 35.3% 35.3%
28.3% 28.3% 29.4%
30.0%
20.0%
10.0%
0.0% 0.0%
0.0%
Children TAY Adults Older Adults
FY 2008-09 (145) FY 2009-10 (102)
FY 2008-09 Unknown/Missing 0.0% (n = 0); FY 2009-10 Unknown/Missing 0.0% (n = 0)
Figure 6.8. FSP consumers by gender
100.0%
90.0%
80.0%
70.0%
57.6%
56.5%
60.0%
50.0%
43.5% 42.4%
40.0%
30.0%
20.0%
10.0%
0.0%
Female Male
FY 2008-09 (138) FY 2009-10 (99)
FY 2008-09 Unknown/Missing 4.8% (n = 7); FY 2009-10 Unknown/Missing 2.9% (n = 3)
35
Priority Indicator 7: Penetration of Mental Health Services
Indicator Summary
This indicator details rates of service to all consumers relative to estimates of need for service among Californians
earning less than 200% of the federal poverty income level. This metric is intended to show the extent to which
service levels are in line with the level of need.
Indicator Calculation
For all Santa Cruz County mental health consumers, the number of all mental health consumers served and among
demographic category (i.e., race/ethnicity, age, and gender) was divided by estimates of need for service
(Holzer Targets) among Santa Cruz County residents earning less than 200% of the federal poverty level
overall and among each demographic category (see Figures 7.1-7.5 below).
Data Sources
Client & Service Information (CSI) Data Fields: H-01.0 County/City/Mental Health Plan Submitting Record; H-02.0
County Client Number; C-05.0 Gender; C-09.0 Ethnicity; C-10.0 Race; S-05.0 Mode of Service; S-16.0 From /
Entry Date; S-17.0 Through / Exit Date; S-23.0 Date of Service.
Estimates of need for mental health services (Holzer Targets) among Californians earning less than 200% of the
federal poverty income level.
Review of Existing Data
Data sources likely to be sustained
Data appropriate for analysis of all mental health consumers. The estimates of need for service (Holzer
Targets) used are not appropriate points of comparison for FSP service levels.
Data available across multiple service years
Less than 10% missing or unknown values (see Appendix C for details of recoding race/ ethnicity data
fields)
Analytic Potential of Indicator
Analysis across time possible
Analysis among specific service populations possible
36
Figure 7.1. Penetration of mental health services
100.0%
80.0%
65.5% 62.2%
60.0%
40.0%
20.0%
0.0%
FY 2008-09 FY 2009-10
Consumers Served/Holzer
Target
FY 2008-09 (4,282/6,535)
FY 2009-10 (4,092/6,580)
Figure 7.2. Penetration of services by race/ethnicity
300.0%
259.6%
241.5%
250.0%
200.0%
150.0%
123.5% 117.4% 114.9%
89.6% 104.0% 89.4%
100.0%
84.2% 84.0%
50.0% 37.6% 33.1% 33.3%
35.7% 29.4% 33.3%
0.0%
White Hispanic/Latino Asian Pacific Islander Black Amerian Indian Multirace Other
FY 2008-09 FY 2009-10
FY 2008-09 FY 2009-10
White (2,240/2,501) (2,079/2,470)
Hispanic (1,357/3,608) (1,314/3,682)
Black (105/85) (101/86)
Asian (35/119) (40/121)
Pacific Islander (1/3) (1/3)
American Indian (26/25) (21/25)
Multirace (355/147) (379/146)
Other (54/47) (42/47)
37
Figure 7.3. Penetration of mental health services by age
100.0%
91.3%
84.6%
80.0% 75.4% 72.8%
54.3%
60.0%
51.2% 45.3% 47.1%
40.0%
20.0%
0.0%
Children TAY Adults Older Adults
FY 2008-09 FY 2009-10
FY 2008-09 FY 2009-10
Children (1,332/1,767) (1,304/1,790)
TAY (970/1,062) (889/1,051)
Adult (1,821/3,355) (1,727/3,372)
Older Adult (159/351) (173/367)
Figure 7.4. Penetration of mental health services by gender
100.0% 92.7%
88.9%
80.0%
60.0%
46.3% 43.3%
40.0%
20.0%
0.0%
FY 2008-09 FY 2009-10 FY 2008-09 FY 2009-10
Female Male
FY 2008-09 FY 2009-10
Female (1,771/3,827) (1,668/3,852)
Male (2,511/2,708) (2,424/2,727)
38
Priority Indicator 8: Access to a Primary Care Physician
Indicator Summary
This indicator details the level of access to a primary care physician reported among FSP consumers, during FYs
2008-09 and 2009-10.
Indicator Calculation
The ratio of Santa Cruz County FSP consumers indicating access to a primary care physician at any point during a
fiscal year to all Santa Cruz County FSP consumers served during a fiscal year was calculated (see Figure 8.1). This
ratio was also calculated within demographic categories (i.e., race / ethnicity, age, and gender), for each FY (see
Figures 8.2-8.4, below).
Data Sources
Data Collection and Reporting (DCR) Data Fields: 1.01 Global ID; 1.02 Assessment ID; 1.04 Date Partnership Status
Change; 1.05 Partnership Status; 1.07 Age Group; 1.08 Assessment Type; 2.01 CSI Date of Birth; 2.02 Gender; 2.03
CSIRace1; 2.04 CSIRace2; 2.05 CSIRace3; 2.06 CSIRace4; 2.07 CSIRace5; 2.10 CSI Hispanic; 3.01 County ID; 3.05
Partnership Date; 3.06 Assessment Date; 11.01 PhysicianCurr.
Review of Existing Data
Data source likely to be sustained
Data relevant to population of interest (FSPs). Relevant data not available to assess primary care access
among all mental health consumers (e.g., CSI).
Data available across multiple service years
Less than 10% missing or unknown values within key DCR fields (see Appendix C for details of recoding
race/ethnicity data fields)
Analytic Potential of Indicator
Analysis across time possible
Analysis among specific service populations possible
39
Figure 8.1. FSP access to a primary care physician
100.0%
92.0%
84.2%
80.0%
60.0%
40.0%
20.0%
0.0%
FY 2008-09 (96) FY 2009-10 (46)
FY 2008 -09 Unknown/Missing = 0.0% (n = 0); FY 2009-10 Unknown/Missing = 0.0% (n= 0)
Figure 8.2. FSP access to a primary care physician by race/ethnicity
100.0%
80.0%
67.0%
60.0% 57.1%
33.3%
40.0%
20.9%
20.0%
0.0% 0.0% 0.0% 0.0% 4.4% 0.0% 3.3% 2.4% 4.4% 2.4% 0.0% 0.0%
0.0%
White Hispanic/Latino Asian Pacific Islander Black Amerian Indian Multirace Other
FY 2008-09 (91) FY 2009-10 (42)
FY 2008 -09 Unknown/Missing = 5.2% (n = 5); FY 2009-10 Unknown/Missing = 4.3% (n= 2)
Figure 8.3. FSP access to a primary care physician by age group
100.0%
80.0%
60.0% 58.7%
41.7% 41.3%
40.0% 35.4%
22.9%
20.0%
0.0%
0.0% 0.0% 0.0%
Children TAY Adults Older Adults
FY 2008-09 (96) FY 2009-10 (46)
FY 2008 -09 Unknown/Missing = 0.0% (n = 0); FY 2009-10 Unknown/Missing = 0.0% (n= 0)
40
Figure 8.4. FSP access to a primary care physician by gender
100.0%
80.0%
56.5% 57.8%
60.0%
43.5% 42.2%
40.0%
20.0%
0.0%
Female Male
FY 2008-09 (92) FY 2009-10 (45)
FY 2008 -09 Unknown/Missing = 4.2% (n = 4); FY 2009-10 Unknown/Missing = 2.2% (n= 1)
41
Priority Indicator 9: Perceptions of Access to Services
Indicator Summary
This indicator provides insight into consumer and family perceptions of access to mental health services.
Indicator Calculation
Family members/caregivers and TAY respondents’ ratings (1–Strongly Disagree to 5–Strongly Agree) of two self-
report items (specified under the “Data Sources” section below) were averaged to calculate aggregate ratings of
perceptions of access to mental health services (see Figure 9.1 and Tables 9.1-9.2 below). Aggregate ratings were
calculated for each fiscal year. Ratings of 3.5 or greater generally indicate positive perceptions. This calculation
method is in line with previous DHCS practices.
Adult and Older Adult respondents’ ratings (1-Strongly Disagree to 5-Strongly Agree) of fourteen self-report items
(specified under the “Data Sources” section below) were averaged to calculate aggregate ratings of perceptions of
access to mental health services (see Figures 9.1 and Tables 9.1-9.2 below). Aggregate ratings were calculated for
each fiscal year. Ratings of 3.5 or greater generally indicate positive perceptions. This calculation method is in line
with previous DHCS practices.
Data Sources
Consumer Perception Surveys
Family members/caregivers and TAY self-report items analyzed (YSS/YSS-F):
The location of services was convenient for us.
Services were available at times that were convenient for us.
Adult and older adult self-report items analyzed (MHSIP):
The location of services was convenient (parking, public transportation, distance, etc.).
Staff were willing to see me as often as I felt it was necessary.
Staff returned my call in 24 hours.
Services were available at times that were good for me.
I was able to get all the services I thought I needed.
I was able to see a psychiatrist when I wanted to.
Review of Existing Data
Data source likely to be sustained
Data relevant to population of interest (i.e., convenience or random sample of all mental health consumers).
Data available across multiple service years
More than 10% missing or unknown values within key CPS scales
Analytic Potential of Indicator
Analysis among specific service populations possible
County analysis possible for FY 2008-09 (convenience sample), but data is not available at the county level for
FY 2009-10
42
Figure 9.1. Perceptions of access to service
Older Adults 4.15 (n=10)
Adults 4.22 (n=399)
TAY 4.17 (n=235)
Children 4.47 (n=270)
1.00 2.00 3.00 4.00 5.00
Table 9.1. Perceptions of access to services by race/ethnicity
Family Member/
TAY Adult Older Adult
Caregiver
White 4.50 (n=119) 4.14 (n=91) 4.21 (n=267) 4.17 (n=7)
Hispanic / Latino 4.47 (n=148) 4.19 (n=126) 4.19 (n=86) 4.00 (n=1)
Asian 4.08 (n=6) 4.38 (n=4) 4.12 (n=13) 4.17 (n=2)
Pacific Islander 4.79 (n=7) 4.50 (n=4) 4.04 (n=4) N/A
Black 4.18 (n=14) 4.09 (n=17) 4.21 (n=12) N/A
American Indian 4.36 (n=11) 4.09 (n=22) 3.94 (n=28) N/A
Other 4.45 (n=53) 4.24 (n=82) 4.12 (n=58) 4.17 (n=2)
FY 2008-09 Missing/Unknown Values: Family Member/Caregiver = 7.3% (n = 28), TAY = 8.2% (n = 31), Adult =
11.2% (n = 59), Older Adult = 14.3% (n = 2)
Table 9.2. New mental health consumers by gender
Family Member/
TAY Adult Older Adult
Caregiver
Female 4.49 (n=103) 4.31 (n=102) 4.30 (n=184) 4.17 (n=9)
Male 4.42 (n=146) 4.07 (n=117) 4.14 (n=183) 4.00 (n=1)
FY 2008-09 Missing/Unknown Values: Family Member/Caregiver = 8.1% (n = 22), TAY = 6.8% (n = 16), Adult = 8.5%
(n = 34), Older Adult = 9.1% (n = 1)
43
Priority Indicator 10: Involuntary Status
Indicator Summary
This indicator provides insight into the rates of involuntary status among all mental health consumers during FY
2008-09. Involuntary status refers to a legal designation that can be applied to individuals who are found to be a
danger to themselves and/or others, and/or gravely disabled.
Indicator Calculation
The California Department of Health Care Services (DHCS) reports incidents of involuntary status per 10,000
consumers for Santa Cruz County. Such rates are reported here (see Figure 10.1, below).
Data Sources
The California Department of Health Care Services provides reports of incidents of involuntary status (see
http://www.dmh.ca.gov/statistics_and_data_analysis/Involuntary_Detention.asp)
Review of Existing Data
Data source likely to be sustained
Data relevant to population of interest (all mental health consumers). Relevant data is not available to
specifically assess the involuntary status among FSP consumers.
Data available across multiple service years
Analytic Potential of Indicator
Aggregate data does not allow for analysis among specific (e.g., demographic) service populations
44
Figure 10.1. Involuntary status per 10,000 consumers, FY 2008-09 (NOTE: horizontal scale
reduced for ease of viewing)
Additional 14-day Intesive (suicidal) 0.1
14-day Intensive Treatment 16.3
72-Hour Evaluation and Treatment - Child 0.0
72 Hour Evaluation and Treament - Adult 0.0
0 10 20 30 40 50 60 70 80 90 100
45
Priority Indicator 11: Consumer Perceptions of Improvement in Well-
Being as a Result of Services
Indicator Summary
This indicator provides insight into consumer and family perceptions of well-being (i.e., outcomes, functioning, and
social connectedness) as a result of mental health services.
Indicator Calculation
Family members/caregivers and TAY respondents’ ratings (1–Strongly Disagree to 5–Strongly Agree) of eleven self-
report items (specified under the “Data Sources” section below) were averaged to calculate aggregate ratings of
perceptions of wellbeing as a result of mental health services (see Figure 11.1 and Tables 11.1-11.2, below).
Aggregate ratings were calculated for each fiscal year. Ratings of 3.5 or greater generally indicate positive
perceptions. This calculation was developed to approximate domains of wellbeing many respondents noted in
their feedback to our initial reports.
Adult and Older Adult respondents’ ratings (1–Strongly Disagree to 5–Strongly Agree) of fourteen self-report items
(specified under the “Data Sources” section below) were averaged to calculate aggregate ratings of perceptions of
wellbeing as a result of mental health services (see Figure 11.1 and Tables 11.1-11.2, below). Aggregate ratings
were calculated for each fiscal year. Ratings of 3.5 or greater generally indicate positive perceptions. This
calculation was developed to approximate domains of wellbeing many respondents noted in their feedback to our
initial reports.
Data Sources
Consumer Perception Surveys
Family members/caregivers and TAY self-report items analyzed (YSS/YSS-F):
My child is better at handling daily life.
My child gets along better with family members.
My child gets along better with friends and other people.
My child is doing better in school and/or work.
My child is better able to cope when things go wrong.
I am satisfied with our family life right now.
My child is better able to do things he or she wants to do.
I know people who will listen and understand me when I need to talk.
I have people that I am comfortable talking with about my child's problems.
In a crisis, I would have the support I need from family or friends.
I have people with whom I can do enjoyable things.
Adult and older adult self-report items analyzed (MHSIP):
I deal more effectively with daily problems.
I am better able to control my life.
I am better able to deal with crisis.
I am getting along better with my family.
I do better in social situations.
I do better in school and/or work.
I do things that are more meaningful to me.
I am better able to take care of my needs.
I am better able to handle things when they go wrong.
I am better able to do things that I want to do.
I am happy with the friendships I have.
I have people with whom I can do enjoyable things.
46
I feel I belong in my community.
In a crisis, I would have the support I need from family or friends.
Review of Existing Data
Data source likely to be sustained
Data relevant to population of interest (i.e., convenience or random sample of all mental health consumers)
More than 10% missing or unknown values within key CPS scales
Analytic Potential of Indicator
Analysis among specific service populations possible
County analysis possible for FY 2008-09 (convenience sample), but data is not available at the county level for
FY 2009-10
47
Figure 11.1. Perceptions of improvement in well-being as a result of services
Older Adults 3.85 (n=10)
Adults 3.93 (n=393)
TAY 4.03 (n=238)
Children 4.06 (n=271)
1.00 2.00 3.00 4.00 5.00
Table 11.1. Perceptions of improvement in well-being as a result of services by
race/ethnicity7
Family Member/
TAY Adult Older Adult
Caregiver
White 4.06 (n=119) 4.04 (n=92) 3.91 (n=267) 3.85 (n=7)
Hispanic / Latino 4.12 (n=148) 4.02 (n=127) 3.90 (n=86) 4.00 (n=1)
Asian 3.99 (n=6) 4.05 (n=5) 3.78 (n=13) 3.79 (n=2)
Pacific Islander 4.01 (n=7) 3.86 (n=4) 4.07 (n=4) N/A
Black 3.67 (n=13) 3.88 (n=17) 4.11 (n=12) N/A
American Indian 4.19 (n=11) 4.04 (n=23) 3.76 (n=28) N/A
Other 4.10 (n=53) 3.98 (n=83) 3.80 (n=58) 3.25 (n=2)
FY 2008-09 Missing/Unknown Values: Family Member/Caregiver = 7.0% (n = 27), TAY = 8.1% (n = 31), Adult = 9.8%
(n = 51), Older Adult = 14.3% (n = 2)
Table 11.2. Perceptions of improvement in well-being as a result of services by gender8
Family Member/
TAY Adult Older Adult
Caregiver
Female 4.10 (n=104) 4.05 (n=104) 4.03 (n=184) 3.84 (n=9)
Male 4.01 (n=146) 4.01 (n=118) 3.81 (n=183) 4.00 (n=1)
FY 2008-09 Missing/Unknown Values: Family Member/Caregiver = 7.7% (n = 21), TAY = 6.7% (n = 16), Adult = 6.6%
(n = 26), Older Adult = 0.0% (n = 0)
7 Consumer perception data for FY 2009-10 cannot be disaggregated by county.
8 Consumer perception data for FY 2009-10 cannot be disaggregated by county.
48
Priority Indicator 12: Satisfaction with Services
Indicator Summary
This indicator provides insight into consumer and family perceptions of satisfaction with mental health services.
Indicator Calculation
Family members/caregivers and TAY respondents’ ratings (1–Strongly Disagree to 5–Strongly Agree) of two self-
report items (specified under the “Data Sources” section below) were averaged to calculate aggregate ratings of
perceptions of access to mental health services (see Figure 12.1 and Tables 12.1-12.2, below). Aggregate ratings
were calculated for each fiscal year. Ratings of 3.5 or greater generally indicate positive perceptions. This calculation
method is in line with previous DHCS practices.
Adult and Older Adult respondents’ ratings (1–Strongly Disagree to 5–Strongly Agree) of fourteen self-report items
(specified under the “Data Sources” section below) were averaged to calculate aggregate ratings of perceptions of
access to mental health services (see Figure 12.1 and Tables 12.1-12.2, below). Aggregate ratings were calculated
for each fiscal year. Ratings of 3.5 or greater generally indicate positive perceptions. This calculation method is in line
with previous DHCS practices.
Data Sources
Consumer Perception Surveys
Family members/caregivers and TAY self-report items analyzed (YSS/YSS-F):
Overall, I am satisfied with the services my child received.
The people helping my child stuck with us no matter what.
I felt my child had someone to talk to when he/she was troubled.
The services my child and/or family received were right for us.
My family got the help we wanted for my child.
My family got as much help as we needed for my child.
Adult and older adult self-report items analyzed (MHSIP):
I like the services that I received here.
If I had other choices, I would still get services from this agency.
I would recommend this agency to a friend or family member.
Review of Existing Data
Data source likely to be sustained
Data relevant to population of interest (i.e., convenience or random sample of all mental health consumers).
More than 10% missing or unknown values within key CPS scales
Analytic Potential of Indicator
Analysis among specific service populations possible
County analysis possible for FY 2008-09 (convenience sample), but data is not available at the county level for FY
2009-10
49
Figure 12.1. Satisfaction with services
Older Adults 4.27 (n=10)
Adults 4.30 (n=401)
TAY 4.19 (n=240)
Children 4.43 (n=274)
1.00 2.00 3.00 4.00 5.00
Table 12.1. Satisfaction with services by race/ethnicity9
Family Member/
TAY Adult Older Adult
Caregiver
White 4.48 (n=119) 4.21 (n=92) 4.33 (n=267) 4.38 (n=7)
Hispanic / Latino 4.45 (n=150) 4.16 (n=129) 4.31 (n=86) 4.00 (n=1)
Asian 4.28 (n=6) 4.43 (n=5) 4.13 (n=13) 4.00 (n=2)
Pacific Islander 4.74 (n=7) 4.54 (n=4) 4.17 (n=4) N/A
Black 4.34 (n=14) 4.17 (n=17) 4.11 (n=12) N/A
American Indian 4.70 (n=11) 4.18 (n=23) 4.27 (n=28) N/A
Other 4.43 (n=53) 4.21 (n=83) 4.29 (n=58) 4.50 (n=2)
FY 2008-09 Missing/Unknown Values: Family Member/Caregiver = 5.8% (n = 22), TAY = 4.3% (n = 16), Adult = 6.8%
(n = 34), Older Adult = 7.7% (n = 1)
Table 12.2. Satisfaction with services by gender10
Family Member/
TAY Adult Older Adult
Caregiver
Female 4.50 (n=105) 4.26 (n=104) 4.46 (n=184) 4.30 (n=9)
Male 4.35(n=147) 4.10 (n=120) 4.17 (n=183) 4.00 (n=1)
FY 2008-09 Missing/Unknown Values: Family Member/Caregiver = 10.0% (n = 28), TAY = 12.2% (n = 31), Adult =
13.8% (n = 59), Older Adult = 16.7% (n = 2)
9 Consumer perception data for FY 2009-10 cannot be disaggregated by county.
10 Consumer perception data for FY 2009-10 cannot be disaggregated by county.
50
Appendix A – Priority Indicator Matrix
51
Appendix B – Preceding Reports
The evaluation team completed extensive groundwork before arriving at the conclusions contained
in this report. To date, the team has documented evaluation planning in six reports based on
statewide data made available online at UCLA and MHSOAC websites. Stakeholders were invited to
provide feedback on draft reports by responding to questions in an accompanying guidance
document. The invitation was shared online and through mass e-mail. Small stakeholder groups
participated in one of two webinars, or online orientations to the report, that provided an overview
to the report’s purpose and the types of feedback sought.
Report title: Defining Priority Indicators
Report version: Draft for stakeholder review
Here, the evaluation team began to refine the core set of priority indicators proposed by the
California Mental Health Planning Council to assess target outcomes of mental health consumers
and the performance of the mental health system.
Report title: Defining Priority Indicators
Report version: Final, revised with stakeholder input
In the revised report, the evaluation team illustrated how stakeholder feedback was integral to
indicator development. This report incorporated changes driven by stakeholders’ comments about
the comprehensiveness and appropriateness of the indicators.
Report title: Compiling Data to Produce All Priority Indicators
Report version: Draft for stakeholder review
In this report, the evaluation team proposed how priority indicators could be calculated using
existing statewide data. The report also detailed all possible data sources and specific variables or
data fields that might be used to build comprehensive priority indicators.
Report title: Compiling Data to Produce All Priority Indicators
Report version: Final, revised with stakeholder input
The initial report was revised to include information regarding measurement methods and the
adequacy of existing data sources, gathered through a stakeholder feedback process similar to that
used for the final Defining Priority Indicators report.
Report title: Initial Statewide Priority Indicator Report
Report version: Draft for stakeholder review
The evaluation team used select data from fiscal years 2008-09 and 2009-1011 to calculate priority
indicators as outlined in the previous report (Compiling Data to Produce All Priority Indicators).
Report title: Initial Statewide Priority Indicator Report
Report version: Final, revised with stakeholder input
The revised report incorporated several calculation updates guided by the MHSOAC. Updated
calculations largely focused on outcome data collected after enrollment. The report also reflected
input from consumer stakeholders about the use and accessibility of report features (illustrations,
descriptions, etc.) and language.
11 The team sorted through datasets from 2005 through 2011 in search of one or more fiscal years (FY) in
which data cells were largely filled where expected. Data from FY 2008-09 and 2009-10 fit this requirement.
52
Appendix C – Recoding Pre-DIG Race Data to Post-DIG Format
Stakeholder feedback to previous evaluation team reports suggested inconsistency and potential
inaccuracy among Race and Ethnicity data fields may be due in part to changes in the format of
these fields in the CSI and DCR data systems. In 2006, DMH implemented changes to the Race and
Ethnicity fields due to Uniform Data System/Data Infrastructure Grant (DIG) requirements from the
federal government (see DMH Information Notice: 06-02). Although DMH provided training about
these changes, Race and Ethnicity information seems to be reported inconsistently across counties.
Because demographic information in the CSI system is transferred to corresponding fields in the
DCR system, Race and Ethnicity information in both systems was analyzed but interpreted with
caution. To ameliorate potential shortcomings of this change, the evaluation team used pre-DIG
information to fill gaps in missing post-DIG Race and Ethnicity fields for analyses involving
demographic information. The table below details the recoding process.
Before Recode After
Recode (if
Post-DIG
field
empty)
Data Post-DIG Data
Pre-DIG Field Definition Definition
Value Field Value
Empty formerly
White 1 Race White or Caucasian 1
Ethnicity / Race
Empty formerly
Hispanic 2 Ethnicity Yes (Hispanic or Latino) Y
Ethnicity / Race
Empty formerly Black or African
Black 3 Race 3
Ethnicity / Race American
Empty formerly American Indian or
American Native 5 Race 5
Ethnicity / Race Alaska Native
Empty formerly
Amerasian A Race Other Asian O
Ethnicity / Race
Empty formerly
Hawaiian Native P Race Native Hawaiian P
Ethnicity / Race
Empty formerly
Multiple X Race Multiracial Multiracial
Ethnicity / Race
Empty formerly Other Asian or Pacific
4 Race Other Asian O
Ethnicity / Race Islander
53