BHSOAC
Mhsa Contract Deliverable 2b
Read the report at Behavioral Health Services Oversight & Accountability Commission ↗
Mental Health Services Act Evaluation:
Templates for Reporting Priority Indicators
Contract Deliverable 2B, Phase II
UCLA Center for Healthier Children, Youth and Families
EMT Associates, Inc.
Initially submitted on September 30, 2011
Revised version submitted on October 20, 2011
Final version submitted on November 2, 2011
The following report was funded by the
Mental Health Services Oversight and Accountability Commission.
Focus
The Mental Health Services Act (MHSA) evaluation team was charged with developing
templates and reports on statewide and county specific data that would improve understanding of
how the MHSA impacted consumers. Per contract language, the team will:
Design and complete statistical analyses and reports that measure impact of
MHSA at individual and system levels on indicators specified in the Matrix of
California’s Public Mental Health System Prioritized Performance Indicators at
the state and county levels. Draft templates, documentation of analysis, and
initial statewide reports will be circulated to key stakeholders and made
available to the public for input by posting on the web and making a hard copy
available upon request.
Individual client outcomes for full service partnerships (FSPs) by age group
must be addressed for each domain (education / employment, homelessness /
housing and justice involvement) as specified. Note: this impact analysis at the
individual level is limited to available data (i.e., a small segment of public
mental health clients, and full services partners, is reflected in this data.) Mental
Health system performance must address family/client/youth perception of well-
being, demographics of FSP population, FSP access to primary care,
penetration rate and changes in admissions for the entire public community
mental health population, involuntary care, and annual numbers served through
[Community Services and Supports programs] CSS.
The evaluation team submits the following report, which incorporates stakeholder insights, in
fulfillment of this charge.
Stakeholder Feedback
As noted in the contract language, input from key stakeholders and mental health service
advocates is key to developing final reports. To this end, all reports that the evaluation team
submits to the Mental Health Services Oversight and Accountability Committee (MHSOAC) –
including earlier versions of this report – are considered drafts until such input is received and
incorporated, to a reasonable extent, into subsequent reports. The evaluation team enlisted
feedback about this report from a wide range of stakeholders from July 29, 2011 through August
31, 2011 to create a more comprehensive, accurate report. Stakeholders received an e-mail
announcing the report’s availability on both the MHSOAC1 and UCLA2 web sites. A call for
feedback and an illustration of how the evaluation team would develop the report using
stakeholder input were embedded in the report introduction to clarify stakeholders’ roles in
report creation and to increase transparency about this process (see Illustration 1). A complete
account of organizations whose representatives responded to correspondence is included at the
end of this report (see Appendix D).
1 http://www.mhsoac.ca.gov/Announcements/announcements.aspx
2 http://healthychild.ucla.edu/MHSA_evaluation.asp
1
Illustration 1
Steps Leading to Statewide and County Specific Data Reports
Initial Statewide Evaluation
12/31/12
9/30/12
6/30/12
County reports incorporate
county context Three written County specific and statewide reports, on all priority indicators Step 7
(demographics, funding, etc)
Revised written report from STEP 5 in response to stakeholder input
Step 6
3/31/12
Initial draft written report submitted including data for all priority indicators
at the statewide level for the most recent one year period available
Step 5
12/31/11
Revised deliverable from STEP 2 in response to stakeholder input obtained.
Step 4
s
p
e 9/30/11
St
Revised deliverable from STEP 1 in response to stakeholder input
Step 3
9/30/11
Draft written documentation of the process for compiling the
data to produce the reports for all priority indicators
Step 2 Stakeholder/Consumer
Feedback
6/30/11
Draft proposed standardized template for reporting
all priority indicators
Step 1 Currently
Completed
6/30/11
Due Dates
2
Webinars were also conducted for two groups – MHSA stakeholders and the California Mental
Health Directors Association Indicators, Data, Evaluation, and Accountability (IDEA) Ad-Hoc
Committee – to describe the reports’ purposes, input needs, and feedback process. Webinars
were not designed to collect feedback; rather they were intended to provide a visual synopsis of
deliverables. Given the number of webinar participants, it was not feasible to account for all
stakeholders’ comments. The evaluation team requested that all feedback be written and shared
through e-mail correspondence. The evaluation team encouraged all stakeholders to respond to
points of interest in the reports as well as use the accompanying guidance document to think
critically about questions posed by the evaluation team. Groups were invited to use their existing
internal processes for reviewing and responding to mental health-related reports; the evaluation
team did not impose any review protocol. The team only requested that feedback be specific (e.g.,
noting page numbers, specific priority indicators, or specific measures), rather than a set of
general comments, to optimize feedback use. Stakeholders largely accommodated this request.
The nature of responses ranged greatly. A table illustrating the types of feedback garnered by
each indicator and measure is located in Appendix E. Among stakeholder input were direct
responses to guidance questions, feedback that provided historic reasons for data quality,
concerns about the accuracy of particular indicators given data quality, and even the
ramifications of a university group assessing mental health service consumer outcomes instead of
the consumers themselves. The evaluation team anticipated feedback diversity – particularly
recommendations that would be at odds with each other – and devoted ample time to negotiating
what could and could not be addressed given the available time and data with which we were
provided to conduct the evaluation. The team articulated this point early in the reports in the
service of transparency and to moderate expectations.
Overview
The following report outlines a strategy to assess the Mental Health Services Act (MHSA)
impact throughout California. The strategy would be used to create county-level and state-level
reports on outcomes related to Community Services and Supports (CSS) program outcomes
using data that all counties collect regularly. The report summarizes how impact would be
measured across four target age groups – children, TAY, adults, and older adults – using
“priority indicators” that reflect target domains in which MHSA impact should be evident. Thus,
the report does not include data analysis; rather it explains data that exists.
The report begins with a brief history of priority indicators and their intended use. After
explaining select terms that the evaluation team will use throughout this and other reports, we
more fully describe priority indicators, including the criteria used for review and the data that
could be used to create priority indicators, and insights about the state of this data per
stakeholders. We then describe the indicators in detail, including their relevant measures, data
sources, and limitations. Detailed justification of each priority indicator follows. The report
concludes with possible, practical ways the indicator set can improve what we know about
MHSA impact. The report should be read as one step in a process toward creating measurable
3
indicators of consumer outcomes and mental health system performance with available data. The
report is not a culmination; rather it explores what exists at this early stage of the process that
could prove useful in developing a statewide evaluation.
Background: Priority Indicators
To capture how the MHSA impacts consumers throughout the state, the California Mental Health
Planning Council (referred to throughout as The Planning Council) proposed a set of
performance outcomes for CSS programs. The CSS outcomes were re-conceptualized as
indicators for mental health activities and services throughout California. These priority
indicators are broadly defined as key measures of MHSA impact – the reduction of negative
outcomes or increase in positive outcomes at the individual (consumer outcomes), system
(county mental health system performance), and community levels. For example, rates of
consumer homelessness and incarceration should decrease under the MHSA while client
satisfaction with services and mental health promotion throughout communities should increase.
The set of priority indicators was derived from discussions involving the Planning Council and
mental health service stakeholders with the goal of streamlining the MHSA’s monitoring and
planning activities. The need for such indicators was also discussed in the report Evaluation
Brief: Summary and Synthesis of Findings on CSS Consumer Outcomes, submitted in preparation
for the MHSA evaluation. The Planning Council decided to create priority indicators using data
that was already collected across counties, reflected current statues related to the Act, was
included in the federal data reporting system, and seemed intuitive to mental health service
consumers and other stakeholders. The current indicator set – ultimately adopted by the Mental
Health Services Oversight and Accountability Commission – is illustrated in Appendix A and
has since garnered attention as a way to monitor quality improvement. The Planning Council
sees the benefit of these indicators stating,
Tracking one’s performance on key indicators over time and/or across
programs and/or against other comparable counties can provide useful
information to those planning, operating, and monitoring services.3
This report further hones the indicator set. Through data sorting and verification, the indicator set
is revised to give a fuller picture of how MHSA contributes to consumers’ lives and shapes
mental health service system performance.
Objectives
The objectives of this report are two-fold. The evaluation team was charged with examining 1) if
data already collected by county agencies could sufficiently measure individual-level and
system-level priority indicators, identifying gaps and redundancies among indicators (if any).
Using these findings, and with stakeholder input, the evaluation team would 2) create data
templates for reporting priority indicators.
3 From the report “Performance Indicators for Evaluating the Mental Health System” published by the California
Mental Health Planning Council (January 2010; p.3).
4
Creating a Measurement Framework
Defining Terms
The evaluation team aims to increase understanding around this and other deliverables related to
the MHSA evaluation. To ensure that language is clear and accessible to readers throughout the
report, we include a glossary for reference (Appendix B). When possible, our team shares how
we interpret terms to include the reader and aid understanding especially where concepts become
more complex.
Conceptualizing “Individual-level” and “System-level” Indicators
The performance indicator framework developed by The Planning Council distinguishes between
“individual-level indicators” and “system-level indicators.” These terms are widely used in
performance monitoring systems, but it is important to clarify the terms to ensure understanding
of their relevance, relationship, and priority in the measurement system described here.
The individual receiving mental health services is considered the consumer whether they are a
child, transition age youth (TAY; 16-25 years of age), adult, or older adult. A review of relevant
documents indicates that methods of measuring individuals’ mental health vary and can include
indicators such as fixed attributes a consumer brings to services (i.e., demographics, education
level); internal attributes (i.e., psychological and social development); behavior (i.e., the extent to
which one exercises self-restraint); or one’s perception (i.e., assessment of personal growth).
Each measurement is a mental health indicator that is specific to and bound by the consumer. To
achieve a broader understanding of the consumer, individual-level indicators can be derived from
the target person or others in immediate contact with the consumer; a parent or teacher might
provide responses about a target child, for example, who might also provide feedback about his
or her behavior. Based on these points, we define individual-level indicators as measurement of
mental health and cues of mental health service impact on consumers (and by extension, their
families).
Mental health systems can be explained as the overall context (service, procedures and policies)
within which mental health service agencies operate. Systems encompass all agencies, the
consumer and families they serve, their operations (service delivery, budgeting, administration,
client and staff satisfaction, etc.), and the resources and policy supports required to maintain
these systems. Researchers have offered a handful of indicators that better craft what is meant by
“system-level.” These include,
Formal commitments to a [mental health services] approach,
sustainability of an initiative or policy agenda, incentives to encourage
incorporation of [mental health] principles at the [agency] level,
opportunities for [stakeholder engagement] in governance and policy
5
making, and accountability for positive [consumer] development outcomes
and provision of essential supports at system and [agency] levels.4
In sum, we define system-level indicators as aggregated measures of the service population, the
structure and process of mental health services, and the outcomes and experiences of care.
We recognize that overlap exists between indicator levels when responses are aggregated. For
example, individuals’ self-reported rates of well-being – an individual-level indicator – can
provide an assessment of system-level performance when combined as a group response.
Whereas, a system level indicator such as the number of mental health consumers served is not
useful as a consumer level indicator. System indicators entail a level of measurement or
aggregation of consumer-level data at the system level, and measurement of consumer or
agencies is necessary for individual level indicators.
Reviewing Priority Performance Indicators, Measures and Data Sources
The UCLA/EMT team considered several performance measurement criteria (outlined below)
when evaluating the quality and utility of existing County Mental Health System Performance
Indicators (i.e., consumer and system level indicators; see Appendix A). To ensure consistency
with, and build upon, previous work to develop a comprehensive performance measurement
framework, we reviewed the criteria used by the California Department of Mental Health’s
Quality Improvement Committee5 (QIC) and The Planning Council6 to establish indicators for
the MHSA performance measurement system. The team also conducted a thorough review of
literature regarding mental health service performance measurement. The review provided
necessary background to evaluate the criteria used by the QIC and The Planning Council as well
as identify gaps and redundancies among the performance indicators those criteria were used to
develop. In this manner, a wide set of mental health consumer and system level measurement
domains and relevant indicators were cast. The quality and utility of measures and data sources
that could potentially be used to operationalize indicators was reviewed according to several
criteria outlined below. The data quality/utility review revealed that there are key elements of
service delivery and outcomes for which data sources do not contain adequate measurement
properties or are not readily available. In such cases additional data collection options are
presented. These methods and criteria guide systematic evaluation of consumer and system level
indicators and their underlying measures, to ensure all relevant domains of County Mental
Health Systems are captured in the most rigorous and comprehensive manner possible to
ultimately produce meaningful and actionable results for users (e.g., consumers/families,
policymakers, and providers) who strive to improve the quality of mental health services.
Specific criteria used to evaluate each indicator, measure and data source are detailed below.
4 John W. Gardner Center for Youth and Their Communities (2009, October). Positive youth development:
Individual, setting and system level indicators (Issue Brief). Stanford, CA: Dukakis, K., London, R., McLaughlin,
M., & Williamson, D.; p. 5.
5Chapter 93, Statutes of 2000, an omnibus Health Trailer Bill to the Budget Act of 2000, recognized the Quality
Improvement Committee (QIC) in law.
6California Mental Health Planning Council’s Performance Indicators for Evaluating the Mental Health System,
January 2010.
6
Criteria Used to Assess Performance Indicators
Performance Indicator Coverage. To ensure all actionable points in the process of MHSA
implementation are assessed, measures of County Mental Health System Performance should
pertain to one of the following domains:7
Performance Indicator Quality. County Mental Health System Performance measures were
evaluated for:
Quality, or the extent to which they are meaningful, unambiguous and widely understood
by all stakeholders, and drive improvement;
Support in the research base, suggesting the indicator had been informative and useful
across different mental health systems;
Ability to be operationalized using data accessible/obtainable by the evaluation team,
such as existing databases, or additional primary data collection, so as not to add
significant burden to the measurement framework;
Based on a high level of data integrity (i.e., data collection is embedded within the
normal procedures of County MHSA program, collected with fidelity, reliability), so as
not to impress undue burden on the evaluation resources of counties;
Linked to critical goals and key drivers of MHSA (i.e., core values), such that the
measurement framework is reflective of the overall orientation of the MHSA initiative.
7 Lutterman, T., Ganju, V., Schacht, L., Shaw, R., Monihan, K. (2003). Sixteen state study on mental health
performance measures. DHHS Publication No. (SMA) 03-3835. Rockville, MD: Center for Mental Health Services,
Substance Abuse and Mental Health Services Administration.
7
Performance Indicator Practicality. Measures were also assessed for their practicality, or the
extent to which they are useful for evaluation purposes and statistical analysis. These relate
largely to the degree which indicator concepts are clearly defined, evidence-based, feasible in
terms of data availability, understandable and actionable. Indicators must provide information to
help improve and maintain MHSA services as well as provide statistical indication of change in
services and their impact in consumers and families. Among evaluative needs, indicators should
reflect the following criteria:
Able to drive improvement (e.g., produce actionable results)
Useful for identifying opportunities for improvement (e.g., gaps or redundancies in
services)
Useful for tracking and comparing performance against both internal (e.g., organizational
goals) and external standards (e.g., national benchmarks)
Data Quality. Data used to represent priority performance indicators must also be evaluated for
quality and utility. Specifically, data must be consistent, trustworthy, and hold properties which
allow for the creation of each indicator and robust statistical analysis. Criteria used to evaluate
measures and data sources of each measure include:
Adequate base rate (i.e., how often an event occurs, or level at which a scaled response is
given on average, must not be so low as to make the indicator useless or meaningless)
Adequate variance (i.e., values of a given measure must be sufficiently distributed about
the mean such that statistical analysis can be conducted; values cannot all be clustered at
the same point)
Validity
o The measure is face valid, can conceptually and logically be said to measure what
it was intended to;
o The performance measure is internally valid and can logically be tied to a
particular program intervention or outcome;
o The indicator is externally valid and can logically be generalized to other
populations or programs.
Reliability (i.e., the indicator is consistent across time and cases)
Availability and completeness (i.e., indicator relevant data must be obtainable and
complete for populations of interest for the period of time under study)
Review of Performance Indicators
A systematic review of existing public mental health system prioritized performance indicators
(see Appendix A) was conducted utilizing the indicator coverage, quality, practicality, and data
quality criteria specified above. This review yielded several distinct areas of measurement (e.g.,
education, housing, justice involvement, service access and performance), as well as gaps and
redundancies in the existing measurement framework. The review of consumer and system level
indicators and proposed additional areas of measurement is summarized below.
Consumer-level Indicators
Consumer outcomes identified by The Planning Council reflect three broad, accessible indicator
8
categories of desired mental health intervention outcomes to be examined primarily across Full
Service Partnership consumers. The categories (i.e., Education / Employment, Homelessness /
Housing, and Justice Involvement) stem from previous studies and policies (e.g., Assembly Bill
2034) and were informed by indicators already in place for children’s systems of care (later
applied to systems for TAY, adults, and older adults). That is, consumer outcomes were
grounded in the premise that children should have stable homes, be in school, and stay out of
trouble. Similarly, TAY and adults should have stable homes, be employed, and stay out of
trouble. The Planning Council further limited indicators to those for which data was already
systematically collected across counties.
We suggest adding one indicator to the three proposed consumer outcome categories (home,
school/employment, justice involvement). Averting psychiatric hospitalizations is a factor to
consider when describing desired outcomes for mental health consumers. We are particularly
interested in consumers’ visits to and reliance on emergency facilities like hospitals and
psychiatric centers to manage their mental health – arguably the point at which management has
failed. Thus, we note that Emergency Care (e.g., the reduction of visits to related centers)
should be considered as an individual-level (consumer) outcome.
Feedback. Some stakeholders provided counterpoints to the ways by which consumer-level
indicators would be measured, yet their responses generally indicated agreement with the four
proposed indicators. As is detailed in the following pages, concerns emerged about the
completeness of the Emergency Care priority indicator. As the evaluation team defined it,
Emergency Care examines the number of visits to emergency facilities (e.g., hospitals and
psychiatric centers) for mental health management. Stakeholders noted that reasons for
emergency visits should also include physical injuries that can co-occur with mental health
episodes. Thus, a more complete Emergency Care priority indicator examines mental health
emergencies and the physical injuries or physical health disorders that co-occur with these
emergencies.
Stakeholders offered a fifth consumer-level indicator, which could arguably be categorized as
a consumer-level or system-level indicator. Social connections that might nurture associations
within families and communities were suggested as an additional strategy to evaluate MHSA
impact on its consumers. Whether or not these social connections are fostered through mental
health service organizations was not described in the feedback, leaving much to be discussed.
However, there is obvious use in considering social connections as a priority indicator given
the need for consumers to sustain mental health using this safety net.
System-level Indicators
A review of existing county mental health system performance indicators across all mental
health service consumers (see Appendix A) using the criteria specified above yielded three
domains of measurement, including system Access, Performance and Structure. Within each
domain the existing indicators are focused on measurement of system processes (e.g., services
administered, consumers reached) or system outcomes (e.g., consumer/family satisfaction,
penetration rate). These domains and levels of measurement are in line with previous
9
evaluations of mental health systems.8,9 However, a review of the mental health system
measurement literature revealed several gaps or redundancies among existing system
performance indicators. To reduce conceptual and measurement redundancy, and address gaps
in system performance measurement, additional or revised indicators proposed, include:
All proposed Priority System Performance Indicators are outlined in Template 2 and detailed below.
Feedback. Stakeholders largely expressed agreement that proposed system level indicators
are comprehensive and appropriate for monitoring mental health system processes. Most
stakeholders articulated concern regarding the use of specific existing data as the primary
sources for compiling system indicators, citing issues with data availability and reliability.
Stakeholders also provided feedback suggesting revisions to a limited number of system
indicators. Specifically, stakeholders asked us to consider alternative time periods with which
to measure and compare events or services, such as the number of new or high need
consumer served currently or over the past year. Some also proposed examining consumer
wellbeing among specific consumer groups (e.g., age groups, ethnic groups).
8 Lutterman, T., Ganju, V., Schacht, L., Shaw, R., Monihan, K. (2003). Sixteen state study on mental health
performance measures. DHHS Publication No. (SMA) 03-3835. Rockville, MD: Center for Mental Health Services,
Substance Abuse and Mental Health Services Administration.
9Hermann, R.C., Mattke, S., Somekh, D., Silfverhielm, H., Goldner, E., Glover, G. Pirkis, J., Mainz, J., Chan, J.A.
(2006). Quality indicators for international benchmarking of mental health care. International Journal for Quality in
Health Care, September 2006, 31-38.
10
Orientation to Templates
The subsequent templates represent a framework of proposed Priority Consumer and System
Performance Indicators. The templates are intended to represent a menu of indicators, selected
through a detailed review of existing and potential indicators against the indicator coverage,
quality, practicality, and data quality criteria specified above. The columns from left to right
detail the measurement domains, performance indicators, potential measures, and potential or
proposed data sources. Additional data collection options are proposed for further consideration,
where supplementary data may strengthen an indicators or existing data is insufficient. Proposed
indicators, which are revisions of or complements to the performance indicators established by
the California Mental Health Planning Council are highlighted. Proposed external data sources
(e.g., California Health Interview Survey) or new primary data collection are identified where
necessary.10
Two templates, reflecting consumer level priority indicators (Template 1) and mental health
system level indicators (Template 2) are presented. Each template includes 1) the domain in
which MHSA impact should be evident (e.g., education and employment), 2) priority indicators
that reflect said domains, 3) possible ways to measure the priority indicator, and 4) every
potential data source that includes at least one variable deemed useful in calculating the priority
indicator. Although the Planning Council envisioned consumer outcomes to be measured across
Full Service Partnership consumers (children, TAY, adults, and older adults) and system
outcomes to be measured across all mental health service consumers (see Appendix A), data
sources in both tables reflect possibilities for outcome calculations across all mental health
services consumers (via the Consumer Services and Information [CSI] system) as well as
persons enrolled in Full Service Partnerships (via the Data Collection and Reporting [DCR]
system). Templates developed using a thorough review of all data dictionaries related to mental
health services, should be read as an inventory of available information. These templates have
been revised to reflect stakeholder feedback regarding data source appropriateness and validity.
Blue fields indicate stakeholder feedback. Identifying this information is one step in a process
toward sorting and selecting variables, verifying data collection associated with selected
variables, and testing data fidelity and reliability in an evaluation of MHSA impact.
Please note that templates are not analysis plans that describe what can be learned for specific
mental health consumer groups (e.g., women, Native Americans, "high need," etc.). Rather, they
present priority indicators and their proposed measurement across four prescribed age groups –
children, TAY, adults, and older adults. Attention to consumers by demographic group exists
within system level indicators (refer to Appendix A), but exactly what information indicators can
provide cannot be stated without a full review of mental health service data. Following review of
the data (to be received during mid-October), the evaluation team can make more definitive
statements about which demographic groups (age groups excepted) can be described using
existing data.
10 Contractually, the evaluation team is responsible for coordinating data collection related to the Participatory
Research component of this project (Phase 3 Deliverables 2.a.1 and 2.b.2.). Beyond this requirement, the team only
offers suggestions about where new data collection can create more complete mental health service datasets.
11
Template 1. Initial Proposed Template for Reporting Core Priority Indicators: Individual-
level (Consumer) Outcomes for Full Service Partnerships11
Blue fields indicate stakeholder feedback.
Domain Indicator Potential Measure(s) Potential/Proposed Data
Source(s)
1. 1.1 Average attendance – Number (increase) of days at school Data Collection and
Education/ score per year annually Reporting (DCR) System12
Employment Consumer Services and
Information (CSI) system
Youth Services Survey for
Families (YSS-F)
Proposed data collection
1.2 Proportion Number (increase) of the consumers DCR
participating in paid and participating in paid and unpaid CSI
unpaid employment employment;
Number of days employed
2. 2.1 Housing Number (increase) of days that children or DCR
Homelessness/ situation/Index – score TAY (younger than 18 years) live in the CSI
Housing family home or a foster home; Youth Services Survey
Number (increase) of TAY or adults with (YSS)
independent residential statuses;
Number (increase) of older adults with
stable housing;
Number of days in housing
3. Justice 3.1 Justice involvement Number (decrease) of consumer arrests; DCR
Involvement Number (decrease) of incarcerations CSI
YSS-F
MHSIP-Adult
MHSIP-Older Adult
4. Emergency 4.1 Emergency Number (decrease) of consumer visits to the DCR
Care intervention for mental hospital or psychiatric health facility for CSI
health episodes mental health episodes annually
4.2 Emergency Number (decrease) of consumer visits to the CSI
intervention for co- hospital for physical injuries or physical
occurring physical injury health disorders that co-occur with mental
health episodes
5. Social 5.1 Proportion who Number of family members the consumer Proposed data collection
Connections identify family13 support identifies as reliable supporters, or persons
who are consistently present for the
consumer
5.2 Proportion who Number of community (non-family) Proposed data collection
identify community members that the consumer identifies as
support reliable supporters;
Number of organizations the consumer
identifies as providing quality services
when needed
11 Data sources that reflect all mental health service consumers have been added in the event that knowledge broader
than what is learned about Full Service Partnership Consumers is sought. Templates are arranged according to The
Planning Council’s vision, illustrated in Appendix A.
12The Data Collection and Reporting (DCR) system collects data for consumers who are enrolled in Full Service
Partnerships only.
13 “Family” may or may not include caregivers depending on each consumer’s designation.
12
Template 2. Initial Proposed Template for Reporting Comprehensive Priority Indicators:
System-level Outcomes
Domain Indicator Potential Measure(s) Potential/Proposed Data
Source(s)
6. Access 6.1 Demographic Age, gender, race/ethnicity, language Consumer Services and
Profile of Consumers spoken of consumer population (overall Information (CSI) system;
Served and FSP) Data Collection and Reporting
(DCR) system
6.2 New Consumers Age, gender, race/ethnicity of new DCR;
by Demographic consumer population in comparison to CSI
Profile those receiving services for more less
than 6 months, 1 year, and more than 1
year
6.3 High Need Homeless (currently and past 12 DCR;
Consumers Served months); CSI
Unemployment (currently and past 12
months);
Consumer with justice involvement;
Consumers with multiple psychiatric
hospitalizations
6.4 Access to Primary Consumers who have a primary care DCR;
Care Physician physician currently; CSI;
Consumers who have had a primary care Proposed data collection (e.g.,
physician for the past 12 months surveys) recommended
6.5 Consumer / Family Perceived access to services YSS;
Perceptions of Access YSS-F;
to Services MHSIP-Adult;
MHSIP-Older Adult;
Primary data collection (e.g.,
surveys, interviews, or focus
groups; proposed data collection)
7. 7.1 Consumers Served Ratio – Numerator: Consumers served / Quarterly Progress Reports;14
Performance Annually through CSS* Denominator: CSS consumers targeted in Annual Updates
county plan
7.2 Involuntary Care* Ratio – Numerator: seclusions / Annual Report on Involuntary
Denominator consumers served; Detentions;
Ratio – Numerator: restraints / DCR;
Denominator consumers served CSI
7.3 24-hour Care* Ratio – Numerator: utilization of MHRC, DCR;
SNF, SH / Denominator: TAY, Adult, CSI
Older-adult populations;
Ratio – Numerator: utilization of CTF,
RCL 14, MHRC / Denominator: Child
population;
Consumers in IMD, MHRC, SNF, SH by
race/ethnicity;
Readmission to acute care facility within
30/180 days
7.4 Appropriateness of Treatment protocols for co-morbidity; DCR;
Care* Hospital readmission rate; CSI;
Average length of stay in acute care; YSS;
14 Key informants strongly suggest replacing data collected for quarterly reports (CSS Exhibit 6) with annual
updates, which were not a part of the initial data dictionary review and might face a shift in standards in light of
Assembly Bill 100. The evaluation team will explore the differences in the reports’ data quality and regularity in
future publications.
13
Domain Indicator Potential Measure(s) Potential/Proposed Data
Source(s)
Consumer/family perceptions of YSS-F;
appropriateness of care MHSIP-Adult;
MHSIP-Older Adult;
Primary data collection (e.g.,
surveys, interviews, or focus
groups; proposed data collection)
7. 7.5 Continuity of Care* Use of crisis services; DCR;
Performance Services provided in community settings; CSI;
(continued) Consumer Satisfaction Surveys –
Documented discharge plans
YSS, YSS-F, Adult, and Older
Adult;
Primary data collection (e.g.,
surveys, interviews, or focus
groups; proposed data collection)
7.6 Penetration Rate Ratio – Numerator: Consumers / DCR;
Denominator: targeted for service or CSS Exhibit 6 Quarterly Progress
populations of interest (e.g., age, gender, Reports;
race/ethnicity, socioeconomic status); California Health Interview Survey
Ratio – Numerator: Consumers / (CHIS; proposed external data
Denominator: Holzer Targets source)
Census data
7.7 Consumer Client/family perception of improvement YSS;
Wellbeing in functioning (current, over time, among YSS-F;
high need groups); MHSIP-Adult;
Client/family perception of quality of life MHSIP-Older Adult;
(current, over time, among high need Primary data collection (e.g.,
groups) surveys, interviews, or focus
groups; proposed data collection)
7.8 Satisfaction Consumer/family satisfaction with the YSS;
care or service YSS-F;
MHSIP-Adult;
MHSIP-Older Adult
8. Structure 8.1 Workforce Demographic composition of mental Cultural Competence Plans;
Composition* health workforce Ratio – Numerator: WET Plans;
Staff / Denominator: consumers; Primary data collection (e.g.,
Consumer/family member employment surveys, interviews, or focus
in the mental health system (i.e., number, groups; proposed data collection)
FTE, % of workforce)
8.2 Evidence- Use of evidence-based practices; CSI;
Based/Best Practice Fidelity of best practices to established Primary data collection (e.g.,
Programs and models; surveys, interviews, or focus
Services* Receipt and experience of best practice groups; proposed data collection)
services/supports among consumers/
families
8.3 Cultural Client and family perceptions of cultural YSS;
Appropriateness of appropriateness YSS-F;
Services MHSIP-Adult;
MHSIP-Older Adult
8.4 Recovery, Consumer/family member/staff Recovery Oriented Systems
Wellness, and perceptions of recovery orientation of Indicators Measure (ROSI;
Resilience Orientation system and services proposed data collection);
Developing Recovery Enhancing
Environments Measure (DREEM;
proposed data collection)
* Asterisks refer to indicators that are processes (not outcomes).
14
Mental Health Indicator Detail
To clarify the meaning, importance and potential utility of each domain and indicator, this
section provides detailed descriptions of the indicators summarized in the templates. This
discussion is based on research and professional literature, research briefs, and technical reports.
Individual-level (Consumer) Outcomes for Full Service Partnerships Indicator
Detail
Domain 1: Education/Employment
This domain encompasses indicators of education for children and transitional age youth (TAY)
younger than 18 years of age as well as employment indicators among TAY who are 18 and
older, adults and older adults.
1.1 Indicator: Average attendance – score per year15,16 (CMHPC Indicators #2 and #8)
Population: Children and TAY
Rationale for Inclusion: The number of days a youth attends school during a school year has
been used as an indicator of healthy development during adolescence. School attendance has
been associated with academic functioning, subjective well-being, and life satisfaction. Youth
who are more vulnerable to negative mental health outcomes such as low self-concept and
limited sense of social support have been linked with poor academic success including lower
assessments of school importance to achieve goals and limited motivation to self-regulate
learning behaviors. Further, mental health distress outside of the school environment has been
thought to redirect youths’ attentions away from attending school. This indicator will help
identify the extent to which MHSA programs bolster youths’ school attendance.
Measure: Number (increase) of days at school annually
Data Source(s): Data Collection and Reporting (DCR) System, Consumer Services and
Information (CSI) system, Consumer Satisfaction Surveys – Youth Services Survey for Families
(YSS-F) version, proposed new data collection
Stakeholder-informed Challenges and Limitations: Stakeholders reported that accurate school
data was difficult to access and normalize due to the types of schooling in which consumers were
enrolled. Alternative education and home schooling would have different attendance
requirements, for example. A ratio of school days attended to total school days would address
this issue (explored in Deliverables 2C and 2D), however no strategy exists to collect the total
number of school days from each school district in the state.
15 Suldo, S., & Shaffer, E., (2008). Looking beyond psychopathology: The dual-factor model of mental health in
youth. School Psychology Review, 37, 52-68.
16 Kearney, C., (2008). School absenteeism and school refusal behavior in youth: A contemporary review. Clinical
Psychology Review, 28, 451-471.
15
1.2 Indicator: Proportion participating in paid and unpaid employment17,18 (CMHPC
Indicators # 8 and #13)
Population: TAY, Adults, and Older Adults
Rationale for Inclusion: Unemployment has been identified as a negative outcome of untreated
mental illness. Successful employment has been linked to social networks, life stability, and
stamina. Some research has shown that vocational training, in combination with mental health
services, has been associated with positive employment outcomes such as higher likelihood of
being hired in competitive work and having an opportunity to work full-time. A count of all
consumers who engage in employment will help identify the amount of consumers employed
over time and account for the effectiveness of employment programs for consumers.
Measure: Number (increase) of the consumers participating in paid and unpaid employment
Data Source(s): DCR system, CSI system
Stakeholder-informed Challenges and Limitations: Stakeholders suggested adding the number of
days or period of time during which a consumer was employed. Stakeholders also noted that
employment information in the DCR system was very limited unlike the CSI system, which
accounts for all types of employment (part-time, volunteer, etc.). The evaluation team would
only have a robust understanding of consumer employment for persons who were not enrolled in
FSPs – those who are more likely to be employed in part-time and volunteer work. Last,
stakeholders offered that employment rates, like the economy, occur in cycles that could be
overlooked in analyses if unaccounted for.
Domain 2: Homelessness/Housing
This domain encompasses indicators of homelessness and the variety of housing situations
among all consumers (children, TAY, adults, and older adults).
2.1 Indicator: Housing Situation/Index – Score19 (CMHPC Indicators #1, #7, #12, #17)
Population: Children, TAY, Adults, and Older Adults
Rationale for Inclusion: Untreated mental illness has been linked to homelessness and the ability
to live independently. Supportive housing provided through MHSA programs is designed to give
independent living opportunities to “low-income adults, or older adults with serious mental
illness, and children with severe emotional disorders and their families who, at the time of
assessment for housing services, meet the criteria for MHSA services in their county of residence
and are homeless or at risk for homelessness.” Further, housing provisions might curb
17 Secker, J. & Membery, H. (2003). Promoting mental health through employment and developing healthy
workplaces: The potential of natural supports at work. Health Education Research, 18, 207-215.
18 Cook, J., Lehman, A., Drake, R., McFarlane, W., Gold, P., Leff, H., Blyler, C., Toprac, M., Razzano, L., Burke-
Miller, J., Blankertz, L., Shafer, M., Pickett-Schenk, S., & Grey, D. (2005). Integration of psychiatric and vocational
services: A multisite randomized, controlled trial of supported employment. American Journal of Psychiatry, 162,
1948-1956.
19 MHSA housing program: Background, and information about the application, commitment, and funding processes.
(2001, February 1). Retrieved May 20, 2011 from http://www.calhfa.ca.gov/multifamily/mhsa/process/MHSABackground.pdf
16
homelessness, which will subsequently decrease consumers’ vulnerability to justice involvement.
Identifying consumers’ housing situations will improve understanding of access to housing and
the range of living situations currently used.
Measure(s): Number (increase) of days that children and TAY live in the family home or a
foster home; number (increase) of TAY, adults, within dependent residential statuses; number
(increase) of older adults with stable housing
Data Source(s): DCR system, CSI system, Consumer Satisfaction Surveys – Youth Services
Survey (YSS) version
Stakeholder-informed Challenges and Limitations: Stakeholders noted three challenges with data
related to the Homelessness/ Housing indicator. 1) Both homelessness and housing can change
often among the group, yielding complex data. Stakeholders suggested becoming familiar with
this particular data before conducting analyses because of its ever-changing nature. 2) Housing
data from the CSI system is not updated as regularly as data from the DCR system, and lag might
create inaccuracies in the evaluation. 3) Children who are homeless by way of their parents are
not measured by the DCR. That is, a child who is with one or both parents can be categorized as
“housed” without any indication or measure of his or her parents’ living situation. Thus, housing
information in the DCR is not absolutely accurate. Beyond data challenges, stakeholders offered
that adding the number of days a consumer is housed would be useful to understand housing
stability.
Domain 3: Justice Involvement
3.1 Indicator: Justice Involvement20, 21 (CMHPC Indicators #1, #7, #12, #17)
Population: Children, TAY, Adults, and Older Adults
Rationale for Inclusion: Research has shown that a percentage of former inmates who became
mental health service consumers had been arrested previously for behaviors stemming from
preexisting disorders. That is, an episode left these consumers vulnerable to arrest and
incarceration. Among youth, some studies have found significantly higher occurrence of
externalizing behaviors, attention deficit, and defiance among those who had been arrested
compared to those who had not. This indicator will follow consumers’ interactions with the
justice system to explore how participation in MHSA programs shapes number of arrests.
Measure: Number (decrease) of consumer arrests or incarcerations
Data Source(s): DCR system, CSI system, Consumer Satisfaction Surveys – Youth Services
Survey for Families (YSS-F) version, Adult version, and Older Adult version
20 Daly, R., (2011, January 7). Study examines relationship of arrests, mental illness. Psychiatric News, 46, 9-10.
21 Center for Behavioral Health Services & Criminal Justice Research. (2009, September). The effects of mental
health problems on juvenile arrests (Research Brief). New Brunswick, NJ: Hirschfield, P., Maschi, T., Raskin White,
H., & Goldman Traub, L.
17
Stakeholder-informed Challenges and Limitations: Stakeholders described different ways to
conceptualize “arrests” to create a more specific indicator definition. Some suggested counting
incarcerations instead of arrests given that intercession by FSP teams sometimes prevents
incarceration. Others suggested counting new arrests given that consumers might be re-arrested
for the same offense due to probation violations. Also, the term “episodes” was contested as it
did not account for chronic mental health issues. In sum, further revision of what is meant by
Justice Involvement is needed. In addition to clarifying the indicator, stakeholders suggest
adding the number of days a consumer is held to create a rate that can be followed over time.
Domain 4: Emergency Care
4.1 Indicator: Emergency Intervention for mental health episodes (Proposed
Indicator)
Population: Children, TAY, Adults, and Older Adults
Rationale for Inclusion: Hospital stays and visits to psychiatric facilities for emergency
interventions might indicate poor or lack of mental health management. Mental health services
and related supports might curb the need for hospitalization related to mental health episodes.
This indicator can account for consumers’ hospitalizations and provide trends of reliance on
hospitals for mental health management.
Measure: Number (decrease) of consumer visits to the hospital or psychiatric facility for mental
health issues annually
Data Source(s): DCR system, CSI system
4.2 Indicator: Emergency Intervention for Co-occurring Physical Injury (Proposed
Indicator)
Population: Children, TAY, Adults, and Older Adults
Rationale for Inclusion: Hospital stays and visits for physical injuries that may or may not co-
occur with mental health issues. It is a secondary measure of mental health management and can
provide more accurate counts of episodes that might be missed by a count of facility visits for
mental health issues alone.
Measure: Number (decrease) of consumer visits to the hospital for physical injuries or physical
health disorders that co-occur with mental health episodes
Data Source(s): DCR system, CSI system
18
Stakeholder-informed Challenges and Limitations: Stakeholders suggested that creating an
appropriate Emergency Care indicator could be difficult given that 1) visits to the emergency
room and hospitalization are distinct in the DCR system, and 2) the CSI is not regularly updated
with visits and releases from medical facilities. Among additions to the proposed priority
indicator, stakeholders proposed adding the number of days a consumer spends in emergency
care given that FSP interventions might yield a shorter psychiatric stay. Stakeholders
recommended dividing the types of hospitalizations (mental health episodes from physical health
issues), when possible, with the understanding that these might co-occur. Stakeholders requested
new data collection to create a measure of co-occurring mental and physical health issues among
consumers as well as drug and alcohol use.
Domain 5: Social Connections
5.1 Indicator: Proportion Who Identify Family Support (Proposed Indicator)
Population: Children, TAY, Adults, and Older Adults
Rationale for Inclusion: Social Connections – a priority indicator that emerged from stakeholder
feedback – has been included in the service of identifying the family and community supports
that exist for consumers. Supports can provide consumers a sense of place, belonging, and
security that might have been absent or diminished over time. This priority indicator can be
considered both a consumer-level indicator when consumers actively seek and establish positive
connections that create a supportive community and system-level indicator when organizations
create paths toward support systems on consumers’ behalves.
Measure: Number of family members (stable or increase) the consumer identifies as reliable
supporters, or persons who are consistently present for the consumer
Data Source(s): Proposed new data collection
5.2 Indicator: Proportion Who Identify Community Support (Proposed Indicator)
Population: Children, TAY, Adults, and Older Adults
Rationale for Inclusion: This indicator assesses the extent of consumers’ social support networks
beyond the family. It is a measure of individuals and organizations that provide steadfast support
around mental health issues, education, and services.
Measure: Number of community (non-family) members (stable or increase) that the consumer
identifies as reliable supporters; number (increase) of organizations the consumer identifies as
providing quality services when needed
Data Source(s): Proposed new data collection
Stakeholder-informed Challenges and Limitations: The Social Connectedness priority indicator
was suggested during the feedback period. It is unclear, at present, if existing data accommodates
this indicator.
19
Mental Health System-level Outcomes for all Consumers Indicator Detail
Domain 6: Access
This domain encompasses indicators of consumers’ and families’ ability to obtain timely and
convenient care or service based on needs.
Processes
6.1 Indicator: Demographic Profile of Consumers Served (CMHPC Indicator #3022)
Rationale for Inclusion: Demographic description of those receiving services (all consumers and
FSP) within and across counties will provide a better understanding of who is accessing services.
Such information may provide insight into ways to improve service outreach and implementation.
Potential Measure(s): Age, gender, race/ethnicity, language spoken of consumer population
(overall and FSP)
Potential Data Source(s): DCR system, CSI system
Stakeholder Informed Challenges & Limitations: Demographic information (i.e., ethnicity,
gender) contained in the DCR is imported from the CSI database. Thus, any FSP consumer not
registered in the CSI system will have incomplete demographic information in the DCR
database.
6.2 Indicator: New Consumers by Demographic Profile (CMHPC Indicator #3423)
Rationale for Inclusion: Demographic description of all new consumers (i.e., those not receiving
services for prior 6 months) within and across counties, and in comparison to the existing service
population, will provide description of how access of services may be changing. Specifically,
this indicator may serve as a gauge of the penetration of outreach and engagement services,
including what has been done to engage underserved populations.
Potential Measure(s): Age, gender, race/ethnicity of new consumer population in comparison to
those receiving services for more less than 6 months, 1 year, and more than 1 year
Potential Data Source(s): DCR system, CSI system
6.3 Indicator: High Need Consumers Served24 (Proposed Indicator)
Rationale for Inclusion: Previous studies have indicated a high occurrence of mental illness
amongst the homeless, those who are unemployed25, and those in poverty. Homeless individuals
and those in poverty tend not to seek necessary supportive services. Thus, connecting these
22California Mental Health Planning Council’s Performance Indicators for Evaluating the Mental Health System,
January 2010.
20Health Care for the Homeless Clinicians’ Network. 2000. “Mental Illness, Chronic Homelessness: An American
Disgrace.” Healing Hands 4:1-6.
21Linn, M., Sadifer, R., and Stein, S. (1985). Effects of unemployment on mental and physical health. American
Journal of Public Health, 75, 502-506.
22California Mental Health Planning Council’s Performance Indicators for Evaluating the Mental Health System,
January 2010.
20
groups with appropriate services becomes difficult and requires extensive outreach and
engagement to do so. Profiling service to these groups will provide greater understanding of
access to services among these high need groups, within counties and across the state.
Potential Measure(s): Homeless (past 12 days and past 12 months); Unemployment (past 12
days and past 12 months); Consumers with justice involvement; Consumers with multiple
psychiatric hospitalizations
Potential Data Source(s): DCR system, CSI system
Stakeholder Informed Challenges & Limitations: Stakeholders expressed concerns that variables
included in the CSI database, such as Living Situation and Employment, may not be updated
regularly for non-FSP consumers. If these items are found to be unreliable, high need consumer
served through non FSP programs may not be accurately described.
Outcomes
6.4 Indicator: Access to Primary Care Physician26, 27 (CMHPC Indicator #3128)
Rationale for Inclusion: Individuals with mental illness tend to experience poor health, as
compared to the general population. The medical needs of those with mental illness are often not
met due to poor access to general health care. This indicator will provide indication of the extent
to which FSP services have been successful in connecting consumer with regular sources of
primary health care.
Potential Measure(s): Consumers who have a primary care physician currently/past 12 months
Potential Data Source(s): DCR system, CSI system, primary data collection (e.g., surveys,
interviews, or focus groups; proposed additional data collection)
Stakeholder Informed Challenges & Limitations: Stakeholders expressed concerns that CSI data
regarding access to a primary care physician may not be reliable. If this information is not
reliably reported, access to a physician among all mental health consumers cannot be accurately
assessed and comparisons to FSP consumers cannot be made.
26Druss, B.G. and Rosenheck, R.A. (1998). Mental Disorders and Access to Medical Care in the United States.
American Journal of Psychiatry, 155(12), 1775-1777.
27 Statutory outcome: Improve health and mental health (WIC 5801(d)(2), WIC 5806(a), WIC 5840(a), WIC
5840(c))
28California Mental Health Planning Council’s Performance Indicators for Evaluating the Mental Health System,
January 2010.
21
6.5 Indicator: Consumer / Family Perceptions of Access to Services29, 30 (Proposed
Indicator)
Rationale for Inclusion: Subjective evaluations of services can provide indications that barriers
may exist to accessing care or service. Additional qualitative data collection can provide
indications of the specific problems that may hinder access to care.
Potential Measure(s): Perceived access to services
Potential Data Source(s): Consumer Satisfaction Surveys – YSS, YSS-F, Adult, and Older
Adult; primary data collection (e.g., surveys, interviews, or focus groups; proposed data
collection)
Domain 7: Performance
This domain includes indicators of the extent to which county mental health system processes
met the values and expectations of consumers and families, communities, providers and the
MHSA initiative overall.
Processes
7.1 Indicator: Consumers Served Annually through CSS31 (FSP, GSD, Outreach &
Engagement; CMHPC Indicator #4332)
Rationale for Inclusion: Tracking the number of individuals targeted and served though CSS
services will provide a snapshot of system implementation and highlight progress toward
achieving service goals.
Potential Measure(s): CSS consumers targeted in county plan compared to those who were
served.
Potential Data Source(s): Quarterly Progress Reports (i.e., CSS Exhibit 6), Annual Updates
Stakeholder Informed Challenges & Limitations: Stakeholders expressed concerns that Exhibit 6
is not a reliable source for information regarding consumers targeted or served through CSS
programs due to the various ways in which counties define these categories. If data regarding
consumers targeted or served is found unreliable across counties, the statewide ratio of target
consumers to served consumers cannot be accurately estimated.
29Onken, S., Dumont , J., Ridgeway, P., Dornan, D., and Ralph, R. (2002). Mental Health Recovery: What Helps
and What Hinders? NASMHPD and NTAC.
30 Statutory outcome: Reduce disparities in access (MHSA Section 3(d), WIC 5878.3(b), (WIC 5813.5(d), WIC
5840(a), WIC 5830(a)(1))
31Statutory outcome: Implement MHSA county plans(WIC 5847(b))
32 California Mental Health Planning Council’s Performance Indicators for Evaluating the Mental Health System,
January 2010.
22
7.2 Indicator: Involuntary Care33 (CMHPC Indicators #35 & #3634)
Rationale for Inclusion: Tracking the number of consumers requiring therapeutic seclusion or
restraint, as compared to populations served (e.g., age groups) will provide indication of the
extent to which mental health systems employ these therapeutic strategies.
Potential Measure(s): Ratio of Involuntary Services to Consumers served and overall
Count/State population
Potential Data Source(s): Annual Report on Involuntary Detentions, DCR system, CSI system
7.3 Indicator: 24-hour Care35 (Revision of CMHPC Indicators #37-4136)
Rationale for Inclusion: The use of Long-Term Strategies for Community Placement and
Alternatives to Institutions for Mental Diseases (IMDs) among various populations/groups (e.g.,
age, race/ethnicity) can provide a picture of how consumers are cared for within county mental
health systems and across the state.
Potential Measure(s): Utilization of Institutions for Mental Disease (IMD)/Mental Health
Rehabilitation Centers (MHRC)/Specialized Nursing Facilities (SNF)/State Hospitals (SH)
compared to Child/TAY/Adult/Older-adult populations; Utilization of CTF, RCL 14, MHRC
compared to child population; Consumers in IMD/MHRC/SNF/SH by race/ethnicity;
Readmission to acute care facility within 30/180 days
Potential Data Source(s): DCR system, CSI system
7.4 Indicator: Appropriateness of Care37 (Proposed Indicator; Revision of CMHPC
Indicator #2438)
Rationale for Inclusion: This indicator will focus on the extent to which care or service is
relevant to consumer/family needs. Several factors may provide evidence of the appropriateness
of care or service, including: 1) the existence of treatment protocols for co-morbidity, as serious
mental illness often co-occurs with substance use disorders; 2) greater length of stay in acute
care facilities may indicate inadequate services or supports; 3) consumer/family perceptions of
the appropriateness of care they receive, including involvement and a sense of empowerment in
the treatment decision making process, will provide a key reflection of services as received.
33 Statutory outcome: Implement Recovery Vision (WIC 5813.5(d))
34California Mental Health Planning Council’s Performance Indicators for Evaluating the Mental Health System,
January 2010.
35 Lutterman, T., Ganju, V., Schacht, L., Shaw, R., Monihan, K. (2003). Sixteen state study on mental health
performance measures. DHHS Publication No. (SMA) 03-3835. Rockville, MD: Center for Mental Health Services,
Substance Abuse and Mental Health Services Administration.
36 California Mental Health Planning Council’s Performance Indicators for Evaluating the Mental Health System,
January 2010.
37McEwan, K., and Goldner, E. (2001). Accountability and performance indicators for mental health services and
supports: A resource kit. Department of Psychiatry, University of British Columbia. Retrieved March 16, 2011
(http://www.hc-sc.gc.ca/hppb/mentalhealth/service)
38 California Mental Health Planning Council’s Performance Indicators for Evaluating the Mental Health System,
January 2010.
23
Potential Measure(s): Treatment protocols for co-morbidity; Hospital readmission rate; Average
length of stay in acute care; Consumer/family perception of appropriateness of care
Potential Data Source(s): DCR system; CSI system; Consumer Satisfaction Surveys – YSS,
YSS-F, Adult, and Older Adult; document review and primary data collection (e.g., surveys,
interviews; proposed data collection)
Stakeholder Informed Challenges & Limitations: Assessing treatment protocols for co-morbidity
would require extensive document review to establish their existence in each county, and
additional data collection (e.g., surveys or interviews) to assess whether they are appropriately
applied.
7.5 Indicator: Continuity of Care39 (Proposed Indicator)
Rationale for Inclusion: This indicator will center on the extent to which county mental health
systems provide uninterrupted, coordinated care and services across programs, providers,
organizations, and levels of care/service.
Potential Measure(s): Use of crisis services; Services provided in community settings;
Documented discharge plans
Potential Data Source(s): DCR system; CSI system; Consumer Satisfaction Surveys – YSS,
YSS-F, Adult, and Older Adult; primary data collection (e.g., surveys, interviews, or focus
groups; proposed data collection)
Stakeholder Informed Challenges & Limitations: Comprehensively assessing continuity of care
may require extensive document review or additional data collection (e.g., surveys or
interviews).
Outcomes
7.6 Indicator: Penetration Rate40 (Revision of CMHPC Indicator #3341)
Rationale for Inclusion: The penetration of CSS services among targeted groups (e.g., age,
race/ethnicity, individuals in poverty, homeless) will provide an important indication of the
extent to which services are reaching those most in need.
Potential Measure(s): Ratio of CSS consumers served, compared to eligible for services among
targeted populations; Ratio of CSS clients served, as compared to Holzer Targets (i.e., estimates
of the prevalence of serious mental illness/serious emotional disturbance)
39Hermann, R.C., Mattke, S., Somekh, D., Silfverhielm, H., Goldner, E., Glover, G. Pirkis, J., Mainz, J., Chan, J.A.
(2006). Quality indicators for international benchmarking of mental health care. International Journal for Quality in
Health Care, September 2006, 31-38.
40 Statutory outcome: Increase number of individuals receiving public mental health services (MHSA Section 3(d),
WIC 5813.5(a), WIC 5830(a)(4))
41California Mental Health Planning Council’s Performance Indicators for Evaluating the Mental Health System,
January 2010.
24
Potential Data Source(s): DCR system, CSS Exhibit 6 Quarterly Reports, California Health
Interview Survey (CHIS; proposed external data source), Census data (proposed external data
source)
Stakeholder Informed Challenges & Limitations: Stakeholders have expressed concern about the
reliability of Exhibit 6 data. If such data proves unreliable, the penetration of CSS services
among populations in need cannot accurately be assessed.
7.7 Indicator: Consumer Wellbeing42 (Revision of CMHPC Indicator #3343)
Rationale for Inclusion: Perceptions of improvements in functioning, the appropriateness of care
they receive, participation in treatment, quality of life, and satisfaction with services among
consumer groups (e.g., age, gender, race/ethnicity, individuals in poverty), can provide
indications of the quality of service within county mental health systems and across the state.
Potential Measure(s): Client/family perception of improvement in functioning (current, over
time, among high need groups); Client/family perception of quality of life (current, over time,
among high need groups)
Potential Data Source(s): Consumer Satisfaction Surveys – YSS, YSS-F, Adult, and Older
Adult; Primary data collection (e.g., surveys, interviews, or focus groups; proposed data
collection)
Stakeholder Informed Challenges & Limitations: Different sampling methods for consumer
satisfaction survey respondents have been employed over time, which may have implications for
analysis and any conclusions drawn. Additional qualitative and quantitative data collection may
be needed to create an indicator of wellbeing which is sensitive to county context and the
backgrounds of consumers and their families.
7.8 Indicator: Consumer Satisfaction (Revision of CMHPC Indicator #2544)
Rationale for Inclusion: Consumer/family satisfaction with the care and service they receive will
provide an important reflection of the ability of county mental health systems to achieve stated
values and goals.
Potential Measure(s): Consumer/family satisfaction with the care or service
Potential Data Source(s): Consumer Satisfaction Surveys – YSS, YSS-F, Adult, and Older Adult
42 Statutory outcome: Improve health and mental health (WIC 5801(d)(2), WIC 5806(a), WIC 5840(a), WIC
5840(c))
43 California Mental Health Planning Council’s Performance Indicators for Evaluating the Mental Health System,
January 2010.
41 ibid.
25
8. Domain: System Structure
This domain includes indicators of the mental health system workforce, and the type and quality
of programs/services.
Processes
8.1 Indicator: Workforce Composition (Revision of CMHPC Indicator #4545)
Rationale for Inclusion: This indicator addresses the extent to which the mental health system
workforce is appropriately configure to serve the diverse populations of county mental health
systems.
Potential Measure(s): Staff to consumer ratio, demographic composition of mental health
workforce, consumer/family member employment (i.e., number, FTE, % of workforce)
Potential Data Source(s): Cultural competence plans; WET Plans; Primary data collection (e.g.,
surveys, interviews, or focus groups; proposed data collection)
Stakeholder Informed Challenges & Limitations: Stakeholders expressed concern about the
reliability of workforce data in Cultural Competence Plans and WET Plans over time at the
program level. Without a valid source of data regarding workforce makeup, the appropriateness
of the workforce for serving the current consumer population cannot be assessed.
8.2 Indicator: Evidence-Based/Best Practice Programs and Services46 (Proposed Indicator)
Rationale for Inclusion: Care or services that are implemented based on the best available
evidence will lead to improved client outcomes. This indicator will center on whether
county/regional/statewide mental health services and supports adhere to best practice criteria
established through scientific evidence and/or expert consensus.
Potential Measure(s): Use of evidence-based practices; Fidelity of best practices to established
models; Receipt of best practices services/supports among consumers/families
Potential Data Source(s): CSI system, Primary data collection (e.g., surveys, interviews, or focus
groups; proposed data collection)
Stakeholder Informed Challenges & Limitations: Stakeholders expressed concern that the fidelity
with which evidence-based services are implemented is not captured by the relevant CSI item.
Thus, it is recommended that additional data collection (e.g., surveys, interviews, or focus
groups) be conducted to assess the fidelity of reported evidence-based program implementation.
45California Mental Health Planning Council’s Performance Indicators for Evaluating the Mental Health System,
January 2010.
46Anthony, W., Rogers, E., Farkas, M. (2003). Research on evidence-based practices: future directions in an era of
recovery. Community Mental Health Journal, 39, 101-114
26
Outcomes
8.3 Indicator: Cultural Appropriateness of Services47 (CMHPC Indicator #2348)
Rationale for Inclusion: This indicator addresses the extent to which the care or service is
configured to best address the diverse cultures served by county mental health systems.
Potential Measure(s): Client and family perceptions of cultural appropriateness
Potential Data Source(s): Consumer Satisfaction Surveys – YSS, YSS-F, Adult, and Older Adult
8.4 Indicator: Recovery, Wellness, and Resilience Orientation49,50 (Proposed Indicator)
Rationale for Inclusion: Recovery, wellness, and resilience orientation is a core value of the
MHSA initiative. The recovery process generally includes learning ways to manage mental
illness, but also involves learning attitudes and skills about living, learning, working, having
meaningful relationships, a place in the community and connection to the world. This indicator is
focused on the extent to which county mental health systems are structured to provide guidance
and support to consumers and families in their transition from living with mental illness as the
most important part of their lives to being only a part of who they are.
Potential Measure(s): Consumer/family member/staff perceptions of recovery orientation of
system and services
Potential Data Source(s): Recovery Oriented Systems Indicators Measure (ROSI; proposed data
collection), Developing Recovery Enhancing Environments Measure (DREEM; proposed data
collection)
Stakeholder Informed Challenges & Limitations: Any measure of recovery orientation must be
sensitive to, and informed by, the unique needs and circumstances of diverse cultural, ethnic,
regional, and age groups across the state.
Potential Implications of Indicators for MHSOAC, Counties, Consumers
and Their Families, and Other Stakeholders
Based upon the revised tentative indicator set presented in this report we offer some conversation
topics that might be fueled by the indicator set or ways that indicators might be brought into
ongoing conversations about MHSA performance, accountability and improvement. In other
words, we propose a handful of ways the priority indicators can work for the greater MHSA
community, whether in programming, service, or planning efforts. The final set will depend on
47Thomas, D. (2002). Evaluating the cultural appropriateness of service delivery in multi-ethnic communities.
Evaluation Journal of Australasia, 2, 50-56.
48California Mental Health Planning Council’s Performance Indicators for Evaluating the Mental Health System,
January 2010.
49 Statutory outcome: Implement Recovery Vision (WIC 5813.5(d))
50Anthony, W. (2000). A recovery-oriented service system: setting some system level standards. Psychiatry
Rehabilitation, 24, 159-169.
27
data availability, quality and reliability. Data integrity will be examined through a
comprehensive data review process which will commence upon receipt of relevant data. To the
extent existing data (already collected across counties) is complete and appropriate, we expect
the indicators detailed in this report will hold several implications for MHSOAC, counties,
consumers and their families, and other stakeholders working to improve MHSA services. The
questions outlined below, many informed by stakeholders, are by no means exhaustive but rather
are examples of those that might be answered using priority indicators.
Evaluating Mental Health System Processes and Outcomes
Given the multiple ways that MHSA is designed to shape the mental health system (through
improving consumer access, service performance, and system structure), system-level indicators
reflect diverse points of MHSA impact. The indicators can offer broad insights into system
processes (e.g., appropriateness of care) and relevant outcomes (e.g., consumers’ aggregated
experiences). As illustration, indicators regarding mental health system processes or outcomes
might address the following points.
Discussion Points Relevant Indicators
Processes
To what extent have demographic (gender, language, income, age)
6.1, 6.2, 8.1
disparities been addressed among MHSA service consumers?
Have rates of hospital visits, involuntary care, and 24-hour care changed
7.2, 7.3, 7.4, 7.5
since the MHSA was established, and from whom?
What best practices for consumer care are most prevalent within and
8.2
across counties?
Outcomes
What is the quality of care and services according to consumers and
7.8
their families?
How can various cultural groups be most appropriately served? 8.3
Processes and outcomes
What services and supports do consumers use after a mental health
episode or psychiatric hospitalization, and how do they perceive the 6.4, 6.5, 7.5, 7.8, 8.3
service experience?
Questions of mental health system performance can provide necessary context for greater
understanding of how system processes are related to consumer outcomes, which will be
essential for improvement of monitoring, transparency and effectiveness of MHSA services.
Improved monitoring, transparency and effectiveness, transparency
Among consumer outcomes, the indicator set might more fully explain how consumers navigate
available mental health interventions, the provisions made for consumers, and service quality.
28
Such increased understanding will provide the opportunity to address questions of monitoring,
transparency and effectiveness at the consumer level, such as those outlined below.
Discussion Points Relevant Indicators
To what extent do consumers have access to independent living or
2.1
supported living opportunities?
For children and TAY, what roles do schools play in mental health
1.1
support?
Broader questions that can inform what we know about system monitoring, effectiveness, and
transparency at the system level are as follows:
Discussion Points Relevant Indicators
To what extent are mental health services being accessed within and
6.1, 6.2, 6.3, 6.4
across target consumer populations or county mental health systems?
To what extent are county mental health systems aligned with the values
and expectations of consumers, families, communities, providers and the 6.5, 7.4, 7.8, 8.3, 8.4
MHSA initiative overall?
What is the makeup of the mental health system workforce, and the type
8.1, 8.2
and quality of programs/services they provide?
Reduce burden/cost
We propose that the indicator set will also add positive perspectives to the ongoing discussion of
mental health’s social costs. Some questions are beyond the indicator set, suggesting that
indicators evoke peripheral discussions that are important to maintaining MHSA performance
and quality. Particularly, the indicators might inform the following questions:
What is the cost saved/absorbed by hospitals that consumers use for emergency mental
health intervention?
What are the social ramifications of reducing homelessness through mental health
intervention?
At the system-level, it is equally important to consider such questions as:
Are services reaching those most in need?
Are recovery, wellness, and resilience being promoted?
29
Decision making and feedback loop for continuous improvement
Priority indicator findings will drive important decisions made about the MHSA, its consumers,
and its systems. At the consumer level, priority indicators can help make the case for more,
fewer, or different types of programs in particular domains, for example. At the system-level, the
MHSA administration might use findings formatively, meaning to shape existing practices in the
mental health system based on what information priority indicators provide. This could be
achieved by redistributing funds to areas that require more support, facilitating the revision of
programs and supports that fail to meet expected performance levels, or modifying models to
capitalize on a set of best practices that have been shown to consistently produce desired results.
At both levels, any information gained from priority indicators is an ongoing assessment of the
state of mental health and related services to maintain the highest quality of life possible for
consumers and sound system performance.
This report and its companion “Compiling Data to Produce All Priority Indicators” (deliverable
2D) are concrete examples of this continuous feedback loop for quality improvement. Outlined
in the next section, feedback from experts and stakeholders has ensured that the resulting
proposed indicator set is comprehensive and appropriate for monitoring MHSA consumer
outcomes and mental health system processes.
Types of Stakeholder Feedback and Corresponding Revisions
Through the feedback process detailed earlier in this report, stakeholders across the state
provided unique and well informed perspectives, thoughtful reaction and insight regarding the
initial indicator set and the methods and data sources proposed to compose priority indicators.
Stakeholders’ specific concerns regarding limitations or challenges of individual indicators can
be found in the Mental Health Outcomes Indicator Detail section above. For a table of
categorized stakeholder feedback, see Appendix E. Feedback regarding the indicator template
presented to stakeholders in the initial draft of this report fell largely into the two domains below:
Comprehensive and appropriate indicators. While many stakeholders expressed a
perspective of the proposed set of indicators as largely comprehensive and appropriate for
monitoring consumer outcomes and mental health system processes, they also provided
feedback regarding a limited number of alternative indicators. For example, many
stakeholders expressed the importance of considering Consumer Wellbeing as a consumer
level priority indicator, although this indicator was proposed under the performance domain
of the system level indicators.
o Corresponding revisions. Based upon feedback regarding alternatives, some
indicators proposed in this report reflect revisions to ensure they are most appropriate
for monitoring their respective element of the MHSA system or consumer experience.
One such revision was reconceptualizing Wellbeing as a consumer level indicator,
rather than a system outcome. Many such revisions to incorporate stakeholder
feedback regarding alternative indicators have been integrated throughout this report.
Data source availability and quality. Possibly the most common feedback theme was
centered on the availability and accurate and reliable data to inform the proposed indicators.
30
While many stakeholder concerns in this regard will be verified by a thorough review of
existing data, the unique insights and historical knowledge of data integrity within specific
counties and throughout the state was central to our revisions of the proposed sources of data
to be used to compose priority indictors. As an example, stakeholders expressed concern that
many indicators could require additional primary data collection to comprehensively capture
factors such as Workforce Composition, as they noted that existing sources were incomplete
or unreliable with regard to some indicators.
o Corresponding revisions. Based upon feedback regarding data concerns several
indicators were revised to include alternative existing data sources, potential external
data sources (e.g., census data) or proposals for additional primary data collection.
Additionally, a data quality review process outlined in a companion report (i.e.,
“Compiling Data to Produce All Priority Indicators”; deliverable 2D) will provide
more detailed knowledge of where the holes are among existing data sources and
present logical and practical possibilities for filling them.
Next Steps
The priority indicators proposed in this report have been elaborated down to the item level, along
with more comprehensive proposals for composing or calculating each indicator, in a subsequent
report entitled “Compiling Data to Produce All Priority Indicators” (deliverable 2D). Stakeholder
feedback was central to revising this companion document to ensure the calculation of each
indicator is appropriate and accurate for monitoring MHSA consumer outcomes and mental
health system processes.
This report and its companion will be the foundation for forthcoming report 2E, which includes
results for all priority indicators at the statewide level, for the most recent one-year period. This
report will then be followed by three county level reports on all priority indicators, submitted on
a quarterly basis.
31
Appendix A
32
Appendix B
Glossary
Criteria
A set of standards on which decisions are made
Domain
An overarching category within which related items are grouped
Indicator
A gauge or measure of a particular trend or condition
Outcome
Change brought about by a guiding course of action
Process
The breadth of actions taken to achieve an outcome or set of outcomes
33
Appendix C
Data Sources Reviewed
Data Collection & Reporting System for FSP (DCR)
Key Event Tracking (KET)
Partnership Assessment Form (PAF)
Quarterly Assessment Forms (3M)
Performance Outcomes & Quality Improvement (POQI)
Youth Services Survey (YSS)
Youth Services Survey for Families (YSS-F)
Adult Survey
Older Adult Survey
Client Services and Information System (CSI)
County Reports
Revenue and Expenditure Reports (R&E)
Annual Updates
Quarterly Progress Goals and Report (includes CSS Exhibit 6)
Annual Report on Involuntary Detentions
Cultural Competence Plans
34
Appendix D
Participating Organizations and Agencies
(In Alphabetical Order)
Individuals and groups from the following entities responded to an e-mail announcing the
availability of MHSA Evaluation Team contract deliverables 2A and 2C (early versions of 2B
and 2D). It is possible that more persons than are listed received the call through message
forwarding. The following list was created to the best of the team’s knowledge and e-mail
verification.
APS Healthcare
Association of Community Human Services Agencies
Bonita House
California Association of Social Rehabilitation Agencies
California Community Colleges Chancellor's Office
California Department of Aging
California Department of Mental Health
California Institute for Mental Health
California Mental Health Director’s Association
California Mental Health Planning Council
California Network of Mental Health Clients
Contra Costa County Health Services Department
California Council of Community Mental Health Agencies
EMQ Families First
Humboldt County
California Mental Health Directors Association Indicators, Data, Evaluation
Accountability (IDEA) Committee
Los Angeles County Department of Mental Health
Mental Health America of California
Mental Health America of Los Angeles
MHSA Partners
Mental Health Services Oversight and Accountability Commission
Monterey County
National Alliance on Mental Illness – California
Nevada County
Orange County Behavioral Health Services
San Bernardino County Department of Behavioral Health
San Diego County
San Francisco Department of Public Health
San Joaquin County Mental Health Board
Seeds of Hope
Shasta County Health and Human Services
Turning Point Community Programs
United Advocates for Children and Families
35
Appendix E
Table 1: Types of Stakeholder Feedback Received (Summary)
Domain Domains Domains
Alternative Alternative Challenges with
incomplete incomplete Request for
domain/measures data sources associated data
(changes (additions clarification
suggested suggested noted
suggested) suggested)
• • • •
1. Employment/ Education
2. Homelessness/ Housing • • •
Situation
• • •
3. Justice Involvement
• •
4. Emergency Care
• •
5. Access
•
6. Performance
• • • • • •
7. Structure
Additional domains suggested: Additional data sources:
Alcohol and Other Drug Use (AOD) Claiming process
Co-occurring physical health disorders Recovery Oriented Systems Indicators Measure (ROSI) / Developing
Social connections Recovery Enhancing Environments Measure (DREEM)
36
Table 2: Types of Stakeholder Feedback Received (Detailed)
Indicator Indicator Indicator
Alternative Alternative Challenges with
incomplete incomplete Request for
domain/measures data sources associated data
(changes (additions clarification
suggested suggested noted
suggested) suggested)
•
1. Employment/ Education
• •
1.1 Education
• • •
1.2 Employment
2. Homelessness/ Housing • • •
Situation
• •
3. Justice Involvement
• • •
4. Emergency Care
5. Access
5.1 Demographic •
Profile
5.2 New Consumers by • •
Demographic Profile
37
Table 2: Types of Stakeholder Feedback Received (Detailed)
Indicator Indicator Indicator
Alternative Alternative Challenges with
incomplete incomplete Request for
domain/measures data sources associated data
(changes (additions clarification
suggested suggested noted
suggested) suggested)
5.3 High Needs • • •
Consumers Served
5.4 Access to Primary • • •
Care Physicians
5.5 Consumer/ Family
Perceptions of Access to
Services
6. Performance
6.1 Consumers Served •
Annually Through CSS
6.2 Involuntary Care
6.3 24-Hour Care
6.4 Appropriateness of • •
Care
• •
6.5 Continuity of Care
38
Table 2: Types of Stakeholder Feedback Received (Detailed)
Indicator Indicator Indicator
Alternative Alternative Challenges with
incomplete incomplete Request for
domain/measures data sources associated data
(changes (additions clarification
suggested suggested noted
suggested) suggested)
• •
6.6 Penetration Rate
6.7 Consumer •
Wellbeing
6.8 Satisfaction
7. Structure
7.1 Workforce • • •
Composition
7.2 Evidence Based/Best
• • •
Practice Programs and
Services
7.3 Cultural
• • •
Appropriateness of
Services
7.4 Recovery, Wellness,
• •
and Resilience
Orientation
39