BHSOAC
Deliverable 2c 1
Read the report at Behavioral Health Services Oversight & Accountability Commission ↗
DRAFT REPORT FOR STAKEHOLDER REVIEW. RESPONSE DEADLINE: AUGUST 31, 2011
Mental Health Services Act Evaluation:
Compiling Data to Produce All Priority Indicators
Contract Deliverable 2C
UCLA Center for Healthier Children, Youth and Families
EMT Associates, Inc.
June 30, 2011
The following report was funded by the
Mental Health Services Oversight and Accountability Commission.
Focus
The Mental Health Services Act (MHSA) evaluation team was charged with developing
templates and reports on statewide and county specific data that would improve understanding of
how the MHSA impacted consumers. More specifically, and per contract language, the team
will:
Design and complete statistical analyses and reports that measure impact of MHSA at
individual and system levels on indicators specified in the Matrix of the California Public
Mental Health System Prioritized Performance Indicators at the state and county levels.
Draft templates, documentation of analysis, and initial statewide reports will be
circulated to key stakeholders and made available to the public for input by posting on
the web and making a hard copy available upon request.
Individual client outcomes for full service partnerships (FSPs) by age group must be
addressed for each domain (education/employment, homelessness/housing and justice
involvement) as specified. Note: this impact analysis at the individual level is limited to
available data (i.e., a small segment of public mental health clients, full services
partners, is reflected in this data.) Mental Health system performance must address
family/client/youth perception of well-being, demographics of FSP population, FSP
access to primary care, penetration rate and changes in admissions for the entire public
community mental health population, involuntary care, and annual numbers served
through [Community Services and Supports programs] CSS.
The evaluation team submits the following report in fulfillment of this charge. We do so
acknowledging that this report is not final until key stakeholders have reviewed and provided
their insights about issues related to measuring the impact of MHSA.
Stakeholder Feedback
As noted in the contract language, input from key stakeholders and mental health service
advocates is key to developing a final report. To reflect input from a range of stakeholder groups
in the report’s development, the evaluation team will enlist feedback from existing groups (e.g.,
FSP Advisory Committee, Equality California, Racial and Ethnic Mental Health Disparities
Coalition, California Mental Health Directors Association, National Alliance on Mental Illness,
California Mental Health Planning Council, the California Network of Mental Health Clients,
United Advocates for Children and Families, and other providers and representatives of un-
served and underserved populations) over a one-month period. The evaluation team will avoid
imposing additional work on these groups and instead will allow groups to rely on their existing
internal processes for reviewing and responding to mental health-related reports. The evaluation
team will only provide a set of questions tailored to each group’s expertise to maximize the
amount and quality of feedback gained about target issues in this report. Thus, the following
report is not a final product. Instead, it is a starting-point from which stakeholders can begin a
conversation about measuring mental health impact since the MHSA’s initiation.
2
Feedback Process
This report constitutes the beginning steps in a process designed to solicit feedback for numerous
consumers and stakeholders. As such, it should be viewed as a draft. The final report, which is
due on 9/30/11 will incorporate the feedback we receive (see Figure 1: Steps Leading to
Statewide and County Specific Data Reports on the following page).
While we welcome feedback on all aspects of the report, along with this report we have provided
a brief ―guidance‖ document. The goal of this guidance is to provide everyone who chooses to
comment, suggestions regarding the aspects of the report where we would like feedback.
Given the timeframe for our contract we would like to receive feedback anytime between
7/29/2011 and 8/31/2011. After this period we will compile all the feedback, identify common
themes and concerns, and revise the reports accordingly. We expect some recommendations
from different individuals or organizations to be at odds with each other. We will negotiate these
differences by incorporating into the report as many recommendations or alternative views as
make sense given the context.
Format of feedback
With the exception of general comments and reactions, whether it be to our guidance questions
or your own suggestions, feedback should make reference to a specific page(s) in the document
so the evaluation team can appropriately address the suggestion or concern. Comments can be
emailed to the addresses below.
Starting July 29th, you can download the documents from the following websites if you need
them, along with the guidance questions.
MHSA Website
http://www.mhsoac.gov/Announcements/announcements.aspx
UCLA
http://healthychild.ucla.edu/mhsa_evaluation
Email
Ashaki Jackson: ashakijackson@mednet.ucla.edu
Rob Blagg: rblagg@emt.org
OR
MHSAevaluation@gmail.com
3
Steps Leading to Statewide and County Specific Data Reports
Initial Statewide Evaluation
12/31/12
9/30/12
6/30/12
County reports incorporate
county context Three written County specific and statewide reports, on all priority indicators Step 7
(demographics, funding, etc)
Revised written report from STEP 5 in response to stakeholder input
Step 6
3/31/12
Initial draft written report submitted including data for all priority indicators
at the statewide level for the most recent one year period available
Step 5
12/31/11
Revised deliverable from STEP 2 in response to stakeholder input obtained.
Step 4
s
p
e 9/30/11
St
Revised deliverable from STEP 1 in response to stakeholder input
Step 3
9/30/11
Draft written documentation of the process for compiling the
data to produce the reports for all priority indicators
Step 2 Stakeholder/Consumer
Feedback
6/30/11
Draft proposed standardized template for reporting
all priority indicators
Step 1 Currently
Completed
6/30/11
Due Dates
4
Overview
This draft report proposes processes for creating useful measures of priority indicators that can
be used to monitor how the Mental Health Services Act (MHSA) impacts consumer outcomes
and mental health service system performance statewide and at the county level. The priority
indicators themselves were proposed in a preceding report (Templates for Reporting Priority
Indicators, Deliverable 2A). These proposed indicators were intended to identify measurable
Community Services and Supports (CSS) program outcomes, including consumer outcomes and
measurable characteristics of mental health service system capacity and performance. This report
details how each priority indicator can be represented using survey and service description (e.g.,
outputs) data already collected within each county. The report identifies data sources, identifies
specific data items (variables) within those sources, and proposes methods of combining data
into more adequate indicators where appropriate. No analyses are included in this report; rather
data is organized in preparation for analyses that will take place subsequent to this report.
Criteria for testing individual or multiple-item measures of indicators are identified. These
quality tests will be applied to cull and refine proposed measures once access to the necessary
data is acquired.
The report is organized by the following topics.
Brief discussion of indicator development prior to this project
Profile of the data sources used for this project – mental health-related surveys and
reports that are regularly submitted by California counties
Discussion of the criteria used to select, review, and refine measures
Explication of the calculations proposed to create priority indicator measures
Notes on examining data for quality and completeness
Description of consumer stakeholder group roles in refining measures and calculations
Two tables are presented that summarize MHSA domains, priority indicators (i.e., consumer and
system level), measures, relevant data sources and items, and necessary calculations. Where data
quality concerns can be anticipated, or gaps are evident, recommendations for new data
collection to attain measurement goals are included.
The report concludes with next steps in refining the measure and indicators to ensure accurate
and comprehensive monitoring of consumer outcomes and mental health service system
performance.
Background
The California Mental Health Planning Council (CMHPC) proposed a set of priority indicators
to assess the impact of the MHSA on consumers and county service systems throughout the state.
Council members designed individual-level priority indicators to create greater clarity about
consumers’ dispositions (e.g., employment, education, housing, justice involvement) following
interventions coordinated through the MHSA. Similarly, council members proposed that system-
5
level priority indicators (related to consumer access, agency performance, agency structure)
would explain how operations changed or were enhanced (if at all) by the Act. These indicators
were ultimately adopted by the Mental Health Services Oversight and Accountability
Commission.
As part of ongoing efforts to define priority indicators and identify how these indicators might be
measured using data currently collected by counties, the evaluation team reviewed existing
datasets to identify appropriate data sources and items. This process began with the set of
indicators identified in Templates for Reporting Priority Indicators, Deliverable 2A. This
document defines and provides the rationale for indicators recommended for monitoring county
use of MHSA funds, and performance of MHSA initiatives. In this report, the evaluation team
outlines a data extraction and measurement process to operationalize conceptually complete
indicators of consumer outcomes and system performance related to the MHSA.
Objectives
The evaluation team conducted a search of available data with two goals: to 1) locate variables,
relevant to each priority indicator, that are regularly collected; and 2) outline short protocols for
converting existing data into priority indicators. The present report briefly documents this
process, in which we provide guidelines about how to create relevant measures using current
data to the extent possible. Throughout the report we note challenges in calculation, and areas in
which new data might need to be collected.
Process for Reviewing Available Data
We reviewed several data dictionaries and instruments (e.g., surveys, forms) associated with
their respective existing datasets or reports. The datasets and reports, listed below, reflect
information that is regularly collected across counties at present. In the absence of access to raw
data, we closely examined the qualities (e.g., item wording, response options, intended response
population) of each item (variable), sorting which would be most appropriate to represent each
priority indicator.
Client & Service Information (CSI)
The CSI system is a repository of county, client (e.g., age, gender, preferred language, education,
employment status, living arrangement, etc.), and service information (number and length of
service contact). The data is collected from all consumers who receive mental health services,
including consumers involved in the Full Service Partnership.
Data Collection and Reporting (DCR) System
The DCR system houses data for consumers who are served through Full Service Partnership
programs. Data from assessments – the Partnership Assessment Form (PAF), Key Event
Tracking (KET), and Quarterly Assessment (3M) – are collected for consumers in specific age
6
categories. The PAF reflects consumer history and baseline information, including consumer
education and/or employment, housing situation, legal issues, health status, and substance use.
The KET reflects any important changes in the consumer’s life such as housing, education and/or
employment, and legal issues during service receipt. The 3M collects follow-up information on
key areas such as education, health status, substance use, and legal issues each quarter.
Consumer Satisfaction Surveys (YSS for youth responses, YSS-F for family responses)
These consumer surveys are instrument sets customized for consumer groups (e.g., youth, adults,
and older adults). Instruments are composed of widely validated tools such as the Child Behavior
Checklist, Youth Self Report, and Restrictiveness of Living Environment Scale for youth
assessment; the Global Assessment of Functioning, Behavior and Symptom Identification Scale,
and the California Quality of Life for adults; and the Brief Symptom Inventory, Senior Outcomes
Checklist 10, and Index of Independent Activities of Daily Living for older adults. The data,
designed to inform treatment planning and service management, is collected from individuals
with ―serious, persistent‖ mental illness, have received services for 60 days or more, and are not
categorized as ―medication only.‖ At minimum, data is customarily collected at intake, annually,
and at discharge, however this schedule has changed in recent years. Findings are reported to the
Department of Mental Health semi-annually.
Mental Health Statistics Improvement Program (MHSIP for adult responses and MHSIP for
older adult responses)
The MHSIP consumer surveys are designed to assess client satisfaction, service accessibility,
quality, and outcomes. Adult consumers and older adult consumers respond to the survey
questions using a 5-point scale of agreement (e.g., 5 = strongly agree and 1 = strongly disagree).1
Quarterly Reports (Exhibit 6)2
Quarterly reports, including Exhibit 6, reflect consumer counts—the number of people who
targeted and receive MHSA services. Counts are aggregated from different consumer pools,
including Outreach and Engagement, and Community Services and Supports (CSS), among
others. Service types and demographics are not included in reports. Data is reported quarterly
then compiled into annual reports.
Calculation of Measures
Meaningful and useful measures must be carefully conceptualized, designed, and constructed.
Doing so facilitates how variables are combined, or calculated, then interpreted. The indicators
included in the measurement template proposed in Templates for Reporting Priority Indicators,
1 Key informants have informed the evaluation team that the MHSIP surveys are sometimes also referred to as
―POQI-Adult‖ and ―POQI-Older Adult.‖
2 In the previous deliverable (2A), we noted that key informants preferred the use of Annual Updates instead of
Quarterly Reports although the state of Annual Updates was uncertain due to Assembly Bill 100. At the time of this
report, the data dictionary associated with Annual Updates was unavailable, limiting what we are able to report here.
The recommendation is described more fully in the discussion.
7
Deliverable 2A were expressed as absolute figures (e.g., counts, frequencies) or ratios (e.g.,
normalized data). These two types of indicators have distinct requirements for the data used to
operationalize them, and implications for stakeholders who would use indicators to drive quality
improvement. The following table outlines the implications for these two types of indicators.
Absolute Indicators
Counties often report data in terms of absolute figures, which might be expressed as the
frequency (count) of a specific event or an indication of consumer status (e.g., attendance in
school, employment status, housing status, receipt of service, type of service, etc.). Absolute
figures can provide a) description of a services or outcomes at points in time, and b) provide the
basis for critical analyses of differences across consumer groups, across time, and attribution of
impact.
Absolute figures are essential to any assessment of the parameters (e.g., carrying capacity, limits,
or sustainability) of mental health systems and services, and the impact on consumers and
families. For example, at the mental health system level, the total number of consumers receiving
24-hour care provides the possibility to consider service levels relative to a county’s overall
resource capacity. Similarly, absolute measures of consumer outcomes, such as the number of
consumers attending school, will allow for comparison to other consumers and mental health
systems. Absolute figures can provide an important perspective of the capacity, performance,
and impact of mental health systems. When information regarding local MHSA context is
lacking, absolute indicators can also be useful for stakeholders trying to understand the relative
magnitude of county services and impacts, or reasons for prioritizing efforts. For example,
identifying the 10 counties with fewest consumer arrests would require absolute figures, whereas
ratios (e.g., normalized data) are more useful when making comparisons between counties.
Absolute measures of MHSA performance and impact provide for:
consistent tracking;
data aggregation to key levels (e.g., county, state); and
ability to form additional ratios other than those included in the priority indicator
template.
Ratio Indicators
Ratios relate two absolute figures to each other and provide context to both. For example, the
efficiency of Community Services and Supports (CSS) can be expressed in terms of the number
of consumers served through CSS relative to those who were targeted for service. Alternatively,
to shift focus to the impact of the CSS program, the number of consumers served through CSS
could be compared to all consumers receiving mental health services.
Ratio indicators serve to:
8
relate two absolute figures to each other;
make relationships visible and interpretable by a broad audience; and,
provide for comparison of different scales of operation relative to a specific service (e.g.,
number of incarcerated consumers per individual served)
Ratios may also be particularly useful for comparing counties or regions. Absolute figures
sometimes do not provide the context in which performance or impact may be best understood.
This may be particularly true among the diverse counties and regions of California. For example,
the magnitude of a service will not always correlate with the size of the county in which it was
administered. As illustration, it may be factually correct that county A served twice as many
consumers as county B, but this would be misleading if county B were a quarter that size and
twice as efficient in the administration of their services. For some indicators an absolute figure
may be the most meaningful piece of information, but for others additional context is needed to
accurately understand the implications of figure.
Single vs. Multiple Item Measures
To produce adequately robust measures, we applied a process to construct single or multiple-
item measures as appropriate. In some cases a single data item is adequate to capture an
indicator, such as when its meaning is clear and it has adequate variance and precision. For
example, the number of days child or TAY consumers attended school in the past year may only
require a single item (e.g., ATTENDANCEPAST12 – PAF). But often, single items are not
adequate, or can be improved by combining several data items that express different empirical
facets of an indicator. For example, to construct a robust and accurate measure of consumer or
family perceptions of improvement in functioning, multiple survey (e.g., MHSIP) items are
necessary to ensure several facets of this measure are assessed. When carefully constructed these
multi-item indicators improve the reliability, validity and variance characteristics of the resulting
measure.
Orientation to the Tables
The subsequent tables detail how proposed Priority Consumer and System Performance
Indicators may be constructed. The tables are divided at the individual (Table 1) and system
(Table 2) levels, and are intended to present options for constructing measures to represent
priority performance indicators, from existing data or proposed additional data collection.
Although the Planning Council envisioned consumer outcomes to be measured across Full
Service Partnership consumers and system outcomes to be measured across all mental health
service consumers (see Appendix 1), data sources in both tables reflect possibilities for outcome
calculations across all mental health services consumers (via the Consumer Services and
Information [CSI] system) as well as persons enrolled in Full Service Partnerships (via the Data
Collection and Reporting [DCR] system).
9
The columns from left to right detail measurement domains, performance indicators, possible
calculation of measures based on what is available (recommended and alternative), the databases
or reports from which relevant items can be drawn, and the specific items within each dataset or
report which may be used to construct measures. Not all data sources provide accurate indicator
measurement, however we list these sources and their associated variables to generate discussion
about how existing data might be re-envisioned or manipulated to represent a priority indicator.
To be clear, the Data Source and Dimension columns list all of the possible variables that might
be used to calculate or estimate a measure. These columns should be considered an inventory of
possible measures. In the absence of appropriate data sources, we recommend new data
collection.
To make the most efficient use of existing and proposed additional data and data collection
processes, and provide flexible performance measurement options at the state and county levels,
we present ―recommended‖ and ―alternative‖ measures of priority indicators. Recommended
measures are those that would most accurately reflect indicators, while attempting to take
advantage of existing data systems. We consider these calculations optimal. Alternative
calculations, considered as substitutes or supplements to the recommended calculation, are based
on variables that currently exist within at least one of the Department of Mental Health datasets,
and with manipulation can provide an approximate measure of the desired performance
indicator. In the event that existing data can only provide an approximate measure of an
indicator, additional data collection is proposed. The following tables detail how measures of
each priority indicator can be constructed from existing or proposed additional data collection.
10
Table 1. Process for Compiling Data and Calculating Priority Indicators: Individual-level (Consumer) Outcomes for Full Service
Partnerships3
Domain Indicator Calculation of Measure(s) Data Source(s) Dimension – Potential Items
1. Education/ 1.1 Average Recommended Ratio DCR Children
Employment attendance – score per Numerator: Number of days AttendanceRate Estimate – ATTENDANCECURR (PAF) (3M)
year at school during a 9-month ATTENDANCEPAST12 (PAF)
school year* / Denominator: TAY
(Children, TAY) Number of days during AttendanceRate Estimate – ATTENDANCECURR (PAF) (3M)
consumer’s school year ATTENDANCEPAST12 (PAF)
YSS Children (Youth report)
Alternative Estimate 1
Current and previous expulsions – LES12EXPSUS,
Number of days during
LES12PSTEXPSUS, MOR12EXPSUS, MOR12PSTEXPSUS
consumer’s school year* –
Number of expulsions or YSS-F Children (Parent or guardian report)
suspensions during the year Current and previous expulsions – LES12EXPSUS,
LES12PSTEXPSUS, MOR12EXPSUS, MOR12PSTEXPSUS
Alternative Estimate 2 Proposed new data Children and TAY
Attendance rate estimate for collection Number of days absent
three quarters Total number of school year days at consumers’ school
1.2 Proportion Recommended Ratio 1 DCR TAY
participating in paid Numerator: Number of Paid employment – Current_In-HouseAvgHrWeek (PAF),
and unpaid employed consumers Current_OtherEmploymentAvgHrWeek (PAF),
employment (reporting work hours) / Current_SupportedAvgHrWeek (PAF),
Denominator: Total number Current_TransitionalAvgHrWeek (PAF), Past12_Competitive (PAF),
(TAY over 18, adults, of consumers eligible for Past12_In-House (PAF), Past12_In-HouseAvgHrWeek (PAF),
and older adults) employment (over 18 years Past12_OtherEmployment (PAF),
old) Past12_OtherEmploymentAvgHrWeek (PAF), Past12_Supported
(PAF), Past12_SupportedAvgHrWeek (PAF), Past12_Transitional
Recommended Ratio 2 (PAF), Past12_TransitionslavgHrWeek (PAF)
Numerator: Number of Unpaid employment – Current_Non-paidAvgHrWeek (PAF),
employed consumers Past12_Non-paid (PAF), Past 12_Non-paidAvgHrWeek (PAF)
3 Data sources that reflect all mental health service consumers (e.g., CSI) have been added in the event that broader information than what is learned about Full Service Partnership
consumers is sought.
*Asterisk indicates information from new data collection or a data source not yet identified.
11
Domain Indicator Calculation of Measure(s) Data Source(s) Dimension – Potential Items
receiving pay for work / Adults
Denominator: Total number Paid employment – Current_In-HouseAvgHrWeek (PAF),
of consumers eligible for Current_OtherEmploymentAvgHrWeek (PAF),
employment (over 18 years Current_SupportedAvgHrWeek (PAF),
old) Current_TransitionalAvgHrWeek (PAF), Past12_Competitive (PAF),
Past12_In-House (PAF), Past12_In-HouseAvgHrWeek (PAF),
Recommended Ratio 3 Past12_OtherEmployment (PAF),
Numerator: Number of Past12_OtherEmploymentAvgHrWeek (PAF), Past12_Supported
employed consumers not (PAF), Past12_SupportedAvgHrWeek (PAF), Past12_Transitional
receiving pay for work / (PAF), Past12_TransitionslavgHrWeek (PAF)
Denominator: Total number Unpaid employment – Current_Non-paidAvgHrWeek (PAF),
of consumers eligible for Past12_Non-paid (PAF), Past 12_Non-paidAvgHrWeek (PAF)
employment (over 18 years Older Adults
old)
Paid employment – Current_In-HouseAvgHrWeek (PAF),
Current_OtherEmploymentAvgHrWeek (PAF),
Current_SupportedAvgHrWeek (PAF),
Current_TransitionalAvgHrWeek (PAF), Past12_Competitive (PAF),
Past12_In-House (PAF), Past12_In-HouseAvgHrWeek (PAF),
Past12_OtherEmployment (PAF),
Past12_OtherEmploymentAvgHrWeek (PAF), Past12_Supported
(PAF), Past12_SupportedAvgHrWeek (PAF), Past12_Transitional
(PAF), Past12_TransitionslavgHrWeek (PAF)
Unpaid employment – Current_Non-paidAvgHrWeek (PAF),
Past12_Non-paid (PAF), Past 12_Non-paidAvgHrWeek (PAF)
2. Homelessness/ 2.1 Housing situation/ Recommended Ratios 1 DCR Children
Housing Index- score Numerator: Number of days Current housing situation – CURRENT (PAF) (KET)
that children or TAY (under Previous housing situations (week, month) –
(Children, TAY, 18) live in a family home EMERGENCYSHELTER-PASTTWELVEDAYS,
adults, and older annually/ Denominator: 365 EMERGENCYSHELTER_PASTTWELVEOCCURENCES,
adults) days EMERGENCYSHELTER_PRIORTWELVE (PAF),
HOMELESS_PASTTWELVEDAYS (PAF),
Recommended Ratios 2 HOMELESS_PASTTWELVEOCCURENCES (PAF),
Numerator: Number of days YESTERDAY (PAF)
that children or TAY (under DCR TAY
18) live in a foster home
Current housing situation – CURRENT (PAF) (KET)
annually/ Denominator: 365
Previous housing situations (week, month) –
days
EMERGENCYSHELTER-PASTTWELVEDAYS,
EMERGENCYSHELTER_PASTTWELVEOCCURENCES,
Recommended Ratios 3
EMERGENCYSHELTER_PRIORTWELVE (PAF),
Numerator: Number of days
HOMELESS_PASTTWELVEDAYS (PAF),
TAY, adults, or older adults
HOMELESS_PASTTWELVEOCCURENCES (PAF),
are homeless/ Denominator:
YESTERDAY (PAF)
12
Domain Indicator Calculation of Measure(s) Data Source(s) Dimension – Potential Items
365 days Adults
Current housing situation – CURRENT (PAF) (KET)
Recommended Ratios 4 Previous housing situations (week, month) –
Numerator: Number of TAY EMERGENCYSHELTER-PASTTWELVEDAYS,
or adults with independent EMERGENCYSHELTER_PASTTWELVEOCCURENCES,
residential statuses/ EMERGENCYSHELTER_PRIORTWELVE (PAF),
Denominator: Total number HOMELESS_PASTTWELVEDAYS (PAF),
of FSP TAY and FSP adults HOMELESS_PASTTWELVEOCCURENCES (PAF),
YESTERDAY (PAF)
Recommended Ratios 5 Older Adults
Number of TAY, adults, and
Current housing situation – CURRENT (PAF) (KET)
older adults who are not
Previous housing situations (week, month) –
homeless/ Denominator:
EMERGENCYSHELTER-PASTTWELVEDAYS,
Total number of FSP TAY.
EMERGENCYSHELTER_PASTTWELVEOCCURENCES,
FSP adults, or FSP older
EMERGENCYSHELTER_PRIORTWELVE (PAF),
adults
HOMELESS_PASTTWELVEDAYS (PAF),
HOMELESS_PASTTWELVEOCCURENCES (PAF),
YESTERDAY (PAF)
2. Homelessness/ 2.1 Housing situation/ Recommended Ratios 1 YSS Children
Housing (cont’d) Index- score Numerator: Number of days Specific housing – PARENT, FAMILYMEM, FOSTERHM,
that children or TAY (under THERAPEUTIC, SHELTER, HOMESHELT, GROUPHM,
(Children, TAY, 18) live in a family home RESIDENTX, HOSPITAL, JAIL, CORRECTIONS, HOMELESS,
adults, and older annually/ Denominator: 365 LIVEOTHER, *WHERE (follow-up to LIVEOTHER)
adults) days YSS TAY
Specific housing – PARENT, FAMILYMEM, FOSTERHM,
Recommended Ratios 2 THERAPEUTIC, SHELTER, HOMESHELT, GROUPHM,
Numerator: Number of days RESIDENTX, HOSPITAL, JAIL, CORRECTIONS, HOMELESS,
that children or TAY (under LIVEOTHER, *WHERE (follow-up to LIVEOTHER)
18) live in a foster home CSI4 TAY
annually/ Denominator: 365
PATIENT STATUS CODE (consumer’s housing if recently
days
discharged)
Recommended Ratios 3
Numerator: Number of days Adult
TAY, adults, or older adults PATIENT STATUS CODE (consumer’s housing if recently
are homeless/ Denominator: discharged)
365 days
Older Adult
Recommended Ratios 4 PATIENT STATUS CODE (consumer’s housing if recently
Numerators: Number of discharged)
4 Key informants have expressed concerns that CSI data designed to capture all mental health service consumers, is of questionable quality.
13
Domain Indicator Calculation of Measure(s) Data Source(s) Dimension – Potential Items
TAY or adults with
independent residential
statuses/ Denominator: Total
number of FSP TAY or FSP
adults
Recommended Ratios 5
Number of TAY, adults, or
older adults who are not
homeless/ Denominator:
Total number of FSP TAY,
FSP adults, or FSP older
adults
3. Justice 3.1 Justice Recommended Ratio 1 DCR Children
Involvement Involvement Numerator: Number of Recent arrest – DATEARRESTED (PAF)
consumer arrests annually / TAY
(Children, TAY, Denominator: 365 days Previous arrests (year) – ARRESTPRIOR12 (PAF
adults, and older
Older Adults
adults) Recommended Ratio 2 – by
Recent arrest – DATEARRESTED (PAF)
age group
YSS TAY
Numerator: Number of child,
Previous arrests (year) – MOR12AREST, MOR12PSTAREST
TAY, adult, or older adult
YSS-F Children (Parent or guardian response)
arrests/ Denominator: Total
Previous arrests (year) – LES12AREST, LES12PSTAREST
number of FSP children, FSP
TAY, FSP adults, or FSP CSI TAY
older adults P-08.0 CONSERVATORSHIP/ COURT STATUS (if consumer is a
ward of the court)
Recommended Ratio 3 – by Older Adults
age group S-20.0 LEGAL CLASS ADMISSION (if consumer has been
Numerator: Number of child, admitted to acute 24-hour mental health services)
TAY, adult, or older adult
MHSIP Adult Adults
arrests / Denominator:
Recent arrests – ARREST
County estimate of all
Previous arrests – LES12AREST, LES12PSTAREST
children, all TAY, all adults,
or all older adults MHSIP Older adult Older Adults
Recent arrests – ARREST
Previous arrests – LES12AREST, LES12PSTAREST
4. Emergency 4.1 Emergency Recommended Ratio 1 CSI Children
Care psychiatric Numerator: Number of S-06.0 SERVICE FUNCTION (Identifies the specific type of
hospitalizations and mental health episode- service received by the client within 24 Hour, Day, and/or
interventions for related hospitalizations Outpatient mode of service)
14
Domain Indicator Calculation of Measure(s) Data Source(s) Dimension – Potential Items
mental health annually/ Denominator: TAY
episodes Number of consumers visits S-06.0 SERVICE FUNCTION (Identifies the specific type of service
to the hospital for any reason received by the client within 24 Hour, Day, and/or Outpatient mode
(Children, TAY, annually of service)
adults, and older Adults
adults) Recommended Ratio 2 S-06.0 SERVICE FUNCTION (Identifies the specific type of service
Numerator: Number of received by the client within 24 Hour, Day, and/or Outpatient mode
emergency psychiatric of service)
interventions (numerator)/
Older Adults
Denominator: Number of
S-06.0 SERVICE FUNCTION (Identifies the specific type of service
consumer visits to a non-
received by the client within 24 Hour, Day, and/or Outpatient mode
hospital intervention center
of service)
annually
Proposed new data Children, TAY, Adults, Older Adults
collection Number of non-psychiatric hospital visits
Number of visits to a non-hospital facility for mental health
interventions
Table 2. Process for Compiling Data and Calculating Priority Indicators: System-level Outcomes for All Mental Health Consumers5
Domain Indicator Calculation of Measure(s) Data Source(s) Dimension – Potential Items
5. Access 5.1 Demographic Recommended Descriptives CSI Age – C-03.0 Date of Birth
Profile of Mean, mode, range, percentiles of Gender – C-05.0 Gender
Consumers Served age, gender, race/ethnicity of Race/ethnicity – C-09.0 Ethnicity; C-10.0 Race
consumer population
DCR Age – Date of Birth
Gender – Gender
Race/ethnicity – Ethnicity_A; Ethnicity_B
Alternate Descriptives 1 CSI Employment Status – P-03.0 Employment Status
Mean, mode, range, percentiles of
age, gender, race/ethnicity of
DCR Income – Wages_Curr; Wages_Past12
individuals living below the
poverty line or unemployed
Alternate Descriptives 2 CSI Homelessness – P-09.0 Living Arrangement
Mean, mode, range, percentiles of
age, gender, race/ethnicity of DCR Homelessness – Homeless_PastTwelveDays;
homeless Homeless_PastTwelveOccurences; Homeless_PriorTwelve
5.2 New Recommended Descriptives DCR Age – Age_Group
Consumers by Mean, mode, range, percentiles of Gender – Gender
5 Data sources that reflect Full Service Partnership consumers have been added in the event that specific knowledge about systems from this population is sought.
15
Domain Indicator Calculation of Measure(s) Data Source(s) Dimension – Potential Items
Demographic age, gender, race/ethnicity of new Race/ethnicity – Ethnicity_A; Ethnicity_B
Profile consumers (< 6 months) Length of Service – PartnershipDate
Additional Descriptives CSI Age – Age_Group
Mean, mode, range, percentiles of Gender – Gender
age, gender, race/ethnicity of Race/ethnicity – Ethnicity_A; Ethnicity_B
existing consumers (> 6 months) Length of Service – S-15.0 Admission Date; S-16.0 From/Entry Date; S-
17.0 Through/Exit Date; S-18.0 Discharge Date
DCR Age – Age_Group
Gender – Gender
Race/ethnicity – Ethnicity_A; Ethnicity_B
Length of Service – PartnershipDate
5.3 High Need Recommended Count 1 DCR Homeless – Homeless_PastTwelveDays;
Consumers Served Total homeless - FSP consumers Homeless_PastTwelveOccurences; Homeless_PriorTwelve
served6
Recommended Count 2 CSI Homeless – P-09.0 LIVING ARRANGEMENT
Total homeless - all consumers
served
Recommended Count 3 DCR Unemployed – Current_Unemployed
Total unemployment - FSP
consumers served
Recommended Count 4 CSI Unemployed – P-03.0 EMPLOYMENT STATUS
Total unemployment - all
consumers served
5.4 Access to Recommended Ratio 1 DCR Primary Care Physician – PhysicianCurr;
Primary Care Numerator: FSP consumers who PhysicianPast12
Physician have a primary care physician
currently and over the past 12
months/Denominator: Total
number of FSP consumers
Recommended Ratio 2 Additional Data An item to collect data regarding all mental health consumers access to a
Numerator: Consumers who have Collection primary care physician may be added to the CSI or incorporated into
a primary care physician currently another data collection mechanism.
and over the past 12
months/Denominator: Total
number of consumers
5.5 Consumer / Recommended Rating MHSIP Access to Services – LOCATION; TIMEGOOD; HELPWANT;
Family Perceptions Average items to create aggregate HELPNEED
of Access to measure of Perceived Access to
6 Homelessness has customarily been a challenge to measure, particularly beyond the mental health service consumer population. Should housing information about mental health
service consumers remain of questionable quality, new data collection strategies (e.g., new surveys of literal and functional homelessness or shelter counts) could be suggested.
16
Domain Indicator Calculation of Measure(s) Data Source(s) Dimension – Potential Items
Services Services
Alternative Description Primary data collection e.g., surveys, interviews, focus groups; proposed additional data collection
Qualitative and quantitative
analysis of several dimensions of
access to services
6. 6.1 Consumers Recommended Ratio Quarterly Progress CSS exhibit 6
Performance Served Annually Numerator: CSS consumers served Reports; Annual
through CSS / Denominator: CSS consumers Updates
targeted
6.2 Involuntary Recommended Ratio Annual Report on 72 hr Evaluation and Treatment (Adults, Children)
Care Numerator: Involuntary Involuntary Detentions 14 & 30-day Intensive Treatment
Detentions (i.e., Evaluation & 180-day Post Certification Treatment
Treatment, Temporary & Temporary & Permanent Conservatorships
Permanent Conservatorships) /
CSI County Client Number (CCN)
Denominator: consumers served
Alternate Ratio Annual Report on 72 hr Evaluation and Treatment (Adults, Children)
Numerator: Involuntary Involuntary Detentions 14 & 30-day Intensive Treatment
Detentions (i.e., Evaluation & 180-day Post Certification Treatment
Treatment, Temporary & Temporary & Permanent Conservatorships
Permanent Conservatorships) /
Census Population Total Statewide Population (projected)
Denominator: Population
Projection Data
6.3 24-hour Care Recommended Ratios 1 CSI Residential Information - Hospital, PHF, and SNF (S-20.0 – S-22.0)
Numerator: Utilization of MHRC, Age -Date of Birth (C-03.0)
SNF, SH among TAY, Adults, or
Older Adults/ Denominator: Total DCR Residential Information – Long-TermCare_PastTwelveOccurences; Long-
FSP TAY, Adult, Older Adults
TermCare_PriorTwelve; NursingPhysical_PastTwelveDays;
NursingPhysical_PastTwelveOccurences; NursingPhysical_PriorTwelve;
Recommended Ratio 2 Yesterday; Current; PsychiatricHospital_PastTwelveDays;
Numerator: Utilization of CTF,
PsychiatricHospital_PastTwelveOccurences;
RCL 14, MHRC / Denominator:
PsychiatricHospital_PriorTwelve
Total FSP children or total county
Age – Age_Group
child population
Alternate Count 1 DCR Residential Information – See above
Consumers in IMD, MHRC, SNF, Race/Ethnicity – CSIRace1-5
SH by race/ethnicity CSI Residential Information – See above
Race/Ethnicity – Race(C-10.0)
Alternate Counts 2 DCR Acute Care – MedicalHospital_PastTwelveDays;
Readmission to acute care facility MedicalHospital_PastTwelveOccurences;
within 30 and 180 days PsychiatricHospital_PastTwelveDays;
PsychiatricHospital_PastTwelveOccurences;
PsychiatricHospital_PriorTwelve
17
Domain Indicator Calculation of Measure(s) Data Source(s) Dimension – Potential Items
CSI Acute Care – 24 Hour Mode of Service (S-15.0 – S-19.0)
6.4 Recommended Rating POQI Appropriateness of Care – GETALL; RECOVER; SIDEFFCT;
Appropriateness of Consumer/family perceptions of RESPECT; GOALS; CULTURE; MEMANAGE; SELFHELP
Care appropriateness of care
Alternate Descriptive 1 DCR Acute Care – MedicalHospital_PastTwelveDays;
Average length of stay in acute MedicalHospital_PastTwelveOccurences
care CSI Acute Care – 24 Hour Mode of Service (S-15.0 – S-19.0)
Alternate Count 2 Primary data collection e.g., surveys, interviews, or focus groups; proposed additional data
Treatment protocols for co- collection
morbidity
6.5 Continuity of Recommended Count DCR Emergency Care – MenRelated; PhyRelated; ReferredBy;
Care Use of crisis services CSI Emergency Care – Hospital, PHF, and SNF (S-20.0, S-21.0, S-22.0)
Alternate Rating 1 DCR Residential Information –Yesterday; Current; ApartmentAlone;
Reintroduction into community AssistedLiving; CommunityCare; CongregatePlacement;
FosterHomeNon-relative; GroupHome; IndividualPlacement;
ResidentialTreatment
Primary data collection e.g., surveys, interviews, or focus groups; proposed additional data
collection
Alternate Count 2 Primary data collection e.g., surveys, interviews, or focus groups; proposed additional data
Discharge plans collection
6.6 Penetration Recommended Ratio Quarterly Progress CSS exhibit 6
Rate Numerator: CSS consumers / Reports; Annual
Denominator: high need Updates
populations California Health Demographic Information
Interview Survey
(CHIS; proposed
external data source)
Alternate Ratio DCR FSP Consumer – CountyFSPID
Numerator: FSP consumers /
Primary data collection e.g., surveys, interviews, or focus groups; proposed additional data
Denominator: individuals eligible
collection
for services among targeted
populations;
6.7 Wellbeing Recommended Rating MHSIP Functioning – DAILYPRB; CONTROL; CRISIS; BETTRFAM;
Improvement in functioning BETTRSCH; MEANINGFUL; BETTRNEED; BETTRHANDLE;
(current/over time); DOWANTS; HAPYFREND; DOTHINGS; BELONG; SUPPORT
Primary data collection e.g., surveys, interviews, or focus groups; proposed additional data
collection
Recommended Rating MHSIP Quality of Life – LIFESAT; LIVRANG; PRIVACYL STAYLONG;
Quality of life (current/over time) SPARETIM; ENJOY; FUN; RELAX; SEEFAMLY; FAMCT;
FAMGEN; VISIT; TIMERND; DOPEOPLE; TIMEPEOP; SEEPEOP;
AMTFREND
18
Domain Indicator Calculation of Measure(s) Data Source(s) Dimension – Potential Items
6.8 Satisfaction Recommended Rating MHSIP Satisfaction – LIKESVCS; CHOICES; RECOMMEND; STAFWILL;
Consumer/family satisfaction with COMFQUEST; COMPLAIN;
the care or service
7. Structure 7.1 Workforce Recommended Ratio Numerator: Primary data collection e.g., surveys, interviews, or focus groups; proposed additional data
Composition Number of staff / Denominator: collection
Number of consumers
Alternate Ratio 1 Cultural Competence Demographic Profile of Workforce – Document review
Compare demographic Plans
composition of MH workforce to
that of the consumer population
Alternate Count 2 WET Plans Consumer/family member employment – Document review
Consumer/family member
employment (i.e., number, FTE, %
of workforce)
7.2 Evidence- Recommended Additional Data Primary data collection e.g., surveys, interviews, or focus groups; proposed additional data
Based/Best Practice Collection 1 collection
Programs and Existence of best practice core
Services programs
Recommended Additional Data CSI Best Practices – S-25.0 Evidence-Based Practices / Service
Collection 2 Strategies 1
Fidelity of best practices to Primary data collection e.g., surveys, interviews, or focus groups; proposed additional data
established models collection
Alternate Additional Data Primary data collection e.g., surveys, interviews, or focus groups; proposed additional data
Collection collection
Receipt of best practices
services/supports among
consumers/families
7.3 Cultural Recommended Rating MHSIP Cultural Appropriateness – CULTURE
Appropriateness of Client and family perceptions of
Services cultural appropriateness
Primary data collection e.g., surveys, interviews, or focus groups; proposed additional data
collection
7.4 Recovery, Recommended Additional Data Recovery Oriented Recovery Orientation
Wellness, and Collection Systems Indicators
Resilience Consumer, family member, and Measure (ROSI;
Orientation staff perceptions of recovery proposed additional
orientation of system and services data collection)
Developing Recovery Recovery Orientation
Enhancing
Environments Measure
(DREEM; proposed
additional data
collection)
19
Mental Health System Indicator Measurement Detail
To clarify the rational and potential utility of the measures of each indicator, this section
provides detailed descriptions of the calculations summarized in the tables. This discussion is
based on a thorough review of all relevant existing data and, where appropriate, some alternative
data sources.
Individual-level (Consumer) Outcomes for Full Service Partnerships
Measurement Detail
Consumer indicators are individual-level priority indicators designed to create greater clarity
about consumers’ dispositions (e.g., employment, education, housing, justice involvement)
following interventions coordinated through the MHSA.7
Education/ Employment
1.1 Indicator: Average attendance – score per year
Rationale for measure: Dividing the number of consumers’ days at school during a 9-month
school year (numerator) by the total number of days during consumers’ school year
(denominator) will yield attendance rates for child consumers and TAY consumers 18 and
younger within each county. The rates will then be averaged across all counties to identify
statewide average attendance rates for each age group.
1.2 Indicator: Proportion participating in paid and unpaid employment
Rationale for measures: Employment is measured in three ways: 1) Dividing the number of
employed consumers over 18 years old (numerator) by the total number of all consumers over 18
years old who are eligible for employment (denominator) will provide the statewide proportion
of eligible consumers who are employed at the time of data collection. 2) Dividing the number of
consumers over 18 years old who are employed for pay (numerator) by total number of
consumers who are eligible for employment (denominator) will provide statewide employment-
for-pay proportions. 3) Dividing the number of consumers over 18 years who are employed
without pay (numerator) by total number of consumers who are eligible for employment
(denominator) will provide statewide employment-without-pay proportions for TAY (18 years
and older) and adult groups.
7 In early planning, consumer indicators were designed for FSP clients only. However, RFP language for this project
suggests that community activities around consumer mental health, which extend beyond FSP, are equally important
to consider in MHSA reach and impact. Thus, consumer indicators may be relevant to FSP clients and non-FSP
clients in the current report.
20
Homelessness/Housing
2.1 Indicator: Housing situation/ Index- score
Rationale for measures: To capture the variety of consumers’ housing situations, five counts
should be conducted. Among these, we recommend a count of days that 1) child consumers and
TAY consumers under the age of 18 (considered herein as dependent youth) live in a family
home annually; 2) child consumers and TAY consumers under the age of 18 live in a foster
home annually; 3) TAY over 18 (legally considered adults), adult consumers, and older adults
are homeless. Further, we recommend 4) a count of TAY over 18 and adults with independent
residential statuses as well as 5) a count of TAY over 18, adults, and older adults who are not
homeless (have any type of housing). Counts are not summative; rather they provide statewide
statuses of the housing types being used by consumers, to what extent, and the level of need
(homelessness).
Justice Involvement
3.1 Indicator: Justice Involvement
Rationale for measures: Number of consumer arrests within 12 months will be collected to track
statewide rates that may or may not be related to consumers’ mental health episodes.
Emergency Care
4.1 Indicator: Emergency Hospitalizations for Mental Health Episodes
Rationale for measures: Dividing the number of mental health episode-related hospitalizations
(numerator) by the number of consumers’ hospital visits within 12 months will give an indication
of episode severity, crisis, and rate of acute hospitalization for mental health
management/intervention. Indirectly, the ratio will give an indication of consumers’ quality of
life related to mental health.
Mental Health System-level Outcomes for all Consumers Measurement
Detail
System-level priority indicators (related to consumer access, agency performance, agency
structure) explain how operations changed or were enhanced (if at all) by the MHSA.
Access
5.1 Indicator: Demographic Profile of Consumers Served
Rationale for measure: Mean, mode, range, percentiles of age, gender, race/ethnicity of FSP
population will provide demographic description of those receiving FSP services within and
21
across counties, and allow for comparison to populations in need (e.g., overall MH service
population).
5.2 Indicator: New Consumers by Demographic Profile
Rationale for measure: Mean, mode, range, percentiles of age, gender, race/ethnicity of new
consumers (i.e., less than 6 months of service receipt) will provide an understanding of who new
consumers are and some indication of the populations which are being reached. Additionally,
demographic description of existing consumers (i.e., more than 6 months of service receipt) will
provide context for evaluating the makeup of new consumers and might give indication that
historically underrepresented groups are seeking and/or receiving services.
5.3 Indicator: High Need Consumers Served
Rationale for measure: Accurate counts of homeless and unemployed consumers served through
the FSP program can provide understanding of the extent to which these high need consumer
groups are being served. Alternately, the numbers of homeless and unemployed among all
mental health consumers will provide evidence of service to these groups overall, and provide a
relative basis with which to evaluate the extent of service to high need consumer groups through
FSP.
5.4 Indicator: Access to Primary Care Physician
Rationale for measure: Tracking the number of FSP consumers who access to a primary care
physician will provide evidence of the extent to which FSP services may be helping to connect
consumers with a medical home and the health care they need.
5.5 Indicator: Consumer / Family Perceptions of Access to Services
Rationale for measure: Aggregate ratings of consumer and family perceptions of the extent to
which they are able to connect with the services they need, will provide important evidence of
the accessibility of MHSA services from the perspective of the consumer.
Performance
6.1 Indicator: Consumers Served Annually through CSS
Rationale for measure: The number of consumers served annually through CSS relative to those
who were targeted for service will allow for CSS service rates to be understood in the context
(e.g., type and extent of need among various consumer populations) of the county in which the
services were provided. In this case, grounding service rates in county context will provide a
more accurate account of service levels/performance than a simple count of consumers.
22
6.2 Indicator: Involuntary Care
Rationale for measure: The ratio of Involuntary Detentions (i.e., Evaluation & Treatment,
Temporary & Permanent Conservatorships) to consumers served will allow for greater
understanding of this service relative to the consumer population. Such measures provide for
more accurate evaluation of services within and between counties, as well as statewide.
Alternatively, the ratio of Involuntary Detentions to various populations (i.e., adults, homeless,
unemployed) will allow for evaluation of the performance of this service within and between
consumer groups.
6.3 Indicator: 24-hour Care
Rationale for measure: The ratio of 24-hour care to consumer populations (i.e., TAY, Adult,
Older-adult populations – MHRC, SNF, SH; Child population – CTF, RCL 14, MHRC) will
provide and accurate assessment of the performance/extent of these services relative to the size
of population for which they were intended. As an alternative, demographic profiles of
consumer receiving these services may provide useful information regarding the consumer
groups who utilize such intense services most. Another measurement options would involve
counts of readmissions to acute care facilities, which can provide indication of often consumers
require this type of care.
6.4 Indicator: Appropriateness of Care
Rationale for measure: Aggregate consumer and family ratings of appropriateness of care will
provide an understanding of how services are perceived on average. As an alternative, average
length of stay in acute care (i.e., among each age group) can provide evidence of the extent to
which such intensive services are utilized, which may be more or less appropriate for different
consumer groups. Another alternative would be the existence of standard protocols for treating
co-morbidity. Issues such as substance abuse often co-occur with mental health issues, thus the
existence of treatment protocols for co-morbidity will provide evidence of the of the existence of
appropriate care for such consumers. However, this option would likely require additional data
collection.
6.5 Indicator: Continuity of Care
Rationale for measure: Use of crisis services among consumers may provide evidence of the
connection of such services with those they have previously or currently received. An alternative
measure of the extent to which consumers have been reintroduced to the community may be
created from data regarding residential status and living situation. Another option would be to
assess the existence of discharge plans, which may provide evidence to the continuity of
consumers’ paths to recovery. However, these alternate measures may require additional data
collection.
23
6.6 Indicator: Penetration Rate
Rationale for measure: The ratio of consumers who receive services through CSS to the number
of persons considered ―high need‖ (e.g., homeless, unemployed) will present the reach of CSS
programs into various populations within each county. Alternately, the ratio of FSP consumers
served to individuals eligible for services in each county would provide indication of the extent
to which FSP services are reaching those they were intended for. However, this alternate
measure may require additional data collection in order to establish the number of individuals
eligible for FSP service within each county.
6.7 Indicator: Consumer Wellbeing
Rationale for measure: Consumer and family member aggregate ratings of improvement in
functioning and quality of life will provide important measures of the perceived impact of
services on average from the consumer perspective. As these measures only tap two elements of
wellbeing, additional qualitative primary data collection may supplement these ratings by
providing more rich understanding of how services impact consumers’ wellbeing.
6.8 Indicator: Satisfaction
Rationale for measure: Consumer and family member aggregate ratings satisfaction with care or
service will provide indication of consumers’ perceived services on average.
Structure
7.1 Indicator: Workforce Composition
Rationale for measure: The ratio of staff to consumers will generate a measure of the size of the
workforce relative to the consumer population in each county. As an alternative, comparison of
the demographic makeup of the workforce and consumer populations will provide insight into
how well the workforce reflects those they serve. Another option would be to consider consumer
and family member employment in the mental health system (i.e., number, FTE, % of
workforce), which would provide evidence of the extent to which consumers have been
integrated into the service process.
7.2 Indicator: Evidence-Based/Best Practice Programs and Services
Rationale for measure: The number of evidence based or best practice programs implemented in
each county would provide indication of the extent to which established high quality programs
are being implemented within counties and across the state. Additionally, the extent to which
evidence based or best practice programs are being implemented with fidelity would provide
indication of the quality of these programs as implemented. Alternatively, the frequency with
which evidence based or best practice services are received would provide important evidence of
24
the use these programs from the consumer perspective. However, all three measures would
require additional data collection.
7.3 Indicator: Cultural Appropriateness
Rationale for measure: Consumer and family member aggregate ratings of cultural
appropriateness of services will provide an important measure of the perceived adequacy of
services with regard to consumers’ cultural needs. However, only a single survey item directly
taps cultural appropriateness of services, thus it may be necessary to augment existing data
collection or consider additional data collection, in order to create a more robust measure.
7.4 Indicator: Recovery, Wellness, and Resilience Orientation
Rationale for measure: Measure of recovery, wellness, and resilience orientation, may provide
evidence of the extent to which county mental health systems, and the state overall are adhering
to and achieving stated values and goals. No comprehensive measure of the recovery, wellness,
and resilience orientation is currently collected, however options for established measures exist
(e.g., Recovery Oriented Systems Indicators Measure, Developing Recovery Enhancing
Environments Measure).
Conclusions
Per our objectives, the evaluation team located items (variables) which may be used to construct
recommended and alternative measures relevant to priority indicators, and outlined protocols and
rational for calculating each measure. Also, where existing data was not sufficient, measures and
indicators for which additional data collection may be helpful (i.e., supplementary) or necessary
were noted. All measurement domains, priority performance indicators, calculation of measures
(recommended and alternative), the databases or reports from which items can be drawn, and the
specific items within each dataset or report were displayed in a series of tables.
Overall, this report was an important step in defining and refining the priority performance
indicators to the very practical item level. While this framework for constructing indicators is
comprehensive of all priority performance indicators, flexibility exists with regard to how each
measure may be constructed, which is reflected in the alternate measure and method of
calculation highlighted throughout the table. In order to refine the measure of each indicator and
solidify the methods of calculation for each measure, the UCLA/EMT team must conduct a
thorough data quality review.
Next Steps
Data Quality Review
As of this report, access to data listed in the report tables has not been granted. Once data
associated with each item (variable) can be reviewed, we will systematically determine data
25
quality and completeness as well as item appropriateness for each measure/indicator using the
following criteria. The data quality review will also take into account input from experts in the
field who hold expertise regarding data collection and analysis generally, and specific to the data
sources specified in this report. This process will drive further development of the indicator
template and recommendations regarding existing and additional data collection. The criteria,
also outlined in the report Templates for Reporting Priority Indicators, Deliverable 2A, must
include:
Adequate base rate (i.e., the rate at which an event occurs or level at which a scaled
response is given on average, must not be so low as to make the indicator useless or
meaningless)
Adequate variance (i.e., values of a given measure must be sufficiently distributed
through the range of the measure to support analysis)
Validity
o The measure is face valid, can conceptually and logically be said to measure what
it was intended to
o The performance measure is internally valid and can logically be tied to a
particular program intervention or outcome
o The indicator is externally valid and can logically be generalized to other
populations or programs
Reliability (i.e., the indicator is consistent over time and cases)
Availability and completeness (i.e., indicator relevant data must be obtainable and
complete for populations of interest for the period of time under study)
Ability to be aggregated to county and state levels
Stakeholder Input
To note, the team has received strong feedback regarding the use of Annual Updates to replace
Quarterly Reports. Specifically, key informants (stakeholders) perceive quarterly reports (or CSS
Exhibit 6) to be less useful in providing accurate counts. These informants have also mentioned
that Annual Reports might change given recent policy changes, thus exploration of an associated
database might not yield the information needed to incorporate variables into this project
currently. This feedback has been valuable in understanding the appropriateness and availability
of data needed (and strongly suggested) to evaluate MHSA impact.
The results of a data quality review will feed naturally into the next steps the priority indicators
development process. Specifically, the feedback gained through stakeholder review of the
proposed priority indicator template (Deliverable 2A) and this document (Deliverable 2C),
detailing procedures to construct measures of priority indicators, will be considered in light of
the observed qualities of existing data. The comprehensive process by which the evaluation team
will collect, respond to, and incorporate key stakeholder feedback is underway. The
consideration of these two important sources of input will lead to an indicator refinement process
26
which takes into account the diverse needs of MHSA stakeholders overall, MHSA performance
data users in particular, and the parameters of existing data.
The feedback process, described early in this report, will also provide for experts in the field and
key stakeholders to contribute to the development of a plan for appropriate and rigorous analysis
of all priority indicators, including the examination of MHSA impact on specific populations
(e.g., age groups, race/ethnicity, economic/living situation, language, etc) and in the context of
each counties’ unique characteristics (e.g., demographics, funding, economic factors, etc). The
feedback process, which is still being developed in collaboration with mental health organization
leaders, will be detailed in a subsequent report within the process description.
27