EvaluationMasterPlan Final 040413

Mental Health Services Oversight and Accountability Commission Evaluation Master Plan JOAN MEISEL, PHD, MBA March 25, 2013 FINAL Contents Preface .................................................................................................................................................................................. 3 EXECUTIVE SUMMARY .................................................................................................................................................. 4 GLOSSARY .......................................................................................................................................................................... 17 INTRODUCTION ................................................................................................................................................................ 18 PART 1: FINDINGS FROM THE INFORMATION GATHERING .................................................................... 22 PART 2: OVERALL MODEL AND SCOPE ............................................................................................................. 27 Levels of Outcomes ............................................................................................................................................... 30 Evaluation Questions ............................................................................................................................................. 32 PART 3: ORGANIZATION OF EVALUATION ACTIVITIES AND PRIORITY SETTING ......................... 35 Three Evaluation Methods ........................................................................................................................................ 35 Performance Monitoring ...................................................................................................................................... 35 Evaluation Studies .................................................................................................................................................. 36 Developmental and Exploratory Work Efforts ................................................................................................. 37 Priority Setting ............................................................................................................................................................. 38 PART 4: EVALUATION ACTIVITIES ........................................................................................................................ 40 Evaluation Method 1: Performance Monitoring ................................................................................................... 40 Background and Context....................................................................................................................................... 40 MHSOAC Prior and Approved Work................................................................................................................... 44 Summary ................................................................................................................................................................... 55 Evaluation Method 2: Evaluation Studies ............................................................................................................... 57 Background and Context....................................................................................................................................... 57 High Priority Evaluation Studies .......................................................................................................................... 59 Medium Priority Studies ....................................................................................................................................... 62 Summary ................................................................................................................................................................... 67 Evaluation Method 3: Developmental and Exploratory Work Efforts ........................................................... 69 High Priority Work Efforts..................................................................................................................................... 69 Medium Priorities ................................................................................................................................................... 72 Summary ................................................................................................................................................................... 78 PART 5: SPECIAL EVALUATION CONSIDERSATIONS FOR MHSA COMPONENTS ........................... 79 Community Services and Supports (CSS) ............................................................................................................... 79 Prevention and Early Intervention (PEI)................................................................................................................. 81 MHSOAC Evaluation Master Plan Page 1 Background and Context: ..................................................................................................................................... 81 PEI in the Evaluation Master Plan ....................................................................................................................... 84 Other Recommendations Related Specifically to PEI Evaluation ................................................................ 85 Innovation (INN) .......................................................................................................................................................... 85 Background and Context: ..................................................................................................................................... 85 Innovation in the Master Plan ............................................................................................................................. 86 Other Recommendations Related Specifically to INN Evaluation ............................................................... 87 Technological Needs (TN) .......................................................................................................................................... 87 Background and Context: ..................................................................................................................................... 87 TN in the Master Plan ............................................................................................................................................ 88 Other Recommendations Related Specifically to TN Evaluation ................................................................. 88 Workforce, Education, and Training (WET) ........................................................................................................... 88 Background and Context: ..................................................................................................................................... 88 WET in the Master Plan ......................................................................................................................................... 89 Other Recommendations Related Specifically to WET Evaluation.............................................................. 89 Summary ....................................................................................................................................................................... 90 PART 6: FINAL WORDS ............................................................................................................................................. 91 Overriding Issues That Affect All Elements of the Plan ...................................................................................... 91 Timeline and resources .............................................................................................................................................. 94 REFERENCES .................................................................................................................................................................... 96 APPENDIX 1: List of interviewees ............................................................................................................................. 97 APPENDIX 2: Rankings for Evaluation Studies and Developmental and Exploratory Work ...................... 98 MHSOAC Evaluation Master Plan Page 2 Preface This document assumes a reasonable level of knowledge amongst its readers about California’s public mental health system, the Mental Health Services Act (MHSA), and the Mental Health Services Oversight and Accountability Commission (MHSOAC). It is designed specifically as a plan for the MHSOAC. While it is hoped that the Evaluation Master Plan will be accessible to all stakeholders who are interested in the MHSOAC’s role in evaluation, it is purposely concise and targeted at its singular purpose – to create a simple, usable work plan for the MHSOAC over the next few years. The author thanks the Commissioners for the opportunity to work on this important task. The advice and input from Commissioners is appreciated, particularly from Commission Chair Larry Poaster and Evaluation Committee co-chairs Richard Van Horn and David Pating. Thanks are also due to the members of the MHSOAC Evaluation Committee who provided thoughtful feedback at a number of presentations. I am grateful for the serious and frank discussions with the leadership and staff from the four counties I visited - Humboldt, Los Angeles, San Bernardino, and San Mateo. Things always look a bit different from the county perspective. The work was ably supported by the staff of the MHSOAC, particularly Sherri Gauger, Aaron Caruthers, Renay Bradley, and Deborah Lee. And a special thanks to Carol Hood whose knowledge of the history of the MHSA and early MHSOAC evaluation efforts has been invaluable and whose support and friendship I greatly appreciate. Joan Meisel MHSOAC Evaluation Master Plan Page 3 EXECUTIVE SUMMARY This Evaluation Master Plan was developed for the Mental Health Services Oversight and Accountability Commission (MHSOAC) to chart its course on evaluation activity for the next 3-5 years. Evaluation is one of the core activities by which the MHSOAC fulfills its oversight and accountability role. The primary audience for this aspect of MHSOAC evaluation activity is state policy makers (Governor, Legislature, and state agencies) and the general public. Its oversight and accountability responsibilities entail ensuring that the expenditure of Mental Health Services Act (MHSA) funds is in accord with the requirements of the MHSA and that the funds are leading to improvements for the consumers of the public mental health system. The evaluation function is one of the ways in which it meets those responsibilities. The MHSOAC evaluation activity also serves a valuable quality improvement role. The results of evaluations can be used to improve policy and practice to ensure that clients and families receive the most effective services possible. In this regard the audiences for MHSOAC activities are those at the local level who are receiving services, those providing services, and those paying for services. Evaluation results can also track how well the system is doing in respecting the state’s racial/ethnic and cultural diversity. The MHSOAC has already completed a variety of evaluation efforts; and, a number of studies are currently in progress with three more approved for funding. As the MHSOAC undertakes a more robust role in evaluation it is faced with innumerable possible evaluation projects. The MHSOAC desires a plan which would chart a course of evaluation action rather than making decisions on an ad hoc annual basis as it has done to date. This is a challenging time in which to develop an Evaluation Master Plan. Rarely has the public mental health system in California faced as much change and as much uncertainty as now. Amidst a major organizational change that shifts most of the responsibilities of the now extinct California Department of Mental Health (DMH) to the California Department of Health Care Services (DHCS) the system has to accommodate the forensic realignment, the early implementation of the Affordable Care Act (ACA) expanded Medi-Cal population, the ACA Dual Eligible pilots, and the increasing focus on the integration of mental health and substance use disorders (SUD) and the integration of behavioral health with health care. Another complexity is charting a course through other statewide activity currently being done on evaluation. In addition to ongoing evaluation responsibility and activity by the California Mental health Planning Council (CMHPC), the California Institute for Mental Health (CiMH), and the California External Quality Review Organization (CAEQRO), there are four current one-time evaluation-related efforts. These are the California Health and Human Services Agency’s development of a joint plan for a coordinated evaluation of client outcomes in the community-based mental health system; the DHCS plan for outcomes for specialty Early Periodic Screening, Diagnosis, and Treatment (EPSDT) mental MHSOAC Evaluation Master Plan Page 4 health services; the California State Auditor’s extensive audit, performance and outcome review of the MHSA; and the CiMH and the Alcohol and Drug Policy Institute (ADPI) development of a business plan for DHCS with one area of focus being evaluation, outcomes, and accountability. The challenge for the MHSOAC is to develop a plan that addresses current evaluation issues and questions; attempts to anticipate the evaluation issues and questions that will be most important in the evolving health care environment; and supports the revising and updating of the plan over time. Findings from interviews with key stakeholders and site visits to four counties revealed substantial agreement about evaluation issues. Approximately 40 interviews were conducted in the preparation of this plan, and visits were made to Humboldt, Los Angeles, San Bernardino, and San Mateo counties. Major themes that emerged from the interviews and visits are as follows: Evaluation results need to be objective to maintain credibility but being able to “tell the story” about the success of mental health services should be part of the goal. The major use of evaluation should be to support efforts at continuous quality improvement. Many existing evaluation products (produced by the MHSOAC and others) are either not used at all or not used effectively. All have serious concerns about the reliability and credibility of the current state data bases. Counties are making significant upgrades to their data systems and are working on producing and using outcome data1. Comparisons between counties or between programs within a county should be done with great caution and appropriate attention to contextual factors. The continuing devolution of control over the mental health system to the counties increases the importance of local advocates having evaluation information and expertise. The movement toward the integration of behavioral health care with physical health care creates threats and opportunities with regard to evaluation. Not enough attention has been paid to measuring the extent to which the system has incorporated the values underlying the MHSA. Three evaluation principles are particularly relevant to the development of this Evaluation Master Plan: building evaluation incrementally, making results usable, and incorporating the input of two special stakeholder groups. The Evaluation Master Plan rests on principles for evaluation articulated by the MHSOAC in a number of published documents. The November, 2010 Policy Paper: Accountability through Evaluative Efforts Focusing on Oversight, Accountability and Evaluation2 document states “The MHSOAC is committed to an approach of continuous evaluation, learning from and building upon each progressive completed 1 Some counties are producing reports that document outcomes for specific programs and/or for sets of programs such as Full Service Partnerships (FSPs) or Prevention and Early Intervention (PEI) programs. 2 This MHSOAC document and others referenced in this Plan can be found at www.mhsoac.org. MHSOAC Evaluation Master Plan Page 5 evaluation.” Rather than attempting an all-encompassing one-time evaluation it has tackled evaluation questions one at a time; this strategy has proven successful thus far. This Plan has been designed in accord with this approach– it suggests a series of concrete specific evaluation activities that can build upon what the MHSOAC has already done. The second principle is exemplified by one of the objectives in the 2013 Charter for the MHSOAC Evaluation Committee, “Ensure that information from evaluation efforts and reports is usable for continuous quality improvement within California’s community-based mental health system, programs, and projects, and for revising MHSA policy guidelines.” This Report emphasizes the need for MHSOAC staff (with assistance from others as needed) to provide the Commission with interpretation of evaluation results emphasizing their implications for quality improvement. Without a concerted effort to make this translation from evaluation results to implications for policy and practice, the evaluation activities will not achieve their ultimate goal of improving the mental health system. The third principle insists on soliciting and respecting the views of the two stakeholder groups whose importance has been a cornerstone of the MHSA: persons with lived experience and their families and representatives of racial/ethnic and cultural groups who have been underserved or inappropriately served. The above cited November 2010 Policy Paper notes that in fulfilling its oversight and accountability responsibilities the MHSOAC will work “closely and collaboratively with …stakeholders including clients and their family members [and] representatives from underserved communities.” The MHSOAC has fulfilled this role thus far by focusing studies on access to care for underrepresented groups and by directly involving persons with lived experience in participatory research. The Plan includes additional efforts in both these areas. The overall paradigm underlying this Evaluation Master Plan is quite simple – determining the outcomes or results from the MHSA inputs. The MHSA provided a set of inputs to an existing mental health system – essentially money and policy and practice guidelines based on a set of underlying principles and values. The inclusion of a robust local planning process added a new element to the existing way of doing business. The inputs of the MHSA were intended to create changes in the mental health system which would lead to improved outcomes for the individuals and families served and for the general community. In the first step the values and principles articulated in the law and subsequent regulations and guidelines were to be put into practice in counties and programs (the system level). For example, the principle of recovery was to be reflected in recovery-oriented practices, and client and family direction was to lead to more involvement in treatment planning. In the second step the changed system would lead to improved results for individuals being served – reflected in the individual level outcomes. A more recovery oriented system should lead to an improved quality of life and better functional and clinical outcomes and more client and family direction into enhanced feelings of empowerment. MHSOAC Evaluation Master Plan Page 6 The diagram below outlines this basic paradigm. The diagram is not meant to be all inclusive, but it describes the reasoning behind how and what the MHSA was intended to accomplish. PARADIGM FOR EVALUATION MASTER PLAN IN PUTS OUTCOMES MH System Individuals and Family Community MHSA Members Being Served Services for SMI/SED and those at Prevalence of mental illness Stakeholder risk (access, quality, efficiency, Functional status planning p rocess satisfaction) - Living situation Seven negative outcomes -Recovery/ resilience orientation - Education/employment - For those with SMI/SED Values & principles - Integrated service experience - Social connectedness - For those at risk - Client/family driven - Family connectedness Regulations and - Culturally competent Receipt of services or guidelines - Community collaboration Quality of life supports New mon ey and - Outreach and welcoming - Well being - For those with SMI/SED - Cost effective and efficient - For those at risk - Identity services - “Serve first” rather than “fail first” - Hopefulness - CSS - Use of evidence based practices - Empowerment Stigma and discrimination - PEI - Physical health - WET System characteristics - CF/TN - Racial/ethnic and cultural disparities Clinical status - Penetration rates Innovation projects Negative events Infrastructure MHSOAC - Use of 24-hour services - Workforce - Use of emergency rooms - Housing alternatives - Abuse of substances - Information systems - Trouble with the law Existing System - Victimization - Children: Adult System of - Out-of-home placement Care - Disruptive behavior - Aggressive behavior Children’s System - School truancy o f Care F inancing structure MHSOAC Evaluation Master Plan Page 7 There are three levels of outcomes in the paradigm; individual, system, and community. The initial discussions about evaluating the MHSA introduced the idea of individual, system, and community levels. This categorization has continued, despite the fact that the distinctions were never clearly defined, there is overlap among them, and they have come to mean different things to different people. There is no “correct” way of making these distinctions. The critical thing is to be clear about how one is using the terms. The following describes the way the terms are used here. System Level. The system level refers to the features of the programs that serve individuals and the infrastructure that supports them. The usual referent for the system in this context is a county which organizes and creates policies for the programs within its systems of care. It can, however, also refer to the state level as a system of county services and a state infrastructure. The characteristics that are listed under the services cover the general categories of access, quality, efficiency and satisfaction, with the particular items listed weighted heavily towards the values of the MHSA. Individual Level. The individual level is the most straightforward referring to what happens to the persons (and families) who actually receive services. The sum of the individual outcomes on any particular outcome becomes a measure of the success of a program or a county or the state on that outcome. So, for example, a program or a county or a state could be evaluated by the percentage of its clients who are employed or who are in independent living. The critical point is that the outcome is based on what happens to individuals, so for the purposes of this paradigm they are in the “individual” level. Community Level: The community level has been the most slippery of levels to define. For some this is just another term for what is called here the system level. The meaning as used here is different than that; it refers to a population based level of analysis as opposed to the analysis of those who are served by the system. It includes outcomes for those beyond those who have received a service. The MHSA supports a continuum of activities from prevention through early intervention and treatment and it attempts to integrate its philosophy and activities into an integrated mental health system. Thus the system and community outcomes can be understood as the result of all the components of the MHSA. Additionally many of the individual outcomes will be the same for persons receiving Community Services and Supports (CSS), early intervention, and selective prevention services. The basic evaluation questions are straightforward. One could ask innumerable evaluation questions about all the items in the diagram. For the purpose of gaining an overall view of the Evaluation Master Plan it is useful to consider the most important general evaluation questions. While evaluation questions are generally applied to outcomes there are some questions which can be asked about the inputs. This is particularly the case for the local planning process which is critical to the MHSOAC Evaluation Master Plan Page 8 transformation sought by the MHSA.3 The second type of evaluation question about the inputs relates to the tracking of how the MHSA funds have been expended. So, the evaluation questions related to inputs are as follows: Has the local stakeholder process been effective? Has the MHSA money be spent as intended?4 The other evaluation questions relate to the three levels of outcomes: systems, individual, community. For the system level the basic evaluation questions are as follows: Has the mental health system improved in terms of service access, quality, efficiency and satisfaction? Has the infrastructure (workforce, technology, housing alternatives) improved? Have the values and principles of the MHSA been incorporated into the system? Are more people being served (penetration rates)? Have the disparities in amount and type of services been reduced? For the individual level there is one basic evaluation question: Are persons served / family members doing better? Are persons served / family members satisfied with services? At the community level there are four basic evaluation questions: Has the prevalence of mental illness been reduced? Have the negative outcomes for those with a SMI/SED and those at risk been reduced? Have the proportion of persons with an SMI/SED and/or those at risk who receive services and/or natural supports increased? Have stigma and discrimination been reduced? All of the evaluation activities included in the Evaluation Master Plan relate back to at least one of these basic evaluation questions. While the major focus of the Evaluation Master Plan is on the MHSA, the scope of the plan is broader. There are two major ways in which the scope of this Plan extends beyond the MHSA. The MHSA was intended to make a fundamental change in the way the mental health system operates, i.e. to transform the system and not just increase “business as usual.” The MHSA impact extends beyond the funding of specific MHSA programs; the goal is to imbue the whole system with MHSA values and principles. 3 The local planning process is different from the principles and values in not having a directly measurable result at the system level. The input of a value such as recovery can be measured by assessing the recovery orientation of programs while the local planning process input would affect the whole system and its results not be so easily measured separately. 4 This question has been treated as an evaluation issue by the MHSOAC but could also be viewed as a monitoring and accountability issue. How the responsibility for answering this question is apportioned within the MHSOAC is an internal organizational issue. MHSOAC Evaluation Master Plan Page 9 Creating another funding silo would be contrary to the intention of integrating the MHSA into the system of care. To do justice to the goals of the MHSA the whole system and not just MHSA-funded efforts must be included in evaluation efforts. A second reason is that the MHSA explicitly mandates the MHSOAC to consider in its oversight and accountability the outcomes described in statute describing the adult and children’s systems of care and to consider these systems of care in its evaluations. The above does NOT minimize the requirement for the MHSOAC to monitor the use of the MHSA funds and the outcomes associated with the use of those funds, particularly with regard to the specific components. How the money is spent and whether it is spent in accord with the statute, regulations, and guidelines needs to be tracked. But the scope of the evaluation cannot be limited to just what is directly funded by the MHSA. The evaluation activities are organized by three evaluation methods. The use of diverse methods for answering evaluation questions can be confusing to the recipient of evaluation results. Without some understanding of the methods one can’t know whether the results really answer evaluation question(s) and what limitations or cautions are needed in interpreting the results. Because this issue of the strengths and limitations of methods is so critical, the Evaluation Master Plan organizes the evaluation activities by the type of method. This is intended to eliminate misunderstandings about what the evaluation activities can tell us and to foster an understanding of the scope of what is in the Plan. The three basic evaluation methods used are Performance monitoring Evaluation studies Developmental and exploratory work efforts The MHSOAC has already used all three of these methods so that the activities within the Plan will build upon prior work Performance monitoring is a common approach to assuring accountability and promoting quality improvement. The process involves identifying a desired result or outcome. Then an indicator of that result that can be measured is identified. A good indicator is a way of measuring the intended result that is scientifically sound (reliable), is valid and meaningful (really measures what is the intended result), is feasible (has a data source which is accurate and consistent), and is useable (provides information which can be used to improve quality). A performance measurement indicator is presented as a proportions or rate, with the number achieving the desired result as the numerator and the population being measured as the denominator. A performance measurement is applied to some population or some entity. The population could be MHSOAC Evaluation Master Plan Page 10 everyone in a program, or everyone in all the programs in a county, or the population covered by a payer, or the population of a county under 200% of poverty. The indicator will define clearly what is being measured for what population by what data source for what period of time. Evaluation studies are what is most commonly understood as “evaluation” and is strictly speaking the most accurate use of the term. An evaluation study measures the results (effectiveness or efficiency) of a particular intervention. The intervention can be a program or an element of a program, a process, an initiative, or the implementation of a value. The intervention can be narrow or broad, e.g. the addition of an employment specialist to a team or the introduction of a welcoming initiative in a county. The better specified the intervention the more useful the evaluation will be. An evaluation can be qualitative or quantitative or as is often the case a mix of the two. The methodology of evaluation studies varies considerably from what would be termed a rigorous research design to a fairly simple pre and post program evaluation design. Developmental and exploratory work efforts are appropriate for evaluation questions which are of considerable interest but which don’t lend themselves to either of the two evaluation methods already described. These are generally issues which if studied would help in understanding something important about the mental health system and/or would be useful in subsequent performance monitoring or evaluation work. One example would be to determine whether FSP outcomes differ depending on demographic or clinical characteristics of the persons served. If so, then a future evaluation study to compare the effectiveness of FSP programs could include a risk adjustment factor which would make the program comparisons more valid. Another example would be determining a reliable way to assess the recovery orientation of programs. This would allow a subsequent evaluation study of the relationship between program recovery orientation and outcomes. The Evaluation Master Plan classifies evaluation activities into high and medium priorities based on a set of criteria. There are obviously far more possible evaluation activities than can be accomplished by the MHSOAC over the next 3-5 years. A set of criteria was applied to yield the priority ratings for the evaluation activities in the plan. Any set of criteria is subject to argument, and the criteria used here have not been previously approved by the MHSOAC; but, they were developed after consideration of the values and priorities in the MHSOAC documents on accountability and evaluation. There was no absolute cutoff between the high and medium priority rankings. The distinction between high and medium was used primarily in the setting of a recommended order in which the MHSOAC would conduct the various activities. Separate criteria are used for the evaluation questions and for the evaluation activity. The criteria applied to the evaluation questions are as follows: MHSOAC Evaluation Master Plan Page 11 Consistency with MHSA: Are the question(s) consistent with the language and values of the Act? Potential for quality improvement: Will answers to the question(s) lead to suggestions for and implementation of policy and practice changes? Importance to stakeholders: Are the question(s) a high priority to key stakeholders? Possibility of partners: Are there other organizations that might collaborate and/or partially fund the activity? Forward looking: Are the question(s) relevant to the evolving health care environment? Challenges: Do the question(s) address an area which is creating a challenge for the system? The criteria for the evaluation activity are the following: Feasibility: How likely is the evaluation activity to produce information that answers the evaluation question(s)? Cost: How many resources are needed to do the activity well? Timeliness: How long will it take to complete the evaluation activity? Leveraging: Does the evaluation activity build upon prior work of the MHSOAC or others? The table below lists the evaluation activities included in the plan. The activities are arranged under the three different evaluation methods. The text of the Plan includes a brief description of the nature of each activity with a statement of the relevant evaluation question(s), a review of relevant prior work, suggestions about technical matters, subject matter expertise needed, and ratings on the priority criteria. Evaluation Master Plan Activities Evaluation Method and Activity Priority PERFORMANCE MONITORING5 Step 1. Revisit, clarify, and/or revise existing priority indicators High Step 2. Develop a process for adding other indicators High Step 3. Incorporate indicators from other work groups High Step 4. Incorporate specific indicators from PEI, INN, TN, and WET components Medium Step 5. Incorporate community indicators Medium Step 6. Incorporate additional general indicators Medium Step 7. Consider adding indicators that measure change over time with individual clients Medium EVALUATION STUDIES Study 1: Person Level: Collect, summarize, and publicize the outcomes from counties that have High gathered such information Study 2: System Level (PEI) Determine outcomes of selected early intervention and selective High prevention programs Study 3: System Level (Access and Quality): Determine effectiveness of methods for engaging and High serving TAY clients Study 4: System Level (Quality) Determine effectiveness of selected programs for older adults High Study 5: System Level (Quality) Determine scope of implementation and effectiveness of evidence- High 5 The ordering of the steps under the Performance Monitoring evaluation method should not be viewed as rigid but as a suggestion for a logical ordering. Some steps can be done concurrently rather than in the order described here.. MHSOAC Evaluation Master Plan Page 12 based practices (EBP) for children and their families. Study 6: System Level (Quality): Determine the effectiveness of peer-led and consumer run services Medium Study 7: System Level (Quality): Determine the effectiveness of screening all persons receiving Medium services for substance use issues Study 8: System Level (Efficiency and Quality): Determine the effectiveness of obtaining routine Medium physical health status indicators on clients in FSPs Study 9: System Level (Efficiency) Refine and repeat FSP cost and cost offset study Medium Study 10: Person Level: Determine outcomes of promising and/or community-based practices being Medium developed by counties, particularly for un-served, underserved, or inappropriately served populations DEVELOPMENTAL/EXPLORATORY WORK Work Effort 1: PEI: Determine status of county efforts to evaluate one PEI project and make High recommendations, as needed, to ensure adequate evaluations. Work Effort 2: PEI: Develop an ongoing method for describing and cataloguing programs funded by High PEI Work Effort 3: System level (Quality, efficiency): Explore feasibility of classifying FSP programs in a High meaningful and useful fashion Work Effort 4: Community level: Develop indicators for the community level Medium Work Effort 5: Person level: Develop system to track outcomes for adults in less intensive services Medium than FSP Work Effort 6: Person and system (Quality) levels: Determine the interaction between the Medium characteristics of the populations served in FSPs and the outcomes obtained Work Effort 7: TN: Develop and implement a plan for routine monitoring and special studies of the Medium impact of technological need (TN) expenditures. Work Effort 8: System (Quality): Explore the extent of and variation in recovery orientation of programs Medium While the Evaluation Master Plan views the MHSA and the mental health system as an integrated whole it also includes consideration of evaluation items specific to the MHSA components. While the Evaluation Master Plan considers the MHSA as an integrated system, the Plan addresses the MHSOAC additional responsibility for oversight of the specific MHSA components. This dual function creates challenges, but the Master Plan accommodates both. A section of the Plan reviews each component (1) detailing how issues related to that component are included in the basic evaluation activities and (2) specifying other recommendations for MHSOAC actions related to that component. The table below shows these other recommended MHSOAC actions.6 MHSOAC ACTIONS RELEVANT TO SPECIFIC MHSA COMPONENTS Component Action PEI Urge Department of Public Health to fund evaluation of the projects to be included in the statewide California Reducing Disparities Project PEI Utilize program level data collection system from Developmental/Exploratory Work Effort #4 to collect basic statewide PEI information on numbers and characteristics of persons served PEI Do not develop a separate PEI Evaluation Framework. INN Support counties to widely disseminate the results of successful INN programs INN Collect information on the successful spread of effective INN programs 6 The Community Services and Supports (CSS) component is not listed in the table. The Plan notes that the CSS component is well represented in the activities under the three evaluation methods. The discussion of the CSS component is therefore limited to the MHSA Housing Program, and no specific actions related to that program are in the Plan. MHSOAC Evaluation Master Plan Page 13 TN7 Collaborate with the CAEQRO on the development and implementation of a plan to track impact of TN funding WET Urge OSHPD to track the implementation of county WET activity WET Obtain routine updates from OSHPD on WET activity The final part of the Evaluation Master Plan includes specific actions on a few general evaluation issues. Successful action on these overriding issues will affect the Commission’s ability to effectively implement this Plan. These are briefly noted below. The MHSOAC needs to devote more attention to using evaluation information. Strategy 6 of the MHSOAC Logic Model is “utilize evaluation results for quality improvement.” It is incumbent on the MHSOAC to take this step if it is to maximize the benefit of the resources it is devoting to evaluation. Completed evaluation reports are not the end of the process; in some ways they are the beginning. Results need to be interpreted and implications drawn. The report from each evaluation activity should be accompanied by a staff abstract which summarizes the major results, places them into context, and draws implications for consideration by the Commissioners. This may require consultation with subject experts to draw the most useful information from the evaluation activity results. The MHSOAC must continue to address the data system situation. The current data bases that the MHSOAC uses for much of its evaluation effort are technologically outdated, inconsistently used by programs and counties, and inadequately supported by the relevant state agencies. The transition of the maintenance of the data bases from the DMH to the DHCS offers the possibility of improvement or of a continued slow decline. Many stakeholders are devoting energy and focus on the development of a new data enterprise architecture. While the MHSOAC can support this effort,8 it must urgently continue to monitor and reinforce the importance with DHCS (and the Department of Finance) of the immediate need for DHCS to maintain and support the existing data sources in the interim. Unless these systems are supported and maintained, much of the Evaluation Master Plan will be compromised. For many reasons the MHSOAC should involve stakeholders more in its evaluation work. Involving others will add to the credibility of the work, will promote the wider distribution of the results, and will facilitate acknowledgement of the good evaluation work being done by the Commission. Additionally the range of subject matter expertise needed to cover all the areas in the Master Plan will require others to supplement internal capacity. The MHSOAC should consider collaboration with other entities whenever possible. Other organizations and entities have many of the same interests in evaluation as does the MHSOAC. Collaboration on an evaluation activity is not easy, but the benefits can be worth the effort. 7 TN is part of the Capital Facilities/Technological Needs (CF/TN) component. Since there are no activities in the Evaluation Master Plan related to CF the component is referred to as TN. 8 The Master Plan suggests that a potentially more successful approach with the Administration would be to propose a feasibility study of a new system rather than the actual development of such a system. MHSOAC Evaluation Master Plan Page 14 Joint sponsorship of a study can increase the constituency for and credibility of the results and potentially cut the expense of a study. Examples of such partnerships are included in a few of the evaluation activities. The MHSOAC should continue to refine its method of selecting and monitoring contractors. The Commission should continue its effort of widening the distribution of its RFPs which will increase the number of creditable bidders from which to choose. Consideration should be given to placing a greater weight in the evaluation of bidders on having “knowledge of and experience with California’s mental health system.” While not micromanaging, the MHSOAC staff should take a more active role in assuring that the study design and methodology are both scientifically sound and responsive to MHSOAC needs and that the data sources that will be used in the study will yield reliable results. The Evaluation Master Plan lays out an ambitious agenda. The Master Plan calls for seven steps in maintaining and upgrading the Performance Monitoring system, 10 Evaluation Studies, and eight Developmental and Exploratory Work Efforts. If, as anticipated, most of the Evaluation Studies and Developmental and Exploratory Work Efforts extend beyond one year, the MHSOAC could be actively engaged with more than ten projects during the course of a year, not including the ongoing work on the Performance Monitoring system. The Evaluation Master Plan cannot be implemented as envisioned with the existing level of internal staff resources devoted to evaluation. To maintain and upgrade the Performance Monitoring system will require an ongoing commitment of internal resources with the addition of some subject matter expertise. Each of the Evaluation Studies and Developmental and Exploratory Work Efforts requires internal resources to further refine the study or work effort parameters, draft and issue an RFP, review proposals and develop contracts, and monitor the contractor’s work. The recommendations in the Master Plan would require a more active role for internal staff in contract monitoring, more efforts to include subject matter and other stakeholders in the work (requiring coordination), and a more intensive effort at interpreting and drawing implications from the results of the evaluation activities. In addition to the activities in the Master Plan, internal evaluation staff must attend numerous meetings with other constituencies and stakeholders, coordinate with DHCS and others over data issues, and plan and coordinate the work of the Evaluation Committee. Participation in these additional activities contributes to the quality of MHSOAC evaluation efforts, but requires additional time and work on the part of MHSOAC evaluation staff. The speed with which the evaluation activities can be implemented will be a function of the amount of funds available for contracts as well as the capacity of the internal staff. To accomplish the full set of evaluation activities would require additional funds for contracting as well as an augmentation of internal staff resources. The MHSOAC has generally initiated two or three contracts a year funded out of its annual $1 million set aside for evaluation. The Evaluation Master Plan MHSOAC Evaluation Master Plan Page 15 would add four to five new contracts a year if all the Evaluation Studies and Developmental and Exploratory Work Efforts were started within four years. The amount of resources devoted to contracts needs to be calibrated with the capacity of the internal staff, or the results from contracts will not be as creditable or useful as they might be. There may also be draws on the MHSOAC evaluation funds from unexpected sources. The most likely is the need to devote resources to the amelioration and then maintenance of the existing data systems. While this is not the responsibility of the MHSOAC it may be incumbent on the Commission to assist the DHCS as it has already been doing through its contract with the California State University, Sacramento for work on the Data Collection and Reporting (DCR) system. MHSOAC Evaluation Master Plan Page 16 GLOSSARY ACA Affordable Care Act ACO Accountable Care Organization ADPI Alcohol and Drug Policy Institute CAEQRO California External Quality Review Organization CalHFA California Housing Finance Agency CalMHSA California Mental Health Services Authority CHIS California Health Interview Survey CHKS California Healthy Kids Survey CiMH California Institute for Mental health CMHDA California Mental Health Directors Association CMHPC California Mental Health Planning Council CPS Consumer Perception Survey CRDP California Reducing Disparities Project CSI Client and Services Information system CSS Community and Services and Supports component of MHSA DCR Data Collection and Reporting system DHCS California Department of Health Care Services DMH California Department of Mental Health EBP Evidence based practice EPSDT Early Periodic Screening, Diagnosis, and Treatment FSP Full Service Partnership HEDIS Health Effectiveness Data and Information System HHS U.S. Department of Health and Human Services HSRI Human Service Research Institute INN Innovation component of MHSA KET Key Event Tracking from in DCR system MAP Measures Application Partnership MHSA Mental Health Services Act MHSIP Mental Health Statistics Improvement Program MHSOAC Mental Health Services Oversight and Accountability Commission NAMI National Alliance on Mental Illness NBHQF National Behavioral Health Quality Framework NQF National Quality Forum NCQA National Committee for Quality Assurance NOMS National Outcomes Monitoring System OSHPD California Office of Statewide Health Planning and Development PEI Prevention and Early Intervention component of MHSA PSDA Plan, Do, Study, Act RFP Request for proposals SAMHSA U.S. Substance Abuse and Mental Health Services Administration SED Serious emotional disturbance SMI Serious mental illness SOC System of Care SUD Substance use disorders TAC Technical Assistance Collaborative TAY Transitional aged youth TN Technological Needs, part of Capital Facilities/Technological Needs component of MHSA UACF United Alliance for Children and Families WET Workforce and Education component of MHSA MHSOAC Evaluation Master Plan Page 17 INTRODUCTION Evaluation is one of the core activities by which the MHSOAC fulfills its oversight and accountability role. The MHSOAC focused during the first five years on the implementation of the MHSA – on the planning process, the initial three-year plans for each component, and the accounting for the expenditure of MHSA funds. Once the initial plans were approved and implementation begun, the MHSOAC began to switch attention to evaluating the results of the MHSA activities. Its November 8, 2010 Policy Paper: Accountability through Evaluative Efforts Focusing on Oversight, Accountability, and Evaluation9 stated “The MHSOAC will now be broadening its focus from MHSA implementation to greater emphasis on program evaluation focusing on outcomes and the appropriate and effective use of MHSA funds.” The MHSOAC Logic Model, adopted by the Commission on July 28, 2011 outlines its oversight and accountability strategies. Two of these relate directly to evaluation: Strategy 5 is “to evaluate the impact of the MHSA” and Strategy 6 is “to use evaluation results for quality improvement.” Two others require the active involvement of evaluation staff: Strategy 2 (“ensure collection and tracking of data and information”) and Strategy 7 (“communicate impact of MHSA”). While the initial efforts are to be focused on the MHSA, the November 2020 Policy Paper makes clear that “as the MHSA is more fully integrated into the community mental health system, the focus of the MHSOAC’s oversight and accountability will be expanded to the public community mental health system.” The MHSA indicates that the outcomes to be evaluated by the MHSOAC include those in the Adult and Children’s Systems of Care. The MHSOAC has launched an ambitious early effort at evaluation. Subsequent to the November 2010 Policy Paper, the MHSOAC contracted with the UCLA Center for Healthier Children, Youth, and Families, along with its subcontractors, for a major evaluation of the MHSA. The contract called for a set of evaluation activities to extend over the next few years. The evaluation activities included reports on the expenditures by MHSA component, reports on a set of priority performance indicators, a study of the average costs and cost offsets of Full Service Partnerships (FSPs), and outcome results from early implementation of Community Services and Supports (CSS) and Prevention and Early Intervention (PEI) county programs. The contracts are nearing completion. In subsequent sections of this report the relevant studies from this contract are cited. Beginning in FY 2010-11 an additional $1 million was set aside annually for evaluation efforts. The Evaluation Committee reviewed staff recommendations based on a set of criteria with final approval of specific evaluation studies made by the Commission. Two years of funded projects have been completed or are in progress. For FY 12-13, some of that funding supports a new staff position, a chief of Research 9 This and other MHSOAC reports can be found at www.mhsoac.org. MHSOAC Evaluation Master Plan Page 18 and Evaluation, and another set of studies was recently approved by the Commission. MHSOAC funded projects completed to date and in progress are referred to as relevant in the rest of this document. This is both an opportune and a challenging time in which to develop an Evaluation Master Plan. With the added focus on and a resource commitment to evaluation, the MHSOAC sought a plan for how to proceed for the next three to five years. Rarely has the public mental health system in California faced as much change and as much uncertainty as now. The known changes include the abolition of the Department of Mental Health, the expansion of Medi-Cal under the Affordable Care Act (and its early implementation in California under the Medicaid Section 115 waiver), the realignment of a large portion of the state’s forensic population to the counties, the Dual Eligible (Medi-Cal and Medicare) pilots, and the implementation of the rest of the Affordable Care Act (ACA) beginning in 2014. While the fact of these changes is known, the actual consequences of them remain uncertain. How will the California Department of Health Care Services (DHCS) absorb the responsibilities of overseeing and providing leadership for the mental health and substance abuse systems? How will the coverage of mental health and substance abuse services under the expanded Medi-Cal eligibility and the California Health Benefit Exchange affect the demand for mental health services? How will all the pressures for increased coordination of physical and behavioral health care impact the public mental health system? How will the behavioral health needs of the forensic population be managed effectively and safely? The challenge for the MHSOAC is to develop a plan that addresses current evaluation issues and questions; anticipates the evaluation issues and questions that will be most important in the evolving health care environment; and, supports the revising and updating of the Plan over time. Another complexity is charting a course through all the other activity currently being done on evaluation within the public mental health system. The MHSOAC is not alone in its interest in evaluation. Significant effort is being devoted to multiple evaluation efforts. CiMH is building a system to evaluate evidence-based practices using a Palettes of Measures approach and is using a Breakthrough Series use of a PDSA (Plan, Do, Study, Act) structure for a number of learning collaboratives. The California External Quality Review Organization (CAEQRO) conducts extensive reviews of county mental health plans and produces an annual state report as required by Medicaid. The Consumer Perception Survey (CPS), which meets federal block grant requirements, has most recently been conducted by the California Institute for Mental Health (CiMH). The state reports on the Uniform Reporting System as part of the federal National Outcome Monitoring System (NOMS) – also to comply with federal block grant requirements. And two highly regarded organizations – Technical Assistance Collaborative (TAC) and Human Services Research Institute (HSRI) - recently completed a needs assessment of the state’s mental health and substance abuse systems as part of the state’s Medicaid 1115 waiver request10. Finally the California Mental Health Planning Council (CMHPC) continues to have the statutory responsibility “to review and approve performance outcome measures.” 10 Available at www.dhcs.ca.gov/provgovpart/Pages/BehavioralHealthServicesAssessmentPlan.aspx MHSOAC Evaluation Master Plan Page 19 Added to this are four current special one-time evaluation-related efforts. The MHSAOAC is to “work in collaboration with the State Department of Health Care Services and the California Mental Health Planning Council, and in consultation with the California Mental Health Directors Association, in designing a comprehensive joint plan for a coordinated evaluation of client outcomes in the community-based mental health system…The California Health and Human Services Agency shall lead this comprehensive joint plan effort.” (AB 1467, 2012). The DHCS is charged with creating a plan for a performance outcome system for specialty EPSDT mental health services to be informed by a stakeholder advisory committee. The plan is to be presented to the Legislature by October 1, 2013, with an implementation plan to be part of the Governor’s 2014-15 Budget (Mental Health Realignment Trailer Bill SB 109, 2012). The California State Auditor is conducting an extensive audit, performance and outcome review of the MHSA including reviews of how four counties are collecting and analyzing data to determine the effectiveness of programs and how the data is being used to modify and improve programs. The expected completion date is May 2013. CiMH and the Alcohol and Drug Policy Institute (ADPI) are developing a business plan for DHCS to use in establishing priorities as it assumes responsibility for the state’s role in public mental health and substance abuse services. One major area of focus is evaluation, outcomes, and accountability. The recommendations to DHCS are expected in early 2013. This Evaluation Master Plan has been created to help the MHSOAC fulfill its oversight and accountability functions and to promote quality improvement. The need for information about mental health services is different depending on who is using the information and for what purpose. There is always tension between the use of evaluation for compliance or for quality improvement. And the interests differ among the state, counties, providers, and consumers and family members. While a single evaluation system that would meet everyone’s needs might be ideal, in practice such a system does not seem either feasible or optimal. What a clinician wants to know about the effectiveness of his services with a particular client is different from what the program manager wants to know about the effectiveness of all her staff. And these are different again from what the county administrator wants to know about the relative efficiency of her contract programs or what the state wants to know about overall state level impacts. While the idea of being able to “roll up” the data from one level to the next sounds simple, doing so is not only exceedingly difficult but also loses the special perspective of each level’s needs and interests. The MHSOAC’s primary audience is state policy makers (Governor, Legislature, and state agencies) and the general public. Its oversight and accountability responsibilities entail ensuring that the expenditure of MHSA funds is in accord with the requirements of the MHSA and that the funds are leading to improvements for the consumers of the public mental health system. The evaluation function is one of the ways in which it meets that responsibility. MHSOAC Evaluation Master Plan Page 20 Beyond meeting that accountability responsibility, the MHSOAC also views its evaluation role as providing information to all stakeholders to be used in efforts to improve the quality of the system. This Report includes findings from information gathered in the development of the plan as well as recommended evaluation activities. Part 1 states the findings from the Information Gathering phase of the plan’s development. Interviews were conducted with roughly 40 individuals and/or organizations with experience and interest in the evaluation of the mental health system (see Appendix 1 for a list of interviewees). Site visits were conducted to four counties (Humboldt, Los Angeles, San Bernardino, and San Mateo) to review the data system and evaluation efforts and challenges at the county level. The evaluation endeavors of a few states and trends in national evaluation activity were also briefly explored. Part 2 contains the overall model and scope of the Evaluation Master Plan which derives from the principles and values articulated in MHSOAC documents and feedback from MHSOAC Commissioners and staff. The model includes the overall evaluation questions and the three levels of analysis (individual, system, community) used in the Plan. Part 3 contains brief descriptions of the three evaluation methods (performance monitoring, evaluation studies, developmental and exploratory work efforts) that provide the structure for the evaluation activities in the Plan. It also contains a set of criteria used to prioritize the evaluation activities. Part 4 contains the actual evaluation activities to be conducted over the next 3-5 years. Part 5 includes special considerations for particular MHSA components. A final part contains recommended actions for the MHSOAC on a few overarching evaluation issues and briefly addresses resources needed to implement the Evaluation Master Plan. MHSOAC Evaluation Master Plan Page 21 PART 1: FINDINGS FROM THE INFORMATION GATHERING There is substantial agreement about the role and function of the MHSOAC in evaluation. Stakeholders agree that the MHSOAC has a unique role in providing evaluation results to state level policy makers and to the general public. Many express the desire that the MHSOAC “tell the story” about the value of mental health services to consumers and family members, but there is an accompanying recognition that the MHSOAC needs to be viewed as objective in its evaluations or risk losing its credibility. And no matter what the results of its evaluations, they must be presented in a way that is understandable to policy makers and to the general public. For the MHSOAC to meet its responsibilities, the findings and implications of its evaluations must be clear to those with less sophistication no matter how technical and rigorous the studies might be. Stakeholders believe the major use of evaluation should be to support continuous quality improvement. The premise of this approach is that service providers try to do a good job and will use evaluation information to improve their services if evaluation is done with that aim. In this view the use of evaluation for compliance purposes interferes with its potential utility for quality improvement. The potential for quality improvement exists at all levels of the system. Some advocate that the most likely avenue for improvement is in the interactions between staff and clients. For this reason they contend that evaluations at the program level are advisable to take advantage of where the motivation and opportunity for change is strongest. The MHSOAC’s use of program level data is dependent on the particular evaluation activity. If an evaluation activity is designed to assess the impact of a particular type of program then it will need to look at results at the program level. But tracking and using program level data as part of an ongoing monitoring effort is generally more a role for the counties than for the state. Many existing evaluation products are either not used at all or not used effectively. As noted in the Introduction many entities conduct either routine or special evaluation efforts. Interviewees were either unaware of these activities or if aware could not articulate what the evaluations showed, how the information was disseminated, or how it was used. This lack of knowledge of and use of evaluation information extends to the evaluation activities of the MHSOAC as well as to evaluations conducted by others. While there is understandable worry about duplication of evaluation effort, the bigger problem is that no one’s efforts are being used to either confirm current effectiveness or to change policy or practice. Unless the use of evaluation information is improved, the value of the evaluation effort will be marginal. What is needed to produce effective evaluation results is clear. The critical elements for successful evaluation at any level are data that can be trusted and technical expertise to analyze and draw valid and meaningful conclusions from the data. Beyond that the entity’s MHSOAC Evaluation Master Plan Page 22 culture must support evaluation which is bolstered by leadership that is actively interested in asking questions and delving into the evaluation results. The final requirement is sufficient resources to support an infrastructure that includes information technology and evaluation staff who can serve as intermediaries between the programs and the technical data people. “Everything starts and rests on the data,” and there are serious concerns in this area. Two of the three major state data bases – the Client and Service Information (CSI) and the Data and Collection Reporting (DCR) System - were not adequately supported by the Department of Mental Health (DMH) in recent years. As the resources of the DMH declined the number of persons (a) with the expertise to technically manage and support the data systems and (b) with the experience of using the information in those data systems was diminished. By the time the systems and support people were transferred to the DHCS there were already significant concern. Given the large task of incorporating the responsibilities of the DMH, DHCS has not yet been able to address how to fix and then support on an ongoing basis the transferred data bases. Making the task more difficult is the loss of specific expertise about the systems and no clear direction about how to proceed. The DCR system was created by the DMH specifically to gather outcome information about enrollees in Full Service Partnerships (FSPs). The MHSOAC has an ongoing contract with California State University, Sacramento to provide support in making the DCR system more accurate and useful. This effort has been very successful in clarifying data elements through the creation of a data dictionary and user manual, in highlighting problematic procedures, in developing clearer protocols for entering data, in helping providers create incentives for staff to enter accurate and timely information, and in helping counties use the data to understand and monitor their FSP programs. But these efforts will be to no avail if the technical data system itself is not supported by the DHCS. Funds from the Capital Facilities/Technological Needs (CF/TN) Component of the MHSA have supported the enhancement of county data systems. Most of the counties have embarked on the adoption of Electronic Health Records (EHRs). The extent and progress of this effort varies by county depending upon when they started and on other challenges unique to each county. The effort has been more time and resource intensive than anticipated, as is usually the case with such technology upgrades. Three large vendors are doing most of the installations, and the counties have formed user groups to encourage the vendors to make adaptations that meet their needs. For example, special modules need to be added to accommodate outcome data elements. Many support the creation of a new enterprise data system architecture reliant on newer technology. Even with better support the state’s data systems use antiquated technology. Given the dramatic changes in data system technology some wonder at the wisdom of continuing to support antiquated systems. Technology experts claim that a newer state architecture would make the data requirements less onerous on the counties. MHSOAC Evaluation Master Plan Page 23 Further, none of the current data systems provides a full and accurate account of the public mental health system. The Short-Doyle/Medi-Cal claims system covers only Medi-Cal clients and outpatient services covered by Medi-Cal,11 and the DCR covers only FSP enrollees. The CSI, while covering everyone who receives a service, is often not updated as required so that it does not allow an accurate assessment of the current status of clients. Estimates of the time and resources to create a new data system architecture vary and would be dependent on how extensive a system would be created. The Evaluation Master Plan recommends that a first step be to conduct a feasibility study of alternative approaches. But even with a decision to proceed with a new system, there appears to be no alternative to maintaining the existing systems until such a system were in place. There is understandable reluctance to use any evaluation reports to make comparisons either between programs or counties. The diversity of populations, services, funding, and other contextual factors makes the comparison across entities or programs problematic. Certainly making decisions at the state level based on such comparisons is not advised. But making information public can have benefits. It can raise questions that help to understand the reasons for noted differences, and it can highlight what appear to be better performers for others to emulate. There will be resistance to establishing any benchmarks or performance standards until the issues of differences can be addressed.12 The continuing devolution of control over the mental health system to the counties increases the importance of local advocates having evaluation information and expertise. With the demise of the DMH and the removal of almost all approval by state authorities over MHSA plans, control of the mental health system rests with county Boards of Supervisors. Local Mental Health Boards or Commissions have a legislatively mandated local oversight function but are purely advisory to the Boards of Supervisors. These Boards and Commissions, with a required minimum of 51% consumers and family members, are variably effective. Differences in effectiveness are believed to be related to how much membership fluctuates (are there any steady members), how sophisticated members are about data (are there at least a few with some experience and expertise), and how the county uses the Board/Commission (do they advocate for the needs of the system; do they visit programs; do they engage in serious discussion about plans, budgets, new programs, and evaluation results). The California Mental Health Planning Council and CiMH have undertaken training activities with the local Boards and Commissions with admittedly mixed results. The requirement for a robust local planning process as part of the MHSA plan development broadened considerably the number of local stakeholders and the level of their participation. In some instances the 11 A minority of Medi-Cal inpatient services are in the Short-Doyle-Medi-Cal system but most are in the Inpatient Hospital Consolidation data system. 12 See Developmental and Exploratory Work Effort #3 (Explore feasibility of classifying FSP programs in a meaningful and useful fashion) and Work Effort #6 (Determine the interaction between the characteristics of the populations served in FSPs and the outcomes obtained). MHSOAC Evaluation Master Plan Page 24 local MH Board acted as the convener of the planning process, but often a separate planning group was established. The initial planning efforts are generally viewed as having been robust, energetic, and effective; a current MHSOAC evaluation project is under development to assess the effectiveness of this effort. The ongoing role of these local planning efforts after the transition from planning to implementation has not been studied, e.g. have they transitioned to an effective oversight and quality improvement role? The movement toward the integration of behavioral health care with physical health care creates threats and opportunities with regard to evaluation. Behavioral health policy makers need to attend to the implications of this trend for its data systems and evaluation efforts. The Medi-Cal Specialty Mental Health Services Consolidation and the Freedom of Choice waiver under Section 1915(b) of the federal Social Security Act give county mental health departments the first right of refusal to be the designated Mental Health Plan with the authority to provide or contract for all Medi-Cal medically necessary specialty mental health services. This design – the “carve out” – maintains the traditional strong county role in organizing and providing services for persons with serious mental illness or children/youth with a serious emotional disturbance. While this has produced an organized system of mental health services that controls costs it has created a silo within general health care. With attention shifting to coordinating health care for the individual, the weakness of the carve-out from a quality of care perspective is receiving increasing attention. The carve-out model has facilitated the development and maintenance of data reporting systems that support both county level and state level analysis. Should the carve-out model be either modified or eliminated the collection and analysis of public mental health service data could be threatened. On the other hand, the focus on the much higher physical health care costs attributable to persons with a serious mental illness and substance abuse issues creates an opportunity to demonstrate through good evaluation that effective mental health care can reduce overall health care costs. And of great importance to consumers and their families this creates an opportunity to highlight and track improvements in the length of life spans of persons with a serious mental illness. Stakeholders have interests in specific kinds of evaluation. Stakeholders interviewed for this project indicated interest in specific kinds of evaluation. The following represent some of that variation. Participatory research: Consumers and family members support the inclusion of people with lived experience in the design, implementation, analysis, and interpretation of evaluation studies. Persons from racial/ethnic and cultural groups who have been traditionally underserved or inappropriately served want all evaluations to consider the special needs of these populations. Immediate feedback: People at the program level would like real-time evaluation systems so that they could use the results in their interactions with consumers during the course of treatment. MHSOAC Evaluation Master Plan Page 25 Natural supports: Some policy makers want to better understand how people with serious mental illness use their natural supports rather than or in addition to the mental health system and how we can enhance the capacity of those natural supports. Regional evaluation networks: Some county evaluation staff perceive value in supporting forums in which they can share their experiences and potentially undertake regional evaluation efforts. There is concern that there has not been enough attention to measuring the extent to which the system has incorporated the values underlying the MHSA. The implicit assumption in the MHSA and the implementation guidelines was that if the system adopted the values and principles underlying the Act, the outcomes would be better for consumers and families. Some effort has been made to measure whether these values have been embraced, e.g. Consumer Perception Survey (CPS) questions about consumer involvement in decision making. But a concerted effort to measure how well these values have been adopted in programs has not occurred. A corollary has been too little attention paid to adjusting the outcomes measured at the individual level to reflect dimensions of recovery outside of functional outcomes, e.g. hope, identity, and empowerment. The strategy of using data to improve the quality of health care is well entrenched with new attention to behavioral health. For at least the last two decades the tracking of the performance of health care organizations has expanded in order to improve the quality of services. Innumerable private and public entities now produce and use performance measurement systems. The Institute of Medicine Crossing the Quality Chasm: A New Health Care System for the 21st Century13 supported work already begun and prompted new endeavors. The Affordable Care Act calls for a National Strategy for Quality Improvement in Health Care, and many of the initiatives in the Act support evaluation efforts designed to improve the quality of care. Initial sets of performance measurement standards included few if any behavioral health measures. This has changed in recent years with greater attention nationally to the creation of such measures. And Substance Abuse and Mental Health Services Administration (SAMHSA) is now creating a National Behavioral Health Quality Framework which will include a Behavioral Health Barometer to highlight key behavioral health indicators. (See the Section on Performance Monitoring for more information about these national efforts.) 13 Available at www.iom.edu/Reports/2001/Crossing-the-Quality-Chasm-A-New-Health-System-for-the-21st- Century.aspx MHSOAC Evaluation Master Plan Page 26 PART 2: OVERALL MODEL AND SCOPE The Evaluation Master Plan rests on principles for evaluation articulated by the MHSOAC. The MHSOAC has issued a number of documents14 related to its policies on evaluation: Measurement and Outcome Technical Resource Group, May, 2008 Policy Paper: Accountability through Evaluation Efforts Focusing on Oversight, Accountability and Evaluation, November 8, 2010 Establishing Priorities and Focusing Evaluations, September 14, 2011 Charters for the Evaluation Committee – most recent is the 2013 Charter The basic principles include conducting evaluations which are technically sound, reflect the values underlying the MHSA, measure outcomes articulated in the MHSA including the adult and children’s systems of care, are culturally and linguistically competent, focus on issues which can improve the quality of services, and incorporate the views of persons with lived experience and their families. Three evaluation principles are particularly relevant to the development of this Evaluation Master Plan: building evaluation incrementally, making results usable, and incorporating the input from two special stakeholder groups. The November 8, 2010 document states “The MHSOAC is committed to an approach of continuous evaluation, learning from and building upon each progressive completed evaluation.” The MHSOAC has successfully conducted a series of evaluation efforts, each of which addressed a particular issue or utilized a particular evaluation method. Rather than attempting an all-encompassing one-time evaluation it has tackled evaluation questions one at a time; this strategy has proven successful thus far. This Plan has been designed in accord with this approach– it suggests a series of concrete specific evaluation activities that can build upon what the MHSOAC has already done. The second principle is exemplified by one of the objectives in the 2013 Charter for the Evaluation Committee, “Ensure that information from evaluation efforts and reports is usable for continuous quality improvement within California’s community-based mental health system, programs, and projects, and for revising MHSA policy guidelines.” This Report recommends a bigger role for MHSOAC staff (with others as needed) to interpret evaluation results as they impact policy and practice. Without a concerted effort to make this translation, the evaluation activities will not achieve their ultimate goal of improving the mental health system. The third principle insists on soliciting and respecting the views of the two stakeholder groups whose importance has been a cornerstone of the MHSA: persons with lived experience and their families and representatives of racial/ethnic and cultural groups who have been underserved or inappropriately served. The above cited November 2010 Policy Paper notes that in fulfilling its oversight and accountability responsibilities the MHSOAC will work “closely and collaboratively with …stakeholders 14 These documents can be found on the MHSOAC website, www.mhsoac.org. MHSOAC Evaluation Master Plan Page 27 including clients and their family members [and] representatives from underserved communities.” The MHSOAC has fulfilled this role thus far by focusing studies on access to care for underrepresented groups and by directly involving persons with lived experience in participatory research. The Plan includes additional efforts in both these areas. The overall paradigm underlying this Evaluation Master Plan is quite simple – determining the outcomes from the MHSA inputs. The MHSA provided a set of inputs to an existing mental health system – essentially money and policy and practice guidelines based on a set of underlying principles and values. The inclusion of a robust local planning process added a new element to the existing way of doing business. The inputs of the MHSA were intended to create changes in the mental health system which would lead to improved outcomes or results for the individuals and families served and for the general community. In the first step the values and principles articulated in the law and subsequent regulations and guidelines were to be put into practice in counties and programs (the system level). For example, the principle of recovery was to be reflected in recovery-oriented practices, and client and family direction was to lead to more involvement in treatment planning. In the second step the changed system would lead to improved results for individuals being served – reflected in the individual level outcomes. A more recovery oriented system should lead to an improved quality of life and better functional and clinical outcomes and more client and family direction into enhanced feelings of empowerment. The diagram below outlines this basic paradigm. The diagram is not meant to be all inclusive, but it describes the reasoning behind how and what the MHSA was intended to accomplish. MHSOAC Evaluation Master Plan Page 28 PARADIGM FOR EVALUATION MASTER PLAN IN PUTS OUTCOMES MH System Individuals and Family Community MHSA Members Being Served Services for SMI/SED and those at Prevalence of mental illness Stakeholder risk (access, quality, efficiency, Functional status planning p rocess satisfaction) - Living situation Seven negative outcomes -Recovery/ resilience orientation - Education/employment - For those with SMI/SED Values & principles - Integrated service experience - Social connectedness - For those at risk - Client/family driven - Family connectedness Regulations and - Culturally competent Receipt of services or guidelines - Community collaboration Quality of life supports New mon ey and - Outreach and welcoming - Well being - For those with SMI/SED - Cost effective and efficient - For those at risk - Identity services - “Serve first” rather than “fail first” - Hopefulness - CSS - Use of EBPs - Empowerment Stigma and discrimination - PEI - Physical health - WET System characteristics - C/TN - Racial/ethnic and cultural disparities Clinical status - Penetration rates Innovation projects Negative events Infrastructure MHSOAC - Use of 24-hour services - Workforce - Use of ER - Housing alternatives - Abuse of substances - Information systems - Trouble with the law Existing System - Victimization - Children: A dult System of - Out-of-home placement Care - Disruptive behavior - Aggressive behavior Children’s System - School truancy of Care F inancing structure MHSOAC Evaluation Master Plan Page 29 Levels of Outcomes There are three levels of outcomes in the paradigm. The initial discussions about evaluating the MHSA introduced the idea of individual, system, and community levels. This categorization has continued, despite the fact that the distinctions were never clearly defined, there is overlap among them, and they have come to mean different things to different people. Many of the outcomes at the system level consist of summing up outcomes for individuals, e.g. a measure of the use of 24-hour services is the sum of those who were in such a setting. The “program” level does not appear in the conceptualization when it is clearly a critical part of the system. And perhaps most confusing is what “community” means beyond the system. There is no “correct” way of making these distinctions. The critical thing is to be clear about how one is using the terms. The following describes the way the terms are used here. System Level The system level as used here refers to the features of the programs that serve individuals and the infrastructure that supports them. The usual referent for the system in this context is a county which organizes and creates policies for the programs within its systems. It can, however, also refer to the state level as a system of county services and an infrastructure. The characteristics that are listed under the services cover the general categories of access, quality, efficiency and satisfaction with the particular items listed weighted heavily towards the values of the MHSA. While service system characteristics refer mostly to treatment services the intention of the MHSA is to view the whole continuum of efforts from prevention through early intervention through all levels of treatment as one integrated system. So while there is no specific mention of PEI in the system box the noted characteristics could apply to the prevention and early intervention efforts as well as the treatment services. Another type of outcome listed measures the nature and numbers of persons served (penetration rates) and the nature of any disparities in service amounts or type by demographic characteristics of the population, particularly racial/ethnic and cultural characteristics. Individual Level This level is the most straightforward referring to what happens to the persons (and families) who actually receive services. For the purposes of evaluation the individuals must receive enough service (of any type) that they become part of a data collection system that tracks their individual characteristics and outcomes. Those who receive traditional services will be part of the county and state data collection systems. Those who receive early intervention or selective prevention services can have outcome results even if they are not entered in the treatment data systems if there is a record of their individual characteristics, the services they receive, and the results of those services. For example, children who receive the generally time-limited evidence-based practices suggested in the PEI guidelines may not be MHSOAC Evaluation Master Plan Page 30 entered into the treatment data system but might be entered into a separate PEI data base or be part of an EBP developer’s data system.15 The sum of the individual outcomes on any particular outcome becomes a measure of the success of a program or a county or the state on that outcome. So, for example, a program or a county or a state could be evaluated by the percentage of its clients who are employed or who are in independent living. The critical point is that the outcome is based on what happens to individuals, so for the purposes of this paradigm they are in the “individual” level. Community Level This has been the most slippery of levels to define. For some this is just another term for what is called the system level. The meaning as used here is different; it refers to a population based level of analysis rather than the analysis of those who are served. It includes outcomes for population-based groups and not just those who received a service. It is included with this meaning for a number of reasons: The goals of the MHSA extend beyond improving the service system. The shift from treatment to prevention reflects the shift from a “fail first” to a “help first” philosophy. The reduction in stigma and discrimination extends beyond the service system to the general community. The reduction in the gap between the need for help and the provision of help can be addressed by others in the community who may be reached through the MHSA messages. Some policy makers believe that the public mental health system will not be able to provide services and supports to all those with a serious mental health disorder. Many such persons now receive help from natural supports, and some of the MHSA Innovation projects are intended to expand and strengthen these environmental supports. With the shift to a more integrated health care system more persons may receive mental health services through a health care provider than the specialty mental health system. This could create the paradox of specialty mental health penetration rates going down while community based data on “receiving help” could go up. It is important to track this community level trend to correctly interpret indicators of the use of specialty mental health services. Tracking the use of services from any source at the community level is another critical way of tracking access for those with barriers to care. It could be that more persons from racial/ethnic and cultural groups receive some service outside the public mental health system which would be a positive trend. Attention to a population-based approach was highlighted by RAND in its Prevention and Early Intervention Framework, 2012. RAND noted that the overall effectiveness of the efforts undertaken by the MHSA could only be measured through a population-based strategy, although it recognized the difficulty in attributing any changes in outcomes specifically to the MHSA. 15 For example, CiMH sponsored EBP community collaboratives for children/youth include a component of data collection and Los Angeles is implementing a full range of EBP for its PEI component with a separate data base. MHSOAC Evaluation Master Plan Page 31 There are three populations to consider at the community level. One is the general population which is used to determine the prevalence of mental illness and mental health problems. The second is the population which has a serious mental illness (SMI) or a serious emotional disturbance (SED), i.e. all of these individuals not just those who are receiving services. And the third is those who are at risk of a SMI or SED. These three populations can be identified through surveys that sample the general population. For the latter two of these populations (those with an SMI/SED and those at risk) the outcomes of interest are the seven negative outcomes in the MHSA PEI section and the receipt of some service or support for the mental health problem. How this can be done is described more fully later in this document (see Developmental and Exploratory Work #5). Evaluation Questions The basic evaluation questions are straightforward. One could ask innumerable evaluation questions about all the items in the diagram. For the purpose of gaining an overall view of the Evaluation Master Plan it is useful to consider the most important general evaluation questions. Evaluation questions most often are about outcomes, but there are some questions which can be asked about the inputs. This is particularly the case for the local planning process which is critical to the transformation sought by the MHSA.16 The second type of evaluation question about the inputs relates to the tracking of how the MHSA funds have been expended. So, the evaluation questions related to inputs are as follows: Has the local stakeholder process been effective? Has the MHSA money be spent as intended?17 For the system level the basic evaluation questions are as follows: Has the mental health system improved in terms of access, quality, efficiency and satisfaction? Has the infrastructure (technology, workforce, housing alternatives) improved? Have the values and principles of the MHSA been incorporated into policy and practice? Are more people being served (penetration rates)? Have the disparities in amount and type of services been reduced? 16 The local planning process is different from the principles and values in not having a directly measurable result at the system level. The input of a value such as recovery can be measured by assessing the recovery orientation of programs while the local planning process input would affect the whole system and its results not be so easily measured specifically. 17 This question has been treated as an evaluation issue by the MHSOAC but could also be viewed as more of a monitoring and accountability issue. How the responsibility for answering this question is apportioned within the MHSOAC is an internal issue. MHSOAC Evaluation Master Plan Page 32 For the individual level there is one basic evaluation question: Are person served / family members doing better? Are persons served / family members satisfied with services? At the community level there are four basic evaluation questions: Has the prevalence of mental illness been reduced? Have the negative outcomes for those with a SMI/SED and/or those at risk been reduced? Have the proportion of persons with an SMI/SED or those at-risk who receive services and/or natural supports increased? Have stigma and discrimination been reduced? All of the evaluation activities included in the Evaluation Master Plan relate back to at least one of these basic evaluation questions. While the major focus of the Evaluation Master Plan is on the MHSA, the scope of the plan is broader. There are two major ways in which the scope of this Plan extends beyond the MHSA. The intention of the MHSA was to make a fundamental change in the way the mental health system operates, i.e. to transform the system and not just increase the amount of “business as usual.” The MHSA impact extends beyond funding specific MHSA programs; the goal is to imbue the whole system with MHSA values and principles. Creating another funding silo would be contrary to the goals of the MHSA which was to integrate the MHSA into the system of care. To do justice to the goals of the MHSA the whole system and not just MHSA-funded efforts must be included in evaluation efforts. A second reason is that the MHSA explicitly mandates the MHSOAC to consider in its oversight and accountability the outcomes described in statute describing the adult and children’s systems of care and to consider these systems of care in its evaluations. The above does NOT minimize the requirement for the MHSOAC to monitor the use of the MHSA funds and the outcomes associated with the use of those funds, particularly with regard to the specific components. How the money is spent and whether it is spent in accord with the statute, regulations, and guidelines needs to be tracked. But the scope of the evaluation cannot be limited to just what is directly funded by the MHSA. The Evaluation Master Plan views the MHSA and the mental health system as an integrated whole while also including items specific to the MHSA components. As noted above, the MHSA was intended to support a continuum of activities from prevention through early intervention through treatment and to integrate the philosophy and activities supported by the MHSA funds into an integrated system. Thus the system and community outcomes are relevant to the whole system of care including PEI and CSS as well as to the infrastructure elements. Most of the individual outcomes will be the same for persons receiving CSS, early intervention, and selective prevention services. MHSOAC Evaluation Master Plan Page 33 At the same time some elements of the MHSA components raise specific evaluation issues which require a more focused look. Thus the Evaluation Master Plan also includes evaluation activities that are specific to particular MHSA components. MHSOAC Evaluation Master Plan Page 34 PART 3: ORGANIZATION OF EVALUATION ACTIVITIES AND PRIORITY SETTING The evaluation activities are organized by three evaluation methods. One of the confusions about evaluation is the diversity of methods that can be used in answering an evaluation question. The evaluation methods used need to be understandable to people so that they know what questions the evaluation is able to answer and what limitations or cautions are needed in interpreting the results. Understanding the evaluation methods is important enough to serve as the organizing principle for the evaluation activities in this Plan. Hopefully this can eliminate misunderstandings about what the evaluation activities can tell us and foster an understanding of the scope of what is in the Plan. The three methods are as follows: Performance Monitoring Evaluation Studies Developmental and Exploratory Work Efforts The MHSOAC has already used all three of these methods so that the activities within the Plan build upon prior work Three Evaluation Methods Performance Monitoring Performance monitoring is a common approach to assuring accountability and facilitating quality improvement. The process involves identifying a desired result which can be a process, a structure, or an outcome. Then an indicator of that result that can be measured is identified. A good indicator has a way of measuring it that is scientifically sound (reliable), is valid and meaningful (really measures what is the intended result), is feasible (has a data source which is accurate and consistent), and is useable (provides information which can be used to improve quality). A measurement of an indicator will define clearly what is being measured for what population by what data source for what period of time. A performance measurement indicator is usually presented as a proportions or rate, with the number achieving the desired result or outcome as the numerator and the population being measured as the denominator. The National Committee for Quality Assurance (NCQA) defines a performance measure as “a set of technical specifications that define how to calculate a ‘rate’ for some important indicator of quality.” A performance measurement is applied to some population or some entity. The population could be everyone in a program, or everyone in all the programs in a county, or the population covered MHSOAC Evaluation Master Plan Page 35 by a payer, or the population of a county under 200% of poverty. Examples would be percent homeless (of those in a particular FSP or in all the FSP programs in a county), penetration rates (percent of those receiving a mental health service of those who have Medi-Cal or those under200% of poverty). Some commonly used performance indicators refer to process characteristics of a program or administrative entity. Examples are contract requirements for the number to be served or wait times until a first appointment or a clinician productivity rate or the number of a health plan’s members who are screened for a substance use issue. Another attribute of a performance measure is the time period over which it is measured. The most common is at a particular point in time, e.g. the Consumer Perception Survey (CPS) is completed at a point in time and so the characteristics of the population surveyed are for that point in time. The other commonly used method is to calculate an event over a particular period of time, often annually; for example, one could measure the demographic characteristics of all the clients newly entering the system over a one year period. Systems of performance measures in health care most often use administrative or claims data that are readily available. The ones of most importance to the MHSOAC efforts include the CSI, DCR, CPS, and approved Short-Doyle/Medi-Cal claims. Some systems, e.g. the NCQA Health Effectiveness Data and Information Set (HEDIS), rely on surveyors who might review chart abstracts. Performance measurement systems are used differently depending on the user. An indicator can be used for compliance, i.e. in a performance contract. For MHSOAC, a performance monitoring system allows a view of the mental health system on important indicators of quality of performance and outcomes over time and across entities. Rates of employment, for example, for those in the mental health system can be compared across years or between counties. While any such comparison must be made with extreme caution, ensuring ample provision of context, it does allow for raising questions about what might be responsible for differences over time or among counties. Programs managers and county administrators can use the comparison data for motivational purposes, and it can lead to the identification of programs/counties which are good performers and to the exploration of what seems to make a difference. Once data is trusted by everyone in the system, benchmarks or goals for performance measures can be used as additional motivation. Thus the data can be very useful for quality improvement efforts with no compliance component involved. Evaluation Studies Evaluation studies are what is most commonly understood as “evaluation” and is strictly speaking the most accurate use of the term. An evaluation study measures the results (effectiveness or efficiency) of a particular intervention. The intervention can be a program or an element of a program, a process, an initiative, or the implementation of a value. The intervention can be narrow or broad, e.g. the addition of an employment specialist to a team or the introduction of a welcoming initiative in a county. The better specified the intervention the more useful MHSOAC Evaluation Master Plan Page 36 the evaluation will be. If the results of an intervention are positive the ability to replicate the results can only be done reliably if the intervention has been specified well enough to duplicate it. There is often a tradeoff in deciding whether to evaluate a well specified narrow intervention or a potentially more meaningful and far reaching less well specified intervention. An evaluation can be qualitative or quantitative or as is often the case a mix of the two. The methodology of evaluation studies can vary from what would be termed a rigorous research design to a fairly simple program evaluation design. While the MHSOAC should attempt to obtain as much rigor in evaluation design as possible, the level does not have to be what would be required for a study to be published in a peer reviewed journal. Randomized control study design, which is the gold standard for rigorous research studies, is often problematic in the public mental health system, but should always be the first choice where it is possible. When it is not feasible; alternatives need to be considered. For example, the evaluation used to substantiate the positive results of the initial AB 34 pilot programs for homeless persons with a mental illness used a simple pre and post design comparing the 12-month period prior to enrollment in the pilot program to the 12 months after enrollment. There were no control groups and no substantiation of the memories of the clients about the pre-enrollment data. Despite the lack of rigor the positive pre to post enrollment results were important in providing support for the funding extension. Evidence-based practices are developed through the use of evaluation studies. The standards for an evidence-based practice vary by the entity listing such practices, but they all assume at least well conducted evaluation studies. This is something to consider as evaluation methodologies are being designed. Developmental and Exploratory Work Efforts Some evaluation questions that are of considerable interest don’t lend themselves to either of the two evaluation methods already described. These are generally issues which if studied would help in understanding something about the mental health system and/or would be useful in subsequent performance monitoring or evaluation work. For example, we know that FSPs, while meeting regulations and guidelines, vary considerably in their level and kind of staffing as well as in the kinds of consumers they serve. Comparisons of the effectiveness or efficiency of FSPs (an evaluation study) would need a reliable way to classify FSP programs. Determining whether such a classification that would be both meaningful to staff and empirically reliable could be devised would be developmental work in preparation for a possible evaluation study. Another developmental effort would be to determine whether FSP outcomes differ depending on demographic or clinical characteristics of the persons served. This would be a first step to a possible future effort to risk adjust ta program’ clients when comparing the outcomes of FSPs. Another example would be to the exploration of how best to assess the recovery-orientation of programs. MHSOAC Evaluation Master Plan Page 37 Priority Setting The Evaluation Master Plan classifies evaluation activities into high and medium priorities based on a set of criteria. There are obviously more possible evaluation activities than can be accomplished by the MHSOAC over the next 3-5 years. An initial set of recommended activities is established in the Plan based on a set of evaluation criteria. Any set of criteria is subject to argument, and the criteria used here have not been previously approved by the MHSOAC. They were developed based on the values and priorities in the MHSOAC prior documents on accountability and evaluation. Most importantly the ratings on these criteria are clearly subjective and open to challenge. There was no absolute cutoff between the high and medium rankings. The distinction between high and medium was used primarily in the setting of a recommended order in which the MHSOAC would conduct the various activities. Criteria have been used that relate to the evaluation questions and to the evaluation activity18. The criteria applied to the evaluation questions are as follows: Consistency with MHSA: Are the questions consistent with the language and values of the Act? Potential for quality improvement: Will answers to the question(s) lead to suggestions for and implementation of policy and practice changes? Importance to stakeholders: Are the questions a high priority to key stakeholders? Possibility of partners: Are there other organizations that might collaborate and/or partially fund the activity? Forward looking: Are the question(s) relevant to the evolving health care environment? Challenges: Do the question(s) address an area which is creating a challenge for the system? The criteria for the evaluation activity are the following: Feasibility: How likely is the evaluation activity to produce information that answers the evaluation question(s)? Cost: How many resources are needed to do the activity well? Timeliness: How long will it take to complete the evaluation activity? Leveraging: does the evaluation activity build upon prior work of the MHSOAC or others? These criteria can be used by the MHSOAC to reprioritize the evaluation activities and/or to review additional evaluation activities now or in the future. The MHSOAC will want to reconsider the order of activities and add and revise the list over the timeline of the plan in response to environmental events. The MHSOAC can use this set of criteria (with whatever modifications it makes to them) in an orderly way to revise priorities over the next few years. 18 The term “evaluation activity” as used here and throughout the document refers to any of the Performance Monitoring steps, the specific Evaluation Studies, and the specific Developmental and Exploratory Work Efforts. MHSOAC Evaluation Master Plan Page 38 Obtaining full consensus on priority setting is difficult no matter how much discussion and how elaborate a rating system is used. A simple rating system was used for the priority setting in the Plan; a score of 3 was assigned to the highest rating on the criteria, a 2 to a moderate rating, and a 1 to the lowest rating, and the sores were then summed across the criteria. This method does not weight the criteria. MHSOAC Evaluation Master Plan Page 39 PART 4: EVALUATION ACTIVITIES Evaluation Method 1: Performance Monitoring Background and Context As noted in the Findings section, the efforts of the MHSOAC to develop a system for tracking performance indicators is consistent with the national effort to use such measures to assess and compare the quality of care provided by health care organizations, health plans, and health systems. This section briefly notes a few major developments at the national and state levels, reviews the work already undertaken and planned by the MHSOAC, and then outlines the next steps in the Master Plan for this evaluation method. National and state performance monitoring systems The performance monitoring system begun by the MHSOAC is aligned with the growing use of such systems in the health care field. A thorough compilation of all the efforts in the area of performance measures and indicators is beyond the scope of this report. The intention here is to illustrate how widespread the use of performance monitoring has become and how the field of behavioral health is slowly being introduced to this evaluation and accountability method. National Committee for Quality Assurance (NCQA): The NCQA is one of the best known entities in the field of performance indicators in the health field. The NCQA is a private non-profit association which accredits and certifies health care organizations. The NCQA developed the Healthcare Effectiveness Data and Information Set (HEDIS) which consists of 75 measures over eight domains. Measurement procedures are detailed and complex; the NCQA certifies auditors who conduct the reviews for certification. Over 90% of the country’s health plans seek accreditation by the system. There is a formal and methodical process for adding additional indicators to the set; criteria are relevance, scientific soundness, and feasibility. The NCQA website says “NCQA did not create the field of health care performance measurement, but we have refined the process and led the development of objective measures since the mid-1990s when our standardized measurement tool, the Healthcare Effectiveness Data and Information Set (HEDIS), was broadly adopted by the industry.” “Public reporting of performance data holds health care providers accountable to both consumers and purchasers of care; transparency builds trust….HEDIS allows for standardized measurement, standardized reporting and accurate, objective side-by-side comparisons.” National Quality Forum (NQF): The National Quality Forum (NQF) is a voluntary consensus standard- setting organization that lists measures sponsored by numerous other public and private entities. The NQF is playing a role in the Affordable Care Act’s (ACA’s) national quality improvement strategy. It has established a The Measure Applications Partnership (MAP) as a public-private partnership of 52 MHSOAC Evaluation Master Plan Page 40 organizations to provide input to the U.S. Department of Health and Human Services (HHS) on performance measures for public reporting, performance-based payment programs, and other purposes. The Health and Human Services Department has contracted with NQF as a “consensus-based entity” to “convene multi-stakeholder groups to provide input on the selection of quality measures” for various uses.19 More behavioral health items are being added to the national health care sets of indicators. The health care performance measurement systems originally contained few behavioral health indicators. This has changed with the growing awareness of the overall health care cost implications of covering persons with behavioral health problems. The current list for HEDIS 13 (for 2013), for example, includes nine behavioral health items, some appearing for the first time. Examples of the indicators included arei: Percent of persons with schizophrenia or bipolar who are using antipsychotic medications who are screened for diabetes Percent adhering to antipsychotic medications for individuals with schizophrenia Mental health utilization There are an additional 20 or so Phase 1 behavioral health measures being considered for adoption by NCQA for future inclusion in HEDIS. The NCQA has established a shorter set of standards for new Accountable Care Organizations (ACOs). These include four behavioral health indicators: Antidepressant medication management Follow-up care for children prescribed Attention Deficit Hyperactivity Disorder medications Follow-up after hospitalization for mental illness Initiation and engagement of alcohol and other drug dependence treatment The NQF MAP project has agreed on 11 behavioral health indicators to recommend as a result of a detailed review of 22 measures. Most of the indicators are ones sponsored by NCQA and are already included in HEDIS 13. Examples of the other indicators included in the 11 measuresii are: Medical assistance with smoking and tobacco use cessation Tobacco use screening Tobacco use cessation MAP has also recommended the following behavioral health indicators for inclusion in the Dual Eligible (Medicaid and Medicare) pilots being conducted by selected states, including California. Alcohol screening and intervention Depression screening 19 From www.nationalquality.org MHSOAC Evaluation Master Plan Page 41 Substance use treatment Tobacco use screening and cessation The Substance Abuse and Mental Health Services Administration (SAMHSA) plays a role in establishing indicators for publicly funded behavioral health care systems. States are required by SAMHSA to submit data on their overall state system to the National Outcomes Monitoring System (NOMS) in order to receive federal mental health and substance abuse block grants. Some of the items originated with the Mental Health Statistics Improvement Program (MHSIP) which is included in the California Consumer Perception Survey (CPS). The NOMS contains an extensive list of indicators including the following: Penetration rates per 1000 residents Living situation Use of SAMHSA certified evidence based practices for adults and children Social connectedness Various consumer perceptions of care (from the MHSIP) Outcomes from services (from the MHSIP) This data collection system allows for comparisons across states on these indicators. These comparisons are at best suggestive given the diversity in definitions and reliability of data collection across the states. SAMHSA is creating a National Behavioral Health Quality Framework (NBHQF).The aims of the SAMHSA Quality Framework are better care, healthy people/healthy communities, and affordable care. The quality strategy established six priorities to help focus efforts by public and private partners: patient safety, patient and family engagement, care coordination, effective prevention and treatment practices, population health, and improving affordability of care. SAMHSA is in the process of creating a Behavioral Health Barometer which will highlight key behavioral health indicators. It is intended to provide information at three levels: grantees, behavioral health systems, and the general population or subpopulations. Other states use performance monitoring systems for varying reasons. Oklahoma has a centralized public mental health system with five state-operated and 10 contracted non-profit community mental health centers serving those on Medicaid and the uninsured. Subsequent to approval of a Medicaid Plan Amendment the state began in 2008 using a portion of Medicaid dollars as a supplemental payment for programs meeting certain benchmarks. The system started with six, then expanded to 12 standards. Eleven of the measures were tracked based on existing Medicaid claims data while a 12th (access to treatment) was measured using a secret shopper approach. The state uses a Client Assessment Record as a required tool that measures client functioning in nine domains. Payments (calibrated by the volume of services) were made to providers who met benchmarks, and additional bonus payments were made to those who exceeded the benchmark by at least one standard MHSOAC Evaluation Master Plan Page 42 deviation. Overall percentages increased on all of the measures. Increases were modest for most indicators, but some showed dramatic increases, e.g. the percentage of persons with a non- crisis outpatient service follow-up within eight days of a crisis outpatient visit rose from 30% in July 2008 to 80% in June 2010. New York has a Balanced Scorecard which presents information on achievement of target goals on a series of management and outcome measures. The items fall into three overall domains – services, outcomes, and system management – and vary in their coverage – just state operated programs or all programs. Some of the indicators are common ones, e.g. rates of hospital readmissions, rates of hospitalization while in an ACT program, rates of employment for those receiving services, rates of seclusions and restraints in state operated inpatient settings, and rates of satisfaction with services. Others have not been discussed in California, i.e. rates of completed suicides during inpatient stay or within 72 hours after discharge for all inpatient facilities. Others reflect specific management objectives, i.e. the number of programs with current licenses, numbers of state programs with appropriate accreditation, percent occupancy in supported housing units, and the number of new Personalized Recovery Oriented Services programs. New York also has a Dashboard that shows county level data on a set of demographic, service use, Medicaid expenditures, medication indicators, psychiatric hospital readmissions, and wellness and community integration. Data is for all programs licensed or funded by the state. Data comes from a variety of sources, e.g. a Patient Characteristic Survey done for one week every two years. Again, most of the measures tracked are common ones, e.g. rates of homelessness at last update, rates in competitive employment, but others are not currently collected in California, i.e. rates of chronic medical conditions, percentage of adults on four or more medications for more than 90 days. Ohio has traditionally had a robust state level evaluation and research unit. Fiscal pressures in recent years plus push back from providers led to the dismantling in 2009 of the extensive outcome data collection system. A new system is currently being piloted with voluntary participation of the 50 local authorities. The system is designed essentially to obtain information necessary for NOMS reporting, but the state intends to work with stakeholders to add selected health care items. The system will have a web-based data entry portal and also the capacity for batch uploads. Data will be collected at entry and annually or at discharge. Data will be available on the standard measures and is being promoted to the authorities as a useful tool for quality improvement. Arizona has a dashboard that displays indicators in four domains – outcomes, access, service delivery, and coordination/collaboration. Data is collected from a variety of sources including routine demographic data at admission, individual and family survey data, analysis of claims data, audits of client records, and data provided by the Regional Behavioral Health Authorities (RHBAs). The data is displayed on line with historical trends and with the ability to drill down to specific outcomes or geographical regions. The outcome data elements are the ones required by NOMS and are calculated based on data entered into a client information system which gathers information at admission and then at annual updates or at discharge. For outcomes, the system shows both current rates and also MHSOAC Evaluation Master Plan Page 43 differences between admission and update/discharge. Access items are measured using a survey methodology similar to the CPS. For system measures rely on data submitted by the RHBAs. MHSOAC Prior and Approved Work. MHSOAC initiated a process to obtain data on 12 priority indicators adopted by both the CMHPC and the MHSOAC. The California Mental Health Planning Council (CMHPC) has the statutory authority to approve performance indicators for the state’s public mental health system (WIC 5772(a)(1). CMHPC adopted a set of indicators in January, 2010. The indicators reflected desired outcome areas for individuals, the system, and the community. Some indicators were stated in terms of rates or proportions or numbers (e.g. number of arrests, average school attendance) while others were stated more generally as outcomes or as data that could be interpreted to determine if desired outcomes were achieved (e.g. demographic profile of new clients served). A smaller set of indicators (four person-level and eight system level) was adopted as the highest priority ones by both CMHPC and MHSOAC. The MHSOAC contracted with the UCLA Center for Healthier Children, Youth and Families with EMT Associates as a subcontractor to identify how to measure these priority indicators and to produce a series of reports that included data on the indicators. The Initial Priority Indicator Report was submitted in the fall of 2012. It contains measurements of the 12 priority indicators for FY 08-09 and FY 09-10. The indicators, service populations, and data sources are shown in the table below. Priority Indicators Included in Initial Priority Indicator Report (2012) Indicator Service Data Sources Population INDIVIDUAL LEVEL Average school attendance per year All/FSP CPS, DCR Proportion of consumers employed All/FSP CSI, DCR Homelessness and housing rates All/FSP CSI, DCR Arrest Rate All/FSP CPS, DCR SYSTEM LEVEL Demographic profile of consumers served All/FSP CSI, DCR Demographic profile of new consumers All/FSP CSI, DCR Penetration of mental health services All CSI, Holzer estimates Access to a primary care physician FSP DCR Perceptions of access to services All CPS Involuntary status All DMH Report Consumer well-being All CPS Satisfaction with services All CPS MHSOAC Evaluation Master Plan Page 44 Some of the indicators were measured on everyone served in public mental health programs (or a sample of everyone) and some only on those in FSPs; this decision was based largely on data availability. The main state data sources used for the two service populations were as follows: For the entire population served: the Client and Service Information (CSI) and the Consumer Perception Survey (CPS) which is conducted annually on a sample of persons enrolled in the public mental health system; and, For those enrolled in FSPs: the Data Collection and Reporting (DCR) system. County level data was compiled on each indicator in a series of supplemental reports. The conclusion from the Initial Priority Indicator Report is that the existing data sources can provide reasonable information about most of the indicators, but there are significant problems. A review of the contractor’s analysis and recommendations leads to the following conclusions. For seven of the indicators, existing data sources can provide information which can be considered reasonably accurate while for the remaining five a lack of clear data element definitions, a lack of data completeness, and a lack of timeliness in data reporting seriously jeopardize the results. The original set of priority indicators needs to be revisited. o For a few indicators the data elements either do not meaningfully measure the intended outcome or the data is not sufficiently reliable so that an alternative indicator may be needed o Efforts to make sense of the indicators by using comparisons over time or across entities has not been explored enough to know which indicators will prove to be useful o Stakeholders recommended additional indicators and/or measurement strategies o Changes in the health care environment have heightened the importance of indicators not originally considered The MHSOAC contract with the UCLA Center for Healthier Children, Youth and Families provides for additional work on the priority indicators. A report on the priority indicators for FY 04-05 and FY 05-06 will be completed in the spring of 2013. Because this predates the MHSA there will be no reporting on any indicator in the DCR. Thus the data will be on the full population of mental health consumers with data largely from the CSI and the CPS. This data will allow for comparisons between pre-MHSA and post-MHSA time periods. This should allow analysis of results on the system level indicators which tend to be more reliable and valid at this point than the individual level ones. Two more reports on the set of priority indicators will be completed as part of this contract – one to add FY 10-11 (due 9/30/13) and the final one to add FY 11-12 (due 3/31/14). Revised and/or new indicators might be added to these reports if prior developmental work has been completed. MHSOAC Evaluation Master Plan Page 45 Continuing the refinement and development of the Performance Monitoring system is a high priority in the Evaluation Master Plan. The following describes the recommended process for building off of and further developing the performance monitoring system. It is organized in terms of the progression of steps. Some could be done internally by MHSOAC, by DHCS staff, by inclusion in the subsequent work being carried out by the existing contractor, but some may require either contract amendments, additional contracts potentially with others, and/or additional internal MHSOAC evaluation staff. The following list of tasks is built on a set of assumptions that applies to the whole process. The following steps assume the following: The MHSOAC will meet with DHCS to discuss roles regarding compilation of the data after the conclusion of the current contract with the UCLA Center for Healthier Children, Youth, and Families. The MHSOAC will collaborate with other organizations as it refines and/or eliminates some indicators and adds others. Obtaining agreement from the CMHPC is required, and working with others to develop consensus broadens the knowledge and understanding of the performance monitoring system. Other current efforts to work towards agreement on the most important outcomes may provide a forum for at least part of this work. The MHSOAC will proceed cautiously in adding indicators. Given the large number of possible indicators, stakeholders will urge adoption of many that are important to them. It is critical to the credibility of the performance monitoring system that the current set of indicators be “cleaned up” first and that new ones be added only after pilot testing demonstrates that the indicator is reliable and meaningful. The MHSOAC will place considerable weight in making decisions about indicators on the usefulness of the resulting data. As will be recommended later, the MHSOAC staff should produce analysis of the indicator reports and draw policy and practice implications for the Commission. Note: The ordering of the steps outlined below should not be viewed as rigid but as a suggestion for a logical ordering. Some steps can be done concurrently. STEP 1: REVISIT, CLARIFY, AND/OR REVISE EXISTING PRIORITY INDICATORS The original set of 12 priority indicators needs to be reviewed and revised. The initial effort illuminated some of the issues with the data sources and with the relationship between what outcome the indicator sought to measure and what the indicator actually measured. The contractors assessed the quality of the data for each indicator on the dimensions of availability, completeness, sustainability, relevance, availability for multiple years, and ability to be analyzed at multiple levels. The initial 12 indicators need to be reexamined in light of the data and measurement issues that emerged, and a start needs to be made on trying to interpret and use the information. MHSOAC Evaluation Master Plan Page 46 The individual level indicators are the most problematic. There were four individual level outcome goals stated in the MHSOAC May 10, 2011 Initial Priority Outcomes and Indicators document: increase educational progress, increase employment, improve housing situation, and reduce justice involvement. The specific indicators were best guesses at ways to measure the achievement of these desired outcomes. The UCLA Priority Indicators Report used those specific indicators as well as exploring others designed to measure the same general outcome. Feedback from stakeholders suggested other indicators for measuring these four individual level outcomes. Two kinds of measurement issues arose with the individual level indicators: updating and relevance/reliability. Updating. The data elements assessed at a consumer’s entry into the system are usually entered into CSI and for FSPs clients also into the DCR system. Both data systems have procedures for updating the status of the client while s/he is receiving services. The CSI data is supposed to be updated annually and/or when the client leaves the system. The DCR has updates on some elements when a change in status occurs (Key Event Tracking or KET) and on other items quarterly (3M). The updates are not uniformly completed, particularly with the CSI system. The individual indicators should be measured on clients after they have been in the system so we can, at least theoretically, track the influence of the services provided.20 Using data on clients when they enter the system tells us only about the initial status of clients. All of the client-level indicators in the initial report that used only data at admission will need to be reviewed. A first step will be to accurately assess the percentages of completed follow-ups that are available for each indicator. Depending on these results additional effort will need to be devoted to obtaining more complete updates going forward and/or the indicators will need to be altered.21 Relevance and reliability. The original individual level indicators were selected as hopefully reliable and valid ways of measuring the intended outcomes. While this sometimes worked, it did not always. The four individual outcome domains are briefly reviewed below on these dimensions. Increase educational progress. The indicator is “increase in number of days in school.” Since no data sources contain this specific data element, two substitute measurements were used by UCLA. A reasonable substitute was found for those in an FSP using DCR data element consisting of a staff rating on how frequently the child attends school with five choices ranging from “always attends” to “never 20 The original California Mental Health Planning Council document “Performance Indicators for Evaluating The Mental Health System” suggested that the individual-level outcomes for FSP members be tracked as change scores over time on the same individuals. The subsequent MHSOAC approved list of Initial Priority Outcomes and Indicators (May 10, 2011) did not specify whether the measurements would be a point in time or measured over time on the same individuals. The complexity of the task as well as the inadequacies of the data systems resulted in the priority indicators being measured at a point in time only. Thus the current performance indicators do not track change over time on individual clients. They rather track the status of individuals at a point in time over time or across entities (counties or programs). Step 6 returns to the original concept and explores including such change measures as indicators. 21 There is general consensus on the outcome domains so that the issue is fundamentally one of finding the best metric and not of needing to alter the desired outcomes. MHSOAC Evaluation Master Plan Page 47 attends.” This is a reasonable metric for FSP children/youth if the above cited issue of lack of follow-up data were corrected. The second substitute measure comes from the CPS which asks how many times the child/youth was expelled or suspended in the 12 months prior to starting services and in the time since receiving services. While this is a meaningful and technically adequate measure, it does not capture the full meaning of school attendance since truancy is likely as large a problem as expulsion and suspension. Increase employment. The indicator was straight forward in measuring the proportion of consumers participating in employment, but the reliability of the data in the CSI in particular was questionable. Reduce justice involvement. The suggested indicator was “number of arrests.” UCLA used arrest data from both the CPS and CSI. The arrest data proved to be problematic, and stakeholders suggested that incarceration would be in any case a better indicator of justice involvement than arrests. The FSP Cost and Cost Offset report indicated significant savings from pre to post in days of incarceration. Days incarcerated for those in an FSP is likely a better indicator for justice involvement than number of arrests. Improve housing situation. Two suggested indicators were days homeless and independence in residential setting. An indicator measuring homelessness is clearly needed, but how to measure this most accurately remains an issue, particularly with the CSI data. The classification of housing situation alternatives on the CSI and the DCR needs further analysis to better define the indicator of independence of housing. Obtaining reliable, relevant, updated data for indicators at the individual level may be restricted for the near future to FSP enrollees. The initial CMHPC individual level indicators were designed for use with FSP enrollees only. A more intensive level of services and a more extensive data collection system made this limitation seem reasonable. The MHSOAC Initial Priority Outcomes and Indicators also limited the individual indicators to persons in FSPs. Efforts to extend the indicators to the broader population of persons enrolled in any mental health service may not be feasible with the current limitations in the CSI system, particularly as it relates to updating. An expansion of the types of questions on the CPS might be an alternative means for generating appropriate indicators for the four individual level priority outcome areas. The measurement issues for the system level indicators are less serious but still need attention. For most of the system level indicators, e.g. the demographic characteristics of the clients in the system or the penetration rate, data from the admission forms is adequate so the updating issue is not relevant.22 The challenge for the system level indicators is to deal with changes in measurement methods. 22 The one exception is “access to primary care” for FSP clients. MHSOAC Evaluation Master Plan Page 48 One of the chief uses of the performance monitoring data is to compare performance over time. To do this meaningfully, the ways that the data is collected and analyzed must remain consistent. Two areas in changes in methodologies require attention. California in 2006 changed the collection of race and ethnicity data in CSI to align with federal requirements. Concerns remain about the accuracy of current reporting despite considerable training by the state DMH. The contractors dealt with the issue by translating the new reporting system into the pre-2006 system. While this is a reasonable strategy particularly for upcoming comparisons with 2004 and 2005 data it is not sustainable for the future. The other change is the sampling methodology for the CPS. A convenience sample was used prior to FY 09-10; i.e. the survey was completed by everyone who received a service during a set period of time. SAMHSA now requires a random sampling method, and in FY 09-10 California utilized such a method. The survey for FY 11-12 was administered under the direction of CiMH with a return to a convenience sample. Until a decision is made on how the sampling will be done in the future, comparisons across time will be problematic. The review of the existing priority indicators should also consider how the information can be analyzed and interpreted. The indicators monitor performance in order to enhance quality improvement efforts. The results need analysis and interpretation to raise questions, to call attention to selected areas of interest or concern, and to recommend changes in policy and/or practice. The analysis and interpretation of the performance monitoring results relies on comparing indicators over time, comparing indicators across counties, and comparing indicators where possible with other states or national data. All of these analyses and interpretations must be done cautiously and with appropriate attention to the context. But if done with caution such analysis and interpretation can be useful. Without it there is little utility to gathering the data in the first place. This should not be considered a definitive effort at this point because of the uncertainty about the data. It would be an exercise in “what if,” i.e. what might we say about the results if we had confidence in the data. For example, if the rates of employment among adult consumers in California were significantly lower than in the rest of the nation what recommendations might follow? Or, if consumer satisfaction was significantly lower in selected counties what questions for additional analysis might that raise? Or, if some counties showed higher rates of new clients among underserved racial/ethnic and cultural groups what policy or practice recommendations might follow? The process of analysis and interpretation should be helped by the addition of the FY 04-05 and FY 05-06 data in the report due from the contractors in the Spring of 2013. Drawing conclusions about changes from this earlier period to the data in the recent report for FY 09-10 and FY 10-11 should further illustrate the potential usefulness of monitoring the performance indicators. MHSOAC Evaluation Master Plan Page 49 This review and revision of the 12 existing priority indicators should be done in collaboration with DHCS and the CMHPC. MHSOAC staff should be heavily involved in the details of this review with the UCLA contractors since the staff has the context for how the information will ultimately be used. It is also strongly recommended that representatives of DHCS, who are the keepers of the major data sources, be actively involved in this process since the Master Plan relies on their being the ones who provide the data to the MHSOAC after the conclusion of the current UCLA contract. The CMHPC should also be involved given their statutory responsibility for approving indicators. The recommendations for change in the initial indicators should be reviewed by the Evaluation Committee. STEP 2: DEVELOP A PROCESS FOR ADDING OTHER INDICATORS A formal process for considering new indicators will maintain the integrity of the performance monitoring system. Suggestions about additional indicators are likely to be extensive, coming from multiple stakeholders. Having a clear process for evaluating suggestions will keep the process organized and allow a thoughtful gradual increase in the number of indicators while minimizing false starts and unrealistic expectations. The following process is only a suggestion; the actual process will need to be created by the MHSOAC staff based on its organization and resources. Two critical parts of the process should be the use of piloting and having others involved (in an advisory role) in the process. Piloting an indicator means collecting the data element on a few counties before moving to statewide implementation. The experience with the initial report on the performance indicators suggests that substantial work is necessary before an accurate and relevant metric can be established for a particular indicator. Working closely with the pilot counties facilitates the discovery of problems and possible solutions with the data elements that are being measured. It is therefore recommended that work be done to pilot indicators before they are officially presented to the CMHPC and the MHSOAC for approval. Having others involved in an advisory role will help in determining the validity of the indicator. Multiple options exist for how to involve others: a permanent subgroup of experts to advise on all parts of this process, subgroups of experts to review particular suggested indicators, or review of suggested indicators by the MHSOAC Evaluation Committee. Having others comment on suggested indicators and measurement strategies should assist not only in obtaining reliable data but also in what the implications of the data might be. One possible process is outlined here. Staff suggests a list (perhaps annually) of potential new indicators after considering ideas of diverse stakeholders and advice of expert group(s), and/or contractor Evaluation Committee reviews and comments on the list Piloting of indicators is conducted MHSOAC Evaluation Master Plan Page 50 Results of pilots are analyzed by staff and recommendations made about what to add to the formal performance monitoring system Evaluation Committee reviews recommendations Staff recommendations reviewed by Commissioners and tentatively approve subject to approval by the CMHPC CMHPC reviews and approves Indicators added to the Performance Monitoring system Each of the following steps (#3 - #6) involves a different type or source of possible new indicators. The order in which they are presented is only suggestive, based on current priorities. Some of the steps can be done concurrently. STEP 3: INCORPORATE INDICATORS FROM OTHER WORK GROUPS The work product of other consensus-based groups should have a high priority for inclusion in the performance monitoring system. Gaining consensus among stakeholders on desirable goals and indicators facilitates the building of the performance monitoring system. Two current activities, both with statutory authority, are designed to build such consensus. The MHSOAC is a participant in both these efforts. One is working on a set of specialty mental health outcomes for the Early and Periodic Screening, Diagnosis, and Treatment (EPSDT) program which provides services for children and youth who are Medi-Cal beneficiaries up to the age of 21. The other is a mandate in AB 1467 for the DHCS to work jointly with the MHSOAC , and in collaboration with the CMHDA, to establish performance outcomes for services. How much of the output of these two, and any other official consensus-seeking groups, will lend itself to being incorporated into the performance monitoring system is unclear, but any recommended outcomes that can be translated into a reliable indicator should be. If, for example, one of the recommended outcomes of the EPSDT group is “out of home placement” this could be included in the performance monitoring system since this data could be obtained at a county and state level. But if a recommendation from that group is for certain outcome measures to be used by programs providing EPSDT services the question would be if there is a feasible plan to get this outcome data on everyone in the state. STEP 4: INCORPORATE SPECIFIC INDICATORS FOR THE PEI, INN, TN, AND WET COMPONENTS The indicators in the performance monitoring system should be valid as outcomes for the whole MHSA not just the treatment system. Most of the current and possible new indicators measure the results of the whole system from prevention through early intervention to treatment, all built upon a common infrastructure. This reflects the view of the MHSA as part of a fully integrated mental health system. MHSOAC Evaluation Master Plan Page 51 At the same time the MHSOAC retains the responsibility for monitoring the implementation of the specific MHSA components and plays a particularly critical role with the INN component having the authority to approve INN projects. The performance monitoring system can be used in the tracking of key elements of the implementation of these components and outcomes as appropriate. Developing appropriate component indicators and measurement strategies requires subject matter expertise. Each of the components has separate regulations and/or guidelines and in some instances are under the direction of other entities; i.e. housing funds are largely administered by the California Housing Finance Agency (CalHFA), and the five-year statewide WET Plan will now be done by the Office of Statewide Health Planning and Development (OSHPD). Understanding these requirements and the content of the component subject matter is needed to develop the most meaningful and reliably measurable indicators. The development of any component-specific indicators can occur concurrently or sequentially so long as the relevant expertise is included in the work. STEP 4: INCORPORATE COMMUNITY LEVEL OUTCOMES The development of community level indicators warrants a special effort. The community level of analysis depends largely on a different measurement strategy, i.e. the use of population based surveys as opposed to the usual state data bases that track services and outcomes for those who receive public mental health services. The Developmental and Exploratory Work Effort #4 describes a process for developing these indicators. As with the developmental work required before adding specific component indicators the work on the community level indicators could be integrated into an overall process for developing new indicators, but special expertise would need to be added. There are community level outcomes suggested in the CMHPC approved list (Indicators # 47 – 51) but accompanying measurement strategies have not been developed. STEP 5: INCORPORATE ADDITIONAL GENERAL INDICATORS The major focus of indicator development will likely be an expansion of outcomes for the whole mental health system. Ideas for new indicators can and will come from many sources, but attention might most appropriately be paid initially to the following five. The original list of 51 indicators adopted by the CMHPC which retains the statutory authority for reviewing and approving performance indicators. The recommendations of the MHSOAC funded contractor who is most familiar with the data sources and received stakeholder input as it refined the indicators. MHSOAC Evaluation Master Plan Page 52 Items coming from general health care monitoring systems in recognition of the need to adapt to these systems; thus, the importance of the HEDIS and other activities of the NCQA and NQF. Behavioral health performance monitoring systems from public and private entities nationally, other countries, and other states. 23 Input from key stakeholders: policy makers, consumers and family members, representatives of underserved and inappropriately served ethnic/racial and cultural groups,24 counties, and providers. Priority should be given to indicators which use existing data sources. Given concerns about already extensive administrative burdens every effort should be made to utilize existing data sources and efforts. For example, data collected by the CAEQRO may not be comprehensive (covers only Medi-Cal clients and Medi-Cal eligible services) nor complete (only covers paid claims not those being adjudicated) it does provide readily available useful information. As the environment changes new priorities will require new data elements. For example with physical health care items which are not currently part of state data bases. Making changes to existing data bases may be problematic but also necessary to keep up with demands for a more holistic view of quality of care. One promising alternative used in New York and Ohio would be to conduct one-week surveys which allow for the gathering of this information or to add questions of consumers in the CPS. A list of possible additional indicators is included here as a suggestion of a place to start. Whatever process the MHSOAC staff put in place to review possible new indicators (Step 2 above) will be responsible for deciding what indicators to review first. This list was developed based on the above five sources and on a perceived general consensus on what might be desirable to include, but it is not meant to restrict the options as MHSOAC staff develops its formal process. Possible New Indicators Indicator Level Measurement Method Data Source Source of Recommendation Use of non-acute System: a) Unduplicated count of persons with a) CSI CMHPC Indicators #37, locked 24-hour Quality, relevant service codes # 42 services efficiency b) % of expenditures in relevant service b) Cost categories reports UCLA/EMT Long length of stay System: Number of persons with lengths of stay CSI CMHPC Indicator # 38 (LOS) in non-acute Quality, LOS longer than 6, 12, 18 months locked 24-hour efficiency services Acute care System; Readmission to acute care setting within a) CSI CMHPC Indicator # 41 recidivism Quality, 30, 60, 90 days b) Medi-Cal 23 The review of such systems for this report has been cursory; a more comprehensive review should be undertaken as the performance monitoring system is developed further. 24 These stakeholders were involved in the original work of the UCLA contractors, and their input was included in earlier UCLA deliverables. Their input will need to be sought again through a formal process as the opportunity for new indicators arises. MHSOAC Evaluation Master Plan Page 53 efficiency Claims Outpatient follow-up System: % with an outpatient visit within 7, 30 days a) CSI HEDIS measure after acute care Quality, b) CAEQRO episode access, Medi-Cal efficiency Claims Consumer and family System: Scores on relevant CPS items CPS UCLA/EMT centered care Quality CMHPC Indicator #27 for Appropriateness Use of evidence- System: Number of clients receiving an EBP CSI (Lists has Federal NOMS based practices Quality to be updated) UCLA/EMT Cultural System: Scores on relevant CPS items CPS CMHPC Indicators #23, appropriateness Quality #27 UCLA/EMT Recovery orientation System: Scores on relevant CPS items CPS MHSOAC (5-10-11) Quality UCLA/EMT Meaningful use of Consumer New item DCR Recommendation based time on county studies and stakeholder input Social Consumer Scores on relevant CPS items CPS UCLA/EMT connectedness Seclusions and System Available only for state hospitals State data CMHPC Indicator #36 restraints bases Relative System: Percent of expenditures on state hospital Cost reports CMHPC Indicator #42 expenditures on Quality, (civil commitments), acute, and state hospital (civil), Efficiency IMD/MHRC/SNF levels of care acute and IMD/MHRC/SNF levels of care Use of Emergency System; Percent using (and $ expended) DCR UCLA/EMT Departments for Quality emergency departments for physical physical health health problems Medi-Cal problems claims Discontinuance from System: % discontinuing FSP without meeting DCR CMHPC Indicator #32 FSPs without Quality goals meeting goals Length of time in System: Median and range of length of stay in FSP DCR CiMH work on increasing FSPs Efficiency for successful discharges and for current FSP though put enrollees Percent of clients System: Percent of clients who receive a substance CSI NHSOAC Report on Co- receiving substance Quality abuse service DCR occurring Disorders abuse services (2008) Percent with known System: Percent of clients in FSPs with record of New HEDIS 13 diabetes and Quality screening in chart cardiovascular NQF MAP screening Percent offered System: Percent of clients in FSPs who smoke who New NQF MAP tobacco cessation Quality are offered assistance with smoking cessation MHSOAC Evaluation Master Plan Page 54 Number of clients System: Annual number of clients who die while New Ron Mandersheid25 who die Quality receiving services STEP 6: CONSIDER ADDING INDICATORS THAT MEASURE CHANGE OVER TIME WITH INDIVIDUAL CLIENTS Consideration should be given to including indicators that measure change over time on the same individuals. The indicators discussed thus far all measure a cross section of the population either at a point in time or over a specified period of time. The persons included in the indicator will be different each time the indicator is measured. For example, the denominator in the measurement of the rate of persons served who are employed during a particular year will be different from one year to the next. And the persons sampled in the CPS to obtain ratings of satisfaction with services will differ from one year to the next. A more meaningful way to address the impact of services is to track outcomes over time on the same people. One would measure the change in the numbers employed from the time of admission to a subsequent time after the receipt of services. As noted in Footnote #22 the original set of individual level indicators anticipated the ability to track changes on the same individuals for persons in FSPs. One of the high priority Evaluation Studies is to do this on a pilot basis to determine if the data is sufficiently reliable to make such a strategy feasible. Should it prove so, indicators could be added that measure change over time. Summary The continued development of the performance monitoring system is a high priority. The MHSOAC has made a valiant start to defining and measuring a short list of priority indicators. This effort is in line with the use of such systems in the health care system and increasingly within behavioral health. The steps to be taken include the following: Reviewing and revising as needed the original 12 priority indicators including o Developing the capacity to make better use of the results of the system o Reviewing the FY 04-04 and FY 05-06 system indicators which will be available in early 2013 Developing a formal process to consider additions to the original list so that the system remains creditable and manageable Incorporating any indicators that are well developed by other work groups in an effort to build consensus around outcomes and measurement strategies Doing the developmental and exploratory work necessary to design and pilot test indicators particularly relevant to other MHSA components not well reflected in the current set Considering and evaluating suggestions for other indicators Testing the feasibility and reliability of measuring change over time in the same individuals on the priority outcomes 25 Personal communication MHSOAC Evaluation Master Plan Page 55 Resources need to be committed each year to this evaluation method. The first item above is a top priority and is something that will be a continual process. The next four should be the high priority with the other steps following as resources allow. The order of the other steps can be determined by the MHSOAC staff. While it would be possible to consider all suggestions for additional indicators in one project the effort is more likely to maintain its momentum if the efforts are more gradual. Additionally, some of the other steps would benefit from subject matter expertise which can be gathered more efficiently as needed for specific time periods and tasks. The speed with which the tasks can be accomplished will depend on the level of resources that the MHSOAC devote to it. The MHSOAC should review systematically ways to disseminate performance results so that they can be used more broadly. Many public and private performance monitoring systems have web-based systems that present results with the capacity to “drill down” to data on a geographical or subject basis. While the vast majority of MHSOAC future work is on the content of the performance monitoring system, attention should be paid to how the data will be disseminated. The MHSOAC should at a minimum produce an annual report focused on the results of the performance monitoring system including the implications for policy and practice. Both these activities can be postponed for another year while the basic revisions on the priority indicators are done and the process for adding other indicators is developed and used. MHSOAC Evaluation Master Plan Page 56 Evaluation Method 2: Evaluation Studies Background and Context This evaluation method assesses the efficiency or effectiveness of a particular intervention. This evaluation method is technically speaking what most consider evaluation. An evaluation study can be broad or narrow in its scope, can be more or less technically rigorous, can use quantitative and/or qualitative data, and can take varying lengths of time to complete. States that have robust mental health departments conduct or sponsor evaluation studies although this capacity has shrunk in recent years. The MHSOAC, along with the DMH, has funded a number of evaluation studies on the MHSA. The prior studies include the following: The Nichols Petris Center on Health Care Markets and Consumer Welfare (funded by the California DMH and the California Healthcare Foundation) evaluated the impact of FSPs on selected outcomes (2010). They used two methods – one a sophisticated statistical method that compared the outcomes for FSP clients to those for clients in the regular system and the other that measured change among FSP enrollees from the 12 months prior to admission to the 12 months post enrollment. Both methods showed positive impacts from FSP services. The MHSOAC funded FSP Cost and Cost Offset Study, conducted by the UCLA Center for Healthier Children, Youth and Families and subcontractor EMT delineated the average costs of FSP programs by age group and by county for FY 08-09 and FY 09-10. Prior 12 month costs were compared to 12 months post enrollment in the areas of psychiatric and non-psychiatric hospitalizations, non-psychiatric skilled nursing facility care, and incarcerations. It concluded that avoided costs offset 75% of the cost of the FSPs in 2008-09 and 88% in 2009-10. The UCLA Center for Healthier Children, Youth and Families with subcontractor Clarus recently completed The MHSA Participatory Evaluation (funded by the MHSOAC) that studied three services: peer support, employment support, and crisis intervention services. Outcomes measured were housing, employment and wellness/recovery/resilience. Also included were reported consumer experiences with access to services, appropriateness of services, and continuity of care. A web-based survey (979 completed surveys) and semi-structured interviews (40 interviews) sought the views of persons with lived experience. While there were no differences in housing or employment status between those who received the three services and those who did not (but wanted them), respondents generally perceived the services as helpful. And persons receiving these services reported higher ratings on personal recovery and resilience than those who did not receive services. The UCLA Center for Healthier Children, Youth and Families conducted early studies (funded by the MHSOAC) on the outcomes of CCS (2011) and of PEI (2011). These reports reviewed and ranked for methodological rigor studies done by counties on the outcomes to date for these two components. Because the reports were done early in implementation of these components the quantity and quality of the data were limited. The results from the CSS study are described MHSOAC Evaluation Master Plan Page 57 briefly under High Priority Study #1 below. While sample sizes were particularly small for PEI studies there were encouraging signs of decreased behavior problems, improved social competence and skills, and improved parenting skills and family functioning. Current MHSOAC evaluation studies underway or in the planning stages include the following: A study of the early intervention programs will categorize and gather information on the numbers served and costs of early intervention programs funded by PEI. Three clusters of early intervention programs will be identified and the outcomes of those programs determined. A participatory evaluation of the impact and effectiveness of the local planning process for the purposes of quality improvement. Impact of the MHSA on reducing disparities in access. This study will explore trends in access to services (for all clients and new clients) by demographic characteristics including racial/ethnic and cultural groups, age, and gender; and, will gather consumer and family member perceptions of the impact of the MHSA on reducing identified disparities. The Evaluation Master Plan provides a longer range set of evaluation studies than the prior annual selection process. For the last three years, MHSOAC staff has presented ideas for evaluation studies to the Evaluation Committee which has reviewed these (and suggested others) with the staff making final recommendations to the Commission. This process yielded important topics for study, but has not been able to weigh and prioritize studies over a longer time horizon. The annual funding cycle has also made it more difficult to contemplate evaluation studies that will take longer than a year to accomplish.26 The following sections lay out a longer range plan for formal evaluation studies. The descriptions of the studies below are not comprehensive or definitive but enough to indicate the purpose, the scope, and possible methodologies. More work is needed to develop the ideas for any of the following studies. Two areas in particular should be more fully explored before embarking on any of these studies. A more thorough literature review should be done on each study topic to ensure that the proposed study addresses key issues in the field and incorporates prior study results. As noted below the very importance and relevance of the study topics makes them the object of other work which needs to be considered when designing and implementing these studies. The feasibility of some of the studies depends on counties having implemented programs that fit the parameters of what is to be evaluated. Counties will need to be queried to determine the extent to which such programs are in place, have usable data, and/or are willing to participate in a more rigorous evaluation study. 26 The MHSOAC funding does allow for two years of funding for some of the projects but the framework for the studies has generally been for yearlong projects. MHSOAC Evaluation Master Plan Page 58 The general methodology to be used in each study should be reasonably well developed before an RFP is issued. The preliminary work - literature review, review of existing programs, solicitation of county and/or program interest - can be done by internal MHSOAC staff or by separate contracts just for this purpose with subject experts. A more developed methodology included in an RFP reduces the chance of subsequent controversy about the scope and nature of the contractor’s work.27 High Priority Evaluation Studies The following studies are recommended as a high priority based on the criteria presented in Part 3. The actual rankings for each study can be found in Appendix 2. Study #1: Person Level: Collect, summarize, and publicize the outcomes from counties that have gathered such information Evaluation question How effective are treatment programs in producing positive outcomes for individuals and families served? Some counties have conducted outcome studies on treatment programs either to meet requirements of participation in training on an EBP, and/or to demonstrate success to local stakeholders, and/or as a quality improvement strategy. The earlier MHSOAC funded UCLA/EMT Evaluation Brief: Summary and Synthesis of Findings on CSS Consumer Outcomes was published in May, 2011. The results of the summary suggested only four counties had data that met adequate methodological standards – Contra Costa, Los Angeles, San Diego, and San Francisco. The programs that were evaluated were largely FSPs, and the most prevalent method was a comparison on selected outcomes between 12-month pre- enrollment to 12 months post- enrollment. The conclusions from the review were that the programs Reduced homelessness for TAY and adults Increased independence or residential living for children, TAY , and adults Reduced psychiatric hospitalization Reduced arrests There were trends, but not statistically significant results, for decreases in incarceration, reductions in physical health emergencies, improvements in education outcomes, and improvements in mental health functioning and quality of life. There were no differences or even trends in employment outcomes. Some counties have continued and improved their capacity to measure outcomes over time for consumers served, particularly those in FSPs. In addition to these standard outcomes for adult FSPs, 27 The ability to do this pre-RFP work is dependent on there being adequate internal staff or contractor time to do the work. Without additional resources this kind of pre-RFP work does not seem feasible. In addition, this pre-RFP work requires significant forethought regarding use of funds and timing of work, as funds need to be encumbered by specific deadlines if they are not to be lost. Thus, project selection/prioritization/approval, pre-RFP work, development and release of an RFP, and execution of a contract all need to occur in advance of the deadline by which funds would be reverted. MHSOAC Evaluation Master Plan Page 59 some counties are collecting data on more specific outcomes for EBPs, particularly with children. Collecting and summarizing the results of county evaluation efforts is a relatively inexpensive way of promoting positive effects from MHSA-funded programs, of highlighting areas where improvement is needed, and of encouraging counties to undertake evaluation studies. It would be helpful if this kind of information could be routinely collected from counties by the MHSOAC. Study 2: System Level (Quality) Determine outcomes of selected early intervention and selective prevention programs Evaluation questions How effective are early intervention programs? (this is a continuation and expansion of current evaluation activity) How effective are selective intervention efforts? Under the PEI guidelines counties could implement programs in both of these categories (as well as for primary prevention). Early intervention programs provide services to persons with early signs of a serious mental illness; selective prevention programs are targeted to persons who are at high risk of developing serious mental health problems. MHSOAC has contracted with the UCLA Center for Healthier Children, Youth and Families for an evaluation of three types or clusters of early intervention programs: person experiencing a first break, older adults with depression, and children exposed to a trauma. Efforts will be made to use existing data collected by programs, but primary data collection remains a possibility. Demonstrating successful outcomes for these PEI efforts is critical to the overall success of the MHSA shift from a “fail first” to a “help first” orientation. This strategy of studying clusters of similar programs to combine and/or collect uniform meaningful outcomes can produce positive results for wide dissemination. This evaluation effort should be an ongoing activity – at least for the length of Master Plan horizon. The experience with this first set of evaluations of the three clusters of EI projects should be used to determine how many additional evaluation efforts can be conducted each year, but the expectation would be that the MHOAC could begin at least one additional cluster every year. Most of the projects will extend beyond one year. Study 3: System Level (Access and Quality): Determine effectiveness of methods for engaging and serving transitional aged youth (TAY) clients Evaluation questions What types of implemented programs or programs elements are sufficiently well-defined to be evaluated? How effective are those well-defined interventions in engaging TAY and producing positive outcomes? MHSOAC Evaluation Master Plan Page 60 The mental health field has increasingly acknowledged the poor outcomes achieved by youth with a serious mental illness as they transition to adulthood (Clark, Koroloff, Geller & Sondheimer, 2008). Neither the children’s nor the adult mental health systems of care have been able to address adequately the unique needs of this population. The mandate in the MHSA to develop approaches and programs for this population has fostered substantial progress on how to engage and serve this group. Understanding the TAY culture is critical to establishing the kind of personal and trusting relationships necessary to making progress. The CMHDA subcommittee Resource Guides on TAY (2005), the CMHPC TAY with Emotional and Behavioral Disabilities: Moving Towards Self-sufficiency, and the CiMH FSP TAY Toolkit (2010) reflect the attention being paid to the population. First this study must determine what programs to use in a formal evaluation. TAY program experts can assist in identifying criteria for this selection process. The key criterion is having an intervention which is well defined. The task is thus (a) to identify well defined model interventions, e.g. Transitions to Independence Process (TIP), that are being used in various counties; and/or if there are not enough of those, (b) to identify key well-defined elements of interventions that can be used to cluster more generic programs. The next step would be to elicit participation of programs that fit the criteria. General consensus exists on the kinds of outcomes to use for the evaluation study, i.e. stability in housing, progress in education, employment, staying out of trouble, no substance use risks, adequate health status, self-ratings of well-being (e.g. social connectedness, hopefulness, positive identity),and lack of clinical symptoms. The evaluation should also include measures of program tenure and ratings of satisfaction with services. Study 4: System Level (Quality) Determine effectiveness of selected programs for older adults Evaluation question How effective are well-defined interventions for older adults? As with the TAY population, the MHSA requirement for resources being devoted to older adults increased the services available for this population within the public mental health system. Designing and implementing services for this population was a challenge for the traditional mental health system, and the first step for many counties was to consider what should be in an older adult system of care. Efforts were facilitated by the existence of a number of best and promising practices. The CiMH FSP Toolkit for Older Adults (2011) has an extensive listing of resources in addition to highlighting the approaches that have been used successfully with older adults. The first step in this study, as with the TAY study, would be to consult with subject matter experts to help in the choice of programs to include in the evaluation study. Then the participation of volunteer counties and programs should be sought followed by an actual study. Study 5: System Level (Quality) Determine the scope of implementation and effectiveness of evidence- based practices (EBPs) for children and their families. Evaluation questions MHSOAC Evaluation Master Plan Page 61 How broadly have EBPs been implemented in the state with children and families? How effective has the implementation been, i.e. are outcomes as good as in the original models? The implementation of evidence-based practices has been a priority within the California children’s system of care. The existence of many EBPs for this age group including developers eager to provide training and assess fidelity, the championing of this effort by CiMH, and the advocacy of using EBPs in the PEI guidelines have led to many counties adopting at least one such practice. CiMH has sponsored some but not all of these efforts. It has collected outcome data from many of the programs in its EBP community development teams, but submission of data has been voluntary unless required by the EBP developer. Los Angeles has begun routine data collection on the large number of EBPs it has funded through PEI. This substantial body of work underway in both implementation and outcome data collection has not been summarized statewide. California appears to be making progress implementing EBPs within the children’s system of care; documenting that progress would be a good way to advance and encourage the trend and to ensure that practices are being adopted with reasonable fidelity. The study would first survey and catalogue the extent of EBP implementation with children and families. Subject experts would need to advise on what practices to include as evidence based and whether to also include best and promising practices. Surveying counties would follow to identify the extent of implementation, fidelity assessments, and data collection. The study would then summarize and potentially add to the collection of data on program effectiveness. Partnering with CiMH would facilitate and strengthen the study. Medium Priority Studies Study #6: System Level (Quality): Determine the effectiveness of peer-led and consumer run services Evaluation questions: How prevalent are peer-led and consumer run centers28 in the state? What are the nature of the services offered, the size of the organizations, their organizational structures, their roles within the county system, the services offered, and the challenges faced? What are the benefits to the clients and the staff of such centers? The MHSA has been influential in the increase in consumers and family members working in mental health programs. The Working Well Together Project, a collaboration among National Alliance on Mental Illness (NAMI), CiMH, and United Advocates for Children and Families (UACF) provides training 28 The National Mental Health Consumers Self-Help Clearinghouse defines consumer-driven programs as follows: Consumer-driven programs must include a significant contribution from mental health consumers in design, administration, executive leadership, service provision and/or day-to-day program decision making. Some, but not all, of these organizations have consumer involvement as an essential part of their charter or mission statement, requiring, for instance, a majority of consumers on their Board of Directors or staff. MHSOAC Evaluation Master Plan Page 62 and technical assistance to counties, consumers, and family members interested in the hiring and retention of consumers and family members. One of the services studied in the recently completed MHSOAC contracted participatory research study by the UCLA Center for Healthier Children, Youth and Families and subcontractor Clarus was peer and parent/family partnership support (see above for a brief description of the study). These included “any services, supports, guidance, advocacy, mentoring, or assistance provided by an individual who has lived experience with mental health services. These services may be provided as part of a clubhouse, wellness or recovery center, consumer or family led organization, or other similar program.” Eighty-one percent of those surveyed who received such services said they helped them feel better, and 77% said they helped them with their recovery. This evaluation study has a different focus – it deals with consumer-run centers rather than consumer and family members offering supports as part of other programs. Taking an active role in managing a center is a special type of consumer involvement which extends beyond being a consumer or family member staff person. The challenges facing these centers differ from those where consumers are employees of a non-consumer run agency (Mowbray, Robinson, and Holter, 2002). A national survey in 2002 identified 1,133 consumer operated services with 534,000 clients/members served in one year (Goldstrom, Campbell, Rogers, Lambert, Blacklow, Henderson, & Manderscheid, 2006). A set of structural and process components thought to be of importance by experts has been identified (Holter, Mowbray, Bellamy, MacFarlane, & Dukarski (2004). There is some evidence of the effectiveness of consumer operated centers. Clients who received services from both a community mental health center and a consumer-operated self-help agency scored higher on five recovery-oriented measures than those receiving only the center-based services. Results were attributed to the empowerment that results from operating the organization (Segal, Silverman, & Temkin, 2010). An ethnographic study of a consumer-run program highlighted the importance of the participants’ experience of being accountable to their peers in a way that fostered a “shared project of recovery” (Lewis, Hopper, & Healion, 2012). Two early studies of satisfaction with state supported consumer-run drop-in centers - nine in Pennsylvania and two in Michigan - found that consumers who attended were highly satisfied, but that the centers faced challenges including not enough paid staff or hours of operation, management challenges, and lack of transportation in the Pennsylvania centers (Kaufmann Ward-Colasante, & Farmer, 1993) and funding constraints, insufficient accessibility, variable support by the different mental health centers in the Michigan sites (Mowbray & Tan, 1993). The California DMH funded Mental Health Services Act Implementation Study, Phase II (2007) examined consumer run services in six of the seven case study counties. Counties were progressing at different speeds in their implementation of the centers, and there was not consensus among the counties or even within the counties about how the centers should be designed and what services should be offered. The Phase III Study (Fall, 2008), one year later, gathered information about the sponsoring agency, the roles of the center directors, the staffing of the centers, the amount of staff turnover, the roles the centers played in the county’s system of services, whether or not a consumer had to be connected to the county mental health system to attend, how the centers viewed the concept of providing a “drop-in” MHSOAC Evaluation Master Plan Page 63 environment that fosters socialization, the schedules of activities, and rules on attendance and prohibited behavior. The centers were having difficulty engaging and sustaining attendance and faced challenges in ensuring adequate transportation. While viewed as valuable additions to a county’s system of care, practical challenges in operating efficiently and effectively remained.29 The CMHPC received 14 responses to a survey about Wellness and Recovery Centers in 2011.30 The study covered the following areas: how the Center was funded, the program’s design, the menu of services, the relationship of the Center to the community safety-net, and the staffing and organization. All the Centers have a regular structure of activities, all have an active Advisory Council with most having at least half the members being consumers, all have paid consumers/family members, and a few were consumer run. The major finding was that there has been an evolution from the drop-in center model to Wellness and Recovery Centers which actively promote recovery-oriented and skill-building activities. All view peer involvement as critical to their success. The purpose of this study would be to build upon the prior work to determine how the consumer-run centers have continued to evolve over the last few years. The goals are twofold: (a) to describe the nature of the goals, organization, services, and challenges faced by these centers; and, (b) to document the perceived effectiveness of the centers for clients as well as staff. The study would likely contain both qualitative and quantitative elements. Study 7: System Level (Quality): Determine the effectiveness of screening all persons receiving services for substance use disorders (SUD) Evaluation questions: How extensive are procedures for screening all youth, TAY, adult, and older adult mental health clients for SUD? How effective is the screening of mental health clients for SUD? The challenges to implementing integrated care for persons with co-occurring mental health and SUD have been well documented as have recommendations for resolving the issues. The MHSOAC Report on Co-occurring Disorders (COD) (November, 2008) states The central finding of the COD workgroup is that co-occurring disorders are pervasive and disabling, yet individuals with co-occurring mental illness and substance abuse are among California’s most underserved. The Report details the efforts being made at that time by a Co-occurring Joint Action Council working to institute a “One Person, One Team, One Plan” approach. The report highlights some evidence-based and 29 Los Angeles County, one of the case study counties, has had considerable experience in developing and supporting both Wellness Centers and Client-run Centers as part of their adult system of care. They began with 14 county-operated and 7 contracted Wellness Centers and 8 Client-run Centers in 2005. In 2010-11 they had 62 Wellness and Client-run Centers. 30 Wellness and Recovery Centers: An Evolution of Essential Community Resources, CMHPC, July, 2011. MHSOAC Evaluation Master Plan Page 64 promising practices for treating co-occurring disorders including the Screening and Brief Intervention for Substance Abuse Treatment (SBIRT) program in San Diego. As noted in the report, some California counties have undertaken major initiatives to implement the Integrated Dual Diagnosis Treatment (IDDT) model or the Comprehensive, Continuous, Integrated System of Care model (Minkoff & Cline, 2004). The current CiMH/ADPI work on a business plan for the DHCS includes as a priority issue the integration SUD with mental health and both of these with primary care services. The transition of mental health and substance abuse services to the DHCS creates the opportunity to resolve some of the long standing structural and financing barriers to integrated care. The goal of the CiMH recommendations, which parallel those of the MHSOAC’s earlier report, are to address policy and practice barriers to integrated care and to support, study, and disseminate the results of integration projects. The scope of the issue has broadened as the public mental health system attempts to address the TAY, older adult, and AB 109 populations who all bring a significant prevalence and complexity of SUD. County mental health systems are moving ahead despite the organizational and financial challenges to incorporate policy and practice changes, to conduct training and consultation, and to co-ordinate and integrate with the substance use system (most often now under a single administrative structure). Little has been done, however, to document this progress and/or any tangible outcomes from the effort. This study takes a first step in gathering one piece of simple information that can help in highlighting the issue and documenting the progress being made. The study would focus on the questions of whether all clients (youth, TAY, adults, older adults) are being screened for substance use issues and what the results of that screening have been. The first part of this study would be to determine (most likely through a survey) the extent to which routine screening is being done within counties. A survey could determine whether a county has any policies for screening clients for SUD, whether it has any data on the actual numbers screened, and any data on the results of the screenings (identifications and dispositions). A second part would be to work with a few volunteer counties to determine the effectiveness of the efforts at screening in terms of (a) generating referrals for substance abuse services and having clients begin and then successfully complete such services. and/or (b) creating or changing treatment plans within the mental health program to address the SUD. Data already collected by the counties or programs included in the study would be used to the extent possible, with primary data collection conducted if needed. Finally the study would address challenges faced by counties and programs in trying to institute rigorous screening initiatives. Study 8: System Level (Efficiency and Quality): Determine the effectiveness of obtaining routine physical health status measures on clients in FSPs Evaluation question How extensive and effective are efforts to obtain health status indicators on clients in FSPs? MHSOAC Evaluation Master Plan Page 65 From both cost and quality perspectives the physical health status of persons with mental illness has become a critical focus of attention. Evidence of the shortened life span of persons with serious mental health problems has catapulted to the forefront concerns about comorbid physical health conditions. The higher than average co-occurrence of serious chronic health conditions among those with mental health problems is well documented and contributes to the higher overall health system costs for these individuals. The mental health community in California has taken an assertive role in promoting the integration of behavioral health care with primary care as a means of addressing the need. The CiMH31 paper The Business Case for Bidirectional Integrated Care was followed by a set of Webinars on the topic in conjunction with the Alcohol and Drug Policy Institute. CiMH has also produced Clinically Informed Consensus Guidelines for Improved Integration of Primary Care and Mental Health Services32 and is now working with selected counties with a PDSA model to implement integration efforts. Dual eligible pilot programs, while maintaining the carve-out for mental health services, will foster enhanced collaboration and coordination. And Los Angeles County is undertaking an evaluation of three different models of integration. As noted earlier in this document, national health care standards are increasingly including indicators that track health status indicators and the appropriateness of health care for persons with a mental illness. The NCQA is in the first stage of piloting a HEDIS standard requiring cardiovascular screening and monitoring and diabetes screening and monitoring for persons with a diagnosis of schizophrenia. A practical strategy of mental health programs to increase their awareness of physical health comorbidities and to demonstrate to health care organizations that they are attentive to such issues is to become aware of the physical health status of their clients. This information can come from consumers themselves, from health-related personnel within the mental health system, and/or through information sharing protocols with primary care providers. Successful implementation of such a strategy is most likely where there is an ongoing relationships with clients, e.g. in FSPs. This study would (a) determine what policy, practice, and resources it would take for an FSP to obtain and record health status information on all of its clients; and, (b) determine whether that had an impact on mental health and physical health outcomes. Thus the focus would be not on how the clients received their physical health services (since that is being studied by many others) but whether greater attention to the issue within mental health programs can make a difference. Study 9: System Level (Efficiency) Refine and repeat the FSP cost and cost offset study Evaluation questions: What is the average (and range) of costs for FSPs by age group? What is the average cost offset from FSPs by category of offset, by age, and by time in the program? 31 The cited documents are available at www.cimh.org. 32 The document includes a useful annotated bibliography on the topic. MHSOAC Evaluation Master Plan Page 66 The MHSOAC funded UCLA/EMT FSP Cost and Cost Offset study produced valuable information about the efficiency of FSP services. This study should be repeated in two years with a refined and expanded methodology. Additional areas of potential offset should be included, e.g. the costs of all mental health services in the pre and post period; methodological issues (e.g. average vs marginal costs of some services) should be addressed; and, a 24 month follow-up period included. Study 10: Person Level: Determine outcomes of promising and/or community-based practices being developed by counties, particularly for un-served, underserved, or inappropriately served populations One of the concerns about a large reliance on EBPs is that it can stifle creative and innovative efforts at the local level, particularly with regard to programs for un-served, underserved, or inappropriately served persons. Local communities are developing practices that appear to be effective but don’t rise to the level of an evidence or best practice in part because of a lack of funds to undertake traditional evaluations. This study would identify a few such practices and submit them to more rigorous evaluation. Other efforts are underway to evaluate such programs. The INN component supports programs and program elements that fit this description, and they are required to have an evaluation component. The California Reducing Disparities Project (CRDP) strategic plans include practices that would fit this category of community- developed practices; and, the Master Plan recommends advocating that the CRDP projects be required to include evaluations. But even with these efforts, it is likely there will be support to have more resources devoted to the development, testing, and dissemination of new community developed practices. The process might include asking counties to nominate possible practices, selecting a few that appear to have the best chance of success, and then conducting an evaluation study of their effectiveness. Numerous other studies have been suggested; a few additional ones are listed. Other study topics are of importance but the amount of work that the MHSOAC can support (even with enhanced resources) is limited. Thus these are mentioned but not recommended at this time. System (Access): Determine effectiveness of a county welcoming initiative System (Access): Determine changes in the number served as a result of selected outreach activities (including PEI projects) System (Quality): Determine if the recovery orientation of a program can be improved System (Satisfaction): Determine if satisfaction with services is related to consumer outcomes and if so in what way Summary Beginning all the high and medium priority evaluation studies would require the MHSOAC to begin two new ones each year. The MHSOAC has already sponsored studies that have demonstrated the success of certain elements of the MHSA. While often difficult to pursue rigorous evaluations in the real world environment, the results MHSOAC Evaluation Master Plan Page 67 can be useful in both portraying successes and in highlighting effective and efficient practices. The evaluation study method is invaluable to the MHSOAC’s monitoring and oversight role. The MHSOAC should take advantage of the other work being done in these topic areas. The evaluation study topics have been prioritized, in part, because of their relevance to the immediate concerns of policy makers, practitioners, and consumers and families. The importance of these topics means that others have worked on and written about them. The MHSOAC evaluation studies need to build on these other efforts, and where possible include subject experts in the designing and monitoring of the studies. The interest of other stakeholders can also lead to collaboration and potential joint sponsorship of some of the studies. MHSOAC Evaluation Master Plan Page 68 Evaluation Method 3: Developmental and Exploratory Work Efforts Developmental and exploratory work is an important element in the Evaluation Master Plan. Developmental work is necessary at times to strengthen future evaluation work. The MHSOAC has included in its funding priorities the last few years some work that would be considered more developmental and capacity building than formal evaluation. The most important of these efforts is the ongoing work of the California State University Sacramento to support improvements in the DCR system. This contract has resulted in improvements in the use of the system through clarification of data elements, feedback to counties on their data submissions, and training on how to improve data quality. This kind of work makes any future evaluation efforts by the Commission using the DCR more reliable and is therefore a valid use of evaluation resources. Exploratory work is sometimes necessary to determine whether it makes sense to pursue an evaluation study. Some of the evaluation questions of greatest interest cannot be answered until there is more information available about the subject. That is, we cannot answer whether something is effective until we know more about what the something is. Resources are used more efficiently if the feasibility of an evaluation study is determined before embarking on it. High Priority Work Efforts Work Effort #1: PEI: Determine status of county efforts to evaluate one PEI project and make recommendations, as needed, to ensure adequate evaluations33. Evaluation questions: Ultimate: Are PEI projects effective in meeting their goals Intermediate: What is the status of the evaluation efforts with regard to the one required PEI evaluation? The PEI Guidelines require that each county (small counties excluded) conduct a formal evaluation of one of their projects. As noted in the Evaluation Studies section, the MHSOAC funded UCLA Center for Healthier Children, Youth and Families with subcontractor Clarus study found only a few counties had adequate evaluation methodologies. The purpose of this work effort is to assess how well the one PEI project is being evaluated. The key questions are as follows: Are the evaluation designs adequate Are the goals appropriate Are there measureable outcomes, complete data collection, and robust analysis In addition to determining the status of the evaluations the work should also address the reasons for any limitations or deficiencies, i.e. is there a lack of resources, a lack of focus and priority setting, a lack 33 The MHSOAC has already approved funding for a similar study for FY 12-13 to assess the adequacy of evaluations for Innovation projects. An RFP is anticipated in Spring 2013. MHSOAC Evaluation Master Plan Page 69 of knowledge or expertise, and/or delayed project implementation. Based on the reasons for any significant deficiencies the work should develop a plan for correcting them. Suggested corrective action might include discussions with the county, specific technical assistance (perhaps through CiMH), and/or suggestions that the county contract with an outside evaluator. Work Effort 2: PEI: Develop an ongoing method for describing and cataloguing programs funded by PEI Evaluation questions: Ultimate: How are PEI funds being expended? What PEI programs are effective in achieving their intended outcomes? Intermediate: What is the best way to describe and catalogue PEI programs? The initial PEI guidelines contained a number of different dimensions to describe a PEI project – what community need it addressed, what priority population did it target, and what MHSA goal did it intend to achieve (particularly which of the seven negative outcomes from untreated or inappropriately treated mental illness). The dimensions overlapped - for example, a key community need was “psycho-social impact of trauma” and a priority population was persons who were “trauma-exposed” – but the overlaps were not complete. The MHSOAC PEI Trends Report catalogued the programs in the initial plans into 13 focus areas based on meaningful combinations of these three dimensions. This allowed for a description of the percentage of programs in any particular focus area although programs could appear in more than one. The report also catalogued the site at which the program was to occur since the emphasis in the guidelines was to provide the PEI activity to the extent possible through generic community entities (e.g. schools, recreational areas) as opposed to in the mental health system. The RAND Corporation PEI Evaluation Framework suggested that county PEI programs can be categorized by the kinds of activities that they undertake – both the structure and the process. RAND did not conduct a categorization, but outlined what it believed would be possible categories. The current MHSOAC contract with the UCLA Center for Healthier Children, Youth and Families is categorize early intervention programs by target population, mental health issue addressed, kind of program, and relevant program features. Given the diversity of county PEI activities and the above mentioned overlap of program descriptions (community need, target population, program features) developing a uniform statewide system for describing PEI programs will be a complicated and likely difficult task. The purpose of this exploratory work is to develop alternatives for a categorization system and after pilot testing recommend a best method. A first step in this work would be to review the strengths and weaknesses of the categorization schemes already proposed or used. There will be inevitable tradeoffs. For example, the PEI Trends Report allows a program to be put into more than one focus area while an alternative approach would be include it in only one area based on where the majority of the activity or effort is directed. A categorization system would likely have more than one dimension. Programs could be categorized by program focus, by the predominant site at which the activity occurs, and by type of prevention (primary or selective). MHSOAC Evaluation Master Plan Page 70 Any categorization system should not be unduly burdensome on counties and should have a usefulness that balances the extra work involved in collecting the data. A primary assumption should be that all data would be collected at the program level in a summary fashion leaving the decisions on how to accumulate the data from the programs to the counties. Attempting a statewide data system that collects information at the individual level on persons served by PEI programs would be unsustainable. Sufficient information on the numbers reached, the nature of the efforts, and the characteristics of those reached can be accomplished using summary data. The most promising categorization system(s) should be piloted in a few counties. The costs of gathering the information should be weighed against the usefulness of the resulting information. Once the classification system has been designed it can be incorporated into the Annual Update process to ensure ongoing information. Work Effort 3: System level (Quality, efficiency): Explore feasibility of classifying FSP programs in a meaningful and useful fashion Evaluation questions Ultimate: Are some FSPs more successful than others? If so, what makes the difference? Intermediate: Can FSPs be classified in a meaningful and useful fashion? The initial FSP model evolved from the AB 34/2034 work with adult consumers which itself evolved from the prior AB 3777 pilots for adults, and both are variants on the original Assertive Community Treatment (ACT) model. Over time the FSP models have evolved to meet the needs of counties and to accommodate the needs of the other three age groups – children/youth, TAY, and older adults. The MHSA regulations and guidelines deal with the minimum standards, e.g. 24/7 coverage, co-ordination of services when a client enters an institutional setting, and some characteristics of the service delivery model such as a team approach and the capacity to deliver intensive services. CiMH notes “Since their inception in 2005, FSP programs continue to develop and improve by identifying and implementing key practices that consistently promote good outcomes for consumers and their families.” The FSP Toolkits developed by CiMH for each age group include a section on Philosophy which outlines the underlying principles upon which the FSP concept is based, such as for adults “whatever it takes,” outreach and engagement, a welcoming environment, and ensuring cultural relevance of interventions. A second section describes the service array which includes for adults using a strengths- based approach, connecting with the family, increasing social supports, and reducing involvement in the criminal justice system. A third section deals with the team structure “which refers to the staffing, operations, decision-making, organization of the [adult] teams, and to their overall approach to the coordination of client care.” The fourth section covers housing suggesting strategies for ensuring permanent housing alternatives for clients. Counties have been flexible in their implementation of the FSP models within the constraints of these general guidelines, principles, and recommended practices. As expected, models of service for children MHSOAC Evaluation Master Plan Page 71 and older adults have diverged from the adult model because of different client needs and different service structures. The desire to compare the outcomes of FSPs both as part of an oversight responsibility and to identify best performers for quality improvement purposes are stymied by this variety in FSPs. Without some way to classify FSPs, comparisons will rightly be subject to criticism. The purpose of this study would be to explore different methods for classifying FSPs within each age group.34 The work would be done with a few volunteer counties who would be willing to have their FSPs possibly gather additional data which would describe the nature of their services. Inclusion of service usage from either the county or the CSI data bases would also be used to measure intensity and type of services. The potential classification systems could then be used later to determine if they relate to client outcomes in any consistent fashion. Medium Priorities Work Effort 4: Community level: Develop indicators for the community level Step #4 in the Performance Monitoring section is the addition of community indicators developed through a developmental/exploratory work effort. The meaning of the community level within the evaluation schema has been described in Part 2 on the Overall Model and Scope. The use of performance indicators is the most appropriate method for tracking outcomes at the community level. The kinds of indicators to be measured at the community level include, but are not limited to, the seven negative outcomes resulting from untreated or mistreated serious mental illness, the prevalence of mental illness, service penetration rates including for the traditionally underserved, physical health comorbidities, and the extent of stigma and discrimination. Three types of populations need to be considered at the community level: everyone in the community, those in the community with a serious mental health issue, and those in the community who are a high risk of a serious mental health issue. The major data sources for these community indicators are population based surveys. The goal is to establish a set of indicators that can be tracked over time using existing survey data bases. The RAND Corporation PEI Framework document includes a list of potentially relevant surveys. A few examples are included here to provide a sense of what indicators from these surveys might look like. Examples from the California Health Interview Survey (CHIS) include For the general population, the percentage that have a mental health disorder; percentage without any psychological or emotional issue 34 One would want to start with one age group. The interest in comparing FSPs is clearest with adults because outcomes are already clearly defined and because there is at least a generally accepted basic model. But the diversity of program models within the other age groups might actually make starting with one of them easier. MHSOAC Evaluation Master Plan Page 72 For persons identified on the survey with a mental disorder: percentage who have received any service from a physician or a mental health professional or taken a psychiatric medication; percentage who are employed; percentage with suicide ideation or attempt Examples from the California Healthy Kids Survey(CHKS) include: Percent with emotional problems serious enough to interfere with usual activities Percent with suicide attempts or ideation Percent getting help when needed for emotional or substance use issues Percent feeling connected at school; percent having a trusted adult outside of family or school The developmental work entails selecting a set of potentially relevant specific measures and then assessing them for reliability, relevance, likelihood of showing change overtime, and likelihood of continuing to be collected. Actual data on the most promising items should then be gathered from prior surveys, analyzed, and interpreted to determine if they are meaningful and useful. Based on that work recommendations should be made about which limited number of indicators should be added to the Performance Monitoring system. This work effort should not be too costly or time intensive to accomplish. Additional work will likely be needed to work with survey administrators to both ensure that needed survey elements are continued and that potentially new ones are added that might more directly measure outcomes of interest. Work Effort 5: Person level: Develop system to track outcomes for adults35 in less intensive services than FSPs Evaluation questions: Ultimate: How effective are our services for adults who receive less intensive services than what is provided in an FSP? Intermediate: Can we develop a system for tracking relevant outcomes for a set of adult clients (and/or level of service) that is less intensive than FSP? Much of the MHSOAC evaluation effort for adult clients thus far has focused on the effectiveness of FSP services. FSPs are usually the most intensive community services provided in a county system of care. For counties that use a level of care structure for organizing their service system it represents the most intensive level of services. Focusing evaluation activity on FSPs is reasonable from both policy and practical perspectives. On the policy level, a majority of CSS funds are devoted to FSPs, and the needs of the adult FSP enrollees are the most intense and complex. Practically the existence of a special data system (DCR) to track the progress of the clients in FSPs makes it easier to conduct evaluation studies. 35 The work is limited to adults at least in this initial stage because of the more defined structure of the FSP services for this age group. The greater diversity in FSPs for the other age groups makes the value of this work less clear at this point in time. Level of care discussions have been most relevant for the adult system of care. MHSOAC Evaluation Master Plan Page 73 Many are interested in expanding evaluation efforts to more adult clients than those in FSPs. Currently assessing the effectiveness of the mental health system for these other clients occurs only within the evaluation of specific programs and does not look at the overall progress of clients independent of the particular program in which they may be receiving services. To obtain a better view on the effectiveness of the system of care for these individuals a focused evaluation structure would be useful. The challenge to establishing such a system is threefold. The first is the definition of what clients would be included in such a system. It has been reasonable (but burdensome) to ask FSP programs to fill out periodic forms (KET, 3M) about the status of their clients because they know their clients well and see them often. It is not reasonable (at least at this point) to have staff do anything as extensive as the DCR type reporting for clients with whom they have less interaction. There are programs that provide a next level down in intensity of services from an FSP where staff has an ongoing relationship with clients. Within the CMHDA level of care structure this would be a level three service (California Adult System of Care Committee Recommended Guidelines for Level of Service, CMHDA, 2008) The first task is to create a uniform definition of who/what would be included in such a system. The definition could be based on client characteristics, e.g. a level of care assessment or a program requirement for a specific frequency or intensity of services. This would define the set of clients who would be in the group to be tracked. The initial effort should start small and not necessarily include everyone who could be considered as needing a level three service. The second challenge is to identify what information to collect and with what frequency. Some counties, who have already begun to work on this issue, have suggested that the most meaningful outcomes for this set of clients might be social connectedness and productive use of their time. Self-administered recovery oriented measures might also be reasonable for this group of clients. In terms of frequency semi-annual updates (and at discharge) might be appropriate. The third challenge is how to collect the data. The DCR system might be able to accommodate the kind of data collection activity that such an expansion would entail36. The systems already in place in a few counties would be another alternative that should be explored. This exploratory work should be done with a few volunteer counties who are interested in devising a system for tracking outcomes for these adult clients. They could work together with the MHSOAC to address the three challenges and pilot a new system. This would be a multiyear effort with the next stage being dependent on the learning from the prior efforts. Work Effort 6: Person and system (Quality) levels: Determine the interaction between the characteristics of the populations served in FSPs and the outcomes obtained Evaluation questions Ultimate: What kinds of clients have better and worse outcomes in FSPs? 36 While the use of the DCR for this purpose is technically possible, it should definitely not be considered for this purpose unless and until the current problems with the system are remedied and adequate support is provided to maintain the system. MHSOAC Evaluation Master Plan Page 74 Ultimate: What FSPs do better in achieving positive outcomes for their clients? Intermediate: What characteristics (demographic or clinical) of the clients served in FSPs are related to outcomes? Intermediate: Is it feasible to develop a risk adjustment methodology to use in comparing outcomes across FSPs? Differences in outcomes in an FSP can result from differences in the programs and/or differences in the clients. Unless one can control for differences in outcomes that result from client characteristics, comparisons of outcomes between FSPs programs is open to question. For example, if persons with multiple years of institutional care have poorer outcomes than those never institutionalized, then comparing outcomes between two programs would have to adjust for how many such persons were in each program. To begin to develop a risk adjustment methodology one needs first to determine the relationship between client characteristics (e.g. years in institutional care) and client outcomes. Statistical analysis of the outcomes for multiple clients in different FSPs is the method that would be used to begin to answer this question. A contractor with exploratory statistical expertise would be an ideal contractor for this work. It would require close cooperation among volunteer FSP programs and the contractor to assure that all the data is reliable and accurate and to be willing to engage in feedback loops which can hopefully lead to reliable client characteristic predictors of good outcomes. Not all the relevant data may be on the DCR, e.g. clinical characteristics, relevant historical factors, engagement in services, and self-rated recovery concepts. Thus participating programs would need to be willing to engage with the contractor to gather some additional data elements. Work Effort #7. (Infrastructure - TN) Develop and implement a plan for routine monitoring and special studies of the impact of technological need (TN) expenditures. Evaluation questions Ultimate: How have the MHSA expenditures on TN enhanced services for persons in the public mental health system? Intermediate: How have the MHSA expenditures on TN enhance the counties’ information technology (IT) capacity? Intermediate: How have the TN expenditures enhanced consumer and family empowerment? The TN funds are intended to be used by counties to update and modernize their IT systems and to support projects that allow consumers and family members easier access to culturally and linguistically competent health information. The UCLA Center for Healthier Children, Youth and Families and EMT have recently submitted a draft Capital Facilities and Technological Needs Report (funded by the MHSOAC) that reviews expenditures on MHSOAC Evaluation Master Plan Page 75 this component.37 The information presented here was taken by the contractor from the original plans of the counties and so reflects the counties’ intended projects and not actual expenditures.38 IT projects o 89% of the counties proposed a project related to electronic health records (EHR) o 47% a telemedicine/ telehealth project o 31% a pilot project for quality assurance monitoring o 57% a project for imaging or paper conversion 80% of the counties proposed a project related to consumer and family empowerment o 71% a project for client or family access to computing resources o 58% a project related to a personal health record o 40% other online information resource projects The site visits to four counties as part of the development of this Master Plan confirmed significant investment in data systems to upgrade capacity to track, analyze, and report on service usage and outcomes. This work effort is designed to ultimately provide a comprehensive and understandable assessment of how the MHSA TN funds fit into overall county IT system enhancements and how these IT improvements improve services for clients. The first step is to develop a plan and the second to implement the plan. Both steps will require intensive collaboration with others; a small work group would be useful for assisting throughout this work effort. Participants from counties (IT, evaluation, and clinical managers) and the CAEQRO would be essential. The sections of the plan should include the following: A classification system for the projects funded by the TN expenditures.39 A process for how to track progress on these projects. One possible process would be to add a selected item(s) to the Annual Update. Another that might provide a richer set of information would be to have the CAEQRO include in their annual county visits a review of the MHSA-funded TN project. A method for portraying how these projects fit into the counties’ overall IT strategies. This might also be a task amenable to being done by the CAEQRO. Suggested means to relate IT systems to overall county service systems. This may include concrete ideas or may be a recommendation for another developmental and exploratory work effort to pilot some alternatives. A special section on the family empowerment projects 37 The Report is still under review by the MHSOAC so no information on actual expenditures is included here. 38 The denominators for the following figures are not uniform as not all counties reported on all plan elements, but this provides a general sense of how the counties intended to spend their TN funds. 39 The categories used by the UCLA Center for Healthier Children, Youth and Families and EMT should be reviewed but may or may not be determined to be most useful. MHSOAC Evaluation Master Plan Page 76 Once the plan is developed the MHSOAC can determine how to allocate resources to implement the plan. Work Effort 8: System (Quality): Explore the extent of and variation in the recovery orientation of programs Evaluation questions: Ultimate: What approaches are most successful in improving the recovery orientation of programs? How do client outcomes relate to the recovery orientation of programs? Intermediate: What is the best way to reliably measure the recovery orientation of a program? The principle of recovery is a cornerstone of the MHSA. Many of the county training activities are designed to enhance the recovery orientation of program staff. And many of the program and system changes and initiatives are designed to make them conducive to a recovery orientation. The few measurement methods that have been developed to assess the recovery orientation of programs have been derived either directly from or with substantial consumer input. The most frequent measurement method is a self-administered survey completed by the program’s clients. The survey items are scales of attributes which consumers believe assist them in their recovery. A few of the instruments have been subjected to some reliability and validity testing but the field does not have a gold standard scale. There is beginning work on using these scales to measure whether efforts to influence a program’s environment through training or other change efforts are successful. Making a contribution in this area would not only address the implementation of a critical MHSA value but also make a contribution to the broader field. The first step would be to review all the existing scales and measurement methods and select a few for use in measuring recovery orientation; the most prominently used are the Recovery Oriented System Indicators (ROSI) and the Recovery Self Assessment (RSA), and the Recovery Enhancing Environment Scale (REE)40. Programs would be solicited to have the scales administered to their consumers (and in some instances also to staff) to assess the level of recovery orientation of their programs. The distribution of results would be informative particularly if it could be related to type of program and/or type of training received and/or characteristics of the staffing (e.g. how much presence of peer staff). Once the first stage of study is completed, the instrument(s) can be used in at least two ways in more evaluation-type studies: to measure change resulting from structured culture change initiatives and to relate recovery orientation of programs to client outcomes. Other possible Developmental and Exploratory Work Efforts of importance: Two other developmental and exploratory projects received interest during interviews and from feedback and would be worth pursuing if resources were available. 40 See the Human Services Research Institute HSRI) Compendium of Recovery Measures, Volume II, 2005. MHSOAC Evaluation Master Plan Page 77 Person level: Explore development of a means to measure the attainment of client determined goals System level (Quality and Satisfaction): Pilot an immediate consumer feedback method that gives the provider ongoing information about what is working/not working Summary The priority Developmental and Exploratory Work Efforts should receive as much resource commitment as the Evaluation Studies. Some of the more important evaluation questions cannot be answered without the kind of developmental and exploratory work efforts described above. The relationship between the characteristics of the persons served and outcomes and the relative effectiveness of different programs require this level of prior analysis. The ability to measure the recovery-orientation of a program and use the information as a quality improvement tool and the capacity to measure outcomes in a systematic way for persons other than those receiving the most intensive level of services would add to our knowledge base and allow for better evaluation studies in the future. These work efforts require considerable collaboration with participating programs and counties. The developmental and exploratory nature of these projects require identifying interested counties and programs to join with the MHSOAC in the design and implementation of the work. These efforts are not as cut and dried as evaluating the impact of a program; they are likely to entail back and forth gathering of information, analyzing information, and trying out something different based on what has been learned. This can be exciting work if the participants approach it with flexibility and a sense of inquiry. There is actually no dearth of providers and county managers who would be interested in participating – it is just a question of reaching out and finding them. MHSOAC Evaluation Master Plan Page 78 PART 5: SPECIAL EVALUATION CONSIDERSATIONS FOR MHSA COMPONENTS While the Evaluation Master Plan views the MHSA as an integrated system, the MHSOAC has a responsibility for oversight of the specific components. This dual function creates challenges, but the Master Plan can accommodate both. Most of the performance indicators reflect the activity of the full continuum from prevention to early intervention to treatment. Better and earlier identification of persons with mental illness or at risk of mental illness will impact the outcomes for the whole system. A better trained workforce and better information technology will make the system more efficient and produce better overall outcomes. More appropriate housing for persons with disabilities will make the housing outcomes better. The specific components can also be viewed on their own. Specific references to component-specific activities are noted in the Plan Step 4 in the Performance Measurement section entails incorporating indicators recommended by experts in the subject matter of each of the components Evaluation Study #2 relates specifically to PEI Developmental and Exploratory Work Effort #1 and #2 relate specifically to PEI, and #7 specifically to TN This part of the Plan provides context for the components, reviews the places in the Master Plan that are particularly relevant to the component, and makes additional recommendations for some of the components. Because activity related to the components appears in the main part of the plan under the three evaluation methods, it would be duplicative to repeat the information here. Rather references are made to those other parts of the Plan. But to avoid too much shuffling through the Plan, some information is repeated here for the ease of the reader. A discussion of each of the components follows. As noted below, the Community Services and Supports (CSS) component is included extensively in the Evaluation Master Plan. The discussion of that component is therefore limited to the MHSA Housing Program which has not been addressed elsewhere in the Plan. Community Services and Supports (CSS) The CSS component of the MHSA is well represented in the three evaluation methods. The dominance of treatment related evaluation questions and the presence of more available data leads to the CSS component being represented most extensively in the Master Plan. The performance indicators that are measured on all persons in the mental health system while mostly reflecting the performance of the entire integrated mental health system from prevention through treatment are MHSOAC Evaluation Master Plan Page 79 weighted towards treatment-related system variables. And the indicators that are measured on FSP clients are totally within the CSS component. Many of the evaluation studies and the developmental and exploratory studies are directed towards treatment issues. The MHSA Housing Program has nearly $400 million of MHSA funding from counties to meet supportive housing needs. The MHSA Housing Program, begun in 2007, is administered by the California Housing Finance Agency (CalHFA). MHSA funds were made available to counties who then dedicated funds to this collaborative program. Projects need to be approved by CalHFA. The MHSA Housing Program provides both capital and operating subsidy funding for the development of permanent supportive housing for individuals who experience mental illness and who are homeless or at risk of homelessness. It funds projects that are identified in the county CSS plans, and the DMH had the responsibility for reviewing and approving each applicant’s proposed target population and supportive services plan. DMH reported semi-annually on the progress of the Program. According to the report of October 2011, 127 applications had been received from 33 counties since the beginning of the program. Of these, 104 had received loan approval. As of the report date, 1,121 units were ready for occupancy or had been occupied. Completion of the already approved applications will create a total of more than 1,500 supportive housing units. The MHSOAC made recommendations in January 2012 for improvements in the MHSA Housing Program. The results of the Senate Office of Oversight and Outcomes (SOOO) review of the MHSA Housing Program were presented to the Commission. The identified biggest issue was the restriction on counties not to spend more than one-third of the funds on rental subsidies since counties have limited other resources or avenues for funding for this purpose. The MHSOAC approved the SOOO recommendations to continue to grant waivers from this restriction on a case by case basis and consider waiving altogether the restriction for counties who have already spent at least 80% of their allocated dollars. The problem of the funding base for small counties not being sufficient to inaugurate any projects (11 counties received less than $1 million) remains unresolved. The MHSA Housing Program will likely not continue unless counties provide additional funding. The hope was initially that the $400 M would be the start of an ongoing program. But county dissatisfaction with the operation of the program and the general shortage of funding makes its continuation unlikely. Since the MHSOAC has recently reviewed the program and taken action no further evaluation activity is included in the Evaluation Master Plan. Should the program continue under another administrative structure the MHSOAC can reconsider this decision. The MHSOAC should continue in the meantime to track the reports to the Legislature from the MHSA Housing Program. MHSOAC Evaluation Master Plan Page 80 Prevention and Early Intervention (PEI) Background and Context: The PEI component is one of the unique and most exciting parts of the MHSA. The PEI component was the second to be rolled out with initial plans being approved by the MHSOAC between July 2008 and November 2010. The PEI component has raised substantial interest nationally because it is one of the most significant investments in prevention and early intervention that has been made in the mental health field. And it is a high priority to the MHSOAC as it played such an important role in contributing to the DMH guidelines and in approving initial plans. The MHSA section on PEI required PEI programs to emphasize strategies to reduce seven negative outcomes which resulted from untreated mental illness. The kinds of negative outcomes are most relevant to persons who have a serious mental illness that remains untreated. Evidence indicates that the longer the time period between the first signs of serious mental illness and the initiation of treatment the more disabling the illness is likely to be. The guidelines that were issued by the Department of Mental Health with active participation of the MHSOAC broadened the range of programs and the populations that were to be served under the PEI component. Projects could be designed for selective prevention (targeting a high risk group) or primary prevention (directed to the whole population) in order to bring about mental health outcomes for defined “key community needs” and “priority populations” with known risk for mental illness. One requirement of the initial plans was the inclusion of a formal evaluation of one of the PEI projects with an exception for small counties. In addition to county projects, PEI funding supports four statewide initiatives. A portion of the PEI funds were set aside in the initial planning to support four41 statewide projects in an effort to focus some of the funds in a way that could make a sizeable change. Three of the statewide projects are funded primarily via a Joint Powers Agreement among counties (CalMHSA). CalMHSA has contracted with 25 programs under the three major initiatives – stigma and discrimination reduction, school mental health, and suicide prevention. The fourth statewide project – the California Reducing Disparities Project (CRDP) will be administered through the Department of Public Health and is focused on four different racial/ethnic groups and the LGBTQ community. Five Strategic Planning Workgroups (SPWs) compiled “community-defined evidence and population-specific strategies for reducing disparities in behavioral health.” These five reports are being combined into an overall plan which will be the basis for implementation of this fourth statewide project. 41 A fifth statewide project was conceived originally to fund a statewide training and technical assistance effort. Funding was provided directly to counties to achieve this goal. MHSOAC Evaluation Master Plan Page 81 The challenge for the Master Plan is to incorporate PEI as an integral part of the whole public mental health system while also considering its special effort. Besides the usual complexity that might be expected with trying to evaluate prevention activity the intention behind the PEI component create special challenges. PEI is designed to gradually shift the whole public mental health system from a “fail first” system to one which embraces early recognition of and response to risk or indications of mental illness before they become severe and disabling. Theoretically over time the savings resulting from reducing the negative consequences of untreated problems through prevention and early intervention can be put into even more PEI activity. Thus PEI activity is viewed as part of a whole integrated system of care from prevention to early intervention to treatment. From this perspective the outcomes for PEI are reflected in the outcomes for the whole mental health system e.g. in performance monitoring system. A reduction in homelessness would be a result of both PEI and of CSS, of prevention, early intervention, and treatment. In this view the outcomes for the PEI system are ultimately the same as for the direct treatment system and should not be thought of as separate. But because the investment in PEI is so significant and so unique there is also the desire to separately track and report on how the programs and initiatives within the component are doing - thus the interest in a separate break-out analysis for PEI. The MHSOAC has produced special PEI reports. A PEI Trends Report in 2011 was produced by MHSOAC staff based on the counties initial three-year plans. It delineated the number of counties and programs in each of 13 program areas which were a combination of priority populations, community needs and MHSA-specified negative outcomes. The report also described target populations and program features of counties’ intended PEI projects. The MHSOAC contracted for an early review of possible outcomes from county PEI projects. In 2011, the ULCA Center for Healthier Children, Youth and Families with Clarus Research issued a Summary and Synthesis of PEI Evaluations and Data Elements. The report was based on information in the original three year PEI plans and Annual Updates. The report concluded that while the intended outcomes for the projects to be evaluated were generally appropriate the descriptions of measurement tools and research methods lacked specificity. The Annual Updates indicated that for FY 09-10, 448,000 persons were reached by a PEI activity in the 30 counties that reported data. The demographic characteristics of this population were also presented. The contractors culled the plans and updates for outcomes information and found 37 reports on outcomes; but, only five had “high utility” data, i.e. measurement methods that were adequately described and data analyzed appropriately. The greatest amount of evaluation activity was for Emotional and behavioral health problems among at-risk children, youth, and young adults received the most evaluation attention. The authors concluded that the implementation was too early, and the counties evaluation capacity insufficient to draw any definitive conclusions although there were some positive trends. They recommended instituting a more uniform system for MHSOAC Evaluation Master Plan Page 82 collecting program and evaluation data from the counties and providing technical assistance to counties so that they could do a better job with their formal evaluations. A current MHSOAC contract will provide outcomes for three different kinds of early intervention programs. The MHSOAC has a current contract with the UCLA Center for Healthier Children, Youth and Families, to catalogue early intervention programs by target population, mental health issue addressed, kind of program, and relevant program features. The contract also includes evaluating outcomes for three clusters of programs with common features. The three likely clusters are persons with a first break psychosis, older adults with depression/suicide ideation, and children experiencing mental illness as a result of trauma. The RAND Corporation is conducting a large multi-year evaluation of three CalMHSA administered statewide projects. CalMHSA contracted with RAND Corporation to conduct a large-scale multi-year evaluation of the three statewide projects they are implementing. The evaluation plan is quite extensive so the following just skims the surface of what is included. At the program level: RAND will evaluate the process and outcomes of each of the programs by program activity. It will assess short-term outcomes (the immediate targets of change) and “key” outcomes (reducing the negative outcomes). The evaluation methods will include document and material reviews, attendance records, case studies, key informant interviews, and surveys. At the initiative level: o For all three initiatives RAND will summarize the program level results and do a baseline and follow-up survey to measure stigma, knowledge, help provision, help seeking and barriers to help seeking, and mental illness scores o For the stigma and discrimination reduction (SDR) initiative RAND will include focus groups of persons with mental illness; for the suicide initiative it will include vital statistic reports of suicide; and for the school mental health initiative, existing school based surveys (e.g. CHKS) and statewide surveys (CHIS). The RAND contract also includes the development of a plan for long-term outcome monitoring of the three areas. RAND separately developed a PEI Evaluation Framework CalMHSA also contracted with RAND Corporation (with MHSOAC funding) to create a statewide PEI Framework which could be used for the county level efforts funded directly from the PEI funds. The report includes a thorough compilation of potential data sources for measuring population-based outcomes. The framework includes the measurement overall of the seven negative outcomes as well as general well-being. RAND argues that population-based measures are most appropriate for the overall MHSOAC Evaluation Master Plan Page 83 evaluation of the PEI effort, but do not exclude the possibility of collecting that program level outcomes using the basic structure of the framework. The authors recommend the development of a system for statewide collection of data regarding program outputs and features. PEI in the Evaluation Master Plan There are two aspects to the PEI evaluation effort. One is the inclusion of PEI efforts in each of the three types of evaluation methods. These are noted below by priority level with a reference to where they are described in the Plan. The second aspect is a set of recommendations relevant to PEI evaluation that are not included in one of the methods. Performance Monitoring: As noted above the goals of the MHSA are relevant for both the prevention/early intervention and the treatment system, i.e. reducing the development and consequences of mental illness that is not detected and treated early. Therefore many of the existing performance indicators are quite relevant to the overall goal of the PEI component, e.g. the prevalence of mental illness and penetration rates. There are, however, no specific PEI related indicators in the existing set. One of the steps in the performance monitoring section of the plan calls for the development of community level outcomes that would be particularly relevant to the statutory PEI goal of reducing negative outcomes from untreated mental illness and reducing stigma and discrimination. (See Developmental and Exploratory Work Effort #4 below.) As with all the performance indicators it is difficult to attribute changes over time to the MHSA since there are so many other variables that affect the indicators. This is particularly the case with the community level indicators. The importance of continuing to monitor these indicators even if they do not show much change is to keep our eye on the ball, i.e. to not forget the ultimate goal of the MHSA. Evaluation Studies: Evaluation Study #1 which is building off of county evaluation efforts extends to PEI projects as well as CSS programs. As counties expand their outcomes work to include more PEI programs this will yield more results. Evaluation Study #2 continues the work of identifying and evaluating early intervention programs and extends the effort to selective prevention programs. This should be an ongoing effort. Demonstrating the effectiveness at the program level of the early intervention and targeted selective prevention activity will be the mainstay of the MHSOAC evaluation efforts specifically related to the PEI component. Developmental and Exploratory Work Efforts: There are three developmental/exploratory efforts related to PEI. Two are high priority items. Developmental and Exploratory Work Effort #1 is a review of county efforts with regard to the required evaluation of one PEI project. The scope of the review includes determining the MHSOAC Evaluation Master Plan Page 84 reasons for any deficiencies and making recommendations for how to address those deficiencies. Developmental and Exploratory Work Effort #2 will develop a way to classify PEI programs so that program level data can be collected statewide. This activity is already being done for early intervention programs under the existing MHSOAC contract with UCLA Center for Healthier Children, Youth and Families. This is a formidable task but if done effectively could allow the collection of information to be included in the Annual Updates. Developmental and Exploratory Work Effort #4 is the development of a small set of community level indicators including the rates of the seven negative outcomes from untreated mental illness. While these are cited specifically in the PEI section of the Act, the tracking of these community indicators is applicable across the whole continuum of prevention, early intervention, and treatment. Other Recommendations Related Specifically to PEI Evaluation 1) Urge the Department of Public Health to fund evaluation of the projects to be included in the statewide California Reducing Disparities Project. The three-year lifespan of the projects should allow for meaningful evaluation. The Department of Public Health should include funding for the evaluation of the projects it supports. The evaluations should be conducted with assistance from the five Strategic Planning Workgroups (SPWs) who best understand the issues and who can provide credibility for the evaluations. 2) Utilize program level data collection system from the Developmental and Exploratory Work Effort #2 to collect basic statewide PEI information on the numbers and characteristics of persons served. That Work Effort will produce a system to categorize PEI programs. Once this classification system is developed it would be possible to have PEI programs report summary information to the counties who could pass it on to the state through the Annual Update process. If this is can be done with reasonable accuracy and without an undue burden on counties and programs then it will be possible to collect summary information about numbers and characteristics of those served which could be accumulated to statewide totals. 3) Do not develop a separate PEI Evaluation Framework. The RAND work has been productive in highlighting the importance of community level outcomes, in compiling an annotated list of potential measurement tools, and in suggesting potential ways of categorizing programs. The next step should be to continue the evaluation activities already underway and to start the other studies and work in the Plan. The PEI evaluation questions can be addressed through the proposed activities in this plan. Innovation (INN) Background and Context: As with PEI, the Innovation component presents a unique challenge with regard to evaluation. As noted in the Innovation Trends Report (2012), “Evaluation is at the core of MHSA Innovation, since all programs are pilots to be tested,” i.e. evaluation is not an add-on but the essential element of every MHSOAC Evaluation Master Plan Page 85 innovation project. The strategy underlying the INN component is to invest in new or modified practices; and, if they are shown to be effective, for the county to adopt them and disseminate them to the rest of the mental health system. Thus the purpose is system change not just demonstrating a successful intervention. Every INN project must include an evaluation. The ultimate test of the success of the INN component is not just positive results on those evaluations, but evidence that positive results lead to changes in practice in the county and elsewhere. The evaluation of the INN component is a multi-year endeavor which requires at a minimum having the local evaluations demonstrate the effectiveness of the INN projects and assessing the success in the dissemination of those deemed effective as manifested by their adoption both in the county and more broadly in the state. The MHSOAC has been active in supporting the implementation of the INN component with constant emphasis on the critical role of evaluation The MHSOAC provided extensive technical assistance on the INN component to counties and assisted CiMH in providing training to help counties understand Innovation’s purpose and develop appropriate pilots and evaluations. The MHSOAC also worked closely with CiMH to develop an e-learning curriculum specifically focused on evaluation of INN programs. The MHSOAC produced an Innovations Trend Report (January, 2012) which detailed the characteristics of the initial 86 INN programs proposed in county plans and approved by the MHSOAC before the passage of AB 100. Most of the INN programs are adaptations of existing practices in a new setting or community. Half of the programs have the primary purpose of improving the quality or outcomes of services while one-third were designed primarily to test ways to improve access to services including for underserved populations. The INN pilots assess treatment, crisis response, early intervention, prevention, and infrastructure activities. Many programs are for multiple age groups so the following tally of age groups covered totals more than 100%. Children/youth 34% TAY 83% Adults 72% Older adults 61% An already approved Evaluation Study will assess the adequacy of local evaluations of INN programs. The MHSOAC has approved an INN Evaluation Study for FY 12-13 with an RFP scheduled for release in the Spring of 2013. The study will assess the adequacy of the county evaluations of their INN programs and provide assistance where there are problems. Enough time has elapsed from the initiation of the INN component for counties to be well along in the design and implementation of their INN evaluations. If these evaluations are to be useful to the counties and the state it is essential that their results be creditable and disseminated. So this first task of determining the status of these evaluations is critical. The goal is to ensure that all counties’ INN programs feature robust evaluations. Innovation in the Master Plan Performance Monitoring MHSOAC Evaluation Master Plan Page 86 Step #3 in the Performance Monitoring section is to include relevant and meaningful indicators that are specific to components. A relevant indicator for the INN component might be the extent of dissemination and adoption of successful INN programs both within the originating county and elsewhere throughout the mental health system statewide. A requirement could be added to the Annual Update to include information on how the county has disseminated the results of their INN evaluations and whether they have adopted the practices from any other county’s INN projects. This would have the advantage of both evaluating the spread of innovations and also highlighting the fact that counties should be attentive to what other counties are doing with their INN projects. Evaluation Studies There is no Statewide Evaluation study in the Master Plan specifically for INN. Summarizing the results of the local evaluations as well as reporting on how successful the dissemination and adoption of successful INN results has been is recommended below. A full evaluation of the INN component, focused on the overall strategy of introducing system change through the support of local innovations, should eventually be done; but, the likelihood of such a study yielding creditable and useful results within the next few years is not high enough to warrant its being a priority during the planning horizon of this Plan. Other Recommendations Related Specifically to INN Evaluation 1) Support counties to widely disseminate the results of successful INN programs. Results can be disseminated in journal articles, online presentations, videos, public media, at conferences, at MHSOAC meetings, via an online clearinghouse, and at other statewide and regional meetings. The MHSOAC can also publicize the results of local INN evaluations through its channels. 2) Collect information on the successful spread of effective INN programs. A summary indicator of the actual dissemination of successful INN programs could be calculated once an item is added to the Annual Update on counties’ adoption of INN programs whose evaluations were positive. This will be the beginning of a long-range process of determining the usefulness of the INN strategy for creating positive change in the public mental health system. Technological Needs (TN) Background and Context: The Technological Needs (TN) component has not received much statewide attention to date. TN was combined with Capital Facilities (CF) into one CRF/TN component in the MHSA. These two are separated here because there is no evaluation activity pertinent to the capital facilities part. Under a contract with the MHSOAC, the UCLA Center for Healthier Children, Youth, and Families with subcontractor EMT have recently submitted a draft Capital Facilities and Technological Needs Report that reviews expenditures on this component. When this report is finalized by the MHSOAC it will be the first detailed accounting of the uses of the TN funds. MHSOAC Evaluation Master Plan Page 87 The four county site visits conducted as part of the development of this plan suggested that counties are making major resource commitments to upgrading their IT systems. The additional funds provided by MHSA are assisting this effort and may represent one of the untold major benefits from the MHSA. TN in the Master Plan Developmental and Exploratory Work The major evaluation activity for the TN component is Work Effort #7 which calls for the development (and subsequent implementation) of a plan to categorize the TN projects undertaken by the counties, to track the progress on the projects, to relate the projects to overall county IT enhancements, and to explore how to relate IT improvements to an improved service system. A special section of the plan will be devoted to projects intended for the use of IT projects for client and family empowerment. Performance Monitoring The TN plan that is developed is intended to include a method for tracking the progress of the TN projects. If this tracking process includes a simple measure of progress it might be included as an indicator in the performance monitoring system. Other Recommendations Related Specifically to TN Evaluation 1) Collaboration with CAEQRO on the development and implementation of a plan to track the impact of TN funding. As noted in the Developmental and Exploratory Work Effort #7, the CAEQRO includes a review of the county IT systems as part of its routine site visits. The CAEQRO should be included in the development of the TN plan and should be considered as a possible partner in the implementation of the plan. Workforce, Education, and Training (WET) Background and Context: The first WET five-year plan was developed by DMH in 2008 after a thorough needs assessment and stakeholder process. “The Five-Year Plan provides the means for developing and maintaining a culturally competent workforce, to include clients and family members, which is capable of providing client- and family-driven services that promote wellness, recovery and resilience, and lead to measurable, values-driven outcomes.”42 The Plan included goals, objectives, and actions and a series of performance indicators to document the success of the plan’s implementation. The counties submitted and gained approval for their WET plans and reported on their WET activities in Annual Updates, but this information was never summarized. 42 “MHSA Five-Year WET Development Plan,” DMH, 2008. MHSOAC Evaluation Master Plan Page 88 The responsibility for administering the WET component of the MHSA has been transferred to the Office of Statewide Health Planning and Development (OSHPD)43. OSHPD manages a wide range of workforce issues within the health care field allowing for the MHSA WET component to be better aligned with other health care workforce activities. Besides administering the stipend and loan programs, OSHPD manages the contract for the Working Well Together (statewide technical assistance project) and provides the liaison function to the WET Regional Partnerships. Since assuming responsibility OSHPD has made two presentations to the MHSOAC, one in July 2012 on Current Transition Activities and Future Goals and one in September 2012 on Reducing Disparities. Information from the latter showed the distribution of the applicants for the loan stipend program in terms of the percentage of awards to persons from an under-represented group (67% in FY 11-12) and to persons speaking a language other than English (59% in FY 11-12). Similar information was provided about the applicants for the Mental Health Loan Assumption Program (MHLAP) in FY 11-12: 71% to persons from under-represented groups, 60% to persons speaking another language besides English, and 53% to consumers or family members OSHPD has begun a formal planning process for the development of a new WET five-year plan. OSHPD has posted on its website its intended work plan for development of the new five year WET plan. The process began in December 2012 with a presentation on the work plan to the WET Advisory Committee44. Activity under the work plan begins in January 2013 with an evaluation by OSHPD WET staff of the current WET programs and the creation of a Five Year Plan Advisory Sub-Committee. An extensive stakeholder engagement begins in March 2013 to gather input about the vision, goals and objectives, workforce needs, performance indicators, etc. to be included in the new five-year plan. The results of a contractor’s needs assessment are added by August 2013. Drafts of the plan are then circulated to all interested stakeholders through another extensive stakeholder process that extends through October 2013. The WET Advisory Committee approves the Five Year Plan in December 2013 followed by its submission to the CMHPC for review and approval in January 2014. The Plan is then submitted to OSHPD with final approval coming from the Health and Human Services Agency in March 2014. WET in the Master Plan The only reference to the WET component in the Master Plan is the possible inclusion of a performance indicator relative to WET activities (Step 3 in the Performance Monitoring method section). This step as it relates to WET should be postponed until the conclusion of the new Five Year Plan which will be based on a new needs assessment and will likely include possible performance indicators. Other Recommendations Related Specifically to WET Evaluation 1) Urge OSHPD to track the implementation of county WET activity. OSHPD appears to be focused largely on its state role in administering the WET component, i.e. the programs that it directly controls. 43 The Health Professions Education Foundation, a 501(c)(3) non–profit public benefit corporation created by statute 1987. The transfer was officially to OSHPD. 44 MHSOAC has a staff representative on this Advisory Committee. MHSOAC Evaluation Master Plan Page 89 It is not clear how extensive its review of current WET programs will be, i.e. will it be just of the state level programs or also include a review of county WET activity. As a participant on the WET Advisory Committee, MHSOAC should urge OSHPD to include a review of county WET activity in its evaluation of current WET programs and to include clear direction for tracking and evaluating county WET activity as part of the new Five Year Plan. 2) Obtain routine updates from OSHPD on WET activity. The OSHPD has a great deal of knowledge and expertise in the area of work force development and appear committed to the inclusion in its work of the major mental health stakeholders, including the MHSOAC. Duplicating the work of OSHPD would be counterproductive, but ensuring that its oversight of the WET component remains a responsibility of the MHSOAC. This responsibility can be fulfilled by its role on the WET Advisory Committee and through periodic updates from OSHPD. Summary The Evaluation Master Plan accommodates the challenge of evaluating the MHSA as an integrated system while also addressing individual components. This part has focused on the ways in which each of the MHSA components has been incorporated into the overall model’s three evaluation methods and how evaluation questions that are unique to the components can be addressed. A resulting set of component-specific actions are thus part of the Evaluation Master Plan. MHSOAC Evaluation Master Plan Page 90 PART 6: FINAL WORDS This part includes recommendations for other evaluation efforts besides the activities listed in the main part of the Master Plan. The heart of the Master Plan is the activities listed in Part 4 under the three evaluation methods. Some additional actions are included in Part 5 under the special evaluation considerations for the MHSA components. Included here are other recommendations that apply to the following: Overriding issues that affect all elements of the Plan Timing and resource considerations for Plan implementation Overriding Issues That Affect All Elements of the Plan The MHSOAC needs to devote more attention to using evaluation information. Strategy 6 of the MHSOAC Logic Model is “utilize evaluation results for quality improvement.” The translation of evaluation results into recommendations for policy and practice changes is not always a straightforward or easy task. But it is incumbent on the MHSOAC to take this step if it is to maximize the resources it is devoting to evaluation. Completed evaluation reports are not the end of the process; in some ways they are the beginning. Results need to be interpreted and implications drawn. This requires being able to put the evaluation results into the context of California’s public mental health system so that recommendations are appropriately targeted at points where there is leverage for change. The report from each evaluation activity should be accompanied by a staff abstract which not only summarizes the major results but more importantly puts them into context and draws implications for consideration by the Commissioners. This may require consultation with subject experts to draw the most useful information from the evaluation activity results. This recommendation applies not just to evaluation activity conducted by the MHSOAC but also to reports of evaluations done by other entities. A review of past MHSOAC agendas suggests that it receives evaluation reports done by other entities. These presentations should be accompanied by MHSOAC staff consideration of implications and potential recommendations. It should be noted that the MHSOAC is already undertaking a more active role in interpreting results as evidenced by forthcoming Fact Sheets on major evaluation studies and by a clear mandate in the 2013 Evaluation Committee Charter. The MHSOAC must continue to address the data system situation. As noted in the Findings section, the current data bases that the MHSOAC uses for much of its evaluation effort are technologically outdated, inconsistently used by programs and counties, and MHSOAC Evaluation Master Plan Page 91 inadequately supported by the relevant state agencies. The transition of the maintenance of the data bases from the DMH to the DHCS offers the possibility for improvement or a continued slow decline. Energy and focus are currently in place to begin the development of a new data enterprise architecture. It is unclear whether there will be sufficient Administration support to proceed with this effort. While the MHSOAC can support this effort as being in everyone’s best interests, it has the responsibility to also take a more measured approach to the data system issue. Two efforts are strongly recommended: The MHSOAC should recommend to DHCS and the Health and Human Services Agency to sponsor a feasibility study of developing the new system suggested by stakeholders. This would likely be a first step in any case and may seem less daunting than committing to a new system without any real sense of the resource and time parameters that would be entailed. The MHSOAC should continue to monitor and reinforce the importance with DHCS and the Department of Finance of the immediate need to devote sufficient resources to the maintenance and support of the existing data sources in the interim until a new system is in place. Unless they do so, much of the Evaluation Master Plan will be compromised. For many reasons the MHSOAC should involve stakeholders more in its evaluation work. The evaluation activity of the MHSOAC is not as widely known as it could be. One way of engendering a greater awareness is by involving other people who represent constituencies who would be interested in the evaluation work. Participation in an evaluation activity engenders engagement and builds a constituency for the evaluation. The range of subject matters that are included in the Evaluation Master Plan is wide; one would not expect to find the diversity of subject matter expertise within the relatively small MHSOAC staff. The Plan notes in a number of places that subject matter expertise is particularly important for a particular evaluation activity which is another way of broadening the involvement of others. The MHSOAC Evaluation Committee contains invaluable technical expertise at all levels of the mental health system and reflects multiple perspectives on evaluation issues. The advice and involvement of persons on the Committee could be used more widely by the MHSOAC while maintaining a strictly advisory role. People will only continue to devote time and energy to something like Committee membership if they feel they are making a real contribution. Another group of stakeholders who can be more involved are persons with lived experience. The participatory evaluation study conducted by the UCLA Center for Healthier Children, Youth and Families under contract with the MHSOAC, developed a small cadre of such people who after extensive training in research methods were able to participate effectively in the study. The MHSOAC should consider developing a group of evaluation trained persons with lived experience to participate as needed in its evaluation activities. The MHSOAC should consider collaboration with other entities whenever possible. MHSOAC Evaluation Master Plan Page 92 Other organizations and entities have many of the same interests in evaluation as does the MHSOAC. Collaboration on an evaluation activity is not easy, but the benefits can be worth the effort. Joint sponsorship of a study can increase the constituency for the results and potentially cut the expense of a study. And it will increase the credibility of the evaluation. In particular, the CAEQRO and CiMH are active in evaluation activities that are relevant to MHSOAC interests. The MHSOAC might work with the CAEQRO to incorporate some of its performance monitoring and/or information gathering interests into its routine annual county site visits. And the MHSOAC might collaborate with CiMH on topic areas where CiMH has a special expertise or interest, e.g. the children’s EBP evaluation study. Counties are invaluable partners in many of the evaluation activities in the Plan. In some instances the MHSOAC activity involves gathering and summarizing the results of evaluation studies done by counties. In others it requires soliciting volunteers to participate in either an Evaluation Study or Developmental and Exploratory Work Efforts that will likely require additional effort on their part. Counties are usually more than willing to participate in such endeavors so long as they can play a meaningful role in the design and implementation of the evaluation activity and take an active part in the analysis and interpretation of the results. Providers are also potential partners, particularly large ones who have substantial evaluation expertise and who already conduct serious evaluation efforts of their own. Such investments on the part of provider organizations are a testament to their interest in evaluation. The MHSOAC should continue to refine its method of selecting and monitoring contractors. The Commission is in the process of widening the distribution of its RFPs which will increase the number of creditable bidders from which to choose. Consideration should be given to placing a greater weight in the evaluation of bidders to having “knowledge of and experience with California’s mental health system.” California’s mental health system has a rich history, a complicated legal and regulatory structure, and multiple important stakeholders at the state and local level. Familiarity with all this enhances the chance that an evaluation activity will be completed efficiently. While it requires resources, the active involvement of MHSOAC staff in all aspects of a contractor’s work will increase the chances that the results will be creditable and meaningful. While not micromanaging, MHSOAC staff should understand fully the study design, methodology, and data sources that will be used in the study and monitor progress on a regular basis to alleviate any last minute surprises. The evaluation activities will undoubtedly require modification over time. No original plan can or should remain static: this is particularly the case in an environment like the current one which is changing and uncertain. The results of other evaluation projects, like that of the California State Auditor, could require the MHSOAC to conduct an Evaluation Study not contemplated in the Master Plan. Or a decision by the DHCS on what to do about data systems could affect what is proposed in the Master Plan. Even without any such dramatic changes, the MHSOAC may wish to reprioritize the activities in the Plan and/or add new ones. MHSOAC Evaluation Master Plan Page 93 The advantage of having the Master Plan is that evaluation activity can continue in the midst of any such changes. The MHSOAC will not have to start afresh every time there is a change. There will be a structure in place which can be built upon. Timeline and resources The Evaluation Master Plan lays out an ambitious agenda. The Master Plan calls for seven steps in the Performance Monitoring system, 10 Evaluation Studies, and eight Developmental and Exploratory Work Efforts. To accomplish all this in, for example, four years would require starting two to three new evaluation studies and two new developmental and exploratory work efforts a year while managing the ongoing work of maintaining and upgrading the Performance Monitoring system. If, as anticipated, most of the Evaluation Studies and the Developmental and Exploratory Work Efforts extend beyond one year, the MHSOAC could be actively engaged with more than ten projects during the course of a year, not including the ongoing work on the Performance Monitoring system. The Evaluation Master Plan cannot be implemented as envisioned with the existing level of internal resources devoted to evaluation. To maintain and upgrade the Performance Monitoring system will require an ongoing commitment of internal resources with the addition of some subject matter expertise. Each of the Evaluation Studies and Developmental and Exploratory Work Efforts requires at a minimum internal resources to further refine the study or work effort parameters, draft and issue an RFP, review proposals and develop contracts, and monitor the contractor’s work. The recommendations in the Master Plan would require a more active role for internal staff in contract monitoring, more efforts to include subject matter and other stakeholders in the work (requiring coordination), a more intensive effort at interpreting and drawing implications from the results of the evaluation activities, and more involvement in disseminating those implications. In addition to the activities in the Master Plan, internal evaluation staff must attend numerous meetings with other constituencies and stakeholders, coordinate with DHCS and others over data issues, and plan and coordinate the work of the Evaluation Committee. Participation in these additional activities contributes to the quality of current MHSOAC evaluation efforts, but also requires additional time and work on the part of MHSOAC evaluation staff. The speed with which the evaluation activities can be implemented will be a function of the capacity of the internal resources and the amount of funds available for contracts. To accomplish the full set of evaluation activities would require additional funds for contracting as well as an augmentation of internal staff resources. The MHSOAC has generally initiated two or three projects a year funded out of its annual $1 million set aside for evaluation. The Evaluation Plan would require at least two to three times that level of funding for study and work effort contracts and for special contracts for subject experts. MHSOAC Evaluation Master Plan Page 94 In addition to the activities in the Master Plan there may be draws on the MHSOAC evaluation funds from unexpected sources. The most likely is the need to devote resources to the amelioration and then maintenance of the existing data systems. While this is not the responsibility of the MHSOAC it may be incumbent on the Commission to assist the DHCS as it has already been doing with the contract with the California State College at Sacramento for work on the DCR system. The amount of resources devoted to contracts needs to be calibrated with the capacity of the internal staff, or the results from contracts will not be as creditable or useful as they might be. MHSOAC Evaluation Master Plan Page 95 REFERENCES Clark, H.B.R., Koroloff N., Geller J., & Sondheimer D.L. (2008). Research on transition to adulthood: Building the evidence base to inform services and supports for youth and young adults with serious mental health disorder. Journal off Behavioral Health Services & Research, 35(4): 365. Goldstrom, I.D., Campbell, J., Rogers, J.A., Lambert, D.B., Blacklow, B., Henderson, M.J. & Manderscheid, R.W. (2006). National estimates for mental health mutual support groups, self-help organizations, and consumer-operated services. Admin & Policy in Mental Health and Mental Health Services Research, 33 (1): 92-103. Holter, M.C., Mowbray, C.T., Bellamy, C.D., MacFarlane, P., & Dukarski, J. Critical elements of consumer run services: Results of a national survey. Comm Mental Health J, 40(1): 47-63. Kaufmann, C.L., Ward-Colasante, C., & Farmer, J. (1993). Development and evaluation of drop-in centers operated by mental health consumers. Psychiatric Services, 44(7): Lewis, S.E., Hopper, K., & Healion, E. (2012). Partners in recovery: social support and accountability in a consumer-run mental health center. Psychiatric Services, 63(1): 61-5. Minkoff, K. & Cline, C.A. (2004). Changing the world: The design and implementation of comprehensive continuous integrated systems of care for individuals with co-occurring disorders. Psychiatric Clinics of N Amer, 27(4): 727-43. Mowbray, C.T. & Cheribeth, T. (1993). Consumer-operated drop-in centers: Evaluation of operations and impact. J of Mental Health Admin, 20(1): 8-19. Mowray, C.T., Robinson E.A.R., & Holter M.C. (2002). Consumer drop-in centers: Operations, services, and consumer involvement. Health Social Work, 27(4): 248-61. Segal, S.P., Silverman, C.J., & Temkin, T.L. (2010). Self-help and community mental health outcomes: a recovery focused randomized controlled trial. Psychiatric Services, 61(9): 905-10. MHSOAC Evaluation Master Plan Page 96 APPENDIX 1: List of interviewees MHSOAC Commissioners--Larry Poaster, David Pating, Richard Van Horn, Tina Wooten, Ralph Nelson MHSOAC Evaluation Committee members - Denise Hunt, Debbie Innes-Gomberg (also CMHDA), Dave Pilon, Karyn Dresser, Sergio Aguilar-Gaxiola, Stephanie Oprendek (also CiMH), Toby Ewing , Steve Leoni, Karen Stockton (also CMHDA), Tim Smith, Deborah Lee (MHSOAC staff), Kathleen Derby (also NAMI) Mental Health Association of California Council of Community Mental Health Agencies - Rusty Selix, Abram Rosenblatt California Mental Health Planning Council - Jane Adcock, John Ryan California External Quality Review Organization/APS Healthcare – Sandra Sinz, Michael Reiter CalMHSA – Wayne Clark, Bill Arroyo, Stephanie Welch National Association of Mental Health – Kathleen Derby California Mental Health Directors Association– Pat Ryan, Molly Brassil, Don Kingdon, Sandra Santana- Moore California State University, Sacramento – Tim Croisdale, Kate Cordell Department of Mental Health regarding Client and Services Information system – Bryan Fisher Racial and Ethnic Mental Health Disparities Coalition - Stacie Hiramoto California Institute for Mental Health—Sandra Naylor Goodwin, Neal Adams, Jennifer Clancy UCLA Center for children, Youth, and Families (and subcontractors) —Jane Yoo Others - Davis Ja MHSOAC Evaluation Master Plan Page 97 APPENDIX 2: Rankings for Evaluation Studies and Developmental and Exploratory Work The rankings on the criteria for the evaluation studies and the developmental and exploratory work in the Master Plan are shown below. The ranking of the study or work on each criteria was rated as high, medium, or low. Scores were the sum of the ratings (with “3” for high, “2” for medium and”1” for low. EVALUATION STUDIES HIGH PRIORITY Study #1: Individual Level: Collect, summarize, and publicize the outcomes from counties that have gathered such information CRITERIA Rating Evaluation questions Consistency with MHSA: are the questions consistent with language and/or values of the Act? H Potential for quality improvement: will answers to the question(s) lead to suggestions for policy and practice M changes? Importance to stakeholders: are the question(s) of importance to key stakeholders? H Possibility of partners: are there other organizations who would be interested in collaborating and/or partially H funding the evaluation? Context: are there changes in the environment which make the question(s) particularly relevant? L Challenges: do the question(s) address an area which is creating a challenge for the system? M Evaluation methodology Feasibility: how likely is the evaluation method(s) to produce information that answers the evaluation H question(s)? Cost: how many resources are needed to do the evaluation well? H Timeliness: how long will it take to complete the evaluation? H Leveraging: does the evaluation build on prior work by the MHSOAC or others? H Total = 27 Study 2: System Level (Quality) Determine outcomes of selected early intervention and selective prevention programs CRITERIA Rating Evaluation questions Consistency with MHSA: are the questions consistent with language and/or values of the Act? H Potential for quality improvement: will answers to the question(s) lead to suggestions for policy and practice H changes? Importance to stakeholders: are the question(s) of importance to key stakeholders? M Possibility of partners: are there other organizations who would be interested in collaborating and/or partially M funding the evaluation? Context: are there changes in the environment which make the question(s) particularly relevant? H Challenges: do the question(s) address an area which is creating a challenge for the system? M Evaluation methodology Feasibility: how likely is the evaluation method(s) to produce information that answers the evaluation question(s)? H MHSOAC Evaluation Master Plan Page 98 Cost: how many resources are needed to do the evaluation well? M Timeliness: how long will it take to complete the evaluation? M Leveraging: does the evaluation build on prior work by the MHSOAC or others? H Total: 25 Study 3: System Level (Access and Quality) :Determine effectiveness of methods for engaging and serving TAY clients CRITERIA Rating Evaluation questions Consistency with MHSA: are the questions consistent with language and/or values of the Act? H Potential for quality improvement: will answers to the question(s) lead to suggestions for policy and practice H changes? Importance to stakeholders: are the question(s) of importance to key stakeholders? H Possibility of partners: are there other organizations who would be interested in collaborating and/or partially M funding the evaluation? Context: are there changes in the environment which make the question(s) particularly relevant? L Challenges: do the question(s) address an area which is creating a challenge for the system? H Evaluation methodology Feasibility: how likely is the evaluation method(s) to produce information that answers the evaluation M question(s)? Cost: how many resources are needed to do the evaluation well? M Timeliness: how long will it take to complete the evaluation? M Leveraging: does the evaluation build on prior work by the MHSOAC or others? M Total=25 Study 4: System Level (Quality) Determine effectiveness of selected programs for older adults CRITERIA Rating Evaluation questions Consistency with MHSA: are the questions consistent with language and/or values of the Act? H Potential for quality improvement: will answers to the question(s) lead to suggestions for policy and practice H changes? Importance to stakeholders: are the question(s) of importance to key stakeholders? H Possibility of partners: are there other organizations who would be interested in collaborating and/or partially M funding the evaluation? Context: are there changes in the environment which make the question(s) particularly relevant? H Challenges: do the question(s) address an area which is creating a challenge for the system? M Evaluation methodology Feasibility: how likely is the evaluation method(s) to produce information that answers the evaluation H question(s)? Cost: how many resources are needed to do the evaluation well? H Timeliness: how long will it take to complete the evaluation? H Leveraging: does the evaluation build on prior work by the MHSOAC or others? L Total: 26 Study 5: System Level (Quality) Determine scope of implementation and effectiveness of evidence-based practices (EBP) for children and their families. CRITERIA Rating Evaluation questions Consistency with MHSA: are the questions consistent with language and/or values of the Act? H Potential for quality improvement: will answers to the question(s) lead to suggestions for policy and practice H MHSOAC Evaluation Master Plan Page 99 changes? Importance to stakeholders: are the question(s) of importance to key stakeholders? M Possibility of partners: are there other organizations who would be interested in collaborating and/or partially H funding the evaluation? Context: are there changes in the environment which make the question(s) particularly relevant? M Challenges: do the question(s) address an area which is creating a challenge for the system? M Evaluation methodology Feasibility: how likely is the evaluation method(s) to produce information that answers the evaluation H question(s)? Cost: how many resources are needed to do the evaluation well? M Timeliness: how long will it take to complete the evaluation? M Leveraging: does the evaluation build on prior work by the MHSOAC or others? H Total: 25 MEDIUM PRIORITY Study #6: System Level (Quality): Determine the effectiveness of peer-led and consumer run services CRITERIA Rating Evaluation questions Consistency with MHSA: are the questions consistent with language and/or values of the Act? H Potential for quality improvement: will answers to the question(s) lead to suggestions for policy and practice M changes? Importance to stakeholders: are the question(s) of importance to key stakeholders? H Possibility of partners: are there other organizations who would be interested in collaborating and/or partially M funding the evaluation? Context: are there changes in the environment which make the question(s) particularly relevant? L Challenges: do the question(s) address an area which is creating a challenge for the system? M Evaluation methodology Feasibility: how likely is the evaluation method(s) to produce information that answers the evaluation M question(s)? Cost: how many resources are needed to do the evaluation well? M Timeliness: how long will it take to complete the evaluation? M Leveraging: does the evaluation build on prior work by the MHSOAC or others? H Total: 22 Study 7: System Level (Quality): Determine the effectiveness of screening all persons receiving services for substance use issues CRITERIA Rating Evaluation questions Consistency with MHSA: are the questions consistent with language and/or values of the Act? H Potential for quality improvement: will answers to the question(s) lead to suggestions for policy and practice H changes? Importance to stakeholders: are the question(s) of importance to key stakeholders? M Possibility of partners: are there other organizations who would be interested in collaborating and/or partially M funding the evaluation? Context: are there changes in the environment which make the question(s) particularly relevant? H Challenges: do the question(s) address an area which is creating a challenge for the system? M Evaluation methodology Feasibility: how likely is the evaluation method(s) to produce information that answers the evaluation M question(s)? Cost: how many resources are needed to do the evaluation well? L MHSOAC Evaluation Master Plan Page 100 Timeliness: how long will it take to complete the evaluation? M Leveraging: does the evaluation build on prior work by the MHSOAC or others? M Total: 22 Study 8: System Level (Efficiency and Quality): Determine the effectiveness of obtaining routine physical health status measures on clients in FSPs CRITERIA Rating Evaluation questions Consistency with MHSA: are the questions consistent with language and/or values of the Act? H Potential for quality improvement: will answers to the question(s) lead to suggestions for policy and practice H changes? Importance to stakeholders: are the question(s) of importance to key stakeholders? M Possibility of partners: are there other organizations who would be interested in collaborating and/or partially L funding the evaluation? Context: are there changes in the environment which make the question(s) particularly relevant? H Challenges: do the question(s) address an area which is creating a challenge for the system? M Evaluation methodology Feasibility: how likely is the evaluation method(s) to produce information that answers the evaluation M question(s)? Cost: how many resources are needed to do the evaluation well? M Timeliness: how long will it take to complete the evaluation? L Leveraging: does the evaluation build on prior work by the MHSOAC or others? L Total=20 Study 9: System Level (Efficiency) Refine and repeat FSP cost and cost offset study CRITERIA Rating Evaluation questions Consistency with MHSA: are the questions consistent with language and/or values of the Act? H Potential for quality improvement: will answers to the question(s) lead to suggestions for policy and practice L changes? Importance to stakeholders: are the question(s) of importance to key stakeholders? H Possibility of partners: are there other organizations who would be interested in collaborating and/or partially L funding the evaluation? Context: are there changes in the environment which make the question(s) particularly relevant? L Challenges: do the question(s) address an area which is creating a challenge for the system? M Evaluation methodology Feasibility: how likely is the evaluation method(s) to produce information that answers the evaluation H question(s)? Cost: how many resources are needed to do the evaluation well? M Timeliness: how long will it take to complete the evaluation? M Leveraging: does the evaluation build on prior work by the MHSOAC or others? H Total=24 Study 10: Individual Level: Determine outcomes of promising and/or community-based practices being developed by counties, particularly for un-served, underserved, or inappropriately served populations CRITERIA Rating Evaluation questions Consistency with MHSA: are the questions consistent with language and/or values of the Act? H Potential for quality improvement: will answers to the question(s) lead to suggestions for policy and practice H changes? MHSOAC Evaluation Master Plan Page 101 Importance to stakeholders: are the question(s) of importance to key stakeholders? H Possibility of partners: are there other organizations who would be interested in collaborating and/or partially M funding the evaluation? Context: are there changes in the environment which make the question(s) particularly relevant? L Challenges: do the question(s) address an area which is creating a challenge for the system? M Evaluation methodology Feasibility: how likely is the evaluation method(s) to produce information that answers the evaluation M question(s)? Cost: how many resources are needed to do the evaluation well? M Timeliness: how long will it take to complete the evaluation? L Leveraging: does the evaluation build on prior work by the MHSOAC or others? M Total: 21 DEVELOPMENTAL AND EXP0LORATORY WORK HIGH PRIORITY Work Effort #1: PEI: Determine status of county efforts to evaluate one PEI project and make recommendations, as needed, to ensure adequate evaluations. CRITERIA Rating Evaluation questions Consistency with MHSA: are the questions consistent with language and/or values of the Act? H Potential for quality improvement: will answers to the question(s) lead to suggestions for policy and practice H changes? Importance to stakeholders: are the question(s) of importance to key stakeholders? M Possibility of partners: are there other organizations who would be interested in collaborating and/or partially M funding the evaluation? Context: are there changes in the environment which make the question(s) particularly relevant? L Challenges: do the question(s) address an area which is creating a challenge for the system? M Evaluation methodology Feasibility: how likely is the evaluation method(s) to produce information that answers the evaluation H question(s)? Cost: how many resources are needed to do the evaluation well? M Timeliness: how long will it take to complete the evaluation? H Leveraging: does the evaluation build on prior work by the MHSOAC or others? M Total= 23 Work Effort 2: PEI: Develop an ongoing method for describing and cataloguing programs funded by PEI CRITERIA Rating Evaluation questions Consistency with MHSA: are the questions consistent with language and/or values of the Act? H Potential for quality improvement: will answers to the question(s) lead to suggestions for policy and practice M changes? Importance to stakeholders: are the question(s) of importance to key stakeholders? H Possibility of partners: are there other organizations who would be interested in collaborating and/or partially M funding the evaluation? Context: are there changes in the environment which make the question(s) particularly relevant? M Challenges: do the question(s) address an area which is creating a challenge for the system? H Evaluation methodology MHSOAC Evaluation Master Plan Page 102 Feasibility: how likely is the evaluation method(s) to produce information that answers the evaluation M question(s)? Cost: how many resources are needed to do the evaluation well? M Timeliness: how long will it take to complete the evaluation? M Leveraging: does the evaluation build on prior work by the MHSOAC or others? M Total=23 Work Effort 3: System level (Quality, efficiency): Explore feasibility of classifying FSP programs in a meaningful and useful fashion CRITERIA Rating Evaluation questions Consistency with MHSA: are the questions consistent with language and/or values of the Act? H Potential for quality improvement: will answers to the question(s) lead to suggestions for policy and practice H changes? Importance to stakeholders: are the question(s) of importance to key stakeholders? H Possibility of partners: are there other organizations who would be interested in collaborating and/or partially M funding the evaluation? Context: are there changes in the environment which make the question(s) particularly relevant? L Challenges: do the question(s) address an area which is creating a challenge for the system? M Evaluation methodology Feasibility: how likely is the evaluation method(s) to produce information that answers the evaluation H question(s)? Cost: how many resources are needed to do the evaluation well? M Timeliness: how long will it take to complete the evaluation? M Leveraging: does the evaluation build on prior work by the MHSOAC or others? M Total: 23 MEDIUM PRIORITY Work Effort 4: Community level: Develop indicators for the community level CRITERIA Rating Evaluation questions Consistency with MHSA: are the questions consistent with language and/or values of the Act? H Potential for quality improvement: will answers to the question(s) lead to suggestions for policy and practice L changes? Importance to stakeholders: are the question(s) of importance to key stakeholders? M Possibility of partners: are there other organizations who would be interested in collaborating and/or partially L funding the evaluation? Context: are there changes in the environment which make the question(s) particularly relevant? M Challenges: do the question(s) address an area which is creating a challenge for the system? L Evaluation methodology Feasibility: how likely is the evaluation method(s) to produce information that answers the evaluation H question(s)? Cost: how many resources are needed to do the evaluation well? H Timeliness: how long will it take to complete the evaluation? H Leveraging: does the evaluation build on prior work by the MHSOAC or others? M Total: 21 MHSOAC Evaluation Master Plan Page 103 Work Effort 5: Person level: Develop system to track outcomes for adults45 in less intensive services than FSPs CRITERIA Rating Evaluation questions Consistency with MHSA: are the questions consistent with language and/or values of the Act? H Potential for quality improvement: will answers to the question(s) lead to suggestions for policy and practice M changes? Importance to stakeholders: are the question(s) of importance to key stakeholders? H Possibility of partners: are there other organizations who would be interested in collaborating and/or partially M funding the evaluation? Context: are there changes in the environment which make the question(s) particularly relevant? M Challenges: do the question(s) address an area which is creating a challenge for the system? M Evaluation methodology Feasibility: how likely is the evaluation method(s) to produce information that answers the evaluation M question(s)? Cost: how many resources are needed to do the evaluation well? M Timeliness: how long will it take to complete the evaluation? L Leveraging: does the evaluation build on prior work by the MHSOAC or others? M Total: 21 Work Effort 6: Person and system (Quality) levels: Determine the interaction between the characteristics of the populations served in FSPs and the outcomes obtained CRITERIA Rating Evaluation questions Consistency with MHSA: are the questions consistent with language and/or values of the Act? H Potential for quality improvement: will answers to the question(s) lead to suggestions for policy and practice H changes? Importance to stakeholders: are the question(s) of importance to key stakeholders? M Possibility of partners: are there other organizations who would be interested in collaborating and/or partially M funding the evaluation? Context: are there changes in the environment which make the question(s) particularly relevant? L Challenges: do the question(s) address an area which is creating a challenge for the system? M Evaluation methodology Feasibility: how likely is the evaluation method(s) to produce information that answers the evaluation M question(s)? Cost: how many resources are needed to do the evaluation well? M Timeliness: how long will it take to complete the evaluation? L Leveraging: does the evaluation build on prior work by the MHSOAC or others? L Total: 21 Work Effort #7. (Infrastructure - TN) Develop and implement a plan for routine monitoring and special studies of the impact of technological need (TN) expenditures. CRITERIA Rating Evaluation questions Consistency with MHSA: are the questions consistent with language and/or values of the Act? H Potential for quality improvement: will answers to the question(s) lead to suggestions for policy and practice M 45 The work is limited to adults at least in this initial stage because of the more defined structure of the FSP services for this age group. The greater diversity in FSPs for the other age groups makes the value of this work less clear at this point in time. Level of care discussions have been most relevant for the adult system of care. MHSOAC Evaluation Master Plan Page 104 changes? Importance to stakeholders: are the question(s) of importance to key stakeholders? M Possibility of partners: are there other organizations who would be interested in collaborating and/or partially H funding the evaluation? Context: are there changes in the environment which make the question(s) particularly relevant? M Challenges: do the question(s) address an area which is creating a challenge for the system? L Evaluation methodology Feasibility: how likely is the evaluation method(s) to produce information that answers the evaluation M question(s)? Cost: how many resources are needed to do the evaluation well? L Timeliness: how long will it take to complete the evaluation? M Leveraging: does the evaluation build on prior work by the MHSOAC or others? H Total=21 Work Effort #8: System (Quality): Explore the extent of and variation in the recovery orientation of programs CRITERIA Rating Evaluation questions Consistency with MHSA: are the questions consistent with language and/or values of the Act? H Potential for quality improvement: will answers to the question(s) lead to suggestions for policy and practice H changes? Importance to stakeholders: are the question(s) of importance to key stakeholders? H Possibility of partners: are there other organizations who would be interested in collaborating and/or partially M funding the evaluation? Context: are there changes in the environment which make the question(s) particularly relevant? L Challenges: do the question(s) address an area which is creating a challenge for the system? M Evaluation methodology Feasibility: how likely is the evaluation method(s) to produce information that answers the evaluation M question(s)? Cost: how many resources are needed to do the evaluation well? L Timeliness: how long will it take to complete the evaluation? M Leveraging: does the evaluation build on prior work by the MHSOAC or others? M Total= 21 Endnotes MHSOAC Evaluation Master Plan Page 105 i The list of behavioral health performance measures for the HEDIS 13 are the following: Anti-depression medication management Follow-up care for children prescribed ADHD medications Follow-up after hospitalization for mental illness Diabetes screening for person with schizophrenia or bipolar using antipsychotic medications Diabetes monitoring for persons with schizophrenia and diabetes Adherence to antipsychotic medications for individuals with schizophrenia Cardiovascular monitoring for persons with schizophrenia and cardiovascular disease Identification of alcohol and other drug services Mental health utilization ii The list of behavioral health performance measures recommended by the NQF MAP process are the following: Initiation and engagement of alcohol and other drug dependence treatment Medical assistance with smoking and tobacco use cessation Tobacco use screening Tobacco use cessation Adherence to antipsychotic medications for individuals with schizophrenia Diabetes screening for people with schizophrenia or bipolar disorder who are prescribed antipsychotic medications Cardiovascular health screening for people with schizophrenia or bipolar disorder who are prescribed antipsychotic medications Cardiovascular health monitoring for people with cardiovascular disease and schizophrenia Diabetes monitoring for people with diabetes and schizophrenia Follow-up after hospitalization for schizophrenia (7- and 30-day) Follow-up after hospitalization for mental illness MHSOAC Evaluation Master Plan Page 106