Ucla EvaluationPhases2And3

2 Statewide Mental Health Services Act Evaluation Phases II and III 2102/81/6 University of California at Los Angeles (UCLA) Evaluation, Management & Training Associates, Inc. (EMT) 1 Overview • Introductions • MHSOAC 2 MHSOAC • UCLA/EMT • Evaluation Framework • Deliverables • Orientation to Evaluation • CDC Framework • Utilization Focused Evaluation 2102/81/6 • Utilization‐Focused Evaluation • Participatory Research 2 1 Overarching Evaluation Framework • Present the framework within which our team will: W k ith lti l t k h ld ( li t f ili 2 • Work with multiple stakeholders (e.g., clients, families, counties) and report information relevant to different categories of users • Map information needs in complex service systems providing continuous services • Work with integrated data collection systems and conduct analysis that supports continuous quality 2102/81/6 y pp q y improvement • Partner with key stakeholders to develop recommendations for a performance monitoring system with respect to processes, outcomes and 3 impacts Objectives of Statewide MHSA Evaluation Contract ‐Phase II • Deliverable #1 • Report of Activities and Costs of Local MHSA Funds 2 p • Deliverable #2 102/81/6 • Reports on Prioritized Indicators • Deliverable #3 • Summary and Synthesis of Existing Evaluations on CSS and PEI 2 • Deliverable #4 • Final Report 4 2 Objectives of Statewide MHSA Evaluation Contract ‐Phase III • Deliverable #1 • FFSSPP CCoosstt ‐OOffffsseett aanndd BBeenneeffiitt RReeppoorrttss • Deliverable #2 • Reports on General System Development Impact on Individual and System (all age groups) • Reports on Consumer, Family and Caregiver Involvement in MHSA and its Impact on Client Outcomes •• PPaarrttiicciippaattoorryy RReesseeaarrcchh – ssiiggnniiffiiccaanntt aassppeecctt ooff PPhhaassee IIIIII , bbuutt ccoonnssiisstteenntt with the UCLA/EMT utilization‐focused evaluation approach to Phases II and III • Deliverable #3 • Final Report and Recommendations on Next Steps for Evaluation UCLA Center for Healthier Children, Families and Communities (CHCFC) • Is a multidisciplinary research, policy and training institute at UCLA with faculty from the UCLA Schools of Medicine, Public Health, and Public Affairs. • The mission of the UCLA CHCFC is to promote children's and families lifelong health, mental health,, developpm ent and well‐beingg byy creatingg and translating innovative ideas and research into optimal environments, systems and policies. 22102/81/6 5 22102/81/6 6 3 Evaluation y Management y Training (EMT) • EMT’s Mission: d f l h • Kick off meeting scheduled 3/30 2 • To promote and facilitate the use of relevant information to improve social policy and enhance the resolution of social problems • Evaluated over 50 CBO recovery programs for adults • All programs serving people of color T d d f l i 2102/81/6 • Two decades of evaluation partnership with Tribal communities • e.g., Nez Perce, several Alaska Native Tribes. • Currently partnering with ICF on 7 national Native Aspirations effort Statewide MHSA Evaluation Contracts – Launch Dates • Phase II 2 • Contract with MHSOAC executed on 2/7 • Kick off meeting held 1/27 • MHSOAC Evaluation Committee Chair also joined the meeting • Phase III • Kick off meeting scheduled 3/30 2102/81/6 8 4 Our Orientation to Evaluation and the Statewide MHSA Evaluation • Utilization‐focused (Patton, 2008) • Focus on producing information which is useful for 2 Focus on producing information which is useful for decision makers and other key stakeholders • End‐users of evaluation data are integrally involved in 102/81/6 development and implementation of the evaluation • Data collected and feedback delivered must be: • Meaningful • Useful 2 • Timely 9 22102/81/6 Evaluation Framework 10 http://www.cdc.gov/mmwr/preview/mmwrhtml/rr4811a1.htm (36 of 52)2/13/2007 5 Statewide Evaluation Phases But we are dropping in after the Starting from the beginning….. horse has left the barn! Logical Sequence of Activities Firs t Delive rable Sta rtswith Ste p 4 stakeholder groups and recommendations may vary, we will make the final call when conflicting recommendations for revisions are made 2 Logical Sequence of Activities First Deliverable Starts with Step 4 1. Engage stakeholders 4. Gather credible evidence (existing reports about CSS and PEI outcomes) 2. Describe the program 5. Justify conclusions (write a 3. Focus the evaluation design summary report about CSS and PEI 4. Gather credible evidence outcomes) 5. Justify conclusions CSS (FSP) Report Due May 1, 2011 6 Ensure use and share lessons PEI Report Due August 31 2011 2102/81/6 6. Ensure use and share lessons PEI Report Due August 31, 2011 learned 11 But what about engaging stakeholders? Stakeholder Engagement Principles: 2 • Stakeholder groups will generally have time to review and provide feedback • All recommendations will be considered • Because we will be receiving feedback from different stakeholder groups and recommendations may vary 2102/81/6 12 6 Engaging Stakeholders Examples (there will be many more!) Potential Stakeholder Product Role 2 Group Advisory Board, Regional Conceptual Framework Review, Feedback Coordinators Client and family Draft reports including Review, Feedback members** PEI and FSP CMHDA IDEA Recommended Committee, Evaluation Outcomes (backed by Experts (e g Brian data available and 102/81/6 Review, Feedback 2 Experts (e.g., Brian data available and Yates, Steve Hahn‐Smith, accessible in existing UCSD Team Members, data sources) for FSP San Francisco County Cost‐Offset and Benefit Team) Analysis 13 ** Still need to identify optimum process for identifying and collaborating with existing stakeholder groups Evaluation Framework • Building Collaboration 2 • Identify and Engage Key Stakeholder Groups • Individuals who are or have been involved with the 102/81/6 public mental health system, their families, and caregivers • County mental health management • County service providers 2 • Advisory Board 14 7 Participatory Research • Example: • Priority Indicators 2 • Proposing variables from existing measures that can be compiled to create measures • Proposing creation of system‐level measures through coding of existing reports augmented with key stakeholder interview data (e.g., web survey of select group of county mental health t ff) 2102/81/6 staff) • Draft matrices will be vetted through existing key stakeholder groups for review and feedback • Draft web survey will be vetted through existing key 15 stakeholder groups for review and feedback Participatory Process • Example: • Logic Model f d l 2 • Map information needs in complex system • Part of “Defining the Program” Process • Overarching conceptual framework and related evaluation logic models graphically depicting each component of the study, instruments, data sources • Participatory research process will involve b i i d ft t i ti t k h ld f 2102/81/6 bringing drafts to existing stakeholder groups for review and feedback 16 8 What to expect… in a nutshell • We will use existing data whenever and wherever possible • A lot of data has been collected –we h ave been tasked with analyzing it across the counties and regions • When we create new measures, we will do so out of existing data whenever and wherever possible • We understand that each county has unique features and characteristics, and that these features are not reflected in cross‐site reports linked to client impact •• TThhiiss ssuuggggeessttss tthhaatt tthheerree mmaayy bbee aa nneeeedd ttoo ggaatthheerr aaddddiittiioonnaall system‐level data (e.g., recovery‐orientation) • We will provide plenty of notice before launching a web or telephone survey (at least 30 days), and we will try to provide adequate time for stakeholders to respond 2 102/81/6 Questions? 2 18 22102/81/6 17 9