EvaluationMasterPlan

7/16/2012 EEVVAALLUUAATTIIOONN MMAASSTTEERR PLAN MH S O AC EVA L U A TI O N C O MMI TTE E 1 MA Y 23, 2012 AGENDA • Background and purpose/scope • Process • Discussion, Questions, Advice 2 BACKGROUND AND PURPOSE/SCOPE • Current situation • Significant effort is being devoted to multiple evaluation efforts • MHSOAC, CalMHSA, CiMH, EQRO, consumer surveys, URS reporting, 1115 Waiver Needs Assessment • Purposes and audiences vary •• MMeennttaall hheeaalltthh eennvviirroonnmmeenntt iiss uunnuussuuaallllyy uunncceerrttaaiinn • Demise of DMH • Changes in county responsibilities – 1115 expanded eligibility, forensic realignment • Health reform • Purpose and scope of master plan development • Assist MHSOAC determine its evaluation work plan over a reasonable planning horizon • Place this within a context of other evaluation efforts 3 1 7/16/2012 BACKGROUND AND PURPOSE/SCOPE • Why this is hard • Needs for information are different depending on who is using the information and for what purpose • Tension between evaluation for purposes of compliance, oversight, reporting to Legislature, public, policy making , and quality improvement • Needs are different if viewed from perspective of state or county or pprroovviiddeerr oorr ccoonnssuummeerr//ffaammiillyy mmeemmbbeerr • Sophistication about evaluation and how to use information is mixed • There are questions about usefulness of existing evaluation efforts • Information already available is not always used • Multiple uncoordinated evaluation efforts underway • We don’t yet know how useful current evaluation efforts will be • Challenges abound about data systems • Are costly to maintain and support • Many question the accuracy of some of the data systems • We currently have multiple data sources covering different populations 4 BACKGROUND AND PURPOSE/SCOPE • Why it is important to address these issues now • With the demise of DMH, essential to ensure the data systems are continued with adequate support • With fiscal pressures need to coordinate and plan eevvaalluuaattiioonn eeffffoorrttss ttoo eennssuurree eeffffiicciieenntt uussee ooff rreessoouurrcceess • As more data becomes available have to increase our capacity to use the information productively • Principles for developing the plan • Guided by needs of the users • Simple and straight forward • Address issues of data reliability and quality • Be incremental 5 PROCESS • Information gathering • Reports of evaluation activity already undertaken, in process, or planned • Interviews with key stakeholders • Summary review of data systems – populations included, data eelleemmeennttss , eessttiimmaatteess ooff aaccccuurraaccyy, lleevveell ooff eeffffoorrtt ttoo mmaaiinnttaaiinn and support, potential usefulness • Evaluation activity in selected other states – how organized, how resourced, what kinds of information collected, how results are used • Evaluation activity in selected counties – how organized and resourced, what kinds of evaluation efforts, how data and information are used • Review of national evaluation requirements, trends, and opportunities 6 2 7/16/2012 PROCESS • Consideration of options – not yet clear on how best to frame these • Kinds of activities • Data system maintenance and support • Data analysis capacity • Data reporting capacity • Program evaluation opportunities: ranging from controlled trials to qualitative exploration • Analysis of system transformation and adherence to MHSA values • Analysis of information for policy recommendations • Required oversight activities – EQRO, URS, consumer satisfaction • Performance indicators • Training and support for users of information • Who should be responsible for each • State and/or counties • If state, what role for MHSOAC and others • How much coordination of effort 7 PROCESS • Plan • Draft • Review • Final • TTii mellii ne – bb y endd off call enddar year 8 DISCUSSION/QUESTIONS/ADVICE • What works, i.e. examples of evaluation information being used productively for either oversight or quality improvement or justifying mental health expenditures? • HHow ddo we jju sttiiffy siig niiffiic antt expenddiitt ures ffor evaluation in the face of a shrinking service system? • Are we using productively information we already have about the mental health system? 9 3