Measuring CalWORKs Performance

Measuring CalWORKs Performance MAC TAYLOR • L E G I S L A T I V E A N A L Y S T • SEPTEMBER 2017 AN LAO REPORT 2 Legislative Analyst’s Office www.lao.ca.gov AN LAO REPORT EXECUTIVE SUMMARY The California Work Opportunity and Responsibility to Kids (CalWORKs) program represents one of the state’s major efforts to assist low-income families with children. This report examines the state’s approach to assessing the performance of CalWORKs, in light of recent legislative action to establish a new framework for performance measurement in the program. Measuring Program Performance: Why and How? Various Rationales for Performance Measurement. There are several rationales for the Legislature to engage in performance measurement, including to (1) evaluate whether a program is meeting its objectives, (2) verify whether program implementation is consistent with legislative intent, (3) communicate statewide priorities for a program, and (4) encourage continual improvement in program performance. What Can Be Measured? There are three main types of information that can be measured when assessing a program’s performance—inputs, processes (also referred to as outputs), and outcomes. Inputs represent the resources provided for a program to operate, processes refer to the specific activities undertaken in a program, and outcomes refer to the results of a program—the objectives the program is intended to achieve. Ideally, Performance Measurement Should Focus on Outcomes. In practice, there are a variety of practical issues that arise in meaningfully measuring outcomes. However, when approached with appropriate caution, including outcomes in performance measurement puts the focus on a program’s end goals, rather than simply on the processes used to try to achieve those goals. Performance Measurement in CalWORKs Is Relatively Limited Federal Work Participation Rate (WPR) Is Driving Factor in Current Performance Measurement. Currently, performance measurement in CalWORKs is relatively limited and emphasizes the WPR, the federal government’s only performance measure for the program. This emphasis largely results from fiscal penalties the state and counties face if federal targets for the WPR are not met. At the same time, the state has enacted policies in CalWORKs that reflect priorities beyond the WPR, some of which have the potential to work at cross-purposes with meeting federal targets. This creates the need to balance state priorities with the federal WPR requirement. Beyond WPR, Current Reporting Emphasizes Process. Beyond the WPR, data reported on the CalWORKs program emphasize processes over outcomes. The state has made some attempts to increase the focus on outcomes in performance measurement in the past. However, for a variety of reasons, including past fiscal challenges, these efforts are currently not fully implemented. Legislature Recently Enacted New Performance Measurement System In recent budget legislation, the Legislature established a framework for a new performance measurement system for CalWORKs, to be known as the CalWORKs Outcomes and Accountability www.lao.ca.gov Legislative Analyst’s Office 3 AN LAO REPORT Review (Cal-OAR). Under Cal-OAR, data on various performance indicators will be collected and published and counties will regularly undergo self-assessment and develop system improvement plans with targets for the performance indicators. Budget legislation directs the Department of Social Services to convene a workgroup beginning in the fall of 2017 to develop plans for how Cal-OAR will operate. Issues for Consideration In concept, we think that Cal-OAR will significantly increase performance measurement in CalWORKs and has the potential to place greater emphasis on outcomes. We outline several issues for consideration by the workgroup and the Legislature as planning and implementation of Cal-OAR progress in coming months and years. Aligning Cal-OAR Performance Measures With Other Federal Workforce Programs. In choosing a set of performance measures for Cal-OAR, we suggest that the workgroup give priority consideration to “common measures” used in federal workforce programs authorized by the Workforce Opportunity and Innovation Act. These common measures are consistent with the goals of CalWORKs and including these measures in Cal-OAR would increase the consistency with which performance is measured across workforce programs in the state. Taking Steps to Account for Other Local Factors That Affect Outcomes. As a way to address some of the practical challenges of outcome measurement, some programs use statistical techniques to adjust program performance and set targets to reflect local conditions that are beyond the program’s control. As it plans for meaningful outcome measurement in Cal-OAR, the workgroup could consider what role similar statistical adjustments might play in CalWORKs. We identify several potential ways such techniques could be used. Balancing State and Federal Priorities in Cal-OAR. As a way of striking a balance between federal and state priorities in CalWORKs, the workgroup could consider including the WPR as one performance measure in Cal-OAR, alongside other measures that align more directly with state priorities. Additionally, the Legislature may wish to consider revisiting county fiscal penalties associated with the WPR in the future and consider ways to account for both Cal-OAR performance measures and the WPR. Creating More Uniform County Data Collection and Reporting. Much of the data on CalWORKs are housed in one of three automation systems used by counties to administer the program. Due to recent federal guidance, counties will be moving to a single automation system by 2023. We recommend that the Legislature require the administration to report on how the transition to a single automation system will increase uniformity of data collection and streamline data sharing between the state and counties. Cleanup Needed to Align Past Actions With Cal-OAR. Once implemented, Cal-OAR will overlap with some previous actions taken by the Legislature related to performance measurement in CalWORKs, requiring cleanup to address aspects of these previous actions that are now redundant or unnecessary. 4 Legislative Analyst’s Office www.lao.ca.gov AN LAO REPORT INTRODUCTION CalWORKs Assists Very Low-Income Families to a federal performance measure known as the With Children. The California Work Opportunity “work participation rate” (WPR)—rather than on and Responsibility to Kids (CalWORKs) program the program’s end results, or outcomes. represents one of the state’s major efforts to assist Recent Budget Legislation Creates CalWORKs low-income Californians, specifically families with Outcomes and Accountability Review (Cal-OAR). children. During 2016-17, the program served an As part of the human services trailer bill of the average of nearly 460,000 families each month, 2017-18 budget package, Chapter 24 of 2017 with almost 880,000 children receiving assistance. (SB 89, Committee on Budget and Fiscal Review), With $2.8 billion in estimated spending from state the Legislature adopted a framework for a new and local funding sources during the same year performance measurement and accountability (and an additional $2.4 billion in federal spending), system for CalWORKs that will be known as CalWORKs represents a significant allocation of Cal-OAR. In this report, we outline the rationales the state’s resources. Legislative oversight of the for measuring program performance and CalWORKs program’s effectiveness is important to what kinds of things can and ideally should be promote the best possible outcomes for CalWORKs measured. We then provide an overview of the recipients and the efficient use of state resources. current approach to performance measurement Performance Measurement in CalWORKs in CalWORKs as well as the major features of Is Relatively Limited. Currently, performance Cal-OAR. Finally, we raise several issues to be measurement in CalWORKs is relatively limited, considered as the Cal-OAR system is further and largely focuses on processes involved with the developed and implemented in the coming years. program’s operation—particular emphasis is given BACKGROUND CalWORKs Provides Monthly Cash Grant reimbursement for transportation and certain and Welfare-to-Work Services. The CalWORKs other expenses. Adults enrolled in CalWORKs are program provides financial assistance in the form limited to a lifetime maximum of 48 months of of a monthly cash grant to families with children assistance. Once adults in a case reach this limit, whose income is inadequate to meet their basic the children in the case may continue to receive a needs. Unless they qualify for an exemption (for monthly cash grant, but that grant is reduced by example, to care for a disabled family member), the adults’ portion. adults in CalWORKs cases are generally subject to Counties Are Responsible for Local a requirement that they be employed or participate CalWORKs Administration. The CalWORKs in specified activities—known as “welfare-to-work program is locally administered by the state’s activities”—intended to lead to employment. 58 counties. The California Department of Social Individuals subject to the work requirement Services (DSS) oversees operations in the counties. are entitled to receive services to help meet the While some aspects of county CalWORKs requirement, including subsidized child care and programs are required by law, such as subsidized www.lao.ca.gov Legislative Analyst’s Office 5 AN LAO REPORT child care, counties do have flexibility in what CalWORKs Implemented in Response to types of welfare-to-work activities and other Creation of Federal Temporary Assistance for services are offered to CalWORKs participants. For Needy Families (TANF) Program. CalWORKs example, not all counties operate programs that was created in 1997 in response to the 1996 federal place CalWORKs participants in jobs where the welfare reform legislation that created the federal wages are partially or fully covered by the program, TANF program. The federal TANF program commonly referred to as “subsidized employment.” provides flexible block grant funding to states to Counties Rely on One of Three Automation implement programs that are consistent with the Systems to Administer CalWORKs. To perform federal goals for TANF, described below. eligibility determinations and other administrative State and Federal Goals for CalWORKs. State functions in CalWORKs (as well as other major health law establishes the first goal of the CalWORKs and human services programs), counties rely on one program to be reducing child poverty, followed of three automation systems, each of which serves by achieving the four goals of the federal TANF a consortium of counties. These three automation program, listed in Figure 1. systems house most of the data that are currently Federal Performance Requirements for TANF reported to the state on the CalWORKs program. Limited to the WPR. One of the relatively few The state is currently in the process of reducing the requirements the federal TANF program places number of automation systems from three down to on states is that, at a minimum, 50 percent of two, a process expected to be completed by 2020. families that receive assistance must be working Recently, federal agencies mandated that the state or engaged in welfare-to-work activities for a migrate all counties to a single statewide automation minimum number of hours each week. (For system by 2023 in order for the federal government two-parent families, federal law specifies a higher to continue to contribute funding for a portion of the minimum requirement of 90 percent.) Federal law systems’ maintenance costs. outlines which activities are allowable in order Figure 1 State and Federal Goals for CalWORKs State Statutory Goals of CalWORKs 9 Reduce child poverty in the state. 9 Achieve the goals of TANF. 9 Meet the requirements of federal law. Federal TANF Goals 9 Provide assistance to needy families so that children can be cared for in their own homes. 9 Reduce the dependence of needy parents by promoting job preparation, work, and marriage. 9 Prevent and reduce the incidence of out-of-wedlock pregnancies. 9 Encourage the formation and maintenance of two-parent families. TANF = Temporary Assistance for Needy Families. 6 Legislative Analyst’s Office www.lao.ca.gov AN LAO REPORT for a family’s participation to count toward the federal performance measure for state TANF requirement. The state is free to define different programs. States that fail to comply with the WPR rules for allowable welfare-to-work activities (as requirement may face fiscal penalties in the form described later on, California has defined different of reductions in TANF block grant funding that rules for allowable activities), so long as the grow with each successive year of noncompliance. percentage of families meeting the participation For states that face penalties, federal law outlines requirement by engaging in activities allowable administrative processes for states to appeal under federal law exceeds the minimum the penalty amount, claim good cause for threshold. This requirement—referred to as noncompliance, or enter corrective action plans to the WPR requirement—represents the primary reduce or eliminate penalty amounts before they are enforced. MEASURING PROGRAM PERFORMANCE: WHY AND HOW? In this section, we describe the reasons To Verify Whether Program Implementation performance measurement in public programs Is Consistent With Legislative Intent. In addition like CalWORKs is worth pursuing and discuss to establishing the objectives of a program, some of the ways that performance measurement the Legislature sometimes specifies how those can be implemented. objectives should be pursued. Performance measurement can help the Legislature assess Rationales for Performance Measurement whether a program has been implemented in There are several rationales for the Legislature a manner that is consistent with state law and to engage in measuring program performance, as legislative intent. For example, state law specifies described below. that CalWORKs participants are entitled to To Evaluate Whether the Program Is Meeting receive child care subsidies to enable them to the Legislature’s Objectives. All state programs meet the program’s work requirement. As a result, are intended to meet certain objectives or address the Legislature may wish to measure the extent certain problems. Measuring the extent to which a to which CalWORKs participants have access to program is meeting its objectives is central to the child care subsidies. Legislature’s ability to identify and make changes To Communicate Statewide Priorities for to the program to increase its effectiveness or, the Program. State programs often have multiple in some cases, discontinue the program if it is objectives. Measuring performance relative to a determined that it cannot meet its objectives. particular objective will tend to focus attention For example, in the case of CalWORKs, the on that objective. The Legislature’s choice of Legislature has an interest in understanding which objectives are measured and how they are whether the program achieves the intended measured can play a role in communicating which outcomes of reducing child poverty and reducing objectives should take the highest priority. For the dependence of needy parents. example, as has been noted, the WPR is the federal government’s primary performance measure for www.lao.ca.gov Legislative Analyst’s Office 7 AN LAO REPORT state TANF programs. As will be discussed in CalWORKs program, inputs include things like more detail later, the state’s priorities, as reflected funding provided for cash grants; funding provided in CalWORKs policies, do not always align with for county administration of the program; and the WPR. However, the absence of performance the caseworkers, facilities, and contracted services measures related to these state priorities means paid for with these funds. Input measures quantify that the state and counties have tended to focus program components like funding and staff, attention on the WPR and the federal priorities and often appear in budgetary documents. Input it represents, potentially to the detriment of state measures provide some information about how the priorities. program is operating but little to no information To Encourage Continual Improvement in about whether it is meeting its objectives. When Program Performance. Performance measurement considered with outcomes, inputs can play a critical can be used to encourage continual program role, however, in assessing how efficiently the improvement if it is paired with goalsetting program is operated, by identifying the resources processes that establish reasonable targets for required to achieve observed results. performance, identify specific actions to take to Processes. Processes refer to the specific achieve those targets, and measure progress toward activities undertaken by the program that are meeting targets. intended to achieve the program’s objectives. Processes can be thought of as the operations of What Can Be Measured? the program that are made possible through the In general, there are three main types of inputs provided. Process measures quantify these information that can be measured when assessing program operations. Examples of process measures a program’s performance—(1) inputs, (2) processes include things like the number of families receiving (also referred to as outputs), and (3) outcomes. cash assistance, the number of adults participating These three key types of program information are in various welfare-to-work activities, or the number interrelated, as shown in Figure 2. of children enrolled in subsidized child care. Like Inputs. Inputs are the resources provided inputs, processes are relatively easy to quantify. for a program to operate. For example, in the Process measures also provide information about Figure 2 CalWORKs Inputs and Processes Contribute to Program Outcomes Inputs Processes Outcomes • Amount of funding budgeted • Number of families receiving • Family income relative to for cash assistance. cash assistance. poverty thresholds. • Amount of funding budgeted • Number of adults participating • Employment status of families. for county administration and in welfare-to-work activities. services. • Growth in earnings received • Number of adults receiving by families. • Number of caseworkers. services to address barriers to employment. • Whether families return to aid. 8 Legislative Analyst’s Office www.lao.ca.gov AN LAO REPORT how a program is operating, but generally do Outcomes Are Affected by Factors Other not provide direct information about whether a Than the Program . . . Even when data are program is meeting its objectives. available, another significant challenge with Outcomes. Outcomes refer to the results of the outcome measurement is that outcomes are program—the objectives the program is intended to generally affected by other factors in addition to achieve. For example, some outcome measures for the program’s operation. For example, whether a CalWORKs might include the income of families family enrolled in CalWORKs obtains employment relative to poverty thresholds following their that pays high enough wages for the family to participation in the program, their employment no longer require assistance depends not just on status, growth in their wage levels, and the services received through CalWORKs, but also extent to which families return to assistance in on the characteristics of the family (such as level the future. Outcome measures are most directly of education and health status) and on conditions related to answering whether a program is working in the labor market. Families with higher levels of as intended to achieve its objectives. Given the education, better health, and who reside in an area Legislature’s overarching interest in assessing with a strong local labor market with relatively whether programs are meeting their objectives, abundant employment opportunities are more performance measurement in public programs likely to have a positive employment outcome than should ideally focus on outcomes. However, as a family with lower levels of education, poorer we describe below, there are a variety of issues in health, and living in an area with a weak labor meaningfully measuring these outcomes. market. This is true regardless of whether the family participates in CalWORKs. Outcome Measurement . . . Making It Challenging to Directly Link Can Present Challenges . . . Program Operations to Outcomes . . . The presence Data on Outcomes Not Always Readily of other factors that affect program outcomes Available. One challenge associated with outcome can make it difficult to distinguish the role of the measurement is that data on desired outcomes are program from the role of other factors in bringing not always available. For example, as mentioned about the observed outcome. For example, if a previously, the first goal of the CalWORKs family obtains employment with high enough program is to reduce child poverty in the state. wages to allow the family to leave CalWORKs Measuring child poverty requires relatively assistance, this would typically be considered a detailed information on families’ circumstances positive outcome. However, it may not be clear and resources. However, services provided in whether the positive outcome came about because CalWORKs to improve participating families’ the family had above-average skills, because of employment and earnings may take effect gradually improving conditions in the local labor market, over time, including after a family is no longer because the CalWORKs program helped the participating in the program. Once a family leaves family address barriers to employment, or some the program, much less detailed information on the combination of all three. This means that, in family’s situation is collected. This constrains the general, measured outcomes cannot necessarily be state’s ability to measure whether the family or the fully attributed to the program, since other factors children are in poverty, or how a family’s income likely played some role as well. These challenges are changes relative to poverty thresholds over time. compounded as time passes after a family leaves www.lao.ca.gov Legislative Analyst’s Office 9 AN LAO REPORT CalWORKs, since the extent to which other factors including CalWORKs as we describe below, tends will affect outcomes for families increases over to focus on processes rather than on outcomes. time. As a result, measuring long-term outcomes is However, outcome measurement remains particularly challenging. important because measuring outcomes puts the . . . Or Compare Outcomes Across focus on the program’s end goals, rather than Jurisdictions. The presence of other factors that simply on the processes used to try to achieve affect program outcomes also makes it challenging those end goals. When data are available, including to compare outcomes across counties. This is outcomes as a part of program performance because the differences in outcomes between two measurement helps the Legislature to track counties may be due to actual differences in the outcomes for families in the program and allows effectiveness of the respective counties’ programs, program administrators to consider how program or differences in the labor markets and participant administration might be changed to improve characteristics between the two counties, or a outcomes. This is done recognizing that there will combination of both. For example, a county with always be some uncertainty in the extent to which a weak local labor market and a caseload with the program is responsible for observed outcomes relatively significant barriers to employment— or how changes in program administration factors beyond the county’s control—may have affect outcomes. Given this uncertainty, outcome worse employment outcomes for CalWORKs measurement is most useful when it is done in participants than a county with a strong local conjunction with the measurement of processes labor market and a caseload with relatively fewer that are expected to lead to desired outcomes and significant barriers to employment, even if the are naturally more directly under the program’s CalWORKs programs in the counties are operated control. Additionally, as will be described in similarly. greater detail later, steps can be taken to adjust for the effect of other factors, potentially making it . . . But Adds Value When Approached easier to measure program performance relative With Appropriate Caution to outcomes and make comparisons across In light of the challenges described above, jurisdictions. performance measurement in many programs, PERFORMANCE MEASUREMENT IN CALWORKS TODAY Federal WPR the Driving Factor in the state’s approach to performance measurement CalWORKs Performance Measurement in CalWORKs. To meet federal requirements, the state devotes significant resources to tracking Potential for Fiscal Penalties Focuses the WPR at both the state and county level. Attention on WPR. As noted previously, the WPR Additionally, state law specifies that, in the event is the only required federal performance measure that the federal government imposes a penalty for state TANF programs. The prominence of on the state for failure to comply with the WPR the WPR, and the fiscal penalties attached to requirement, the state will share any penalty noncompliance, have made it a driving factor in 10 Legislative Analyst’s Office www.lao.ca.gov AN LAO REPORT equally with those counties that contributed to the State Priorities Diverge From Federal state’s failure to meet the WPR requirement. In Priorities, Requiring Balance other words, 50 percent of any penalty amounts At the same time the state has worked to come ultimately imposed on the state for failure to meet into compliance with the WPR requirement, the the WPR requirement would be passed on to state has enacted policies that reflect priorities counties that did not themselves meet the federal beyond simply meeting the federal WPR WPR requirement. This provision is intended to thresholds. In particular, significant changes made motivate counties to assist the state in its efforts to in recent years have broadened the focus of the comply with the WPR requirement. program. State Has Failed to Meet WPR Requirement State Made Significant Changes to CalWORKs in Past Years, but Recently Came Into in 2012 and 2013. In 2012, the Legislature enacted Substantial Compliance. Due to a variety of a significant change to the state rules that govern factors, California has consistently failed to meet allowable welfare-to-work activities. Specifically, the WPR over much of the past decade. This has the state rules were altered to provide greater resulted in a significant amount of penalties (a flexibility in what counts as an allowable welfare- total of $1.8 billion) being assessed—but not yet to-work activity for a period of 24 months, allowing enforced—on the state. However, as a result of participants to engage in activities and receive certain actions taken by the state and by taking services that best align with addressing their advantage of administrative remedies available in barriers to employment. Because these state rules federal law, the state has now achieved substantial differ from federal rules for allowable welfare- compliance with the WPR requirement, has to-work activities, families taking advantage of the already received relief from more than $1 billion flexibility afforded during this period might not of the total penalties, and is on track to eliminate count toward the state’s WPR requirement. Once the majority of the remaining penalties that have 24 months of assistance under the more flexible been assessed. We describe some of the major state rules are exhausted, adult recipients may steps—most of them administrative—the state continue to receive assistance but are required to has taken to reach substantial compliance in the participate under the federal welfare-to-work rules, box on page 12. Currently, the state is estimated which are relatively less flexible and generally have to be exceeding the overall WPR threshold that a heavier emphasis on employment, as opposed to applies to all families by several percentage points. education, training, and other activities intended (We note that the state continues not to meet the to remove barriers to employment. A family’s separate WPR threshold for two-parent families, participation under the federal rules generally leaving the state subject to some penalties. would count toward meeting the state’s WPR However, penalties for noncompliance with the requirement. two-parent requirement are much smaller than In 2013, the Legislature enacted further penalties for noncompliance with the all-families changes to CalWORKs related to welfare-to-work. requirement, since two-parent families make up The first of these was requiring the development a relatively small share of all families receiving of a standardized appraisal, known as the Online assistance.) CalWORKs Appraisal Tool (OCAT), which is used with all new CalWORKs participants to identify potential barriers to employment before they begin www.lao.ca.gov Legislative Analyst’s Office 11 AN LAO REPORT State Strategies to Address Work Participation Rate (WPR) Noncompliance History of California WPR Compliance. California failed to meet the all-families WPR threshold for federal fiscal years (FFYs) 2006-07 through 2013-14 and, as shown in the figure below, has failed to meet the two-parent WPR threshold since FFY 2011-12. In recent years, the state has taken several steps to address WPR noncompliance. As a result of these steps, the state achieved compliance with the all-families requirement in 2014-15 but remains out of compliance with the two-parent requirement. Steps to Alter Who Is Counted in WPR Calculation. Of all the actions the state took to reach substantial compliance, the most significant involved administratively adding households to the WPR calculation that are not enrolled in the California Work Opportunity and Responsibility to Kids (CalWORKs) program but that are employed with enough hours to meet the federal the Temporary Assistance for Needy Families (TANF) work requirement. This is done by using state CalWORKs funds to provide a modest supplemental food benefit, referred to as the Work Incentive Nutrition Supplement (WINS), to working families that are enrolled in CalFresh but do not receive regular CalWORKs benefits. Because the WINS benefit is funded with CalWORKs funds, families that receive it are included in the state’s WPR calculation. The state also took steps to fund regular CalWORKs benefits for certain families that are unlikely to meet the federal TANF work requirement with non-CalWORKs funding, effectively removing these families from the WPR calculation. Taken together, these steps account for most of the increase in the state’s all-families WPR from 2013-14 to 2014-15. Steps to Increase Participation in Federally Allowable Activities. In addition to changing which families are included in the WPR calculation, the state and counties have made efforts to improve the engagement of CalWORKs participants in activities that meet the federal work requirement. For example, in 2013 the state approved additional funding for subsidized employment, which is an allowable activity (counts toward the WPR requirement) under federal law. Recent History of California WPR Compliance Federal Fiscal Year 2010-11 2011-12 2012-13 2013-14 2014-15 2015-16 All-Families WPR Required ratea 29.0% 50.0% 50.0% 50.0% 50.0% 50.0% Actual rate 27.8 27.2 25.1 29.8 55.7 60.7b Met requirement? No No No No Yes Yes Two-Parent WPR Required ratea — 90.0% 90.0% 90.0% 90.0% 90.0% Actual rate 33.9% 30.8 30.9 25.5 61.4 69.9b Met requirement? Yes No No No No No a California’s effective required rates were reduced in 2010-11 (including down to zero percent for two-parent families) due to a federal credit for past caseload reductions. b Preliminary. WPR = work participation rate. 12 Legislative Analyst’s Office www.lao.ca.gov AN LAO REPORT welfare-to-work. The second was the creation of the Beyond WPR, Current CalWORKs Family Stabilization program within CalWORKs, Reporting Emphasizes Process which provides intensive case management Monthly Reporting on Caseload and additional services (depending on the Characteristics, Participation Rates, and administering county) intended to help CalWORKs Certain Limited Outcomes. Chapter 270 of 1997 participants experiencing a crisis situation that (AB 1542, Ducheny), the legislation that established would hinder full participation in welfare-to-work. CalWORKs, required DSS to develop and Additional funding for subsidized employment implement a system of performance monitoring was also approved at this time. Taken together, that would: (1) look at the extent to which the these changes increase the program’s emphasis on state and counties were achieving the program’s identifying participants’ barriers to employment goals, (2) evaluate whether negative outcomes were early and provide the potential for additional occurring, and (3) identify any needed adjustments flexibility in activities and services to address them. to the program. Chapter 270 also required DSS Some State Policies Potentially Work at to periodically publish data reported by counties Cross-Purposes With Meeting WPR Requirement, on caseload characteristics and welfare-to-work Creating Need for Balance. Some of the policies outcomes. Pursuant to these requirements, counties described above have the potential to work at cross- report on various aspects of program operations purposes with meeting the WPR requirement. including the number of families receiving cash In particular, the 24-month time clock allows assistance, the number of individuals participating CalWORKs recipients to participate in activities in each of the allowable welfare-to-work activities, that do not necessarily count toward meeting the number of individuals receiving mental the WPR. This has required implementing the health or substance abuse services, the number of 24-month time clock in such a way as to balance individuals who obtain employment, the number of the state priority of providing flexibility against cases discontinued from assistance due to increased the federal priority of having enough participants earnings, and the wages of current CalWORKs in federally allowable activities to meet required participants. Most of the data points currently WPR thresholds. We note that the actual effect reported are process-oriented. Other reported of implementing the 24-month time clock on the information, such as the number of individuals WPR is unclear at this time, but is one subject of an who obtain employment, the number of cases evaluation of recent program changes that is due discontinued from assistance due to earnings, and to the Legislature by January 2018. Additionally, the wages of current CalWORKs participants, are actions taken in recent years to reach substantial more outcome-oriented. compliance with the WPR requirement have provided a certain degree of “cushion” to allow State Has Previously Made Attempts to the state to implement policies consistent with its Promote Greater Outcome priorities that might negatively affect the WPR. Measurement and Accountability Despite this, because meeting the WPR is a federal Following a reauthorization of the federal requirement, the need to maintain balance between TANF program in 2005, the Legislature enacted state priorities and meeting the WPR requirement additional performance measurement and will continue into the future. www.lao.ca.gov Legislative Analyst’s Office 13 AN LAO REPORT accountability policies largely intended to increase County Peer Review (CPR) Process. The the state’s performance relative to the WPR second change to performance measurement and requirement. accountability in CalWORKs was the creation Pay for Performance Incentive System of the CPR process, enacted by Chapter 75 of Enacted . . . The first of these performance 2006 (AB 1808, Committee on Budget). Under improvement policies is the Pay for Performance this process, each county is periodically visited program, enacted by Chapter 78 of 2005 (SB 68, by representatives from DSS and peer counties Committee on Budget and Fiscal Review). Under to review the county’s performance on existing this program, counties may receive incentive performance measures, including the federal WPR funding on top of funding for regular program and Pay for Performance measures, and receive operations if they achieve certain benchmarks technical assistance. From 2007 through 2010, relative to specified performance measures. When eight CPRs were conducted. With each CPR, DSS Pay for Performance was enacted, the measures issued a report summarizing what was covered initially included (1) the percentage of current during the visit and identifying both challenges CalWORKs cases that had employment, (2) a and promising practices for the visited county. county-specific WPR, and (3) the percentage of The CPR process was temporarily discontinued in participants with earnings one quarter after leaving 2010 and reinstated in 2013. Since that time, the assistance. A fourth measure—the percentage of emphasis of county visits has shifted to oversight of CalWORKs participants with earnings sufficient implementation of CalWORKs reforms enacted in to receive the maximum benefit from the federal 2012 and 2013. earned income tax credit—was added later. In Recent County-Led Efforts order to receive incentive funding from the Pay Related to Performance Measurement for Performance program, counties would either have to make a minimum level of improvement In 2015, counties collectively contracted with on a given performance measure or rank in the a group of researchers to conduct an assessment top 20 percent of counties. Incentives funds were of the CalWORKs program, referred to as the required to be spent in the CalWORKs program. CalWORKs Strategic Initiative. The CalWORKs . . . But Never Funded. After the Pay for Strategic Initiative’s work has examined issues Performance program was enacted, $40 million relating to (1) county policies and procedures that was initially appropriated for incentive related to the recent program changes, (2) county payments was reverted to achieve budgetary staffing and staff development needs, (3) strategies savings and no counties received an incentive for client engagement, (4) county partnerships, payment. No funding has been appropriated for and (5) how data and performance measurement incentive payments since that time. As required by practices might support more effective program law, DSS published county performance relative to operations. In part, the Strategic Initiative’s work Pay for Performance measures beginning in 2005, relative to performance measurement contributed but stopped publishing the measures (except for to the development of the recent Cal-OAR county WPR) around 2010. legislation, described in detail below. 14 Legislative Analyst’s Office www.lao.ca.gov AN LAO REPORT OVERVIEW OF CAL-OAR As described previously, the Legislature process through which counties would use enacted legislation to establish Cal-OAR as part of these performance measures to assess their own the 2017-18 budget package. In terms of its main performance and set goals for improvement. Each components, Cal-OAR is based on the California county will repeat this process on a three-year Child and Family Services Review (C-CFSR), cycle. The main components of this process are the county performance measurement and described below and displayed in Figure 3. accountability system used in child welfare services. County Self-Assessment. First, counties We describe the main features of Cal-OAR below. will undergo a self-assessment, in collaboration with local stakeholders, that will review their Requires Development of performance relative to the Cal-OAR measures; New Performance Measures identify the factors that contribute to that The Cal-OAR legislation requires the performance, including strengths and weaknesses development of performance measures, referred to in county practices; and consider potential areas of in the legislation as “indicators,” that are consistent focus for future improvements. with program goals. These measures are to include County System Improvement Plan. Following both process measures and outcome measures, the self-assessment, counties will develop a system although the legislation does not specifically define improvement plan that describes the steps counties the measures. Process measures are intended to will take to improve in selected areas, and will focus on things like the types of services provided, feature related targets for Cal-OAR performance how those services are provided, and the extent to measures. For some process measures, counties will which recipients utilize these services. Outcome measures Figure 3 are intended to focus on things Major Features of Cal-OAR like recipients’ employment, educational attainment, Annual Report to the Legislature program exits and reentries, and potentially other measures of family and child well-being. Department of Social Services Review and Oversight County-level data on these measures will be reported and YEAR 1 YEAR 2 YEAR 3 published no less frequently County Self Assessment than semiannually. Annual County Annual County County System Progress Report Progress Report Improvement Plan Establishes County Assessment, Planning, and Regularly Published Performance Measure Data Reporting Process on Three-Year Cycle The Cal-OAR legislation Cal-OAR = CalWORKs Outcomes and Accountability Review. further establishes a www.lao.ca.gov Legislative Analyst’s Office 15 AN LAO REPORT be required to meet statewide targets determined by the development of county system improvement DSS (in consultation with stakeholders). For other plans that merit legislative consideration. measures, counties may establish targets based Implementation Guided by Workgroup Process on individual circumstances. The development of county system improvement plans will involve Key Implementation Details to Be Determined a peer review component similar to the CPR by Workgroup. Many details related to how process in current law, in which peer counties will Cal-OAR will operate have yet to be determined. share best practices and provide other technical Budget legislation requires DSS to convene assistance to the county developing the system a workgroup in the fall of 2017 consisting of improvement plan. County system improvement counties, legislative staff, interested welfare plans are required to be approved by local elected advocacy and research organizations, current and officials, such as county boards of supervisors. former recipients, and others, to produce a plan Annual Progress Reports. After approval for Cal-OAR implementation. The workgroup of system improvement plans, counties will be will determine which performance measures to required to prepare annual updates on progress use, considering what data are already collected toward meeting goals identified in the system and how the administrative burden of additional improvement plans. reporting can be limited; what elements county self-assessments and system improvement plans DSS Oversight Role should include; and how county peer reviews The DSS will play an important oversight will be structured. The workgroup will also role under the Cal-OAR legislation. County examine state and county workload and costs self-assessments, system improvement plans, and associated with implementing Cal-OAR and make annual reports will all be required to be submitted recommendations on potential financial incentives to DSS for review and approval. Additionally, DSS for counties based on their performance. Prior to is required to facilitate the sharing of best practices implementation, DSS is required to annually update and provide technical assistance to support the legislative budget committees on the progress and implementation of the system improvement plans. timelines to implement Cal-OAR. The DSS is also required to regularly monitor Implementation by July 2019. Budget county performance relative to the Cal-OAR legislation requires that DSS implement measures. In the event that DSS determines that Cal-OAR by July 2019, based on plans developed a county is consistently failing to meet statewide by the workgroup, with the cycle of counties targets for process measures or other targets conducting self-assessments and preparing system established through system improvement plans, improvement plans beginning after baseline data the department is required to provide targeted are collected. Following completion of the first technical assistance and, if performance does not Cal-OAR cycle, DSS will establish, in consultation improve, may require counties to take corrective with the workgroup, statewide standard thresholds actions to improve performance. Finally, DSS for process measures. The workgroup will also is required to report to the Legislature annually convene following implementation to examine on trends in county performance and makes potential changes to the performance measures or recommendations on any issues identified through potentially additional performance measures. 16 Legislative Analyst’s Office www.lao.ca.gov AN LAO REPORT ISSUES FOR CONSIDERATION In concept, we think that Cal-OAR will with obtaining employment or higher wages, significantly increase the extent of performance in many cases by providing services to address measurement in CalWORKs and has the barriers to employment. For programs reauthorized potential to place greater emphasis on outcomes by WIOA, state and local performance is assessed measurement. Below, we outline several issues for using a set of common measures, described in consideration by the workgroup and the Legislature Figure 4. related to how Cal-OAR can be structured as Recent Federal Actions Expand Use of WIOA planning and implementation progress in coming Common Measures. The federal government has months and years. recently taken actions to expand the application of WIOA common measures to other programs. Aligning Performance Measures With Specifically, beginning in 2017, California is newly Other Federal Workforce Programs required to use WIOA common measures to Federal Workforce Innovation and assess the performance of CalFresh employment Opportunity Act (WIOA) Programs Assessed and training programs. These programs are Using “Common Measures.” The federal WIOA, administered by the majority of the state’s passed in 2014, reauthorized several federal 58 counties and provide employment and training programs related to workforce development, services to CalFresh recipients that are similar in including those that provide funding for local concept to CalWORKs welfare-to-work services. employment services and adult education. These Additionally, in recent years legislation has been programs have goals that overlap with those of introduced in Congress, but not enacted, that CalWORKs, in that they seek to assist individuals would require states to assess the performance of Figure 4 WIOA Common Measures 9 Employment. The percentage of participants who are in unsubsidized employment during the second quarter after leaving the program. Employment is also measured in the fourth quarter after leaving the program. 9 Earnings. The median level of earnings among participants who are employed during the second quarter after exit. 9 Credential Attainment. The percentage of participants enrolled in education and training who attained a recognized postsecondary credential or high school diploma or equivalent while participating or within one year of exiting the program. 9 Skill Gains. The percentage of program participants who, during a program year, are in an education or training program that leads to a recognized postsecondary credential or employment and who are achieving measurable skill gains toward such a credential or employment. 9 Effectiveness in Serving Employers.a a Measure still under development. WIOA = Workforce Innovation and Opportunity Act. www.lao.ca.gov Legislative Analyst’s Office 17 AN LAO REPORT TANF programs using outcome measures that are Some Other Programs Use Statistical similar to some of the WIOA common measures. Techniques to Adjust Performance and Priority Consideration Should Be Given Expectations in Light of Local Conditions. Below, to WIOA Measures. In choosing a key set of we briefly describe approaches used in some other performance measures for Cal-OAR, giving programs to adjust for the effect of other factors priority consideration to the common measures beyond the program when measuring outcomes. used in federal WIOA programs would make As one example, in programs authorized under sense for several reasons. First, these measures WIOA, federal law requires that a statistical are consistent with the CalWORKs program’s model be used to determine states’ “expected goals of increasing employment and wages. performance” on common measures based on We note that one of the measures included in the conditions in the state, such as the strength of the currently inoperative CalWORKs Pay for the labor market and characteristics of program Performance program measure is very similar to participants. This expected performance level is the employment measure used in federal WIOA then used as a starting point for determining state programs. Second, using WIOA common measures targets for performance that are tailored to a state’s increases the consistency with which performance individual circumstances. A similar process is is measured in workforce-related programs in the used to determine local targets within California. state more broadly. In our August 2016 report, In another example, federal law requires that state Improving Workforce Education and Training Data performance in child welfare services programs be in California, we recommended that the state adopt adjusted to account for the conditions in the state common performance measures that would be used and the characteristics of populations served. Each across all of the state’s workforce education and state’s adjusted performance is then compared to training programs, including CalWORKs. Using national targets to determine whether performance WIOA common measures in CalWORKs would expectations have been met. Statistical adjustment promote this goal. Finally, as noted above, there is also used in similar ways in some state TANF has been movement at the federal level to increase programs. We describe the role of statistical the application of the WIOA common measures to adjustment in performance measurement in the additional programs, including potentially to state TANF program of one state—Minnesota—in the TANF programs. Using WIOA common measures nearby box. in CalWORKs could reduce the risk that the state Statistical Adjustments Can Be Beneficial, would have to significantly change performance Subject to Constraints of Available Data. By measures as the result of potential future federal accounting for the effect of factors beyond a actions. program that affect outcomes, such adjustments can make it easier to link program operations Consider Steps to Account for to outcomes. Accounting for the effect of factors Other Factors That Affect Outcomes outside the program can also “level the playing As noted previously, outcome measurement can field” among different jurisdictions that operate be challenging because other factors beyond the a program in differing local circumstances, program can affect outcomes, sometimes making helping to explain what might cause differences in it difficult to distinguish the role of the program in performance in different jurisdictions. causing an outcome versus the role of other factors. 18 Legislative Analyst’s Office www.lao.ca.gov AN LAO REPORT At the same time, we note that statistical for small jurisdictions. These data limitations can adjustments are always constrained by the limit the accuracy and reliability of statistical availability of data on the factors that affect adjustments, or may limit to which outcomes or in outcomes. Complete and comprehensive data on which jurisdictions they can be applied. all of the factors beyond a program that may affect Consider How Statistical Adjustments for outcomes will generally not be available. For some Outcomes Might Be Used in Cal-OAR. Through factors, only approximate data will be available the process of developing the details of Cal-OAR, and for other factors there may be no available the working group could consider what role data. Additionally, data may be particularly limited statistical adjustments could play in measuring Performance Measurement in the Minnesota Family Investment Program (MFIP) Minnesota’s Temporary Assistance for Needy Families program, referred to as MFIP, is administered locally by 87 counties and four tribal employment service providers. Below we describe at a high level how performance in MFIP is measured. State-Defined “Self-Support Index” Is Primary Performance Measure. The performance of local MFIP administrators is measured and reported on several measures, including the federal work participation rate (WPR). Among these measures, however, the MFIP performance measurement system focuses on a state-defined measure referred to as the self-support index. The self-support index is calculated as a percentage of program recipients in a given quarter that three years later (1) have left assistance for reasons other than exceeding the program’s time limits or being subject to a sanction or (2) are working at least 30 hours a week. Local Performance Individually Measured Against Statistically Determined Range. Local performance on the self-support index is determined by whether a local administrator’s percentage falls below, within, or above, an individualized “range of expected performance” for that administrator. The range of expected performance for each local administrator is determined using a statistical model that predicts a range of expected performance based on various factors including economic conditions and demographic characteristics of recipients in the local area. Adjusting the ranges of expected performance to reflect these factors is intended to make performance targets more realistic to reflect local conditions, such that achieving a given percentage on the self-support index may be viewed as successful for one local administrator but not another. For example, in 2016 the lowest expected range of performance for a local administrator was 50.2 percent to 61.9 percent. The highest expected range of performance for a local administrator for the same year was 77.9 percent to 91.4 percent. Local Administrators May Earn Incentives, or Potentially Face Penalties, Based on Self-Support Index Performance. Local administrators that achieve a self-support index above their expected range of performance receive a bonus of 2.5 percent of their usual funding. Local administrators that fall below their expected range of performance are required to submit improvement plans to the state and eventually, if performance does not improve, have their funding reduced as a penalty. www.lao.ca.gov Legislative Analyst’s Office 19 AN LAO REPORT CalWORKs performance, in light of the potential • To Improve DSS Review of County benefits and data limitations. Statistical adjustment Performance. Finally, DSS could use could potentially be used in Cal-OAR in different statistical adjustment as a tool to perform ways and in varying degrees. Some ways statistical oversight over county self-assessments adjustment might be used include: and system improvement plans. For example, adjusting county performance • To Enable Statewide Standards for to reflect local conditions could help DSS Outcome Measures. As one example, the to determine whether or not differences Cal-OAR legislation specifies that the in county performance are more likely department and workgroup can consider the result of local conditions or more in future iterations of Cal-OAR whether to likely the result of practices that could be establish statewide performance standards improved. Similarly, statistical adjustment for outcome measures, provided that all could be used to help DSS evaluate whether counties could be reasonably expected to targets chosen by counties in their system meet such standards given varied local improvement plans are reasonable given conditions. If there is future interest in those counties’ local circumstances. statewide outcome standards, statistical adjustment could be used to account for In each of these examples, the use of statistical the effect that local conditions have on adjustments, including the data sources and models counties’ ability to meet such standards, used to adjust for the effect of other factors on similar to techniques used in other outcomes, would need to be revisited periodically programs. and refined as policies and conditions in which CalWORKs operates in the state and in the • To Provide Additional Perspective counties change over time. on County Performance Relative to Outcome Measures. Another potential Performance Measures Should Reflect use for statistical adjustment might be for Balance of State and Federal Priorities DSS to use similar techniques to publish Cal-OAR Presents Opportunity to Put in Place adjusted county performance on selected Additional Performance Measures Consistent outcome measures alongside unadjusted With State Priorities. As noted previously, the county performance. In some cases, local state’s CalWORKs priorities and policies go beyond conditions might affect unadjusted county just meeting the federal WPR requirement and, performance in such a way that counties in some cases, have the potential to work at cross- that appear to perform poorly might be purposes with meeting that requirement. The shown to perform well after accounting for enactment of Cal-OAR presents an opportunity those local conditions and counties that to institute additional performance measures that appear to perform will might be shown to are consistent with state priorities that are not have room for improvement after adjusting necessarily reflected in the WPR, such as providing for local conditions. This information could services to remove barriers to employment or help counties to determine which areas to providing flexibility in activity assignments. focus on in system improvement plans. Putting additional measures in place that are consistent with these state priorities will elevate 20 Legislative Analyst’s Office www.lao.ca.gov AN LAO REPORT the prominence of these state priorities and better penalties for Cal-OAR performance measures, such enable the state to track how these priorities are that the WPR may remain the only performance being achieved. measure for which counties could face a fiscal Need for Balancing State Priorities Against penalty if they do not meet established thresholds. Compliance With Federal Requirement Remains. This would continue to place emphasis on the WPR At the same time the state is working to improve relative to other performance measures in Cal-OAR performance relative to Cal-OAR measures, it must that have no associated fiscal penalties. After still meet the WPR requirement or face penalties, Cal-OAR implementation, the Legislature may requiring the state to balance the implementation wish to revisit county penalties related to the WPR of its priorities against the need to meet the federal and consider ways to account for both Cal-OAR requirement. Because of state actions taken to performance measures and the WPR. substantially comply with the WPR requirement, Consider Ways to Create More Uniform the state is exceeding the main WPR threshold, County Data Collection and Reporting such that the state’s WPR could decrease somewhat without returning to noncompliance. However, it Multiple Consortia Systems Approach Has is unclear how the implementation of Cal-OAR Advantages . . . Maintaining multiple automation will affect the emphasis that counties place on the systems for CalWORKs eligibility determination WPR requirement in their CalWORKs operations, and case management has certain advantages. or what effect this may have on the state’s WPR. Specifically, this approach provides additional It is possible that the additional performance flexibility for counties to adapt the automation measurement and county self-assessment and systems to meet local needs. Operating multiple planning will lead to increased engagement consortia systems has also resulted in opportunities overall and a higher WPR. It is also possible that for innovation and competition among the reduced emphasis on the WPR from implementing consortia, leading to new best practices that have Cal-OAR could lead to a lower WPR, resulting been adopted across the consortia systems. in greater risk of federal penalties. Given this . . . And Drawbacks. However, there are uncertainty, it will be necessary to continue also some drawbacks to the multiple consortia to monitor how the state’s WPR is affected as approach. Much of the data on CalWORKs is Cal-OAR is implemented and emphasis shifts housed in the consortia systems and, because these to state priorities to a greater extent through systems are operated and managed by the county performance measurement. consortia, DSS does not have direct access to Consider Including WPR as Cal-OAR these data. Instead, DSS is often required to make Performance Measure. As a way to strike a balance specific requests for the consortia to provide data between federal and state priorities for CalWORKs, to the state to use for program oversight purposes. the working group could consider including the Having different counties using different systems WPR in Cal-OAR as one measure of engagement, in some cases also results in a lack of consistency alongside other measures that align more directly in data collection across the state. The three with state priorities. consortia systems do not collect all the same data Consider Revisiting County Fiscal Penalties elements and, even within a single consortia, not After Cal-OAR Implementation. The Cal-OAR all counties use all the data elements included in legislation does not specifically create county fiscal the system, or use them in the same way. These www.lao.ca.gov Legislative Analyst’s Office 21 AN LAO REPORT issues with consistency in data collection from Cleanup Needed to Align county automation systems have implications Past Actions With Cal-OAR for how county performance will be reported in Once implemented, Cal-OAR will overlap Cal-OAR that will need to be addressed as part of with some previous actions taken by the the workgroup process. Legislature relative to performance measurement Required Transition to Single System Presents and accountability. The Pay for Performance Opportunity to Increase Uniformity. As described incentive program, enacted but never funded or previously, the state has recently been required to fully implemented, is redundant with Cal-OAR. develop a single, statewide automation system, to Similarly, the CPR process is very similar to the be known as the California Statewide Automated peer review envisioned by the Cal-OAR legislation Welfare System, or CalSAWS. The development to be part of the development of the county system of CalSAWS presents an opportunity for the state improvement plan. The workgroup should consider to consider ways to standardize data collection whether any aspects of these previously enacted and streamline the sharing of county data with program components would be useful to consider DSS in ways that will facilitate the operation of as part of Cal-OAR (for example, whether any of Cal-OAR. We recommend that the Legislature the Pay for Performance measures might be useful require the administration to report on how as performance measures in Cal-OAR), at which CalSAWS development will result in increased point they can be repealed. uniformity in data collection and streamline data sharing between the state and counties, and how these changes will enable the implementation of performance measurement through Cal-OAR. CONCLUSION The CalWORKs program plays a key role in the potential to focus performance measurement in the state’s efforts to assist low-income families CalWORKs more on outcomes that are consistent with children, and significant resources are with the program’s goals. As Cal-OAR is developed dedicated to achieve the program’s goals. In an and implemented in coming years, consideration environment of differing federal and state priorities should be given to how Cal-OAR can align with and decentralized program administration, performance measurement systems used in other effective oversight of program performance by major federal workforce programs, account for the Legislature takes on great importance. In the factors beyond the CalWORKs program that affect past, program performance measurement has outcomes, balance state and federal priorities for emphasized process measures, in particular the CalWORKs, and lead to more uniform county data federal WPR. The recently enacted Cal-OAR has collection and reporting. 22 Legislative Analyst’s Office www.lao.ca.gov AN LAO REPORT www.lao.ca.gov Legislative Analyst’s Office 23 AN LAO REPORT LAO Publications This report was prepared by Ryan Woolsey and reviewed by Ginni Bella Navarre. The Legislative Analyst’s Office (LAO) is a nonpartisan office that provides fiscal and policy information and advice to the Legislature. To request publications call (916) 445-4656. This report and others, as well as an e-mail subscription service, are available on the LAO’s website at www.lao.ca.gov. The LAO is located at 925 L Street, Suite 1000, Sacramento, CA 95814. 24 Legislative Analyst’s Office www.lao.ca.gov