Improving Workforce Education and Training Data in California

Improving Workforce Education And Training Data in California MAC TAYLOR • L E G I S L A T I V E A N A L Y S T • AUGUST 2016 AN LAO REPORT 2 Legislative Analyst’s Office www.lao.ca.gov AN LAO REPORT EXECUTIVE SUMMARY Background State Has Extensive Array of Workforce Programs. In 2016-17, eight state agencies received a total of more than $6 billion in state and federal funding to administer almost 30 workforce education and training programs. Community colleges and schools are the major providers of workforce education and training, relying primarily on state funds to support these programs. The Department of Social Services and the Employment Development Department (EDD) also receive a notable amount of funding to support workforce programs, primarily from federal funds. Other agencies also administer various workforce programs, though these programs tend to be smaller. The state tasks the California Workforce Development Board (CWDB), an appointed body with broad stakeholder representation, with developing an overarching strategic workforce plan every four years. This plan is intended to serve as a framework for the development of policy, spending, and operation of all workforce programs in the state. Programs Typically Report on Participants and Outcomes. Most workforce education and training programs require service providers to report information about their program participants, including demographic information. Many programs also require information about participants’ near-term outcomes, such as the share of students completing training programs or earning certificates. Increasingly, programs also require information about longer-term outcomes, such as subsequent employment and earnings. State Agencies Must Link Some of Their Workforce Data to Meet Current Reporting Requirements. To collect information about program participants’ longer-term outcomes, state agencies often must share and link data with one another. For example, CCC must link data on students completing workforce education programs with EDD data to determine whether those students successfully transition into the workforce. California’s agencies currently link their data using agency-to-agency agreements. Agencies in other states link their data using different methods, with some states relying upon a central repository of data managed by a central agency and other states using a federated data system in which agencies maintain their data in-house but permit other agencies access to some of that data through a shared interface. Assessment Outcome Measures Historically Have Not Been Standardized. Historically, state and federal laws have required service providers to report different types of outcome information even for similar workforce programs with comparable goals. Such differences frustrate efforts by providers, state agencies, and the Legislature to aggregate data across different programs, compare program outcomes, and assess the overall system’s performance. Such differences also increase providers’ and state agencies’ administrative burdens. Some Recent Progress Using Common Outcome Measures, but Work Remains. To address these concerns, the federal government standardized outcome measures for all programs funded www.lao.ca.gov Legislative Analyst’s Office 3 AN LAO REPORT through the recently reauthorized Workforce Innovation and Opportunity Act (WIOA). The state’s workforce plan indicates that it eventually intends for state-funded workforce programs to align their outcome measures with the new federal WIOA measures. Though somewhat different from the WIOA measures, the California Community Colleges (CCC) recently standardized outcome measures for most of its career technical education programs. Despite these developments, many state workforce programs still require providers to track different performance measures. Current Efforts to Link Data Inefficient and Burdensome. A further problem is that the state’s current data-sharing model, the agency-to-agency model, is fragmented and time consuming for state agencies to navigate. It also is not comprehensive, as not all relevant agencies link with all other agencies serving the same participants. Compared to the agency-to-agency model, both the central repository and federated models have notable advantages, including greater accuracy, efficiency, and comprehensiveness. Recommendations Recommend Requiring Development and Use of Common Measures for All Programs. We recommend the Legislature convene a task force to adopt common workforce participation and outcome measures. We recommend including the standardized WIOA measures, with a few additional state-specific measures. After adopting common measures, we recommend the Legislature require all of the state’s workforce programs to collect and report data for those measures. We recommend streamlining all existing data and reporting requirements accordingly, amending statute as necessary. Recommend Improving Data Linkages Using Systemwide Model. We recommend the Legislature direct the CWDB to study and report on potential approaches for replacing existing agency-to-agency agreements with a statewide, streamlined data-linking system for all workforce programs in the state. After reviewing this report, the Legislature could authorize a preferred data-linking system in statute. Once the new system is in place, we recommend the Legislature require state agencies and workforce providers to participate in it, contributing specified data. Recommend Using the Resulting Data to Inform Budget and Policy Decisions. In concert with adopting common measures and creating a streamlined data-linking system, we recommend the Legislature direct the CWDB to develop a small number of standardized reports designed to communicate results clearly to policy makers, service providers, current and prospective program participants, and the public. We recommend the Legislature annually review these reports as part of its budget and policy processes, taking program results into consideration when adjusting budgets and refining state laws. We also recommend directing state workforce agencies to develop and adopt a uniform protocol by which researchers could gain access to the data. Such a protocol should contain appropriate safeguards for participant privacy and confidentiality. By conducting third- party analysis of workforce data, researchers could help inform state policy and fiscal decisions, thereby increasing the value of the data collected. 4 Legislative Analyst’s Office www.lao.ca.gov AN LAO REPORT INTRODUCTION Workforce education and training programs programs and the system overall is effective. provide job-specific training, basic skills education, Though the state’s workforce system has had these and related support services to help individuals shortcomings for many years, recent changes participate in civic life and the labor market. We in federal law and some recent state actions are estimate that over three million people access requiring workforce programs to collect more services funded by the state’s almost 30 workforce standardized data in a more coordinated way. education and training programs annually. Though These changes signal an opportunity for the state to the state’s workforce system is extensive, policy reconsider its approach to workforce data. makers currently cannot answer basic questions We begin this report by providing background about it. For example, even getting an exact about the state’s workforce programs and their count of the people who use the state’s workforce data reporting requirements. We then provide programs is difficult. With programs administered information about standardizing performance by so many agencies and providers, data housed measures and linking data across entities. Next, we in so many places, data requirements varying evaluate how well current data collection practices by program, and key data elements linked only provide information about programs. We conclude in limited situations, policy makers routinely by making recommendations regarding the next struggle to assess whether individual workforce steps the Legislature could take to improve the usefulness of workforce data in California. CALIFORNIA’S WORKFORCE PROGRAMS Eight State Agencies Administer Workforce California Spends More Than $6 Billion Programs. The California Community Colleges Annually on Almost 30 Workforce Programs. (CCC) Chancellor’s Office and the California Figure 1 (see next page) lists 29 publicly funded Department of Education (CDE) are the main state- workforce programs currently operating in level administrators of workforce education and California. The Appendix contains descriptions training programs. The California Department of of each of these programs. The funding amounts Social Services (DSS), the California Employment shown understate spending on workforce education Development Department (EDD), and the and training because they do not include any California Department of Rehabilitation (DOR) portion of the unrestricted funding that high also administer large workforce programs. Three schools receive through the Local Control Funding other state agencies—the California Department Formula (LCFF). Under this formula, high of Corrections and Rehabilitation (CDCR), the schools receive a per-pupil funding rate intended California Conservation Corps, and the California to recognize the higher costs of career technical Prison Industry Authority—administer relatively education (CTE). The state’s accounting system, smaller workforce programs targeted to more select however, currently does not have the capability populations. to track LCFF spending specifically on CTE. Statewide, districts likely spend between hundreds www.lao.ca.gov Legislative Analyst’s Office 5 AN LAO REPORT Figure 1 Funding for Workforce Education and Training Programs in California 2016-17 (In Millions) State Other Total Program Agency General Fund Fund Sourcesa Funding Apportionments for workforce education and training CCC $2,122b — $2,122 Adult Education Block Grant CDE/CCC 505c — 505 Career Technical Education Incentive Grants CDE 300d — 300 CalWORKs employment and training services DSS 233 $1,094 1,327 Strong Workforce Program CCC 200 — 200 Office of Correctional Education programs CDCR 199 — 199 Office of Offender Services workforce programs CDCR 114 43 156e Vocational Rehabilitation DOR 59 364 423 Apprenticeships CDE/CCC 54 — 54 Career Technical Education Pathways Program CDE/CCC 48f — 48 Project Workability for students in special education CDE 40 — 40 CCC Student Services for CalWORKs Recipients CCC 44 — 44 Core Training Program Corps 42 48 91 Economic and Workforce Development Program CCC 23 — 23 California Partnership Academies CDE 21 — 21 Adults in Correctional Facilities CDE 15 — 15 Nursing program support CCC 13 — 13 Specialized Secondary Programs CDE 5 — 5 Agriculture Incentive Grants CDE 4 — 4 Adult, Youth, and Dislocated Worker Services (WIOA Title I) EDD — 418 418 Wagner-Peyser Employment Services (WIOA Title III) EDD — 127 127 Carl D. Perkins Career and Technical Education Act Program CDE/CCC — 123 123 Adult Education and Family Literacy Program (WIOA Title II) CDE/CCC — 85 85 Employment Training Panel EDD — 73 73 CalFresh Employment and Training Program DSS — 63 63 Jobs for Veterans State Grant EDD — 20 20 CDE Student Services for CalWORKs Recipients CDE — 10 10 Proposition 39 pre-apprenticeships EDD — 3 3 Offender Development programs CalPIA 3g 2h 5 Totals $4,044 $2,473 $6,517 a Largely federal funds with some special funds. b Extrapolated from best available data. Assumes community colleges spend one-third of apportionment funding on core adult education areas. c $5 million is one-time funding for technical assistance to regional consortia. d Reflects second-year funding for three-year, $900 million grant program. e Reflects funding for wraparound services, which include workforce education and training. f Enacted legislation sunsets program July 1, 2017 and folds funding into Strong Workforce Program. g Transfer from CDCR. h Funded through sale of CalPIA goods. Assumes program will sell the same value of goods as in 2015-16. CCC = California Community Colleges; CDE = California Department of Education; DSS = California Department of Social Services; CDCR = California Department of Corrections and Rehabilitation; DOR = California Department of Rehabilitation; Corps = California Conservation Corps; WIOA = Workforce Innovation and Opportunity Act; EDD = California Employment Development Department; and CalPIA = California Prison Industry Authority. of millions of dollars and billions of dollars on identified in Figure 1, 62 percent ($4 billion) is CTE. The large range is due to varying definitions state funding. State funding primarily supports of what to count as CTE instruction and support. workforce education and training programs State and Federal Governments Provide provided through CCC and CDE. The remainder of Significant Workforce Funding. Of the $6.5 billion the funding is mostly federal, with a small amount 6 Legislative Analyst’s Office www.lao.ca.gov AN LAO REPORT from state special funds (such as the Clean Energy California Workforce Development Board Job Creation Fund). Federal funding primarily (CWDB) Tasked With Developing the Plan. The supports workforce programs provided through CWDB is the state’s designated coordinating body DSS, EDD, and DOR. The largest federal workforce for workforce issues and consists of 53 members programs are the California Work Opportunity appointed by the Governor. The board consists and Responsibility to Kids (CalWORKs) of a broad group of stakeholders, including employment and training services program, representatives of the Legislature, business, labor, supported with Temporary Assistance to Needy education, and corrections. The U.S. Department Families funding, and the various programs funded of Labor required states to submit their workforce by the Workforce Innovation and Opportunity plans for the 2016 through 2020 period by April 1, Act (WIOA). Some workforce programs, including 2016. States were to begin transitioning to the new CalWORKs, Offender Services, and Vocational plans beginning July 1, 2016 and are required to Rehabilitation, receive funding from both the state fully implement the plans by July 1, 2017. and federal governments. State Plan Outlines Primary Goals and State Workforce Plan Provides Overarching Identifies Policy Strategies to Meet Goals. Vision. The WIOA has particular importance for California’s workforce plan revolves around a state workforce policy because it requires each few key goals. Most notably, the plan sets a goal state to prepare an overarching strategic workforce of producing a million “middle-skill,” industry- plan every four years. Federal law requires that recognized, postsecondary credentials in areas the plan lay out a vision and goals for preparing a with labor market value by 2027. (The plan defines skilled workforce and meeting employers’ needs. these credentials as associate degrees, certificates, The state also is required to set specific growth and professional certifications and licenses that do targets for WIOA-funded programs related to not require a baccalaureate degree.) The state plan WIOA’s performance accountability measures also sets a goal of doubling the number of people (discussed further below). The state negotiates enrolled in apprenticeship programs. To meet these these targets with the federal government. Upon goals, the plan identifies various policy strategies, its development, the plan is intended to serve as such as increasing support services to participants California’s framework for all workforce-related in workforce programs, increasing the number policy and program decisions as well as federal and of “earn and learn” opportunities (including state spending decisions. apprenticeships), and building data capacity across agencies administering workforce programs. WORKFORCE DATA Historical State and Federal Data participants in their programs, typically including Reporting Requirements some associated demographic data, such as the participants’ age, gender, and ethnicity. In some State and Federal Programs Require Data cases, providers must collect data on whether on Participants and Outcomes. Most workforce participants have financial need and qualify for education and training providers are required grant or loan assistance. In addition, programs to submit information about the number of www.lao.ca.gov Legislative Analyst’s Office 7 AN LAO REPORT often must report certain outcomes. Both state point averages or “other appropriate standards of and federal programs typically require providers achievement.” to report on participants’ near-term outcomes, Even for Same Types of Outcomes, Measures such as program completion, test scores, and Can Differ. Even when programs share the same certificates or diplomas awarded. Increasingly, data goals and require providers to track the same types requirements also include participants’ longer-term of outcomes, specific outcome measures can vary. outcomes, such as subsequent employment, For example, programs often are required to report earnings, and enrollment in further education on their participants’ subsequent employment or training. Federal workforce programs require and wages. These measures, however, differ data on participants’ longer-term outcomes more among programs in timing, with some measures commonly than state programs. focusing on outcomes within three or six months State Often Asks for Different Sets of Data of program completion and others looking out Across State Programs. Some workforce programs three or five years. In addition, some measures with very similar goals require providers to consider only employment in a job related to the collect somewhat different performance data. For training a participant received, while others include example, CCC and CDE jointly administer the any employment or wage gains. Such differences CTE Pathways Program and the Career Pathways frustrate efforts to aggregate data across different Trust program. (Career Pathways Trust grants programs, compare outcomes among programs, were awarded in 2013-14 and 2014-15 and schools and assess the overall system’s performance. Such can spend the funding through 2016-17.) Both differences also increase administrative burdens programs provide funding to schools and colleges for agencies and providers, which may need to to create career pathways from high school into track outcomes differently for various funders and postsecondary education and the workforce, programs. yet state law requires that each program report Recent Moves Toward Common Measures different information. State law requires CTE Pathways Program grantees to report on the wages In recent years, efforts at various levels of of program graduates, whereas Career Pathways government have sought to address the problem of Trust grantees must report on students’ transitions collecting different outcomes for similar workforce related to employment, apprenticeships, and job programs. We describe these efforts below. training. Similarly, CDE administers both the Federal Government Expanded the Use of California Partnership Academies (CPA) and the Common Measures for Its Workforce Programs. Specialized Secondary Programs (SSP). Each of Although some federal workforce programs used these programs provides high schools with funding common data measures in the past, WIOA updated to create programs that focus on a career theme. and expanded the use of common measures to all For the CPA, the state requires grantees to annually workforce programs funded under its provisions. report attendance, credits earned, test scores, grade As shown in Figure 2, these measures include skill point averages, graduation rates, and postsecondary gains, employment, and earnings. plans. For the SSP, grantees have more flexibility State Working Toward Common Measures. in what data they annually report to the state. The CWDB has committed in the state workforce For example, grant recipients may choose to plan to assist state programs in aligning their report either improvement in participants’ grade outcome measures with the WIOA measures. 8 Legislative Analyst’s Office www.lao.ca.gov AN LAO REPORT Separately, Chapter 13 of 2015 (AB 104, Committee California’s Workforce Data Collection on Budget) requires CDE and the CCC Chancellor’s and Reporting Systems Office to develop common measures of effectiveness Each Agency Typically Has One Major Data for adult education programs. In a November 2015 System. State agencies that operate workforce report, the two agencies identified a minimum set programs each have one primary data system of enrollment and outcome information aligned to which providers report information about with WIOA requirements that all adult education program participants. For example, at CDE, providers must report and recommended that the the majority of student data is collected in the state create a centralized clearinghouse to track California Longitudinal Pupil Achievement Data student outcomes. System (CALPADS). Similarly, CCC’s primary Community Colleges Using Common data system is the Chancellor’s Office Management Measures for Workforce Programs. The Information System (COMIS), DOR’s primary data Chancellor’s Office developed its own set of system is Accessible Web-based Activity Reporting common measures for most CCC workforce Environment (AWARE), and CDCR’s primary education and training programs and implemented data system is the Strategic Offender Management them beginning in 2014-15. The Chancellor’s Office System (SOMS). Each of these systems compiles recently aligned several of these measures with the data from numerous local service providers into a WIOA common performance measures and is in statewide database. the process of modifying more of its measures to In Addition, Ad Hoc Data Systems Proliferate. align with WIOA measures. Rather than adding new data elements to their Figure 2 WIOA Common Performance Measuresa 9 Skill Gains • Percentage of program participants who, during a program year, are in an education or training program that leads to a recognized postsecondary credential or employment and who are achieving measurable skill gains toward such a credential or employment. 9 Learning Outcomes • Percentage of program participants who obtain (1) a recognized postsecondary credential or (2) a secondary school diploma or its recognized equivalent upon completing the program or within one year of exiting the program. 9 Employment • Percentage of program participants who are in unsubsidized employment during the second and fourth quarters after exiting from the program. 9 Earnings • Median earnings of program participants who are in unsubsidized employment during the fourth quarter after exiting from the program. 9 Effectiveness in Serving Employersb a Applies to Adult, Youth, and Dislocated Worker (WIOA Title I), Adult Education and Family Literacy (WIOA Title II), Wagner-Peyser (WIOA Title III), and some of the Rehabilitation Act programs. b The U.S. Department of Labor is in the process of developing measure. WIOA = Workforce Innovation and Opportunity Act. www.lao.ca.gov Legislative Analyst’s Office 9 AN LAO REPORT primary systems, which can be costly and time information from adult education providers in consuming, agencies sometimes create ad hoc order to comply with their annual reporting data systems to fulfill new reporting requirements requirement to the Legislature and administration. as they implement new programs. This practice Some Agencies Display Selected Data Publicly. is especially common for programs with unique Agencies sometime provide aggregate information data requirements, limited-term programs, and about their program participants to the public. programs with tight reporting deadlines. For For example, the CCC Chancellor’s Office makes example, for the 12 workforce education and publicly available some aggregate information on training programs CDE administers either alone students, course taking, and student outcomes or jointly, it uses at least six separate internal data through its online Data Mart, Student Success systems to collect and store performance data. Scorecard, Salary Surfer, and Wage Tracker tools. Moreover, in 2015-16, the first year of the Adult Likewise, CDE makes publicly available some Education Block Grant, the CCC and CDE used aggregate information about students, course a temporary data system to collect performance taking, and short-term outcomes on its online DataQuest and Ed-data.org tools. DATA LINKING The Value of Linked Data federal programs increasingly require information about program participants’ eventual employment, Data Linking Enables Providers and Policy wage gains, and enrollment in further education. Makers to Track Participation Across Programs. Linking data across state agencies over certain Data linking refers to matching participant records periods of time can provide this information. For from two or more programs or agencies and using example, to learn how many students continued those linked records to track participation and their education and training and/or went to work in outcomes. In the workforce area, this process is a particular industry, a high school CTE program essential for illuminating which programs are could link its student information with CCC and working well. For example, linked data can show EDD data for periods ranging from one semester to if students who complete high school-equivalency several years after high school graduation. programs at adult schools successfully transition Federal Workforce Act Requires Data to community colleges or other training programs. Linking Across Some State Agencies. As part of Linked data also can show if students who begin a WIOA implementation, the federal government program in one agency complete a similar program is requiring states to establish an integrated data in another agency (perhaps due to relocating). Data system to which WIOA-funded programs would linking also can identify gaps and duplication of contribute participant and outcome data. The services. purpose of the requirement is to improve state Data Linking Provides Information About agencies’ ability to (1) coordinate service delivery Longer-Term Impacts of Programs. In addition across programs and (2) record outcomes across to facilitating tracking across programs, data programs in a single report. linking enables policy makers to track participant outcomes over time. As noted earlier, state and 10 Legislative Analyst’s Office www.lao.ca.gov AN LAO REPORT Examples of Data Linking shows wage and employment outcomes for certain workforce programs. (The Dashboard likely will be Various entities already are linking data across available in late 2016.) Some state agencies also are workforce education and training providers to linking data. For example, the CCC Chancellor’s track program participation and outcomes. Office has developed the state-level CCC Salary Some National, Regional, and State-Level Surfer and the college-level College Wage Tracker, Data Linking. At the national level, the U.S. both of which use linked CCC and EDD data to Department of Education has developed the make information on median wages by program College Scorecard, which links data from of study publicly available. At the provider level, colleges and universities with federal student the Bay Area Community College Consortium, financial aid and tax data. For a given college, consisting of 28 community colleges in the region, the Scorecard provides information about is exploring the development of a data system that admissions, student demographics, completions, would link information about students enrolled costs (by family income), student aid (including in the region’s community colleges and adult student loan repayment rates), and the average education programs. earnings of students after attending the college. At the regional level, the Western Interstate Three Models for Linking Data Commission for Higher Education is piloting the Entities typically link participant data to Multistate Longitudinal Data Exchange, which measure longer-term outcomes using one of three links several western states’ higher education and models. Figure 3 (see next page) shows these employment data to provide information about models. student outcomes across state lines. At the state Agency-to-Agency Model. Under this model, level, several states routinely link K-12 education, two agencies enter into a memorandum of postsecondary education, and employment and understanding to share selected data for specified wage data through longitudinal systems to provide purposes on a one-time or recurring basis. information about students’ progression through Participating agencies determine the terms and the educational system and post-graduation conditions, as well as the mechanics, for linking outcomes. Florida, Texas, and Virginia have among their data. Additionally, the memorandum of the most comprehensive of these systems. understanding specifies who can access the data Linked Data in California. Although WIOA and for what purposes it will be used. In California, requires states to develop a comprehensive data linking for workforce education and training approach to linking data, California does not programs occurs only through agency-to-agency have a linked education or workforce data system. agreements. As the top of Figure 3 shows, CCC, Various entities, however, have made efforts to CDCR, DOR, and DSS each have entered into link information to track program participation separate agency-to-agency data sharing agreements and outcomes over time. These efforts include with EDD to collect information about their Cal-PASS Plus, created and funded through the participants’ wage gains. Notably, CDE has not CCC Chancellor’s Office, which links data for some linked its data with EDD records. In California, K-12 and higher education agencies with each other many other agency-to-agency agreements and with workforce outcome data. In addition, the relate to the sharing of many other elements of state tasked the CWDB in 2014 to produce a tool workforce data. known as the Workforce Metrics Dashboard, which www.lao.ca.gov Legislative Analyst’s Office 11 Template_LAOReport_fullpage.ait ARTWORK #150656 AN LAO REPORT Figure 3 Illustration of Three Data Linking Options Agency-to-Agency Model State agencies establish individual EDD & CCC EDD & CDCR EDD & DOR EDD & DSS interagency agreements with rules specifying how the data they collect will be linked and used.a State agencies can generate reports based on data matches between two agencies. Reports Reports Reports Reports Central Repository Model State agencies establish a comprehensive interagency CCC CDCR CDE EDD Others agreement with rules specifying how the data they collect will be linked and used. State agencies submit data to a central repository. A central agency links and Central Agency stores the data. Repository Central agency can generate reports from the data and state agencies also can request reports from the central agency. Reports Federated Model Automated Data Retrieval System State agencies establish a comprehensive interagency agreement with rules specifying how the data they collect will be linked and used. CCC CDCR CDE EDD Others Upon request by a state agency, an automated data retrieval system temporarily accesses and links data from multiple agencies to produce reports. Reports Reports Reports Reports Reports a Typically, one agency provides a data file to another agency, which matches records from the first agency to its own records and returns a new file with combined data from the two agencies. EDD = Employment Development Department; CCC = California Community Colleges; CDCR = California Department of Corrections and Rehabilitation; DOR = California Department of Rehabilitation; DSS = Department of Social Services; and CDE = California Department of Education. 12 Legislative Analyst’s Office www.lao.ca.gov AN LAO REPORT Central Repository Model. Under a central EDD both document Social Security numbers in repository model, agencies send their data to their records and can use these numbers to identify a central agency, such as a higher education individuals who both (1) attended a community coordinating board, which maintains a data college program and (2) earned reportable income repository or “warehouse.” As the middle section following completion of the program. Use of Social of Figure 3 shows, once an agency submits its data, Security numbers is not universal. State law does the central agency links it to other data in the not allow schools to collect these numbers for warehouse. The central agency (and sometimes K-12 pupils, complicating CDE’s ability to link other authorized agencies) can then query and records with CCC and EDD. Similar issues arise for use the data. In 1999, the Legislature directed the programs that serve undocumented immigrants. California Postsecondary Education Commission Workarounds Used When Common Identifiers to develop a central repository that linked data Are Not Available. Agencies have developed two across state education institutions. This repository, workarounds for reporting employment outcomes which included some data going back to the late in the absence of a common identifier. One method 1980s, operated until 2011. Many other states— is to survey former participants about their including Florida, Texas, and Washington— longer-term outcomes. This survey data tends to currently use this model to link their education and yield information that is less reliable than official workforce data. wage data and typically is used by agencies that Federated Model. Under a federated model, do not link data through any of the three models sometimes called an “integrated data system,” described above. The other workaround is to use agencies maintain their data in-house and permit “fuzzy matching.” Fuzzy matching uses several other agencies access to some of that data through personal characteristics, such as name, date of a shared interface (see the bottom section of birth, gender, and address to link information Figure 3). Using automated processes, the shared about an individual across separate data systems. interface can link data from multiple agencies upon As the name implies, fuzzy matching is not as the request of one or more agencies and with the precise as a common identifier, but it commonly is approval of all affected agencies. While this model understood to be the next best option, providing a is newer and (to date) less common than the central reasonably valid and reliable way for linking data. repository model, several states, including Virginia Agencies that use any of the data-linking models and Illinois, use it to link their data. described above often use this method to identify participants when common identifiers are not Other Considerations When Linking Data available. Regardless of the data-linking model entities Agencies Must Address Privacy and Security use, issues arise about how to identify participants Concerns When Sharing Data. State and federal and protect their privacy. We describe these laws allow sharing of education data across considerations in more detail below. agencies for legitimate educational and research Common Identifiers Facilitate Data Linking. purposes. These laws also require agencies to A key step in linking data is identifying individuals safeguard participant privacy. To comply with who appear in two or more data sets. This requires legal requirements, agencies sharing data with a common individual identifier that agencies can each other typically develop formal data sharing use to match the records. For example, CCC and agreements to establish (1) rules about how and to www.lao.ca.gov Legislative Analyst’s Office 13 AN LAO REPORT whom data can be disclosed and (2) steps agencies data. We discuss these issues in more detail in the will take to prevent unauthorized access to the nearby box. ASSESSMENT Despite having made some progress, Recent Progress Toward Common the state still lacks data measures that are Measures, but Work Remains applied systemwide and it still does not have a Some Programs Use Common Measures . . . coordinated data-linking approach. These two When fully implemented, the federal requirement key shortcomings hinder policy makers’ ability to that common measures be applied to all examine performance across California’s workforce WIOA-funded workforce programs will improve education and training system. We discuss these the usefulness of the state’s workforce data, issues in more detail below. allowing policy makers to examine participation and outcome data consistently across those programs. Efforts to establish common measures Data Privacy and Security State and Federal Privacy Laws Permit Data Sharing for Legitimate Educational and Research Purposes. The federal Family Educational Rights and Privacy Act (FERPA), the California Information Practices Act, portions of the California Education Code, and several other federal and state laws govern the privacy and confidentiality of student records. These laws generally prohibit schools from disclosing personally identifiable student information to a third party (someone other than the student and his or her educational institution) without a student’s or parent’s permission. The laws, however, make several exceptions. Under FERPA, for example, a school may disclose personally identifiable information in connection with a student transferring schools or qualifying for financial aid. A school also may disclose records to researchers conducting studies on behalf of the school, and to federal or state agencies evaluating publicly funded education programs. State laws largely align with the protections outlined by FERPA yet further require that any third party receiving personally identifiable student information for research purposes be approved by an institutional review board. Agencies Are Responsible for Data Security. Specifically, federal and state agencies are required to adopt administrative, physical, and technical safeguards to protect personal information from potential threats to its confidentiality and security. Agencies have employed various methods to safeguard data, including encrypting data, removing personal identifiers after linking two data sets but before sharing the results with a third party, and omitting results that have only a few observations (such as data on members of a very small group within a school). Agencies are responsible for providing adequate personnel training and oversight to ensure these data-security systems work. 14 Legislative Analyst’s Office www.lao.ca.gov AN LAO REPORT for certain state-funded programs will further relevant agencies link with all other agencies enhance the usefulness of California’s workforce serving the same participants. Moreover, the data. Similarly, recent efforts to use common adult model is inefficient both administratively and education outcome measures across all providers is technically. It requires that each agency’s legal resulting in more useful data. counsel negotiate individual agreements with . . . But State Still Does Not Apply every other participating agency. In addition, Common Measures to All Programs. The state’s the model often requires an agency to develop a WIOA-funded programs, CalWORKs employment unique technical approach for working with every and training program, CCC workforce programs, other participating agency because data are stored and various other state programs (such as the CTE in different formats. Similarly, the surveys some Pathways Program and the Career Pathways Trust) agencies have developed to report outcomes in the still do not all share a set of common measures. absence of linked data are relatively burdensome and unreliable. In some cases, programs do Current Method of Linking Data Falls Short not report any outcome data, despite reporting Current Efforts to Link Data Inefficient and requirements. The nearby box provides an example Burdensome. The state’s current linking model, of a longstanding program that has been unable to the agency-to-agency model, is a fragmented report longer-term outcomes due to lack of linked and time-consuming method to link data across data. agencies. It also is not comprehensive, as not all Longstanding Program Demonstrates Fundamental Flaws in Existing System Career Technical Education Pathways Program Has Certain Data Requirements. Chapter 433 of 2012 (SB 1070, Steinberg) reauthorized the Career Technical Education (CTE) Pathways Program, originally created by Chapter 352 of 2005 (SB 70, Scott). The program provides grants to consortia that include community colleges and high school districts. The primary goal of the program is to help regions develop sustainable policies to improve CTE pathways among schools, colleges, and industry organizations. The state has provided $586 million to date for the program. State law requires grantees to submit outcome data annually on (1) transitions from high school to postsecondary education and training and (2) wages of program participants. State law requires the California Community Colleges (CCC) Chancellor’s Office, in turn, to submit annual reports on the program to the Legislature and Governor. Lack of Systematic Data Sharing Results in Data Requirements Not Being Met. The CCC Chancellor’s Office repeatedly has reported that evaluators cannot reliably assess longer-term outcomes of the program because grantees generally do not submit the required information. This is because no systematic way exists to link individual high school student data with longer-term outcomes such as postsecondary enrollment and wages. The 2014 CTE Pathways Program Annual Report noted, “few K-12 districts and colleges tracked student-level information for the Initiative, and in some cases when data were tracked, the information gathered was insufficient to follow students from one segment of the education system to the next or from school to employment.” www.lao.ca.gov Legislative Analyst’s Office 15 AN LAO REPORT Systematically Linked Data Could Improve would be much more systematic than under the Decision Making. Policy makers often must rely agency-to-agency model. on anecdotal evidence to make decisions because Central Repository Model Has Certain programs are limited in the outcomes they can Advantages Over Federated Model . . . On the reliably report without linked data. By contrast, one hand, the central repository model has the availability of systematically linked data could been used by many states for many years and provide information about the types and volume uses proven technology. It also may yield more of workforce education services currently provided reliable data than the federated model because across the state; the number of distinct individuals the central agency typically reviews and validates receiving these services; patterns of movement the data submitted by participating agencies to for participants across these services; short-term the repository. In addition, data queries under and long-term outcomes of programs, including this model tend to be conducted more quickly, as participants’ employment and wage gains at responding to them does not involve pulling new various intervals following service delivery; and data from several agencies. alignment of workforce education services with . . . But Federated Model Might Be More regional economic needs. Common and linked data Suitable Today, and in California Context. On also could permit service providers to compare the other hand, the federated model has become participants’ outcomes across programs and more common in recent years due to certain examine factors that contribute to program success. advances in technology and information science. Largely as a result of these advances, the federated Other Linking Models Offer Notable model improves data security by avoiding Advantages Over Current Method creation of a separate, large, linked, continuously Both Central Repository and Federated maintained data repository for which security Models Better Than Agency-to-Agency Model. breaches are particularly serious. In the states Both the central repository and federated models currently using the federated model, the model would be more efficient than the agency-to-agency also appears to have cultivated greater acceptance model because data sharing would be governed by by participating agencies due to avoiding conflicts a comprehensive master agreement rather than the regarding control and stewardship of data. This numerous separate agreements that each agency is because data are not stored centrally or long otherwise would have to negotiate. Moreover, both term under the federated model. In California, the central repository and federated models would an additional advantage of this model could be provide more accurate and complete information greater acceptance by policy makers who might about participant outcomes because data linking be reluctant to create a state agency to oversee a central, longitudinal education data repository. RECOMMENDATIONS To help policy makers, providers, current workforce education and training programs, we and prospective participants, and the public gain recommend (1) finalizing and applying common access to linked, longitudinal data that would performance measures for all of the state’s help them make better decisions regarding workforce education programs; (2) establishing a 16 Legislative Analyst’s Office www.lao.ca.gov AN LAO REPORT statewide structure for linking data across agencies Streamline Data Linking Among Agencies to determine programs’ longer-term outcomes; and Replace Agency-to-Agency Agreements With a (3) improving how data are used by communicating Systemwide Model. We recommend the Legislature results, incorporating information into policy and direct the CWDB to study and report on potential budget processes, and facilitating policy research. solutions to developing a statewide, streamlined We discuss each of these recommendations in more data-linking approach for all workforce programs. detail below. The CWDB already has begun working with various agencies to explore such a system, so Develop and Use Common Measures we believe the task force could report back to Convene Group to Establish Common the Legislature by July 2017. At that point, the Performance Measures. We recommend the Legislature could weigh the trade-offs of potential Legislature direct the CWDB, which includes solutions. Regardless of the preferred solution, the representation from all major workforce providers, system could be developed through collaboration to lead a task force that would resolve remaining among the participating agencies. We recommend inconsistencies among performance measures the Legislature require workforce education and for the state’s workforce programs. We anticipate training programs, as a condition of receiving that the resulting measures would include the new state funding, to participate in the new system, as federal WIOA measures plus a small number of applicable, to provide specified performance data. additional common measures that reflect other state priorities. The task force’s work should include Use Data to Inform State Budget the development of detailed data definitions, and Policy Decisions reporting schedules, and other specifications Once the state implements common needed to ensure that agencies collect and report measures and a streamlined data-linking model, data to state and federal authorities in a consistent the Legislature can start using the resulting and efficient way. To ensure broad stakeholder information to inform its workforce funding and input and buy-in, we recommend the Legislature policy decisions. direct the group to consult with workforce and Require Transparent, Periodic Performance economic development officials, employers, and Reporting. Regardless of the model the state adopts other agencies that administer workforce programs to link workforce data, we recommend tasking as well as provide a public comment period. the CWDB with creating a small number of Require Programs to Use the Measures. standardized reports that will communicate results Following identification of statewide, common clearly to policy makers, providers, participants, performance measures for workforce education and the general public. Several states, for example, programs, we recommend the Legislature amend have implemented online performance dashboards state law, as needed, to align data reporting to help users find and explore performance data. requirements with the common measures. Beyond state policy makers, local providers could Legislation also could clarify to which programs use the data to better guide program development, the common measures will apply by offering a clear and interested individuals could use the reports to definition of workforce education and training choose workforce education and training programs programs. that are suitable for them. www.lao.ca.gov Legislative Analyst’s Office 17 AN LAO REPORT Incorporate Information Into Budget and Require Access to Data for Policy Research. Policy Process. We recommend the Legislature We recommend directing workforce education annually review statewide workforce education agencies to develop and adopt a uniform performance data as part of its budget and policy protocol by which researchers could gain processes. Specifically, the relevant legislative access to data. Such a protocol should contain committees could ask state agencies administering appropriate safeguards for participant privacy and workforce programs to discuss their performance confidentiality. By conducting third-party analysis results in hearings. This would give committees the of workforce data, researchers could help inform opportunity to ask agencies questions about their state policy and fiscal decisions, thereby increasing programs’ effectiveness and describe their plans for the value of the data collected. improvement. CONCLUSION The state’s current hodgepodge of workforce agreements, and use labor-intensive surveys and data requirements is cumbersome for agencies other data collection strategies to compensate for to manage and not very useful for informing lack of outcome data that could be available from state policy. Our recommendations to implement other agencies. Taken together, we believe that our common measures and a statewide approach to recommendations for improving the alignment, data linking would reduce the administrative integration, and accessibility of workforce data burden for state agencies that often struggle to would provide more useful information to help meet disparate reporting requirements for their guide state policy and funding decisions and programs, negotiate cumbersome data-sharing ultimately result in better workforce education outcomes for participants and the state. 18 Legislative Analyst’s Office www.lao.ca.gov AN LAO REPORT APPENDIX Workforce Education and Training Programs in California Agency Description California Community Colleges (CCC) Apportionments for workforce education Ongoing Proposition 98 funds allocated to community college districts for credit and and training noncredit courses in basic skills, English as a second language (ESL), and career technical education (CTE). Workforce education and training comprises about one-third of CCC apportionment funding. Strong Workforce Program Ongoing Proposition 98 General Fund for community colleges to provide regionally focused CTE and workforce programs leading to certificates, degrees, and other credentials. Funds can be used for various purposes, including purchasing equipment, convening consortia, redesigning curriculum, and making other enhancements to CTE instruction. Student Services for CalWORKs Ongoing Proposition 98 funding and federal funding for community colleges to provide child Recipients care, work study, and job placement services to students receiving CalWORKs assistance. (CalWORKs provides cash aid and services to low-income individuals and families.) The CCC Chancellor’s Office distributes funding to colleges based on enrolled CalWORKs recipients. Economic and Workforce Development Ongoing Proposition 98 funding to help community colleges identify regional workforce Program education and training needs in collaboration with (1) employers; (2) two advisory committees representing colleges and industry; and (3) business, industry, and economic development partners. Funds can be used for hiring CCC industry liaisons, supporting collaboration among stakeholders, and providing technical help to regional consortia in using workforce data. Nursing program support Ongoing Proposition 98 funding for community colleges to increase the number of nursing program graduates. The CCC Chancellor’s Office allocates a portion of funds as a per-student supplement to expand or maintain capacity, improve student readiness for courses, help students prepare for national licensing exam, and provide faculty professional development. The Chancellor’s Office allocates another portion as a fixed amount for student assessment and retention activities. California Conservation Corps Core Training Program Ongoing non-Proposition 98 General Fund to provide Corps members with education and training services, including high school diploma and General Educational Development (GED) test, technical skills, career guidance, and job search assistance. California Department of Corrections and Rehabilitation (CDCR) Office of Correctional Education Ongoing non-Proposition 98 General Fund to provide academic and CTE programs to programs incarcerated adults at adult state prisons. The overall objective is to reduce recidivism. Prisons offer basic skills, CTE, and high school diploma and equivalency programs. Office of Offender Services workforce Primarily ongoing non-Proposition 98 General Fund to support various programs that programs prepare offenders for release and provide employment preparation, transitional employment, and job placement assistance upon release. The in-prison Transitions Program provides a curriculum for offenders on how to get and retain a job as well as information about services offered at America’s Job Centers of California. Reentry programs include (1) the Caltrans Parolee Work Crew Program (overseen by CDCR and Caltrans) that hires parolees to clear litter from roadways, and (2) the Female Offender Treatment and Employment Program that provides CTE training and employment services to female offenders. (Continued) www.lao.ca.gov Legislative Analyst’s Office 19 AN LAO REPORT Agency Description California Department of Education (CDE) Adults in Correctional Facilities Ongoing Proposition 98 funding to county offices of education (COEs) and school districts that provide educational programs to inmates at county jail facilities. Coursework varies and the state does not track participation by subject area. Providers create memoranda of understanding with jails and apply to CDE to receive funding based on average daily attendance. Agriculture Incentive Grants Ongoing Proposition 98 funding for high schools to support nonsalary agricultural education costs. Funds are commonly used to purchase equipment and pay for student field trips. Requires local match. California Partnership Academies Ongoing Proposition 98 funding to high schools to operate small learning communities that integrate a career theme into academic classes in grades 10 through 12. Conditions of funding include a private sector match, an internship or work experience for students, and a common planning period for academy teachers. Student Services for CalWORKs Ongoing Proposition 98 and federal Temporary Assistance to Needy Families (TANF) Recipients funding for adult education programs and Regional Occupational Centers and Programs (ROCP) to provide adult education and training that leads to employment for students receiving CalWORKs assistance. CDE distributes funding to providers based on enrolled CalWORKs recipients. CTE Incentive Grants Proposition 98 funding for a three-year competitive grant program to support CTE. School districts, COEs, charter schools, and joint powers agencies (JPAs) may apply. Applicants that do not currently operate CTE programs, regions with high dropout rates, and rural areas receive funding priority. Requires a local match and ongoing commitment to fund programs after grant ends. Local Control Funding Formula (LCFF) Ongoing Proposition 98 funding for a 2.6 percent LCFF add-on to the base rate for high school students. Some combination of base and add-on funds is intended to support the costs of offering CTE instruction. The add-on originally was calculated to reflect ROCP funding. (Districts also have discretion to use LCFF funds to support adult education.) Project Workability Ongoing Proposition 98 funding for pre-employment training and employment placement for high school students in special education. Students are placed in employment and the program fully subsidizes their wages until they complete high school or turn 22 years old. Regional Occupational Centers and Education agencies may choose to use their general purpose Proposition 98 funding for Programs regionally focused CTE at high schools and regional centers. (Prior to 2013-14, the state funded ROCP through a categorical program.) Primarily serves high school students ages 16 through 18. Specialized Secondary Programs Ongoing Proposition 98 funding for short-term competitive grants for school districts to pilot programs that prepare students for college and career. Ongoing Proposition 98 funding also supports two high schools specializing in math, science, and the arts. CDE and CCC Joint Programs Adult Education Block Grant Ongoing Proposition 98 funding allocated by the CCC Chancellor’s Office to regional consortia of community colleges, schools districts, COEs, and JPAs. Consortia may offer adult education in seven areas of instruction: basic skills, CTE, ESL and citizenship, programs for adults with disabilities, workforce programs for older adults, certain caregiving programs for older adults, and pre-apprenticeship programs. Funding is based on consortia’s prior-year funding, performance, and regional need. Adult Education and Family Literacy Ongoing federal Workforce Innovation and Opportunity Act (WIOA) Title II funding allocated Program by CDE to numerous adult education providers, including adult schools, community colleges, libraries, and community-based organizations. CDE distributes funding based on student learning gains and other outcomes. (Continued) 20 Legislative Analyst’s Office www.lao.ca.gov AN LAO REPORT Agency Description Apprenticeships Ongoing Proposition 98 funding allocated by the CCC Chancellor’s Office to schools and community colleges to help support the classroom instruction component of apprenticeship training. Apprenticeships are paid, educational work programs that pair students with skilled workers for supervised, hands-on learning, typically in the skilled trades. Apprenticeships last from two to six years and commonly are sponsored by businesses or labor unions that help design and support the programs and recruit apprentices. Career Pathways Trust One-time Proposition 98 funding for two rounds of competitive grants administered by CDE in 2013-14 and 2014-15. Grants fund regional consortia of schools and community colleges partnering with local businesses to improve linkages between CTE programs and local workforce needs. Authorizes several types of activities, such as creating new CTE pathways, articulation agreements, and curriculum. Requires local match. Grantees have three years to spend the funding. Carl D. Perkins Career and Technical Ongoing federal funding allocated by CDE to schools, community colleges, and correctional Education Act Program facilities. May be used for a number of CTE purposes, including curriculum and professional development and the purchase of equipment and supplies for the classroom. Of these monies, 85 percent directly funds local CTE programs and the other 15 percent supports statewide administration and leadership activities, such as support for CTE student organizations. CTE Pathways Program Limited-term Proposition 98 funding (partly from the Quality Education Investment Act) administered by the CCC Chancellor’s Office to improve linkages among CTE programs at schools, community colleges, universities, and local businesses. Program has funded various activities, including developing CTE courses that meet college acceptance requirements, supporting CTE student organizations, and supplementing some related CTE programs, including the California Partnership Academies. Funding renewed annually since 2005. Scheduled to sunset June 30, 2017. California Department of Rehabilitation Vocational Rehabilitation Ongoing federal WIOA Title IV funding (and some state General Fund) to provide vocational rehabilitation services for adults and youth with disabilities, including employment, education, and job placement assistance. Funds career assessment and counseling, job search and interview skills training, career training, and assistive technology such as hearing aids. California Department of Social Services CalFresh Employment and Training Ongoing federal Supplemental Nutrition Assistance Program funding, with a county match, Program to operate employment and training programs for CalFresh recipients in select counties. (CalFresh provides federally funded food assistance to low-income individuals and families.) Services are generally similar to CalWORKs services. Locally administered by county human services departments. CalWORKs employment and training Ongoing federal TANF funding, state funding, and county funding for employment services services for very low-income families with children on CalWORKs. Services include job search assistance, mental health and substance abuse treatment, referrals to education and training, and on-the-job training. Locally administered by county human services departments. California Employment Development Department (EDD) Adult, Youth, and Dislocated Worker Ongoing federal WIOA Title I funding for America’s Job Centers of California (formerly Services known as OneStops). These centers provide workforce information, resources, and employment services to adults, youth, and dislocated workers. Services include job search assistance, career assessment, career counseling, on-the-job training, and adult education and training. Funds also support education and job programs, including YouthBuild and Job Corps, for youth ages 16-24 who are not in school or employed. (Continued) www.lao.ca.gov Legislative Analyst’s Office 21 AN LAO REPORT Agency Description Employment Training Panel Reimbursements from the state Employment Training Tax to support retraining programs for current employees and companies facing out-of-state competition, training programs for recipients of unemployment benefits, and training programs for employers that meet certain criteria, such as those that are located in regions of the state where the unemployment rate is significantly higher than the state average. Proposition 39 pre-apprenticeship Special funds from the Clean Energy Job Creation Fund for competitive grants to regional support, training, and placement workforce partners to implement and support “green” pre-apprenticeships that lead to industry-valued credentials, entry into apprenticeship, or direct employment in the energy- efficiency workforce. Funds may be used to provide training, support services, and job placement assistance. Regional Workforce Accelerator Program Discretionary federal WIOA Title I funds the state has chosen to use for providing competitive grants to regional workforce partners that use innovative strategies to address gaps in education and workforce, with the goal of replicating and applying the strategies to other regions of the state. SlingShot Discretionary federal WIOA Title I funds the state has chosen to use for providing competitive grants to regional workforce partners to support alignment of job seekers and market demand. Grantees must submit a plan that identifies a workforce challenge in the region and a strategy to address it. Requires local match. Jobs for Veterans State Grant Ongoing federal WIOA Title I funding to provide workforce services to veterans at America’s Job Centers of California. Services include assessments of education, skills, and abilities; career planning; and work-readiness skills training. Funding distributed based on the number of veterans seeking employment. Wagner-Peyser Employment Services Ongoing federal WIOA Title III funding to provide services to connect job seekers with available positions in the labor market. EDD works with employers to list job openings on an open online database known as CalJOBS. California Prison Industry Authority (CalPIA) Offender Development programs Funds collected through the sale of CalPIA inmate-produced goods support inmate CTE and employability programs. CalPIA partners with trade unions to provide CTE to inmates and operates the Inmate Employability Program, which requires CalPIA factory supervisors to help inmates develop work habits and job application materials, such as portfolios. 22 Legislative Analyst’s Office www.lao.ca.gov AN LAO REPORT www.lao.ca.gov Legislative Analyst’s Office 23 AN LAO REPORT LAO Publications This report was prepared by Natasha Collins and Judy Heiman and reviewed by Jennifer Kuhn. The Legislative Analyst’s Office (LAO) is a nonpartisan office that provides fiscal and policy information and advice to the Legislature. To request publications call (916) 445-4656. This report and others, as well as an e-mail subscription service, are available on the LAO’s website at www.lao.ca.gov. The LAO is located at 925 L Street, Suite 1000, Sacramento, CA 95814. 24 Legislative Analyst’s Office www.lao.ca.gov