Improving Higher Education Oversight

Improving Higher Education Oversight M A C TAy l o r • le g i s lA Ti v e A nAl y sT • J A n uA r y 6 , 2012 An LAO RepOR t 2 Legislative Analyst’s Office www.lao.ca.gov An LAO RepOR t ExEcutIvE Summary The Governor vetoed funding for the California Postsecondary Education Commission (CPEC) in the 2011-12 budget, citing the agency’s ineffectiveness in higher education oversight. In his veto message, the Governor acknowledged the well-established need for coordinating and guiding state higher education policy and requested that stakeholders explore alternative ways these functions could be fullfilled. The commission shut down in fall 2011, transferring a federal grant program to the California Department of Education (CDE) and extensive data resources to the California Community Colleges (CCC) Chancellor’s Office. The future of higher education oversight is unclear in the wake of CPEC’s closure. The public segments have stepped in to assume some roles previously performed by CPEC, raising concerns about how institutional and public interests will be balanced. In this report, we focus on the need for oversight that enables policymakers and others to monitor how efficiently and effectively the postsecondary system is serving the state’s needs, and make changes to improve its performance. We present a conceptual model of how the state can improve higher education oversight, and provide specific recommendations for improvement, both short and long term. These recommendations respond to supplemental report language approved in the 2011 legislative session seeking our office’s suggestions on the structure and duties of a statewide higher education coordinating body for California. In preparing the report, we consulted with a broad range of stakeholders as directed in the language. LAO Recommendations. We recommend several steps the Legislature could take to begin estab- lishing an effective oversight structure: • Defining the state’s postsecondary education needs. This could involve setting specific goals, as many states have done, or identifying key areas or outcomes of interest to the state. In either case, we would encourage the Legislature to prioritize among identified needs. • Assigning a technical work group to define measures of higher education performance and collect baseline data. • Using performance results to inform policy decisions. We suggest regular oversight hearings as part of a feedback cycle for budget and policy decisions. • Establishing an independent oversight body with limited and clear responsibilities. We offer several specific recommendations on the structure and duties of this body. We recognize the difficulty of creating a new public organization in the current fiscal environment, and offer a number of short-term measures to strengthen oversight in the interim: • Amending statute to ensure pertinent data remain available to policymakers and researchers. www.lao.ca.gov Legislative Analyst’s Office 3 An LAO RepOR t • Increasing direct legislative oversight and limiting new long-term funding commitments until an effective oversight structure is in place to support the Legislature’s decision-making. • Monitoring segments’ allocation decisions, including investment in new programs and other major program changes, until mechanisms are in place for outcome review. Taken together, we believe these recommendations would improve higher education oversight in the short term while providing an opportunity for the Legislature to establish an effective, outcome- oriented process to improve the performance of the higher education system in meeting the state’s priorities. 4 Legislative Analyst’s Office www.lao.ca.gov An LAO RepOR t IntrOductIOn This report responds to supplemental report the need for oversight as a set of functions, more language, approved in the 2011 legislative deliberate and robust than coordination, whereby session, seeking our office’s recommendations public policymakers articulate what they want on the structure and duties of a statewide higher from the state’s higher education system, assess education coordinating body for California. The what it is producing, and make changes to bridge Legislature requested the report after rejecting the gap between the two. Earlier models of coordi- an administration proposal to phase out CPEC. nation were designed to guide the development However, the Governor effectively reversed the of a growing higher education system. California Legislature’s decision and vetoed state funding for now needs a new model to guide the efficient and CPEC, forcing its closure in fall 2011. effective use of its established and extensive higher The elimination of CPEC makes more urgent education resources. the task of ensuring effective coordination of The report includes both longer-term recom- California’s higher education system. In fact, we mendations for creating a new state oversight believe it exposes a more fundamental question structure with a formal agency, as well as interim about the extent to which the state needs to ensure steps the Legislature could take to help guide not just coordination but oversight of its higher the state’s postsecondary education policy in the education system. The Master Plan and statutory absence of a new agency. In preparing the report, provisions related to CPEC directly and indirectly we consulted with a broad range of stakeholders as refer to both concepts, but neither term is well directed in the supplemental report language. defined. For this report, we focus primarily on BackgrOund need for coordination and misalignment of student education pathways, and Oversight Well Established other inefficiencies which make it unlikely that the collective efforts of the state’s public and private Promoting an Integrated System. In a 2010 postsecondary institutions will meet the state’s report, we detail the need for coordination of needs. the state’s higher education system and trace Protecting the Public Interest. Moreover, insuf- the history of repeated Legislative attempts to ficient state oversight could allow state priorities strengthen this function. That report, Greater to be subordinated to those of the institutions and Than the Sum of Its Parts—Coordinating Higher other interests. These have been long-standing Education in California, underscores the potential concerns. As long ago as 1960, the Master Plan value of coordination. If California’s diverse for Higher Education in California found that the postsecondary education segments act as an segment’s own governing boards could not be relied integrated system in which each part makes its on to oversee the state’s higher education system on own contributions toward achieving a common their own. Accordingly, the Master Plan called for purpose, then their combined efforts may add up a state oversight body, observing that it “is increas- to more than what the institutions could achieve ingly obvious that enforcement will require more independently. Insufficient coordination, on the sanctions than are available at present.” other hand, gives rise to duplication of efforts, www.lao.ca.gov Legislative Analyst’s Office 5 An LAO RepOR t Duties Assigned to Coordinating Body. Several Persistent Oversight concerns Span decades specific coordination functions were recom- The structure of California’s higher education mended in the Master Plan and statutorily assigned system reaffirmed in the Master Plan—comprising to a Coordinating Council and later to CPEC. three distinct public segments differentiated by The Legislature assigned numerous additional eligibility pools and functions, and a number of duties to CPEC over the years, to the extent that independent and proprietary institutions—has core oversight functions were diluted by myriad been widely credited with facilitating the orderly reporting requirements and other tasks. This is now growth of high-quality, relatively low-cost generally understood to have weakened CPEC’s educational opportunities in the 1960s. By the effectiveness. early 1970s, however, concerns surfaced about the Functions Prioritized. In 2008, recognizing state’s ability to sustain the Master Plan’s vision that CPEC could not perform all of the functions of a unified, coordinated system. These concerns and tasks assigned to it, the Legislature adopted intensified over the ensuing decades as numerous statutory language prioritizing four functions: reviews, studies, and reports identified “mission (1) reviewing and assessing proposals for new creep” across segments, decline of the transfer public campuses and facilities, (2) reviewing and function, a dearth of comprehensive data, and assessing proposals to create new programs at the inadequate planning as serious problems. public higher education segments, (3) serving as Some Concerns Directed at Coordinating the designated state educational agency to carry Body. The reports also cited broad and incom- out federal education programs and, (4) collecting patible roles for CPEC, the composition of its and managing higher education data. Missing governing board, and lack of state support and from this list are other duties generally considered follow-through as barriers to improving oversight. central to oversight, including planning, evaluating • Several reports identified an intrinsic effectiveness, and participating in the executive conflict between CPEC’s coordinating and legislative budget processes. We believe the function, which required that it maintain omission of these key oversight duties from the positive relationships with the segments, priority list reflects the Legislature’s lack of confi- and its charge to produce objective and dence in CPEC’s ability to perform them effectively. critical policy analysis with which the Data Role Primary. In recent years, it became segments may strongly disagree. evident that data functions—including maintaining a public website for data access—were the CPEC • Some noted that segmental representatives functions most highly valued by most stakeholders on the commission tended to dominate including the segments, policymakers, analysts, CPEC’s agenda, raising issues about the and researchers. Many other CPEC functions were commission’s objectivity. largely disregarded. Although the Legislature had • Declining state financial support hampered prioritized facility and program review, CPEC had CPEC’s effectiveness. Between 2001-02 and not been influential in these areas in recent years. 2009-10, its General Fund budget declined While the commission’s administration of a federal by more than 60 percent (adjusted for teacher quality grant was well respected among inflation), seriously eroding its capacity. participating institutions, this role was largely invisible to the broader public. 6 Legislative Analyst’s Office www.lao.ca.gov An LAO RepOR t • More importantly, policymakers General Services collected office furnishings and sometimes ignored CPEC’s recommenda- equipment, and is seeking a new tenant for the tions, further marginalizing the organi- office space. The Department of Finance (DOF) zation and making it difficult to attract has estimated closeout costs at roughly $850,000, effective leadership. including final payments for salaries, leave payouts, and other benefits, and suggested that the Governor Responding to these longstanding concerns may include the final amount in his 2012-13 budget about CPEC, several Governors and Legislators proposal. have attempted to reform, combine, replace, or eliminate CPEC. More than a dozen proposals data transferred to ccc chancellor’s Office from the Legislature and the administration The commission gained the ability to collect were introduced in as many years. No significant and aggregate student records from the three reforms, however, were enacted. segments relatively recently. Although Chapter 916, governor Eliminates cPEc through Budget Statutes of 1999 (AB 1570, Villaraigosa), required the segments to provide student records, the University Governor’s “Blue Pencil” Eliminates CPEC of California (UC) and California State University Funding. The Governor proposed eliminating (CSU) did not begin doing so until the mid-2000s. CPEC in his May Revision budget proposal. Under The data warehouse CPEC developed with these this proposal, the agency would have been phased records proved useful for understanding higher out over a period of six months with the exception education processes and outcomes in California. of one function—management of a federal teacher Prior to closing down, CPEC transferred its data quality training grant—that would have been warehouse to the CCC Chancellor’s Office. moved to CDE. The Legislature rejected the May What Data Resources Did CPEC Have? Revision proposal, and instead adopted supple- The commission compiled two main categories mental report language calling for this review. In of higher education data: signing the 2011-12 Budget Bill, the Governor used line-item veto authority to eliminate all General • Individual Student Records. The Fund support for CPEC. Although the commis- commission maintained individual sion’s statutory authority remains intact, the agency student records from each of the public was forced to close in fall 2011. higher education segments dating back CPEC Winds Down. Although the Governor’s to 1993 for the CCC and 2000 for UC funding veto was effective July 1, CPEC’s closeout and CSU. Information contained in these required about five months to complete. During records includes high school of origin, that time, CPEC transferred its reports and postsecondary enrollment history, program historical materials to the State Archives and the of study, transfers, completions, degrees California State Library, and its federally funded awarded, and demographic information. Improving Teacher Quality State Grants Program The commission was able to link data across to CDE. Most of CPEC’s 21 staff members retired the three segments using unique student from state service or found other positions. When identifiers. For example, a community it closed its doors on November 19, CPEC laid off college record for a transfer student could be two remaining staff members. The Department of matched with the corresponding university record. www.lao.ca.gov Legislative Analyst’s Office 7 An LAO RepOR t • Aggregate Data From Other Public Where Did the Data Go? Shortly before its Sources. Fulfilling its role as a clearing- closure, CPEC transferred its database servers to house for California higher education the CCC Chancellor’s Office. (The administration, data, CPEC collected publicly available however, did not transfer the associated funding data sets from federal sources including and personnel.) CDE, Census Bureau, and Bureau of Labor Under federal privacy laws, each segment is Statistics, as well as the state Employment permitted access only to its own student data and Development Department (EDD) and should not have access to individually identified other sources. The commission extracted student records from other segments. (See nearby California data from these data sets and box for a description of federal and state privacy made the resulting information readily provisions.) Accordingly, CPEC’s transfer of the available through its public website. data warehouse to CCC could be considered unallowable disclosure of student records to a third How Did CPEC Use These Data? Commission party. Recognizing this, UC and CSU formally staff used these data resources to meet the agency’s requested that CPEC return to them their own research and reporting responsibilities. In some students’ records. Instead of returning the data cases, research was conducted in collaboration directly to each segment, however, the universities with other agencies, including CDE, colleges and requested that CPEC transfer their data to CCC, universities, and EDD. In addition, CPEC made which has agreed (through a formal interagency much of its data available to the public through agreement) to manage the data on their behalf. preconfigured reports and an interactive reporting Through this legal maneuver, the segments believe tool on its website. (To protect student privacy, only they have satisfied the privacy protections in the aggregate data with no individual student identi- law while preserving the value of the interseg- fiers was made available to the public.) mental data resources CPEC had assembled. Examples of reports available from CPEC’s Although it is now stored and hosted in new public website include: locations, the public website remains available at • Freshman Pathways: flow of recent high CPEC’s same Internet address. Individual student school graduates to public colleges and data is on a secure server under the management universities, by campus. of CCC Information Systems staff, with separate “partitions” on the server for each segment’s data. • Transfer Pathways: flow of community Eventually, CCC plans to move the data to the college students to public four-year colleges Corporation for Education Networks in California and universities, by campus. (CENIC), a membership organization that operates • Trend Analyses: data over time. the high-speed Internet “backbone” through which schools and other educational institutions • Custom Data Reports: California students in California connect to the broader Internet. by gender, age, ethnicity, student level, While CENIC will provide the infrastructure for institutions, and segment. the databases, management will remain with CCC • Pre-Configured Reports: various quick under current agreements. This may not be a viable data reports located on the commission’s long-term solution, however, as discussed later in website. this report. 8 Legislative Analyst’s Office www.lao.ca.gov An LAO RepOR t Federal and State Privacy Laws The Family Education Rights and Privacy Act (FERPA) is the major federal law affecting access to student records. The main objectives of FERPA are to (1) ensure students and their parents have access to all information in a student’s official academic record, and (2) prevent unauthorized access to individually identifiable information in a student’s records. Under this law, individually identifiable student data may not be disclosed without the student’s or parent’s written consent in most circumstances. The entity directly collecting student data has the responsibility for ensuring compliance with the law. According to federal privacy officials, only statewide education authorities with program evaluation responsibilities may maintain intersegmental data under FERPA. California’s privacy laws also include provisions limiting who can receive individually identifiable data from a state agency or department. In addition, state laws require review by an independent review board for any new release of individually identifiable information to a researcher. Separately, CDE is seeking a federal grant in fall 2011. The next update is due in fall 2012, to support a statewide kindergarten through and will require CCC to use personally identi- postsecondary (K-20) data system. If its appeal is fiable information in student records from each successful, this funding could help support inter- segment. Federal privacy officials were especially segmental data resources. concerned about CCC’s legal authority to perform Role of CCC May Be Problematic. The transfer this update or any other studies that involve using to CCC of CPEC’s data warehouse may not be student identifiers across segments without specific a viable long-term solution for management of statutory authority for postsecondary data. intersegmental data. Federal privacy officials with Alternatives for Maintaining Data Resources. whom we consulted expressed concerns about The CCC is providing a valuable service to the state whether the current arrangement (in which CPEC by accepting CPEC’s data warehouse, immediately “technically” returned data to each segment making the CPEC website and associated data while physically transferring it to CCC) meets resources available to the public and securing federal privacy requirements. They agreed this student record data on behalf of the segments. With arrangement could comply with federal privacy no other transition plan in place, it is possible that laws if the CCC were determined to be a statewide the extensive data resources CPEC had assembled education authority with assigned responsibility would otherwise have become unavailable. For the for data collection and program evaluation across long term, however, there remain legal and policy postsecondary education. Such a designation would issues to be resolved. likely require a statutory change to provide the Federal privacy officials have confirmed that necessary authority. CPEC’s data warehouse legally could be transferred Questions regarding data access will have to to a different state oversight body for postsecondary be resolved this year if the data warehouse is to education. They pointed out that state depart- be kept current. With the segments’ cooperation, ments of education commonly receive student CPEC completed its annual update of student data data resources when other entities holding student www.lao.ca.gov Legislative Analyst’s Office 9 An LAO RepOR t records (K-12 or postsecondary) are terminated. strengthened CPEC’s statutory authority to require As statewide education authorities with data data, after some frustration with segment responses collection and program evaluation responsibilities, to state data requests. education departments can take over management Future of coordination unclear of the records without constituting disclosure of the information. It is unclear, however, whether The Governor’s veto message acknowledges CDE currently possesses the statutory authority the importance of higher education coordination to maintain intersegmental data. The state could and cites CPEC’s ineffectiveness, rather than a grant the necessary authority to CDE, CCC, or diminished need for coordination, as the reason for another entity designated as a statewide education the veto. In the message, the Governor requests the authority. three public higher education segments, along with The segments have laid some groundwork for stakeholders, to explore alternative ways to improve a Joint Powers Authority (JPA) to oversee interseg- coordination and development of higher education mental data. These efforts are on hold due to the policy. difficulty of creating new public organizations in In the wake of CPEC’s closure, the segments the current fiscal environment. In addition, discus- have stepped in to assume roles previously sions with federal officials suggest a JPA could performed by CPEC in two areas: encounter the same or greater legal problems as the • The CSU Board of Trustees has initiated current arrangement. a revision of the university’s comparison Policy Considerations. In addition to the groups for executive compensation. legal questions, a significant policy issue concerns Previously, CPEC was responsible for devel- control of intersegmental data. Under the current oping comparison groupings for various arrangement, each segment considers that it has purposes, including tuition levels and sole control over access to its own student records. executive compensation, using a process If outside analysts wish to use information from agreed upon by administration, legislative, these records for state policy purposes (for example, and segment representatives. The CSU’s to study transfer outcomes by institution), they process departs from the agreed-upon need the approval of the segments involved. After methodology in several significant ways, approval, CCC (or CENIC as a contractor) would and involves no direct legislative or admin- match the specified data and provide a file to the istration participation (although comments analysts. In contrast, CPEC was able to perform from LAO and DOF were solicited after such studies on behalf of the state and provide prompting by a legislative committee). data access to researchers without having to secure individual approval from the segments. • The segments have reestablished the We believe there is a potential conflict of student data warehouse that CPEC had interest in relying on the segments for permission created as a public resource. The segments to study their performance and that of their contend that policymakers, legislative students. For this reason, we believe it is in the analysts, and researchers will need their state’s best interests for a third party independent of explicit permission to conduct a study or the segments to maintain the data. The Legislature evaluation using these data to the extent previously assigned this role to CPEC, and later the data are not already publicly available. 10 Legislative Analyst’s Office www.lao.ca.gov An LAO RepOR t As described in our earlier report on coordi- and data resources may be an early indication that nation, institutions and their governing boards the absence of a formal coordinating structure have their own interests and priorities, which do increases the opportunities for them to emphasize not always match the broader public interest. The their institutional interests. segments’ recent actions on comparison groups ImPrOvIng HIgHEr EducatIOn OvErSIgHt Purpose of Oversight. In our view, higher as cohesive as it had been in earlier periods. Our use of education oversight enables policymakers (and the term vision was deliberate. Our view of oversight to some extent, other audiences) to monitor how presumes public policymakers have envisioned what efficiently and effectively the system is serving the success looks like, and thus what the system should be state’s needs, and make changes to improve its achieving. This is not a trivial requirement. The higher performance. This view of oversight focuses on education system is a complex enterprise with multiple outcomes, and is also concerned with program, missions and many constituencies. Policy decisions administrative, and resource allocation decisions often involve difficult tradeoffs among legitimate at all levels. The state has a role in oversight of competing interests. For decades, policy experts have private as well as public higher education, although called for the state to articulate its needs in the form of the policy tools available to the Governor and clear goals and priorities for higher education. That this Legislature differ between the two. remains a work in progress in part reflects the difficulty This section presents a conceptual view of how of the task. the state can improve higher education oversight. Master Plan Provides Insufficient Guidance. For Figure 1 provides an illustration of some of the more than 50 years, the Master Plan has been looked terms we use in this section. The last section to as the primary expression of the state’s vision for provides more specific recommendations for higher education. Its emphasis on access, affordability, improvement, both short and long term. and quality are well known and invoked widely in policy discussions. However, these principles, System Should Be guided by State’s needs compelling as they may be, are insufficient to guide In an earlier policy brief we assessed the state’s policymakers in the 21st century. The Master Plan vision for higher education, concluding it was no longer principles lack specificity, are not prioritized, and fail Figure 1 Illustration of Key Terms for Higher Education Oversight Term Example Vision The purpose of the postsecondary education system in California is to align the knowledge and skills of the adult population with the civic and workforce needs of the state of California. Goal Achieve measurable, value-added student learning outcomes. Measure, Mean change in scores for individual students from freshman to senior year on specified portions Metric, or of the Collegiate Assessment of Academic Proficiency or Collegiate Learning Indicator Assessment (CLA). Target System target is for each segment to achieve mean gain scores of 50 points on performance task and 100 points on writing task of CLA (for example). Segments allocate targets to individual campuses to achieve system target. www.lao.ca.gov Legislative Analyst’s Office 11 An LAO RepOR t to take into account the many changes in California’s goals for California that were consistent population and economy over the last half-century. with Master Plan principles (see Figure 2). More recently, the Legislature’s Joint • State Lacks Specific Goals. Several Committee on the Master Plan for Higher analyses in recent years, including those Education found that the state “has no from our office, have recommended clari- articulated, comprehensive statement of fying and quantifying higher education goals for California’s system of higher goals as a way to provide better policy education,” and that “[t]he lack of these guidance than the Master Plan is able to goals makes it difficult to develop sound offer. Our 2010 report highlighted efforts systems of criteria for advancement or clear in other states and offered sample priority systems of accountability….” Figure 2 Sample Public Agenda for Higher Education in California Mission The purpose of the higher education system in California is to align the knowledge and skills of the adult popula- tion with the civic and workforce needs of the state of California. Master Plan Principles • Access to higher education for all adults who could benefit from postsecondary instruction with community colleges serving as the main entryway for the majority of undergraduates. • Affordability through general support to public institutions and financial aid for students attending public and private colleges. • High quality and cost containment through orderly growth, differentiation of functions, and coordination. Priority Goals Access and Success 1. Increase awareness of and student preparation for the demands of postsecondary education. 2. Achieve measurable value-added student learning outcomes. 3. Increase rates of program completion, transfer, licensure and certification, and job placement. Affordability 4. Adopt and maintain a fee policy that defines the share of educational costs that students pay. 5. Maintain financial aid so that all students are financially able to attend higher education. Quality and Cost Containment 6. Provide educational programs whose content, quality, and costs are aligned with state needs. 7. Reduce cost per completion in each public segment. Accountability Although the Master Plan does not include accountability, policymakers have come to recognize it as necessary for achieving priority goals. 8. Expand statewide longitudinal data collection and analysis to inform state policy decisions. 9. Align policies and funding mechanisms with priority state goals. 12 Legislative Analyst’s Office www.lao.ca.gov An LAO RepOR t • Fails to Set Priorities. In addition to almost exclusively on access, with little lacking specific goals, the state’s approach attention to progression and completion. to higher education policy does little to Additionally, policymakers in 1960 could help the administration or Legislature scarcely have envisioned the recent prolif- in prioritizing allocation of resources. eration of for-profit and online education. The state generally has not differentiated Moving Forward From Here. The Legislature funding amounts between the university has a range of possible approaches for articulating systems (despite their very different its expectations for higher education. Ideally, the outcomes), and has allowed the public Legislature could develop specific and measurable higher education segments broad discretion goals which are more likely to improve perfor- to allocate General Fund augmentations mance than vague goals. We would acknowledge and reductions within their budgets. This that it is difficult to agree on goals. There is has impeded any strategic, systemwide sometimes little analytical basis for setting a higher education funding policy. Capital specific goal in a given area, and efforts to do so outlay decisions are especially problematic could result in arbitrary or unrealistic goals. As without established priorities. The well, elected officials may place different value on absence of a rigorous independent review various aspects of higher education, making it predicated on clear priorities leaves the problematic to win agreement on priorities. Legislature vulnerable to undue influence Yet common ground can often be found from interests that would benefit from large between specific goals and general principles. investments. Indeed, one of the primary The Legislature has previously agreed on motivations for developing the Master Plan important questions to guide higher education was to manage the “almost uncontrolled policy, and has put in place a number of mostly aspirations and proposals of local commu- ad-hoc reporting requirements through statute, nities” for new facilities. budget language, and supplemental reports. Many of these reflect broadly shared concerns • Vision Outdated. The framework estab- about particular aspects of the higher education lished in the Master Plan has not kept pace system and its performance. Examples include with changes in California’s population efficiency of the transfer process, preparation of and economy. For example, in 1960 the math and science teachers, and effectiveness of writers of the Master Plan report assumed campus financial aid programs. that the people of California would provide adequate resources to support the higher Progress Should Be monitored education system and keep it tuition-free using Standardized measures for all residents of the state. This has not Whatever level of goal specificity the been the case for several decades. Likewise, Legislature is able to achieve, it can begin economists and demographers have cited monitoring the system’s performance in a more the need for an emphasis on completion systematic way. For example, the Legislature of degrees and certificates in response to could regularly review existing data about access, changes in the structure of the economy. affordability, quality, and student progression and The state’s policy focus to date has been completion for the higher education segments. www.lao.ca.gov Legislative Analyst’s Office 13 An LAO RepOR t (See Figure 3 for examples.) A consolidated report Whether linked to specific goals or used more drawing on various sources and presenting infor- generally to track performance in areas of interest mation concisely could facilitate such a review. to public policymakers, the choice of measures Several states use performance “dashboards” or matters. Poorly designed measures could give “scorecards” (such as the one illustrated on the misleading or confusing signals to policymakers cover of this report) to track measures of interest. and the public. For measures to be useful, there Other states as well as associations of state must be broad agreement that they accurately governments have developed integrated sets of reflect the parameters they are intended to measures for higher education productivity and measure, and they must be defined consistently to efficiency that California could employ. (For ensure comparability over time across segments. example, the National Governor’s Association Selecting Measures and Collecting Baseline and Complete College America have developed Data. While the Legislature may formally adopt measures of student progression, student higher-level goals, it would be impractical to completion, and system and institutional efficiency include in statute the level of detail needed to define and effectiveness.) Some of these are quite sophis- measures for monitoring progress across a diverse ticated and adjust for differences among colleges or higher education system. Instead, statute can states, while others are simpler and more intuitive. delegate technical decisions about specific measures The Legislature could adapt one or more of these and reporting protocols. frameworks for its own needs. Routine monitoring As an illustration, consider a goal of improving of the higher education system’s performance could student completion. Measuring this would require improve oversight and inform further discussion several decisions: of goals and priorities. Measuring performance • Whether to count transfers and certificate and publicizing results also focuses institutions’ awards as successful completion, in attention on the measures of importance to the addition to degree awards. state and creates an incentive for them to improve their performance. • Whether to focus on the absolute number of completions, the increase in number Figure 3 Examples of Existing Measures for Articulated Values Articulated Values Existing Measures Access • Eligibility studies • Application, admission, and enrollment reports Affordability • Total instructional costs; student and state shares • Total and net costs of attendance Quality • Passage rates on licensure and certification examinations • Value-added measures from system accountability reports Student progression • Year-to-year persistence rates • Time to degree Student completion • Numbers of completions • Completion rates 14 Legislative Analyst’s Office www.lao.ca.gov An LAO RepOR t over time, the percent increase over a $100,000 of spending. The state could allocate a baseline, the rate for a particular cohort of portion of the target to each segment and compare students over a given time period, or a rate performance against these targets. Although per 100 full time-equivalent students in a a technical group is best suited to perform the given year. analytical work of measuring current performance and assessing the feasibility of reaching certain • For any rate, which students to count in targets, the setting of targets is a policy decision. the denominator—many students enroll in Using Comparisons. Alternatively, the courses with no intention of completing a state could compare performance directly with full academic program. other states or comparison institutions. Direct comparisons across institutions can be problematic, • Which data elements and data sources to however, because of differences in institutional use, to ensure validity and comparability missions, program mix, student populations, across institutions and over time. resources, and other factors. Emerging research These are only some of the types of questions focuses on input-adjusted and value-added analysts would have to consider when measuring measures of institutional performance. Adjusted progress against each goal. outcome measures (which may adjust for the The Legislature could delegate these questions academic achievement or socioeconomic status of to a technical group including representatives from incoming students, for example) and longitudinal the administration, legislative staff, the segments, measures of learning (testing students as freshmen and independent researchers with expertise in and again as seniors) can provide more accurate higher education performance measurement. information than traditional measures regarding This group could consider existing, well-defined how well institutions are serving their students, measurement frameworks from national associa- leveling the playing field for comparisons. Higher tions as a starting point. The technical group could education researchers and policymakers disagree also be responsible for collecting baseline data for about whether these measures are ready to be comparisons over time and recommending perfor- broadly employed, especially for high-stakes appli- mance targets. cations such as performance funding. Some believe Setting Targets. Another question facing more research and development is needed to refine policymakers is how they will evaluate perfor- them. There is broad agreement, however, that mance once it is reported. States with a goal-setting policy analysts should use these measures to inform approach usually set numeric statewide targets for policy discussions, along with clear communication each measure of performance. In our completion about their limitations. example, a numeric target could be to increase by 1 million (above a baseline projection) the data Should Be collected Systematically cumulative number of degrees and certificates Data is a vital tool for coordination and students earn by 2025. Alternatively, it could be to oversight, both for measuring progress toward increase awards by 2 percent annually; or to match goals and more broadly for understanding costs, a benchmark such as the top quartile of states, changes in student enrollment patterns, effects of best-performing states, or comparison institutions financial aid and other state programs, and other in completion rates or number of completions per aspects of the state’s higher education system. www.lao.ca.gov Legislative Analyst’s Office 15 An LAO RepOR t California has made progress on higher education in assessing the data. In addition, the audience data collection. In recent years, CPEC worked with for such analyses includes the administration, the the segments and CDE and other state agencies Legislature, and the higher education segments. If to systematically collect student data and make it analyses were conducted within any one of these accessible to policymakers, analysts, researchers, spheres, the others might reasonably be concerned and the public. The closure of CPEC confronts about bias stemming from partisan, ideological, or policymakers with the central question of who financial interests. will collect data to assess the higher education Recommending Changes. A more challenging system and its progress toward state goals. To aspect of the analysis involves recommending enable effective oversight, the state will need to changes in policy that address these areas. assign this function to another entity and direct Recognizing that different observers could the segments to continue providing student records reasonably arrive at different prescriptions for the in standardized formats to be used for routine and same symptoms, the analysis should be presented ad-hoc analyses and reports. publicly in a format that enables policymakers and others to readily evaluate results and recommenda- Policy and Budget decisions Should Be tions. Public transparency is especially important Informed by analysis of Progress because there are multiple audiences for an analysis Graphic Sign Off Regular measurement of system performance of higher education performance, and several levels Secretary provides crucial data for assessing progress toward of responsibility for making changes. Analyst established goals. Even more importantly, it can Making Policy and Budget Choices. Based Director shed light on the effectiveness of state policy on progress towards meeting state goals and the Deputy and budget decisions, thus paving the way for continued analysis of state’s needs and priorities, continuous improvement and adjustment. Figure 4 policymakers at various levels can then review and illustrates this cycle. modify policies and funding to better align perfor- Independent Assessment of Data. While mance of the higher education system with the data collection and dissemination is largely an needs of the state of California. The Legislature’s administrative responsibility, assessing that data is an analytical task requiring expertise, sound Figure 4 judgment, and independence. It is here that the Feedback Cycle for need for an independent oversight agency is most Policy and Budget Decisions pronounced. For example, if college attainment rates were to decline, it would be necessary Collect Data Assess Data to understand whether this was the result of changes in student preparation, changes in student enrollment patterns, issues with transfer and alignment, affordability challenges, or other factors. Such analyses could be politically sensitive, Adjust Targets or Recommend Changes Goals, If Needed reflecting on decisions made by campuses, segments, or policymakers. Independence from the segments and both branches of government Make Policy and Budget Choices would protect against undue political influence 16 Legislative Analyst’s Office www.lao.ca.gov ARTWORK #110482_Figure 4 - Feedback Cycle for Policy and Budget Decisions An LAO RepOR t and Governor’s roles in this process are paramount, through budgets, statutes, committee hearings, and could be facilitated by an established process and other mechanisms. The segments’ and other to periodically review progress and consider policy agencies’ governing boards and administrative recommendations. For example, the Legislature leaders are responsible for ensuring their institu- could hold annual hearings where the oversight tions contribute toward meeting the state’s expecta- body reports its findings and recommendations, tions. Just as their missions and functions are representatives of the segments and other postsec- differentiated, so too will their individual contribu- ondary agencies offer comments on the analysis, tions to meeting state goals differ. In addition, and other stakeholders have an opportunity to there is a role for an independent perspective provide input. Ideally, discussion would encompass in monitoring the segments’ contributions and the broad system of postsecondary education, assisting the Governor, Legislature, and governing including financial aid programs and private insti- boards—through data collection, analysis, and tutions as well as the public segments. policy advice—to meet the state’s postsecondary Tiered Responsibility. State policymakers, the needs. segments, campuses, and other parts of the postsec- Adjust Goals, Indicators, and Targets as ondary system (the Student Aid Commission, Necessary. The Legislature will need to periodi- for example) have distinct roles in overseeing cally review goals, indicators, and targets to reflect postsecondary education. The Legislature and changing priorities, technology, and expectations. Governor establish the state’s expectations for It could do so as part of the annual hearing process higher education statewide and communicate them suggested above. rEcOmmEndatIOnS Need for Oversight Remains. The Governor system is serving the state’s needs, and (2) makes eliminated CPEC because it had proved ineffective, changes to improve its performance. This presumes not because the need for oversight had diminished. policymakers have identified the state’s needs, In fact, we find that the state’s higher education suggesting some metrics against which to measure system requires oversight that is more robust than performance. what the state has had to date. Accordingly, our foremost recommendation The Legislature, the administration, segment is for the Legislature to articulate the state’s governing boards, and an independent agency postsecondary education needs in some form. This all have distinct oversight roles. The Legislature could involve setting specific goals, as many states could take several steps to begin establishing an have done, or identifying key areas or outcomes effective oversight structure. The Legislature could of interest to the state. In either case, we would also implement a number of short-term steps to encourage the Legislature to prioritize among improve oversight in the interim. identified needs. Best practices from other states suggest that Strategic actions developing such an agenda involves broad stake- Articulate State’s Needs. As discussed holder participation and formal deliberation by above, we envision oversight as a process which policymakers. California has begun the deliberative (1) monitors how effectively the postsecondary process on several fronts, including the Legislature’s www.lao.ca.gov Legislative Analyst’s Office 17 An LAO RepOR t Joint Committee on the Master Plan for Higher various types of benchmarks. The ultimate selection Education, a number of legislative proposals to of targets, however, would need to be decided by the establish an accountability framework, and efforts Governor and Legislature. by several nongovernmental organizations to Use Performance Results to Inform Policy develop recommendations for statewide goals. The Decisions. We recommend the Legislature delegate Legislature adopted the Joint Committee’s report data analysis, interpretation of results, and devel- calling for state goals in Chapter 163, Statutes of opment of policy recommendations to an entity 2010 (ACR 184, Ruskin). The Legislature could independent of the segments, and direct the organi- build on those efforts to gain broad agreement on zation performing these functions to report measures postsecondary education needs for California, and and results clearly and concisely. We also recommend establish them in statute. the Legislature convene regular oversight hearings For example, the Legislature could use hearings to review progress and consider policy changes. In on proposed legislation as a vehicle for stakeholder considering actions, we recommend the Legislature participation in refining goals. (See nearby box for a focus on setting expectations and incentives, and that summary of current legislative efforts.) Participation segment and other institutional governing boards by the Governor’s office is key, given that previous focus on operational and programmatic improvement Governors have vetoed bills establishing goals and efforts. accountability frameworks despite broad support Establish a Formal Oversight Body. For the within the Legislature. long run, we reiterate our earlier recommendation Assign a Technical Work Group to Define to establish an effective, independent oversight Measures, Collect Baseline Data. We recommend structure for higher education. In the past, we have the Legislature delegate technical decisions about recommended reforming or replacing CPEC. The specific measures and reporting protocols to a Governor’s elimination of CPEC this year could pave technical working group with representatives from the way to create a far more effective organization in the administration, legislative staff, and the segments, the future. as well as independent researchers with experience Our 2010 report on higher education coordi- in higher education performance measurement. The nation includes a number of specific recommenda- group could also collect baseline information and tions for improvement, summarized in Figure 5. recommend targets for the state’s performance on Recommendations relative to CPEC’s organizational the measures, considering current performance and structure and duties include: current Legislative Efforts to Improve Oversight • Assembly Bill 2 (Portantino, 2011) would establish a new accountability framework for achieving prescribed educational and economic goals. • Senate Bill 721 (Lowenthal, 2011) would likewise establish an accountability framework for achieving prescribed educational and economic goals. • Senate Bill 885 (Simitian, 2011) would encourage the design and implementation of a high-quality, comprehensive, and longitudinal preschool through higher education (P-20) statewide data system that meets specified goals. 18 Legislative Analyst’s Office www.lao.ca.gov An LAO RepOR t • Increase the body’s independence from the • Nearly all stakeholders we consulted public higher education segments. identified the need for better linkages with K-12, workforce, and economic devel- • Develop a more unified governing board opment agencies, with respect to both data appointment process. and policy development. • Assign to it limited and clear responsibilities. At the same time, we recognize the difficulty of creating a new public organization in the current We believe those earlier recommendations fiscal environment. We will continue to refine our remain relevant today. In addition, we have recommendations on the structure and duties for a continued to explore oversight structures and new independent entity to perform some of the key roles in consultation with stakeholders, and are functions required for effective higher education developing additional options for the Legislature to oversight. In the meantime, we turn below to consider. For example: interim measures to ensure oversight in the absence • Pared-down duties could focus on postsec- of an independent entity. ondary effectiveness and efficiency. recommended actions in the Short term • An organization with a stronger oversight role in capital outlay decisions could better We recommend a number of immediate balance institutional or other interests with solutions to guide higher education policy in the a statewide view. interim. These include ensuring continued access to higher education data, relying on existing resources • The organization could ensure a more for budget and policy recommendations, limiting integrated approach to the state’s overall major new investments in facilities and programs, higher education policy by, for example, and monitoring major reallocation decisions within including in its purview private postsec- the segments. ondary education. Figure 5 • A more Summary of Recommendations From 2010 LAO Report streamlined organization Set a Clear Public Agenda for Higher Education could maintain a • Set specific statewide goals (see sample goals) minimal research • Use goals as framework for an accountability system staff and make Strengthen Coordination Mechanisms • Align funding formulas with state goals expanded use • Simplify articulation and transfer processes of independent • Improve oversight for major policy decisions researchers to • Reform program approval process • Consider regional coordination conduct policy Rebuild State’s Capacity for Policy Leadership studies and • Maintain coordinating board’s independence from executive and legislative analyze data. branches and increase its independence from higher education segments • Revise appointment process for the coordinating board • Assign clear responsibility for shepherding the public agenda • Create a more comprehensive statewide student data resource with enhanced research and analysis capabilities www.lao.ca.gov Legislative Analyst’s Office 19 An LAO RepOR t Ensure Data Remain Available to Policymakers or off-campus centers) are of particular concern in and Researchers. Maintaining intersegmental data the absence of an oversight structure. It is difficult for at the CCC appears to be the best short-term option legislative committees to devote the time and resources for providing continued data availability. To ensure needed to identify each proposal’s long-term costs, compliance with federal privacy laws, we recommend alignment with state needs and institutional missions, the Legislature amend statute to assign the Board duplication of other resources, and priority relative to of Governors responsibility for postsecondary data other long-term demands on the General Fund. While collection and program evaluation. Specific statutory our office and DOF will continue to analyze these language could be developed in consultation with proposals, neither agency has the resources to routinely federal privacy officials and representatives from the conduct the type of in-depth evaluation that an office segments, the administration, and the Legislature. focused exclusively on higher education could provide. If the Legislature’s intent is that this be a stopgap Accordingly, we recommend the Legislature limit measure until a more permanent home is found, significant new commitments of resources until a more statutory authority could be time-limited, expiring comprehensive higher education oversight structure is within a few years unless further amended by the in place to support the Legislature’s decision making. Legislature and Governor. Monitor Major Program Changes. In the long As part of the legislation, the Legislature could term, we believe the institutional governing boards direct the three public segments to (1) continue should make programmatic decisions and be held providing data (or otherwise making it available) accountable for institutional outcomes such as student for intersegmental data evaluation and (2) work learning, graduation, and employment. Until the with economic development, workforce, and public oversight mechanisms are in place for outcome review, education officials to improve the state’s capacity for however, interim mechanisms are needed to monitor longitudinal outcome analysis. We believe there should allocation decisions of segments and campuses, be a clear expectation that student record data from including investment in new programs. the public segments remain a public resource that While the Legislature can weigh in on major new is available for public purposes so long as all parties programs through the budget process, many program maintain appropriate privacy safeguards. changes are made through reallocation of existing Increase Direct Legislative Oversight. Ideally, resources within campuses. We recommend the the Legislature would use the results of independent Legislature require the segments to report annually analyses to align budget and policy decisions with the on new program approvals, program deletions, state’s needs and priorities. In absence of an oversight and substantive program changes, until such time structure to conduct these analyses, the Legislature as a structure is in place to independently measure will need to assume a greater role in assessing budget outcomes on behalf of the Legislature. and policy proposals. Committee leadership could conclusion ask subcommittees to spend more time scrutinizing proposals. They could also make greater use of state Taken together, we believe these recommenda- and national policy experts who can provide context tions would improve higher education oversight in and lessons from the research and other states’ the short term while providing an opportunity for the experiences. Legislature to establish an effective, outcome-oriented Capital outlay and other long-term proposals (such process to improve the performance of the higher as the establishment of new professional programs education system in meeting the state’s priorities. 20 Legislative Analyst’s Office www.lao.ca.gov An LAO RepOR t www.lao.ca.gov Legislative Analyst’s Office 21 An LAO RepOR t 22 Legislative Analyst’s Office www.lao.ca.gov An LAO RepOR t www.lao.ca.gov Legislative Analyst’s Office 23 An LAO RepOR t LAO Publications This report was prepared by Judy Heiman and steve Boilard. The legislative Analyst’s office (lAo) is a nonpartisan office which provides fiscal and policy information and advice to the legislature. Cover concept drawn from national Center for Higher education Management systems (nCHeMs) Cost Model, available at lumina Foundation’s College Productivity website: http://collegeproductivity.org/page/resources. To request publications call (916) 445-4656. This report and others, as well as an email subscription service, are available on the lAo’s website at www.lao.ca.gov. The lAo is located at 925 l street, suite 1000, sacramento, CA 95814. 24 Legislative Analyst’s Office www.lao.ca.gov