Interim Evaluation of Community College Basic Skills Transformation Grants

Interim Evaluation of Community College Basic Skills Transformation Grants Summary State Funded One-Time Transformation Grants Initiative. Chapter 22 of 2015 (SB 81, Committee on Budget and Fiscal Review) created a Basic Skills Transformation Grant initiative to improve basic skills education at the community colleges. Chapter 22 directed our office to evaluate the program’s effectiveness based upon data compiled by the Chancellor’s Office. The legislation directed our office to submit an interim evaluation by December 1, 2019 and a final evaluation by December 1, 2021. This brief fulfills the interim reporting requirement. Grants Were for Promoting Certain Evidence-Based Improvement Strategies. The state provided a total of $90 million for the grants, which were provided on a competitive basis. Grant recipients could receive up to $1.5 million to spend over three years. Grant recipients were required to choose two of six evidence-based strategies, with the goal of increasing the proportion of students who (1) complete college-level math and English in a sequence of three or fewer courses and (2) earn an industry-relevant degree or certificate within two years. Subsequent Legislation Made Major Changes to Basic Skills Education. Two years after creating the Basic Skills Transformation Grants, the Legislature enacted Chapter 745 of 2017 (AB 705, Irwin). The legislation changed the rules for basic skills education in three main ways. The legislation (1) made transferable college-level courses the default placement for students, (2) required the use of high school coursework and/or grades in making placement decisions, and (3) set expectations that students complete transferable college-level math and English coursework within one academic year. Difficult to Disentangle Effects of Transformation Grants From AB 705. The Transformation Grants data collected by the Chancellor’s Office was of poor quality, such that we could not use it to evaluate the initiative. Instead, we relied on conversations with colleges and other sources of student outcomes data, which suggested the grants generally were achieving their intended objective of improving course completion rates. The distinct effects of Transformation Grants, however, can no longer be disentangled from the changes to basic skills education that all colleges are making as required by AB 705. Recommend Legislature Rethink Its Evaluation Goals. Given the enactment of AB 705 makes the final evaluation of the Transformation Grant initiative obsolete, we recommend the Legislature eliminate this evaluation requirement. The Legislature could reconsider its evaluation goals to determine if any information of interest is not currently available through the Chancellor’s Office. If such information is identified, the Legislature then could authorize a new corresponding evaluation. GABRIEL PETEK LEGISLATIVE ANALYST DECEMBER 2019 analysis full gutter AN LAO REPORT INTRODUCTION State Created Program to Improve Basic including the types of strategies implemented, Skills Education at Community Colleges. the size of cohorts served by the new strategies, Chapter 22 of 2015 (SB 81, Committee on Budget and the outcomes for students served by these and Fiscal Review) created a grant program to strategies. Chapter 22 also directed our office to improve basic skills education at the community evaluate the program’s effectiveness based upon colleges. Grant recipients were to implement the data compiled by the Chancellor’s Office. certain evidence-based improvement strategies. The legislation directed us to submit an interim Chapter 22 required grant recipients to collect evaluation by December 1, 2019 and a final and report to the California Community Colleges evaluation by December 1, 2021. This brief fulfills (CCC) Chancellor’s Office a variety of information, the interim reporting requirement. BACKGROUND In this section, we provide background on college-level courses based on standardized tests. basic skills education at the community colleges, Students were required to take these tests prior to describe the Basic Skills Transformation Grants enrolling in community college courses. Students created in 2015, and describe more recent policy would then be placed into the appropriate course changes that require community colleges to make based on their test results. (As we discuss later in broader changes to the way they support students this section, these tests are no longer used due to in completing transferable college-level math and recent policy changes.) English courses. State Provides Ongoing Funding for Basic Skills Courses. Most basic skills funding is Basic Skills Education generated through apportionments, which Community Colleges Offer Math and English provide funding to community college districts on Courses Below College Level. To obtain an a per-student basis. In the past, the state also associate degree and/or transfer to a four-year has provided categorical funding specifically for university, students must complete college-level basic skills education. From 2007-08 through courses in math and English (among other 2017-18, the state provided funding through the requirements). For students who are not deemed Basic Skills Initiative. Districts could use the funds prepared for a college-level course, colleges offer from this program for counseling and tutoring preparatory courses, including courses in elementary students enrolled in basic skills courses, as well and secondary reading, writing, and math, as well as as designing curriculum and training the faculty courses in English as a second language. Students teaching these courses. Over most of this period, deemed unprepared may be required to complete the state provided $20 million annually for the one or more of these basic skills courses before initiative. In 2018-19, this funding was consolidated enrolling in college-level courses. For example, a into a larger block grant—the Student Equity and student whose educational goal requires completion Achievement Program. Community colleges can of college statistics may need to first complete use the $475 million provided through the block pre-algebra, elementary algebra, and/or intermediate grant for basic skills, but they also can use the algebra. Basic skills courses do not apply towards funds for other counseling and support, as well as meeting degree or transfer requirements. for implementing strategies to close equity gaps. Students Previously Assessed Based on (As we discuss on the following page, the state has Standardized Tests. Traditionally, students also provided one-time grant funds for basic skills.) were assessed as prepared or unprepared for 2 LEGISLATIVE ANALYST’S OFFICE analysis full gutter AN LAO REPORT Students Deemed Unprepared Are Less $1.5 million in grant funding that it could spend over Likely to Graduate. Among degree, certificate, three years. Colleges applied and were awarded or transfer-seeking students who entered the funding through a competitive process. community college system in 2011-12, only Initiative Focused on Promoting Certain 25 percent were deemed prepared for college-level Evidence-Based Improvement Strategies. coursework in both math and English. Prepared Chapter 22 identified six evidence-based strategies students have much higher completion rates than for improving basic skills education (Figure 1). unprepared students. Of the students entering the It required grant recipients to adopt or expand system in 2011-12, 70 percent of prepared students the use of at least two of them. Colleges could completed a degree, obtained a certificate, or use grant funding for implementing their selected transferred to a four-year university within six years, strategies. Statute specifies the overarching intent compared to 41 percent of unprepared students. of implementing the strategies is to increase The share of students classified as prepared varies the proportion of students who (1) complete significantly by racial/ethnic group. The share college-level math and English in a sequence of prepared students is relatively low for black of three or fewer courses and (2) earn an and Latino students (13 percent and 20 percent, industry-relevant degree or certificate within two respectively), while over half of white and Asian years. These changes were driven by findings from students are classified as prepared. Age also is community colleges in California and other states related to preparation, with younger students more showing that certain basic skills strategies resulted likely to be classified as prepared upon entering the in higher proportions of students completing community college system. college-level math and English courses in shorter periods of time. Basic Skills Transformation Grants Total of 64 Colleges Received Grants. Of the State Funded One-Time Basic Skills 114 community colleges operating during the grant Transformation Grants. The 2015-16 budget period, 85 colleges applied for grants. As part of included $60 million one-time Proposition 98 funding the application process, colleges were required for grants to assist colleges in adopting or expanding to describe their readiness to implement new the use of evidence-based models for basic skills strategies, set targets for the number of students assessment, placement, instruction, and student to be served by these new strategies, and describe support. Due to the large demand from colleges for their plan for addressing the training and support these grants, the Legislature provided an additional faculty would need to implement these strategies. $30 million one-time Proposition 98 funding for this The CCC Chancellor’s Office deemed 64 of the purpose in 2016-17. A college could receive up to 85 applications as meeting minimum requirements. Figure 1 Grant Recipients Could Choose Among Six Evidence-Based Strategies Colleges receiving Basic Skills Transformation Grants were required to adopt or expand at least two of the following strategies: 9 Using multiple measures to assess and place students into math and English courses. 9 Increasing placement of students directly into transferable college-level courses and providing corequisite basic skills instruction. 9 Requiring students to master only those math and English skills needed for their chosen programs of study. 9 Contextualizing basic skills instruction to relate to students’ programs of study. 9 Integrating student support services with instruction. 9 Developing shorter course sequences for attaining college-level math or English skills by using technology, the above strategies, or other strategies that the college can substantiate are effective. www.lao.ca.gov 3 analysis full gutter AN LAO REPORT Grant funding was distributed to these colleges in Chancellor’s Office Provided Implementing July 2016, with the expectation that colleges would Guidance. In fall 2017, the CCC Chancellor’s begin implementation during the 2016-17 academic Office began providing guidance to colleges in year. implementing AB 705. The Chancellor’s Office expects colleges to place all students that Recent Policy Changes graduated high school within the past ten years Subsequent Legislation Made Major Changes directly into transferable college-level math and to Basic Skills Education. Two years after English, but provide concurrent academic support creating the Basic Skills Transformation Grants, the to students with lower GPAs. The specific GPA Legislature enacted Chapter 745 of 2017 (AB 705, cut-offs are based on research conducted by Irwin). The legislation changed the rules for basic the Multiple Measures Assessment Project, a skills education in three main ways. collaborative of community college researchers and practitioners. In English, for example, the • Assessment. Colleges must now use Chancellor’s Office recommends concurrent students’ high school coursework, course academic support for students with a high school grades, and/or grade point average (GPA) in GPA lower than 2.6. The Chancellor’s Office making placement decisions. Colleges may not expects colleges to fully implement the new policies use standardized tests to determine student by fall 2019 for math and English courses. placement unless the assessment has been New Law Has Implications for Colleges approved by the CCC Board of Governors. Receiving Basic Skills Transformation Grants. To date, the Board of Governors has not Because of the changes AB 705 makes to approved any assessment for this purpose. basic skills education, some colleges receiving • Placement. Colleges also must now make Transformation Grants are having to rethink transferable college-level courses the default their basic skills approaches. Most notably, placement for students. (Students may Transformation Grant recipients previously were choose to enroll in courses below their default allowed to focus on shortening basic skills course or recommended placement.) This change sequences (rather than placing students directly in state policy was driven by a growing into college-level courses). Transformation Grant research consensus that placing students recipients also originally had a less stringent timing directly into transferable college-level courses expectation, with students encouraged to complete was the most effective basic skills strategy college-level material within a three-course implemented to date. sequence, typically taking a year and a half. Now, • Timing. Assembly Bill 705 sets the colleges are to de-emphasize these longer course expectation that students complete sequences, as well as have students complete transferable college-level coursework within college-level work more quickly. one year. FINDINGS In this section, we first describe our Office was of poor quality and could not be used findings regarding implementation of the to evaluate the grant program. To complete Basic Skills Transformation Grants initiative. our evaluation of the grant program, we spoke Chapter 22 required grant recipients to collect with several community college administrators and report certain data from 2015-16 through and analyzed trends in student outcomes using 2019-20, with the CCC Chancellor’s Office required statewide data collected by the Chancellor’s Office. to compile that data annually. As noted later in this section, the data collected by the Chancellor’s 4 LEGISLATIVE ANALYST’S OFFICE analysis full gutter AN LAO REPORT Implementation data to summarize the strategies used by colleges or evaluate the effects of the initiative. Colleges Criteria Used for Determining Awardees are required to report two additional years of Was Reasonable. Based on our review of the data (2018-19 and 2019-20) for the final program CCC Chancellor’s Office’s request for applications, evaluation due in 2021. Without greater oversight we have no concerns with the criteria used for and quality control from the CCC Chancellor’s Office, determining grant awardees. The information this additional data also is likely to be unusable. the Chancellor’s Office requested from colleges Colleges Used Funding Primarily for appears to be well aligned with the statutory Curriculum Development and Training. requirements of the initiative. The Chancellor’s Though colleges were required to report which Office also appears to have made reasonable evidence-based strategies they implemented, they judgment calls in reviewing applications—rejecting were not required to report how they spent their those that did not include all required information grant funds. Based on our conversations with and may, therefore, have been less well developed. colleges, they indicated they used grant funds For example, the Chancellor’s Office rejected primarily to modify existing math and English applications that lacked information on the number curricula, train faculty in using the redesigned of students currently served by evidence-based curricula, and train staff in supporting students basic skills strategies and the number of students through their course sequences. These purposes the college was proposing to serve using the are consistent with the Legislature’s intent for how Transformation Grant funding. colleges were to use the grant funds. Grant Recipients Had Somewhat Higher Some Colleges Did Not Sustain Activities Completion Rates Than Other Colleges. Notably, When Grant Funds Were Exhausted. Some the colleges receiving grants had slightly better colleges indicated to us that they used their student outcomes than those colleges with rejected one-time grant funds to hire additional support applications or not applying. The percentage staff, with the goal of making some support of first-time students enrolled in fall 2015 who services (such as tutoring) more accessible for completed transferable college-level English within students. In some cases, colleges shared with us one year was 24 percent for Transformation Grant that they had since found ongoing fund sources recipients. This compares with 22 percent for (such as the Student Equity and Achievement colleges whose applications were rejected and Program) to continue employing these staff. In 20 percent for colleges that did not apply for a other cases, colleges indicated they were not able Transformation Grant. to continue these services after their Transformation Data Collected by CCC Chancellor’s Office Is Grant funds expired. of Poor Quality. To collect the required data from grant recipients, the Chancellor’s Office developed Trends in Student Outcomes a detailed spreadsheet for colleges to complete and More Students Are Being Placed Directly submit once per year. The spreadsheet includes Into Transfer-Level Courses. Over the last few information on the strategies a college implemented months, several groups have released reports and its student outcomes, disaggregated by race/ examining how community college course offerings ethnicity, high school GPA, and the student’s have changed in response to basic skills reforms. initial placement. In preparing our report, we had Reports by the Public Policy Institute of California access to data submitted for the first two years (PPIC) and the RP Group (a nonprofit organization of implementation (2016-17 and 2017-18). The operated by community college researchers) have spreadsheets shared with us, however, contained found significant increases in the percentage numerous errors and inconsistencies. As evident of students being placed into transferable from the spreadsheets, the Chancellor’s Office did college-level coursework. The PPIC report also not thoroughly review the data or follow-up with found that students who enrolled in transferable colleges to have them submit accurate data. Due college-level coursework with co-requisite support to this poor data quality, we are unable to use this www.lao.ca.gov 5 analysis full gutter AN LAO REPORT had higher completion rates than students who Completion Rates for Transfer-Level Math and enrolled in basic skills courses. A Campaign for English Courses Have Improved. Given the poor College Opportunity review of the fall 2019 course quality of the data reported by Transformation Grant section offerings at 47 colleges also found a recipients, we examined an alternative data source significant increase in transferable college-level to track course completion rates. Specifically, we course offerings and co-requisite courses. This used the state’s main data system for all colleges— report, however, identified a few concerns of known as the Management Information System implementation falling short. In particular, the report (MIS)—to examine trends in the share of students noted that some colleges continued to include a completing transferable college-level math and significant number of basic skills courses in their English courses. Such analysis allows us to see course catalogs and appeared to be offering too trends in student outcomes, but we cannot link few courses in transferable college-level statistics these outcomes directly with specific strategies and quantitative reasoning—the most common grant recipients adopted. As Figure 2 shows, math requirement for community college students. systemwide completion rates in both math and Figure 2 Course Completion Rates Are Improving Percentage of First-Time Students Who Complete Transferable College-Level Courses Within Specified Time Math Math (1 Year) (2 Year) 40% 40% 35 35 30 30 25 25 20 20 15 15 10 10 5 5 2013 2014 2015 2016 2017 2018 2013 2014 2015 2016 2017 All Asian Black Latino White English English (1 Year) (2 Year) 40% 40% 35 35 30 30 25 25 20 20 15 15 10 10 5 5 2013 2014 2015 2016 2017 2018 2013 2014 2015 2016 2017 6 LEGISLATIVE ANALYST’S OFFICE analysis full gutter AN LAO REPORT English courses have improved over the past five improvements occurred in the fall 2018 cohort, years. The proportion of first-time students who which could be driven by colleges beginning early enrolled and completed a transferable college-level implementation of AB 705. As Figure 2 shows, English course within one year increased from completion rates increased for all racial/ethnic 18 percent in the fall 2013 cohort to 31 percent in groups, although previously existing achievement the fall 2018 cohort. In math, these rates increased gaps remain. We found no significant differences from 8 percent in the fall 2013 cohort to 14 percent in improvement between Transformation Grant in fall 2018. For both math and English, the largest recipients and all other colleges. ISSUES FOR LEGISLATIVE CONSIDERATION Final Evaluation of Transformation Grants to examine the early effects of AB 705. The Is No Longer Pertinent. When the Legislature Chancellor’s Office collects MIS information authorized the evaluations of the Basic Skills annually and makes it publicly available. The Transformation Grant initiative back in 2015, Chancellor’s Office also is required to report it presumably desired to learn which of the six some MIS data, including course completion and intervention strategies were most effective at graduation rates, to the Legislature as part of an improving student completion rates in transferable annual performance report. math and English courses. Now, with the Recommend Legislature Rethink Its enactment of AB 705 in 2017, all colleges must use Evaluation Goals. Given the enactment of the same two prescribed intervention strategies AB 705 makes the final evaluation of the (high school grades and default placement into Transformation Grant initiative obsolete, we transferable courses). As a result, the evaluation recommend the Legislature eliminate this evaluation focus has shifted to how effective these two requirement. If the Legislature is interested in (rather than six) strategies are at improving course seeing if AB 705 improves course completion completion and graduation rates. With these and graduation rates, it can monitor outcomes developments, the distinct effects of Transformation using CCC’s annual performance report. As this Grants can no longer be disentangled from the information already is available annually, the changes to basic skills education that all colleges Legislature could reconsider its evaluation goals to are making as a result of AB 705. determine if any information not available through Outcome Data for All Colleges Will Continue MIS might be of interest. If such information is to Be Available Through MIS. Because all identified, the Legislature then could authorize a colleges must implement the new strategies, new corresponding evaluation. research groups have been using MIS data www.lao.ca.gov 7 analysis full gutter AN LAO REPORT LAO PUBLICATIONS This report was prepared by Edgar Cabral and reviewed by Jennifer Pacella and Anthony Simbol. The Legislative Analyst’s Office (LAO) is a nonpartisan office that provides fiscal and policy information and advice to the Legislature. To request publications call (916) 445-4656. This report and others, as well as an e-mail subscription service, are available on the LAO’s website at www.lao.ca.gov. The LAO is located at 925 L Street, Suite 1000, Sacramento, CA 95814. 8 LEGISLATIVE ANALYST’S OFFICE