Recommendations

California Air Resources Board Improved Program Measurement Would Help California Work More Strategically to Meet Its Climate Change Goals February 2021 REPORT 2020‑114 CALIFORNIA STATE AUDITOR 621 Capitol Mall, Suite 1200 | Sacramento | CA | 95814 916.445.0255 | TTY 916.445.0033 For complaints of state employee misconduct, contact us through the Whistleblower Hotline: 1.800.952.5665 Don’t want to miss any of our reports? Subscribe to our email list at auditor.ca.gov For questions regarding the contents of this report, please contact Margarita Fernández, Chief of Public Affairs, at 916.445.0255 This report is also available online at www.auditor.ca.gov | Alternative format reports available upon request | Permission is granted to reproduce reports Elaine M. Howle State Auditor February 23, 2021 2020-114 The Governor of California President pro Tempore of the Senate Speaker of the Assembly State Capitol Sacramento, California 95814 Dear Governor and Legislative Leaders: As directed by the Joint Legislative Audit Committee, my office conducted an audit of the California Air Resources Board (CARB). Our assessment focused on transportation programs intended to reduce greenhouse gas (GHG) emissions, and the following report details the audit’s findings and conclusions. In general, we determined that CARB must do more to help the State work strategically toward its climate change goals. CARB has not done enough to measure the GHG emissions reductions its individual transportation programs achieve. Specifically, CARB has not collected or evaluated sufficient data to allow it to determine whether or how its incentive programs, which pay consumers in exchange for purchasing low- and zero-emission vehicles, reduce GHG emissions beyond what CARB’s regulations already require. For example, CARB has done little to measure the extent to which its incentive programs lead to emissions reductions by causing individuals and businesses to acquire clean vehicles that they otherwise would not. As a result, CARB has overstated the GHG emissions reductions its incentive programs have achieved, although it is unclear by how much. Given the ambitious nature of the State’s climate change goals and the short time frame to meet them, California is in need of more reliable tools with which to make funding decisions. Additionally, CARB has not consistently collected or analyzed data to determine whether some of its programs provide the socioeconomic benefits that CARB has identified for those programs, such as maximizing participants’ economic opportunities. Because these programs may cost significantly more than other incentive programs from the perspective of reducing GHG emissions, CARB must do more to measure and demonstrate specific benefits to disadvantaged communities and low-income communities and households that the programs intend to serve. Finally, despite requirements in state law and its own guidelines, CARB has been slow to measure the jobs its programs create and support—or the benefits of the specialized job training that certain programs are supposed to provide. As with the need to assess accurately programs’ GHG reductions, knowing whether its programs are achieving the expected important but more expensive socioeconomic benefits is crucial to providing the State with the information it needs to allocate its limited resources effectively in pursuit of its various goals. Respectfully submitted, ELAINE M. HOWLE, CPA California State Auditor 621 Capitol Mall, Suite 1200 | Sacramento, CA 95814 | 916.445.0255 | 916.327.0019 fax | www.auditor.ca.gov iv California State Auditor Report 2020-114 February 2021 Selected Abbreviations Used in This Report CARB California Air Resources Board CalEPA California Environmental Protection Agency LAO Legislative Analyst’s Office cap-and-trade fund Greenhouse Gas Reduction Fund UC Berkeley University of California, Berkeley California State Auditor Report 2020-114 v February 2021 Contents Summary 1 Introduction 7 Chapter 1 CARB Has Not Done Enough to Collect and Analyze Data to Demonstrate Its Programs’ Effectiveness in Reducing GHG Emissions 19 Recommendations 38 Chapter 2 CARB Has Not Adequately Demonstrated Its Programs’ Socioeconomic Benefits 41 Recommendations 53 Other Areas We Reviewed 55 Appendix A CARB’s Transportation GHG Emissions Reduction Programs 57 Appendix B Scope and Methodology 61 Response to the Audit California Air Resources Board 65 California State Auditor’s Comments on the Response From the California Air Resources Board 73 vi California State Auditor Report 2020-114 February 2021 California State Auditor Report 2020-114 1 February 2021 Summary Results in Brief Audit Highlights . . . Fighting climate change is a key public policy concern for Our audit of CARB’s transportation California. The State has set ambitious goals for reducing programs for reducing GHG emissions greenhouse gas (GHG) emissions—the primary source of air highlighted the following: pollution linked to climate change—over the next decade and beyond. At the forefront of those efforts is the California Air » CARB has not done enough to measure Resources Board (CARB), which state law has given responsibility the GHG emissions reductions its incentive for controlling emissions from motor vehicles and for designing programs achieve. programs to reduce statewide GHG emissions. • It does not account for overlap However, California may not successfully meet its upcoming of its incentive programs and GHG reduction goal, which will require the State to reduce GHG regulatory programs. emissions by nearly 40 percent over the next decade. Although other sources of GHG emissions have been declining in recent • It has generally not determined the years, emissions from transportation have increased since 2013, and effects its incentive programs have on GHG emissions from transportation accounted for 40 percent of consumers’ behavior and thus, has all statewide emissions in 2018. To help CARB fight climate change overstated GHG emissions reductions by reducing GHG emissions, the Legislature has allocated more its incentive programs achieve. than $2 billion from the State’s Greenhouse Gas Reduction Fund (cap-and-trade fund) to CARB’s transportation programs since » Although cap-and-trade revenue is fiscal year 2013–14. substantial, it is finite and can be unpredictable—it experienced a The State’s cap-and-trade program—one of the key elements of its significant drop in 2020. climate change strategy—raises revenue by setting statewide limits on GHG emissions from major sources. The program allows the » Although CARB is required to use entities responsible for those sources to comply with the set limits cap-and-trade funds in a way that by reducing their emissions or by paying the State for allowances maximizes economic benefits and fosters to emit GHGs. The payments take place during quarterly auctions job creation, it has done relatively little to of GHG allowances, which have generated billions of dollars in measure specific socioeconomic benefits. annual revenue that the State then deposits in the cap-and-trade fund. Although it is substantial, cap-and-trade revenue is finite and • It has been slow to measure the can be unpredictable. Partly as a result of the ongoing COVID-19 jobs its programs create or support pandemic, the cap-and-trade auction in May 2020 generated and the benefits of job training its quarterly proceeds of only $25 million—compared to an average programs require. of more than $700 million for each of the previous 11 quarters. This drop in revenue caused a funding reduction of $81 million to CARB’s programs for the year. Although the auction has rebounded somewhat since then, proceeds remain below the historical average. This uncertainty, together with the short time frame remaining before the 2030 date for achieving the State’s GHG goals, increases the challenge of meeting those GHG goals. In California’s 2017 Climate Change Scoping Plan, CARB set forth key objectives for reducing GHGs from California’s various transportation sectors, including passenger vehicles, heavy-duty trucks, buses, and freight. To achieve its objectives, CARB has 2 California State Auditor Report 2020-114 February 2021 designed and implemented a range of programs targeted at reducing GHG emissions from specific vehicle types. Many of CARB’s programs fall into two general categories: regulatory programs and incentive programs. CARB establishes its regulatory programs through a formal public rulemaking process, and some of these programs require vehicle manufacturers to produce and sell certain types of vehicles or for their vehicles to meet GHG emissions standards. Incentive programs are voluntary programs that often provide monetary payments to consumers who purchase low- and zero-emission vehicles. CARB implements these programs—sometimes at the direction of the Legislature—and reviews the programs’ funding each year. Although they are different in how CARB operates them, regulatory and incentive programs may work toward the same objective. For example, CARB operates a regulatory program aimed at increasing the manufacture and sale of zero-emission passenger vehicles (ZEVs). The regulation underlying the program requires that auto manufacturers sell enough ZEVs each year to make up a required proportion of their overall sales. CARB also operates incentive programs that provide rebates or other financial support to consumers who purchase ZEVs. The intent of these rebates is to encourage customers to purchase ZEVs, which tend to be more expensive than gasoline-powered vehicles. All of these programs work simultaneously toward achieving CARB’s objective of putting five million ZEVs on California roads by 2030. Given the ambitious nature of the State’s GHG goals, it may be reasonable for CARB to operate multiple programs that work toward a shared transportation objective. However, to ensure that it is operating the most effective mix of programs to achieve the State’s goals, it is important for CARB to identify the GHG reductions that each individual program achieves. Our review determined that although CARB generally approaches the projected GHG reductions from its individual programs in a reasonable way, it has not accounted for overlap between some of its programs. For the eight regulatory programs we reviewed, we found that when proposing the new regulation, CARB generally identified the relationship between the regulation and other existing regulatory programs in order to isolate the expected additional GHG reductions. However, the proposed regulations did not assess how the regulatory programs might overlap with its incentive programs that work toward the same objective. Because CARB does not know whether funds for incentives will be available in the future to help manufacturers and consumers offset vehicle costs, CARB designs certain regulatory programs to achieve their GHG reductions without assistance from the incentive programs. Although reasonable, this approach means that the GHG California State Auditor Report 2020-114 3 February 2021 reductions it claims from its incentive programs should be over and above what the regulatory programs achieve, and CARB must be able to measure those additional GHG reductions. However, CARB has not done enough to measure the emissions reductions its incentive programs achieve in their own right. CARB generally does not formally acknowledge the potential overlap with regulatory programs or discuss how it accounts for that overlap in its incentive programs’ designs. In addition, CARB does not collect and measure data for passenger and heavy-duty vehicles in a way that allows it to assess the extent to which clean vehicle manufacturing and sales—and therefore GHG reductions—exceed the reductions that its regulatory programs require. If it did, CARB might be able to assess its incentive programs’ GHG reductions based on any extra reductions that have occurred. For example, manufacturers are currently exceeding requirements in a regulatory program that requires them to sell ZEVs, raising the possibility that CARB’s incentive programs are augmenting the regulatory program’s impact. However, CARB does not know precisely how many additional ZEVs are being sold or how that number compares to the number of vehicles its incentive programs help pay for. CARB also has generally not determined the effects its incentive programs have on consumers’ behavior. Specifically, it generally does not know how often many of its incentive payments influence consumers to purchase a cleaner (lower-emission) vehicle than they otherwise would have purchased. Having this information is crucial to making accurate calculations of the GHG reductions of those programs because it would indicate whether the incentive caused the vehicle purchase and therefore produced the reductions. However, of the five incentive programs we reviewed where CARB provides a payment or other financial assistance to purchase a cleaner vehicle, CARB collects information about behavioral changes for only one: its Clean Vehicle Rebate Program. Even for that program, CARB has made only limited use of the behavioral data it collects. Finally, CARB may be missing opportunities to use other sources of data, such as federal tax credits for clean vehicles, to learn more about how effective its programs are in changing behavior. Although such analyses may be challenging, they may allow CARB to modify programs to increase their cost-effectiveness and to have a greater impact on emissions reductions. CARB’s inability to measure the GHG reductions from its cap-and-trade-funded incentive programs diminishes the usefulness of its annual reports to the Legislature on the GHG reductions from these programs. CARB’s current reporting assumes that the emissions reductions from all of the vehicles funded by an incentive program would not otherwise occur. By not taking into account the effects that regulations and other factors have on emissions, CARB 4 California State Auditor Report 2020-114 February 2021 overstates the incentive programs’ GHG reductions, although it is unclear by how much. One effect of this overstatement is to obscure the programs’ cost-effectiveness in reducing GHG emissions. Without accurate information in the annual reports— which would make these reports a reliable emissions measurement tool—the Legislature’s ability to make decisions about investments towards the State’s GHG goals may be hampered. Specifically, if the annual reports contained accurate information, these reports could better help the Legislature make decisions about whether to continue funding a given program at its current level, decrease the funding and use those resources elsewhere, or significantly increase funding. Further, improved and clear metrics will help CARB to know when its incentive programs have successfully achieved their goals of helping low- and zero-emission vehicle technology become sustainable. As part of strengthening its program measurement overall, CARB must also do more to ensure that the data it collects on those programs are accurate and that the calculations CARB makes from the data are free of errors that can further distort the emissions reductions it reports. State law directs CARB, to the extent feasible, to use cap-and-trade funds in a way that maximizes economic benefits and fosters job creation. More specifically, state law requires CARB to establish programs that increase access and provide benefits to Californians living in environmentally disadvantaged communities as well as low- and moderate-income communities. In part, those requirements specify that minimum proportions of cap-and-trade spending must go to geographically defined disadvantaged and low-income communities. Although CARB has exceeded these minimum spending requirements in recent years, it has done relatively little to measure the specific socioeconomic benefits of its programs. Some programs that CARB operates focus primarily on producing socioeconomic benefits, as opposed to maximizing GHG reductions. These programs may cost significantly more than other incentive programs because they offer higher incentive payments per vehicle and may require more administrative effort. Partly due to these additional costs, we expected CARB to demonstrate the programs’ value by clearly defining and measuring the specific socioeconomic benefits. Although CARB has identified benefits that include maximizing economic opportunities for participants, increasing participants’ credit scores, and lowering their driving costs, it does not consistently collect data to determine whether the programs actually provide those benefits. For example, CARB collects information related to the auto loans in its Financing Assistance for Lower-Income Consumers program, but it has not collected California State Auditor Report 2020-114 5 February 2021 information to measure whether participants’ credit scores increased or if they subsequently qualified for housing loans. CARB did make an effort to design participant surveys for a separate program—Clean Cars 4 All—that asked specific questions about changes to participants’ employment opportunities or income. However, even though the surveys could help CARB determine whether the program is providing the intended socioeconomic benefits, CARB does not require the entities that administer the day-to-day operations of the programs to use those specific survey questions, and it does not know whether they do so. CARB has also missed opportunities to use data it has already collected to determine whether participants receive the intended benefits of its programs. Finally, CARB has been slow to measure the jobs its programs create or support, and it has done little to measure the benefits of the job-training activities that its own guidelines require. Despite requirements since 2015 in both state law and CARB’s own funding guidelines that cap-and-trade programs must encourage job creation, CARB only began formally collecting information related to jobs in 2019. Further, at the time of our review, it had collected this information in its reporting database for just three of the nine programs we reviewed for which it should have done so. Additionally, despite clear language in its funding guidelines that programs should also support on-the-job training and requirements for reporting the outcomes of that training, CARB has not always collected detailed information about such training or its participants. As with the need to accurately assess programs’ GHG reductions, knowing whether these important but more expensive socioeconomic benefits are occurring is crucial to providing the State with the information it needs to allocate its limited resources effectively. Summary of Recommendations To improve its ability to isolate each of its incentive programs’ GHG reductions, by February 2022 CARB should establish a process to formally identify its incentive programs’ overlap with other programs that share the same objectives. To improve its ability to identify the effectiveness of each of its incentive programs in reducing GHG emissions, by August 2021 CARB should develop a process to define, collect, and evaluate data on the behavioral changes that result from each of its incentive programs. 6 California State Auditor Report 2020-114 February 2021 To better assist the State in achieving its GHG goals, CARB should use the information we describe above to refine its GHG emissions estimates for its incentive programs in its annual reports to the Legislature, the funding plans approved by its board, and any longer-term planning documents or reports. To better demonstrate that its incentive programs are as effective as possible in achieving specific socioeconomic benefits, by February 2022 CARB should develop a process to define, collect, and evaluate data that will translate to metrics showing the socioeconomic benefits that result from each of the incentive programs. To provide transparency to the Legislature and other stakeholders, beginning in 2022 and using the metrics and data described above, CARB should make funding and design recommendations in its funding plans and annual reports based on which programs are effective in producing socioeconomic benefits and at what cost. Agency Comments CARB agreed with our recommendations and indicated that it is taking steps to implement them. California State Auditor Report 2020-114 7 February 2021 Introduction Background The California Air Resources Board (CARB) is the state agency charged with combating air pollution and regulating sources of greenhouse gas (GHG) emissions in order to mitigate the adverse impacts of climate change.1 State law gives CARB responsibility for controlling emissions from motor vehicles and requires CARB to coordinate efforts related to attaining and maintaining air quality standards. As part of its responsibilities, CARB is required to design emissions reduction measures and to monitor and regulate sources of GHG emissions in order to reduce them. Serious and escalating problems in California have been linked to climate change, including wildfires, water shortages, threats to agriculture, and health threats from air pollution. California’s Fourth Climate Change Assessment report from 2018, the most recent such assessment, explains that temperatures are warming and that available science indicates that many people will endure more illness and be at greater risk of early death in California because of climate change. Accordingly, climate change is a major and time-sensitive public policy concern, and one for which the State has established specific goals. In 2005 Governor Schwarzenegger issued an executive order that described the need to reduce GHG emissions and established emissions reduction targets. As part of the California Global Warming Solutions Act of 2006, a landmark California law that established the State’s GHG reduction program, the Legislature declared that global warming—an aspect of climate change—poses a serious threat to the economic well-being, public health, natural resources, and environment of California. The Legislature also identified California as a national and international leader in environmental stewardship efforts and stated that California’s GHG reduction program would place it at the forefront of national and international efforts to reduce GHG emissions. Over the past 15 years, California has enacted certain laws and executive orders intended to reduce GHG emissions. Two of these enactments specifically require CARB to ensure that California reduces its GHG emissions to certain levels by specified dates. For example, the California Global Warming Solutions Act of 2006 required CARB to adopt a statewide GHG emissions 1 Global warming is the long-term heating of Earth’s climate generally attributed to human activities, primarily the burning of fossil fuels. Climate change encompasses global warming, but it also refers to the broader range of changes happening to the planet as a consequence of global warming. We use the term “climate change” in the report instead of “global warming” because it encompasses the broad consequences of human activities on the climate. 8 California State Auditor Report 2020-114 February 2021 limit equivalent to the State’s 1990 emissions level and to design reduction measures that would enable the State to meet that limit by 2020. In 2007 CARB established the 2020 limit at 427 million metric tons (MMT) of GHG emissions per year, and it later increased that limit to 431 MMT. Similarly, in April 2015, Governor Brown issued an executive order to establish a goal of reducing GHG emissions to 40 percent below 1990 levels by 2030, and the Governor’s Office described the goal as the most aggressive benchmark enacted by any government in North America. The Legislature subsequently passed legislation in 2016 requiring CARB to ensure that California meets that goal, which means achieving GHG levels of 260 MMT by 2030. Other relevant laws and executive orders include those aimed at supporting the development and deployment of low-emission heavy-duty trucks and increasing the volume of zero-emission vehicles (ZEVs) sold in the State. For example, in a September 2020 executive order, Governor Newsom directed CARB to adopt new regulations to increase ZEV sales with the goal that by 2035 all new passenger cars and trucks sold in California will be ZEVs. The Governor subsequently proposed allocating $1.5 billion in special funding as part of the State’s fiscal year 2021–22 budget in order to help implement the State’s ZEV objectives. However, California may not meet its goal of reducing GHG emissions to 260 MMT per GHG Emissions by Source year by 2030. According to CARB’s report California Greenhouse Gas Emissions for 2013 (MMT) 2018 (MMT) 2000 to 2018, published in 2020 (2020 GHG Transportation 161 169 emissions report), in 2018 California’s emissions All other sources 286 256 were 425 MMT, meaning that although it has TOTAL 447 425 achieved the 2020 goal of 431 MMT, the State still needs to reduce annual emissions by nearly Source: CARB’s 2020 GHG emissions report. 40 percent over the next decade to reach the 2030 goal. The text box shows that, although other sources of GHG emissions have declined in recent years, transportation-related emissions have increased slightly, leading to a problematic trend. In 2013 transportation-related emissions were responsible for 36 percent of California’s total GHG emissions; as of 2018, they accounted for 40 percent of the total. Figure 1 shows the evolution of California’s GHG emissions goals and the trend in GHG emissions since 1990. As the Figure demonstrates, the State will fall short of meeting the 2030 goal unless emissions reductions occur at a faster pace. California State Auditor Report 2020-114 9 February 2021 Figure 1 California Has Implemented Goals to Reduce GHG Emissions 500 481 2020 450 EMISSIONS GOAL 431 429 431 MMT 425 400 2006: State law directs CARB to adopt a GHG emissions limit that is equivalent to the State's 1990 GHG emissions level, to be achieved by 2020. 350 355 PRESENT: The State has met its 2020 goal but must reduce 300 emissions by nearly 40 percent to meet its 2030 goal. 2030 EMISSIONS GOAL 260 MMT 250 2016: State law requires CARB to ensure that statewide GHG emissions are reduced to 40 percent below 1990 levels by 2030. 200 1990 1992 1994 19961998 2000 200220042006 2008 2010 201220142016 2018 20202022 2024 20262028 2030 )TMM( snoissimE GHG Year Reported Emissions Projection Based on Past Emissions Source: CARB’s GHG emissions reports, California’s 2017 Climate Change Scoping Plan, state law, CARB Resolution 14-16, and our projections of future GHG emissions based on the average annual change in GHG emissions over the past 10 years. According to CARB’s 2020 GHG emissions report, the other 60 percent of California’s GHG emissions come from sources such as industrial operations and electric power generation. The vast majority of California’s GHG emissions reductions since it established the statewide emissions limits in 2006 have come from electric power generation. CARB has acknowledged the need for greater contributions from the transportation sector in order to meet the 2030 GHG emissions reduction goals. CARB’s 2020 GHG emissions report also shows that within the transportation sector, two categories of vehicles accounted for more than a third of California’s total emissions in 2018. Passenger vehicles, which include cars and small trucks driven on California’s roads, account for 70 percent of transportation emissions and more 10 California State Auditor Report 2020-114 February 2021 than 28 percent of the State’s total emissions. The second-highest level of emissions comes from heavy-duty vehicles, that is, large trucks and buses, which account for 20 percent of transportation emissions and 8 percent of total state emissions. Other smaller transportation categories include aviation, rail, and ships, which combined generally account for the rest of California’s transportation emissions. One of the key elements of California’s strategy to reduce GHG emissions is a statewide cap-and-trade program that sets a cap on statewide GHG emissions and provides funding for CARB’s programs. The cap-and-trade program effectively sets a statewide limit on GHG emissions from major sources, such as electricity generation, and allows the entities responsible for those sources to meet the limit by reducing their emissions or by paying for allowances to emit GHGs. The payments take place during quarterly auctions, which have generated billions of dollars in revenue. The State deposits the revenue in its Greenhouse Gas Reduction Fund (cap-and-trade fund). State law allocates cap-and-trade revenue to various state agencies, including CARB, for the purpose of supporting programs intended to further reduce GHG emissions. CARB’s Transportation Programs CARB has established objectives to reduce CARB’s Key Objectives for transportation-related GHG emissions to meet California’s Transportation Sector the State’s overall emissions goals. To meet those goals, CARB designs, implements, and oversees a • Transition to ZEVs, including 1.5 million ZEVs by 2025 and variety of programs. State law requires CARB to 5 million ZEVs by 2030. prepare a scoping plan for achieving the maximum • Adopt more stringent GHG emissions requirements for all technologically feasible and cost-effective GHG light-duty vehicles, including passenger vehicles. reductions and to update that plan at least every • Reduce GHG emissions from medium- and five years. In its 2017 scoping plan, CARB published heavy-duty vehicles. a set of objectives for achieving California’s GHG goals. The text box summarizes CARB’s key • Transition to clean transit (bus) options. objectives for reducing transportation emissions in • Reduce vehicle miles traveled by passenger vehicles. California, as outlined in that plan. • Improve freight system efficiency and deploy freight vehicles and equipment capable of zero or As the text box shows, putting millions of ZEVs near-zero emissions. on California roads is part of California’s plan to reach its GHG emissions reduction goals. These • Transition to cleaner fuels that emit lower amounts of GHGs. ZEV objectives were established through executive orders from Governor Brown. Figure 2 shows that Source: CARB’s 2017 scoping plan and Governor’s as of October 2020, more than 700,000 ZEVs executive orders. have been sold in California, but the market still needs to grow significantly for the State to meet its objectives. California State Auditor Report 2020-114 11 February 2021 Figure 2 California’s ZEV Sales Must Continue to Increase to Meet Long‑Term Objectives 2030 ZEV OBJECTIVE 5 MILLION 5.0 4.5 4.0 3.5 3.0 2.5 2.0 2025 ZEV OBJECTIVE 1.5 MILLION 1.5 1.0 0.0 2012 2013 2014 2015 2016 2017 2018 2019 2020 )snoilliM ni( selaS VEZ evitalumuC Year Source: Executive orders and analysis of market data on light-duty ZEV sales from the California Energy Commission. Note: 2020 sales here are as of the end of October 2020. To help California achieve the ZEV objective, CARB operates multiple programs aimed at increasing ZEV ownership and supporting the ZEV market. The programs CARB has implemented in this area fall into two general categories: regulatory programs and incentive programs. CARB establishes its regulatory programs through a formal public rulemaking process; some of these programs require vehicle manufacturers to produce and sell certain numbers and types of vehicles or meet GHG emissions standards. For example, CARB operates a regulatory program aimed at increasing ZEV ownership that requires certain auto manufacturers to sell enough ZEVs to make up a required proportion of their overall sales in the State. These ZEVs include full battery-electric, hydrogen-fueled electric, and plug-in hybrid electric vehicles. Incentive programs are voluntary programs that often provide monetary payments to consumers who purchase low- and zero-emission vehicles. CARB approves these programs— sometimes at the direction of the Legislature—and reviews the programs’ funding each year. For example, CARB operates incentive programs that provide rebates or other financial support to consumers who purchase ZEVs. The intent of these rebates is to encourage customers to purchase ZEVs, which tend to have a higher sales price than gasoline-powered vehicles. The largest of these programs is the Clean Vehicle Rebate Project (CVRP), which has received more than $940 million from the cap-and-trade fund. 12 California State Auditor Report 2020-114 February 2021 CVRP is a statewide program that allows consumers to apply for rebates if they purchase battery-electric vehicles, hydrogen-fueled electric vehicles, plug-in hybrid electric light-duty vehicles, or zero-emission motorcycles. The maximum rebate amounts range from $750 to $7,000, depending on the type of vehicle purchased and the consumer’s income level. All of these programs work simultaneously toward achieving increasing ZEV sales. As Figure 3 illustrates, other factors like consumer preference and outside programs also play a role. Therefore, when a consumer buys a new ZEV in California, any combination of these factors may contribute to the consumer’s decision. The regulatory programs cause manufacturers to produce and sell the vehicles, while incentive programs, along with other factors, influence the consumers’ purchasing decisions. Further, some incentive programs overlap with one another, and consumers may use multiple incentives when purchasing a single vehicle. CARB’s programs do not operate in a vacuum; they can directly target the same objectives or even the same vehicles. CARB also operates regulatory and incentive programs that work toward shared objectives in other transportation areas. For example, CARB operates regulatory programs that require manufacturers to produce lower-emission heavy-duty vehicles, such as trucks. In one of these programs, manufacturers can earn compliance credits based on the lowered emissions from their heavy-duty vehicles. CARB simultaneously operates the Hybrid and Zero-Emission Truck and Bus Voucher Incentive Project (HVIP), which provides vouchers to commercial vehicle users, such as school districts, small businesses, and transit agencies, to help them purchase low- and zero-emission trucks and buses. Additionally, manufacturers can receive a higher number of compliance credits for producing HVIP-eligible trucks than they do for producing standard trucks. Both programs work toward the objective of reducing GHG emissions from heavy-duty vehicles. CARB also has certain pilot programs, which receive lower amounts of funding and are generally smaller in scope than larger programs like CVRP and HVIP. For example, CARB’s bus pilot program provides funding for local transportation agencies to acquire small numbers of zero-emission buses in order to reduce GHG emissions and demonstrate the practicality and viability of widespread adoption of that technology. During this audit, as Table A in Appendix A shows, we reviewed eight regulatory programs and 10 incentive programs operated by CARB, as well as one program—the Sustainable Communities program—intended to help the State reach its 2030 goals by reducing vehicle miles traveled and therefore GHG emissions for which CARB provides some oversight. CARB operates all 10 of the incentive programs we reviewed through program administrators who manage the day-to-day operations of the programs. In some cases, these program administrators are private nonprofit organizations; in others, they are local air districts. CARB oversees the programs through grant agreements that lay out specific program requirements and it is responsible for ensuring the success of these programs. California State Auditor Report 2020-114 13 February 2021 Figure 3 Various Factors Simultaneously Contribute to Increased ZEV Sales OTHER FACTORS Federal tax credits, REGULATORY PROGRAM high-occupancy vehicle Requires manufacturers to sell access stickers, and ZEVs as a certain percentage of consumer preferences also their overall sales each year. contribute to ZEV sales. ZEV SALES OBJECTIVE Increase ZEVs on California roads to 1.5 million by 2025 and to 5 million by 2030. INCENTIVE PROGRAMS* Make ZEVs less expensive for consumers: » CVRP provides rebates to consumers. » Clean Cars 4 All provides cash to lower-income consumers for ZEV replacement vehicles. » Financing Assistance helps low-income consumers get low-interest loans to purchase ZEVs. Source: Analysis of CARB’s regulatory and incentive programs, other state programs, federal programs, and executive orders. * Consumers may receive multiple incentives in order to purchase a single vehicle. Measuring Programs’ GHG Reduction‑Related Benefits Given the ambitious nature of the State’s goals for GHG emissions reductions and the progress still needed to meet those goals, it may be reasonable to have multiple programs that work together to address a shared transportation objective—such as putting more zero-emission cars on California’s roads or reducing GHG emissions from heavy-duty vehicles. However, a 2018 report by the Legislative Analyst’s Office (LAO) concluded that operating multiple programs with shared objectives can make it difficult to evaluate the effects of each program. The report notes that there is a 14 California State Auditor Report 2020-114 February 2021 wide range of state and federal emissions programs that can overlap, including regulatory and incentive programs aimed at increasing the sale of ZEVs, as we discuss above. The report concludes that the interaction between the various programs can make it difficult to evaluate the effects of each program. For example, although ZEV sales in California are increasing, the regulatory and incentive programs’ shared objective can make it difficult to know how many cars are sold because of the regulation and how many because of the incentives. The LAO’s report stated that this limitation may, in turn, make it difficult for the State to determine which programs the State should expand to achieve future goals most effectively. The LAO report also described free‑riders—consumers who receive a vehicle rebate for purchasing a ZEV but would have purchased the vehicle even without the rebate. For example, such a consumer may already believe that the savings from reduced fuel costs outweigh the additional upfront costs to purchase a ZEV. Further, some consumers’ primary motivation for buying a ZEV may be the desire to drive a cleaner vehicle or they may be less likely to depend on a rebate for their purchase decisions because they have higher incomes. According to the LAO’s report, failure to account for these consumers could result in overstated estimates of direct emissions reductions for some programs. In fact, research indicates that it is important to measure behavioral responses to economic incentives such as rebates in order to determine their effectiveness. Benefits Not Related to GHGs In addition to reducing GHG emissions, CARB designs and operates programs intended to achieve socioeconomic benefits. State law directs a minimum percentage of cap-and-trade funding to two populations—disadvantaged communities and low‑income communities or households. State law tasks the California Environmental Protection Agency (CalEPA) with identifying disadvantaged communities in California. CalEPA has classified these communities based on their sensitivity to environmental pollution using a range of factors, including air pollution and public health data. State law defines low-income communities as those areas in which median household income falls below certain statewide thresholds. State law requires that a minimum of 25 percent of cap-and-trade funding be allocated to projects located in disadvantaged communities and a minimum of 10 percent to low-income communities or households. State law also directs CARB to design programs that achieve other benefits beyond GHG emissions reductions and minimum required spending. For example, state law directs CARB to use cap-and-trade California State Auditor Report 2020-114 15 February 2021 funds in a way that maximizes economic benefits and that fosters job creation by promoting in-state GHG emissions reduction projects carried out by California workers and businesses. In addition, the Legislature has directed CARB to establish programs that are accessible to and provide benefits for Californians living in disadvantaged, low-income, and moderate-income communities, including increasing their access to affordable ZEVs and near-zero-emission vehicles. CARB’s own funding guidelines state that agencies that receive cap-and-trade funding must, to the extent feasible, foster job creation and training with an emphasis on disadvantaged and low-income communities and households. To achieve socioeconomic benefits, including reduced transportation costs, some of CARB’s programs focus on achieving equitable outcomes instead of maximizing GHG emissions reductions. For example, its Financing Assistance for Lower-Income Consumers program (Financing Assistance program) provides low-income consumers with grants and low-interest loans to purchase ZEVs or hybrid vehicles that the consumers otherwise might not be able to afford. CARB states that it also expects the program to provide economic benefits for its participants, including increased credit scores and the ability to qualify for housing loans, presumably because of their improved credit. Similarly, CARB operates an Agricultural Worker Vanpools program that provides agricultural workers in disadvantaged and low-income communities with transportation to their worksites in low-emission vans, in place of those workers using their individual gasoline-powered vehicles or vanpooling in gasoline-powered vans. Because these programs are less cost-effective in achieving emissions reductions, it is critical for CARB to measure other benefits and make funding decisions accordingly. For example, the Financing Assistance program spends seven times more to reduce GHG emissions by one metric ton than the CVRP program does. Therefore, CARB must clearly define and measure the specific benefits—other than GHG emissions reductions—that it intends programs like Financing Assistance to achieve. Reporting and Resources CARB’s board approves an annual funding plan that describes the goals and expected benefits for many of its cap-and-trade-funded incentive programs. The funding plan covers incentive programs, including CVRP, HVIP, and Financing Assistance, and in it CARB annually updates its board on the status of its programs. The updates include summaries of the activities performed to date and overviews of the purposes of the programs. CARB then 16 California State Auditor Report 2020-114 February 2021 proposes funding allocations for each program. Total Cap‑and‑Trade Funds Allocated by CARB In fiscal year 2019–20, the Legislature allocated to the Transportation Incentive Programs $485 million in cap-and-trade funding for CARB’s We Reviewed transportation programs, bringing the total allocation to more than $2 billion in cap-and-trade • CVRP ($944 million) funding since fiscal year 2013–14. The text box • HVIP ($488 million) lists the total cap-and-trade funds CARB allocated to the incentive programs we reviewed for fiscal • Funding Agricultural Replacement Measures for Emission years 2013–14 through 2019–20. CARB also uses Reductions (FARMER) ($251 million) the funding plan to propose new programs, and • Advanced Technology Demonstration Projects CARB staff stated that the funding plans contain ($230 million) the justification or design for most of the incentive • Clean Cars 4 All ($102 million) programs we reviewed. The funding plan also projects the future GHG emissions reductions each • Zero-emission bus pilot program ($85 million) program will achieve if given the requested level • Car-sharing pilot program ($55 million) of funding. CARB bases its projections of GHG emissions reductions on the number and types of • Zero-Emission Drayage Truck pilot program ($40 million) vehicles it expects the programs to support. • Financing Assistance program ($34 million) • Agricultural Worker Vanpools ($6 million) CARB also reports annually to the Legislature regarding the estimated benefits its Source: CARB’s 2020 annual report to the Legislature, fiscal cap-and-trade-funded programs have achieved.2 year 2020–21 funding plan, expenditure records, and our program review. The annual report covers programs administered by Note: The zero-emission bus pilot program’s funding amount CARB as well as programs administered by other includes zero-emission truck pilot funding. state agencies. State law requires CARB to develop funding guidelines for all agencies that receive cap-and-trade funding. The annual report contains data on the estimated GHG reductions the majority of CARB’s (and other agencies’) programs achieved as well as the programs’ cost. It is also the primary mechanism for CARB to inform the Legislature of the benefits of its programs for disadvantaged and low-income communities. Limited and potentially unpredictable funding underscores the State’s tight time frame for meeting the 2030 goal. CARB operates a range of programs aimed at reducing transportation GHG emissions, and many of these programs receive their funding from the State’s cap-and-trade fund. As we note above, in fiscal year 2019–20, the Legislature appropriated $485 million from the cap-and-trade fund to CARB’s transportation programs. However, in part because of the ongoing COVID-19 pandemic, the quarterly cap-and-trade auction in May 2020 generated quarterly 2 State law requires the Department of Finance (Finance) to submit an annual report to the Legislature on the status and outcomes of cap-and-trade funded programs. The law also requires CARB to develop funding guidelines for the agencies administering programs, and those guidelines state that CARB compiles data from the administering agencies and coordinates with Finance to prepare the report. Therefore, although Finance submits the report to the Legislature, because of CARB’s lead role in preparing it, we refer to it as “CARB’s annual report” at times in this report. California State Auditor Report 2020-114 17 February 2021 proceeds of only $25 million, compared to the average of more than $700 million that it had received for each of the previous 11 quarters. The May 2020 decrease caused a reduction in funding to CARB’s programs for the year of $81 million, from $557 million to $476 million. Although the auction revenue has rebounded somewhat since, with the last two auctions raising $474 million and $587 million, it remains below the historical average—compounding the State’s short time frame as another challenge to meeting its GHG goals. This uncertainty also highlights the importance of CARB’s ability to assess and maximize the effectiveness of its transportation programs as well as to help the Legislature guide the State’s resources to programs that will best help California achieve its GHG reduction goals. 18 California State Auditor Report 2020-114 February 2021 California State Auditor Report 2020-114 19 February 2021 Chapter 1 CARB HAS NOT DONE ENOUGH TO COLLECT AND ANALYZE DATA TO DEMONSTRATE ITS PROGRAMS’ EFFECTIVENESS IN REDUCING GHG EMISSIONS Chapter Summary The State is at risk of not meeting its GHG reduction goals for 2030, and CARB has acknowledged the need for greater emissions reductions from the transportation sector. Because transportation-related emissions are the largest single source of GHG emissions in California and because they have increased in recent years, it will be necessary to accelerate the pace of reduction if the State is to achieve its goals. Given the ambitious nature of these goals and the progress still needed to meet them, it may be reasonable for CARB to operate multiple programs that address a shared transportation objective. However, to ensure that it is operating the most effective mix of programs, it is important for CARB to identify the GHG reductions that each individual program achieves. CARB does not formally account for the incentive programs’ overlap with relevant regulatory programs, nor does it collect and analyze all of the data necessary to isolate the incentive programs’ GHG emissions benefits. CARB has also not done enough to measure or collect data about behavioral changes of the participants in its consumer-focused incentive programs, further limiting its ability to identify the programs’ individual impacts on emissions. The State needs more accurate program measurement to guide its GHG investments and increase the chances of meeting its GHG goals. The measurement concerns we identified limit the usefulness of the information CARB reports annually to the Legislature about the GHG reductions from its cap-and-trade-funded incentive programs. Specifically, CARB overstates the programs’ GHG reductions, although it is not clear by how much. These reporting issues also obscure the programs’ true cost-effectiveness. Finally, CARB must do more to ensure that the emissions reductions it reports are accurate and supported by valid underlying data. CARB Has Not Identified the GHG Reductions Its Incentive Programs Provide CARB’s regulatory and incentive programs are intended to work toward shared objectives for reducing GHGs, such as increasing the number of ZEVs on California’s roads. CARB has taken steps 20 California State Auditor Report 2020-114 February 2021 to identify the new GHG reductions its regulatory programs are intended to achieve and has generally designed those reductions to be achievable through the regulatory programs alone. However, it has not done enough to demonstrate the amount of GHG reductions it projects and measures for its incentive programs. CARB does not collect certain data or perform measurements that would help it determine those additional impacts, including the extent to which the incentive programs motivate consumers to purchase lower-emission vehicles that they otherwise would CARB’s lack of measurement of not have purchased. CARB’s lack of measurement of participants’ participants’ behavioral changes behavioral changes also limits its ability to project the longer-term limits its ability to project the benefits of its incentive programs. longer‑term benefits of its incentive programs. CARB Cannot Distinguish the Individual GHG Benefits of Multiple Programs That Work Toward Shared Objectives Given the ambitious nature of the State’s goals for GHG reductions, it may be reasonable for CARB to operate multiple programs that work toward a shared transportation objective, such as putting more ZEVs on California’s roads or reducing GHG emissions from heavy-duty vehicles. However, to ensure that it is operating the most effective mix of programs, it is important for CARB to identify the GHG reductions that each individual program achieves. The short time frame remaining for the State to meet its 2030 GHG goals also underscores the need for the best information possible to guide the use of the State’s limited resources. We engaged an environmental consultant to help us review CARB’s methods for projecting the GHG emissions reductions it expects its programs to achieve. For the eight regulatory programs and nine of the incentive programs we reviewed, our consultant determined that CARB’s methodologies for estimating emissions reductions were generally reasonable on a program-by-program basis.3 However, because of the potential overlap among many of these programs, we—along with our consultant—also evaluated whether CARB accounted for that overlap when designing the programs and when measuring the programs’ actual estimated GHG reductions. For the eight regulatory programs we reviewed, which CARB approved or modified between 2007 and 2020, CARB generally identified relevant overlap between those proposed regulatory programs and other existing transportation regulatory programs. For example, when it proposed a new regulatory program in 2018 to require transit agencies to purchase only zero-emission buses in the 3 The solicitation for one of our 10 selected incentive programs, the Zero-Emission Drayage Truck Pilot, was not released until late November 2020, and we therefore did not include it in our review of GHG emissions reductions. California State Auditor Report 2020-114 21 February 2021 future, CARB took steps to account for GHG reductions from the existing regulatory programs that set emissions requirements for buses. By doing so, CARB isolated the additional GHG emissions reductions that the new regulation would achieve. In contrast, although the proposed regulatory programs sometimes acknowledged the existence of incentive programs that share the same objectives, CARB did not formally assess how relevant incentive programs might overlap with the proposed regulatory programs. When we asked CARB why it did not assess these overlaps, key staff and program managers for two regulatory programs explained that they did not consider incentive programs when determining the regulation’s costs. Specifically, CARB does not assume that incentive funds will be available in the future to help manufacturers and consumers offset vehicle costs. Therefore, CARB designed these regulatory programs to achieve the expected reductions without help from incentive programs that target the same types of vehicles. Although reasonable, this approach means that any GHG reductions it claims from incentive programs that work toward the same objectives should be above and beyond what the regulatory programs achieve. Because incentive programs can represent significant costs, it is important that CARB be able to measure those additional GHG reductions to ensure that it is pursuing the most effective approaches to meeting the State’s goals. However, CARB has not done enough to measure the emissions CARB has not done enough to reductions its incentive programs achieve beyond the reductions measure the emissions reductions brought about by its regulatory programs. We reviewed its incentive programs achieve documentation related to a selection of CARB’s incentive programs beyond the reductions brought and found that CARB generally did not acknowledge the potential about by its regulatory programs. overlap with regulatory programs or discuss how it accounted for the overlap in the incentive programs’ design. CARB also does not measure precisely how many additional GHG benefits its incentive programs may actually provide. For example, as we discuss in the Introduction, CARB has a regulatory program that requires car manufacturers to sell ZEVs and it operates incentive programs that provide payments to consumers to encourage them to purchase ZEVs. According to CARB’s 2019 Zero Emission Vehicle Credits report, released in August 2020, car manufacturers were overcomplying with the ZEV regulation. This overcompliance could indicate that CARB’s ZEV incentive programs achieve additional GHG emissions reductions by driving consumer demand and ZEV sales beyond what would be sold under the regulation alone. However, CARB has not determined whether its incentive programs are responsible for the additional sales. CARB offered two primary reasons for not performing such a determination, but these reasons suggest a lack of coordination in measuring how the respective programs perform. First, CARB does 22 California State Auditor Report 2020-114 February 2021 not track the number of ZEVs sold over and above the number required by the regulation because the regulatory program is not designed to measure or require reporting on the number of vehicles sold. Instead, the regulatory program uses a system in which manufacturers earn a certain number of credits per vehicle sold in California, and CARB monitors manufacturers’ credit balances to ensure that they have complied with the regulation. In addition, some ZEVs earn more credits than others. For example, a vehicle with a 350-mile battery range earns the manufacturer four credits, while a vehicle with a 50-mile range earns only one credit. Although longer-range battery-electric vehicles do not have a greater direct impact on GHGs, because they may be more attractive to consumers, the regulatory program awards more credits for these vehicles to reward manufacturers for producing and selling them. In contrast, CARB measures the GHG emissions reductions achieved by its incentive programs using the number of vehicles for which CARB provides a rebate. CARB staff explained that although it is feasible to report the credits generated in a year and the number of ZEVs sold, CARB has not done so because of the reporting requirements and how it evaluates compliance with the regulation. Because it has not taken steps to coordinate the performance measurements for regulatory and incentive programs, CARB does not know how the additional ZEVs being sold in the State compare to the number of rebates CARB provides through its incentive programs. Being able to make that comparison would be an important step in analyzing the extent to which CARB’s incentive programs achieve additional emissions reductions. Second, the manager for CVRP, the largest incentive program, stated that CARB cannot conclusively identify how many ZEVs are on the road because of CVRP, as many factors play a role in a Without conducting analyses to consumer’s decision to purchase one. The manager explained that validate its assumptions, CARB if the market for ZEVs is increasing, CARB assumes that CVRP cannot accurately identify the is having a positive impact on the market. However, without GHG emissions reductions that conducting analyses to validate these assumptions, CARB cannot its ZEV incentive programs are accurately identify the GHG emissions reductions that its ZEV actually achieving. incentive programs are actually achieving. We identified similar gaps in CARB’s analysis of how HVIP, an incentive program for low- and zero-emission trucks and buses, relates to CARB’s regulatory program that establishes GHG emissions limits for heavy-duty trucks and buses. As it does in the ZEV regulatory program, CARB uses a credit system to monitor its regulation of GHG emissions for heavy-duty vehicles. Manufacturers can earn credits by producing low- or zero-emission heavy-duty vehicles, such as trucks and buses, and HVIP provides incentives to encourage companies and transit agencies to buy these vehicles. However, unlike their practice for the ZEV program, California State Auditor Report 2020-114 23 February 2021 staff do not track the total number of credits that all manufacturers have generated under the heavy-duty GHG regulatory program across the entire industry; instead, CARB explained that it assesses each manufacturer individually. The CARB branch chief who oversees compliance with this regulation explained that CARB has been focused on ensuring that individual manufacturers comply with the regulation but acknowledged that it would be possible to calculate the industrywide credit balance. Until it does, CARB does not know whether the industry as a whole has overcomplied with the requirements in a way that might indicate an additional effect from incentives. Moreover, CARB has not evaluated the extent to which manufacturers are using HVIP-eligible vehicles to comply with the GHG regulations for heavy-duty vehicles. Without assessing the relationship between HVIP and the related regulatory program, CARB cannot demonstrate whether HVIP is achieving GHG reductions over and above what the regulatory program would achieve on its own. As with the ZEV programs, a greater level of coordination here is needed. In fact, the branch chief confirmed that CARB collects the data it needs to determine how frequently manufacturers have used vehicles eligible for incentives like HVIP to comply with regulatory requirements, but that the analysis would require coordination across different divisions of CARB. CARB has also not isolated the GHG reduction benefits of CARB has not isolated the GHG incentive programs that overlap with one another. In addition to reduction benefits of incentive sharing the same transportation objective of increasing ZEVs in programs that overlap with one California, three of the incentive programs we reviewed—CVRP, another, which could lead to Clean Cars 4 All, and Financing Assistance—also overlap because the double or triple counting of qualifying consumers are able to receive financial support from emissions reductions associated one, two, or all three programs for a single purchased vehicle. with a single vehicle. Therefore, if CARB does not account for this overlap, it could lead to the double or triple counting of emissions reductions associated with a single vehicle. Despite this risk, our consultant found that CARB’s methodologies for estimating the emissions reductions from the Clean Cars 4 All and CVRP programs do not account for the overlap or specify how CARB avoids simultaneously assigning the emissions reductions from a single vehicle to multiple programs. Because the Clean Cars 4 All and Financing Assistance programs are relatively small compared to CVRP and because not all consumers are eligible for incentives from multiple programs, the effect of the overlap on CARB’s emissions estimates may be relatively small. However, given that the Governor’s budget for fiscal year 2021–22 proposed directing increased funding to existing transportation programs such as Clean Cars 4 All and Financing Assistance, the importance of this measurement issue could significantly increase if CARB does not take steps to address it. 24 California State Auditor Report 2020-114 February 2021 CARB Does Not Know How Frequently Its Incentive Programs Change Purchasing Behavior Knowing the extent to which its incentive programs change participants’ behavior is crucial for CARB to accurately assess the effectiveness of those programs in achieving additional GHG reductions. Research has shown that it is important to measure behavioral responses to economic incentives such as rebates in order to determine an incentive’s effectiveness. For example, determining how frequently an incentive payment influences consumers to purchase a cleaner vehicle than they otherwise would choose is a way of accounting for individuals who would purchase cleaner vehicles anyway—sometimes called free‑riders, as we discuss in the Introduction. CARB can then apply this information to its calculations of the vehicles’ impacts on GHG emissions to determine the reductions for which the incentive program is responsible. Conversely, failing to collect data on or measure behavioral effects could lead CARB to overstate the emissions reductions from its incentive programs. Such overstatement will occur if CARB gives an incentive program credit for the GHG reductions from all vehicles for which the program paid incentives, even if some consumers would have purchased some of those vehicles anyway. In such a scenario, the State would not be able to rely fully on the estimated GHG reductions reported for individual programs or on the relative cost-effectiveness of those reductions across programs. Problems with the reliability of this information would prevent CARB and the Legislature from being able to use it to direct the State’s limited resources away from less effective programs and toward more effective ones. Despite the importance of knowing whether incentive payments CARB has not done enough to influence individuals’ decisions, CARB has not done enough to measure or collect data about measure or collect data about how its consumer-focused incentive how its consumer‑focused programs have changed participants’ behavior. As Table 1 shows, for incentive programs have changed five programs we reviewed in which CARB provides an incentive participants’ behavior. payment or other financial assistance to consumers who purchase a cleaner vehicle, CARB collects information about behavioral changes of participants for only one—CVRP. Specifically, CVRP participants who receive rebates are given a survey that asks whether they would have acquired their clean vehicle without the rebate. Two of the other programs have participant surveys that include questions about participants’ satisfaction with their vehicles, among other topics. However, CARB was unable to adequately explain or provide documentation for its rationale as to why those surveys do not include questions about what led the participant to purchase the vehicle. For another program, the program manager stated that CARB does not have copies of the California State Auditor Report 2020-114 25 February 2021 survey questions that the external program administrators use; however, we found that the sample survey questions that CARB provides to the program administrators do not contain questions about whether participants would have purchased their vehicle without receiving an incentive. The CARB program manager for the last program—FARMER—could not remember why CARB does not require a survey of participants, but a program staff member noted that the air districts that help administer FARMER on CARB’s behalf stated that it was difficult to collect all of the required program information from participants and that the districts expressed hesitation to add more questions. Table 1 CARB Rarely Collects or Uses Data on Participants’ Behavior for Key Incentive Programs IS THERE AN EXISTING SURVEY IS DATA COLLECTED THAT COULD COLLECT ABOUT PARTICIPANT HAS CARB USED THE DATA TO PROGRAM PARTICIPANT BEHAVIORAL DATA? BEHAVIORAL CHANGES? ADJUST THE PROGRAM? Clean Cars 4 All Yes No* No CVRP Yes Yes Yes FARMER No No No Financing Assistance Yes No No HVIP Yes No No Source: CARB’s program grant agreements, surveys, survey data, fiscal year 2019–20 funding plan, and interviews with program managers. * The program manager stated that CARB does not have copies of the survey questions that the grantees use; however, the sample survey questions that CARB provides to grantees do not contain questions about whether participants would have purchased their vehicle without participating in the program. Program managers for the three incentive programs with surveys that do not collect behavioral change data expressed their belief that participants would not have purchased their vehicles without the incentives, but they could not demonstrate whether their belief is justified. For example, the manager for HVIP stated that there are not very many purchases of HVIP-eligible vehicles—zero- and low-emission trucks and buses—without an incentive and that HVIP-eligible vehicles represent a small part of the total market for heavy-duty vehicles. The manager’s claim suggests that HVIP is the reason consumers purchased the vehicles and that they would not have done so without the incentive payment. However, the manager also stated that CARB does not have any data to show what proportion of eligible vehicles were purchased with an HVIP voucher or what proportion of total heavy-duty vehicle purchases in California are for HVIP-eligible vehicles. When we asked for such data, the manager acknowledged that CARB has only anecdotal information from vehicle manufacturers who indicated that all or a majority of their zero- or low-emission vehicle sales occur because of incentives like HVIP. Further, the manager stated that 26 California State Auditor Report 2020-114 February 2021 CARB presumes there are few purchases of HVIP-eligible vehicles as a proportion of total heavy-duty vehicles because they are more expensive than conventional vehicles. However, without actual data, improved surveys, or both, CARB cannot confirm its assumptions that HVIP participants would not have purchased their vehicles without the incentive. CARB is also missing an opportunity to use data to design programs that maximize their effects on consumer behavior and therefore on GHG emissions. As Table 1 shows, CARB has made changes to only one program, CVRP, based on behavioral considerations. The manager for CVRP explained that over the seven years that CARB has collected survey data, it has used the data to inform and support program changes, and provided documentation of one such change—a limit on the price of vehicles for which consumers can receive CVRP rebate—approved in October 2019. Specifically, the external program administrator for CVRP noted that according to the program’s survey data, as the cost of a vehicle increases, the influence of a rebate on the purchasing decision decreases. The CARB program manager told us that CARB considered this survey result when determining the maximum price for eligible vehicles. However, our review of the survey data raised other questions about participant behavior that CARB could not answer. Between 2013 and 2019, 41 percent to 54 percent of respondents stated that they would have purchased their vehicle even without receiving the CVRP rebate. The CVRP manager explained that CARB knows that there are free-riders in any incentive program. However, the manager also stated that CARB has not reached any conclusions about what these responses indicate for CVRP’s effectiveness and although the response rates can help justify general changes to CVRP, CARB has not specifically used these response rates to CVRP Rebate Amounts as of July 2020 make any policy changes to the program. Standard maximum rebate amounts: In 2015 CARB approved an increase to the CVRP • Hydrogen-fueled electric vehicle: $4,500 rebate for lower-income consumers; the text box • Battery electric vehicle: $2,000 lists the current amounts of both the standard • Plug-in hybrid electric vehicle: $1,000 and increased rebates. Afterward, CARB added • Zero-emission motorcycle: $750 a question to the CVRP survey to ask consumers who received the increased rebate whether they Maximum rebate amounts for low-income consumers: would have acquired their vehicles without the • Hydrogen-fueled electric vehicle: $7,000 increased rebate amount. The responses to the • Battery electric vehicle: $4,500 survey indicated that about half of the participants • Plug-in hybrid electric vehicle: $3,500 who had lower incomes and received higher • Zero-emission motorcycle: $750 rebate amounts answered that they would have acquired their clean vehicles even without the extra Source: CVRP implementation manual as of July 2020. payment—a higher proportion than responded this way regarding the smaller, standard rebate. This result raises questions about whether the increased California State Auditor Report 2020-114 27 February 2021 amount is effective in changing consumer behavior. However, when we asked CARB what the survey results indicated about the program, the CVRP manager explained that CARB has not used this aspect of the data to make any program changes though the manager went on to say that CARB would likely review it to examine the efficacy of the additional rebate. CARB may be missing opportunities to use other sources of data to learn more about how effective its programs are in changing behavior. For instance, in addition to CARB’s CVRP rebate, the federal government provides a federal income tax credit for certain electric vehicle purchases. The federal income tax credit can be as much as $7,500 for certain battery electric vehicles, which is more than CARB’s maximum CVRP rebate of $4,500 for the same type of vehicle. Because the federal income tax credit program is limited to a certain total number of qualifying vehicles per manufacturer, the applicability of the federal income tax credit program for these types of electric vehicles will phase out over time. CARB could use the expiration of the federal income tax credit to help it assess the tax credit’s effect on behavior and use that information to evaluate the effectiveness of CVRP. For example, if consumers stop buying vehicles that are no longer eligible for the tax credit, it might mean that incentives cause consumers to purchase vehicles but that CVRP’s current rebate amounts are not a great enough incentive on their own. In addition, CVRP rebates make up a smaller percentage of ZEV sales in California than they did previously. Figure 4 shows that CVRP rebates have declined in proportion to total ZEV sales in the State over the last six years. Although CARB attributes this trend to an income limit for CVRP eligibility that CARB implemented in 2016 at the direction of the Legislature, it has not analyzed its data or the broader ZEV market to confirm the reason for the trend. There may be value in learning why consumers who purchase ZEVs without rebates decide to do so, but the CVRP manager explained that CARB has not conducted such an analysis because to do so CARB would likely need help from car dealerships, who are generally uncooperative and would not participate unless paid to do so. However, the manager did not describe any formal efforts CARB has made to collect such information. Although potentially Although potentially challenging, challenging, collecting and analyzing data from these additional collecting and analyzing data sources—along with better use of its own survey tools—could from additional sources—along help CARB understand the extent to which its programs change with better use of its own survey consumers’ behavior and thereby help CARB better measure its tools—could help CARB understand programs’ actual emissions reductions. Understanding its programs’ the extent to which its programs current effects on participant behavior might also help CARB make change consumers’ behavior. changes in those programs to increase their cost-effectiveness and have a greater impact on emissions reductions. 28 California State Auditor Report 2020-114 February 2021 Figure 4 CVRP Rebates Account for a Decreasing Share of ZEV Sales in California 180,000 160,000 140,000 120,000 53% 52% 100,000 80,000 49% 60,000 26% 27% 39% 40,000 27% 47% 48% 20,000 74% 73% 61% 51% 73% 0 2013 2014 2015 2016 2017 2018 2019 selaS VEZ launnA )selcihev fo rebmun( CVRP Rebate Issued No CVRP Rebate Issued Source: Analysis of published CVRP rebate information from CARB’s program administrator and ZEV sales information from the California Energy Commission. CARB Has Not Assessed Its Programs’ Effect on Other Behavioral Changes That Are Necessary to Achieve GHG Reductions We also reviewed whether CARB measures its programs’ impacts on other aspects of participant behavior that CARB depends on to achieve GHG reductions. For example, some of CARB’s pilot programs fund low- or zero-emission transportation services, such as car-sharing programs and vanpools. CARB intends for these programs to reduce GHG emissions by getting people to change their driving habits. CARB has planned to use the data gathered and lessons learned from these programs to identify opportunities to develop expanded programs in the future. A second kind of CARB pilot program provides funding for clean vehicles, such as zero-emission buses, to transit agencies and other entities to support larger-scale deployments of ZEVs, thereby accelerating their introduction and market penetration. Ultimately, this second kind of pilot program seeks to encourage more widespread use of these clean vehicles in California. Knowing whether these programs have the intended effects is crucial to determining their effectiveness at reducing GHG emissions, both now and in the future. However, CARB has not done enough to accurately measure the programs’ effects on participant behavior. For example, CARB operates a car-sharing pilot program that provides funding to establish car-sharing fleets in or near disadvantaged communities to offer alternative modes of transportation and encourage the use of clean cars. Through California State Auditor Report 2020-114 29 February 2021 this program, CARB has awarded funds to several program administrators throughout the State to implement various kinds of car-sharing projects that use clean vehicles. CARB’s emissions reduction projections for the pilot program assume that all of the trips taken in the shared clean vehicles replace trips that would otherwise be taken in a conventional car. However, as our consultant identified, this assumption would overstate the program’s emissions reduction if trips that participants made in clean vehicles instead replaced trips on public transit or by walking. Therefore, CARB should determine if the assumptions in its methodology are supported by participants’ behavior. One way to do so would be to collect information from participants about how they previously made the trips they now take in the shared clean vehicles. However, it is not clear whether CARB will have appropriate data It is not clear whether CARB will to reliably measure the program’s behavioral changes. The branch have appropriate data to reliably chief who oversees the car-sharing pilot program and the program’s measure the car‑sharing pilot manager told us that CARB does not have data about changes program’s behavioral changes. in program participants’ behavior. CARB requires the program administrators for the individual car-sharing pilots to conduct initial and follow-up surveys of program participants. However, CARB was not able to provide follow-up surveys for two of the pilots, explaining that it is working with its program administrators to determine timelines for the follow-up surveys and that all of the pilots have seen delays. When we reviewed the follow-up surveys for the four pilots CARB has collected so far, we found that the surveys for only two contained questions that could allow CARB to identify what kind of transportation participants replaced with shared vehicles. Subsequently, in January 2021, CARB provided a recently approved, additional follow-up survey for one of the two pilots that was previously missing the necessary information. Although the new survey does contain one such question, this pilot began in 2015 and we are concerned that CARB did not better ensure the collection of this information earlier in the program’s existence, which would have allowed CARB to more closely monitor the program and be certain that it has the data it needs to evaluate its assumptions. Further, it is not clear that CARB will have better information when the car-sharing pilot program has ended because grant agreements do not consistently specify what data the follow-up surveys must contain. Without ensuring that it has this information and that the information is complete and usable, CARB cannot verify its assumption or determine precisely how effective its car-sharing pilot program is at reducing GHGs. The branch chief explained that a program evaluation contract CARB signed with the University of California, Berkeley (UC Berkeley) in May 2020 should provide 30 California State Auditor Report 2020-114 February 2021 insight into the impacts of pilot programs, including the car-sharing pilot program, on users’ lives. However, results from this evaluation will likely not be available until 2022. CARB does not have sufficient data CARB also does not have sufficient data on the longer-term on the longer‑term behavioral behavioral changes that some of its pilot programs for heavy-duty changes that some of its pilot vehicles intend to achieve. For instance, CARB operates a bus pilot programs for heavy‑duty vehicles program that provides grant funding to enable transit agencies intend to achieve. and other entities to acquire multiple zero-emission buses and associated infrastructure, such as charging equipment. CARB intends for the pilot program to help the participating agencies address challenges in deploying zero-emission buses and to encourage those agencies and others to transition more of their fleets to zero-emission buses, thereby reducing GHGs. To know whether the pilot program is having the intended effect, CARB stated that it measures behavior changes for participants in its bus pilot program by tracking whether participating transit agencies go on to invest in additional clean buses during or after the end of the pilot program. If they do, the program may be responsible for more considerable GHG reductions over the longer term. Achieving such long-term reductions is a goal of CARB’s pilot programs in general. Further, the information that CARB uses to measure behavioral changes related to the bus pilot program does not allow it to fully determine whether the program has had its intended effect. Specifically, the information indicates which participating agencies applied for other incentive programs from CARB or other agencies to purchase additional buses. However, that information alone does not allow CARB to conclude that the bus pilot program was the reason for the subsequent vehicle purchases. Without collecting and reviewing data about why participating agencies do or do not choose to acquire more clean buses after participating in the bus pilot program, CARB cannot be certain that the pilot program is responsible for changing agencies’ behavior. CARB staff also referred us to another current contract with a third party for some of its heavy-duty vehicle programs, which it expects will help it determine whether its pilot programs are successful, including the bus pilot program. However, the contract’s scope of work does not include any specific requirement to ask participants whether they would have changed their purchasing behavior if they had not participated in the pilot program. The scope of work includes an evaluation of whether the vehicles in the pilot program can replace conventional vehicles and the practicality of widespread adoption of those vehicles. Although CARB entered into this contract in 2017, final reports that include the contractor’s analysis of the technology and participant experiences for the program’s various individual pilots were not yet completed at the time of our review in October 2020. California State Auditor Report 2020-114 31 February 2021 Better Program Measurement Would Help the State Maximize Its Efforts to Reduce GHG Emissions CARB’s inability to measure the GHG reductions from its incentive programs diminishes the usefulness of the information the State uses to fight climate change. Currently, CARB’s reporting to the Legislature on the effectiveness of its incentive programs does not accurately estimate the emissions reductions that each of its programs achieves. CARB has responded to other specific questions from the Legislature about how the State can use its programs to pursue longer-term emissions reductions. However, CARB’s lack of information about how its programs change consumer behavior and its imprecise metrics for defining success have limited the usefulness of its responses to the Legislature. As we describe in the Introduction, CARB reports annually to the Legislature on the estimated GHG emissions reductions its programs achieve as well as the programs’ costs. For the nine incentive programs our consultant reviewed, CARB generally estimates GHG reductions based on the number and types of vehicles for which it paid incentives during the year.4 For instance, CARB estimates the GHG reductions from CVRP by calculating the difference between the emissions for the clean vehicles participants purchased and the emissions of an average new vehicle CARB assumes the consumer would otherwise have purchased. CARB then multiplies those reductions by the total number of vehicles the incentive helped pay for. This approach is generally consistent across CARB’s incentive programs that we reviewed. Although this is reasonable on a per vehicle basis, CARB’s approach leads it to overestimate its incentive programs’ total GHG reductions because it does not account for the effects on industry and consumer behavior of regulatory programs and other factors we have discussed, such as the federal tax credit. Instead, CARB assumes that the emissions reductions from all of the vehicles funded by an incentive program would not otherwise have occurred. By attributing all vehicles to a program without considering these other factors, CARB is overstating the programs’ CARB is overstating the incentive benefits in its annual reports to the Legislature though it is unclear programs’ benefits in its annual by how much. One effect of this overstatement is to obscure the reports to the Legislature though it programs’ cost-effectiveness in reducing GHG emissions. Taking is unclear by how much. the steps we describe in the previous section to account for the incentive programs’ overlap with regulatory programs, as well as to improve efforts to ensure that incentive programs cause consumers 4 The solicitation for one of our 10 selected incentive programs, the Zero-Emission Drayage Truck Pilot, was not released until November 2020, and we therefore did not include it in our review of GHG emissions reductions. 32 California State Auditor Report 2020-114 February 2021 to make choices they otherwise would not, would help CARB get closer to identifying—and reporting—the actual emissions reductions its incentive programs achieve. As the primary report on the status and outcomes of all programs funded by cap-and-trade—including the programs that CARB operates—CARB’s annual report must contain accurate estimates of the GHG emissions reductions the funds have achieved. The Legislature has referenced GHG emissions reductions that programs achieve when it has codified programs into state law. For example, effective in 2018 the Legislature established in law the Clean Cars 4 All program, which previously had been a pilot program that CARB operated only in certain parts of California. The Legislature indicated that one of the reasons for expanding the program and making it permanent was the GHG emissions benefits that the pilot program had achieved. The annual reports also contain information that the Legislature has relied on, in part, when considering other changes to state law regarding how Without accurate information cap-and-trade funds are spent. Without accurate information in in CARB’s annual reports, the the annual reports—which would make these reports a reliable Legislature’s ability to make emissions measurement tool—the Legislature’s ability to make decisions about its investments decisions about its investments to help the State achieve its GHG to help the State achieve its GHG goals may be hampered. Specifically, if the annual reports contain emissions goals may be hampered. accurate information, these reports can better help the Legislature make decisions about whether to continue funding a given program at its current level, decrease funding and use those resources elsewhere, or significantly increase funding to help the State achieve its GHG emissions goals. In addition to the annual reports, the Legislature has required that CARB publish information about how certain programs will help CARB meet the State’s goals. For example, the Supplemental Report of the 2018–19 Budget Act required CARB to update its forecast annually to indicate how much CVRP funding will be necessary to meet the State’s objective of five million ZEVs on the road by 2030. However, because CARB stated that it does not have data to calculate how CVRP drives ZEV sales or how other factors may change the future need for CVRP rebates, CARB could not be fully responsive to this question in the analysis it included in its fiscal year 2019–20 funding plan. CARB responded to the Legislature’s request by assuming that five million vehicles would be sold and used historical rebate payment information to calculate the price to the CVRP program. Because it assumed that CVRP would provide rebates for half of the ZEVs sold in California based on its historical data, CARB projected that it would ultimately need to pay out 2.1 million rebates totaling $5.6 billion to get five million ZEVs on the road, although it noted that it would not reach that goal until after 2030, given the current ZEV market. This approach suggests that CVRP payments are merely a by-product of ZEV California State Auditor Report 2020-114 33 February 2021 sales and does not provide the Legislature information about how to use CVRP funding to actually generate ZEV sales. The CVRP program manager indicated that CARB took the approach it did with the data it had available. However, we believe that if CARB pursued some of the analysis and data we describe in the previous section, it could offer useful information about how to invest in its incentive programs. The Legislature has also asked CARB about its plans for meeting longer-term objectives. Effective January 2019, it directed CARB to establish a three-year investment strategy that includes estimated funding needs for zero- and near-zero emission heavy-duty vehicles and equipment to meet certain state goals, including lessening the impacts and effects of climate change. The strategy identified several key areas in which CARB’s heavy-duty incentive programs must be successful in order to meet the State’s goals, including support for the future evolution of cleaner technology. In the strategy, CARB explained that current technology is insufficient to meet California’s long-term air quality and climate change goals. Therefore, CARB intends its incentive programs to support the development, improvement, and commercialization of cleaner technology until the technology can either be supported with other programs or exist without the support of the programs. However, CARB has not yet adequately defined the metrics it will CARB has not yet adequately use to determine when the incentive programs have successfully defined the metrics it will use to accomplished these goals. Instead, CARB’s strategy identifies determine when incentive programs anecdotal evidence about production costs and observations about have successfully accomplished the use of technologies as metrics for determining the programs’ their respective goals. success. CARB program managers stated that the metrics do not include quantifications or indicators of success because it is still working to identify and develop what those should be. CARB explained that before the publication of the next strategy document for fiscal year 2021–22, it will continue to work with stakeholders to identify what data are available that could be used to measure whether a technology can transition to a different program or stand on its own. One of the program managers stated that ideally, when the next strategy is published, a few of the metrics will have more details about how CARB plans to quantify success and determine whether a technology is able to transition out of the program. We are concerned that CARB has not yet identified quantifiable metrics to determine when its heavy-duty programs have succeeded in supporting a new technology. The state law requiring CARB to develop this investment strategy took effect in January 2019 and CARB has identified the need to evaluate the effectiveness of its investments in HVIP since at least 2014. Until it has developed these informal metrics into measurable objectives, CARB will 34 California State Auditor Report 2020-114 February 2021 be limited in its ability to know when an incentive program has succeeded in supporting new technology to the point that it can either be supported by a different program or stand on its own. To ensure the effective future use Lastly, to ensure the effective future use of state resources to fight of state resources to fight climate climate change, CARB must have plans for how it will phase out change, CARB must have plans or adjust incentive programs when those programs have met their for how it will phase out or adjust goals. However, CARB’s investment strategy for heavy-duty vehicle incentive programs when those programs does not include details for phasing out relevant incentive programs have met their goals. programs, and staff were not able to provide them when we asked whether CARB had formal projections for when HVIP would no longer be necessary to support a specific kind of technology. Planning for how long CARB’s incentive programs will be needed is part of ensuring the success of its other programs as well. For example, in 2019 CARB established a regulatory program that will require, beginning in January 2029, all new bus purchases by transit agencies in California to be for zero-emission buses. CARB recognized that a transition of the bus fleet to entirely zero-emission buses could be challenging for transit agencies and, as a result, identified its commitment to using incentive programs to help with the transition. However, some transit agencies have told CARB that the zero-emission buses currently in production do not serve all of their needs. Accordingly, the program manager explained to us that HVIP may still be needed to support and develop the market for zero-emission buses after the regulatory compliance dates have passed. The manager also acknowledged that CARB does not have a projected date for when HVIP incentive payments for zero-emission buses will no longer be necessary. Although making these projections will likely be challenging for the newest technologies, CARB must work to establish clear metrics for when incentive programs have achieved their goals as well as strategies for how to use those resources thereafter. Without doing so, CARB risks continuing to fund programs that are no longer needed to actively support GHG reduction objectives. CARB’s Measurement of Emissions Reductions Suffers From Errors and Issues With Its Programs’ Underlying Data CARB has not done enough to ensure that the reductions in emissions it reports are fully supported by program data. Our review of CARB’s emissions reductions in its most recent annual report found several errors in the calculations CARB reported to the Legislature—some of them significant. In other cases, CARB cannot provide documentation to support the accuracy of the data it uses in its aggregate reports. Finally, CARB’s inability thus far to measure emissions by region creates uncertainty about a statewide transportation program it administers with regional planning California State Auditor Report 2020-114 35 February 2021 organizations, and it undermines the State’s ability to identify and pursue the best regional strategies to ensure that the program is successful in reducing GHGs. CARB Has Not Done Enough to Ensure the Accuracy of the Data It Reports The emissions reductions that CARB has included in its annual report to the Legislature have not always been accurate and fully supported by its program data. CARB estimates emissions reductions using the methodologies we describe in the preceding section—generally based on emissions reductions calculated from the number and type of vehicles for which it provided incentives— and the data it receives from the program administrators who handle the daily operations of CARB’s incentive programs. Those data contain the number of incentives paid for low- and zero-emission vehicles and specifics about the vehicles, such as vehicle type, engine specifications, and model year. With assistance from our consultant, we reviewed CARB’s calculations of emissions reductions for the most recent annual report, issued in March 2020. Our review found that CARB’s calculations were generally accurate, based on the methodologies for those programs. However, we found instances in which CARB incorrectly applied its own methodologies and therefore made errors in its calculations. One of these errors had a relatively small impact. Specifically, CARB overstated the GHG reductions by 3 percent for heavy-duty vehicles with a specific type of engine—one of several types of vehicles supported by the HVIP program. However, errors affecting other programs had larger impacts. For example, CARB used the wrong time frame for its CVRP and Financing Assistance programs when calculating the emissions reductions for vehicles those programs helped pay for. As a result, CARB underestimated the amounts it reported to the Legislature for those programs’ emissions reductions by more than 40 percent during the second half of the 2019 reporting cycle, which is included in the 2020 annual report. After we raised these errors in the CVRP, Financing Assistance, and HVIP calculations with CARB, program staff informed us that they had already found and corrected internally some of the errors and would work to correct the others. However, we believe these are errors CARB should We believe CARB should have have identified before publishing the annual report. Further, at identified the errors in its annual the time of our review, CARB had not provided any corrected report before publishing it. emissions estimates to the Legislature. Additionally, for the Clean Cars 4 All program, staff were unable to provide documentation to support CARB’s emissions reduction calculations for the entire second half of the 2019 reporting cycle, which is also included in its 2020 annual report. Without retaining this supporting 36 California State Auditor Report 2020-114 February 2021 documentation, even for recent years, CARB cannot demonstrate that the program accomplishments included in its annual reports are accurate. CARB has not always ensured that CARB also has not always ensured that the program activity it the program activity it includes in includes in its annual reports is fully supported by underlying its annual reports is fully supported program data. CARB staff aggregate detailed data from the by underlying program data. program administrators to summarize a program’s total incentive payments and emissions reductions in its annual reports. For the five programs we reviewed that provide incentive payments or other financial incentives to support consumers’ vehicle purchases, we used a selection of data from the years 2016 through 2020 to compare the programs’ underlying data to the totals given in CARB’s annual reports. We found that the data for two of the programs did not align with the total incentive payments in its annual reports. For CVRP, we found discrepancies between the cap-and-trade spending totals in the program data and the totals CARB reported to the Legislature. Although these discrepancies were relatively small—an average difference of 7 percent of the total dollar amount reported by CVRP—they occurred in each of the four reporting cycles we reviewed going back to 2016. Program staff were unable to adequately explain the discrepancies. We did not fully assess the underlying data for the Financing Assistance program because CARB staff informed us that they had already identified inaccuracies in the data that had affected the annual public report. The program manager explained that errors, including incentive payment amounts and omitted data, had led CARB to report inaccurate information over the history of the program, which began in 2015. Finally, CARB has not taken sufficient steps to ensure the accuracy of the program data it uses in its calculations. Because CARB relies on its program administrators to compile and provide program data, we reviewed how CARB ensures that the data they compile are accurate. Program staff explained that CARB relies on what it refers to as desk reviews to ensure that the data are valid and supported by appropriate documentation. During these reviews, CARB examines the documentation for specific incentive payments to determine whether the vehicle data for those incentive payments are accurate, among other tasks. Significant inaccuracies in the data could affect estimated emissions reductions as well as undermine CARB’s efforts to track the specific types of vehicles its incentives pay for as part of longer-term efforts to support emerging technology. Even so, CARB has completed these desk reviews for only two of the five programs we reviewed that make such payments to consumers—HVIP and CVRP. California State Auditor Report 2020-114 37 February 2021 Further, we have concerns about the frequency of the reviews and the evidence CARB maintains to support its conclusions. CARB CARB has not conducted any desk has not conducted any desk reviews for HVIP since 2014 and has reviews for HVIP since 2014 and conducted only two reviews for CVRP—in 2014 and 2020. The has conducted only two reviews for report from the 2020 CVRP desk review indicates that CARB staff CVRP—in 2014 and 2020. did not identify documentation issues, and CARB staff told us that the issues it identified in its 2014 HVIP review would not have been significant enough to affect emissions reporting for the program. Nonetheless, the documentation CARB maintained from the reviews is not sufficient for us to independently determine whether that is the case. Managers for HVIP and CVRP stated that they also work less formally with the program administrators to review supporting documentation on an ongoing basis. However, given the importance of the program data to CARB’s reporting and broader GHG objectives, more frequent and widespread formal reviews are needed across CARB’s incentive programs. A Regional GHG Reduction Program Is Failing, but CARB Has Not Collected Data to Determine Why As we discuss in the Introduction, CARB also provides statewide oversight for the Sustainable Communities program, which is intended to help the State reach its 2030 GHG goals by reducing vehicle travel and therefore GHG emissions. State law requires certain regional planning organizations (governed by local officials) to prepare strategies designed to achieve regional GHG emissions reduction targets. The strategies are aimed at reducing per capita GHG emissions from passenger vehicles, which they do largely by reducing vehicle travel. Pursuant to state law, CARB established specific GHG reduction targets for the 18 distinct regional planning organizations throughout California, which account for nearly all of the population in the State. State law also requires CARB to review the strategies submitted by the regional planning organizations. In approving the strategies, CARB must determine whether the strategies, if implemented, will achieve the GHG reduction targets it set. Although CARB has updated the Legislature on the progress of the Sustainable Communities program, that update did not contain key information needed to evaluate the program’s success. Beginning in 2017 the Legislature required CARB to assess the progress made by each regional planning organization toward meeting the GHG emissions reduction targets set by CARB and to report that information to the Legislature every four years. In its first and only report thus far, released in November 2018, CARB reported that California is not on track to achieve the program’s GHG reduction targets. However, CARB did not report the progress made by each regional planning organization in meeting the regional targets, 38 California State Auditor Report 2020-114 February 2021 as required by state law. CARB explained in the report that it was unable to find a data source to report that information accurately, and it instead reported changes in GHG emissions statewide. In doing so, CARB concluded that passenger vehicle emissions had not declined as significantly as expected and that vehicle travel had actually increased. Accordingly, CARB concluded that the regional planning organizations either were not implementing the strategies or were not achieving the expected results. However, because CARB did not identify GHG emissions changes by region, it could not determine which of these explanations was more responsible for the lack of emissions reductions or whether any regional strategies were actually working. CARB also could not determine which specific types of activities might be successful in reducing GHGs via reductions in vehicle travel. CARB’s inability to measure GHG CARB’s inability to measure GHG emissions by region creates emissions by region creates uncertainty about the future success of the Sustainable uncertainty about the future Communities program. When we asked CARB about its progress success of the Sustainable in finding a data source for regional GHG emissions, it indicated Communities program. that it had not found one and does not know whether it will have regional data in time for the 2022 report. As a result, we are concerned that CARB and the State may not be able to identify and pursue the best regional strategies for reducing emissions in time to help the State meet its 2030 GHG goals. Recommendations To improve its ability to isolate each of its incentive programs’ additional GHG reductions, by February 2022 CARB should establish a process to formally identify its incentive programs’ overlap with other programs that share the same objectives. As part of that process, CARB should document how it will account for the overlap to allow the most accurate program measurement possible. As part of its work to measure both incentive and regulatory programs’ additional GHG reductions, by February 2022 CARB should begin collecting and analyzing the data it needs to assess the extent to which the requirements in its regulatory programs are being exceeded by manufacturers. To the extent applicable, that analysis should focus on the components of the requirements that overlap with CARB’s incentive programs, such as the extent to which manufacturers are complying with regulations for heavy-duty vehicles via low- and zero-emission vehicles. To improve its ability to identify the effectiveness of each of its incentive programs in reducing GHG emissions, by August 2021 CARB should develop a process to define, collect, and evaluate data on the behavioral changes that result from each of its incentive California State Auditor Report 2020-114 39 February 2021 programs. Having done so, by February 2022 CARB should collect and analyze relevant survey information for all consumer-focused incentive programs, as well as information about the behavioral effects of programs that other entities offer, such as the federal tax credit. To better assist the State in achieving its GHG goals, CARB should use the information we describe to refine its GHG emissions estimates for its incentive programs in its annual reports to the Legislature, the funding plans approved by its board, and any longer-term planning documents or reports. To promote transparency and inform stakeholders, beginning in December 2021, CARB should prepare an annual report for its board and the Legislature on its progress in isolating the GHG emissions reductions attributable to each of its regulatory and incentive programs. As a part of this report, CARB should identify any measurement challenges that persist and highlight any administrative barriers that prevent it from obtaining the information it needs to perform better analysis. To strengthen the accuracy and integrity of its emissions reduction reporting, CARB should immediately begin retaining all supporting documentation it uses to perform calculations of GHG reductions for its cap-and-trade-funded incentive programs for a period of at least five years. In conjunction with this change, CARB should also document the justification for any instances in which the underlying data it uses to compile its annual reports vary from the information it publishes in those reports. To better ensure the accuracy of its program data, by August 2021 CARB should develop a formal schedule and procedures for reviewing the supporting documentation maintained by its program administrators. These procedures, which CARB should begin using with the 2022 annual report, should specify a minimum number of records to review in relation to the program’s size, should specify how staff will collect and maintain evidence to support conclusions, and should be standardized across all of CARB’s incentive programs. To ensure that the State is positioned to assess the status of the Sustainable Communities program, by April 2021 CARB should report to the Legislature whether it will have a usable source for measuring regional GHG emissions in time for the 2022 report. If CARB believes it may not, it should identify any administrative or bureaucratic barriers it faces in accessing data it needs for the estimates and request relevant action by the Legislature to make those data available. 40 California State Auditor Report 2020-114 February 2021 California State Auditor Report 2020-114 41 February 2021 Chapter 2 CARB HAS NOT ADEQUATELY DEMONSTRATED ITS PROGRAMS’ SOCIOECONOMIC BENEFITS Chapter Summary State law requires CARB to spend a portion of its cap-and-trade funds to provide benefits to disadvantaged communities as well as low-income communities. The history of these requirements demonstrates the Legislature’s desire to target cap-and-trade funding in a way that provides benefits for Californians most in need. However, although in recent years CARB has exceeded the minimum requirements in state law for program spending in disadvantaged and low-income communities, it has significant opportunities to better evaluate and demonstrate its programs’ effectiveness. CARB operates incentive programs that it intends to have specific, tangible socioeconomic benefits for participants. However, it does not consistently collect data to determine whether the programs actually provide those benefits. CARB has also missed opportunities to use data it has already collected to determine whether participants are receiving the intended benefits. Finally, CARB has been slow to measure the jobs its programs are required to support, and it has not done enough to measure the benefits of the job training activities that its own guidelines require. Given that programs designed to provide socioeconomic benefits may cost more than other CARB programs and do not focus primarily on GHG reductions, it is crucial that CARB effectively define and measure the socioeconomic benefits they provide. The Legislature Has Shown a Desire to Focus Benefits on Californians Most in Need Requirements in state law for cap-and-trade spending in disadvantaged and low-income communities have become more specific over time. As discussed in the Introduction, disadvantaged communities are classified based largely on their sensitivity to environmental pollution, while the low-income community classification is based solely on income statistics. In 2013 state law required CalEPA to identify disadvantaged communities in California. CalEPA performed this identification by zip code based on a range of factors that focused primarily on sensitivity to the physical environment, such as air pollution. At that time, state law 42 California State Auditor Report 2020-114 February 2021 required a minimum of 25 percent of all cap-and-trade funds to benefit these disadvantaged communities, with 10 percent to be spent on projects located within those communities. The original approach to identifying disadvantaged communities proved to be too broad. In 2014 CalEPA determined that identifying communities by zip code did not allow for enough precision and changed the identification of disadvantaged communities from zip codes to census tracts—smaller geographical areas determined by the U.S. Census Bureau with assistance from local entities and intended to allow statistical comparisons. This change allowed for a more detailed assessment of each community’s status because there are more than four times as many census tracts as there are zip codes in California, and census tracts are established with the goal of having roughly equivalent populations. Even with this change, some of the cap-and-trade funds counted toward the required minimum spending under the state law in effect at that time even if those funds benefited disadvantaged communities but were not spent on projects located within those communities. Under this approach to defining eligible spending, many cap-and-trade dollars were not spent within the disadvantaged communities most in need. For example, our review of two of CARB’s largest incentive programs found that in 2016 significant proportions of spending categorized as benefiting disadvantaged communities took place outside those communities’ borders. Specifically, program data indicate that HVIP spent nearly one-third of its $8.4 million in disadvantaged community spending in areas physically outside of disadvantaged communities; for CVRP it was more than 80 percent, or $34 million, of its total $42 million in reported disadvantaged community spending. Acknowledging the need to better target cap-and-trade In 2016 the Legislature amended spending, in 2016 the Legislature amended state law to focus this state law to focus cap‑and‑trade cap-and-trade spending primarily on projects located within the spending primarily on projects boundaries of, and benefiting individuals living in, disadvantaged located within the boundaries of, communities. The amended law significantly limited the amount and benefiting individuals living in, of cap-and-trade spending that could take place outside of disadvantaged communities. disadvantaged communities and still be considered as benefiting those communities. Effective January 2017, the law now requires that all cap-and-trade funds that count toward the disadvantaged community funding requirement be spent on projects located within the boundaries of the disadvantaged communities. Also effective in 2017, the Legislature established a cap-and-trade spending requirement for a second group of communities, referred to as low-income communities. Here too the Legislature encouraged a higher degree of precision in focusing the benefits of this cap-and-trade funding on Californians who need them most. Specifically, the amended law defines low-income communities as California State Auditor Report 2020-114 43 February 2021 census tracts where the median household income is below certain statewide thresholds, defined in part as 80 percent of the statewide median income. The law also requires 10 percent of cap-and-trade dollars to benefit these newly defined communities. When establishing the low-income community funding requirement, the Legislature again demonstrated a desire to ensure that the new benefits went to Californians most in need. Although defining these communities at the census-tract level was more precise than previous requirements for disadvantaged communities, the Legislature also acknowledged that some of the best GHG reduction strategies are those that benefit low-income households directly, regardless of where those households are located. Accordingly, the amendment, which took effect in 2017, also gives agencies that receive cap-and-trade funds the option to measure and report low-income community spending based on the actual households that the programs benefit, as opposed to reporting only at the census-tract level. In recent years, CARB has exceeded the minimum spending requirements for disadvantaged and low-income communities. In its 2020 annual report, CARB reported that the State has spent a cumulative 39 percent of its cap-and-trade funding in and benefiting disadvantaged communities and 21 percent of funding in and benefiting low-income communities since August 2017. Further, our review of selected programs found that CARB generally exceeded the separate minimum 25 percent and 10 percent requirements for each of those programs. For example, in 2019, program data show that CARB spent nearly 50 percent of its total cap-and-trade funding for HVIP in disadvantaged communities and more than 15 percent in low-income communities, both well over the required minimum. However, CARB’s reporting could be more precise. Specifically, for the consumer-based programs we reviewed, CARB reported spending in low-income communities at the census-tract level and not in terms of actual households. Reporting at the household level, where possible, would provide By reporting at the household more valuable information about programs that serve individuals level, where possible, CARB would and families. Further, as we discuss in the following sections, provide more valuable information spending in disadvantaged and low-income communities is about programs that serve generally the only information CARB measures or reports regarding individuals and families. the socioeconomic benefits of its programs, even though its programs have specific intended benefits that focus on individuals and households. Given the Legislature’s desire to focus and track the non-GHG benefits of cap-and-trade-funded programs, we believe CARB has significant opportunities to better evaluate and demonstrate the programs’ effectiveness, including by reporting at the household level. 44 California State Auditor Report 2020-114 February 2021 CARB Cannot Sufficiently Demonstrate That Its Programs Achieve Socioeconomic and Jobs Benefits As required by state law and its own guidelines, CARB operates certain transportation incentive programs to achieve socioeconomic benefits for participants. Although CARB uses cap-and-trade revenue to operate these programs, the programs place more emphasis on the socioeconomic benefits than on maximizing GHG reductions. As a result, the programs may cost significantly more than other incentive programs relative to the GHG reductions they achieve—underscoring the importance of ensuring that they actually provide the other intended benefits. Although CARB has identified However, although CARB has identified specific socioeconomic specific socioeconomic benefits it benefits it intends its programs to achieve, it does not consistently intends its programs to achieve, collect the data it needs to demonstrate those benefits, and it has it does not consistently collect been slow to collect required socioeconomic data, such as data the data it needs to demonstrate related to job creation. Further, CARB has missed opportunities those benefits. to use data it currently collects to ensure that participants are receiving the intended benefits. CARB Does Not Evaluate Programs to Determine Whether They Achieve Intended Socioeconomic Benefits CARB’s programs that are aimed primarily at producing socioeconomic benefits receive cap-and-trade funding but do not have GHG reduction as their primary goal. CARB categorizes the programs it develops for disadvantaged and low-income communities as equity programs. Equity programs that focus on socioeconomic benefits may cost significantly more than CARB programs that focus primarily on reducing GHGs relative to the GHG reductions they achieve because they offer higher incentive payments or require more intensive program administration to provide socioeconomic benefits to participants. For example, CARB’s data indicate that the cost to reduce one ton of GHG emissions through CARB’s Financing Assistance program is seven times the cost of doing so through its CVRP program. Some of the higher costs come either from the larger cash incentives that the programs provide per vehicle to the low- or middle-income Californians who participate or from the more intensive program administration required to achieve the additional socioeconomic benefits the programs are expected to provide. For example, grant agreements we reviewed for the Financing Assistance program allow 14 percent or more of the total grant funding to go towards personnel costs for the program administrators, who work closely with participants, while the CVRP grant agreement limits administrative costs—including but not limited to personnel costs—to 7 percent. Because equity programs are less California State Auditor Report 2020-114 45 February 2021 cost-effective in terms of emissions reductions, it is critical that Because equity programs are less CARB demonstrate each program’s value by clearly defining and cost‑effective in terms of emissions measuring its socioeconomic benefits. reductions, it is critical that CARB demonstrate each program’s value Although CARB has identified specific socioeconomic benefits that by clearly defining and measuring it intends for its equity programs to provide participants, it does its socioeconomic benefits. not consistently collect data to determine whether the programs actually provide those benefits. As we discuss above, CARB’s program administrators for its Financing Assistance program work closely with participants, providing training on financial literacy with required topics such as credit, bank services, and vehicle expenses to ensure consumer protection and increase the chance of successful loan repayment. For example, in its fiscal year 2017–18 funding plan, CARB stated that it intended for the Financing Assistance program to provide economic benefits to participants such as increased credit scores, the ability to qualify for housing loans, and access to more reliable transportation. However, the metrics CARB identified that it will use to measure progress do not allow CARB to actually measure these benefits. Instead, they include the number of consumers who participate, costs and types of vehicles purchased, and loan repayment rates, none of which allow CARB to determine whether the program achieves the intended benefits. Further, over the five years CARB has operated the Financing Assistance program, it has not collected data that would allow it to report on benefits achieved. For example, despite repeatedly citing participants’ loan repayment rates as a measure of the program’s success in its annual funding plans, CARB does not collect data about or report on these rates. CARB stated it does not have access to this information because the program administrators do not manage the participants’ loans, leading us to question why CARB identifies loan repayment as a metric to measure program success. We expect CARB to identify metrics for which it can collect the data and measure the benefits when designing and requesting funding for the program. CARB also does not collect data that allows it to measure the socioeconomic benefits of participation in its car-sharing pilot program. In its annual funding plans, CARB has described improvements in participants’ access to mobility as a metric with which to measure the program’s success. When we asked CARB how it measures this metric, the program manager explained that CARB expects to see participants taking trips that they would not have been able to without the program, such as traveling outside of their neighborhoods for medical appointments. However, CARB does not consistently collect this information from participants. Specifically, although the grant agreements require initial and follow-up participant surveys be conducted by the program administrators, they do not require program administrators to ask questions about changes in mobility. Accordingly, the surveys 46 California State Auditor Report 2020-114 February 2021 do not consistently collect information about such changes. The program manager also confirmed that CARB has not evaluated its car-sharing pilot program to determine such mobility. CARB’s failure to collect and measure relevant program data greatly limits its ability to inform the Legislature and other stakeholders about whether the programs are effective. As Figure 5 shows, despite the specific socioeconomic goals CARB identified for its programs, its public reporting on their benefits is generally limited to the broad spending requirements we discuss in the first section of this chapter. In May 2020, CARB entered into a contract with UC Berkeley to, among other things, evaluate the socioeconomic benefits of the car-sharing program, including increased access to transportation and increased economic opportunity. However, CARB does not expect a final report on the results from the evaluation until 2022, at which point the program will have been operating for five years without an appropriate evaluation of the benefits it provides to participants. CARB also does not consistently use the data that are available to demonstrate benefits from programs and determine whether the programs are achieving their goals. CARB’s Clean Cars 4 All program provides incentives for lower-income consumers who scrap their old vehicles and purchase new clean vehicles, ostensibly providing more reliable and less costly transportation. Consequently, CARB states that the intended benefits of its Clean Cars 4 All program include maximizing economic opportunities for participants, reducing their cost of driving, and minimizing work interruptions because of unreliable transportation. However, as Figure 6 shows, CARB has missed clear opportunities to assess whether the Clean Cars 4 All program actually provides these socioeconomic benefits. CARB administers the program through grant agreements with local air districts, which perform the day-to-day activities of the program. These grant agreements require the air districts to conduct surveys of program participants. When entering into multiple grant agreements with several air districts, which it did between 2015 and 2020, CARB provided the air districts with identical sets of example surveys. The example surveys include questions that indicate how CARB intended to measure the socioeconomic benefits it expects the program to provide, such as whether participants’ employment opportunities have changed as a result of the program and whether their income has changed. CARB could use the results from these survey questions to evaluate whether the program is providing the socioeconomic benefits it expects. However, although the grant agreements require the air districts to conduct surveys of participants, they do not require the air districts to ask specific questions or use the example surveys. Instead, the grant agreements allow the air districts to design their own survey questions. The California State Auditor Report 2020-114 47 February 2021 program manager explained that CARB does not have copies of the actual surveys the air districts use because it has not asked for them. As a result, CARB does not know what questions the surveys ask. Figure 5 CARB Establishes Specific Socioeconomic Goals but Only Reports on Broad Outcomes CARB’S SOCIOECONOMIC GOALS FINANCING ASSISTANCE CAR SHARING CLEAN CARS 4 ALL Improve participants’ Enable participants to economic status by Maximize economic opportunities for travel outside of their increasing credit scores and participants and reduce their communities in ways they enabling participants to transportation costs. previously could not. qualify for housing loans. METRICS CARB DOES NOT EVALUATE OR REPORT • Improvements in credit scores and access to additional • Increases in mobility. • Improvements in employment financing for housing. • Improvements in access to food, status and income. • Decreases in transportation health care, or other needs. • Increases in vehicle reliability and costs and more decreases in driving costs. reliable transportation. METRICS CARB REPORTS Dollars spent in disadvantaged and low-income communities. Source: CARB’s funding plans, 2020 annual report to the Legislature, and interviews with CARB staff. 48 California State Auditor Report 2020-114 February 2021 Figure 6 CARB Has Missed Opportunities to Measure Benefits for Its Clean Cars 4 All Program 1 CARB intended Clean Cars 4 All to achieve specific socioeconomic benefits when requesting funding for the program. “Clean Cars 4 All will maximize economic opportunities and reduce the cost of driving.” 3 CARB created detailed survey questions designed to measure these benefits, and over Cle2an Cars 4 All’s five years, it provided them repeatedly to its program administrators. “How much more or less is your income since participating in the program?” CARB required its Clean Cars 4 All program administrators to perform surveys—and still does. However, CARB has not required program administrators to ask the specific questions 4 CARB designed, and CARB does not know what questions they actually ask. ? ?? ? ? ? ? CARB told us that collecting all the ? ? survey information is not useful because of ? low response rates and because responses do not always reflect how the program works. However, CARB has not co5nducted any analysis of the survey responses. ? ?? ? ?? Source: CARB’s funding plans, grant agreements, and interviews with CARB staff. California State Auditor Report 2020-114 49 February 2021 CARB has also missed opportunities to use data it currently has to CARB has missed opportunities determine whether participants are receiving the intended benefits to use data it currently has to of its programs and to monitor the effectiveness of those programs. determine whether participants Although greater financial stability for participants is a shared goal are receiving the intended benefits of both the Financing Assistance program and Clean Cars 4 All, of its programs and to monitor the CARB does not analyze data it collects on program participants effectiveness of those programs. that could help it determine whether the programs are contributing to improved financial stability. For both programs, CARB collects data about participants’ personal financial circumstances, including details about their household income and vehicle loans—such as loan amounts and interest rates. CARB’s manager for Clean Cars 4 All told us that CARB uses the data in these reports to monitor the performance of the air districts, determine whether the districts are meeting statutory obligations in administering the program, and determine funding needs and annual statutory goals. However, that manager also confirmed that CARB has not made any changes to the program based on the data. Similarly, the manager for the Financing Assistance program stated that CARB uses the data in the quarterly reports from its external program administrators to ensure that the program is working but confirmed that CARB has not made any changes to the program based on the data contained in the reports. The manager explained that the programs each have a low volume of data, meaning that it is not useful for analyzing program effectiveness. Given that Financing Assistance is a small-scale pilot program, a low volume of data for the program is reasonable; however, the scale of the program does not prevent CARB from analyzing the available data and using that analysis to inform changes if necessary. The program data we reviewed show that interest rates for participants’ loans vary significantly, with some rates as low as 1 percent and others exceeding 25 percent. Given that helping consumers overcome the barrier of obtaining vehicle financing by providing low-interest loans is a primary purpose of the Financing Assistance program and that CARB intends for Clean Cars 4 All to maximize economic benefits for participants, we expected that CARB would be analyzing the loan information to determine whether the programs are operating as expected and in ways that are beneficial to participants. However, CARB has not formally done so for either program. The Financing Assistance manager stated that CARB expects participants to pay off their loans and avoid vehicle repossession. When we asked whether CARB has formally collected data on vehicle repossessions, CARB reviewed past reports from its two external program administrators and learned that there were five repossessions reported by one and zero by the other. However, CARB has not formally analyzed the information and is accordingly unable to demonstrate whether these numbers indicate that the program is succeeding. These numerous instances of uncollected and unused data prevent CARB 50 California State Auditor Report 2020-114 February 2021 from determining whether its programs are providing the specific intended socioeconomic benefits. As a result, it also has not been able to inform stakeholders, such as the Legislature, about the extent to which these programs are succeeding. CARB Has Been Slow to Measure Job Creation and Has Done Little to Evaluate Job Training CARB has also done relatively little to evaluate the effects of its programs on required job creation and training. As discussed in the Introduction, state law directs CARB, to the extent feasible, to use cap-and-trade funds in a way that fosters job creation by promoting GHG emissions reduction projects carried out by California workers and businesses. In addition, CARB’s own funding guidelines from 2015, the first year it published these guidelines for cap-and-trade-funded programs, direct it to design programs that result in jobs and job training whenever possible. CARB’s 2018 updated funding guidelines expanded on this requirement, specifying that programs’ job creation and training efforts should target disadvantaged and low-income communities and households. Both sets of guidelines include elements for measuring and reporting on the programs’ impact on jobs. The 2015 guidelines specify that CARB should track and report the number of job recipients and trainees funded by its programs, and the 2018 guidelines refined the required reporting to direct the external entities administering CARB’s programs to use a specific job calculator tool to estimate the number of jobs each program supports more broadly. Despite clear direction and a Despite clear direction and a stated requirement to estimate the stated requirement to estimate number of jobs its programs support, CARB has failed to do so the number of jobs its programs in a timely or comprehensive way. CARB began collecting data in support, CARB has failed to do so in 2019 for only some of its programs about the numbers of jobs they a timely or comprehensive way. fund and support, and it still does not do so for other programs. Specifically, of the nine programs we reviewed for which jobs reporting requirements apply, CARB had collected jobs data in its reporting database for only three at the time of our review. Further, although it published estimates for some of its incentive programs in its fiscal year 2019–20 funding plan, as of its most recent annual report to the Legislature in March 2020 CARB had not reported any jobs creation numbers for its individual programs. Instead, CARB reported the aggregate jobs numbers for all cap-and-trade programs across multiple agencies, numbers that our review of program data showed were incomplete. When we asked about its general lack of jobs reporting, CARB staff referenced the difficulty of establishing and implementing the measurement process. Staff also indicated that they have California State Auditor Report 2020-114 51 February 2021 been unable to require some program administrators to follow the jobs reporting requirement because CARB entered into multiyear agreements with those program administrators before developing the reporting requirement. We find these explanations unconvincing, given that CARB has had since 2015 to implement the funding guidelines and according to a literature review, the jobs calculation tool CARB requires its program administrators to use is based on a method that is relatively simple and widely used for such analyses. We also note that CARB could seek to amend its grant agreements or, failing that, use its own tool to determine jobs benefits rather than relying on its program administrators to do so. Finally, CARB explained that staff had collected jobs data for two of the programs we reviewed where those data were missing, but they had neglected to enter them into CARB’s reporting database. CARB also has requirements for providing job training benefits and reporting on them. As we discuss above, CARB’s funding guidelines require it to foster job training for its cap-and-trade-funded programs and to focus its training efforts on disadvantaged and low-income communities and households. The guidelines further specify that CARB should report information on employment outcomes from programs that provide job training, including information on the quantity and quality of the job trainings as well as the wages participants are paid and any credentials they receive as a result of the training activities. Although the jobs training requirements in CARB’s funding guidelines apply whenever possible, we understand that these types of benefits may be more relevant for some programs than for others. Specifically, it might be reasonable for a consumer-focused rebate program like CVRP not to have an explicit job training component. In contrast, pilot programs focused on industry and demonstrating advanced clean technologies have a natural overlap with job training. For example, as we discuss in Chapter 1, CARB operates a bus pilot program that helps transit agencies and other entities acquire zero-emission buses and associated infrastructure. The program is focused in disadvantaged communities, and its goal is to place large numbers of zero-emission buses in service while also providing opportunities for on-the-job training for bus operators and maintenance workers to learn the new technology. Among the 10 incentive programs we reviewed, CARB includes a job training element for only four programs. The fact that job training benefits may be relevant to only certain programs underscores the importance of tracking those benefits when they are provided. CARB does not always require its program administrators to When CARB does include job training as a program benefit, it does provide detailed information not always require its program administrators to provide detailed about that training or how it information about that training or how it benefits participants. benefit participants 52 California State Auditor Report 2020-114 February 2021 For example, CARB operates a freight program that provides grant funding to demonstrate advanced emissions-reducing technologies for moving freight within and through California and to encourage industry’s development of those technologies. CARB’s grant agreements with the program administrators require them to provide job training. The grant agreements include training materials and completed training rosters as required deliverables but do not require the program administrators to provide any information to CARB about the outcomes of the trainings for participants, the quality of trainings, or whether participants come from the communities CARB is trying to serve. For the two pilots we reviewed under this program, one of the program administrators reported to CARB that trainings were conducted and provided CARB with the training manual as evidence that the trainings occurred. However, the program administrator provided no further information, such as wages paid to program participants, the training credentials they received, or where they live. At the time of our reporting, the other program administrator reported it had not yet provided any training as part of the program. Similarly, one of CARB’s pilot programs provides grant funding to enable transit agencies and other entities to acquire advanced zero-emission buses and associated infrastructure. We reviewed three of the five bus pilots CARB operates under its bus pilot program and found that the grant agreements for these three pilots include general job training requirements and require the external program administrators to report to CARB the date the training was completed. However, similar to the freight program, Grant agreements for the three the grant agreements for the three bus pilots do not include specific bus pilots do not include specific requirements for reporting on the quality of trainings, how the requirements for reporting on training benefits the participants, or where the participants live. the quality of trainings, how the training benefits the participants, or Notably, we identified one pilot program that has more detailed where the participants live. job training and reporting requirements, but those requirements apparently were included because additional funding was provided. Specifically, CARB and the California Energy Commission (energy commission) are planning to jointly operate a zero-emission truck pilot program that includes funding from both entities. The solicitation documents for the program indicate that the program administrators will be responsible for developing and implementing a workforce training and development plan as part of the program, as well as for including specific data about the training and those who participate. However, the funds supporting these job trainings and detailed reporting requirements are from the energy commission, not CARB’s cap-and-trade funding. In fact, when we asked CARB about the job training for this truck pilot program, it indicated that the program includes these requirements because the energy commission had specific funds to invest in workforce development. Although the energy commission does allocate funds California State Auditor Report 2020-114 53 February 2021 for job training, CARB’s own funding guidelines have, since at least 2015, directed it to include job training as a component of funded projects wherever possible. Therefore, we question why CARB has not included similarly detailed requirements for its own programs to ensure that they meet its own funding guidelines and provide these important benefits. Recommendations To ensure that it communicates clearly to the Legislature about the extent to which programs benefit low-income households as the Legislature intended, by March 2022 CARB should begin reporting its spending in low-income communities at the household level wherever possible in its annual report to the Legislature. To better define incentive programs’ impact beyond GHG emissions reductions, by August 2021 CARB should review its incentive programs to ensure that it has clearly designated which programs focus primarily on socioeconomic benefits. As a result of this process, by February 2022 it should ensure that it includes the benefits expected for each program in its funding plan or other public documents, such as its annual report and individual grant agreements. To better demonstrate the socioeconomic benefits that its incentive programs achieve, by February 2022 CARB should do the following: • Identify clear and measurable metrics it will use to assess each of the socioeconomic benefits it intends its programs to achieve. • Develop a process to collect data, or use existing data, to measure and report on each metric. • In its funding plans and annual reports, CARB should report to the Legislature and its board on the metrics. To provide transparency to the Legislature and other stakeholders, beginning in 2022 and using the metrics and data described above, CARB should make funding and design recommendations in its funding plans and annual reports based on which programs are effective in producing socioeconomic benefits and at what cost. To ensure that the State has reliable information about the extent to which cap-and-trade-funded programs create and support jobs, by August 2021 CARB should begin collecting data on the jobs produced by each of its incentive programs. Where needed, CARB should pursue amendments to its agreements with its program 54 California State Auditor Report 2020-114 February 2021 administrators to make reporting this information mandatory. CARB should include an analysis of these jobs data in its annual reports to the Legislature beginning in 2022. To ensure that its incentive programs promote effective and equitable job training, by August 2021 CARB should develop a process to assess which programs should include a job training element. For those programs it identifies, by February 2022 CARB should direct its staff or its external program administrators to collect and report on the quality of job trainings and outcomes experienced by participants, including who received training, the credentials participants received as a result, any actual or expected wages they received as a result of participating in the training or for developing the relevant expertise, and the number of participants from disadvantaged communities or low-income communities and households. California State Auditor Report 2020-114 55 February 2021 Other Areas We Reviewed To address the audit objectives approved by the Joint Legislative Audit Committee (Audit Committee), we also reviewed the subject areas in Table 2. These areas include the risk that regulatory programs may shift GHG emissions outside of California (as opposed to reducing them overall), CARB’s economic analyses for its proposed regulatory programs, and CARB’s tracking of the costs it incurs to administer its transportation programs. Table 2 indicates the results of our review and presents an associated recommendation that we have not already discussed in other sections of this report. Table 2 Other Areas We Reviewed as Part of This Audit Transportation Programs Generally Do Not Risk Shifting Emissions Elsewhere The Audit Committee directed us to examine whether the reduction of GHG emissions resulting from a selection of CARB’s transportation programs were the result of shifting emissions rather than eliminating them. Such shifting may occur if regulatory requirements cause economic activity to move out of state. In such a scenario, there may appear to be a reduction in emissions, but the emissions have just moved elsewhere. We engaged our consultant to help us determine the extent to which emissions reductions from CARB’s programs may result from shifting emissions elsewhere rather than eliminating them. Our consultant concluded that the transportation-related industries regulated by the programs we reviewed are not likely to shift emissions materially. For example, the regulatory programs generally focus on vehicles sold or operating in California, and individuals and businesses cannot easily shift their use of transportation out of state. As such, even when California’s vehicle requirements are more stringent than those in other states, manufacturers cannot avoid the restrictions by making the vehicles elsewhere. Further, for sectors that could experience a shift in activities from California to another location depending on the magnitude of regulatory costs—such as services provided by ports—our consultant identified that CARB has acknowledged these regulatory costs and has adopted certain programs to address them, which may limit the potential shifting of emissions. CARB Reasonably Projects the Economic Impacts of Its Regulatory Programs, but It Could Do More to Explain the Effects on Consumers The Audit Committee directed us to identify the cost-effectiveness of the transportation programs we selected for review. Our consultant evaluated the economic analyses that CARB conducted when it proposed the eight regulatory programs we reviewed. Our consultant concluded that CARB reasonably approached its assessment of the compliance costs and resulting cost-effectiveness of all eight of these regulatory programs reviewed. However, our consultant also found that CARB may understate costs in some circumstances and that it could expand its consideration of the costs and benefits to consumers. Specifically, CARB often concluded that the regulatory requirements in its programs, such as requiring the sale of ZEVs, represent net savings to consumers. In one specific analysis, CARB concluded that the long-term cost savings for some vehicles were greater than the additional up-front cost of those vehicles. However, given its conclusion that these vehicles save consumers money, CARB could do more to consider all costs and benefits to purchasing a clean vehicle—such as driving range—and if, in fact, such vehicles are more desirable to consumers, explain why it is necessary to require sales of clean vehicles through its regulatory programs. CARB has not fully analyzed these issues and their implications for the regulation’s rationale and design. We recommend in Chapter 1 of this report that CARB develop a process to define, collect, and evaluate information on what factors influence consumers’ purchasing behavior as a key part of measuring the effectiveness of CARB’s incentive programs. Doing so would also allow CARB to better speak to the necessity of its regulatory programs and the effectiveness of particular designs it proposes. 56 California State Auditor Report 2020-114 February 2021 Tracking Administrative Costs Would Allow CARB to Better Assess Its Programs’ Cost‑Effectiveness The Audit Committee directed us to assess, in part, how CARB considers the effects of a proposed program on its administrative costs. For the eight regulatory programs we reviewed, CARB identified its expected administrative costs for the programs when proposing them. However, CARB did not consistently identify its own administrative costs when proposing funding for the 10 incentive programs we reviewed. Some managers explained that they do not estimate administrative costs because they consider overseeing incentive programs to be a part of CARB’s everyday responsibilities or because CARB did not know how much it would cost to administer the programs. However, it is clear that the addition of new programs for CARB to oversee represents new costs; therefore, CARB should include those costs in its estimates of the expenses to operate the programs once implemented. Our review determined that CARB also does not track actual administrative costs for its individual incentive programs. As a result, although CARB has spent approximately $120 million in cap-and-trade funds to administer its programs since 2013, it could not tell us what its administrative costs have been for individual programs. CARB explained that staff may work on more than one program at a time and indicated that it does not yet have a means of tracking staff time or other costs across all programs. However, because these costs are a potentially significant component of the total costs required to operate a given program, CARB should take all reasonable steps to measure them. Moreover, it is a basic and often required task for state agencies to track their costs related to specific programs and activities, and we see no reason why CARB would be unable to do so. Recommendation To ensure that it can account for the total costs of its transportation programs, beginning with fiscal year 2021–22 CARB should develop and implement processes to track the administrative costs it incurs to operate each of its transportation programs. After doing so, it should begin including those costs as part of the cost-effectiveness measurements in its annual reports to the Legislature. We conducted this performance audit in accordance with generally accepted government auditing standards and under the authority vested in the California State Auditor by Government Code 8543 et seq. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on the audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Respectfully submitted, ELAINE M. HOWLE, CPA California State Auditor February 23, 2021 California State Auditor Report 2020-114 57 February 2021 Appendix A CARB’S TRANSPORTATION GHG EMISSIONS REDUCTION PROGRAMS The Audit Committee directed the California State Auditor’s Office (State Auditor) to identify the number and type of CARB’s GHG emissions reduction programs in the transportation sector. This Appendix presents a list of those programs. The green highlights in Table A indicate the programs we reviewed as a part of the audit, which we selected in order to ensure a review of programs affecting a variety of vehicle types and transportation objectives. Table A CARB’s Transportation GHG Emissions Reduction Programs PROGRAM NAME TYPE DESCRIPTION 1 AB 617 Community Incentive Creates incentive for vehicle owners to replace older, high-polluting vehicles Air Protection Program and equipment with newer models that have low or zero emissions. Funds can also support local strategies that air districts and communities identify. 2 Advanced Clean Trucks Regulation Future Regulation Will require manufacturers to make a percentage of their truck and bus sales zero-emission and will require certain entities to report once about their use of contracted services that require trucks or shuttles. 3 Advanced Technology Incentive Encourages the development of advanced technologies to accelerate their Freight Demonstrations market introduction. 4 Agricultural Worker Vanpools Incentive Expands access to clean transportation vanpools for agricultural workers and achieves emissions reduction benefits by providing incentives for cleaner vehicles instead of conventional vehicles. 5 Alternative Diesel Fuels Regulation Regulation Intends to create a framework for low carbon diesel fuel substitutes, such as biodiesel, to enter the commercial market in California while mitigating potential environmental or public health impacts. 6 California Climate Investments Oversight State law requires CARB to develop guidance on reporting and quantification methods for all state agencies that receive cap-and-trade moneys to ensure that requirements in state law are met. Programs administered with these moneys are referred to as California Climate Investments programs. 7 Cap-and-Trade Regulation (fuels) Regulation Establishes a declining cap on emissions from facilities accounting for about 80 percent of the State’s GHG emissions and currently covers about 450 facilities, of which more than 50 are fuel suppliers. Such facilities must either acquire credits to offset their emissions or invest in emissions reduction technology to reduce emissions. 8 Car-sharing pilots Incentive Helps government entities or nonprofits start or expand car-share programs for residents of disadvantaged communities, using low- and zero-emission vehicles, vanpools, and other mobility options. 9 Carl Moyer Memorial Air Quality Incentive Grants funding to private companies and public agencies to purchase cleaner Standards Attainment Program than required engines, equipment, and emissions reduction technologies. Among other things, the program can fund trucks, emergency vehicles, and farm and cargo-handling equipment. 10 Clean Miles Standard and Future Regulation Intends to reduce GHG emissions from transportation network company Incentive Program services. State law required CARB to establish a baseline for emissions of GHGs for vehicles used on the online-enabled applications or platforms by transportation network companies on a per-passenger-mile basis. 11 Clean Mobility in Schools Incentive Funds the electrification of transportation fleets, including vehicles and infrastructure, plus education and awareness to encourage clean mobility in and around schools located in disadvantaged communities. continued on next page . . . 58 California State Auditor Report 2020-114 February 2021 PROGRAM NAME TYPE DESCRIPTION 12 Clean Off-Road Equipment Voucher Incentive Gives vouchers for certain zero-emission freight equipment. Incentive Project (CORE) 13 Clean Vehicle Rebate Project (CVRP) Incentive Supports funding for vehicle rebates on a first-come, first-served basis for light-duty ZEVs. Rebate amounts are increased for lower-income applicants. 14 Drayage Truck Regulation Regulation Requires that all trucks servicing ports and railyards be equipped with 2007 or newer model year engines. 15 Electric Vehicle Supply Regulation Intends to increase drivers’ ease of access to charging infrastructure. Equipment Standards According to CARB, creates a minimum standard of access for public electric vehicle charging and creates a more complete database of location and pricing information for consumer use, among other provisions. 16 Electrify America investments Oversight CARB approves the investment plans for how Electrify America, a subsidiary of Volkswagen, invests funding in California to address the adverse impacts to California’s ZEV program resulting from the sale of Volkswagen vehicles equipped with emissions defeat devices to consumers who believed they were purchasing clean vehicles. 17 Clean Cars 4 All Incentive Creates incentives for lower-income consumers living in and near disadvantaged communities who retire their old vehicles and purchase new or used hybrid, plug-in hybrid, or ZEV replacement vehicles. 18 Financing Assistance for Incentive Supports low interest loans and vehicle price buy-downs to help Lower-Income Consumers lower-income Californians overcome the barrier of obtaining vehicle financing for clean vehicles. 19 Funding Agricultural Replacement Incentive Awards funds to farmers and agricultural businesses for newer, cleaner equipment. Measures for Emissions Reduction (FARMER) 20 Goods Movement Emission Incentive Funds the retrofit, purchase of engines, or replacement of vehicles— Reduction Program (Proposition 1B) including trucks, locomotives, harbor vehicles, transport refrigeration units, cargo handling equipment, and supporting infrastructure. 21 Greenhouse Gas Regulations for Regulation Establishes GHG standards for new medium- and heavy-duty engines and Medium- and Heavy-Duty Engines vehicles sold in California, beginning with model year 2014. California’s and Vehicles (Phase 1) Phase 1 regulation harmonizes with the federal phase 1 standards. 22 Greenhouse Gas Regulations for Regulation Expands the scope and stringency of the GHG regulations established in the Medium- and Heavy-Duty Engines Phase 1 regulations and are applicable starting with the 2021 model year for and Vehicles (Phase 2) engines and vehicles. 23 Greenhouse Gas Vehicle Regulations Regulation Establishes GHG standards beginning with 2009 model year light-duty (“Pavley” regulations) vehicles and medium-duty passenger vehicles and added various GHGs to the emissions that the State was regulating. 24 Hybrid and Zero-Emission Truck and Incentive Funds vouchers to reduce the cost of hybrid and zero-emission trucks and Bus Voucher Incentive Project (HVIP) buses at the time of purchase. 25 Innovative Clean Transit Regulation Regulation Requires transit agencies to purchase increasing percentages of new zero-emission buses, with 100 percent of new buses purchased being zero-emission by 2029. 26 Low Carbon Fuel Standard (LCFS) Regulation Sets carbon intensity standards for transportation fuels in California. LCFS uses a system of tradeable credits to determine compliance. Fuel producers must comply with the regulation either by producing or buying lower carbon-intensity fuels, lowering their fuels’ carbon intensity, or buying credits from other producers. 27 Low-Emission Vehicle Program III Regulation Establishes GHG standards for multiple kinds of emissions for 2017 (LEV III) Greenhouse Gas Regulations and subsequent model year light-duty vehicles and medium-duty passenger vehicles. 28 Lower-Emission School Bus Program Incentive Funds the purchase of new buses to replace old, high-emission public school buses and to equip in-use diesel school buses with retrofit devices that significantly reduce certain toxic emissions. California State Auditor Report 2020-114 59 February 2021 PROGRAM NAME TYPE DESCRIPTION 29 Ocean-Going Vessels at Regulation Requires fleets to reduce their auxiliary engine power usage or equivalent Berth Regulation emissions when at berth at regulated ports within California. 30 Optional Low-NOx Standards for Regulation Establishes optional low-NOx emission standards for heavy-duty engines. Heavy-Duty Engines Manufacturers can certify engines to these standards or to an existing mandatory standard. 31 Outreach, Community Incentive Provides funding and technical assistance to community-based organizations Transportation Needs Assessments, for outreach on other sources of funding, conducting community Technical Assistance, and the transportation needs assessments, strengthening partnerships, developing One-Stop-Shop projects, and applying for CARB’s clean transportation incentive projects. 32 Rural School Bus Project Incentive Funds zero-emission school buses and new school buses that use renewable fuels. It prioritizes older school buses with higher mileage in small- and medium-sized air districts. 33 Sustainable Communities Land Use and Establishes regional GHG emission targets for each metropolitan planning Transportation organization (MPO) in the State. The MPOs are required to develop a Planning Sustainable Community Strategy that shows how the region could meet CARB’s GHG emissions reduction targets. 34 Sustainable Transportation Equity Incentive Grants funding to support planning and capacity-building efforts in Project (STEP) communities in order to prepare those communities to implement clean transportation and land-use projects. STEP additionally provides grants intended to increase community residents’ access to and use of their mobility system so they can get where they need to go without the use of a personal vehicle. 35 Tire Inflation Regulation Regulation Requires automotive service providers to perform a tire pressure service on all passenger cars, light-duty trucks, medium-duty vehicles, and light heavy-duty vehicles while performing any vehicle maintenance or repair service. 36 Tractor-Trailer Greenhouse Regulation Requires certain types and sizes of trailers and their tractors used for Gas Regulation long-distance goods movement to have aerodynamic equipment and low rolling resistance tires when traveling in California. 37 Truck and Bus Regulation Regulation Requires emissions control equipment retrofit for certain trucks and buses and accelerates the turnover of older trucks and buses. 38 Volkswagen Environmental Oversight CARB is the lead agency implementing California’s allocation of this trust, Mitigation Trust which includes developing a plan that describes the State’s goals for the use of the funds, the categories of what can be funded, and the percentages of funds to be allocated to the categories. 39 Zero- and Near Zero-Emission Incentive Funds a variety of heavy-duty vehicles, off-road equipment, and fueling Freight Facility Project infrastructure, as well as other facility and efficiency improvements to reduce emissions facilitywide. Freight facilities include warehouses, distribution centers, seaports, and freight airports, among others. 40 Zero-Emission Airport Regulation Requires airport shuttle operators to transition to 100 percent zero-emission Shuttle Regulation vehicle technology by 2035. 41 Zero-Emission Drayage Truck Pilot Incentive Funds a large-scale deployment of zero-emission trucks that can operate in drayage or regional haul service. 42 Zero-Emission Powertrain Regulation Establishes an optional certification pathway that manufacturers can use to Certification Regulation certify their heavy-duty electric and fuel-cell vehicles and is intended to help reduce variability in the quality and reliability of heavy-duty zero-emission technology, among other things. 43 Zero-Emission Truck and Bus Incentive Funds competitively awarded projects intended to complement HVIP by Pilot Projects supporting larger-scale deployments of zero-emission vehicles. 44 Zero-Emission Vehicle Regulation Requires auto manufacturers to produce a certain amount of ZEVs each Regulation (ZEV) year to meet credit requirements established by the regulation. The manufacturer’s credit requirement is a percentage of their total annual passenger car and light-duty truck sales in the State. Source: CARB’s management, regulatory documents, funding plans, reports, and documents provided to the Legislature and state law. Note: CARB has identified that some programs have a primary goal of reducing GHGs while others reduce GHG emissions as a “co-benefit” to other emissions reductions. 60 California State Auditor Report 2020-114 February 2021 California State Auditor Report 2020-114 61 February 2021 Appendix B Scope and Methodology The Audit Committee directed the State Auditor to conduct an audit of CARB to determine the effectiveness of its oversight of GHG reduction programs related to transportation. Table B lists the objectives that the Audit Committee approved and the methods we used to address them. Table B Audit Objectives and the Methods Used to Address Them AUDIT OBJECTIVE METHOD 1 Review and evaluate the laws, rules, and regulations significant to Reviewed relevant state laws and regulations related to GHG reduction and the audit. transportation programs. 2 Identify the number and type of CARB’s GHG emissions reduction Reviewed a list of programs CARB has created; reviewed program programs in the transportation sector. descriptions in CARB’s annual funding plans and its website; interviewed CARB staff. 3 Determine the extent to which CARB’s programs overlap and how • Identified transportation-related objectives in CARB’s 2017 scoping they may interact with one another. plan and selected 19 programs for review based on vehicle type and transportation sector objective, program type, and potential for overlap. • For the selected groups of programs that work toward shared objectives in CARB’s 2017 scoping plan, we reviewed CARB’s documentation justifying the development of each program and interviewed CARB officials to determine the extent to which CARB identified and addressed overlap among the programs. • Our consultant assessed the programs’ design and CARB’s measurement of the programs’ benefits to assess whether CARB accounted for any overlap. 4 For a selection of CARB’s GHG emissions reduction programs in the transportation sector, identify the following: a. Whether CARB assesses the effects of the programs on • Reviewed CARB’s program data on spending in disadvantaged and communities and households after it has implemented those low-income communities to ensure that spending was appropriately programs. Assess, to the extent possible, how each program categorized. has affected households and communities, including those • Reviewed CARB’s underlying program data and information in CARB’s of different economic status, ethnicities, and locations funding plans and annual reports, and interviewed CARB staff to throughout California. identify the specific socioeconomic benefits it intends its programs to provide to participants and their households, including financial and employment benefits. Determined the extent to which CARB measures and reports on those benefits. continued on next page . . . 62 California State Auditor Report 2020-114 February 2021 AUDIT OBJECTIVE METHOD b. The annual GHG emission reductions that result from each • Assessed the extent to which CARB reports on the GHG emissions program and whether those reductions result from moving reductions that result from its transportation programs. emissions rather than eliminating them. • Reviewed the underlying data and assessed CARB’s processes for collecting and compiling those data to assess the completeness of CARB’s reporting on emissions reductions. • With our consultant’s assistance, assessed the reasonableness of CARB’s methods for projecting emissions reductions and for estimating actual reductions from each program. Reviewed emissions estimates for its most recent annual report to determine whether CARB calculated those data accurately and consistently. • As part of our review of CARB’s regulatory programs, our consultant assessed the risk that projected emissions reductions may be achieved by merely shifting those emissions elsewhere. c. The programs’ cost-effectiveness, including social benefits and • Reviewed and verified the total program expenditures in CARB’s annual costs, and compare them to other GHG emissions reduction reports for a selection of programs. Compared the cost information to programs, such as the cap-and-trade program. the GHG reduction information we reviewed in Objective 4b. • Assessed select incentive programs’ relative costs in the context of our review of socioeconomic benefits under Objective 4a. d. To the extent possible, the number of program participants who For the 10 incentive programs we reviewed, interviewed CARB staff and may have changed their behavior without the program and assessed their data collection mechanisms, such as surveys, for each whether CARB’s strategies to reduce the occurrence of this issue program to determine if they collect information on behavioral changes have been successful. and, if so, how they have used the information. e. To the extent possible, whether the programs’ activities have For incentive programs we reviewed, particularly pilot programs targeted at contributed to the development of a diverse and equitable specific industries, evaluated CARB’s approach to setting job creation goals workforce in the affected industries. and job training goals, and evaluated its data collection and reporting. 5 Assess the process CARB uses to create initial statements of reasons for proposing new GHG reduction programs in the transportation sector, including the following: a. How it considers effects of a program on air quality, the • For the eight regulatory programs we reviewed, assessed CARB’s initial environment, administrative costs, the overall economy, and statements of reasons—the documentation that CARB is required low-income and disadvantaged communities. to publish when proposing regulatory programs that set forth the rationale for CARB’s determinations that the adopted, amended, or repealed regulations are reasonably necessary and that are prepared in a manner consistent with the environmental purposes of CARB’s regulatory activity—as well as selected regulations. • Assessed the contents of this documentation against selected key criteria in the Administrative Procedures Act (APA) governing the regulation development process. The programs we reviewed complied with the selected key requirements in law. • Reviewed the initial statements of reasons and supporting documentation, and interviewed CARB staff, to determine the extent to which CARB considered the effects of the regulations on air quality, the environment, administrative costs, the overall economy, and low-income and disadvantaged communities. • For the 10 incentive programs we reviewed that CARB did not establish through regulation, reviewed documentation related to the design, implementation, and funding of the programs, and interviewed CARB staff to determine the extent to which CARB considered the effects of the programs on air quality, the environment, administrative costs, the overall economy, and low-income and disadvantaged communities. • As described under Objective 4b, our consultant also reviewed CARB’s projected emissions reduction methodologies for regulatory and incentive programs. California State Auditor Report 2020-114 63 February 2021 AUDIT OBJECTIVE METHOD b. How it considers potential interactions and effects with existing • Reviewed selected regulations, the initial statements of reasons for policies and programs. the eight regulatory programs and relevant documentation for the 10 incentive programs we reviewed, and interviewed CARB staff to assess whether and how CARB considers interactions with other programs when proposing new programs. • Our consultant reviewed whether CARB’s program design accounts for potential interactions as a part of the work under Objective 3. 6 Assess whether changes to GHG emissions reduction programs in Assessed whether changes to state law or regulations were required to the transportation sector should be addressed through changes to address issues we identified as a part of our work on the other objectives. state law or regulation. Concluded that without improved program measurement and reporting, specific changes to existing programs in law or regulations would be premature. 7 Review and assess any other issues that are significant to the audit. We did not identify any additional issues that are significant to the audit. Source: Analysis of Audit Committee’s audit request number 2020-114, state law, and information and documentation identified in the column titled Method. Assessment of Data Reliability In performing this audit, we relied on electronic data obtained from CARB’s California Climate Investments Reporting and Tracking System (CCIRTS) database, which CARB populates with data from its program administrators. The U.S. Government Accountability Office, whose standards we are statutorily required to follow, requires us to assess the sufficiency and appropriateness of the computer-processed information we use to support our findings, conclusions, and recommendations. To perform this assessment, we compared a selection of data from CARB’s program administrators for the period from December 2015 through May 2020 to CARB’s annual report to the Legislature. We identified discrepancies in the data for two of the five programs that could affect the precision of CARB’s reports to the Legislature. We also reviewed CARB’s efforts to ensure that the detail in the program data is accurate, which includes formal reviews of how a program administrator carries out its duties under its cap-and-trade grant agreement. However, we found that CARB has only completed formal reviews for two of the five programs we selected for our review. Further, CARB has not formally reviewed one of those two programs since 2014, and it has retained very limited documentation from its two reviews of the other. Although our findings raise questions about the reliability of some of the data CARB reports to the Legislature, we disclose these issues in our report and make recommendations to CARB to address them. Furthermore, our overall conclusion about CARB’s program measurement is that CARB should make improvements that go beyond the validity of the program data itself. For these reasons, although the issues we identified above may affect the precision of some of the numbers we report, there is sufficient evidence in total to support our findings, conclusions, and recommendations. 64 California State Auditor Report 2020-114 February 2021 California State Auditor Report 2020-114 65 February 2021 February 3, 2021 Elaine M. Howle, CPA * California State Auditor 621 Capitol Mall Suite 1200 Sacramento, California 95814 [submitted via electronics link as directed] Dear Ms. Howle, Thank you for the opportunity to review the draft Report 2020-114 California Air Resources Board: Improved Program Measurement Would Help California Work More Strategically to Meet Its Climate Change Goals. The California Air Resources Board (CARB) appreciates the time that your staff has spent with us over the last year to develop the recommendations. CARB has already started implementing a number of steps to address these recommendations and will be taking future steps as described in the attachment consistent with direction from the Legislature. In the case of some recommendations, it will take time to work through the public process to determine the most efficient and effective mechanisms to collect and analyze the additional data and information that you are recommending, and we are committed to doing so. I also want to note that implementing a number of the recommendations will likely come with an assessment that additional staffing and resources will be needed to fulfill CARB’s ability to deliver them. You will find attached a more detailed response on how CARB will address the recommendations included in the report. CARB looks forward to working with your team to track our progress implementing these recommendations and to share that information with the public and the Legislature. If you have any questions, please contact me at (916) 322-7077. Sincerely, Richard W. Corey, Executive Officer Attachment arb.ca.gov 1001 I Street • P.O. Box 2815 • Sacramento, California 95812 (800) 242-4450 * California State Auditor’s comments begin on page 73. 66 California State Auditor Report 2020-114 February 2021 Attachment California Air Resources Board (CARB) Responses to California State Auditor Recommendations in Draft Report California Air Resources Board: Improved Program Measurement Would Help California Work More Strategically to Meet Its Climate Change Goals Recommendations from Chapter 1 RECOMMENDATION: To improve its ability to isolate each of its incentive programs’ additional GHG reductions, by February 2022 CARB should establish a process to formally identify its incentive programs’ overlap with other programs that share the same objectives. As part of that process, CARB should document how it will attempt to account for the overlap to allow the most accurate program measurement possible. CARB RESPONSE: CARB will undertake a process to better document the interaction between incentive and regulatory programs and how to refine methods to better 1 account for emission reductions from incentives going forward. CARB will evaluate seeking funding to commission a study to determine where refinements may be needed to our quantification methodologies. We would like to clarify that, for purposes of tracking progress in meeting health 2 protective federally mandated clean air laws, tools such as EMFAC (which CARB uses to assess emissions levels from specific mobile sources) account for the complementary nature of policies that may impact those sources and avoid overestimating emissions benefits. RECOMMENDATION: As part of its work to measure both incentive and regulatory programs’ additional GHG reductions, by February 2022 CARB should begin collecting and analyzing the data it needs to assess the extent to which the requirements in its regulatory programs are being exceeded by manufacturers. To the extent applicable, that analysis should focus on the components of the requirements that overlap with CARB’s incentive programs, such as the extent to which manufacturers comply with regulations for heavy-duty vehicles via low- and zero-emission vehicles. CARB RESPONSE: CARB will include the extent to which regulated parties are exceeding regulatory requirements as we consider how to implement the preceding recommendation. RECOMMENDATION: To improve its ability to identify the effectiveness of each of its incentive programs in reducing GHG emissions, by August 2021 CARB should develop a process to define, collect, and evaluate data on the behavioral changes that result from each of its incentive programs. Having done so, by February 2022 CARB should collect and analyze relevant survey information for all consumer-focused incentive programs, as well as 1 California State Auditor Report 2020-114 67 February 2021 information about the behavioral effects of programs that other entities offer, such as the federal tax credit. CARB RESPONSE: CARB has work underway that will help address this recommendation. CARB will also seek public input regarding additional options for conducting this work as a part of the FY 2021-22 Funding Plan for its Low Carbon Transportation incentives including seeking public input around which incentive programs are most suited for additional analysis. CARB currently has work underway that will help with this effort through surveys of participants in projects such as the Clean Vehicle Rebate Project 3 (CVRP), Clean Cars 4 All, Car Sharing, Financing Assistance, and Hybrid and Zero- Emission Truck and Bus Voucher Incentive Project (HVIP) among others. CARB has also contracted with UC Berkeley to develop an evaluation model/process for CARB to use as a new standard for assessing the effectiveness, sustainability and outcomes of CARB’s clean mobility equity pilot projects for disadvantaged communities and low-income communities. Researchers will identify both community-preferred and 4 research-preferred metrics and evaluation methodologies that can be consistently applied across CARB’s clean transportation equity projects. The study will also result in policy recommendations on successful project elements to inform future transportation equity funding. Results of the UC Berkeley study are expected starting in May 2022. RECOMMENDATION: To better assist the State in achieving its GHG goals, CARB should use the information we describe above to refine its GHG emissions estimates for its incentive programs in its annual reports to the Legislature, the funding plans approved by its board, and any longer-term planning documents or reports. CARB RESPONSE: As new data become available through CARB’s implementation of the recommendations in this report, we will use those data to update our GHG quantification methodologies. CARB routinely evaluates and updates the quantification methodologies 5 used to calculate the GHG emission reductions from its incentive programs as new information becomes available, and we will incorporate relevant information obtained through the recommendations in that existing process. CARB uses a public process to update these estimates and publishes the approved California Climate Investments quantification methodologies on its website after a public review period. CARB will use these updated quantification methodologies as it prepares future annual Funding Plans, annual reports to the Legislature on California Climate Investments, and other longer- term planning documents. RECOMMENDATION: To promote transparency and inform stakeholders, beginning in December 2021, CARB should prepare an annual report for its board and the Legislature on its progress in isolating the GHG emissions reductions attributable to each of its regulatory and incentive programs. As a part of this report, CARB should identify any measurement challenges that persist and highlight any administrative barriers that prevent it from obtaining the information it needs to perform better analysis. 2 68 California State Auditor Report 2020-114 February 2021 CARB RESPONSE: CARB will report annually on its progress in identifying GHG emission reductions from related transportation incentive and regulatory programs. RECOMMENDATION: To strengthen the accuracy and integrity of its emissions reduction reporting, CARB should immediately begin retaining all supporting documentation it uses to perform calculations of GHG reductions for its cap-and-trade-funded incentive programs for a period of at least five years. In conjunction with this change, CARB should also document the justification for any instances in which the underlying data it uses to compile its annual reports vary from the information it publishes in those reports. CARB RESPONSE: CARB is updating its policies and procedures for the various incentives programs it administers to secure additional data related to data reporting and associated records retention. RECOMMENDATION: To better ensure the accuracy of its program data, by August 2021, CARB should develop a formal schedule and procedures for reviewing the supporting documentation maintained by its program administrators. These procedures, which CARB should begin using with the 2022 annual report, should specify a minimum number of records to review in relation to the program’s size, should specify how staff will collect and maintain evidence to support conclusions, and should be standardized across all of CARB’s incentive programs. 6 CARB RESPONSE: CARB currently conducts program and desk reviews of its program administrators. In response to this recommendation, CARB will evaluate the need for additional resources and funding to increase the frequency of its reviews and pursue opportunities to standardize that approach. RECOMMENDATION: To ensure that the State is positioned to assess the status of its sustainable communities program, by April 2021 CARB should report to the Legislature whether it will have a usable source for measuring regional GHG emissions in time for the 2022 report. If CARB believes it may not, it should identify any administrative or bureaucratic barriers it faces in accessing data it needs for the estimates and request relevant action by the Legislature to make those data available. CARB RESPONSE: The auditor correctly points out that CARB was unable to find a data source to accurately report GHG emissions reductions or vehicle miles travelled (VMT) by region to track SB 375 implementation. Since the 2018 progress report, CARB has initiated a number of efforts to better measure and track SB 375 program progress at a regional level. CARB will provide a status report of this work, including any issues we identify, by April 2021. Recommendations from Chapter 2 RECOMMENDATION: To ensure that it communicates clearly to the Legislature about the extent to which programs benefit low-income households as the Legislature intended, by 3 California State Auditor Report 2020-114 69 February 2021 March 2022 CARB should begin reporting its spending in low-income communities at the household level wherever possible in its annual report to the Legislature. CARB RESPONSE: CARB currently implements three incentive projects where participation is limited by household income – Clean Cars 4 All, CVRP, and Financing 7 Assistance for Lower-Income Consumers. CARB collects household income information from participants in each of these projects. As of the most recent California Climate Investments reporting cycle, CARB now reports low-income benefits into the California 8 Climate Investments Reporting and Tracking System (CCIRTS) at the household level for all three of these projects. In response to this recommendation, CARB will continue to report low-income benefits at the household level for all future reporting cycles for these projects and for any new projects where consumer participation is limited by household income. RECOMMENDATION: To better define incentive programs’ impacts beyond GHG emissions reductions, by August 2021 CARB should review its incentive programs to ensure that it has clearly designated which programs focus primarily on socioeconomic benefits. As a result of this process, by February 2022 it should ensure that it includes the benefits expected for each program in its funding plan or other public documents, such as its annual report and individual grant agreements. CARB RESPONSE: In response to this recommendation, CARB will clarify which Low Carbon Transportation incentive programs provide socioeconomic benefits, including but 9 not limited to public health benefits, green economic opportunities, and greater access to zero emission mobility. CARB’s Low Carbon Transportation equity projects authorized under Health and Safety Code Section 44258.4 (4)(A) are the projects which primarily focus on providing socioeconomic benefits. CARB considers funding additional projects in each Funding Plan and through a public process, and will consider socioeconomic factors in the development of those projects consistent with direction from the Legislature in its budget appropriations. RECOMMENDATION: To better demonstrate the socioeconomic benefits that its incentive programs achieve, by February 2022 CARB should do the following:  Identify clear and measurable metrics it will use to assess each of the socioeconomic benefits it intends its programs to achieve.  Develop a process to collect data, or use existing data, to measure and report on each metric.  In its funding plans and annual reports, CARB should report to the Legislature and its board on the metrics. CARB RESPONSE: In response to this recommendation CARB will work through the public process and with current project grantees, to identify additional socioeconomic metrics associated with clean transportation equity. CARB’s clean transportation equity 4 70 California State Auditor Report 2020-114 February 2021 10 projects currently incorporate surveys, focus groups, vehicle telematics, and other means of documenting overall project effectiveness, the results of which are used to adaptively manage the projects, address users’ needs, and increase community participation, while also informing future project planning. CARB will evaluate the need to modify future grant solicitations to accommodate relevant additional metrics. The ongoing CARB contracted research by UC Berkeley will provide input to inform this effort. 4 Implementation of this recommendation may include an evaluation of the need for additional resources. RECOMMENDATION: To provide transparency to the Legislature and other stakeholders, beginning in 2022, using the metrics and data described above, CARB should make funding and design recommendations in its funding plans and annual reports based on which programs are effective in producing socioeconomic benefits and at what cost. CARB RESPONSE: In response to this recommendation and in accordance with the activities taken under the preceding recommendation, CARB will present its initial findings in the FY 2022-23 Low Carbon Transportation Funding Plan and will continue to report in future annual Funding Plans and annual reports. RECOMMENDATION: To ensure that the State has reliable information about the extent to which cap-and-trade-funded programs create and support jobs, by August 2021 CARB should begin collecting data on the jobs produced by each of its incentive programs. Where needed, CARB should pursue amendments to its agreements with its program administrators to make reporting this information mandatory. CARB should include an analysis of these jobs data in its annual reports to the Legislature beginning in 2022. CARB RESPONSE: CARB will work with its grantees for ongoing Low Carbon 9 Transportation projects to collect and report on the direct jobs for each grantee resulting from CARB California Climate Investments funding to the maximum extend feasible. We 11 will work with grantees to revise existing grant agreements where feasible to ensure refined reporting across programs. CARB currently reports on these jobs benefits for the FARMER Program and Community Air Protection Program in the annual California Climate Investments Report. RECOMMENDATION: To ensure that its incentive programs promote effective and equitable job training, by August 2021 CARB should develop a process to assess which programs should include a job training element. For those programs it identifies, by February 2022 CARB should direct its staff or its external program administrators to collect and report on the quality of job trainings and outcomes experienced by participants, including who received training, the credentials participants received as a result, any actual or expected wages they received as a result of participating in the training or for developing the relevant expertise, and the number of participants from disadvantaged communities or low-income communities and households. 5 California State Auditor Report 2020-114 71 February 2021 CARB RESPONSE: In response to this recommendation, CARB will seek stakeholder input as part of the annual Low Carbon Transportation Funding Plan process, to 9 determine which programs should include a job training element. Currently, job training and workforce training elements have been included in some clean transportation equity projects and heavy-duty demonstration and pilot projects. For projects identified through the annual Funding Plan process as being appropriate to include a job training element, CARB will ensure that grant amendments include the appropriate reporting provisions as noted in the recommendation. Recommendation from Other Areas Reviewed RECOMMENDATION: To ensure that it can account for the total costs of its transportation programs, beginning with fiscal year 2021-22 CARB should develop and implement processes to track the administrative costs it incurs to operate each of its transportation programs. After doing so, it should begin including those costs as part of the cost- effectiveness measurements in its annual reports to the Legislature. CARB RESPONSE: CARB will develop and implement processes to track the administrative costs it incurs to operate each of its transportation programs within FI$Cal and track those costs as part of the cost-effectiveness measurements in its annual reports to the Legislature. 6 72 California State Auditor Report 2020-114 February 2021 California State Auditor Report 2020-114 73 February 2021 Comments CALIFORNIA STATE AUDITOR’S COMMENTS ON THE RESPONSE FROM THE CALIFORNIA AIR RESOURCES BOARD To provide clarity and perspective, we are commenting on CARB’s response to our audit. The numbers below correspond to the numbers we have placed in the margin of CARB’s response. 1 We do not understand the rationale for CARB’s statement that it will evaluate seeking funding to commission a study to determine where refinements may be needed to its quantification methodologies. As we discuss throughout Chapter 1, we identified deficiencies with CARB’s methodologies—such as not accounting for overlap between its incentive and regulatory programs—that lead it to overstate those programs’ GHG emissions reductions. Therefore, CARB’s proposed study seems unnecessary given the deficiencies we already identified, and could delay CARB’s implementation of our recommendation. 2 CARB’s response conflates statewide emissions reporting and its Emission Factor (EMFAC) tool with its measurement of the GHG reductions achieved by its individual transportation programs. During our audit, CARB confirmed that the statewide emissions reductions in CARB’s reporting cannot be attributed to specific programs. The statewide reporting is designed to measure total GHG emissions, but it is not able to assign responsibility for those reductions to individual programs. Further, CARB cannot use the tool to identify or account for overlap in the GHG reductions it projects or reports for each of its transportation programs. Therefore, we stand by our recommendation on page 38. 3 Although CARB states that it currently has work underway through surveys to help address this recommendation, our review found that its surveys generally did not address the crucial question of whether participants would have purchased their vehicles without receiving an incentive. Specifically, as we discuss on page 24 and show in Table 1 on page 25, for five programs we reviewed in which CARB provides an incentive payment or other financial assistance to consumers who purchase a low- or zero-emission vehicle—Clean Cars 4 All, CVRP, Financing Assistance, FARMER, and HVIP— CARB only collects survey information about behavior changes for CVRP. Further, we explain on page 29 that of the four follow-up surveys CARB has collected for its car-sharing pilot program, only two contained questions that CARB could use to validate its GHG emissions reduction assumptions about consumers replacing trips in conventional cars. Therefore, as we recommend on pages 38 and 39, CARB should collect and analyze relevant 74 California State Auditor Report 2020-114 February 2021 survey information for all consumer-focused incentive programs to improve its ability to identify the effectiveness of each of its incentive programs in reducing GHG emissions. 4 We acknowledge CARB’s contract with UC Berkeley for program evaluation on pages 29 and 46 of the audit report. However, the contract with UC Berkeley includes an evaluation of a small subset of CARB’s programs and notably does not include an evaluation of either HVIP or CVRP—two of CARB’s largest incentive programs; it will therefore exclude relevant analysis of the majority of CARB’s programs. Moreover, CARB’s response indicates that results of the UC Berkeley study are expected starting in May 2022, putting those results several months after the completion date of February 2022, which we believe is reasonable to implement our recommendation. Given how crucial participant behavior is to improving CARB’s emissions reduction methodologies, we believe that CARB should take the necessary actions to address our recommendation fully and timely. 5 As we state above in comment 1, we identified deficiencies with CARB’s quantification methodologies—such as not accounting for overlap between its incentive and regulatory programs—that lead it to overstate those programs’ GHG emissions reductions. Therefore, as we conclude on page 20, the processes CARB describes in its response have not done enough to demonstrate the amount of GHG reductions it projects and measures for its incentive programs. As such, our recommendations on pages 38 and 39 reflect our belief that CARB should be proactive in collecting and evaluating the information it needs to better measure programs’ emissions benefits—including information we already identified and named. 6 CARB’s statement that it currently conducts program and desk reviews of its program administrators overstates its efforts in this area. As we note on page 36, CARB has completed desk reviews for only two of the five programs we reviewed that make incentive payments to consumers—HVIP and CVRP. Further, as we state on page 37, CARB has not conducted any desk reviews for HVIP since 2014, and has conducted only two reviews for CVRP—in 2014 and 2020. Finally, we conclude on page 37 that the documentation CARB maintained from the reviews was not sufficient for us to independently determine whether any issues the reviews identified would affect emissions reporting for the programs. The shortcomings of CARB’s current approach are the basis for our recommendation on page 39 that CARB develop a formal schedule for these reviews, specify how staff will collect and maintain evidence to support conclusions, and standardize the review process across all of CARB’s incentive programs. California State Auditor Report 2020-114 75 February 2021 7 To fully implement our recommendation, CARB will need to broaden its planned implementation. Our recommendation on page 53 states that CARB should begin reporting its spending in low-income communities at the household level wherever possible in its annual report, and not solely for the three incentive programs where participation is limited by household income as CARB’s response states. As we state on page 43, the Legislature has acknowledged that some of the best GHG reduction strategies are those that benefit low-income households directly, regardless of where those households are located. Therefore, our recommendation is relevant to any CARB program that provides incentive payments at the household or individual level. 8 Although CARB may have already begun collecting low-income household data for the Clean Cars 4 All, CVRP, and Financing Assistance programs, this data will not become public until March 2021, when it provides its next annual report to the Legislature. CARB uses the CCIRTS database it mentions in its response to collect data on individual programs for its annual reports. CCIRTS is not itself a report. As such, the activity CARB describes in its response when saying it now reports this information into CCIRTS does not satisfy the recommendation we make on page 53 to publicly report on this information. Therefore, at the point CARB publishes its next annual report, we will evaluate the report to assess CARB’s implementation of this recommendation. 9 Although most of the incentive programs we reviewed, including CVRP and HVIP, are currently funded under the Low Carbon Transportation portion of the cap-and-trade program, our recommendations related to program information and benefits are not limited only to these programs. Rather, our recommendations apply to all of CARB’s transportation programs that target GHG emissions reductions and that may have socioeconomic benefits, or to which cap-and-trade reporting guidelines apply. 10 CARB’s characterization of its efforts to evaluate the socioeconomic benefits of its programs is disingenuous. In fact, as we discuss on page 45, CARB does not consistently collect data to determine whether its equity programs actually provide the socioeconomic benefits CARB intends. In an example we discuss on page 45, the metrics CARB identified for its Financing Assistance program do not allow it to measure the benefits it intends the program to provide. Further, despite repeatedly citing participants’ loan repayment rates as a measure of the Financing Assistance program’s success in its annual funding plans, CARB does not collect data about or report on these rates. As we explain on page 46, CARB also does not consistently use the data that are available to demonstrate benefits from programs and determine whether the 76 California State Auditor Report 2020-114 February 2021 programs are achieving their goals. Finally, managers for both the Financing Assistance and Clean Cars 4 All programs confirmed that CARB has not made any changes to the programs based on the data provided by program administrators in quarterly reports to CARB. Therefore, we do not agree with CARB’s assertion that it incorporates these types of data to manage the programs and inform future planning. 11 CARB’s statements are not fully responsive to our recommendation on page 53, which states that CARB should begin collecting data on the jobs produced by each of its incentive programs. We explain on page 50 that CARB’s own 2015 guidelines state that it should track and report the number of job recipients and trainees funded by its programs. As a result, CARB should already be collecting and reporting on this information. Further, as we state on page 51, CARB could seek to amend its grant agreements or, failing that, use its own tool to determine jobs benefits rather than relying on its program administrators to do so. Therefore, CARB should pursue any or all of these reasonable steps to fulfill its own reporting requirements. Finally, although CARB asserts that it reports jobs benefits for the FARMER and Community Air Protection programs, as we discuss on page 50, in its most recent annual report to the Legislature in March 2020, CARB had not reported any job creation numbers for individual programs. Instead, CARB reported aggregate job numbers across all cap-and-trade funded programs across multiple agencies. Further, those aggregate numbers were incomplete because, of the nine programs we reviewed for which jobs reporting requirements apply, CARB had collected jobs data in its reporting database for only three at the time of our review.