Summary

California Department of Transportation Its Capital Outlay Support Program Should Strengthen Budgeting Practices, Refine Its Performance Measures, and Improve Internal Controls April 2011 Report 2010‑122 Independent NONPARTISAN Accountability TRANSPARENT The first five copies of each California State Auditor report are free. Additional copies are $3 each, payable by check or money order. You can obtain reports by contacting the Bureau of State Audits at the following address: California State Auditor Bureau of State Audits 555 Capitol Mall, Suite 300 Sacramento, California 95814 916.445.0255 or TTY 916.445.0033 OR This report is also available on the World Wide Web http://www.bsa.ca.gov The California State Auditor is pleased to announce the availability of an on‑line subscription service. For information on how to subscribe, please contact the Information Technology Unit at 916.445.0255, ext. 456, or visit our Web site at www.bsa.ca.gov. Alternate format reports available upon request. Permission is granted to reproduce reports. For questions regarding the contents of this report, please contact Margarita Fernández, Chief of Public Affairs, at 916.445.0255. CALIFORNIA STATE AUDITOR Elaine M. Howle State Auditor Doug Cordiner B u r e a u o f S t a t e A u d i t s Chief Deputy 555 Capitol Mall, Suite 300 Sacramento, CA 95814 916.445.0255 916.327.0019 fax www.bsa.ca.gov April 28, 2011 2010-122 The Governor of California President pro Tempore of the Senate Speaker of the Assembly State Capitol Sacramento, California 95814 Dear Governor and Legislative Leaders: As requested by the Joint Legislative Audit Committee, the California State Auditor presents this audit report concerning the performance, management, efficiency, and budget of the California Department of Transportation’s (Caltrans) Capital Outlay Support program (support program). This report concludes that, despite a stated goal to reduce overruns in its support project budgets, Caltrans has performed little analysis to determine the frequency or magnitude of support cost budget overruns. Our review of projects that completed construction in fiscal years 2007–08 through 2009–10 indicates that 62 percent of the projects had support costs that exceeded their respective budgets. These overruns totaled more than $305 million of the $1.4 billion in total support cost expenditures for the projects that completed construction during these fiscal years. Our analysis found that the primary cause for support cost overruns was an increase in the hourly rate for support costs. For example, one project was approximately 14,600 hours under budget but exceeded its budgeted dollar amount by nearly $6.8 million, representing a support cost overrun of 83 percent. The changes in the hourly rate for support costs were due, in part, to salary increases of more than 40 percent during fiscal years 2005–06 through 2008–09 for certain Caltrans employees, including engineers. We also found that project managers for 12 of the 40 projects we reviewed monitored their budgets based primarily on the hours charged and not dollars spent. If project managers do not pay attention to costs, escalations in the rate paid per hour could cause a support cost overrun, even if the project remains under its budgeted hours. Further, project managers for 10 of the 40 projects we reviewed did not use a detailed approach to develop a support budget when the project was ready for construction. Moreover, although Caltrans has established a goal of reducing support costs to represent a ratio of 32 percent of the total capital costs (support-to-capital ratio), according to our assessment Caltrans generally did not meet its goal for fiscal years 2007–08 through 2009–10. In addition, Caltrans has failed historically to use a consistent method to calculate this ratio over time, thus decreasing the value of the ratio for assessing Caltrans’ performance in managing the support program. Furthermore, the support-to-capital ratio has limitations and could be defined more precisely to better measure efficiency, given that support costs can vary greatly depending on a project’s size and type. We also noted that Caltrans’ time-reporting system lacks strong internal controls, and better project monitoring and consistent use of performance metrics, such as earned value metrics, could help it minimize support cost overruns. Further, although Caltrans recently sought to hire consultants rather than permanent employees to address a temporary increase in workload, it was not successful in doing so because requests for consultants have historically been revised during the legislative budget process to align with a staffing ratio of 10 percent consultants to 90 percent state staff. Respectfully submitted, ELAINE M. HOWLE, CPA State Auditor Blank page inserted for reproduction purposes only. California Department of Transportation Its Capital Outlay Support Program Should Strengthen Budgeting Practices, Refine its Performance Measures, and Improve Internal Controls April 2011 Report 2010‑122 Blank page inserted for reproduction purposes only. California State Auditor Report 2010-122 vii April 2011 Contents Summary 1 Introduction 7 Audit Results Caltrans Has Not Adequately Prioritized Its Monitoring of Capital Outlay Support Costs, Its Project Support Costs Exceed Budgeted Amounts, and Its Systems Contain Inaccurate Budget Data 27 Project Managers Identified Multiple Reasons for Support Cost Overruns 34 Two Factors Contribute to Support Cost Overruns 35 Because the Commission Does Not Track or Review Construction Support Cost Overruns for STIP Projects, Neither Counties nor Caltrans Is Held Accountable When Construction Support Costs Exceed Estimates 40 Caltrans Has Generally Not Met Its Goal for Its Support‑to‑Capital Ratio, Has Calculated This Ratio Inconsistently, and Could Improve the Manner in Which It Measures Efficiency 40 Changes to Caltrans’ Internal Controls and Project Monitoring Could Improve Its Management of Support Costs 46 Increased Use of Consultants to Handle Large, Temporary Infusions of Funding May Help Caltrans Better Manage Workloads 50 Recommendations 52 Appendix Comparison of Support‑to‑Capital Ratios by District and Project Type and Size 57 Response to the Audit Business, Transportation and Housing Agency 61 California Department of Transportation 62 Comments California State Auditor’s Comments on the Response From the Department of Transportation 69 viii California State Auditor Report 2010-122 April 2011 Blank page inserted for reproduction purposes only. California State Auditor Report 2010-122 1 April 2011 Summary Results in Brief Audit Highlights . . . The California Department of Transportation (Caltrans) is Our review of the California Department of responsible for the design, construction, maintenance, and Transportation’s (Caltrans) Capital Outlay operation of the California State Highway System, as well as that Support Program (support program), portion of the Interstate Highway System within the boundaries highlighted the following: of the State.1 The Capital Outlay Support Program (support program) provides the funding and resources necessary to develop » Caltrans has done little analysis to and deliver the projects to construction, as well as to administer and determine the frequency or magnitude oversee the projects once they are under construction. Support of support cost budget overruns and to program functions include engineering, design, environmental inform stakeholders of the overruns. studies, right‑of‑way acquisition, and construction management of state highway projects. The fiscal year 2010–11 Budget Act allocated » Sixty‑two percent of the projects that $1.8 billion to Caltrans for the support program. completed construction during fiscal years 2007–08 through 2009–10 had Despite a stated goal to reduce overruns in its project budgets, support costs budget overruns, which Caltrans has done little analysis to determine the frequency or totaled more than $305 million of the magnitude of support cost budget overruns. Further, although $1.4 billion of such cost expenditures opportunities exist to inform stakeholders of the extent of these made during that period. overruns, Caltrans has not done so, limiting valuable information on the efficiency and effectiveness of the support program. » Differences between a project’s budgeted Based on our review of the data provided by Caltrans, 62 percent and actual support costs are due of the projects that completed construction in fiscal years 2007–08 primarily to an increase in the hourly through 2009–10 had support costs that exceeded their respective rate for support costs—one project we budgets. These overruns totaled more than $305 million of the reviewed was about 14,600 hours under $1.4 billion in support cost expenditures for the projects that budget yet nearly $6.8 million over the completed construction during these fiscal years. Caltrans’ cost budgeted. California Transportation Improvement and Programming System (CTIPS)—which Caltrans uses, in part, to capture project » Some potential causes for support cost budgets—is currently more reliable than Caltrans indicated had overruns include: been the case in prior fiscal years. Nevertheless, our review of the data provided by Caltrans for fiscal years 2007–08 through 2009–10 • Project managers did not use a found that Caltrans did not ensure that this system effectively and detailed approach to develop accurately tracked a project’s total support budget. According to a support budget when the project the chief of the Division of Transportation Programming, CTIPS was ready for construction. was not intended, at the time those projects were programmed, to capture projects’ total support budgets.2 Further, she stated • Project managers monitored their that Caltrans did not have a process for its headquarters to track budgets based primarily on the hours projects’ total supports budgets and instead relied on its districts charged and not the dollars spent. to do so. As a result, Caltrans risks limiting its ability to compare » Although Caltrans has established a goal of reducing total support costs to 32 percent of the total capital costs, it 1 When we use the name Caltrans, we are referring to the statewide organization and its has historically failed to use a consistent management headquartered in Sacramento. We refer to Caltrans’ districts as districts or identify method to calculate the ratio over time, the locations of specific districts’ headquarters. and has generally not met its goal for the 2 Transportation programming is the commitment of transportation funds to be available over a period of several years for allocation to particular projects. last three fiscal years. continued on next page . . . 2 California State Auditor Report 2010-122 April 2011 » Caltrans’ time‑reporting system lacks budgeted support costs to actual support costs, thereby decreasing strong internal controls, and better its and the public’s accurate assessment of the effectiveness and project monitoring and the use of efficiency of the support program or individual projects. performance metrics could help it minimize cost overruns. Differences between the budgeted and actual support costs of a project can generally involve one of two factors or a combination of the two factors: a difference between the expected and actual number of hours that staff charged to the project and a difference between the expected and actual hourly rate of staff time. Our analysis found that the primary cause for support cost overruns was an increase in the hourly rate for support costs. For example, one project was about 14,600 hours under budget but exceeded its budgeted dollar amount by nearly $6.8 million, an amount representing a support cost overrun of 83 percent. The changes in the hourly rate for support costs were due, in part, to salary increases effective from fiscal years 2005–06 through 2008–09. Specifically, the annual salaries for certain Caltrans employees, including engineers, increased by more than 40 percent during this four‑year period. We also found that project managers often did not update their budgets to account for these and other support cost increases. According to the chief of Caltrans’ Division of Project Management (division chief), until about five years ago, Caltrans placed a greater emphasis on ensuring that capital costs were within budget because these costs were generally the larger part of the project’s total budget. However, she explained that more recently Caltrans has been increasing its emphasis on managing support costs separately from capital costs. Based on discussions with Caltrans project managers, we found several potential causes for support cost overruns. For example, project managers for 12 of the 40 projects we reviewed indicated that they monitored their budgets based primarily on the hours charged and not dollars spent. If a project manager does not pay attention to costs, escalations in the rate paid per hour could cause a support cost overrun, even if the project remains under its budgeted hours. Further, project managers for 10 of the 40 projects we reviewed did not use a detailed approach to develop a support budget when the project was ready for construction. According to the division chief, when budgets are overstated, fewer projects received funding, and when budgets are understated, the subsequent overruns take funding away from other projects, leading to construction delays. A February 2010 memorandum that took effect in July 2010 requires Caltrans to produce quarterly a list of projects that are expected to exceed their budgets and to distribute this list to the districts. The memorandum further requires project managers to prepare funding plans to address these potential cost overruns. Further, because the California Transportation Commission (commission) does not track or review construction support cost overruns for State Transportation Improvement California State Auditor Report 2010-122 3 April 2011 Program (STIP) projects, neither the counties nor Caltrans is held accountable when construction support costs exceed estimates. Given the limited funds available for STIP projects, overruns on current projects impair the State’s ability to undertake future projects. Caltrans has established a goal of reducing support costs to represent a ratio of 32 percent of the total capital costs, referred to as the support to capital costs ratio (support‑to‑capital ratio). Although it has identified this goal as an objective in its current strategic plan, Caltrans has failed historically to use a consistent method to calculate this ratio over time, thus decreasing the value of the ratio for assessing Caltrans’ performance in managing the support program. Using a consistent methodology, we conducted our own assessment of Caltrans’ support‑to‑capital ratios for the last three fiscal years and determined that Caltrans generally did not meet its goal. Finally, the support‑to‑capital ratio has limitations and could be defined more precisely to better measure efficiency, given that support costs can vary greatly depending on a project’s size and type, both of which can have a large impact on the resulting ratio. Additionally, Caltrans’ time‑reporting system lacks strong internal controls, and better project monitoring and the use of performance metrics could help it minimize cost overruns. We found that Caltrans lacks strong internal controls to ensure that staff appropriately charge time to support program projects. Further, although Caltrans has established some project‑monitoring processes and performance metrics, it has not comprehensively implemented these tools. For example, consistent use of earned value metrics could help Caltrans to better manage its support program projects. Earned value management integrates measures of a project’s scope, cost, and schedule to help the project management team assess and measure project performance and progress. However, in reviewing four districts, we noted that their use of earned value management varied greatly. For example, the Los Angeles district has a robust system utilizing earned value metrics. Moreover, although Caltrans recently sought to hire consultants rather than hire permanent employees to address a temporary increase in workload, it was not successful in doing so. According to the chief of Caltrans’ Project Delivery Management Support Office, requests for additional consultants historically have been revised during the legislative budget process to align with a staffing ratio of 10 percent consultants to 90 percent state staff. The deputy director of the San Diego district’s Division of Project Management stated that this ratio creates limitations and that consultants have been an effective way to manage resources. To the extent increases in workload are temporary in nature, it may be more fiscally prudent for Caltrans to address this workload with consultants rather than with permanent state employees. 4 California State Auditor Report 2010-122 April 2011 Recommendations To improve accountability internally and with the public, Caltrans should create and incorporate an analysis of support cost budget variances in its quarterly report to the agency and in its annual report to the Legislature and the governor. The analysis should report on the number of completed projects with budget variances and on the number of open projects for which the estimates at completion predict budget variances. Further, the analysis should report on the overrun and underrun ratios for those projects, and the portions of the variances due to rates and hours. Caltrans should also include in its strategic plan a measurable goal for reducing variances. To improve performance metrics related to the support program, Caltrans should take the following steps: • Devise, use, and publicize a consistent method for reporting the support‑to‑capital ratio on its Web site and in other reports to the public. Further, Caltrans should recalculate past support‑to‑capital ratios using the method devised to allow for comparisons across years. • Develop goals—and then publicly report on its progress in reaching those goals—for the support‑to‑capital ratio based on project type, for STIP and the State Highway Operations and Protection Program (SHOPP), and for project size. To better develop and manage project budgets, Caltrans should direct its project managers to use a detailed approach based on project tasks, such as those included in a project work plan, when finalizing project budgets before construction. To ensure it monitors the status of projects, Caltrans should do the following: • Continue to implement the policies described in its February 2010 memorandum to the districts describing an approach Caltrans will take to monitor support costs within budget. Moreover, Caltrans should direct its project managers to monitor budgets for all projects according to both hours and costs. • Implement earned value management throughout its districts in a manner similar to the implementation in the Los Angeles district. To allow for performance evaluation of project work, Caltrans should ensure that these performance metrics are available at the task level for both active and completed projects. Caltrans should instruct districts to aggregate this information for all projects by task level, to better assess the effectiveness California State Auditor Report 2010-122 5 April 2011 and efficiency of support costs by task level. Caltrans should also make graphical displays of project cost and adherence to schedule available to project managers. To better address costs associated with the support program, Caltrans should do the following: • Ensure that its new project management system contains strong controls that ensure employees charge time only to projects and phases to which the employees are assigned. • Commission an independent study of the costs and benefits of using consultants to address temporary increases in workload and, if the study reveals cost savings, to use consultants. To the extent possible, Caltrans should also use temporary staff appointments for temporary increases in workload when consultants are unavailable. To increase accountability for budget overruns of support costs, the Legislature should consider legislation that would expressly require the commission to review and approve project construction support costs when they differ from the amount budgeted by 20 percent or more. To ensure that Caltrans does not hire permanent state staff beyond its need for such staff, the Legislature should consider appropriating funding for consultants to address temporary increases in Caltrans’ workloads when Caltrans requests such funding. Agency Comments Caltrans generally agreed with our recommendations, except for our recommendation regarding the use of temporary staff and the need to commission a study of the benefits of using consultants to address temporary increases in workload. Additionally, Caltrans raised concerns about recommendations regarding tracking where employees are authorized to charge time and requiring the commission to review and approve project construction support costs. 6 California State Auditor Report 2010-122 April 2011 Blank page inserted for reproduction purposes only. California State Auditor Report 2010-122 7 April 2011 Introduction Background The California Department of Transportation (Caltrans) is responsible for the design, construction, maintenance, and operation of the California State Highway System as well as for that portion of the Interstate Highway System within the boundaries of the State.3 The Capital Outlay Support Program (support program) provides the funding and resources necessary to develop and deliver the projects to construction as well as to administer and oversee the projects once they are under construction. Support program functions include the engineering, design, environmental studies, right‑of‑way—which entails obtaining property rights for the construction of projects—and construction management of state highway projects. The fiscal year 2010–11 Budget Act allocated $1.8 billion to Caltrans for the support program. This amount funds the fiscal year 2010–11 support activities associated with about 2,500 capital outlay projects and about 9,300 positions within Caltrans and its 12 districts. Figure 1 on the following page provides a map of Caltrans’ districts and related regions, and it indicates the four districts we reviewed for purposes of this audit—districts headquartered in Oakland, Fresno, Los Angeles, and San Diego. According to the chief of Caltrans’ Project Delivery Management Support Office, about 500 of the support program staff are based at Caltrans’ headquarters in Sacramento, where they manage the support program. The remaining staff are based at Caltrans’ 12 districts or the Division of Engineering Services (engineering services)and are responsible for managing the individual projects.4 Overview of the Support Program The support program functions for a project begin after the California Transportation Commission (commission) programs funding for the project, and they continue until the project is completed.5 Figure 2 on page 9 outlines the different support tasks involved in a project’s life cycle. As a project progresses through the various support functions, Caltrans and its districts must approve certain milestones. For example, before a project may begin construction, the district division chief for right‑of‑way must sign 3 When we use the name Caltrans, we are referring to the statewide organization and its management headquartered in Sacramento. We refer to Caltrans’ districts as districts or identify the locations of specific districts’ headquarters. 4 Caltrans uses engineering services to support capital projects for the design and construction of structures, such as bridges or wastewater treatment design. Although this division is based in Sacramento, its staff are located in each of the districts. 5 Transportation programming is the commitment of transportation funds to be available over a period of several years for allocation to particular projects. 8 California State Auditor Report 2010-122 April 2011 Figure 1 Map of the California Department of Transportation’s Regions and District Offices, Including Those Visited by the Bureau of State Audits DEL NORTE SISKIYOU MODOC Districts we visited for analysis District offices 2 EEUURREEKKAA SHASTA LASSEN HUM 1 BOLDT TRINITY RREEDDDDIINNGG North Region Districts 1, 2, and 3 TEHAMA PLUMAS GLENN BUTTE SIERRA MENDOCINO LAKE COLUSA S U TTE R MM YU AA BA RRYY N SS EV VV AD II A LL P LL LA EE CER 3 YOLO EL DORADO SONO M M A A RIN NAPA S C O O C L N O A T S N R T O A A SA J CR O AM S A EN A Q TO N UIN AM SS A TT D CA OO O L R A CC VE KK RA TT S OO TU NN O A L L U P M IN N E E MONO Central Region SAN F S R A A N N M C A IS4 T C EO O A OO LA AA S M A E KK D N A LL TA AANNDD STANISLAUS 10 MARIPOSA Districts 5, 6, 9, and 10 CLARA MERCED MADERA BBIISSHHOOPP 6 SANTA CRUZ 9 FRESNO SAN BENITO FFRREESSNNOO INYO TULARE MONTEREY KINGS 5 SSAANN LLUUIISS OOBBIISSPPOO KERN SAN LUIS OBISPO SAN BERNARDINO 8 SANTA BARBARA 7 VENTURA LOSANGELES LLOOSS AANNGGEELLEESS SSAANN BBEERRNNAARRDDIINNOO RIVERSIDE ORANGE IIRRVVIINNEE 12 11 IMPERIAL SAN DIEGO SSAANN DDIIEEGGOO Source: California Department of Transportation. California State Auditor Report 2010-122 9 April 2011 2 erugiF tnenopmoC tcejorP hcaE rof elcyC efiL tcejorP s’noitatropsnarT fo tnemtrapeD ainrofilaC setamitsE dna ,snoitacfiicepS ,snalP ot krow lla sedulcni tnenopmoc sihT ,snalp tcartnoc eht poleved ,etamitse s’reenigne ,snoitacfiiceps noitacolla ,stnemucod dib tcartnoc dna ,drawa tcartnoc ,sdnuf fo .lavorppa tcartnoc ,lluf eraperp ot deriuqer si snartlaC ,snalp etarucca dna ,etelpmoc latnemnorivnE dna stimreP tsoc fo setamitse dna ,snoitacfiiceps seidutS rof tcartnoc a otni gniretne erofeb ti ,deecorp ot tcejorp latipac a roF .tcejorp a ,etats ,laredef laicffio eviecer tsum )DIP( tnemucoD noitaitinI tcejorP noitcurtsnoC reenigne ecffio tcirtsid ehT sa ,lavorppa latnemnorivne dna rof elbisnopser si )reenigne( lla morf susnesnoc sa llew yfitnedi stcirtsid eht ,yllaunnA tcejorP fo dnE sevorppa noisivid lagel ’snartlaC ,ssenetelpmoc eht gnirusne .cilbup eht dna sredlohekats eriuqer lliw taht stcejorp fo r n e e o h e i t n t e s , i n g l d p o n n m i e e t c o m t u n c m r e n t d s o o n i c s p o e e U c r r eh s t i . l f r a o o y v t e e o c n c r a p ffi r r o t p n O t a t o A e t c c ( h a e l t a r h r r t t o e n d n f o t n e c c G a f a o r , y ) t d l e n a e n o r fi e r c o n i t e t e o t h G A n t ,s e n h a t T a l s p h e . s t f t r o a e e m r t y u r c i a t s n u s s e e q e t d d s k i a a n s m e n a h o , s o c o n s d t o l a n d i t a e r a e t c , t e y fi i t n m i i c i l g a b e u n p u q e s s e d — r n a a e s s o l e a a s i v h d la o p u — r t p t s n p t e g n A m n e t i n c u r e e o c j e o p o n D m rP i g l o a e n c t h e n s t e i d h s m e a t d n g n n o n w r a i i o r p v u n n x D e k E a d ,e e p t t s t o i i l c m A s b d .s u e D s n I s fi P i e p s d t o c a l e e j . s v s o e e r r d e p d t u r d o l a c t e u n s s q i o e D d c p r a I o u P e r o h p e s h o f e o T t r fo ecnatpecca rof tcartnoc noitcurtsnoc eht retfA dna ,snoitacfiiceps ,snalp eht sredlohekats dna detelpmoc ,etamitse tsoc troppus dna latipac eht sa tcartnoc eht neeb sah tcejorp latipac snartlaC a ’snartlaC ot mrofnoc setamitse dna tcejorp eht ot etubirtnoc eht rof nalpkrow tcejorp a dna fo evitatneserper .nigeb nac noitcurtsnoc ,dedrawa .sdradnats dna ,serudecorp ,seicilop .sevitanretla sti rof dednemmocer evitanretla ’snartlaC .etatS eht .tcejorp eht gnimmargorp fo noisiviD stsisnoc tnenopmoc siht ,rehtruF noitcurtsnoC gnidulcni ,semoctuo suoirav fo ni noitercsid evah srotcerid tcirtsiD stpecca yletamitlu eht senfier taht troper tcejorp eht DIP rof stcejorp tcirtsid gnizitiroirp noitcurtsnoc eht sefiitnedi ,esoprup s’tcejorp ot ytirohtua evah dna tnempoleved eht rof tcartnoc yaW -fo-thgiR ,detceles sevitanretla eht .sDIP evorppa .snartlaC fo rotcerid sti fo ynam fo noitcurtsnoc eht rof sthgir ytreporp niatbo ot deriuqer si snartlaC sevitanretla esoht woh sebircsed margorP troppuS yaltuO latipaC ehT woh sebircsed dna ,dediced erew lagel dna spam gniraperp sevlovni tnenopmoc sihT .stcejorp noitatropsnart si gnidnuf nehw devlovni semoceb ainrofilaC eht dna sredlohekats fo noissessop lacisyhp dna lagel gniniatbo ,slasiarppa gniraperp ,stnemucod .tcejorp eht rof demmargorp noitatropsnarT fo tnemtrapeD dna ,snoitcurtsbo lacisyhp lla gniraelc ,stnapucco gnitacoler ,ytreporp .susnesnoc dehcaer )snartlaC( .seitilitu gnitacoler eht ,noitacfiitrec yaw-fo-thgir eht edulcni tnenopmoc siht fo selbareviled niam ehT .yaw-fo-thgir eht morf snoitcurtsbo lacisyhp fo ecnaraelc eht dna yaw-fo-thgir lagel eht ,revewoh ;noitcurtsnoc hguorht seunitnoc tnenopmoc yaw-fo-thgir ehT .nigeb yam noitcurtsnoc erofeb eud si noitacfiitrec .launaM serudecorP tnempoleveD tcejorP sti dna ,koobdnaH tnemeganaM tcejorP ’snatlaC :secruoS 10 California State Auditor Report 2010-122 April 2011 a certification that Caltrans has acquired or is acquiring the necessary real property. Caltrans and the Office of the Attorney General, or an attorney representing Caltrans, must review and approve contracts for construction. Each of the 20 projects we reviewed had received the required approvals during various phases of the project. Caltrans’ Division of Project Management is responsible for managing and delivering the projects, and it monitors and reports on the projects’ status. This responsibility includes developing the overall workload and budget for the transportation improvement project portfolio by creating and implementing the tools, formulas, and standards used to estimate the resources necessary to deliver each project. In addition, this division is responsible for developing, conducting, and managing training courses to improve the skills of the staff working on these projects. Caltrans’ Division of Construction is responsible for establishing construction policies, directions, and objectives. This responsibility includes providing guidance to the districts on administering construction contracts and validating that district construction contracts are administered fairly and in good faith. The Division of Construction is also responsible for providing expert assistance to district construction managers on complex, sensitive issues in construction contracts. Within each district, the deputy district director for project management has the overall responsibility for managing the support program. The project managers in each district have the full authority, delegated by the deputy district director for project management, to deliver the project within the approved scope, schedule, and budget. Moreover, the project manager is the advocate and primary point of contact for the project, and he or she is responsible Comparison of Methods for Structuring for resolving problems that may affect the project. Project Teams Caltrans has a number of functional divisions • Functional organization: Staff members are grouped by that perform specific tasks on each project. These specialty and will do their project work independently functional divisions include, for example, the of other special groups. The staff performing the project tasks report to the functional managers, not the Division of Design and the Right‑of‑Way Division. project manager. According to the chief of Caltrans’ Division of Project Management (division chief), the project manager, • Projectized organization: Staff from various organizational along with the project development team, determines units are involved in the project and usually report directly which tasks are necessary for a project and which to the project manager. functional divisions will handle them. The project • Matrix organization: A blend of functional and manager approaches the project development team projectized characteristics. Weak matrices have many of and the managers from each functional division the characteristics of functional organizations, while strong (functional managers) with a list of tasks to be matrices have many of the characteristics of projectized completed by their units. The functional managers organizations. A balanced matrix shares characteristics then assign resources from their units to each task. of both. The division chief further explained that Caltrans Source: Project Management Institute’s A Guide to the Project structures most projects using a weak matrix Management Body of Knowledge, Fourth Edition. approach, which the text box describes, rather than assigning staff and managers only to one project at a California State Auditor Report 2010-122 11 April 2011 time. Caltrans asserts that this practice allows it to be more efficient with its resources, as it has more than 2,000 projects active at any one time. However, the division chief stated that for very large projects, Caltrans may establish a dedicated team. The State Transportation Improvement Program and State Highway Operation and Protection Program Capital improvement projects that increase the capacity of the State’s transportation infrastructure are partially funded through the State Transportation Improvement Program (STIP), and projects that rehabilitate or preserve existing infrastructure are funded through the State Highway Operation and Protection Program (SHOPP). These two programs provide funding for capital outlay projects. The commission is responsible for programming funds for particular transportation projects, and the SHOPP and STIP are the two primary programs through which the commission programs and allocates funds. The SHOPP is a four‑year plan of projects, while the STIP is a five‑year plan, and both programs are approved by the commission every two years. To aid the commission in its duties, Caltrans is required by law to create a fund estimate for the STIP every two years. The fund estimate includes a forecast in annual increments of all federal and state funds available for programming in the next STIP cycle. The fund estimate also includes projections for SHOPP, to which state law gives priority for state transportation funding. Between fiscal years 2010–11 and 2014–15, the fund estimate projected a total SHOPP capacity of $8.4 billion, and it projected $3.8 billion for STIP. In 1997 Senate Bill 45 was enacted and, in addition to making other changes, substantially revised the process for distributing state and federal funds available for transportation projects. For example, the bill mandated that 75 percent of funding for STIP is to be used for regional improvements. Further, it required the commission to set aside sufficient funding for all projects programmed in 1996, but it did not expressly require Caltrans to seek commission approval for additional funding for those projects and the office chief of capital improvement programs informed us that these projects do not have support budgets. Caltrans’ Process for Creating Its Annual Budget Request for the Support Program As we noted earlier, the support program is funded through the annual budget act. For fiscal year 2010–11, Caltrans has 12 funding sources available for support program projects. As Figure 3 on the following page indicates, the two largest funding sources are the State Highway 12 California State Auditor Report 2010-122 April 2011 Account and the Federal Trust Fund. Another major source of funding is reimbursements, including those for projects funded by local agencies, such as regional transportation authorities, which reimburse Caltrans for the cost of the projects. Additionally, bond funds provide funding for the support program. For example, Proposition 1B, which took effect in 2006, authorized the issuance of about $20 billion in general obligation bonds for transportation improvements, $12 billion of which, according to the Legislative Analyst’s Office (legislative analyst), Caltrans would play a crucial role in delivering. The commission approves project budgets for the support program, and these projects may span multiple years or, in some cases, a decade or more. However, Caltrans uses funds authorized in the budget year only for those project support activities occurring within the budget year. Thus, each project receives funding through multiple budget acts. In addition, the amount of support budgeted for a project will vary from year to year depending on the stage of the project. Figure 3 Sources of Funding for Projects in the Capital Outlay Support Program Fiscal Year 2010–11 (Dollars in Millions) Other*—$18 (1%) Bonds†—$134 (7%) Reimbursements—$270 (15%) State Highway Account— $852 (48%) Federal funds— $507 (29%) Source: The fiscal year 2011–12 Governor’s Budget, which contains the actual capital outlay support funding for fiscal year 2010–11. * Other includes the Historic Property Maintenance Fund and the Traffic Congestion Relief Fund. † Bonds include funds from the Seismic Retrofit Bond Fund of 1996 and the Trade Corridors Improvement Fund as well as the Corridor Mobility Improvement Account; Transportation Facilities Account; Highway Safety, Rehabilitation, and Preservation Account; and State Route 99 account of the Highway Safety, Traffic Reduction, Air Quality, and Port Security Fund of 2006. Table 1 illustrates the relationship between the support program and project budgets by using one project each from four districts as examples and, for an overall perspective, by showing the totals for all of Caltrans’ projects. These budgets are typically measured in personnel California State Auditor Report 2010-122 13 April 2011 years and are converted to dollars during the annual budget process.6 The columns in Table 1 represent the personnel years requested in the annual support program budget for each of the sample projects, while the rows represent the personnel years required over five years of the project’s life cycle, although projects may last longer than five years. For example, the roadway rehabilitation project shown for the Oakland district required about 17 personnel years of support in fiscal year 2006–07 but almost none in fiscal year 2010–11. The support program budget for all projects during a given fiscal year thus represents only a fraction of the total personnel years required for those projects. Table 1 Examples of Program and Project Budgets PERSONNEL YEARS BUDGETED FOR FOUR SAMPLED PROJECTS BY PROJECT AND FISCAL YEAR FISCAL YEAR TOTAL WORKLOAD DISTRICT PROJECT 2006–07 2007–08 2008–09 2009–10 2010–11 FOR PROJECT* Oakland Rehabilitate roadway 17.01 4.75 3.77 0.33 0.01 25.87 Fresno Rehabilitate rest areas 1.39 4.05 4.26 1.59 1.70 12.99 Los Angeles Construct soundwalls 0.57 10.45 12.55 0.27 16.14 39.98 Program’s San Diego Widen freeway and bridge 9.65 1.60 13.29 7.61 15.08 47.23 Annual Budget Remaining Remaining projects† 10,956.03 10,273.76 9,861.13 9,092.20 9,312.85 49,495.97 Districts Total Workload For Budget Year 10,984.65 10,294.61 9,895.00 9,102.00 9,345.78 49,622.04 Individual Project’s Lifetime Budget Source: California Department of Transportation’s (Caltrans) estimated personnel year workload for fiscal years 2006–07 through 2010–11. * We have presented the program budgets for five fiscal years, and highlighted four project budgets during these fiscal years as examples. Projects may last longer than five fiscal years and the program budgets specify all project workloads from Caltrans’ 12 districts. † The fiscal year 2010–11 budget request for the Capital Outlay Support Program included nearly 2,500 projects. Caltrans develops budget requests for the support program based on its anticipated workload for the budget year. For example, in January 2010 Caltrans produced a list of projects that it anticipated to be active in fiscal year 2010–11 as well as the personnel years estimated for each project during the fiscal year. It based this list on district work plans, which Caltrans’ district project managers are responsible for approving. These work plans detail the schedule and the staff time required for each project in the district. For fiscal year 2010–11, Caltrans compared each project against historical data to assess the reasonableness of the district work plans. Based on the work plans, Caltrans can then determine the amount of resources, measured in personnel years, that are required to perform the work scheduled for the budget year. 6 Caltrans equates one personnel year to 1,758 hours of staff time. 14 California State Auditor Report 2010-122 April 2011 Once it determines the overall workload, Caltrans assesses the portion of the workload that will be accomplished through contracted consultants, Caltrans staff, and overtime as well as the cost of each of those resources. For contracted consultants, Caltrans relies on its districts to report the hours and cost of each consultant contract that will be used in the budget year. In determining the amount and cost of state staff, Caltrans uses the estimated workload for the budget year to ascertain whether more or fewer personnel years are required. It then calculates the cost of the difference based on the distribution of positions within Caltrans and the salaries for those positions. Caltrans also determines the percentage of work that will be accomplished through overtime rather than through additional positions, and it includes in its budget request several categories of indirect costs, which include costs not attributed to a specific project. For example, the cost of office space and the cost of personnel who do not work on a specific project— including Caltrans’ managers and supervisors—are indirect costs. Figure 4 illustrates the direct and indirect costs as a proportion of Caltrans’ total budget request for the support program. Figure 4 Distribution of Personnel Years for the Capital Outlay Support Program Fiscal Year 2010–11 Indirect Costs: Consultant Contract Administration*—185 personnel years (PYs) (2%) Indirect Costs: Employee Training and Other Non-Project Activities*—313 PYs (3%) Direct Costs: State Staff Overtime—348 PYs (3%) Direct Costs: Consultants—1,232 PYs (11%) Indirect Costs: Management, Supervision, and Other Direct Costs: State Overhead Costs*—1,990 PYs (16%) Staff—7,598 PYs (65%) Sources: California Department of Transportation (Caltrans) and May 2010 Senate Budget and Fiscal Review Subcommittee No. 2 Transportation Hearing Outcomes (May 2010 transportation hearing outcomes). * The personnel years for indirect costs regarding consultant contract administration; employee training and other non‑project activities; and management, supervision, and other overhead costs are not delineated in the May 2010 transportation hearing outcomes. Therefore, we relied on information provided by Caltrans for these figures. California State Auditor Report 2010-122 15 April 2011 Finally, Caltrans may include the cost of extraordinary items in its budget request. For example, according to the May 2010 Senate Budget and Fiscal Review Subcommittee No. 2 Transportation Hearing Outcomes, components of the San Francisco–Oakland Bay Bridge East Span Seismic Safety Project are manufactured in China, Japan, and Korea. Caltrans’ staff coordinate and monitor the fabrication of these components, and in fiscal year 2010–11, the travel costs for this monitoring were expected to be $1.1 million. According to the chief of statewide resources management branch, Caltrans allocates budget resources among its districts and engineering services, using the departmental budget allocation approved by the Legislature and the governor. Further, the division chief explained that the allocation of budget resources to the districts and engineering services is based on the workload expected in those districts for the coming fiscal year. According to Caltrans’ budget allocation methodology, Caltrans adjusts project workloads for various overhead factors before distributing resources to the districts and to engineering services. Additionally, allocations are given for overtime, overhead—such as costs associated with travel, training, and administration—and consultant resources. Caltrans’ Systems for Tracking and Monitoring Projects Caltrans’ districts upload project data into a central database called eXpert Project Management (XPM), which was installed in the districts in 1995. However, the districts do not use the XPM system to manage projects directly. A deputy district director for project management noted that XPM is not available to all users because of licensing and issues related to difficulty of use. As a result, only a select group of users can input project schedules and resources into XPM. Another deputy district director for project management said that when the XPM system was implemented, it was not easy to install, use, or maintain and that the database often crashed. He noted that Caltrans decided in 1998 to allow the districts to choose their own project management software as long as they submit their project data to XPM on at least a quarterly basis. As a result, Caltrans’ districts use various project management software to track and monitor projects. The four districts we visited use three different programs. Each district uploads at least quarterly its project data to Caltrans’ XPM system. According to the division chief, these uploads enable Caltrans to monitor the progress of projects; to provide reports for quarterly meetings on the status of capital projects; to create annual budgets, as described previously; and to generate reports for the commission. 16 California State Auditor Report 2010-122 April 2011 In addition to project management software, Caltrans districts use specialized reports to track and monitor projects and to evaluate performance. For example, the Central Region, which is headquartered in Fresno, generates an annual performance report. This report focuses on performance measures such as meeting delivery milestones for planned projects, performance in relation to statewide goals for the use of overhead, and support to capital cost ratios (support‑to‑capital ratios), which we discuss in the Audit Results. These measures fall within three areas: delivery, efficiency, and effectiveness. The performance report also identifies areas for improvement in the next fiscal year. The Oakland district uses reports from its construction division to track project expenditures, such as labor and operating expenses, and employee time charges. The San Diego district uses a report to track project expenditures. The Los Angeles district’s Project Information and Reporting System generates a report that displays the performance of various levels of support operations, such as the construction and right‑of‑way divisions or the units within those divisions. The Audit Results further describe these reports—part of the implementation of a management technique known as earned value management—which help project managers review various units’ performance in relation to Problems and Requirements That the budgeted and scheduled project work. Project Resource and Schedule Management (PRSM) System Should Address Caltrans’ New Project Management System Problem 1: The California Department of Transportation (Caltrans) cannot fully meet the reporting requirements To address concerns with its current systems, as mandated by the Legislature and the California Transportation Commission. Caltrans expects to implement a new project management system called Project Resource • Requirement: Compare planned to actual costs on and Schedule Management (PRSM). According a project. to both the assistant chief of Caltrans’ Division Problem 2: Substantial time and effort is required to of Project Management, who is also the PRSM develop project schedules. project director (assistant division chief) and • Requirement: Allow project scheduling based on information found on Caltrans’ intranet, the actual staffing requirements and support the use of a system is intended to replace XPM, the current non‑linear distribution of workload. project management system. The assistant division chief also stated that the system is expected to • Requirement: Support earned value analysis interface with other systems, including Caltrans’ and reporting. time‑reporting system and its accounting system. Problem 3: Caltrans lacks the ability to identify skilled Finally, he noted that PRSM will eventually replace individuals and assign them to specific tasks. the need for some other Caltrans’ systems. • Requirement: Capture skills inventory and skill development needs of all resources. In 2000 Caltrans formulated a list of major problems to resolve when determining the • Requirement: Allow individuals to be assigned to requirements for a system to replace XPM. The specific tasks. text box describes some of these major problems Source: Caltran’s—PRSM Problems, Objectives and Requirements. and some of the requirements identified that will address those problems. Caltrans began California State Auditor Report 2010-122 17 April 2011 the procurement process for this system in 2000 and, according to its independent project oversight provider, it anticipates implementation of PRSM in all of its districts by August 2012. According to the assistant division chief, the system was originally intended to replace XPM and the unique project management systems at each of the districts. However, he noted that in 2004 or 2005, the scope of the project was revised to specify that PRSM will replace XPM only. Nevertheless, according to the assistant division chief, PRSM will still replace some of the need for those unique project management systems. He further indicated that each district’s project management system will be maintained in a parallel function during the PRSM implementation until Caltrans is satisfied that PRSM is functioning adequately as a project management system in that district. The assistant division chief stated that he expects that Caltrans will phase out the districts’ project management systems after final implementation of PRSM. The assistant division chief also explained that processes have been developed for transferring data between the XPM project work plans and PRSM, as well as for setting up interfaces between PRSM and Caltrans’ accounting system and its time‑reporting system. He noted that interfaces with existing systems were created and tested and that the data transfer between XPM and PRSM is to be performed by conversion routines, which transform project data in XPM into the proper formats for uploading into PRSM. Further, he stated that during deployment of PRSM at the districts, for any data conversion issues discovered, the routines will be modified as necessary so that Caltrans can ensure that data is converted accurately. Finally, the assistant division chief said that the next phase of data conversion involves actively converting project data in a district from one system to the other. A consulting firm provides independent project oversight of PRSM’s implementation and produces a monthly Independent Project Oversight Report (oversight report) on the status of the project, which, among other things, identifies risks to the project and quantifies whether the project is within budget and identifies schedule delays. Caltrans submits these reports to the Legislature each quarter. In its February 2011 oversight report, the consulting firm noted that adaptation phase activities such as testing and data conversion are scheduled to be completed by the end of May 2011. The report also states that the implementation has encountered delays and problems, including schedule extensions and cost increases. According to the report, total one‑time costs for the system will be about $26.1 million. 18 California State Auditor Report 2010-122 April 2011 According to the assistant division chief, the PRSM project has experienced many delays since procurement began in 2000. He explained, for example, that in 2006, a project schedule indicated an expected implementation date of April 2008; however, the contract with the vendor chosen to implement PRSM was not executed until February 2009. Further, between the November 2010 and the February 2011 oversight reports, the end date for the PRSM project moved from April 2012 to August 2012. The February 2011 oversight report states that the current phase of the project has been delayed due to a number of defects identified during configuration testing and by issues regarding data conversion. The assistant division chief told us that Caltrans anticipates piloting PRSM at the Marysville district in June 2011, and estimates it will implement the system on a departmentwide basis in 2012. Past Studies of the Support Program Reducing capital outlay support costs has been an ongoing challenge for Caltrans. In 1996 Caltrans underwent a peer review that recommended specific actions to improve its efficiency and effectiveness in forecasting the workload and budget for capital outlay support. In addition, the legislative analyst’s review of the fiscal year 2003–04 budget bill noted that actual project support expenditures were higher than budgeted amounts and recommended that the governor’s budget require Caltrans to create targets for each of the support program’s performance measures. Further, in 2004 the California Performance Review found that Caltrans’ ineffective project management, along with fluctuating staffing for highway improvements, resulted in project delays, higher costs, and unsatisfied customers. It also noted that in 2003 Caltrans was criticized by local partner agencies concerned about transportation project delivery management. More recently, in March 2010, the legislative analyst evaluated Caltrans’ support program practices and found that the support program was overstaffed and that it essentially had no cost‑control measures. Following these studies and reports, Caltrans implemented numerous processes to improve its project management, such as the creation of manuals, handbooks and guides, and work breakdown structures. Scope and Methodology The Joint Legislative Audit Committee (audit committee) asked the Bureau of State Audits (bureau) to assess the performance, management, efficiency, and budget of Caltrans’ support program. California State Auditor Report 2010-122 19 April 2011 In addition to reviewing and evaluating the laws, rules, and regulations significant to the audit objectives, we were asked to do the following: • Review and evaluate Caltrans’ organizational structure as it relates to the planning, development, management, delivery, and oversight of capital outlay projects to determine the impact the organizational structure has on the exchange of information between various divisions of the support program; the extent to which the organizational structure allows for a cohesive and comprehensive review of project delivery from a project’s initial budget through completion of work; whether the responsibility for the management and completion of key tasks is placed at the appropriate level within Caltrans; and the extent to which program‑level responsibilities are diffused or fragmented. • Review and evaluate Caltrans’ process for determining its annual support budget to assess how the budgets for the program of projects and individual projects fit into the support budget request. More specifically, we were to determine if the resources requested in the fiscal year 2010–11 budget are supported by capital outlay projects and all projects are included in the appropriate program of projects. • Review Caltrans’ plans or processes for staffing and overseeing the support program at headquarters and at a sample of districts to determine the relationship between staffing levels and the capital programs in those districts. • Determine whether Caltrans uses best practices to manage its support program and projects at its headquarters and at the sample of district offices. This determination should include, but not be limited to, Caltrans’ use of project schedules, project support budgets, technology, software, and regular evaluations of budgeted and actual project expenditures. • At a sample of district offices, perform the following steps on a sample of projects that support Caltrans’ fiscal year 2010–11 budget: Identifying the extent to which there are cost overruns on the support projects selected for review, including, to the extent possible, determining the cause of any support cost overruns to include a specific evaluation of the accuracy of the initial estimates for support budgets and whether any projects not initially included or approved for the budget were subsequently charged to it; reviewing and evaluating Caltrans’ time‑charging policies and practices on support projects to determine whether staff accurately charge time to the projects; determining whether the amounts planned for expenditure and the amounts actually spent on each phase of the support work 20 California State Auditor Report 2010-122 April 2011 are reasonable; comparing Caltrans’ planned and actual staffing levels and costs to industry workload standards and norms, or to an independent consultant’s estimate of staffing levels and costs; and determining whether a framework or process for evaluating the validity and efficiency of staffing requests for support‑type projects exists. • Review and assess any other issues that are significant to the support program. To assist us in our review of the support program, the bureau retained the services of NewPoint Group Management Consultants (consultant), a consulting firm with experience analyzing issues relevant to public agencies, including transportation‑related issues such as assessing STIP projections and assisting with the creation of the 2025 Transportation Development Plan, a policy document designed to guide transportation decisions and investments. To determine the impact Caltrans’ organizational structure has on the exchange of information among the various divisions of the support program, we reviewed Caltrans’ organizational charts, its 2007 Project Communication Handbook, and its established procedures, and we interviewed key personnel. Further, to review the extent to which Caltrans’ organizational structure allows for a cohesive, comprehensive review of project delivery, the extent to which program‑level responsibilities are diffused or fragmented, and whether the responsibility for the management and completion of key tasks is placed at the appropriate level within Caltrans, we first identified key points of communication and review regarding projects. We judgmentally selected 20 of the 40 projects described on page 22 and interviewed the appropriate project manager for each project. Further, we reviewed documentation to determine whether required reviews and approvals occurred at key milestones during each project’s life cycle. In addition, we assessed whether each project had a risk management plan that identified potential risks to its delivery, as Caltrans formally began requiring in March 2004. Our review found that the organizational structure appeared adequate and appropriate for the exchange of information among the various divisions of the support program. To review and evaluate Caltrans’ process for determining its annual budget for its support program, we interviewed Caltrans’ management responsible for developing the support program budget request and reviewed relevant supporting documentation. According to our review, Caltrans’ annual budget request process for the support program was reasonable. To determine whether the capital outlay projects supported the resources requested in Caltrans’ fiscal year 2010–11 budget, we reviewed the list of all projects that Caltrans submitted with its budget request and verified that it agreed with Caltrans’ XPM California State Auditor Report 2010-122 21 April 2011 database. We also reviewed relevant documentation that Caltrans used in its development of the budget request for the support program to ensure that other costs, such as overhead, were reasonable. We determined that both the list of projects and the other costs included in the budget request for Caltrans’ support program were accurate and adequately supported. The audit committee also asked us to determine if all projects were included in the appropriate program of projects; however, the commission, not Caltrans, is responsible for committing particular projects to SHOPP and STIP. Nevertheless, we reviewed a random sample of 10 active STIP and 10 active SHOPP projects that are part of Caltrans’ 2010–11 budget request and determined the projects were included in the appropriate program of projects. To review Caltrans’ plans or processes for staffing and overseeing the support program at headquarters and at a sample of districts, and to determine the relationship between staffing levels and the capital programs—the total value of projects—in the districts, we interviewed Caltrans’ staff. Further, we reviewed examples of districts’ workload forecasts and Caltrans’ resource allocation documents, which are used to staff the support program at the districts. According to our review, Caltrans’ process for staffing appears adequate. Additionally, we assessed Caltrans’ use of consultants by interviewing Caltrans and district staff, reviewing studies of the cost of using consultants at Caltrans compared to state staff, and evaluating Caltrans’ budgets and requests for consultants during fiscal years 2006–07 through 2010–11. Through interviews with managers at the districts we visited and headquarters, we determined that districts generally share staff through brokering agreements, which are essentially contracts between districts for project work. Therefore, a direct assessment of the relationship between the districts’ official staff and the capital programs within each district would not be meaningful. To determine whether Caltrans uses best practices to manage its support program and projects, we reviewed—at headquarters and a sample of districts—the use of project schedules and support budgets, evaluations of budgeted and actual expenditures, and technology and software in three districts and the Central Region. Additionally, we evaluated the process for distributing resources to the districts. We also examined the history and status of PRSM, which Caltrans asserts will change the way its projects are planned, resourced, managed, and reported once the system is implemented. To identify the performance measures it uses for the support program, which includes the support‑to‑capital ratio, we reviewed Caltrans’ current strategic plan. We interviewed key personnel to identify the methodologies used by Caltrans over the past several years to calculate the support‑to‑capital ratio. Further, we conducted our own analysis of the support‑to‑capital ratio for the 766 projects we reviewed that were completed during fiscal years 2007–08 through 2009–10. Additionally, we evaluated the fiscal year 2003–04 analysis by the legislative analyst 22 California State Auditor Report 2010-122 April 2011 regarding Caltrans’ support‑to‑capital ratio. In conducting our audit work, we identified strategies that Caltrans could use to make the support‑to‑capital ratio more meaningful in measuring the support program’s performance. Using these strategies, we calculated by district, by project size, and by program the ratio for projects that had completed construction during fiscal years 2007–08 through 2009–10. We present this analysis in the Appendix. The audit committee asked us to review a sample of projects that support Caltrans’ fiscal year 2010–11 budget to identify, in part, the extent to which support budgets have cost overruns and to determine the accuracy of support budgets. However, projects identified in Caltrans’ fiscal year 2010–11 budget request were ongoing at the time of our audit. To address the audit committee’s request adequately, we focused our review on projects that were generally complete. Specifically, we judgmentally selected—based on the degree to which support costs varied from budgeted amounts—40 projects that completed construction between fiscal years 2006–07 and 2009–10. Additionally, we visited the Fresno, Los Angeles, Oakland, and San Diego districts. To determine which districts we would review, we analyzed data from the support program’s fiscal year 2010–11 budget request. We chose these four districts based on distinct risk factors, such as a large number of projects and large support budgets, and also because they are diverse in terms of geography and size of staff. We selected 10 projects to review from each of the four districts we visited. Our consultant interviewed project managers at each of the four districts we visited to identify the cause of budget overruns for support costs, to evaluate the accuracy of the initial estimates for support budgets, and to determine whether the amounts planned for expenditure and the amounts actually spent on each phase of the support work were reasonable. These project managers included those who directly managed or who could best answer questions about each of the 40 projects we selected for review. We also analyzed state laws and commission regulations regarding the STIP. In addition, our consultant examined budgeted support costs and actual support cost expenditures for 766 projects that completed construction in fiscal years 2007–08 through 2009–10.7 However, according to the division chief, the budget information we reviewed did not consist of the original budgets but was typically what was 7 We analyzed 766 of 877 projects that completed construction in fiscal years 2007–08 through 2009–10. According to the assistant chief of Caltrans’ Division of Project Management, the remaining projects included those that were exempt from budget accountability requirements. He stated that excluding such projects allowed a more accurate representation of support expenditures against the established budget. We also excluded from our analysis 33 projects for which Caltrans’ systems did not include budget data. According to a senior transportation engineer within Caltrans’ Division of Project Management, the original budgeted support hours, costs, and other information are no longer readily available for projects that Caltrans subsequently divided into multiple projects or combined with other existing projects. California State Auditor Report 2010-122 23 April 2011 available at the time the projects were ready to begin construction. A senior transportation engineer in Caltrans’ Division of Project Management noted that some projects have scope and budget changes that occur over the life of the project and that extracting and interpreting historical budget amounts for projects become especially complicated and time consuming when projects later divide into multiple projects or combine with existing projects. He stated that individual confirmation and review of the related project information is required in these cases to ensure that the data is accurate and meaningful. Consequently, we attempted to obtain original budget and other detailed information for 36 randomly selected projects that completed construction during fiscal years 2007–08 through 2009–10 to assess the extent to which support cost overruns were due to increases in the quantity of hours charged to the project or due to the cost per hour of staff time. However, we were unable to analyze 18 of the 36 projects. In fact, Caltrans could not provide budgeted hours for 16 projects. The senior transportation engineer stated that 10 of the 16 projects were divided into multiple projects or combined with other projects such that the original budgeted support hours, costs, and other information was no longer readily available and would require individual research and analysis. He further stated that the remaining six projects were considered minor and no hours were budgeted for them. Additionally, there were two projects that were not appropriate to include in our analysis. Specifically, the senior transportation engineer stated that one project was implemented by a local agency, not Caltrans. Finally, the remaining project, according to information provided by Caltrans, was initiated before the enactment of Senate Bill 45, and was excluded from support budgeting and accountability requirements. To review Caltrans’ time‑charging policies and practices on support projects to ascertain whether staff charged time to the projects accurately, we reviewed applicable policies and interviewed department staff regarding the time‑charging practices of support program staff. For each of the 40 projects we reviewed at the four districts, we attempted to review expenditure reports and time sheets of the employees working on the projects. However, Caltrans does not track or formally document the project to which an employee is authorized to charge time during a given time period. Thus, the four districts we reviewed could not provide us with the documentation necessary for us to determine whether employees’ time charges are appropriate, a finding that we describe further in the Audit Results. In an attempt to compare Caltrans’ planned and actual staffing levels and costs to industry workload standards and norms, our consultant reviewed industry literature as well as support and capital budgets from eight other states. However, our consultant 24 California State Auditor Report 2010-122 April 2011 determined that there is no published or commonly used industry standard for the ratio of support costs to capital costs. Further, we attempted to compare Caltrans’ support cost estimates to those of local transportation agencies that performed work in various phases of projects. However, according to the division chief, Caltrans has no ability to verify local capital outlay support costs for projects when the locals are the implementing agency. Thus, we concluded that there were not enough local projects with known cost estimates to support a comparison with Caltrans’ support cost estimates. In addition, according to the division chief, Caltrans may perform work in some of the same phases as local transportation agencies under a cooperative agreement. Therefore, any conclusion related to differences in costs between Caltrans and local agencies could be unreliable. According to the 2010 California Multi‑Agency CIP Benchmarking Study (study), the ratio of project delivery costs to total construction costs—a measure similar to Caltrans’ support‑to‑capital ratio—for street projects in seven participating cities was 34 percent. As we discuss in the Audit Results, Caltrans’ ratio of support costs to capital costs equaled 35 percent over the last three fiscal years. However, because these street projects were not the subject of this audit, we did not assess the validity of the study. Finally, we attempted to analyze Caltrans’ support‑to‑capital ratio by comparing it to other states; however, state‑by‑state comparisons are challenging because of the different manner in which each state presents its data. For example, Florida’s capital expenditures include aviation costs, but Florida does not indicate what portion of its support costs goes toward these types of projects. In California, aeronautics is a separate budget category, and it is not part of capital outlay or capital outlay support. Therefore, differences in how states report both support and capital expenditures limit the value of such a comparison, which is why we do not present this information in the Audit Results. To determine whether a framework or process exists for evaluating the validity and efficiency of staffing requests for projects in the support program, we reviewed documents used to compile the support program budget by both headquarters and the districts. Our review found that such a framework exists. Finally, the audit committee requested that we review and assess any other issues that are significant to the support program. During our audit work, we identified potential issues related to travel expenditures. To evaluate whether Caltrans’ employee travel expenditures were appropriate and reasonable, we interviewed department staff and reviewed pertinent Caltrans’ and state policies and procedures. Further, we selected a sample of Caltrans employees who received large travel expense reimbursements during fiscal year 2009–10 to determine if the expenditures were reasonable and California State Auditor Report 2010-122 25 April 2011 appropriate. Our review revealed that the travel reimbursements were appropriate, reasonable, and consistent with departmental and state guidelines. In performing this audit, we relied upon various electronic data obtained from Caltrans. To facilitate our analysis of Caltrans’ project data, Caltrans provided information compiled from its various systems for projects that completed construction during fiscal years 2007–08 through 2009–10. Caltrans extracted this data from its California Transportation Improvement and Programming System (CTIPS), Transportation Accounting Management System (TRAMS), and the XPM system. From these systems, Caltrans provided information related to budgeted project support costs and hours as of the time the project was initiated and when it began construction, as well as actual expenditures and hours charged to the project. The U.S. Government Accountability Office, whose standards we follow, requires us to assess the sufficiency and appropriateness of computer‑processed data. To comply with this standard, we assessed each system separately for the purpose for which we used the data in this report. To ensure the data Caltrans provided to us was complete, we selected a sample of projects from Caltrans’ reports that detail when projects completed construction and verified that all the STIP and SHOPP projects we selected were present in the data provided to us. Additionally, to ensure the data Caltrans provided us was accurate, we compared the data provided to the actual information in Caltrans systems for a random sample of 36 projects. In addition, we were able to verify project programmed budget data in CTIPS for 26 of the 36 projects by comparing the information to actual STIP and SHOPP documents. However, we were unable to verify the project budget data for the remaining 10 projects because, according to information provided by Caltrans, these projects were generally the result of projects that subsequently divided into multiple projects or combined with existing projects. As discussed previously, information for these types of projects were no longer available. Further, we were unable to perform such a comparison for data provided from TRAMS and XPM. The TRAMS data Caltrans provided was extracted from a data warehouse that Caltrans uses to produce reports, rather than the original data produced by TRAMS. Because the production system is paperless, we could not assess reliability by tracing to and from source documents. Additionally, a test of system controls would not be meaningful because controls can be overridden in the data warehouse. In addition, the XPM data came from project management systems in Caltrans’ 12 districts, which are administered independently by the districts. It would not be practical to assess the system controls for each of these disparate systems. As a result, we concluded that the data from these three systems was of undetermined reliability for the purposes of this audit. 26 California State Auditor Report 2010-122 April 2011 Blank page inserted for reproduction purposes only. California State Auditor Report 2010-122 27 April 2011 Audit Results Caltrans Has Not Adequately Prioritized Its Monitoring of Capital Outlay Support Costs, Its Project Support Costs Exceed Budgeted Amounts, and Its Systems Contain Inaccurate Budget Data Despite a stated goal to reduce project cost overruns, the California Department of Transportation (Caltrans) has done little analysis to determine the frequency or magnitude of support cost budget overruns.8 Further, although opportunities exist to inform stakeholders of the extent of these overruns, Caltrans has not done so, limiting valuable information on the efficiency and effectiveness of the Capital Outlay Support Program (support program). According to our review of the data provided by Caltrans, 62 percent of the projects that completed construction in fiscal years 2007–08 through 2009–10 had support costs that exceeded their respective budgets. These overruns totaled more than $305 million of the $1.4 billion in total support cost expenditures for these projects that completed construction during these fiscal years. Budget overruns can deprive other projects of necessary funding, potentially causing projects to be delayed. Caltrans’ California Transportation Improvement and Programming System (CTIPS)—which Caltrans uses, in part, to capture project budgets—is currently more reliable than Caltrans indicated had been the case in prior fiscal years. Nevertheless, our review of the data provided by Caltrans for projects that completed construction during fiscal years 2007–08 through 2009–10 found that Caltrans did not ensure that this system effectively and accurately tracked a project’s total support budget. According to the chief of the Division of Transportation Programming, CTIPS was not intended, at the time those projects were programmed, to capture projects’ total support budgets.9 Further, she stated that Caltrans did not have a process for its headquarters to track projects’ total support budgets and instead relied on its districts to do so. As a result, Caltrans risks limiting its ability to compare projects’ actual support costs with the corresponding support budgets, a comparison that is critical to allow Caltrans and the public to easily assess a project’s effectiveness and efficiency related to support costs. 8 When we use the name Caltrans, we are referring to the statewide organization and its management headquartered in Sacramento. We refer to Caltrans’ districts as districts or identify the locations of specific districts’ headquarters. 9 Transportation programming is the commitment of transportation funds to be available over a period of several years for allocation to particular projects. 28 California State Auditor Report 2010-122 April 2011 In Its Management of the Support Program, Caltrans Has Not Sufficiently Prioritized the Monitoring of Support Costs Caltrans’ strategic plan covering 2007 through 2012 includes a goal to deliver quality transportation projects and services efficiently. According to its project development procedures manual (Caltrans’ manual), Caltrans measures its project delivery performance by the quality of the projects delivered and whether projects are on schedule and within budget. Caltrans’ manual also includes a stated goal to avoid project cost overruns. However, as part of its current strategic plan, Caltrans has not included an effective strategy to measure its progress in achieving these goals by specifically analyzing support budget overruns in the program. Caltrans also does not include such an analysis in its performance measures report that it provides quarterly to the Business, Transportation and Housing Agency (agency). Moreover, state law requires Caltrans to submit an annual project delivery report to the governor and Legislature on all highway projects for the State that are included Although not required to do so, in the State Transportation Improvement Program (STIP), that Caltrans has not included in its cost more than $1 million, and for which Caltrans is responsible for annual project delivery report to the project development work. However, although not required to do governor and Legislature a measure so, Caltrans has not included in this report a measure of support of support budget overruns even budget overruns even though doing so would be prudent and though doing so would be prudent transparent; rather, the report includes legally required information and transparent. pertaining to project schedule milestones. According to the assistant chief of Caltrans’ Division of Project Management (assistant division chief), the analysis of budgeted costs versus actual expenditures is handled mostly at the district level. He further stated that not much impact in terms of outcomes would occur if Caltrans included a comparison of budgeted costs to total expenditures. However, we question this perspective, given that incorporating an analysis of support cost budget overruns, including their frequency and magnitude, is an effective strategy that could assist Caltrans in achieving its goal of reducing support budget overruns. Further, by providing a similar analysis in its quarterly reports to the agency and in its annual reports to the governor and Legislature, Caltrans would be effectively providing data to enable a more systematic method of evaluating the success of the projects and would improve the transparency of the support program. According to the 2010 STIP guidelines of the California Transportation Commission (commission), Caltrans will provide the commission with a semiannual report on completed STIP projects. The assistant division chief told us that as of March 2011, Caltrans had just begun working with the commission regarding what to measure and include in the report; however, he expects that the report will compare project budgets against project California State Auditor Report 2010-122 29 April 2011 expenditures and will likely include State Highway Operation and Protection Program (SHOPP) projects as well as STIP projects. He also stated that the report will likely be developed in fiscal year 2011–12. If Caltrans chooses not to include in the semiannual report certain budgetary information, such as data pertaining to SHOPP projects, the value of the report will be diminished. Conversely, including such information would help the commission gauge the support program’s effectiveness and efficiency. Caltrans’ Capital Outlay Support Costs Generally Exceed Budgeted Amounts Given the lack of emphasis Caltrans has historically placed on analyzing support cost overruns and on providing this information to stakeholders, it is not surprising that we identified frequent, significant support cost overruns. In reviewing 766 STIP and SHOPP projects for which construction was completed during fiscal years 2007–08 through 2009–10, we noted that 476, or 62 percent, of the projects had support costs that exceeded their respective budgets.10 In conducting our analysis of the selected projects, we determined whether a project had a support cost budget overrun by performing the calculation shown in Figure 5. Figure 5 The Overrun Ratio for Capital Outlay Support Budgets Total actual Total budgeted Total budgeted – ÷ capital outlay capital outlay capital outlay support costs support costs support costs Source: Bureau of State Audits’ analysis of California Department of Transportation’s capital outlay support budget overruns. As shown in Table 2 on the following page, we found that the support cost overrun ratio for STIP projects equaled or exceeded 39 percent in each of the fiscal years 2007–08 through 2009–10, and that it equaled 46 percent for the entire three‑year period. Similarly, we noted that the support cost overrun ratio for SHOPP projects equaled or exceeded 57 percent in each of the three fiscal years, and this ratio equaled 68 percent across all three years. For example, one STIP project to widen bridges and a roadway had a support cost budget of $3 million; however, at the time construction 10 The term budget in our analysis refers to the amount programmed for project support costs by the commission. 30 California State Auditor Report 2010-122 April 2011 2 elbaT noitcurtsnoC detelpmoC tahT stcejorP 667 rof snurrevO tegduB tsoC troppuS 01–9002 hguorhT 80–7002 sraeY lacsiF STCEJORP PPOHS DNA PITS LATOT STCEJORP )PPOHS( MARGORP NOITCETORP DNA NOITAREPO YAWHGIH ETATS STCEJORP )PITS( MARGORP TNEMEVORPMI NOITATROPSNART ETATS DETELPMOC FO REBMUN FO REBMUN DETELPMOC FO REBMUN FO REBMUN DETELPMOC FO REBMUN FO REBMUN TEGDUB EVOBA STCEJORP DETELPMOC FO REBMUN TEGDUB EVOBA STCEJORP DETELPMOC FO REBMUN TEGDUB EVOBA STCEJORP DETELPMOC FO REBMUN NURREVO RETAERG YB TEGDUB STCEJORP DETELPMOC NURREVO RETAERG YB TEGDUB STCEJORP DETELPMOC NURREVO RETAERG YB TEGDUB EVOBA STCEJORP DETELPMOC LACSIF OITAR *TNECREP 02 NAHT TEGDUB EVOBA STCEJORP OITAR *TNECREP 02 NAHT TEGDUB EVOBA STCEJORP OITAR *TNECREP 02 NAHT TEGDUB STCEJORP RAEY %97 251 681 092 %09 341 071 272 %05 9 61 81 80–7002 85 601 551 432 95 79 241 812 45 9 31 61 90–8002 15 101 531 242 75 39 021 122 93 8 51 12 01–9002 %26 953 674 667 %86 333 234 117 %64 62 44 55 slatoT noitatropsnarT sti dna metsyS gnimmargorP dna tnemevorpmI noitatropsnarT ainrofilaC sti morf )snartlaC( noitatropsnarT fo tnemtrapeD ainrofilaC eht yb dedivorp atad fo sisylana ’stiduA etatS fo uaeruB :ecruoS .metsyS tnemeganaM gnitnuoccA ytilibatnuocca tegdub morf tpmexe ,sgniht rehto gnoma ,erew taht stcejorp dedulcxe eW .01–9002 hguorht 80–7002 sraey lacsfi ni noitcurtsnoc detelpmoc taht stcejorp 778 fo 667 dezylana eW :setoN .snoissimmoc noitatropsnart ytnuoc sa hcus—snartlaC ton—seititne lacol yb detnemelpmi erew taht stcejorp dna stnemeriuqer lanigiro eht retfa tcejorp rehtona htiw denibmoc eb ro ,stcejorp etarapes otni dedivid eb ,segnahc tegdub dna epocs ogrednu nac stcejorp ,tnemeganaM tcejorP fo noisiviD ’snartlaC fo feihc eht ot gnidroccA eht rof noitamrofni tegdub eht taht deton ehS .snigeb noitcurtsnoc erofeb segnahc suoirav ogrednu yllareneg stcejorp rof stegdub lanigiro eht ,tluser a sa ,taht detacidni ehS .dehsilbatse neeb sah tegdub eht retfa edam era tegdub troppus eht ot segnahc fi taht detats ehS .tegdub lanigiro eht ton ,noitcurtsnoc nigeb ot ydaer saw tcejorp eht emit eht ta elbaliava saw tahw yllacipyt si deweiver ew stcejorp 667 .evitavresnoc era elbat siht ni detneserp soitar nurrevo tegdub eht ,suhT .drawpu tegdub eht esiver ot yllacipyt era yeht ,dehsilbatse neeb sah tegdub lanigiro lacsfi ni noitcurtsnoc detelpmoc taht stcejorp lla rof stegdub tsoc troppus latot dna serutidnepxe tsoc troppus latot eht neewteb ecnereffid eht gnizylana yb oitar nurrevo tsoc troppus eht detaluclac eW taht stcejorp esoht no ylno desab si oitar nurrevo tsoc troppus ehT .oitar gnitluser eht no tcapmi retaerg a evah ,srallod fo smret ni ,stcejorp egral ,hcaorppa siht ot gnidroccA .01–9002 hguorht 80–7002 sraey .nurrevo tsoc a decneirepxe fo feihc eht ot gnidrocca ,rehtruF .stsoc tcejorp lautca rof demmargorp tnuoma eht fo tnecrep 02 nihtiw era taht secnereffid tcefler ot degnahc eb ton yam stsoc tcejorp PITS taht seriuqer wal etatS * sti revo tnecrep 02 si tcejorp PPOHS a nehw stcejorp PITS ot ralimis stcejorp PPOHS eganam ot si ecitcarp ’snartlaC ,stnemeriuqer nettirw on era ereht hguohtla ,tnemeganaM tcejorP fo noisiviD ’snartlaC .tegdub troppus California State Auditor Report 2010-122 31 April 2011 was completed on the project in fiscal year 2008–09, actual support costs totaled more than $6.5 million, representing a 114 percent increase. We also calculated the support cost overrun ratio by including projects in which actual support cost expenditures did not exceed support cost budgets. By including these projects in our analysis, we found that the support cost overruns were still significant over the three‑year period; support cost overruns for STIP projects equaled 35 percent, while support cost overruns for SHOPP projects equaled 25 percent, and the support cost overrun for all projects was 21 percent. Further, the support cost overrun ratios shown in Table 2 are likely conservative because we did not base our analysis on the original support cost budgets. Specifically, the chief of Caltrans’ Division of Project Management (division chief) stated that projects can undergo scope and budget changes and that the original budgets for projects generally undergo various changes before construction begins. She further noted that projects can either be divided into separate projects or be combined with another existing project after the original budget has been established. Moreover, she noted that projects that are divided or combined represent about 10 percent to 15 percent of Caltrans’ projects. She stated that although Caltrans has a tool to provide final budget information for projects that were divided or combined, the tool is not designed to provide initial budget information and that it is time consuming to extract the initial budget for these projects. Thus, she told us that the budget information for the 766 projects we reviewed was typically what was available at the time the projects were ready to begin construction, and the data did not represent the original budgets. As a result, we elected to base our analysis of support cost overruns on the budget available at the time the project was ready to begin construction because doing so allowed our review to include projects that were later divided or combined. The division chief further stated that if changes to the support budget are made after the original budget has been established, the changes typically revise the budget upward. Had Caltrans been able to provide us with the original support cost budgets for the projects that subsequently were either divided or combined, we could have based our analysis on those budgets, and the support cost overruns would likely have been higher than those shown in Table 2. The dollar value of support cost overruns was significant for projects that completed construction during fiscal years 2007–08 through 2009–10. Specifically, the average support cost overrun for the The average support cost STIP projects we reviewed was about $1.5 million per project, and overrun for the STIP projects we the SHOPP project overruns averaged $329,000. These overruns reviewed was about $1.5 million were substantial, particularly when considering that the average per project, while overruns for support budgets for the projects we reviewed were $4.3 million SHOPP projects averaged $329,000. and $1.3 million for STIP and SHOPP projects, respectively. 32 California State Auditor Report 2010-122 April 2011 Further, the STIP projects’ support cost overruns totaled more than $82 million, while the overruns for the SHOPP projects totaled nearly $223 million, representing a combined overrun of more than $305 million over the last three fiscal years. Total support cost expenditures for these projects that completed construction during this period were $1.4 billion. Due to a Lack of Emphasis on Support Costs, Caltrans’ Systems Do Not Provide Adequate Data on Project Support Budgets Caltrans has not ensured that its systems effectively and accurately track projects’ total budgets. According to the chief of capital improvement programs (chief of capital improvement), until 2004 for STIP projects and 2008 for SHOPP projects, Caltrans did not capture full project support budgets in a central database. Specifically, the budget data used in the analysis of support cost budgets for completed STIP and SHOPP projects were based on data from Caltrans’ CTIPS database, which Caltrans uses to capture the STIP and SHOPP documents in an automated repository and to manage the programming and allocations of funds for the STIP and SHOPP. However, according to the chief of the Division of Transportation Programming (chief of programming), CTIPS was not intended, at the time the projects we reviewed were programmed, to capture projects’ total support budgets. Moreover, the chief of capital improvement stated that federal, local, and other funding resources were incorporated inconsistently into CTIPS. He indicated that for STIP projects that were initiated before 2004, Caltrans did not consistently require project managers to be accountable for support budgets because Caltrans placed a greater priority on timely project delivery and capital construction Caltrans did not have a process for costs, with less emphasis on support costs. In fact, the chief of its headquarters to track projects’ programming stated that Caltrans did not have a process for its total support budgets and instead headquarters to track projects’ total support budgets and instead relied on its districts to do so. relied on its districts to do so. The chief of capital improvement also stated that some STIP projects that completed construction in the years we reviewed— fiscal years 2007–08 through 2009–10—were excluded from support budgeting and accountability requirements; thus, in effect, these projects have no established support budgets. Although Senate Bill 45, enacted in 1997, required the commission to set aside sufficient funding for all projects programmed in the prior year, it did not expressly require Caltrans to seek commission approval for additional funding for those projects, and the chief of capital improvement informed us that these projects do not have established support budgets. Caltrans estimated that it has 24 such projects yet to complete construction. According to Caltrans’ records available for these projects, their total value, including California State Auditor Report 2010-122 33 April 2011 the estimated support and capital costs, is nearly $250 million. Although Senate Bill 45 did not expressly require Caltrans to do so, it would have been prudent to establish and monitor the budgets for these projects as a management tool to hold project managers accountable for projects’ support costs. The division chief, who noted that she was not part of the implementation of Senate Bill 45, assumed that support budgets were not applied to these projects partly because Senate Bill 45 was a significant change, and it would have been difficult to apply the new requirements to projects already under way. Similarly, Caltrans’ SHOPP manager explained that for SHOPP projects initiated before the 2008 SHOPP was approved, the support budgets in CTIPS were informational only; thus, Caltrans did not enforce rigid accountability of these support budgets. He stated that at the time the projects were programmed by the commission, Caltrans placed a greater priority on the capital aspects of a project and less emphasis on the support costs. In addition, Caltrans’ current project management system does not keep accurate records of the support budgets for capital projects. Although Caltrans’ project management system, eXpert Project Management (XPM), provides information for support budgeting, XPM is primarily a project scheduling and resource tool. In fact, according to the assistant division chief, XPM estimates support costs by converting estimated hours to dollars using a rate matrix. He also stated that although XPM can provide initial resource estimates, the actual budget for the project is often arrived at after negotiations with local transportation agencies. As a result, we determined that XPM data was not an authoritative source to use in a comparison of budgeted support costs to actual support costs. Further, Caltrans’ practice of splitting and combining projects makes Caltrans’ practice of splitting and it difficult to compare actual support costs to budgeted support costs. combining projects makes it difficult For example, as we discuss later, we obtained detailed data for a sample to compare actual support costs to of projects completed during fiscal years 2007–08 through 2009–10 to budgeted support costs. determine the factors causing support cost overruns. Caltrans provided data for 36 projects; however, according to a senior transportation engineer within Caltrans’ Project Management Division, the agency could not provide accurate budget data for 10 of the projects we requested because of the time required to perform the research and analysis needed to gather the requested data. A senior transportation engineer within Caltrans’ project management division stated that these 10 projects had been split into multiple projects or combined with other projects such that the original budgeted support hours, costs, and other information were no longer readily available and would require research and analysis on each project. 34 California State Auditor Report 2010-122 April 2011 The assistant division chief told us that Caltrans plans to implement new business processes and project coding in its upcoming project management system called Project Resource and Schedule Management (PRSM), which should allow Caltrans to better Until it better emphasizes and compare budgeted support costs with actual support costs for focuses on support cost budgets, projects that are split or combined. However, until it better Caltrans risks reporting project emphasizes and focuses on support cost budgets, Caltrans risks support costs inaccurately and reporting project support costs inaccurately, regardless of whether limits the ability of the public to PRSM is implemented effectively. In addition, Caltrans limits the easily assess a project’s adherence ability of the public to easily assess a project’s adherence to its to its budgeted support costs. budgeted support costs. According to its SHOPP manager, Caltrans is now placing greater emphasis on looking at entire project budgets, including support costs, and it is now holding the districts and project managers accountable for their entire project budgets that CTIPS identifies. Further, our review of 10 active SHOPP projects and 10 active STIP projects revealed that the data in CTIPS had reliable budget information. However, both the chief of capital improvement and the SHOPP manager indicated that the majority of STIP and SHOPP projects initiated in 2004 and 2008, respectively, have not completed construction. As a result, we cannot identify the impact Caltrans’ new emphasis on support costs and accountability will have on support cost overruns. Project Managers Identified Multiple Reasons for Support Cost Overruns To further review the reasons for support cost budget overruns, we visited the Fresno, Los Angeles, Oakland, and San Diego districts. We determined that we would review an additional 10 projects that had completed construction in each of the four districts we visited, for a total of 40 projects. We judgmentally selected these projects based on the differences in the budgeted and actual support to capital costs ratio (support‑to‑capital ratio). As we discuss later in this report, the support‑to‑capital ratio is the total support costs divided by the total capital costs. For each district, we selected five projects for the SHOPP and five projects from the STIP. Of the five projects selected for each program, we selected four projects in which the actual support‑to‑capital ratio deviated significantly from its budgeted ratio relative to the other projects that completed construction in the district. We also selected one project from each program in which the actual support‑to‑capital ratio remained near or below the budgeted ratio. The project managers identified various causes for support cost overruns. Generally, the project managers indicated that each project had unique circumstances that were not contemplated in California State Auditor Report 2010-122 35 April 2011 developing the original support cost budgets and that could have resulted in budget overruns. For example, some project managers attributed support cost overruns to changes in project scope; contractor arbitration or litigation; design changes or project redesigns; funding delays; project delays; support costs not adjusted for escalations; unforeseen environmental issues; unforeseen right‑of‑way issues, such as property condemnations; and unforeseen project complexity, such as local road corrections and relocations. Two Factors Generally Contribute to Support Cost Overruns Differences between the budgeted and actual support costs of a project generally can be caused by one or a combination of two factors—a difference between the expected and actual quantity of hours charged by staff to the project, or a difference between the expected and actual hourly cost of staff time. We found that an We found that an increase in increase in the hourly rate for support costs was the primary cause the hourly rate for support costs for the cost overruns in the projects we reviewed. Specifically, the was the primary cause for the annual salaries for certain Caltrans employees, including engineers, cost overruns in the projects increased by more than 40 percent during this time. We also we reviewed. found that project managers often did not update their budgets for support costs to account for such cost increases as salary escalation. Finally, we found that some overruns in budgeted hours may be due to inadequate project support budgeting. According to the division chief, cost overruns take funding away from other projects, leading to construction delays. Support Cost Overruns Were Due Primarily to Increases in Labor Costs To determine the factors causing project cost overruns, we selected and reviewed a random sample of 18 projects—separate from the selection of 40 projects discussed earlier—that completed construction during fiscal years 2007–08 through 2009–10. Table 3 on the following page displays how the actual expenditures for project support differed from the initial approved budgets for the 18 projects we reviewed. To calculate the difference in cost caused by more or fewer hours being charged to a project than were allocated, we multiplied the difference between the budgeted and actual hours by the original hourly support cost rate. Conversely, to determine the difference caused by a change in the hourly support cost rate, we multiplied the difference between the budgeted and actual hourly rates by the actual support hours charged to the project. 36 California State Auditor Report 2010-122 April 2011 3 elbaT stcejorP noitatropsnarT fo tnemtrapeD ainrofilaC eht rof serutidnepxE lautcA dna stegduB troppuS lanigirO neewteB secnereffiD fo sesuaC 01–9002 hguorhT 80–7002 sraeY lacsiF gniruD detelpmoC TSOC TSOC LAUTCA DETEGDUB REDNU RO REVO SRUOH LAUTCA LANIGIRO RO REVO SERUTIDNEPXE ECNEREFFID ECNEREFFID REP TSOC REP TSOC EGATNECREP( TEGDUB SRUOH SRUOH EGATNECREP( TEGDUB REDNU LAUTCA LANIGIRO †ETAR OT EUD †SRUOH OT EUD RUOH RUOH )TEGDUB REDNU RO REVO DEGRAHC DETEGDUB )TEGDUB REDNU RO REVO SERUTIDNEPXE *TEGDUB NOITPIRCSED TCEJORP TCIRTSID tnuomA detegduB gnideecxE secruoseR ffatS fo tsoC eht ot yliramirP euD snurrevO tsoC troppuS tnacfiingiS htiW stcejorP 099,521,1$ 004,870,1$ 29$ 55$ %781 376,91 991,03 625,01 %283 093,402,2$ 093,187,2$ 000,775$ aera tser gnidliubeR 60 049,732 605,98 59 34 58 790,2 755,4 064,2 213 644,723 644,234 000,501 ecnef gnillatsnI 70 081,704 329,333 97 35 66 242,6 286,51 044,9 741 301,147 301,642,1 000,505 noitagirri dna gnitnalP 70 461,656 975,681‑ 97 52 83‑ 284,7‑ 780,21 965,91 69 585,964 585,759 000,884 noitagirri dna gnitnalP 40 196,194 704,711 101 26 81 788,1 425,21 736,01 29 890,906 890,172,1 000,266 liar draug gnillatsnI 70 668,165,7 313,197‑ 011 45 01‑ 185,41‑ 329,531 405,051 38 355,077,6 355,839,41 000,861,8 yawhgih gninediw dna gningilaeR 70 dna enal nrut gnitcurtsnoC 078,246 629,611‑ 38 44 41‑ 176,2‑ 653,61 720,91 36 449,525 449,853,1 000,338 60 langis cffiart gnillatsni 484,315,2 479,938‑ 59 25 12‑ 190,61‑ 052,95 143,57 34 015,376,1 015,606,5 000,339,3 yawhgih gninediW 40 270,438 364,81‑ 88 36 1‑ 392‑ 065,33 358,33 83 806,518 806,849,2 000,331,2 noitazilibats gnipolS 70 499,205 668,13‑ 29 07 2‑ 554‑ 374,22 829,22 92 821,174 821,870,2 000,706,1 yawdaor gnitatilibaheR 40 tnuomA detegduB gnideecxE tcejorP eht ot degrahC sruoH fo ytitnauQ eht ot yliramirP euD snurrevO tsoC troppuS tnacfiingiS htiW stcejorP 063,232 625,485 88 27 721 001,8 474,41 473,6 871 688,618 688,672,1 000,064 slangis cffiart gnillatsnI 11 178,31 841,343 89 79 76 345,3 808,8 562,5 07 910,753 910,768 000,015 reirrab gnitcurtsnoC 40 537,460,1‑ 492,974,1 68 541 921 181,01 570,81 498,7 63 955,414 955,165,1 000,741,1 yawdaor gnitatilibaheR 40 531,743,1‑ 284,289,1 37 311 211 094,71 440,33 455,51 63 743,536 743,893,2 000,367,1 metsys gniretem cffiart gnillatsnI 40 snurrednU tsoC troppuS htiW stcejorP ro snurrevO tsoC troppuS tnacfiingisnI htiW stcejorP 202,071 382,251‑ 59 87 61‑ 749,1‑ 969,9 619,11 2 919,71 919,949 000,239 yawdaor gnitatilibaheR 11 055,541‑ 195,59 69 521 81 767 120,5 452,4 9‑ 959,94‑ 140,084 000,035 langis nrut gnillatsnI 70 186,302 344,123‑ 89 67 23‑ 552,4‑ 802,9 364,31 21‑ 367,711‑ 732,998 000,710,1 yawdaor gnitatilibaheR 60 076,469,1‑ 650,920,2‑ 39 261 13‑ 194,21‑ 343,82 438,04 06‑ 527,399,3‑ 572,936,2 000,336,6 egdirb gnitatilibaheR 11 472,270,11$ 573,616,1$ secnereffiD tsoC latoT %78 %31 etaR dna sruoH ot euD ecnereffiD tsoC latoT fo noitroporP tcejorP trepXe ,metsyS gnimmargorP dna tnemevorpmI noitatropsnarT ainrofilaC sti morf )snartlaC( noitatropsnarT fo tnemtrapeD ainrofilaC eht yb dedivorp atad fo sisylana ’stiduA etatS fo uaeruB :ecruoS .metsyS tnemeganaM gnitnuoccA noitatropsnarT dna ,metsyS tnemeganaM stcejorp 81 eht ot weiver ruo detimil ew ,stcejorp 63 eht fo 81 rof atad niatrec dekcal snartlaC esuaceb ,ygolodohteM dna epocS ruo ni dessucsid sa ,revewoH .modnar ta stcejorp 63 detceles yllanigiro eW :etoN .elbat siht ni detneserp .noissimmoC noitatropsnarT ainrofilaC eht yb devorppa tcejorp eht rof noitacolla laitini eht ot srefer tegdub lanigirO * eseht dednuor ew ,sesoprup noitatneserp rof ,esuaceb snmuloc ruoh rep tsoc lautca dna detegdub eht ni nwohs setar ylruoh eht gnisu devired yltcaxe eb tonnac snmuloc eseht ni detneserp stnuoma ehT † .rallod tseraen eht ot setar ylruoh California State Auditor Report 2010-122 37 April 2011 As Table 3 shows, in 14 of the 18 projects, the actual support costs In 14 of the 18 projects we reviewed, greatly exceeded the project’s original budget by amounts ranging the actual support costs greatly from 29 percent to 382 percent. Of the remaining four projects, exceeded the projects’ original one was within 2 percent of meeting its budget, and another budget by amounts ranging from three completed construction under budget. An increase in 29 percent to 382 percent—in the hourly rate for support costs was the primary cause for the one instance, the project was cost overruns in 10 of the 14 projects that greatly exceeded their about 14,600 hours under budget budgets. For example, in one instance, the project was about yet cost nearly $6.8 million more 14,600 hours under budget but exceeded its budgeted cost by than budgeted. nearly $6.8 million, representing a cost overrun of 83 percent. The changes in the support cost hourly rate were due in part to salary increases agreed to in a memorandum of understanding effective from July 2003 through July 2008 between the State and the Professional Engineers in California Government. Specifically, from fiscal years 2005–06 through 2008–09, the annual salaries of certain Caltrans employees, including engineers, increased by more than 40 percent. According to the memorandum of understanding that granted the salary increases, the purpose of the increases was to establish pay parity between the state engineering classifications and their counterparts in California’s larger local agencies and the University of California. Project Managers Did Not Always Update Support Cost Budgets We also found that project managers often did not update their budgets for support costs to account for cost increases, such as salary increases. According to Caltrans’ 2003 Project Changes Handbook, districts must submit a programming change request to Caltrans’ Division of Project Management for processing and approval when districts need a programming change, such as a change to a project’s budget. Of the 40 projects we reviewed at four districts, we identified 35 projects for which actual expenditures exceeded support budgets by 20 percent or more; however, the districts did not submit programming change requests for any of these projects to seek a modification to the budget for support costs. According to Caltrans’ SHOPP manager, before 2008, SHOPP projects were not included in the process for project change requests. Although these projects were not included in the project change request process, we identified that three of the 17 SHOPP projects with budget overruns greater than 20 percent had modifications to the support budget. Nevertheless, given the frequency of support cost overruns, we would have expected project managers to update SHOPP project budgets and, furthermore, to submit programming change requests for STIP projects when appropriate to reflect increased support costs. 38 California State Auditor Report 2010-122 April 2011 In addition, some of the districts’ project managers told us that they monitored their budgets based primarily on the hours charged and not the dollars spent. According to Caltrans’ 2007 Project Management Handbook, project managers are responsible for controlling both the support budgets and the capital budgets for projects. Thus, in monitoring their projects’ support budgets, we would expect project managers to monitor both the hours and the dollars spent. For example, when Caltrans posts contract progress estimates for capital costs on its Web site, it reports on both the quantity of work performed Project managers for 12 of and dollar amounts spent. However, project managers for 12 of the the 40 projects we reviewed 40 projects we reviewed at the four districts indicated that they at the four districts said they monitored their budgets based primarily on actual versus budgeted monitored their budgets based personnel hours or on a combination of actual versus budgeted primarily on personnel hours personnel hours and actual versus planned project schedule progress, or schedule progress without without comparing actual costs to budgeted costs. If a project manager comparing costs. does not pay attention to costs, escalations in the rate paid per hour could cause a support cost overrun, even if the project remains under its budgeted hours. Moreover, overruns require additional funding originally allocated for other projects, which may result in the delay of those projects. According to the division chief, until about five years ago, Caltrans placed a greater emphasis on ensuring that capital costs were within budget, because these costs were generally the larger part of the project’s total budget. However, she explained that more recently Caltrans has been increasing its emphasis on managing support costs separately from capital costs, due in part to increasing accountability requirements from external factors such as bond funding; Caltrans’ increased accountability to its local funding partners, such as counties using funding from local ballot measures to support projects; and the impact of engineers’ recent salary increases on support budgets, as discussed previously. In February 2010 Caltrans issued a memorandum to the districts describing the approach Caltrans will take to monitor support costs relative to their budgets. The memorandum, effective July 2010, requires Caltrans to produce quarterly, a list of projects that are projected to exceed their budgets and to distribute this list to the districts. The memorandum further requires the project managers to prepare funding plans to address these potential cost overruns. According to the memorandum, the districts may approve funding plans for projects that have not yet exceeded their budgets; however, if a project is already over budget, Caltrans must approve the funding plan. Without approval, Caltrans will not allow further expenditures to be charged to the project. According to the division chief, part of the reason for the memorandum was that, although Caltrans was having quarterly meetings with the districts and receiving project updates, it was still finding out about issues with some projects too late in the process to ensure that the districts were proactively addressing California State Auditor Report 2010-122 39 April 2011 concerns, such as by revising the initial budget before an overrun occurred. However, because the memorandum is a recent policy development that affects only the management of active projects, we cannot determine whether this policy is effective at controlling support budget overruns until some of the projects reach completion. Overruns in Support Hours May Be Due to Inadequate Use of Available Detail When Budgeting for Project Support In our analysis of cost overruns, we found that four of the 18 projects Four of the 18 projects that greatly that greatly exceeded their support budgets did so due primarily exceeded their support budgets did to an increase in the quantity of hours charged to the project. In so due primarily to an increase in one instance, a project in the Oakland district exceeded its original the quantity of hours charged to the support budget by more than 10,000 hours, or 129 percent. According project—one project exceeded its to the chief of the Oakland district’s Office of SHOPP Design, this budget by more than 10,000 hours. project was to rehabilitate a local street. He explained that such projects are typically much more labor intensive than freeway rehabilitation projects. He noted that extensive surveying over the entire project length, consisting of 2.3 miles, and coordination with the local government, business owners, and residents was necessary. We also found that inadequate project budgeting for support may have contributed to overruns in budgeted support hours for other projects as well. Specifically, for 10 of the 40 projects we reviewed at four districts, the project managers indicated that they used a “top‑down” approach—in which the budget for support was based on a percentage of the total capital costs—to develop the support budget when the project was ready to contract for construction. However, according to its publication titled How Caltrans Builds Projects, at the point that a project is ready to contract for construction, Caltrans should have completed the environmental and design work and developed a complete set of project plans. At this stage, project managers could use a more detailed approach to developing the support budget, based on tasks, such as those included in a project work plan to establish more accurate support cost budgets. In fact, the Project Management Institute’s A Guide to the Project Management Body of Knowledge (PMBOK Guide) states that “activity cost estimates” are factors in a budget.11 Without consistent use of detailed budgeting at the point that a project is ready for construction, Caltrans risks misstating its project support budgets. According to the division chief, when budgets are overstated, fewer projects receive funding, and when budgets are understated, the subsequent overruns take funding away from other projects, leading to construction delays. 11 Recognized for its development of standards for project management, the Project Management Institute publishes the PMBOK, which provides guidelines for managing individual projects. 40 California State Auditor Report 2010-122 April 2011 Because the Commission Does Not Track or Review Construction Support Cost Overruns for STIP Projects, Neither Counties nor Caltrans Is Held Accountable When Construction Support Costs Exceed Estimates Caltrans projects are budgeted based on estimates of the project development phases, such as the environmental phase, the design phase, the right‑of‑way phase, and the construction phase—including construction support. After the allocation of construction funds to the project by the commission, project budgets may not change to reflect variations in construction expenditures, among other things, except in the cases of supplemental project allocations made by the commission. Caltrans’ chief of capital improvement programs informed us that support budgets are never updated after construction begins. Without supplemental project allocations, support budget overruns during the construction phase are not tracked or reviewed by the commission. Additionally, although support cost overruns during the project development phases, such as the environmental and design phases, reduce the funds available to the entity with the overrun, such as Overruns during the construction a county or Caltrans, overruns during the construction phase do phase do not reduce funds available not reduce funds available to the entity. According to state law, the to the entity, however, such commission may not change project costs to reflect changes in overruns impair the State’s ability construction expenditures once a project is in construction without a to undertake future projects. supplemental allocation. Therefore, the commission cannot adjust the funds available to the entity because the commission does not review cost overruns during this phase. These two conditions insulate Caltrans and counties from the consequences of construction cost overruns, including construction support cost overruns. We examined 55 STIP projects that completed construction during fiscal years 2007–08 through 2009–10 and found that 48 percent of the total support costs for these projects were incurred during the construction phase; and that the overrun ratio for these projects was 46 percent. Given the limited funds available for STIP projects within the State, overruns on current projects impair the State’s ability to undertake future projects. We believe that increased oversight and accountability during the construction phase of STIP projects could reduce these cost overruns. Caltrans Has Generally Not Met Its Goal for Its Support‑to‑Capital Ratio, Has Calculated This Ratio Inconsistently, and Could Improve the Manner in Which It Measures Efficiency Caltrans has established a goal of reducing support costs to represent a ratio of 32 percent of the total capital cost for the support program. Although it has identified this as an objective in its current strategic plan, Caltrans has historically failed to use a California State Auditor Report 2010-122 41 April 2011 consistent method to calculate this ratio, decreasing the value of the ratio for assessing its performance in managing the support program over time. We calculated Caltrans’ support‑to‑capital ratios for completed projects for the last three fiscal years and determined that it generally did not meet its goal. Finally, the support‑to‑capital ratio could be defined more precisely to better measure efficiency, given that a project’s size and type can have a large impact on the resulting ratio. Although Caltrans Has Established a Goal for Its Support‑to‑Capital Ratio, It Has Used Inconsistent Time Frames in Its Calculations and Could Improve Its Methodology In its current strategic plan, Caltrans has a performance measure that compares support costs to capital costs, referred to as the support‑to‑capital ratio. As Figure 6 indicates, this performance measure, which measures past efficiency and is calculated only for those projects that have completed construction, allows a comparison of the ratio of capital outlay support costs to capital outlay expenditures over the life of an individual project or single set of projects. Figure 6 The Support‑to‑Capital Ratio Total project or ÷ Total project or program capital outlay program capital costs support costs Source: California Department of Transportation (Caltrans). Note: Capital outlay support costs include the sum of environmental costs, design and engineering costs, right‑of‑way costs, and construction management costs. Capital costs include the sum of construction capital costs and right‑of‑way capital costs. The assistant chief of Caltrans’ Division of Project Management explained that—at the program level—the ratio is based on the aggregate of the support and capital costs used in the ratios calculated for the individual projects. Caltrans calculates this ratio only for those projects that complete construction in a given year. As Caltrans’ current strategic plan states, one of the objectives that Caltrans has identified to assist it in achieving its goal of delivering quality transportation projects and services is to reduce the support‑to‑capital ratio to 32 percent or lower. However, in reviewing Caltrans’ performance data, which is published quarterly on its Web site, we noted that Caltrans reported that its program‑level support‑to‑capital ratios consistently exceeded the 32 percent target for fiscal years 2004–05 through 2009–10. During this time, Caltrans reported support‑to‑capital ratios that ranged from 33.4 percent in fiscal year 2004–05 to 37.3 percent in fiscal year 2009–10. 42 California State Auditor Report 2010-122 April 2011 Since Caltrans first began reporting the support‑to‑capital ratio for the support program in 1995, it has used inconsistent methodologies in calculating the ratio. Specifically, the assistant division chief explained that until fiscal year 2001‑02 Caltrans’ methodology for deriving the support‑to‑capital ratio was to total the program’s annual support expenditures and divide this amount by the program’s total construction expenditures within the same fiscal year. This methodology was unique to this reporting and had the advantages of providing real‑time program level data specific to a budget year. However, he explained that this methodology was inadequate for several reasons, including that readers of the report misinterpreted the ratio to represent the support‑to‑capital ratio over the life of a project rather than for the support program in a given fiscal year. Consequently, he explained that in subsequent performance reporting to the agency, Caltrans used another method for fiscal years 2004–05 through 2009–10 that aggregated the total support and capital costs for all projects that completed construction within the respective fiscal year as well as the preceding four fiscal years, referred to as the five‑year cumulative method. The assistant division chief explained that this method was used to smooth out any variability in the ratio from year to year and to produce a smoother trend line over time. Although it was Caltrans’ intent to use the five‑year cumulative method, it failed to do so consistently. According to the division chief, Caltrans used three years of Caltrans’ use of inconsistent data when calculating the ratio in fiscal year 2004–05, four years methods to derive the of data for fiscal year 2005–06, and an entirely different method to support‑to‑capital ratio over calculate the ratio in fiscal years 2008–09 and 2009–10. Caltrans’ use the years has likely hindered of inconsistent methods to derive the support‑to‑capital ratio over any meaningful analysis of the the years has likely hindered any meaningful analysis of the support support program’s performance. program’s performance. Additionally, state law requires Caltrans to report annually certain information regarding the support program to the governor and Legislature. Although this law does not prescribe a specific method, and we could not identify any legally prescribed methods for Caltrans to use in deriving the support‑to‑capital ratio included in this report, we believe Caltrans could provide more meaningful information to the governor and Legislature by including support costs incurred through the construction phase in its calculation. Specifically, according to the division chief, Caltrans currently calculates the support‑to‑capital ratio based only on costs incurred up to the award of the construction contract, which includes costs associated with the environmental, design, and right‑of‑way components of a project’s life cycle. Caltrans uses this methodology to calculate the support‑to‑capital ratio that it reports for STIP projects on an annual basis. However, the current report provides the governor and Legislature with an indication of Caltrans’ current efficiency only as it relates to ongoing STIP projects. To allow the governor and Legislature to evaluate more effectively the support program’s performance, Caltrans should expand its methodology to include a separate support‑to‑capital ratio California State Auditor Report 2010-122 43 April 2011 for STIP projects that have completed construction, which would provide an indication of past efficiency, and another ratio using the same methodology for SHOPP projects, which would provide more complete information on the support program in general. For instance, we conducted our own assessment of 766 projects that completed construction during fiscal years 2007–08 through 2009–10 and for which construction costs were available. Table 4 presents the support‑to‑capital ratio for STIP and for SHOPP as well as the fiscal year in which the projects were completed. As Table 4 shows, Caltrans failed to meet its 32 percent goal in all but one year— fiscal year 2009–10—for the STIP. Adopting a similar methodology in calculating the support‑to‑capital ratio for its projects will enable Caltrans to provide the governor and Legislature with more meaningful information with which to evaluate the support program’s performance. Table 4 Support‑to‑Capital Ratios for 766 Projects That Completed Construction Fiscal Years 2007–08 Through 2009–10 STATE TRANSPORTATION STATE HIGHWAY OPERATION AND IMPROVEMENT PROGRAM PROTECTION PROGRAM (STIP) PROJECTS (SHOPP) PROJECTS TOTAL STIP AND SHOPP PROJECTS NUMBER OF NUMBER OF NUMBER OF SUPPORT‑TO‑CAPITAL COMPLETED SUPPORT‑TO‑CAPITAL COMPLETED SUPPORT‑TO‑CAPITAL COMPLETED FISCAL YEAR RATIO PROJECTS RATIO PROJECTS RATIO PROJECTS 2007–08 35% 18 39% 272 38% 290 2008–09 43 16 34 218 36 234 2009–10 30 21 34 221 33 242 Totals 34% 55 36% 711 35% 766 Source: Bureau of State Audits’ analysis of data provided by the California Department of Transportation’s (Caltrans) Transportation Accounting Management System for projects completed in each fiscal year. Notes: We analyzed 766 of 877 projects that completed construction in fiscal years 2007–08 through 2009–10. We excluded projects that, among other things, were exempt from budget accountability requirements and projects that were implemented by local entities—not by Caltrans—such as county transportation commissions. We calculated the support‑to‑capital ratio by analyzing the difference between the total support cost expenditures and total capital cost expenditures for the 766 projects that completed construction in the fiscal years 2007–08 through 2009–10. Caltrans’ Support‑to‑Capital Ratio Has Limitations in Measuring Performance Although Caltrans aims to reduce the support‑to‑capital ratio to 32 percent or lower by 2012, this performance measure could be more effective if Caltrans refined it to account for project size and scope. As Table 5 on the following page demonstrates, smaller projects—those with less than $1 million in total capital costs—had a 103 percent support‑to‑capital ratio for STIP and SHOPP projects completed during fiscal years 2007–08 through 2009–10. However, large projects—those with $50 million or more in total capital costs—had a support‑to‑capital 44 California State Auditor Report 2010-122 April 2011 5 elbaT noitcurtsnoC detelpmoC tahT seziS tnereffiD fo stcejorP 667 rof soitaR latipaC‑ot‑troppuS tcejorP 01–9002 hguorhT 80–7002 sraeY lacsiF NOITAREPO YAWHGIH ETATS NOITATROPSNART ETATS DNA PITS LATOT MARGORP NOITCETORP DNA MARGORP TNEMEVORPMI STCEJORP PPOHS STCEJORP )PPOHS( STCEJORP )PITS( FO REBMUN FO REBMUN FO REBMUN EGATNECREP EGATNECREP LATIPAC‑OT‑TROPPUS DETELPMOC LATIPAC‑OT‑TROPPUS DETELPMOC LATIPAC‑OT‑TROPPUS DETELPMOC LATOT FO LATOT FO OITAR STCEJORP OITAR STCEJORP OITAR STCEJORP EULAV LATIPAC STCEJORP *EZIS TCEJORP %301 652 %201 542 %911 11 %3 %33 )noillim 1$ naht ssel( llamS noillim 1$ neewteb( muideM 63 694 53 164 14 53 67 56 )noillim 05$ dna 42 41 51 5 82 9 12 2 )erom ro noillim 05$( egraL %53 667 %63 117 %43 55 %001 %001 slatoT .metsyS tnemeganaM gnitnuoccA noitatropsnarT eht morf detareneg atad )snartlaC( s’noitatropsnarT fo tnemtrapeD ainrofilaC eht fo sisylana ’stiduA etatS fo uaeruB :ecruoS ytilibatnuocca tegdub morf tpmexe erew ,sgniht rehto gnoma ,taht stcejorp dedulcxe eW .01–9002 hguorht 80–7002 sraey lacsfi ni noitcurtsnoc detelpmoc taht stcejorp 778 fo 667 dezylana eW :etoN .snoissimmoc noitatropsnart ytnuoc sa hcus—snartlaC yb ton—seititne lacol yb detnemelpmi erew taht stcejorp dna stnemeriuqer ni noitcurtsnoc detelpmoc taht stcejorp 667 eht rof serutidnepxe tsoc latipac latot eht dna serutidnepxe tsoc troppus latot eht neewteb ecnereffid eht gnizylana yb oitar latipac‑ot‑troppus eht detaluclac eW .01–9002 hguorht 80–7002 sraey lacsfi .stsoc latipac yaw‑fo‑thgir dna stsoc latipac fo mus eht si ezis tcejorP * California State Auditor Report 2010-122 45 April 2011 ratio of 24 percent. According to the assistant division chief, smaller projects will likely have higher support‑to‑capital ratios because a certain amount of support is necessary regardless of project size. For As project size increases, the this reason, as project size increases, the amount of support needed amount of support needed relative relative to capital costs decreases. to capital costs decreases. Additionally, the analysis by the Legislative Analyst’s Office (legislative analyst) of Caltrans’ fiscal year 2003–04 budget indicated that grouping STIP and SHOPP projects together to measure performance would intermingle projects with different expected support‑to‑capital ratios, rendering any targets meaningless. Thus, the legislative analyst recommended that Caltrans calculate separate measures for STIP and SHOPP projects. As a result, we expected that Caltrans would have established different performance measures based on project size and project type. However, according to the division chief, Caltrans had developed those measures internally but had not reported them on its Web site or in formal documents such as its report to the Legislature and the governor. She stated Caltrans is considering reporting on these measures in the future. Refining the use of the support‑to‑capital ratios, as we suggest in Table 5, would allow Caltrans to monitor support costs more effectively by helping it identify projects that have potential issues related to support costs. Further, in its analysis of Caltrans’ fiscal year 2003–04 budget, the legislative analyst concluded that while calculating the support‑to‑capital ratio for all the projects that completed construction in a given year should be the primary measure of the support program’s efficiency, this measure demonstrates Caltrans’ past efficiency rather than its current performance. Therefore, the legislative analyst suggested a second measure to provide some indication of Caltrans’ performance on projects that are currently ongoing. This measure would provide the aggregate performance for all projects that began construction in a given year and would include projected support cost overruns as well as projected total capital costs. In addition, a draft 2011 study commissioned by Caltrans and prepared by a professor from the Department of Civil and Environmental Engineering at the University of California, Davis, suggests that the support‑to‑capital ratio is limited, in part because it does not reflect agency productivity or efficiency. Instead, the study argues that Caltrans should use a measure based on a productivity index, which can offer a means for improving insight into the trends in labor performance over time, as well as helping to identify factors that drive changes. Generally, the productivity index uses the number of hours worked as an input and the number of projects delivered as an output, using a base year for comparison. This productivity index, together with the use of earned value metrics, described later in this report, 46 California State Auditor Report 2010-122 April 2011 and the more comprehensive support‑to‑capital ratio calculations we suggest that Caltrans implement, could assist Caltrans in better gauging its efficiency and effectiveness. According to the division chief, Caltrans’ Division of Project Management plans to implement basic concepts of the recommendations made in the 2011 study, but it needs to come up with a more detailed plan on how to implement the recommendations. She further explained that it is the division’s intent to present a detailed plan to Caltrans management by the summer or fall of 2011. In the meantime, Caltrans continues to use the support‑to‑capital ratio to measure performance, and the Legislature and the public will continue to assess Caltrans’ performance based on that measure. The Appendix includes an analysis of the support‑to‑capital ratio for each of Caltrans’ 12 districts. Changes to Caltrans’ Internal Controls and Project Monitoring Could Improve Its Management of Support Costs According to our review, Caltrans lacks strong internal controls to ensure that staff charge time to support projects appropriately. Further, although Caltrans has established project monitoring and performance metrics, it has not comprehensively implemented these tools. For example, consistent use of earned value metrics could help Caltrans to better manage its support projects. Caltrans Lacks Strong Internal Controls to Ensure That Staff Always Charge Time to the Correct Project Caltrans’ time‑reporting system Caltrans’ time‑reporting system does not prevent its employees does not prevent its employees from from charging time to projects to which they are not assigned, charging time to projects to which and it lacks strong internal controls to ensure that its employees they are not assigned. charge time appropriately. According to Caltrans’ policy regarding departmental charging practices, accurate charging practices are essential for providing financial information in a cost‑effective manner, maintaining financial control of budgets, managing projects effectively, and accurately billing the federal government and local agencies for reimbursable work, among other things. Caltrans uses an online reporting system into which employees manually enter their hours worked. However, according to the office chief of transaction services, Caltrans’ time‑reporting system does not have a mechanism in place to prevent employees from charging to projects to which they were not previously assigned, as long as those projects are active. Caltrans’ policies require that its employees accurately fill out their time sheets, and the employee’s functional manager or supervisor is responsible for ensuring that all time sheets are reviewed and approved on a weekly basis. California State Auditor Report 2010-122 47 April 2011 According to the deputy director of the Fresno district’s Division of Project Management (Fresno deputy director), although project managers have no official authority over reviewing or approving employees’ time sheets, Caltrans relies on the project managers to intercept any improper charges that the functional managers fail to identify. He explained that the project manager reviews expenditure reports that list all employees who have charged time to their project. However, Fresno’s deputy director further stated that since the project resources are assigned to various functional units—consisting of a group of employees—there is There is neither a list that neither a list that documents the names of individual employees documents the names of individual who are authorized to charge to a specific project nor a list that employees who are authorized tracks changes in the projects employees are authorized to charge. to charge to a specific project nor Moreover, he explained that project managers manage 10 to a list that tracks changes in the 20 projects on average, making it difficult to catch every improper projects employees are authorized charge and to constantly be aware of who is authorized to charge to charge. to a particular project. For example, we noted that many staff may charge to a given project. At one district, in one fiscal year, 14 people charged time to one project, while in that same fiscal year, more than 120 people charged time to another project. Further, the chief of Caltrans’ Division of Accounting noted that it would be difficult to identify improper charges based on earlier time sheets, because such charges may have been entered onto those time sheets in error and not subsequently identified. Without a system that distinguishes an authorized charge from an unauthorized charge and prevents mischarging, Caltrans does not have an efficient way to ensure that employees are always charging time to the appropriate project. For example, according to the chief of the Oakland district’s Division of Project Management and Bay Area Toll Authority support services (Oakland division chief), if an improper charge is suspected, the cost of the research required to ascertain what work the employee was doing that is in question and whether the charge is correct may exceed the potential savings. He explained that it would not be reasonable to conduct this research for every questionable time charge and that there is at present no other way to confirm the appropriateness of time charges. According to Caltrans’ assistant division chief, who is also the PRSM project director, the PRSM system will interface with Caltrans’ time‑charging system to facilitate more accurate time charges to projects. Specifically, according to its guidelines for the delivery of capital projects, the PRSM system will assign every resource a unique cost center number. Cost centers are organized into functional categories, such as project development, engineering services, construction, transportation planning, and right‑of‑way acquisition, and these categories appear in a pull‑down list when staff record their time. The guidelines note that this time‑charging structure needs to be in place so that each district’s resources are standardized and similar to those of other districts. 48 California State Auditor Report 2010-122 April 2011 The guidelines state that this requirement is especially true when collecting expenditure data, as the cost center is the definition of who did the work, which functional category did what work, and how allocations are being spent. The assistant division chief further explained that employees will no longer manually enter the project phases into their time sheets, making the time‑charging system a closed system that prohibits employees from charging to project phases to which they have not been previously assigned. However, he also stated that Caltrans anticipates piloting PRSM in June 2011, and the February 2011 Independent Project Oversight Report indicates a departmentwide implementation of the system in August 2012. Thus, we cannot evaluate whether this control will be effective. Nonetheless, without strong internal controls, Caltrans risks having its employees charge time to incorrect projects or functional categories. Standardizing Its Approach to Using Earned Value Metrics Could Help Districts Better Manage Projects Earned value management is a commonly Earned Value Management Metrics used project management tool that can assist districts in assessing and measuring project • Schedule variance measures schedule performance and indicates when a project is falling behind its performance. The Project Management Institute, baseline schedule. which is recognized for its development of standards for the practice of project management, • Cost variance measures cost performance and indicates publishes A Guide to the Project Management the relationship of physical performance to the Body of Knowledge (PMBOK Guide), which amount spent. provides guidelines for managing individual • Schedule and cost variance values can be converted projects. The PMBOK Guide states that earned to efficiency indicators to reflect the cost and schedule value management integrates measures of performance of any project for comparison against project scope, cost, and schedule to help other projects. the project management team assess and measure • Schedule performance index measures the progress project performance and progress. Further, achieved compared to progress planned on the it describes earned value management as a project. A value greater than 1.0 would indicate that project management technique that requires the more work was complete than was planned, while a formation of an integrated baseline against which value less than 1.0 indicates less work was completed performance can be measured for the duration of than was planned. the project. The text box describes some selected • Cost performance index measures the value of earned value management metrics. work completed compared to the actual cost or progress made on the project, and is considered the In 1996 Caltrans underwent a peer review by most critical earned value management metric. A representatives of large, comparable private value less than 1.0 indicates a cost overrun for work and public engineering enterprises. The peer completed, while a value greater than 1.0 indicates a review’s final report, issued in September 1996, cost underrun for work completed. identified several areas for improvement in Source: Project Management Institute’s A Guide to the Project project management, including recommendations Management Body of Knowledge, Fourth Edition. related to the use of earned value management. Specifically, the report recommends the use California State Auditor Report 2010-122 49 April 2011 of labor, schedule, and cost performance indices to determine how well the project is being done or has been done. Caltrans responded to these recommendations in an action plan, updated in December 1998, noting that it had addressed the recommendations by providing project management courses that included training in the proper use of earned value management and requiring that earned value reporting be performed for every major state‑programmed highway project. Caltrans does not have a standardized earned value management policy, and we note that some Caltrans districts are using earned value management metrics in different ways. According to the Oakland division chief, the Oakland district does not calculate earned value metrics itself, but relies on the earned value data provided by Caltrans on a semimonthly basis. Further, he stated he felt this was sufficient for the district’s needs, although the statewide reports could be improved by adding features to improve understanding of the underlying data. The deputy director in the San Diego district’s Division of Project Management (San Diego deputy director) noted that the district does not use any earned value performance metrics but does track expenditures in relation to the budget for project phases over time. The Central Region, headquartered in Fresno, creates project status reports that list the schedule and cost performance indices for each phase of a project. However, the report lacks the aggregated metrics used by the Los Angeles district, as we discuss later in this section. An office chief of the Irvine district’s Division of Project Management noted that the district computes earned value performance indices for programmed projects. The Irvine district also creates earned value reports for each project that requires close monitoring. This report includes a graphical display of the earned value metrics, which the peer review recommended as necessary to provide project managers with information that will enable them to quickly and easily recognize project status. The Los Angeles district issued a directive regarding the effectiveness of earned value management in January 2010, and this document states that the district will use earned value metrics as part of a comprehensive effort to increase efficiency and improve accountability in delivering the support program. The district’s online tutorial for its earned value management system states that the district’s goal in implementing earned value management is to increase efficiency and accountability by measuring and controlling support costs. According to the deputy director of the Los Angeles district’s Division of Project Management, project and functional managers use its Project Information and Reporting System on a regular basis to determine the status of their projects. He explained that although almost all of the project managers use the system weekly, some project managers 50 California State Auditor Report 2010-122 April 2011 and functional managers are not yet in the habit of using the earned value management tools. He noted that this usage would change as the managers realize the benefits that these tools can offer. One district’s implementation of The Los Angeles district’s implementation of earned value earned value management appears management and other data analysis techniques appears to be robust. to be robust—another district Unlike the Irvine district, which, according to the office chief in uses earned value reports for only its Division of Project Management, uses earned value reports for 20 percent of its projects. about 20 percent of its projects, the Los Angeles district makes its system available for all projects. Further, the Los Angeles district’s reporting system creates reports displaying project schedule and cost performance indices for the project as a whole and for each of the respective project tasks, whereas the Central Region’s system generates reports displaying the cost and schedule performance indices only for an overall project and its individual phases, such as the design phase or the construction phase. The performance indices presented by the Los Angeles district’s system enable project performance reviews and performance reporting and are displayed for both budgeted hours and budgeted costs. These performance measures can be collected and combined for individual phases on all projects, or for a group of phases on all projects. This collection of performance metrics enables performance reviews of entire sections of the district, such as those working on right‑of‑way management or on construction support. However, it was not possible for us to quantify the impact this system has on the support costs for projects because it had been in use for only roughly one year at the time we completed our fieldwork. Since projects often span multiple years, it will take some time before the impact of this system is detectable in the portfolio of projects in this district. Increased Use of Consultants to Handle Large, Temporary Infusions of Funding May Help Caltrans Better Manage Workloads Although Caltrans recently attempted to address a temporary increase in workload by seeking approval from the Department of Finance for consultants rather than hiring permanent employees, it faced challenges in doing so. In particular, for fiscal year 2007–08, Caltrans requested an additional 595 consultant personnel years, a request that was based on a five‑year projection and the principle of combining stable staffing levels with the use of flexible resources, such as consultants, to manage workload peaks and valleys associated with Proposition 1B funding.12 Nevertheless, the authorized budget included 12 Proposition 1B, which took effect in 2006, authorized the issuance of about $20 billion in general obligation bonds for transportation improvements. According to the legislative analyst, Caltrans would play a crucial role in delivering transportation improvements requiring $12 billion of these funds, and Caltrans’ amount represented a 33 percent increase in the total value of the projects Caltrans had been working on in fiscal year 2006–07. California State Auditor Report 2010-122 51 April 2011 an additional 486 state staff personnel years and only 50 consultant personnel years. Further, in an analysis of the 2007–08 Governor’s Budget, the legislative analyst provided a rough estimate that for fiscal year 2007–08 and beyond, Caltrans could need as many as 4,800 personnel years in additional resources to plan and construct projects funded by Proposition 1B. In its fiscal year 2007–08 budget request, Caltrans proposed that consultants would undertake nearly 15 percent of its projected workload; however, Caltrans ultimately received authorization for less than 11 percent, which was a reduction from the previous year’s authorization. From fiscal years 2007–08 through 2010–11, Caltrans has been authorized to staff roughly 10 percent of its workload with consultants, despite generally proposing higher levels. According to the chief of Caltrans’ Project Delivery Management Support Office, requests for specific levels of consultant resources, based on actual project needs have historically been revised during the legislative budget process to align with a 10 percent to 90 percent consultant‑to‑state staff ratio. The San Diego deputy director stated that this ratio of 10 percent limits staffing flexibility in his district. He explained that consultants have been an effective tool to manage resources and to help keep in‑house staffing needs at a sustainable level. Although Caltrans could have addressed its increase in workload by hiring state staff in temporary positions, such as limited‑term and intermittent appointments, it has indicated that doing so would not have been practical. Specifically, the assistant division chief told us that Caltrans generally considers all options in staffing its workload but that most of its project workload is highly technical and professional in nature, and this work is generally not conducive to staffing with temporary employees. He cited two primary reasons for this situation. First, a substantial amount of training is necessary to bring new employees to a productive level, including engineers and other classifications that already have a technical background. Second, people seeking a professional career with public agencies Caltrans believes its best resource are usually looking for longer term or permanent employment. He for addressing short‑term workload indicated that Caltrans’ best resource for addressing short‑term demands is the consultant workload demands is the consultant community, which includes community, which includes firms firms that already perform Caltrans’ work and understand Caltrans’ that already perform Caltrans’ work technical requirements as well as its project delivery processes. and understand its requirements. Studies comparing the costs of permanent state staff to the costs of consultants have produced inconclusive results, and Caltrans has examined the impact of using consultants to manage workloads. For instance, a 2007 report commissioned by Consulting Engineers and Land Surveyors of California titled Cost to the Taxpayers of Obtaining Architectural and Engineering Services: State Employees Versus Private Consulting Firms, found no significant difference in 52 California State Auditor Report 2010-122 April 2011 the amount the State must pay to use a Caltrans engineer versus a private engineer. However, the report does note that there are important factors that tend to increase the cost of using Caltrans’ engineers, such as costs resulting from project delay and the cost of idle capacity when demand is below capacity. In contrast, Professional Engineers in California Government, the union that represents Caltrans’ engineers, among others, disagrees and indicated in a letter to the Caltrans division chief in March 2011 that outsourcing is considerably more expensive and those agencies that rely on outsourcing experience delays in project delivery and costs above estimates. Caltrans also contracted for a cost comparison, which was issued in 1992 by the Institute of Transportation Studies at the University of California at Berkeley, of using contracted engineering services versus in‑house engineers and found no statistically significant difference in terms of cost. Finally, according to the assistant division chief, each year Caltrans districts perform workload forecasts to determine whether future staffing needs are temporary or permanent; among other purposes, this information is incorporated into the budget request made to the Department of Finance and the Legislature. Recommendations To improve accountability internally and with the public, Caltrans should do the following: • Create and incorporate an analysis of support cost budget variances in its quarterly report to the agency and in its annual report to the Legislature and the governor. The analysis should report on the number of completed projects with budget variances and on the number of open projects for which the estimates at completion predict budget variances. Further, the analysis should report on the overrun and underrun ratios for those projects, and the portions of the variances due to rates and hours. Also, Caltrans should include in its strategic plan a measurable goal for reducing variances. • Establish budgets for those STIP projects programmed before the passage of Senate Bill 45 so that overruns may be reported in the quarterly report to the agency and in the annual report to the Legislature and the governor. • Develop a system to report on the total budgets of support program projects—including initial project support budgets— of projects that have been divided into multiple projects or combined into a larger project. California State Auditor Report 2010-122 53 April 2011 To improve performance metrics related to the support program, Caltrans should take these steps: • Devise, use, and publicize a consistent method for reporting the support‑to‑capital ratio on its Web site and in other reports to the public. Further, Caltrans should recalculate past support‑to‑capital ratios using the method devised to allow for comparison across years. • Develop goals—and publicly report on the progress against those goals—for the support‑to‑capital ratio, based on project type (STIP or SHOPP) and project size. • Continue to explore the use of additional metrics, such as a measure based on a productivity index as described in a March 2011 draft study by the University of California, Davis. To better develop and manage project budgets for support, Caltrans should do the following: • Instruct project managers to submit requests to update the budget when assumptions on which the budget was based are no longer valid, regardless of the phase of the project. • Direct its project managers to use a detailed approach based on project tasks, such as those included in a project work plan, when finalizing project support budgets before construction. To ensure that it monitors the status of projects, Caltrans should take these steps: • Continue to implement the policies described in its February 2010 memorandum to the districts describing an approach Caltrans will take to monitor support costs within budget. Moreover, Caltrans should direct its project managers to monitor budgets for all projects according to both hours and costs. • Implement earned value management throughout its districts in a manner similar to the implementation in the Los Angeles district. To allow for performance evaluation of project work, Caltrans should ensure that these performance metrics are available at the task level for both active and completed projects. Caltrans should instruct districts to aggregate this information for all projects by task level, to better assess the effectiveness and efficiency of support costs by task level. Caltrans should also make available to project managers graphical displays of project cost and schedule performance. 54 California State Auditor Report 2010-122 April 2011 To better address costs associated with the support program, Caltrans should do the following: • Ensure that the PRSM system contains strong controls that ensure employees only charge time to projects and phases for which they are assigned. • Commission an independent study of the costs and benefits of using consultants to address temporary increases in workload and, if the study reveals cost savings, use consultants. To the extent possible, Caltrans should also use temporary staff appointments for temporary increases in workload when consultants are unavailable. To ensure that it receives more complete information on the support program, the Legislature should require Caltrans to include in its annual report an expanded methodology for reporting support‑to‑capital ratios to include, in addition to a support‑to‑cost ratio analysis based on costs incurred up to the award of the construction contract of STIP projects, a separate support‑to‑capital‑ratio analysis for STIP projects that have completed construction. Further, the Legislature should require Caltrans to report on similar ratios for SHOPP projects based on costs incurred up to the award of the construction contract and for those projects that completed construction. To increase accountability for budget overruns of support costs, the Legislature should consider legislation that would expressly require the commission to review and approve project construction support costs when they differ from the amount budgeted by 20 percent or more. To ensure that Caltrans does not hire permanent state staff beyond its need for such staff, the Legislature should consider appropriating funding for consultants to address temporary increases in Caltrans’ workloads when Caltrans requests such funding. California State Auditor Report 2010-122 55 April 2011 We conducted this audit under the authority vested in the California State Auditor by Section 8543 et seq. of the California Government Code and according to generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives specified in the scope section of the report. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Respectfully submitted, ELAINE M. HOWLE, CPA State Auditor Date: April 28, 2011 Staff: Laura G. Boll, Project Manager Ralph M. Flynn, JD John Lewis, MPA Jim Adams, MPP Sarah T. Bragonje, MPA Jessica E. Kubo A.J. Meyer Consultant: NewPoint Group Management Consultants Legal Counsel: Stephanie Ramirez‑Ridgeway, JD For questions regarding the contents of this report, please contact Margarita Fernández, Chief of Public Affairs, at 916.445.0255. 56 California State Auditor Report 2010-122 April 2011 Blank page inserted for reproduction purposes only. California State Auditor Report 2010-122 57 April 2011 Appendix COMPARISON OF SUPPORT‑TO‑CAPITAL RATIOS BY DISTRICT AND PROJECT TYPE AND SIZE To evaluate the potential differences in capital outlay support among districts of the California Department of Transportation (Caltrans), we assessed the districts’ support to capital costs ratios (support‑to‑capital ratios) for projects completed during fiscal years 2007–08 through 2009–10.13 We further refined the ratios based on project type and size—State Transportation Improvement Program (STIP) or State Highway Operation and Protection Program (SHOPP). We categorized each project as small, medium, or large based on the sum of the individual project’s actual construction and right‑of‑way capital costs. It is possible to explain differences in the ratios among districts by reviewing project size. Specifically, as we explain in the Audit Results, the presence of large projects tends to indicate that a district will have a lower support‑to‑capital ratio. For example, of the 92 projects for which construction was completed in the Marysville district during the period we reviewed, only two were large projects. However, the size of these two projects, and their respective support‑to‑capital ratio of nearly 13 percent, played a significant role in causing the total ratio for the district to fall below 30 percent, even though the support‑to‑capital ratios for small‑ and medium‑sized projects were much higher at roughly 79 percent and 34 percent, respectively. A district dominated by small projects, or by small and medium projects, generally has a higher support‑to‑capital ratio. According to the assistant chief of Caltrans’ Division of Project Management, a certain amount of support is necessary, such as that for traffic plans and bid packages, regardless of project size. Thus, the ratio of support costs to capital costs is generally greater for smaller projects than it is for larger projects. For example, as Table A on the following page shows, all of the projects for which construction was completed in Eureka, San Luis Obispo, and Irvine districts were small‑ and medium‑sized. Consequently, these three districts experienced the highest support‑to‑capital ratios of Caltrans’ 12 districts. 13 When we use the name Caltrans, we are referring to the statewide organization and its management headquartered in Sacramento. We refer to Caltrans’ districts as districts or identify the locations of specific districts’ headquarters. 58 California State Auditor Report 2010-122 April 2011 Table A Support‑to‑Capital Ratios Categorized by District, Project Size, and Project Type Fiscal Years 2007–08 Through 2009–10 STATE TRANSPORTATION STATE HIGHWAY OPERATION IMPROVEMENT PROGRAM (STIP) AND PROTECTION PROGRAM PROJECTS (SHOPP) PROJECTS TOTAL STIP AND SHOPP PROJECTS NUMBER OF SUPPORT‑TO‑CAPITAL NUMBER OF SUPPORT‑TO‑CAPITAL NUMBER OF SUPPORT‑TO‑CAPITAL DISTRICT PROJECTS RATIO* PROJECTS RATIO* PROJECTS RATIO* Totals for District 1, Eureka 2 67.49% 50 44.82% 52 44.91% Small† (projects with less than 2 67.49 10 65.15 12 65.40 $1 million in capital costs) Medium† (projects with between $1 million and $50 million in 0 – 40 44.12 40 44.12 capital costs) Large† (projects with more than 0 – 0 – 0 – $50 million in capital costs) Totals for District 2, Redding 4 38.82% 49 31.25% 53 31.97% Small 0 – 11 77.60 11 77.60 Medium 4 38.82 37 35.93 41 36.27 Large 0 – 1 8.29 1 8.29 Totals for District 3, Marysville 5 59.84% 87 28.10% 92 29.74% Small 2 102.11 35 77.93 37 79.10 Medium 3 58.55 50 31.88 53 33.77 Large 0 – 2 12.92 2 12.92 Totals for District 4, Oakland 6 51.98% 109 37.17% 115 39.36% Small 1 263.97 35 108.92 36 113.24 Medium 4 54.86 74 34.76 78 36.01 Large 1 48.48 0 – 1 48.48 Totals for District 5, San Luis Obispo 8 50.36% 52 39.13% 60 41.27% Small 2 87.89 17 98.25 19 97.25 Medium 6 49.21 35 34.94 41 37.74 Large 0 – 0 – 0 – Totals for District 6, Fresno 7 26.58% 58 36.14% 65 31.90% Small 2 104.85 24 97.24 26 97.68 Medium 2 37.46 34 32.18 36 32.25 Large 3 26.02 0 – 3 26.02 Totals for District 7, Los Angeles 5 34.14% 89 32.44% 94 32.77% Small 0 – 17 114.39 17 114.39 Medium 3 46.30 71 32.01 74 32.70 Large 2 31.44 1 25.34 3 29.08 Totals for District 8, San Bernardino 5 35.18% 57 33.73% 62 34.10% Small 1 154.59 27 97.50 28 99.75 Medium 3 50.38 29 34.64 32 37.06 Large 1 23.73 1 13.58 2 18.65 Totals for District 9, Bishop 1 26.07% 8 25.34% 9 25.61% Small 0 – 0 – 0 – Medium 1 26.07 8 25.34 9 25.61 Large 0 – 0 – 0 – California State Auditor Report 2010-122 59 April 2011 STATE TRANSPORTATION STATE HIGHWAY OPERATION IMPROVEMENT PROGRAM (STIP) AND PROTECTION PROGRAM PROJECTS (SHOPP) PROJECTS TOTAL STIP AND SHOPP PROJECTS NUMBER OF SUPPORT‑TO‑CAPITAL NUMBER OF SUPPORT‑TO‑CAPITAL NUMBER OF SUPPORT‑TO‑CAPITAL DISTRICT PROJECTS RATIO* PROJECTS RATIO* PROJECTS RATIO* Totals for District 10, Stockton 7 26.46% 37 47.74% 44 32.19% Small 0 – 19 125.84 19 125.84 Medium 5 31.02 18 40.21 23 34.90 Large 2 22.57 0 – 2 22.57 Totals for District 11, San Diego 3 66.68% 58 39.05% 61 39.66% Small 1 119.72 20 79.28 21 81.42 Medium 2 55.47 38 35.99 40 36.38 Large 0 – 0 – 0 – Totals for District 12, Irvine 2 61.52% 57 49.61% 59 50.24% Small 0 – 30 152.68 30 152.68 Medium 2 61.52 27 40.79 29 41.99 Large 0 – 0 – 0 – Totals 55 34.01% 711 35.71% 766 35.33% Small 11 118.93 245 102.29 256 102.95 Medium 35 41.09 461 34.91 496 35.64 Large 9 28.44 5 15.13 14 23.76 Source: Bureau of State Audits’ analysis of the California Department of Transportation’s (Caltrans) data generated from the Transportation Accounting and Management System. Notes: We analyzed 766 of 877 projects that completed construction in fiscal years 2007–08 through 2009–10. We excluded projects that, among other things, were exempt from budget accountability requirements and projects that were implemented by local entities—not by Caltrans—such as county transportation commissions. * We calculated the support‑to‑capital ratio by analyzing the difference between the total support cost expenditures and the total capital cost expenditures for the 766 projects that completed construction in fiscal years 2007–08 through 2009–10. † For the purposes of this analysis, we categorized each project as small, medium, or large based on the sum of that project’s construction capital costs and right‑of‑way capital costs. 60 California State Auditor Report 2010-122 April 2011 Blank page inserted for reproduction purposes only. California State Auditor Report 2010-122 61 April 2011 (Agency response provided as text only.) Business, Transportation and Housing Agency 980 9th Street, Suite 2450 Sacramento, CA 95814‑2719 April 11, 2011 Elaine M. Howle* Bureau of State Audits 555 Capitol Mall, Suite 300 Sacramento, CA 95814 Dear Ms. Howle: Attached is a response from the California Department of Transportation (Caltrans) to your draft audit report Department of Transportation: Its Capital Outlay Support Program Should Strengthen Budgeting Practices, Refine Its Performance Measures, and Improve Internal Controls to Minimize Cost Overruns and Increase Accountability (#2010‑122). Thank you for allowing Caltrans and the Business, Transportation and Housing Agency (BTH) the opportunity to respond to the report. As noted in its response, Caltrans supports the vast majority of the report’s recommendations and has already implemented portions of them. Additionally, Caltrans plans to complete full implementation of all recommendations by the summer of 2012. We appreciate your identification of opportunities for improvement related to Caltrans’ operation of its Capital Outlay Support program. If you need additional information regarding Caltrans’ response, please do not hesitate to contact Michael Tritz, BTH Deputy Secretary for Audits and Performance Improvement, at (916) 324‑7517. Sincerely, (Signed by: Traci Stevens) TRACI STEVENS Acting Undersecretary * California State Auditor’s comments begin on page 69. 62 California State Auditor Report 2010-122 April 2011 Department of Transportation Office Of The Director P.O. Box 942873, MS‑49 Sacramento, CA 94273‑0001 April 8, 2011 Ms. Traci Stevens Acting Undersecretary Business, Transportation and Housing Agency 980 9th Street, Suite 2450 Sacramento, CA 95814 Dear Ms. Stevens: Thank you for the opportunity to review and comment on the Bureau of State Audits’ (BSA) draft audit report entitled “Department of Transportation: Its Capital Outlay Support Program Should Strengthen Budgeting Practices, Refine its Performance Measures, and Improve Internal Controls to Minimize Cost Overruns and Increase Accountability.” At the request of the Joint Legislative Audit Committee, the BSA conducted an audit of the performance, management, efficiency, and budget of the California Department of Transportation’s (Caltrans) Capital Outlay Support (COS) Program. We were pleased that the BSA’s report identified the following positive conditions at Caltrans: • The budget request process for the support program was reasonable. • Both the list of projects and the other costs included in the budget request for Caltrans’ support program were accurate and adequately supported. • The process for staffing appears adequate. • The organization structure appears adequate and appropriate. • Travel reimbursements were appropriate, reasonable and consistent with policy. The BSA also concluded, however, that Caltrans had not adequately prioritized its monitoring of capital outlay support costs, its project support costs exceed budgeted amounts, and its systems contain inaccurate project budget data. The BSA’s audit report includes recommendations that will assist Caltrans in focusing efforts to increase the importance of monitoring, measuring, and updating support budgets as well as increase accountability at both the project level and the program level. We support the majority of the report’s recommendations and have already implemented portions of them. Typically, project support budgets have been established and committed to by programming the initial support budget in a programming document early in the project development process when there are a lot of unknown risks and factors yet to be considered. During the development process, the project scope often changes based on public input, avoidance of environmental and property impacts, and technical engineering studies. Adjusting those initial support budgets as scope and schedules change during the development of a project has not been the highest priority. Caltrans’ primary focus has been to meet project deadlines and stay within the overall project budget. California State Auditor Report 2010-122 63 April 2011 Ms. Traci Stevens April 8, 2011 Page 2 Over the past five fiscal years, Caltrans has delivered 1,391 out of 1,394 projects (over 99 percent) in Caltrans’ Contracts for Delivery, with a total contract value of $14.6 billion, on schedule ready to go to construction. Since 2007, Caltrans has increased attention on the support components of project budgets. Examples of recent support budget emphasis include: • Programming of the four major support budget components, starting with the 2008 State Highway Operations and Preservation Program (SHOPP). • Including preliminary engineering support costs starting with the fiscal year 2009–10 Contract for Delivery. • Implementing a February 25, 2010, memorandum entitled “Managing Capital Outlay Support” which reinforced the districts responsibility for increased accountability for pro‑active management of their support budgets. • Modifying change policy practices to include support cost changes. Although the BSA’s review found a support cost budget overrun ratio on 62 percent (as shown in the report’s Table 2) of the projects, many of those projects were started well before our emphasis on support costs. Further, Caltrans has made improvements in estimating support costs. Senate Bill 45, which was enacted in 1997 and implemented with the 1998 State Transportation Improvement Program (STIP), was the first time that program support budgets were established for projects. At that time, there was little historical data and information to adequately evaluate newly established support budgets. In the data sample of 766 projects evaluated in the BSA review, our own analysis revealed that, when the support budget overrun percent is calculated based on the year that the environmental document was completed (closer to when the support budget was initially programmed), the overrun percentages are substantially higher for projects initiated in earlier years than in more recent years. Detailed responses to each of the BSA’s recommendations are listed below: Recommendation No. 1: To improve accountability internally and with the public, Caltrans should: a) Create and incorporate an analysis of support cost budget overruns in its quarterly report to the agency, and in its annual report to the Legislature and the governor. The analysis should report on the number of completed projects with budget overruns and the number of open projects where the estimate at completion projects a budget overrun, the overrun ratio for those projects, and the portion of the overruns due to rate and hours. Further, Caltrans should include a measurable goal for reducing overruns in its strategic plan. Caltrans Response: Caltrans agrees that an analysis and report needs to be completed on project budgets at project completion. Caltrans is developing a template that will be incorporated into the CTC Quarterly 1 Delivery Report for the period starting July 1, 2011. The year‑end CTC Quarterly Delivery Report is an attachment to the annual report to the Legislature, which is also shared with the Business, Transportation and Housing Agency (BTH). 64 California State Auditor Report 2010-122 April 2011 Ms. Traci Stevens April 8, 2011 Page 3 1 While the focus of the recommendation is on support budget overruns, establishing a measureable goal in our strategic plan that addresses only overruns could lead to inflated support cost estimates. To avoid this potential risk, Caltrans plans to establish a performance measure that targets support expenditures that are within a specified range of the support budget. This performance measure will be in place by July 1, 2011, and will include the support component. b) Establish budgets for those STIP projects allocated funding prior to the passage of Senate Bill 45, so that overruns may be reported in the quarterly report to the agency, and in the annual report to the Legislature and the governor. Caltrans Response: Caltrans concurs with this recommendation. Currently, there are 24 STIP projects that were programmed prior to the passage of Senate Bill 45. Of 2 these 24 projects, 12 are currently in construction and 12 remain to be delivered to construction. Caltrans will set up support budgets for all 12 projects that remain to be delivered to construction. This will be implemented by July 1, 2011. c) Develop a system to report on the total budgets of support program projects – including initial project support budgets – of projects that have been divided into multiple projects or combined into larger projects. Caltrans Response: Caltrans concurs with this recommendation. The ability to divide and combine projects is an important delivery tool. This allows projects to be broken into smaller, or combined into larger, projects for delivery due to funding constraints, streamlined delivery or constructability issues as well as providing small businesses increased opportunities. Approximately 10 percent of Caltrans projects result in a divided or combined project. While Caltrans has processes in place to track these projects, we are developing improved business practices to allow for easier tracking of project budgets and will implement these practices by July 1, 2011. In the longer term, an IT system or upgrade will be required. Recommendation No. 2: To improve performance metrics related to the support program, Caltrans should: a) Devise, utilize, and publicize a consistent method for reporting the support‑to‑capital ratio on its Web site and in other reports to the public. Further, Caltrans should recalculate past support‑to‑capital ratios using the method devised to allow for comparison across years. b) Develop goals ‑ and publicly report on the progress against those goals ‑ for the support to capital ratio, based on project type (STIP or SHOPP) and project size. Caltrans Response: Caltrans concurs with these two recommendations. Although the support‑to‑capital ratio can be an important indicator from a program perspective, it needs to be viewed in the context of size and complexity for individual projects. California State Auditor Report 2010-122 65 April 2011 Ms. Traci Stevens April 8, 2011 Page 4 By July 31, 2011, Caltrans will refine a consistent methodology to normalize data across years and will establish preliminary goals. By December 31, 2011, Caltrans will incorporate these indicators into the CTC Quarterly Delivery Report. Caltrans will post on its Web site the method devised to allow for a comparison of support‑to‑cost ratios across years. c) Continue to explore the use of additional metrics, such as a measure based on a productivity index as described in a March 2011 draft study by the University of California, Davis. Caltrans Response: Caltrans concurs with this recommendation, and points to its support‑to‑capital ratio goal of 32 percent as evidence of the need for additional metrics. Caltrans has used a ratio of support to capital as a measure since at least 1992, and the ratio has been around 35 percent. This has been reported in its strategic plan reports. Approximately five years ago, Caltrans set a number of goals as part of its strategic plan objectives, including an aggressive goal for reducing its support‑to‑capital ratio to 32 percent. However, since the goal was set five years ago there has been an unprecedented level of changes in terms of construction costs. In 2004, the construction capital cost index increased by 40 percent, followed by an additional 24 percent the following year. Four years later it dropped by 19 percent. This ratio is impacted dramatically by swings up and down in construction costs. In this environment, the support‑to‑capital measure has become a less desirable measure of productivity or efficiency. Comparison of changes from one year to the next has become a reflection of cost escalation and it is increasingly difficult to gauge production or efficiency from this measure. Thus, Caltrans has been moving away from using the support‑to‑capital ratio as a measure of performance but will continue to use it as an indicator. Caltrans will produce an additional metric by July 1, 2012. This effort will be done jointly with our partners and stakeholders. Recommendation No. 3: To better develop and manage project budgets for support, Caltrans should: a) Instruct project managers to submit requests to update the budget when assumptions on which the budget was based are no longer valid, regardless of the phase of the project. Caltrans Response: Caltrans concurs with this recommendation. Caltrans has a process in place to update project budgets and has made significant improvement over the last 5 years. However, we recognize the need for continued improvement in setting and updating support budgets when changes in assumptions and conditions occur. To emphasize these improvements in a consistent manner, a new Project Management Directive (Directive) entitled “Management of Capital Outlay Support” has been developed and will be issued by July 1, 2011. 66 California State Auditor Report 2010-122 April 2011 Ms. Traci Stevens April 8, 2011 Page 5 b) Direct its project managers to use a detailed approach based on project tasks, such as those included in a project work plan, when finalizing project support budgets prior to construction. Caltrans Response: Caltrans concurs with this recommendation. For the SHOPP program, programming documents will be updated to reflect the budget at the time of the CTC allocation for construction. For the STIP program we will update support budgets for construction within the limitations of the STIP statutes and CTC guidelines. Recommendation No. 4: To ensure it monitors the status of projects, Caltrans should: a) Continue to implement the policies described in its February 2010 memorandum to the districts describing an approach Caltrans will take to monitor support costs within budget. Moreover, Caltrans should direct its project managers to monitor budgets for all projects according to both hours and costs. Caltrans Response: Caltrans concurs with this recommendation. Caltrans is currently collecting data in accordance with the February 2010 memorandum. To ensure there is follow‑through on the plans submitted, this information will become part of our Quarterly Project Delivery Video Teleconferences, effective with the next quarterly meeting, which will occur in May 2011. b) Implement earned value management throughout its districts in a manner similar to the implementation in the Los Angeles district. To allow for performance evaluation of project work, Caltrans should ensure that these performance metrics are available at the task level for both active and completed projects. Caltrans should instruct districts to aggregate this information for all projects by task level, to better assess the effectiveness and efficiency of support costs by task level. Caltrans should also make available graphical displays of project cost and schedule performance to project managers. Caltrans Response: Caltrans concurs with this recommendation. Caltrans has had basic earned value reports in 3 place for many years and several districts have already implemented some level of earned value management. Caltrans will implement a statewide standard approach to earned value management with the implementation of its Project Resourcing and Schedule Management project (PRSM). PRSM is scheduled to be fully implemented by the summer of 2012. Recommendation No. 5: To better address costs associated with the support program, Caltrans should: a) Ensure that the PRSM system contains strong controls that ensure employees only charge time to projects and phases for which they are assigned. Caltrans Response: Caltrans concurs in part with this recommendation. Given the complexity and numbers of projects 4 that teams of employees are working on, limiting staff ability to charge to certain projects will actually encourage mischarging and impede the flexibility to work as a team to deliver critical products. When PRSM is fully implemented (summer of 2012) project managers will be able to California State Auditor Report 2010-122 67 April 2011 Ms. Traci Stevens April 8, 2011 Page 6 assign cost centers to their projects. This will allow only those employees with approved cost centers to charge to their projects. Accurate charging practices are extremely important and Caltrans has implemented several practices in this area: • Deputy Directive 41, entitled “Departmental Charging Practices,” was issued on October 12, 2009, and states in part, “…employees must accurately record time and expenses in performing their duties…accurate charging practices are essential for …effective project management....” This directive will be updated to emphasize the importance of accurate time charging. • All time charges to projects are reviewed and approved by the first‑line supervisor and the project managers have the responsibility to validate these charges. • In March of 2011, a new tool was deployed to make project‑specific labor monitoring simpler and more accessible. 5 • Also in March of 2011, Caltrans released specific guidance on charging practices for Capital Outlay Support (COS). The “COS Proper Charging Guide” provides step‑by‑step procedures for charging time in our current automated system, Staff Central, as well as instructions for supervisors and project managers on running labor monitoring reports by project or unit. The “COS Proper Charging Guide” includes a description of each charge code used in Staff Central and highlights those appropriate for use by COS. • As a preventative measure, Staff Central only allows charges to a task associated with an active project phase. For example, all Sub Object codes associated with Phase 0 of a project will only 4 be chargeable when that phase is active. In all other circumstances, the user will be prohibited from charging time. b) Commission an independent study of the costs and benefits of using consultants to address temporary increases in workload and, if the study reveals cost savings, use consultants. To the extent possible, Caltrans should also utilize temporary staff appointments for temporary increases in workload when consultants are unavailable. Caltrans Response: Caltrans disagrees with this recommendation. 6 Caltrans believes that both State staff and consultants are vital to the successful delivery of transportation projects. Using consultants for temporary increases in workload, for specialty work in which the Caltrans does not have expertise, and for seasonal work is crucial to our success. Consultants bring their own tools, equipment and office space. In comparison, temporary staff appointments require training to be able to perform the highly technical work and the purchasing of tools, equipment and office space. These factors further exacerbate temporary staff as a cost‑effective solution. Hiring State engineers as temporary staff appointments does not meet Caltrans’ critical needs. As discussed in the audit, cost comparison studies have been performed by several entities over the past two decades. For example, the Institute of Transportation Studies, University of California at Berkeley (July of 1992) concluded that there was no significant difference in cost between 68 California State Auditor Report 2010-122 April 2011 Ms. Traci Stevens April 8, 2011 Page 7 consultants versus in‑house forces when performing engineering work. Also, the Consulting Engineers and Land Surveyors performed a study in 2007, which determined that there was no significant cost difference between a Caltrans engineer versus a private engineer. However, the Professional Engineers in California Government indicate that outsourcing is considerably more expensive than the cost of using Caltrans engineers. While another study could be done, the results of previous studies already provide a basis for making effective decisions to address temporary increases in workload. Recommendation No. 6: To ensure it receives more complete information on the support program, the Legislature should require Caltrans to include in its annual report an expanded methodology for reporting support‑to‑capital ratios to include, in addition to a support to cost ratio based on costs incurred up to the award of the construction contract of STIP projects, a separate support to capital ratio for STIP projects that have completed construction. Further, the Legislature should require Caltrans to report on similar ratios for SHOPP projects based on costs incurred up to the award of the construction contract and for those projects that completed construction. Caltrans Response: Caltrans agrees with this recommendation and will incorporate it into Caltrans’ next annual report due to the Legislature by November 15, 2011. Recommendation No. 7: To increase accountability for budget overruns of support costs, the Legislature should consider legislation that would expressly require the commission to review and approve project construction support costs when they differ from the amount budgeted by 20 percent or more. Caltrans Response: 7 As proposed, this recommendation will not achieve the stated objective. Recommendation No. 8: To ensure that Caltrans does not hire permanent state staff beyond its long‑term need for such staff, the Legislature should consider appropriating funding for consultants to address temporary increases in Caltrans workloads when they are requested. Caltrans Response: Caltrans concurs with this recommendation and will continue to identify the most cost‑effective mix of resources needed to deliver the planned transportation program. Ultimately, the resource mix is determined by the approved budget for the fiscal year. Caltrans appreciates the opportunity to provide a response to the draft audit report. If you have any questions or require further information, please contact Richard Land, Deputy Director, Project Delivery, at (916) 654‑6490, or Carlos Aguila, Acting Assistant Director, Audits and Investigations, at (916) 323‑7911. Sincerely, (Signed by: Cindy McKim) CINDY McKIM Director California State Auditor Report 2010-122 69 April 2011 Comments CALIFORNIA STATE AUDITOR’S COMMENTS ON THE RESPONSE FROM THE DEPARTMENT OF TRANSPORTATION To provide clarity and perspective, we are commenting on the response to our audit report from the Department of Transportation (Caltrans). The numbers below correspond to the number we placed in the margin of Caltrans’ response. 1 As noted on page 39, according to the chief of Caltrans’ Division of Project Management (division chief), when budgets are overstated, fewer projects receive funding, and when budgets are understated, the subsequent overruns take funding away from other projects, leading to construction delays. Therefore, we revised the recommendation language on page 52 to include analysis of budget variances for both overruns and underruns. Further, as noted in the recommendation, Caltrans should report on budget variances for both completed projects and open projects. Finally, we agree that Caltrans should report this analysis in its Quarterly Delivery Report to the commission; however, as noted in the recommendation on page 52, this analysis should also be reported in Caltrans’ quarterly report to the agency and its annual report to the Legislature and governor. 2 As we note on page 32, some State Transportation Improvement Program (STIP) projects were excluded from support budgeting and accountability requirements; thus, in effect, these projects have no established support budgets. Further, on pages 32 through 33, Caltrans estimated that it has 24 such STIP projects yet to complete construction and their total, including the estimated support and capital costs, is nearly $250 million. We see no reason why Caltrans should not establish support budgets for the 12 STIP projects that are in construction as well. 3 As noted on page 49, Caltrans does not have a standardized earned value management policy, and we noted that some Caltrans districts are using earned value management metrics in different ways. Further, as noted on page 18, the Project Resource and Schedule Management (PRSM) system has experienced many delays since procurement began in 2000. For example, there was an expected implementation date of April 2008. However, according to Caltrans’ response on page 66, PRSM is currently scheduled to be fully implemented by the summer of 2012. In light of these delays and the importance of implementing the use of earned value metrics consistently throughout the districts, we believe that Caltrans should implement 70 California State Auditor Report 2010-122 April 2011 a statewide standard approach to analyze earned value metrics, using information currently available, either in preparation for or in lieu of the PRSM system. 4 Caltrans does not seem to appreciate our concerns regarding its lack of strong controls to ensure employees only charge time to projects and phases for which they are assigned. As noted on page 46, Caltrans’ time‑reporting system currently does not have a mechanism in place to prevent employees from charging to projects to which they were not previously assigned, as long as those projects are active. Further, on page 47 we conclude that without a system that distinguishes an authorized charge from an unauthorized charge and prevents mischarging, Caltrans does not have an efficient way to ensure that employees are always charging time to the appropriate project. 5 Caltrans notes that in March 2011, it deployed a new tool to make project‑specific labor monitoring simpler and more accessible, and also released specific guidance on charging practices for the support program. However, because this new tool and guidance were issued after we completed our fieldwork we did not have an opportunity to review their effectiveness. 6 As noted on page 52, the Institute of Transportation Studies at the University of California at Berkeley study was conducted more than 18 years ago. Further, the Consulting Engineers and Land Surveyors of California, and the Professional Engineers in California Government, the union that represents Caltrans engineers, are not independent entities. Thus, we believe it prudent for Caltrans to conduct an independent study of the costs and benefits of using consultants to address temporary increases in workload. We also believe such an in‑depth and independent analysis would assist the Legislature in making difficult budgeting decisions. 7 As we point out on page 40, Caltrans’ support budget overruns during the construction phase are not tracked or reviewed by the California Transportation Commission (commission). Further, according to state law, the commission may not change project costs to reflect changes in construction expenditures once a project is in construction without making a supplemental allocation. However, as noted on page 40, Caltrans’ chief of capital improvement programs informed us that support budgets are never updated after construction begins. Therefore, the commission cannot easily adjust the funds available to the entity because the commission does not review cost overruns during this phase. We also conclude on page 40 that we believe that increased oversight and accountability during the construction phase of STIP projects could reduce support cost overruns. California State Auditor Report 2010-122 71 April 2011 Because there is no oversight during the construction phase, we were unable to gain a clear understanding of the reasons or explanations associated with support cost overruns during this phase. Thus, we recommended on page 54 that the Legislature consider legislation that expressly requires the commission to review and approve project construction support costs when they differ from the amount budgeted by 20 percent or more. It is our expectation that the commission will conduct a responsible oversight that would require explanation and evidence as to the cause(s) for support cost overruns and result in comprehensive review of budget adjustments before approval. Moreover, it is our expectation that when the commission determines that cost overruns are not appropriate, it would ensure the overrun budgeted costs affect only the share of STIP funds for those counties in which the overrun occurred, which would infuse the process with accountability. 72 California State Auditor Report 2010-122 April 2011 cc: Members of the Legislature Office of the Lieutenant Governor Milton Marks Commission on California State Government Organization and Economy Department of Finance Attorney General State Controller State Treasurer Legislative Analyst Senate Office of Research California Research Bureau Capitol Press