Summary

Information Technology: The State Needs to Improve the Leadership and Management of Its Information Technology Efforts June 2001 2000-118 rotiduA etatS ainrofilaC S T I D U A E T A T S F O U A E R U B The first five copies of each California State Auditor report are free. Additional copies are $3 each, payable by check or money order. You can obtain reports by contacting the Bureau of State Audits at the following address: California State Auditor Bureau of State Audits 555 Capitol Mall, Suite 300 Sacramento, California 95814 (916) 445-0255 or TDD (916) 445-0255 x 216 OR This report may also be available on the World Wide Web http://www.bsa.ca.gov/bsa/ Alternate format reports available upon request. Permission is granted to reproduce reports. C S A ALIFORNIA TATE UDITOR ELAINE M. HOWLE STEVEN M. HENDRICKSON STATE AUDITOR CHIEF DEPUTY STATE AUDITOR June27,2001 2000-118 TheGovernorofCalifornia President pro Temporeof theSenate SpeakeroftheAssembly StateCapitol Sacramento, California95814 Dear Governorand Legislative Leaders: As requested by the Joint Legislative Audit Committee, the Bureau of State Audits presents its audit report concerning the State’s management of information technology (IT). This report concludes that the Department of Information Technology (DOIT), which is responsible for overseeing the State’s efforts to plan, develop, and manage IT, needs to provide stronger leadership and guidance to departments as they develop new IT projects. Additionally, DOIT has not sufficiently addressed other responsibilities such as completing a statewide inventory of projects to assist in the coordination of similar IT efforts and to enhance its oversight, releasing technical standards that establish common rules for IT projects, and consistently using state-mandated advisory councils to provide input on its operations. Although DOIT is implementing a new strategic vision for overseeing departments’ IT efforts, it is too early to assess the value that its new processes will add. While DOIT is responsible for overseeing the State’s IT efforts, departments are primarily responsible for planning, developing, and ensuring the quality of their respective IT projects. However, our review of four major projects revealed that departments are not always using best practices to manage such projects. The four IT projects we reviewed experienced varying degrees of cost overruns and delays; however, two of these projects, managed by the Department of Transportation and the Department ofHealth Services, had significant project management problems. Respectfullysubmitted, ELAINEM.HOWLE StateAuditor BUREAU OF STATE AUDITS 555 Capitol Mall, Suite 300, Sacramento, California 95814 Telephone: (916) 445-0255 Fax: (916) 327-0019 CONTENTS Summary 1 Introduction 5 Chapter 1 DOIT Needs to Provide Stronger Leadership Over Departments’ Information Technology Efforts 15 Recommendations 32 Chapter 2 Some State Departments Are Not Managing Their Information Technology Efforts According to Best Practices 35 Recommendations 50 Chapter 3 DOIT Has Not Fully Satisfied Its Other Responsibilities 53 Recommendations 66 Appendix A Evaluation of Departments’ Project Management of Selected Information Technology Projects 69 Appendix B Major Completed and In-Development Information Technology Projects 73 Responses to the Audit Department of Information Technology 97 California State Auditor’s Comments on the Response From the Department of Information Technology 141 Business, Transportation and Housing Agency 149 Department of Transportation Health and Human Services Agency 155 Department of Health Services Employment Development Department State and Consumer Services Agency 167 Franchise Tax Board SUMMARY RESULTS IN BRIEF T he State has a significant investment in information technology (IT)––more than an estimated $2 billion Audit Highlights . . . annually––and in the past has experienced several major failures in planned IT systems. When it passed legislation that Our review of the State’s leadership and management resulted in the creation of the Department of Information Tech- of its information technology nology (DOIT) in 1996, the Legislature envisioned that DOIT (IT) projects revealed the would provide the leadership, guidance, and oversight needed to following: protect the State’s investment in IT. Although DOIT is developing (cid:1) The Department of new processes to meet its responsibilities, it has not consistently Information Technology delivered what it has been asked to do by the Legislature. (DOIT), which is responsible for overseeing The State needs strong leadership from DOIT to ensure that the State’s efforts to plan, develop, and evaluate IT, departments develop IT systems effectively. However, DOIT is needs to provide stronger not consistently ensuring that departments plan their IT efforts leadership and guidance carefully. Planning is crucial to the proper development of IT to state departments. projects. State law requires DOIT to develop a statewide IT plan (cid:1) Four major projects we to outline the vision and direction of the State’s IT efforts, but it reviewed experienced has not updated its 1997 plan to address pressing issues and varying degrees of cost include current measurable objectives. Additionally, the state- overruns and delays, but wide plan does not include priorities to ensure that the most two of these projects had significant project important projects are considered first. Further, DOIT has not management problems. sufficiently reviewed the strategic plans of departments’ IT (cid:1) projects. DOIT could increase the level of leadership and guid- DOIT has not sufficiently met other responsibilities, ance it provides to increase the probability of success for the such as completing a State’s IT efforts. statewide inventory of projects, releasing key DOIT is also responsible for the review and approval of the standards that establish common rules for projects, proposed IT projects of departments. However, DOIT cannot and using state-mandated demonstrate that it has consistently and sufficiently performed advisory councils project reviews or that it considers all risks before departments consistently. develop their projects. In its oversight role for IT projects in The State has more than 2,500 development, DOIT requires departments to report their completed and in-development progress and, in some cases, requires departments to hire IT projects of which 235 had development costs greater than independent consultants to oversee projects. However, we $1 million. As such, the State found that DOIT does not always use these reports effectively, needs strong leadership to nor does it request the detail necessary to properly monitor ensure departments develop IT projects. DOIT has launched several programs for overseeing systems effectively. projects from planning through evaluation that may address these issues, but it has yet to fully implement these programs. 1 Although DOIT is responsible for overseeing departments’ IT efforts, departments are primarily responsible for planning, developing, and ensuring the quality of their IT projects. When we reviewed four major IT projects recently under development, we found that two projects had significant management problems. The Department of Transportation (Caltrans) undertook a major project to redesign how it collects and accounts for toll bridge charges without first establishing a supportable justification for the redesign. This particular project, which started in 1993, is about five years behind schedule and is expected to cost $28 million more than originally estimated. The Department of Health Services (Health Services) has been enhancing a management system for cases involving children from low-income families and individuals who have certain genetic diseases but is currently rethinking the technical direction of this project. Meanwhile, the project is expected to cost more than $10 million, almost double the original estimated cost of $5.6 million. The projects at the Employment Development Department and Franchise Tax Board were generally better managed but still experienced some cost overruns and delays. In addition, DOIT has not consistently fulfilled other responsibilities intended to improve the State’s management of IT projects. One area of particular concern is that DOIT lacks a mechanism to consistently assist departments in identifying and coordinating similar IT projects. Further, state law requires DOIT to maintain an inventory of IT projects, which could improve coordination and oversight, but it has not continuously maintained such an inventory and has only recently surveyed departments to collect sufficient information. In addition, DOIT has only recently prepared drafts of several key standards that establish common rules for IT projects, and has not yet begun to develop other standards. Also, DOIT has not consistently used two state-mandated councils established to provide advice on its activities. DOIT’s lack of progress on these requirements and on some past initiatives could lessen its credibility. Further, DOIT’s lack of an internal strategic plan to prioritize its use of resources while facing the turnover of key managers and other challenges has contributed to DOIT not making more progress in meeting its statutory responsibilities. 2 RECOMMENDATIONS To provide strategic guidance for IT activities, DOIT, in conjunc- tion with others, should update the statewide plan to address pressing IT concerns, priorities, and measurable objectives. DOIT should then ensure that it reviews department-level strategic plans for appropriateness and consistency with the State’s plan. To improve its oversight of departments’ IT efforts, DOIT should continue to reengineer its project review, approval, and monitoring processes and promptly implement new processes as needed. However, for any new oversight process, DOIT must include sufficient documentation of project analyses to demonstrate that it thoroughly evaluated the projects, require departments to report relevant information about the progress of their efforts, and ensure that DOIT staff receive and effectively use project progress and oversight reports. To organize and focus its efforts, DOIT should adopt an internal strategic plan to identify key responsibilities and establish priorities. Further, DOIT should establish a formal mechanism to promote coordination of IT projects. To facilitate this coordination, DOIT should complete the IT project inventory based on its survey of departments, ensure that the reported data is accurate, update the inventory as it receives new data, and consider how departments and the Legislature can effectively access this information. In addition, to improve compatibility and help standardize IT processes, DOIT should expedite its efforts to implement standards by focusing on standards it determines are high priority. Further, it should work with departments to ensure that all necessary standards have been implemented. DOIT also should consistently use the two advisory councils as intended by state law. Finally, to help ensure that IT projects are implemented effec- tively, departments need to better manage the development of their IT projects. In particular, Caltrans and Health Services should follow best practices when obtaining the services of vendors and follow sound project management practices. AGENCY COMMENTS DOIT believes that our recommendations are consistent with the new enterprise direction it is taking and has reported progress on each of the recommendations. DOIT raises some concerns with 3 the audit work that established the basis for the recommenda- tions; however, its response contains numerous incorrect or misleading statements. Our comments on DOIT’s response begin on page 141. Caltrans, Health Services, and the Employment Development Department all agree with the recommendations that we made to them. In its response, the Franchise Tax Board provided us n additional information on its project that we reviewed. 4 INTRODUCTION BACKGROUND B ecause of its vast power to process information quickly, information technology (IT) continues to revolutionize most aspects of industry and government. The computer’s ability to quickly process large amounts of information is vital to virtually all government services––whether providing subsistence payments to the needy, collecting taxes, operating the State’s water storage and delivery systems, or authorizing health care for low-income families. The public’s apprehension of the year 2000 computer “glitch” indicated how much we depend on IT in our daily lives. Developing a major IT project can be a complex, costly, and time-consuming task involving the efforts of many state employees and often the assistance of outside consultants. Because it demands a significant commitment of resources—both financial and human— developing an IT project is disruptive to an organization and may shift its focus away from its primary mission. IT development is not without risks. Many IT projects cost more and take longer to complete than originally planned, while others are abandoned when concerns mount regarding cost overruns or system malfunctions. Estimates indicate that the State spends more than $2 billion annually on IT projects. Before 1996 the Office of Information Technology within the Department of Finance had primary responsibility for the review, approval, and oversight of IT projects in the State. However, concerns over the Office of Information Technology’s ineffectiveness led to a change in how the State monitored departments’ IT efforts. In 1995, following legislative hearings and various reports on failed IT projects, including a report from the Bureau of State Audits in 1994, state law was enacted to reengineer oversight of IT development. A major focus of the state law, which became effective in January 1996, was establishing the Department of Information Technology (DOIT). Designed to improve the State’s ability to apply IT effectively, DOIT was responsible for giving departments guidance and leadership in all their IT efforts. 5 THE STATE HAS EXPERIENCED SEVERAL COSTLY INFORMATION TECHNOLOGY FAILURES Past IT failures have been very costly to the State. In 1997, after spending more than $111 million, the State abandoned development of a system to establish a statewide automated network for tracking child support payments. A new statewide system––known as the California Child Support Automation System––is scheduled for full implementation in 2005. The delay in implementing this system left the State subject to the penalty of a reduced federal share of administrative expenditures for child support. The penalties are expected to continue through fiscal year 2004-05 and could reach $1 billion, according to recent estimates by the Legislative Analyst’s Office. Although a private vendor that the State hired was responsible for many of the problems associated with the project’s failure, two departments—the Department of Social Services and the Health and Welfare Agency Data Center (now the Health and Human Services Agency Data Center)—were also at fault, particularly because they mismanaged the project and did not take sufficient action to address quality assurance warnings. Other major project failures, including those at the Department of Motor Vehicles and the Department of Corrections, have cost the State and taxpayers about $400 million. Moreover, the costs related to these failures are even greater considering the inefficiencies and service delivery problems that departments continue to experience because they could not implement the projects intended to remedy those problems. A major reason for DOIT’s existence is the Legislature’s Authority of the intention that such failures would not be repeated. State’s Chief Information Officer • Develop policies and requirements for carrying out DOIT’s responsibilities. DOIT HAS BROAD AUTHORITY TO (cid:127) Review proposed department IT projects for compliance with statewide strategies, policies, PERFORM ITS RESPONSIBILITIES and standards, including project management methods. DOIT’s charter is to provide leadership, guidance, (cid:127) Monitor department IT projects and and oversight for the IT projects initiated by recommend remedial measures to ensure state departments. DOIT’s role is crucial, continued compliance with statewide strategies, policies, and standards, including especially when it oversees how departments project management methods. develop their projects, because it must identify (cid:127) Suspend, reinstate, or terminate IT projects. and correct problems before they become costly disasters. The director of DOIT, who is also the State’s chief information officer (state CIO), is 6 given broad authority to carry out DOIT’s responsibilities and reports to the governor. The governor appoints the state CIO with the concurrence of the senate. The current state CIO was appointed in February 1999. In addition to attempting to fulfill its responsibilities mandated by law, DOIT put significant energy into leading the State’s year 2000 efforts. These efforts were needed to resolve a problem caused by an old design convention calling for computer systems to use only the last two digits of a year. This convention could have produced invalid results or system failure on January 1, 2000, because systems would read the “00” as 1900 rather than 2000. In late 1997 the Legislature designated DOIT to oversee, coordinate, and report on the departments’ progress in fixing their IT systems. Because of the potential impact to the State, departments were required to focus on the year 2000 effort and could not undertake new IT projects unless mandated by law. Thus departments had to defer development of many IT projects until they addressed how to resolve the year 2000 problem in their completed projects. In fiscal year 1996-97, its first full year of operation, DOIT was budgeted to have 20 positions. By fiscal year 1999-2000, DOIT had 61 positions, of which 17 have limited terms. In 2000 DOIT analyzed its responsibilities and received approval to increase authorized staffing to 80 positions for fiscal year 2000-01. Because of the complex nature of its work, 47 of the budgeted positions are at the manager level or above. According to DOIT and a consultant it hired, this level of staffing is sufficient for it to perform its statutory responsibilities. As of January 2001 DOIT had filled only 62 of the 80 positions. DOIT was created with a sunset provision, which was initially July 1, 2000, but was extended to July 1, 2002. Sunset provisions are uncommon in California but are used by some states to periodically evaluate the need for various state departments to exist based on their effectiveness. Unless the current law is amended, DOIT will cease operations. DOIT has a budget of $11.8 million for fiscal year 2000-01. 7 THE STATE’S INFORMATION TECHNOLOGY GOVERNANCE STRUCTURE IS COMPLEX The State’s ability to manage its IT efforts is complicated by the interrelationships of many state agencies, departments, and other parties. (cid:127) The governor establishes a vision for how the State will use IT and provides direction to departments on how to implement this vision. (cid:127) The Legislature authorizes project development by appropri- ating funds. (cid:127) The state CIO monitors most state departments’ implementa- tion of the governor’s vision and oversees DOIT. State law makes the state CIO the leader for the State’s IT efforts. (cid:127) Agency secretaries oversee the planning, development, and operation of IT systems within their agencies. (cid:127) Departments plan, develop, and operate IT projects to carry out their responsibilities. (cid:127) Department CIOs are primarily responsible for coordinating their departments’ IT activities. (cid:127) Department project managers manage the development of their departments’ IT projects. (cid:127) Department of Finance approves funding for IT projects primarily through its specialized IT review unit called the Technology Investment Review Unit. The Department of Finance has authority over departments’ budget activities. (cid:127) Department of General Services (General Services) adminis- ters the procurement of IT goods and services from private vendors, primarily through its procurement division. General Services has authority over departments’ procurement activities. (cid:127) Data centers provide data processing services to departments and, in some instances, help evaluate vendors’ bids prior to vendor selection and contract award. The data centers include the Stephen P. Teale Data Center, the Health and Human Services Agency Data Center, and the Hawkins Data Center in the Department of Justice. 8 (cid:127) Advisory councils are intended to assist the state CIO by providing advice and input on IT strategy and policies. (cid:127) Director of eGovernment advises the governor on policy and coordinates with DOIT and other state departments on the State’s electronic government (eGovernment) initiative, which is designed to promote the provision of services and informa- tion by state government to the public through the Internet. (cid:127) Private vendors and consultants provide software develop- ment services, computer hardware, and other specialized technical services to state departments. Some entities are exempt from the purview of DOIT. These include the California State University, the University of Califor- nia system, the Public Employees’ Retirement System, and the California State Lottery. These entities initiate, approve, and procure their own IT projects and are ultimately responsible for their development. INFORMATION TECHNOLOGY DEVELOPMENT REQUIRES SEVERAL STEPS The IT development process involves a number of steps, from planning through evaluation of the project, as shown in Figure 1. Departments are responsible for developing IT projects with the approval and oversight of the control agencies—DOIT, the Department of Finance, and General Services. To obtain approval for an IT project, a department must prepare a feasibility study report. This report provides the business reasons to justify the investment of state resources in the project, the reasons for undertaking the project, the means of ensuring the success of the project, and a comprehensive analysis of its costs and benefits. Often a department must work with a private vendor to develop an IT project. Unless the department is delegated the authority to make its own purchases, General Services helps the department procure the appropriate vendor. However, the department is responsible for the proper management of all development efforts. Once development is completed, the department must evaluate whether it achieved the anticipated benefits of the IT project. 9 FIGURE 1 The State’s Information Technology Development Process PLANNING APPROVAL PROCUREMENT IMPLEMENTATION EVALUATION DOIT is to develop a DOIT is to approve Unless the department is The department and The department statewide strategic plan, technical aspects of the delegated the authority the vendor, if used, prepares a while each department project and determine to make its own develop and implement postimplementation is to develop IT consistency with the purchases, General the IT project. DOIT evaluation report to strategies. When statewide plan, policies, Services assists the and, in certain assess how well the proposing an IT project, and standards. The department in planning instances, independent completed project met a department prepares a Department of Finance the acquisition strategy consultants, provide its goals. DOIT is to feasibility study report approves the business and in preparing a bid project oversight. review and approve this for approval from DOIT case and determines if solicitation document, report. and the Department of the project is a sound usually a request for Finance. investment of state proposal. General funds. Services issues this document, receives vendors' proposals, conducts discussions with vendors, negotiates contract language changes, coordinates with the department to evaluate proposals, and determines the winning vendor. In some instances, the preparation of the request for proposal may come before a department develops a feasibility study report. THE GOVERNOR’S ELECTRONIC GOVERNMENT INITIATIVE In September 2000 the governor issued an executive order to direct the State’s eGovernment initiative, which is intended to encourage departments to offer services and information to the public through the Internet. The premise is that eGovernment will act as a catalyst for departments to design better ways to provide government services. The executive order also established a director of eGovernment to advise the governor and coordinate these activities with DOIT and departments. A successful eGovernment program depends on a statewide infrastructure that is compatible and allows the sharing of information among systems. Thus the eGovernment initiative will be a major focus of the State’s future IT activities. 10 SCOPE AND METHODOLOGY The Joint Legislative Audit Committee requested that the Bureau of State Audits examine the State’s processes for approving, funding, planning, and monitoring IT projects to determine whether they are adequate to protect the State’s multibillion- dollar investment in these efforts. We were asked to review a number of specific areas, including strategic planning for IT activities, the project approval process and coordination of similar IT activities, the management of a sample of projects under development, the procurement of private vendors to assist in development of IT projects, and the sufficiency of DOIT’s resources to perform its duties. In addition, we were asked to compile an inventory of the State’s major IT projects. To gain an understanding of the requirements for planning, designing, procuring, and implementing IT projects, we reviewed relevant state laws and policies. We learned that many of the areas we were asked to review were under the direct responsibility of DOIT. For example, state law makes DOIT responsible for developing or ensuring the development of the statewide vision, strategies, plans, policies, requirements, standards, and infrastructure for IT and ensuring that departments’ proposed IT projects are aligned with appropriate strategies, policies, and standards. Thus we focused our work on DOIT and to a lesser extent on General Services, which is responsible for conducting procurements, and the Department of Finance, which is responsible for approving the funding for IT projects. In addition, we evaluated the management of one IT project each at the Employment Development Department, the Franchise Tax Board, the Department of Health Services, and the Department of Transportation. To further understand the State’s IT environment, we inter- viewed 17 CIOs: 12 from departments that report to DOIT and 5 from state organizations that are exempt from DOIT’s review. We asked all 17 CIOs a series of questions to learn their concerns and perspectives about the State’s IT development process. We also asked the 12 department CIOs about the role of DOIT and the other control agencies involved in IT development. Further, we used the results of interviews of CIOs to help us decide where to focus our audit efforts. We also considered IT practices used by other states to help us understand the State’s IT practices. 11 To evaluate the planning process for IT projects, we reviewed the State’s and selected departments’ strategic IT plans. Further, we also selected a sample of DOIT-approved projects to assess the adequacy of the project approval process. For some projects, departments can request an exemption from DOIT’s review. Because DOIT reported approving only eight requests during 2000, of which at least four appear to be for hardware or commercial software, we did not review how DOIT granted exemptions. In addition, we evaluated how DOIT assesses the departments’ risk planning for their projects. To learn about DOIT’s operations, we interviewed its management and staff and examined its procedures for approving IT projects. To evaluate the effectiveness of DOIT’s oversight of IT projects under development, we determined if DOIT received project status reports and independent consultant reports on these projects and reviewed its analysis of these documents. We also reviewed whether DOIT required departments to evaluate the benefits received from completed IT projects. To assess whether departments have properly planned, approved, procured, managed, and evaluated their IT projects, we reviewed four IT projects recently under development. We reviewed a 1998 report from the Legislative Analyst’s Office on IT best practices, interviewed department staff and the private vendors they hired, and analyzed various project documents, such as planning documents, to determine if departments were following best practices during project development. In addition to the Legislative Analyst’s Office’s best practices, we considered those included in state requirements, the Project Management Body of Knowledge (PMBOK), and the Control Objectives for Information and related Technology (COBIT). We then rated how well the departments managed these four projects in accordance with the best practices. We discuss the results of this review in Chapter 2 and present the detailed ratings in Appendix A. To determine if DOIT was fulfilling its other responsibilities, we interviewed DOIT management and staff, attended meetings of department CIOs, and reviewed reports on the State’s IT devel- opment process. In addition, we reviewed DOIT’s prior annual reports to the Legislature and followed up to determine if DOIT had completed objectives that it had established. Further, we assessed whether DOIT had coordinated similar IT activities as well as performed other statutory responsibilities, including obtaining input from IT advisory councils. 12 Finally, to create an inventory of the major statewide IT projects in the planning or implementation stages, we compiled information from DOIT’s project survey of 111 state agencies, departments, boards, and commissions. For purposes of this report, we refer to all these entities as departments. We used the information as the basis for our inventory of IT projects but found that many departments did not report complete information to DOIT. We contacted departments to ask them to provide missing information and to confirm that we had properly presented their IT projects. In addition, we verified the accuracy of selected information at 5 departments for a sample of projects. We n present our inventory of IT projects in Appendix B. 13 Blank page inserted for reproduction purposes only. 14 CHAPTER 1 DOIT Needs to Provide Stronger Leadership Over Departments’ Information Technology Efforts CHAPTER SUMMARY A lthough the Department of Information Technology (DOIT) is responsible for overseeing the State’s efforts to plan, develop, and evaluate information technology (IT) projects, its leadership and guidance should be increased. DOIT has not updated the statewide IT plan to address current issues or specify current measurable objectives. Such a strategic plan— which DOIT is charged with developing—is critical to focus the State’s IT efforts. DOIT also does not sufficiently review and approve departments’ IT strategic plans. As a result, DOIT is missing opportunities to ensure that departments’ IT planning efforts are appropriate and consistent with statewide IT strate- gies. Moreover, DOIT has not always given departments clear guidance in developing and managing IT projects. In addition to not fulfilling its IT planning responsibilities, DOIT does not have a well-documented project review and approval process. DOIT has not ensured that departments assess the risks involved with IT projects before they start develop- ment. As a result, DOIT could fail to detect poorly conceived or planned projects. Once it approves IT projects for development, DOIT is responsible for overseeing departments in properly managing the projects. However, DOIT is not always using departments’ progress reports to effectively oversee IT efforts. In addition, some departments make it difficult for DOIT to properly oversee their efforts because they do not report problems when they should. Finally, DOIT’s new strategic vision for planning, approval, and oversight may address some of these weaknesses. To implement this vision, DOIT is reengineering its existing processes and began a pilot of its revised feasibility study report process in May 2001. Because DOIT has yet to finalize and implement this vision, it is too early to determine the value that it will add. 15 DOIT SHOULD IMPROVE ITS PLANNING RESPONSIBILITIES AND PROVIDE CLEAR GUIDANCE Although a statewide IT plan is required by law, DOIT has not recently updated the plan to include pressing issues or current measurable objectives. Good strategic planning would set the tone for departments’ IT efforts and enable the State to better coordinate the divergent efforts of its departments. DOIT issued the existing statewide IT plan in 1997, establishing strategies, objectives, and action items. However, DOIT should revise the plan to guide the State in dealing with critical IT issues and changes in technology. For example, over the next couple of years, various departments that provide health care services or use the related data will need to change how their IT systems collect, process, and report individuals’ health information to comply with administrative requirements stemming from a federal law passed in 1996. Further, the governor’s electronic government (eGovernment) initiative will require all departments to consider ways to deliver services to citizens over the Internet. These issues and changes in technology have significantly changed how departments should plan their IT efforts. Although DOIT has made presentations to departments regarding these issues, it has not updated the statewide IT plan to clarify how these and other potentially important matters fit into the State’s overall strategy. Few department chief information officers (CIOs) with whom we met reported using the statewide plan to guide the creation of their departments’ IT plans. In fact, many CIOs indicated they had not recently read or were not familiar with the statewide IT plan, and some stated that the plan did not include enough details on standards to be relevant to their departments’ current project development efforts. In addition, because proposed IT projects must conform to statewide strategies, departments need a current statewide IT plan to guide their development decisions. When DOIT updates the statewide IT DOIT has not updated plan, departments will be able to ensure that their IT strategies the statewide IT plan are properly aligned with the State’s IT vision. since 1997, and few departments we met with Additionally, DOIT has not established current measurable reported using it to guide objectives that would allow the State to use the statewide IT plan their IT efforts. to gauge its progress in key areas. The 1997 statewide IT plan identifies 36 action items to implement the IT strategy; however, only 6 items still add value because they are ongoing tasks or tasks that are to be carried out by this year or later. Because all 16 other action items are outdated, the State is left with few relevant items by which to measure its progress. For example, one action item recognizes the need for DOIT to encourage one of the State’s data centers to move to a more suitable facility. Because the specified data center moved into a new facility during 2000, this action item is no longer current. In the absence of an up-to-date statewide IT plan, DOIT has not provided measurable objectives in the annual reports that state law requires it to submit to the Legislature. These measurable objectives should define the results DOIT will achieve and when it will achieve them. As a result, the State lacks current, effective measures for its progress on strategic IT issues. Unlike the plans of other organizations, DOIT’s statewide IT plan does not include priorities for large projects that ensure that the most important projects are considered first. We identi- fied several organizations that use their strategic planning process to identify priority projects for their organizations. For example, the California State University, one of the entities not When updated, the subject to DOIT’s purview, surveyed its campuses to determine statewide IT plan should their IT needs and then prioritized IT funding on 11 key prioritize IT projects that projects. Transportation projects in California also go through are most important or an extensive prioritization process at local, regional, and state needed. levels to identify which projects will receive available funding. In addition, the state of New York identified 75 priority services and transactions in its plan to provide services to its citizens over the Internet. DOIT’s statewide IT plan also does not identify departments that lack sufficient funds to replace aging systems or systems that vendors no longer support. We recognize that establishing priorities for the State’s IT efforts may be a consider- able undertaking. Additionally, we recognize that DOIT will need to work in collaboration with departments, the governor, the Legislature, and the Department of Finance to establish priorities rather than establish them unilaterally. Nevertheless, we believe that including priorities in the statewide IT plan would be beneficial and that DOIT can play a key role in facili- tating this process. Without a plan that communicates statewide priorities for new systems, the State may not focus on strategic IT efforts that are most important or needed. DOIT acknowledges that the statewide IT plan is out-of-date, stating that instead of updating the plan, it focused on ensuring that the State was properly prepared through the year 2000 effort, reorganizing its operations, and addressing other issues that it considered important, such as eGovernment. Although DOIT initially proposed to revisit the plan once it completed its 17 internal strategic plan, it now intends to work on both plans concurrently. To begin the process of updating the plan, DOIT organized a work group of department CIOs that started weekly meetings in March 2001. DOIT expects to issue the statewide IT plan sometime during fiscal year 2001-02. Until DOIT estab- lishes an effective statewide IT plan, the State continues to operate without a clear direction for its IT efforts. DOIT Should Ensure That Departments Conduct Appropriate IT Strategic Planning DOIT has not sufficiently reviewed and approved the strategic plans for departments’ IT projects although it stated that it recently began to do so. Consequently, it has not consistently guided department planning efforts at the earliest stages to ensure the development of viable projects. DOIT noted recently that the review of departments’ IT plans in the past did not merit assignment of its resources because of higher priorities, including the year 2000 effort, and thus these reviews occurred only sporadically. However, since 1996 DOIT has been responsible for approving departments’ IT strategies. Despite the absence of DOIT involvement, departments are continuing to prepare their strategies and implement projects accordingly. Of eight departments we reviewed, all had prepared plans between 1997 and 2000, but DOIT had reviewed none. Without DOIT’s review and approval, departments’ IT strategic plans may have weaknesses, be inconsistent with the statewide IT plan, or in the absence of an updated statewide plan, reflect philosophies that DOIT believes are inappropriate. Insufficient IT plans among state departments are not new occurrences. The former Office of Information Technology (OIT) DOIT does not review under the Department of Finance found numerous weaknesses departments’ IT plans when it analyzed departments’ IT plans before 1996. For example, despite past problems OIT found that the IT plan prepared by the Employment with such plans. Development Department in June 1994 provided no basis for prioritizing projects, lacked a coordinated approach, and failed to fulfill 9 of 19 criteria on the review checklist that OIT used to assess the adequacy of departments’ IT strategic plans. To help with what it believed was an increasing workload of IT strategic plan reviews, DOIT requested and received additional staff but subsequently assigned them to other tasks. For the fiscal year 1998-99 budget, DOIT requested and received two data processing managers and approximately $437,000 to contract for outside assistance in reviewing departments’ IT plans. 18 However, DOIT reassigned the staff to work on the year 2000 effort, which was a higher priority at the time. DOIT stated that it began reviewing departments’ IT plans on a case-by-case basis in May 2001, and it plans to redefine this process sometime in early 2002. Departments Receive Unclear Guidance for Sources of DOIT Policy for Preparing Managing IT Projects Feasibility Study Reports Departments receive unclear guidance because Management Memos DOIT does not always consolidate, update, or 1. 98-02 Fundamental Decision Criteria for clarify its policies. State law charges DOIT with Approval of IT Projects updating its policies to reflect the State’s 2. 98-12 Acquisition and Management of changing IT needs and publishing them in the Major IT Projects State Administrative Manual or in Management State Administrative Manual 3. Section 4819.35 Feasibility Study Report Memos. Departments need clear and up-to-date policies to guide them when researching the 4. Sections 4920-4930.1 Feasibility Study Policy feasibility of IT projects. Although DOIT has Statewide Information Management Manual published policies, it has not consolidated them 5. Volume I Section 5.0 Feasibility Study to improve departments’ ability to follow them Report Policy when drafting feasibility study reports. 6. Volume II Section 5.0 Feasibility Study Currently, departments must search at least six Report Guidelines sources of policy to understand DOIT’s guidance and directives to submit their feasibility study reports. This lack of policy consolidation also extends to other com- monly used IT approval documents. For example, to prepare IT procurement documents, which allow departments to seek vendors to assist in their development efforts, departments must consult seven sources. These multiple sources of policy can create confusion and misunderstanding. DOIT contends that, because it works with CIOs to develop policies and processes, this practice reduces misunderstanding and confusion. Never- theless, departments could more readily understand DOIT’s guidance on elements of the IT development process if they were defined in fewer policy sources. We also found that DOIT still publishes some rescinded policies that conflict with current policies. Departments’ IT efforts are guided by DOIT policies, which must be clear and not in contra- diction with each other so that departments can understand them. However, DOIT has not removed or marked as rescinded several sections of published policy that have been superseded by policies it has subsequently released. For example, DOIT published sections of Volumes I and II of the Statewide Information Management 19 Manual in July 1998 to replace seven sections of the State Ad- ministrative Manual but did not remove those seven sections from publication. Moreover, the State Administrative Manual does not always reflect current policy and IT trends because the last update to any IT section was in 1997, and many parts were Because the last update last updated before 1994. Thus departments could be following to the IT section of the guidance contained in DOIT policies that have been rescinded, State Administrative conflict with existing policies, or are no longer relevant. Manual was in 1997, it does not always reflect Additionally, although DOIT provides guidance to departments on current policy and IT the alternative procurement process through a management trends. memorandum and discussions with departments, it has not updated contradictory guidance contained in the State Administrative Manual. The alternative procurement process is one in which a department describes its business need and vendors propose their designs of the IT project to meet this need rather than the traditional process of procuring a vendor after approval of the feasibility study report. In 1998 DOIT published a memorandum requiring that departments use the alternative procurement process for all IT projects that cost more than $5 million and are critical to department operations. However, in the section of the State Administrative Manual for which DOIT is responsible, the alternative procurement process is described as a pilot, and it is left up to departments to request authorization to use it if the departments believe it might be beneficial. Additionally, although DOIT’s memorandum apparently man- dates the use of the alternative procurement process for large, critical projects, DOIT and the Department of General Services (General Services) have yet to evaluate and formalize the pilot process as called for in the State Administrative Manual. This administrative policy states that within two years after approval of the first pilot procurement, the directors of DOIT and General Services would assess the process to determine its appro- priateness and suitability for continued use. The first alternative procurement, according to General Services, was awarded in 1994. Thus the process should have been evaluated by now. DOIT assumes that the previous management’s 1998 memo- randum requiring that departments use the process was part of the assessment needed. General Services told us that assessing the alternative procurement process is overdue. An evaluation of the alternative procurement process would be valuable because it could be used to provide information to departments on how the process could be most effectively used. DOIT stated 20 that it is currently collaborating with General Services and the Department of Finance to further refine the process, including changes to the State Administrative Manual. Based on our interviews with staff from DOIT, the Technology Investment Review Unit within the Department of Finance, and General Services, the alternative procurement process is not only beneficial for the State but should be used for large IT projects that are critical to departments’ operations. Most of the CIOs we interviewed said they have used the alternative procurement process. Although DOIT and General Services are both responsible for assessing the value of the pilot project, DOIT should take the lead because state law makes it responsible for procurement policy. However, DOIT will need the assistance of General Services, which administers procurements for state departments, to make this evaluation meaningful. DOIT SHOULD DOCUMENT ITS BASIS FOR APPROVING PROJECTS AND ENSURE THAT DEPARTMENTS PROPERLY ASSESS RISKS Weaknesses in DOIT’s review of feasibility and risk for proposed IT projects could result in its failing to detect poorly conceived efforts. One weakness is that DOIT often does not document the basis for its decisions to approve IT projects. In addition, despite the fact that IT projects are Objectives of Evaluating IT Projects inherently risky, DOIT does not ensure that Against DOIT’s Fundamental departments appropriately assess their risks. Decision Criteria (cid:127)Supports department’s mission as defined in its business plan. DOIT’s Project Review and Approval Process Is (cid:127)Is compatible with state, agency, and Not Well Documented department IT plans. DOIT does not adequately document its process (cid:127)Supports business functions that have been optimized and redesigned. for reviewing proposed IT projects. Thus it cannot demonstrate it has consistently and sufficiently (cid:127)Demonstrates a clear return on the State’s investment. analyzed whether departments are properly (cid:127)Secures the commitment and sponsorship of conceiving and planning IT projects. When executive management. establishing DOIT, the Legislature made it (cid:127)Has a project manager and team with responsible for ensuring that proposed IT projects appropriate training and experience. comply with appropriate statewide strategies, (cid:127)Is structured with clear project management policies, standards, and project management and control methodologies for tracking practices. DOIT issued a policy defining project progress. fundamental decision criteria to help ensure that (cid:127)Employs best practices for procurement. departments developed IT projects properly. In 21 addition, DOIT developed a checklist incorporating various criteria, including the fundamental decision criteria, to ensure that projects meet all applicable criteria for support and approval. In early 2000 DOIT also developed a project summary intended to support its analyses of projects. However, DOIT could not provide us with sufficient evidence that it thoroughly analyzed any of the 10 projects we reviewed. For 10 proposed The development costs for each project ranged from about projects reviewed, with $280,000 to more than $11 million and totaled roughly development costs $35 million. Although DOIT provided its approval letters, totaling $35 million, which included the conditions imposed on the projects, its DOIT could not provide documentation of its analyses of these projects was insufficient sufficient evidence that it for several reasons. For 5 of the 10 projects we found no thoroughly analyzed evidence—such as checklists, project summaries, or other them. documents—of DOIT’s analyses. For the remaining 5 projects we found some analysis but not enough to demonstrate why DOIT approved the projects. In the five projects showing some analysis occurred, DOIT gave us the questions it asked departments about their projects. Instead of providing grounds for project approvals, however, the questions and responses only supplied additional information or clarity not found in the departments’ feasibility study reports. In addition, some of these five projects included other analysis documents, such as a checklist or project summary. However, none of this analysis was sufficient to demonstrate why DOIT approved the projects. For example, one of the five projects contained a project summary, but it was very brief and the analyst noted when recommending approval for the project that DOIT had already issued a letter supporting the project. Thus DOIT’s documentation of its analyses of these five projects was insufficient. DOIT indicated that it approved many of these projects while it was focusing on the year 2000 effort and that it did not have enough resources to consistently perform thorough project reviews and oversight or to completely document the work performed. However, given the State’s investment in these projects, it is crucial that DOIT ensures it documents that it has thoroughly analyzed proposed projects. In contrast, when we reviewed the process that the Department of Finance’s Technology Investment Review Unit (unit) uses to review and approve the funding of proposed IT projects, we 22 found it to be consistent and well documented. The unit focuses its evaluation on the investment value and funding of a project, Lacking evidence of while DOIT’s review focuses on the proposed technology. The approval decisions, DOIT unit documents its review in a project analysis using guidelines would be hard-pressed to in its employee review handbook, and the analysis clearly explain why it approved indicates the reasons the unit recommends approval of a a project if that project project’s funding. Therefore, we found appropriate evidence to should fail or experience support the unit’s approval decision on each of the same 10 pro- significant difficulties. posed IT projects we reviewed. Without thorough documentation of its approval decisions, DOIT would be hard-pressed to explain why it approved a project costing the State millions of dollars if the project were to fail or experience significant difficulties. In lieu of a complete record of its approval, DOIT must rely on the memories of its staff, but high staff turnover makes it even more difficult to reconstruct the circumstances of an approval. Furthermore, the current level of documentation could prevent DOIT from learn- ing from past mistakes and improving its project approval process. Although a proposed revision to the approval process for feasibility study reports might address some weaknesses, DOIT has not yet finalized and implemented the revision. DOIT began a pilot of the new process in May 2001, and the pilot is tentatively scheduled to last two to three months. DOIT expects to use this new process for all proposed IT projects beginning in early 2002. Therefore, we could not evaluate whether this new process will fully address these weaknesses. DOIT Is Not Adequately Ensuring That Departments Assess Project Risk To assess the risks associated with implementing IT projects, DOIT has a risk assessment model but applies it in a way that weakens its usefulness. IT projects are inherently risky because many unanticipated events occur during their development, and often outcomes are unknown. Thus a key step in planning is to identify these risks and develop ways to mitigate or lessen their potential impact. DOIT developed a risk assessment model as an automated tool to help departments identify project risk. This model categorizes project risks into five distinct areas. DOIT requires departments to use the risk assessment model and reviews the risk assessments as part of the project approval process. DOIT states that its staff use risk assessments, as well as all other information provided by departments, to recommend 23 whether to approve projects and suggest actions that DOIT and the departments might take to Five Areas of Project Risk That DOIT’s minimize project risks and ensure project success. Risk Assessment Model Evaluates (cid:127) Strategic risk: the degree to which the However, in our review of the 10 proposed IT proposed project is in alignment with business strategies. projects we discussed in the previous section, we found little evidence that DOIT evaluates depart- (cid:127) Financial risk: the probability that the organization will be able to secure funding ments’ risk assessments. DOIT indicated that its for the entire project life cycle from analysis should appear in various documents, sponsoring agencies. such as the project summaries, the checklist that (cid:127) Project management risk: the impact on all includes the fundamental decision criteria, and areas of project management necessary to complete the project, including a realistic questions to departments. However, as discussed time frame, sufficient resources, necessary previously, DOIT did not sufficiently document skill levels, and a sound project management approach. its analysis of the 10 IT projects we reviewed. DOIT has stated that documenting processes was (cid:127) Technology risk: the degree to which the project must rely on new, untested, or not its top priority while it was working on the outdated technologies, including hardware, year 2000 effort because of conflicting priorities software, and networks. and limited resources. Nevertheless, without a (cid:127) Change management and operational risk: thorough review of IT project risk, DOIT cannot the amount of change needed within the organization as well as the effort required for ensure that departments have identified and continued operations at project completion. mitigated all project risks. In addition, although DOIT received risk assessment summary reports for 9 of the 10 projects we reviewed, it received sufficient information to evaluate the risk-planning efforts of only 3 projects. For the remaining 6 projects that submitted reports, the departments did not answer all questions required, and 3 included only summaries of the final risk scores and omitted responses to questions about risk. Without sufficient responses to these questions, DOIT cannot identify specific risks that may need further questioning and clarification. For example, 1 of the 8 questions used to assess project management risk deals with the level of experience of the project management team, a crucial factor in evaluating project risk. Additionally, DOIT does not require departments to answer the same questions when submitting their risk assessments—the number to be answered ranged from 5 to 38. Because some departments do not consistently answer questions about risk, DOIT may not be able to use the risk assessment to ensure that departments consider all risks before developing their projects. Further, DOIT allows departments to apply the risk assessment model late in the approval process of large, critical IT projects that are required to use the alternative procurement process. 24 Therefore, IT project risks are not formally assessed before departments have started undertaking significant planning efforts. In these instances departments begin planning their IT projects and conduct procurement with potential vendors, including evaluating proposals, selecting technical solutions, and negotiating contracts, before performing formal risk Under DOIT guidelines, assessment. For example, more than two years of planning can more than two years of pass on some projects before departments must conduct a risk planning can pass before assessment, per DOIT’s guidelines. departments assess risk on large, critical projects. DOIT states that until departments complete the alternative procurement process, they do not know enough details about the projects to evaluate most of the factors that the risk assess- ment model measures. We disagree. Of the 38 questions that DOIT may require departments to answer, we identified at least 18 that departments could address when starting an alternative procurement to give some indication of risk. For example, departments could answer questions about whether projects are aligned with their business plans, the experience of the project management team, and expected development costs. By not considering risks early in the alternative procurement process, departments might overlook problems that could be corrected before they are too expensive and time-consuming to remedy. As part of its revised feasibility study report process, DOIT indicated that it also plans to redesign its risk assessment model. DOIT COULD IMPROVE ITS OVERSIGHT OF DEPARTMENTS’ INFORMATION TECHNOLOGY EFFORTS In several of the State’s major IT project failures over the past decade––for example, the statewide automated network for tracking child support payments and a failure at the Department of Motor Vehicles––poor project management by departments was a common factor. One of DOIT’s key responsibilities is to monitor departments’ implementation efforts and thus help avoid such failures. To carry out this responsibility, DOIT is to oversee the development and management of IT projects to ensure that departments comply with policies and standards. However, DOIT has not always fulfilled its oversight responsibility and, as a result, has not positioned itself to prevent potentially unnecessary and costly project delays. 25 DOIT Is Not Always Using Departments’ Project Reports as Oversight Tools DOIT does not always use the periodic progress reports or the reports submitted by independent validation and verification (IV&V) consultants to properly oversee departments’ IT projects. When DOIT approves a project, it typically requires the depart- ment to submit periodic progress reports to summarize the status of the project. Additionally, departments sometimes use an IV&V consultant to evaluate the primary vendor’s perfor- mance by reviewing planning documents, assessing the quality of the project’s system design, going over the computer code being written, and a variety of other tasks. The IV&V consultant makes recommendations to the department on how to obtain a better quality product from the primary vendor and thus can give DOIT insights on potential problems. Because DOIT does DOIT may be late in not always use periodic progress reports or IV&V consultant detecting problem IT reports effectively, it may be late in detecting problems and is projects because it does more reactive than proactive in dealing with troubled projects. not always use departments’ reports We reviewed nine projects that recently submitted special effectively. project reports, which are generally required when a project exceeds or expects to exceed the original cost or timeline by more than 10 percent. Based on our review of the periodic progress and IV&V reports for these nine projects, dating from October 1999 to September 2000, and our discussion with DOIT analysts, we found limited evidence that DOIT used the reports as tools for overseeing these projects. Moreover, it is question- able whether DOIT uses the IV&V reports for oversight because DOIT only had 33 percent (12 of 36) of the IV&V reports that it should have received for the nine projects we reviewed during a one-year period, as shown in Table 1. We also observed that DOIT did not always have the periodic progress reports that departments are supposed to submit. For some projects, DOIT required quarterly reports and for other projects, monthly reports. Specifically, for two of the nine projects that we reviewed, DOIT had none of the department’s quarterly reports for one project and only 2 of the 12 monthly reports for the other. For three other projects, DOIT was missing 1 or 2 reports. It is difficult for DOIT to conduct effective oversight if it does not have the departments’ reports on the progress of their IT projects. According to DOIT, all departments were to report on their projects monthly as of the fall of 2000. 26 TABLE 1 Status of IV&V Reports on Nine Projects Experiencing Significant Cost and/or Schedule Overruns (October 1999 Through September 2000) IV&V Reports Department Project Name Received Expected Controller, State Automated Travel Expense 0 4 Reimbursement System Corrections, Department of Revocation, Scheduling, and NA NA Tracking System Employment Development Tax Engineering 5 12 Department and Modernization Franchise Tax Board Accounts Receivable 1 4 Collection System* Health and Human Services Electronic Benefits Transfer NA NA Agency Data Center Health and Human Services Statewide Fingerprint 6 12 Agency Data Center Imaging System Health Services, Department of Medi-Cal Management 0 4 Information System/Decision Support System Transportation, Department of Advanced Toll Collection NA NA and Accounting System Water Resources, Department of Cost Allocation and NA NA Repayment System Totals 12 36 * Instead of an IV&V consultant, this project had a project oversight consultant who focused primarily on the financial, personnel, schedule, and documentation matters related to project management. NA = Not Applicable—DOIT did not require IV&V reports on these projects. Additionally, although DOIT now requires monthly reporting, it does not require specific information that would help it measure a project’s progress against planned timelines and resource usage. For example, although DOIT requires a monthly narrative on project status, including project schedule dates and expenditures by fiscal year, it does not require monthly reporting of two types of critical information. First, it does not require the department to compare the project’s monthly actual costs and revised estimates of total projected costs with the budget. Second, DOIT does not require departments to compare actual and revised projected completion dates for project phases with the original schedule. This information would be readily available if departments had 27 sound project management plans to appropriately track and monitor their efforts. As a result, DOIT is not informed when a project first starts to miss targets for cost and completion date. DOIT may receive this additional information when depart- ments submit special project reports. However, departments submit special project reports only after significant changes have occurred or are expected to occur to their IT projects— typically increases or changes in cost or schedule of more than 10 percent. Relying exclusively on special project reports does not assist DOIT in the ongoing oversight of projects before they have become recognized problems. The Advanced Toll Collection and Accounting System (ATCAS) project provides a good example of the need for DOIT to adequately monitor a project and require regular reports to learn when a project is experiencing significant cost overruns, delays, and problems. When approving the feasibility study report in 1996 that the Department of Transportation (Caltrans) submitted to justify the need for the project, DOIT stipulated that Caltrans provide periodic progress reports. However, the Because it did not require reports were due only at specific milestones: at the successful periodic reports on a completion of three specific phases of the ATCAS development. Caltrans project over a However, when Caltrans redefined these phases in May 1998, it four-year period, DOIT did not report the changes to DOIT. It also did not report on its was not informed when progress to DOIT during the following two years of the project the costs significantly because it did not successfully complete any of the redefined increased—ultimately phases. As a result, Caltrans submitted no progress reports to from $27.8 million to DOIT for a four-year period after DOIT approved the project, $56.1 million. even though the project experienced significant changes from what was envisioned. Finally, in August 2000 Caltrans submitted a special project report requesting approval for cost increases and delays on ATCAS. Ultimately, estimates for one-time costs—costs to fully develop this project but not for continuing operations and maintenance—had increased from the original $27.8 million to $56.1 million. DOIT would have been aware of project slippage as it occurred if it had required monthly reporting of two critical pieces of information: the project’s monthly actual costs and revised estimates of total projected costs compared with the budget, and actual and revised projected completion dates for project phases compared with the original schedule. DOIT has begun to reengineer its project oversight by using performance metrics to evaluate the status of IT projects under development. DOIT believes the use of metrics to track key 28 project characteristics will allow it to rate how well projects are progressing and act as an early warning of troubled projects. DOIT used a similar process to monitor departments’ efforts during the year 2000 effort. However, it is too early to determine the extent to which this process will enhance DOIT’s oversight. DOIT Is Not Receiving Information on All Projects Under Its Oversight Departments do not always submit special project reports when they should, making it difficult for DOIT to properly oversee their efforts. For example, the ATCAS project experienced major cost increases and schedule delays after DOIT approved the feasibility study report in 1996. Despite the need to do so sooner, Caltrans did not report these problems until August 2000 when it submitted a special project report. In another instance the Department of Corrections significantly altered the scope of an approved project, causing one-time costs to increase from an originally estimated $3.7 million in July 1997 to $6 million in December 2000. However, the Department of Corrections submitted a special project report requesting funding after rather than before it made the alterations. In both cases the Department of Finance approved the continuation of project expenditures but expressed concern about the lack of reporting and asked DOIT to remind the departments of their responsibility to report properly. Departments are primarily responsible for the planning, devel- opment, and quality assurance of their projects. In passing the Departments that do not legislation that established DOIT, the Legislature intended that report to DOIT as they DOIT would be a means of helping to ensure the success of the should frustrate the intent State’s IT efforts. Therefore, departments must come forward of DOIT’s oversight role. with their problems so that DOIT can be in a position to assist them. However, if departments do not report to DOIT as they should, they frustrate the intent of DOIT’s oversight role. DOIT Has Not Ensured That Departments Evaluate Completed Projects DOIT has not ensured that departments submit reports evaluating their IT projects after completion. Moreover, for the relatively small number of postimplementation evalua- tions (evaluations) it has reportedly received, DOIT has not demonstrated that it performed the analyses necessary to ensure that projects are meeting departments’ goals. As a 29 result, departments have not been held accountable for the promised benefits from planned IT projects. DOIT believes that the current evaluation report process does not provide value, and it plans to reengineer the process. Under state policy, departments must evaluate the success of completed IT projects and compare the original timelines, costs, and benefits with the actual by preparing evaluation reports. In Evaluations of completed addition, departments must provide analyses if projects are not projects are needed to successful or had significant differences between original and hold departments actual timelines, costs, or benefits. In these instances the analy- accountable for the ses must include the actions the departments intend to take to estimated costs and improve outcomes. DOIT’s records show it receiving roughly promised benefits of these 87 evaluations from departments since 1996. However, the projects. Legislative Analyst’s Office believes that there could be more than 400 completed IT projects for which departments have not submitted evaluation reports. By requiring departments to compare the actual costs and benefits against those included in feasibility study reports that departments must submit to justify planned IT projects, DOIT could make departments accountable for their early promises. DOIT could not provide any evidence that it analyzed any of the 87 reports it did receive. Further, as indicated by the Legislative Analyst’s Office in its analysis of the fiscal year 2001-02 budget, DOIT does not keep sufficient records of completed IT projects and gives the review of these reports low priority. DOIT believes that most people associated with IT in California recognize that the current evaluation report process is weak and does not provide much value. DOIT acknowledges that until recently it did not devote much time or effort to reviewing evaluation reports because it plans to reengineer the process. Part of DOIT’s reengineering is to incorporate into the evaluation process the lessons learned––successes as well as problems––during project development. DOIT states that it recently has begun to review a few evaluation reports to prepare for its reengineering effort, which it expects to complete by fiscal year 2003-04. DOIT’S NEW STRATEGIC VISION FOR THE MANAGEMENT OF PROJECTS MAY IMPROVE ITS PROCESSES DOIT believes a key element of its strategic vision to improve the oversight of departments’ IT efforts is its enterprise portfolio management program. This new program encompasses the entire life cycle of the State’s IT development. DOIT states that 30 the new program involves a continuous improvement process to ensure that IT policies, processes, and procedures keep pace with changing technology and indicates that it is working with departments, control agencies, and the private sector to reengineer its existing practices. It starts with developing a statewide inventory of IT projects, which DOIT anticipates using to measure the progress of departments’ efforts against their IT plans. Implementing a review of department IT plans and continuously updating IT policies, procedures, and standards are additional components of the new program. DOIT also intends to improve the planning and approval of IT projects by reengineering the feasibility study report process. The next step in the life cycle is project oversight. DOIT’s new project over- sight vision is intended to support project management best practices and rigorous performance metrics. Each project is closed out through the evaluation report process. Upon comple- tion, projects will become part of the DOIT inventory of IT projects, thus completing the technology life cycle. Finally, the last two steps in the program are maintenance and operations related to completed projects and asset management. DOIT envisions revising each step in the project life cycle to enable better management over IT projects. Although DOIT’s new strategy appears to have merit, it is too early to evaluate because DOIT has only begun to make changes. The proposed inventory of IT projects, which is discussed in Chapter 3, has only recently begun. DOIT has acquired the information to complete the inventory but will need to ensure the information in the inventory is complete and accurate. In addition, as mentioned earlier in this chapter, DOIT has recently Although DOIT’s new begun a pilot of its new process for reviewing feasibility study strategy appears to have reports. This pilot involves the new projects of 10 departments. merit, it is too early to DOIT plans to use this process for all new projects beginning in assess the value it may early 2002. Further, DOIT recently provided us with the perfor- add. mance metrics it intends to use for project oversight, but it is too early to determine the extent to which the metrics will enhance its oversight. Moreover, DOIT will have to establish new internal procedures and statewide policies before it can implement the changes. As of May 2001, DOIT had provided us only draft procedures for how its staff will review departments’ IT plans and evaluation reports. Therefore, until DOIT actually completes its planning and implements a new oversight framework, it is not possible to assess the value it will add. 31 RECOMMENDATIONS To provide strategic guidance for the State’s IT activities, DOIT, in conjunction with the departments, the governor, the Legislature, the Department of Finance, and other relevant parties, needs to update the statewide IT plan to address the current IT environment. In particular, the plan should establish measurable objectives to show how the State intends to reach its goals. Also, the plan should communicate priorities for approval and funding of projects. To facilitate the establishing of such priorities, DOIT should work in collaboration with the entities previously mentioned. To ensure departments’ IT strategic plans are consistently evaluated for their compliance with the statewide IT strategy, DOIT should implement a process to review department plans. To provide appropriate department guidance and direction for the IT development process, DOIT should consolidate the various sources of policy and guidance, remove outdated policies from published documents, and revise policies as needed to reflect changing state needs. In addition, DOIT should resolve the contradiction between its management memorandum and the State Administrative Manual over the applicability of the alternative procurement process. Finally, DOIT should work with General Services to evaluate the alternative procurement process and provide information to departments about how the process could be most effectively used. DOIT should continue its efforts to improve its project review and approval process. However, it should ensure that the changes result in a process that will subject proposed IT projects to a thorough evaluation. Further, DOIT should ensure that departments are properly assessing IT projects by comparing departments’ feasibility study reports with established criteria, such as the fundamental decision criteria. Moreover, to ensure that it can defend its approval of costly IT projects, DOIT should thoroughly document its approval decisions. To ensure departments assess and mitigate project risks, DOIT should require complete risk assessment reports from departments. Further, DOIT should properly analyze the responses and document how it resolves any concerns. Finally, DOIT should require departments using the alternative procurement process to assess risks at the beginning of their 32 projects. If DOIT believes its current model is inappropriate for alternative procurements, it should modify its risk model to more appropriately address alternative procurements. To ensure that it receives and effectively uses the proper infor- mation to monitor departments’ IT projects, DOIT should take the following actions: (cid:127) Continue with its efforts to restructure the oversight process to ensure that the process allows DOIT to properly monitor and guide projects. (cid:127) Modify the required progress reports to include two types of critical information: the project’s monthly actual costs and revised estimates of total projected costs compared with the budget, and actual and revised projected completion dates for project phases compared with the original schedule. DOIT should use this modified progress reporting to closely monitor projects that may be required to submit special project reports. (cid:127) Ensure that analysts sufficiently review and document their oversight of projects and track the receipt of required reports. To hold departments accountable for the benefits expected from their IT projects, DOIT should ensure that departments submit postimplementation evaluation reports. Further, DOIT should continue with its effort to reengineer the evaluation process including the incorporation of lessons learned from project n development. 33 Blank page inserted for reproduction purposes only. 34 CHAPTER 2 Some State Departments Are Not Managing Their Information Technology Efforts According to Best Practices CHAPTER SUMMARY T he last chapter reviewed the Department of Information Technology’s (DOIT) leadership, guidance, and oversight responsibilities for the State’s information technology (IT) projects. Departments, however, are primarily responsible for planning, developing, and ensuring the quality of their own projects. Based on our in-depth review of four major IT projects that were nearing completion, we found that departments are not always using best practices to manage their IT projects. We categorized these best practices into the areas of planning and procurement; design, development, and implementation; and quality assurance. In particular, two projects we reviewed at the Department of Transportation (Caltrans) and the Department of Health Services (Health Services) had significant weaknesses, particularly because the departments did not always plan the projects appropriately. For example, Caltrans did not develop a supportable justification for a complex project begun in 1993, which is still in progress. Two other projects at the Employment Development Department and the Franchise Tax Board were better managed but also experienced some cost and schedule overruns. OUR REVIEW OF BEST PRACTICES FOCUSED ON PROJECTS AT FOUR DEPARTMENTS We reviewed four major projects that were nearing completion to evaluate how well the departments managed these projects. Because they had experienced varying degrees of cost and timeline overages compared with the departments’ original estimates, all these projects had submitted special project reports, which DOIT and the Department of Finance subse- quently approved. 35 (cid:127) Advanced Toll Collection and Accounting System (ATCAS) A Caltrans initiative to replace the existing toll collection and accounting system and install electronic toll collection on all state-owned toll bridges. Electronic toll collection is intended to eliminate the need for participating drivers to stop at tollbooths to pay tolls. Instead, as vehicles with special elec- tronic sensors pass the tollbooth, the sensors automatically charge the drivers’ accounts. The current projected cost is $56.1 million, 102 percent more than the original projected cost of $27.8 million. ATCAS began in June 1993 and is expected to be completed in December 2001. (cid:127) Tax Engineering and Modernization (TEAM) An Employment Development Department effort to redesign the processing of employer tax returns and payments. The projected cost is $71.7 million, which is 6 percent more than its original projected cost. The project began in June 1997 and was completed in April 2001. (cid:127) Children’s Medical Services Network Enhancement 47 (CMS Net E47) A Health Services project to enhance an existing system by linking it with the State’s medical and dental fiscal intermedi- aries. The enhancement will implement electronic billing and create a statewide patient, service authorization, and claims settlement database for two programs that serve children from low-income families and individuals who have certain genetic diseases. The project is currently estimated to cost $10.2 mil- lion and is 82 percent over the original estimate. The project began in January 1998 and is expected to be completed in December 2002; however, elements of the project related to case management and eligibility for Medi-Cal and Healthy Families were implemented in April 2001. Currently, 46 counties use these elements. (cid:127) Accounts Receivable Collection System (ARCS) A Franchise Tax Board project intended to consolidate its various automated and manual collection systems into one system, thus making the Franchise Tax Board’s collection efforts more effective and efficient for all its tax programs. The project cost $36.3 million, 10 percent more than the original estimate. ARCS began in April 1998 and was completed in March 2001. The best practices we used to evaluate the departments’ manage- ment of these projects fell into three categories: planning and procurement; design, development, and implementation; and 36 quality assurance. We reviewed the departments’ management of their projects using certain best practices, including those identified in a 1998 Legislative Analyst’s Office report and in state requirements. We then rated their use of these require- ments and best practices as strong, acceptable, weak, or inconsis- tent. Generally, if the department’s project management exceeded minimum standards in the area, we awarded a strong rating, and if it met the minimum standards for an area, we gave it an acceptable rating. We rated a department weak if project management was deficient in an area that caused significant problems, and we gave a rating of inconsistent to a department that initially did not meet minimum requirements but acted during the course of the project to correct the problem. Our ratings of each department’s use of these requirements and practices are presented in Table 5 in Appendix A. The ATCAS and CMS Net E47 projects have experienced notable difficulties in several of the areas we evaluated and as a result are expected to cost significantly more than the initial projections. In addition, ATCAS is seriously behind its originally scheduled Employing best practices completion date. Both the TEAM and ARCS projects were generally for project management better managed but also had areas with minor weaknesses. will provide a greater However, even though TEAM and ARCS were generally better chance of meeting managed, both experienced cost overruns and delays. The cost original cost and overruns and schedule delays for all four projects are displayed in schedule estimates. Appendix A. While we were not always able to link the cost and schedule overruns to a department’s lack of best practices, employing best practices for project management will provide a greater chance of meeting original cost and schedule estimates. PLANNING AND PROCUREMENT WEAKNESSES CREATED PROBLEMS IN THE PROJECTS WE REVIEWED Proper planning and procurement are critical to successfully developing an IT project. We found several aspects of weak planning in two of the IT projects we reviewed—the Caltrans ATCAS and Health Services CMS Net E47 projects. For example, Caltrans did not develop a supportable justification and a well-defined problem statement for the project. In addition, it did not employ a project management plan to help it identify and resolve problems as the project developed. These planning omissions played a part in creating a cost overrun of 102 percent and a delay of 59 months for the project now scheduled to be completed in December 2001. Health Services’ weak planning and procurement practices also contributed to its 37 project’s potential 82 percent cost overrun and 15-month delay. This project, according to its project manager, will not be completed until December 2002. To determine how well the departments planned and procured the four projects, we evaluated how well the departments complied with 11 best practices. Planning begins with clarifying project goals by defining the pressing need or service that the department hopes to resolve, the project scope, and the project management methodology. These factors are among several that constitute sound planning practices. Proper planning Best Practices for Planning and is essential to set the stage for the project’s design, Procuring an IT Project development, implementation, and final quality. A (cid:127) Define a problem statement and poorly planned project can result in significant cost supportable business case. increases and delays, and in some instances, failure. (cid:127) Outline the business problem and allow the vendor to propose a solution. The procurement of a vendor to develop an IT project (cid:127) Clearly define and document the scope is also an important element of planning. Currently, of the project. sections of both state law and policy require that (cid:127) Clearly define and document the use of departments follow best practices for procurement. project management methodology. For example, the State’s contract laws require that (cid:127) Sponsor the project through appropriate departments award contracts for large IT projects to commitment and involvement of management, and include sufficient vendors that propose the most value-effective solu- resources. tion––an objective basis not limited to cost alone. (cid:127) Base procurement on best value, not Another process, known as an alternative procure- lowest cost. ment, puts more responsibility for project risk on the (cid:127) Use a vendor that provides and vendor rather than the department by allowing maintains sufficient qualified staff. vendors to propose the solution for a department’s (cid:127) Require a letter of credit from the vendor business need. Since 1993 state policy has allowed on a large project. departments to use this process. Beginning in 1998, (cid:127) Clearly define and document the risk management plan. DOIT and the Department of General Services have required that departments use an alternative procure- (cid:127) Write a strong contract to adequately protect the State. ment process, unless a different method is approved, (cid:127) Enforce the terms of the contract. for any IT project deemed to be critical for the department’s responsibilities and with an estimated cost of $5 million or more. The alternative procure- ment process is an approach that considers limiting risk and developing the project in phases, business-based specifica- tions structured to encourage maximum vendor innovation in providing business solutions, and selection of the proposal that is based on cost and other considerations so that it represents a best value for the State. 38 Advanced Toll Collection and Accounting System The main weakness of Caltrans’ planning approach was that it failed to develop a supportable justification—called a business case—and a well-defined problem statement for the project. Both are important because they would have helped Caltrans better understand the expected benefit of ATCAS, assess the magnitude of the problem, and determine if the proposed solution was realistic and achievable. Although one of the purposes of the ATCAS project was to replace the obsolete hardware and software components of its existing toll collection and accounting system, Caltrans also wanted to enhance the system with electronic toll collection, which would involve identifying participating vehicles through an electronic sensor Caltrans did not for the automatic payment of tolls. However, Caltrans’ feasibility sufficiently justify the study report did not clearly differentiate the costs of these two need for developing an distinct objectives. Moreover, Caltrans did not report that it electronic toll collection would take six years before it recouped only half the cost of the system for use on state- electronic toll collection system. Our review of details presented owned bridges. on numerous pages of the feasibility study report indicates that the estimated one-time cost of developing electronic toll collec- tion was $18.3 million, but the expected benefits over the first six years (Caltrans’ calculations did not go beyond that period) were only $8.9 million. These savings were to come from fewer toll collectors, interest earnings on deposits of participating drivers, and time and fuel savings to the motoring public. Given the optional and costly nature of electronic toll collection, it was crucial that Caltrans clearly disclose the cost justification for the system before committing taxpayer dollars. Subsequently, Caltrans revised its feasibility study report to more clearly disclose the costs related to electronic toll collection but still did not demonstrate that it was cost beneficial. In addition, Caltrans did not have a project management plan until two years after development of ATCAS began. A project management plan provides specific steps to administer and track the progress of a project. Used appropriately, such a plan can help managers ensure that they identify and remedy difficulties before they become significant problems. In 1995, approxi- mately two years after the project began, Caltrans developed a project management plan, but the plan had many shortcomings that led us to question how it would promote the delivery of a quality IT project on time and within budget. Specifically, the plan lacked a description of how Caltrans planned to monitor the progress of ATCAS. It also did not clearly identify the tasks required to successfully complete the project. In response to 39 DOIT’s concerns, Caltrans updated the project management plan in October 2000, but it still fell short because it continued to lack the details necessary for Caltrans to effectively manage the project. Specifically, the updated plan still did not address how to monitor progress, did not consider staffing needs and budgets, and did not describe how to approve key elements of the project. In contrast, we found these elements in the current versions of the project management plans of the Franchise Tax Board and Employment Development Department. Another planning weakness is that Caltrans developed the technical specifications to the proposed project rather than letting vendors propose their designs. It is a best practice—and now a state policy in large, critical IT projects—to allow vendors to propose solutions to departments’ business needs, shifting more responsibility for the project’s success to vendors, who are subject-matter experts. Although it may be appropriate for a department to design a small, narrowly defined project, this approach is not always prudent in large, complex projects. It appears that Caltrans’ original technical specifications ulti- mately did not describe the project as it exists now. Given the numerous changes that ATCAS experienced, Caltrans’ decision to define the specifications itself rather than having the vendor do so likely contributed to the cost and schedule overruns on this project. Tax Engineering and Modernization Planning of the TEAM project demonstrated strength in several areas: its defined problem statement and supportable business case; technical solutions proposed by the vendor rather than the department; and the quality of its contract with the vendor that The Employment formed a partnership designed to share the risk, investment, and Development benefits of the proposed project. In particular, the Employment Department’s contract Development Department structured the contract so that the with its vendor was vendor would only be paid from the savings that TEAM would structured to pay the produce by eliminating the need for more than 350 positions. vendor only for To be paid, the vendor had to demonstrate savings from imple- demonstrated savings menting certain phases of TEAM. Thus the contract established from implementing the a strong incentive for the vendor to develop a viable project. TEAM project. This arrangement has been successful in that TEAM has been recently completed, although almost two years late, and the Employment Development Department has saved 330 positions, which is close to the savings originally estimated. 40 However, we noted a significant staffing problem that delayed the project. We found that high turnover among critical vendor staff––the project manager and the quality assurance manager— as well as insufficient and inadequately skilled vendor staff likely contributed to most of the nearly two-year delay in development of TEAM. More specifically, the vendor had four different project managers over TEAM’s four-year development. In addition, the vendor failed to fill the vacant quality assurance position for nearly 11 months, which contributed to the vendor delivering poor quality products. The vendor’s current project manager indicated that when he took over in April 2000, he recognized the need to add staff to the project and to replace existing staff with better skilled staff. In addition, we noted that the Employment Development Department’s oversight consultant monitored and also reported on the vendor’s staffing problems. Another issue was that the Employment Development Department was inconsistent in its development of a clearly defined and For the first two years of documented project management plan. Specifically, the initial the TEAM project, the project management plan included in the feasibility study report Employment Development did not include the methodology used to monitor the progress of Department did not have the TEAM project. In addition, the plan did not include a schedule a clearly defined and that identified all the tasks necessary to complete the project. The documented project plan also did not describe how approvals on key elements of the management plan. project would be handled. With these elements missing, it was more difficult to manage and monitor TEAM’s progress. A second project management plan, developed more than two years later at the recommendation of the independent consultant that the Employment Development Department hired to perform project oversight, was more comprehensive and included the required elements. Children’s Medical Services Network Enhancement 47 After working on this project for more than three years and experi- encing delays and significant problems, Health Services is now rethinking the technical direction of CMS Net E47. In a recent assessment that DOIT required, a technical consultant concluded that the project as currently planned was technically feasible but did not represent a sound architectural plan. The consultant reached this conclusion because the development software and database management system were not widely used or easily supportable, and because neither are supported by Health Services’ current IT infrastructure. The consultant recommended that Health Services use the alternative procurement process to obtain a 41 vendor to complete the remaining elements of the project. At this time project costs are expected to exceed $10 million, almost double the original estimated cost of $5.6 million. Thus our review focuses on the planning and procurement weaknesses in Health Services’ approach that led to this point. We observed that Health Services’ primary weakness in planning and procurement was how it obtained the services of vendors to develop the project. When planning CMS Net E47 in 1995, Health Services determined that it needed a vendor to develop the project. However, rather than following the best practice of outlining its business problem and requesting solutions from The Department of vendors, Health Services developed the specifications itself. As a Health Services’ result, Health Services missed the opportunity to obtain contracting practices innovative approaches from industry experts and to transfer the provided little incentive development risk from the State to the vendor. In addition, for vendors to complete rather than selecting the vendor on the basis of best value—the the work and exposed the best combination of experience, solution, and cost—Health State to unnecessary risk. Services awarded the contract to the vendor that submitted the lowest bid. Further, Health Services did not ensure that the contract included language to protect itself if the vendor did not complete its work. The contract also did not require that a portion of payments to the vendor be retained until the vendor performed satisfactorily. Consequently, the procurement method provided little incentive for the vendor to make sure that the project functioned as intended. Additionally, as a result of the lack of adequate contract terms, the State may find itself in a compromised situation if a conflict arises. We discuss the effect that structuring the contract in this manner had on the department’s quality assurance efforts later in the chapter. Accounts Receivable Collection System Two noteworthy areas in the Franchise Tax Board’s planning were how it addressed the lack of qualified programmers and structured the contract with the vendor who assisted in the development of ARCS. Specifically, the Franchise Tax Board considered the alternative of developing the new project itself, capitalizing on its experience with a similar project. However, it rejected this idea because its analysis showed, among other disadvantages, that the project’s complexities would cause a delay in implementing collection programs and receipt of the projected increased collections. The Franchise Tax Board also determined that the project would cost at least twice as much if it solely used in-house staff for development. The Franchise Tax Board’s solution was to use a combination of in-house and 42 contracted services. To develop ARCS, it integrated vendor staff and state employees into the development team, because it felt that it would not be able to hire fully qualified programmers. By hiring and training state employed junior programmers to work with more experienced vendor staff, a core of experienced and trained state staff would be available to maintain the project after development, and the Franchise Tax Board would avoid having to depend on long-term maintenance contracts with the vendor. This approach might have worked, but during the last six months of the ARCS project, 14 state staff and 4 managers, including the project manager, left the project to work on another large IT project that the Franchise Tax Board was required to undertake––the legislatively mandated child support system. This exodus undermined its strategy, and as a result the Franchise Tax Board must now rely on the vendor to maintain and enhance ARCS. Another positive approach that the Franchise Tax Board took was structuring the vendor’s contract to share increased revenues that ARCS was intended to produce and thus gave the vendor a powerful incentive to develop a successful project. The Franchise Tax Board anticipated the project would increase revenues by approximately $76.5 million in the first four years at a one-time start-up cost of $33 million. Further, it structured the contract to have the vendor pay the start-up cost until the project produced actual revenues. Thus far this arrangement has proven successful, resulting in Best Practices for Designing, increased revenues of more than $150 million, Developing, and Implementing an exceeding the original estimate, according to the IT Project Franchise Tax Board’s independent oversight (cid:127) Design, develop, and test the project consultant. appropriately according to specifications. (cid:127) Establish measurable objectives for the project. DEPARTMENTS’ DESIGN, DEVELOPMENT, (cid:127) Prioritize project elements, such as budget, AND IMPLEMENTATION EFFORTS NEED schedule, and functionality, up front. IMPROVEMENT (cid:127) Develop smaller projects with milestones. (cid:127) Use sound project management practices Once a department plans a project, obtains during implementation. approval, and procures a vendor to assist in (cid:127) Establish appropriate policies and development, its attention focuses on project procedures to manage and control changes to the developing project. management as it implements the project. Project management was troublesome for two of the (cid:127) Develop an adequate information security system to detect and prevent inappropriate projects we reviewed, particularly for the ATCAS access. project and, to a lesser extent, the CMS Net E47 project. For example, Caltrans moved ahead with its ATCAS project despite the fact that the project had 43 not passed critical interim tests. Additionally, Health Services did not consistently use sound project management practices during the development of its project. To successfully implement an IT project, a department should consider seven design, development, and implementation factors. For example, when a department establishes quantifiable goals, it can measure the project’s progress and determine whether the project is meeting its objectives. Employing sound project management practices is also critical to ensuring that the department keeps control of the project during development. Project cost increases and schedule delays, along with poor-quality products, can result when a department manages IT project development poorly. Advanced Toll Collection and Accounting System The ATCAS project has experienced several significant problems in its development. For instance, legislation passed in 1997 required Caltrans to change the basis for assessing toll fees from vehicle classification to the number of axles. This switch required reworking the hardware and software for electronic toll collection. Caltrans also had problems with the performance of its vendor, which culminated in a dispute settlement in December 1999. Other problems resulted from how Caltrans managed the development of ATCAS. For example, Caltrans did not always perform testing of ATCAS Despite not having components as it should have. Testing the key components of a completed acceptance project is critical to ensuring that a project functions as intended tests of the vendor’s and is not disruptive when implemented. Since 1996 Caltrans design for electronic toll and its vendor have been working on a prototype on one bridge collection, Caltrans to test and demonstrate the vendor’s model design for electronic installed the system on toll collection. In July 2000 Caltrans decided to partially deploy other toll bridges. ATCAS on other bridges, which it completed in December 2000. However, Caltrans went ahead with this deployment without completing acceptance tests to ensure that the vendor’s prototype functioned as intended. For example, although all the State’s toll bridges had the ATCAS hardware and software installed on at least one lane, Caltrans has not finished acceptance tests of the accounting portion of the software and the system that is supposed to detect and identify vehicles not participating in the electronic toll collection program on these bridges. Caltrans indicated it performed other validations of the vendor’s prototype to ensure that no major problems would occur and expects to complete acceptance tests in June 2001. 44 Nevertheless, as a result of Caltrans’ decision to implement ATCAS before completing acceptance tests, problems may exist that have not been identified at each of the bridges and may need to be corrected later. Caltrans has also failed to use sound project management Because it did not practices during the development of the ATCAS project. For sufficiently document example, Caltrans repeatedly assigned project managers who critical decisions or had little or no experience managing an IT project of the size or communicate progress, complexity of ATCAS. In addition, none of the managers Caltrans did not ensure assigned to the project were certified as project managers, an that its management or indication of specific training in project management principles. DOIT had the information Further, Caltrans did not demonstrate that it sufficiently needed to effectively monitored the project’s progress. Although Caltrans conducted monitor the project. weekly project team meetings and executive steering committee meetings, Caltrans could provide only limited notes and minutes to demonstrate that it had documented the issues discussed and agreements reached in these meetings. In fact, in October 1997 Caltrans’ technical consultant for the project recommended that meetings be documented, but apparently Caltrans chose to ignore the recommendation. As a result of not documenting critical discussions and decisions, Caltrans’ management did not have sufficient information to properly oversee the project. In addition, had Caltrans used sound project management practices, it would have communicated the progress of ATCAS not only to department management but to DOIT as well, allowing DOIT to effectively monitor the project. In fact, DOIT had previously required Caltrans to submit progress reports after the completion of specific phases, but when Caltrans redefined the phases, it did not report the changes or its progress to DOIT. Caltrans’ less than rigorous application of sound project management practices contributed to cost and schedule overruns for the ATCAS project. Tax Engineering and Modernization This project had difficulties with its change control process and the information security system for TEAM. Before the current project manager came on board in February 1999, the Employment Development Department did not have any formal process to control and monitor project changes. When a change control process is lacking, a department cannot ensure that only the staff that will use the completed IT project can initiate the changes, that the appropriate management staff approves 45 changes, and that changes are reviewed before they are implemented. The current process allows the project team to appropriately track and monitor changes. We also observed certain weaknesses in IT security over the TEAM project. The details of these weaknesses have been shared with management of the Employment Development Depart- ment. The Employment Development Department intends to implement appropriate security procedures by June 2002. The first project manager Children’s Medical Services Network Enhancement 47 that the Department of Health Services assigned Although we rated Health Services’ design, development, and lacked the appropriate implementation acceptable overall, we had several concerns. skills or background Health Services did not consistently use sound project necessary for the project. management practices during the development of the project. Specifically, the project manager assigned to the CMS Net E47 project from March 1998 through November 1999 lacked the appropriate skills or background necessary for a large, complicated project. In addition, the job classification of this project manager––a staff information systems analyst––was not at a level that would provide the authority needed to effectively manage and direct staff of various job classifications. During the period the first project manager was assigned, certain basic project management tasks were not performed consistently. These tasks included updating the risk management plan, tracking problems to resolution, developing procedures to control and monitor changes, and documenting project meetings. Health Services hired an independent verification and validation (IV&V) consultant in November 1998 to ensure that appropriate project management practices were followed. Health Services believes that after November 1998 the IV&V consultant helped it address these issues. Health Services finally hired a state employee with the appropriate qualifications and job classification to manage the project in December 2000. In the interim, from December 1999 through November 2000, Health Services hired a consulting firm to provide a project manager. We also observed certain weaknesses in the IT security over the CMS Net E47 project. The details of these weaknesses have been shared with management of Health Services. Health Services is studying how to implement appropriate security procedures. 46 Accounts Receivable Collection System Despite problems it encountered due to changes in requirements and schedule slippage, the Franchise Tax Board used sound project management practices during its implementation of the ARCS project. For example, after planning the project by getting high-level executive sponsorship and user involvement, the Franchise Tax Board and the vendor committed enough staff with appropriate skill levels to ensure the success of the project. In particular, the Franchise Tax Board and the vendor both employed project managers with sufficient experience and skills. They also monitored the progress of system design, develop- ment, and testing. Specifically, for each phase of the project, the ARCS team created a general scope and approach document and then formulated a detailed work plan that became the basis for the management of the project. The project also had a change control process that allowed the project managers to monitor, control, and track changes to the project or the project plan. Moreover, the consultant the Franchise Tax Board hired to provide oversight on the ARCS project stated in its final report of April 2001 that ARCS management staff did an excellent job of managing a complex and difficult project. DEPARTMENTS GENERALLY FARED WELL IN THEIR QUALITY ASSURANCE EFFORTS Quality assurance procedures help departments identify and assess the significance of problems that occur as projects are implemented. Two of the projects experienced problems with their quality assurance. Caltrans failed to use an independent consultant to monitor the develop- Best Practices for Ensuring the ment of its ATCAS project. Also, Health Services Quality of an IT Project hired a consulting firm that provided both project (cid:127) Create an effective quality assurance management and independent oversight services mechanism and use it during each phase for the CMS Net E47 project. This resulted in an of development. incompatible situation because the consulting (cid:127) Heed the advice of the oversight consultant or explain why it is not applicable. firm was supposed to provide objective oversight of the project. (cid:127) Pay the vendor only after accepting tested project deliverables. We reviewed how each of the four departments’ quality assurance procedures complied with three best practices. We noted that consultants can perform various quality assurance functions to evaluate the progress of an IT project under development. For example, project oversight consultants help departments identify and assess the 47 significance of problems that occur as projects are implemented. An IV&V consultant takes this a step further by following a prescribed process to assess the primary vendor’s performance and making recommendations to the department on how to obtain a better quality product. Quality assurance helps minimize risk by identifying potential problems earlier rather than later in the project’s development and implementation phases. Advanced Toll Collection and Accounting System Despite the fact that ATCAS was a complex and costly project, Caltrans failed to employ an IV&V consultant for almost the entire project, and when it did hire one in 1997, Caltrans changed the consultant’s duties. Rather than assessing, among other things, the quality of the technical work and overseeing the project, the IV&V consultant provided technical assistance. In January 2001 Caltrans finally contracted with another IV&V consultant to provide project oversight and quality assurance consulting services to support the ATCAS effort. However, Caltrans states it hired the contractor only after DOIT required it to do so. Using an IV&V consultant earlier in the project might have ensured that the vendor’s model design for electronic toll collection passed Caltrans’ testing and might have avoided some The Employment of the cost overruns and delays that the project experienced. Development Department’s decision to Tax Engineering and Modernization hire an independent verification and The Employment Development Department’s quality assurance validation consultant efforts on the TEAM project were effective because it hired an helped ensure the IV&V consultant to help it ensure that the primary vendor met vendor’s products met the quality standards of the contract. The IV&V consultant hired quality standards. by the Employment Development Department used industry standard methodology to review all phases of the project, assisted the department in developing a formal project plan and requirements to achieve the business objectives, and tracked the project requirements through all phases of the development process. In addition, the consultant frequently reported findings and made recommendations to the appropriate staff levels of the Employment Development Department. Children’s Medical Services Network Enhancement 47 We noted a concern over how Health Services used two individuals from the same consulting firm to help it manage the CMS Net E47 project and to oversee project management. In November 1998 Health Services hired a consulting firm to 48 provide an individual as the IV&V consultant to oversee the project. In November 1999 Health Services asked the same firm to provide a project manager to assist it in the development of its CMS Net E47 project. However, because Health Services had hired both individuals—the project manager and the IV&V consultant—from the same firm, it may have made it difficult for the IV&V consultant to objectively oversee the performance of the project manager. It was not until August 2000, nine months after the potential conflict of interest occurred, that this The Department of issue was resolved when the IV&V consultant resigned from the Health Services paid a project. Even though it appears that the IV&V consultant’s vendor for hours worked, recommendations were valuable and much needed, a potential not for deliverables conflict of interest existed. As of April 2001, Health Services still produced, and thus had not hired a new IV&V consultant. accepted the majority of risk for the vendor’s work. We also found that Health Services did not ensure the quality of its project because it used a procurement and payment method that did not sufficiently protect the State. To safeguard state investments, a department should pay a vendor only after accepting the deliverable and verifying that it meets the terms of the contract. However, Health Services paid the vendor 100 percent of billed hours the vendor invoiced each month. Additionally, Health Services did not withhold a reserve to be paid on completion of acceptance testing or on approval of final implementation. Specifically, beginning in June 2000, Health Services used a vendor from the California Multiple Award Schedule (CMAS) program—a procurement process in which departments neither solicit nor evaluate bids from vendors. CMAS offers products or services and prices from an existing list of vendors that have been competitively assessed. Health Services’ contract required that the vendor develop and provide application modules—computer programs—to Health Services for the project team to review and test. Our review found that Health Services paid the vendor based on the amount billed, even though the modules were not completed. By not structuring its agreement with the vendor to pay only after testing and accepting the deliverable, Health Services accepted the majority of risk for the vendor’s work—a risk that, according to a document prepared by DOIT, was increased by Health Services’ piecing together the project through CMAS contracts. Accounts Receivable Collection System The Franchise Tax Board generally complied with best practices in the quality assurance area. Although the project is completed and generally functioning as intended, employing an IV&V 49 consultant could have minimized potential problems. When development began in 1998, DOIT required the Franchise Tax Board to hire a project oversight consultant. The Franchise Tax Board chose to hire an oversight consultant rather than contract with an IV&V consultant. The oversight consultant focused on reviewing the project’s finances, personnel, schedule, and docu- mentation, not the requirements, design, testing, and imple- mentation details that an IV&V consultant is required to review. As a result, the Franchise Tax Board did not have the benefit of information that would have enabled it to make better informed decisions had problems developed with the quality of the vendor’s work. RECOMMENDATIONS To improve the management of IT projects and to help ensure that projects are completed on time and within budget, Caltrans should take the following actions: (cid:127) Develop a problem statement for each IT project that adequately describes the problem the project is intended to solve with quantifiable goals, and establish a supportable business case for each project that justifies its funding. (cid:127) Develop an effective project management plan before begin- ning to develop each project so it can monitor the progress of the project. (cid:127) Allow vendors to propose solutions and the technical specifications for its large and complex IT projects. (cid:127) Ensure that testing is completed at appropriate phases to identify and resolve problems before moving ahead. (cid:127) Ensure that it uses sound management practices during the development of each project, such as assigning qualified individuals with appropriate experience and training to manage the project, documenting key discussions and decisions, and monitoring progress through periodic reports. (cid:127) Use IV&V consultants on complex IT projects. 50 To improve the management of IT projects and to help ensure that projects are completed on time and within budget, the Employment Development Department should take the follow- ing actions: (cid:127) Ensure that the vendor provides sufficient staff with the necessary training and experience. (cid:127) Use an effective project management plan before beginning to develop each project so it can monitor the progress of the projects. (cid:127) Ensure that it establishes and uses a process to control and monitor project scope changes that requires changes to be adequately reviewed before they are made. (cid:127) Correct the IT security weaknesses we identified. Health Services should take the following actions to improve the management of IT projects and to help ensure that projects are completed on time and within budget: (cid:127) Select vendors that propose the best solutions at the best value. (cid:127) Structure contracts with vendors to protect the interests of the State, including provisions to pay vendors only after deliverables have been tested and accepted. (cid:127) Use sound project management practices during the design, development, and implementation phases of projects and specifically ensure that it assigns project managers with the appropriate skills and authority. (cid:127) Correct the IT security weaknesses we identified. (cid:127) Ensure independent oversight of its projects by hiring IV&V consultants from firms that are different from those providing other services to the project. To improve the management of IT projects and to help ensure that projects are completed on time and within budget, the Franchise Tax Board should use not only project oversight consultants but also IV&V consultants throughout the development of its complex projects to make better informed n decisions should problems occur. 51 Blank page inserted for reproduction purposes only. 52 CHAPTER 3 DOIT Has Not Fully Satisfied Its Other Responsibilities CHAPTER SUMMARY I n addition to the Department of Information Technology’s (DOIT) problems addressing its project planning and oversight responsibilities as described in Chapter 1, DOIT has not fully met its other responsibilities. The Legislature envisioned that DOIT would improve the State’s ability to apply information technology (IT) effectively; however, DOIT’s inability to prioritize its efforts contributes to it only partially carrying out its responsibilities. For example, DOIT has not established a process to consistently address the coordination of IT project development. Also, DOIT still needs to compile an accurate inventory of IT projects, which could be a valuable resource in coordinating departments’ activities and enhancing its oversight. In addition, DOIT has only recently prepared drafts of several key standards for IT development activities that establish common rules and could encourage the use of best practices, and has not yet begun to develop other standards. Further, DOIT has inconsistently used the two statutorily required IT advisory councils to discuss key policies and strategies. DOIT’s failure to fulfill its responsibilities and complete several initiatives designed to better manage the State’s IT efforts lessens its credibility with outside parties. Finally, DOIT has not developed an internal strategic plan that would help it prioritize its operations and organize its activities that are currently scattered over a variety of initiatives. DOIT HAS NOT TAKEN SUFFICIENT ACTION TO COORDINATE INFORMATION TECHNOLOGY PROJECTS Despite the mandate of state law, DOIT does not have an established process to ensure that departments do not independently develop statewide IT applications or duplicate other departments’ efforts. Instead, departments have mostly relied on informal networking to identify similar projects at other departments. In addition, DOIT has not continuously maintained an IT project inventory as required by state law. The project inventory, if properly designed and updated, would help 53 coordinate activities and enhance the State’s ability to make a conscious, proactive evaluation of how it allocates its limited resources for IT projects. Without consistent coordination, the State lacks assurance that it can identify overlapping or redundant IT efforts. Additionally, without coordination, departments do not benefit from each others’ knowledge of technology and development approaches for projects addressing similar business functions. DOIT Lacks an Established Process to Consistently Identify Coordination Opportunities DOIT lacks an established process to consistently address the coordination of IT project development. Although DOIT says it uses its project approval and oversight activities to identify departments with similar projects, these activities do not include a consistent method for identifying common elements across several projects. As a result, DOIT may not always identify opportunities where departments could discuss similar projects. As part of its revised feasibility study report process, DOIT plans to improve its ability to identify projects with similar characteristics by Benefits of Project Coordination creating a new database that will capture (cid:127) The State can avoid the duplication of information from more than 200 questions that it processes and functions in its IT systems. will ask departments about their proposed (cid:127) DOIT can identify opportunities to leverage projects. DOIT stated that it will be able to IT development efforts across departments to reduce costs and make departments’ IT identify similar projects once it implements its systems compatible. new database. However, DOIT has not finalized (cid:127) DOIT can partner a department that is and implemented this process yet. weak in a specific IT technology with a department that is strong in that area. DOIT’s past efforts in this area have been guided (cid:127) Departments can learn from other departments’ experiences. by a 1997 committee report on coordinating work on administrative systems—those relating to (cid:127) Departments can see other types of procurement that may suit their current departments’ accounting, human resources, needs. procurement, and inventory functions. Among (cid:127) Departments can share experiences the committee’s recommendations to DOIT was about how well vendors performed on past designing and implementing pilot projects to IT projects. determine the feasibility of embarking on statewide projects. In its 1998 annual report, DOIT indicated that it had authorized pilot projects of statewide systems at four departments. However, DOIT could not demonstrate that it evaluated these pilots to determine if statewide projects are warranted. In addition, other departments have pursued projects that will provide service across many departments, such as the payroll 54 and travel expense claims projects at the State Controller’s Office and the procurement automation project at the Department of General Services. Further, DOIT indicates that it has participated in statewide efforts to coordinate IT activities, including the statewide geographic information systems effort and the governor’s electronic government (eGovernment) initiative, which is intended to encourage departments to offer services to citizens over the Internet through the State’s Web site. Despite these efforts, DOIT lacks a consistent and proactive approach to ensure that coordination occurs to the extent needed. Based on its project review during the fall of 1999, DOIT claims that there is very little commonality among IT activities. How- ever, our review revealed that a number of departments are developing projects in similar business areas. For example, various departments use IT systems to manage client cases. Specifically, during our review of major IT projects—those with one-time development costs greater than $1 million—we found 13 departments that were developing or had developed 25 major case management projects. Assorted departments also license individuals and businesses for a variety of purposes. We found 9 departments that were developing or had developed 16 major licensing projects. With so many multimillion-dollar projects addressing similar business functions at different departments, DOIT needs to help departments share with each other their knowledge of technology, development approaches, and vendors. Because DOIT has not effectively coordinated IT efforts, many chief information officers (CIOs) we spoke with at various departments indicated that they rely on informal networking to identify possible coordination opportunities. Some departments have established relationships with others that perform similar functions to identify opportunities to share information and In the absence of DOIT’s cooperate on IT projects. For example, staff from the Board of leadership, many Equalization, the Franchise Tax Board, and the Employment departments we spoke Development Department have met to identify coordination with indicated that they opportunities. As a result of these meetings, the departments rely on informal participated in several coordinated IT activities and have networking to identify discussed how they could establish common methods for coordination identifying businesses and accepting credit card payment of opportunities. taxes. Although these actions by departments’ CIOs and staff provide some coordination of IT activities, they do not replace the more formal mechanism under DOIT’s leadership that is needed. 55 DOIT’s Failure to Consistently Maintain a Project Inventory Inhibits Coordination and Oversight Even though state law instructed DOIT to create an inventory of IT projects with approved feasibility study reports, DOIT has not continuously maintained an up-to-date inventory that contains sufficient information to be useful in project coordination and oversight. To be beneficial, the inventory would need to contain a variety of data, including project descriptions, costs, timelines, technologies used, and vendors employed. A complete, central- ized record of IT projects would be a powerful tool in the State’s efforts to coordinate the activities of various departments and in the efforts of the Legislature and control agencies to make conscious, proactive evaluations of how the State spends its limited IT resources. In addition, the inventory would improve DOIT’s ability to monitor whether departments develop their projects on time and within budget and allow department CIOs to identify other departments developing similar projects. In its 1996 and 1997 annual reports to the Legislature, DOIT indicated that it was using a database that met the statutory requirement for a project inventory. One portion of the database reportedly allowed Internet access to an index of proposed IT projects. The other part of the database apparently captured ongoing information from independent oversight reports and other reports about projects that were in development. However, DOIT stopped using the database in 1998 because it believed the database was not recording project data accurately. DOIT reported that it collected information on 1,200 computer systems during the year 2000 effort to help identify departments’ progress in resolving computer issues. However, DOIT indicated that the database only contained data on projects impacted by the year 2000 problem that departments had identified as mission critical. Therefore, DOIT has recently In November 2000 DOIT taken steps to create a more complete project inventory. In surveyed departments to November 2000 DOIT requested that 111 departments provide obtain the information it information on up to 74 items for each project, including believed was needed to development costs, start dates, vendors, and technologies used. develop a useful project DOIT believed the breadth of the data available in the year 2000 inventory. database made it the appropriate starting point for a survey of the departments’ IT activities. However, when DOIT created its survey, it did not include additional detailed information that it had previously received and was available in its project approval and reporting documents. DOIT requested that departments respond to the survey by January 2001. By April 2001 all departments had responded. 56 Because we were asked to create a statewide Problems and Inconsistencies inventory of major IT projects, we compiled With Information Departments information on 15 key fields from departments’ Submitted to DOIT responses to DOIT’s survey. However, we found (cid:127) Costs or project status initially not included that departments initially failed to include one- for 55 percent of projects. time development costs or project status for (cid:127) Completed project costs based on rough 55 percent of the projects reported to DOIT. To estimates or not provided, because address these omissions, we sent portions of the previously departments were only required to report IT costs on an annual basis without surveyed information back to departments to tying costs to specific projects. encourage them to identify cost and status for all (cid:127) Cost savings apparently reported as projects and to provide additional information negative project costs instead of zero. on the 15 key fields for major projects. (cid:127) Differing standards used by departments to Through these follow-up contacts, we found determine which IT activities should be reported, because they believed DOIT’s that departments’ responses contained many definition of an IT project was unclear. For inconsistencies. We also asked five departments example, some departments listed minor to provide documentation for selected portions activities, like spreadsheets and databases, as projects, and others only listed projects of their responses to DOIT’s survey for 23 major they were still developing. projects. We found that 21 projects contained (cid:127) Systems that were the responsibility of one at least one discrepancy each, where the department reported by another. For reported information did not match underlying example, CALSTARS, which is the State’s financial system, was included in several documentation. We found that 6 of these departments’ responses. 21 projects had discrepancies on more than (cid:127) Many fields not completed because half of the items we reviewed. departments may have been burdened by DOIT’s request for information on 74 items for projects. We summarize the key items from the survey responses of the 111 departments in Appendix B. However, we found that many departments omitted data from their survey responses, misinterpreted the instructions, or included errors. We gave departments an opportunity to review the information they submitted, and we made any changes they indicated. Nevertheless, the information we present may still contain inaccuracies. Because of these inaccuracies, we are skeptical of the information departments provided to DOIT. DOIT needs to expend considerable effort to obtain consistent, accurate project information. In addition, DOIT needs to create a process to update project information so that data from special project reports and new feasibility study reports are added to the existing inventory. DOIT says it plans to have its project analysts review departments’ survey responses and then work with departments to capture missing data. DOIT also plans to allow departments to correct and update their project information on-line. DOIT indicates its on-line process is currently in the pilot phase and will not be available to all departments until it evaluates the pilot process. 57 In its March 2001 Capability Assessment Report to the Legislature, DOIT indicated that it recently used internally It is unclear at this time maintained project records to create an inventory of 153 IT when DOIT will have a projects under development that require its oversight. This complete and accurate report made public the names of the projects and DOIT’s project inventory planned level of involvement but did not include key attributes available for departments of the projects—including project description, technologies to coordinate their used, complexity, and cost—that are necessary for coordination efforts. or oversight. Further, although DOIT has provided the names of 153 projects with approved feasibility study reports, this list represents only a small portion of the more than 2,500 completed and in-development IT projects that departments reported in DOIT’s survey. Although some of these projects may not have required DOIT’s approval of their feasibility study reports, they represent a significant part of the State’s IT environment and should be considered when making oversight and coordination decisions. Therefore, it is unclear at this time when DOIT will have a complete, accurate, and up-to-date project inventory available for departments to coordinate their efforts, and for the Legislature and control agencies, including DOIT, to improve their project oversight. DOIT HAS NOT FINALIZED SEVERAL KEY STANDARDS AND PLANS TO DEVELOP OTHERS State law directs DOIT to develop standards to guide departments’ IT efforts. DOIT has recently developed several key standards. Four of five we identified are still in draft form and the remaining standard is yet to be created. Standards establish common rules and can encourage the use of best practices for collecting, sharing, protecting, and storing data, as well as ensuring the accessibility and usability of systems. By minimizing the need to modify IT projects that must work together and reducing the risk of unauthorized access, among other benefits, standards can decrease operating costs. Despite the value of these standards and the fact that from its beginning DOIT has been required to oversee the development of IT standards, DOIT has not finalized standards in several critical areas. Departments’ records show that since DOIT’s inception in 1996, the State has completed or begun development of about 200 major IT projects—those with development costs greater than $1 million. By not finalizing the standards, DOIT may have missed opportunities to ensure that departments develop IT systems in a consistent and coordinated fashion. 58 DOIT states that the standards it has developed to date have mostly been directed toward the eGovernment initiative. As part of the eGovernment initiative, senior IT staff from various departments, with DOIT’s assistance, were to develop five key standards—security, infrastructure, application development, accessibility, and data. However, as Table 2 shows, one standard has not been developed and the others are still in draft form. TABLE 2 Key Standards Not Completed Status of Standard Standard Benefit According to DOIT Security Protect state IT projects and information Final draft created from damage or unauthorized disclosure or use. Ensure the privacy of information. Infrastructure Ensure that state IT projects meet the Final draft created availability requirements of their users. Establish a statewide level of service. Facilitate the efficient operation and maintenance of projects. Application Promote the usability of projects, including Preliminary draft Development the use of skilled personnel and proper created infrastructure. Accessibility Ensure that individuals with disabilities can Preliminary draft access systems. created Data Facilitate the sharing of data among Not drafted projects without translating, editing, or reformatting. Although DOIT currently indicates that security and infrastructure standards are final drafts, it does not expect these standards to be through the review and approval process until October 2001. DOIT does not plan to finalize these standards until then because it believes that standards development requires an extensive process to obtain the input and buy-in of the affected entities. Because the application development and accessibility standards are in preliminary draft form and the data standard is not yet drafted, it is unclear when DOIT will issue these standards. DOIT plans to issue other standards in the future, including standards for software licensing and asset management, e-mail, office automation, and document exchange. Until standards are finalized, departments will continue to conceive and develop IT projects without the framework needed to ensure that their efforts meet common rules and are consistent with best practices. 59 DOIT HAS INCONSISTENTLY USED ITS ADVISORY COUNCILS DOIT has not consistently used the two IT advisory councils required by state law to address strategic IT policies and management issues. One required council—the private commission—should consist of IT practitioners from private, academic, nonprofit, and governmental sectors and is intended to provide advice on long-term trends and strategies, key policies, emerging technologies, and best practices. The second required council—the public committee—should consist of representatives from state agencies and is intended to advise DOIT on successful IT management, identify critical success factors, and recommend policy changes. DOIT’s annual reports from 1996 to 1998 indicated that both councils met on a regular basis and that subcommittees of the councils were studying various issues and producing reports. During 1999 a form of both councils met to help the State with the year 2000 effort. It is unclear if DOIT regularly met with the private commission in 2000, but DOIT has more recently begun meeting with it regularly to discuss pressing issues. Although DOIT’s last official meeting with the version of the private commission that had addressed the year 2000 effort was in January 2000, DOIT claims that it continued to meet with certain private industry and local After meeting with the government representatives during 2000. However, it is unclear private advisory whether unofficial meetings with these representatives commission in constitute the statutorily required private commission. In January 2000, DOIT did addition, DOIT was unable to provide evidence that it ensured it not officially meet with was receiving the advice envisioned in state law, including long- the commission again term IT trends and strategies. During this time, DOIT was until January 2001. attempting to change its feasibility study report process and reorganize its operations and would have benefited from the commission’s insights. DOIT did not officially meet again with the private commission until January 2001, which was when DOIT first met with the current version of the commission known as the California Information Technology Commission. DOIT subsequently scheduled monthly meetings for the rest of the year. The commission’s first few meetings this year included discussions on a draft of software piracy policy, recruitment and retention of skilled IT staff, and best practices to meet energy challenges. DOIT uses a council composed of department CIOs to meet the intent of the law for a public committee. In February 2000 DOIT established a CIO Advisory Council comprised of the CIOs that chair 13 DOIT-sponsored work groups. DOIT formed the work 60 groups to develop policies and address a variety of topics, including strategic planning, acquisition, and approval of feasibility study reports. Although DOIT indicates that the CIO Advisory Council was put on hold until October 2000 so its DOIT could not provide members could focus on the eGovernment initiative, DOIT us any written findings or appears to have met regularly with the council since then. recommendations from However, because DOIT did not sufficiently document its the CIO Advisory meetings with the council, we could not determine if DOIT Council—the group ensured that the council provided it with the advice intended by created to serve as a law. In addition, DOIT could not show us any written findings public committee. or recommendations made by the council, even though state law indicates they must be made available to interested parties. Consequently, interested parties, including the public and the Legislature, are unable to readily evaluate the direction that the council provides DOIT. Further, many of the 13 CIO work groups are too new to determine if they will reach the goals established for them. Although DOIT created the work groups early in 2000, they were initially used to assist in the State’s eGovernment efforts to produce draft standards and other related duties. It was not until later in 2000 that DOIT was able to direct these work groups to other purposes, such as recruiting and retaining skilled IT staff and redesigning the feasibility study report. According to DOIT, only 5 work groups had met before February 2001 but all 13 had met as of May 2001. In addition to the CIO Advisory Council and work groups, DOIT has created other opportunities to receive feedback from CIOs and communicate its message to them. DOIT typically sponsors biweekly meetings with department CIOs to present IT information, such as changes in the feasibility study report approval process, the implementation of the State’s new Web site, and more recently, the State’s efforts to conserve energy. DOIT has also co-sponsored other forums that have included presentations on best practices, how government can provide services over the Internet, privacy and computer security, and using geographic information systems. DOIT acknowledges that pressing concerns with the year 2000 effort and the eGovernment initiative dominated the use of the councils over the past two years. However, DOIT has now directed the councils to provide advice on how DOIT should carry out its statutory responsibilities. It appears that DOIT has started to receive the input from private- and public-sector leaders on its other IT efforts, although it is too early to tell whether the councils’ activities provide the advice envisioned by 61 state law. The councils’ input may help DOIT address the weak- nesses in strategic planning, project review and approval, and project oversight discussed in Chapter 1 of this report. DOIT’S LACK OF PROGRESS ON ITS RESPONSIBILITIES COULD LESSEN ITS CREDIBILITY Although DOIT has made progress on some past initiatives, DOIT’s lack of progress on other initiatives and legislative requests could lessen its credibility with departments and the Legislature. Since its inception DOIT has pledged action on key initiatives or planned tasks in its annual reports to the Legislature. However, as shown in Table 3, DOIT has not fulfilled all its promises. For example, DOIT indicated in its 1998 annual report that it would enable departments to update the statewide project inventory over the Internet, but this capability still does not exist. DOIT states that the initiatives presented in Table 3 were established by the previous administration. DOIT believes that the current administration cannot be held accountable for the promises and initiatives of that administration. Nevertheless, DOIT’s lack of progress may lessen its credibility. DOIT has made progress on certain initiatives. For example, DOIT is required to promote reforms in IT classifications so that the State can better reward skilled workers and facilitate the State’s ability to succeed in the changing IT environment. Over the past year DOIT has worked with other departments and a CIO work group to change the minimum qualifications for some IT positions to allow for a broader pool of qualified applicants. In addition, DOIT is redesigning its process for approving feasibility study reports and has recently begun a pilot of the new process. The Legislature has had to make special requests Nevertheless, as discussed throughout this report, DOIT has of DOIT to develop addressed some but not all responsibilities the Legislature has policies in many areas given it. Some of the responsibilities it has not sufficiently such as project addressed include properly overseeing departments’ IT efforts, management, creating standards, and developing and maintaining a project procurement, project inventory. In addition, the Legislature has had to make sizing, and feasibility special requests of DOIT to issue policies on project study reports. management training, procurement, project sizing, project delegations, and streamlining the feasibility study report process, among other topics. Because it has not complied with legislative requests and requirements, DOIT could lose credibility 62 TABLE 3 Selected DOIT Initiatives That Have Not Been Completed Initiative Status Develop statewide Although DOIT indicated in its 1996 annual report that it IT systems would identify departments that would participate in potential statewide pilots, DOIT could not demonstrate that it evaluated these pilots to determine if statewide projects are warranted. Create security Although DOIT indicated in its 1997 annual report that it standards would publish general standards for the IT security requirements for new projects, security standards are still in draft form. Improve the State’s Although DOIT indicated in its 1997 annual report that it ability to efficiently would improve the ability to readily locate the e-mail use e-mail addresses of state employees, DOIT states that the Department of General Services has taken responsibility for this initiative and has implemented an on-line directory. However, DOIT indicates it is still a work in progress. Create an on-line project Although DOIT indicated in its 1998 annual report that it tracking database planned to expand its project tracking system so that departments could submit their projects over the Internet, DOIT has not developed that capability but plans to use it as part of its new process for overseeing departments’ development efforts. Improve project Although DOIT indicated in its 1998 annual report that it risk assessment would incorporate the results of a department survey into its risk assessment model, DOIT states that due to staff turnover, it is unable to locate the survey that would guide those changes and therefore cannot show how it incorporated the results. with departments and the Legislature. For instance, some CIOs with whom we met expressed concerns because DOIT has not fulfilled its responsibility of issuing standards to help departments improve communication and coordination between IT systems, and some CIOs questioned the value of DOIT’s project oversight because it did not provide feedback on project reports. DOIT NEEDS TO IMPROVE ITS INTERNAL PLANNING PROCESSES Although good management practices suggest that DOIT develop and implement an internal strategic plan to guide its efforts and maximize the efficient use of its resources, it has not consistently used one. DOIT’s authorizing legislation requires it to be involved in a variety of activities, and meeting these responsibilities stretches its resources. In addition to its oversight 63 of IT projects, DOIT is required to issue policies and standards, organize advisory councils, and coordinate the State’s IT activities. Yet, DOIT has operated without the direction of an internal strategic plan that sufficiently prioritizes its activities, even though it has faced difficult challenges like the year 2000 effort and staffing problems. In September 1999 DOIT initiated a planning and realignment effort that produced a high-level operational plan. However, this plan was simply a list of key activities DOIT would undertake from July 2000 through March 2001. Although the list identified numerous short-, mid-, and long-term activities, it did not differentiate between activities that must be completed and those that were less critical. In addition, this list lacked measures for DOIT to determine if its efforts were effective. Without the direction of an internal strategic plan to define and prioritize what it needs to do, DOIT’s efforts have been scattered over a variety of initiatives, and it has performed inconsistently. A prime example of DOIT’s need to analyze workload and prioritize efforts is the challenge presented during the year 2000 effort. Although two executive orders from the governor established the year 2000 effort as DOIT’s top priority, they did not relieve DOIT of its responsibility to oversee departments’ IT efforts or its other statutory duties. In March 1999, in response to the executive orders, DOIT reduced the staff it assigned to core responsibilities, such as project approval and oversight, from 15 to 7 and limited its oversight of ongoing projects to large and high- risk projects. Faced with insufficient resources to focus on the year 2000 effort and still adequately cover its other responsibilities, DOIT should have requested additional staff for its other responsibilities. However, according to DOIT, it did not request additional staff because its main focus at the time was completing its year 2000 efforts. DOIT faces another significant challenge in prioritizing its Because a federal reform efforts in a federally mandated health insurance reform that that affects how will require changes to the IT systems of various state departments’ IT projects departments. Over the next couple of years, this reform—the handle health insurance 1996 Health Insurance Portability and Accountability Act—will information may require change how these departments collect, process, and report a significant commitment individuals’ health information. The governor’s budget for fiscal of its resources, DOIT will year 2001-02 requests $92 million to assist departments in the need to carefully statewide planning and implementation of the reform. Because prioritize its efforts. this reform requires significant changes to certain business processes and their related IT systems, industry experts are warning that it could be more disruptive than the year 2000 64 effort for the departments that are affected. Although the State has not yet identified all of them, the number of affected departments could be significant, possibly including departments that indirectly handle health care data, such as the State Teachers’ Retirement System. DOIT has not been asked to take the lead role; however, this effort may require considerable commitment of its resources. Unless DOIT carefully plans and prioritizes its efforts to oversee the changes, DOIT may again be ineffective in accomplishing its other mandated responsibilities while focusing on this problem. DOIT’s ability to identify its strategic priorities has been hurt by the turnover of key manage- Turnover of Key Managers During ment staff. We observed that since July 2000, 8 of Fiscal Year 2000-01 DOIT’s 11 key managers have left, taking with (cid:127) Chief deputy them their experience, knowledge, and insight. (cid:127) Deputy director of project oversight and These departures have significantly changed initiation (vacant)* DOIT’s leadership and depleted the breadth of its (cid:127) Chief procurement officer managers’ experience with strategic issues and (cid:127) General counsel (vacant)* their relationships with departments. DOIT has (cid:127) Budget officer hired individuals to fill half of these positions, (cid:127) Personnel officer but several management positions have been vacant for as long as six months. Since the end of (cid:127) Legislative affairs coordinator (vacant)* the year 2000 effort, DOIT has been working to (cid:127) Public information officer (vacant)* reorganize its efforts. The turnover in leadership has been problematic, particularly at a time when * All vacancies as of May 2001. DOIT operates without a strategic plan to guide new managers on how to deal with DOIT’s many unmet responsibilities. With high management turnover and no strategic plan, we are concerned that current DOIT management seems to have not taken sufficient advantage of the processes established by prior management. In a recent report to the Legislature, DOIT indicated that policies and practices in place before the year 2000 effort offered, at best, a starting point for developing current processes and claimed that relatively few standard processes and procedures had been developed. Yet before the year 2000 effort, DOIT made several contributions to the State’s IT efforts, including issuing a statewide IT plan in 1997 that contained several initiatives that could continue to guide the State’s IT efforts. For example, DOIT could update the initiative related to developing IT standards to describe DOIT’s current efforts in this area. Further, DOIT had previously published a detailed policy guide, policy directives, and several studies to provide guidance and improve the State’s IT management. Thus 65 DOIT’s current leadership appears to have added to its workload by insisting on revamping the processes and policies developed by the previous management. It will be difficult for DOIT to make measurable progress if it is continuously rethinking policies and practices rather than building on existing practices when planning future activities. DOIT acknowledges that it needs a strategic plan to guide its operations and meet its responsibilities. In October 2000 DOIT hired a consultant to analyze its internal operations. In January 2001 the consultant submitted a report to DOIT that, In January 2001 its among other things, recommended that DOIT rigorously consultant recommended prioritize its efforts so that it achieves a consistent level of high that DOIT rigorously performance in its key processes while focusing new efforts on prioritize its efforts and addressing identified high-risk areas. DOIT established the focus new efforts on position of director of strategic planning to research internal identified high-risk areas. and external factors related to its plan, review advice and recommendations from the private commission, and develop and document its strategic plan. DOIT expects to complete an internal strategic plan in November 2001. However, since this plan has not been completed, it is unclear whether DOIT’s strategic plan will include elements needed for effective operations—prioritization of duties and the establishment of measurable, reasonable goals. Finally, we are concerned that an organization charged with such critical responsibilities has not made more progress on its mission during the more than five years it has existed and has not adequately prioritized its efforts to address unmet responsibilities. State law created DOIT in 1996 because a need existed to facilitate the application of IT in the State. State law gave DOIT broad authority to coordinate the State’s IT efforts and oversee most departments’ IT projects. Although DOIT has faced challenges with the year 2000 effort and staffing issues, these issues only partially explain DOIT’s inability to fulfill its statutory responsibilities. RECOMMENDATIONS To promote coordination on IT projects and avoid redundant efforts, DOIT should establish a formal mechanism to initiate discussions between departments that are developing projects 66 based on similar technologies or processes. To facilitate this coordination and improve project oversight, DOIT should complete its IT project inventory based on its survey of depart- ments. DOIT should ensure that departments’ reported data are accurate and should update this information when departments report new information so that the project inventory stays current. DOIT also needs to consider how departments and the Legislature can effectively access this information, taking into consideration privacy issues and other concerns that may limit the release of this information. To improve compatibility and properly guide IT development, DOIT should expedite its work on implementing standards by determining which standards need to be addressed first and focusing their efforts accordingly. Further, DOIT should work with departments to ensure that all necessary standards have been implemented. To ensure that DOIT is fully employing the IT advisory councils and receiving the benefits intended by law, DOIT should continue to meet with the private commission and the public committee on a regular basis to guide its strategic planning efforts, provide input on new policies, and ensure that the State follows best practices. DOIT should ensure that the public committee makes all findings and recommendations in writing, as required by state law. DOIT should also monitor the progress of its CIO work groups to ensure that they reach their established goals. To ensure that it completes initiatives, DOIT should establish timelines and goals for meeting future initiatives. If DOIT does not believe it can complete initiatives within established guidelines, it should communicate its priorities and resource requirements to the Legislature. In addition, it should notify the Legislature when changes in the State’s IT environment prompt adjustments to these priorities or resource requirements. To organize and focus its efforts, DOIT should adopt an internal strategic plan to identify key responsibilities and establish priorities. This plan should clearly describe how the organization will address its many responsibilities, particularly those that we observed it has not sufficiently accomplished. Further, it should build on past efforts to the extent possible rather than reinventing processes and practices when planning its future activities. 67 We conducted this review under the authority vested in the California State Auditor by Section 8543 et seq. of the California Government Code and according to generally accepted government auditing standards. We limited our review to those areas specified in the audit scope section of this report. Respectfully submitted, ELAINE M. HOWLE State Auditor Date: June 27, 2001 Staff: Karen L. McKenna, CPA, Audit Principal John Baier, CPA Faye Borton Nathan Checketts, CIA Jeremy Evans Arn Gittleman, CPA Vern Hines Kumar Sah, CISA Michelle Tabarracci, CISA Mike Urso 68 APPENDIX A Evaluation of Departments’ Project Management of Selected Information Technology Projects I n Chapter 2 we described our review of four information technology (IT) projects at different departments to determine how they used best practices and state requirements in IT project development. We selected these projects because they were large, complex, and nearing completion, which allowed us to evaluate various phases of the project development process. The following is a more detailed description of each project: • The Advanced Toll Collection and Accounting System (ATCAS) is a Department of Transportation (Caltrans) initiative to replace the existing toll collection and accounting system and install electronic toll collection on all state-owned toll bridges. Caltrans believes that the electronic toll collection system will eliminate the need for participating drivers—whose vehicles would be equipped with electronic sensors—to stop and pay at tollbooths. Under the new system, Caltrans would charge participants’ accounts each time they drive through specially equipped toll bridge lanes. In 1990 legislation passed requiring Caltrans to develop specifications and standards for the system and facilities, and Caltrans began development of ATCAS in June 1993. Caltrans estimates ATCAS will be completed in December 2001. (cid:127) The Tax Engineering and Modernization (TEAM) project, developed by the Employment Development Department, was chartered to reengineer and streamline employers’ tax returns and related payment processes, reduce or eliminate the flow of paper documents, improve efficiency in processing tax documents, increase electronic filing options, reduce costs, and improve responsiveness to customers. The Employment Development Department intended TEAM to eliminate the need for about 350 positions. To provide an incentive to the vendor to reach this goal, the payment to the vendor was based on the savings achieved. TEAM began in June 1997 and was completed in April 2001. 69 (cid:127) The Children’s Medical Services Network Enhancement 47 (CMS Net E47) is a Department of Health Services (Health Services) project to enhance the existing Children’s Medical Services Network––a case management system for two programs that serve children from low-income families unable to afford specialized medical care and individuals who have certain genetic diseases. The enhancement will link the existing system to the State’s medical and dental fiscal intermediaries. The project is intended to implement electronic billing and create a statewide patient, service authorization, and claims review database. CMS Net E47 began in January 1998, and Health Services expects to complete it in December 2002; however, elements of the project related to case management and eligibility for Medi-Cal and Healthy Families were imple- mented in April 2001. Currently, 46 counties use these elements. (cid:127) The Accounts Receivable Collection System (ARCS) was developed to expand the benefits that the Franchise Tax Board had obtained from its collection system for banks and corpo- rations. That collection system created modeling techniques to prioritize tax accounts with the greatest potential for revenue collection. ARCS was designed to unify the Franchise Tax Board’s collection efforts for all tax programs, consolidating debtor accounts, creating an enterprisewide bankruptcy and lien system, and coordinating a consistent collection approach. ARCS began in April 1998 and was completed in March 2001. Each of the four projects had cost increases and schedule overruns, as shown in Table 4. The most notable of these cost overruns occurred with the ATCAS and CMS Net E47 projects. In addition, ATCAS is significantly behind the originally scheduled completion date. The other two projects were better managed and had less significant cost increases and delays. The original and current costs for each project shown in Table 4 are for one-time development costs and do not include continuing costs for operations. 70 TABLE 4 Cost Overruns and Delays in Four IT Projects We Reviewed (Dollars in Millions) Employment Development Health Franchise Caltrans Department Services Tax Board Project ATCAS TEAM CMS Net E47 ARCS Original cost estimate ($) 27.8* 67.7 5.6 33.0 Current cost estimate ($) 56.1 71.7 10.2 36.3 Increase in cost ($) 28.3 4 4.6 3.3 Increase in cost (%) 102 6 82 10 Start date June 1993 June 1997 January 1998 April 1998 Original completion date January 1997 June 1999 September 2001 June 2000 Revised completion date December 2001 April 2001 December 2002 March 2001 Original development time (months) 44 25 45 27 Revised development time (months) 103 47 60 36 Expected delay (months) 59 22 15 9 * This amount is from the 1995 feasibility study report and, according to Caltrans, excludes $1.8 million for spare parts, and an unknown cost for Caltrans staff working on the project. Had Caltrans included the additional known costs of $1.8 million in the feasibility study report, the original cost estimate would have been $29.6 million, and the increase in cost would have been $26.5 million, or 90 percent. Table 5 shows in detail how we rated, from strong to weak, each of the four departments for their use of best practices in IT project development. After assigning a rating to each individual factor, we assigned an overall rating for each of the three phases: planning and procurement; design, development, and imple- mentation; and quality assurance. 71 TABLE 5 Evaluation of Four IT Projects Employment Development Health Franchise Caltrans Department Services Tax Board Best Practices (ATCAS) (TEAM) (CMS Net E47) (ARCS) Planning and Procurement Define a problem statement and supportable business case W S A S Outline the business problem and allow the vendor to propose a solution W S W A Clearly define and document the scope of the project W A A A Clearly define and document the use of project management methodology W I A A Sponsor the project through appropriate commitment and involvement of A A W A management, and include sufficient resources Base procurement on best value, not lowest cost A A W A Use a vendor that provides and maintains sufficient qualified staff I I W A Require a letter of credit from the vendor on a large project A NR W NR Clearly define and document the risk management plan W I I A Write a strong contract to adequately protect the State A S W A Enforce the terms of the contract W A NA A Overall Weak Acceptable Weak Acceptable Design, Development, and Implementation Design, develop, and test the project appropriately according to specifications I A A A Establish measurable objectives for the project W A S A Prioritize project elements, such as budget, schedule, and functionality, up front A A A A Develop smaller projects with milestones W A A A Use sound project management practices during implementation W A I A Establish appropriate policies and procedures to manage and control changes A I I A to the developing project Develop an adequate information security system to detect and prevent A W W A inappropriate access Overall Weak Acceptable Acceptable Acceptable Quality Assurance Create an effective quality assurance mechanism and use it during each W S I A phase of development Heed the advice of the oversight consultant or explain why it is not applicable NA A A A Pay the vendor only after accepting tested project deliverables A A W A Overall Weak Acceptable Inconsistent Acceptable S — Strong The department successfully demonstrated that it met the minimum requirements. Further, the department demonstrated additional controls or processes that exceed the minimum standards. A — Acceptable The department successfully demonstrated that it met the minimum requirements. The department may have encountered some minor problems during the IT project, but these problems did not cause significant cost or schedule overruns. I — Inconsistent Initially, the department did not successfully demonstrate that it met the minimum requirements. However, during the course of the IT project, the department took steps to meet the minimum requirements. W — Weak The department did not successfully demonstrate that it met the minimum requirements. The department’s failure to do so likely created significant problems related to the cost, schedule, or functionality of the IT project. NR — Not required These projects are benefit or performance based. Consequently, the State pays the vendor only when benefits are realized, and a letter of credit is not required. NA— Not applicable We could not evaluate the best practice. Health Services did not write a contract to adequately protect the State, and Caltrans did not have an oversight consultant. 72 APPENDIX B Major Completed and In-Development Information Technology Projects A s discussed in the Scope and Methodology, we were asked to create an inventory of the State’s major information technology (IT) projects as part of this audit. We based our project inventory on information we collected from the Department of Information Technology’s (DOIT) IT project survey of 111 state organizations. DOIT selected these departments, boards, and commissions because it was aware of IT projects at these organizations. DOIT indicates that it plans to invite orga- nizations excluded from this survey to participate in subsequent surveys. DOIT also excluded the State’s public universities and colleges from the survey because those entities are not required to report to it. For purposes of this report, we refer to state organizations in our project inventory as departments. Because many departments that DOIT surveyed initially submit- ted incomplete responses, we engaged in several rounds of follow-up contacts to fill in missing information or clarify unclear terms in the project titles. As a result of these follow-up efforts, we received sufficient information to complete Tables 6 through 8. Yet, as discussed in Chapter 3, our follow-up and our review of five departments’ documentation of 23 projects identified weaknesses in the way departments reported their information. Considering these problems, the information in these tables may contain errors, be incomplete, or be reported in a manner that is inconsistent with other departments. However, we offered all departments the opportunity to review their information, and we made any changes they indicated. Table 6 shows completed IT projects (ongoing systems) with one-time development costs greater than $1 million, listed by department. In Table 7 we give information on IT projects with one-time development costs greater than $1 million that depart- ments are currently developing. DOIT’s survey asked departments to report project information, including the percentage of the project that was complete, as of November 2000. However, some departments updated their project information during our follow-up. Departments that had no major projects in either category are listed at the end of both tables. 73 Table 8 summarizes each department’s reported IT activities. We separated reported projects into five groups: three for completed projects (greater than $1 million, less than $1 million, or no reported costs) and two for projects in development (greater than $1 million and less than $1 million). We also give the total number of completed and in-development projects reported by each department. 74 TABLE 6 Completed Projects With Development Costs Greater Than $1 Million (Dollars in Millions) Projects Listed Year Development Maintenance Costs Child Support Services, Department of Pre Statewide Interim Case management 1997 $23.3 † Systems Management (PRISM) Consumer Affairs, Department of DCA Field Office LAN’s Licensing 1997 2.6 $ 0.5 Contractors’ State Licensing Board Licensing 1991 2.0 0.2 License Applications/Cashiering System Consumer Affairs System (CAS) Licensing 1986 1.8 1.1 Licensing Module Family Support Enforcement Project Law enforcement 1996 1.8 0.3 CAS Family Support Module CAS Enforcement Module Licensing 1992 1.4 1.1 Contractors’ State Licensing Board Licensing 1991 1.3 0.2 License System (Phase I) Corrections, Department of Interim Parolee Tracking Law enforcement 1998 29.3 4.2 Standard Automated Preventive Workload 2000 10.0 0.2 Maintenance System Automated Canteen Sales Financial 1996 1.8 0.4 and Inventory Inmate Trust Accounting General accounting 1989 1.0 0.7 Developmental Services, Department of Doctors Orders and Hospital information system 1998 10.8 0.2 Communication (DOCS) Cost Recovery Financial 1991 5.8 0.8 Hospital Client Support Hospital information system 1989 2.8 0.9 System (HCSS) Employment Development Department New Employee Register Child support 1998 26.2 21.7 Job Training Automation Enhancement 1997 6.7 0.1 Compliance Automation Workflow, case management 1999 5.2 1.6 Workstations (CAW) Assembly Bill 3086 Workflow 1999 2.1 1.2 Unemployment Insurance (UI) Benefit claims 2000 1.4 0.2 Continued Claim Y2K System Replacement *A list of departments that reported no completed projects with development costs greater than $1 million begins on page 81. † The cell was left blank in the survey or the response was incomplete. Continued 75 Projects Listed Year Development Maintenance Costs by Department Description Implemented Costs Fiscal Year 1999–2000 Equalization, Board of Migration to Teale Data Center‡ Financial 1999 $ 24.0 $11.0 Automate Compliance Management Case management, financial 1997 10.6 1.2 System (ACMS) Fish and Game, Department of Geographic Information Systems Geographic information systems 1999 8.5 1.0 Franchise Tax Board Business Entities Tax Revenue management 1997 42.0 0.2 Pass-Through-Entity Automated Case management 1998 34.8 8.6 Screening & Support (PASS) Taxpayer Information (TI) Revenue management 1993 32.7 0.0 Collections Account Processing Collections 1996 7.3 0.3 Employment Information Database Revenue management 1997 3.6 0.1 Enterprise Wide Customer Telecommunication 1999 3.0 0.4 Service Platform Mainframe Capacity Augmentation Audit, filing, public service, 1999 2.3 0.1 collections Limited Liability Companies (LLC) General accounting, audit, 2000 1.9 2.4 financial, collections, filing Imaging Revenue management 1994 1.8 0.1 Collections, DMV Collections 1995 1.4 0.0 Enterprise Duplex Printing System Collections, audit 2001 1.3 § Direct Filing Portal Revenue management, 2001 1.2 § tax assessment Un-interruptible Power Source Audit, collections 2001 1.2 § (UPS) Replacement Electronic Filing Revenue management 1995 1.1 0.0 General Services, Department of Procurement Division (PD) Procurement 1994 4.9 0.2 Information Network Office of Fiscal Services (OFS) Financial, human resources 1998 4.0 0.0 Activity-Based Management System (ABMS) Executive Branch (EB) Workflow, workload 2000 1.1 § E-Mail System management Health and Human Services Agency Data Center Child Welfare System/Case Case management 1998 119.5 53.8 Management System (CWS/CMS) Statewide Automated Welfare System – Case management, financial, 1998 114.7 21.5 Interim Statewide Automated Welfare workload management, System (SAWS – ISAWS) Consortium eligibility determination ‡ Migration to Teale Data Center includes the following projects: Integrated Revenue Information System (IRIS), Aircraft/Vessels (ATAV), Board Roll, Consumer Use Tax FAA Information File (ATAI), Homeowners, Legal Entity Ownership Program (LEOP), Private Railroad Car, Timber Tax, Special Taxes Automated Revenue System (STARS), Excise Tax Beer Vendors (ETBV), Excise Tax Importers (ETIM), Excise Tax Purchase & Sales (ETPS), Cigarette Tax Stamp Orders (CTSO), and Cigarette Tax Stamp Tracking (CTST). § The project was completed after fiscal year 1999-2000. Continued 76 Projects Listed Year Development Maintenance Costs by Department Description Implemented Costs Fiscal Year 1999–2000 Health Services, Department of Medi-Cal Management Information Data warehousing, 2000 $43.9 $7.6 System/Decision Support System management information system decision support California Medicaid Management Health care 1980 23.5 15.0 Information System (CA–MMIS) Women, Infants, and Children/ Case management, financial 1997 18.5 19.0 Integrated Stateline Information services, reporting architectures, System (WIC/ISIS) on-line purchase, tracking Medi–Cal Eligibility Data System (MEDS) Health care administration 1984 14.0 25.3 Denti–Cal Claims processing 1986 4.5 0.5 Exchange E–mail System E-mail messaging system 1999 3.0 1.7 Desktop Assessment Project Ongoing inventory 1999 1.7 0.2 Highway Patrol, California Department of Computer Aided Dispatch System (CAD) Law enforcement 1999 13.1 2.3 Housing and Community Development, Department of Codes and Standards Automated Office automation, document 2000 11.2 4.0 System Re-engineering Project tracking, licensing, registration and titling, financial, law enforcement, title search, workflow Insurance, Department of Integrated Data Base (IDB) Registration and titling 1999 8.6 1.5 Justice, Department of ll Cal–ID Equipment Replacement Law enforcement, automated 2000 42.5 1.2 fingerprint identification system (AFIS) Information Technology Needs Law enforcement 2000 14.3 1.2 Statewide Integrated Narcotic Law enforcement, 1997 14.1 2.5 System (SINS) telecommunication, case management California Law Enforcement Law enforcement, 1999 10.7 2.5 Telecommunications System telecommunication (CLETS) Replacement Applicant Live Scan Fingerprint 1998 10.4 1.6 National Criminal History Fingerprint, criminal history 1999 8.0 0.0 Improvement Program (NCHIP) Year 1&2 Statewide Alcohol Information Telecommunication, 1999 6.8 0.4 Network (SAIN) law enforcement, case management CLETS Hardware Replacement Law enforcement, 2000 6.3 0.0 telecommunication NCHIP Year 3 Fingerprint, criminal history 2000 4.3 0.0 NCHIP Year 4 Fingerprint, criminal history 2000 3.8 0.0 ll The department only reported projects completed since 1997. Continued 77 Projects Listed Year Development Maintenance Costs by Department Description Implemented Costs Fiscal Year 1999–2000 Justice, Department of (continued)ll Southwest Border States Anti–Drug Law enforcement 1999 $ 3.1 $0.9 Information System Integration Juvenile Accountability Incentive Fingerprint 2000 3.0 0.0 Block Grant (JAIBG) Violent Crime Information Network Law enforcement, 2000 2.4 0.6 (VCIN)/Megan’s Law CD Distribution automated fingerprint Change (AB 2799) identification system (AFIS), telecommunication Consolidated Firearms Information Law enforcement, workflow 1999 1.9 2.0 System (CFIS) Expansion of Cal-DNA/CODIS Law enforcement 2000 1.4 0.0 Live Scan Augmentation Fingerprint 2000 1.4 0.0 Automated Cartridge System (ACS) Computer operations 1997 1.3 0.2 Cal–Gang Law enforcement 1999 1.2 0.3 Implementation of AB 1659 Law enforcement, automated 2000 1.1 0.0 fingerprint identification system (AFIS) Lottery Commission, California State On–line Games Processing System Lottery product sales 1993 70.0 30.0 Scratch Ticket Processing System Lottery product sales 1997 29.0 30.0 Epicore Automated Financial Financial 1999 3.0 0.5 Accounting System Lottery Retailer Master File Lottery product sales 1993 1.4 0.1 Military Department-Office of the Adjutant General Wide Area and Local Area Network 2000 4.5 § Networks (WAN/LAN) Motor Vehicles, Department of Driver Safety Program Postlicensing control 1996 126.3 0.0 Automation Project Applying Technology to Financial, vehicle 1998 11.9 0.0 Remittance Processing registration, titling Field Office Printer Replacement Infrastructure 1999 4.8 0.0 Occupational Licensing Licensing 1999 3.7 0.0 System Redesign Smog Impact Fee Refund One-time $300 SMOG 2000 1.8 0.1 impact fee refund Motor Carrier Permit (MCP) System Registration 1998 1.6 0.1 Network Component Replacement Infrastructure 1999 1.6 0.0 Telephone Systems Upgrade/ Telecommunication 1999 1.4 0.2 Staffing Augmentation Telephone Service Center Licensing, vehicle registration 2000 1.3 0.0 Upgrade (Pilot) and titling, telecommunication llThe department only reported projects completed since 1997. §The project was completed after fiscal year 1999-2000. Continued 78 Projects Listed Year Development Maintenance Costs by Department Description Implemented Costs Fiscal Year 1999–2000 Motor Vehicles, Department of (continued) Vehicle Registration Internet Vehicle registration renewal 2000 $ 1.2 $0.0 Renewal System Public Employees’ Retirement System CalPERS Online Member & Employer Health benefit data support 1998 13.2 0.9 Transaction System (COMET) System COMET Employer and Participant Demographics, employment and 2001 13.1 § retirement information data base Investment Accounting Portfolio management 2000 5.3 0.8 and Reconciliation reconciliation Actuary and Employers Division (AESD) Actuarial valuation system 1996 3.5 0.2 PeopleSoft Financials Accounting 1999 3.2 0.9 Automated Communication Exchange Web application, 2000 2.8 § System (ACES) electronic transactions, inquiry COMET Transmission Log (T-Log) Batch process 2001 1.5 § Public Utilities Commission Office Automation 2000 Workload management 1998 2.1 0.4 Transportation Management Licensing 1988 1.2 0.3 Information System (TMIS) Rehabilitation, Department of Field Computer System (FCS) Vocational Case management 1990 1.0 0.3 Rehabilitation Case Management System Secretary of State Limited Partnership/ Document tracking 1984 8.5** † Limited Liability Companies Uniform Commercial Code Filings Document tracking 1995 4.0 0.0 Online Status System CalVoter I: Elections-Voter Elections enforcement 1999 3.5 7.3 Registration Database Calvoter II: Elections Elections 2000 2.9 0.3 Cal-Access: Political Reform – Document tracking 2000 2.4 1.1 Campaign Filings and Political Reform – Lobbyist Filings Management Services – Fiscal: Financial 1999 1.4 2.1 Financial Information Resource Management System (FIRMS) Social Services, Department of Field Automation System Application expansion 1997 1.3 0.0 (FAS) - Residential Student Aid Commission, California Grant Delivery System Financial 2000 4.7 0.5 Teachers’ Retirement System, State STRS Enterprise Database Case management 1985 6.1 1.5 §The project was completed after fiscal year 1999-2000. ** The department reported total costs for this project because it could not readily separate development costs from other costs. † The cell was left blank in the survey or the response was incomplete. Continued 79 Projects Listed Year Development Maintenance Costs by Department Description Implemented Costs Fiscal Year 1999–2000 Transportation, Department of District 07 Automated Transportation Traffic operations 2000 $6.8 $1.2 Management System (ATMS) Version II Toll Registration and Audit Toll bridge operations 1989 5.0 0.5 Collection System (TRAC) District 00 Transportation Systems Highway inventory 1998 4.3 0.4 Network and Inventory District 00 Transportation Accounting Financial 1984 4.1 2.5 Management System (TRAMS) District 11 Ramp Meter Traffic management, 1972 4.0 0.5 congestion management District 00 Expert Project Project management 1997 3.3 0.1 Manager (XPM) District 11 Reversible Lanes Traffic management, 1988 3.0 0.7 congestion management District 00 Intermodal Transportation Transportation planning 1998 2.8 0.1 Management System (ITMS) District 00 Current Billing Financial 1991 2.3 1.6 and Reporting (CBARS) District 00 Caltrans Financial 1995 2.3 0.7 Accounts Payable (CAPS) District 00 Highway Accident Database Planning 1998 2.1 0.2 District 04 Toll Bridge Accounting (TBA) Financial 1986 2.0 0.3 District 04 Toll Registration and Audit Financial 1986 2.0 0.2 Collection (TRAC)/SAIC Programs for Bay Bridge District 00 Time Reporting (TRS) Human resources 1993 1.8 1.4 District 00 California Contract management 1993 1.8 0.7 Certification System District 00 Traffic Volumes (TV) Inventory 1998 1.8 0.3 District 00 Right of Way Rentals (RWR) Financial 1998 1.6 0.4 District 00 Right of Way Property Project management 1992 1.6 0.3 Management (RWPM) District 00 Pavement Highway inventory 1998 1.4 0.3 Management System District 59 Bridge Inspection Engineering archive 1993 1.2 0.2 Reporting System District 00 Federal Aid Data Financial 1994 1.1 0.6 System (FADS) District 00 Highway Inventory 1997 1.1 0.2 Performance Monitoring District 04 Toll Bridge Financial 1986 1.0 0.1 Reconciliation (TBR) Treasurer, State Item Processing - Replacement Financial 1999 4.3 0.2 Continued 80 Projects Listed Year Development Maintenance Costs by Department Description Implemented Costs Fiscal Year 1999–2000 Veterans Affairs, Department of Farm & Home Loan System (CalVet) Complete mortgage banking 1999 $ 1.7 $0.1 Veterans Home Information Complete health care system 1998 1.5 0.1 System – Barstow Water Resources Control Board, State Water Rights Information Support water rights program 1999 2.3 0.4 Management System (WRIMS) Water Resources, Department of SAP-1 Enterprise Resource Planning Enterprise resource planning 1999 17.4 4.9# System Implementation/ (Business 2000 Phase 1) SAP-2 Enterprise Resources Planning Enterprise resource planning 2000 13.2 0.0 System Implementation/ (Business 2000 Phase 2a) SAP-CARA (Cost Allocation State water project and Repayment System) billing system 2000 5.4 # The following departments reported no completed projects with development costs greater than $1 million: Administrative Law, Office of Conservation, Department of African American Museum Controller, State Aging, Department of Corporations, Department of Agricultural Labor Relations Board Corrections, Board of Air Resources Board Criminal Justice Planning, Office of Alcohol and Drug Programs, Department of Education, Department of Alcoholic Beverage Control, Department of Education, Office of the Secretary for Arts Council, California Education, State Board of Boating and Waterways, Department of Emergency Medical Services Authority Building Standards Commission, California Emergency Services, Office of Business, Transportation and Housing Agency Employment Training Panel California Victim Compensation and Government Claims Energy Resources, Conservation and Development Board (formerly the State Board of Control) Commission Coastal Commission, California Environmental Health Hazard Assessment, Office of Coastal Conservancy, State Environmental Protection Agency, California Colorado River Board of California Exposition and State Fair, California Community Colleges, Board of Governors Fair Employment and Housing, Department of Community Services and Development, Department of Fair Political Practices Commission Conservation Corps, California Finance, Department of #The $4.9 million is for both the SAP-1 and the SAP-CARA programs. Continued 81 The following departments reported no completed projects with development costs greater than $1 million: Financial Institutions, Department of Pesticide Regulation, Department of Food and Agriculture, Department of Postsecondary Education Commission, California Forestry and Fire Protection, Department of Prison Terms, Board of Health and Human Services Agency, California Public Employment Relations Board Health Planning and Development, Office of Statewide Real Estate Appraisers, Office of Horse Racing Board, California Real Estate, Department of Housing Finance Agency, California Resources Agency Industrial Relations, Department of San Francisco Bay Conservation and Development Commission Inspector General, Office of the Santa Monica Mountains Conservancy Insurance Advisor, Office of Science Center, California Integrated Waste Management Board, California State and Consumer Services Agency Law Revision Commission, California State Lands Commission Library, California State Stephen P. Teale Data Center Managed Health Care, Department of Tahoe Conservancy, California Managed Risk Medical Insurance Board Teacher Credentialing, Commission on Mandates, Commission on State Technology, Trade and Commerce Agency, California Medical Assistance Commission, California Toxic Substances Control, Department of Mental Health, Department of Traffic Safety, Office of Narcotic Addict Evaluation Authority Unemployment Insurance Appeals Board, California Parks and Recreation, Department of Youth and Adult Correctional Agency Peace Officer Standards and Training, Commission on Youth Authority, Department of the Personnel Administration, Department of Youthful Offender Parole Board Personnel Board, State 82 83 TABLE 7 Projects in Development With Estimated Costs Greater Than $1 Million (Dollars in Millions) Original Revised Original Current Projected Projected Projected Projected Original Current Projects Listed Original Actual Completion Completion Percent One-Time One-TimeProjected Projected by Department* Description Start Date Start Date Date Date Complete Cost Cost Savings Savings California Victim Compensation and Government Claims Board (formerly the State Board of Control) Restitution Track System (10% Rebate) Case management Sep–99 Sep–99 Dec–00 Feb–01 95% $1.2 $ 0.7 $1.6 $0.5 Conservation, Department of Division of Recycling Integrated Information Integrated recycling Jul–99 Mar–01 Jun–03 Mar–04 1 8.5 8.5 1.7 1.7 System (DORIIS) information Controller, State Automated Travel Expense Automated Jul–00 Jul–00 Jul–02 Jul–02 50 4.0 4.0 28.5 28.5 Reimbursement System travel expense reimbursement system Corrections, Department of Cable Plant Telephone System Telecommunication Apr–00 Apr–00 Jun–03 Jun–03 4 22.4 22.4 0.0 0.0 Parole Law Enforcement Automated Law enforcement Jul–97 Jul–97 Nov–98 Jun–01 80 3.7 6.0 0.0 0.0 Data System (LEADS) Phase II Enhanced Peace Officer Training Training Mar–00 Jul–00 Jun–02 Jun–02 80 1.3 1.3 0.0 0.0 Revocation Scheduling and Tracking System Law enforcement Jul–98 Jan–99 Jun–99 May–01 70 0.8 1.8 0.3 -1.3 Developmental Services, Department of California Developmental Disabilities Case management, Jul–00 Jul–00 Jun–03 Jun–05 10 9.9 9.9 0.0 0.0 Information System (CADDIS) contract management, financial, general accounting Health Insurance Portability and Case management, Jul–00 Sep–00 Jun–05 Jun–05 1 2.6 2.6 0.0 0.0 Accountability Act contract management Education, Department of Principal Apportionment System Financial Feb–97 Apr–98 Jun–01 Jun–02 31 3.6 4.4 0.0 0.0 Rewrite (PASR) * A list of departments that reported no projects in development with estimated costs greater than $1 million is on page 91. Continued 84 Original Revised Original Current Projected Projected Projected Projected Original Current Projects Listed Original Actual Completion Completion Percent One-Time One-TimeProjected Projected by Department Description Start Date Start Date Date Date Complete Cost Cost Savings Savings Employment Development Department Tax Engineering and Document tracking, Jun–97 Jun–97 Jun–99 Apr–01 75% $67.7 $71.7 $2.3 $2.3 Modernization (TEAM) workload management Personal Identification Number (PIN) Benefit programs May–00 Oct–00 Jun–00 Jun–01 55 2.3 2.3 0.0 0.0 Independent Contractor Reporting Child support Nov–99 Nov–99 Jan–01 Jan–01 90 1.3 1.3 0.0 0.0 (Senate Bill 542) Equalization, Board of Return Processing Automation Project Financial, workflow, Apr–00 Apr–00 Apr–02 Apr–02 1 28.4 28.4 27.6 27.6 workload management Revenue Database Workflow, workload Jan–01 Jan–01 Sep–04 Sep–04 10 15.3 15.3 35.2 35.2 Consolidation/AB 2114 management Field Office Automation Workflow, workload Oct–96 Jul–97 May–06 Jun–03 20 5.0 5.0 64.5 64.5 management PC Hardware and Software Refresh Boardwide infrastructure Sep–00 Sep–00 Jun–01 Jun–01 5 2.5 2.5 0.0 0.0 maintenance Fish and Game, Department of IT Computing Services Department IT services Aug–01 ‡ Jun–02 ‡ 0 5.0 ‡ 0.0 ‡ Office Automation Standards Office automation Aug–01 ‡ Jun–03 ‡ 0 5.0 ‡ 0.0 ‡ Automated Licensing Data System Licensing Jul–01 ‡ Jun–02 ‡ 0 4.8 ‡ 0.2 ‡ Franchise Tax Board Integrated Nonfiler Compliance (INC) Revenue management, Jul–98 Jul–98 Dec–01 Dec–01 75 46.3 46.3 67.5 67.5 tax assessment Accounts Receivable Collection Revenue management, Apr–98 Apr–98 Jun–00 Mar–01 90 33.0 36.3 39.7 68.0 System (ARCS) tax assessment Modernization and Reengineering Revenue management, Jun–98 Jun–98 Jun–01 Jun–01 50 6.6 8.1 9.2 0.0 Cashiering System (MARCS) tax assessment Network Backbone Support Mar–00 Mar–00 Dec–01 Jul–05 20 6.3 6.1 0.0 0.0 Data Center Infrastructure Improvement Audit Jul–01 ‡ Nov–02 Nov–02 0 4.6 4.6 0.0 0.0 General Services, Department of Procurement Division (PD) Cal–Buy Procurement Oct–00 Oct–00 Oct–01 Jun–01 50 9.4 5.3 13.0 7.3 Executive Branch (EB) Network Workload Jul–00 Nov–00 Apr–01 May–01 90 1.5 3.0 1.5 1.5 EB California Homepage eGovernment Oct–00 Sep–00 Dec–01 Dec–01 50 1.4 3.7 1.6 1.6 ‡ This project has not started; therefore, these dates or costs are not applicable. Continued 85 Original Revised Original Current Projected Projected Projected Projected Original Current Projects Listed Original Actual Completion Completion Percent One-Time One-Time Projected Projected by Department Description Start Date Start Date Date Date Complete Cost Cost Savings Savings Health and Human Services Agency Data Center Statewide Automated Welfare System Case management, Oct–99 Feb–00 Dec–03 Dec–03 6% $245.7 $318.0 $518.9 $593.0 (SAWS) – Welfare Case Data System human resources, (WCDS) Consortium telecommunication, eligibility determination, welfare SAWS – Los Angeles Eligibility Automated Case management, Oct–95 Nov–95 Mar–99 Apr–01 94 76.5 114.0 374.6 374.6 Determination, Evaluation and Reporting financial, workload (LEADER) Consortium management, eligibility determination SAWS – Welfare Data Tracking Case management, Mar–99 Mar–99 Jul–00 Dec–01 59 9.7 11.9 0.0 0.0 Implementation Project (WDTIP) financial Statewide Fingerprint Imaging System (SFIS) Fingerprint Jul–95 Nov–95 Sep–03 Apr–01 98 7.4 17.3 125.0 125.0 Electronic Benefit Transfer (EBT) Financial Jul–97 Jul–97 Feb–01 Nov–04 0 6.9 6.9 521.0 521.0 SAWS Consortium IV Case management, Dec–97 Dec–97 Aug–04 Jul–04 0 5.8 6.0 467.4 467.4 human resources, telecommunication, eligibility determination SAWS – Interim Statewide Automated Eligibility determination Jul–99 Jul–99 Jun–01 Jul–01 60 2.6 6.4 0.0 7.0 Welfare System (ISAWS) – Welfare Reform and Infrastructure Modifications (WRIM) SAWS – CalSERV Middleware Case management Jul–99 Jul–99 Feb–03 Feb–03 0 1.1 1.3 0.0 0.0 Health Planning and Development, Office of Statewide Medical Information Reporting Document tracking, Sep–00 Sep–00 Dec–02 Dec–02 10 5.2 5.2 0.0 0.0 for California (MIRCal) workflow, workload management, hospital patient data Health Services, Department of DHS Health Insurance Portability and Accountability Act (HIPAA) All health care Jun–00 Jun–00 Nov–02 Nov–02 0 21.9 † 0.0 0.0 Business Systems Replacement General accounting Jul–98 Mar–99 Apr–99 Jul–01 90 5.7 5.1 8.1 8.1 † The cell was left blank in the survey or the response was incomplete. Continued 86 Original Revised Original Current Projected Projected Projected Projected Original Current Projects Listed Original Actual Completion Completion Percent One-Time One-TimeProjected Projected by Department Description Start Date Start Date Date Date Complete Cost Cost Savings Savings Health Services, Department of (continued) Children’s Medical Services Case management, Jul–96 Jan–98 Sep–01 Dec–02 22% $5.6 $10.2 $67.3 $20.1 Net/Enhancement 47 (CMS Net 47) document tracking Richmond Lab IT Support Telecommunication Jul–00 Jul–00 Jun–06 Jun–06 20 4.7 4.7 0.0 0.0 Computer Utilization for Radiation Licensing Nov–98 Nov–98 Jun–02 † 18 3.6 † 2.8 † Information and Enforcement (CURIE) Statewide Immunization Information Patient care Jul–95 Jan–01 Jul–00 Jul–03 0 2.6 3.0 10.4 10.4 System [SIIS (IZ Registry)] Newborn Hearing Screening Project, Case management, Jul–99 Jul–99 Dec–02 Dec–02 35 0.9 2.4 7.5 7.5 Data Management System document tracking Highway Patrol, California Department of Statewide Integrated Traffic Law enforcement Aug–98 Aug–00 Jul–01 Apr–02 10 3.1 2.3 0.0 0.0 Records System (SWITRS) California Automated Law enforcement Jan–95 Apr–97 Jun–99 Jul–02 65 2.0 2.0 0.0 0.0 Reporting System (CARS) Graphical User Interface (GUI) Law enforcement Mar–00 Oct–00 Jul–02 Apr–03 5 1.3 1.3 0.0 0.0 Activity and Attendance System Human resources Dec–92 Jun–92 Mar–98 Dec–01 75 1.1 1.4 0.7 0.1 Industrial Relations, Department of Division of Workers’ Compensation Workers’ Jul–97 Jul–97 Dec–99 Jan–01 90 0.5 1.0 2.9 0.0 (DWC) Workers’ Compensation compensation Information System (WCIS) Justice, Department of Automated Criminal History Law enforcement Jun–00 Jun–00 Jun–03 Jun–03 20 15.1 15.1 0.0 0.0 System (ACHS) Migration Statewide Integrated Narcotic Law enforcement Jul–00 Jul–00 Jun–05 Jun–05 10 8.1 8.1 0.0 0.0 System (SINS) Enhancements Consolidated Firearms Information Law enforcement Dec–99 Dec–99 Dec–01 Dec–01 75 3.9 3.9 0.0 0.0 System (CFIS) Enhancements California Automated Palm Print Law enforcement, Jul–99 Jul–99 Jun–01 Jun–01 10 2.7 2.7 0.0 0.0 automated fingerprint identification system (AFIS) National Sex Offender Law enforcement Jan–99 Jan–99 Jun–00 Dec–01 25 2.3 2.6 0.0 0.0 Registry (NSOR) † The cell was left blank in the survey or the response was incomplete. Continued 87 Original Revised Original Current Projected Projected Projected Projected Original Current Projects Listed Original Actual Completion Completion Percent One-Time One-TimeProjected Projected by Department Description Start Date Start Date Date Date Complete Cost Cost Savings Savings Motor Vehicles, Department of Administrative and Financial Financial, Jul–99 Nov–99 Jun–02 Jun–02 30% $14.0 $16.4 $0.2 $0.3 Systems Redesign (AFSR) general accounting Vehicle License Fee (VLF) Rebate Financial, registration, Aug–00 Aug–00 Jun–01 Apr–01 95 6.0 6.0 0.0 0.0 titling Field Office Terminal Replacement Infrastructure Jul–98 ‡ Jun–01 Sep–03 0 3.5 7.3 0.0 0.0 Personnel Board, State Automated Case Tracking System Case management, Jul–00 Jul–00 Dec–00 Feb–01 66 1.4 1.4 0.0 0.0 document tracking, workflow, workload Pesticide Regulation, Department of Enforcement and Compliance Action Tracking System Enforcement Jul–98 Jul–99 Dec–99 Mar–01 75 1.2 1.2 -1.2 -1.2 Public Employees’ Retirement System Straight Through Processing (STP) Retirement, Jan–01 Jan–01 FY 2002–03 FY 2004–05 0 31.0 31.0 0.0 † (Investment settlement seamless process) member benefits CalPERS Online Member & Employer Benefits Jul–99 Jul–99 Dec–03 Jan–04 30 17.5 2.4 † † Transaction System (COMET) Phase IV, management Project 2 (Contribution Reporting Systems) Customer Contact Center Contact center May–00 May–00 Jun–01 Jun–02 1 15.1 15.3 † † COMET Database Redesign Phase IV, Benefits Jul–98 Jul–98 Mar–01 Sep–01 72 14.3 14.3 0.0 † Project 1 (Employer Contracts & management Participant Registration) eBusiness/Automated Communication Retirement, Jul–00 Jun–00 Jun–01 Jun–01 30 3.6 5.8 0.0 † Exchange System (ACES) FY 00/01 health insurance eService/Member Self–service FY 00/01 Retirement Jul–00 Jul–00 Jun–01 Jun–01 36 2.1 2.4 0.0 † Middleware Tool Implementation Retirement, Jul–00 Jul–00 Jun–01 Jun–01 15 2.0 2.5 0.0 † member benefits COMET Phase IV, Project 3 Benefits Jul–98 Jul–98 Mar–01 Sep–01 72 1.8 1.8 0.0 † (SCO Transaction Log) management Service Level Improvements Project Business process Jul–00 Jul–00 Feb–02 Feb–02 18 0.7 1.9 0.0 † reengineering ‡ This project has not started; therefore, these dates or costs are not applicable. † The cell was left blank in the survey or the response was incomplete. Continued 88 Original Revised Original Current Projected Projected Projected Projected Original Current Projects Listed Original Actual Completion Completion Percent One-Time One-TimeProjected Projected by Department Description Start Date Start Date Date Date Complete Cost Cost Savings Savings Real Estate, Department of Licensing Tracking System Licensing Jul–94 Jul–97 Jun–00 Jun–01 80% $ 2.7 $ 6.7 $0.0 $0.2 Rehabilitation, Department of Field LAN/WAN Project Human resources Aug–00 Aug–00 Sep–01 Sep–01 40 3.5 3.5 2.0 2.0 case management, asset management Resources Agency California Land Science Information Environmental Feb–98 Feb–98 Jun–03 Jun–03 70 1.4 † –1.4 † Partnership (CaLSIP) planning, management Social Services, Department of Community Care Licensing Division Licensing Aug–97 Aug–97 Mar–01 Jun–01 70 2.4 2.4 0.0 0.0 Information System Integration – Phase I & II Community Care Licensing Division Licensing Aug–99 Mar–99 Jul–01 Mar–02 40 2.2 2.2 0.0 0.0 Information System Enhancement – Phase III Student Aid Commission, California Grant Delivery System (GDS) Financial Dec–00 Dec–00 Dec–02 Dec–02 0 9.4 9.4 0.0 0.0 Entitlement Implementation (SB 1644) Teacher Credentialing, Commission on Teacher Credentialing Service Licensing Jul–01 ‡ Dec–02 Dec–02 0 6.4 6.4 0.0 0.0 Improvement Project Teachers’ Retirement System, State State Teachers’ Automation Case management, Aug–95 Aug–95 Mar–00 Mar–01 90 21.7 21.7 0.0 0.0 Redesign Team (START) financial Corporate Imaging Project Document tracking, Jul–01 ‡ Sep–03 Sep–03 0 4.4 4.4 0.0 0.0 workflow Transportation, Department of District 07 Showcase Program Intelligent transportation Jul–94 Jun–95 Jun–02 Jun–02 65 30.0 30.0 0.0 0.0 systems (ITS) research demo District 00 Advanced Toll Collection Toll collection Oct–92 Jun–93 Jan–97 Dec–01 65 27.8 56.1 2.4 2.5 & Accounting System (ATCAS) * The cell was left blank in the survey or the response was incomplete. ‡ This project has not yet started; therefore, these dates or costs are not applicable. Continued 89 Original Revised Original Current Projected Projected Projected Projected Original Current Projects Listed Original Actual Completion Completion Percent One-Time One-TimeProjected Projected by Department Description Start Date Start Date Date Date Complete Cost Cost Savings Savings Transportation, Department of (continued) District 00 Transportation Operations Human resources Jan–98 Apr–98 Jun–01 Jun–01 5% $18.6 $18.6 $5.4 $1.8 Project System (TOPSS–II) District 00 Transportation Permits Apr–01 Jun–01 Jul–02 Jul–02 5 15.0 15.0 0.0 0.0 Permits (TPMS2000) District 00 Construction Management Project Jul–02 Jul–02 Jun–04 Jun–04 0 11.6 11.6 28.6 28.6 System 1 management District 00 Integrated Maintenance Highway Jan–00 Feb–00 Dec–02 Dec–02 30 8.5 8.5 12.0 12.0 Management System (IMMS) inventory District 07 Los Angeles Regional Traffic Oct–98 Oct–98 Oct–02 Oct–02 70 4.5 4.5 † † Transportation Management Center Systems management District 00 Lane Closure Project Jul–02 Jul–02 Jun–04 Jun–04 0 4.4 4.4 0.0 0.0 management District 00 Caltrans Land Project Jul–02 Jul–02 Feb–04 Feb–04 0 4.1 4.1 5.7 5.7 Management System management District 00 Project Resources & Schedule Project Jul–00 Jul–00 Dec–02 Dec–02 5 4.0 11.5 0.0 0.0 Management (PRSM) management District 00 Project Management Information Project Oct–98 Oct–98 Dec–99 Jun–02 75 4.0 4.0 0.0 0.0 System – Toll Bridge (PMIS–TB) management District 00 Traffic Accidents Database – Traffic Jul–98 May–99 Dec–00 Jul–01 75 2.7 2.1 0.5 0.5 District Access operations District 59 CTBRIDGE (Computer Software Engineering Jan–98 Jan–98 Dec–01 Dec–07 33 1.5 3.1 0.0 0.0 Used by Engineers to Design Bridges According to California Requirements) District 59 Conversion of Right of Way Document Oct–98 Oct–98 Jun–02 Jun–02 67 1.5 1.5 21.1 21.1 Maps and Survey Records to Archivable conversion, Electronic Format archiving District 00 District 04 Traffic Management Project Apr–00 May–99 Apr–04 Apr–04 75 1.5 1.5 0.0 0.0 Center Support management District 00 Extra Work Billing System (EWB) Workload Aug–99 Aug–99 Sep–00 Feb–02 50 1.2 2.2 0.0 0.0 management † The cell was left blank in the survey or the response was incomplete. Continued 90 Original Revised Original Current Projected Projected Projected Projected Original Current Projects Listed Original Actual Completion Completion Percent One-Time One-TimeProjected Projected by Department Description Start Date Start Date Date Date Complete Cost Cost Savings Savings Treasurer, State Debt Management System Redesign Bond issue Aug–00 Aug–00 Aug–01 Aug–01 40% $1.2 $1.0 $0.0 $0.0 Project (STRIPS) Phase 1 Veterans Affairs, Department of Veterans Home Information Complete Jun–98 Sep–98 Apr–99 Jun–02 80 5.4 5.2 -5.4 -5.2 System (Yountville) health care system Veterans Home Information System Complete Oct–99 Nov–99 Oct–00 † 20 1.8 2.1 -1.8 -2.1 (Chula Vista) health care system Water Resources Control Board, State Geographic Environmental Information Track tank Mar–00 Mar–00 Mar–01 Mar–01 60 1.4 1.4 0.0 0.0 Management System (GEIMS) sites System for Water Information Information on May–99 May–99 Jun–99 Jun–00 50 0.8 1.8 2.1 0.8 Management (SWIM) state water quality Water Resources, Department of Exchange: Advanced Enterprise E–mail Advanced enterprise Feb–01 Feb–01 Oct–01 Oct–01 2 1.5 1.5 0.4 0.4 and Groupware Project e–mail, groupware Youth Authority, Department of the California Youth Authority (CYA) E–Rate Telecommunication Jul–00 Jan–01 Jun–01 Jun–01 0 5.9 † 0.0 † Digital High School Program Youthful Offender Database (YODA) Case management, Jul–96 Oct–97 Dec–00 Sep–01 90 1.0 1.9 0.0 0.5 document tracking, workload † The cell was left blank in the survey or the response was incomplete. Continued 91 The following departments reported no projects in development with estimated costs greater than $1 million: Administrative Law, Office of Environmental Health Hazard Assessment, Office of Peace Officer Standards and Training, Commission on African American Museum Environmental Protection Agency, California Personnel Administration, Department of Aging, Department of Exposition and State Fair, California Postsecondary Education Commission, California Agricultural Labor Relations Board Fair Employment and Housing, Department of Prison Terms, Board of Air Resources Board Fair Political Practices Commission Public Employment Relations Board Alcohol and Drug Programs, Department of Finance, Department of Public Utilities Commission Alcoholic Beverage Control, Department of Financial Institutions, Department of Real Estate Appraisers, Office of Arts Council, California Food and Agriculture, Department of San Francisco Bay Conservation and Development Commission Boating and Waterways, Department of Forestry and Fire Protection, Department of Santa Monica Mountains Conservancy Building Standards Commission, California Health and Human Services Agency, California Science Center, California Business, Transportation and Housing Agency Horse Racing Board, California Secretary of State Child Support Services, Department of Housing and Community Development, Department of State and Consumer Services Agency Coastal Commission, California Housing Finance Agency, California State Lands Commission Coastal Conservancy, State Inspector General, Office of the Stephen P. Teale Data Center Colorado River Board of California Insurance Advisor, Office of Tahoe Conservancy, California Community Colleges, Board of Governors Insurance, Department of Technology, Trade and Commerce Agency, California Community Services and Development, Department of Integrated Waste Management Board, California Toxic Substances Control, Department of Conservation Corps, California Law Revision Commission, California Traffic Safety, Office of Consumer Affairs, Department of Library, California State Unemployment Insurance Appeals Board, California Corporations, Department of Lottery Commission, California State Youth and Adult Correctional Agency Corrections, Board of Managed Health Care, Department of Youthful Offender Parole Board Criminal Justice Planning, Office of Managed Risk Medical Insurance Board Education, Office of the Secretary for Mandates, Commission on State Education, State Board of Medical Assistance Commission, California Emergency Medical Services Authority Mental Health, Department of Emergency Services, Office of Military Department-Office of the Adjutant General Employment Training Panel Narcotic Addict Evaluation Authority Energy Resources, Conservation and Development Commission Parks and Recreation, Department of TABLE 8 Summary of Departments’ IT Projects Completed Projects Projects In Development Development Development Development Development Total Costs Greater Costs Less No Reported Costs Greater Costs Less Number of Than Than Development Than Than Reported Department $1 Million $1 Million Costs $1 Million $1 Million Projects Administrative Law, Office of 0 0 0 0 0 0 African American Museum 0 0 0 0 0 0 Aging, Department of 0 2 0 0 0 2 Agricultural Labor Relations Board 0 0 0 0 0 0 Air Resources Board 0 2 0 0 0 2 Alcohol and Drug Programs, 0 19 0 0 3 22 Department of Alcoholic Beverage Control, Department of 0 0 0 0 0 0 Arts Council, California 0 1 0 0 0 1 Boating and Waterways, Department of 0 4 0 0 0 4 Building Standards Commission, California 0 0 0 0 0 0 Business, Transportation and 0 4 0 0 0 4 Housing Agency California Victim Compensation and 0 5 0 1 0 6 Government Claims Board (formerly the State Board of Control) Child Support Services, Department of 1 0 0 0 0 1 Coastal Commission, California 0 4 0 0 0 4 Coastal Conservancy, State 0 0 0 0 0 0 Colorado River Board of California 0 0 0 0 0 0 Community Colleges, Board of Governors 0 10 0 0 3 13 Community Services and Development, 0 5 0 0 0 5 Department of Conservation Corps, California 0 7 0 0 10 17 Conservation, Department of 0 5 0 1 4 10 Consumer Affairs, Department of 6 114 0 0 10 130 Controller, State 0 0 6 1 1 8 Corporations, Department of 0 62 0 0 0 62 Corrections, Board of 0 6 0 0 0 6 Corrections, Department of 4 61 0 4 2 71 Criminal Justice Planning, Office of 0 17 0 0 1 18 Developmental Services, Department of 3 22 0 2 3 30 Education, Department of 0 13 4 1 2 20 Education, Office of the Secretary for 0 0 0 0 0 0 Education, State Board of 0 0 0 0 0 0 Emergency Medical Services Authority 0 4 0 0 1 5 Continued 92 Completed Projects Projects In Development Development Development Development Development Total Costs Greater Costs Less No Reported Costs Greater Costs Less Number of Than Than Development Than Than Reported Department $1 Million $1 Million Costs $1 Million $1 Million Projects Emergency Services, Office of 0 4 0 0 8 12 Employment Development Department 5 19 1 3 6 34 Employment Training Panel 0 9 0 0 0 9 Energy Resources, Conservation and 0 21 0 0 4 25 Development Commission Environmental Health Hazard 0 0 0 0 0 0 Assessment, Office of Environmental Protection 0 0 0 0 0 0 Agency, California Equalization, Board of 2 6 0 4 8 20 Exposition and State Fair, California 0 3 0 0 0 3 Fair Employment and Housing, 0 2 0 0 1 3 Department of Fair Political Practices Commission 0 0 0 0 2 2 Finance, Department of 0 12 16 0 2 30 Financial Institutions, Department of 0 0 0 0 0 0 Fish and Game, Department of 1 8 0 3 1 13 Food and Agriculture, Department of 0 125 0 0 26 151 Forestry and Fire Protection, 0 37 0 0 7 44 Department of Franchise Tax Board 14 30 0 5 9 58 General Services, Department of 3 114 0 3 14 134 Health and Human Services Agency 2 5 0 8 0 15 Data Center Health and Human Services 0 0 0 0 0 0 Agency, California Health Planning and Development, 0 15 0 1 1 17 Office of Statewide Health Services, Department of 7 180 1 7 36 231 Highway Patrol, California Department of 1 47 0 4 8 60 Horse Racing Board, California 0 1 0 0 1 2 Housing and Community Development, 1 15 0 0 0 16 Department of Housing Finance Agency, California 0 6 0 0 1 7 Industrial Relations, Department of 0 51 0 1 2 54 Inspector General, Office of the 0 0 0 0 1 1 Insurance Advisor, Office of 0 0 0 0 0 0 Insurance, Department of 1 1 0 0 13 15 Integrated Waste Management 0 22 0 0 0 22 Board, California Justice, Department of 19 11 0 5 2 37 Law Revision Commission, California 0 0 0 0 0 0 Continued 93 Completed Projects Projects In Development Development Development Development Development Total Costs Greater Costs Less No Reported Costs Greater Costs Less Number of Than Than Development Than Than Reported Department $1 Million $1 Million Costs $1 Million $1 Million Projects Library, California State 0 0 0 0 0 0 Lottery Commission, California State 4 18 0 0 5 27 Managed Health Care, Department of 0 11 0 0 6 17 Managed Risk Medical Insurance Board 0 3 0 0 0 3 Mandates, Commission on State 0 0 0 0 1 1 Medical Assistance Commission, California 0 2 0 0 0 2 Mental Health, Department of 0 44 0 0 0 44 Military Department-Office of the 1 20 0 0 0 21 Adjutant General Motor Vehicles, Department of 10 13 0 3 1 27 Narcotic Addict Evaluation Authority 0 0 0 0 0 0 Parks and Recreation, Department of 0 7 0 0 1 8 Peace Officer Standards and Training, 0 4 0 0 0 4 Commission on Personnel Administration, Department of 0 4 0 0 0 4 Personnel Board, State 0 15 0 1 1 17 Pesticide Regulation, Department of 0 20 0 1 1 22 Postsecondary Education 0 0 0 0 1 1 Commission, California Prison Terms, Board of 0 3 0 0 0 3 Public Employees’ Retirement System 7 84 13 9 0 113 Public Employment Relations Board 0 3 0 0 1 4 Public Utilities Commission 2 20 0 0 1 23 Real Estate Appraisers, Office of 0 0 0 0 0 0 Real Estate, Department of 0 0 0 1 1 2 Rehabilitation, Department of 1 25 0 1 0 27 Resources Agency 0 1 0 1 0 2 San Francisco Bay Conservation and 0 0 0 0 0 0 Development Commission Santa Monica Mountains Conservancy 0 0 0 0 0 0 Science Center, California 0 0 0 0 0 0 Secretary of State 6 1 0 0 0 7 Social Services, Department of 1 32 1 2 3 39 State and Consumer Services Agency 0 0 0 0 0 0 State Lands Commission 0 13 0 0 3 16 Stephen P. Teale Data Center 0 7 0 0 0 7 Student Aid Commission, California 1 2 0 1 1 5 Tahoe Conservancy, California 0 4 0 0 0 4 Continued 94 Completed Projects Projects In Development Development Development Development Development Total Costs Greater Costs Less No Reported Costs Greater Costs Less Number of Than Than Development Than Than Reported Department $1 Million $1 Million Costs $1 Million $1 Million Projects Teacher Credentialing, Commission on 0 28 0 1 0 29 Teachers’ Retirement System, State 1 0 0 2 0 3 Technology, Trade and Commerce 0 8 0 0 1 9 Agency, California Toxic Substances Control, Department of 0 19 0 0 6 25 Traffic Safety, Office of 0 2 0 0 6 8 Transportation, Department of 23 298 0 16 27 364 Treasurer, State 1 10 0 1 2 14 Unemployment Insurance Appeals 0 0 0 0 1 1 Board, California Veterans Affairs, Department of 2 8 0 2 11 23 Water Resources Control Board, State 1 7 0 2 0 10 Water Resources, Department of 3 78 0 1 1 83 Youth and Adult Correctional Agency 0 0 0 0 0 0 Youth Authority, Department of the 0 31 0 2 3 36 Youthful Offender Parole Board 0 7 0 0 3 10 State Totals 134 1,963 42* 101 281 2,521 * The departments reported that data are no longer available for these projects because they are older projects. 95 Blank page inserted for reproduction purposes only. 96 Agency’s comments provided as text only. Department of Information Technology 801 K Street, Suite 2100 Elias S. Cortez, Director Sacramento, CA 95814 June 13, 2001 Ms. Elaine M. Howle, State Auditor* Bureau of State Audits 555 Capitol Mall, Suite 300 Sacramento, California 95814 Dear Ms. Howle: The Department of Information Technology (DOlT) appreciates the opportunity to provide a response to your draft report dated June 7, 2001. These recommendations are consistent with DOlT’s direction in implementing the State of California’s Enterprise Portfolio Management Sys- tem (Information Technology Circle of Life) which addresses the full technology lifecycle from concept through implementation and on into maintenance and operations. DOlT has either 1 initiated or implemented these recommendations. This enterprise direction ensures that DOlT actively engages its customers throughout the entire project lifecycle, providing vigorous and ongoing oversight while sustaining collaborative efforts that stress the use of industry best practices, standard metrics, and concise yet rigorous documentation and performance measure- ment. 2 We are however disappointed with the BSA’s approach to the audit and the limited valuable recommendations for what was surely a costly and time-consuming review. Specifically, the BSA auditors tended to focus on reviewing policies, procedures, and documentation, mostly from previous administration practices, rather than assessing the value and results of DOlT’s current administration activities. Many of the BSA’s findings are based on the lack of artifacts. 2 We find little in the two chapters of the report we received that address or recommend any substantive reforms of the previous administration’s practices in the way the State manages its information technology investments, such as IT governance, which is a challenge for most government IT enterprises. To the contrary, one of the BSA’s recommendations is that DOlT should “. . . build upon past efforts to the extent possible rather than reinventing processes and practices when planning its future activities.” 3 In essence, the BSA is suggesting that the current administration build upon the flawed pro- cesses and practices of the previous administration. Unfortunately, it is these very practices and processes of the previous administration that failed to identify, prevent, or minimize the failure of the Statewide Automated Child Support Enforcement System (SACSS) in 1997 which was cited by the BSA as one of the State’s costly failures. This and other failures resulted in nearly $500 million of wasted taxpayer money. Surely returning to such practices would be counterproductive. * California State Auditor’s comments begin on page 141. 97 Furthermore, the BSA’s Analysis of Audit Request states that Assemblymember Wildman is “concerned about ongoing problems state departments are having implementing IT systems.” 4 The BSA appears to focus primarily on DOlT and does not address the IT governance challenge in managing the state’s information technology investments. The BSA’s focus on DOlT may lead one to the conclusion that all of the Administration’s challenges with information technology could be solved with more oversight. While we recognize the value of information technology project review, approval, and monitoring by an independent party like DOlT, we also recognize that such controls have their limitations. A more effective enterprise, cross-cutting approach to managing the state’s information technology investments will address both the departments directly respon- sible for managing those investments as well as the oversight organizations like DOlT, the De- partment of Finance, and the Department of General Services. Our disappointment with the results of the audit is amplified by the BSA’s focus on compliance 5 issues rather than effectiveness or outcomes. The auditors relied upon a typical compliance audit approach of comparing DOlT’s actions against the requirements of the law. While compliance is obviously important, it does not tell the whole story because it does not address current effective- ness. The audit report does not highlight how the progressive enterprise approach by DOlT is adding value and improving the state’s management of its investments in information technology. 6 The fact of the matter is that the BSA did not identify any failed, major IT projects that were approved or sponsored by this Administration. 7 In addition, the BSA did not address one important objective of audit regarding the constraints on sufficiency of resources. Specifically, objective #8 of the audit listed in the BSA’s June 27, 2000 Analysis of Audit Request states that the BSA audit would “evaluate the sufficiency of the re- sources allocated to DOlT (by the prior administration) to fulfill its statutory responsibilities.” The lack of sufficient resources has recently been a major issue for DOlT since its inception and has certainly contributed to some of the deficiencies noted by the BSA. To address this issue, the current Administration and Legislature allocated an additional 40 PYs for fiscal year 2000-01 and DOlT is just beginning to benefit from the additional resources. However, recruitment and reten- tion of staff to fill the authorized positions is impeded by the impending threat from the sunset date for DOlT and the general nature of the IT job market, an 18% factor in the State. The BSA acknowledges that recruitment and retention of staff is a severe problem for DOlT but 8 again fails to explore the cause of that problem. Many of the high caliber staff sought by DOlT is reluctant to join an organization that may be eliminated by the impending sunset date of DOlT which is June 30, 2002 according to current state law. Likewise, existing staff seek positions in other state organizations in advance of the sunset date. The BSA’s failure to analyze the cause of the problems they cite leaves the Legislature with little new information and virtually no recom- mendations regarding the types of reforms needed to improve the situation. The administration seeks to address this problem by repealing of the sunset provisions through AB 1559 (Diaz). 98 Finally, it is important to note that many of the instances in which the BSA finds that DOlT failed to develop artifact of its review of a project occurred in the midst of the Y2K readiness efforts. DOlT and this administration made a conscious decision to re-direct scarce DOlT resources from other responsibilities to focus on the higher priority and higher risk Y2K remediation efforts. For ex- ample, the Governor issued Executive Order D-3-99 which stated that the Year 2000 solutions shall be the State’s top technology priority. DOlT made resources available to the Y2K effort by spending less time producing artifact of reviews of projects. This redirection of resources was partially in response to concerns raised by the BSA in its July, 1998 report regarding Y2K. There- 9 fore, we find the BSA’s criticism of the lack of artifact of such reviews during the Y2K time period in conflict with what the Governor, DOlT, and BSA concluded was the highest IT priority for the State at the time. DOlT recognized many of the recommendations noted by the BSA before the BSA began its review and took steps to rectify the situation. Based upon the success of DOlT’s project manage- ment office (PMO) approach to the Y2K crisis, we began a series of initiatives that emphasize an enterprise approach to our responsibilities. For example, DOlT worked with the Department of General Services to encourage state and local agencies to negotiate an enterprise-wide software license thereby leveraging the State’s enormous buying power. These efforts resulted in esti- mated savings of tens of millions of dollars for state and county entities. Other enterprise initia- tives include Enterprise Program Management System (IT Circle of Life) and Enterprise Geo- graphical Information Systems (GIS). In summary, we agree with the BSA that a great deal of important work remains to reconstruct 0 previous administration’s flawed IT management infrastructure. As the BSA points out in its report, the current administration and DOlT are in the process of addressing these needs in a proactive and collaborative manner. More importantly, DOlT is focusing its business process re- engineering activities to ensure that DOlT is adding value and increasing the level of assurance that the state’s investment in information technology is properly managed. We appreciate the professionalism of the BSA’s staff and the opportunity to provide input to the audit report. If you have any questions or require any additional information, please call me at (916) 445-3050. Sincerely, (Signed by: Elias S. Cortez) ELIAS S. CORTEZ Director Attachments 99 Executive Summary Background “I believe that the Department of Information Technology has proven itself to be an indispensable leader of the State’s technology management strategy.” Governor Gray Davis The Davis Administration has raised the bar of accountability for information technology (IT) management to its highest levels to deliver quality service to all Californians. This Administration’s leadership has empowered the Department of Information Technology (DOIT) to ensure that all IT initiatives under its watch represent sound IT investments and utilize IT best practices to secure their successful implementation. 6 California has had zero major IT failures for projects initiated under the Davis Administration. As evidenced by the recent government IT report cards, we have raised California’s IT man- agement capabilities from below average during the prior Administration to above average during the Davis Administration by embracing industry best practices. We are committed to continue this upward trend. DOIT has established itself as the leader in government IT. More importantly, the Administra- tion has provided effective leadership and executive sponsorship to transform a dysfunctional, “stovepipe” IT program into an enterprise focused technology leader by investing precious resources to implement successful IT initiatives such as year 2000 remediation (Y2K), Elec- tronic Government (eGovernment), software management, and enterprise project manage- ment, which enhance our State programs and services. Over the past two years, DOIT has provided an exceptional return on investment at a very efficient rate of less than 1% of statewide IT expenditures. Contrary to BSA’s findings, the challenge that California government faces is NOT IT leader- q ship BUT governance. Most decentralized government entities also face this challenge. Very few have implemented successful governance models without going through the growing pains of addressing long term systemic IT management infrastructure issues. The Davis Administration’s enterprise approach to government IT coupled with DOIT’s Enterprise Portfolio Management initiative clearly exemplify the governance model that will successfully transform the previous command-and-control governance structure to a collaborative and tightly knit model. Such an approach dictates heavy emphasis on vetting, planning, risk management, and value effective IT investments that will maximize government funds through enterprise strategies rather than “stovepipe” solutions with inherently costly redundancies. California is confronted with an extremely complex and distributed IT organizational model. As the business and technology dynamics change, DOIT’s target is continuously moving to 100 accommodate such dynamics. To that end, DOIT relies heavily on statewide workgroups of Chief Information Officers as well as several intergovernmental and private sector councils to formulate its crosscutting IT leadership, guidance, and oversight capabilities. This approach has enabled the Davis Administration to navigate successfully through grave circumstances such as Y2K while also providing the flexibility to adapt to pioneering efforts such as eGovernment and Geographic Information Systems. Also, this approach provides the State with the strategic capabilities to confront and manage other major IT mandates such as the Health Insurance Portability and Accountability Act (HIPAA) that may require sweeping IT initiatives. The audit has raised some valid issues, which the Davis Administration has recognized upon taking office in 1999. The fragmented approach used by the prior Administration resulted in IT losses of over $500 million. The systemically flawed IT management infrastructure left behind by the prior Administration impacted every aspect of the then IT management life cycle from planning to maintenance and operation. Unfortunately, out of about 300 projects received by w DOIT annually, the majority of the 10 projects that the BSA examined were based on these systemically flawed processes. The statewide plan of 1997 developed by the prior Administration does require substantial revisions to clearly reflect and leverage the invaluable capabilities the Davis Administration introduced to California government during Y2K and eGovernment implementation. Upon successful completion of Y2K and eGovernnment, DOIT launched the Enterprise IT Strategic Planning program by forming a statewide Chief Information Officer workgroup focused on revamping the 1997 statewide strategic plan. This effort will result in enterprise planning capa- bilities that tightly couple the departmental plans with the statewide plan and will continuously refresh California government service delivery capabilities through technology integration. The revised statewide plan will reflect the State of California Enterprise IT needs and should be available during 2001-2002. In addition to launching the Enterprise IT Strategic Planning program, DOIT is implementing three other programs that are natural iterations of the DOIT Enterprise IT Portfolio Management initiative: (cid:226) IT Policies and Standards Program (cid:226) IT Project Submittal and Approval Program (cid:226) IT Oversight and Online Early Warning Program These programs are designed to transform the prior “stovepipe” nature of government IT in California to that of enterprise IT management capabilities, i.e. eGovernment, which will clearly address the systemic and procedural flaws of past IT practices. To completely revamp the systemic aspects of previous IT management practices, DOIT has launched four additional programs to complete the enterprise IT management life cycle from 101 planning to maintenance and operations and on into asset management. These programs include: (cid:226) IT Risk Management and Corrective Action Program (Special Project Report) (cid:226) IT Post Implementation Evaluation Program (cid:226) IT Maintenance and Operations Program (cid:226) IT Asset Management Program These Enterprise Programs launched by DOIT are targeted for statewide implementation during 2001-2002. The collaborative governance structure coupled with the comprehensive re- engineering of enterprise IT management based on Y2K and eGovernment successes and lessons learned will clearly position the Legislature to leverage the State’s IT capabilities as it considers broad reaching policies for legislative implementation. Summary of Recommendations and Responses The BSA’s recommendations are clearly in line with the Davis Administration and DOIT’s enterprise IT direction. As a result, DOIT prepared the following responses to the BSA recom- mendations: BSA Recommendation: To provide strategic guidance for the State’s IT activities, DOIT, in conjunction with the departments, the governor, the Legislature, the Department of Finance, and other relevant parties, needs to update the statewide IT plan to address the current IT environment. In particular, the plan should establish measurable objectives to show the State intends to reach its goals. Also, the State should consider establishing a methodology for prioritizing projects for approval and funding. DOIT Response Summary: In Progress. Updating the Statewide Strategic IT Plan represents a major element within the Enterprise IT Strategic Planning component of DOIT’s IT Circle of Life, which tracks the IT technology lifecycle from concept through implementation and on into maintenance and operations. In March 2001 DOIT convened the CIO Workgroup for Strategic Planning, launching the development of a new Statewide Strategic IT Plan. This plan reflects the State of California’s enterprise IT environment, leveraging the current Administration’s successes in Y2K and eGovernment using current and future IT opportunities and trends. It also focuses on measurable objectives that enable the State to gauge its implementation progress while serving to guide IT project approval and funding priorities. DOIT envisions that the State Strategic IT Plan will offer guidance to Department CIO’s as they partner with DOIT in developing technology strategies to support their department program needs. By incorporating those strategies into agency strategic business plans to ensure execu- tive sponsorship and then rolling them up to the statewide level, the State achieves a holistic alignment of strategic technology objectives with the Administration’s program priorities. 102 BSA Recommendation: To ensure departments’ IT strategic plans are consistently evaluated for their compliance with the statewide IT strategy, DOIT should implement a process to review department plans. DOIT Response Summary: In Progress. The Agency Information Management Strategy (AIMS) process represents a major element within the Enterprise IT Strategic Planning compo- nent of DOIT’s IT Circle of Life. In recognition of this fact, DOIT is reviewing existing Agency Information Management Strategy (AIMS) policy and procedures in conjunction with the state- wide strategic IT planning process. DOIT also has begun to pilot a new process for reviewing department AIMS. Using existing policy as a baseline, this process targets specific enterprise crosscutting issues, such as eGovernment, to guide the various departments’ strategic IT initia- tives. DOIT plans to use feedback from this pilot to reengineer current AIMS policy and proce- dures. This major reengineering effort focuses on partnering Department CIO’s with DOIT in jointly preparing technology strategies to support statewide and department program needs. By incorporating those strategies into agency strategic business plans to ensure executive sponsor- ship and then rolling them up to the statewide level, the State achieves a holistic alignment of strategic technology objectives with the Administration’s program priorities. BSA Recommendation: To provide appropriate department guidance and direction about the IT development process, DOIT should consolidate the various sources of policy and guidance, remove outdated policies from published documents, and revise policies as needed to reflect changing state needs. In addition, DOIT should resolve the contradiction between its management memorandum and the State Administrative Manual over the applicability of the alternative procurement process. DOIT Response Summary: In Progress. State IT Policies, Processes and Standards represent a primary component within DOIT’s IT Circle of Life, supporting all technology lifecycle phases from concept through implementation and on into maintenance and operations. Therefore, to establish understanding and buy-in from a diverse set of stakeholders, DOIT employs a “best practices” collaborative approach, using a series of CIO subcommittees, to vet proposed policies and procedures before formally publishing them. DOIT chooses this approach owing to the complexities of California IT governance and the fact that IT policies and practices cannot be “force fed”. Once proposed policies and procedures are appropriately vetted and approved, each CIO subcommittee is responsible for consolidating Management Memo and State Adminis- tration Manual (SAM) content for its assigned IT governance area. This approach will resolve any misunderstanding by eliminating outdated references and apparent contradiction. 103 BSA Recommendation: DOIT should continue its efforts to improve its project review and approval process. However, it should ensure that the changes result in a process that will subject proposed IT projects to a thorough evaluation. Further, DOIT should ensure that departments are properly assessing IT projects by comparing departments’ feasibility study reports with established criteria, such as the fundamental decision criteria. Moreover, to ensure that it can defend its approval of costly IT projects, DOIT should thoroughly document its approval decisions. DOIT Response Summary: Initial Reengineering Done, Pilot in Progress. IT Project Submittal and Approval comprises a major component within DOIT’s IT Circle of Life. As a result, DOIT has taken the leadership role in partnering with its sister control agencies and other department representatives to develop a new, web-enabled IT Project Submittal and Approval process. DOIT goals include streamlining project review and approval while at the same time adding greater rigor to the review process. Based on industry standards and best practices, as acknowledged by as many as 56 CIO’s, the new IT Project Submittal and Approval process requires that IT project proposals reflect the nine Project Management Institute (PMI) project management knowledge areas used to plan and execute successful IT projects. The process also requires agencies to provide specific program- matic information to ensure that business objectives are clearly understood and addressed by the IT project proposal. DOIT is currently piloting its reengineered IT Project Submittal and Approval process with support from ten departments. Project information provided during the IT Project Submittal and Approval process will be used to populate DOIT’s Enterprise Portfolio Management System (EPMS), which serves as the central repository for all IT Circle of Life components. The redesigned IT Project Submittal and Approval process provides greater focus on quality planning and sets the stage for ongoing project over- sight. It also provides the foundation for comprehensive enterprise-wide IT asset management by supporting the consistency and integrity of the State’s IT asset inventory. BSA Recommendation: To ensure departments assess and mitigate project risks, DOIT should require complete risk assessment reports from departments. Further, DOIT should properly analyze the responses and document how it resolved any concerns. Finally, DOIT should require departments using the alternative procurement process to assess risks at the beginning of the project. If DOIT believes its current model is inap- propriate for alternative procurements, it should modify its new risk model to more appropriately address alternative procurement. DOIT Response Summary: Already in Progress. IT Project Submittal and Approval as well as IT Project Oversight comprise two key components of DOIT’s IT Circle of Life. The IT Project Submittal and Approval process focuses on the importance and need to fully understand and address risk mitigation issues, whereas IT Project Oversight demands effective and ongoing risk management. DOIT’s metrics driven IT project review and approval requires full risk assess- 104 ment reports to accompany all department IT project proposals during the submittal process. DOIT is in the process of assessing its current risk assessment model (RAM) to ensure it reflects industry best practices by replacing existing subjective evaluations with more objective metrics. DOIT has engaged an industry leader to assist in developing a comprehensive set of project oversight and risk assessment metrics. Using this material as a base, as well as lessons learned during Y2K, DOIT plans to implement a revised RAM as part of the IT Project Submittal and Approval Process. Once the new model is in place, agencies will be required to provide dynamic and ongoing risk assessment and management by updating the RAM during the various stages of project development and implementation. BSA Recommendation: To ensure that it receives and effectively uses the proper infor- mation to monitor departments’ IT projects, DOIT should take the following actions: (cid:226) Continue with its efforts to restructure the oversight process to ensure that the process allows DOIT to properly monitor and guide projects (cid:226) Modify the required progress reports to include a comparison of two types of critical information: the project’s monthly actual costs and revised estimates of total projected costs with the budget, and actual and revised projected completion dates for project phases compared with the original schedule. Use this modified progress reporting to closely monitor projects that may be required to submit special project reports. (cid:226) Ensure that analysts sufficiently review and document their oversight of projects and track the receipt of required reports. 1 DOIT Response Summary: Done, On-going. IT Project Oversight and the Special Project Report represent two critical components of DOIT’s IT Circle of Life. The reengineering of DOIT’s IT project oversight process and tools includes implementing a web-enabled Early Warning System that displays crucial project metrics, including key cost and schedule data. Developed by industry leaders, these metrics are based on project management best practices, such as the red/yellow/green “traffic light indicators” so effectively used during Y2K, and allow agencies to report project status regularly while offering DOIT the ability to assess project health. These metrics also provide an objective link to the Special Project Report process, by which significant changes to department IT projects are tracked and approved. DOIT and departments also partner to select and contract with Independent Project Oversight Consultants, who provide independent and objective project performance assessments. BSA Recommendation: To hold departments accountable for the benefits expected from their IT projects, DOIT should ensure that departments submit post implementa- tion evaluation reports. Further, DOIT should develop a process to promptly review and evaluate the reports submitted. 105 DOIT Response Summary: In Progress. The Post Implementation Evaluation Report (PIER) process represents a significant component within the IT Circle of Life. DOIT currently requires state agencies to submit a PIER following any reportable project implementation. With its re- e cently acquired staff increase, DOIT is able to review all project PIER’s submitted by those agencies. DOIT is revisiting the current PIER process to determine the extent changes are required to provide more rigorous capture of lessons learned and to build upon best practices, which can be shared in DOIT newsletters as well as such venues as the Executive Leadership Forums and CIO biweekly meetings. With the implementation of DOIT’s Enterprise Portfolio Management System (EPMS), much of the information to be provided within the PIER will already exist in automated form through previous project planning and oversight processes. DOIT’s primary goals for the PIER process include updating the State’s IT asset inventory, feeding objective project performance information into the statewide policy planning process and showcasing best practices through DOIT’s various communication channels. BSA Recommendation: To promote coordination on IT projects and avoid redundant efforts, DOIT should establish a formal mechanism to initiate discussions between departments that are developing projects based on similar technologies or processes. To facilitate this coordination and improve project oversight, DOIT should complete its IT project inventory based on its survey of departments. DOIT should ensure that de- partments’ reported data is accurate and update this information when departments report new information so that the project inventory can be current. DOIT also needs to consider how departments and the Legislature can effectively access this information, taking into consideration privacy issues and other concerns that may limit the release of this information. DOIT Response Summary: In Progress, Priority. DOIT’s Enterprise Portfolio Management System (EPMS) facilitates the capture of relevant IT project information from concept through 1 implementation and on into maintenance. This web-enabled central repository represents the State of California IT investment portfolio and IT Asset Management strategy. DOIT is actively reengineering most of the components in the IT Circle of Life. As each component is imple- mented, DOIT, agencies and the Legislature will gain access to timely and accurate IT project and IT asset information, which can be leveraged in making informed IT decisions. DOIT’s new Maintenance and Operations policies and guidelines have been drafted and are currently entering their review cycle. DOIT bases its Maintenance and Operations program on experiences drawn from pilot implementations conducted with selected departments over the past three years. DOIT anticipates providing guidance to departments in extending objective performance measures from existing project oversight processes on into the Maintenance and Operations phase. This practice allows the State to gain more insight into improving program performance, optimizing existing resources and business processes, and providing feedback into the ongoing IT planning processes. 106 DOIT’s current IT baseline inventory forms the foundation for statewide IT Asset Management. It also provides comprehensive baseline information in support of statewide strategic IT plan- ning. DOIT’s Enterprise Portfolio Management System is designed to support automated and ongoing refresh of critical Asset Management information. BSA Recommendation: To improve compatibility and properly guide IT development, DOIT should expedite its efforts to implement standards by determining which stan- dards need to be addressed first and focus their efforts accordingly. Further, DOIT should work with departments to ensure that all necessary standards have been imple- mented. r DOIT Response Summary: Done, On-going. As stated earlier, State IT Policies, Processes and Standards comprise a major component within the IT Circle of Life. As a result, DOIT’s process to develop, document and review State IT standards involves full participation of state and external stakeholders. In addition, DOIT captures lessons learned via the PIER review process to provide additional standards input. Though time-consuming, this participative process ensures the successful adoption of full security and infrastructure standards as well as prelimi- nary drafts of accessibility and eGovernment applications standards. DOIT has also adopted internal processes and standards to notify agencies when their project proposals involve activi- ties subject to published DOIT standards. BSA Recommendation: To ensure that DOIT is fully employing the IT advisory councils and receiving the benefits intended by law, DOIT should continue to meet with the private commission and the public committee on a regular basis to guide its strategic planning efforts, provide input on new policies and ensure that the State follows best practices. DOIT should ensure the public committee makes all findings and recommen- dations in writing, as required by state law. DOIT should also monitor the progress of its CIO work groups to ensure that they reach their established goals. DOIT Response Summary: Done, Ongoing. DOIT meets with the private commission and the public committee on a regular basis to guide its strategic planning efforts, receive input on new policies, and ensure that the State utilizes industry best practices and processes. All the t findings and recommendations from these two advisory councils are documented. BSA Recommendation: To ensure that it completes initiatives, DOIT should establish timelines and goals for meeting future initiatives. If DOIT does not believe it can com- plete initiatives within established guidelines, it should communicate its priorities and resource requirements to the Legislature. In addition, it should notify the Legislature when changes in the State’s IT environment prompt adjustments to these priorities or resource requirements. 107 DOIT Response Summary: Done, Ongoing. DOIT, with support from legislative staff, has developed a realistic timeline for policy development and publication. DOIT will report its progress to the appropriate legislative staff to meet those commitments and will promptly notify the Legislature if DOIT needs to readjust priorities or resource requirements due to changing circumstances. DOIT has also offered to meet with the Legislative Analyst Office and legisla- tive staff, as needed, to keep them engaged. BSA Recommendation: To organize and focus its efforts, DOIT should adopt an internal strategic plan to identify key responsibilities and establish priorities. This plan should clearly describe how the organization would address its many responsibilities, particu- larly those that we observed it has not sufficiently accomplished. Further, it should build on past efforts to the extent possible rather than reinventing processes and prac- tices when planning its future activities. y DOIT Response Summary: Done, Updates Ongoing. DOIT has always operated from an internal strategic plan. However, as of January 2001 DOIT began drafting a new internal Strate- gic Business Plan, establishing priorities and key responsibilities to “operationalize” DOIT’s IT Circle of Life and supporting Enterprise Portfolio Management System. This plan offers a com- prehensive approach to organizing and addressing DOIT’s multi-faceted responsibilities. How- ever, rather than building on fragmented and archaic processes and practices of the prior Admin- istration, DOIT’s current reengineering efforts undertake a more comprehensive and consistent approach on enterprise crosscutting issues that span the entire technology lifecycle. Conclusion Although the BSA recommendations are in line with the Davis Administration and DOIT’s direction, the BSA’s shift from the original scope of examining the “state’s” IT capabilities to that of DOIT’s statutory and documentary compliance has greatly diffused the original audit intent. 5 The opportunity to provide an independent perspective about the previous Administration’s flawed IT practices would undoutedly add value to the Davis Administration’s enterprise IT initiatives. Although the Davis Administration and DOIT inherited flawed practices, they managed to successfully complete several enterprise IT initiatives, i.e., Y2K and eGovernment, while delivering on many other statewide IT solutions without failures. DOIT’s emerging direction is to focus on strategic and systemic realignment of IT management infrastructure by establishing successful programs that streamline IT projects and the relative government services delivery. Our goal is to raise the bar of accountability for IT project successes, enhance State government’s ability to exceed the quality expectations of its constituents, and provide solu- tions that position California for strategic and systemic IT governance. 108 Department of Information Technology Response to the Bureau of State Audits Report “Information Technology: The State Needs to Improve the Leadership and Management of Its Information Technology Efforts” Introduction DOIT’s mandated responsibilities, as articulated in AB 1686, encompass a daunting task of extraordinary scale and complexity. The State of California invests anywhere from $2 billion to $4 billion annually in information technology. Both the number and complexity of State IT projects are increasing. Managing those projects as separate, independent projects (i.e., in a stovepipe fashion), is no longer effective to succeed in today’s business environment. More- over, the fragmented approach exercised by the prior Administration has resulted in numerous and substantial inefficiencies for the State, including: (cid:127) Previous large-scale IT failures (cid:127) Redundancies and duplicated costs in data generation and processing (cid:127) Cross-agency and department disconnects at the functional and operating levels (cid:127) An inability to share and leverage investments in information systems and data due to lack of open standards (cid:127) A dampened ability to develop and deploy world-class automation to better serve the citizens and businesses throughout California Since DOIT was first established, an even greater need has arisen for IT leadership and oversight in State government. Today, the quantity and complexity of IT projects have grown, security and privacy issues have captured public attention, and the Internet has become ubiquitous. Most importantly for the State, eGovernment has emerged from its infancy as a catalyst for enabling technology that is changing how government programs will work and provide services in the future. Today, the State of California commits billions of dollars annually to IT project implementation as well as ongoing systems and infrastructure support. DOIT has taken an active leadership role in promoting an enterprise IT framework, including architecture, standards and policies, which upholds the State’s strategic direction for IT implementation and management. The result is cost savings and avoidance amounting to millions of dollars through u architectural consistency and economies of scale. In parallel to its leadership function, DOIT has enhanced its independent oversight framework to ensure State agencies adopt the poli- cies, architecture, standards and practices needed for successful planning and implementation 109 of IT projects. In addition to ensuring that taxpayer money is well spent, DOIT also meets the 6 challenge that State agencies have no failed projects. By continuously focusing its oversight capabilities on State IT projects, DOIT generates significant savings to the State through prevention or restoration of struggling IT projects. Borrowing successful methodologies from the private sector, in addition to the body of knowledge related to IT industry best practices, DOIT has redefined itself, its operating model and vision. Perhaps more importantly, DOIT Executives have crafted an IT model for the State that has revolutionized the manner in which services are provided to other state departments and ultimately to the citizens of this State. Until recently, DOIT’s primary organizational challenge has been one of inadequate staffing. DOIT was established in 1995 with a total of 15 positions to address the overwhelming dilemma of the State’s IT problems. Now, DOIT has a total staff of nearly 80 positions to support a customer base that if treated as a separate country would represent one of the largest econo- mies in the world. The fact is that, until recently under the current Administration, DOIT did not have the resources to adequately address the issues required in its authorizing legislation. The recent Y2K triumph represented an unprecedented achievement in the history of the State of California, yet it generated an unexpected yet substantial by-product for DOIT and the future of the state’s information systems. It also served as the catalyst for a dramatic departure from the dispersed model used by the previous DOIT by positioning and elevating the new DOIT to deliver quick, internationally recognized, successful initiatives such as eGovernment, Geo- graphic Information Systems, Enterprise Program Management and effective IT energy man- agement among others. The Y2K transition was unique in that it unified program managers and CIO’s alike around a potential disaster of unprecedented proportions. Yet, borrowing exten- sively from private sector best practices in establishing an Enterprise Portfolio Management approach, DOIT’s leadership adopted project management best practices while achieving executive sponsorship at a level previously unattained. Seizing upon the momentum generated by the Y2K activity and capitalizing on the best prac- tices that led to its success, DOIT utilized its positive project management methodology to enhance the management of new IT initiatives that were focused on enhancing state program services delivery to its customers. DOIT has incorporated this approach into today’s Enterprise 1 Portfolio Management strategy, which provides the framework for DOIT’s leadership and oversight functions. By transforming best practices into integrated policies that mirror the full technology lifecycle, DOIT has expanded upon its comprehensive vetting practices and broad communication channels to maintain stakeholder buy-in and ongoing ownership. DOIT has raised the bar on IT project oversight and is reforming the framework that all technology projects will utilize for ensure their successful implementation. This methodology’s development originated from DOIT’s comprehensive and progressive project oversight on the first enterprise-level eGovernment initiative, the Department of Motor Vehicles “DMV Online” project. In fulfilling its leadership role, DOIT elevated the approach for 110 managing complex, crosscutting technology efforts to support this Administration’s efforts in transforming government services to benefit the people. The best practices stemming from this enterprise effort enhanced the capability of DOIT’s leadership and oversight roles by enabling DOIT to expand the support gained from the Y2K Business Council to further hone its project tracking and monitoring program. Moreover, DOIT engaged industry leaders in oversight and project management best practices to reinforce its metrics, performance measurement prac- tices and project management methodology. Oversight best practices based on current and factual data allow DOIT as an oversight entity to recognize and interpret the dynamics underly- ing IT project behavior. Such insight contributes toward sound project management and, when necessary, focused corrective action based on facts rather than guesswork. This Administration has engaged DOIT in its priority eGovernment initiatives. DOIT has fol- lowed the Administration’s vision and direction by strategically positioning itself to ensure successful eGovernment initiatives even as it began its own comprehensive realignment of internal resources from the previous stovepipe practices to a progressive and proactive leader in IT management. i Governing Magazine duly noted these recent successes by stating, “. . . the state is doing reasonably well. Statewide IT planning efforts are in place, and most of the larger agencies do a good job, with a great deal of input from stakeholders. Major project problems in the past have led to the development of a good project management system, which puts reasonable constrains on cost and time overruns. Post-implementation audits are done to make sure projects deliver their promised benefits.” In addition, Governing.com’s “Grading the States 2001: A Management Report Card” recognized DOIT’s achievements by upgrading its performance rating to a B-. Today, with competition from the private sector, the State faces a technology staff recruitment and retention challenge that is inching upward toward twenty percent. When combined with the deleterious impact of the existing sunset clause, this statistic casts an ominous shadow over DOIT’s leadership and oversight program. Nevertheless, this Administration has continued to offer its vision and executive sponsorship, not only by investing in DOIT but also by engaging DOIT in a leadership role in all of its successful IT ventures. A number of compelling reasons exist for DOIT to realign itself from the previous Administration’s fragmented practices by taking an enterprise-wide, full life cycle approach in its oversight activities. They include the following: (cid:127) Ensure fully integrated enterprise solutions instead of “stovepipe” technology (cid:127) Attain synergy through engaging, collaborating with other departments (cid:127) Substantiate that all project management methodology ingredients stem from best practices in the private sector and other government arenas (cid:127) Verify that needs are program driven while technology serves as an enabler 111 (cid:127) Maintain continuous and rigorous oversight from thought through implementation and beyond (cid:127) Assign accountability to the program owner in providing executive sponsorship to ensure successful IT initiatives (cid:127) Make sure all stakeholders are at the table to ensure quality solutions are derived through comprehensive vetting processes (cid:127) Verify that technology solutions are developed in manageable phases and achieve success through sound planning (cid:127) Integrate technology platforms as a key strategy for solutions development rather than continuing to build islands of information (cid:127) Utilize a project management methodology that employs objective metrics and measure- ments to ensure successful IT solutions 1 These objectives comprise critical success factors for DOIT’s Enterprise Portfolio Manage- ment program. The central theme underlying DOIT’s strategic vision is the successful imple- mentation of an Enterprise Portfolio Management program that spans the entire statewide technology lifecycle, more commonly referred to as the IT Circle of Life. As mandated by Senate Bill 1 (Chapter 508, Statutes of 1995) and Assembly Bill 1686 (Chapter 873, Statutes of o 1999), DOIT has established, through its IT Baseline Survey, a statewide inventory containing IT project and infrastructure information across all State agencies. This Asset Management inventory anchors the starting point of the technology lifecycle and establishes the baseline against which statewide IT advancement is measured, as directed by the Statewide IT Strate- gic Plan and Agency Information Management Strategies (AIMS), and guided by DOIT insti- tuted IT policies and standards. Because the state advances technologically through discrete and manageable projects, the next step confirms that projects are well conceived, planned and organized through the project initiation and approval process. Once viable projects have been set in motion, DOIT focuses on ensuring their successful completion through effective project oversight, as supported by established project management best practices and rigorous performance metrics. When necessary, corrective measures are reported, approved and applied through the Special Project Report process to ensure impacted projects remain on track. Finally, each project is closed out through the Post Implementation Evaluation Report process, and the implemented technology enters into the Maintenance and Operation phase, thus becoming woven into the fabric of the IT Asset Management framework and completing the full IT Circle of Life. DOIT’s leadership role promotes an enterprise-wide Information Technology (IT) framework that includes reengineering existing policy, processes and standards that impact and shape the management of the State’s IT resources. The IT Circle of Life offers the framework for program performance measurement, which in turn provides the basis for continuous improvement by ensuring that all State IT policies, processes, procedures and standards keep pace with changing technologies while continuing to focus on service delivery. 112 Each IT Circle of Life component collects specific information into a central enterprise reposi- tory where it can be leveraged by the other components. Partnering with state agencies, control agencies and the private sector, DOIT is actively evaluating and reengineering each component using industry best practices and tools to capture information related to IT projects and assets, as follows: (cid:127) Enterprise IT Strategic Planning/Agency Information Management Strategy. DOIT and the CIO Strategic Planning Workgroup have begun the efforts needed to engage and guide a consultant in developing a Statewide IT Strategic Plan and reengineering the State’s existing Agency Information Management Strategy (AIMS) policies and procedures to ensure accountability. This major reengineering effort focuses on DOIT and Department CIOs’ joint preparation of technology strategies to support statewide and department program needs. By incorporating those strategies into agency strategic business plans to ensure executive sponsorship and rolling them up to the statewide level, the State achieves a holistic alignment of strategic technology objectives with the Administration’s program and policy priorities. (cid:127) State IT Policies, Processes and Standards. DOIT and the Enterprise Advisory Council (formerly known as the CIO Advisory Council) meet biweekly to discuss proposed policies, processes and standards before finalizing them. Owing to the complexities of California IT governance and the fact that sound IT policies and practices cannot be “force fed”, DOIT’s comprehensive vetting process ensures not only the development of sound policy and procedures but also full buy-in from this diverse set of stakeholders. (cid:127) IT Project Submittal and Approval. Serving as the linchpin in the IT Circle of Life, DOIT’s reengineering of the IT project submittal and approval process began almost fifteen months ago and is now being piloted among ten key departments. DOIT’s objectives are to stream- line project review and approval while at the same time adding greater rigor to the review process. Through DOIT’s positive and proactive project management methodology, the redesigned project submittal and approval process not only provides greater focus on quality planning but also sets the stage for sound, ongoing project oversight. In addition, the process provides the foundation for comprehensive enterprise-wide IT asset management by supporting the consistency and integrity of the State’s IT asset inventory. (cid:127) Oversight/Early Warning System. Within DOIT’s Enterprise Portfolio Management strat- egy, the project oversight function forms the basis for DOIT’s web-based early warning system by providing performance-based project status (metrics). This system uses the same “traffic light” indicators that proved effective during the Statewide Y2K readiness effort. In addition, DOIT has begun evaluating a web-enabled tool known as the “Project Health Indicator”. Also known as the “Project Dashboard”, this tool transforms regularly updated project oversight metrics into meaningful performance information that will enable DOIT to provide more effective guidance to problem projects before they have a chance to get out of control. DOIT’s primary focus is to ensure that project oversight is grounded in factual project performance data as opposed to the more subjective project status reporting. 113 (cid:127) Special Project Report. Using regularly updated project oversight metrics, DOIT plans to link the Special Project Report process, by which DOIT tracks and approves significant changes to department IT projects, directly to the Oversight/Early Warning System. Such an approach introduces objectivity into the change management process by basing project planning enhancements or modification on firm, factual data. Even so, DOIT anticipates that given the emphasis on up front quality planning and oversight, the future trend will be fewer SPR’s. (cid:127) Post Implementation and Evaluation Report (PIER). DOIT is revisiting the current process to determine the extent changes are required to capture lessons learned and build upon best practices. With the implementation of DOIT’s Enterprise Portfolio Management System, much of the information to be provided within the PIER will already exist in auto- mated form through previous project planning and oversight. DOIT’s primary goal of the PIER process becomes one of feeding objective project performance information into the statewide policy planning process and showcasing best practices through newsletters and such venues as the Executive Leadership Forum. Project financial and programmatic outcomes will continue to be reviewed by the Department of Finance. (cid:127) Maintenance and Operations. The new Maintenance and Operations policies and guide- lines have been drafted and are currently starting their review cycle. The DOIT Mainte- nance and Operations program is based on experiences drawn from pilot implementations with selected departments conducted over the past three years. DOIT expects to provide guidance to departments in extending objective performance measures from existing project oversight processes on into the Maintenance and Operations phase. This practice allows the State to gain more insight into improving program performance as well as opti- mizing existing resources and business processes, thus providing feedback into the ongoing IT planning process. o (cid:127) Asset Management. The current IT baseline inventory is the foundation for Asset Manage- ment and provides comprehensive baseline information that supports strategic IT planning. The Enterprise Portfolio Management System provides the basis for automated and ongo- ing updates to critical Asset Management information. As each IT Circle of Life component is implemented, the State of California improves its ability to manage its IT investments at an enterprise level. Some of the benefits include: (cid:127) Incorporation of industry best practices into DOIT’s leadership, guidance and oversight activities. Such best practices include project management, performance metrics, lessons learned, and enterprise architecture and standards. (cid:127) Sharing of enterprise-level information to help state officials make better informed IT deci- sions (cid:127) Clearer and more objective IT management performance indicators 114 (cid:127) Central on-line capability to track IT projects from concept through implementation and on into maintenance (cid:127) Ability to save millions of taxpayer dollars in implementing IT solutions by reducing licensing costs, leveraging IT training, sharing project management knowledge, reducing outdated technology costs, etc. (cid:127) Centralized enterprise database containing all IT project costs and other historical informa- tion (cid:127) Ability to track departments’ compliance with State policy, process and procedures (cid:127) Ability to collect project information for all delegated projects (cid:127) Availability of a single information source for control agencies to review and approve IT project proposals. (cid:127) Statewide adherence to standard acquisition and procurement processes (cid:127) Ability to ensure cost-effective, efficient, and reliable processes are deployed in a timely manner, thereby saving departments time and resources presently expended on outdated processes (cid:127) Capability to centrally monitor IT project status to ensure IT project success The following represent major recent accomplishments by DOIT: (cid:1) The Department of Information Technology (DOIT) led California through a successful Year 2000 (Y2K) transition. (cid:1) In its oversight role DOIT has published new policy on IT Project Oversight. (cid:1) Initiated the reengineering of the Feasibility Study Report Process. (cid:1) Piloted the Project Registration System with 13 departments to identify the level of detail information needed to adequately track IT projects. (cid:1) Engaged a leading consultant firm and established a workgroup of State IT professionals to develop IT project management metrics to accurately gauge achieved and sustained project performance. (cid:1) Developed an IT Baseline Survey to gather the most current information on all IT projects and on-going maintenance efforts. (cid:1) Established a partnership with the Project Management Institute and State project manage- ment professionals to adopt the Project Management Body of Knowledge (PMBOK) as a guideline on IT project management. (cid:1) Developed a process for standards development and maintenance for the Statewide Technical Architecture and Standards, and drafted comprehensive standards for eGovernment systems. (cid:1) Established an Acquisition Policy and Oversight Division within DOIT. (cid:1) Established a formal partnership with the Department of General Services’ Procurement Division to identify acquisition policy and process improvements. 115 (cid:1) Initiated review of opportunities for consolidated purchase of software licenses to reduce costs to the State. (cid:1) Developed an Operational Recovery Program to provide guidance to State agencies on how to prepare for and respond to the damage or destruction of critical IT systems. (cid:1) Sponsored a Geographical Information Systems forum to raise awareness of how this technology can be used to more effectively deliver programs. (cid:1) Established a Chief Information Officer (CIO) Advisory Council to foster Statewide collabo- ration, provide advice on technology directions and assist DOIT with its ‘go forward’ strat- egy. (cid:1) Established a Strategic Planning Division within DOIT to develop, implement and maintain a comprehensive program to govern and facilitate IT planning. (cid:1) Collaborated with DPA and established a task force to develop and implement solutions to the longstanding IT recruitment and retention issues. (cid:1) Established a Statewide IT Recruitment and Retention Section within DOIT to address the recruitment and retention of IT professionals on a Statewide basis. (cid:1) Initiated bi-weekly CIO Meetings. Through these meetings DOIT has fostered Statewide strategic planning, enterprise solutions and change management among State IT and business professionals. (cid:1) Co-sponsored technology leadership events (i.e. the Executive Leadership Forums, Quar- terly Supplier Forums, Information Security Forums, and Energy Forums). These forums have successfully contributed to improved private-public cooperation, understanding of emerging technologies, and communication among all technology stakeholders. 116 Summary of Recommendations and Responses The BSA’s recommendations are clearly in line with the Davis Administration and DOIT’s enterprise IT direction. As a result, DOIT prepare the following responses to the BSA recom- mendations: BSA Recommendation: To provide strategic guidance for the State’s IT activities, DOIT, in conjunction with the departments, the governor, the Legislature, the Department of Finance, and other relevant parties, needs to update the statewide IT plan to address the current IT environment. In particular, the plan should establish measurable objectives to show the State intends to reach its goals. Also, the State should consider establishing a methodology for prioritizing projects for approval and funding. DOIT Response Summary: In Progress. Updating the Statewide Strategic IT Plan represents a major element within the Enterprise IT Strategic Planning component of DOIT’s IT Circle of Life, which tracks the IT technology lifecycle from concept through implementation and on into maintenance and operations. In March 2001 DOIT convened the CIO Workgroup for Strategic Planning, launching the development of a new Statewide Strategic IT Plan. This plan reflects the State of California’s enterprise IT environment, leveraging the current Administration’s successes in Y2K and eGovernment using current and future IT opportunities and trends. It also focuses on measurable objectives that enable the State to gauge its implementation progress while serving to guide IT project approval and funding priorities. DOIT envisions that the State Strategic IT Plan will offer guidance to Department CIO’s as they partner with DOIT in developing technology strategies to support their department program needs. By incorporating those strategies into agency strategic business plans to ensure executive sponsorship and then rolling them up to the statewide level, the State achieves a holistic alignment of strategic technology objectives with the Administration’s program priorities. Detailed Response: In March 2001 DOIT organized and convened the CIO Workgroup for Strategic Planning and began creating the new Statewide Strategic IT Plan. Meeting weekly, this key workgroup created a project charter and drafted a Statement of Work to acquire specialized contractor services to develop the Statewide Strategic IT Plan and publish a Statewide Strategic IT Planning methodology. Reflecting the State of California’s enterprise IT environment, this plan leverages the current Administration’s successes in Y2K and eGovernment. The scope of work covers recent technology advances and crosscutting enterprise strategic issues, such as eGovernment, Geographic Information Systems (GIS) and the Health Improvement Portability and Account- ability Act (HIPAA). It also addresses statewide priorities for technology implementation. In addition to documenting the current IT environment as well as technology advances, issues and standards, this plan establishes measurable objectives that enable the State to gauge its progress in key areas. At the same time, it serves as a guide for prioritizing IT project approval and funding. 117 Moreover, DOIT expects to work closely with the Governor, the Legislature, the Department of Finance and the Office of EGovernment to ensure technology opportunities support their collaborative efforts in defining a statewide strategic vision for government programs and services. Serving as the appropriate driver for the Statewide Strategic IT Plan, this vision addresses three key elements: 1) statewide business process reengineering; 2) eGovernment blueprint and priorities; and, 3) major program and policy issues. These elements are essential in enabling DOIT to create a comprehensive Statewide IT Strategic Plan that supports this Administration’s priorities. Such a statewide strategic planning framework also enables the State to: (cid:127) Define statewide strategic issues (cid:127) Establish measurable strategic performance objectives (cid:127) Provide the basis to prioritize projects at the state level (cid:127) Serve as a yardstick when reviewing agency strategic plans (cid:127) Identify and address crosscutting enterprise issues To supplement the statewide strategic planning framework, the Strategic Planning Workgroup is using the strategic direction set forth in Governor Davis’ 2001 “State of the State Address.” Additionally, the team works closely with the Governor’s Office on Innovation in Government and the Director of eGovernment in reviewing and synthesizing material from individual agency Strategic Business Plans to extract common themes and crosscutting issues supported by technology advances. Meanwhile, Project Review and Oversight staff scrutinize project proposal and planning efforts to ensure they address current crosscutting enterprise issues and initiatives, such as eGovernment, GIS and HIPAA. Similarly, Technology and Infrastructure staff review project planning documents to ensure the proposed technology infrastructure considers critical IT issues and advances. BSA Recommendation: To ensure departments’ IT strategic plans are consistently evaluated for their compliance with the statewide IT strategy, DOIT should implement a process to review department plans. DOIT Response Summary: In Progress. The Agency Information Management Strategy (AIMS) process represents a major element within the Enterprise IT Strategic Planning compo- nent of DOIT’s IT Circle of Life. In recognition of this fact, DOIT is reviewing existing Agency Information Management Strategy policy and procedures in conjunction with the statewide strategic IT planning process. DOIT also has begun to pilot a new process for reviewing depart- ment AIMS. Using existing policy as a baseline, this process targets specific enterprise crosscut- ting issues, such as eGovernment and the Health Insurance Portability and Accountability Act (HIPAA), to guide the various departments’ strategic IT initiatives. DOIT plans to use feedback from this pilot to reengineer current AIMS policy and procedures. This major reengineering effort 118 focuses on partnering Department CIO’s with DOIT in jointly preparing technology strategies to support statewide and department program needs. By incorporating those strategies into agency strategic business plans to ensure executive sponsorship and then rolling them up to the statewide level, the State achieves a holistic alignment of strategic technology objectives with the Administration’s program priorities. Detailed Response: In January 2001, DOIT began reviewing existing AIMS policy and proce- dures in conjunction with the statewide strategic IT planning process to ensure they support each individual department’s strategic IT objectives as well as those of the State as a whole. First, DOIT drafted a new AIMS review process based on existing policy, as published in the State Administrative Manual (SAM), and modified it to address specific crosscutting issues such as eGovernment. Using the first AIMS documents to arrive in May 2001, DOIT began piloting this process. DOIT plans to continue evaluating this process as agencies submit their AIMS documents or AIMS updates through September 2001. DOIT plans to utilize information gathered during this period and feedback received through the CIO Strategic Planning Workgroup, to reengineer existing AIMS policy and internal DOIT review procedures. DOIT targets the policy release in early 2002. The Statewide Strategic IT Plan publication, revised policies and revised proce- dures will fully support agency strategic IT planning during the 2002/03 planning cycle. Meanwhile DOIT continues to review projects initiated through the FSR process to ensure their consistency with agency strategic IT planning objectives and statewide strategic business issues. The BSA asserts that DOIT has not consistently reviewed and approved the strategic plans for departments’ IT projects. This practice was adhered to by the former Administration owing to lack of appropriate staffing. Now, with sufficient staff on board, DOIT not only has p reviewed all AIMS documents submitted thus far during the 2001/02 planning cycle but is also using its review effort as a primary basis for reengineering the existing AIMS process. BSA Recommendation: To provide appropriate department guidance and direction about the IT development process, DOIT should consolidate the various sources of policy and guidance, remove outdated policies from published documents, and revise policies as needed to reflect changing state needs. In addition, DOIT should resolve the contradiction between its management memorandum and the State Administrative Manual over the applicability of the alternative procurement process. DOIT Response Summary: In Progress. State IT Policies, Processes and Standards repre- sent a primary component within DOIT’s IT Circle of Life, supporting all technology lifecycle phases from concept through implementation and on into maintenance and operations. There- fore, to establish understanding and buy-in from a diverse set of stakeholders, DOIT employs a “best practices” collaborative approach, using a series of CIO subcommittees, to vet proposed 119 policies and procedures before formally publishing them. DOIT chooses this approach owing to the complexities of California IT governance and the fact that IT policies and practices cannot be “force fed”. Once proposed policies and procedures are appropriately vetted and approved, each CIO subcommittee is responsible for consolidating Management Memo and State Administration Manual (SAM) content for its assigned IT governance area. This approach will resolve any misunderstanding by eliminating outdated references and apparent contradiction. Detailed Response: DOIT is taking the leadership role in engaging the Agency Information Officers (AIO), agency CIO’s, control agencies, local government and the private sector in developing policies, processes and standards to align the State’s technology tools and ser- vices with its current business needs. The current administration uses an industry “best practices” collaborative approach in which DOIT plays a leadership role in working with a variety of state CIO Workgroups to develop, support and implement new policies, processes and standards that are enterprise-wide in nature. The CIO input and support is critical to successful and timely implementation. The private sector (California Information Technology Council) also has the opportunity to review proposed policies, processes and standards to validate the purpose and value of the proposed changes before they become final. This process was extremely successful during the state Y2K effort and has proven itself in the development of eGovernment standards and most recently in the development of the IT Project Submittal and Approval Process, which replaces the existing Feasibility Study Report process. DOIT also conducts pilots, another industry best practice, to refine the final product. The pilot tests the proposed model – which includes policy, processes, standards and tools – to ensure all components meet their stated objectives before DOIT applies them on an enterprise basis. This approach reduces confusion and ensures successful change management. Currently DOIT provides state departments IT guidance by issuing policies via Management Memos, which are later incorporated into the State Administrative Manual (SAM). The State Information Management Manual (SIMM) provides guidance on how to apply the policy; there- fore SAM and SIMM complement each other without representing a duplication of effort. Fi- nally, IT Directives are used to clarify existing policies and/or standards. DOIT will update SAM and SIMM once the policies have been properly vetted and piloted. DOIT will revise all corresponding sections of SAM and SIMM as each IT Circle of Life compo- nent is reengineered. DOIT agrees with BSA that some inconsistencies exist among Management Memos, SAM and SIMM. Such inconsistencies occur when DOIT Management Memos rescind existing policy or processes, yet the old policy or process is not physically removed from SAM. This minor paperwork issue does not imply in any way that DOIT is shirking its responsibility of providing 120 necessary guidance to the departments. Rather, DOIT works with the CIO’s to develop, vet and pilot many IT policies and procedures before formally issuing them by means of Manage- ment Memo, SAM or SIMM. This practice reduces confusion and misunderstanding while expediting the implementation and acceptance process for new policies. Departments are well aware of this practice because many participate in the pilot effort while others are informed in the biweekly CIO Meetings. When departments do have questions regarding proposed policy or procedures, they typically contact DOIT for clarification and direction. DOIT has established a DOIT Policy Committee to take an enterprise view of its policies, processes and standards. The Policy Committee chair is the Enterprise Services Division Deputy and includes the DOIT CIO Workgroup Sponsors (i.e., the DOIT executive responsible for the program area). The Policy Committee function is to coordinate the development and tracking of all new policy, processes and standards to ensure they meet the established timeframes for publication. The IT Circle of Life State IT Policies, Processes and Standards component begins its reengineering efforts by analyzing existing policies, processes and standards to determine which ones are outdated and which ones will be replaced or removed. The Policy Committee Chair is also the Chair for the CIO Advisory Council, which includes the CIO Workgroup Leads, the DOIT CIO Workgroup Sponsors and the Agency Information Officers. The CIO Advisory Council along with DOIT’s sister control agencies will comprise the vetting group for all proposed policy and standards before the proposed policy or standard is sent to the Governor’s Office for approval. As BSA notes in its report, DOIT maintains that the issuance of policy in 1998 requiring the use of alternative procurements indicates that the previous Administration had completed its as- sessment of that process in accordance with SAM Section 5215. However, DOIT, the Depart- ment of General Services and the Department of Finance are currently collaborating on addi- tional refinements to the alternative procurement process, including changes to the existing IT project submittal and approval activities associated with both planning and procurement, SAM Section 5215, previously issued DOIT policies and procedures, and Department of Finance Budget Letters. DOIT anticipates these changes will resolve any misunderstanding that may currently exist and further institutionalize the use of a procurement process that has received strong support from both the Administration and the Legislature. BSA Recommendation: DOIT should continue its efforts to improve its project review and approval process. However, it should ensure that the changes result in a process that will subject proposed IT projects to a thorough evaluation. Further, DOIT should ensure that departments are properly assessing IT projects by comparing departments’ feasibility study reports with established criteria, such as the fundamental decision criteria. Moreover, to ensure that it can defend its approval of costly IT projects, DOIT should thoroughly document its approval decisions. 121 DOIT Response Summary: Initial Reengineering Done, Pilot in Progress. IT Project Submittal and Approval comprises a major component within DOIT’s IT Circle of Life. As a result, DOIT has taken the leadership role in partnering with its sister control agencies and other department representatives to develop a new, web-enabled IT Project Submittal and Approval process. DOIT goals include streamlining project review and approval while at the same time adding greater rigor to the review process. Based on industry standards and best practices, as acknowledged by as many as 56 CIO’s, the new IT Project Submittal and Approval process requires that IT project proposals reflect the nine Project Management Institute (PMI) project management knowledge areas used to plan and execute successful IT projects. The process also requires agencies to provide specific programmatic information to ensure that business objectives are clearly understood and ad- dressed by the IT project proposal. DOIT is currently piloting its reengineered IT Project Submittal and Approval process with support from ten departments. Project information provided during the IT Project Submittal and Approval process will be used to populate DOIT’s Enterprise Portfolio Management System (EPMS), which serves as the central repository for all IT Circle of Life components. The redesigned IT Project Submittal and Approval process provides greater focus on quality planning and sets the stage for ongoing project oversight. It also provides the foundation for comprehensive enterprise-wide IT asset management by supporting the consistency and integrity of the State’s IT asset inventory. Detailed Response: DOIT is committed to implementing the IT Circle of Life as one of its highest priorities. The IT Project Submittal and Approval Process is the first component to reach the pilot stage, with others to follow. The use of industry best practices in the IT Project Submittal and Approval Process sets the standard for incorporating industry proven best practices, such as those published in the PMI’s Project Management Body of Knowledge (PMBOK) in the area of project management, risk management, quality, communication, acqui- sition planning, performance metrics and project tracking. DOIT will continue to solicit the assistance of agency CIO’s, local government entities and private sector participants to ensure that only proven and beneficial best practices are incorporated into the State’s IT processes. As each component of the IT Circle of Life is reengineered, DOIT must also reengineer its internal processes to support the new processes. As a result, DOIT is currently revising its project review and approval procedures to support the newly developed IT Project Submittal and Approval Process and to ensure complete documentation is available on the disposition of all IT project proposals submitted to DOIT. DOIT continues to provide agencies with the oppor- tunity to discuss their plans related to new IT projects before submitting a formal IT project proposal. This concept phase allows the agencies to gain valuable input on their proposed projects prior to formal submittal. 122 DOIT would like to note, however, that even though BSA states they reviewed ten projects w approved by the Administration, this is not a representative sample of the number of projects submitted for DOIT review. Moreover, the ten projects referenced were reviewed under proce- dures inherited from the previous Administration. At present, DOIT regularly reviews between 200 and 300 project documents per year and conducts a thorough analysis of each document that is submitted. This analysis generally includes requests for additional information and clarification from the departments, as well as multiple subsequent meetings with the depart- ments and the Department of Finance, often requiring submittal of revised documents. DOIT maintains that these activities, along with the presence of document support letters that specifi- a cally reference the Administration’s conditions of approval, are evidence of the completed analysis. BSA Recommendation: To ensure departments assess and mitigate project risks, DOIT should require complete risk assessment reports from departments. Further, DOIT should properly analyze the responses and document how it resolved any concerns. Finally, DOIT should require departments using the alternative procurement process to assess risks at the beginning of the project. If DOIT believes its current model is inap- propriate for alternative procurements, it should modify its new risk model to more appropriately address alternative procurement. DOIT Response Summary: Already in Progress. IT Project Submittal and Approval as well as IT Project Oversight comprise two key components of DOIT’s IT Circle of Life. The IT Project Submittal and Approval process focuses on the importance and need to fully under- stand and address risk mitigation issues, whereas IT Project Oversight demands effective and ongoing risk management. DOIT’s metrics driven IT project review and approval requires full risk assessment reports to accompany all department IT project proposals during the submittal process. DOIT is in the process of assessing its current risk assessment model (RAM) to ensure it reflects industry best practices by replacing existing subjective evaluations with more objective metrics. DOIT has engaged an industry leader to assist in developing a comprehensive set of project oversight and risk assessment metrics. Using this material as a base, as well as lessons learned during Y2K, DOIT plans to implement a revised RAM as part of the IT Project Submittal and Approval Process. Once the new model is in place, agencies will be required to provide dynamic and ongoing risk assessment and management by updating the RAM during the various stages of project development and implementation. Detailed Response: DOIT concurs that each IT project proposal must include a complete risk assessment report, which must be rigorously evaluated as part of the project review and approval process. Moreover, DOIT maintains that agencies must complete risk analyses that are consistent with and applicable to the specific IT projects proposed, and that DOIT must evaluate the results using the same criteria. For instance, some existing RAM questions may not apply to all project types (e.g., questions about system implementation, when the proposal seeks approval for hardware replacement), whereas some may apply to a subsequent project phase and provide little to no value during the IT Project Submittal and Approval Process (e.g., 123 questions regarding specific IT development teams, where the proposal seeks approval for a planning effort). Recognizing the importance of risk assessment and the need to continuously improve project review processes, DOIT has included, as part of the IT Project Submittal and Approval Pro- cess, the requirement for agencies to prepare and maintain comprehensive risk management plans, which include risk identification, assessment, mitigation and response activities. DOIT recognizes the importance of having a tool that transforms project progress information into meaningful measures, which provide department directors, CIO’s and project managers with valuable project status information and pinpoint areas where potential problems can arise. DOIT is in the process of implementing such a tool called the “Project Health Indicator.” By graphically displaying crucial measurement data in a variety of chart formats, this web-enabled tool allows department directors, CIO’s and project managers to identify areas of concern related to a specific project. Because of its component-based architecture and its metrics hierarchy, the Project Health Indicator can “zoom-in” to several levels of detail to assist project managers in isolating potential areas of concern. The Project Health Indicator tracks both positive and negative project performance trends, alerting the project manager when estab- lished thresholds are exceeded. DOIT’s Oversight and Early Warning System component within the IT Circle of Life will include the Project Health Indicator component to assist DOIT oversight staff in tracking IT project status. The BSA notes that, although DOIT received risk assessment summary reports for 9 of the 10 projects reviewed, it received sufficient information to evaluate the risk-planning efforts of only 3 projects. Once again, DOIT would like to point out that these 10 projects were reviewed s using procedures inherited from the prior Administration. DOIT’s current review process ensures that each assessment responds to all questions in accordance with the type of project being proposed. Moreover, DOIT’s review process ensures that each response is consistent with the project lifecycle phase represented. BSA Recommendation: To ensure that it receives and effectively uses the proper infor- mation to monitor departments’ IT projects, DOIT should take the following actions: (cid:127) Continue with its efforts to restructure the oversight process to ensure that the process allows DOIT to properly monitor and guide projects (cid:127) Modify the required progress reports to include a comparison of two types of critical information: the project’s monthly actual costs and revised estimates of total pro- jected costs with the budget, and actual and revised projected completion dates for project phases compared with the original schedule. Use this modified progress reporting to closely monitor projects that may be required to submit special project reports. 124 (cid:127) Ensure that analysts sufficiently review and document their oversight of projects and track the receipt of required reports. 1 DOIT Response Summary: Done, On-going. IT Project Oversight and the Special Project Report represent two critical components of DOIT’s IT Circle of Life. The reengineering of DOIT’s IT project oversight process and tools includes implementing a web-enabled Early Warning System that displays crucial project metrics, including key cost and schedule data. Developed by industry leaders, these metrics are based on project management best prac- tices, such as the red/yellow/green “traffic light indicators” so effectively used during Y2K, and allow agencies to report project status regularly while offering DOIT the ability to assess project health. These metrics also provide an objective link to the Special Project Report process, by which significant changes to department IT projects are tracked and approved. DOIT and departments also partner to select and contract with Independent Project Oversight Consultants, who provide independent and objective project performance assessments. Detailed Response: As indicated by the BSA, DOIT has begun to reengineer its project oversight processes and tools. Specifically, DOIT is in the process of implementing a web- enabled Early Warning System that allows department directors, CIO’s, project managers and DOIT to quickly and clearly monitor the status of their IT projects. This system graphically displays crucial performance measures in a variety of chart formats. DOIT’s new oversight program includes the two metrics (cost and schedule) recommended by BSA, as well as many other performance measures. Because of its component-based architecture and its metrics hierarchy, this system can “zoom in” on all levels of detail for any metric. This ability to isolate areas of potential concern helps project managers keep their projects on course while allowing DOIT to provide appropriate monitoring and guidance. Project oversight is a key element of DOIT’s Enterprise Portfolio Management (EPM) strategy, which spans the entire statewide technology lifecycle. Once viable projects are set in motion (through the project review and approval process), DOIT focuses on ensuring their successful completion through effective project oversight, as supported by the Project Management Institute’s best practices and objective performance metrics. The Oversight/Early Warning System component provides a mechanism for departments to report project status on a regular basis while offering the ability to evaluate project health. Successful oversight of complex IT projects requires collection and analysis of metrics that span various phases of the project. Metrics based on project management best practices are essential to their success. Projects will be evaluated based on metrics that have been col- lected in the following major categories: Project Initiation, Time Management, Scope Manage- ment, Human Resource Management, Cost Management, Procurement Management, Commu- nication Management, Risk Management, Quality Management and Project Completion. These metrics along with the Early Warning System raise the bar on accountability for IT project successes and enhance DOIT’s ability to provide oversight on all IT projects statewide. 125 As part of its oversight role, DOIT participates in Executive Steering Committees to work collaboratively on various oversight issues pertaining to IT projects. The committees meet monthly and on an ad hoc basis to discuss project issues, review the results of oversight activities and assess project risks and changes. In addition to participating on the Executive Steering Committee, DOIT oversight activities include: (cid:127) Reviewing monthly status reports for each IT project (cid:127) Reviewing specified deliverables produced during project design and development (cid:127) Attending meetings where project risks and risk mitigation measures are discussed in depth (cid:127) Participating with appropriate State executives in resolving sensitive issues encountered on various projects (cid:127) Reporting project status from a statewide perspective and providing recommendations for project performance enhancement DOIT also monitors department project management activities to ensure compliance with state- level information technology policies and standards. Through such monitoring, DOIT provides guidance and leadership to ensure that IT projects proceed in accordance with the approved project plan and budget. In keeping with best practices, effective oversight of IT activities provides value to state pro- grams and the constituencies they serve. Therefore, DOIT partners with departments to jointly develop Statements of Work and select Independent Project Oversight Consultants (IPOC) to perform independent oversight on IT projects. Each Statement of Work requires oversight contractors to submit all project reports to DOIT and the departments concurrently. DOIT reviews these reports and utilizes the information to evaluate, monitor and report the project’s health. The IPOC activities and the information they are expected to provide are listed below: Performance Monitoring (cid:127) Schedule and progress (e.g., “Are milestone/deliverable dates being met?”) (cid:127) Resources and cost (e.g., “How do resource expenditures compare to plan?”) (cid:127) Requirements growth and stability (e.g., “Are new requirements/change orders causing the overall size of the system to grow?”) (cid:127) Product quality (e.g., “Are deliverables sufficiently detailed, complete, accurate and trace- able to system requirements?”) (cid:127) Technical adequacy (e.g., “Will the proposed hardware and software provide sufficient capacity to meet operational performance requirements at full load?”) Performance Metrics Evaluation (cid:127) Actual effort expended compared to planned effort (cid:127) Actual delivery dates compared to planned delivery dates 126 (cid:127) Actual acceptance/approval dates (for each deliverable) compared to planned delivery dates (cid:127) Deliverable review times and deliverable review results (cid:127) Key staff experience compared to proposed experience (cid:127) Key staff turnover compared to industry standards/averages Deliverables Review (cid:127) Technical architecture deliverables, performance modeling and benchmark results, and any other contractor work products bearing upon issues of technical adequacy (cid:127) Review and comparison of functionality specified in the scope of work to actual product functionality delivered (cid:127) Functional specifications and test plans Contract Management Monitoring (cid:127) Contractor and State compliance with applicable contract terms and conditions (cid:127) Contractor and State adherence to project scope definition (cid:127) State response to contractor performance issues (if any) (cid:127) Change order processing Risk Management (cid:127) Risk identification, evaluation, tracking and mitigation (cid:127) Risk reporting and escalation within the stakeholder organizations (cid:127) Risk profiling for funding/oversight entities (cid:127) Independent “point-in-time” assessments of project activities to validate thoroughness of risk identification efforts DOIT maintains regular communication with Independent Project Oversight Consultant (IPOC) vendors and uses their progress reports to detect problems and be more proactive in dealing with troubled projects. An IPOC was required for the Department of Transportation’s (DOT) ATCAS project as a result of a Special Project Report (SPR) that was submitted to DOIT in August of 2000. Since that date, the DOT and the IPOC have met all reporting requirements. The IPOC has been provid- ing monthly reports, directly to DOIT, identifying the progress, project document/task reviews, risks encountered with related mitigations, overall project status, as well any recommendations. Previous administration only required periodic progress reports, which were due at specific milestones. This practice is no longer in place at DOIT. DOIT reviews department monthly progress reports, as well as the IPOC’s reports to evaluate, monitor and report the status of every project. BSA Recommendation: To hold departments accountable for the benefits expected from their IT projects, DOIT should ensure that departments submit post implementation evaluation reports. Further, DOIT should develop a process to promptly review and evaluate the reports submitted. 127 DOIT Response Summary: In Progress. The Post Implementation Evaluation Report (PIER) process represents a significant component within the IT Circle of Life. DOIT currently requires state agencies to submit a PIER following any reportable project implementation. With its re- e cently acquired staff increase, DOIT is able to review all project PIER’s submitted by those agencies. DOIT is revisiting the current PIER process to determine the extent changes are required to provide more rigorous capture of lessons learned and to build upon best practices, which can be shared in DOIT newsletters as well as such venues as the Executive Leadership Forums and CIO biweekly meetings. With the implementation of DOIT’s Enterprise Portfolio Management System (EPMS), much of the information to be provided within the PIER will already exist in automated form through previous project planning and oversight processes. DOIT’s primary goals for the PIER process include updating the State’s IT asset inventory, feeding objective project performance information into the statewide policy planning process and showcasing best practices through DOIT’s various communication channels. Detailed Response: The State Administrative Manual (SAM) section 4947 requires that agencies submit a Post Implementation Evaluation Report (PIER) following the completion of each information technology project. Accordingly, DOIT requires that state agencies submit a PIER following implementation of any reportable project. Now that DOIT is adequately staffed, it has begun reviewing PIER’s and additional reviews are in progress. DOIT plans to reengineer and further enhance the PIER policy and processes as part of the IT Circle of Life. The PIER process that was inherited from the Department of Finance, Office of Information Technology, was largely fiscal-based and did not include any evaluation of industry best practices more appropriate to DOIT’s role. The reengineered PIER process will provide the ability to leverage lessons learned, populate the IT asset inventory to allow for enterprise level licensing and capture information with which to update the State’s IT inventory. It will also provide for historical and statistical information to assist in more accurately estimating project duration and cost. According to the BSA, DOIT’s records show that it received roughly 87 PIER’s from depart- ments since 1996 yet offer no evidence that it analyzed any of the 87 reports received. The BSA also cites a claim made by the Legislative Analyst Office that there could have been more than 400 completed IT projects for which departments did not submit evaluation reports. DOIT e would like to point out that these 87 PIER’s were reviewed using procedures inherited from the prior Administration. Moreover, previous Administration practices did not provide for enforce- ment of PIER submittals. Now that DOIT has received the staffing necessary to perform comprehensive review and oversight, it plans to address these deficiencies in reengineering and enhancing the PIER policy and processes. Meanwhile, DOIT Project Review and Over- sight staff continue to review PIER’s submitted in accordance with existing practices. 128 Maintenance and Operations, and IT Asset Management, comprise the remaining two compo- nents of DOIT’s IT Circle of Life. BSA Recommendation: To promote coordination on IT projects and avoid redundant efforts, DOIT should establish a formal mechanism to initiate discussions between departments that are developing projects based on similar technologies or processes. To facilitate this coordination and improve project oversight, DOIT should complete its IT project inventory based on its survey of departments. DOIT should ensure that de- partments’ reported data is accurate and update this information when departments report new information so that the project inventory can be current. DOIT also needs to consider how departments and the Legislature can effectively access this information, taking into consideration privacy issues and other concerns that may limit the release of this information. DOIT Response Summary: In Progress, Priority. DOIT’s Enterprise Portfolio Management System (EPMS) facilitates the capture of relevant IT project information from concept through 1 implementation and on into maintenance. This web-enabled central repository represents the State of California IT investment portfolio and IT Asset Management strategy. DOIT is actively reengineering most of the components in the IT Circle of Life. As each component is imple- mented, DOIT, agencies and the Legislature will gain access to timely and accurate IT project and IT asset information, which can be leveraged in making informed IT decisions. DOIT’s new Maintenance and Operations policies and guidelines have been drafted and are currently entering their review cycle. DOIT bases its Maintenance and Operations program on experiences drawn from pilot implementations conducted with selected departments over the past three years. DOIT anticipates providing guidance to departments in extending objective performance measures from existing project oversight processes on into the Maintenance and Operations phase. This practice allows the State to gain more insight into improving program performance, optimizing existing resources and business processes, and providing feedback into the ongoing IT planning processes. DOIT’s current IT baseline inventory forms the foundation for statewide IT Asset Management. It also provides comprehensive baseline information in support of statewide strategic IT plan- ning. DOIT’s Enterprise Portfolio Management System is designed to support automated and ongoing refresh of critical Asset Management information. Detailed Response DOIT is taking the leadership role in implementing the IT Circle of Life within the State of California. At the core of this enterprise model core resides the Enterprise Portfolio Management System (EPMS), which houses the State’s IT project and IT asset information. The EPMS supports many industry best practices, including project management, performance metrics, project tracking, cost tracking and procurement processes. The EPMS offers major benefits to the manner in which state agencies plan, develop, monitor, implement and maintain IT solutions. This new business model significantly improves how the State of California manages IT investments and enhances the quality of service to its citizens. 129 DOIT is currently piloting the first major EPMS component, the IT Project Submittal and Ap- proval Process, with ten state departments to ensure the new policies, processes, standards and tools meet their stated objectives. State agencies will be able to register all IT projects into a central repository and submit IT project proposals electronically to all three control agencies (Department of Information Technology, Department of Finance and Department of General Services) by releasing the information once the proposal is complete. The response from the control agencies will be made available to the submitting agency online as soon as the control agency releases that information. As DOIT completes other components of the IT Circle of Life, more information related to IT project proposals, project progress and asset management information will be captured and readily available to control agencies, departments and the Legislature. Selected information may also be made available to the public. The use of the EPMS central repository information provides the basis to recognize and capitalize on opportunities for enterprise solutions early in the planning process. Coordination of similar activities between multiple agencies is an important and worthwhile endeavor, which offers tremendous cost savings potential for the State of California. BSA notes that the State undertakes multiple case management and licensing projects. DOIT has been directly involved in the review and oversight of these efforts. The technology similari- ties between these activities are severely limited by the great variation between the business processes, external interfaces and legal requirements of each program supported. The com- monality between case management systems, and licensing systems, is far greater between systems supporting similar program functions in other government agencies and the private sector than there is between state agencies. For several years DOIT’s consistent position has been to direct agencies to seek out and obtain experience and guidance from these peers, d perhaps adopting those external solutions that were proven effective. This position is a depar- ture from previous attempts to build enterprise systems to align with a standard State frame- work, regardless of complexity and risk such efforts might entail. DOIT, in conjunction with BSA, developed a baseline survey to collect information that per- tained to production applications and all projects under development, including those under- taken under departmental delegation. While the surveys were populated with production applications from the Y2K database, departments were instructed to list all development projects in addition to all production applications not listed on the survey. The survey data has been loaded into an interim database, which will be used by DOIT in reviewing the information for completeness and reasonability. DOIT will also use the data to crosscheck development applications reported by departments with those maintained in the project inventory database, identifying those under departmental delegation. DOIT will do a revalidation and expand the survey. 130 DOIT currently maintains an electronic project inventory that contains all projects approved via 1o an FSR. This inventory is continuously updated and provides the information needed for project coordination and oversight. This inventory is used to report any project that is in trouble or showing early warning signs while documenting a corresponding corrective action plan. DOIT then targets those projects for higher levels of project oversight. By incorporating project oversight into the IT Circle of Life and the associated Enterprise Portfolio Management System, DOIT is providing continuous improvement of its project oversight capabilities. Inventory data will be incorporated into the DOIT Enterprise Portfolio Management System (EPMS) thereby becoming part of DOIT’s integrated corporate database. The EPMS will support the strategies surrounding the State of California Information Technology “Circle of Life”, which incorporates all State policies and processes impacting IT projects from concept through implementation and on into maintenance. As a result, corresponding IT project informa- tion will be updated as the project proceeds through each phase of the Circle. DOIT, the departments and the Legislature will gain access to project and application information via the EPMS, which will facilitate informed decision making as it relates to existing and new policies, processes, procedures and tools that support the State’s IT projects. The business value and functional need for statewide “enterprise” IT systems is more important than the potential technical benefits. The previous administration’s enterprise workgroup concluded that the business value of such a system to the State of California was limited, while the cost and difficulty of such systems are high. This position has recently been reinforced through the administration decision to abandon, at least temporarily, the multi-year effort to procure and implement a new statewide payroll/personnel system. While DOIT remains cogni- zant of its statutory responsibilities to identify appropriate opportunities for such systems, it continues to consider such systems with proper regard for the State’s business needs and priorities. f The previous administration’s review of IT projects had no consistent procedure for identifying common elements across projects and only occasionally identified opportunities where depart- ments could discuss similar projects. Those practices are not in place any more. DOIT, through its project approval and oversight activities, has embraced opportunities to share its knowledge of similar applications and uses of technology with department customers. Also, through the project reporting criteria related to crosscutting statewide systems, DOIT becomes aware of proposed development and/or purchase of systems to support enterprise activities. As DOIT continues to reengineer its processes, the following strategies are envisioned to assist in ensuring that coordination between DOIT and departments occur consistently and proactively: 131 (cid:127) The Enterprise Portfolio Management “Circle of Life” program, particularly the IT Project Submittal and Approval, Oversight/Early Warning System, PIER and Asset Management components (cid:127) The automated IT Project Registration System, which requires registrants to answer a set of questions regarding program functions supported and proposed technology type, includ- ing whether the department has reviewed similar project implementations and solutions within other government entities (cid:127) Enterprise software licensing to leverage economies of scale in State software purchasing. Geographic Information Systems (GIS) represent a prime example of DOIT’s leadership and guidance in recognizing the opportunity to capitalize on enterprise solutions. State agencies use GIS to improve their effectiveness to plan, deliver and monitor the conditions associated with their programs. Currently, there is no mechanism to coordinate shared information re- sources that encourages collaboration between agencies and, ultimately, results in more efficient and effective program delivery. During the past year, DOIT and various state agencies partnered to develop a plan for a Statewide Geographical Information Infrastructure. This plan involves three primary compo- nents: a governance structure, technical system architecture, and a funding mechanism. The purpose of the Statewide Geographic Information Infrastructure is to assemble and maintain a statewide framework of shared geographic information that enables collaborative decision- making and crosscutting program delivery among agencies, various levels of government, and the public The overall program has executive sponsorship from several agency secretaries and depart- ments heads, including, Office of Planning & Research, Department of Information Technology, Resources, Cal EPA, and Health & Human Services. Additionally, the working committee, the California Mapping and Coordinating Committee (CMCC), which comprises more that 40 members from various state entities, meets regularly to implement the directions of the GIS Executive Sponsors. The working teams have developed a business process for statewide GIS, including initial charters for the GIS Executive Sponsors Committee, the Office of the State’s Geographical Information Officer (GIO), the CMCC, and the proposed External State- wide GIS Council. The Statewide Council will include representatives from Federal, State and Local government. This external council will serve as the primary collaborative forum to develop guiding principles and policies that promote development, dissemination, and shared use of geographic informa- tion in California. The external council will identify California’s geographic information needs, priorities and standards while promoting cooperation among federal, state and local agencies and the private sector in addressing these needs. The primary goal will be to leverage existing and new partnerships and funding sources to accomplish the most cost-effective and efficient means of developing and maintaining geographic information. From an IT perspective, the GIS infrastructure is a collaboration of enterprises. Each enterprise contributes information for the greater overall benefit of the participants. This infrastructure provides shared information resources that encourage collaboration between agencies, result- ing in more efficient and effective program delivery. 132 As another prime example of enterprise leadership and guidance, DOIT established the Acqui- sition Policy and Oversight (APO) Division on July 1, 2000. The Division is committed to col- laborative partnership with State entities, control agencies, local government entities and the vendor community. In support of the Administration’s long-term goal to strengthen and improve existing acquisition policies, DOIT created a Chief Information Officer (CIO) IT Acquisition Workgroup and an Enterprise Workgroup made up of CIO’s from various departments and local government entities. The purpose of these workgroups is to identify acquisitions that support best value, enterprise acquisition opportunities and the enhancement of the procure- ment process. DOIT recently released Management Memo 01-06, Information Technology Acquisition Plan- ning Policy. This policy requires State agencies to conduct acquisition planning and develop an overall IT acquisition strategy to acquire IT goods and services. This policy includes the identification of methods and approaches for conducting ongoing asset management to sustain an IT infrastructure that supports program operations. Leveraged agreements such as volume purchase agreement, enterprise agreements, and master agreements represent best practices utilized to achieve saving by leveraging buying power. In an effort to assist State entities in the management of their IT assets and to leverage the State’s buying power, APO, through the promotion of leveraged procurements, has been proactively supporting the development of enterprise agreements with major State suppliers such as Computer Associates, Microsoft, IBM and Oracle. On May 31, 2001, the first Enterprise License Agreement (ELA) was established with Oracle Corporation. This ELA database license covers full use of the Oracle database by State agencies to develop and deploy their applications. The agreement provides help desk services that include distribution of software and updates, asset tracking that will facilitate better control over the software. The agreement also provides State entities (excluding the universities) 270,000 named users, 100,000 power units, and reduced maintenance fees through Fiscal Year 2010/2011. For a period of five years the State, City and County governments receive a 50% discount on all other Oracle products (regardless of order size). Additionally, at the beginning of this year the State entered into a Volume Purchase Agreement with Microsoft, which provides an initial baseline discount of fifty percent (50%) through August 1, 2001. Currently, DOIT and the Department of General Services Procurement Division have partnered together with State agencies to establish a Multiple Award Contract (MAC) for Geo- graphical Information Systems (GIS) software, maintenance and other related services to be in place by June 24, 2001. A Master Service Agreement (MSA) to provide Statewide IT Consult- ing Services for independent oversight activities, project and quality assurance activities will be in place by September 1, 2001. 133 The BSA asserts that DOIT does not have an established process to ensure that departments do not independently develop statewide IT applications or duplicate other departments’ efforts. DOIT would like to point out that the IT strategic planning processes and the project review and approval processes inherited from the prior Administration provided neither the tools nor the data to support implementation of enterprise initiatives. Having recognized this deficiency, DOIT is in the process of developing its Enterprise Portfolio Management System and IT Asset Management capabilities, which will facilitate coordination among departments while supporting recognition and planning of enterprise initiatives. Meanwhile, DOIT’s Project Review and Oversight (PRO) Division staff work closely with departments to share information about other departments that are developing like systems. f During the Feasibility Study Report (FSR) review process, PRO staff meet on a regular basis to discuss IT projects currently under review and share information relating to those projects. This information is then shared with the departments that are considering similar IT projects. In addition, the existing PRO database contains information about all FSR projects submitted for approval. This database contains the type of technology the project proposal is addressing and is readily accessible by all PRO and DOIT Staff. Similarly, the Technology and Infrastructure Division offers guidance by working closely with other departments and PRO Division staff in sharing technical standards and best practices associated with particular technologies under consideration. BSA Recommendation: To improve compatibility and properly guide IT development, DOIT should expedite its efforts to implement standards by determining which stan- dards need to be addressed first and focus their efforts accordingly. Further, DOIT should work with departments to ensure that all necessary standards have been imple- mented. r DOIT Response Summary: Done, On-going. As stated earlier, State IT Policies, Processes and Standards comprise a major component within the IT Circle of Life. As a result, DOIT’s process to develop, document and review State IT standards involves full participation of state and external stakeholders. In addition, DOIT captures lessons learned via the PIER review process to provide additional standards input. Though time-consuming, this participative process ensures the successful adoption of full security and infrastructure standards as well as preliminary drafts of accessibility and eGovernment applications standards. DOIT has also adopted internal processes and standards to notify agencies when their project proposals involve activities subject to published DOIT standards. Detailed Response: DOIT has developed and implemented an effective process to develop, document and review state information technology standards. This process allows full involve- ment of state and external stakeholders. It offers the first real consensus among state 134 agencies and its suppliers regarding the value and viability of such standards in the State’s diverse and supplier-competitive environment. DOIT acknowledges that the process it has adopted is more time-consuming than one in which DOIT were to internally develop and publish such standards, with or without the assistance of consultant contractors. However, the suc- cessful implementation and statewide adoption of technical standards can be successful only if the state is willing to invest the time and effort involved in a fully participative process. r In the past year, DOIT has coordinated the development and review of two full standards, security and infrastructure, and preliminary drafts of two others, accessibility and eGovernment applications. In addition, DOIT has published and trained state agency staff on a fully redevel- oped set of procedures and guidelines for operational recovery planning, and is nearing completion of policies, guidelines and procedures for software asset management. DOIT has enabled more streamlined development of such standards in the past year by establishing a structure of internal and external advisory committees, by adopting a full lifecycle process for standards development, publication and maintenance, and by implementing a consistent, documented procedure for standards review and comment. These components, which did not exist before 1999, will enable continued, rapid progress in this critical area of DOIT leadership and guidance for IT. DOIT has adopted internal processes and standards to identify when agency project proposals involve activities subject to published DOIT standards, to specifically notify the agency of the requirement to conform to those standards, and to ensure that on-going DOIT and independent contractor oversight activities monitor project implementation and system operation to maintain conformance with those standards. BSA Recommendation: To ensure that DOIT is fully employing the IT advisory councils and receiving the benefits intended by law, DOIT should continue to meet with the private commission and the public committee on a regular basis to guide its strategic planning efforts, provide input on new policies and ensure that the State follows best practices. DOIT should ensure the public committee makes all findings and recommen- dations in writing, as required by state law. DOIT should also monitor the progress of its CIO work groups to ensure that they reach their established goals. DOIT Response Summary: Done, Ongoing. DOIT meets with the private commission and the public committee on a regular basis to guide its strategic planning efforts, receive input on new policies, and ensure that the State utilizes industry best practices and processes. All the findings t and recommendations from these two advisory councils are documented. Detailed Response The private commission, known as the California Information Technology Commission (CITC), is an elite body of experts that serve as an advisory commission on initiatives that are planned, initiated or will be implemented. Under the previous Administration, the CITC consisted primarily of marketing representatives. Today’s CITC is comprised of thirty- five industry technology leaders, including Chief Information Officers and Chief Technology 135 Officers, carefully selected from a cross-section of public, private, academic, non-profit and other organizational structures that provide advice and best practice information to assist the State in responding proactively to IT issues and trends. The CITC advises DOIT on key policies impacting the State’s IT direction and provides insight to strategic technologies that are proven to solve complex business problems. Commission meetings are public and designed to solicit input from the public. Agendas and meeting notices outlining the business of the Com- mission are posted on the DOIT’s web page ten days prior to each meeting. The following represent quotes from CITC members citing DOIT’s important contributions to technology leadership within state government: “DOIT is vital to keeping the state up to date on info tech issues and will undoubtedly continue to provide quality guidance it is known for.” – Apple Computer “Continuing its current course, DOIT is in a unique position to make a lasting contribution toward the challenges that the State and the people of California face as the State ventures further into the era of eGovernment transformation.” – IBM “DOIT provides needed leadership and oversight and ensures that IT investments are managed properly, including enterprise initiatives such as eGovernment implementation, Geographic Information Systems (GIS), and other major enterprise-wide initiatives.” – Compaq “Its coordination and review of projects saves state revenue by spotting inefficiencies and recommending remedies to them. It also enhances the state’s productive engagement with the Information Technology industry by setting benchmarks and standards to which we can aspire.” – Unisys “As a member of the California Information Technology Commission, HP has appreciated the opportunity to see first-hand the potential that DOIT has to provide leadership and vision for many of the technology challenges facing California in the near future.” – Hewlett Packard “The department plays a critical role as a central planning and oversight agency for the state’s information technology programs.” – Microsoft “Under the current leadership demonstrated by the DOIT, California is on the threshold of establishing public sector IT standards which have been proven by leading commercial industry stakeholders in California’s Information Technology marketplace. This California based market- place leads the world in IT practices and has come of age. Under the DOIT leadership, these practices and standards are finally making their way into public sector IT policy.” – CTA “Pacific Bell supports the leadership of the Department in exercising oversight of state agen- cies’ information technology projects. Centralized strategy planning and policy processes on a statewide basis will bring cost savings that benefit all Californians.” – Pacific Bell 136 “Since the beginning of the Davis Administration, Cisco Systems has worked with DOIT on initiatives from Y2K compliance to the Project Registration System. DOIT has shown leader- ship in improving the State’s ability to apply, manage and implement information technology.” – Cisco Systems “The DOIT’s development of acquisition policies, standards, and adherence to industry best practices has had a significant impact on improving the efficiency and effectiveness of critical IT project teams.” – Logicon DOIT established the public committee, referred to as the Enterprise Coordination Council (ECC) and formerly known as CIO Advisory Council, to foster collaboration across the State and to provide technical advice and strategic direction to ensure all agencies share a single IT vision. DOIT meets with the ECC regularly to guide its strategic planning efforts, receive input on new policies, and ensure that the State follows IT best practices. The current council is comprised of CIO’s that chair the thirteen discrete DOIT-sponsored CIO Advisory workgroups and the Agency Information Officers (AIO’s) representing Business, Transportation and Hous- ing Agency, State and Consumer Services Agency, Health and Human Services Agency, Resources Agency, Environmental Protection Agency and DOIT. The CIO Advisory Workgroups focus on Recruitment and Retention, Training, FSR Reengineering, IT Acquisition and Security, to name some of the thirteen DOIT-sponsored workgroups. All of the CIO Advisory Workgroups are key to DOIT’s mission and objectives, with CIO’s and senior IT staff representing in excess of fifty departments serving on the vari- ous workgroups. Each workgroup has an average of fifteen members with some much larger, such as the FSR Reengineering, which has more than eighty members representing over forty departments. Members representing the departments of Finance, General Services, Correc- tions, Motor Vehicles, Employment Development, Social Services, Health Services, State Personnel Board, Health and Human Services Data Center, Teale Data Center, Governor’s Office of Planning and Research, and the Governor’s Office for Innovation in Government participate, to name only some that serve on the various CIO Advisory Workgroups. t ECC meeting agendas and notes are maintained. All proposed policies are submitted to the council for their review and comments. DOIT executives and senior managers are members of the CIO Advisory Workgroups to ensure that continuity of the Workgroups’ efforts are focused on agency and statewide strategic goals. New CIO Advisory Workgroups are established as key statewide challenges or new initiatives arise. Since their initial meetings, the CITC and ECC have reviewed proposed policies that include the California Software Management Policy, IT Management Policy and Feasibility Study Report (FSR) Reengineering effort. This review included the metrics to support the FSR reengineering effort. The council also evaluated the Enterprise Portfolio Management System (EPMS), which supports the State’s IT “Circle of Life”. The advisory council has provided input to the collective efforts underway regarding Statewide IT Recruitment and Retention, including the IT Classifi- cations minimum qualifications and alternative approaches that may be incorporated to 137 recruit and retain IT Staff. More specifically, DOIT’s Recruitment and Retention Unit along with the CIO Advisory Workgroup has taken the lead of the State’s effort to revise the minimum qualifications for 30 of the State’s 34 primary IT classifications. The Recruitment and Retention Unit recently coordinated participation in a Technology job fair resulting in resumes from over 350 qualified individuals interested in State IT careers. Prior to revising the minimum qualifica- tions, less than ten per cent of these individuals would have been eligible for State technology jobs. A subsequent recruitment activity resulted in over 100 recent college graduates submit- ting resumes and expressing interest in State IT careers. Again, the revised qualifications acknowledge all of an individuals related work experience, providing much broader career options for all, especially the college graduates. BSA Recommendation: To ensure that it completes initiatives, DOIT should establish timelines and goals for meeting future initiatives. If DOIT does not believe it can com- plete initiatives within established guidelines, it should communicate its priorities and resource requirements to the Legislature. In addition, it should notify the Legislature when changes in the State’s IT environment prompt adjustments to these priorities or resource requirements. DOIT Response Summary: Done, Ongoing. DOIT, with support from legislative staff, has developed a realistic timeline for policy development and publication. DOIT will report its progress to the appropriate legislative staff to meet those commitments and will promptly notify the Legislature if DOIT needs to readjust priorities or resource requirements due to changing circumstances. DOIT has also offered to meet with the Legislative Analyst Office and legisla- tive staff, as needed, to keep them engaged. Detailed Response: DOIT and legislative staff met and agreed to a prioritized and realistic plan for DOIT policy publication. During the 2001/02 Budget process, DOIT worked directly with representatives of the Legislature, including staff for the budget subcommittees and the Legislative Analyst Office, to identify and prioritize DOIT policy and standards development efforts. These working sessions resulted in an agreed upon schedule to develop and formally publish policy based upon mutual agreement of priorities and the availability of DOIT and external resources to perform those efforts. DOIT will continue to engage the appropriate legislative staff to providing the respective progress in meeting these commitments. Further- more, DOIT will promptly inform the Legislature if it has a need to readjust priorities due to unforeseen circumstances, such as changes in the State’s IT environment or higher than anticipated impact from HIPAA. BSA Recommendation: To organize and focus its efforts, DOIT should adopt an internal strategic plan to identify key responsibilities and establish priorities. This plan should clearly describe how the organization would address its many responsibilities, particu- larly those that we observed it has not sufficiently accomplished. Further, it should build on past efforts to the extent possible rather than reinventing processes and prac- tices when planning its future activities. 138 y DOIT Response Summary: Done, Updates Ongoing. DOIT has always operated from an internal strategic plan. However, as of January 2001 DOIT began drafting a new internal Strate- gic Business Plan, establishing priorities and key responsibilities to “operationalize” DOIT’s IT Circle of Life and supporting Enterprise Portfolio Management System. This plan offers a com- prehensive approach to organizing and addressing DOIT’s multi-faceted responsibilities as well as prioritizing its efforts and establishing measurable objectives. However, rather than building on fragmented and archaic processes and practices of the prior Administration, DOIT’s current reengineering efforts undertake a more comprehensive and consistent approach on enterprise crosscutting issues that span the entire technology lifecycle. Detailed Response: As of January 2001, DOIT began drafting a revision to its internal Strate- gic Business Plan to reflect organizational objectives beyond Y2K. This plan publishes time frames for issuing standards and policies, documents communication strategies through DOIT’s on-going advisory councils, and defines the strategic framework DOIT will use to coordinate the State’s IT activities. It also establishes and prioritizes measurable objectives to maximize DOIT’s value to the State IT governance structure while ensuring consistent perfor- mance. In doing so, DOIT defines a baseline against which it can ensure it has the resources necessary to achieve those objectives. Leveraging successful methodologies from the private sector, including the Project Manage- ment Institute’s (PMI) Project Management Body of Knowledge relating to best practices in the IT industry, DOIT is working to redefine itself, its operating model and vision. Central to this approach is the framework in which all of DOIT’s specific tactical and strategic objectives will be realized, the IT Circle of Life. This innovative operating model enables the State to imple- ment an enterprise-wide Asset Management strategy via the Enterprise Portfolio Management System (EPMS). The EPMS enables the State to make technically sound and financially viable investments in IT for the optimum benefit of state government and the state’s citizens. DOIT’s Strategic Business Plan addresses to “operationalizing” the State’s Information Technology Circle of Life and supporting EPMS by defining a comprehensive set of organizational strategic initiatives extending through Fiscal Year 2004/2005. DOIT recognizes that recent key management turnover has had a significant impact and caused critical institutional program knowledge loss and the disruption to internal planning efforts. As a result, DOIT has intensified its efforts to support Legislative action that eliminates the existing 2002 sunset date. Confronting the uncertainty that such a sunset date brings is not only deleterious to staff morale, but creates an undue challenge for DOIT’s internal strategic planning efforts. 3 DOIT disagrees that it has not taken sufficient advantage of the processes and priorities established by prior management. Severe resource limitations forced the previous Administra- tion to focus on exceptions and high-visibility projects; hence, policies and practices during that 139 time reflect a fragmented approach to technology leadership, guidance and oversight. While reengineering statewide policies and procedures, DOIT has considered those policies and practices employed. However, DOIT is currently better staffed and therefore able to undertake a more comprehensive and consistent approach to enterprise crosscutting issues that span the entire technology lifecycle. DOIT has engaged a top management consulting firm to evaluate its capability to perform the program areas (e.g., strategic planning) and to develop recommendations that improve DOIT’s efficiency and effectiveness. This company will review DOIT’s existing goals and objectives, and evaluate how well its policies and procedures, organizational structure, and staffing levels enable it to achieve these goals. This firm will also develop recommendations on automated tools that may assist DOIT in this process, as well as develop education and training programs for DOIT and state agency staff. 140 COMMENTS California State Auditor’s Comments on the Response From the Department of Information Technology T o provide clarity and perspective, we are commenting on the Department of Information Technology’s (DOIT) response to our audit report. The numbers correspond with the numbers we have placed in DOIT’s response. 1 Throughout its response, DOIT indicates that it is addressing our recommendations through its new enterprise portfolio manage- ment program. However, it is important to recognize that this program is in its infancy. DOIT at times discusses elements of the program as though they were in place, but as we note in several places of our report, DOIT has only just begun to initiate its reengineered processes. While DOIT recently began piloting certain processes, such as the approval of feasibility study reports, it plans to implement elements of its reengineered processes over the next several years. Therefore, we are unable at this time to assess the value that its efforts will add to the oversight of the State’s information technology (IT) development. 2 Although DOIT believes our recommendations are of limited value, it also states that our recommendations are consistent with the new enterprise direction it is taking and reports progress on each of the recommendations. Additionally, it is clear from our discussions with DOIT that it is using the term “artifacts” to describe documents or other pieces of evidence that we had requested to support its actions related to past, current, and planned activities. Generally accepted government auditing standards require auditors to obtain sufficient, competent, and relevant evidence to afford a reasonable basis for findings and conclusions. Because DOIT often lacked documents or evidence to demonstrate or support its actions, it is not surprising that DOIT criticizes our focus on what it terms “artifacts.” 141 3 DOIT is mischaracterizing our recommendation. Our point is that DOIT should not reinvent past processes and procedures that had merit. We recognize that many improvements are needed to DOIT’s operations; however, to discount all past efforts as flawed is both wasteful and shortsighted. Additionally, we are troubled that throughout DOIT’s response, it blames the processes of the previous administration for problems we cite in our report. The current state chief information officer (state CIO) was ap- pointed in February 1999. Yet each of the weaknesses we noted were the result of processes used since then. Further, DOIT has only begun to initiate changes to its processes. 4 DOIT’s comments about the focus of our audit are misleading. Chapter 2 specifically addresses the audit objective of how well departments are managing the development of their IT projects. We informed DOIT management that Chapter 2 would address our findings and recommendations related to other departments that we reviewed, but that we could only share with DOIT what we reported related to its operations. Moreover, as discussed on page 11 of our report, many of the specific areas the Joint Legislative Audit Committee asked us to review were under the direct responsibility of DOIT. The extent to which we have findings and recommendations related to DOIT is indicative of the extent that it has weaknesses in its operations. 5 DOIT dismisses our audit as a “typical compliance audit,” yet the areas we reviewed, and for which we found DOIT’s performance lacking, are responsibilities that are critical to its mission in providing leadership over the State’s IT efforts and not merely compliance issues. For example, as we discuss on page 16 and page 18 of our report, DOIT has not provided a current statewide IT plan that would act as a guide for the State’s IT efforts nor has it sufficiently reviewed the strategic plans for departments’ IT efforts. Further, DOIT indicates throughout its response that it is addressing our recommendations through its new enterprise portfolio management program. However, DOIT has only recently begun a pilot of some elements of the program and has not even begun to reengineer other elements of the program. Thus, an audit cannot assess the effectiveness or outcomes of what DOIT holds out to be its major accomplishments until such accomplishments are achieved. 142 6 DOIT’s conclusion that there have been no failed major projects among those that were approved or sponsored by the current state CIO seems premature. As noted previously, the current state CIO was appointed in February 1999. It is unlikely that many major projects have been approved, funded, started, and com- pleted since then. Further, it appears that DOIT has a very narrow view of whether a project is successful. A broader view focuses on whether completed projects achieve the original intended ben- efits and reasonably meet the original cost and schedule targets. Because DOIT has not ensured that departments complete postimplementation evaluation reports to assess the benefits and costs of completed systems, it has little basis for commenting on the extent to which projects have been successful under the administration of the current state CIO. 7 DOIT is wrong. The audit analysis did not state that we would evaluate the sufficiency of resources allocated by the prior administration. Rather, the audit analysis states that we would evaluate the sufficiency of DOIT’s resources to fulfill its statutory responsibilities. Moreover, focusing on the sufficiency of DOIT’s current resources is appropriate because doing so would determine whether a recommendation for change is warranted. In accomplishing this objective, we obtained DOIT’s Capability Assessment Report that it submitted to the Legislature in March 2001. DOIT prepared this report in response to the Legislature’s requirement that it provide an independent assessment of the activities and resources needed by DOIT to successfully manage IT projects. In this report, both DOIT and a consultant it hired concluded that DOIT’s authorized staffing of 80 positions for fiscal year 2000-01 was sufficient to perform its statutory responsibilities. We discuss this conclusion as well as some history on DOIT’s staffing on page 7 of our report. 8 DOIT indicates its sunset date is a primary cause of staff turnover. However, it appears as though other factors play a role. On page 65 of our report, we discuss turnover in key management posi- tions stating that, since July 2000, 8 of DOIT’s 11 key managers have left. Individuals in these positions serve at the will of the state CIO and, therefore, are subject to removal at any time. Further, many of those individuals have accepted positions with similar employment conditions at other state departments. Thus we question whether the 2002 sunset date is the primary cause of high turnover. 143 9 DOIT has not accurately depicted one of our comments regarding the year 2000 effort. As discussed on page 64, we acknowledged that the governor told DOIT that the year 2000 effort was its top priority. However, the governor’s executive orders did not relieve DOIT of its responsibility to oversee departments’ IT efforts. Therefore, as we stated on page 64, faced with insufficient re- sources to focus on the year 2000 effort and still adequately cover its other responsibilities, DOIT should have requested additional staff for its other responsibilities. Further, while many departments’ projects were deferred because of the year 2000 effort, it was still important for DOIT to meet its responsibilities for those that were not deferred, particularly projects that were large and considered high-risk. 0 DOIT is misleading when it indicates that we concluded “that a great deal of important work remains to reconstruct previous administration’s flawed IT management infrastructure.” Any conclusion regarding the extent to which the previous administration’s infrastructure was flawed is solely that of the current administration of DOIT. q DOIT appears to be trying to deflect the leadership problems we observed by blaming its problems on the State’s IT governance structure. However, this governance structure is one that any control agency within a decentralized government must operate within, and those that are effective demonstrate strong leadership. During our audit, we asked DOIT if it had sufficient authority to effectively oversee and guide the State’s IT efforts and if there were any barriers to exercising its authority. In a written response dated October 5, 2000, DOIT stated that it believed it had sufficient and appropriate authority to discharge its responsibilities. Additionally, it stated that it “does not believe that statutory changes are necessary to assist it in discharging its statutory responsibilities to oversee and guide state IT.” Further, DOIT also stated that the most severe barrier to DOIT implementing its full statutory authority is the continued existence of its sunset provision and did not mention any other barriers. Thus if DOIT considered that changes in the governance structure or the removal of barriers were needed for it to properly carry out its responsibilities, DOIT should have recommended the changes it believed were necessary. By asserting that governance is the problem, DOIT appears to be redirecting the responsibility for its problems to the environment in which it operates within. Our audit confirms that DOIT needs to improve its leadership as indicated throughout our report. 144 w DOIT’s statements about our sample on pages 101 and 123 are misleading. We focused our review on projects approved during fiscal year 1999-2000 because this was the most recent period in which projects had completed the approval process and were funded for development. Further, we attempted to obtain the number of projects approved by DOIT during this period from its records, but found that DOIT was not adequately keeping track of such information. The Department of Finance’s records indicate it received 124 feasibility study reports for proposed projects during fiscal year 1999-2000—substantially less than the 300 projects that DOIT states. Of these 124 feasibility study reports, 64 were approved, and the remainder were disapproved, withdrawn, duplicates, or returned to the department. Because our sample of 10 projects was about 15 percent of the projects approved during that fiscal year, our sample provides a reasonable basis to evaluate DOIT’s approval process. Moreover, we believe that a sample of 10 projects was appropriate because the results were consistent. DOIT could not provide sufficient evidence that it thoroughly analyzed any of the projects we reviewed. e DOIT is misleading when it states that it is able to review all postimplementation evaluation reports submitted by departments. As we discuss on page 30 of our report, DOIT states that it re- cently began to review a few evaluation reports. When we asked DOIT for evidence of its reviews in May 2001, DOIT provided us information that showed it had only recently reviewed 3 evaluation reports. Further, as discussed later in its response, on page 128, DOIT indicates that it had reviewed all 87 reports that it received from departments since 1996. However, DOIT was unable to provide us any evidence that it had actually reviewed these 87 reports. r Contrary to what it indicates, DOIT is not “done” with implement- ing any portion of our recommendation on standards. As we described in Table 2 on page 59 of our report, four of five key standards we identified are still in draft form, and the process to develop the remaining standard has not started yet. Additionally, DOIT still plans to develop other standards in the future including those for asset management and e-mail. Finally, as we recommended in the report, we believe DOIT should work with departments to ensure that all necessary standards have been implemented. 145 t Contrary to its assertion, DOIT was unable to provide us any written findings and recommendations generated by the CIO Advisory Council (which DOIT now calls the Enterprise Coordina- tion Council), as we discussed on page 61 of our report. DOIT’s failure to maintain this information, as required by state law, restricts the ability of interested parties, including members of the public and the Legislature, to review the advice this council provides DOIT. y Although DOIT claims it has always operated under an internal strategic plan and began revising it in January 2001, DOIT was unable to provide us more than a high-level operating plan for July 2000 through March 2001. As discussed on page 64 of our report, this plan is a list of numerous activities that did not differentiate between activities that must be completed and those that were less critical. Additionally, the list lacked measures for DOIT to determine if its efforts are effective. Further, it is doubtful DOIT had an internal strategic plan, because it told us at our exit conference that it did not provide a plan to the con- sultant hired in October 2000 to analyze its internal operations. In fact, the consultant’s report in January 2001 recommended that DOIT “rigorously prioritize its efforts.” u DOIT overstates the extent to which it has taken an active leadership role in various efforts, including promoting architectural consistency. It has not formally issued technical standards that would promote this consistency. Additionally, DOIT has not fulfilled other leadership responsibilities such as ensuring it has up-to-date policies to guide departments’ efforts. i DOIT has not presented the whole story from Governing Magazine’s 2001 report that discusses California’s IT activities. DOIT omitted the report’s key criticism of the State’s IT efforts—“Inefficiencies today occur primarily because of a lack of comprehensive data format and content standards.” This criticism of the lack of data standards, as we discuss on page 59 of our report, addresses an area of responsibility where state law has clearly given DOIT the leadership role. Additionally, it appears as though the authors of this report were not always provided accurate information. The report credits California as having postimplementation audits to make sure that projects deliver their promised reports. Yet, we found, as discussed on page 30 of our report, that DOIT has not ensured that departments submit postimplementation evaluation reports and that it has only recently begun to review a few of them. 146 o DOIT states that it has established a statewide inventory containing IT project and infrastructure information across all state agencies. However, as we discuss on page 57 of our report, the departments’ information that is the basis for the inventory contain many problems and inconsistencies. As a result, DOIT needs to expend considerable effort to obtain consistent and accurate project information. As we discuss on page 58 of our report, it is unclear at this time when DOIT will have a complete, accurate, and up-to-date inventory available for departments to coordinate their efforts and for the Legislature and control agencies, including DOIT, to improve their oversight. p DOIT’s comments about the reviews of departments’ IT plans are misleading because it indicates that the insufficient reviews were a practice of the prior administration, thus implying that DOIT has been reviewing these plans since the current state CIO was appointed in February 1999. However, as discussed on page 18 of our report, DOIT indicated that reviewing these plans in the past did not merit assignment of its resources because of higher priori- ties, including the year 2000 effort, and thus these reviews occurred only sporadically. DOIT also stated that it began reviewing depart- ments’ IT plans on a case-by-case basis in May 2001—as of that time, it had completed a draft review of one department IT plan— and it plans to redefine this process sometime in early 2002. a The activities that DOIT asserts provide evidence of its “completed analysis” were the same ones that we reviewed and found to be insufficient. Specifically, as noted on page 22 of our report, for 5 of the 10 projects we reviewed, we found no evidence documenting DOIT’s analysis—checklists, project summaries, or other documents, such as DOIT’s questions to departments. For the remaining 5 projects, the evidence of DOIT analysis was insufficient to demonstrate why DOIT approved the projects. s DOIT’s statement that its current review process ensures that each risk assessment responds to all questions in accordance with the type of project being proposed is misleading. As we discuss on page 24 of our report, DOIT does not require departments to answer the same questions when submitting their risk assess- ments. Some departments are asked to respond to as few as 5 questions while others must respond to as many as 38. We found no indication that the variance in questions asked was caused by the type of project. Additionally, DOIT’s current review process allows departments to perform risk assessments late in the approval process for those projects using the alternative procurement process. 147 d We recognize that the same systems may not be appropriate at each department, but departments can benefit by sharing their knowledge and experiences with technology, development approaches, and vendors on projects in similar business areas. Further, rather than directing departments to seek out others, which DOIT indicates is its current approach, we believe it is part of DOIT’s leadership role to facilitate this sharing. f DOIT is mischaracterizing its current coordination activities. DOIT states that it was the previous administration’s review of IT projects that had no consistent procedure for identifying common elements across projects. DOIT claims that these practices are not in place anymore and that DOIT, through its project approval and oversight activities, embraces opportunities for coordination. Later in its response, on page 134, DOIT contends that its staff work closely with departments to share information about other departments that are developing like systems. However, as noted on page 54 of our report, DOIT’s current activities lack a mechanism that would consistently identify similarities among IT projects. Such a mechanism could foster coordination beyond analysts’ ability to recognize similarities among projects. Our report also acknowledges that DOIT plans to improve its ability to identify projects with similar characteristics by creating a new database that will capture information from more than 200 questions that it will ask departments about their projects. However, DOIT has not yet implemented this process. 148 Agency’s comments provided as text only. Business, Transportation and Housing Agency 980 9th Street, Suite 2450 Sacramento, California 95814-2719 June 13, 2001 Elaine M. Howle State Auditor Bureau of State Audits 555 Capitol Mall Sacramento, CA 95814 Dear Ms. Howle: Attached is the Department of Transportation's (Caltrans) response to your June 7, 2001, draft audit report dealing with the State's management of information technology (IT). As its response indicates, Caltrans concurs with the report's findings and recommendations related to its Advanced Toll Collection and Accounting System (ATCAS) project. The Business, Transportation and Housing Agency (Agency) also concurs with the report's findings related to Caltrans' project management, and is pleased to note that the BSA acknowledged external circumstances that also adversely affected the ATCAS project. Additionally, the Agency concurs with the report's recommendations, and supports Caltrans' plan for corrective action. We are thankful for your review, and for the opportunity to respond to the draft report. The BSA recommendations will be very helpful in improving the management of current and future IT projects. If you need additional information, please do not hesitate to contact me, or Michael Tritz, Chief of the Agency's Office of Internal Audits, at (916) 324-7517. Sincerely, (Signed by: Al Lee for) MARIA CONTRERAS-SWEET Secretary Attachment 149 Department of Transportation Office of the Director 1120 N Street, MS 49 Sacramento, CA 95814 June 12, 2001 MARIA CONTRERAS-SWEET, Secretary Business, Transportation and Housing Agency 980 - 9th Street, Suite 2450 Sacramento, CA 95814 Dear Secretary Contreras-Sweet: I am pleased to provide our response to the Bureau of State Audits' (BSA) draft audit report dealing with the State's management of information technology (IT). The Joint Legislative Audit Committee requested the audit, which covered a complex IT project at each of four State agen- cies. The primary finding related to the California Department of Transportation's (Department) man- agement of the Advanced Toll Collection and Accounting System (ATCAS) is that the Department did not effectively manage the ATCAS project in accordance with the 1998 Legislative Analyst's Office's "Best Practices" and State requirements for IT projects. Notwithstanding the fact that the ATCAS project was initiated in 1993, the Department concurs with the issues and recommenda- tions noted in the report. Following is each BSA recommendation, followed by a discussion of our corrective efforts. Recommendations The BSA recommended that the Department take the following actions to improve the Department's IT project management practices (BSA recommendations are in bold): Develop a problem statement for each IT project that adequately describes the problem the project is intended to solve with quantifiable goals, and a supportable business case for each project that justifies its funding. The Department concurs with this recommendation and recognizes that as a result of decentral- ized IT project initiation and the lack of IT policy direction, projects in the past did not have a fully 150 MARIA CONTRERAS-SWEET June 12, 2001 Page 2 documented business case. The Department is currently establishing the Program and Project Management Division (PPMD). The Project Initiation and Reporting Office, within the PPMD, will be responsible for enterprise project initiation. All project Feasibility Study Reports statewide will be reviewed for compliance with State and departmental IT policy, and will ensure that the project proposal fully documents the business case. The PPMD is targeted to become operational in July 2001. Develop an effective project management plan before beginning to develop each project so it can monitor the progress of the project. The Department concurs with this recommendation. As a result of similar findings from other external and internal audits, the Department will perform the following: (cid:127) Expand on internal IT policies and procedures. (cid:127) Fine tune and deploy the Department's IT Project Management Methodology (PMM) for enterprise use, by October 1, 2001. (cid:127) Deploy a development tool set that will directly support the enterprise PMM and will enable repeatable processes for project delivery and the project manager's ability to estimate project schedules and cost. (cid:127) Provide a training program on the PMM starting June 11, 2001. Three levels of training are planned: 1) executive level staff, 2) project managers, and 3) project development staff. (cid:127) Provide project management training during the first quarter of FY 2001/2002. This training will be Project Management Book of Knowledge (PMBOK)-based and will provide the educa- tional background necessary to pursue project management certification. Allow vendors to propose solutions and the technical specifications for its large and complex IT projects. The Department concurs with this recommendation. The Department intends to allow vendors to propose solutions and the technical specifications for its large and complex IT projects procured both in conformance with the traditional Request for Proposal process and the Alternative Pro- curement methodology initiated following Management Memo 98-12, issued on June 24, 1998. The first project implemented under the Alternative Procurement process was the Transportation Operations and Project Support System (TOPSS). Subsequently, the Department has worked with the Department of Information Technology (DOIT) and the Department of Finance's Technol- ogy Investment Review Unit (TIRU) in further refining the use of what has evolved into the 151 MARIA CONTRERAS-SWEET June 12, 2001 Page 3 Alternative Procurement Business Justification (APBJ) process. As both procurement processes have evolved, the Department has recognized the need to provide the vendors with the depart- mental standards for the IT architecture and with a business case for the project, in the beginning of the process, to allow vendors to propose solutions and technical specifications. Ensure that testing is completed at appropriate phases to identify and resolve problems before moving ahead. The Department concurs with this recommendation. The Department anticipates the training and deployment of the PMM will help ensure that the appropriate testing will take place on IT projects. The Department's IT division is planning to develop Quality Assurance and IT Architecture units. These units will provide operational support for the PMM. The Department believes it can ensure that the various phases of an IT project are validated through the full project lifecycle by utilizing the Independent Validation and Verification (IV&V) processes, the oversight of the Risk Manage- ment Office, and the Quality Assurance and IT Architecture units. Ensure that it uses sound management practices during the development of each project, such as assigning qualified individuals with appropriate experience and training to manage the project, documenting key discussions and decisions, and monitoring progress through periodic reports. The Department concurs with this recommendation. The Department is creating the PPMD to ensure better management and consistencies in IT project development, enterprise-wide. The PPMD is targeted to be operational in July 2001. The Department will provide IT outreach and educational programs to further promote the staff's awareness of the IT policy and project definition. As part of the educational programs, project managers will receive training regarding IT project documentation, monitoring and reporting requirements. The PPMD will support the following activities from both a policy and operational perspective: (cid:127) IT strategic planning and performance measures. (cid:127) IT project initiation and reporting. (cid:127) IT IV&V and risk management. (cid:127) IT acquisitions of goods and services. (cid:127) Business process re-engineering and data administration. 152 MARIA CONTRERAS-SWEET June 12, 2001 Page 4 The creation of the PPMD, and the deployment and training for the PMM will be major steps toward the improvement of the Department's IT practices. Use independent verification and validation consultants on complex IT projects. The Department concurs with this recommendation. Management Memo 00-11, Information Technology Oversight Policy, was issued on November 3, 2000. This policy establishes a frame- work for effective oversight of State IT activities. The Department understands the need for IV&V to provide an independent reporting entity to the DOIT. The Department is utilizing or is planning to utilize project oversight and/or IV&V contractors on the following projects: (cid:127) ATCAS (cid:127) PRSM (cid:127) Transportation Permits (cid:127) TOPSS (cid:127) Integrated Maintenance Management System (IMMS) The Department will continue to utilize IV&V contractors for specific IT projects as appropriate. As previously stated, one of the functions of the PPMD is to promote, monitor and assist project managers with the engagement of IV&V contractors for their projects. If we can provide further information, or if you have any questions, please contact Gerald Long, External Audit Coordinator, at (916) 322-7122. Sincerely, (Signed by: Tony V. Harris for) JEFF MORALES Director 153 Blank page inserted for reproduction purposes only. 154 Agency’s comments provided as text only. Health and Human Services Agency 1600 9th Street, Room 460 Sacramento, CA 95814 June 13, 2001 Ms. Elaine M. Howle State Auditor Bureau of State Audits 555 Capitol Mall, Suite 300 Sacramento, CA 95814 Dear Ms. Howle: Thank you for forwarding for my review and comment a draft copy of the Bureau of State Audits' report as it relates to one of the Department of Health Services' (DHS) and one of the Employ- ment Development Department's (EDD) information technology projects. I am forwarding to you both DHS' and EDD's responses to the review findings and recommendations. I understand that DHS and EDD have begun taking steps to address the issues raised in the Bureau's report. Thank you once again for sharing the draft copy of your findings and recommendations. If you require further information concerning DHS' or EDD's information technology projects, please do not hesitate to contact me directly. You may also contact Diana M. Bontá, the Director of the Department of Health Services, at (916) 657-1425 and Michael Bernick, the Director of the Employment Development Department at (916) 654-8210 to assist you. Sincerely, (Signed by: Grantland Johnson) GRANTLAND JOHNSON Secretary Enclosure 155 Department of Health Services 714/744 P Street Sacramento, CA 95814 Ms. Elaine M. Howle State Auditor Bureau of State Audits 555 Capitol Mall, Suite 300 Sacramento, CA 95814 Dear Ms. Howle: Thank you for the opportunity to comment on the draft of the Department of Health Services' (Department) portion of your recent audit dealing with the Children's Medical Services Network Enhancement 47 (CMS Net E47) project and the State's management of information technology. The Department recognizes the importance of using sound management practices for its infor- mation technology (IT) projects and ensuring that its projects are completed on time and within budget. The Department agrees with the Auditor's Best Practices recommendations, and will continue taking action to implement them as available resources permit, both for the duration of the CMS Net E47 project, as well as for future IT projects the Department undertakes. The Department is committed to the establishment of a centralized Project Management Office (PMO) in the Information Technology Services Division (ITSD) to enhance the effectiveness and success of the Department's IT projects. In addition to providing ongoing project management direction and oversight to Department initiatives, the PMO will help ensure implementation of the Best Practices as detailed in the Auditor's recommendations. A more detailed overview of the Department's activities and plans to implement the recommen- dations, as well as our comments on specific audit findings, is provided in the enclosed "Com- ments on Draft Audit Report from Bureau of State Audits". If you have additional questions or concerns, please feel free to contact Mr. Rick Vagg, Acting Deputy Director, Information Technology and Services Division (ITSD), at (916) 657-1255. Again, thank you for the opportunity to comment. Sincerely, (Signed by: David Souleles for) Diana M. Bontá, R.N., Dr.P.H. Director Enclosure 156 COMMENTS ON DRAFT AUDIT REPORT FROM BUREAU OF STATE AUDITS Response to BSA Recommendations The Bureau of State Audits recommends that the Department should take the following actions to improve the management of information technology (IT) projects: (cid:127) Select vendors that propose the best solutions at the best value. The Department agrees with this recommendation. In order to ensure that future procure- ments by the Department follow the State's best practices for selecting vendors and best solutions, the Department is establishing a centralized project management office in the Information Technology Services Division to oversee Department initiatives. For the CMS Net enhancement, the Department will procure a new developer for the remain- ing components of the project through an alternative competitive procurement process that follows best practice guidelines. All the development contracts for the CMS Net enhancement project noted by the auditors have ended and the contractors have completed the scheduled work. (cid:127) Structure contracts with vendors to protect the interests of the State, including provi- sions to pay vendors only after deliverables have been tested and accepted. The Department agrees with this recommendation and will strengthen future contracts for IT development projects to incorporate best practices, including selecting the vendor based on best value; ensuring that the contract is deliverables-based; providing contract language to protect itself if the vendor does not complete its work on schedule; and retaining a portion of payments to the vendor until work is performed satisfactorily. The Department's new project management office will provide direction and oversight in contract management, procure- ment, and best practices to support Departmental IT projects. (cid:127) Use sound project management practices during the design, development, and imple- mentation phases of projects and specifically ensure that it assigns project managers with appropriate training and authority. 157 The Department agrees with this recommendation. The new project management office is proposed to be headed by a project management officer established at the Career Executive Assignment II level. A temporary project management officer has been appointed through an interim redirection. Project management positions will be requested at the Data Processing Manager III and IV levels, depending on the complexity and requirements of the project. In filling the positions, the Department will seek experienced project managers who have or are in the process of obtaining Institute of Electrical & Electronic Engineers (IEEE) and/or Project Management Body of Knowledge (PMBOK) certification. The Department has been using recognized industry best practices for project management on the CMS Net enhancement project, beginning in November 1998 with the recommenda- tions made by the Independent Verification and Validation (IV&V) contractor. Examples of sound project management practices adopted by the CMS Net enhancement project include development of procedures to manage and control changes; development of smaller projects with milestones; development of a statistically based testing system for developed software; and establishment of a risk management plan. The Department has appointed the project manager for the CMS Net enhancement project at the Data Processing Manager III level. The audit accurately reports that from November 1999 through November 2000, following the departure of the project manager, the Department hired a contract project manager. During this period, the Department worked to reclassify the State project manager position. The CMS Branch recognized that the project needed a higher level manager and submitted a request to upgrade the position to a Data Processing Manager III on April 11, 2000. This request was approved on October 4, 2000. The current Data Processing Manager III project manager was hired on December 20, 2000. (cid:127) Correct the IT security weaknesses we identified. The Department remains committed to assuring that its systems maintain the highest levels of security. In order to assure this, the Department is continuously reviewing its security procedures to identify weaknesses and approaches to resolving them. The new project management office will work closely with the Department's IT security officer and the Health and Human Services Agency Data Center on security issues. (cid:127) Ensure independent oversight of its projects by hiring IV&V consultants from firms that are different from those providing other services to the project. The Department will assure that this recommendation is followed on all future IT projects. 158 Additional Comments Additionally, the Department has the following comments on the findings of the audit: (cid:127) The Bureau of State Audits finds an 80 percent cost overrun and a 15 month delay. The original proposal for this project was written in 1995 but the project was not funded until 1998. The time and cost generated by this unanticipated delay were not included in the original project plan. In the period between 1995 and the present, there have been a number of unanticipated events, such as Y2K and the implementation of Healthy Families, that have impacted the scope of the project. These include the following: (cid:127) The delay of two years between preparation of the Feasibility Study Report (FSR) and funding the project resulted in additional costs and extended the timeline for completion of the project. These time and cost changes were delineated in a Special Project Report submitted in late 1997. (cid:127) Amendment of the federal Social Security Act to enact the State Children's Health Insur- ance Program, known as Healthy Families in California, established new business require- ments for the CMS Net enhancement. Concurrently, modifications to other State systems to implement Healthy Families imposed additional changes to the design of the CMS Net enhancement project. (cid:127) Y2K remediation and certification of CMS Net required additional time and resources. (cid:127) The original FSR delineated system modifications to CMS Net in order to interface with existing State systems. Actual implementation identified the need to make significant modifications to two other State eligibility systems, the Medi-Cal Eligibility Data System (MEDS) which had to be converted from month-specific to day-specific dates of eligibility to reflect the policies of the programs that use CMS Net, and the Statewide Client Index which required modification in order to receive information from CMS Net. All changes to the cost and time schedule have been delineated in three Special Project Reports, all of which have been approved by the Department of Information Technology and the Technology Investment Review Unit of the Department of Finance. Where indicated, the budget change process has been used. 159 As a result of the decision to undertake a competitive procurement for the new developer to complete the remaining components of the CMS Net enhancement and in order to ensure that all best practices for procurements are met, the completion of the CMS Net enhancement project will be later than December 2002. The new developer will submit a revised project plan that will determine the completion date. (cid:127) The Department is rethinking the technical direction of this project for the remaining components of CMS Net/E47. The audit report indicates that a recent assessment required by DOIT and conducted by a technical consultant concluded that the project as currently planned was technically fea- sible, but did not represent a sound architectural plan. The audit report indicates that the consultant reached this conclusion because the software and database management system were not widely used or easily supportable, and because neither is supported by the Department's current IT infrastructure. The Department agrees with the technical consultant's assessment that the software and database management system used in the CMS Net system is not widely used in the technology industry. The existing CMS Net system, on which the enhancement is being built, was acquired by the Department in 1992 as public domain software from a similar program in San Diego County. This technology is widely used in the health care industry, both in the private and public sectors. For example, this technology is used in UCLA Medical Center, University Hospital at Stanford University, Johns Hopkins Hospital in Baltimore Maryland, and Sunquest Information Systems in Tucson, Arizona. Acquiring this existing system was a sound decision in 1992 because the system was built and immediately available, it met the business requirements, and it was a no-cost acquisi- tion. The CMS Net system continues to meet the program's business needs and is cur- rently used by 46 counties and the CMS Branch. Support for maintaining this architecture is available locally and the Department has contracted for support to maintain the current software and database management system for CMS Net. This technical support includes regular communication and meetings between the contractors; the Department's Informa- tion Technology Services Division; and the Health and Human Services Agency Data Center. 160 The proposed CMS Net enhancement project will add new functions to the CMS Net System such as authorized provider enrollment and client service authorizations. A com- petitive procurement will be conducted to select a new developer to develop these func- tions. This will provide an opportunity for the Department to review available technology options to assure that the most up-to-date and appropriate architectural platform is used for the additional components. There have been significant advances in technology since the CMS Net system was acquired and the options for architectural platforms identified in the technical consultant's assessment were not considered mature by industry standards when the CMS Net enhancement project was proposed. The technical consultant's as- sessment will be an important resource for the Department as the CMS Net enhancement is developed. (cid:127) The Department has not hired a new IV & V consultant to replace the IV&V consultant who resigned in August 2000. When a development contract is selected, a new IV&V consultant will be hired. Comments on the Ratings (Table 5) The Department's comments below address only those areas rated as Weak. Planning and Procurement ♦♦♦♦♦ Outline the business problem, allow vendor to propose solutions. The procurement process for a new developer for the CMS Net enhancement project will meet this best practice. Additionally, the Department will include this practice in the guidelines to be established by the new project management office for Department IT initiatives. ♦♦♦♦♦ Sponsor the project through appropriate commitment and involvement of man- agement, and include sufficient resources. The auditor reports that this weakness refers to staff assignments to support both the existing CMS Net system and the CMS Net enhancement project. The Department will continue its efforts to provide needed resources to support this project. 161 ♦♦♦♦♦ Base procurement on best value, not lowest cost. The Department has already determined that an alternative procurement process based on best practices will be used in selecting a new developer for remaining components of the CMS Net enhancement project. The Department will include this practice in the guidelines that the new project manage- ment office will establish for Department IT initiatives. ♦♦♦♦♦ Use a vendor that provides and maintains sufficient qualified staff. The Department intends to establish criteria for sufficient qualified staff in selecting a new developer for the CMS Net enhancement project. The Department will include this practice in the guidelines that the new project manage- ment office will establish for Department IT initiatives. ♦♦♦♦♦ Require letters of credit from the vendors on a large project. The procurement process for a new developer for the CMS Net enhancement project will meet this best practice. Additionally, the Department will include this practice in the guidelines that the new project management office will establish for Department IT initiatives. ♦♦♦♦♦ Write strong contracts to adequately protect the State. Under the procurement process for a new developer, the CMS Net enhancement project will meet this best practice. Additionally, the Department will adopt contract best management practice guidelines to be developed by the new project management office. Design, Development, and Implementation ♦♦♦♦♦ Develop an adequate security system to detect and prevent inappropriate access. The Department remains committed to assuring that its systems maintain the highest levels of security. In order to assure this, the Department is continuously reviewing its security procedures to identify weaknesses and approaches to resolving them. The Department's new project management office will work closely with the IT security officer and the Health and Human Services Agency Data Center on security issues. Quality Assurance ♦♦♦♦♦ Pay the vendor only on acceptance of tested project deliverables. The contract for the new CMS Net enhancement developer will include provisions that reimburse vendors only when the Department accepts tested deliverables. The Department will include this practice in the guidelines that the new project manage- ment office will establish for Department IT initiatives. 162 Employment Development Department June 13, 2001 Ms. Elaine M. Howle State Auditor 555 Capitol Mall, Suite 300 Sacramento, CA 95814 Dear Ms. Howle: This letter contains the Employment Development Department's (EDD) response to the Bureau of State Audit's (BSA) review of the TEAM project. The EDD agrees with the BSA's findings. For clarity, each BSA recommendation statement is included below, followed by EDD's response. Recommendation: Ensure that the vendor provides sufficient staff with the necessary training and experience. On future projects, the EDD will require that vendors provide sufficient, adequately skilled staff necessary to complete the project within the required time frame. In addition, the EDD will review vendor staffing throughout the project to ensure appropriate staffing levels are maintained. Additionally, for future contracts of this type, the EDD will consider strengthening contract provi- sions that will compel the vendor to provide sufficient skilled staff throughout the project. Recommendation: Use an effective project management plan before beginning to de- velop each project so it can monitor the progress of the projects. Throughout the life of the TEAM Project, the EDD continuously focused on ensuring that the project management planning methods and practices used provided a framework to assist in the successful completion of this project. The EDD requested Logicon (TEAM's IV&V consultant) to provide a final project closure assessment including key issues, lessons learned, and recommen- dations. This review was presented to the EDD and the Department of Information Technology on May 18, 2001. In addition, the project team plans to develop a project assessment document that will be used, along with the BSA and Logicon recommendations, to assess steps necessary to further strengthen our project management and development methods and practices. 163 Ms. Elaine Howle June 13, 2001 Page two We will also continue to review our project management and development methods and practices to reduce the risks and provide a framework to assist project managers in successfully complet- ing projects on time and within budget. Recommendation: Ensure that it establishes and uses a process to control and monitor project scope changes that requires changes to be adequately reviewed before they are made. As stated in the audit report, the EDD did make changes to strengthen our project scope change control review and approval process during the TEAM project. We plan to closely monitor and control scope changes for future projects. The strong project management plan (described earlier) will assist with monitoring progress and controlling scope changes. Recommendation: Correct the IT security weakness we identified. We agree with the audit finding that security of the TEAM system could be improved. As a part of its e-government strategy, the EDD recognizes the need for increased security across the enter- prise. As part of the Expanding Access to EDD Services (EASE) Feasibility Study Report (FSR), the Department has recommended significant changes and improvements to security processes and procedures, as well as, technology changes to all systems running at the EDD that would provide increased levels of security. The security functionality would be managed by a central- ized EDD security organization that is yet to be established. The security organization will have responsibility for processes, procedures, and practices to monitor all activities (authorized and unauthorized) to EDD's automated systems. The EASE is currently undergoing a funding deci- sion by EDD management. Although the projected date for implementing a fully functioning security organization in EDD is June 2002, until the EASE project funding issue is resolved, the target implementation date for technical and administrative security changes has not been set. The major security components to be added as part of EASE are: (cid:127) Technical detailed security architecture design (cid:127) Security Incident Reporting (cid:127) Security Standards for Servers Desktops and Modems (cid:127) Security Awareness Training and Education (cid:127) Improved Business Continuity (cid:127) Network Content Filtering (cid:127) Development of Security Guidelines into the System Development Life Cycle 164 Ms. Elaine Howle June 13, 2001 Page three (cid:127) Secure Mobile computing environment (cid:127) Improve Security Organization (cid:127) Authentication services (cid:127) Access control (cid:127) Network monitoring (cid:127) Intrusion detection Aside from the funding and implementation of the solutions identified in the EASE FSR, some changes could be made to the TEAM system that would improve security. These changes involve software upgrades and increasing capacity to expand security functions. However, no funding currently exists for these changes. Furthermore, because the security infrastructure proposed with the EASE FSR will impact all of EDD's systems, we prefer to invest in this enterprise ap- proach rather than investing in specific systems individually. If you have questions about this response, please call Pam Harris at 654-7014. Sincerely, (Signed by: Sam Rodriguez for) MICHAEL S. BERNICK Director 165 Blank page inserted for reproduction purposes only. 166 Agency’s comments provided as text only. State and Consumer Services Agency Office of the Secretary 915 Capitol Mall, Suite 200 Sacramento, CA 95814 June 14, 2001 Elaine Howle, State Auditor Bureau of State Audits 555 Capitol Mall, Suite 300 Sacramento, California 95814 Dear Ms. Howle: SUBJECT: AUDIT REPORT NO. 2000-118 Enclosed is our response prepared by the Franchise Tax Board to the Bureau of State Audits’ Report No. 2000-118 in which a portion of the report is related to an information technology project managed by the Board. A copy of the response on a diskette is also included. If you have any questions or need additional information, please contact me at 653-4090. Sincerely, (Signed by: Clothilde V. Hewlett) Clothilde V. Hewlett Undersecretary Enclosures 167 Franchise Tax Board P.O. Box 115 Sacramento, CA 95741-0115 M E M O R A N D U M To: Aileen Adams, Secretary June 14, 2001 State and Consumer Services 915 Capitol Mall, Room 200 Sacramento, California 95814 From: Gerald H. Goldberg Subject: Audit Response (2000-118) Franchise Tax Board (FTB) has prepared the following response to the excerpts from the draft audit report from the Bureau of State Audit for the Accounts Receivable Collection system (ARCS). The Accounts Receivable Collection System (ARCS) was designed to create new benefits through a common systems approach that would consolidate and bring full automation to the tax programs, institute standard, proven modeling capability and minimize duplication of effort in cross-training and operational burdens among the collection programs. The ARCS project utilized the benefits of a performance-based procurement. The department worked jointly with the business partner to design and develop the new system. Payments to the business partner were based on tangible benefits achieved by the new integrated system. The benefits- based funding strategy shifts a large portion of the risk of the project's success to the business partner. The ARCS project costs were estimated at $33 million, with increased revenues (benefits) of $76.5 million and an implementation schedule of June 2000. After detailed system designs were developed changes to the baseline functionality became apparent. In order to comply with legislation and incorporate the necessary changes to the system implementation, the schedule was extended. This change in the schedule was necessary for a fully functional system to meet the user needs. The extended schedule allowed FTB to include the necessary additional business functionality that better met the business needs of our users and positioned the department to meet the ongoing business needs. As a result of the changes to the sched- ule, the cost estimate was revised to $36.260 million, 9.8 percent more than the original estimate. ARCS was completed in March 2001 with benefits far exceeding the original esti- mates. ARCS has collected over $150 million in new revenue which would not have been collected under the previous system. ARCS is expected to collect an additional $35 million annually in new revenue. 168 Aileen Adams Page 2 06/14/01 This response has been saved in the requested format on the diskette furnished by the Bureau of State Audit. If you need additional information, please contact Janet Sherwood of my staff at (916) 845-5530. (Signed by: Gerald H. Goldberg) Executive Officer 169 cc: Members of the Legislature Office of the Lieutenant Governor Milton Marks Commission on California State Government Organization and Economy Department of Finance Attorney General State Controller State Treasurer Legislative Analyst Senate Office of Research California Research Bureau Capitol Press 170