Summary

Year 2000 Computer Problem: Progress May Be Overly Optimistic and Certain Implications Have Not Been Addressed August 1998 98023 The first printed copy of each California State Auditor report is free. Additional copies are $3 each. Printed copies of this report can be obtained by contacting: California State Auditor Bureau of State Audits 555 Capitol Mall, Suite 300 Sacramento, California 95814 (916)445-0255 or TDD (916)445-0255 x 248 Permission is granted to reproduce reports. C S A ALIFORNIA TATE UDITOR KURT R. SJOBERG MARIANNE P. EVASHENK STATE AUDITOR CHIEF DEPUTY STATE AUDITOR August 27, 1998 98023 The Governor of California President pro Tempore of the Senate Speaker of the Assembly State Capitol Sacramento, California 95814 Dear Governor and Legislative Leaders: The Bureau of State Audits presents its audit report concerning the State’s reported progress in fixing its critical computer projects for the year 2000. This report concludes that state agencies may not be as far along in fixing their critical computer projects as previously reported. Furthermore, many critical projects may not actually be ready for the next millennium because state agencies are not thoroughly testing their remediation efforts, have not completed all the necessary steps to address interfaces with data exchange partners, such as counties and the federal government, and have not completed business-continuation plans to ensure the uninterrupted delivery of critical services into the next century. Respectfully submitted, KURT R. SJOBERG State Auditor BUREAU OF STATE AUDITS 660 J Street, Suite 300, Sacramento, California 95814 Telephone: (916) 445-0255 Fax: (916) 327-0019 CONTENTS Summary S-1 Introduction 1 Audit Results The State’s Reported Progress Is Overly Optimistic and Several Concerns Remain Unaddressed 7 Recommendations 25 Appendix A Survey of 39 State Agencies 29 Appendix B Comparison of Reported and Actual Project Status 31 Appendix C Analysis of the Time Needed to Begin the Phase Reported in the DOIT’s April 1998 Quarterly Report 33 Appendix D A Comparison of Project Completion Dates 35 Responses to the Audit Department of Information Technology and the Governor’s Office 37 State Board of Equalization 43 State and Consumer Services Agency 45 Franchise Tax Board 47 Department of Social Services 49 State Treasurer’s Office 53 California State Auditor’s Comments on the Response From the State Treasurer’s Office 55 C A L I F O R N I A S T A T E A U D I T O R SUMMARY RESULTS IN BRIEF A s the year 2000 fast approaches, state agencies are rush- ing to fix their critical computer projects to allow the continued delivery of essential products and services to Californians. However, fixing almost 700 of the State’s critical Audit Highlights computer projects may not be as far along as reported in the Our review of the State’s April 1998 quarterly report published by the Department of technological readiness to Information Technology (DOIT) and reported to the Legislature. deliver critical services at the change of the century Furthermore, many state agencies have not addressed all facets revealed that: of the year 2000 problem and, therefore, may not actually be (cid:254) Efforts to fix almost 700 of ready for the next millennium. Specifically, agencies are prema- the State’s most critical turely declaring their critical projects complete that have not computer projects are not progressing as rapidly as been thoroughly tested. Critical projects are those so important reported. that their failure would cause a significant negative impact on (cid:254) the health and safety of Californians, on the fiscal or legal Some projects declared integrity of state operations, or on the continuation of essential complete are not. state agency programs. (cid:254) Critical systems have not been tested in the rigor- Thus far, none of the agencies reporting on completed critical ous “time machine” environment and few projects to the DOIT have rigorously tested their information- such tests are scheduled. technology systems, comprised of one or more critical projects, (cid:254) in an isolated environment where the computer’s internal clock Time allotted for project is set to dates in the next century to make sure the systems will testing is far less than industry norms. continue to function after the year 2000. Moreover, several agencies responsible for remediating large, complex systems (cid:254) Agencies sharing data have yet to even schedule such tests at either of the State’s two with other computer data centers. While all critical projects may not need this type of systems have not taken the steps necessary to testing, we believe the fact that none of the 10 agencies report- protect against failures ing completed critical projects to the DOIT has used such testing caused by corrupted data. on those projects is cause for concern. Moreover, in many cases (cid:254) Managers at most the amount of time agencies are allocating to test their critical agencies have yet to projects falls far short of the 50 percent to 70 percent of total develop tangible business- project time and resources that others in the industry have spent continuation plans in the on testing. event of year 2000-caused delays or failures. In addition, many of the State’s critical computer projects and systems depend on data exchanges with other entities, such as counties and the federal government. Yet not all agencies have completed the necessary steps to ensure that data transmitted C A L I F O R N I A S T A T E A U D I T O R S-1 through these interfaces will work seamlessly with the State’s computer systems into the next century. Even if agencies successfully fix their own critical computer systems, they still may not be able to deliver expected products and services in the next millennium if their data-exchange partners’ systems are not year 2000-ready. Finally, the managers of most state agencies have yet to ensure that their agencies have established appropriate business- continuation plans in the event of failures or delays caused by the year 2000 problem. Agencies appear to be focusing exclu- sively on fixing critical computer systems and choosing not to involve the individuals responsible for program delivery in determining what to do if critical systems do not work as intended or are delayed. However, rather than using staff involved with remediation, we believe the managers responsible for the agencies’ core business processes should establish work groups of program staff and dedicate sufficient resources to develop business-continuation plans to ensure that the agencies maintain the delivery of essential products and services in the event of year 2000-induced failures or delays. RECOMMENDATIONS To ensure uninterrupted delivery of essential products and services to Californians, the Governor’s Office should ensure that all state agencies take the following steps: • Provide the Department of Information Technology (DOIT) with accurate information about the status of their year 2000 remediation efforts. Specifically, the estimated completion dates for each phase of remediation, including final comple- tion, should reflect the agency’s best estimate for the actual completion dates and should be updated whenever circum- stances affecting a project’s status change. • Thoroughly and comprehensively test the remediation for each critical project. For larger, complex projects associated with systems that support the delivery of services to Califor- nians where interruption would be unacceptable, agencies should also consider testing the system in an isolated com- puter environment using a time machine. Moreover, prior to declaring a project complete, tests of any internal interde- pendencies, external data exchanges, 20th and 21st century date recognition, and the impacts from embedded systems S-2 C A L I F O R N I A S T A T E A U D I T O R such as desktop computers, should be complete and the project acceptance tested and approved by agency managers responsible for the business functions. • Protect their computer systems from missing or corrupted data supplied by external parties. Specifically, agencies should identify their data-exchange partners, develop schedules for testing and implementing new date formats, and thoroughly test data supplied by external parties. • Establish business-continuation planning groups, made up of managers from major business units, experts in relevant functional areas, business-continuation and disaster-recovery specialists, operational analysts, and contract specialists. These planning groups should then follow a structured approach to develop a business-continuation plan for each core business process and infrastructure component affected by the year 2000 problem. In addition, to ensure that the administration and the Legisla- ture have accurate information about state agencies’ progress toward fixing their critical projects and systems threatened by year 2000 problems, the DOIT should do the following: • Continue to collect and analyze information state agencies provide on their overall progress. If, after analyzing the reported information, something appears anomalous—such as too little test time—contact the agency for an explanation. • Continue to collect information from agencies on their data-exchange partners. In addition, take appropriate follow-up action if it appears that agencies are not testing their interfaces with data-exchange partners. • Require agencies, as part of their monthly reporting, to indicate whether they have business-continuation plans that ensure that each core business function will continue uninterrupted if the critical computer systems supporting those functions fail to work or are delayed because of year 2000 problems. C A L I F O R N I A S T A T E A U D I T O R S-3 AGENCY COMMENTS The Department of Information Technology (DOIT), responding on its own behalf and that of the Governor’s Office, stated that our observations and conclusions have substantial merit. In addition, three of the four agencies we reviewed concur with our recommendations and believe them to be consistent with industry standards and best practices. The State Treasurer’s Office (STO) believes the methodology we used to determine the accuracy of reported project status differs from that used by the STO and perhaps other agencies and, therefore, believes it projects are closer to being fully remediated than was indicated in our report. S-4 C A L I F O R N I A S T A T E A U D I T O R INTRODUCTION BACKGROUND T he State faces a tremendous challenge as it prepares its computer systems for the year 2000 problem and coping with dates beyond December 31, 1999. Most computer systems are designed to use only the last two digits of a year. The two-digit convention originally conserved computer storage space that once was at a premium but is now relatively inexpen- sive. However, on January 1, 2000, systems using two-digit date fields may produce invalid results or fail because they will read the date “00” as 1900 rather than 2000. This problem will affect many of the State’s computer systems. For example, a computer system using a two-digit date would calculate the age of a person born in 1978 by subtracting the last two digits of the birth year from the last two digits of the current year, 98 minus 78, and determine the person was 20 years old. However, on the first day of the new century, the computer would calculate the person’s age by subtracting 78 from “00,” making the age minus 78. Such errors could negatively affect eligibility for benefits, license requirements, or other date- sensitive areas. The year 2000 problem not only involves the more routine information technology (IT) systems, but it also impacts systems using embedded technology, such as computer chips to control or operate equipment, and on IT systems that depend on the exchange of data with other organizations. Because of the magnitude of the problem and the potentially disastrous consequences to California’s citizens, the governor issued an executive order in October 1997 requiring all state agencies to identify and fix year 2000 problems in their essential computer systems no later than December 31, 1998. Essential or critical systems are defined in the State Administrative Manual as so important that their failure would cause a significant negative impact on the health and safety of Californians, on the fiscal or legal integrity of state operations, or on the continua- tion of essential state agency programs. To address this important issue, the governor has designated the Department of Information Technology (DOIT) to oversee, coordinate, and report on efforts agencies are making to ensure that over 2,700 of the State’s computer systems are fixed to C A L I F O R N I A S T A T E A U D I T O R 1 recognize the year 2000. These efforts involve repairing and testing or replacing components to recognize both the 20th and 21st centuries. The DOIT requires agencies, departments, boards, and commissions (state agencies) under its purview to report the status of their IT systems and associated remediation efforts for the one or more projects comprising each system on a monthly basis. At the end of each quarter, the DOIT in turn reports to the administration and the Legislature. The DOIT’s quarterly report includes specific information about progress on individual remediation projects, such as the particular phase the project is in, and the planned completion date for the project as a whole. The quarterly report also separately tracks the progress of certain projects considered critical. Figure 1 illustrates the project phases necessary to ensure uninterrupted service delivery. An example of a critical application would be the item process- ing system at the State Treasurer’s Office. This system daily reconciles and redeems checks and warrants issued by state agencies, vouchers issued by the Department of Health Services to recipients of the Women, Infants & Children’s Program, and warrants issued by the State Controller’s Office. If the State Treasurer’s Office could not process these items, it would be unable to reimburse banks, which in turn may refuse to honor state food vouchers, checks, and warrants, thus depriving some Californians of needed services. The DOIT also oversees state agencies’ efforts to identify and fix year 2000 problems found in embedded technology. Embedded technology affects a wide variety of systems that use micropro- cessors or “chips” to control, monitor, communicate, or operate equipment. Some examples include fax machines, systems that control vehicle or aircraft traffic, security systems such as those controlling the gates and doors in prisons, and medical equip- ment such as pacemaker monitors. These systems may use chips that have been permanently encoded to calculate date informa- tion using two digits instead of four to designate a year. Any system relying on these chips to correctly determine the day of the week, properly calculate elapsed time, or otherwise distinguish between the 20th and 21st centuries may be at risk if the chips fail to make such calculations accurately. For example, if for security reasons an elevator is programmed to run at different intervals depending on the day of the week, it may totally shut down if it cannot recognize the year 2000. 2 C A L I F O R N I A S T A T E A U D I T O R FIGURE 1 Phases Necessary to Ensure Uninterrupted Service Delivery Parallel to all phases of the remediation Identify Projects with Year 2000 Problems O effort is the development of an N appropriate business-continuation plan G that ensures uninterrupted service O Project Assessment I delivery in the event that: . Determine the remediation strategy N Remediation efforts are not completed G by January 1, 2000 . Project Solution, Design, and Planning B The system fails despite remediation U efforts . Develop detailed plan with activities, costs, timelines S The system fails due to missing or I corrupted data supplied by external N data-exchange partners Project Development and Modification E S Execute the remediation plan S C Project Testing O N Thoroughly test the project, including: T . Testing internal interdependencies I . N Testing external data exchanges . U Testing in a simulated production environment, A using “time machine” or other suitable tests T . Testing for the effect of embedded chips I . O Testing the project for compatibility when it is a N part of a larger system, including tests of all external data exchanges . P Acceptance testing conducted by agency L program delivery managers A N N I Project Implementation N The project is integrated into the business use G Uninterrupted Service Delivery The elevator would not run until the computer chip controlling it is replaced with one that does recognize the year 2000. The DOIT provides guidance concerning the remediation needed for embedded technology, but only recently began requiring state agencies to report on this area. The DOIT’s first required report- ing was due July 31, 1998. C A L I F O R N I A S T A T E A U D I T O R 3 Finally, the DOIT also began tracking state agencies’ efforts to identify their data-exchange partners as of July 31, 1998. California’s state agencies exchange electronic data with thousands of organizations throughout the State, with other states, with the federal government, and with local govern- ments. The exchange of electronic data has the potential to corrupt the systems that state entities are now fixing to properly recognize 20th and 21st century dates. Unless all entities sharing data fix their systems by the year 2000, a single agency’s or entity’s system could prevent data exchange and could even cause linked systems to crash. SCOPE AND METHODOLOGY The Bureau of State Audits (bureau) is required by law to perform the annual single audit of the State of California. The single audit is designed to accomplish two goals. One is to render an opinion on whether the general purpose financial statements prepared by the State conform with generally accepted account- ing principles. The other is to review the State’s internal controls and compliance with state and federal laws and regulations. In conducting the single audit, the bureau frequently uses computer-generated information. Therefore, we assessed the potential risk the year 2000 problem poses to our ability to promptly fulfill our legal and professional responsibilities. Specifically, we reviewed what steps the State is taking to ensure year 2000 compatibility and whether the projects being remedi- ated related to those critical systems are on schedule to be fixed. In addition, we identified which of the State’s reported critical projects affect the single audit and if these projects are depen- dent on data received from others. Finally, we assessed if the State had established business-continuation plans in the event the projects are not fixed in time or fail to work as planned. To gain an understanding of the year 2000 problem and the steps the State is taking to ensure its readiness, we reviewed applicable laws, regulations, and other background information. To identify which systems could potentially affect the single audit, we used a questionnaire to survey the 39 state agencies that reported critical projects that are also either a system or a part of a system to the Department of Information Technology (DOIT) for the quarter ending March 31, 1998 (April quarterly report). Administrative and legislative officials use the quarterly reports to monitor the State’s year 2000 progress, and April was 4 C A L I F O R N I A S T A T E A U D I T O R the most current quarterly report available during our audit fieldwork. Our questionnaire asked agencies why they consid- ered the projects reported on as critical, if the projects affected the single audit, if the projects were dependent on data exchanges with others, and if the agencies had established business-continuation plans in the event the projects fail or are not finished in time. For the results of our survey, see Appendix A. To assess whether systems are on schedule to be ready for the year 2000, we analyzed the information agencies reported to the DOIT. In addition, we reviewed four separate agencies on-site to assess the accuracy of the information they reported to the DOIT. We also assessed whether the four agencies were complying with certain aspects of the governor’s executive order that went into effect in October 1997. Specifically, to determine whether the agencies were focused on solving their year 2000 problems, we verified that all new computer projects had been deferred, unless such projects were either mandated or had received an exemp- tion from the DOIT. We also sampled the purchases of computer software, hardware, and equipment made by the agencies since the executive order went into effect to verify that the contracts and purchase orders related to such items included language requiring they be year 2000 compliant. We found no reportable issues in these areas. The bureau will continue to oversee the State’s year 2000 progress. After we began fieldwork for this audit, the Joint Legislative Audit Committee approved an audit to determine year 2000 progress in the State. We anticipate that our future report will address issues concerning projects that are not considered critical; the progress of other state government entities’ efforts to overcome year 2000 problems, such as the University of California, that do not report to the DOIT; and the remediation status of embedded-chip technology. C A L I F O R N I A S T A T E A U D I T O R 5 Blank page inserted for reproduction purposes only 6 C A L I F O R N I A S T A T E A U D I T O R AUDIT RESULTS The State’s Reported Progress Is Overly Optimistic and Several Concerns Remain Unaddressed SUMMARY E fforts to rid almost 700 state computer projects of year 2000 problems in time to meet the new millennium may not be as far along as reported in the April 1998 quarterly report of the Department of Information Technology (DOIT). We therefore are concerned that completion dates for remedi- ation projects may be overly optimistic in some cases. Also, many state agencies have not addressed other aspects of the year 2000 problem. For example, none of the 39 agencies reporting on completed critical projects to the DOIT has rigorously time- tested their information technology (IT) systems, comprised of one or more critical projects, in an isolated environment that allows the resetting of the computer’s internal clock to make sure they will continue to function after the year 2000. In fact, many of these agencies have yet to even schedule such tests. While all critical projects may not need this type of testing, agencies should consider it for complex applications where significant health, safety, financial, or legal impacts would result from system failure. Furthermore, agencies are declaring some critical projects complete when they are not and are frequently allocating too little time for testing critical projects. Also, many agencies have not taken steps to protect their computer systems from data transmitted from other agencies that have not up- dated their systems to work after the year 2000. Finally, most state agencies do not have business-continuation plans that will assure continued delivery of critical services if their year 2000 remediation efforts fail. SOME AGENCIES OVERSTATED PROGRESS TOWARD RESOLVING YEAR 2000 PROBLEMS In a questionnaire sent to 39 state agencies that report on critical projects to the DOIT, we asked, among other things, if the status of their remediation efforts on 787 critical projects appearing in the DOIT’s April 1998 Quarterly Report on California’s Year 2000 Progress (April quarterly report) was C A L I F O R N I A S T A T E A U D I T O R 7 accurate. Of the 37 agencies that responded, 12 agencies indicated that their progress for 27 critical projects was different from that reported in the DOIT’s April quarterly report. We also visited four state agencies included in our survey and deter- mined that the quarterly report contained incorrect information on 15 of 37 (41 percent) projects we reviewed. Fourteen projects Over 40 percent of the were in an earlier phase of completion than indicated in the projects we sampled were April quarterly report, and one project was in a later phase than not in the phase of reported. See Appendix B for a comparison of the status of the completion reported to sample of projects we reviewed at the four agencies. the DOIT. Many were more than three months For example, we reviewed 12 projects at the Franchise Tax Board behind schedule. (FTB), including 10 of the 147 FTB projects listed in the April quarterly report as critical1 and 2 that, according to FTB officials, should have been included in the April quarterly report as critical projects. For 4 of the 10 projects, the phase of remedia- tion differed from the phase indicated in the April quarterly report—3 projects were in an earlier phase and 1 project was in a later phase. The phase of completion for the remaining 6 projects was accurately reported in the April quarterly report. Similarly, at the Department of Social Services we reviewed 12 projects reported as critical and found that 4 were in earlier phases of remediation than was reported in the April quarterly report and 8 were reported correctly. Because delays in specific phases could lead to delays in com- pleting the overall project, we first measured when the phase originally reported actually began. We next assessed what effect, if any, the delays may have had in the expected completion date of the overall project. As indicated in Table 1, for 11 of the 14 projects that were not as far along as reported in the April quarterly report, the phase originally reported was either not yet started as of July 31, 1998, or was delayed by more than three months. 1The FTB later revised its list of critical projects; in its June monthly report, the FTB indicated that 67 critical projects exist, not 147 projects as originally identified in the April quarterly report. 8 C A L I F O R N I A S T A T E A U D I T O R TABLE 1 Number of Months Elapsed Until the Projects’ Actual Status Matched the Status Reported in the DOIT’s April Quarterly Report Number of Months Until the Phase Reported Actually Began Number of Projects Where the Phase Was Not as Far Along as Reported Less Than 1 1 to 2 2 to 3 More Than 3 14 1 1 1 11 According to their June 30, 1998, monthly reports to the DOIT, all four of the agencies still believe that the overall completion date for 6 of the 14 projects will be the same or earlier than reported in the April quarterly report. For the remaining 8 projects, the agencies extended the respective completion dates for 5 projects from 1 month to 5.5 months, deleted 2 projects, and listed 1 project’s completion date as “undetermined.” Appendix C presents more detailed information on the number of additional months it took for the four agencies to begin the phase as reported by the DOIT for each project, and Appendix D compares the estimated completion dates in March and June 1998 for each project. It appears, based on the reported information as of June 1998, that the overstated progress in the April quarterly report has not affected the anticipated project completion dates for six of the projects. But after assessing the accuracy of state agencies’ short- term predictions, we have concerns that the estimates reported for completing the remediation efforts on many of the State’s critical projects may prove to be overly optimistic. Agencies Often Require More Time to Complete a Project Than Initially Predicted We compared estimates made by agencies in March 1998 about which projects would be completed in April, May, or June 1998 to subsequent monthly reports. In 58 (54 percent) of 107 instances, the agency’s short-term prediction about when a project would be completed turned out to be too optimistic. Table 2 illustrates our findings that actual completion dates were later than predicted for over half of the projects. C A L I F O R N I A S T A T E A U D I T O R 9 TABLE 2 Most Projects Scheduled for Completion by June 1998 Were Not Completed as Planned Month Actually Completed Estimated Month Number March Not Complete of Completion of or as of as of March 31, 1998 Projects Earlier* April May June June 30, 1998 April 1998 28 1 8 2 11 6 May 1998 21 1 1 7 2 10 June 1998 58 3 3 3 22 27 Totals 107 5 12 12 35 43 *It appears that the agencies reporting on these five critical projects had already completed them when they submitted estimated completion times to the DOIT on March 31, 1998. For example, although the agencies’ March 1998 predictions were that 28 projects would be completed in April 1998, only 8 projects actually were. One project was completed early, but 13 others were completed in May and June, and as of June 30, 1998, 6 were still incomplete. Similarly, in March 1998, agencies predicted that 58 projects would be completed in June 1998; however, only 22 were actually finished in June, and 27 were incomplete as of the end of June 1998. AGENCIES ARE NOT GIVING ENOUGH ATTENTION TO THE TESTING PHASE Testing a system to make sure it works as expected is a crucial phase in solving the year 2000 computer problem. The State’s two data centers have set up an isolated computer environment that allows the computer’s internal clock to be set to different dates so that agencies can find out if their systems will recognize dates after January 1, 2000, but thus far only 19 systems have been scheduled for such testing by the governor’s December 31, 1998, deadline. We also have concerns about whether projects that have been reported as complete truly are, and whether the amount of time agencies are budgeting for any type of testing is sufficient. For example, according to the information reported to the DOIT, 151 critical projects have fewer than 30 days allocated for testing to assure that systems will continue to function in the year 2000. 10 C A L I F O R N I A S T A T E A U D I T O R “Time Machine” Testing May Be Difficult to Schedule as More Projects Near Completion State agencies have not tested any of the 60 critical projects reported as complete in an isolated computer environment to determine if the computer systems they are associated with would recognize 20th and 21st century dates, according to our follow-up survey and information supplied by the State’s two statewide data centers. We queried 10 state agencies that reported during our initial survey that they had completed remediation efforts on 60 critical projects about how they had tested them. According to these agencies, 24 projects employed one of the State’s two statewide data centers, with agencies reporting that 23 of these projects were tested in a production environment that included date testing. For the remaining project using a statewide data center, the agency stated that even though it had reported the project complete, the agency had reopened the project and that recoding and testing were under way. In contrast to claims made by these agencies, the State’s two data centers’ information indicates that none of these 24 In contrast to agency projects or their associated systems were tested in the data claims, none of the 24 centers’ isolated computer environment that allows the resetting projects reported as fully of the computer’s hardware clock (also called a “time machine”). remediated were tested Instead, these agencies used a software simulation that is only in the data centers’ able to test the project itself, not how it integrates with other rigorous “time machine” computer applications and systems. This means that the testing environment. performed on the 24 projects was likely to have been less rigor- ous than if time machine testing had been performed. For the remaining 36 projects, 28 belong to the FTB, which operates its own data center; 6 do not use either of the two state data pro- cessing centers; and two projects did not require testing because they represented consultant contracts to assist the agency in identifying, testing, and tracking its year 2000 projects. Neither the 28 critical projects tested at the FTB, nor the 6 projects that did not use a data center were tested through the use of a time machine. Many critical computer applications are run at one of the two state data processing centers: the Stephen P. Teale Data Center (Teale) or the Health and Welfare Data Center (HWDC). Both Teale and HWDC offer computer software that can simulate year 2000 conditions as well as a special isolated computer C A L I F O R N I A S T A T E A U D I T O R 11 environment, called a time machine, as ways that agencies can test their critical projects. Using the data centers’ software, agencies can simulate an environment where the computer projects’ applications are “tricked” into executing in the future through the use of software that simulates future dates (we have referred to such testing as date-simulation testing). Time ma- chine testing is conducted in an isolated environment where the computer’s internal clock is manually set to a date in the future. Time machine testing, unlike date-simulation testing, assures that not only the computer application but also all of the oper- Time machine testing, ating system software operates correctly on the future date set in unlike less rigorous date- the time machine. Time machine testing allows clients to exten- simulation testing, sively test systems comprised of one or more critical projects in a assures that all applica- production environment where the computer’s internal clock tions and operating can be physically set to a variety of different dates occurring system software function both before and after the year 2000. Based on the literature we correctly when set to researched in the area of testing remediation efforts, we believe current and future dates. that each state agency should consider the need for time machine testing, especially for critical projects that are part of systems using complex applications supported by data centers where significant health, safety, financial, or legal ramifications would result from system failure. We contacted both Teale and the HWDC to determine how many of their clients had already done or were planning to do time machine testing. Teale indicated that it has over 100 state clients and that as of July 31, 1998, one client (not among the 10 agencies we followed up with) had used its time machine testing facilities. The HWDC indicated that it has over 40 clients, but to date, only one had used its time machine testing facilities. However, the system tested by the HWDC’s client was not one that included a critical project in the April quarterly report. It therefore was not one we inquired about. While both data centers conceded that time machine testing could be important in ensuring that critical systems will continue to function, the ultimate decision to use such testing is left to the client. We also asked both Teale and the HWDC how many of its clients had reserved time for such testing. The HWDC has been proactive in scheduling testing time, and so far has reserved time for 5 of its clients to use its time machine between August 1998 and December 1998, and 4 clients are scheduled to test systems between January and December, 1999. Teale does not actively schedule its clients; therefore, as of July 31, 1998, only 3 state agencies have requested time machine access, and all testing will occur by the end of 1998. 12 C A L I F O R N I A S T A T E A U D I T O R According to the DOIT’s March 31, 1998, database, 104 critical projects have scheduled completion dates between October 1, Currently, only 19 systems 1998, and December 31, 1998. We do not know how many of are scheduled for time these projects are part of systems that require time machine machine testing before testing, but because they are designated as critical projects, we December 31, 1998, even presume that a significant number are part of systems that though 104 critical would benefit from such testing. However, as of July 31, 1998, projects are scheduled to only 19 systems are currently scheduled for time machine be completed by that testing before the end of the year at either Teale or the HWDC, date. and we question whether all necessary testing can be accom- plished for the remaining systems comprised of one or more critical projects prior to the governor’s December 31, 1998, deadline. We are concerned that in an effort to meet the governor’s deadline, some agencies planning to use time machine testing in remediating their systems may be unable to schedule access as time machine testing opportunities become scarce. Moreover, based on the results of our survey and the information provided us by Teale and the HWDC, we are also concerned that none of the 60 critical projects reported by agencies as complete were part of systems that used time machine testing because some computer systems are complex and time machine testing might reduce the possibility of system failure in the year 2000. Some “Completed” Projects Are Not Really Complete According to the information received in our follow-up survey, there were four projects that had data-exchange partners but the data files for all such partners had not been tested. Further, the testing for two other projects did not include testing dates in both the current and future century. Based on these examples, we have concerns that some projects designated as “complete” are not really complete according to criteria for completion issued by the DOIT in its April quarterly report and shown on the next page. While the DOIT’s criteria were directed at com- puter systems, they should also apply to critical projects that are themselves systems or are part of a system. If agencies cannot answer “yes” to all these questions, systems or critical projects comprising the system should be designated as incomplete. Moreover, if all these factors are not considered and addressed, systems and projects designated as complete may still be at risk for failure because they have not been thoroughly tested. C A L I F O R N I A S T A T E A U D I T O R 13 Some Agencies May Not Be Allocating Enough Time for Thorough Testing We are also concerned that agencies may not be allocating enough time to thoroughly test their projects. According to June 1998 monthly progress reports compiled by the DOIT, state agencies have 151 critical projects for which they have allocated fewer than 30 days for testing. While some projects could possi- bly be tested in fewer than 30 days, we question whether large, complicated projects could be thoroughly tested in such a short time, especially when industry experience is showing that testing is consuming between 50 percent to 70 percent of total Factors to Consider Before Designating a System project time and resources. or Project Complete By comparing the estimated (cid:214) Is the project year 2000 compliant? completion date for the devel- (cid:214) Have repaired and replaced components been tested opment and modification phase successfully in an environment simulating dates in both the (the phase immediately preced- 20th and 21st centuries and operated in the same production ing the test phase) to the environment as the agency’s normal business functions? completion date for testing (cid:214) Are data exchange protocols established to prevent listed in the June 1998 database corruption of the agency’s systems by external sources? maintained by the DOIT, we (cid:214) Have all obligatory special testing requirements, such as determined the amount of time testing in a stand alone environment where the hardware agencies had allocated to testing clock has been moved forward into the 21st century (time projects. According to the data, machine testing), been met? 16 of the projects had no time (cid:214) If the state agency’s system is comprised of one or more at all allocated to the testing critical projects having an independent quality assurance or phase. Of the 151 projects for year 2000 validation process, has the system successfully negotiated that process? which fewer than 30 days elapsed between the completion (cid:214) Are there no additional year 2000-related upgrades or of the two phases, 31 had enhancements to specific hardware, software or micro code required? estimated remediation costs exceeding $100,000 each, with the estimated remediation costs of 7 projects exceeding $500,000 each, indicating that these are larger, more complex projects. We contacted three agencies, each responsible for one of the above projects, and asked them why their respective remedia- tion plans included so little time devoted to testing. For ex- ample, two of the projects included in the DOIT’s June 1998 database indicated that no hours had been set aside for testing, while another project showed a minimal amount of testing time. When we further probed into these projects, however, the two with zero test time indicated that testing had been 14 C A L I F O R N I A S T A T E A U D I T O R performed and, in one case, more was planned. The agency remediating the third project stated that the nine days of testing time shown in the DOIT’s database was correct, but that addi- tional testing of the project was planned. Specifically, for one of the Department of Water Resources’ (DWR) projects with an estimated remediation cost of $320,000, One project we reviewed the DOIT’s June 1998 database indicated that zero hours were budgeted only 17 percent allocated for the testing phase, but one of DWR’s staff told us of its total remediation that was not true. A systems analyst told us the purpose of the time for testing—well project is to upgrade the operating systems for the remote below the 50 percent to terminal units at approximately 100 hydraulic sites located 70 percent industry along the California Aqueduct to make them year 2000-ready. standard. These remote terminal units record water levels and control gate settings along the aqueduct. Although no testing hours were indicated in the DOIT’s database, the system analyst stated that the DWR has tested the new upgraded systems in a Sacramento software laboratory, and the terminals will be tested as they are installed, and that such installations began August 1, 1998. The analyst further stated that the DWR plans to spend about 700 hours total in testing the terminals, which is about 17 percent of the project’s total remediation budget of over 4,200 hours. In addition, the analyst told us that the completion date for the testing phase of the project will be nearly a year later, in December 31, 1998, not January 1, 1998, as indicated in the DOIT’s June database. While the amount of hours allocated to testing (17 percent) may be appropriate for this project, it is well below the industry average of 50 percent to 70 percent for the testing phase for information technology projects. For the other project with an estimated remediation cost of $480,000 and showing zero hours dedicated to testing, we talked to a staff programmer analyst at the Department of Insurance (DOI) who told us that the purpose of the License Information System (LIS) replacement project is to consolidate all DOI mission-critical systems on the Oracle relational database man- agement system, and ensure that the LIS is year 2000-compliant. The analyst stated that the DOIT’s year 2000 monthly report for June shows zero time devoted to testing because unit testing is not a discrete element of application development and modifica- tion; unit testing is a repetitive process and is completed as code is built. According to the analyst, this technique ensures that the end product is code that meets specifications and that leaving testing until the end would mean that thousands of lines of code and hundreds of application modules would be generated and, most probably, none of it would work. C A L I F O R N I A S T A T E A U D I T O R 15 Using the DOIT’s database, we calculated that half of the devel- opment and modification phase equates to roughly six weeks, or One department about 33 percent of the project’s total remediation budget of conducted testing 18 weeks. While the 33 percent of the project time the DOI concurrently with system spent testing seems reasonable, the fact that two of the project’s development and phases were run concurrently is troubling. If two phases are modification—essentially being conducted at the same time, it may indicate that there testing a moving target. were problems with the earlier phase. Moreover, it means that the DOI was testing a phase at the same time it was being devel- oped. In other words, the DOI was testing a moving target. We believe that to properly remediate a project, a linear progression should be followed through each phase. Accordingly, the testing phase is meant to test whether the project’s development and modification phase was successful in identifying and correcting year 2000 problems. Although unit testing is important, testing of the whole system and its interrelationships with other systems is vital. We also contacted the Department of Motor Vehicles (DMV) regarding a project with an estimated remediation cost of $595,000 and allocated time for testing of nine days. The DMV’s year 2000 project manager said this particular project is part of a larger mainframe system responsible for processing vehicle registration transactions. The project manager said that the nine days of testing time reflected in the DOIT’s June database was correct, and was the amount of time spent testing 4 of 34 mod- ules. (The total time from the start of modifications to the end of testing of the 34 modules was over 5 months.) The project manager explained that the DMV used date-simulation software to determine whether the system would process dates beyond January 1, 2000, and future leap years correctly, and that this form of testing takes less time. The project manager also said that, while this particular project was already complete, the DMV intends to further test the project using the Teale time machine in late 1998 and 1999, although specific dates have not yet been set. Further, we were told the time machine testing was not included in the original remediation plan reported to the DOIT because the DMV considers such testing to be an ongoing test to ensure the system is year 2000-ready. Although the DMV reported that the test phase for the project was complete, we question how testing could be considered complete if the testing did not include how the project would work in the larger system of which it is a part. Apparently, the DMV also recognized the need for this type of testing and is now intending to use the Teale time machine facility. This example demonstrates why 16 C A L I F O R N I A S T A T E A U D I T O R nine days of test time is insufficient for a project that took a full year to remediate and that is an integrated part of a larger system. If projects are not thoroughly tested, agencies increase the risk that critical systems will fail and, in turn, interrupt services vital to California’s citizens. Thorough testing should consider all of the elements illustrated in Figure 1 on page 3: internal inter- dependencies, external data exchanges, time machine or appropriate simulation testing, the impact from embedded systems, and finally, full acceptance testing by the agency managers responsible for the business functions supported by the computer projects. AGENCIES NEED TO ENSURE THAT DATA FROM EXTERNAL ENTITIES ARE YEAR 2000-READY External data-exchange partners that share data that is not year 2000-compatible (corrupted data) with the State’s critical Although many state computer systems is another area that could disable the State’s agencies’ critical systems delivery of necessary services. Most agencies exchange electronic rely on data exchanges data with other state agencies, governments, and private sector with other entities, not organizations. While we found that many of the State’s critical all have taken the steps computer systems depend on data exchanges with other entities, necessary to ensure data not all agencies have completed the necessary steps to ensure received will work seam- that the data received will work seamlessly with the State’s lessly with state systems. computer systems into the next century. Even if agencies successfully fix all their critical computer systems, they still may not be able to deliver essential products and services in the next millennium if their data-exchange partners’ systems are not ready for the year 2000. In its April 1998 quarterly report, the DOIT listed the State’s critical computer systems by agency, but it did not require agencies to report which of these critical systems relied on interfaces with external data-exchange partners. To help assess the potential impact of external data interfaces on the State’s remediation efforts, we included questions on data exchange in our survey of 39 state agencies, and we also reviewed this area during our on-site visits to 4 of those agencies. See Appendix A for the results of our survey. C A L I F O R N I A S T A T E A U D I T O R 17 Our survey indicated that 30 of the 37 responding agencies (81 percent) operate one or more critical projects that depend on external data. A total of 207 (38 percent) of the 551 projects confirmed by state agencies as critical to their respective Eighty-one percent of missions are associated with systems having external data- agencies responding to exchange partners. our survey have one or more critical projects We also asked surveyed agencies to tell us why they considered dependent on data their systems to be critical and what the negative effects would exchanged with external be if the data interfaces with other systems failed. For instance, entities such as counties, the Department of Health Services said that its Medi-Cal the federal government, Eligibility Data System (MEDS) provides health care for millions or the private sector. of Californians and is needed for the payment of medical claims to thousands of state health care providers. The MEDS relies on the receipt and exchange of data from 58 counties, various medical service providers, and several state and federal govern- ment agencies to determine eligibility and to pay for medical health services. The failure of the MEDS would stop the work of the counties and service providers and disrupt health care to millions of Californians. From our site visits, we learned that all four of the agencies had a reasoned approach for identifying their data-exchange part- ners. However, while two of the four agencies (the Board of Equalization and the Department of Social Services) had identi- fied their data-exchange partners, they had not yet determined other crucial factors that ensure their systems will not be affected by external data. Although we only visited 4 of the 39 agencies reporting critical systems in the DOIT’s April 1998 quarterly report, we are concerned that many of the remaining 35 agencies may also not have ensured that all external data they need to keep their critical systems functioning are year 2000-compatible. The DOIT Only Recently Began Requiring Agencies to Report on Data-Exchange Partners Initially, the DOIT advised only that state agencies identify their respective data-exchange partners and locate a contact person who could answer questions about data interfaces. To facilitate communication, the DOIT established a web site to record this information, but agencies were not required to report any information regarding data exchange to the DOIT for inclusion in its quarterly report. 18 C A L I F O R N I A S T A T E A U D I T O R In June of 1998, the DOIT published a position paper on the subject of external data-exchange partners describing the chal- lenges the year 2000 problem presents for state agencies depen- dent on data exchange. The DOIT offered an approach for developing and managing a plan to minimize the inherent risks in sharing date-dependent data. The approach outlines California’s data exchange policies pertaining to dates. These policies include a standard date format to be used in exchanges of data, how to determine who is responsible for establishing the date content and format and implementation timetable, and the conventions to be used in scheduling and testing shared data. The DOIT position paper also required that as of July 15, 1998, all state agencies exchanging computer data with systems out- The DOIT recently began side of their control—referred to as external data exchanges— requiring agencies to identify the date format used by each data-exchange partner, report on the total establish a contact person with each partner, develop schedules number of data-exchange for testing and implementing the date formats used by partners, partners, the number of and provide notification to all data-exchange partners of the exchanges affected by the implications of their failure to meet the State’s schedule. year 2000 problem, and how many of these In addition, as part of their monthly reports, beginning July 31, systems have been tested. 1998, state agencies reporting to the DOIT must report on the total number of data-exchange partners, how many of the data exchanges are affected by the year 2000 problem, and how many have been tested and implemented. We applaud these recent efforts by the DOIT to provide specific direction to state agencies in addressing the potential risks data interfaces present to the success of the State’s year 2000 remediation efforts, and we agree with the above requirements. We believe that the requirement that state agencies test the external interfaces with their exchange partners is vital to the State’s remediation efforts of the year 2000 problem. Adequate testing of these interfaces will help ensure that state agencies can continue their core business processes and ensure uninterrupted products and services for Californians. Few State Agencies Have Business-Continuation Plans for Year 2000 Problems Despite the substantial effort by state agencies to fix their critical systems, agencies remain vulnerable to the disruption of their business processes if their remediation efforts are unsuccessful or are not completed in time to meet the new century. Because C A L I F O R N I A S T A T E A U D I T O R 19 most state agencies are highly dependent on information technology to carry out their core business functions, year 2000- Despite substantial induced failures or delays in fixing one or more critical agency efforts to fix their computer systems may severely impact an agency’s ability to critical systems, they deliver essential products and services. Here are some examples remain vulnerable should of what might happen if critical systems either fail to work or remediation efforts fail. are not fixed by January 1, 2000: • The FTB taxpayer information system houses approximately 14 million individual taxpayer accounts. Its planned year 2000 modifications to this date-driven system are extensive. System failure or delay could result in improper interest calculations, payment applications, and refund amounts for millions of Californians. • The Statewide Automated Welfare System (SAWS) is a critical system operated by the Health and Welfare Agency Data Center. The failure of or delay in fixing this system could interrupt the payment of millions of dollars earmarked to provide support for needy families throughout California. • The Department of Water Resources depends on its dispatch system for scheduling the delivery of water and power. If this system fails or is not fixed before the year 2000, it could cause a complete shutdown of the State Water Project and result in water and power shortages affecting millions of Californians. • The State Teachers’ Retirement System uses its enterprise database to store the retirement accounts of thousands of California teachers. This system also receives and sends information to all 58 county offices of education, insurance companies, and various banking partners. Failure or delay in remediating this system could cause miscalculated retirement benefits and interrupted payments to teachers and beneficiaries. The risk that systems will fail or will not be fixed in time is not limited to just the critical computer systems themselves. As discussed earlier, failures and delays can also be caused by state agencies’ data-exchange partners. Many of the State’s critical computer systems depend on data received from outside busi- ness partners. An otherwise fully remediated computer system may fail to operate, fail to produce error messages, or generate incorrect data if it receives data that is not year 2000-compatible from one of its exchange partners. State agencies also depend on 20 C A L I F O R N I A S T A T E A U D I T O R a variety of services provided by the public infrastructure— including power, water, transportation, and voice and data communications. Any interruption in either data exchanges or supportive services because of year 2000 problems could be disastrous to state agencies’ abilities to provide critical services to Californians. Because of these risks, agencies must start business-continuation Because the alternatives planning now to reduce the risk of year 2000 business failures or to avoid service delays. This effort must address potential failures and delays of interruptions caused by others, including data-exchange partners and infrastructure year 2000 failures or service providers. One weak link in the chain of critical depen- delays often require long dencies, and even the most successful remediation efforts spent lead times, it is vital that on a year 2000 program could fail, and major disruptions of business-continuation business operations could result. planning start now. Planning for these types of risks must be done early because of potential long lead times. For example, performing a business function manually normally either requires hiring and training new staff or redirecting current staff. Likewise, the decision to contract out the work normally requires lengthy negotiations. Therefore, specific date triggers must be developed for deploying business-continuation plans, and deployment schedules need to be completed for such alternatives to be used successfully. Plans also need to include enough lead time for testing to see if they are workable, capable of providing the desired results, and can be implemented within the required time periods. Such plans require agencies to act now and not wait until remediation efforts are complete to begin this important task. The DOIT Does Not Require Agencies to Report on Business-Continuation Plans While the DOIT appears to be aware of the importance of business-continuation planning, it has not yet required state agencies to report on the status of their planning efforts. There- fore, in our survey of state agencies reporting critical systems, we asked them to tell us whether they had established business- continuation plans for each of their critical computer systems. By definition, these systems support core business processes that deliver the agencies’ essential products or services. Of the 37 responding agencies surveyed, 25 (67 percent) reported they had no business-continuation plans for systems associated with a total of 338 critical projects, or 61 percent of all projects identi- fied. They therefore cannot ensure that delivery of critical C A L I F O R N I A S T A T E A U D I T O R 21 products and services would continue if these systems failed to Twenty-five of 37 operate or were not fixed on January 1, 2000. See Appendix A agencies responding to for the results of our survey. our survey do not have business-continuation The reasons agencies gave for not having plans varied. In some plans in the event their cases, agencies indicated that business-continuation plans were systems fail or are not needed because remediation of their critical systems would delayed because of be completed well before the next millennium. Although this year 2000 problems. explanation may be appropriate in some very limited instances, we believe it is prudent for all agencies to identify the steps they would take—including using manual processes or contracting with outside vendors for the services—in the event year 2000 problems cause one or more of their critical systems to fail or their remediation efforts are delayed. In other cases, agencies stated that they had limited resources to address business-continuation plans for the year 2000 problem. They also pointed out that information technology staff are too busy working on fixing the year 2000 problem to develop business-continuation plans until after remediation efforts are complete. We agree that the highest and best use of skilled programmers is to have them continue to work on fixing the code of the agencies’ critical systems. However, we do not think it is appropriate that agencies wait until remediation of systems is completed to begin developing business-continuation plans. To the contrary, we believe the people best suited to initiate business continuity planning are the business managers who are ultimately responsible for, and most familiar with, all the agency’s core business processes. It is the business manager who will be called on to explain if core business processes are inter- rupted and delivery of essential products or services is halted. According to a United States General Accounting Office (GAO) publication, the year 2000 problem is primarily a business problem. While the publication is technical in nature, it indi- cates that management needs to be fully aware of the potentially devastating financial, organizational, and political consequences of failure or delays of one or more critical information systems. Managers responsible for the agencies’ core business processes need to establish a work group and dedicate sufficient resources to mitigate the risk posed by critical system failures or delays in remediation efforts. The GAO model for planning suggests this work group include representatives from the agency’s major business units, experts in relevant functional areas, 22 C A L I F O R N I A S T A T E A U D I T O R business-continuation and disaster-recovery specialists, opera- tional analysts, and contract specialists. The GAO model does not include programmers. Agencies’ Existing Contingency Plans May Not Cover Year 2000 Problems Based on our survey, 26 of 37 responding agencies reported that they had business-continuation plans covering a total of 204 critical computer systems. However, we are concerned that these plans are not designed to assure business-continuation under We are concerned that year 2000 circumstances. For example, several agencies stated many agency business- that the business-continuation plans for their critical computer continuation plans only systems for the year 2000 were addressed in existing operational address recovery from a recovery plans. This was also the assertion made at the four disaster, such as fire or agencies we visited. When we reviewed the operational recovery flood, and will be ineffect- plans at these agencies, however, we found that none of them ive in addressing year identified core business processes or addressed how any of the 2000 computer failures. four agencies would continue their respective business processes if critical systems either fail or are not fixed by January 1, 2000. Instead, these plans mostly addressed how the agency would recover data processing functions at a designated alternative site if struck by a natural disaster, such as earthquake or flood. System-recovery planning using backup systems in the event of a disaster were not intended nor designed to address the year 2000 problem. For example, if an agency’s system fails because of year 2000 problems, its backup system would fail as well. We believe that many other agencies that responded to our survey indicating their business-continuation plans for year 2000 were contained in their operational recovery plans also may have not developed an adequate plan for handling year 2000 problems. We subsequently sent a letter to each of the four agencies we reviewed asking what strategies they had for resuming business operations if remediation efforts are not completed by January 1, 2000; remediation is believed to be complete but such efforts fail; or remediation is complete, but data-exchange partners cause system failure. We asked the four agencies to provide such strategies for the 37 critical projects we reviewed. For example, the Franchise Tax Board (FTB) stated that it was in the process of developing a business-resumption plan focusing on critical business activities associated with the receipt and payment of tax remittances. The agency expects to complete its business-continuation and operational recovery plans by late C A L I F O R N I A S T A T E A U D I T O R 23 1998 or early 1999. However, its emphasis is on early remedia- tion of essential systems and business functions. To avoid failing to complete remediation by January 1, 2000, the FTB stated it is using a project-management plan that includes early warnings on failures to meet completion dates, and it is prioritizing resources. If remediation efforts fail, the FTB plans to perform regression testing in its 21st century testing environment and deploy rapid-response maintenance teams to fix critical projects. Finally, the FTB plans to counter problems with data-exchange partners by identifying all partners and documenting the strategies, agreements, and testing to be used for each one; testing high-risk system interfaces; and developing and testing bridge programs. In a further example, the State Treasurer’s Office (STO) stated that, according to its strategy, if remediation efforts are not completed on its four critical projects by January 1, 2000, it will purchase another proprietary software package from a vendor to replace one project, outsource processing for another project, and manually process a third. Remediation is already complete for the fourth. If remediation efforts fail, the STO stated it would manually perform the functions for all four of its projects, including the hiring of temporary staff. If data exchanges cause the STO’s projects to fail, the STO stated it would manually perform the functions for two projects and require their data- exchange partner to re-create the functions of the third, as it is not date-dependent. The fourth requires no action because it has no data-exchange partners. Finally, the STO stated that opera- tional-recovery plans are necessary for all four projects and that two will be completed by the end of December 1998, another will be completed in March 1999, and the fourth has already been completed by Teale. CONCLUSION Efforts to rid almost 700 critical state computer projects of year 2000 problems in time to meet the new century may not be as far along as reported in the April 1998 quarterly report of the DOIT. Also, many state agencies have not addressed other facets of the year 2000 problem. For example, to date no state agencies reporting completed critical projects to the DOIT have rigor- ously time-tested their critical information technology systems in an isolated environment by physically setting the computer’s internal clock to dates set in both the 20th and 21st centuries to make sure they will continue to function after the year 2000; 24 C A L I F O R N I A S T A T E A U D I T O R and many others remediating large, complex systems have yet to schedule such tests. Many agencies have not taken steps to protect their computer systems from corrupted data transmitted from other entities that have not updated their systems to work after the year 2000, such as counties or the federal government. In addition, most state agencies do not have business-continuation plans that will assure uninterrupted delivery of critical services to Californians if their year 2000 efforts fail. RECOMMENDATIONS To ensure uninterrupted delivery of essential products and services to Californians, the Governor’s Office should ensure that all state agencies take the following steps: • Provide the Department of Information Technology (DOIT) with accurate information about the status of their year 2000 remediation efforts. Specifically, the estimated completion dates for each phase of remediation, including final comple- tion, should reflect the agency’s best estimate for the actual completion dates and should be updated whenever circum- stances affecting a project’s status change. • The remediation for each critical project should be thor- oughly and comprehensively tested. For larger, complex projects associated with systems that support the delivery of services to Californians where interruption would be unacceptable, agencies should also consider testing the system in an isolated computer environment using a time machine. Moreover, prior to declaring a project complete, tests of any internal interdependencies, external data exchanges, 20th and 21st century date recognition, and the impacts from embedded systems such as desktop computers, should be complete and the project acceptance tested and approved by agency managers responsible for the business functions. • Protect their computer systems from missing or corrupted data supplied by external parties. Specifically, agencies should identity their data-exchange partners, develop sched- ules for testing and implementing new date formats, and thoroughly test data supplied by external parties. C A L I F O R N I A S T A T E A U D I T O R 25 • Establish business-continuation planning groups, made up of managers from major business units, experts in relevant functional areas, business-continuation and disaster-recovery specialists, operational analysts, and contract specialists. These planning groups should then follow a structured approach to develop a business-continuation plan for each core business process and infrastructure component affected by the year 2000 problem. In addition, to ensure that the administration and the Legislature have accurate information about state agencies’ progress toward fixing their critical systems threatened by year 2000 problems, the DOIT should do the following: • Continue to collect and analyze information state agencies provide on their overall progress. If, after analyzing the reported information, something appears anomalous—such as too little test time—contact the agency for an explanation. • Continue to collect information from agencies on their data-exchange partners. In addition, take appropriate follow- up action if it appears that agencies are not testing their interfaces with data-exchange partners. • Require agencies, as part of their monthly reporting, to indicate whether they have business-continuation plans that ensure that each core business function will continue uninterrupted in the event that the critical computer systems supporting those functions fail to work or are delayed because of year 2000 problems. 26 C A L I F O R N I A S T A T E A U D I T O R We conducted this review under the authority vested in the California State Auditor by Section 8543 et seq. of the California Government Code and according to generally accepted governmental auditing standards. We limited our review to those areas specified in the audit scope section of this report. Respectfully submitted, KURT R. SJOBERG State Auditor Date: August 27, 1998 Staff: Douglas Cordiner, Audit Principal Bill Shepherd, CPA Douglas Gibson, CPA Reed M. McDermott, CPA Gayatri Patel Michelle J. Tabarracci, CISA Jian Wang C A L I F O R N I A S T A T E A U D I T O R 27 Blank page inserted for reproduction purposes only 28 C A L I F O R N I A S T A T E A U D I T O R APPENDIX A Survey of 39 State Agencies T o identify which systems could potentially affect the single audit, we surveyed the 39 state agencies that reported critical systems to the Department of Informa- tion Technology (DOIT) for the quarter ending March 31, 1998 (April quarterly report). Our questionnaire asked the agencies if their systems affected the single audit, if their systems were dependent on data exchanges with others, and if they had established business-continuation plans in the event their systems failed or were not finished in time. We also asked whether each project’s remediation status, as reported in the DOIT’s April quarterly report, was accurate. As Table 3 shows, the 39 agencies listed 787 critical projects. We did not receive a response from one agency, and another submit- ted information that we considered noncomparable because it related to July 1998 instead of March 1998. These two agencies are responsible for a total of 107 critical projects shown in the April quarterly report, leaving 680 critical projects. Moreover, in their responses to our survey, some agencies added critical projects not listed in the April quarterly report and deleted others no longer considered critical. After adjusting for these changes, we obtained information about 551 critical projects. In addition, we asked the 39 agencies two questions related to a project’s effect on the single audit. First, we asked whether a project affected the agency’s financial statements. Second, we asked whether a project would affect the ability of the agency to claim federal funding or comply with federal regulations. As the table shows, agencies responded that 292 of 551 projects (53 percent) affected their financial statements and 156 (28 percent) affected their ability to claim federal funding or comply with federal regulations. The respondents further indicated that 207 of 551 projects (38 percent) involved data exchanged with other state agencies, other governments, or other entities. Finally, for 338 of 551 projects (61 percent), the respondents stated that they did not have business-continuation plans to ensure that critical services continued in the event remediation efforts failed or were delayed. C A L I F O R N I A S T A T E A U D I T O R 29 TABLE 3 Results of Our Survey of 39 State Agencies Critical Projects per the Financial Federal Data Business Projects DOIT’s Crital Crital Statement Funding Exchange Continuation Where April Projects Projects Impact Impact Partners Plans Status Quarterly (+) (-) (as Was Agency Name Report Added Deleted Adjusted) Yesa Noa Yesa Noa Yesa Noa Yesa Noa Different Building Standards Commission 3 + 0 - 3 0 0 0 0 0 0 0 0 0 0 Board of Control 2 0 0 2 1 1 1 1 1 1 0 2 0 Board of Equalization 20 0 5 15 10 5 0 15 4 11 11 2 0 California Highway Patrol 6 0 0 6 1 5 1 5 5 1 6 0 1 Consumer Affairs 6 0 1 5 3 2 0 5 4 1 5 0 2 Corporations 3 0 1 2 1 1 0 2 0 2 2 0 1 Corrections 28 1 6 23 0 23 3 20 6 17 20 3 3 Criminal Justice 11 0 6 5 0 5 3 2 0 5 0 5 0 Developmental Services 18 0 0 18 1 5 2 4 3 9 2 11 0 Education 11 0 0 11 2 9 1 10 4 7 9 2 0 Employment Development 80 0 0 80 55 21 44 31 42 36 0 80 0 Fish and Game 4 0 0 4 1 3 2 2 2 2 4 0 0 Food and Agriculture 22 0 3 19 17 2 2 17 4 15 9 10 0 Forestry 19 5 3 21 12 9 0 21 6 15 21 0 0 Franchise Tax Board 147 36 116 67 51 16 1 66 18 49 12 54 13 General Services 17 2 15 4 2 2 2 2 1 3 1 3 0 Health and Welfare Data Center 18 0 2 16 3 13 5 11 2 14 11 5 3 Health Services 4 11 0 15 15 0 15 0 10 5 3 12 2 Housing and Community Development 25 0 1 24 7 17 1 23 1 23 3 21 0 Industrial Relations 7 0 2 5 2 3 0 5 3 2 5 0 3 Justice 18 0 0 18 0 18 0 18 16 2 0 18 0 Lottery 11 0 1 10 10 0 10 0 7 3 7 3 4 Mental Health 9 0 3 6 5 1 1 5 2 4 6 0 0 OSHPD 24 0 0 24 2 22 0 24 13 11 0 24 0 Parks and Recreation 4 0 0 4 0 4 0 4 0 4 3 1 0 Pesticide Regulation 22 1 15 8 1 7 0 8 0 8 0 8 0 Real Estate 3 2 1 4 0 4 0 4 0 4 4 0 1 Rehabilitation 13 1 5 9 7 2 2 7 2 7 1 8 0 Social Services 38 1 1 38 19 19 18 20 32 6 38 0 0 State Teachers’ Retirement 1 0 0 1 1 0 0 1 1 0 1 0 0 State Treasurer’s Office 8 0 0 8 8 0 1 7 4 4 2 6 0 State Water Resources Control Board 6 0 0 6 2 4 0 6 0 6 4 2 2 Toxic Substance Control 3 0 0 3 0 3 0 3 1 2 0 3 0 Transportation 46 0 0 46 34 12 29 17 9 37 0 46 0 Veterans Affairs 3 0 0 3 3 0 1 2 1 2 0 3 2 Water Resources 19 1 0 20 16 3 11 9 2 18 14 6 3 Youth Authority 1 0 0 1 0 0 0 0 1 0 0 0 0 Total (37 of 39 responded) 680 +61 -190 551 292 241 156 377 207 336 204 338 40 No Response/Incorrect Response State Controller’s Officeb 28 Motor Vehiclesc 79 Grand Total 787 aThe number of yes and no responses may not total to the number of critical projects because in some instances the agencies did not provide information for each project. bThe State Controller’s Office (SCO) did not respond to our survey. Our questionnaire was sent to the SCO on June 17, 1998, and we requested a response by June 30, 1998. Despite repeated efforts to obtain this information that occurred between July 15 and July 30, 1998, the SCO’s chief counsel, whom the information technology officer informed us was the liaison for such a request, did not respond. cThe Department of Motor Vehicles responded to our survey, but incorrectly provided the status of its critical projects as of July 31, 1998, rather than March 31, 1998, as we had requested. Because the information submitted was not comparable with the information received from the other agencies, we did not include it in the table. 30 C A L I F O R N I A S T A T E A U D I T O R APPENDIX B Comparison of Reported and Actual Project Status W e reviewed 37 critical projects at four state agencies: the Franchise Tax Board, the State Treasurer’s Office, the Board of Equalization, and the Department of Social Services. We found that 15 of 37 critical projects were in a different phase of remediation than what appeared in the Department of Information Technology’s (DOIT) April 1998 quarterly report. Fourteen projects were actually in earlier phases of the remediation process on March 31, 1998, than reported by the DOIT, and one project was in a later phase than reported. The actual and reported phases for the 15 projects are shown in Table 4. C A L I F O R N I A S T A T E A U D I T O R 31 TABLE 4 Reported Project Status Compared to Actual Project Status at March 31, 1998 Phase of Remediation, Project per DOIT Phase of Remediation, Agency Name No. Project Name at March 31, 1998a per Auditor’s Analysisa Franchise Tax Board 1 Interagency Offset Solution, Design, & Completed Planning 2 Electronic Tax Form Development & Solution, Design, & Planning Filing Modification 3 Personal Income Tax Testing Development & Modification Billings 4 Revenue Agencies Implementation Development & Modification Cross Reference State Treasurer’s 5 Item Processing Development & Solution, Design, & Planning Office Modification 6 Local Agency Testing Development & Modification Investment Fund 7 Payments and Testing Development & Modification Registration Board of Equalization 8 Natural Corporate Development & Solution, Design, & Planning Objects Modification 9 Electronic Funds Transfer Testing Solution, Design, & Planning 10 Legal Appeals Testing Assessment 11 Data Entry System Implementation Development & Modification Social Services 12 Disaster Response Development & Assessment System Modification 13 Facilities Equipment & Development & Solution, Design, & Planning Embedded Microchips Modification 14 Group Residence Locator Implementation Testing 15 Integrated Fraud Implementation Development & Modification Detection aAs shown in Figure 1 on page 3, the phases for remediating a project are: (1) Assessment (2) Solution, Design, and Planning (3) Development and Modification (4) Testing (5) Implementation (6) Completed 32 C A L I F O R N I A S T A T E A U D I T O R APPENDIX C Analysis of the Time Needed to Begin the Phase Reported in the DOIT’s April 1998 Quarterly Report A s illustrated in Appendix B, 14 of the 37 critical projects we reviewed at 4 state agencies were in earlier remediation stages than reported in the Department of Information Technology’s (DOIT) April 1998 quarterly report. One project was further along than reported in the April quar- terly report. We further analyzed information at each agency to measure the number of months it actually took to begin the phase originally reported for the projects’ status to the DOIT as shown in its April quarterly report. We grouped the projects into four time frames as illustrated in Table 5. C A L I F O R N I A S T A T E A U D I T O R 33 TABLE 5 Number of Months Elapsed Until The Projects’ Actual Status Matched the Status Reported In the DOIT’s April Quarterly Report Number of Months Until the Project Will Be in the Status Shown in the DOIT’s April Quarterly Report Project 3 or Agency Name No. Project Name 0 to 1 1 to 2 2 to 3 more Franchise Tax Board 1 Electronic Tax Form Filing X 2 Personal Income Tax Billings X 3 Revenue Agencies Cross Reference X State Treasurer’s Office 4 Item Processing X 5 Local Agency Investment Fund X 6 Payments and Registration X Board of Equalization 7 Natural Corporate Objects X 8 Electronic Funds Transfer X 9 Legal Appeals Xa 10 Data Entry System Xa Social Services 11 Disaster Response System X 12 Facilities Equipment & Embedded Microchips X 13 Group Residence Locator X 14 Integrated Fraud Detection Xb Totals 1 1 1 11 a According to the information reported to the DOIT as of June 30, 1998, these projects have been deleted. b According to the information reported to the DOIT as of June 30, 1998, this project’s overall completion date is listed as undetermined. 34 C A L I F O R N I A S T A T E A U D I T O R APPENDIX D A Comparison of Project Completion Dates W e compared project-completion dates reported to the Department of Information Technology (DOIT) on March 31, 1998 (as shown in the DOIT’s April quarterly report), and June 30, 1998, respectively, for the 37 projects we reviewed at 4 state agencies—the Franchise Tax Board, the State Treasurer’s Office, the Board of Equalization, and the Department of Social Services—to measure the progress or slippage associated with the projects. At June 30, 1998, the completion date had been shortened for 7 projects, had re- mained the same for 13, had slipped for 10, could not be deter- mined for 4, and did not apply for 3 because the projects were deleted. All 37 projects we reviewed and their completion dates at March 31, 1998, and June 30, 1998, are shown in Table 6. The table is divided into two sections, one section for the projects our review determined were in a different remediation phase as of March 31, 1998, than was reported in the April quarterly report (see Appendices B and C for detail), and the other section for projects whose phase of remediation we determined were accurately reported. C A L I F O R N I A S T A T E A U D I T O R 35 TABLE 6 A Comparison of Project Completion Dates Between the DOIT’s April Quarterly Report and Its June 30, 1998 Monthly Report Project Project Completion Completion Date Date per the April 1998 Difference Project Quarterly Monthly (in Agency Name No. Project Name Report Report Months) Projects We Determined Were In a Different Phase: Franchise Tax Board 1 Interagency Offset 12/01/98 12/01/98a 0 2 Electronic Tax Form Filing 12/31/98 12/31/98 0 3 Personal Income Tax Billings 12/01/98 12/01/98 0 4 Revenue Agencies Cross Reference 12/31/98 12/31/98 0 State Treasurer’s Office 5 Item Processing 03/01/99 03/01/99 0 6 Local Agency Investment Fund 07/01/98 10/01/98 3 7 Payments and Registration 12/01/98 12/31/98 1 Board of Equalization 8 Natural Corporate Objects 02/01/99 12/30/98 -1 9 Electronic Funds Transfer 10/01/98 12/30/98 3 10 Legal Appeals 01/01/00 deleted n/a 11 Data Entry System 07/01/99 deleted n/a Social Services 12 Disaster Response System 07/03/98 11/30/98 5 13 Facilities Equipment & Embedded Chips 06/30/99 06/30/99 0 14 Group Residence Locator 04/01/98 09/17/98 5.5 15 Integrated Fraud Detection 05/01/98 Undetermined n/a Projects Accurately Reported: Franchise Tax Board 16 Timekeepers Reporting System 08/01/98 08/01/98 0 17 Business Entities Tax 11/15/98 11/15/98 0 18 Offers in Compromise Tracking 12/01/98 12/01/98 0 19 Other Systems 08/17/98 08/17/98 0 20 In-State/Out of State Collection 06/01/98 04/01/98 -2 21 Desk Audit Inventory 06/01/98 03/13/98 -2.5 22 Personal Income Tax 06/01/98 05/12/98 -.5 23 Taxpayer Information 09/30/97 09/30/97 0 State Treasurer’s Office 24 Upgrade Bank Reconciliation 07/01/98 12/31/98 6 Board of Equalization 25 Automated Compliance Management 06/30/99 08/28/98 -10 26 Timber Tax 06/01/99 12/04/98 -6 27 A15 Registration 11/13/98 11/20/98 0 28 Special Tax 06/01/99 deleted n/a 29 Fund Distribution 09/30/98 09/25/98 0 Social Services 30 AFDC Quarterly Distribution 08/95/98 Undetermined n/a 31 Income & Eligibility Verification System 09/91/98 Undetermined n/a 32 Child Support Enforcement 10/01/98 12/04/98 2 33 Mainframe Infrastructure 12/31/98 06/30/99 6 34 State Hearing System 06/12/98 06/30/98 .5 35 Licensing Information System 08/01/98 06/15/98 -1.5 36 AFDC Caseload Movement 11/26/97 Undetermined n/a 37 Assistance Dog Allowance System 11/30/97 03/15/98 3.5 aPer our review of this project, it was actually completed in March 1998. 36 C A L I F O R N I A S T A T E A U D I T O R Agency’s response to the report provided as text only: State of California DEPARTMENT OF INFORMATION TECHNOLOGY 8 01 K Street, Suite 2100 Sacramento, CA 95814 (916) 445-5900 FAX (916) 445-6524 August 18, 1998 Kurt R. Sjoberg California State Auditor Bureau of State Audits 555 Capitol Mall, Suite 300 Sacramento, CA 95814 Dear Mr. Sjoberg: Thank you for inviting the Department of Information Technology to review your report to the Legislature on the Year 2000 Computer Problem. I have no doubt that your recent audit is among the important steps being taken by the state to ensure that the Year 2000 will cause no disruption to California’s ability to protect the health and safety of citizens and to deliver essen- tial services. Governor Wilson has asked me to respond to your recommendations, not only for the Department of Information Technology (DOIT), but also to address the recommenda- tions to the Administration as a whole. As reflected in your report, the Wilson Administration has been very active in providing leader- ship and guidance to state departments and agencies regarding the resolution of the Year 2000 challenge. Major aspects of this leadership and guidance include the Governor’s Executive Order W-163-97 and numerous DOIT sponsored meetings, task forces and “summits,” includ- ing intergovernmental forums involving both state and local government agencies, and over 40 specific publications, policies, methodologies and other activities, all accomplished by the DOIT. As I consider the DOIT’s efforts thus far in leading the Year 2000 program, I am ex- tremely proud of the quality of the work that has been accomplished to date and California’s progress in addressing this very critical issue. As we discussed in our meeting on July 13, 1998, my staff and I believe that your report concentrates on the most significant aspects of the Year 2000 problem and that your observations and conclusions have substantial merit. The one caveat I would offer is based on the experience of our Year 2000 Project Office: California employs information technology in multifaceted and heterogeneous ways and, in some special circumstances, statewide generalizations are difficult - perhaps impossible - to make. The Year 2000 poses unprecedented challenges to the state – challenges that demand the best efforts of many diverse agencies within the state. In the spirit of common purpose that must drive the state’s Year 2000 effort, we submit the attached comments that respond to the wide array of recommendations made by your office. 37 Kurt R. Sjoberg California State Auditor Bureau of State Audits Page 2 of 2 August 18, 1998 My staff stands ready to continue working with the Bureau of State Audits to facilitate the state’s Year 2000 problem resolution effort and to provide you information as re- quested. Please contact Claudina Nevis, Deputy Director for Special Projects, at (916) 445-5900 if you require further information. Sincerely, Signature JOHN THOMAS FLYNN Chief Information Officer State of California JTF/CN/kvd Enclosure 38 DEPARTMENT OF INFORMATION TECHNOLOGY RESPONSE TO BUREAU OF STATE AUDITS REPORT ENTITLED “YEAR 2000 COMPUTER PROBLEM: PROGRESS MAY BE OVERLY OPTIMISTIC AND CERTAIN IMPLICATIONS HAVE NOT BEEN ADDRESSED” The Bureau of State Audits’ (BSA) report to the Legislature on the Year 2000 computer problem concentrates on the most important aspects of the state’s efforts to address the formidable management and technical challenges posed by the century change: • The requirement that testing and quality assurance efforts ensure that there will be no significantly negative impact on the health and safety of California’s citizens, to the fiscal and legal integrity of state operations or to the continuation of essential state agency programs. • The necessity for well-defined electronic data exchange protocols to protect the state’s essential systems from corruption by data transmitted through external interfaces with other entities such as counties and the federal government. • The critical role embedded microchip technologies play in delivery of essential programs, and the need for the state to ensure that these de- vices are identified and fixed wherever necessary. • The obligation to develop business continuation plans focused on core business processes to ensure that agencies maintain the delivery of essential products in the event Year 2000-induced failures cause inter- ruptions to normal computer services. Testing Like the BSA, the Department of Information Technology (DOIT) is convinced that meticulous testing of the systems that mission critical programs depend upon will afford the state the best assurance that business operations will not be interrupted due to Year 2000-related problems. The DOIT also shares the concern that rigorous testing must not be slighted in an effort to meet aggressive completion schedules. The state’s information technology (IT) organizations, along with governance bodies such as the BSA and the DOIT, face the common dilemma of determining what constitutes an acceptable level of testing and, ultimately, what constitutes complete remediation. The DOIT’s view is that, as with many Year 2000 issues, answers must be determined on a case by case basis. To assist departments in selecting the most appropriate testing strategy, the DOIT has issued a testing “white paper” containing guidelines. The DOIT has also published “Resolving Year 2000 issues: Best Practices in Testing,” which describes how the private sector is addressing the matter of Year 2000 testing, a process consistent with the DOIT’s testing approach. 39 DOIT Response to BSA Report Page 2 of 4 August 18, 1998 Given the time constraint, agencies may have to evaluate alternative testing strategies in light of the potential risk to safety, health and revenue, the importance of dates and date manipulation with respect to business functions, and the Year 2000 readiness of the suppli- ers of hardware and software components. Weighing tradeoffs between a test in a “time machine” - where the hardware clock has been set to a date in the future - and a test in an environment that simulates dates in the 21st century requires specific knowledge of business functions, application code and the hardware and software employed. Consequently, the DOIT has elected to not dictate a specific testing strategy because of a concern that this could lead to inadvertent adherence to the policy at the expense of rigorous and comprehensive analysis and testing. The DOIT believes, consistent with prevailing business practices in all sectors concerning Year 2000, that in many cases the most prudent way to maximize the short time left until the end of the century is for agencies to adopt risk-based testing strategies that prioritize testing resources – time, people, hardware and dollars – where they will yield the highest business value. It is incumbent upon agencies - IT organizations and their program business partners - to evaluate alternative testing strategies in light of business risk. That said, once a testing strategy has been adopted, it should be seen to its planned conclusion in order to forestall system failures and to demonstrate due diligence. External Interfaces The DOIT shares the BSA’s concerns about the vulnerability of the state’s computer systems to corruption by data received from the thousands of other government and private sector organizations that exchange electronic data with the state. As the BSA acknowledges in its report, the DOIT has established policies and reporting requirements to assist the state in managing its external interfaces. This includes policies related to Interface Rule Maker Man- agement, Interface Date Standard, and schedule for Testing of Interfaces. The DOIT now requires information from departments regarding external interfaces, and will continue to foster successful remediation and risk mitigation efforts in this area. The DOIT will also monitor the state’s progress through monthly reporting by agencies, on-site project reviews and independent oversight. Embedded Systems The DOIT has been proactive in developing a program that provides substantive information about embedded systems, offers guidance in remediation and requires agencies to report cost and status monthly. For example, on June 17, 1998, the DOIT advised departments and agencies of the requirement to identify a department wide project manager for embedded system efforts, and to report the status of their efforts to the DOIT on a regular basis. The DOIT also initiated and has received 40 DOIT Response to BSA Report Page 3 of 4 August 18, 1998 valuable information from two pilot projects in two separate departments (Corrections and Developmental Services). The projects were designed to foster a comprehensive assessment of the nature and extent of embedded systems within the scope of the pilot, and the degree to which there existed a Year 2000 problem. Preliminary analysis of the results suggests that the state’s situation regarding embedded systems parallels the experience of the private sector with similar percentages of embedded systems requiring remediation. The pervasiveness of embedded microchips in virtually every machine exposes the state to the possibility that malfunction or failure of such devices might negatively impact citizen health and safety, the environment, and the ability of the state to continue business operations without interruption. As the BSA correctly points out, many agencies are just beginning to address remediation of embedded systems. However, in this respect we believe that the state is probably ahead of much of the private sector, in part because of the DOIT’s proactive stance as noted above, which has resulted in the establishment of (1) a methodology for assessing and remediating embedded systems, (2) the California Embedded Systems Center, which is a repository of information regarding specific vendors’ equipment with associated Year 2000 compliance information, which is now available to state government, counties and municipalities and (3) lessons learned from the two pilot projects note above. Business Continuation Plans The Department of Information Technology (DOIT) agrees with the BSA’s statement that the Year 2000 problem is primarily a business problem with potentially devastating financial, organizational and political consequences of the failure of one or more critical information systems or delays in having such systems Year 2000 compliant in a timely manner. The DOIT also concurs that the individuals best suited to initiate business continuity planning are the business managers who are ultimately responsible for, and most familiar with, the agency’s core business processes in their entirety. The DOIT believes that the business managers and the BSA share a common position with respect to contingency planning that is supported by sections 4840-4842 of the State Administrative Manual and specifically by sections 4841.1 and 4842.1, which require each agency employing information technology to assess and manage associated risks. In addition, the DOIT will require, as recommended by the BSA, that agencies indicate, as part of their monthly Year 2000 progress reporting, whether they have appropriate business continuation plans in place to ensure that core business functions will continue uninterrupted in the event a critical computer system fails to perform because of a Year 2000 problem. This information must be provided by agencies beginning with the reports due on September 30, 1998. Moreover, the DOIT will continue working with key state agencies to address the issue of statewide business continuation planning. 41 DOIT Response to BSA Report Page 4 of 4 August 18, 1998 Discrepancies with Data Reported by the DOIT The DOIT agrees with the BSA’s analysis process and will clarify our reporting requirements to prevent future discrepancies. Agencies report the status of their remediation projects to the DOIT monthly. The BSA examined 37 projects tracked by the DOIT. Based on its examination of these projects in the field, the BSA determined that 15 of them were in different phases of the remediation life cycle from that reported by the DOIT in its April 1998 Quarterly Report. The DOIT’s understanding is that most of the projects in question consisted of subprojects that were themselves in different phases. When the BSA reported discrepancies, in most cases the agency and the BSA differed as to the overall project status, with the BSA taking the position that the subproject in the earliest phase best represents the status of the project as a whole. At the time of the BSA’s audit, agencies were required to report planned milestone dates to the DOIT. In the intervening months, the DOIT has enhanced its reporting requirements so that the data collected now more accurately reflects project progress. Currently, for each project milestone that the DOIT tracks, agencies not only report the date they plan to reach the milestone but also the date that they actually reached the milestone. Based on data supplied to the DOIT as of July 31, 1998, most of the discrepancies that the BSA noted in its report have been resolved. Nine of the 15 projects cited now conform to the BSA’s evaluation, three of the projects have been deleted from the database, one project appears to be in an earlier phase, and one project appears to be in a later phase. The DOIT has not been able to associate the remaining project with any project in its database, and will work with the BSA and the department involved to resolve the remaining differences. Closing Comments It is important to realize that although the DOIT relies to some extent on formal reporting by departments as to the status of their various Year 2000 efforts, the DOIT has also actively engaged departments in many other ways, and in doing so has established and maintained effective communication channels regarding Year 2000 progress and issues. In addition to intensive project status reviews involving the DOIT and departmental staff, the DOIT’s Year 2000 Project Office staff are in daily contact with Year 2000 project managers and others in state departments. This high level of communication is essential, because as the BSA report notes, the Year 2000 is a statewide problem, which transcends any one department. Finally, we view the BSA report as a complement to the DOIT’s Year 2000 program in the sense that the report highlights a number of key issues and maintains a focus on the serious nature of the state’s Year 2000 remediation efforts. In that regard, the DOIT will, soon after the release of the BSA report, issue a management memo to all agencies and depart- ments describing the DOIT’s policies with respect to the important issues discussed in the BSA report. 42 Agency’s response to the report provided as text only: State of California JOHAN KLEHS First District, Hayward STATE BOARD OF EQUALIZATION DEAN F. ANDAL 450 N Street, Sacramento, California Second District, Stockton (P.O. Box 942879, Sacramento, California 94279-0070) ERNEST J. DRONENBURG, JR. (916) 327-4975 FAX (916) 324-2586 Third District, San Diego KATHLEEN CONNELL Controller, Sacramento August 18, 1998 ______ JOHN CHIANG Acting Member Fourth District, Los Angeles Mr. Kurt R. Sjoberg, State Auditor ______ Bureau of State Audits E.L. SORENSEN, JR. 555 Capitol Mall, Suite 300 Executive Director Sacramento, CA 95814 Dear Mr. Sjoberg: The Bureau of State Audits (BSA) has conducted a review of statewide efforts being taken by agencies to ensure Year 2000 (Y2K) compliance for computer systems. As part of the review, BSA made an on-site examination of the Board of Equalization’s (BOE) Y2K remediation efforts. BOE has reviewed the BSA’s report and is confident that the efforts being taken at BOE will ensure the agency makes a smooth and seamless transition into the next century. The report raised concerns regarding agencies’ external interface contacts and an apparent lack of established test plans and procedures for incoming Y2K remediated external files. BOE has a full time External Interface Manager responsible for the oversight of this process and a project plan has been developed that addresses all areas of concern. In response to recommendations regarding Y2K contingency planning, BOE is expanding its ongoing Business Resumption Planning efforts to include the development of continuation plans for essential services in the event failures or delays are experienced as a result of the year 2000. All the other comments in the report relating to the overall statewide Year 2000 effort have been noted. If you have any questions or concerns, please let me know. Sincerely, Signature of John Warass, Chief Deputy Director For: E. L. Sorensen, Jr. Executive Director ELS:jlm: cc: Mr. John Waraas Mr. Jerry Becker Ms. Darlene J. Allen 43 Blank page inserted for reproduction purposes 44 Agency’s response to the report provided as text only: STATE AND CONSUMER SERVICES AGENCY African American Museum Buildings Standards Commission Office of the Secretary Consumer Affairs 915 Capitol Mall, Suite 200 Fair Employment & Housing Sacramento, California 95814 Fair Employment & Housing Commission Franchise Tax Board General Services Insurance Advisor Science Center Personnel Board August 18, 1998 Public Employees’ Retirement System Teachers’ Retirement System Kurt R. Sjoberg, State Auditor Bureau of State Audits 555 Capitol Mall, Suite 300 Sacramento, California 95814 Dear Mr. Sjoberg: RE: YEAR 2000 COMPUTER PROBLEM: PROGRESS MAY BE OVERLY OPTIMISTIC AND CERTAIN IMPLICATIONS HAVE NOT BEEN ADDRESSED Enclosed is our response prepared by the Franchise Tax Board to the Bureau of State Audits’ Report No. 98023 entitled “Year 2000 Computer problem: Progress May Be Overly Optimistic and Certain Implications Have Not Ben Addressed,” as well as a copy of the response on a diskette. If you have any questions or need additional information, please contact me at 653-4090 Sincerely, Signature George Valverde Deputy Secretary Enclosures 45 Blank page inserted for reproduction purposes 46 Agency’s Response to the report provided as text only: State of California KATHLEEN CONNELL Chair FRANCHISE TAX BOARD P.O. Box 1468 DEAN ANDAL Member Sacramento, California 95812-1468 (916) 845-6191 Fax (916) 845-0913 CRAIG L. BROWN Member August 17, 1998 To: Doug Cordiner, Principal Auditor Bureau of State Audits From: Linda Roth, Director Business Entities Tax Systems Management Bureau Franchise Tax Board Subject: Draft review of Year 2000 Computer Problem On behalf of the Franchise Tax Board and Year 2000 Project Office, we would like to thank you for the opportunity to review the draft of the Year 2000 Computer Problem report. We appreciate the time you and your team took to hear and address our concerns regarding the report. We hope that our input will help to convey the message that the Year 2000 remediation effort and its impact on the State’s ability to serve the people of California must be addressed with the seriousness it requires. We concur with your recommendations; they are consistent with industry standards and best practices. The Franchise Tax Board has already started working towards implementing the suggested processes as part of the Year 2000 Project Office oversight program. If we can be of further assistance, please feel free to contact Marla Hyde at (916) 845-3933 or me at (916) 845-6191. 47 Blank page inserted for reproduction purposes 48 Agency’s Response to the report provided as text only: State of California–Health and Welfare Agency DEPARTMENT OF SOCIAL SERVICES 744 P Street Sacramento, California 95814 August 17, 1998 Kurt R. Sjoberg Bureau of State Audits 555 Capitol Mall, Suite 300 Sacramento California 95814 Dear Mr. Sjoberg: SUBJECT: DRAFT REPORT - “YEAR 2000 COMPUTER PROBLEM: PROGRESS MAY BE OVERLY OPTIMISTIC AND CERTAIN IMPLICATIONS HAVE NOT BEEN ADDRESSED” The Secretary, Health and Welfare Agency, has requested California Department of Social Services’ comments regarding the findings and recommendations contained in the above cited Bureau of State Audits (BSA) report. Our comments are enclosed. We appreciate the many opportunities your staff has provided our department to furnish information and respond to the auditors’ findings during the audit process. If you have any questions regarding our comments, please have your staff contact Michael Howland, Deputy Director, Information Systems Division at (916) 654-5699. Sincerely, Signature of Ann Bersinger, Chief Deputy Director For: Eloise Anderson Director Enclosure c: Tom Robertson, HWA Lawrence Bolton, Legal Division Jarvio Grevious, Administration Division 49 CALIFORNIA DEPARTMENT OF SOCIAL SERVICES (CDSS) Comments in Response to Draft Audit Report Finding: Some Agencies’ Reported Progress toward Resolving Year 2000 Problems was Overstated. CDSS Response: California Department of Social Services (CDSS) concurs with the finding. As recommended, CDSS will ensure that future monthly reports to the Department of Information Technology (DOIT) provide accurate information about the status of the department’s year 2000 remediation efforts. All monthly DOIT updates will reflect the best current estimates for completion dates of each phase of remediation, testing, and completion. As noted in the audit report, CDSS has already corrected some milestone and comple- tion dates and provided these to the DOIT in the June monthly report. CDSS will provide information to the DOIT by October 1, 1998 for any completion dates that are currently listed on the monthly reports as ‘undetermined.’ because correspond- ing completion dates for interfacing systems have not been provided by external data exchange partners. Finding: Agencies Need to ensure that Data from External Entities are Year 2000 Complaint. CDSS Response: CDSS concurs with the finding. CDSS recognizes the criticality of resolving all interface management issues. We are actively managing this effort, and will continue to do so, as indicated below: • Tests of internal interdependencies and external dependencies will be performed for each CDSS year 2000 project prior to its being declared complete. • CDSS is developing schedules and plans for testing external interfaces. This includes thorough follow-up testing of data supplied by external parties. • We are establishing contacts with all data exchange partners and will be using a standardized memorandum of understanding to document agreements regarding date formats, testing, and schedules. • We will comply will all DOIT requirements for all external interface development, coordination, testing, and reporting. 50 Finding: Few State Agencies have Business Continuation Plans for Problems Caused by Year 2000 CDSS Response: CDSS concurs with the finding. As indicated in the schedule that CDSS has given to the Bureau of State Audits (BSA), we will develop business continuity plans no later than June 30, 1999. As recommended, CDSS will establish business continuation planning teams for each core business process and infrastructure component that may be affected by the Year 2000 Problem. CDSS will use the General Accounting Office model, as specified by BSA, as a guideline for developing a business continuation plan for each core business process. 51 Blank page inserted for reproduction purposes 52 Agency’s Response to the report provided as text only: MATT FONG, TREASURER State of California 915 Capitol Mall, Room 110 Sacramento, California 95814 (916) 653-2995 FAX (916) 653-3125 August 18, 1998 Mr. Kurt R. Sjoberg, State Auditor California State Auditor 555 Capitol Mall, Suite 300 Sacramento, CA 95814 Dear Mr. Sjoberg: The State Treasurer's Office (STO) is pleased to be among the respondents to the Bureau of State Audits (BSA) Year 2000 survey and to be chosen as one of the four agencies to be reviewed by the BSA. The problems presented by the new millennium are significant at best and can be disastrous if an agency has not taken steps to remediate its non-compliant sys- tems. I read with interest the draft report as the BSA outlined their concerns regarding the status of the collective state agencies and departments. I commend the BSA for their efforts to "bring to light" areas where state agencies and departments can improve their chances of success. However, in reviewing the main points in this draft and assessing the STO's progress, it is evident our agency is further down the road to total Year 2000 compliancy than is indicated in this report. The draft report speaks to several general areas of concern, and I would like to address those areas on behalf of the STO. The first area involves the accuracy of status reporting to the Department of Information Technology (DOIT). It is clear from this report that BSA used a different methodology for determining project status and remediation phase than used by the STO and perhaps other state agencies. While the BSA methodology of using the "least completed task" for determining * project phase may be appropriate for existing system modification projects, it does not accurately reflect the project phase for systems being fully replaced through new development and acquisition of proprietary packages. With one exception, the STO is replacing its' critical processes through new applications and pre-written software packages. We are willing to work with DOIT and BSA to provide a more objective methodology for assessing the progress of projects that replace hardware, operating systems and proprietary software products. Another area of concern deals with the lack of Year 2000 testing and, in particular, "time machine testing." The report states the Teale Data Center and the Health & Welfare Data Center *California State Auditor’s comments on this response are on page 55. 53 have machines available for time-machine testing, but that none of the 39 agencies surveyed have rigorously time-tested their critical systems. In late 1997, the STO acquired its own computer that would also function as a "time machine." Some early time-testing began in late 1997, and we started time-testing critical applications in February 1998. To date, four of our critical systems have completed full time-testing; two critical systems are scheduled for time testing in early October; and the remaining critical system is scheduled for January 1999. Business Continuity plans were also an area of concern as noted in the report. The draft accurately relays our assertion that Business Continuity plans are necessary for our projects and that we are actively developing those plans in conjunction with the scheduled implementation of the individual systems. The last area of concern relates to departments and agencies receiving assurances from their data exchange partners that shared data files will be Year 2000 compliant. The STO has taken an active role to determine its data exchange partners and to insure that our shared data files are Year 2000 compliant. Internally, we have worked with the State Controller's Office (SCO) and the Department of Health Services (DHS) to remediate our shared files. We implemented Year 2000 compliant shared files with the SCO in July 1997, and are currently working with DHS. We also identified interfaces with the depository banks that do business with the STO. We have received written assurances from each of them that they are actively remediating their systems and plan to be complete by the end of 1998. The Federal Deposit Insurance Corporation (FDIC) also monitors their remediation activity for compliance with federal standards and regulations. In summary, we appreciate the opportunity to be a part of the BSA survey. The report makes some valid observations and points out areas where departments and agencies can improve their Year 2000 remediation efforts. The STO is "ahead of the curve" in most of these areas and is working diligently to complete our remediation process. We are on-track to complete our Year 2000 projects and will be prepared to enter the new millennium. Warmest Regards, Signature Matt Fong State Treasurer 54 COMMENTS California State Auditor’s Comments on the Response From the State Treasurer’s Office T o provide clarity and perspective, we are commenting on the State Treasurer’s Office (STO) response to our audit report. The numbers correspond to the numbers we have placed in the response. Our methodology was to compare the project phase as reported to the Department of Information Technology (DOIT) to the phase demonstrated at the specific department as of March 1998. We found, as shown on page 32, that three projects were in earlier stages than reported based upon the evidence the STO provided to us. Our report statement is based on those agencies declaring critical projects completed as of March 31, 1998. The STO was not one of those agencies reporting completed projects at that time and therefore was not included in our statistic. The text the treasurer is referring to was deleted from our report. C A L I F O R N I A S T A T E A U D I T O R 55 cc: Members of the Legislature Office of the Lieutenant Governor Attorney General State Controller Legislative Analyst Assembly Office of Research Senate Office of Research Assembly Majority/Minority Consultants Senate Majority/Minority Consultants Capitol Press Corps