Review of State's Efforts to Meet Year 2000 Computer Change

LIT'TLE HOOVER COMMISSION REVIEW OF STATE'S EFFORTS TO MEET YEAR 2000 COMPUTER CHANGE Mqy 1998 State of California HOOVER COMMISSION LIT~rLE Richard R. Terzian May 29, 1998 Chairman Michael E. Alpert Vice Chairm,tn Carl D. Covitz Daniel W. Hancock Sally Havice Assemblymember The Honorable Pete Wilson Gary H. Hunt Governor of California Quentin L. Kopp Senator Gwen Moore The Honorable John Burton The Honorable Ross Johnson Angie Papadakis President Pro Tempore of the Senate Senate Minority Leader and members of the Senate Charles Poochigian Assemblymember John Vasconcellos The Honorable Antonio "illaraigosa The Honorable Bill Leonard Senator Speaker of the Assembly Assembly Minority Leader Stanley R. Zax and members of the Assembly Je anrune L. English Executive Director Dear Governor and Members of the Legislature: The Little Hoover Commission believes that the Year 2000 computer problem poses a unique and substantial threat to the State's business operations. And while substantial progress has been made, the Commission believes the efforts to modernize the State's computer systems would benefit from more detailed oversight and creative support on the part of California's top policy makers. Considerable progress has been made by information technology staff in individual departments. The Department of Information Technology (DOlT) through its Year 2:)00 Program has provided valuable assistance to the project managers assigned the task of modernizing the computers. And most recently some lawmakers have demonstrated the ability of public scrutiny to reinvigorate public agencies charged with resolving high priority problems. In particular, some legislative budget subcommittees have scrutinized the Year 2000 efforts of individual departments, and some policy committees have provided valuable oversight, including the Senate Select Committee on Procurement, Expenditures and Information Technology; the Senate Select Committee on Economic Development; and the Assembly Information Technology Committee. But the State is not yet halfway toward finding, fixing and testing all of the computer applications that could be crippled by date-related problems. And given the high cost of failure, this task cannot really be considered finished until the programs have been tested and solid plans are in place to deal with inevitable problems. To give state agencies every opportunity to finish the job, they should not be given new technology related mandates that will distract them from efforts to become Year 2000 compliant. Milton Marks Commission on California State Government Organization and Economy +http}/wwwJhc.ca.gov/lhc.hun 660 J Street, Suite 260+Sacramento, CA 9581H-916-44S-212S+fax 916-322-7709+e-maillittle.hoover@lhc.ca.gov The Year 2000 problem -- sometimes known by the abbreviation Y2K -- results from the way computers save and count dates. Put most simply, many computer programs and microchips were not developed to count accurately beyond December 31, 1999. Unless modified, some technology applications will fail, while others will make mistakes in calculations and other functions. The software problem is daunting because of its size. The Department of Information Technology estimates that 72 million lines of code need remediation -- nearly all of them part of "mission-critical" systems. As of DOlT's April reporting period, 7.9 million lines, or 11 percent, had been remediated. Correctly, technologists and policy makers in recent months have become increasingly concerned about the date-related problems with embedded chips, which are used in a wide variety of technology applications: fax machines, hospital equipment, telephones and air conditioners. It is not always easy to determine which machines contain date-sensitive chips, and vendors have responded poorly to requests from state agencies for information and c;ssistance. In fact, in some cases it has been reported that presumably identical pieces of equipment have reacted differently to date testing because of the existence of chips suppli'3d by different subcomponent manufacturers. For the State, the Y2K problem poses an enormous risk to the public services that Californians rely upon. The collection of revenue and the distribution of benefits could be interrupted. Public safety could be jeopardized by malfunctioning prison security, transportation or emergency response systems. Beyond the highly visible problems are thousands of other technological applications that will be costly to replace and costly should they fail. In both the public and the private sector, reasonable people have developed widely divergent expectations for the likelihood and the consequences of failure -- from prognostications of physical and fiscal catastrophes to calm assurances that the problems will be resolved in time. Similarly, the responses from the State's technologists reveal a dichotomy defined by confident optimism amidst unquantifiable uncertainty. The Year 2000 project managers who testified before the Commission, as well as DOlT's staff, maintain they have the task well in hand. But they also concede that much of the work is left to be done, that some of the problems have yet to be identified, and that they will not completely know the efficacy of their work until they successfully pass failure dates. Specifically: • The Department of Corrections has not even identified all of the problems associated with embedded chips. And while it has contingency plans for staffing towers in the event that electrified fences do not function and manually operating electronic gates and cell doors, operating in the manual mode can only be considered a short-term alternative in the successful operation of the nation's largest prison system. • The University of California estimates its Y2K remediation efforts will require 100,000 person hours. But 81,000 hours of that effort have yet to be made. Similarly, many departments have not budgeted time or money for fixing problems that are detected in the testing phase, even though Y2K project managers have come to believe that testing itself will take more time and cost more money than first thought. 2 • The Health and Welfare Agency Data Center expressed concern that even if all of its software is rewritten, and even if its clients' systems are fully prepared, maintaining services could be imperiled by security systems that are not operating, or telephones and modems that do not work. To be sure the Y2K project managers are taking their jobs seriously. And DOlT appears to be responsive to new issues and concerns raised by analysts and policy makers. For instance, the department's new contingency plan requirement properly corrects a deficiency identified earlier this year by the Legislative Analyst. That attitude is especially helpful because the State has never had to solve a problem of this scope and nature. In that spirit, the Little Hoover Commission makes the following recommendations that it believes will make it easier for departments to complete the job while ensuring that troubled projects receive needed attention. Increase Flexibility In both the public and the private sector, information technology projects have a reputation for being overdue, overbudget and oversold. In its April 1998 Quarterly Report, DOlT said that the schedule for some Year 2000 projects may be too optimistic and that many of the costs associated with repairing or replacing Year 2000-impacted systems are not known. Given the firm deadline and the short time remaining to make repairs, the State needs to build flexibility into the system. Specifically, the State should: II Create a special fund. The State is currently finalizing the annual budget during which nearly all of the Year 2000 repairs will have to be financed and completed. It is clear that not all of the problems have been found -- particularly in regard to embedded chips -- and that testing will uncover still more problems. These repairs cannot be delayed because of the traditional months-long appropriations process. While fiscal restraint is important, the costs of computer failure are high and prudence dictates that the State invest in all essential repairs in a timely manner. As such, the State should set up a fund similar to the one that was established in fiscal year 1997-98 -- but in a more substantial amount in line with the current estimate of remediation costs -- to provide for quick allocation of resources to meet immediate needs. Such a fund would reduce the chances that the quality of remediation efforts are sacrificed because technologists are afraid to ask for the necessary resources or cannot get ready access to those resources. • Review personnel rules. DOlT reports that efforts to finish Y2K projects on deadline - and in particular in the last few weeks of the century -- could be compromised as workers schedule time off that they are required to take because of limits on accrued vacation time. The Department of Personnel Administration should interpret its rules in a way that gives departments the discretion to allow personnel working on Y2K projects to put off vacations until after the project is completed. The Department of Personnel Administration also should consult with technology-dependent departments to ensure that other personnel practices -- and especially examination and selection rules -- are enabling the departments to hire and retain a qualified information technology workforce. 3 II Limit new technology applications. Departments already are prevented from initiating new technology applications until they have shown that all of their systems are Year 2000 compliant. Often times, however, new statutes require modifying computer applications. To prevent new statutes from distracting agencies' information technology units, the Legislature should consult with affected departments to determine if proposed legislation would result in computer-related demands that will diminish that department's efforts to become Year 2000 compliant. IntensifY Oversight The Department of Information Technology maintains that it lacks the staff to review all or even most of the Y2K projects -- and not even all of the mission-critical projects. In turn, DOlT has created an "oversight hierarchy" with escalating degrees of review. The first level of oversight applies to all agencies; departments submit forms that are reviewed by an automated system designed to identify projects that are falling behind schedule. Next, ;ome selected projects are "peer reviewed" by other state agencies. In addition, DOlT has compiled a list of systems for which failure could jeopardize state revenue and public safety; eight of those systems will be selected each month and will be reviewed by DOlT staff or their consultants. Finally, the managers of troubled projects are called into DOlT's office for a detailed review. In short, of the nearly 600 mission-critical systems that must be remediated before the December 31, 1998 deadline prescribed in Executive Order W-1 63-97, fewer than one-third could be expected to receive any oversight other than the self-review forms. Given the cost of failure, more intensive oversight is warranted. Specifically, the State should: • Require more detailed reporting. DOlT's April monthly report was a significant improvement over its previous quarterly reports. Still, future reports could be further refined to tell the Governor, the Legislature and the public simply and clearly what it needs to know. On a department-by-department basis -- and for important applications on a project-by-project basis -- the reports should identify those that are not meeting deadlines or may not be Year 2000 compliant in time. Lists of troubled projects should include a brief summary of the function that application performs, the consequences of failure for the State and the public, and the steps being taken to remedy the situation. For example, DOlT's April 1998 quarterly report states that 46 mission critical systems are not scheduled to be completed until well after the Executive Order's December 1998 deadline. Policy makers and the public need to know specifically which systems those are, what is being done to fix them and what happens if they are not fixed in time. Incorporate more independent review. Many organizations, including the federal government, have concluded that self-evaluations are unreliable. Further, independent reviews of some previously fixed systems found problems that had been overlooked. The State should seriously consider independent reviews of two kinds of projects: (1) Those truly essential programs whose failure would bring immediate hardship to the most vulnerable in society or would compromise the ability to collect revenue or distribute benefits; and, (2) large mission-critical systems that are falling behind schedule or are not scheduled to be remediated until 1999, leaving little time to detect and fix last-minute glitches. Currently, DOlT's reviews are not intensive and not independent, which may be appropriate for the vast majority of projects that do not fit 4 into these two categories. But for those projects that are the most important and the most troubled, investing in intensive independent review is warranted. [] Clarify the ultimate goal and responsibility. Executive Order W-163-97 states that "each agency" is responsible for fixing Year 2000 problems in "essential systems" by December 31, 1998. The order also directs agencies to "achieve compliance through existing resources." Department directors need to be told clearly and specifically that the primary goal is uninterrupted business operations and that it is their responsibility, and not DOlT's, to ensure that goal is reached. The prudent use of public resources is an important means to that end. But the goal of maintaining public service should not be jeopardized because of the need for additional funds. II Conduct a compr~hensive year-end review. An oversight hearing near the end of 1998 should be held to determine which projects have failed to meet the deadline and to ensure that plans are in place to rectify the outstanding problems before the turn of the century. This process will be particularly important to maintain public focus on this issue during the transition to a new administration and Legislature. Address External Factors The State has taken important steps to ensure that its operations do not come to a halt because one of its partners or service providers have not become Year 2000 compliant. Technologists, however, identified some areas in which more effort is needed to ensure that entities other than the State are cooperating completely. Specifically, the State should: • Get firm with vendors. A number of departments report that computer vendors have been reluctant to identify which programs or embedded chips are compliant. DOlT's legal task force should explore ways to make it clear to these companies that full and complete disclosure -- and even active participation on the vendor's part in identifying and correcting Year 2000 problems -- is a prerequisite for doing business with the State now and in the future. Furthermore, vendors should be reminded of legal obligations contained in the false claims sections of the Government Code (§ 12650 et seq.) that require them to truthfully represent the nature of goods and services they have provided to the State. • Review progress by public utilities. Some of the state operations have expressed a valid concern about the ability of power and telecommunications providers to ensure uninterrupted service into the millennium. The State should take the appropriate action to ensure that essential utility services have invested in necessary Y2K repairs. Conduct a Post-Millennium Review Because of the scale and nature of the Year 2000 Program, the State will have gained valuable experience in the technology field no matter how successfully it navigates its computer applications into the new century. The State will learn more from the experience if it assesses its efforts and incorporates that knowledge into future planning. Specifically, the State should: 5 • Assess the costs of diverting resources. After a department has completed all testing and is Year 2000 ready, it should prepare a summary for the Legislature on new projects that were delayed, maintenance that was deferred or other efforts that were compromised because time, talent and resources were diverted to making Year 2000 repairs. Those assessments should guide the Legislature in determining which of those projects should receive additional funds. • Re-assess DOlT's oversight role. DOlT staffers were candid about the fact that they see themselves primarily as a resource for other departments. They want to be a clearinghouse for information and a source for technical assistance. They believe that to the degree they are adversarial in terms of their oversight that they diminish their role as a technical resource. Both functions are important, but DOlT's oversight clearly cannot be forsaken merely to ensure good relations with other state agencies. A fundamental purpose for creating the department was to provide focused and knowledgeable oversight of computer projects. In the case of the Year 2000 Program, the Legislature stepped in to provide some of the public scrutiny that DOlT has been reluctant to provide. Over the long-term, however, the State needs to consider how to best accomplish the oversight originally intended to be conducted by DOlT. The Little Hoover Commission believes these reforms would help the State to maintain business operations in the short-run and improve its use of information technologies in the long run. It appreciates the cooperation afforded ')y a number of departments while conducting its review. And it stands ready to work with the Governor and the Legislature to implement the recommendations. Sincerely, fe.-.c- -:; RiChar: R. Terzian Chairman 6