Summary

February 2015 California Department of Consumer Affairs’ BreEZe System Inadequate Planning and Oversight Led to Implementation at Far Fewer Regulatory Entities at a Significantly Higher Cost Report 2014-116 COMMITMENT INTEGRITY LEADERSHIP The first five copies of each California State Auditor report are free. Additional copies are $3 each, payable by check or money order. You can obtain reports by contacting the California State Auditor’s Office at the following address: California State Auditor 621 Capitol Mall, Suite 1200 Sacramento, California 95814 916.445.0255 or TTY 916.445.0033 OR This report is also available on our Web site at www.auditor.ca.gov. The California State Auditor is pleased to announce the availability of an online subscription service. For information on how to subscribe, visit our Web site at www.auditor.ca.gov. Alternate format reports available upon request. Permission is granted to reproduce reports. For questions regarding the contents of this report, please contact Margarita Fernández, Chief of Public Affairs, at 916.445.0255. For complaints of state employee misconduct, contact the California State Auditor’s Whistleblower Hotline: 1.800.952.5665. Elaine M. Howle State Auditor Doug Cordiner Chief Deputy February 12, 2015 2014-116 The Governor of California President pro Tempore of the Senate Speaker of the Assembly State Capitol Sacramento, California 95814 Dear Governor and Legislative Leaders: As requested by the Joint Legislative Audit Committee, the California State Auditor presents this audit report concerning how the California Department of Consumer Affairs (Consumer Affairs) planned, developed, and implemented BreEZe—an information technology (IT) system Consumer Affairs envisioned would support all of the primary functions and responsibilities of 37 of its 40 boards, bureaus, committees, and a commission (regulatory entities). This report concludes that the BreEZe project has been plagued with performance problems, significant delays, and escalating costs, which based on a January 2015 estimate were $96 million—more than triple the original cost estimate—for implementation of a system at only half of the regulatory entities originally planned for BreEZe. As of that date, only 10 regulatory entities had transitioned to BreEZe, eight more intend to transition to it in March 2016, and it is unknown whether or when the remaining 19 will transition to the system. Although doing so is critical to the successful development of IT systems, Consumer Affairs failed to adequately plan, staff, and manage the project for developing BreEZe. For example, while an up-to-date assessment of business needs is essential to developing adequate system requirements, Consumer Affairs failed to properly perform such an assessment for the regulatory entities when developing the system requirements for BreEZe, which specify what the system should do. Instead, Consumer Affairs relied on requirements from earlier projects that were abandoned and incorrectly assumed that the entities could use similar business processes to, for example, process license applications. Further, although staff of the California Department of Technology (CalTech) in its oversight role raised nearly 180 significant and persistent concerns about the BreEZe project in monthly reports between December 2010 and September 2014 in areas including project management, staffing, system requirements, and vendor performance, it allowed the project to continue without significant intervention. We believe the volume and significance of these concerns should have prompted both CalTech and Consumer Affairs to analyze fully the costs and benefits of suspending or terminating the project versus proceeding. Additionally, during the procurement process for the BreEZe project, the California Department of General Services (General Services) and Consumer Affairs approved revisions to the BreEZe contracts’ terms and conditions proposed by the vendor, Accenture LLP, which transferred significant risk to the State. For example, the revised language limited Consumer Affairs’ ability to terminate the contracts and eliminated protections Consumer Affairs otherwise would have had against the possibility of intellectual property rights violations. We question the prudence of some of the decisions General Services and Consumer Affairs made regarding the terms and conditions, as they substantially increased Consumer Affairs’ financial risks related to these contracts. Despite assertions by the Board of Registered Nursing (BRN) that it was exceeding maximum time frames for processing certain license applications since implementing BreEZe, we found little evidence that it tracks the information needed to support such claims. In addition, we interviewed executive officers of the 10 regulatory entities that have implemented BreEZe and most told us that they are generally dissatisfied with their BreEZe experience because it did not meet their expectations in a variety of ways, and eight including BRN reported that the system has decreased their operational efficiency. Respectfully submitted, ELAINE M. HOWLE, CPA State Auditor 621 Capitol Mall, Suite 1200 Sacramento, CA 95814 916.445.0255 916.327.0019 fax www.auditor.ca.gov Blank page inserted for reproduction purposes only. California State Auditor Report 2014-116 v February 2015 Contents Summary 1 Introduction 9 Chapter 1 Poor Planning and Decision Making Have Led to Increased Costs and Extended Timelines and Have Placed the Future of BreEZe in Doubt 23 Recommendations 49 Chapter 2 The State Entities Responsible Did Not Ensure That the BreEZe Contract Terms Sufficiently Protect the State 53 Recommendations 60 Chapter 3 The Board of Registered Nursing Made Some Unsubstantiated Reports of Difficulties From Implementing BreEZe, and Other Regulatory Entities Also Report Challenges With BreEZe 63 Recommendations 78 Appendix Summary of Significant BreEZe Project Concerns Noted by Independent Oversight Reports, December 2010 Through September 2014 81 Responses to the Audit California Department of Consumer Affairs 99 California Department of Consumer Affairs, Board of Registered Nursing 107 California State Auditor’s Comments on the Response From the Board of Registered Nursing 113 California Department of Technology 115 California State Auditor’s Comments on the Response From the California Department of Technology 125 vi California State Auditor Report 2014-116 February 2015 Blank page inserted for reproduction purposes only. California State Auditor Report 2014-116 1 February 2015 Summary Results in Brief Audit Highlights . . . The California Department of Consumer Affairs (Consumer Our audit concerning the California Affairs) encompasses 40 boards, bureaus, committees, and Department of Consumer Affairs’ (Consumer a commission (regulatory entities) that regulate and license Affairs) planning, development, and professional and vocational occupations to protect the health, implementation of BreEZe—an information safety, and welfare of the people of California. Annually, these technology (IT) system envisioned to support regulatory entities process more than 350,000 applications for all primary functions and responsibilities professional licensure and an estimated 1.2 million license renewals. of its regulatory entities—revealed The regulatory entities establish the minimum qualifications and the following: levels of competency for licensure, register or certify practitioners, » Consumer Affairs failed to adequately investigate complaints, and discipline violators. Although these plan, staff, and manage the project for entities are responsible individually for activities related specifically developing BreEZe. to the professions they oversee and they are semiautonomous bodies whose members are appointed by the governor and the • It did not effectively assess the Legislature, Consumer Affairs establishes general administrative regulatory entities’ business needs to policies for them and provides them with administrative support. determine system requirements. Historically, the regulatory entities have used multiple computer • Inadequate system requirements led systems to fulfill their required duties and meet their business to significant delays at key stages of needs. However, significant issues with these systems reportedly the project. resulted in excessive turnaround times for licensing and • It relied on faulty assumptions in enforcement activities, impeding the ability of the regulatory selecting a commercial “off-the-shelf” entities to meet their goals and objectives. In 2009, after system as the foundation for BreEZe, undertaking several unsuccessful efforts to develop or procure which contributed to an increase in an information technology (IT) system that would improve the project costs—from $28 million capabilities of the regulatory entities it administratively supports, in 2009 to $96 million as of Consumer Affairs proposed, and the California Department of January 2015 for half of the entities Technology (CalTech) approved, BreEZe—a system Consumer originally planned. Affairs envisioned would support all of the primary functions and responsibilities of its regulatory entities.1 Unfortunately, this has not • It did not have adequate staffing been the case. to execute and implement BreEZe through critical project phases. The work Consumer Affairs undertook on the BreEZe project has » Between December 2010 and lacked adequate planning. Although an up‑to‑date assessment September 2014, the California of business needs is critical to the successful development of an Department of Technology’s (CalTech) IT project, Consumer Affairs failed to properly perform such independent oversight raised nearly an assessment for its regulatory entities when developing the 180 significant project concerns, yet both system requirements, resulting in requirements that did not CalTech and Consumer Affairs’ officials allowed the project to continue without significant intervention. 1 Although Consumer Affairs consists of 40 regulatory entities, only 37 of these entities were originally scheduled to implement BreEZe. Specifically, the Bureau of Real Estate and the Bureau of Real Estate Appraisers were brought under Consumer Affairs as a result of the governor’s continued on next page . . . reorganization plan, effective July 2013, after the BreEZe project was approved and underway. According to Consumer Affairs, it planned to implement BreEZe at these regulatory entities once the system was fully implemented at the 37 regulatory entities. Another entity, the Arbitration and Certification Program, does not issue licenses and will not be included in BreEZe. 2 California State Auditor Report 2014-116 February 2015 » Despite significant problems with adequately reflect their individual needs. According to our IT the BreEZe project, CalTech approved expert, system requirements define a business problem to be solved additional funding for it. and specify what the system should do. For example, a system requirement for a regulatory entity could be that the system allow » The California Department of General the entity to record the date it receives an application. In planning Services and Consumer Affairs revised the the BreEZe system, Consumer Affairs should have taken steps to BreEZe contracts’ terms and conditions, at ensure that the system requirements were based on the current the request of the project vendor, in ways business needs of its regulatory entities, so that the resulting system that significantly increased the financial would aid the entities in conducting their business operations risk to the State. and in fulfilling their regulatory responsibilities. Instead, when » As of January 2015 only 10 regulatory developing the requirements for BreEZe, Consumer Affairs relied entities had transitioned to BreEZe, on requirements for earlier projects that were abandoned. eight more intend to transition in March 2016, and it is unknown if the Because Consumer Affairs did not properly determine the business remaining 19 regulatory entities will needs of its regulatory entities, it incorrectly assumed, for example, implement BreEZe. that the entities could use similar business processes to process applications and issue licenses. This misconception, coupled with » Most executive officers of the the fact that Consumer Affairs wanted BreEZe to be developed 10 regulatory entities that had quickly, informed Consumer Affairs’ decision to select an existing transitioned to BreEZe reported that it commercial “off‑the‑shelf,” or COTS, system as the foundation has decreased their regulatory entity’s for BreEZe. Consumer Affairs believed that this type of product, operational efficiency. rather than a custom‑developed system, would require only moderate modifications and resources to implement. These faulty » Due to lack of evidence, the Board of assumptions have led to significant project delays and a substantial Registered Nursing’s claim that the increase in the estimated costs of the project, from $28 million implementation of BreEZe caused in 2009 to $96 million as of January 2015, for implementation inefficiency in processing applications of a system that will include only half of the regulatory entities could not be substantiated. originally planned for BreEZe. Thus, it appears that Consumer Affairs’ selection of this COTS product may not have been the most appropriate and most cost‑effective decision. In part, because the foundation of BreEZe—its system requirements—was inadequately developed, the BreEZe project has experienced delays at key stages of the project. The most extreme delay involved the key milestone of user acceptance testing—testing that future users of the system conduct to confirm that the system operates as its requirements specify. User acceptance testing for the 10 regulatory entities included in the first implementation of BreEZe (phase 1) was originally planned to occur over an eight‑week period; instead it spanned 11 months, from the end of November 2012 to October 2013, significantly exceeding the original time frame. This likely occurred in part because the BreEZe system had almost 1,700 unresolved system defects at the beginning of user acceptance testing. According to our IT expert, many of these defects were likely attributable to the poor development of the system requirements. Although user acceptance testing is one of the final and more critical procedures undertaken before system implementation to ensure that the system operates appropriately, in this case it morphed into a redesign of the requirements and California State Auditor Report 2014-116 3 February 2015 a rework of the system. Specifically, in conducting the testing of the system, some of the 10 regulatory entities included in the first phase of implementation, as well as Consumer Affairs itself, learned that the system did not operate as they expected or needed. Had Consumer Affairs performed a complete, current assessment of the regulatory entities’ needs when determining the system requirements for BreEZe, some of the delays the project has experienced might have been avoided. Further, although CalTech began providing independent oversight of the BreEZe project approximately one year after the project’s inception, neither CalTech nor Consumer Affairs responded appropriately to the significant and persistent concerns that the CalTech staff and consultants charged with overseeing the project were raising. In addition to having the statutory authority to suspend or terminate IT projects, state law assigns responsibility for IT project oversight to CalTech; this project oversight mainly consists of two types of independent oversight. Independent verification and validation (IV&V) is used to ensure that a system satisfies its intended use and user needs. Independent project oversight (IPO) is used to ensure that effective project management practices are in place and in use. In their reports from December 2010 through September 2014 on the BreEZe project, the CalTech IV&V consultant and the IPO specialist raised nearly 180 significant concerns relating to project management, staffing, system requirements, and vendor performance. According to our IT expert, the volume and significance of these concerns should have prompted both CalTech and Consumer Affairs to analyze fully the costs and benefits of suspending or terminating the project versus proceeding. However, although Consumer Affairs officials and CalTech management were fully aware of these concerns, neither group took sufficient action to ensure that these concerns were appropriately addressed; instead, they allowed the project to continue for more than three years without significant intervention. Given CalTech’s authority and the numerous concerns the IV&V consultant and the IPO specialist raised about the project, we question why CalTech did not take steps to ensure that Consumer Affairs heeded its advice. For instance, CalTech could have formally warned Consumer Affairs that it would suspend the project if Consumer Affairs did not bring the project back into alignment with its planned scope, cost, and schedule. As an example, the estimated cost to complete the project had almost tripled to $78 million and the project had experienced significant delays in its schedule before completion of user acceptance testing. We believe these problems, along with the significant cost increases the project had already experienced, should have been enough to elicit 4 California State Auditor Report 2014-116 February 2015 CalTech’s greater involvement in the project. Instead, it approved Consumer Affairs’ Special Project Report (SPR) 2, which requested additional funding for the project, in October 2013.2 Consumer Affairs submitted SPR 3 to CalTech in June 2014; in it, it requested additional funding and estimated the costs to complete the project through phase 2 at $118 million. However, it was not until after Consumer Affairs informally estimated the cost of completing the project had risen to $300 million that same month that CalTech changed its oversight approach on the BreEZe project.3 Although CalTech approved SPR 3 in July 2014, according to the BreEZe project director, Consumer Affairs withdrew its submission of SPR 3 upon direction from CalTech and the California Department of Finance in September 2014. As discussed previously, CalTech has the authority and responsibility to oversee IT projects. If CalTech had chosen to suspend the project, BreEZe development would have been paused temporarily, giving Consumer Affairs additional time to conduct a cost‑benefit analysis and correct fundamental problems, such as requirements issues, that occurred during planning and development. However, in October 2014 the CalTech director— who has overseen the BreEZe project since Consumer Affairs executed its contracts with the project vendor, Accenture LLP (Accenture)—told us that CalTech has not halted BreEZe for several reasons: because BreEZe is moving in the right direction, because the system’s problems are not incurable, and because the system is working and functional. 4 However, Consumer Affairs’ SPR 3.1, which it submitted to CalTech in January 2015, indicates the project is not moving in the right direction and proposes a rescoping of the project because of significant concerns relating to staffing and increasing project costs, and because its contracts with Accenture are no longer financially feasible for Consumer Affairs. For these reasons, among others, the future implementation of BreEZe is uncertain at best and, as it relates to the regulatory entities originally included in the final phase (phase 3), likely unfeasible. As of January 2015, 10 regulatory entities had implemented the system, with the first of three phases occurring in October 2013. Another eight regulatory entities are included 2 An SPR provides a summary of proposed changes to the original project cost, schedule, or scope. An SPR is generally required when the project costs or total financial program benefits deviate or are anticipated to deviate by 10 percent or more, or a major change occurs in project requirements or methodology. 3 The BreEZe project team developed the estimate informally and not in the same manner as an SPR requires. 4 There are three contracts related to the BreEZe project—one contract for design, development, and implementation; another contract for maintenance support; and a third contract for the system license. When we discuss a specific contract, we identify it as either the design, maintenance, or system license contract. California State Auditor Report 2014-116 5 February 2015 in phase 2, which is currently planned for March 2016. However, Consumer Affairs has indicated that it needs additional staffing to successfully implement BreEZe at the phase 2 regulatory entities, and as of January 2015 lacked the funding to fill those positions. Additionally, it is unknown whether or when the remaining 19 phase 3 regulatory entities will implement BreEZe. Specifically, CalTech officials indicated that it completed renegotiating Consumer Affairs’ design contract with Accenture on December 1, 2014, and according to Consumer Affairs’ director, these 19 regulatory entities had been removed entirely from the project. Although the director of Consumer Affairs maintains that the department intends to implement BreEZe at those 19 regulatory entities, it lacks a plan to do so. In fact, SPR 3.1 indicates that the project will end after the phase 2 regulatory entities implement BreEZe, and only after its successful implementation of that phase will Consumer Affairs reassess the best implementation approach for the phase 3 regulatory entities. However, the director of Consumer Affairs acknowledged that the department has not assessed the extent to which the business needs of the 19 regulatory entities will require changes to the system. Moreover, Consumer Affairs has not conducted a formal cost‑benefit analysis to determine whether BreEZe is the most cost‑beneficial solution for meeting those needs. Additionally, the contracts Consumer Affairs executed with Accenture for developing BreEZe do not adequately protect the State. Consumer Affairs executed the BreEZe contracts with Accenture in September 2011, under the direction of the California Department of General Services (General Services). Although its role at that time was to administer state IT procurements and conserve the fiscal interests of the State, General Services and Consumer Affairs agreed to revise the contracts’ terms and conditions during the procurement process, at Accenture’s request, in ways that significantly increased risk to the State. During the request for proposal (RFP) bidding period (RFP phase), General Services provided every potential bidder with the opportunity to submit a protest for issues such as the selection of prequalified bidders or RFP requirements before submitting a bid and to have General Services review its concerns. During the RFP phase in the BreEZe procurement process in January 2011, only Accenture submitted a protest, in which it proposed modifications to the State’s standard IT general provisions and model contract language (standard IT contract).5 Of the 44 modifications to the State’s standard IT contract that Accenture proposed, General Services accepted 18, proposed its own revisions to 19, and rejected just seven. Subsequently in April 2011, in accordance with state law, Consumer Affairs entered into a negotiation with Accenture during 5 At the time of the BreEZe procurement, General Services had several modules of standard contract language related to IT contracts. 6 California State Auditor Report 2014-116 February 2015 which further changes were made to the contract, with General Services’ approval. However, some of those accepted changes to the standard IT contract’s terms and conditions decrease Consumer Affairs’ ability to obtain rights to work product that Accenture builds if Consumer Affairs terminates the contracts early, and they reduce Consumer Affairs’ financial protections in the event of intellectual property rights violations. Although General Services cited reasons for approving the modified terms and conditions in the BreEZe contracts, we question the prudence of some of the decisions it and Consumer Affairs made, as they increased Consumer Affairs’ financial risks related to these contracts. CalTech’s current authority over procurements for IT projects, a role that was not in place at the time the BreEZe contracts were being negotiated, together with its authority for approving and overseeing IT projects, position it well to ensure that future IT procurements do not jeopardize the State’s financial interests. Various stakeholders of the Board of Registered Nursing (BRN), one of the 10 phase 1 regulatory entities, raised concerns about the timeliness with which it has processed applications after implementing BreEZe in October 2013. According to BRN, it has faced significant delays in processing license applications and has been forced to modify its business processes since implementing the BreEZe system. However, although BRN asserted that it was exceeding the maximum time frames for processing certain applications and was facing a backlog of applications after implementing the system, we found little evidence demonstrating that it consistently tracks the information needed to support such claims. For the selection of applications we reviewed, BRN processed these applications, on average, well within the allowable maximum time frames. However, we did determine that as of September 2014, BRN had a significant number of applications that were pending its review— more than 7,000, of which 63 had already exceeded the respective maximum processing time frames. Yet because BRN does not formally track this information, it cannot adequately assess its workload. Additionally, BRN indicated that it has faced, and continues to face, obstacles in its implementation of the BreEZe system; for example, the system requires that staff take additional steps to enter applicant information. However, BRN does not track the information needed to assess the impact of such obstacles. Further, because it believes its efficiency in processing applications has decreased since implementing BreEZe, it has requested additional staff it believes it needs to process applications within required time frames. However, this request is based on data from the two fiscal years preceding BRN’s implementation of BreEZe. Thus, because the analysis BRN used to support its need for the additional positions California State Auditor Report 2014-116 7 February 2015 does not reflect its current workload and business processes since implementing the BreEZe system, the additional positions it requested are not adequately justified. Most of the executive officers of the 10 phase 1 regulatory entities are generally dissatisfied with their BreEZe experience because it has not met their expectations. We interviewed the executive officers of each of the regulatory entities that have implemented the system regarding various aspects of their experience with the project, including their satisfaction with BreEZe and their overall experience with the system. Each regulatory entity reported experiencing issues with certain aspects of the BreEZe project. For example, the majority were unsatisfied with the testing they were able to conduct before implementing the system, and most found the training to be inadequate. In addition, all 10 of the executive officers indicated that BreEZe’s reporting capability was unsatisfactory. Of greater concern, most executive officers reported that BreEZe has decreased their regulatory entity’s operational efficiency. Recommendations CalTech To help ensure the success of the BreEZe project going forward, CalTech should ensure that Consumer Affairs responds promptly to, and adequately addresses, concerns the IPO specialist and the IV&V consultant raise. If Consumer Affairs receives the necessary funding and resources to successfully implement BreEZe at the phase 2 regulatory entities and the project continues to face escalating costs, CalTech should require Consumer Affairs to analyze the costs and benefits of moving forward with the project as planned versus suspending or terminating the project. To ensure that future IT project procurements do not jeopardize the State’s financial interests, CalTech should document its reasons for approving any deviations from standard contract language. Consumer Affairs Consumer Affairs should develop a process to ensure that it undertakes all required oversight activities with respect to BreEZe so that it can prevent or identify and monitor any problems as they arise. This includes taking steps to sufficiently respond to any concerns the IPO specialist and the IV&V consultant raise. 8 California State Auditor Report 2014-116 February 2015 To ensure that BreEZe is a cost‑effective solution to meet the business needs of the phase 3 regulatory entities, should it elect to pursue implementing BreEZe at these entities, Consumer Affairs should first complete a formalized cost‑benefit analysis. This analysis should include an assessment of the potential changes those regulatory entities may require be made to the BreEZe system and the associated costs. Consumer Affairs should continue to work with the phase 1 regulatory entities to ensure that the issues they are facing with BreEZe are being resolved in a timely manner. BRN To ensure that it has adequate data to effectively use its resources and manage its workload, BRN should do the following: • Formally track and monitor the timeliness of its processing of applications by type and track the cause of any delays. • Formally track and monitor the applications pending its review by type and original receipt date. Conduct an analysis no later than June 30, 2015, of its application processing since implementing BreEZe to identify its workload capability. To the extent that it determines additional resources are necessary, BRN should submit a request for these resources that is appropriately justified. Agency Comments Consumer Affairs and BRN agreed with our recommendations and outlined the actions they plan to take to implement them. Although CalTech states that our report’s recommendations are for the most part appropriate and in line with actions and initiatives that it has already undertaken, it explained that it has general concerns with the report and did not indicate whether it agrees with our recommendations. Our comments on CalTech’s response begin on page 125. California State Auditor Report 2014-116 9 February 2015 Introduction Background The primary function of the California Department of Consumer Affairs (Consumer Affairs) is to protect and serve consumers and ensure a competent, fair marketplace. Consumer Affairs encompasses 40 boards, bureaus, committees, and a commission (regulatory entities) that regulate and license professional and vocational occupations to protect the health, safety, and welfare of the people of California. The regulatory entities that comprise Consumer Affairs license doctors, dentists, contractors, and cosmetologists, among other professions. Each year these regulatory entities process more than 350,000 applications for professional licensure and an estimated 1.2 million license renewals. The regulatory entities also establish the minimum qualifications and levels of competency for licensure, register or certify practitioners, investigate complaints, and discipline violators. Although these entities are semiautonomous bodies that are responsible individually for activities related specifically to the professions they oversee and their members are appointed by the governor and the Legislature, Consumer Affairs establishes general administrative policies for them and provides them with administrative support. For example, Consumer Affairs processes payments for goods and services and travel reimbursements for the regulatory entities. Historically, Consumer Affairs’ regulatory entities have used multiple computer systems, referred to as legacy systems, to fulfill their required duties and meet their business needs. The Applicant Tracking System (ATS) and Consumer Affairs System (CAS) are the two database applications that Consumer Affairs uses to maintain its core licensing and enforcement information for regulatory entities. The ATS, which tracks and monitors activities for cashiering, initial license application, and examinations for 32 regulatory entities, was developed in the 1990s and last upgraded in 2003. The CAS, an enterprise licensing and enforcement system, was developed in the 1980s to track license‑related activities such as initial licenses, renewals, complaints, investigations, and enforcement for 38 regulatory entities. According to Consumer Affairs, significant issues with these legacy systems have resulted in excessive turnaround times for licensing and enforcement activities, thus impeding the ability of the regulatory entities to meet their business goals and objectives. In its November 2009 Feasibility Study Report (FSR) for the proposed BreEZe system, Consumer Affairs identified many deficiencies within the existing systems that were affecting consumers. For example, Consumer Affairs reported that the existing licensing 10 California State Auditor Report 2014-116 February 2015 and enforcement systems lacked automated workflow and case management functionality that could help staff ensure that licensing and enforcement activities are accurately and completely documented. Additionally, the existing licensing and enforcement systems did not interface with partner agencies, such as the California Department of Justice, to share information. Consumer Affairs stated that these deficiencies, among others, had resulted in various negative effects, including a prolonged licensing process, an average complaint resolution time of over two years for existing licensees, and lost documents. In 2008 and 2009 Consumer Affairs and one of its regulatory entities, the Board of Registered Nursing (BRN), came under scrutiny from the media for delays in completing enforcement activities. The Los Angeles Times reported that BRN was facing significant enforcement backlogs that extended more than three years and thereby allowed nurses with criminal convictions or with documented histories of incompetence, drug theft, or abuse to continue working while action was pending. For example, the Los Angeles Times reported the results of an investigation it conducted that found more than 100 cases in which the State did not seek to revoke or restrict licenses until the nurses involved had accumulated three or more criminal convictions. In July 2009 the governor stated that Consumer Affairs’ enforcement backlogs were absolutely unacceptable. That summer Consumer Affairs initiated a major effort to reform its enforcement programs, which were plagued with various investigative, legal, procedural, and technical impediments. These impediments hindered Consumer Affairs’ ability to process complaint and enforcement caseloads within a reasonable time, and in turn they diminished Consumer Affairs’ ability to protect consumers and the integrity of the licenses it issued. This initiative, the Consumer Protection Enforcement Initiative (CPEI), intends to overhaul the enforcement process at the healing arts boards—boards that regulate a variety of professions from doctors and nurses to physical therapists and optometrists—through administrative improvements, increased staffing and information technology (IT) resources, and legislative changes. Once it has fully implemented the CPEI, which is largely dependent on its implementation of BreEZe, Consumer Affairs expects the healing arts boards to reduce the average enforcement completion time from 36 months to between 12 and 18 months. California State Auditor Report 2014-116 11 February 2015 History and Evolution of the BreEZe System Consumer Affairs began efforts to develop an updated IT system in 1993. More recently, beginning in 2000, it initiated several IT projects: the Professional Licensing and Enforcement Management System; iLicensing; and the Complaint Resolution Information Management System, a system that was formerly being developed by and for the Medical Board of California. According to the director for the BreEZe project, these projects were ultimately canceled during either the planning or request for proposal (RFP) stages. In 2009 Consumer Affairs modified the system requirements for the projects and combined them into one new, integrated, enterprisewide enforcement and licensing system, referred to as BreEZe. Consumer Affairs views BreEZe as a technological solution to assist it in achieving the goals the CPEI sets forth. As stated in the 2009 FSR for BreEZe, Consumer Affairs believed that BreEZe, at an estimated cost of roughly $28 million, would provide the regulatory entities with an enterprisewide system that supported all applicant tracking, licensing, renewal, enforcement, monitoring, cashiering, and data management capabilities. BreEZe would also be Web‑enabled to allow license application, license renewal, and payment processing on the Internet for applicants and licensees. Further, it would allow the public to file complaints and look up licensee information and complaint status through the Internet. Part of BreEZe’s implementation, according to the 2009 FSR, would involve establishing interfaces to electronically share data with external and internal systems as well as converting existing data from ATS and CAS and migrating it into BreEZe, conducting user training, and creating system documentation. In its FSR for BreEZe, Consumer Affairs also stated its belief that a commercial “off‑the‑shelf” system (COTS) was the appropriate solution and would involve acquiring a systems integrator—a person or company that specializes in bringing together component subsystems into a whole and ensuring that those subsystems work together—to work with appropriate software and service providers to implement the COTS software package that meets all final functional and technical requirements for all participating Consumer Affairs’ regulatory entities.6 Table 1 on the following page presents the objectives of BreEZe. 6 Although Consumer Affairs consists of 40 regulatory entities, only 37 of these entities were originally scheduled to implement BreEZe. Specifically, the Bureau of Real Estate and the Bureau of Real Estate Appraisers were brought under Consumer Affairs as a result of the governor’s reorganization plan, effective July 2013, after the BreEZe project was approved and underway. According to Consumer Affairs, it planned to implement BreEZe at these two regulatory entities once the system was fully implemented at the 37 regulatory entities. Another entity, the Arbitration and Certification Program, does not issue licenses and will not be included in BreEZe. 12 California State Auditor Report 2014-116 February 2015 Table 1 BreEZe Project Objectives OBJECTIVES Track all licensing and enforcement activities within an automated system. Reduce average initial license processing time by 15 percent. Reduce average renewal license processing time by 50 percent. Reduce average complaint resolution time by 5 percent. Address existing backlog issues through proactive management of licensing timelines. Provide accurate performance reporting to stakeholders. Provide accurate enforcement cost reports. Increase data usage during statistical analysis to 90 percent. Reduce average statistical report turnaround time by 50 percent. Reduce average information request turnaround time by 50 percent. Centrally locate and protect licensee personal data. Provide access to cross-entity data. Provide ability for licensees to submit one information update for all licenses. Allow consumers to access licensees’ professional standards violations. Allow legislative changes to be quickly implemented. Reduce the number of cashiering systems from three to one. Achieve 60 percent of applications and renewals submitted via the internet. Enable applicants and licensees to submit, update, and pay via the Internet. Provide the ability to view real-time licensing and enforcement action updates. Sources: California Department of Consumer Affairs’ Feasibility Study Report, November 2009. In September 2011, after receiving approval from the California Department of Technology (CalTech) for its first Special Project Report (SPR) for the BreEZe system, Consumer Affairs entered into a nine‑year contract overseen and approved by the California Department of General Services (General Services) with a systems integrator vendor—Accenture LLP (Accenture)—to assist Consumer Affairs in implementing the COTS.7 As part of this agreement, Accenture subcontracted with Iron Data, the COTS vendor, to provide the BreEZe system for Consumer Affairs. In total, Consumer Affairs executed three contracts with Accenture, totaling $45.7 million, to develop and implement the BreEZe system: a design, development, and implementation contract; a maintenance support services contract; and a software 7 An SPR is a report that provides a summary of proposed changes to the original project cost, schedule, or scope. An SPR is generally required when the project costs or total financial program benefits deviate or are anticipated to deviate by 10 percent or more, or a major change occurs in project requirements or methodology. California State Auditor Report 2014-116 13 February 2015 license contract.8 According to the project director, Consumer Affairs is not responsible for funding the project costs; rather, the total costs of the project are funded by the regulatory entities’ special funds, and the amount each regulatory entity pays is based on the total number of licenses it processes in proportion to the total number of licenses that all regulatory entities process. Consumer Affairs also executed contracts with other nonstate entities to provide services and expertise to assist with implementation of the BreEZe system. For example, Consumer Affairs contracted with private companies to obtain database consulting services and system testing managers. In total, Consumer Affairs awarded about $6.3 million in contracts to these entities, in addition to the Accenture contracts. Table 2 on the following page lists each contract’s purpose, duration, amount, and total expended as of September 30, 2014. Initially, Consumer Affairs had planned for BreEZe to be implemented across the regulatory entities in five sequential phases, with a specified group of entities included in each phase. However, in its contract negotiations with Accenture, Accenture proposed—and Consumer Affairs agreed—that the number of phases be reduced to just three, with each phase containing its own separate design, configuration, and implementation work efforts while sharing project management processes and refined system requirements. As shown in Table 3 on page 15, 10 regulatory entities implemented phase 1 in October 2013, while another eight are scheduled to implement the system in March 2016 (phase 2), leaving 19 regulatory entities for phase 3. However, as we describe further in Chapter 1, CalTech and Consumer Affairs recently finalized negotiations with Accenture to exclude from the current design contract the phase 3 regulatory entities. Consumer Affairs indicated in SPR 3.1, which it submitted to CalTech in January 2015, that after the successful implementation of phase 2, it will reassess the best implementation approach for the remaining 19 regulatory entities; thus, the timing, benefits, costs, and feasibility of that implementation are unknown. The Budget Act of 2013 required Consumer Affairs to submit a report to the Legislature on the status of the BreEZe project no later than October 1, 2014. The report was to include information on the implementation of BreEZe by the healing arts boards, funding allocations, preliminary usage information among new and existing licensees, and a workload analysis for the positions established to support this project. However, according to the BreEZe project director, because the project was not completed by that date Consumer Affairs did not submit the report. 8 For purposes of our report, when we discuss a specific contract, we identify it as either the design, maintenance, or system license contract. 14 California State Auditor Report 2014-116 February 2015 Table 2 All Contracts Related to the BreEZe Project and Expenditures Through September 30, 2014 AMOUNT TOTAL CONTRACT EXPENDED UNDER VENDOR PURPOSE OF CONTRACT DURATION AMOUNT CONTRACT Accenture LLP Design development and implementation of the 9/22/2011–12/31/2020 $34,303,065.43 $5,704,000.00 BreEZe system Accenture LLP BreEZe solution license support 9/22/2011–11/30/2021 7,853,489.75 515,000.00 Accenture LLP Maintenance support services 9/22/2011–9/2/2017 3,544,050.00 519,000.00 LCS Technologies Provide the California Department of Consumer Affairs 5/11/2012–9/11/2012 226,840.00 186,000.00 (Consumer Affairs) with Oracle database administrator consultant—one consultant LCS Technologies Oracle database administrator consultant services— 7/15/2013–7/1/2014 242,000.00 242,000.00 one consultant LCS Technologies Provide Consumer Affairs with Oracle 7/7/2014–7/6/2015 220,000.00 36,000.00 database administrator—one consultant LCS Technologies Provide Consumer Affairs with a Crystal Reports 7/16/2014–7/15/2015 172,480.00 24,000.00 Enterprise solution architect/Oracle PL/SQL information technology (IT) consultant— one consultant Visionary Integration Provide Consumer Affairs with data conversion senior 8/1/2011–2/28/2015 799,250.00 679,000.00 Professionals* analyst services—one analyst (plus one-year option to renew) Sacramento IT Consulting Provide Consumer Affairs with a Natural/Adabas data 12/16/2013–8/15/2015 360,000.00 110,000.00 conversion consultant—one consultant Informatix Provide Consumer Affairs with project management 6/1/2010–5/31/2015 1,148,760.80 865,000.00 consultant services—one consultant Infiniti Consulting Group* Provide Consumer Affairs with a user acceptance test 10/15/2011–10/31/2014 710,000.00 675,000.00 manager senior analyst—one analyst (plus one-year option to extend) Infiniti Consulting Group Analyze, document, and redesign license renewal 4/22/2013–4/21/2014 115,650.00 94,000.00 forms to make them suitable for optical character recognition software—one management consultant and one forms design specialist Infiniti Consulting Group Provide Consumer Affairs with four IT interface 6/17/2013–6/16/2015 1,386,000.00 605,000.00 systems analysts—four analysts Infiniti Consulting Group Provide Consumer Affairs with user acceptance 2/1/2014–8/31/2015 920,000.00 247,000.00 testing (UAT) manager and software release manager consultant—two consultants, one UAT manager, and one software release manager MetaVista Consulting† Independent verification and validation (IV&V) 1/3/2011–8/31/2015 991,100.00 601,000.00 consultant—one consultant Interagency Agreement With Independent project oversight consultant— 7/1/2013–9/30/2017 443,700.00 245,000.00 the California Department of one consultant Technology (CalTech) Totals $53,436,385.98 $11,352,000.00 Sources: Contracts provided by Consumer Affairs and its accounting records. * These contracts include an option to extend the contract for an additional one-year period. In both cases, the additional cost to the State if the option is exercised is up to $230,000. † The MetaVista Consulting contract is a contract between CalTech and MetaVista to provide IV&V services specifically on the BreEZe project. As such, we included the costs for that contract here. California State Auditor Report 2014-116 15 February 2015 Table 3 California Department of Consumer Affairs’ BreEZe System Release Schedule IMPLEMENTED IMPLEMENTATION PLANNED IMPLEMENTATION UNKNOWN PHASE 1 PHASE 2 PHASE 3 * NUMBER OF NUMBER OF NUMBER OF RELEASE DATE: OCTOBER 2013 LICENSEES RELEASE DATE: MARCH 2016 (PLANNED) LICENSEES RELEASE DATE: UNKNOWN LICENSEES Board of Barbering and Cosmetology 484,420 Board of Optometry 26,500 Acupuncture Board 15,490 Board of Behavioral Sciences 90,600 Board of Vocational Nursing and 141,800 Board of Accountancy 134,670 Psychiatric Technicians Board of Podiatric Medicine 2,650 Bureau of Security and 1,290,960 Board of Chiropractic Examiners 46,430 Investigative Services Board of Psychology 20,950 California Board of 17,680 Board of Guide Dogs for the Blind 110 Occupational Therapy Board of Registered Nursing 514,640 Dental Board of California 178,420 Board of Pharmacy 257,810 Medical Board of California 153,820 Dental Hygiene Committee 28,970 Board for Professional Engineers, 236,050 of California Land Surveyors, and Geologists Naturopathic Medicine Committee 540 Physical Therapy Board of California 46,200 Bureau of Automotive Repair 149,530 Osteopathic Medical Board of 7,890 Veterinary Medical Board 33,800 Bureau of Electronic and Appliance 137,710 California Repair, Home Furnishings and Thermal Insulation Physician Assistant Board 9,900 Bureau for Private 2,150 Postsecondary Education Respiratory Care Board 20,430 California Architects Board 47,540 California Athletic Commission 2,780 Cemetery and Funeral Bureau 35,330 Contractors State License Board 286,620 Court Reporters Board 13,030 Landscape Architect 5,270 Technical Committee Professional Fiduciaries Bureau 800 Speech-Language Pathology 32,720 and Audiology and Hearing Aid Dispensers Board Structural Pest Control Board 118,240 Telephone Medical Advice 60 Services Bureau Totals 1,305,840 1,764,330 1,522,340 Sources: The California Department of Consumer Affairs (Consumer Affairs), its BreEZe Web site, BreEZe Special Project Reports (SPR) 3 and 3.1, and interviews with Consumer Affairs’ BreEZe project director. Note: Although Consumer Affairs consists of 40 regulatory entities, only 37 of these entities were originally scheduled to implement BreEZe. Specifically, the Bureau of Real Estate and the Bureau of Real Estate Appraisers were brought under Consumer Affairs as a result of the governor’s reorganization plan, effective July 2013, after the BreEZe project was approved and underway. According to Consumer Affairs, the plan was to implement BreEZe at these two regulatory entities once the system was fully implemented at the 37 regulatory entities. Another entity, the Arbitration and Certification Program, does not issue licenses and will not be included in BreEZe. * Consumer Affairs indicated in SPR 3.1, which it submitted to the California Department of Technology in January 2015, that the project will end after the phase 2 regulatory entities implement BreEZe, and only after that successful implementation will Consumer Affairs reassess the best implementation approach for the 19 phase 3 regulatory entities. 16 California State Auditor Report 2014-116 February 2015 Issues With BRN’s Implementation of BreEZe After BRN implemented BreEZe in October 2013, it reported experiencing significant delays in the processing of certain types of license applications, which stakeholders asserted had negatively affected the employment of both new and experienced nurses. This issue caused BRN to be the subject of news articles and legislative hearings. As the regulatory entity responsible for implementing and enforcing the Nursing Practice Act—the law pertaining to nursing education, licensure, practice, and discipline—BRN regulates registered nursing and advanced practice registered nurses to ensure consumer protection. State regulations require BRN to process applications for new and renewal licenses according to specific time frames, which we describe further in Chapter 3. BRN’s deputy chief of licensing and administrative services reported that before implementing BreEZe, BRN processed an application to take a nursing examination, which is the first step in the process to receive an original license, in approximately three to eight weeks, using ATS and CAS. In contrast, BRN’s 2014 Sunset Review Report stated that after the implementation of BreEZe, processing times for these application types had extended significantly beyond the 90 days (approximately 13 weeks) specified in regulation as the upper limit for such processing, and that this had contributed to a backlog of applications. Roles and Responsibilities of CalTech and General Services CalTech and General Services have had certain responsibilities for overseeing the State’s IT project procurements and, as shown in Table 4, each entity’s roles have changed since the commencement of the BreEZe project in 2009. At the time that Consumer Affairs executed the BreEZe contracts in 2011, General Services had authority over state agencies’ IT project procurements. Legislation effective in 2011 required General Services and CalTech to review all IT RFPs. Subsequent legislation effective July 2013 transferred General Services’ share of this authority as well as General Services’ authority over IT project procurement to CalTech. As shown in Table 4, presently CalTech is authorized to undertake all key responsibilities related to procuring large IT projects whereas General Services’ responsibilities are limited to contracts for the acquisition of IT goods and services.9 This shift of responsibilities 9 According to state law, all contracts for the acquisition of IT projects exceeding specified thresholds—referred to as reportable IT projects—shall be made by or under the supervision of CalTech. According to CalTech’s Web site, the specified thresholds vary by department and generally range from $200,000 to $5 million. Contracts for the acquisition of IT projects that fall below these specified thresholds are overseen by the respective department and must be reviewed by General Services on a selective basis. California State Auditor Report 2014-116 17 February 2015 from General Services to CalTech, and the increase in CalTech’s responsibilities, reflects the Legislature’s conclusion that the unique aspects of IT projects and their importance to state programs warrant a separate acquisition authority. Table 4 Information Technology Oversight Roles and Responsibilities by Key Milestone of the BreEZe Project FIRST INITIAL REQUEST FINAL RFP AND IMPLEMENTATION FIRST CONTRACT FEASIBILITY STUDY REPORT FOR PROPOSAL CONTRACT AWARD, OF BREEZE, AMENDMENT, KEY RESPONSIBILITY (FSR), NOVEMBER 2009 (RFP), MAY 2010 SEPTEMBER 2011 OCTOBER 2013 FEBRUARY 2014 PRESENT Information technology California Department of CalTech CalTech CalTech CalTech CalTech (IT) project approval Technology (CalTech)† and oversight* Review of IT RFP California Department General Services CalTech and CalTech CalTech CalTech of General Services General Services (General Services) Review of IT Project General Services General Services General Services CalTech CalTech CalTech Procurement Sources: Deering’s California Codes Annotated, 2009–2014; California Government Code, sections 11545 and 11546 (2009, 2011, 2013, 2014); and California Public Contract Code, sections 12100 and 12104 (2009, 2011, 2013, 2014). * IT project approval and oversight includes numerous activities, such as evaluating IT projects based on the business case justification; resource requirements; proposed technical solutions; project management; oversight and risk mitigation approach; and compliance with statewide strategies, policies, and procedures. † Until July 2013 CalTech was known as the California Technology Agency and prior to that the Office of the Chief Information Officer. In addition to its authority to approve IT projects, CalTech is responsible for IT project oversight. Although CalTech’s project oversight policy defines both independent verification and validation (IV&V) and independent project oversight (IPO), CalTech generally provides IPO while state entities undertaking IT projects are responsible to contract for IV&V. However, on the BreEZe project, CalTech has assigned a staff member to perform IPO and it contracted with a private firm to provide IV&V to Consumer Affairs. To ensure independence, contracting directly for IV&V is a practice that CalTech employs based on its assessment of a project’s risk, criticality, and complexity as well as the associated state entity’s staff experience. For purposes of our report, we refer to the individuals who performed these activities as the IV&V consultant and IPO specialist. IV&V provides a client, such as Consumer Affairs, with technically proficient “eyes and ears” to oversee a system vendor while an IT system is being developed and implemented, and it also provides early warning of process and technical discrepancies, issues, and problems that might not otherwise be detected until late in testing or implementation. Without this early feedback, detection of anomalies and the resulting system changes required to correct them are typically delayed until later in the system development 18 California State Auditor Report 2014-116 February 2015 process, resulting in greater costs and schedule delays. In contrast, IPO provides an independent review and analysis of project management practices to determine if the project is being well managed, will be completed within the estimated schedule and cost, and will provide the functionality the client requires, in this case, Consumer Affairs. IPO consists of three main components: review and assessment, reporting, and tracking. Other Troubled IT Projects California Agencies Have Initiated Several California state agencies have experienced difficulty in developing IT systems. For instance, the California State Controller’s Office’s (state controller) 21st Century Project’s MyCalPAYS system faced many difficulties and the state controller has twice terminated project contracts. According to a May 2009 SPR, MyCalPAYS was intended to replace existing statewide human resource management systems with a fully integrated solution, including organizational management and payroll functions, among many others. In April 2005 the state controller contracted with SAP Public Services, Inc. (SAP) for off‑the‑shelf software that would be customized to meet the State’s needs. A year later the state controller hired BearingPoint Inc. as the systems integrator, the primary contractor, to customize the software as necessary to build the new system. According to an August 2013 California Senate Office of Oversight and Outcomes report, in January 2009 the State terminated the contract with BearingPoint Inc. because it failed to develop accurate, reliable data conversion programs and scripts. In February 2010 the state controller hired SAP to develop, test, deliver, and implement the MyCalPAYS system. In June 2012 the first pilot of the project went live with 1,300 state controller employees and revealed numerous errors, including employees being paid too much or too little. In February 2013 the state controller terminated its contract with SAP, and the director of CalTech—the oversight authority for the project—suspended the project. The director of CalTech explained that the department took action on MyCalPAYS because the contractor refused to acknowledge issues on the project and to comply with contract provisions that required the vendor to cure defects the state controller documented. In May 2013 the Legislative Analyst’s Office (LAO) reported that at the time of the contract termination, the State had spent over $262 million on the project. The California Department of Motor Vehicles’ (DMV) IT Modernization Project also faced troubles and was ultimately terminated before its completion. According to an April 2013 LAO report, the goal of the project was to modernize DMV’s driver license and vehicle registration system. In 2007 California State Auditor Report 2014-116 19 February 2015 DMV awarded a $76 million contract to Electronic Data Systems, with CalTech performing oversight. Electronic Data Systems was later purchased in 2008 by HP Enterprise Services. In May 2012 DMV informed HP Enterprise Services that it had serious concerns regarding HP Enterprise Services’ ability to successfully complete the project. Specifically, according to the LAO report, DMV raised concerns regarding vendor staff experience, system delays, and programming language replacement delays. After observing a lack of progress between DMV and HP Enterprise Services for eight months, CalTech exercised its oversight authority to terminate most of the project, except for one nearly finished component. As the project was left incomplete, DMV and CalTech are collaborating to determine the best way to complete the remaining portions of the project. According to CalTech, the total cost of the project as of December 2014 was approximately $208 million. Another recent state IT project that faced challenges during its rollout is the Continued Claims Redesign subproject (subproject) of the Employment Development Department’s Unemployment Insurance Modernization (UI MOD) project. The goal of the subproject was to enhance Web processes, provide claimants with increased self‑service, increase efficiency, reduce mail processing costs, and provide for better detection of fraud. The contract for the subproject was awarded in February 2010 to Deloitte Consulting, LLP. According to status reports, the subproject experienced nine months of delays because of unacceptable levels of defects before its internal system was accepted. As of the October 2014 status report, the subproject was developing a new rollout strategy to use in implementing the online portion of the system. According to a May 2012 LAO report, the UI MOD project, including the subprojects, was initially estimated to cost $96 million, with the Continued Claims Redesign subproject to be completed by June 2008. However, as of December 2014, CalTech listed the UI MOD project cost at nearly $190 million and, according to the October 2014 project status report, the completion date of the Continued Claims Redesign subproject is unknown. Finally, Covered California’s California Healthcare Eligibility, Enrollment and Retention System (CalHEERS) project faced difficulty before and after its implementation. In May 2012 the Covered California News Center announced its intent to award the CalHEERS contract to Accenture for $359 million. The contract includes the initial development and implementation of the system and once the system is in place, the continued development and initial operating costs over approximately three and a half years. CalHEERS is a Web portal that provides eligibility information about affordable coverage and offers health plan choice information, among other things. Because Covered California is an independent public entity and is statutorily excluded from the regular oversight 20 California State Auditor Report 2014-116 February 2015 of many state control agencies, CalTech officials explained that CalTech lacked statutory authority to provide oversight on the project. Rather, Covered California contracted with a private company to provide IV&V for CalHEERS. A May 2013 consultant report, before Covered California opened for business, raised various concerns regarding the project. These concerns included discrepancies between the schedule and key release dates as well as inadequate project management processes. According to a January 2014 progress report produced by the Nicholas C. Petris Center on Health Care Markets and Consumer Welfare at the University of California, Berkeley, after the system was implemented, Web site issues were reported including generally slow response times and repeated error messages. Scope and Methodology The Joint Legislative Audit Committee (audit committee) directed the California State Auditor (state auditor) to conduct an audit of Consumer Affairs’ planning, development, and implementation of its online system for licensing and enforcement—BreEZe—to determine whether the chosen solution was justified and whether the solution is meeting the needs of Consumer Affairs’ regulatory entities. The audit committee also asked the state auditor to determine the BreEZe system’s effect on processing license applications at BRN. Table 5 outlines the audit committee’s objectives and our methodology for addressing each objective. Table 5 Audit Objectives and the Methods Used to Address Them AUDIT OBJECTIVE METHOD 1 Review and evaluate the laws, rules, • Reviewed relevant laws, regulations, and other background materials. and regulations significant to the • Reviewed state policies regarding procurement and information technology (IT) projects. audit objectives. 2 Assess whether the California Department • Interviewed key staff of selected Consumer Affairs’ boards, bureaus, committees, and a of Consumer Affairs (Consumer Affairs) commission (regulatory entities). followed laws, rules, regulations, policies, • Reviewed BreEZe project oversight documentation, such as independent verification and and/or best practices when planning, validation (IV&V) consultant and independent project oversight (IPO) specialist reports from developing, and implementing BreEZe, January 2011 through September 2014. including the level of outreach provided • Interviewed key staff at Consumer Affairs, California Department of Technology (CalTech), and to stakeholders and the adequacy of the California Department of General Services (General Services). training provided to staff. • Consulted an IT expert to identify IT best practices and assessed Consumer Affairs’ compliance with the identified best practices. • Reviewed relevant project management planning documents for the BreEZe system. • Reviewed relevant documents related to the origin and development of the BreEZe system, including training materials for phase 1 regulatory entities. California State Auditor Report 2014-116 21 February 2015 AUDIT OBJECTIVE METHOD 3 Review and assess Consumer Affairs’ • Interviewed the BreEZe project director to determine justification for selecting BreEZe. justification for selecting BreEZe. • Reviewed documentation regarding Consumer Affairs’ justification for selecting BreEZe. • Reviewed BreEZe procurement documentation to determine if Consumer Affairs appropriately assessed and vetted bids. 4 Evaluate whether Consumer Affairs • Reviewed BreEZe project documentation and evaluated whether the oversight Consumer provided sufficient oversight and testing Affairs provided adhered to its internal project planning documents and IT best practices during the planning, development, and during BreEZe planning, development, and implementation. implementation of BreEZe, including • Interviewed key staff at Consumer Affairs and CalTech, including the BreEZe whether the executive office was involved oversight consultant and specialists. in making key decisions and ensuring • Reviewed IV&V and IPO reports to determine whether those oversight services were provided that BreEZe met the needs of the Board of consistently throughout the course of the project. Registered Nursing (BRN). • Reviewed documentation related to results of BreEZe system testing during system development. 5 Review and evaluate all contracts and • Interviewed key staff related to procurement and contract management at Consumer Affairs, contract amendments that Consumer General Services, and CalTech. Affairs entered into with BreEZe’s • Reviewed the three contracts Consumer Affairs has with Accenture LLP related to the developer and determine whether goals planning, development, and implementation of the BreEZe system—design, development, have been met based on the expenditures and implementation; maintenance support; and system license—all amendments, and to date. To the extent possible, determine related procurement documents, to determine the contract requirements and adequacy of whether the State has adequate recourse recourse options. in the event that BreEZe vendors and/ • Consulted an IT expert to assess the reasonableness of the contracts and adequacy of or developers do not fulfill contract contract terms. requirements. • Reviewed 21 key deliverables Consumer Affairs accepted from November 2011 through June 2014. Based on this review, we did not find any reportable issues. 6 Determine the estimated and actual • Interviewed key staff at Consumer Affairs. implementation timeline for BreEZe • Reviewed documentation to determine the BreEZe estimated project implementation timeline and Consumer Affairs’ estimated and and the actual implementation timeline as of January 2015. actual costs. • Reviewed original and revised BreEZe project cost estimates through January 2015. These cost estimates are presented in Table 6 on page 35. • Obtained BreEZe’s financial tracking documentation and reconciled the invoiced costs to the California State Accounting and Reporting System accounting records. 7 Determine and evaluate whether • Interviewed Consumer Affairs and BRN staff. BreEZe’s system design and requirements • Reviewed BRN-specific changes made to the BreEZe system from November 2012 as part impacted BRN’s implementation of user acceptance testing through phase 1 implementation in October 2013, and from timeline and business processes. In implementation through July 31, 2014. addition, determine whether Consumer • Identified the types of defects BRN noted or enhancements BRN requested and whether those Affairs modified BreEZe as a result defects were resolved or enhancements were accepted or rejected. of the regulatory entities’ business processes or whether BRN modified its • Determined whether enhancements were processed in accordance with established processes. business practices to conform with the • Observed changes BRN made to its processes following the implementation of BreEZe. implementation of BreEZe. 8 Determine the average amount of time • Reviewed BRN’s available documentation regarding its tracking of application processing. it took BRN to process licenses before • Interviewed key BRN executive staff to determine how BRN tracked its processing of and after Consumer Affairs implemented license applications. BreEZe. If processing times increased after • Obtained database reports from BreEZe and the legacy systems that identified all BRN implementation, determine the causes for applicants for licensure from July 1, 2011, through July 30, 2014. the increases. • Selected 30 license applications records before BreEZe implementation and 30 applications after BreEZe implementation, and calculated and compared BRN’s time to process the applications before and after implementing BreEZe. 9 Assess the current backlog of license • Because of BreEZe and legacy system information limitations, we counted by hand all of the applications, if any, and determine applications BRN had on site that staff had not yet reviewed as of September 2014, and we what steps Consumer Affairs and BRN calculated the number of days the application had been waiting to be processed. are taking or have taken to address any • Interviewed BRN and Consumer Affairs staff to determine steps taken to address BRN’s backlog. backlog and to provide services for new and renewing applicants. continued on next page . . . 22 California State Auditor Report 2014-116 February 2015 AUDIT OBJECTIVE METHOD 10 Determine and evaluate whether • Interviewed key staff at Consumer Affairs and BRN. Consumer Affairs and/or BRN had • Documented Consumer Affairs’ efforts to address any gaps in staffing, including reviewing contingency plans in place to address Special Project Reports requesting additional staff. potential delays in implementing • Consulted with an IT expert to identify best practices relating to contingency plans. BreEZe and gaps in staff capacity or the processing of license applications. 11 Review and assess any other issues that • Developed a questionnaire and interviewed the 10 phase 1 regulatory entities to identify are significant to the audit. whether their needs were considered in developing the BreEZe system, their satisfaction with their level of involvement in developing and testing the system, their perspective on the adequacy of training, and their overall satisfaction with BreEZe. • Conducted limited research of other state IT projects. • Obtained and reviewed Statements of Economic Interests for key staff involved in the BreEZe project from Consumer Affairs, CalTech, and General Services to assess whether any financial conflicts may exist. Our review covered the period 2009 through 2013, and we did not identify any reportable issues. Source: California State Auditor’s analysis of the Joint Legislative Audit Committee’s audit request number 2014-116, and analysis of information and documentation identified in the column titled Method. California State Auditor Report 2014-116 23 February 2015 Chapter 1 POOR PLANNING AND DECISION MAKING HAVE LED TO INCREASED COSTS AND EXTENDED TIMELINES AND HAVE PLACED THE FUTURE OF BREEZE IN DOUBT Chapter Summary Although doing so is critical to the successful development of information technology (IT) systems, the California Department of Consumer Affairs (Consumer Affairs) failed to adequately plan, staff, and manage the project for developing BreEZe, an IT system it envisioned would support all of the primary functions and responsibilities of its 40 boards, bureaus, committees, and a commission (regulatory entities). For example, the foundation of any IT system is its requirements, which define the business problem to be solved and specify what the system should do. However, Consumer Affairs did not adequately assess its regulatory entities’ business needs to determine BreEZe’s system requirements; instead, it used high‑level, overly general requirements that it had gathered for prior projects it had initiated but abandoned. As a result, the BreEZe project has been plagued with performance problems, delays, and escalating costs, which, based on a January 2015 estimate, were $96 million—more than triple the original estimate—for implementation of a system that will include only half of the regulatory entities originally planned for BreEZe. Moreover, staff of the California Department of Technology (CalTech), in its oversight role, raised significant concerns about the BreEZe project, ranging from the project’s lack of adequate resources to the system’s poor planning and development. Despite the seriousness of these concerns and the various points in the project’s life when they could have used their authority to intervene, both CalTech and Consumer Affairs officials allowed the project to continue without adequately addressing these concerns. Had either party exercised its authority earlier than the summer of 2014—nearly five years after the project began—such as during the planning phase, BreEZe could have been suspended and either rescoped, adequately resourced, or terminated, thereby avoiding potentially unnecessary costs. Because of the many issues the BreEZe project has encountered, particularly its cost increases, Consumer Affairs and CalTech renegotiated the contract with the project vendor, Accenture LLP (Accenture), to remove the regulatory entities originally scheduled to implement the system in the final phase (phase 3). As of January 2015 only 10 regulatory entities had transitioned to BreEZe (phase 1); eight more intend to transition to it in March 2016, and it 24 California State Auditor Report 2014-116 February 2015 is unknown whether or when the remaining 19 will transition to the system. Although Consumer Affairs has indicated it will reassess the best implementation approach for these 19 regulatory entities after the successful completion of phase 2, it has yet to assess the extent of the changes these entities may require be made to the system and it lacks an implementation plan. We believe that should Consumer Affairs elect to implement BreEZe at these 19 regulatory entities, it should include this assessment as part of an analysis of the costs and benefits of moving forward with this implementation to ascertain whether its proposed approach is feasible and cost‑beneficial. Consumer Affairs Did Not Adequately Identify the Business Needs of Its Regulatory Entities When Planning the BreEZe Project Consumer Affairs failed to properly assess the regulatory entities’ business needs when developing the system requirements for BreEZe. According to our IT expert, system requirements define the business problem to be solved and specify what the system should do. For example, a system requirement could be that the system allows a regulatory entity to record the date it receives an application. In its 2014 Chaos Report, the Standish Group presented results of a survey of IT executives that identified a clear statement of requirements is one of the major factors leading to project success and that incomplete requirements are a main factor leading to impaired and ultimately canceled projects. 10 Therefore, Consumer Affairs should have taken steps to ensure that the system requirements for BreEZe were based on the current business needs of its regulatory entities so that the resulting system would Consumer Affairs relied on system be designed and built in a manner that would aid those regulatory requirements from earlier projects entities in conducting their business operations and fulfilling their that were abandoned when regulatory responsibilities. However, when developing the system developing the requirements requirements for BreEZe, Consumer Affairs relied on requirements for BreEZe. from earlier projects that were abandoned. According to the BreEZe project director, Consumer Affairs used the requirements it had identified from three earlier IT project initiatives; the first started as early as 2000 and the last abandoned in 2009. The project director stated that the regulatory entities had been involved in developing the requirements for those earlier unsuccessful initiatives. He also asserted that the system requirements for each of the initiatives were combined into one comprehensive project—BreEZe. Thus, it is likely that the requirements Consumer Affairs had identified for its previous 10 The Standish Group is a primary research advisory organization that focuses on software project performance. California State Auditor Report 2014-116 25 February 2015 initiatives and used as the basis for BreEZe were outdated, further hindering Consumer Affairs’ ability to develop a system that would sufficiently meet its regulatory entities’ needs. Although Consumer Affairs invited the majority of its regulatory entities to participate in refining BreEZe System Request for Proposal the requirements for BreEZe during the request Working Sessions Requirements Statistics for proposal (RFP) process for the project, the • Reviewed over 900 requirements with bidders in the BreEZe project director stated that only some working sessions. regulatory entities participated. During this process in October 2010, Consumer Affairs held working • Deleted over 350 requirements. sessions over a three‑week period with the potential • Revised nearly 500 requirements. bidders and regulatory entities to review and • Created approximately 230 new requirements. modify the system requirements. Consumer Affairs revised hundreds of the more than 900 system Source: BreEZe System Independent Project Oversight Report, December 2010. requirements as a result of the working sessions. The text box presents the statistics of these revisions. However, according to the BreEZe project director, these sessions typically resulted in high‑level requirements that described general functionality, not the specific business processes of the regulatory entities. As we discuss in Chapter 3, most of the executive officers of the phase 1 regulatory entities are generally dissatisfied with BreEZe because it did not meet their expectations in a variety of ways. In addition, Consumer Affairs incorrectly assumed that the regulatory entities could all use similar business processes. For example, the BreEZe project team had the misconception that the regulatory entities used similar processes to collect and record licensing fees, process license applications, and issue licenses. The BreEZe project director explained that Consumer Affairs did not understand the depth and breadth of the regulatory entities’ operations and licensing functions, nor did it understand the lack of flexibility within the regulatory entities to make changes to their processes. Further, he indicated that the assumption of these similarities in the regulatory entities’ processes led Consumer Affairs to intentionally write the system requirements at a high level to attract a vendor with a commercial “off‑the‑shelf” (COTS)‑based solution, such as the BreEZe system. Pressure to Move the Project Forward Caused Consumer Affairs to Expedite the Planning Process and Choose an “Off‑the‑Shelf” Solution The BreEZe project director stated that part of the reason the BreEZe project team, which is composed of Consumer Affairs staff and was responsible for developing the project’s Feasibility Study Report (FSR) and its requirements, did not reach out to the regulatory entities before issuing the RFP was because of pressure the project team 26 California State Auditor Report 2014-116 February 2015 received from Consumer Affairs’ executive office, control agencies, and agency officials to expedite the development of the FSR for BreEZe, which it completed during October and November 2009. The RFP was released six months later, in May 2010. The BreEZe project director perceived that much of the pressure resulted from an internal review of Consumer Affairs’ enforcement processes at the healing arts boards, which regulate a variety of professions, from doctors and nurses to physical therapists and optometrists. Consumer Affairs undertook this review in 2009 in response to delays in certain investigation and enforcement actions that the governor described as “completely unacceptable.” We describe this initiative, the Consumer Protection Enforcement Initiative, in the Introduction. However, when we inquired with Consumer Affairs’ director and its current agency—the California Business, Consumer Services and Housing Agency—neither entity could confirm whether there had been pressure to expedite the FSR. Because Consumer Affairs wanted BreEZe to be developed quickly, it identified as an alternative to its legacy systems a COTS solution, which is an existing, commercially available software product that can be put to use more quickly than a product designed from scratch. At least in part because of faulty assumptions that the COTS In 2009 Consumer Affairs estimated product would require only minor modifications and that minimal that BreEZe would cost about resources and time would be necessary to make these modifications, $28 million. In January 2015 it the estimated costs for the BreEZe project have drastically outpaced estimated it could cost $96 million initial projections. Although Consumer Affairs initially estimated in for implementation and would 2009 that the project would cost about $28 million, in January 2015 include only half of the regulatory it estimated it could cost $96 million for implementation of BreEZe entities that originally planned to and that implementation would include only half of the regulatory use the system. entities that originally planned to use the system. It appears that Consumer Affairs’ selection of the BreEZe COTS product may not have been the appropriate and most cost‑effective decision. In particular, according to the project director, the BreEZe project team believed that similarities in the regulatory entities’ processes and their respective leadership’s commitment to the project vision, including implementing a COTS solution and standardizing the regulatory entities’ processes, would mitigate some of the issues created by using such high‑level requirements. However, the BreEZe project director indicated that there was a disconnect between the expectations of the regulatory entities and the expectations of Consumer Affairs, in that Consumer Affairs expected that the regulatory entities would modify their business processes and the regulatory entities were not expecting to need to do so. In addition, Consumer Affairs’ director noted that Accenture’s approach to understanding the business needs of the phase 1 regulatory entities resulted in a system design that did not reflect the business requirements of the regulatory entities. California State Auditor Report 2014-116 27 February 2015 Consumer Affairs’ director further stated that had Consumer Affairs known of these challenges at the time it was developing the FSR for BreEZe, which contained its analysis of the need for a new IT system and the proposal for a COTS, the department might had chosen to procure the development of a new, fully customized system to meet its needs. Although Consumer Affairs described the development of a fully customized system as an alternative to a COTS in its FSR, it cited unacceptable risks for this type of system, stating that it would have the highest overall costs of any IT solution and would take the longest time to develop and implement. The BreEZe project director explained that Consumer Affairs estimated at the time it developed the FSR that it would cost about $30 million to procure a COTS product, while the development of a new system would have cost around $80 million. Nevertheless, as we previously stated, as of January 2015, the estimated cost of the COTS solution has more than tripled to $96 million and will include only half of the regulatory entities as originally planned. Consumer Affairs Failed to Maintain a Baseline of Requirements for the BreEZe Project A fundamental problem related to the BreEZe system design that the independent verification and validation (IV&V) consultant raised as early as February 2011 was that Consumer Affairs failed to maintain an effective baseline of requirements—the official agreed‑upon set of requirements defining how the system should operate. According to our IT expert, these requirements define the scope of the solution to be delivered and are the basis for cost and schedule estimation, planning, execution, and monitoring. The RFP established the initial baseline of requirements for the BreEZe system, which was included in the design contract.11 Subsequent development activities that refine the requirements must be documented and then they become the new baseline. Our IT expert noted that establishing and maintaining a baseline Our IT expert noted that establishing of requirements is essential and that before a vendor builds a and maintaining a baseline of system, there should be a standard process to refine, confirm, and requirements is essential and that maintain the requirements baseline. This means that for the BreEZe before a vendor builds a system, project, Accenture and Consumer Affairs should have agreed to there should be a standard process and documented clear and measurable requirements for the BreEZe to refine, confirm, and maintain the system that must be satisfied so that both entities can determine requirements baseline. 11 There are three contracts related to the BreEZe project—one contract for design, development, and implementation; another contract for maintenance support; and a third contract for the system license. When we discuss a specific contract, we identify it as either the design, maintenance, or system license contract. 28 California State Auditor Report 2014-116 February 2015 whether the system operates as needed. However, according to the BreEZe project director, Consumer Affairs intentionally did not maintain a baseline of requirements that reflected the requirements for the phase 1 regulatory entities because the design of the system was meant to be established through an iterative process as Accenture defined in its RFP response. According to our IT expert, although an iterative approach can be appropriate, because Consumer Affairs failed to maintain and update a system requirements baseline to reflect any changes identified during this iterative process, it did not effectively support the design and implementation of the system. Because establishing and maintaining a baseline of requirements is a critical process for effectively building a system, our IT expert believes the project should have been suspended when concerns were raised about the quality of the baseline of requirements, and that it should have been revised accordingly, as we discuss later in this chapter. The BreEZe Project Has Experienced Significant Delays The BreEZe project has experienced significant delays at key stages of the project. Figure 1 displays the project’s planned and actual key User acceptance testing for phase 1 milestones since its commencement. As shown in the figure, the regulatory entities spanned most extreme delay involved the key milestone of user acceptance 11 months, significantly exceeding testing—testing that future users conduct of the system that confirms the originally planned eight weeks. that the system operates as the requirements specify. User acceptance testing for phase 1 regulatory entities spanned 11 months, from the end of November 2012 through October 2013, significantly exceeding the originally planned eight weeks. According to the IV&V consultant, this extensive testing period likely resulted in part from the BreEZe system having almost 1,700 unresolved system defects at the beginning of user acceptance testing. According to our IT expert, the bulk of the defects were likely attributable to the poor system requirements. Although user acceptance testing is one of the final and most critical procedures undertaken before system implementation to ensure that the system operates appropriately, this phase morphed into a redesign of the requirements and a rework of the system. Specifically, in conducting the testing of the system, some of the 10 phase 1 regulatory entities as well as Consumer Affairs itself learned that the system did not operate as they expected or needed. For example, during user acceptance testing in January 2013, the average rate at which the system failed on the first attempt to provide the anticipated result was 61 percent. Had Consumer Affairs performed a complete, current assessment of the regulatory entities’ needs to determine the system requirements for BreEZe, it is likely that user acceptance testing could have been completed within a more reasonable time frame, thus avoiding some of the 15‑month implementation delay in phase 1. California State Auditor Report 2014-116 29 February 2015 1 erugiF senotseliM yeK lautcA dna dennalP tcejorP eZEerB 1 esahP PFR sesaeler secivreS lareneG snigeb gnitset metsys tnemtrapeD ainrofilaC sriaffA remusnoC fo detelpmoc si ngised deliated 1 esahP defiilauqerp era sreddiB )sriaffA rem st u im sn b o u C s ( ytilibisaeF eZEerB )RSF( tropeR ydutS —ecnatpecca metsys lluF secivreS lareneG tcejorp detamitsE ylno 2 dna 1 sesahp fo gninepo devorppa noillim 5.72$ :tsoc 2 RPS stimbus sriaffA remusnoC gnitset metsys 1 esahP htiw noitaitogen noillim 9.77$ :tsoc tcejorp detamitsE detelpmoc si stimbus sriaffA remusnoC evisnopser ylno 2 esahp detcejorP 1 )RPS( tropeR tcejorP laicepS PLL erutneccA rodnev noitatnemelpmi ecnatpecca resu 1 esahP noillim 8.54$ :tsoc tcejorp detamitsE )erutneccA( detnemelpmi 1 esahP snigeb gnitset dedrawa era stcartnoC *erutneccA ot ecnatpecca resu 1 esahP tcartnoc laniF etelpmoc si gnitset stnemeriuqer eludehcs tcejorP desaeler era devorppa si 3102 2102 1102 0102 9002 2 esahP 1 esahp etelpmoC noitatnemelpmi gnitset ecnatpecca resu tcejorp e e l v u o d r e p h p c A s yfilauqerP noitatnemelpmi 1 esahP sreddib etelpmoc sesahp eerht lla—ecnatpecca metsys lluF tcartnoc lanfi esaeleR stnemeriuqer 1 esahp nigeB gnitset ecnatpecca resu rodnev tceleS noitatnemelpmi 3 esahP 1 esahp etelpmoC gnitset metsys tcartnoc drawA 1 esahp nigeB secivreS lareneG fo tnemtrapeD ainrofilaC gnitset metsys ’sriaffA remusnoC sesaeler )secivreS lareneG( )PFR( lasoporp rof tseuqer ngised deliated 1 esahp etelpmoC ceD voN tcO tpeS guA luJ nuJ yaM rpA raM beF naJ ceD voN tcO tpeS guA luJ nuJ yaM rpA raM beF naJ ceD voN tcO tpeS guA luJ nuJ yaM rpA raM beF naJ ceD voN tcO tpeS guA luJ nuJ yaM rpA raM beF naJ ceD voN ygolonhceT fo tnemtrapeD ainrofilaC 3 RPS sevorppa )hceTlaC( 3 RPS stimbus sriaffA remusnoC †noillim 4.811$ :tsoc tcejorp detamitsE 6102 5102 4102 ceD voN eht retfa dne lliw tcejorp eZEerB ehT yllufsseccus seititne yrotaluger 2 esahp noitatnemelpmI .metsys eht tnemelpmi yrotaluger 3 esahp eht yb eZEerB fo stsoc eht suht dna ,dennalpnu si seititne .nwonknu era sriaffA remusnoC 3 RPS swardhtiw nuJ yaM rpA raM naJ tcO tpeS guA luJ nuJ yaM rpA raM beF naJ LAUTCA DENNALP gnitseT fo noitaruD gnitset ecnatpecca resu 1 esahP sriaffA remusnoC 1.3 RPS stimbus detamitsE :tsoc tcejorp ‡noillim 1.69$ .slaicffio ’sriaffA remusnoC htiw sweivretni dna ;4102 rebmetpeS fo sa eludehcs tcejorp devorppa ;tropeR ytivitcA thgisrevO tcejorP tnednepednI 1102 rebmevoN ;PFR eZEerB ;sRPS dna RSF eZEerB ’sriaffA remusnoC :secruoS .esnecil metsys eht rof tcartnoc driht a dna ;troppus ecnanetniam rof rehtona ;noitatnemelpmi dna ,tnempoleved ,ngised rof tcartnoc eno—tcejorp eZEerB eht ot detaler stcartnoc erutneccA eerht era erehT * detcejorp eht edulcni ton seod tnuoma siht ,suhT .seititne yrotaluger 2 esahp eht ta eZEerB fo noitatnemelpmi eht hguorht detcejorp sriaffA remusnoC stsoc esoht ylno serutpac 3 RPS ni detneserp etamitse tsoc latot ehT † .3 esahp ni dedulcni yllanigiro seititne yrotaluger eht ta eZEerB gnitnemelpmi rof stsoc ton seod tnuoma siht ,suhT .tcejorp eZEerB eht morf yleritne 3 esahp evomer ot ,4102 ,1 rebmeceD no erutneccA htiw snoitaitogen dedulcnoc seititne eseht retfa hceTlaC ot sriaffA remusnoC yb dettimbus saw 1.3 RPS ‡ .3 esahp ni dedulcni yllanigiro seititne yrotaluger 91 eht ta eZEerB gnitnemelpmi rof stsoc detcejorp eht edulcni 30 California State Auditor Report 2014-116 February 2015 CalTech and Its Contractor Raised Significant Concerns About the BreEZe Project Throughout the Oversight Process CalTech did not begin providing independent oversight of the BreEZe project until roughly one year after it approved the FSR for the project. However, once oversight began, Consumer Affairs failed to respond appropriately to the significant and persistent concerns CalTech raised, while for its part, CalTech did not intervene to ensure that Consumer Affairs took corrective action. Under state law, CalTech is responsible for oversight of IT projects. CalTech generally uses two types of independent oversight: IV&V—used to ensure that a system satisfies its intended use and user needs—and independent project oversight (IPO)—used to ensure that effective project management practices are in place and in use. On the BreEZe project, which CalTech acknowledged is large, complex, and costly, its own staff have provided IPO services, whereas CalTech contracted with MetaVista Consulting—an IT consulting firm—to provide IV&V services. As shown in Figure 2, the IV&V consultant and the IPO specialist raised significant concerns about the project from early 2011, after they began providing oversight, through phase 1 implementation in October 2013. Moreover, in their reports from December 2010 through September 2014, they collectively raised what our IT expert categorized as nearly 180 significant concerns in areas including project management, staffing, system requirements, and vendor performance. We present these concerns in detail in the Appendix. According to our IT expert, the volume and significance of these The volume and significance concerns should have prompted both CalTech and Consumer of concerns raised by the IV&V Affairs to analyze fully the costs and benefits of proceeding consultant and the IPO specialist versus suspending or terminating the project. However, although about the BreEZe project should the IV&V consultant and the IPO specialist presented these have prompted both CalTech and concerns to the BreEZe project team and CalTech management Consumer Affairs to analyze fully in monthly reports, neither entity took sufficient action to ensure the costs and benefits of proceeding that all concerns were appropriately addressed and CalTech versus suspending or terminating allowed the project to continue for more than three years without the project. significant intervention. California State Auditor Report 2014-116 31 February 2015 2 erugiF 3102 rebotcO hguorhT noitpecnI eZEerB morF snrecnoC thgisrevO yeK detceleS dna senotseliM yeK fo enilemiT eZEerB 1 esahP secivreS lareneG fo tnemtrapeD ainrofilaC snigeb gnitset metsys tseuqer sesaeler )secivreS lareneG( tnemtrapeD ainrofilaC lasoporp rof sriaffA remusnoC fo )sriaffA remusnoC( detelpmoc si ngised deliated 1 esahP defiilauqerp era sreddiB stimbus ytilibisaeF eZEerB )RSF( tropeR ydutS secivreS lareneG tcejorp detamitsE fo gninepo devorppa noillim 5.72$ :tsoc 2 RPS stimbus sriaffA remusnoC gnitset metsys 1 esahP stimbus sriaffA remusnoC htiw noitaitogen tsoc noillim 9.77$ :tsoc tcejorp detamitsE detelpmoc si 1 )RPS( tropeR tcejorP laicepS evisnopser ylno eht noillim 8.54$ :tsoc tcejorp detamitsE saw hcihw ,rodnev PLL erutneccA ecnatpecca resu 1 esahP )erutneccA( detnemelpmi 1 esahP snigeb gnitset dedrawa era stcartnoC *erutneccA ot ecnatpecca resu 1 esahP etelpmoc si gnitset laniF tcartnoc eludehcs tcejorP stnemeriuqer devorppa si desaeler era SENOTSELIM TCEJORP YEK SNRECNOC THGISREVO 3102 2102 1102 0102 9002 gnitset ecnatpecca resu hgiH etar eruliaf si 1 esahP revo htiw detnemelpmi gnitubirtnoc si erawtfos ytilauq-rooP ffats delliks etauqeda skcal maet tcejorP seussi devlosernu 000,1 syaled gnitset ecnatpecca resu ot stnemeriuqer etauqedani sniatnoc PFR gnilbuort era secitcarp tnemeganam ksiR yrotaluger eht enilesab ton did sriaffA remusnoC nehw stnemeriuqer lanoitcnuf ’seititne dewollof gnieb ton era sdradnats gnireenigne erawtfoS 1 esahp fo ngised eht gnivorppa gnitroper tsoc ro nalp tnemeganam tsoc evah ton seod tcejorP eludehcs no sksat 1 esahp etelpmoc ot elbaliava si gnffiats tneicffiusnI ot serusaem skcal sriaffA remusnoC detelpmoc krow fo tnuoma enimreted era stnemeriuqer hcihw yfitnedi ot yaw on si erehT ytitne yrotaluger hcae rof detnemelpmi gnieb secruoser ecnarussa ytilauq etis-no oN detareneg ton era stroper nalP noisrevnoC ataD erutneccA snalp tnemeganam tcejorp gniwollof ton si sriaffA remusnoC selbareviled etauqedani gnittimbus si erutneccA dna tsilaiceps thgisrevo tcejorp tnednepednI tnatlusunoc noitadilav dna noitacfiirev tnednepednI tcO tpeS guA luJ nuJ yaM rpA raM beF naJ ceD voN tcO tpeS guA luJ nuJ yaM rpA raM beF naJ ceD voN tcO tpeS guA luJ nuJ yaM rpA raM beF naJ ceD voN tcO tpeS guA luJ nuJ yaM rpA raM beF naJ ceD voN htiw snigeb gnitset ecnatpecca resU seussi devlosernu 286,1 thgisrevO )hceTlaC( s’ygolonhceT fo tnemtrapeD ainrofilaC tsilaiceps thgisrevo tcejorp tnednepedni oN tnatlusnoc noitadilav dna noitacfiirev tnednepedni oN nrecnoC thgisrevO eht desiaR ytraP hcihW setacidnI roloC txeT .tnatlusnoc noitadilav dna noitacfiirev tnednepedni eht dna tsilaiceps thgisrevo tcejorp tnednepedni hceTlaC eht yb stroper dna ,sRPS dna RSF eZEerB ’sriaffA remusnoC :secruoS .esnecil metsys eht rof tcartnoc driht a dna ;troppus ecnanetniam rof rehtona ;noitatnemelpmi dna ,tnempoleved ,ngised rof tcartnoc eno—tcejorp eZEerB eht ot detaler stcartnoc erutneccA eerht era erehT * 32 California State Auditor Report 2014-116 February 2015 The IV&V Consultant and the IPO Specialist Raised Concerns About Consumer Affairs’ Decisions That Significantly Affected Phase 1 of the BreEZe Implementation One of the concerns the IV&V consultant raised in October 2012 was that Consumer Affairs was not maintaining an updated requirements baseline, as discussed previously, and this issue became a significant problem for the BreEZe project. For example, it prevented the IV&V consultant from conducting a critical oversight activity, in that the consultant could not verify whether the system being developed functioned as Consumer Affairs and the regulatory entities needed. This oversight activity, which the IV&V consultant reported as a concern in May 2012, is an ongoing process that occurs throughout the development of a system. The IV&V consultant reported that many of the problems identified during testing of the BreEZe system were related to specific functional requirements of the regulatory entities that Consumer Affairs had not included in the requirements baseline. CalTech and the regulatory entities also expressed concern over Consumer Affairs ending user acceptance testing before all of the problems with BreEZe’s operation had been addressed. Overall, More than 1,000 defects of more than 1,000 defects of various severity levels remained various severity levels remained unresolved at the end of the testing period for phase 1. According to unresolved at the end of the CalTech’s IT project oversight and consulting division’s (oversight testing period for phase 1 of division) branch chief (branch chief), given the severity of some the BreEZe implementation. of the outstanding issues, CalTech advised Consumer Affairs that user acceptance testing should be extended to allow Accenture time to resolve the most significant issues. Separately, several regulatory entities expressed concerns before they began using the system that BreEZe was not functioning as needed. However, Consumer Affairs asserted that it fixed the most critical system issues during the original extended user acceptance testing period and, despite the numerous concerns, it moved forward with the phase 1 implementation. Given CalTech’s authority and the significant concerns it had raised about the status of the project, we question why it did not take steps to ensure that Consumer Affairs heeded its advice to extend user acceptance testing. For instance, CalTech could have formally warned Consumer Affairs that it would suspend the project if the department did not bring the project back into alignment with its planned scope, cost, and schedule. As illustrated in Figure 1 on page 29, the estimated project costs had increased from roughly $28 million to almost $78 million and the project experienced significant delays in its schedule before Consumer Affairs ended user acceptance testing. In addition, CalTech could have required Consumer Affairs to conduct an analysis of the BreEZe project so California State Auditor Report 2014-116 33 February 2015 that CalTech could determine whether the benefits of suspending or terminating the project outweighed the costs of moving forward as planned; however, it has not required such an analysis. The IV&V Consultant and the IPO Specialist Reported That Consumer Affairs Lacks Sufficient Staff to Adequately Manage the BreEZe Project One of the key concerns that both the IV&V consultant and the IPO specialist raised has been that the BreEZe project team lacks appropriate resources. For example, in a June 2013 report, the IPO specialist indicated that the project team was inadequately staffed to complete phase 1. In addition, the IPO specialist reported periodically, beginning in April 2012, that he also had concerns with the resources the regulatory entities committed before and during user acceptance testing for phase 1, as the staff were not assigned exclusively to the project and their availability was in question in the face of project delays. Further, a CalTech branch chief in the oversight division who oversaw the IPO specialist’s work from May 2012 through February 2014 told us that Consumer Affairs did not and still does not have enough skilled personnel to handle the complex BreEZe contracts with Accenture. As a result Consumer Affairs has not had of these combined factors, it appears Consumer Affairs has not had adequate staffing to execute and adequate staffing to execute and implement BreEZe through critical implement BreEZe through critical project phases. project phases. As of June 2014 Consumer Affairs’ executive management acknowledged that primarily because of an expanded project scope, Consumer Affairs needs additional staff to manage the development and implementation of the project through phase 2. As explained in the Special Project Report (SPR) 3, since the inception of the BreEZe project in 2009, the project has been able to borrow from within Consumer Affairs and from the regulatory entities to fill critical project positions and keep the project moving.12 It appears, however, that given the project’s evolution, that is no longer possible. Furthermore, Consumer Affairs did not have sufficient staff to conduct organizational change management at the phase 1 regulatory entities, which is critical for preparing the affected employees to adapt to changes. In the case of BreEZe, organizational change management, including business process analysis, would have identified the changes the regulatory entities 12 An SPR is a report that provides a summary of proposed changes to the original project cost, schedule, or scope. An SPR is generally required when the project costs or total financial program benefits deviate or are anticipated to deviate by 10 percent or more, or a major change occurs in project requirements or methodology. 34 California State Auditor Report 2014-116 February 2015 needed to make to their internal operations so that they could prepare for those changes. According to the BreEZe project director, Consumer Affairs did not undertake organizational change management because the department redirected the funding for that activity to support other higher‑priority project needs, which he defined as every other activity that was completed for the BreEZe project. In other words, Consumer Affairs considered organizational change management to be a lower priority than everything else the project has completed. By ignoring this essential step in project management for phase 1, Consumer Affairs missed a key opportunity to improve the regulatory entities’ transition to using BreEZe and may have contributed to the executive officers of several phase 1 regulatory entities indicating that they felt unprepared to implement the system. Another negative effect of the BreEZe project team not having enough staff is that in August 2014, Consumer Affairs began considering making payments to Accenture to extend the project schedule to complete necessary work. Specifically, in two instances, Consumer Affairs requested that Accenture add a total of 28 days to the project schedule for designing phase 2 at an estimated cost to the State of nearly $2 million. According to Consumer Affairs’ documents requesting the additional days, it does not have sufficient resources available to review certain planning documents for phase 2 within the existing schedule. Despite Significant Cost Increases, CalTech Did Not Significantly Increase Its Involvement on the BreEZe Project Until Summer 2014 Although CalTech was well aware CalTech was well aware of the significant problems with the BreEZe of the significant problems with project, yet it approved additional funding for it. As shown in the BreEZe project, it approved Table 6, estimated project costs have increased from an original additional funding for it. estimate of approximately $28 million in 2009 to more than $96 million in 2015 for implementation of the system at only half of the regulatory entities included in Consumer Affairs’ original cost estimate. Specifically, Consumer Affairs indicated in two SPRs it submitted in June 2011 (SPR 1) and July 2013 (SPR 2) that the project’s estimated costs had increased by nearly 70 percent (SPR 1) and by more than 180 percent (SPR 2) from the original estimate of nearly $28 million as reported in its 2009 FSR. According to CalTech’s analysis of SPR 2, there were three key reasons for this increase as of June 2013. First, the single most important driver for the cost increase was the extended project duration of more than two years past the original completion date of October 2013. Second, the project team planned to hire additional project contractors because it had underestimated the complexity of the BreEZe system. Finally, Consumer Affairs increased the budget for Accenture to perform unanticipated tasks. However, as we discuss California State Auditor Report 2014-116 35 February 2015 later, CalTech did not ensure that Consumer Affairs addressed the CalTech director’s concerns about project management, project governance, and contract management before approving two SPRs that identified significant project cost increases. Table 6 BreEZe Estimated Cost Increases Since the Project’s Inception From November 2009 Through January 2015 (Dollars in Millions) AMOUNT CHANGE PERCENTAGE CHANGE TOTAL COST FROM INITIAL FROM INITIAL DOCUMENT DATE OF ESTIMATE ESTIMATE ESTIMATE IN 2009 ESTIMATE IN 2009 Feasibility Study Report November 10, 2009 $27.5 $0.0 0% Special Project Report 1 June 22, 2011 45.8 18.3 66.5 Special Project Report 2 July 22, 2013 77.9 50.4 183.3 Special Project Report 3* June 25, 2014 118.3 90.8 330.2 Special Project Report 3.1† January 7, 2015 96.1 68.6 249.5 Sources: California Department of Consumer Affairs’ (Consumer Affairs) BreEZe Feasibility Study Report, and Special Project Reports (SPR). * The total cost estimate presented in SPR 3 captures only those costs Consumer Affairs projected through the implementation of BreEZe at the phase 2 regulatory entities. Thus, this amount does not include the projected costs for implementing BreEZe at the 19 phase 3 regulatory entities. Further, although SPR 3 was initially approved by the California Department of Technology (CalTech) in July 2014, Consumer Affairs withdrew the report in September 2014. According to the BreEZe project director, Consumer Affairs withdrew SPR 3 at the direction of CalTech and the California Department of Finance. † SPR 3.1 was submitted by Consumer Affairs to CalTech after these entities concluded negotiations with Accenture LLP on December 1, 2014, to remove phase 3 from the BreEZe project. At the time that CalTech approved SPR 2, it believed the additional project funding was necessary to help ensure BreEZe’s success. When we asked CalTech officials why the department did not become more involved in the project at the time it reviewed and approved SPR 2 in October 2013, CalTech’s oversight division deputy director (deputy director) pointed to CalTech’s analysis for approving SPR 2. In addition, she stated that the additional funding Consumer Affairs had requested was warranted so that the future phases of BreEZe would be better planned and executed than phase 1. Although this may have been the intent of the additional funding, it did not resolve the requirements issues that had negatively affected the BreEZe project. Moreover, in its approval of SPR 2, CalTech did not address all of the key problems it identified in its analysis of the SPR, such as inadequate resources assigned to the project. Thus, we believe that rather than approving SPR 2 and the significant cost increase, CalTech should have taken action to compel Consumer Affairs to address the requirements issues including, if necessary, suspending the project. Furthermore, CalTech had not ensured before approving SPR 3 that Consumer Affairs had addressed the conditions CalTech established in approving SPR 2 in October 2013. For example, 36 California State Auditor Report 2014-116 February 2015 when CalTech approved Consumer Affairs’ SPR 3 in July 2014, it knew that Consumer Affairs should have provided it with a baseline workplan, which is essentially the project schedule, for phase 2 before starting design activities for phase 2, which began in December 2013, according to the BreEZe project director. However, according to an IPO report in June 2014, Consumer Affairs still had not provided the baseline workplan. Even though CalTech’s analysis for SPR 3 acknowledged that Consumer Affairs had not met all of the conditions in CalTech’s approval of SPR 2, CalTech nonetheless approved SPR 3, which increased the estimated cost of the project from roughly $78 million to more than $118 million, along with 14 conditions. Consumer Affairs was to address these conditions in exchange for CalTech’s approval of SPR 3. Further, unlike the previous estimates that reflected the entire cost of the project, the $118 million cost estimate only reflected the costs of the project through phase 2. It was not until after Consumer Affairs estimated the cost of completing the project at $300 million in June 2014 that CalTech changed its oversight approach on the BreEZe project.13 According to the BreEZe project director, Consumer Affairs withdrew its submission of SPR 3 upon direction from CalTech and the California Department of Finance in September 2014. According to CalTech’s oversight division deputy director, CalTech’s vision going forward is to complete the project only for the phase 2 regulatory entities. In fact, she stated that CalTech finalized negotiations with Accenture on December 1, 2014, and removed phase 3 from the design contract, which we describe later. Despite the Serious Oversight Concerns, CalTech Allowed Consumer Affairs to Press Forward With BreEZe Both Consumer Affairs and CalTech acknowledge that they were aware of the oversight concerns raised about the BreEZe project. As was mentioned previously, Consumer Affairs officials were aware of the IV&V consultant’s and the IPO specialist’s concerns regarding the BreEZe project, but they did not take appropriate action to address all of the concerns. Similarly, CalTech did not require Consumer Affairs to remedy the problems the IV&V consultant and the IPO specialist raised, although according to the state chief information officer (CalTech’s director), he has met with IT project teams, such as the BreEZe project team, and Consumer Affairs officials monthly since June 2012 to discuss the issues raised in the IPO reports and project status reports. 13 The BreEZe project team developed the estimate informally and not in the same manner as an SPR requires. California State Auditor Report 2014-116 37 February 2015 Consumer Affairs’ BreEZe project director indicated that he did not always act on the concerns raised in the IPO reports, largely because he believed many of the comments were factually inaccurate. According to the project director, Consumer Affairs communicated this concern regarding inaccuracy to CalTech management in 2013. However, when we asked CalTech’s branch chief in the oversight division whether Consumer Affairs had discussed the accuracy of the reports with anyone at CalTech, she responded that Consumer Affairs had complained only about the sternness of the language in the IPO reports. The BreEZe project director stated that Consumer Affairs does act on concerns the IPO specialist raises that it believes are valid. He also explained that Consumer Affairs has not tracked the IV&V and IPO concerns from the inception of the BreEZe project. Rather, Consumer Affairs has chosen to respond to the concerns that CalTech has asked it about. The BreEZe project director indicated that Consumer Affairs has taken this approach because it lacks resources and he believes that the concerns raised in the reports are not very meaningful. Our IT expert believes that Consumer Affairs’ failure to act in the face of the significant and persistent issues that the IV&V consultant and IPO specialist raised constitutes poor management of the project. For its part, CalTech did not use its authority to ensure that Consumer Affairs addressed the significant and persistent warnings of the IV&V consultant and the IPO specialist, which would have better prepared Consumer Affairs to successfully complete a complex project such as BreEZe. Although state law State law provides CalTech with the provides CalTech with the authority to suspend, terminate, and authority to suspend, terminate, reinstate technology projects, it did not take any of these actions and reinstate technology projects; on the BreEZe project or warn Consumer Affairs in writing that however, it did not take any of these it would take one or more of these actions if the project did not actions on the BreEZe project. meet specific performance measures. However, according to the CalTech director, in the February or March 2013 monthly portfolio meeting, he verbally cautioned officials at Consumer Affairs that CalTech had just suspended or terminated two other projects and that BreEZe would be next if CalTech did not see the department improve its rigor in project management, project governance, and contract management. Nonetheless, CalTech still had concerns in these three areas in July 2014, when it approved SPR 3. According to CalTech’s director, it views its role on a project that is in progress as providing oversight in a manner that supports a successful completion of the project. Separately, according to the branch chief in the oversight division, CalTech believes that departments can seek its assistance, particularly the assigned IPO specialist, with questions or issues that may arise on IT projects. She explained that CalTech also monitors a project to ensure that it is within budget and scope of resources and that a major crash is not imminent; to the extent that these issues do not arise, 38 California State Auditor Report 2014-116 February 2015 CalTech will work with the department in a supportive role. Yet, we question why CalTech did not intervene instead of approving SPR 3, given that Consumer Affairs was not appropriately addressing the concerns the IV&V consultant and the IPO specialist raised; these concerns were numerous and were identified in CalTech’s analysis of SPR 2 in 2013. Although CalTech was aware that Consumer Affairs was not heeding the concerns the independent oversight staff raised, it allowed Consumer Affairs to continue with the project. Despite its awareness that Consumer Affairs has experienced noteworthy difficulties in developing the BreEZe system, CalTech demonstrated through its approval of SPR 3 in July 2014 that the BreEZe project should continue. However, we question this position and CalTech’s decision to allow the BreEZe project to receive additional funding in light of the many concerns its oversight staff have raised. Further, CalTech does not have guidance that would help it evaluate whether projects should continue. Although there is no written guidance, the oversight division’s deputy director explained that she developed a framework since arriving at CalTech in April 2014 for the oversight division to use in evaluating the health of projects. Depending on the magnitude and types of problems a project may have, such as experiencing significant delays in the project schedule or project cost increases, the oversight division escalates concerns through the monthly IPO reports, project steering committee reports, CalTech portfolio meetings, meetings with relevant agency leadership, and with the Governor’s Office. She further stated that when the factors are not being resolved within a reasonable time frame, the oversight division will recommend to the director of CalTech that the project be placed in a pause, suspension, or termination status. Nevertheless, when we asked for the BreEZe project recommendations the oversight division has made to the CalTech director, the oversight division’s deputy director stated that she has not seen any documentation about a recommendation concerning pausing the project or other action for BreEZe. When we asked CalTech’s director in October 2014 his thoughts on halting the BreEZe project, he cited several reasons for letting the project continue uninterrupted: BreEZe is moving in the right direction, the system’s problems are not incurable, and Consumer Affairs and Accenture have been able to work through issues toward resolution. In addition, he stated that the BreEZe project is working and functional. However, according to the September 2014 IPO report, while BreEZe was in production for the 10 phase 1 regulatory entities, it identified that the project’s overall health was poor. The IPO report also states that the online application feature, which the IPO specialist considers a key part of BreEZe, had not been “turned on” for some phase 1 regulatory entities. The IPO California State Auditor Report 2014-116 39 February 2015 specialist concluded that it was inefficient of these entities to not be using the online functionality one year after phase 1 implementation. CalTech’s reasons for approving SPR 2, which are shown in the text box, provide other reasons that California Department of Technology’s it has allowed the BreEZe project to continue. Reasons for Approving the BreEZe System However, CalTech’s explanation for approving Special Project Report 2 SPR 2 and allowing the BreEZe project to continue is perplexing given the actions CalTech and • The BreEZe business case remained valid, and BreEZe could be a benefit to the State. Consumer Affairs have recently taken to remove phase 3 of BreEZe from the design contract with • The project had already completed one of the Accenture. In fact, Consumer Affairs’ SPR 3.1, three phases, and the system continued to stabilize. which it submitted to CalTech in January 2015, • The California Department of Technology (CalTech) indicated the project is not moving in the right expected that future phases would take advantage of the direction and proposed a rescoping of the project benefits of lessons learned and avoid some of the issues to remove phase 3 because of significant concerns associated with phase 1. related to staffing and increasing project costs, and • The project and Special Project Report (SPR) 2 underwent because its contracts with Accenture are no longer extensive scrutiny by all control agencies, and all open financially feasible for Consumer Affairs. issues were receiving attention. As discussed previously, CalTech has the authority • Approving SPR 2 would give the California Department of Consumer Affairs (Consumer Affairs) the funding it needed and responsibility to oversee and approve IT to continue implementing BreEZe for the remaining projects. If CalTech had chosen to suspend regulatory entities, as well as pay for approved changes the project, it could have temporarily paused through the unanticipated tasks budget. BreEZe development and could have provided Consumer Affairs with additional time to conduct Source: CalTech’s analysis of Consumer Affairs’ SPR 2. a cost‑benefit analysis and to correct fundamental problems, such as the baseline of requirements and other requirement issues, it encountered during planning and development. However, according to the CalTech director in October 2014, who has overseen the BreEZe project since Consumer Affairs executed its contracts with Accenture, CalTech has not suspended BreEZe for several reasons, including the following: Consumer Affairs badly needs a new system, suspending the project could cause project delays that might result in Consumer Affairs needing to pay Accenture for those delays, and stopping the project now may affect fixes that were due after phase 1. In addition, CalTech’s director told us that should CalTech terminate BreEZe, it would be abandoning Consumer Affairs. Specifically, the director indicated that although CalTech has the authority to terminate the BreEZe project, it cannot terminate the contracts Consumer Affairs has with Accenture and that terminating the project would leave Consumer Affairs without the means to meet its contractual obligations. We do not agree with the CalTech director’s assessment that terminating the project will leave Consumer Affairs stranded with a contract. CalTech has the authority to terminate the project, and Consumer Affairs could subsequently terminate the contract with Accenture 40 California State Auditor Report 2014-116 February 2015 for convenience—meaning that the State may terminate the contract at its own discretion even if the contractor has performed. In other words, Consumer Affairs could unilaterally terminate the contract, even if Accenture did not breach it, but it would be obligated to pay Accenture for certain costs, such as the costs of completed and partially completed work and winding down operations. In 2011, before Consumer Affairs Despite CalTech’s reasons for not intervening in the beginning awarded the BreEZe contracts, stages of the BreEZe project, had it done so the project might the IV&V consultant reported have become more stable and might have better met the needs significant concerns with the system of the regulatory entities included in phase 1 of the system’s requirements, concluding they were implementation. For instance, in the spring of 2011, before inadequate and posed a high risk. Consumer Affairs awarded the BreEZe contracts, the IV&V consultant reported significant concerns with the requirements Consumer Affairs had defined for the system, essentially concluding that they were inadequate and posed a high risk. If CalTech had chosen to suspend the project at that time and required Consumer Affairs to address the issues with the system requirements, it is likely the project would not have faced the subsequent delays that it has experienced in its schedule. CalTech Did Not Ensure Oversight for BreEZe Until More Than One Year After the Project’s Commencement Consumer Affairs did not secure IV&V services for BreEZe and CalTech did not assign an IPO specialist to the project for over a year after CalTech approved the project’s FSR. State IT policies require that IPO begin immediately following project approval and continue through project closeout. State IT policies hold state entities responsible for establishing required oversight activities, which include both IPO and IV&V services, and since CalTech approved the project’s FSR in November 2009, it was aware of the need for this oversight and should have assigned an IPO specialist at that time, according to CalTech’s oversight division deputy director, given that its practice at the time was to initiate an agreement to do so. For IV&V services, CalTech defers to the corresponding standard of the Institute of Electrical and Electronics Engineers, Inc. (IEEE), which states that IV&V shall be performed in parallel with all system life cycle stages, starting at the very beginning of the process when the system requirements are being defined and continuing throughout the project.14 However, the IPO specialist and the IV&V consultant did not begin work until December 2010 and 14 The IEEE is a leading developer of international standards that support many products and services, including IT. California State Auditor Report 2014-116 41 February 2015 January 2011, respectively, more than a year after the FSR was approved. By that time, Consumer Affairs was about to issue the eighth of what would ultimately be 10 amendments to the RFP. CalTech noted that in state IT policy, the purpose of IPO is to identify and quantify any issues and risks affecting schedule, cost, and system functionality. Similarly, the IEEE standard for IV&V indicates that without early feedback from independent oversight, detection of anomalies and the resulting system changes required to correct them are typically delayed until later in the project, resulting in greater costs and schedule delays. Not surprisingly, cost increases Cost increases and schedule delays and schedule delays are some of the problems that have plagued the are some of the problems that have BreEZe project. plagued the BreEZe project. The CalTech oversight division deputy director stated that CalTech’s practice has been to add oversight staff by a project’s start as identified in its FSR, which for the BreEZe project was December 2009. Moreover, she noted that before July 2014, CalTech assigned an IPO specialist to a project after initiating an interagency agreement with a department, whereas effective July 2014 departments submit service requests for IPO services. In addition, according to available documentation, Consumer Affairs canceled an existing agreement with a private sector IPO in June 2009 because CalTech informed Consumer Affairs that it would assume responsibility for providing IPO services in the following month, which did not occur. Thus, apparently CalTech was aware of the need for IPO services to begin early in the project, but it did not secure them until more than a year later. In July 2014 CalTech streamlined the IPO request procedure by changing it from a process requiring an interagency agreement to one that can be initiated simply by a service request, which the deputy director believes will help ensure that IPO oversight starts earlier than it did under the old system. The CalTech oversight division deputy director told us she is unaware of the circumstances concerning IV&V oversight on the BreEZe project before January 2011. Separately, the BreEZe project director stated that the start of IV&V services was delayed for two reasons. First, he prepared the RFP for IV&V services at the same time that he was preparing the RFP for the overall project, and the project RFP was a higher priority. The BreEZe project director’s need to prioritize among these RFPs is another result of Consumer Affairs’ insufficient project resources, as discussed earlier in this chapter. Second, Consumer Affairs found the initial bids for its IV&V RFP to be higher than it expected, which led it to rescope the IV&V RFP and delay obtaining these services. While the BreEZe project director told us he found many of the IPO specialist’s concerns to be of limited merit, had both forms of oversight been provided from the start of the project and had Consumer Affairs promptly addressed the concerns the 42 California State Auditor Report 2014-116 February 2015 oversight staff raised regarding the system requirements, the project would likely not have faced cost increases and schedule delays of the magnitude that it has. Consumer Affairs Did Not Comply With the State’s Policies for IT Project Management CalTech’s California Project Management Methodology defines a series of IT project management plans that are meant to serve as a formal framework for managing a project, including its cost, scope, and staffing. These plans should be developed during the planning stage of a project. Table 7 shows the 12 necessary plans and indicates whether Consumer Affairs completed and followed each plan. As shown in the table, for the nine plans Consumer Affairs developed, four were not created until after it had awarded the contracts to Accenture, and in seven cases, they were not created until 21 months to 54 months after it received project approval. In September 2011 the IV&V consultant reported a concern that the BreEZe project still lacked a significant number of project plans that were key references for performing his oversight activities. Consequently, execution of the BreEZe project was under way before Consumer Affairs had fully determined some of the processes it would use to manage the project. The completion of the project management plans is a key activity of the planning stage, and if properly completed, these plans would have defined the work that Consumer Affairs would complete in the contract execution stage of the project. As a result, delays by Consumer Affairs in completing the project management plans may have led to omissions on its part. According to CalTech officials, state entities are not required to submit these plans to CalTech; however, in its IPO role, CalTech does review, evaluate, and provide feedback on any such plans. Available documentation suggests that CalTech undertook these activities to some extent on the BreEZe project. In addition, according to CalTech officials, it monitors and provides feedback on a project’s ability to execute these Consumer Affairs failed to develop plans. Not surprisingly, in the areas in which Consumer Affairs IT project management plans for failed to develop the plans entirely—costs, scope management, costs, scope management, and and human resources—CalTech’s IPO specialist and the IV&V human resources, raising significant consultant identified significant concerns. In other cases, such as concerns by CalTech’s IPO specialist organizational change management, Consumer Affairs developed and the IV&V consultant. the plans but failed to implement them. Consequently, several of the phase 1 regulatory entities reported that they were ill‑prepared to adjust to the effects of the new system on their business processes. California State Auditor Report 2014-116 43 February 2015 Table 7 California Department of Consumer Affairs’ Adherence to Necessary Project Plans MONTHS AFTER IF COMPLETED, FEASIBILITY DID CONSUMER WAS THE STUDY REPORT MONTHS AFTER AFFAIRS PLAN DATE PLAN WAS APPROVED CONTRACT AWARD FOLLOW THE PURPOSE OF PLAN COMPLETED? FIRST EFFECTIVE (NOVEMBER 2009) (SEPTEMBER 2011) PLAN? Project Management Plan Scope management plan Documents how the project scope will No – – – – be defined, verified, and controlled, and how the work breakdown structure will be created and defined. Configuration change Describes how to document, control, Yes August 2011 21 0 No control plan and manage any changes to key project components and deliverables throughout the project life cycle. Human resource Identifies how and when labor needs No – – – – management plan will be met to ensure that the project has sufficient staff with appropriate skill sets and experience. Communication Determines the information and Yes September 2011 22 0 Partially management plan communications needs of project stakeholders and how to meet them. Risk management plan Indicates how to approach, plan, and Yes October 2011 23 1 Partially execute risk management activities for the project. Cost management plan Used to plan, estimate, budget, and No – – – – control costs so that the project can be completed within the approved budget. Quality management plan Identifies which quality standards are Yes June 2010 7 0 No relevant to the project and describes how to satisfy them. Schedule management plan Establishes how the project schedule Yes August 2011 21 0 Yes will be managed and controlled by estimating the duration of activities. Procurement management plan Determines which project needs can Yes May 2010 6 0 Partially best be met by purchasing or acquiring products, services, or results outside the project organization, and which project needs can be accomplished by the project team during project execution. Contract management plan Documents the products, services, and Yes May 2014 54 32 Partially results requirements needed to meet the project’s objectives. Other Necessary Plans Organizational change Assesses stakeholders’ awareness and Yes October 2011 23 1 No management plan influence, describes any resistance or concerns, and identifies optimum communication and actions to be taken. Maintenance and operations Describes how the project will be Yes December 2012 37 15 Partially transition plan transitioned from the team executing development to the operations team. Sources: California Department of Technology’s Statewide Information Management Manual, interviews with California Department of Consumer Affairs (Consumer Affairs) officials, and California State Auditor’s review of Consumer Affairs’ BreEZe project management plans and associated oversight reports. 44 California State Auditor Report 2014-116 February 2015 The BreEZe project director cited various reasons for the deficiencies we identified in the execution of the project plans. He said that in a number of cases, inadequate staffing was the reason the project team did not complete project management activities. For some of these management activities, the project director claimed that other activities or reports provide functional equivalencies. For example, Consumer Affairs’ configuration change control plan specifies that it is to complete seven reports regarding requests for changes to the system; however, rather than submitting these reports, the contract manager substituted one log tracking the requested changes. Although this log does contain most of the same information that would have been included in the seven separate reports, it is missing the information required for one of them. In another example of functional equivalency, the project director substituted informal daily interactions with Accenture for the periodic performance management reviews the contract management plan specifies. However, that plan notes that a number of other parties are expected to attend these meetings, including the entire project management team and other appropriate team members. These meetings would likely have offered the team an opportunity to address issues and concerns that were larger than those that arise day to day. Another project management problem occurred when the project director officially acted as the project manager for the greater part of 2014 and thus was responsible for an even greater range of significant duties. Consumer Affairs combined the positions of project director and project manager in January 2014, following the implementation of phase 1 of BreEZe, because the former project director retired. This bundling of positions by Consumer Affairs during 2014 removed an important layer of management oversight, thereby reducing the effectiveness of project management. In fact, the IPO specialist identified this as a concern in January 2014, noting that the positions of project director and project manager In 2014 the BreEZe project are distinct roles that focus on different areas, with the project director officially acted as the director overseeing and providing a second opinion on matters project manager, resulting in for the project manager. Consumer Affairs then separated the the removal of an important layer positions again in June 2014 following CalTech’s direction that the of management oversight, thereby combination of the positions was not supportable over the long reducing the effectiveness of term, and as of mid‑September 2014, Consumer Affairs had hired a project management. project manager. Finally, the State Administrative Manual requires all departments to use the California Project Management Methodology for their large IT projects, and it also requires that all project managers and team leads receive training in this methodology. However, according to Consumer Affairs’ chief information officer, the project director is the only one of the three Consumer Affairs’ BreEZe project team leads who has met this requirement. When staff serving in California State Auditor Report 2014-116 45 February 2015 leadership roles do not fulfill the minimum project management training requirements, assurance is lacking that they have the knowledge necessary to successfully manage a large IT project. Further, this lack of training may have contributed to some of the problems the BreEZe project has encountered. Implementation of BreEZe at All Originally Scheduled Regulatory Entities Is No Longer Planned The future implementation of BreEZe by the phase 2 regulatory entities is at risk, and it is unknown whether or when the 19 phase 3 regulatory entities will implement the system. As we explained in the Introduction, 10 regulatory entities implemented the system in October 2013 as part of phase 1, and another eight regulatory entities are included in phase 2, which is currently planned for March 2016. The remaining 19 regulatory entities—among them some of the largest in terms of licenses issued as we illustrated in Table 3 on page 15—were originally included in phase 3; however, it is now unknown whether or when these entities will begin using BreEZe. Specifically, CalTech informed us that it completed renegotiating Consumer Affairs’ design contract with Accenture on December 1, 2014, and these 19 regulatory entities had been removed entirely from the design contract. Under certain conditions, CalTech has the authority to negotiate IT contracts, including amendments, under Public Contract Code, Section 6611. This section of law pertains to contracts, such as the BreEZe contracts, that are conducted through a negotiation process. Although it has had this authority since July 2013, CalTech has not previously exercised it on the BreEZe project; however, according to the BreEZe project director, it did previously coach Consumer Affairs for earlier negotiations. According to CalTech’s director, CalTech increased its involvement in 2014 because of the BreEZe project’s increasing costs, large scope, and high risk. He also acknowledged various concerns with the design contract, described further in Chapter 2, and stated that the BreEZe project has been a learning experience with respect to contracts, vendors, and expectations. In terms of a new contract amendment that will remove phase 3, a CalTech deputy director explained that SPR 3.1 and a notification to the Legislature are required before one can be executed, and she does not expect to finalize these actions until Only after the successful the first quarter of 2015. In fact, SPR 3.1 indicates that the project implementation of phase 2 will will end after the phase 2 regulatory entities implement BreEZe, Consumer Affairs reassess the and only after the successful implementation of that phase will best implementation approach Consumer Affairs reassess the best implementation approach for for the remaining 19 phase 3 the 19 phase 3 regulatory entities. regulatory entities 46 California State Auditor Report 2014-116 February 2015 Consumer Affairs is requesting over Consumer Affairs has explained that it needs additional staffing $18 million in additional funding to successfully implement BreEZe at the phase 2 regulatory for 34 new dedicated staff and entities and to maintain BreEZe into the future; however, as of indicated that if it does not receive January 2015, it lacks the funding to fill these positions. In SPR 3.1, this funding, it will not be able to in which it is requesting additional funding of over $18 million, complete the BreEZe project. Consumer Affairs is requesting 34 new dedicated staff resources. According to SPR 3.1, Consumer Affairs needs the additional dedicated staff in order to mitigate project risk and successfully complete the remaining phase 2 activities with high quality. In addition, Consumer Affairs states that these positions are needed as soon as possible. According to the SPR, these positions will fill critical project roles in areas such as data conversion support, organizational change management, and maintaining existing IT systems for the remaining regulatory entities not using BreEZe after phase 2 is completed. Further, Consumer Affairs indicated that if it does not receive the additional funding for the positions, it will not be able to complete the BreEZe project. Consumer Affairs lacks a plan to implement BreEZe at the 19 phase 3 regulatory entities that were removed from its design contract with Accenture during recent negotiations. According to SPR 3.1, Consumer Affairs will reassess the best implementation approach for the phase 3 regulatory entities after the successful completion of phase 2; thus, Consumer Affairs does not yet have a plan to implement BreEZe for these 19 regulatory entities. In a December 2014 memorandum to the regulatory entities regarding the future of BreEZe, the director explained that Consumer Affairs anticipates a staggered transition to the BreEZe system for the phase 3 regulatory entities, as opposed to having all 19 make the transition at once. Further, the director stated that the current design contract was not sustainable in terms of both financial and human resources and that the decision to end the contract at the conclusion of phase 2 was the fiscally responsible thing to do, as it will allow Consumer Affairs to better manage and contain costs.15 This perspective is echoed in SPR 3.1, in which Consumer Affairs states that following the implementation for phase 2 of continuous improvements identified during phase 1, it became apparent that the current contract structure presented resource and economic challenges that gave Consumer Affairs and its regulatory entities grave concerns about being able to accurately predict and control project costs and schedule beyond phase 2. Moreover, in its analysis of SPR 3, CalTech highlights the complexity of phase 3 as a risk. Specifically, the analysis states that phase 3 contains some of the most complex and largest regulatory 15 Consumer Affairs only renegotiated the design contract. The other two contracts that Consumer Affairs has with Accenture, the maintenance and system license contracts, remain in place. California State Auditor Report 2014-116 47 February 2015 entities, those with the highest number of license types—160 in total, more than the license types of the phase 1 and phase 2 regulatory entities combined. SPR 3 concludes that this makes phase 3 larger than all of the BreEZe work that will have been accomplished through the completion of phase 2. Despite this complexity, the director of Consumer Affairs acknowledged that Consumer Affairs has not determined the extent to which the business needs of the 19 regulatory entities will require changes to the BreEZe system. Because the magnitude of the changes that these regulatory entities may require is unknown and given the project’s history of significant cost increases and schedule delays, it is likely that Consumer Affairs will encounter similar issues if and when it attempts to implement BreEZe at these phase 3 regulatory entities. Consumer Affairs has also not conducted a formalized cost‑benefit analysis, which would include assessing the magnitude of the changes required by those 19 regulatory entities, to inform its decision about implementing BreEZe at them. A cost‑benefit analysis compares the benefits of a project’s outcomes with the cost required to produce them as well as the costs and benefits of an alternative project. This analysis is of particular importance given that the Budget Act of 2014 mandates an ongoing department‑wide budget reduction of $500,000 for Consumer Affairs beginning in fiscal year 2017–18 in recognition of anticipated operational efficiencies resulting from the implementation of BreEZe by the regulatory entities. Consumer Affairs acknowledges in SPR 3.1 that based on the experience of the phase 1 regulatory entities, there will be no staffing or other operational efficiencies resulting from BreEZe that will produce cost savings. Thus, according to the director of Consumer Affairs, although it will be continuing its design contract with Accenture to complete phase 2, those phase 3 regulatory entities may not realize any operational efficiencies if and when they implement the system. The likelihood of such an outcome occurring underscores the importance of Consumer Affairs conducting an analysis of both the costs and benefits of moving forward with phase 3. CalTech Is Developing a New Approach for Its Oversight Role in the Planning of IT Projects CalTech has initiated a project that, if fully implemented, will expand its role in the planning phase of IT projects and potentially contribute to an increase in the successful execution of reportable IT projects.16 The State Technology Approval Reform (STAR) 16 As of July 1, 2013, CalTech is responsible for all contracts for the acquisition of reportable IT projects with costs that exceed an agency’s delegated cost threshold. 48 California State Auditor Report 2014-116 February 2015 project aims to transform CalTech’s current IT project approval process and assist departments in identifying weaknesses in areas critical to IT project success. However, because CalTech has yet to fully implement the STAR project, it is too early to tell whether it will improve the success rate of IT projects. In September 2013 CalTech initiated In September 2013 CalTech initiated the STAR project, with a the STAR project, with a mission mission of improving the planning, quality, value, and likelihood of of improving the planning, IT project success. According to CalTech’s oversight division deputy quality, value, and likelihood of director, the STAR project will revamp the current FSR process, IT project success. which generally requires a department to document the need for a proposed IT project, along with the cost and benefits of the project, following instructions included in the Statewide Information Management Manual. The deputy director also indicated that the new process will bring greater coordination between CalTech and departments during the planning phase of reportable IT projects. She stated that in coordination with a department, CalTech will evaluate the department’s readiness for its IT project using analyses of multiple stages: a business analysis, an alternatives analysis, a procurement analysis, and a solution analysis. According to a June 2014 presentation, CalTech decided to undergo this change for a variety of reasons, including the fact that the State’s approval of IT projects has been identified by the Governor’s Office, the Legislature, and the State’s policy and IT communities as an area that needs to be revised and updated. CalTech believes this new process will reduce project risk, build collaborative partnerships, increase efficiency, better define project data needs, and improve project approval integrity. CalTech also believes it will provide the opportunity to redirect efforts or even stop projects earlier, when needed, before significant budget actions have been taken. According to the deputy director, as of January 2015 CalTech was in the process of piloting the first two stages of the STAR project and plans to continue doing so through June 2015; however, CalTech had not yet finalized the list of IT projects that will be included in the pilot. She explained that CalTech would prefer to work with departments on a voluntary basis until at least January 2016, before mandating that departments use the new process for their IT projects. Again according to the Caltech’s oversight division deputy director, the STAR project is currently developing procedures for examining an IT project’s risk, its business value, and the department’s organizational readiness to implement the project. As guidance for developing these procedures, CalTech is using, among other information, lessons learned from past IT projects, including lessons learned from BreEZe. As an example, CalTech’s oversight division deputy director noted that on the BreEZe project, a decision was made to use standard commercial software requiring little customization, with insufficient consideration paid to the business needs of the regulatory entities, California State Auditor Report 2014-116 49 February 2015 the dependencies of their processes, and impediments to successful implementation. She explained that the initial planning for the BreEZe project depended on all of Consumer Affairs’ regulatory entities being able to use the same system, which was unreasonable given their different needs. As part of the STAR project, the oversight division deputy director indicated that CalTech also plans to address a lack of critical IT project management leadership experience in state departments. To remedy this issue, CalTech is developing as part of the STAR project a voluntary organizational self‑assessment that will help a department understand its internal strengths and weaknesses in areas critical to IT project success. CalTech hopes these self‑assessments will lead to more informed planning for IT projects, perhaps including the use of CalTech consulting services. Recommendations Legislature To ensure that it receives timely and meaningful information regarding the status of the BreEZe project, the Legislature should enact legislation that requires Consumer Affairs to submit a statutory report annually, beginning on October 1, 2015, that will include the following: • Consumer Affairs’ plan for implementing BreEZe at those regulatory entities included in the project’s third phase, including a timeline for the implementation. • The total estimated costs through implementation of the system at the remaining 19 regulatory entities and the results of any cost‑benefit analysis it conducted for phase 3. • A description of whether and to what extent the system will achieve any operational efficiencies resulting from implementation by the regulatory entities. CalTech To help ensure the success of the BreEZe project going forward, CalTech should do the following: • Ensure that Consumer Affairs responds promptly to, and adequately addresses, concerns raised by its IPO specialist and its IV&V consultant. 50 California State Auditor Report 2014-116 February 2015 • Require Consumer Affairs to develop and follow all project management plans and ensure that the BreEZe project team leads receive all required training. • Document key discussions with Consumer Affairs in which significant concerns are raised about Accenture and assessments of the department’s performance by the IV&V consultant and the IPO specialist. If Consumer Affairs receives the necessary funding and resources to successfully implement BreEZe at the phase 2 regulatory entities and the project continues to face escalating costs, CalTech should require Consumer Affairs to conduct an analysis of the costs and benefits of moving forward with the project as planned or of suspending or terminating the project. Continue implementing the STAR project for state departments and ensure that its framework maintains the following requirements: • Undertakes detailed business process analyses before submitting FSRs to CalTech for approval. • Uses those business process analyses to justify the type of IT solution they are proposing, such as a COTS or a fully customized system. • Uses those business process analyses to inform the resulting RFPs and that they ensure that the RFPs contain adequate requirements. To ensure that IT projects have the oversight needed to better position them for success, CalTech should do the following: • Require state departments to follow its IT policies, including developing all necessary plans and receiving all required training. • Ensure that departments obtain IPO and IV&V services as soon as an IT project is approved. Additionally, CalTech should document key discussions with any department in which the IV&V consultant and the IPO specialist raise significant concerns about a project, and ensure that the respective department responds promptly to, and adequately addresses, the concerns that the IPO specialist and the IV&V consultant raise. • Require state departments to appropriately address deficiencies identified in CalTech’s approval of any SPRs as conditions that must be met, and to do so according to specified timelines. If the deficiencies are not adequately addressed within the California State Auditor Report 2014-116 51 February 2015 specified timelines, CalTech should take action to suspend the project until the department has either resolved the identified deficiencies or adequately documented its justification for not addressing the deficiencies. • Develop thresholds relating to IT project cost increases and schedule delays to inform and better justify its decision to allow an IT project to continue. If a department’s IT project reaches or exceeds these thresholds, CalTech should require the department to conduct a cost‑benefit analysis for the project and include this analysis in an SPR. CalTech should consider the results of this analysis in its decision to approve or deny the SPR and, if warranted, take action to suspend or terminate the project so that it does not allow projects with significant problems to continue without correction. Consumer Affairs To make certain that Consumer Affairs has a project team that consists of staff trained in managing IT projects, it should ensure that all the BreEZe project team leads obtain all required project management training. Consumer Affairs should develop a process to ensure that it prepares all required project management documents and undertakes all oversight activities related to BreEZe as CalTech requires so that it can prevent or identify and monitor future problems as they arise. This includes taking steps to sufficiently respond to any concerns that the IPO specialist and the IV&V consultant raise. To the extent that Consumer Affairs chooses to implement BreEZe at the phase 3 regulatory entities, it should first complete a formal cost‑benefit analysis to ensure that BreEZe is a cost‑effective solution to meet these regulatory entities’ business needs. To make certain this analysis is complete, it should include an assessment of the potential changes these regulatory entities may require to be made of the BreEZe system and the associated costs. Consumer Affairs should complete the cost‑benefit analysis before investing any more resources into the implementation of BreEZe at the phase 3 regulatory entities, and it should update this analysis periodically as significant assumptions change. 52 California State Auditor Report 2014-116 February 2015 If Consumer Affairs determines that a new IT project is warranted in the future, it should develop a process to ensure the success of that project that includes, at a minimum, the following: • System requirements that are specific to each regulatory entity as applicable. • A project team that is qualified and experienced. • Development of and adherence to all project management plans. • Planning and implementation of effective organizational change management. • Timely response to concerns that the IV&V consultant and the IPO specialist raise. • Sufficient staffing. California State Auditor Report 2014-116 53 February 2015 Chapter 2 THE STATE ENTITIES RESPONSIBLE DID NOT ENSURE THAT THE BREEZE CONTRACT TERMS SUFFICIENTLY PROTECT THE STATE Chapter Summary In September 2011, under the direction of the California Department of General Services (General Services), the California Department of Consumer Affairs (Consumer Affairs) awarded three BreEZe contracts to Accenture LLP (Accenture). These contracts do not adequately protect the State. 17 General Services’ role at that time was to administer state information technology (IT) procurements and conserve the fiscal interests of the State. However, during the procurement process for the BreEZe project, General Services and Consumer Affairs approved revisions to the contracts’ terms and conditions proposed by Accenture that transferred risk to the State. In addition, the revised language limited Consumer Affairs’ ability to terminate the contracts and eliminated protections Consumer Affairs otherwise would have had against the possibility of intellectual property rights violations. General Services’ staff explained that the contract revisions reflect what Consumer Affairs was willing to accept. Consumer Affairs staff stated that they were concerned that if they did not agree to the changes Accenture proposed, Consumer Affairs would lose the opportunity to contract with Accenture. Nevertheless, General Services and Consumer Affairs approved contract terms and conditions that do not adequately protect the State’s interests. Although All Vendors Had the Opportunity to Protest the State’s Standard IT Contract Language, Only Accenture Did So General Services and Consumer Affairs agreed to changes to the State’s standard IT general provisions and model contract language (standard IT contract) during two phases of the BreEZe procurement process. These changes are not beneficial to the State.18 During the request for proposal (RFP) phase, General Services provided every potential bidder with the opportunity to 17 There are three contracts related to the BreEZe project—one contract for design, development, and implementation; another contract for maintenance support; and a third contract for the system license. When we discuss a specific contract, we identify it as either the design, maintenance, or system license contract. 18 At the time of the BreEZe procurement, General Services had several modules of standard contract language related to IT contracts. 54 California State Auditor Report 2014-116 February 2015 submit a protest for issues such as the selection of prequalified bidders or RFP requirements before submitting a bid and to have General Services review its concerns. During this phase in the BreEZe procurement process, specifically in January 2011, only Accenture submitted a protest, in which it proposed 44 modifications to provisions of the State’s standard IT contract. Of these proposed modifications, General Services accepted 18, proposed its own revisions to 19, and rejected the remaining seven. Subsequently, in April 2011, in accordance with state law, Consumer Affairs entered into negotiations with Accenture—deemed to be the only responsive bidder. According to its former director, Consumer Affairs believed these negotiations were a prudent and necessary next step to fully explore all possible options before it would consider canceling the BreEZe procurement. The contracts that Consumer Affairs executed in September 2011 and that General Services approved with Accenture for developing BreEZe contained certain terms and conditions that are not included in the State’s standard IT contract language. Specifically, General Services and Consumer Affairs accepted terms and conditions that decrease Consumer Affairs’ ability to obtain a system that meets the needs of its boards, bureaus, committees, and a commission (regulatory entities) and that reduce Consumer Affairs’ financial protections in the event Accenture performs poorly. According to Consumer Affairs’ officials, some of the contract changes they accepted resulted in cost savings. General Services’ chief procurement officer provided the department’s perspective for why it agreed to the contract changes Accenture proposed. First, Consumer Affairs had unsuccessfully attempted to procure a licensing system at least twice before the BreEZe General Services and Consumer procurement. Second, Consumer Affairs described its legacy Affairs accepted contract terms systems as broken because they had significant limitations. and conditions that decrease Consequently, General Services was willing to accept the risk of Consumer Affairs’ ability to obtain the contract revisions so that Consumer Affairs could replace its a system that meets the needs of its legacy systems. However, we still question the prudence of some regulatory entities and that reduce of the decisions it and Consumer Affairs made regarding the terms its financial protections in the event and conditions, as they increased Consumer Affairs’ financial risk Accenture performs poorly. related to these contracts. General Services Significantly Weakened the State’s Financial Protections Should Consumer Affairs Relieve Accenture of Its Contractual Duties As a result of the changes to the State’s standard IT contract that General Services accepted for the BreEZe contracts during the RFP phase, state money is not protected if Consumer Affairs decides to terminate the BreEZe contracts for convenience—meaning that the State may terminate the contract at its own discretion, even if the California State Auditor Report 2014-116 55 February 2015 contractor has performed. General Services’ standard IT contract General Services accepted contract language establishes that the State will have rights to the work language from Accenture that product that is delivered or required to be delivered to the State. limits Consumer Affairs’ ability to However, General Services accepted language from Accenture obtain rights to the work product that limits Consumer Affairs’ ability to obtain rights to the work Accenture builds unless the State product Accenture builds unless the State pays the full amount pays the full amount due under the due under the design contract. Specifically, Consumer Affairs design contract. could have to pay the remaining contract amount rather than only certain of Accenture’s costs through the termination date to obtain rights to Accenture’s work product, such as the source code that Accenture wrote for BreEZe, so that it could be used in the future. This source code is critical because it is part of the programming instructions that operate BreEZe. For example, if Consumer Affairs had terminated the design contract for convenience at the end of September 2014, it could have been required to compensate Accenture for the remaining design contract amount of up to roughly $30 million of the total contract amount of $34 million to obtain Accenture’s work product, unless Accenture was willing to offer it for less. General Services’ chief procurement officer stated that his reading of the amended contract language concerning Consumer Affairs’ rights if it terminates the contract indicates that it provides the State greater flexibility than the standard contract terms because it narrows the scope of the payment Consumer Affairs would need to make in the event of a contract termination. However, unless the project were substantially complete, the amount due under this provision could significantly exceed what would be due under the standard IT contract. Consumer Affairs also risks losing the maintenance contract and system license contract if it terminates the design contract in whole. As a result of their negotiations with Accenture, General Services and Consumer Affairs agreed to include a clause in the two contracts that makes those contracts contingent upon the design contract being in effect. Thus, any complete termination of the design contract terminates the remaining contracts, unless Accenture and Consumer Affairs agree otherwise. By including this clause in the maintenance contract and system license contract, General Services and Consumer Affairs have further impaired the State’s ability to terminate the main design contract. General Services Accepted Terms and Conditions That Place the State at Greater Risk of Business Losses and Legal Liability In addition to the problematic contract provisions just discussed, General Services also accepted contract language that affected the State’s indemnification, or protection, from possible future damage, loss, or injury. General Services’ standard IT contract language includes a section regarding intellectual property that requires 56 California State Auditor Report 2014-116 February 2015 contractors to indemnify the State against third‑party claims for violations of intellectual property rights. This language was in the original RFP for the BreEZe procurement; however, during the RFP phase, General Services accepted Accenture’s proposal to delete a provision that required the contractor to certify that state funds would not be used to violate copyright laws. Further, General Services accepted additional contract language Accenture proposed that restricts the State to certain remedies for a contractor’s possible violations of patents, copyrights, and trade secrets, which effectively prohibits the State from invoking other potential remedies elsewhere in the contract or in law. In our opinion, these changes increase the risk of Accenture violating intellectual property rights by reducing its potential liability, which is a key incentive to avoid such violations. Thus, if Accenture unlawfully used copyrighted software rather than writing its own code, Consumer Affairs could be liable to the copyright holder, depending on the facts of the case. Further, the revised contracts language limits the State’s ability to recover its losses from a violation. Consumer Affairs’ Ability to Protect Itself Against Poor Performance by Accenture May Be Hindered The maintenance contract for BreEZe lacks measures to help ensure that Accenture performs work as Consumer Affairs expects. The best practices contained in the 2011 Information Technology Infrastructure Library’s IT Service Design recommend that a service‑level agreement be created between an IT service provider and the IT customer to define the key service targets and responsibilities of both parties. According to our IT expert, this type of agreement should be included in the BreEZe maintenance contract. For example, a service target that we would expect the maintenance contract to include is the amount of time Accenture can take to respond to and resolve technical problems, based on the severity of the problem. CalTech’s deputy director of the statewide technology procurement division (procurement division) agreed that this type of agreement is common. We also noted that although state agencies are not required to follow it, guidance from the U.S. Department of Health and Human Services indicates that a service‑level agreement should also specify performance guarantees, with associated penalties should the service not be performed as contracted. The BreEZe project director stated that as a means to reduce the contract cost, the maintenance contract did not include a service‑level agreement. However, he did not recall the specific cost savings resulting from this omission. Lacking this agreement, Consumer Affairs is likely hindered from effectively managing the services Accenture should be providing under the maintenance California State Auditor Report 2014-116 57 February 2015 contract. For example, without a service target specifying the amount of time within which Accenture must make design changes to BreEZe, Consumer Affairs could face difficulty in holding Accenture accountable for correcting defects to the BreEZe system in a timely manner. The State Entities Responsible Cited Various Reasons for Approving the BreEZe Contract, Despite Its Inadequacies Given the deficiencies with the BreEZe contracts we describe in this chapter, we question whether General Services adequately protected the financial interests of the State in its oversight role relating to these contracts. Until July 2013 state law required General Services to oversee all procurements for IT projects. As part of those responsibilities, state law at that time also specified that General Services develop and enforce policies and procedures to conserve the rights and interests of the State. In addition, the State Contracting Manual indicates that in approving contracts, General Services assists state agencies by conserving the fiscal interests of the State and preventing imprudent financial acts. According to its chief procurement officer, although General Services will not approve agreements that are illegal or unethical in nature or that violate stated policy, each contracting department decides how much risk it is willing to accept. When we discussed General Services’ chief procurement with the chief procurement officer the BreEZe contract concerns officer agreed that some of the we had identified, he explained that Consumer Affairs and the clauses in the BreEZe contracts State retain a measure of protection from any poorly constructed with Accenture place the State at contract sections by contract provisions that allow the State to additional risk. terminate the project.19 He agreed that some of the clauses in the BreEZe contracts with Accenture place the State at additional risk; however, he indicated that he relied on Consumer Affairs’ attestation that it acknowledged the risks within the BreEZe contracts and would take proper steps to address those risks. However, General Services could not provide us evidence that it had ensured that Consumer Affairs had mitigated the risky provisions. When we asked General Services and Consumer Affairs whether their respective legal counsel had any concerns with the final terms and conditions in the BreEZe contracts, each department asserted that any such information is confidential and they would not consent to its disclosure in our report. 19 For example, the BreEZe contracts have a section that allows Consumer Affairs to terminate the contracts if the Legislature does not appropriate funding for the BreEZe project. 58 California State Auditor Report 2014-116 February 2015 Consumer Affairs’ BreEZe project Consumer Affairs’ BreEZe project director provided two key director told us that if Consumer reasons that the BreEZe contract changes we described in Affairs had not revised the BreEZe earlier sections were necessary. Specifically, he told us that if contract sections to include Consumer Affairs had not revised the BreEZe contract sections Accenture’s proposed changes, it to include Accenture’s proposed changes, it would have risked would have risked losing Accenture losing Accenture as a bidder and would have risked needing to as a bidder and would have risked restart the 18‑month procurement process. Accenture’s letter of needing to restart the 18-month January 2011 during the RFP process proposing contract revisions procurement process. tends to support the project director’s claim that Accenture would not bid on the proposal unless the original contract was significantly changed. While Consumer Affairs’ desire to keep the project moving forward is understandable, the numerous problems it has encountered with the project, as described in Chapter 1, illustrate the danger of rushing to implement a complex IT project without proper safeguards that ensure that the State’s interests are protected and the project’s goals can be achieved within reasonable time frames and costs. General Services and the California Department of Technology Have Failed to Ensure That Consumer Affairs Complies With Contract Requirements Despite a provision in the BreEZe contracts requiring Consumer Affairs to forward all requests for changes to the BreEZe system to General Services for review, neither General Services nor its successor with regard to IT project procurement, the California Department of Technology (CalTech), could demonstrate that they had reviewed any of these requests. As described earlier, General Services approved the original BreEZe contracts. In addition, as of July 2013, CalTech had authority over and responsibility for procurements for IT projects such as BreEZe and was responsible for approving the first BreEZe contract amendment in 2014.20 Provisions of the BreEZe contracts require that Consumer Affairs submit certain contract documentation to General Services for review. Among the requests for changes that General Services or CalTech should have reviewed were those that requested changes to the statement of work in the design contract or a work authorization (change requests); both of these actions could 20 Although the contract refers to General Services, we believe the intent of the parties was to involve whichever state entity was responsible for overseeing IT project procurement during the period covered by the contract, which would be CalTech effective July 1, 2013. California State Auditor Report 2014-116 59 February 2015 amend a contract.21 These change requests originate from project staff, vendor staff, or regulatory entities and seek to change aspects of the BreEZe design requirements. If Consumer Affairs and Accenture determine that a change request is an enhancement to BreEZe not covered by the original BreEZe contract or the system warranty, the regulatory entities must pay Accenture for applicable related costs, according to the BreEZe project director. However, as of late October 2014, neither General Services nor CalTech could demonstrate that they had collectively reviewed all of the more than 250 change requests that Consumer Affairs had considered for approval according to a log Consumer Affairs maintained. Specifically, during the time that General Services had IT project procurement authority, Consumer Affairs considered approximately 100 change requests, while under CalTech’s IT project procurement authority it considered roughly 150. Of the approximately 250 change requests, Consumer Affairs approved 130 Of the approximately 250 contract that have already or will result in additional payments to Accenture change requests, Consumer Affairs of almost $3 million. In addition to these costs, without ongoing approved 130 that have already or review by the responsible oversight entity, there is the risk that will result in additional payments to implemented change requests may improperly exceed the scope of Accenture of almost $3 million. the BreEZe contracts. General Services could not explain why it had not reviewed the change requests. Specifically, General Services’ chief procurement officer stated that, based on his review of the procurement file for BreEZe, it does not appear that General Services was aware of or reviewed the change requests. However, documentation that Consumer Affairs provided us showed that it had provided 11 change requests to General Services in April 2012, which General Services acknowledged receiving at that time. General Services’ chief procurement officer did acknowledge that Consumer Affairs had previously entered into a large number of contractual agreements related to BreEZe in November 2011 without General Services’ knowledge, for which General Services later admonished the department in January 2012. Given this history, we would have expected General Services to have been more diligent in ensuring that Consumer Affairs complied with its contract requirements for the nearly 18 months of General Services’ remaining IT project procurement authority. Separately, CalTech believes it is meeting the contract requirement to review the change requests through its independent project oversight (IPO) specialist for the BreEZe project, but we disagree. CalTech’s deputy director of its procurement division stated that 21 Consumer Affairs issues a work authorization after both it and Accenture accept a work request. According to the Accenture project director, work authorizations include requests for change after BreEZe is implemented and are handled under the BreEZe maintenance support services contract. 60 California State Auditor Report 2014-116 February 2015 her division has responsibility for procurement approval and that CalTech’s IT project oversight and consulting division (oversight division) is responsible for project approval and oversight. She explained that as a result, the oversight division holds the responsibility for reviewing the actual implementation of project management processes and procedures necessary for IT projects, which includes change requests and work authorizations. However, when we asked the deputy director of the oversight division for evidence of those reviews, she provided us a summary of two reviews that occurred in October 2013 and June 2014 rather than evidence of reviews occurring contemporaneously with the change requests. Thus, the reviews that she indicated her division had performed could not timely evaluate potential changes to the scope of the contract. Although Consumer Affairs acknowledged that it has not sent the majority of the change requests to the appropriate entities, it still believes it is in compliance with the contract. Specifically, the manager for change requests for the BreEZe project stated that the IPO specialist, who is a CalTech employee, had access to all of the change request documents. She further explained that the contract does not state when Consumer Affairs will forward the change requests for review, nor does it state that the review is required for any subsequent action. Therefore, Consumer Affairs believes it would be appropriate to forward all the change request documents to CalTech when the contract is complete or upon request. We do not believe this interpretation reflects the intent of the language in the contracts concerning change requests, because submitting change requests after the contract is complete, or upon request, would eliminate the possibility that CalTech could intervene during the contract term if it thought a change request was outside the scope of the contract. Moreover, because neither General Services nor CalTech reviewed the vast majority of the change requests, the State lacks assurance that approved changes did not compensate Accenture inappropriately. Recommendations CalTech To ensure that future IT project procurements do not jeopardize the State’s financial interests, CalTech’s procurement division should do the following: • Document its reasons for approving any deviations from standard contract language. California State Auditor Report 2014-116 61 February 2015 • Ensure that contract language gives departments the rights to the source code necessary to complete the project if a contract is terminated for convenience. • Ensure that contract language contains assurances that state funds will not be used for unauthorized purposes. • Ensure that contracting departments can appropriately manage IT maintenance support contracts by including service‑level agreements in these contracts. To ensure that contracting departments comply with requirements, CalTech should periodically monitor the status of contracts and enforce adherence to procurement requirements. For example, CalTech should ensure that contracting departments forward all change requests to it and that it conducts reviews of all change requests to determine whether the changes exceed the scope of the respective contract. Consumer Affairs To ensure that it complies with all terms of the BreEZe contract, Consumer Affairs should develop and follow a process for doing so, including documenting how it has met applicable contract terms, such as forwarding all change requests to CalTech. 62 California State Auditor Report 2014-116 February 2015 Blank page inserted for reproduction purposes only. California State Auditor Report 2014-116 63 February 2015 Chapter 3 THE BOARD OF REGISTERED NURSING MADE SOME UNSUBSTANTIATED REPORTS OF DIFFICULTIES FROM IMPLEMENTING BREEZE, AND OTHER REGULATORY ENTITIES ALSO REPORT CHALLENGES WITH BREEZE Chapter Summary The Board of Registered Nursing (BRN) indicated that since implementing the BreEZe system, it has faced significant delays in processing applications and has been forced to modify its business processes. However, although BRN asserted that it was exceeding maximum time frames for processing certain applications, we found little evidence that it tracks the information needed to support such claims. Our review of a selection of license applications found that BRN, on average, processed these applications well within allowable maximum time frames. Further, as of September 2014, BRN was facing a significant number of applications that were pending its review—more than 7,000. Because BRN does not track this information, it cannot adequately assess its workload and whether it is using staff appropriately. BRN attributes its reported delays in processing applications to a variety of issues it has encountered in implementing BreEZe, including the cumbersome nature of entering applicants’ information into the system. Although some of the issues it points to are valid, BRN does not track the information necessary to assess their impact. Further, BRN asserts that without additional resources, it will not be able to process license applications within applicable time frames; however, it has not adequately justified its recent request for additional positions. Additionally, in September and October of 2014, we interviewed the executive officers of the regulatory entities the California Department of Consumer Affairs (Consumer Affairs) included in phase 1 of the BreEZe implementation and learned that few of them are generally satisfied with the system. In fact, most of the executive officers reported decreased operational efficiencies for their entities after implementing BreEZe. Nonetheless, most executive officers reported that they are hopeful the system will function adequately once their concerns are addressed. 64 California State Auditor Report 2014-116 February 2015 BRN Lacks Information Critical to Assessing Its Efficiency, and Its Claims About Increased Processing Times Are Unsubstantiated Various BRN stakeholders, including the media and members of the Legislature, have raised concerns recently about the timeliness with which BRN has processed applications since implementing the BreEZe system. Media reports regarding BRN, as mentioned in the Introduction, have largely focused on its untimely processing of license applications received from students who recently graduated from nursing programs and were seeking to take the nursing examination for licensure for the first time—referred to as the application for licensure by examination (examination application). BRN’s 2014 Sunset Review Report indicates that the average processing times for examination applications more than doubled from 37 days in fiscal year 2011–12 before it implemented BreEZe to 82 days in fiscal year 2013–14 after implementing the system.22 BRN reported that when it first Additionally, BRN reported that when it first implemented BreEZe, implemented BreEZe, its staff could its staff could not complete an initial review of an application within not complete an initial review of the 90 days specified by its regulations. Although BRN attributes an application within the 90 days its inefficiency in processing applications to the implementation specified by its regulations. of BreEZe, we found that it is unable to substantiate such claims because it does not track the amount of time it takes to process applications. In fact, BRN could provide little evidence demonstrating that it tracks the timeliness of its application processing. For instance, BRN officials provided examples of some reports they said they had used before implementing BreEZe to track the timeliness within which BRN processed applications. However, we found these reports to be of limited value for assessing specific application processing times. For example, the reports generally presented information on the number of applications received and processed but did not contain the average number of days it took to process applications by type. Only one report that BRN provided presented information on the average number of days it took to process applications; however, again this report did not present these averages by type of application, thus hindering BRN’s ability to identify which types of applications take staff longer to process than others. Further, in providing us with this report, BRN officials informed us that the average number of days for processing was overstated, calling into question the accuracy of the data. Moreover, the assistant executive officer explained that since implementing BreEZe, BRN has been unable to track the timeliness of its processing of license applications using the BreEZe system 22 BRN’s Sunset Review Report is submitted to the Senate Business, Professions and Economic Development Committee and the Assembly Business, Professions and Consumer Protection Committee for their consideration. California State Auditor Report 2014-116 65 February 2015 because the system provides poor‑quality data and does not track the receipt date identified in regulations as the starting point for time measurement. The assistant executive officer added that BRN is working with Consumer Affairs to update BreEZe in order to resolve the issue related to the receipt date. Without this ability, BRN’s deputy chief of licensing and administrative services (deputy chief) said that staff perform a visual review of applications to gauge the length of time they have been pending review. Further, although BRN included average application processing times in its Sunset Review Report for both pre‑ and post‑BreEZe implementation, these averages were created specifically for the report, and the time frames are of limited value for evaluating BRN’s ability to process applications within regulated time frames because of their lack of accuracy. For example, BRN officials acknowledged that the average processing times for license applications presented in the Sunset Review Report include time other than what BRN staff spent processing applications. Specifically, when BRN determines a license application does not contain all of the necessary applicant information, it should notify the applicant of this deficiency and the applicant may submit the information to resolve the deficiency. Because the averages provided in the Sunset Review Report include the time an applicant takes to provide the missing information to BRN, these averages should not be used in evaluating BRN’s ability to process applications within regulatory time frames. In fact, BRN states in the Forward section of the Sunset Review Report that in many instances the data provided are a “best estimate” of the true data and consequently the fiscal year 2013–14 data should be viewed with caution, especially when attempting to compare them to data from previous years. Regulations require BRN to review, within 90 days of receipt, whether an examination application is complete. From the point that BRN accepts an application as complete, it has another 390 days to evaluate whether the applicant meets licensure requirements and to notify the applicant of its decision regarding the application. Thus, BRN conducts its evaluation of this application type in two reviews. Two other ways individuals may apply for licensure are through licensure by endorsement (endorsement), which is similar to the examination application but for out‑of‑state registered nurses seeking a California license, and license by renewal (renewal), which is used to renew currently valid licenses for California registered nurses. According to the deputy chief, BRN’s practice is to perform its review for completeness and licensure requirements simultaneously for these two application types, since these applicants are not required to pass an examination, as is the case with the examination application. 66 California State Auditor Report 2014-116 February 2015 Lacking sound data from BRN to substantiate its claims of increased inefficiencies caused by BreEZe, we conducted our own review of selected license applications to get a sense of the impact BreEZe has had on the efficiency with which BRN processes applications. We selected 30 applications processed before, and another 30 applications processed after, BRN’s implementation of BreEZe, for a total of 60 applications. As shown in Figure 3, our analysis of these applications indicates that BreEZe might not have significantly affected BRN’s ability to process applications within maximum time frames; however, it might have delayed its processing of one application type. Specifically, for endorsement applications included in the applications we reviewed, BRN’s processing time increased from an average of 35 days before implementing BreEZe to an average of 60 days after implementation. Nonetheless, the average processing times indicate that for the limited selection of applications we reviewed, BRN has processed the applications well within the maximum time frames. In the case of BRN’s first review of examination applications, Figure 3 shows that BRN’s average processing times for the applications we reviewed increased by only two days, from an average of 29 days before implementing BreEZe to an average of 31 days after implementation. On the other hand, BRN’s average processing time for its second review of the examination applications we reviewed decreased since implementing BreEZe, from an average of 76 days to 52 days, which is significantly less than the maximum allowed time frame of 390 days. Similarly, BRN’s processing of renewal applications we reviewed also BRN needs to formally track license decreased, from an average of 17 days before implementing BreEZe application processing times to to 14 days after implementation. Although these results are based ensure it is properly identifying on a selection of applications, they highlight the need for BRN to areas where it could improve its formally track such information so it can ensure that it is properly use of staff and its procedures for identifying areas where it could improve its use of staff and its evaluating applications that are procedures for evaluating those applications that are now taking now taking longer to process. longer to process. Because average processing times for the applications we reviewed did not align with the delays BRN reported facing in processing applications, particularly examination applications, for the first several months after implementing BreEZe in October 2013 we reviewed an additional six examination applications received during November 2013 through January 2014. We found that BRN took an average of 62 days to conduct the first review of these applications— double the average time it took BRN to conduct the first review of the examination applications we included in Figure 3, but still well within the 90‑day maximum time frame. The increased processing time for the additional applications we reviewed is consistent with the reported difficulties BRN faced when first implementing BreEZe. Following Figure 3, we describe the reasons BRN stated that it faced delays in processing examination applications. California State Auditor Report 2014-116 67 February 2015 Figure 3 Board of Registered Nursing Average License Application Processing Times for a Selection of License Applications Before and After the Implementation of BreEZe syaD 80 Average Number of Days to Process 76 Pre-BreEZe Implemention 70 Post-BreEZe Implemention Maximum Time Frame 60 60 50 52 40 35 30 31 29 20 17 14 10 0 Application for Original License Application for Application for by Examination* Original License by License by Renewal‡ Endorsement† 90 Days 390 Days 90 Days 60 Days First Review Second Review Application Type Sources: California State Auditor’s analysis of 30 applications processed before BreEZe implementation, for the period July 1, 2011, through October 2, 2013, and 30 applications processed after BreEZe implementation for the period October 8, 2013, through July 31, 2014; the Board of Registered Nursing (BRN) Web site; and California Code of Regulations, sections 1410.1 and 1419.2. Note: There are other types of applications the BRN processes that are included in Table 8 on page 69; however, those application types are not included in this table as they are not applications to obtain a California registered nurse license. * Applications for original license by examination have two reviews. Regulations require BRN to review, within 90 days of receipt, whether an examination application is complete. From the point that BRN accepts an application as complete, BRN has another 390 days to evaluate whether the applicant meets licensure requirements and notify the applicant of its decision regarding the application. † An application for original license by endorsement is an application for a California registered nurse license for applicants who are already licensed in another state or Canada and do not need to take the registered nurse license examination. ‡ An application for license renewal is an application for renewal of a currently valid California registered nurse license. While Figure 3 shows that on average BRN processed the applications that we reviewed within the time frames specified in regulations, there were two instances in which BRN exceeded the maximum time frame. One of these applications was an endorsement application that took 27 days longer than the required 90 days BRN has to complete its review. The second application was a renewal that took 73 days for BRN to review, which exceeded the 60‑day maximum review period. When we asked BRN about these specific applications, BRN’s deputy chief offered several reasons for not meeting the review time frame for the endorsement application: the application was received during the time of year when the 68 California State Auditor Report 2014-116 February 2015 largest number of applications are received, BRN was dealing with several issues related to BreEZe that slowed down application processing, and BRN was focused on examination applications. The deputy chief attributed the delay in BRN’s processing the renewal application in question to a hold that the BreEZe system erroneously placed on the application; staff subsequently identified and corrected the error. BRN reports it is unable to track the number of license applications it has received but has yet to review using BreEZe, and therefore it cannot adequately assess its workload with BreEZe. For the purposes of this report, we refer to applications BRN has received but not yet started to review as its pending workload. Because BRN lacks BreEZe data that would allow us to identify its pending workload, we manually counted the number of applications that made up BRN’s pending workload on a specific day during the course of our audit work. As shown in Table 8, on September 2, 2014, BRN’s pending workload was more than 7,000 applications. Of this number, just less than 1 percent, or 63 applications, of which 54 were applications for endorsement, had exceeded the processing time frame specified in regulations. BRN’s deputy chief explained that some of the applications for endorsement, which comprised almost 40 percent of the applications in its pending workload, were not processed within the maximum 90‑day time frame because BRN has focused its resources on processing a large number of examination applications submitted by recent nursing graduates in June, July, and August 2014. Such decisions further underscore the importance of BRN making decisions that are based upon actual data, in this case the amount, type, and age of applications pending review rather than estimating based on a visual review of its files. BRN Asserts That It Has Faced Various Obstacles in Processing Applications Because of BreEZe According to BRN’s deputy chief, BRN reports that it has faced, and continues to face, obstacles BRN provided system requirements in its implementation of the BreEZe system. BRN’s deputy chief to Consumer Affairs and Accenture, explained that BRN provided system requirements to Consumer but the BreEZe system did not Affairs and Accenture LLP (Accenture), but the BreEZe system did include all of these requirements. not include all of these requirements. As described in Chapter 1, system requirements define the business problem to be solved and specify what the system should do, which is critical to successful system development. Further, BRN told us it did not have adequate opportunity to test the BreEZe system before it was implemented, citing, for example, that the cashiering module did not work when BRN tested the system. The BreEZe project director disputed this claim, stating that each phase 1 regulatory entity had the opportunity to test the system before its implementation and that California State Auditor Report 2014-116 69 February 2015 BRN conducted this testing in June 2013, which included processing several complete transactions that BRN subject matter experts had identified as most important. Nevertheless, BRN’s deputy chief maintained that even the BreEZe sandbox—a version of BreEZe available during user acceptance testing that phase 1 regulatory entities (that is, 40 boards, bureaus, committees, and a commission) could use to simulate completing work in the system—did not provide simulation of real‑world transactions. Rather, BRN staff tested subfunctions of the system, such as a new application being initiated by cashiering, in a controlled environment with planned results. Table 8 Status of Applications Pending Review by the Board of Registered Nursing as of September 2, 2014 MAXIMUM NUMBER OF DAYS AVERAGE TIME FRAME NUMBER OF ELAPSED SINCE DAYS ELAPSED TO EVALUATE APPLICATIONS NUMBER OF THE OLDEST SINCE A LICENSE EXCEEDING APPLICATIONS APPLICATIONS APPLICATIONS APPLICATION FOR THE MAXIMUM TYPE OF APPLICATION PENDING REVIEW WERE RECEIVED WERE RECEIVED COMPLETENESS TIME FRAME License by Examination 761 85 36 90 0 License by Examination (Web-Based) 337 107 34 90 3 License by Examination (Repeat) 438 103 36 90 5 License by Endorsement 2,865 92 45 90 54 License by Renewal 230 62 11 60 1 Advanced Practice Certification 1,206 154 48 NA NA Verification of License 1,325 70 24 NA NA Duplicate License Record 90 25 18 NA NA Re-Activation of License 3 8 8 NA NA Re-Activation of License (8+ years) 4 89 58 NA NA Totals 7,259 63 Sources: California State Auditor’s analysis of applications pending initial review by the Board of Registered Nursing (BRN), and California Code of Regulations, sections 1410.1, 1419.2, 1483, and 1493. Note: As described on page 68, because BRN does not track the number of applications it has received but has not yet reviewed, we manually counted the number pending its review as of September 2, 2014. NA = Not applicable. The BRN deputy chief also indicated that the increased time BRN has reported it takes to process applications is due, in part, to new steps in the process as a result of BreEZe. For example, she explained that since implementing BreEZe, BRN must wait three to five days for Consumer Affairs’ central cashiering unit to produce a certain report that BRN must receive before it can begin to process a license application. Although this additional step in BRN’s process is a permanent change, some other steps will be resolved as BreEZe is updated in the future, such as an extra step BreEZe has caused BRN to take in its processing of temporary license applications, which we describe later. However, because 70 California State Auditor Report 2014-116 February 2015 BRN has not been systematically tracking information pertaining Further, according to the deputy chief, when BRN implemented to the timeliness of its application processing before implementing BreEZe, the system did not notify BRN’s examination vendor of BreEZe or after, including which stages of the process might be applicants that BRN had determined were eligible to sit for the contributing to any delays, BRN could not substantiate the effect of license examination, as BRN expected BreEZe to do, and it was these additional steps on its application processing. several months before the problem was corrected. According to the deputy chief, this function did not work properly from the BRN’s executive officer explained that BreEZe operates much implementation of BreEZe in October 2013 until December 2013. differently than the previous legacy systems and that this has also She indicated that Consumer Affairs and BRN worked together and contributed to delays. For instance, according to the BRN deputy resolved the problem. Nevertheless, this issue caused significant chief, the number of screens within BreEZe requiring information delays for BRN in its processing of applications. Specifically, the from BRN staff has increased. To support its contention that deputy chief stated that because the examination vendor was not BreEZe has increased application processing times, the deputy chief notified of applicants’ examination eligibility, BRN and Consumer shared with us the results of a time study BRN staff conducted in Affairs had to create new eligibility transactions for thousands of January 2014 to determine the amount of time it takes to process applicants. BRN cited this issue as the primary factor contributing applications in BreEZe compared to the legacy systems. BRN’s to the delays faced by nursing program graduates applying for time study indicates that entering information into BreEZe has the examination application and in obtaining employment in taken anywhere from 2.5 minutes to 13 minutes longer for each October 2013 through January 2014. application, depending on the type of application. However, BRN’s time study is of limited value because it does not take into account According to BRN officials, BRN had to develop numerous According to BRN officials, BRN had the total number of applications it processes by type. Without workarounds in all areas of its operations because BreEZe was to develop numerous workarounds including this information, BRN cannot adequately assess the not fully functional when implemented. An example of such in all areas of its operations because impact the increase in processing times has had on its workload a workaround concerns temporary licenses that BRN issues BreEZe was not fully functional by application type and make necessary changes to its use of staff. to certain applicants. The deputy chief explained that from when implemented. Furthermore, the deputy chief indicated in November 2014 that BRN’s perspective, the system should expire temporary licenses the average application processing times with BreEZe have likely automatically when the true license is approved, as this function decreased from what they were in the time study because BRN was reportedly available in the legacy systems; however, this staff have become more comfortable using BreEZe and some of the functionality was not included in the design of the BreEZe system, problems with BreEZe have been fixed. Thus, while the results of and staff have had to monitor and manually expire the temporary the time study may highlight differences in application processing licenses. To resolve issues like these, BRN and all of the regulatory times before and after implementing BreEZe, the results are entities can submit issues they identify to Consumer Affairs for apparently outdated. review. If necessary, Consumer Affairs logs the issue as a system investigation request (SIR). According to Accenture’s BreEZe Of additional concern to BRN is that it lacks confidence in the project director, Consumer Affairs and Accenture work together total number of licensed California nurses reported in BreEZe. to determine if a SIR is a defect or an enhancement. A SIR is The deputy chief explained that according to BreEZe data as of a defect when BreEZe is not operating as designed and, thus, September 2014, the total number of licensed nurses in California Accenture is obligated to correct it. The SIR is an enhancement According to BRN’s deputy chief, was the same as the amount reported by its legacy systems about if BreEZe is operating as designed and to resolve such a SIR when BRN implemented BreEZe, a year earlier. This is disconcerting to BRN because, according requires pursuing a change request. According to the BreEZe the system did not notify BRN’s to the deputy chief, it expects annual growth of approximately project director, regulatory entities pay for such enhancements. examination vendor of applicants 10,000 licensed nurses. However, the former business project According to documentation Accenture’s BreEZe project director that BRN had determined were manager for BreEZe explained that the legacy systems overstated provided, as of September 30, 2014, BRN had submitted the eligible to sit for the license the number of licensees because it sometimes double‑counted highest number of SIRs of any regulatory entity included in phase 1. examination. certain individuals, whereas BreEZe counts each individual Specifically, BRN had submitted over 1,000 SIRs, of which roughly only once. Nonetheless, the deputy chief explained that BRN 200 were submitted after implementing BreEZe. According to is not satisfied with this response and maintains that BreEZe is Consumer Affairs, BRN has paid a total of more than $300,000 for understating the number of licensed nurses. enhancements. Our information technology (IT) expert believes that the sheer volume of the changes BRN requested suggest that Consumer Affairs may not have adequately planned the system. California State Auditor Report 2014-116 71 February 2015 Further, according to the deputy chief, when BRN implemented BreEZe, the system did not notify BRN’s examination vendor of applicants that BRN had determined were eligible to sit for the license examination, as BRN expected BreEZe to do, and it was several months before the problem was corrected. According to the deputy chief, this function did not work properly from the implementation of BreEZe in October 2013 until December 2013. She indicated that Consumer Affairs and BRN worked together and resolved the problem. Nevertheless, this issue caused significant delays for BRN in its processing of applications. Specifically, the deputy chief stated that because the examination vendor was not notified of applicants’ examination eligibility, BRN and Consumer Affairs had to create new eligibility transactions for thousands of applicants. BRN cited this issue as the primary factor contributing to the delays faced by nursing program graduates applying for the examination application and in obtaining employment in October 2013 through January 2014. According to BRN officials, BRN had to develop numerous According to BRN officials, BRN had workarounds in all areas of its operations because BreEZe was to develop numerous workarounds not fully functional when implemented. An example of such in all areas of its operations because a workaround concerns temporary licenses that BRN issues BreEZe was not fully functional to certain applicants. The deputy chief explained that from when implemented. BRN’s perspective, the system should expire temporary licenses automatically when the true license is approved, as this function was reportedly available in the legacy systems; however, this functionality was not included in the design of the BreEZe system, and staff have had to monitor and manually expire the temporary licenses. To resolve issues like these, BRN and all of the regulatory entities can submit issues they identify to Consumer Affairs for review. If necessary, Consumer Affairs logs the issue as a system investigation request (SIR). According to Accenture’s BreEZe project director, Consumer Affairs and Accenture work together to determine if a SIR is a defect or an enhancement. A SIR is a defect when BreEZe is not operating as designed and, thus, Accenture is obligated to correct it. The SIR is an enhancement According to BRN’s deputy chief, if BreEZe is operating as designed and to resolve such a SIR when BRN implemented BreEZe, requires pursuing a change request. According to the BreEZe the system did not notify BRN’s project director, regulatory entities pay for such enhancements. examination vendor of applicants According to documentation Accenture’s BreEZe project director that BRN had determined were provided, as of September 30, 2014, BRN had submitted the eligible to sit for the license highest number of SIRs of any regulatory entity included in phase 1. examination. Specifically, BRN had submitted over 1,000 SIRs, of which roughly 200 were submitted after implementing BreEZe. According to Consumer Affairs, BRN has paid a total of more than $300,000 for enhancements. Our information technology (IT) expert believes that the sheer volume of the changes BRN requested suggest that Consumer Affairs may not have adequately planned the system. 72 California State Auditor Report 2014-116 February 2015 Although BRN Has Taken Steps to Make Its Business Process More Efficient, It Also Is Requesting More Staff Since implementing BreEZe, BRN has taken some steps to review and modify its business process in order to increase its efficiency in processing applications. For example, BRN changed its practice of assigning license applications to evaluators based on the alphabetical order of applicants’ names to having evaluators pull applications from a central location with applications stored according to receipt date, a change we observed. In addition, BRN told us it eliminated the support services duties of its application evaluators so they are able to focus on application reviews. In February 2014 Consumer Affairs provided BRN with a team of employees from Consumer Affairs’ consumer information center and the Bureau of Automotive Repair to assist BRN in processing license applications. The team ultimately spent nearly five months assisting BRN and in June 2014, at the request of Consumer Affairs’ director, the team provided 13 suggestions to BRN’s executive officer on how to increase efficiencies in processing applications. For example, to speed up application processing, the team suggested that BRN forward the fingerprint cards submitted by license applicants to the California Department of Justice for clearance once cashiering confirms payment of the applicable fee. As of November 2014, BRN’s assistant executive officer indicated that it was considering this suggestion. When we asked BRN’s assistant executive officer as to whether BRN has taken steps to implement the remaining suggestions, she told us that either the suggestions were too labor‑intensive to consider implementing at that time or BRN’s processes already reflected the suggestions. For example, although one of the suggestions was that BRN produce procedure manuals describing the process it uses to evaluate applications, it has had such procedure manuals in place since 2005. According to BRN, it has requested additional resources it believes it needs to be able to process license applications within maximum time frames. Specifically, BRN’s Sunset Review Report states that BRN submitted a request to the California Department of Finance for another 26 positions for fiscal year 2015–16. BRN explained that the additional staff would cost approximately $2.6 million to hire. But as described earlier in this chapter, we found the data in the Sunset Review Report to be of limited value for evaluating BRN’s ability to process applications within regulatory time frames. Further, because BRN did not support its request for the 26 additional positions in the Sunset Review Report with specific analysis, we asked BRN to provide us with this analysis. The analysis BRN provided to us was based on data from fiscal years 2011–12 and 2012–13, the two years preceding its implementation of BreEZe, which means the analysis does not reflect its workload and business California State Auditor Report 2014-116 73 February 2015 processes since implementing the BreEZe system. Thus, the additional positions it has requested are not adequately justified by this analysis. Most Regulatory Entities That Implemented BreEZe More Than a Year Ago Report That They Continue to Face Significant Challenges With the System Most of the executive officers of the 10 phase 1 regulatory entities told us that they are generally dissatisfied with their BreEZe experience because it has not met their expectations in a variety of ways. Our interview questions addressed their satisfaction with BreEZe, including their experiences during user acceptance testing and whether they believe the training their staff received was adequate. Further, we inquired about the sufficiency of BreEZe’s reporting capabilities and data accuracy, as well as the regulatory entities’ overall experience with the system as of the date we conducted our interviews, which occurred in mid‑September 2014 through mid‑October 2014. Table 9 on the following page presents responses to selected questions from our interviews with the executive officers, showing that each reported experiencing issues with certain aspects of the BreEZe project. For example, the majority were unsatisfied with the testing they were able to conduct before implementing the system, and most found the training for BreEZe was inadequate. Of greater concern is that eight of the 10 executive officers reported that BreEZe has decreased their regulatory entity’s operational efficiency. Our interviews indicated that Consumer Affairs and Accenture did Our interviews with 10 phase 1 not deliver the system that the regulatory entities expected based regulatory entities indicated that on the design phase of the BreEZe project. Most of the executive Consumer Affairs and Accenture officers we spoke with indicated that they wanted a new system did not deliver the system that for a variety of reasons, including to better meet their business the regulatory entities expected needs. For example, the executive officer of the Physician Assistant based on the design phase of the Board said board staff thought BreEZe would lead to efficiencies BreEZe project. in the licensing and enforcement functions of the board. Most of the executive officers also indicated that Consumer Affairs did an adequate job of obtaining their buy‑in to BreEZe. However, several commented that the system they were originally promised by Consumer Affairs officials is certainly not the system they received. Some of them had assumed that the BreEZe system would effectively perform the functions of the legacy systems in use by the regulatory entities. To illustrate, the Board of Psychology had expected BreEZe to include legacy‑like system functions, such as the ability to manually enter data relating to school codes and 74 California State Auditor Report 2014-116 February 2015 examination no‑shows, but it found that BreEZe did not.23 Similarly, several executive officers reported that the planning work they did initially for BreEZe, in which they described their business processes to Accenture and Iron Data (the builder of BreEZe and an Accenture subcontractor), appeared adequate. However, in hindsight, several other executive officers from different regulatory entities indicated that Accenture and Iron Data exaggerated what the system could accomplish. Table 9 Selected Responses From 10 Regulatory Entities to Questions Regarding Their Experiences With the BreEZe System CONCERNS READY TO SATISFIED ADDRESSED CALIFORNIA DEPARTMENT OF OVERALL SATISFIED SATISFIED IMPLEMENT WITH REPORTS EFFECT ON IN ADEQUATE CONSUMER AFFAIRS (CONSUMER AFFAIRS) SATISFACTION WITH DESIGN SATISFIED WITH BREEZE IN AND DATA OPERATIONAL AND TIMELY REGULATORY ENTITY WITH BREEZE PLANNING WITH TESTING TRAINING OCTOBER 2013 ACCURACY EFFICIENCY MANNER Board of Barbering and Cosmetology Fair Yes No No Yes No Decreased No Board of Behavioral Sciences Fair Yes Yes Yes Yes No Decreased Yes Board of Podiatric Medicine Good Yes Yes Yes No No Increased Yes Board of Psychology Fair No No No No No Decreased No Board of Registered Nursing Poor Yes No No No No Decreased No Medical Board of California Fair No No No Yes No Decreased No Naturopathic Medicine Committee Poor No No No Yes No Decreased No Osteopathic Medical Board Poor No No No Yes No Decreased No Physician Assistant Board Fair No No No No No Decreased No Respiratory Care Board Very Good No Yes Yes Yes No Increased Yes Sources: The 10 Consumer Affairs’ regulatory entities included in phase 1 of the BreEZe implementation. Note: The column titled Overall Satisfaction With BreEZe presents the regulatory entities’ responses to a question regarding their overall satisfaction with the BreEZe system as of the dates we conducted our interviews, which occurred in mid-September 2014 through mid-October 2014. Thus, this column does not represent a summary of the responses we received to the remaining questions presented in this table. Rather, the regulatory entities’ responses to the remaining questions in this table represent their experiences with the BreEZe system at different points in time during the course of the project. Some executive officers also commented that they had informed Iron Data or Consumer Affairs of their specific needs for BreEZe, yet BreEZe did not meet those needs at the time of implementation. For example, the executive officer of the Board of Barbering and Cosmetology (Cosmetology) told us that although Cosmetology communicated to Iron Data during the BreEZe design phase its need for an inspection module in BreEZe to include counting and adding features, Iron Data did not include these features in phase 1. The executive officer explained that Cosmetology found it necessary to submit a change request to have these functions implemented at a cost of $116,000. Similarly, the executive officer 23 In December 2014 the BreEZe project director furnished us with evidence that BreEZe now provides these functions. California State Auditor Report 2014-116 75 February 2015 for the Medical Board of California stated that it has had to use workarounds since BreEZe was implemented to meet a legal requirement to disclose malpractice settlements, because the system does not currently provide a feature that allows the Medical Board of California to access the information necessary to make the disclosures, despite Accenture and Iron Data knowing of this need from the beginning of the BreEZe project. Seven of the 10 regulatory entities told us that they experienced a variety of problems during user acceptance testing. Some of the executive officers commented that testing had a significant impact on daily operations or noted they had to commit qualified staff to the process for many more months than originally planned. Several also noted that the BreEZe system they tested during user acceptance testing was not the system they had been promised during the design phase. As a result, the regulatory entities wrote and submitted many SIRs. According to data Accenture provided, Consumer Affairs and the regulatory entities submitted over 7,300 SIRs in total, including over 5,800 during user acceptance testing and nearly 1,500 following the phase 1 implementation of BreEZe. While the Accenture BreEZe project director provided an estimate that less than 30 percent of the SIRs closed during user acceptance testing were system defects and Accenture’s data show that only 17 percent of the SIRs closed during implementation were system defects, the sheer number of instances—over 7,300—in which BreEZe did not function as its users expected or desired is an indication of the extent of the deficiencies in the system’s design. Another example of these design deficiencies is the approximately $3 million that Consumer Affairs and its regulatory entities have approved for enhancements to BreEZe as of December 2014. According to the BreEZe project director, the regulatory entities The BreEZe project director had the opportunity to specify their needs during the design explained that during user phase of the project, but he believes that Accenture did not acceptance testing, which occurred document the information conveyed in related interviews with the from November 2012 through regulatory entities; instead it proceeded with the design according October 2013, Consumer Affairs to memory recall and incomplete informal notes. Further, he realized that the system Accenture explained that during user acceptance testing, which occurred from built was not what Consumer November 2012 through October 2013, Consumer Affairs realized Affairs had asked for. Consequently, that the system Accenture built was not what Consumer Affairs the regulatory entities have had had asked for. Consequently, the regulatory entities have had to to pay extra for changes to the pay extra for changes to the system in order to have it meet their system in order to have it meet their functional requirements. functional requirements. Some executive officers told us that they communicated their concerns about BreEZe to the director of Consumer Affairs throughout user acceptance testing, largely during weekly meetings the director held with the executive officers. Further, although the regulatory entities prioritize SIRs, a few executive officers stated 76 California State Auditor Report 2014-116 February 2015 that Consumer Affairs or Accenture addressed minor concerns adequately and in a timely manner, but larger, more critical SIRs have remained unresolved, sometimes for months. In response, the BreEZe project director noted that SIRs of greater work effort and complexity, regardless of their criticality, naturally take more time to plan, design, develop, test, and implement than minor SIRs. Nonetheless, we believe this approach may negatively affect the regulatory entities because it can allow significant BreEZe deficiencies to persist for extended periods of time, which in turn interferes with the entities’ ability to meet their business needs. All but three of the regulatory entities we interviewed found that the BreEZe training that Consumer Affairs provided was inadequate. Many of these regulatory entities stated that the training was overly general rather than specific to their business processes, and some indicated that the BreEZe system available during training was of limited functionality and thus did not provide an appropriate training environment. Most of these regulatory entities also told us that the training was untimely, as it was provided roughly nine months before user acceptance testing was completed. Consumer Affairs’ former organizational change manager stated that Consumer Affairs provided both high‑level and specific training, distributed tip sheets and job aids, and offered refresher courses when new functionality was introduced. Nonetheless, training as a whole remained inadequate in the view of the regulatory entities. Although more than half of the executive officers of the phase 1 regulatory entities indicated that they were ready to implement the system in October 2013, many nonetheless had concerns about the system itself. For example, the assistant executive officer of the Osteopathic Medical Board stated that while the board was ready, “the [BreEZe] system was not.” In addition, although the executive officer of the Board of Psychology indicated that the board could have been much more prepared for implementation than it was, the board felt it best to proceed with implementation because of the extended user acceptance testing period; Cosmetology’s executive officer expressed a similar belief. A few of the executive officers of A few of the executive officers commented that Consumer Affairs the phase 1 regulatory entities and Accenture pressured them to certify that they were ready to commented that Consumer implement the system. The director of Consumer Affairs noted Affairs and Accenture pressured that he held weekly meetings with the regulatory entity executives them to certify they were ready to regarding the BreEZe project and that these executives approved implement the system. the BreEZe design and later the product itself at the conclusion of user acceptance testing. In addition, as a matter of record, shortly before the BreEZe implementation, Consumer Affairs had the regulatory entities in phase 1 complete implementation readiness certifications, which were forms that list eight different readiness California State Auditor Report 2014-116 77 February 2015 areas for certification, such as regulatory entity staff have completed planning for business process adjustments following BreEZe implementation and staff are familiar with the procedures for processing work under BreEZe.24 However, on these certifications, several executive officers indicated reservations regarding their readiness for BreEZe implementation. For example, the Osteopathic Medical Board executive officer did not indicate her board’s readiness for external user support. In addition, the assistant executive officer of BRN wrote “concerned” along with her initials for two areas—business process adjustments and familiarity with workarounds needed to facilitate board business upon BreEZe implementation—and also included a statement noting her concern with the number of workarounds the system required. While one of the key selling points of BreEZe was its ability to produce reports, all 10 of the executive officers agreed that its reporting capability remains unsatisfactory. According to some executive officers, at the time it was implemented, BreEZe could not produce any reports for their entities. However, according to a Consumer Affairs manager whose unit develops certain customized BreEZe reports, regulatory entities could generate some reports when Consumer Affairs implemented BreEZe, but they did not have the ability to create customizable reports as they had been able to do under the legacy systems. Several executive officers also told us that they do not believe that the data in BreEZe are accurate, including in some cases the number of licensees for their entities and in other cases amounts related to revenue. A Consumer Affairs Because of system reporting cashiering subject matter expert confirmed that BreEZe sometimes deficiencies, BreEZe is not always assigns revenue to the wrong fiscal year, which can result in providing regulatory entities boards’ fund balances being misstated. Because of these reporting the information necessary to deficiencies, BreEZe is not always providing regulatory entities meet their business needs, such the information necessary to meet their business needs, such as as managing their budgets and managing their budgets and producing annual reports. producing annual reports. According to Consumer Affairs’ director, BreEZe has reporting capabilities; however, the design process to get the reports to meet the business needs of the regulatory entities has been difficult. He also indicated that he assigned a business manager to oversee the planning and organization of the reports. The BreEZe project director explained that as of December 2014, of the 24 report development requests outstanding, 14 have clearly defined requirements and could be deployed within four to five months given current staffing levels, barring any higher priority requests being submitted. The BreEZe project director further explained 24 Two of the regulatory entities that Consumer Affairs included in BreEZe phase 1 did not sign this form. The executive officer of the Naturopathic Medicine Committee did not attend the meeting at which the forms were signed, and a staff member for the Medical Board of California attended in place of the executive director and did not sign the form. 78 California State Auditor Report 2014-116 February 2015 that at the time of implementation, BreEZe produced all of the reports the regulatory entities approved during the design phase, although Consumer Affairs has no documentation of the approvals. The project director also indicated that because the regulatory entities make numerous requests for new reports and changes to existing reports, Consumer Affairs is considering IT solutions, such as business intelligence and data warehouse systems, which would provide them with the independent capability to produce the ongoing reporting changes they desire. Most executive officers believe that once all of the remaining problems with the BreEZe system are fixed, it will meet their business needs. For example, the executive officer of the Board of Psychology stated that the BreEZe system does not yet efficiently serve all of its business needs, but it “can get there.” The project director stated he is unable to determine when all outstanding SIRs for the phase 1 regulatory entities will be resolved. The BreEZe project director added that focusing only on SIRs outstanding as of November 2014, he estimated these could be resolved by August 2015—nearly two years after the implementation of phase 1. Recommendations BRN To ensure that BRN has adequate data to effectively use its resources and manage its workload, it should do the following: • Formally track and monitor the timeliness of its processing of applications by type and the cause of any delays. • Formally track and monitor its pending workload of applications by type and original receipt date. • Conduct an analysis no later than June 30, 2015, of its application processing since implementing BreEZe in order to identify the workload capability of each of its units, such as the licensing support unit; to the extent it determines additional resources are necessary, BRN should submit a request for these resources that is appropriately justified. To ensure that BRN continues to process applications within regulatory time frames, it should continue its efforts to refine its business processes to increase efficiency and reduce the amount of time applications are pending its review. California State Auditor Report 2014-116 79 February 2015 Consumer Affairs Consumer Affairs should continue to work with the phase 1 regulatory entities to ensure that the issues they are facing with BreEZe are being resolved in a timely manner, with particular attention to understanding their reporting needs and improving the system’s reporting capabilities. To ensure that future training for BreEZe system rollouts is timely and effective, Consumer Affairs should: • Provide training on the BreEZe system as close to the rollout date as possible to ensure that staff retain the information for using the system as it is implemented. • Work with the regulatory entities to develop training that is specific to each entity’s business processes. We conducted this audit under the authority vested in the California State Auditor by Section 8543 et seq. of the California Government Code and according to generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives specified in the scope section of the report. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Respectfully submitted, ELAINE M. HOWLE, CPA State Auditor Date: February 12, 2015 Staff: John Billington, Audit Principal Laura G. Kearney Myriam K. Czarniecki, MPA, CIA Sean D. McCobb, MBA Charles H. Meadows III Derek J. Sinutko, PhD IT Expert: Catalysis Group Legal Counsel: Joseph L. Porche, Staff Counsel For questions regarding the contents of this report, please contact Margarita Fernández, Chief of Public Affairs, at 916.445.0255. 80 California State Auditor Report 2014-116 February 2015 Blank page inserted for reproduction purposes only. California State Auditor Report 2014-116 81 February 2015 Appendix SUMMARY OF SIGNIFICANT BREEZE PROJECT CONCERNS NOTED BY INDEPENDENT OVERSIGHT REPORTS, DECEMBER 2010 THROUGH SEPTEMBER 2014 As explained in the Introduction, the California Department of Technology (CalTech) is required to perform oversight of information technology (IT) projects. This consists mainly of two types of independent oversight: independent verification and validation (IV&V)—used to ensure that software conforms to requirements and satisfies user needs—and independent project oversight (IPO)—used to ensure that effective project management practices are in place and in use. On the BreEZe project, which CalTech acknowledges is large, complex, and costly, its own specialist provided IPO services whereas it contracted with MetaVista Consulting to provide IV&V services. During the course of the BreEZe project, the IV&V consultant and the IPO specialist produced monthly reports to document their findings regarding the project. Our IT expert reviewed these reports and produced a table summarizing nearly 180 of the most significant concerns that the IPO specialist and the IV&V consultant reported since they began providing oversight on the project in December 2010 and January 2011, respectively. Table A beginning on the following page presents these concerns and categorizes them according to one or more of the key project oversight areas under which they fall, such as project management and the contract. We describe in Chapter 1 some of the significant and persistent concerns the IV&V consultant and the IPO specialist raised. 82 California State Auditor Report 2014-116 February 2015 A elbaT stropeR thgisrevO tnednepednI yb defiitnedI snrecnoC tcejorP eZEerB tnacfiingiS fo yrammuS 4102 rebmetpeS hguorhT 0102 rebmeceD AERA THGISREVO TCEJORP YEK SMETSYS TCEJORP TCARTNOC RODNEV SECRUOSER ELUDEHCS EPOCS GNIREENIGNE TNEMEGANAM YTILAUQ STNEMERIUQER EUSSI FO YRAMMUS THGISREVO FO EPYT ETAD TROPER x x x x x gnirud defiidom yllaitnatsbus erew stnemeriuqeR tcejorP tnednepednI 0102 rebmeceD .0102 rebmevoN ni weiver rodnev )OPI( thgisrevO x x x x .stnemeriuqer fo ytilauq eht htiw seussI x ni detadpu dna detaerc retrahc tcejorP OPI 1102 yraunaJ no noitamrofni evah ton seod 0102 rebmetpeS litnu tiaw ot stnaw eZEerB dna ,ecnanrevog tcejorp ecnanrevog tcejorp a etaerc ot esahp ngised eht ksir rojam a setaerc siht taht setacidni OPI .erutcurts .yletaidemmi desserdda eb dluohs dna x x x ecnamrofrep metsys dna erutcetihcra taht snrecnoC OPI .denfied yletauqedani erew x x x dna egatrohs ffats ot del ezeerf gnirih taht snrecnoC OPI yeht hcihw rof sksat gnimrofrep maet ni detluser .gniniart ro lliks tneicffiusni dah x x x ot deriuqer level lliks eht ssessop ton seod tcejorP OPI ezis siht fo tcejorp a rof eludehcs etauqeda na dliub .ytixelpmoc dna x x .coh-da era sessecorp tnemeganam tcejorP OPI x .gnippils si eludehcS OPI x x x x detamitserednu evah ot sraeppa maet tcejorP OPI .ytixelpmoc tcejorp x x x noitamrofni detimil tuoba snrecnoc desiaR noitacfiireV tnednepednI tcejbus ytiruces dna lacinhcet )TI( ygolonhcet )V&VI( noitadilaV dna gnirud tupni dna noitapicitrap trepxe rettam .stnemeriuqer fo tnempoleved x x x x ton seod maet tcejorp eZEerB eht taht denrecnoC V&VI eht etelpmoc ot secruoser defiilauq tneicffius evah .daeha krow California State Auditor Report 2014-116 83 February 2015 AERA THGISREVO TCEJORP YEK SMETSYS TCEJORP TCARTNOC RODNEV SECRUOSER ELUDEHCS EPOCS GNIREENIGNE TNEMEGANAM YTILAUQ STNEMERIUQER EUSSI FO YRAMMUS THGISREVO FO EPYT ETAD TROPER x x x x reganam egnahc lanoitazinagro na rof noisivorp oN OPI 1102 yraurbeF .tropeR ydutS ytilibisaeF eZEerB eht ni x x x x ytilanoitcnuf )PFR( slasoporp rof tseuqer eZEerB OPI sdeen laudividni eht sserdda yletauqeda ton seod dna senibmoc ylpmis tI .uaerub ro draob yna fo dna sdraob lla rof stnemeriuqer fo tesrepus a setaerc .enilesab a sa etauqedani ti gnikam suaerub x x x a sa stnemeriuqer PFR sefiitnedi tnemssessa ksiR V&VI .ksir fo ecruos x x x ’sriaffA remusnoC fo tnemtrapeD ainrofilaC ehT V&VI ksir eht rof pets noitagitim tsrfi )sriaffA remusnoC( htiw ecneirepxe tnecer fo kcal a htiw detaicossa noitatnemelpmi dna noitisiuqca metsys egral na edulcni ot rodnev gninniw eht eriuqer ot saw .dib sti ni reganam tcejorp decneirepxe x x seigolonhcet htiw ecneirepxe evah ton seod maeT V&VI .noitulos laitnetop htiw detaicossa x x etad fo tuo si noitatnemucod ecafretni tnerruC V&VI rof desu dohtem tnerruc ehT .etelpmocni dna .etairporppani si secafretni eht gnitnemucod x x x x x tcejorp decneirepxe dna elbaegdelwonk fo kcaL V&VI regnol ehT .nrecnoc rojam a era secruoser ffats eht retaerg eht ,desserdda gnieb tuohtiw seog siht ro enod eb ton lliw sksat yrassecen taht doohilekil .tcejorp eht ot etauqeda eb ton lliw x x .eludehcs dniheb gninnur puorg ]sisylana[ ecafretnI OPI 1102 hcraM x .gnilbuort era secitcarp tnemeganam eussi dna ksiR V&VI x x x x eludehcs tcejorp eht netaerht secruoser tneicffiusnI OPI 1102 lirpA esuaceb tsoc desaercni ni tluser lliw dna yltnacfiingis .egaugnal tcartnoc revo snoitaitogen fo x x x x era stnemeriuqer taht rucnoc V&VI dna OPI OPI .)elbatsetnu ,etelpmocni ,suougibma( etauqedani x x gnikat yb stsoc rewol ot skees sriaffA remusnoC OPI 1102 yaM etipsed ,tnempoleved ecafretni rof ytilibisnopser .esitrepxe fo kcal dna egatrohs ecruoser x x x tub tsixe sdradnats gnireenigne erawtfos taht setoN V&VI .dewollof gnieb ton era . . . egap txen no deunitnoc 84 California State Auditor Report 2014-116 February 2015 AERA THGISREVO TCEJORP YEK SMETSYS TCEJORP TCARTNOC RODNEV SECRUOSER ELUDEHCS EPOCS GNIREENIGNE TNEMEGANAM YTILAUQ STNEMERIUQER EUSSI FO YRAMMUS THGISREVO FO EPYT ETAD TROPER x x gniniatbo syaled tnacfiingis gnicneirepxe si tcejorP OPI 1102 enuJ .slavorppa ytitne yrotaluger x x x x naht eludehcs evissergga erom a sesoporp rodneV OPI sriaffA remusnoC dna detseuqer sriaffA remusnoC sgnidnfi OPI dna V&VI etipsed eludehcs taht stpecca decneirepxe dna delliks fo srebmun tneicffiusni no sriaffA remusnoC fo noisnapxe na dna secruoser no gnikat sriaffA remusnoC morf gnitluser epocs eht ni ecirp rodnev ecuder ot krow lanoitidda .noitaitogen tcartnoc x .deganam ylevitceffe gnieb ton era sksiR V&VI x x x .dewollof gnieb ton si ssecorp tnemeganam egnahC V&VI x x x remusnoC eht taht nrecnoc desserpxe sah V&VI OPI 1102 yluJ ot ecneirepxe yrassecen eht skcal maet TI ’sriaffA remusnoC taht won metsys desoporp eht niatniam desuoh hceTO morf egnahc ot esohc sriaffA si taht eno ot noitarugfinoc erawtfos/erawdrah hcihw( retneC ataD detaredeF eht ni deganam eht ni lortnoc sriaffA remusnoC erom swolla .)sehctap fo tnemeganam x x x x ton era sessecorp tnemeganam noitarugfinoC OPI na eb ot mees ton seod erehT .defiitnedi ylraelc evah stnemucod hcihw enimreted ot yaw suoivbo .devorppa yllamrof neeb x x x era stnemeriuqer ssenisub dna lacinhcet deliateD OPI .PFR eZEerB eht ni dedulcni ton x x dna nalp tnemeganam tsoc a evah ton seod tcejorP OPI on si erehT .ecalp ni eno tup ot snalp evah ton seod .gnitroper tsoc x x x x x gnisu stnemeriuqer gninfier neeb sah maet eZEerB OPI DFR esuaceb ;)sDFR( smargaid wofl stnemeriuqer rodnev eht ,tcartnoc eht fo trap ton era stnemucod .noitatnemucod eht esu ot deriuqer ton si x x detaicossa era sksir defiitnedi fo rebmun egral A V&VI 1102 tsuguA tneuqesbus dna PFR eht ni stnemeriuqer eht htiw suoires tuohtiw deretla eb tonnac dna tcartnoc .tnemerucorp eht ot stcapmi evitagen x x x lanoitazinagro ’sriaffA remusnoC tuoba snrecnoC V&VI dna dehsilbatse fo kcal eht ot detaler ,ytirutam .metsys TI na gniniatniam rof ,sessecorp detnemucod California State Auditor Report 2014-116 85 February 2015 AERA THGISREVO TCEJORP YEK SMETSYS TCEJORP TCARTNOC RODNEV SECRUOSER ELUDEHCS EPOCS GNIREENIGNE TNEMEGANAM YTILAUQ STNEMERIUQER EUSSI FO YRAMMUS THGISREVO FO EPYT ETAD TROPER x x gninnur si gninnalp tnemeganam tcejorp stropeR V&VI .eludehcs dniheb x x gnitnemucod ot hcaorppa na sesu tcejorP V&VI retaerg a ni tluser lliw seveileb V&VI taht secafretni ecafretni eht ngiseder ot troffe gnireenigne-er .eZEerB rof x x gnieb noitatnemucod ecafretni taht setaretier V&VI OPI 1102 rebmetpeS ot desu dohtem eht dna etelpmocni si decudorp .etairporppani si secafretni eht tnemucod x x x x x x ytiliba eht skcal sriaffA remusnoC taht denrecnoC V&VI 1102 rebotcO eht ni tnetnoc derised enfied dna yfitnedi ot era taht )sDED( stnemucod noitatcepxe elbareviled .rodnev eht htiw detaitogen gnieb x x x x selbareviled fo noitpircsed )WOS( krow fo tnemetatS V&VI rof elbisnopser si sriaffA remusnoC dna roop si hcae ni dedulcni eb dluohs tahw yltcaxe gninfied .elbareviled x x x .evissergga si eludehcs eZEerB OPI 1102 rebmevoN x x rodnev neewteb sgniteem gnirud taht sevresbO V&VI evah ffats ’sriaffA remusnoC ,sriaffA remusnoC dna ”.gnihsarht“ fo tnuoma tnacfiingis a deyalpsid neewteb snoissucsid htrof-dna-kcab snaem sihT yllaunitnoc ni tluser taht ffats ’sriaffA remusnoC ot edam sdeen rodnev eht taht noisiced a gnignahc .tnempoleved htiw eunitnoc x x x x x )erutneccA( s’PLL erutneccA ,WOS eht ot gnidroccA V&VI nalP tnempoleveD dna noitarugfinoC metsyS erawtfos sti ebircsed dluohs )ygolodohtem( gnidrocca ,revewoH .)CLDS( elcycefil tnempoleved eht ,DED devorppa-sriaffA remusnoC eht ot sessecorp CLDS edulcni ton lliw ygolodohtem noisulcxe eht ni gnitluser( esahp dliub eht dnoyeb ecnanetniam dna ,noitatnemelpmi ,gnitset eht fo rof noitpircsed ehT .)sesahp snoitarepo dna ni dedulcni eb ot era sesahp gniniamer eht ,V&VI eht ot gnidroccA .selbareviled tneuqesbus dezingocer lla htiw tciflnoc tcerid ni si hcaorppa siht .sdradnats yrtsudni x x .secruoser ecnarussa ytilauq etis-no on sah rodneV V&VI . . . egap txen no deunitnoc 86 California State Auditor Report 2014-116 February 2015 AERA THGISREVO TCEJORP YEK SMETSYS TCEJORP TCARTNOC RODNEV SECRUOSER ELUDEHCS EPOCS GNIREENIGNE TNEMEGANAM YTILAUQ STNEMERIUQER EUSSI FO YRAMMUS THGISREVO FO EPYT ETAD TROPER x x x x x x ton PFR morf stnemeriuqer lanoitcnuf setoN OPI 1102 rebmeceD yrotaluger yna rof txetnoc ssenisub ni defiiceps ytilanoitcnuf deriuqer eht ,erofereht ,dna ytitne ton saw seititne yrotaluger eht ot cfiiceps .denfied ylraelc x x x enilesab morf( ni htnom eno si tcejorp eht setoN OPI .eludehcs dniheb shtnom owt dna )eludehcs x x x x x neeb evah taht snalp tnemeganam tcejorP V&VI ,tsom ton fi ,ynaM .dewollof gnieb ton era devorppa fo edistuo rucco seitivitca tnemeganam tcejorp sloot tnemeganam tcejorP .sessecorp denfied eht ,ksir ,egnahc sa hcus( snalp devorppa eht ni denfied eht retfa detatonna era )sgol eludehcs dna ,eussi .lla ta ton ro tcaf x x x x x x rodnev gniweiver ton si sriaffA remusnoC V&VI selbareviled eht ot gnidrocca selbareviled selbareviled ,yllanoitiddA .ssecorp tnemeganam sevresbo V&VI .etar hgih a ta detcejer gnieb era neewteb spag tnacfiingis neeb evah ereht tA .snoitatcepxe s’rodnev dna ’sriaffA remusnoC yltnacfiingis saw elbareviled dettimbus eno tsael tnetnoc deriuqer eht teem ton did dna etelpmocni .DED eht fo x x x x x eht dlosrevo evah yam erutneccA taht raelc si tI OPI 2102 yraunaJ .egats tnemerucorp eht ta loot etargiM asreV naht regral eb lliw troffe noisrevnoc atad ehT sa pleh ton yam sloot detamotua dna detcepxe .detapicitna sa hcum x x x x x x stnemeriuqer sti ni yticfiiceps skcal sriaffA remusnoC OPI yaled dluoc sihT .PFR eht ni deman stroper eht rof .stsoc lanoitidda ot dael dna tcejorp eht x x eht htob morf syaled fo degreme sah nrettap raelc A OPI .sriaffA remusnoC dna rodnev x x x x x lanretxe dna lanretni tsubor a si ereht stressa rodneV OPI fo tsom fo weiver a tub ,ssecorp ecnarussa ytilauq gnikrow ton si ssecorp eht tseggus selbareviled eht .yttops tseb ta si ssecorp eht ot ecnerehda ro x x x x x lliw dna tsisrep seussi ytilauq elbareviled rodneV OPI .syaled eludehcs ot dael ylekil California State Auditor Report 2014-116 87 February 2015 AERA THGISREVO TCEJORP YEK SMETSYS TCEJORP TCARTNOC RODNEV SECRUOSER ELUDEHCS EPOCS GNIREENIGNE TNEMEGANAM YTILAUQ STNEMERIUQER EUSSI FO YRAMMUS THGISREVO FO EPYT ETAD TROPER x x x x x x eht wohs ot dednetni era hcihw ,stroper elfiorP OPI ,ytitne yrotaluger hcae rof stnemeriuqer lanoitcnuf lliw yeht sraeppa ti ,rehtruF .etelpmoc morf raf era .snoitareti lanoitidda ot tcejbus eb x x x .gnippils eludehcs swohs sisylana eludehcS V&VI x x x x x dna llew gnikrow ton si ssecorp elbarevileD V&VI ,sweiver elpitlum deriuqer evah selbareviled tsom .snoissimbuser dna ,setadpu x x x .eludehcs dniheb si tcejorP OPI 2102 yraurbeF x x x x x tsissa ot sriaffA remusnoC no gninael si rodneV OPI .selbareviled htiw x x x x x x tnacfiingis deriuqer evah selbareviled ynaM OPI rof rodnev eht ot kcab tnes erew dna snoisiver eludehcs dna ecruoser ni detluser sah sihT .krower .smelborp x x x x x gnieb era sksat etelpmocni taht ecnedive si erehT OPI .etelpmoc dekram x x x x yam stroper elfiorp ytitne yrotaluger etelpmocnI OPI ni ecnedfinoc esol ot ffats ytitne yrotaluger esuac eht fo ecnatpecca llarevo gnizidrapoej ,metsys eht si ksir siht sserdda ot nalp ehT .metsys detelpmoc tcudnoc ot dne kcab eht no emit lanoitidda wolla ot .snoisses no-wollof x x x x x tnecrep 06 revo swohs eludehcs rodnev fo sisylanA V&VI .citsilaernu si eludehcs eht—kcals orez evah sksat fo x x x x od setadpu nalp krow desoporp eht taht nrecnoC OPI 2102 hcraM eht ,elpmaxe roF .eludehcs llarevo eht tfihs ton 3 esahp dna 2 esahp rof setad noitatnemelpmi lanfi palrevo eht esaercni segnahc ehT .egnahc ton did gnisserdda tuohtiw 2 esahp dna 1 esahp neewteb .nalp ni seussi larutcurts x x x x sessecorp ot gnirehda ton si sriaffA remusnoC OPI .snalp devorppa sti ni debircsed x x x gnisnaelc atad rof elbisnopser si sriaffA remusnoC V&VI si rodneV .atad htiw seussi sefiitnedi rodnev retfa seussi yfitnedi ot stroper yrassecen gnitareneg ton .eludehcs ot gnidrocca x x x .deweiver erew snoitacfiiceps ecafretnI V&VI ecafretni eht tahw fo sliated skcal noitatnemucoD nI .noitcnuf taht mrofrep lliw ti woh dna od lliw eht fo selur ssenisub eht fo sisylana na ,noitidda .detnemucod ton saw smetsys ycagel . . . egap txen no deunitnoc 88 California State Auditor Report 2014-116 February 2015 AERA THGISREVO TCEJORP YEK SMETSYS TCEJORP TCARTNOC RODNEV SECRUOSER ELUDEHCS EPOCS GNIREENIGNE TNEMEGANAM YTILAUQ STNEMERIUQER EUSSI FO YRAMMUS THGISREVO FO EPYT ETAD TROPER x x 1 esahp dna nalp krow enilesab neewteb paG OPI 2102 lirpA .worg ot seunitnoc etad noitatnemelpmi x x x x x ot gnihtyreve seriuqer nalp krow rodnev detadpU OPI sihT .eludehcs no yats ot tcejorp eht rof thgir og doog ton si hcihw ,oiranecs esac tseb a semussa .gninnalp tcejorp x x x x x dettimbus saw taht elbareviled ngised deliated ehT V&VI ssenetelpmoc fo kcaL .DED htiw tnetsisnocni si .tnerappa si x x x x x deraperp gnieb era stpircs gnitset ecnatpecca resU V&VI .srekrow emit-trap fo maet deffatsrednu na yb x x x x .etelpmocni era stpircs gnitset metsys taht snrecnoC V&VI x x x x gnirud defiitnedi smelborp emos taht ecnedivE OPI .devlosernu niamer tolip moor ecnerefnoc eht era seussi rieht taht denrecnoc seititne yrotalugeR .desserdda gnieb ton x x gnffiats si sriaffA remusnoC yaw eht fo esuaceB OPI eht deecxe ylekil lliw ti ,gnitset ecnatpecca resu .detacolla emit x x x x )hceTlaC( ygolonhceT fo tnemtrapeD ainrofilaC OPI snrecnoc fo esuaceb devlovni gnitteg si hceTlaC .ytilauq rodnev tuoba x x x x x .seussi ytilauq rof eciwt detcejer ngised deliateD OPI 2102 yaM x x si ereht dna ,eludehcs dniheb sniamer tcejorP OPI .emoc ot syaled lanoitidda fo ecnedive x .tcejorp eht no tcapmi na evah lliw shguolruF OPI x x x ron nalp eht wollof ton ot seunitnoc noisrevnoc ataD V&VI .emit no ydaer eb ot yrassecen ssergorp ekam x x x gnitset ecnatpecca resu gniyfidom sriaffA remusnoC V&VI .stpircs tset metsys detaerc-rodnev erom esu ot x x x x x x tuohtiw tset metsys gnitrats setapicitna rodneV V&VI .nalp tset sti ni deniltuo airetirc eht gniteem x x x x x x elfiorp cireneg ot decart gnieb era stnemeriuqeR V&VI fo stnemeriuqer lanoitcnuf cfiiceps ot ton ,stroper .seititne yrotaluger eht California State Auditor Report 2014-116 89 February 2015 AERA THGISREVO TCEJORP YEK SMETSYS TCEJORP TCARTNOC RODNEV SECRUOSER ELUDEHCS EPOCS GNIREENIGNE TNEMEGANAM YTILAUQ STNEMERIUQER EUSSI FO YRAMMUS THGISREVO FO EPYT ETAD TROPER x x x degdelwonkca reganam tcejorp eZEerB eht setoN OPI 2102 enuJ ytiliba ezidrapoej ot gninnigeb si erusserp ecruoser .tcudorp ytilauq a reviled ot x x x x .nalp tnemeganam egnahc gniwollof ton llits tcejorP V&VI x x x x x x dna gniman etaerc ot desoppus saw tcejorp ehT V&VI .deneppah ton sah siht dna ,sdradnats gnidoc x x x x x x ,gnitset metsys ni ssergorp rodnev tneicffiusnI OPI 2102 yluJ stekcit nepo fo rebmun egral yb dewols .)stcefed 008( x x x ot elbatsnu oot sniamer tnemnorivne lacinhcet ehT OPI .gnitset ecnatpecca resu nigeb x x x eZEerB eht ,shtnom eerht tsal eht fo esruoc eht revO OPI 2102 tsuguA eludehcs eht nalp-er ot dah sah maet tcejorp ot detcelgen evah yam yeht ylekil si ti ;semit eerht .sksat gnidnatstuo lla fo sisylana hguoroht a tcudnoc x x x x x etelper“ ti sdnfi ngised deliated fo weiver V&VI OPI ”.sgnimoctrohs htiw x x x x 1 esahp no sucof ot stroffe 2 esahp sesuap rodneV OPI .elbatsnu oot si 1 esahp esuaceb x x x x x x x ssenisub deriuqer yfirev ton lliw tset metsyS OPI 2102 rebmetpeS ytilanoitcnuf ssenisub srefed rodneV .ytilanoitcnuf resu gnirud sriaffA remusnoC ot noitamrfinoc .gnitset ecnatpecca x x x x x x tset evah ton od sesac tset metsys fo rebmun A OPI si ereht dna drocer noitucexe tset a ro stpircs nwo sti gniwollof ton si rodnev eht ecnedive .nalp tset x x x x x eht lortnoc ylluferac ot sdeen sriaffA remusnoC OPI seititne yrotaluger eht yb segnahc detseuqer segnahc evah-tsum eht ylno taht erusne ot .gnitset ecnatpecca resu tixe ot deriuqer era eb dluoc gnitset ecnatpecca resu ,esiwrehtO .yletinfiedni degnolorp x x ,skeew xis rof deludehcs yllanigiro ,tset metsyS OPI 2102 rebotcO .shtnom xis retfa gniogno sniamer x x x morf nalp krow tfard ni sesaercni krow gniniameR OPI fo sa sruoh 554,03 ot 21 rebmetpeS no sruoh 527,71 .61 rebotcO x x x x x x x 22 fo 01 dnuof ecart stnemeriuqer fo tiduA OPI .detsil esac tset eht yb defisitas ton stnemeriuqer . . . egap txen no deunitnoc 90 California State Auditor Report 2014-116 February 2015 AERA THGISREVO TCEJORP YEK SMETSYS TCEJORP TCARTNOC RODNEV SECRUOSER ELUDEHCS EPOCS GNIREENIGNE TNEMEGANAM YTILAUQ STNEMERIUQER EUSSI FO YRAMMUS THGISREVO FO EPYT ETAD TROPER x x x x x x nalp krow dna elbareviled gniniart resu lanretnI OPI ot eud rodnev ot denruter elbareviled enilesab deunitnoc fo noitacidni na si hcihw ,seussi larutcurts .smelborp ytilauq x x x x kcal a ot ecart gnitset gnirud defiitnedi gnieb seussI V&VI fo noitatnemelpmi eht rof stnemeriuqer enilesab fo .seititne yrotaluger ot cfiiceps ytilanoitcnuf x x x x x x x enilesab tuohtiw devorppa ngised metsys deliateD V&VI taht ksir tnacfiingis stneserper—stnemeriuqer .ytilibaecart dna gnitset stceffa x x x x x x gnidnatstuo 286,1 htiw detixe tset metsyS V&VI 2102 rebmevoN erofeb desolc eb tsum taht stnedicni/stcefed .noitatnemelpmi 1 esahp x x x x x x x 1 ytiroirp fo erusolc refed ot deerga sriaffA remusnoC V&VI .tixe tset metsys wolla ot stcefed 2 ytiroirp dna x x x x x x 286,1 htiw deretne si gnitset ecnatpecca resU V&VI .stcefed gnidnatstuo x x x krow fo tnecrep 57 dewohs nalp krow rebmetpeS OPI 2102 rebmeceD .tnecrep 36 swohs nalp krow rebotcO .etelpmoc .tnecrep 57 swohs nalp krow rebmeceD x x x x x gnitcapmi gnitset fo kcal ot detubirtta smelborP OPI .gnitset ecnatpecca resu x x x x x x si erawtfos ytilauq roop stroper sriaffA remusnoC OPI .syaled gnitset ecnatpecca resu ot gnitubirtnoc x x fo dne eht ta gol eht ni stcefed nepo 402,2 V&VI .2102 rebmeceD x x x x tcejorp—stcefed ot eud etar eruliaf tset hgiH OPI 3102 yraunaJ tnerruc ;etar eruliaf ssap tsrfi tnecrep 02 detcepxe etar eruliaf ssap dnoceS .tnecrep 16 si etar .tnecrep 56 si x x x x era taht stnemeerga lautum ot gnitroser era smaeT OPI .gnidnib yllautcartnoc ton x x sa noitapicitrap ytitne yrotaluger ni pord suotipicerP OPI .etamitse sti sdeecxe gnitset ecnatpecca resu x x x x esu eht ni deniart ton erew strepxe rettam tcejbuS OPI dna elbuod ni gnitluser si hcihw ,loot gnitset eht fo .yrtne atad elpirt semitemos California State Auditor Report 2014-116 91 February 2015 AERA THGISREVO TCEJORP YEK SMETSYS TCEJORP TCARTNOC RODNEV SECRUOSER ELUDEHCS EPOCS GNIREENIGNE TNEMEGANAM YTILAUQ STNEMERIUQER EUSSI FO YRAMMUS THGISREVO FO EPYT ETAD TROPER x x x x x x x tnemeganam sriaffA remusnoC ,3102 ,11 yraunaJ nO OPI dna reganam tcejorp rodnev ot rettel a deussi sriaffA remusnoC ,rettel eht nI .tnemeganam ot deriuqer saw rodnev taht smeti fo ynatil a detats hcihw ,gnitset ecnatpecca resu gnidulcni ,etelpmoc yeht taht saw esnopser rodneV .gnidnatstuo niamer ,eludehcs ”deerga yllautum“ eht ot gnimrofnoc erew eht—”eludehcs lautcartnoc“ eht ton si hcihw .nalp krow devorppa yllamrof dna enilesab x x x ,rotargetni smetsys eht neve ton ,eno on . . .“ OPI 3102 yraurbeF xelpmoc siht eb dluow eZEerB dezilaer dah ”.tnemelpmi ot x x x x detcepxe na susrev tnecrep 13 si etar liaf/xfi ehT OPI .tnecrep 5 x x x x .tnecrep 81 etar liaf tseteR OPI 3102 hcraM x x x lliw seitud snoitarepo dna ecnanetniam woh raelcnU OPI .sriaffA remusnoC dna rodnev neewteb detacolla eb x x x x x x x x ecnatpecca resu ot trats etal a fo tnemssessA V&VI enilesab on ,snoitarud ksat citsimitpo—gnitset noitatnemelpmi deyaled/wols ,stnemeriuqer fo flehs-eht-ffo laicremmoc fo noitarugfinoc dna ot tset metsys fo epocs eht gniworran ,stcudorp ton ,stnemeriuqer tcartnoc ]level-hgih[ mrfinoc ylno remusnoC .stnemeriuqer cfiiceps-ytitne yrotaluger resu trats ot gniyrt ylevissergga rodnev dna sriaffA remusnoC .elbissop sa noos sa gnitset ecnatpecca egarevel lautcartnoc eht evah ton did ti tlef sriaffA etipsed trats gnitset ecnatpecca resu yaled ot ytilauq roop stressa sriaffA remusnoC .seussi ytilauq ecnatpecca resU .tset otni dereviled stcudorp —yrtne ta etelpmoc ton saw gninnalp gnitset on ,stpircs gnitset ecnatpecca resu enilesab on .stnemeriuqer enilesab x x x x x erew taht stnemele hcihw ta etar eht ,etar liaf/xfi ehT OPI 3102 lirpA yletamixorppa saw ,tset-er a deliaf dexfi sa defiitnedi .tnecrep 52 x x x x x x metsys ylno ,evil-og ta taht detats sah rodneV OPI ”,stekcit tcefed“ ro ,)sRIS( stseuqer noitagitsevni dexfi eb lliw niamod s’rodnev ni eb ot deredisnoc stnemecnahne deredisnoc stcefeD .tsoc on ta tcartta lliw niamod ’sriaffA remusnoC nihtiw ro htiw gnikrow sriaffA remusnoC .stsoc lanoitidda remusnoC“ setutitsnoc tahw yfiralc ot tinu lagel ”.niamod ’sriaffA . . . egap txen no deunitnoc 92 California State Auditor Report 2014-116 February 2015 AERA THGISREVO TCEJORP YEK SMETSYS TCEJORP TCARTNOC RODNEV SECRUOSER ELUDEHCS EPOCS GNIREENIGNE TNEMEGANAM YTILAUQ STNEMERIUQER EUSSI FO YRAMMUS THGISREVO FO EPYT ETAD TROPER x x x x refed ot dezitiroirper gnieb era sRIS taht setoN V&VI 3102 yaM .noitatnemelpmi 1 esahp retfa litnu noitca evitcerroc x x x ton od seititne yrotaluger taht denrecnoc si rodneV V&VI detceffa eb lliw sessecorp ssenisub rieht woh wonk .eZEerB yb x x x roirp on sah reganam tcartnoc ’sriaffA remusnoC OPI 3102 enuJ .reganam tcartnoc a sa ecneirepxe x .enuJ ot yaM morf sesaercni sRIS nepo fo rebmuN V&VI x x x taht raelc edam sah evitucexe deddebme-rodneV OPI 3102 yluJ rehtruf tuohtiw 1 esahp yolped ot si etadnam sih .yaled x x x x erofeb tixe ot tset metsys dewolla sriaffA remusnoC OPI .tem saw noiretirc tixe eht x x x x x x x erew stnemeriuqer ecnamrofrep taht stressa rodneV OPI saw ti nehw nalp erutcetihcra lacinhcet yb defiidom OPI .seergasid V&VI .sriaffA remusnoC yb detpecca siht tnemucod ot tcejorp eht degaruocne tnatlusnoc .tey detnemucod neeb sah ti noitacidni on ;eussi x x x x x htiw gnitset noisserger retne ot seerga tcejorP OPI 3102 tsuguA eb dluow deerga ylsuoiverp erew taht sRIS nepo 88 .dliub lanfi erofeb desolc x x x x x yltnacfiingis slaog/hcaorppa gnitset ecnamrofreP V&VI sriaffA remusnoC“ ot liaf/ssap morf defiidom skramhcneb eht taht denrecnoc V&VI ”.elbatpecca eb ton yam nosirapmoc erutuf rof nesohc .sdeen ssenisub ’sriaffA remusnoC fo evitatneserper x x x x x x era taht selbareviled detpecca sah sriaffA remusnoC V&VI 3102 rebmetpeS .tcerrocni dna ,tnetsisnocni ,etelpmocni x x x x dah V&VI detressa tsilkcehc gnitset ecnamrofreP V&VI taht dna ton did eh syas V&VI .stluser eht defiirev .eslaf si noitressa eht California State Auditor Report 2014-116 93 February 2015 AERA THGISREVO TCEJORP YEK SMETSYS TCEJORP TCARTNOC RODNEV SECRUOSER ELUDEHCS EPOCS GNIREENIGNE TNEMEGANAM YTILAUQ STNEMERIUQER EUSSI FO YRAMMUS THGISREVO FO EPYT ETAD TROPER x x x x x x x ot eerga yllautum rodnev dna sriaffA remusnoC OPI 3102 rebotcO .derrefed eb ot sRIS 610,1 htiw tnemelpmi x ekat ot detamitse gnitset ecnatpecca resu 1 esahP V&VI .syad krow etats 112 sekat yllautca ,skeew 8 x x x x x x x .sRIS nepo 610,1 htiw evil tnew 1 esahP V&VI x x x x x x x litnu denilesab ton ytilanoitcnuf ssenisuB V&VI ot demrofrep saw ytilibaecart oN .3102 ,32 rebotcO rof stnemeriuqer lautcartnoc 087 eht taht mrfinoc oN .gnitset ecnatpecca resu yb detset erew 1 esahp 1 esahp lautcartnoc fo lla mrfinoc ot stsixe ecnedive .tem erew stnemeriuqer x x x x x x .selbareviled rodnev htiw tsisrep seussi ytilauQ OPI 3102 rebmevoN x x x x x x no noitcudorp ni detroper si 10.1 esahP V&VI ,3102 ,52 rebmevoN fo sa hguohtla ,3102 ,52 rebotcO sa defiitnedi erew taht nepo sRIS 13 era ereht saw tahw raelc ton si tI .10.1 esahp ot gnignoleb .esahp siht ni deriuqer yllautcartnoc x x x x x x tcejorP laicepS fo lavorppa fo noitidnoc a sA OPI 3102 rebmeceD fo noitelpmoc deriuqer hceTlaC ,2 )RPS( tropeR erofeb nalp krow enilesab a dna ygolodohtem eht gnieb si tnemeriuqer taht tub ngised fo trats eht si ngised 2 esahp taht setamitse V&VI .derongi .etelpmoc tnecrep 06 x x x s’rodnev eht taht snoitacidni ydaerla era erehT OPI .gnippils si eludehcs detadpu x x x x fo emos ,stcefed 766,11 htiw pu dedne 1 esahP OPI .devloser tey ton era hcihw x x x x x lamrof“ sa sweiver lamrofni gnitaert si tcejorP OPI 4102 yraunaJ elbarevileD eht ni debircsed sa ”sweiver .nalP tnemeganaM x x dengissa won reganam tcejorp ’sriaffA remusnoC OPI rotcerid tcejorp/reganam tcejorp fo elor laud eht eht taht snrecnoc seciov tnatlusnoc OPI .)DP/MP( mrofrep ot htdiwdnab eht evah ton seod DP/MP .ylevitceffe selor htob x x x x ,)1.2( elbareviled ygolodohtem tfard s’rodnev reP OPI syad 02 ylno sriaffA remusnoC wolla ot snalp rodnev htiw gnitset ecnatpecca resu fo ssap tsrfi eht rof seton OPI .noisserger rof edisa tes syad 01 rehtona dna 1 esahp htiw yrotsih eht nevig citsilaernu si siht esnecil erom tnemelpmi lliw 2 esahp taht tcaf eht .1 esahp naht sepyt . . . egap txen no deunitnoc 94 California State Auditor Report 2014-116 February 2015 AERA THGISREVO TCEJORP YEK SMETSYS TCEJORP TCARTNOC RODNEV SECRUOSER ELUDEHCS EPOCS GNIREENIGNE TNEMEGANAM YTILAUQ STNEMERIUQER EUSSI FO YRAMMUS THGISREVO FO EPYT ETAD TROPER x x x x sexfi rof sRIS gningissa sriaffA remusnoC OPI ecnanetniam diaperp eht yb derevoc yllamron( )IDD( tnemelpmi dna poleved ,ngised ot )tcartnoc .tcartnoc tnemecnahne x x x mrofrep ot IDD gnisu taht seton tnatlusnoc OPI ehT OPI a si tcartnoc eht fo epocs eht nihtiw ton si taht krow .secitcarp gnitcartnoc etats fo noitaloiv raelc x x x x .)4102 ,72 yraunaJ fo sa( sRIS 003,1 yltnerruc era erehT OPI detluser evah evil-og retfa defiitnedi stcefed lacitirC ta nepo deniamer taht stcefed fo noitazitiroirper ni ot elba ton si maet ecnanetniam rodneV .evil-og ecap eht ta sdnamed ecnanetniam gniogno teem .seititne yrotaluger eht yb derised x x x x x sRIS derrefed sevom 051 )CFR( egnahC rof tseuqeR V&VI .IDD ot etatS eht ot tsoc on ta dexfi eb ot erew taht x x x eht ot gnirehda ton si tcejorP V&VI .ygolodohtem devorppa x x x x x x x gnitset ecnatpecca resu desoporp eht taht nrecnoC V&VI tneicffius edivorp ton lliw ]2 esahp rof[ hcaorppa evah stnemeriuqer lla taht yfirev ot ecnegilid .dereviled neeb x x x x x x etatS eht setacidni reganam tset ’sriaffA remusnoC V&VI ot gnitset metsys s’rodnev eht nopu yler dluow ’sriaffA remusnoC eht fo lla taht noitacfiirev mrofrep .dereviled erew stnemeriuqer devorppa x x x x x rodnev eht eriuqer ton seod nalp tset devorppa ehT V&VI lla erussa ot liated fo level yrassecen eht ta yfirev ot ti dna tem era stnemeriuqer level ngised devorppa dessap era stset fi noinipo ’sretset rodnev eht eb lliw .)tnemevlovni sriaffA remusnoC tuohtiw( x x x x x x si ygolodohtem eht fo tnenopmoc ngised ehT V&VI esahp ngised eht elihw yraunaJ ni desiver gnieb .gniogno si x x x x x segnahc dnuof tnemssessa tseuqer egnahC V&VI ot segnahc dna stnemucod denilesabnu tsniaga .tcejorp eht fo trap ydaerla saw taht epocs dda senilesab eht fo lortnoc ni ton si sriaffA remusnoC ytilibaecarT .segnahc devorppa eht tnemucod taht .tneicffiusni si California State Auditor Report 2014-116 95 February 2015 AERA THGISREVO TCEJORP YEK SMETSYS TCEJORP TCARTNOC RODNEV SECRUOSER ELUDEHCS EPOCS GNIREENIGNE TNEMEGANAM YTILAUQ STNEMERIUQER EUSSI FO YRAMMUS THGISREVO FO EPYT ETAD TROPER x x x x eb ot eunitnoc stnemecnahne dna stcefed eroM OPI 4102 yraurbeF .1 esahp tsniaga defiitnedi x x x x ecruoser fo esuaceb nwod gniwols 2 esahP OPI revo won era ereht( stcefed 1 esahp xfi ot noitcerider .)evil-og rebotcO ecnis sRIS 002,1 x x x x x x x stnemeriuqer eht devorppa sriaffA remusnoC V&VI tey gnivah tuohtiw )MTR( xirtam ytilibaecart MTR devorppa ehT .ygolodohtem eht devorppa si V&VI dna ,2 esahp rof stnemeriuqer 56 sefiitnedi rof stnemeriuqer erom eb dluohs ereht denrecnoc .2 esahp x ni esaercni laitnatsbus a seveileb tnatlusnoc OPI OPI 4102 hcraM .3 esahp rof dedeen eb lliw secruoser x x x x x smaet rodnev eht dna etats eht htob raelc si tI“ OPI troffe ecnanetniam fo level eht detamitserednu dna seititne yrotaluger 1 esahp rof deriuqer ni esahp eht rof stroffe ecnanetniam lanoitidda ”.deriuqer eb lliw noitcudorp x x x x x x eb tsum stnemeriuqer 56 ylno taht stressa rodneV OPI 4102 yaM .)tcartnoc ni 009,1 susrev( 2 esahp rof detset x x tuoba nrecnoc s’tnatlusnoc OPI eht ot esnopser nI OPI tnemecnahne eht fo noitpmusnoc fo deeps eht revo lortnoc on sah ti taht sdnopser tcejorp ,tegdub .seititne yrotaluger eht yb stseuqer tnemecnahne x lla rof lavorppa enilesab dna gninnalp fo noitelpmoC OPI 4102 enuJ lavorppa hceTlaC fo noitidnoc a saw sesahp tcejorp ton sah noitidnoc siht tub ,3102 rebotcO ni 2 RPS fo .defisitas neeb tey x x yb setamitse deecxe ot detcepxe era stsoc 2 esahP OPI fo noitrop tnacfiingis A .noillim 54$ ot noillim 23$ egnahc noitazinagro dna secruoser tcejorp rof si siht seod esaercni tsoc sihT .lennosrep tnemeganam era snoitacidni lla tub ,stsoc 3 esahp yna sserdda ton .2 esahp naht erom neve tsoc lliw 3 esahp taht . . . egap txen no deunitnoc 96 California State Auditor Report 2014-116 February 2015 AERA THGISREVO TCEJORP YEK SMETSYS TCEJORP TCARTNOC RODNEV SECRUOSER ELUDEHCS EPOCS GNIREENIGNE TNEMEGANAM YTILAUQ STNEMERIUQER EUSSI FO YRAMMUS THGISREVO FO EPYT ETAD TROPER x x x x emulov hgih eht lortnoc ot elbanu si tcejorp ehT OPI stseuqer tnemecnahne ’seititne yrotaluger fo fo esuaceb tcartnoc ecnanetniam eht rednu yrotaluger eht fo erutan suomonotua-imes eht noitpmusnoc retsaf ni gnitluser si sihT .seititne tcejorp eht naht sdnuf ecnanetniam elbaliava fo eerged eht etapicitna ton did tcejorp ehT .detcepxe yrotaluger eht htiw[ tnemngilasim noitatcepxe fo enilesab ot gniliaf morf tluser dluow taht ]seititne .ngised 1 esahp eht x x x x stsoc tcartnoc eZEerB eht fo noitanimaxe nA OPI gnitroper esnepxe tcartnoc tcejorp taht swohs .etaruccani si x x x sriaffA remusnoC“ ,maet tcejorp eht ot gnidroccA OPI tseuqer suaerub dna sdraob eht tahw etatcid tonnac erised seititne yrotaluger eht fI .stnemecnahne rof sriaffA remusnoC ,stnemecnahne eht rof yap ot esuaceb sdnamed rieht etadommocca ot sdnet sekam hcihw ,erutan suomonotua-imes rieht fo tcejorp eht ,suhT .eussi eht ecrof ot tlucffiid ti ylno sdeen ’seititne yrotaluger ot egatsoh si ”.secruoser elbaliava yb deniartsnoc x x noitatnemelpmi 3 esahp rof stsoc tcejorp latoT V&VI .nwonknu niamer x x x x x eht yb detabrecaxe gnieb era seussi secruoseR OPI 4102 yluJ yb depoleved eludehcs noitatnemelpmi evissergga .sriaffA remusnoC yb devorppa dna rodnev eht x x x x x x rodnev eht no kcab gnihsup era seititne yrotalugeR OPI .ytilauq roop tuoba gninialpmoc dna elbatemit x x x x x egnahc lanoitazinagro gnirucorp ni syaleD V&VI eht ecuder yam secruoser )MCO( tnemeganam ngised eht erofeb emit gnidraob no dennalp MCO lla rof ytiliba eht timil yam hcihw ,tixe esahp 2 esahp eht yb detelpmoc eb ot selbareviled .etad tnemyolped California State Auditor Report 2014-116 97 February 2015 AERA THGISREVO TCEJORP YEK SMETSYS TCEJORP TCARTNOC RODNEV SECRUOSER ELUDEHCS EPOCS GNIREENIGNE TNEMEGANAM YTILAUQ STNEMERIUQER EUSSI FO YRAMMUS THGISREVO FO EPYT ETAD TROPER x x x x htiw tcejorp eht no noitirtta fo etar hgih a si erehT OPI 4102 tsuguA selor wen gnimussa secruoser yek fo rebmun a remusnoC eht nihtiw tcejorp eht ot detalernu ssenisub lareves ,yllanoitiddA .noitazinagro sriaffA eht no neeb osla evah )sAIB( stsylana noitargetni snoitisop emos effluhser ot tcejorp eht gnicrof evom fo eno nI .tcejorp eht no selor tnatropmi eseht llfi ot ot detcerider saw reganam ytilauq eht ,sevom eht tuoba snrecnoc desiar OPI .AIB a fo elor eht emussa dna gnikatrednu si tcejorp eht taht nevig evom siht lliw taht selbareviled ngised yek lareves gniweiver .tcejorp eht no stcapmi gnitsal evah x x x remusnoC ,nalp ecnanetniam eht ot gnidroccA OPI rodnev wodahs ot erew srebmem maet sriaffA revo ekat ot eraperp dna ffats ecnanetniam dengissa ton sah tcejorp ehT .ecnanetniam metsys .refsnart egdelwonk siht ni etapicitrap ot ffats yna sriaffA remusnoC fi laitnesse si siht seveileb OPI 2 esahp fo dne eht ta ecnanetniam revo ekat ot si era ffats sriaffA remusnoC esuaceb ,dennalp sa cilbup siht troppus ot deraperp yltnerruc ton .metsys gnicaf x x x fo esuaceb deyaled saw ngised smetsys deliateD OPI .noitnetnoc ecruoser ’seititne yrotaluger x x ot dedeen secruoser setanimile 3 RPS fo lawardhtiW OPI .ecnanetniam metsys gniogno troppus x x x x yrassecen secruoser setanimile 3 RPS fo lawardhtiW OPI .ygolodohtem nopu-deerga ylwen eht troppus ot x x x x x eht rehtie no reganam ecnarussa ytilauq on si erehT OPI 4102 rebmetpeS .emit siht ta smaet rodnev ro etats x x yrotaluger dna tcejorp htob( sriaffA remusnoC OPI on htiw lacitirc era stniartsnoc ecruoser )seititne .3 RPS fo lawardhtiw eht retfa thgis ni feiler x x x wodahs ot enoyna dengissa ton llits sah tcejorp ehT OPI .maet ecnanetniam eht . . . egap txen no deunitnoc 98 California State Auditor Report 2014-116 February 2015 AERA THGISREVO TCEJORP YEK SMETSYS TCEJORP TCARTNOC RODNEV SECRUOSER ELUDEHCS EPOCS GNIREENIGNE TNEMEGANAM YTILAUQ STNEMERIUQER EUSSI FO YRAMMUS THGISREVO FO EPYT ETAD TROPER x x sih detatser sah rotcerid tcejorp eZEerB ehT OPI secruoser deriuqer no gnirb ot tnemtimmoc si tI .eZEerB tnemelpmi yllufsseccus ot yrassecen ni detseuqer secruoser eht fo ynam woh raelc ton .detiurcer eb nac 3 RPS x x 2 esahp troppus ot secruoser tcejorp lanoitidda ehT V&VI tcejorp latot desaercni 3 RPS ni detalucitra erew taht .noillim 811$ ot )2 RPS( noillim 77$ morf stsoc tcejorp 42 lanoitidda na detroppus gnidnuf 3 RPS tnemeganam egnahc lanoitazinagro ,snoitisop gnffiats snoitarepo dna ecnanetniam ,sdeen detapicitnanu eht desaercni dna ,sesaercni eht tuohtiW .epocs 2 esahp rof gnidnuf ksat sraeppa ti ,3 RPS ni desoporp secruoser lanoitidda dluoc sriaffA remusnoC taht ylekilnu ylhgih 2 esahp hsilpmocca ot secruoser tneicffius tcerider .slaog tcejorp .4102 rebmetpeS hguorht 0102 rebmeceD detad tnatlusnoc V&VI dna tsilaiceps OPI hceTlaC eht yb stroper fo sisylana s’trepxe TI s’rotiduA etatS ainrofilaC ehT :secruoS California State Auditor Report 2014-116 99 February 2015 100 California State Auditor Report 2014-116 February 2015 California State Auditor Report 2014-116 101 February 2015 102 California State Auditor Report 2014-116 February 2015 California State Auditor Report 2014-116 103 February 2015 104 California State Auditor Report 2014-116 February 2015 California State Auditor Report 2014-116 105 February 2015 106 California State Auditor Report 2014-116 February 2015 California State Auditor Report 2014-116 107 February 2015 * * California State Auditor’s comments begin on page 113. 108 California State Auditor Report 2014-116 February 2015 1 2 3 4 California State Auditor Report 2014-116 109 February 2015 110 California State Auditor Report 2014-116 February 2015 California State Auditor Report 2014-116 111 February 2015 112 California State Auditor Report 2014-116 February 2015 Blank page inserted for reproduction purposes only. California State Auditor Report 2014-116 113 February 2015 Comments CALIFORNIA STATE AUDITOR’S COMMENTS ON THE RESPONSE FROM THE BOARD OF REGISTERED NURSING To provide clarity and perspective, we are commenting on the response to our audit report from the Board of Registered Nursing (BRN). The numbers below correspond to the numbers we placed in the margin of BRN’s response. BRN misunderstood the information presented in Table 3 on 1 page 15, which is in the Introduction and not in Chapter 3 of our report. Specifically, for each of the 37 boards, bureaus, committees, and a commission (regulatory entities) that were originally scheduled to implement BreEZe, Table 3 shows their respective number of licensees. In the case of BRN, Table 3 shows that BRN has 514,640 licensees. This amount is not limited to registered nurse licensees, as BRN incorrectly understood; rather, it includes other types of BRN licensees, such as nurse practitioners and public health nurses. We are perplexed as to why BRN is singling out certain application 2 types in its response, when the amount of all application types pending its review is clearly a more accurate representation of its pending workload. Specifically, BRN emphasizes that Table 8 on page 69 states that it only had 4,631 applications for initial or renewal licensure, excluding initial international applications. Although this number can be determined from the data in Table 8 by combining the first five types of applications listed, it is not discretely presented. Furthermore, we clearly state on pages 63 and 68 that, as of September 2014, BRN was facing a significant number of applications pending its review—more than 7,000—an amount that includes all application types. Further, BRN incorrectly states that Table 8 does not include initial international applications. For presentation purposes, we included these applications in Table 8 under the respective type of application the international applicant submitted. BRN is incorrect. Our report does not state that BRN overstates 3 its application processing times. Rather, on page 64, we conclude that although BRN attributes its inefficiency in processing applications to the implementation of BreEZe, we found it is unable to substantiate such claims because it does not track the amount of time it takes to process applications. Further, as we point out on page 65, the BRN acknowledged in its 2014 Sunset Review Report that the data in that report concerning application processing times are in many instances “best estimates” and should be viewed with caution, which would lead one to question the accuracy of the data. 114 California State Auditor Report 2014-116 February 2015 4 As we describe in Table 5 on page 21, objective 8 specifically asked us to determine the average amount of time it took BRN to process applications before and after the California Department of Consumer Affairs implemented BreEZe. On page 66 we explain that lacking sound data from BRN to substantiate its claims of increased inefficiencies due to BreEZe, we conducted our own review of selected license applications to obtain a sense of the impact BreEZe has had on the efficiencies with which BRN processes applications. For purposes of addressing the audit objective, we present the results of our review of the selected license applications as averages. California State Auditor Report 2014-116 115 February 2015 * 1 2 * California State Auditor’s comments begin on page 125. 116 California State Auditor Report 2014-116 February 2015 3 4 5 6 5 6 3 7 8 California State Auditor Report 2014-116 117 February 2015 9 118 California State Auditor Report 2014-116 February 2015 10 10 California State Auditor Report 2014-116 119 February 2015 10 11 10 12 120 California State Auditor Report 2014-116 February 2015 10 California State Auditor Report 2014-116 121 February 2015 10 10 122 California State Auditor Report 2014-116 February 2015 4 10 13 California State Auditor Report 2014-116 123 February 2015 4 14 10 15 10 16 124 California State Auditor Report 2014-116 February 2015 10 15 10 15 10 California State Auditor Report 2014-116 125 February 2015 Comments CALIFORNIA STATE AUDITOR’S COMMENTS ON THE RESPONSE FROM THE CALIFORNIA DEPARTMENT OF TECHNOLOGY To provide clarity and perspective, we are commenting on the response to our audit report from the California Department of Technology (CalTech). The numbers below correspond to the numbers we placed in the margin of CalTech’s response. CalTech incorrectly states that it has provided independent project 1 oversight (IPO) on the BreEZe project since inception of the project. As we point out on page 40 of our report, CalTech did not assign an IPO specialist to the project for over a year after it approved the project’s Feasibility Study Report (FSR). We disagree that our report does not recognize the current state 2 of BreEZe and therefore does not provide a clear picture of the current status of the system. On pages 45 through 47 we provide information on the current status of BreEZe, including that future implementation of BreEZe by regulatory entities scheduled for phase 2 of BreEZe is at risk and it is unknown whether or when the 19 regulatory entities originally included in phase 3 will implement the system. In addition, as we describe on page 73, we interviewed the 10 executive officers of the phase 1 regulatory entities during mid‑September through mid‑October 2014 and found that eight reported that BreEZe has decreased their regulatory entities’ operational efficiency. Further, we find it puzzling that CalTech asserts in its response that the BreEZe system was successfully put into production in 2013 when, as shown in the Appendix beginning on page 81, the IPO specialist and independent verification and validation (IV&V) consultant have identified nearly 50 significant issues with the project in their monthly reports following the implementation of BreEZe in October 2013 at the phase 1 regulatory entities. In fact, as we explain on page 38, the September 2014 IPO report identified that the project’s overall health was poor and also states that the online application feature, which the IPO specialist considers a key part of BreEZe, had not been “turned on” for some of the phase 1 regulatory entities. We state on page 36 that it was not until after the California 3 Department of Consumer Affairs (Consumer Affairs) estimated the cost of completing the project at $300 million in June 2014, the same month it submitted Special Project Report (SPR) 3 to CalTech for approval, that CalTech changed its oversight approach on the BreEZe project. According to the BreEZe project director, Consumer Affairs withdrew its submission of SPR 3, 126 California State Auditor Report 2014-116 February 2015 which CalTech previously approved in July 2014, upon direction from CalTech and the California Department of Finance (Finance) in September 2014. This direction from CalTech was a change in its oversight approach in that it was the first time it had directed Consumer Affairs to take an action that significantly altered the project. We discuss CalTech’s involvement in the BreEZe project beginning on page 30. 4 CalTech challenges our conclusion that it should have suspended BreEZe. We stand by our conclusion. As discussed in detail on pages 36 through 40, CalTech did not use its authority to ensure that Consumer Affairs addressed the significant and persistent warnings of the IV&V consultant and the IPO specialist. For example, on page 38 we question why CalTech did not intervene instead of approving SPR 3, given that Consumer Affairs was not appropriately addressing the concerns raised by the IV&V consultant and IPO specialist; these concerns were numerous and were identified in CalTech’s analysis of SPR 2 in 2013. Further, on page 39 we explain that had CalTech chosen to suspend the project, it could have temporarily paused BreEZe development and could have provided Consumer Affairs with additional time to conduct a cost‑benefit analysis and correct fundamental problems, such as requirements issues, it encountered during planning and development. 5 CalTech indicated that suspending the project as suggested by the report would have exposed the State to contractual liability to the prime contractor at a rate of up to $1.2 million per month up to the full value of the contract. Our report indicates on page 40 that CalTech could have suspended the project while it was still in the planning phase, in the spring of 2011, prior to formation of the prime contract. Suspending the project at that juncture would not have exposed the State to any such liability. The requests for proposal contained provisions expressly reserving the State’s right to cancel the solicitation prior to awarding the contract. Similarly, Consumer Affairs could have canceled the negotiations with Accenture LLP (Accenture) conducted under California Public Contract Code, Section 6611, at any time prior to awarding the contract without incurring any liability. This section of law pertains to contracts, such as the BreEZe contracts, that are conducted through a negotiation process. We recognize that suspending the project after the prime contract was awarded could make the State liable for certain costs. However, we believe there are mechanisms in the contract that allow for the containment of potential liability. In particular, there are provisions that allow for a partial or complete stop work order and they require the contractor to take all reasonable steps to minimize the resulting costs and they specify that the State is not liable for lost profits. California State Auditor Report 2014-116 127 February 2015 CalTech indicated that terminating the contract for convenience 6 could have allowed the contractor to claim as much as $38 million. However, our report does not recommend a termination for convenience. Instead, it recommends on page 50 that if Consumer Affairs receives the necessary resources to successfully implement BreEZe at the regulatory entities included in phase 2 and the project continues to face escalating costs, then CalTech should require Consumer Affairs to conduct an analysis of the costs and benefits of moving forward with the project as planned or suspending or terminating the project. Without this analysis, CalTech is not positioned to make an informed decision about whether Consumer Affairs’ current plan to partially complete the project at a total estimated cost of $96 million is the most prudent course of action. We clearly state on page 36 that the $300 million estimate was 7 informally developed by the BreEZe project team and the estimate was not developed in the same manner as required by an SPR. Further, although CalTech reported it had extensive engagement with Consumer Affairs from the beginning of the project, it did not use its authority to ensure Consumer Affairs addressed the significant and persistent warnings of the IV&V consultant and the IPO specialist, as we discuss on pages 36 through 40. CalTech misses our point. Although Consumer Affairs has taken 8 some actions to address concerns raised by the IV&V consultant and the IPO specialist, our point on pages 36 and 37 is that Consumer Affairs did not take appropriate action to address all of the concerns raised and CalTech did not require Consumer Affairs to remedy the problems raised. In fact, on page 37 we describe that the BreEZe project director acknowledged he did not always act on the concerns raised in the IPO report. For instance, the IPO specialist raised concerns related to staffing and resources throughout the course of the project. Further, we explain on pages 34 and 35 that CalTech did not ensure that Consumer Affairs addressed the CalTech director’s concerns about project management, project governance, and contract management before approving two SPRs that identified significant project cost increases. CalTech is incorrect. We do not conclude in the report that CalTech 9 never reviewed or acted on the 1,700 reported initial project defects after the BreEZe system went into production, as CalTech claims in its response. Rather, on page 28 we explain that according to the IV&V consultant, the extensive user acceptance testing period likely resulted in part from the BreEZe system having almost 1,700 unresolved defects at the beginning of the testing period. Further, in contrast to its assertion, we never received information from CalTech that it assisted Consumer Affairs in “reviewing each one of these reported defects.” 128 California State Auditor Report 2014-116 February 2015 10 Throughout its response, CalTech does not explicitly agree or disagree with our recommendations. Instead, in its responses to our recommendations, CalTech generally describes actions it has taken in the past and fails to explain what actions it intends on taking in the future to implement the recommendations. We look forward to CalTech’s 60‑day, six‑month, and one‑year responses to our report in which it should provide evidence of its progress in implementing our recommendations. 11 CalTech appears to miss the point of our recommendation. Although we agree that the IPO and IV&V reports document project issues, risks, and concerns raised on a project, these reports do not document key discussions CalTech may have with Consumer Affairs in which significant concerns are raised about the project vendor Accenture and Consumer Affairs’ performance, or their proposed actions to address such concerns. For example, on page 37, we explain that, according to the CalTech director, in a meeting in February or March 2013, he verbally cautioned officials at Consumer Affairs that CalTech had just suspended or terminated two other projects and that BreEZe would be next if CalTech did not see the department improve its rigor in project management, project governance, and contract management. However, CalTech did not document this discussion or its outcome. By not documenting key discussions, we believe that CalTech is hindered in its ability to hold departments, such as Consumer Affairs, accountable for addressing significant concerns raised. 12 CalTech has misunderstood our recommendation. CalTech refers to the determination made to end the contract with Accenture following phase 2 implementation and a gap analysis that will guide future efforts to address phase 3 regulatory entities. However, our recommendation on page 50 is for a cost‑benefit analysis to be conducted if the necessary resources are received to successfully implement phase 2 and the project continues to face escalating costs. 13 CalTech’s response appears to indicate that it need not implement our recommendation because of existing thresholds for IT project reporting. In as much as these reporting requirements did not prevent the numerous problems with the BreEZe project we identified in the report nor spur CalTech to significantly intervene in the project earlier than it did, we believe they are insufficient for those purposes and our recommendation is needed. 14 Although CalTech did not suspend the BreEZe project, it did change its oversight approach in the summer of 2014, as we indicate on page 36, by directing Consumer Affairs to withdraw SPR 3. California State Auditor Report 2014-116 129 February 2015 This action, which is the type of action we expected CalTech to take sooner, led to SPR 3.1 and the replanning efforts it describes in its response. In response to recommendations we made in Chapter 2 on 15 pages 60 and 61, CalTech indicated that it has processes in place that address our recommendations. However, because our recommendations were based on issues we identified in reviewing the Consumer Affairs’ BreEZe contracts with Accenture, which were executed under the direction of the California Department of General Services (General Services), we did not review and verify whether CalTech’s current processes address our recommendations. Thus, we look forward to CalTech’s 60‑day, six‑month, and one‑ year responses to our report in which it should provide evidence of its progress in implementing our recommendations, such as policies, procedures, or other relevant documentation. CalTech indicated that the State’s current General Provisions, 16 Section 37, is required for all department projects and CalTech is in the process of updating the language. It is correct that the State’s current General Provisions, Section 37, is required for all department projects. The issue, however, is that instead of using the standard language, the contractor for the BreEZe project proposed and Consumer Affairs and General Services approved a deviation from the standard language that increased Consumer Affairs’ financial risks, as we explain on page 55.