Recommendations

June 2016 California Department of Veterans Affairs The State Paid Nearly $28 Million for a Flawed System That Fails to Meet the Needs of Its Veterans Homes Report 2015‑121 COMMITMENT INTEGRITY LEADERSHIP The first five copies of each California State Auditor report are free. Additional copies are $3 each, payable by check or money order. You can obtain reports by contacting the California State Auditor’s Office at the following address: California State Auditor 621 Capitol Mall, Suite 1200 Sacramento, California 95814 916.445.0255 or TTY 916.445.0033 OR This report is also available on our website at www.auditor.ca.gov. The California State Auditor is pleased to announce the availability of an online subscription service. For information on how to subscribe, visit our website at www.auditor.ca.gov. Alternate format reports available upon request. Permission is granted to reproduce reports. For questions regarding the contents of this report, please contact Margarita Fernández, Chief of Public Affairs, at 916.445.0255. For complaints of state employee misconduct, contact the California State Auditor’s Whistleblower Hotline: 1.800.952.5665. Elaine M. Howle State Auditor Doug Cordiner Chief Deputy June 16, 2016 2015‑121 The Governor of California President pro Tempore of the Senate Speaker of the Assembly State Capitol Sacramento, California 95814 Dear Governor and Legislative Leaders: As requested by the Joint Legislative Audit Committee, the California State Auditor presents this audit report concerning the policies and procedures for the planning, development, and implementation of the California Department of Veterans Affairs’ (CalVet) Enterprise‑Wide Veterans Home Information System (system). This report concludes that CalVet paid nearly $28 million for the implementation of a system that has not improved the veterans homes’ process for documenting medical care or reduced the homes’ reliance on paper records, as it was intended to do. In addition, system instability and concerns about functionality resulted in CalVet implementing fewer system functions at some homes than originally planned, thus limiting CalVet’s ability to provide more consistent, efficient care for veterans. CalVet’s project management failed to promptly identify and address the system’s functionality issues. Although it was aware of the problems as early as mid‑2012, CalVet did not begin to address them until late 2013, and the steps it did take did not ultimately fix the problems. Further, although the California Department of Technology (Technology Department) facilitated the contract dispute negotiations between CalVet and the system contractor, it could not provide sufficient documentation to demonstrate its efforts to ensure that the agreements reached were in the best interests of the State. Several key deficiencies contributed to CalVet’s failure to implement a system that meets its needs. CalVet did not exercise adequate oversight of its system project. Specifically, it did not complete or partially completed six of the 12 management oversight plans required by state policy for ensuring effective project management. For the six completed plans, CalVet fully followed only four. For instance, it did not consistently conduct impact analyses on change requests, and therefore was unable to demonstrate that it properly understood the impact of various change requests on the project’s costs, scope, and timelines. CalVet also hired an oversight contractor to provide both independent project oversight (IPO) and independent verification and validation services for its system project, but that oversight contractor’s work was inadequate, and by using the same contractor to perform both services, CalVet did not ensure it had effective oversight. The Technology Department also did not adequately fulfill its oversight responsibilities for CalVet’s system project. Specifically, it did not identify significant concerns with the system until August 2012. The Technology Department indicated that because of the nature of its oversight for this project—consisting of reviewing reports created by the oversight contractor—and because the reports did not indicate any critical errors with the project, the Technology Department did not raise any concerns. However, our information technology expert reviewed the IPO reports and indicated that they were consistently lacking in critical analysis, and that this omission alone should have been a red flag triggering the Technology Department’s closer review and inspection. Respectfully submitted, ELAINE M. HOWLE, CPA State Auditor 621 Capitol Mall, Suite 1200 Sacramento, CA 95814 916.445.0255 916.327.0019 fax www.auditor.ca.gov Blank page inserted for reproduction purposes only. California State Auditor Report 2015-121 v June 2016 Contents Summary 1 Introduction 7 Audit Results The Enterprise‑Wide Veterans Home Information System Does Not Meet the Needs of the Veterans Homes 19 Project Management Failed to Promptly Recognize the Severity of System Problems 25 Failure to Follow Project Management Oversight Plans Contributed to CalVet’s Unsuccessful Implementation of Its System 30 Inadequate Independent Oversight of the System Project Left Stakeholders Without the Information Necessary to Ensure a Successful Implementation 35 The Technology Department Did Not Adequately Fulfill Its Responsibilities in the Oversight of CalVet’s System 38 CalVet Did Not Maintain Adequate Documentation for Its Selection of the System Contractor and for Some Key Deliverables 42 CalVet Identified Some Lessons Learned but Rarely Used Them to Improve Future Phases of Implementation 46 Recommendations 48 Appendix A Comparison of the California Department Of Veterans Affairs’ Estimated and Actual Costs for the Enterprise‑Wide Veterans Home Information System 51 Appendix B An Expanded View of Enterprise‑Wide Veterans Home Information System Use by the California Department of Veterans Affairs’ Veterans Homes 55 Responses to the Audit California Department of Veterans Affairs 59 California Government Operations Agency, California Department of Technology 63 vi California State Auditor Report 2015-121 June 2016 Blank page inserted for reproduction purposes only. California State Auditor Report 2015-121 1 June 2016 Summary Audit Highlights . . . Results in Brief Our audit concerning the development and implementation of the California The mission of the California Department of Veterans Affairs Department of Veterans Affairs’ (CalVet) (CalVet) is to serve California’s veterans and their families. Further, Enterprise‑Wide Veterans Home the mission of CalVet’s veterans homes is to provide the State’s Information System (system) revealed aged and disabled veterans with rehabilitative, residential, and the following: medical care and services in a homelike environment at the State’s » The system has not improved the eight veterans homes. efficiency of the homes’ process for documenting medical care nor has it In 2006, to support the healthcare information needs of its reduced reliance on paper because of five planned new veterans homes and its three existing homes, system flaws. CalVet decided to implement a computerized information system, called the Enterprise‑Wide Veterans Home Information » System instability and concerns System (system), to ensure that veterans receive consistent and about functionality resulted in CalVet integrated care in any veterans home. In its feasibility study report implementing fewer system functions (FSR)—the planning document used to assess the practicality of at some homes, thereby limiting CalVet’s a proposed project—CalVet described the need for a system that ability to provide more efficient care would make its processes more efficient and reduce its reliance for veterans. on paper records. The FSR projected the cost of the system to » CalVet’s project management failed to be $34 million and estimated that it would be complete in 2010. promptly recognize the severity of the Actual implementation of the system began in mid‑2012. However, system’s problems and address them. the system has not achieved the expected or planned efficiencies that CalVet anticipated. Specifically, the system has not improved » The California Department of Technology the homes’ process for documenting medical care nor has it (Technology Department) lacked reduced their reliance on paper records because of flaws that staff documentation to demonstrate that its encountered with the system. Additionally, system instability and efforts to ensure that the settlement with concerns about functionality resulted in CalVet implementing the system contractor was in the State’s fewer system functions at some homes than originally planned, best interest. thereby limiting CalVet’s ability to provide more consistent, efficient care for veterans. Further, although some of CalVet’s » CalVet did not exercise adequate oversight claims of noncompliance with state and federal regulations are of its system project. overstated, its need to take additional steps outside of the system • It did not complete or partially to remain compliant demonstrates that it has not achieved its completed six of the 12 required goal of implementing a system that improved that compliance management oversight plans to with regulations. ensure effective project management. CalVet’s project management failed to promptly identify and • It hired one contractor to provide address the system’s functionality issues. Although it was aware of both independent project oversight such issues as early as mid‑2012, CalVet’s project management did and independent verification and not begin to take steps to address those issues until late 2013. By validation services, and those services that time, it had spent nearly $6 million since staff began notifying were inadequate. them of the functionality problems. The steps CalVet’s project management did take—pausing the implementation of the system, continued on next page . . . sending the system contractor a cure letter to identify specific documents required in its contract that the system contractor had not provided, conducting an assessment of the system, and ultimately ending the contract with the system contractor in 2 California State Auditor Report 2015-121 June 2016 » The Technology Department did December 2014—did not fix the problems. Although the California not adequately fulfill its oversight Department of Technology (Technology Department) facilitated the responsibilities for CalVet’s contract dispute discussions and the negotiations between CalVet system project. and the contractor, the Technology Department could not provide sufficient documentation to demonstrate its efforts to ensure that • It did not identify significant concerns the agreements reached were in the best interests of the State. until August 2012, after Cal Vet When CalVet signed the settlement agreement in December 2014, had already spent $15 million on it had spent $26.2 million on its implementation of the system. By the system. the time it completed its post‑implementation evaluation report in » CalVet’s limited documentation June 2015, its project costs totaled $27.9 million. Although CalVet for the selection of the system is now in the process of identifying a replacement system, it plans contractor prevents it from to continue to use the current system as best it can until it replaces demonstrating it complied with state the system. contracting requirements. We noted several key deficiencies that contributed to CalVet’s • It could not provide proposals for six of failure to implement a system that meets its needs. First, CalVet the seven vendors or the evaluations did not exercise adequate oversight of its system project. for three proposals, including the Specifically, it did not complete or partially completed six of the winning proposal. 12 management oversight plans state policy requires for ensuring effective project management. For the six completed plans, CalVet » CalVet approved payments totaling fully followed only four. For instance, in its configuration change $733,000 for some of its key system control management plan, which describes the process the project implementation deliverables but could team will follow to document, control, and manage changes to not provide adequate documentation of key project components and deliverables throughout the project, receiving these final deliverables. CalVet stated that the change control manager would assign an analyst to conduct impact analyses on each change request to properly understand how each one would affect project costs, scope, and timeline. However, CalVet did not consistently conduct impact analyses on the project’s change requests. Because it did not always follow this plan, it was unable to demonstrate that it properly understood the impact of various change requests on the project’s costs, scope, and timeline. Additionally, the project executive, who is charged with the highest level of project review within the CalVet organization, was not involved with oversight of the management plans. CalVet’s final project manager stated that if project management plans were not being followed during the project, the individual assigned ownership of the plan would have been responsible to escalate the problem to the project manager. Then if the project manager could not resolve the problem, it should have been elevated to the project executive. CalVet did not show that problems regarding the plans were ever escalated to the project executive. Moreover, although CalVet hired one contractor (oversight contractor) to provide both independent project oversight (IPO) and independent verification and validation (IV&V) services for its system project, those services were inadequate. Therefore, CalVet did not identify deficiencies with the system as early as it should have. IPO provides an independent review California State Auditor Report 2015-121 3 June 2016 and analysis of project management practices to determine if the project is being well managed. IV&V provides a client with technically proficient “eyes and ears” to oversee a system vendor while an information technology (IT) system is being developed and implemented. CalVet’s oversight contractor provided both IPO and IV&V services. However, according to our IT expert, separation of IV&V and IPO duties is important and provides a number of advantages to the project and to the State. An important function of IPO is to determine whether IV&V functions are being performed appropriately, and by using the same contractor to perform both functions, CalVet did not ensure that it had effective oversight for this project. For example, our IT expert stated that effective IPO should have raised concerns that IV&V was not managing requirements traceability—the tracing of project requirements throughout the project life cycle to ensure that the system meets specified contract requirements; however, the oversight contractor did not provide this. Although the oversight contractor’s IPO reports should have identified these types of deficiencies in its own IV&V work, it did not. Because the IPO reports did not identify critical work IV&V was not performing, neither CalVet nor the Technology Department received an accurate assessment of whether the system contractor’s processes were effective and whether the system reflected the agreed‑upon quality and solution. The Technology Department also did not adequately fulfill its oversight responsibilities for CalVet’s system project. Since 2008 the Technology Department and its predecessor agencies have been responsible for providing oversight to IT projects, such as CalVet’s system, by reviewing their IPO and other oversight reports. Specifically, the Technology Department did not identify significant concerns with the system until August 2012, after CalVet had already spent $15 million on the system. The deputy director of the Technology Department’s IT Oversight Division indicated that because of the nature of its oversight—that is, reviewing reports that the oversight contractor created—and because the reports did not indicate any critical errors with the project, the Technology Department did not raise any concerns about the project. However, our IT expert reviewed the IPO reports and indicated that they were consistently lacking in critical analysis and that this omission should have raised concerns at the Technology Department. CalVet maintained limited documentation for its process both for selecting a contractor to implement its system and for approving payment of some invoices. This limited documentation prevents it from demonstrating that it made a prudent decision in awarding the contract and in approving payment for deliverables. Specifically, it could not provide the proposals it received from six of the seven vendors that responded to the system’s request 4 California State Auditor Report 2015-121 June 2016 for proposals and it could not provide the evaluations for three of the proposals, including the winning proposal. Because it did not maintain proper documentation, CalVet cannot demonstrate that it complied with state contracting requirements. Additionally, CalVet accepted and approved for payment claims totaling $733,000 for some of its key system implementation deliverables even though it could not provide adequate documentation of receiving these final deliverables. Finally, although CalVet conducted lessons‑learned sessions at various points throughout the project, it cannot demonstrate it used those lessons learned to make improvements in its subsequent implementation. CalVet did, however, capture final lessons learned and has an opportunity to incorporate these lessons in the future. Recommendations CalVet To ensure that its project management of IT projects promptly identifies potential problems and develops resolutions, by September 2016 CalVet should define the project executive and project manager responsibilities to ensure that the individuals who fill those positions take an active role in each project. To ensure that it adequately identifies and monitors problems in its future IT projects, by September 2016 CalVet should establish a formal process for its project executive to verify that the project team prepares all of the required project management plans and other required plans. This formal process should also include a process to periodically verify that the project team is adhering to all these plans. To ensure accountability and independence between the provision of IPO and IV&V services on future IT projects, by September 2016 CalVet should establish a policy requiring it to use separate contractors for IPO and IV&V services when IPO services are not provided directly by the Technology Department. To ensure that it complies with state contracting laws and can demonstrate the basis for its decisions when awarding contracts, by September 2016 CalVet should establish a process to periodically verify that its staff follow state contracting requirements and maintain all required contract documentation. California State Auditor Report 2015-121 5 June 2016 To ensure that it only accepts and pays for deliverables that are complete and that meet the contract requirements, by September 2016 CalVet should establish a process to verify and maintain documentation of all contract deliverables before approving payment. To ensure that it maximizes its opportunity to successfully implement future IT projects, by September 2016 CalVet should establish a formal process to verify that its staff conduct lessons learned sessions for all key phases of future projects. Technology Department To ensure that it can demonstrate that it is acting in the best interest of the State, by December 2016 the Technology Department should create a formal process to summarize its involvement and document key actions taken and decisions reached during agencies’ contract disputes and negotiations for the termination of a contract and maintain those documents according to its records retention schedules. Although the Technology Department indicated that its intent is not to outsource its statutory responsibility for IPO, in any instances when its staff conduct a portfolio review of a project’s IPO, the Technology Department should, by December 2016, establish a process for its review of documents the agency’s IPO contractor creates that verifies whether these reports include critical analysis of project progress and vendor performance so that it can intervene when necessary. Agency Comments CalVet indicated it understands and agrees with each of our recommendations and plans to have several completed by September 2016. The Technology Department agreed with our recommendations and indicated it will take steps to implement them. 6 California State Auditor Report 2015-121 June 2016 Blank page inserted for reproduction purposes only. California State Auditor Report 2015-121 7 June 2016 Introduction Background The mission of the California Department of Veterans Affairs (CalVet) is to serve California’s veterans and their families. Further, the mission of CalVet’s veterans homes is to provide the aged and disabled veterans residing in the State’s eight veterans homes (residents) with rehabilitative, residential, and medical care and services in a homelike environment. The eight veterans homes are located in Yountville, Barstow, Chula Vista, Ventura, Lancaster, West Los Angeles, Fresno, and Redding. As of the end of 2015, they had the budgeted capacity to care for more than 2,300 veterans. Figure 1 on the following page shows the timeline of the opening of each of the homes. CalVet’s veterans homes provide different levels of care to their residents, including domiciliary care, residential care for the elderly, intermediate nursing care, and skilled nursing care. In addition, CalVet provides on‑site outpatient clinics at all of its homes as a service to its residents. These clinics provide comprehensive, multidisciplinary assessments as well as ongoing primary care to address residents’ routine medical needs. Table 1 shows the levels of care available at each of the homes. Table 1 Levels of Care Available at the California Veterans Homes LOCATIONS OF THE VETERANS HOMES WEST LEVEL OF CARE YOUNTVILLE BARSTOW CHULA VISTA LANCASTER VENTURA LOS ANGELES FRESNO REDDING Domiciliary care: Minimum care and supervision for members who are    able to perform all the activities of daily living. Residential care for the elderly: Provides assistance and supervision to members in        activities of daily living. Intermediate nursing care: Provides skilled nursing supervision and supportive   care to members on less than a continuous basis. Skilled nursing care: Provides skilled nursing and supportive care to members on an extended basis, including 24-hour       inpatient care, with medical, nursing, dietary, pharmaceutical, and rehabilitation services, and an activity program. Sources: California Department of Veterans Affairs’ website, Titles 12 and 22 of the California Code of Regulations. wal wal wal wal wal Rene Rene Rene Rene Rene 8 California State Auditor Report 2015-121 June 2016 Figure 1 Location of California Veterans Homes and Timeline of Opening Dates Fresno, Redding— 2013 West Los Angeles, Lancaster, Ventura— 2010 Chula Vista— 2000 Barstow— 1996 Yountville— 1884 SRAEY 211 RREEDDDDIINNGG YYOOUUNNTTVVIILLLLEE FFRREESSNNOO BBAARRSSTTOOWW LLAANNCCAASSTTEERR WWEESSTT LLOOSS AANNGGEELLEESS VVEENNTTUURRAA CCHHUULLAA VVIISSTTAA Source: California Department of Veterans Affairs’ website. History and Evolution of the Enterprise-Wide Veterans Home Information System In late 2006, to support the health care information needs of its planned five new veterans homes and its three existing homes, CalVet sought approval for a new information system that could be California State Auditor Report 2015-121 9 June 2016 used across its eight homes as well as in its headquarters. Former state policy required agencies wishing to launch an information technology (IT) project to prepare a feasibility study report (FSR) that describes the relative need for, cost of, and benefits to be derived from the proposed IT investment. The FSR for the proposed Enterprise‑Wide Veterans Home Information System (system) indicated that the new system would replace an existing implementation of Meditech, a commercial software product that CalVet had used since 1996 to support acute care, ambulatory care, and long‑term care operations at its homes. The FSR defined acute care services as hospital‑like services oriented toward clinically complex patients who need a high level of care; ambulatory care as care provided on an outpatient basis, similar to that provided by a physician’s office; and long‑term care as care delivered over a longer period of time and that includes skilled nursing home, rehabilitation, assisted living, and other services. The FSR stated that the primary reason for the limited use of Meditech was that the software lacked specific functionality for supporting long‑term care operations. In its FSR, CalVet further indicated that considerable technical progress had occurred since the Meditech system was originally installed and that systems were available that could provide coverage across all care settings that CalVet offered. CalVet indicated that newer systems provided improved efficiency, better clinical documentation, and automated medication orders with edits to prevent common medication errors. The proposed new system was intended to be a Key Business Opportunities of the catalyst to transition CalVet from semi‑autonomous Enterprise-Wide Veterans Home homes to an eight‑home integrated system of Information System care. As shown in the text box, in its FSR CalVet described the key business opportunities to be • Improve care management by providing information on realized from the new system. care quality and efficiency. • Improve clinical quality, regulatory compliance, and According to the FSR, CalVet’s implementation of reimbursement by improving clinical documentation. a new system presented an opportunity to take an • Reduce vulnerability to local disasters by reducing affirmative step toward its vision of ensuring that reliance on paper records. veterans receive consistent and integrated care in any CalVet health care facility in California. Source: Enterprise-Wide Veterans Home Information System Feasibility Study Report. It indicated that with the Meditech system, when a veteran transferred between facilities, staff at the new facility were unable to access the veteran’s medical history electronically. This lack of access to care information across facilities hampered clinicians’ ability to provide optimal care and led to increased costs, among other effects. For example, CalVet noted that patients’ tests might need to be repeated when information was not available. The new system was also intended to eliminate the need for cumbersome workarounds and unnecessary paperwork, to automate medication orders, to 10 California State Auditor Report 2015-121 June 2016 reduce the potential for medication errors, and to improve the quality of care veterans receive. The FSR estimated the cost of the system at nearly $34 million and gave a projected completion date of December 2010. In January 2007, the California Department of Finance (Finance) approved CalVet’s FSR for the system. By the time CalVet issued its first special project report (SPR) for approval in August 2008, the project was already experiencing significant delays.1 The FSR stated that CalVet would release the vendor request for proposal (RFP) in September 2007, but delays occurred in the review, validation, and updates to the original system requirements. Further, CalVet added two additional homes (Fresno and Redding) to the scope of the project. These changes, in part, required the creation of the first SPR. Additionally, CalVet took more than six months longer than planned to contract for independent project oversight (IPO) and independent verification and validation (IV&V) services. The SPR indicated that the new release date for the RFP would be November 2008, a delay of 14 months; however, the RFP was not actually released until February 2009, a further delay of three months. Although the RFP was issued, it was canceled in June 2009, as described in the second SPR that CalVet submitted, because only one vendor submitted a proposal and that proposal did not address all of the required system functionality. A request for information (RFI) was issued in July 2009 to a broader vendor pool to determine their ability to meet CalVet’s system requirements. The outcome of the RFI indicated that none of the vendors could meet all of CalVet’s mandatory requirements; as a result, CalVet revised the RFP. For instance, according to the second SPR, the revised RFP listed significantly fewer mandatory requirements and contained a new “highly desirable” category of requirements that were not mandatory. The second RFP was released in November 2009. In December 2010, CalVet awarded the project to Solutions West Consulting, LLC (Solutions West; later Brekken Technology, Inc.) as its system contractor. The project was contained in two contracts. The first contract and two subsequent contract amendments of work included the purchase of system hardware and software, the installation and configuration of the system, and the training of CalVet staff on how to use the new system. The second contract provided for the post‑implementation maintenance and support services for the system by the system contractor. CalVet’s third and final SPR, in January 2013, adjusted the projected cost of the system to $36.7 million. 1 An SPR provides a summary of proposed changes to the original project cost, schedule, or scope. It is generally required when the project cost or total financial program benefits deviate or are anticipated to deviate from the original by 10 percent or more, or when a major change occurs in project requirements or methodology. California State Auditor Report 2015-121 11 June 2016 CalVet’s headquarters and its homes in Barstow, West Los Angeles, Lancaster, and Ventura started using selected modules of the system during the system’s pilot implementation in May 2012. The home in Chula Vista implemented and began using the system in June 2013. The final three homes—in Yountville, Fresno, and Redding—implemented and began using modules of the system in November 2013. The eight homes’ use of the system modules currently varies. According to the manager of the project management office (PMO manager) of CalVet’s Information Services Division (ISD), the system was not fully implemented and used at all of the homes because of certain functionality problems staff experienced with the system. Although both CalVet and the California Department of Technology (Technology Department)—the department that now has responsibility for oversight of state IT projects—had identified project delays during the development and early implementation of the system, according to CalVet’s June 2015 response to questions from the Legislature regarding the project, new leadership in CalVet’s ISD reported critical project problems to the Technology Department in December 2013.2 According to the final project manager for the system, CalVet identified these problems in late 2013 through a review of outstanding unpaid invoices, many of which lacked sufficient documentation to support the work performed. The June 2015 response document stated that the critical project problems CalVet identified related to project oversight, project management, contract management, contract deliverables, and the viability of the system contractor. The response document noted that to resolve these deficiencies, CalVet effectively suspended the project in December 2013 and, after a 12‑month period for reevaluation and remediation, terminated the implementation contract with the system contractor and a system support contract with the software vendor in December 2014. The response document also stated that CalVet and the Technology Department mutually agreed to close out the system project in January 2015. CalVet agreed to pay the system contractor $350,000 to settle the implementation contract. By the time CalVet reached its settlement agreement with the system contractor, it had spent $26.2 million, including the settlement costs. These costs had increased to $27.9 million when CalVet completed its post‑implementation evaluation report in June 2015. Figure 2 on the following page shows key milestones during the course of the project. 2 Until July 2013 the Department of Technology was known as the California Technology Agency, and before that, it was the Office of the Chief Information Officer. 12 California State Auditor Report 2015-121 June 2016 Figure 2 Enterprise-Wide Veterans Home Information System Project Timeline 2007 January 2007 Indicates project oversight’s identification of 2007 I s n y d st isc e ya m tse o tse r p p mr r o o j e j o e c c rt t po c v o reo n r c sj e ieg r ch n tt s ’ . sc iodenncteifircnatsio.n of J C a al n ifo u r a n r ia y D 2 e 0 pa 0 r 7 tment of Finance C ap al p if r o o r v n e i d a D fe e a p si a b r i t l m ity e n st t u o d f y F r in ep an o c r e t (FSR) approved feasibility study report (FESsRt)i mated project cost: $34 million Estimated project cost: $34 million 2008 Decemb2e0r 200087 December 2007 California Department of VeteransC Aafflaiifros (rCnailVae Dt) epartment of Veterans Affairs (CalVet) contracted for independent project oversight and independent verificactoionntracted for independent project and validation services oversight and independent verification 2009 and validation services January 2009 The Office2 of0 th0e S9tate Chief Information Officer (OCIO) approved special project reJpaornt (SuPRa) r1y 2009 Estimated project cost: $35.5 million The Office of the State Chief Information Officer November 2009 (OCIO) approved special project report (SPR) 1 2010 CalVet published its request for pro Epsotsiaml (aRFtPe)d project cost: $35.5 million November 2009 2010 December 2010 2011 November 2010 CalVet published its request for proposal (RFP) Department of General Services OCIO approved SPR 2 (General Services) approved the Estimated project cost: $37.3 million implementation contract December 2010 February 2012 November 2010 May 2012 2012 General S2erv0ice1s a1pproved the maintenance and Pilot implementation at hDeeadpqauratrmteres,n Bta rostfo Gw,e neral Services support contract with the system cOonCtIrOac taoprproved SPR 2 West Los Angeles, Lancaster(, Ganedn Veernatul rSa eacrcveipcteeds) approved the June 2012 Estimated project cost: $37.3 million implementation contract CalVet identifies that it is having issues August 2012 with the pharmacy implementation in an California Department of Technology 2013 executive steering committee (steering (Technology Department) January 2013 committee) meetinFgebruary 2012 identified project schedule February 2013 variance a n m d i s o s v e e d r s b i y g P h p t i r l o c o o je t n c t i t r m a te c a p to m l r emen C c a to l a V mt e im t o n int o t t ea e e d t m i h n e e a ea t s i t nd e gq e t r uh in aa g tr t M ers a , y Ba 2 rs 0 to 1 w 2 , T E e st c i h m n a o te lo d g p y r o D j e e p c 2 t a c r0 o tm st 1 : e $ n 3 2t 6 a .7 p p m r i o ll v io e n d SPR 3G s u e p n p e o ra rt l S co er n v t i r c a e c s t a w p i p th ro t v h e e d s y th st e e m m a c in on te t n ra a c n t c o e r and West Los Angbe B el a ien rs gs t , o d L w roa h pn a p s ce m ad s e frt d oe i m c r a , t ta i h o ne n sd o y r sV d te e em r n s tura accepted June 2013 June 2012 Completed implementation at Chula Vista CalVet identifies that it is having issues NoAveumgbuesr t2 0210312 with the pharmacy implementation in an Completed implementation at California DFerepsnaor, tRmedednintg o, afn dT Yeocuhnntvoillleogy 2013 December 2013 executive steering committee (steering (Technology Department) January 2014 2014 CalVJeat rnepuoraterd ycr it2ic0al 1co3ncern toc ommittee) meeting va Ca ri lV a e n itd c see ei a n n n n m t a t a d d i i fic iss o u s e p r v eu ed e td l e e r p t s bb tr i e eo y g t r w j h t p eo e t r c e t o c ht n e o j s Ce s n c ac yh l t s Vt r te e e a tt dm e c aa u t n c o md ole n r t h tr e a c s t y o s r t e w C m c h a i o c l c o V h m n e r t e m r t a s Fu n c i t l t e o t o t e r t b e d * e e r d u m in a e a r e y t s i t n 2 e g e 0 t r 1 h in a 3 g t F T d h i e s e c b u T r s e u s c i h a o n n r o y s l – b o e M g t y w a D e r e e c p n h a C r 2 t a m l 0 V e 1 e n t 4 t a f n a d c ilitated t p r m e h r o s e o o v p T T E l i o e v n e s i c s g n t ce h i g f d n h mo c o p r n r wa l a i o o t u ta g ic e ls r y o a d i d n l D g g c p e o yt p r n h a o D c e r e j t ep e r m n rp c o e s t aj n b e c r t c e o t a t f m o n s a r t d n e : e d $ n 3 t 6 a .7 p p m r i o ll v io e n d SPR 3 May–August 2014 the system contractor As directed by the Technology Departm B en a t r , s C t a o lV w et c h o a nd s u m cte e d d ication orders JSuepnteem 2b0e1r 320 14 an assessment of the system to debteermininge dwrhoetphepre itd w afsr om the system CalVet suspended the use of functioning properly and met CalVet’s needs * Cthoem orpdeler teentdry i fmunpctlieomnaelitnytation at Chula Vista November 2014 November 20D1e3cember 2014 2015 Maintenance and support • CalVet signed a maintenance and support contract with contractor expired Completedco inmtrapclte wmithe tnheta etxiisotnin ga sto ftware vendor Fresno, Redding, and Yountville December 2013 • CalVet, Technology Department, and the contractor signed a settlement agreemeJnat. nuary 2014January 2015 2014 CalVet reported critical concern to CalVet sent a cure l • e A t s t$ e o 2 f r6 t .t h 2o e m e tih n ll d ieo o ns f yo 2 sn 0 t 1 te 4 h me C a s l yc V so e t t en h mt a r p d a r s co p tj e eo n crt t which resulted C ag a r lV e e e t d a t n o d c l t o h s e e T o e u c t h t n h o e lo p g r y o j D ec e t p F† a e rtm b e r n u t m a u r tu y a – lly M arch 2014 t p h r e o p T o ec se h d n o p l a o u g s y i n D g e t p h a e r t p m ro e j n e t c a t n a d n d in a dispute between CalVet and the system contractor* The Technology Department facilitated resolving critical concerns before discussions between CalVet and moving forward May–August 2014 the system contractor September 2014 As directed by the Technology Department, CalVet conducted an assessment of the system to determine whether it was CalVet suspended the use of functioning properly and met CalVet’s needs * the order entry functionality November 2014 December 2014 2015 Maintenance and support • CalVet signed a maintenance and support contract with contractor expired contract with the existing software vendor • CalVet, Technology Department, and the contractor signed a settlement agreement. January 2015 • As of the end of 2014 CalVet had spent CalVet and the Technology Department mutually $26.2 million on the system project agreed to close out the project† Sources: CalVet’s FSR and SPRs for the Enterprise-Wide Veterans Home Information System (system), FSR and SPR approval letters, contracts for the system, project management documents, cure letter, the implementation contract settlement agreement, post-implementation evaluation report—June 2015, and the Technology Department’s system briefing document. * The cure letter identified the contract requirements that the system contractor had not met and required that the system contractor meet those requirements by February 13, 2014. † According to steering committee meeting minutes, direction was given to close out the system project through the post-implementation evaluation report (PIER) and initiate efforts to replace the system. In June 2015 CalVet submitted its PIER to the Technology Department to support its request for the Technology Department to consider the system project complete and to terminate project reporting. At that point, CalVet’s project costs totaled $27.9 million. According to CalVet’s deputy secretary of veterans homes division, CalVet continues to use the system as best it can until it completes the process of finding a replacement system. California State Auditor Report 2015-121 13 June 2016 Changing Roles and Responsibilities for Project Approval and Oversight Both the Technology Department and the Department of General Services (General Services) have had certain responsibilities for overseeing the State’s IT project procurements, and as shown in Figure 3 on the following page, each entity’s roles have changed since January 2007 when Finance approved CalVet’s FSR for the system. Figure 3 also identifies the roles and responsibilities held by oversight agencies at key milestones during the development and implementation of CalVet’s system project. Effective January 2008, a change in state law transferred the authority to approve, suspend, or terminate IT projects from Finance to the Office of the Chief Information Officer—the predecessor entity to the Technology Department. In December 2010, when CalVet executed the contract to develop and implement the system, General Services had responsibility to review state agencies’ IT RFPs. Then state law effective in 2011 required General Services and the Technology Department to review all IT RFPs. Subsequent legislation operative in July 2013 transferred General Services’ share of this authority as well as General Services’ authority over IT project procurement to the Technology Department. This shift in responsibilities from General Services to the Technology Department, and the increase in the Technology Department’s responsibilities, reflects the Legislature’s conclusion that the unique aspects of IT projects and their importance to state programs warrant a separate acquisition authority. State policy requires departments to implement independent oversight for all reportable projects. The Technology Department, which assumed IT project oversight responsibility under state law effective in 2008, generally provides the IPO services while state entities undertaking the IT projects are responsible to contract for the IV&V services. IPO services provide an independent review and analysis of project management practices to determine if the project is being well managed. IV&V services provide a client with technically proficient “eyes and ears” to oversee a system vendor while an IT system is being developed and implemented, and they also provide an early warning of process and technical discrepancies, issues, and problems that might not otherwise be detected until late in testing or implementation For its system, CalVet contracted with a single outside consultant (oversight contractor) to provide both IPO and IV&V services. 14 California State Auditor Report 2015-121 June 2016 Figure 3 Information Technology Roles and Responsibilities by Key Milestone for the Enterprise-Wide Veterans Home Information System Project California Department of Technology California Department of Finance (Technology Department)* Department of General Services Shared May 2012 First implementation of the Enterprise-Wide Veterans Home Information System December 2013 January 2007 November 2009 Second contract Feasibility study RFP amendment report approved December 2010 January 2015 Contract awarded California Department July 2011 of Veterans Affairs and the Technology First contract Department mutually amendment agree to close project Information Technology (IT) Project Approval and Oversight† Review of IT Request for Proposal (RFP) Review of IT Project Procurement 2007 2008 2009 2010 2011 2012 2013 2014 2015 Sources: Government Code, sections 11545 and 11546 and Public Contract Code, sections 12100 and 12104. * Until July 2013, the Technology Department was known as the California Technology Agency and before that the Office of the State Chief Information Officer. † IT project approval and oversight include numerous activities, such as evaluating IT projects based on the business case justification, resource requirements, proposed technical solution, project management, oversight and risk mitigation approach, and compliance with statewide strategies, policies, and procedures. California State Auditor Report 2015-121 15 June 2016 Scope and Methodology The Joint Legislative Audit Committee (audit committee) directed the California State Auditor to conduct an audit of the development and implementation of CalVet’s Enterprise‑Wide Veterans Home Information System. Table 2 outlines the audit committee’s audit objectives and our methods for addressing them. Table 2 Audit Objectives and the Methods Used to Address Them AUDIT OBJECTIVE METHOD 1 Review and evaluate the laws, rules, • Reviewed relevant laws, rules, and regulations. and regulations significant to the • Reviewed relevant state policies regarding procurement and information technology (IT) projects. audit objectives. 2 Evaluate the level of oversight the • Identified requirements for oversight of development and implementation of IT projects. California Department of Veterans • Obtained and reviewed documentation related to CalVet’s project management and other Affairs (CalVet) management required plans to determine compliance with the California Project Management Methodology. exercised regarding development • Obtained and reviewed executive steering committee (steering committee) meeting minutes to and implementation of the determine the extent of oversight by CalVet’s management, and involvement and decision making Enterprise-Wide Veterans Home by its executive office related to its system project. Information System (system), including whether the executive office • Obtained and reviewed independent project oversight (IPO) and independent validation and was involved in making key decisions verification (IV&V) reports prepared by CalVet’s oversight contractor to determine whether IPO and and ensuring that the system met the IV&V services were provided over the course of the system project and that they adhered to state needs of the veterans homes. policy and contractual requirements. • Our IT expert identified requirements traceability monitoring as an important function of IV&V. Because he did not see evidence that this function was performed by CalVet’s oversight contractor, we obtained and reviewed invoices approved for payment and requested that CalVet provide us with evidence of an accepted deliverable for proof that this work was performed. • Interviewed key staff at CalVet to understand its oversight role. 3 Assess whether CalVet followed laws, • Identified relevant laws, regulations, and policies regarding the selection of the contractor to rules, regulations, policies, and best implement CalVet’s system. practices when selecting vendors • Obtained documentation to confirm CalVet’s request for proposals (RFP) was properly advertised for the system, including, to the in the California State Contracts Register. extent possible, those prohibiting • Requested and reviewed the proposals CalVet received in response to its RFP for implementation a conflict of interest during the of its system and documentation for the evaluations of the proposals. selection process. • Requested and reviewed CalVet’s Statement of Incompatible Activities (SIA) and Conflict-of-Interest Code. • Requested and reviewed the California Fair Political Practices Commission’s Statements of Economic Interests (Form 700) for applicable staff that participated in the selection of the system contractor. The forms we obtained did not identify any reported conflicts. • Requested and reviewed CalVet’s certified SIAs for members of the committee selecting the contractor implementing the system. The forms we obtained did not identify any reported incompatible activities. Interviewed key individuals at CalVet to determine whether any additional methods existed at CalVet to determine potential conflicts of interest. continued on next page . . . 16 California State Auditor Report 2015-121 June 2016 AUDIT OBJECTIVE METHOD 4 Determine the estimated and actual • Reviewed documentation to determine the approved project cost estimates. implementation costs and timeline • Obtained California State Accounting and Reporting System (CalSTARS) accounting records to for the system as well as the number identify the project’s total cost, as reported in CalSTARS. of and reasons for change orders and - Estimated and actual implementation costs are located in Table A.1 in Appendix A. contract amendments. • Reviewed project documentation to determine the estimated project implementation timelines and the actual implementation timeline. - Estimated and actual implementation timelines are located in Table A.2 in Appendix A. • Reviewed the two contract amendments to determine the reason for each. - The first amendment increased the contract costs to include unanticipated tasks—as permitted by the contract—and made minor changes or clarifications to the scope of work. - The second amendment extended the contract by one year, updated the statement of work to reflect the system contractor’s name change, and updated the contact information for the project site coordinators. • Reviewed the list of change requests and determined that there were 304 approved change requests. We reviewed a total of 64 change requests to determine the reason for the changes including all change requests categorized as a significant change, a portion of those categorized as a minor change, and a portion of those that were uncategorized. We identified five main reasons for the change requests we reviewed: - Contract language update or clarification. - Deliverable or requirement change. - Requirement start or due date change. - System enhancement or software change. - Business process change. 5 Determine whether the original • Interviewed key staff at CalVet to determine whether the system contractor met all project requirements, as defined by contract requirements. the scope of work, were delivered • Compared the original statement of work to each amended statement of work to determine what during implementation of the changes were made to the project requirements. system project. • Reviewed change requests to determine how those changes affected the original requirements. • Reviewed selected contract deliverables documentation (that is, deliverable expectation documents and deliverable acceptance documents) for the 33 deliverables to assess whether the system contractor met the project requirements, as defined by the statement of work. • Reviewed CalVet’s post-implementation evaluation report and steering committee minutes to determine the cause for any gaps between the contract requirements and the deliverables documentation. 6 Evaluate the steps CalVet took when • Reviewed CalVet’s executive summary reports, project status reports, special project reports, and project variances were identified. IPO and IV&V reports to determine when CalVet identified project variances and what steps Cal Vet To the extent possible, determine took to address those variances. whether CalVet could have identified • Interviewed key staff at CalVet. problems with the system earlier. • Provided the IPO and IV&V reports to our IT expert for his professional opinion as to whether CalVet should have identified problems with the system earlier. 7 Review the role of the California • Reviewed relevant laws and policies to identify the Technology Department’s oversight Department of Technology responsibilities throughout the life of the system project. (Technology Department) in this • Interviewed key individuals at the Technology Department to determine the Technology project and evaluate whether it Department’s role throughout the life of the system project. fulfilled its roles and responsibilities. • Requested and reviewed documentation from the Technology Department to determine the extent of oversight it provided to CalVet and whether it fulfilled its oversight responsibilities. California State Auditor Report 2015-121 17 June 2016 AUDIT OBJECTIVE METHOD 8 Determine the steps CalVet has • Interviewed key project management at CalVet headquarters and the homes and reviewed taken to improve the functionality available documentation to gain an understanding of the following: of the system and what affect - The system’s lack of functionality. any lack in functionality has had - The impact the lack of functionality has on the homes’ quality of care and cost and efficiency. on quality of care and cost and efficiency of the homes’ operations. - Steps CalVet has taken to address the lack of functionality. In addition, determine whether • Reviewed documentation related to lessons learned to determine what changes, if any, CalVet CalVet has documented lessons made as a result of the lessons learned. learned for use in future phases of system implementation. 9 Identify the level of system • Interviewed key staff at CalVet and the eight veterans homes. functionality and use within CalVet’s • Obtained a list of system functionality implemented and used at headquarters and each veterans veterans homes and administrative home. We asked staff at headquarters and each home to verify their use of system components. offices and evaluate CalVet’s efforts • Reviewed training documentation for a selection of training course evaluations from CalVet and to train staff and otherwise address found generally positive comments about the training. We followed up with staff at the homes resistance to using the system. who mostly expressed positive comments about the training. • Reviewed CalVet’s organizational change management strategy to address resistance to the system brought about by organizational change. We followed up with staff at the homes and found that, in general, staff indicated that there was buy in to the system. 10 Review and assess any other issues • Interviewed key staff at the Technology Department to understand its facilitation of the contract that are significant to the audit. disputes between CalVet and the system contractor. • Obtained and reviewed documentation to assess whether the Technology Department’s actions as the dispute facilitator were in the best interests of the State. Sources: California State Auditor’s analysis of the Joint Legislative Audit Committee’s audit request number 2015-121, and information and documentation identified in the column titled Method. Assessment of Data Reliability In performing the audit, we obtained financial records from the California State Accounting and Reporting System (CalSTARS) for CalVet’s expenditures related to its system. The U.S. Government Accountability Office, whose standards we are statutorily required to follow, requires us to assess the sufficiency and appropriateness of the computer‑processed information that we use to support our findings, conclusions, and recommendations. Specifically, we used expenditure reports from CalSTARS from July 2007 to June 2015. For each fiscal year, we used these data to report CalVet’s total expenditures for its system implementation. However, we did not conduct accuracy or completeness testing on these data because, in accordance with its records retention policy, CalVet did not maintain source documents for many of its historical transactions that we would need to test. As a result, any testing we performed would be incomplete. Further, CalVet stores the source documents it does maintain at locations throughout the State, making this testing cost‑prohibitive. Thus, we determined that the State’s CalSTARS data were of undetermined reliability for the purposes of this audit. Although this determination may affect the precision of the numbers we present, there is sufficient evidence in total to support our audit findings, conclusions, and recommendations. To gain some assurance of the accuracy of 18 California State Auditor Report 2015-121 June 2016 CalVet’s system transactions recorded in CalSTARS, we tested a selection of expenditures and hours worked for fiscal years 2013–14 and 2014–15, and we determined that CalVet accurately recorded the transactions we reviewed. California State Auditor Report 2015-121 19 June 2016 Audit Results The Enterprise-Wide Veterans Home Information System Does Not Meet the Needs of the Veterans Homes The California Department of Veterans Affairs (CalVet) implemented the Enterprise‑Wide Veterans Home Information System (system), but the system does not meet the original primary goals of the project. In its feasibility study report (FSR) requesting approval for the system, CalVet described the need for an information system that would improve the quality of care delivered to veterans by making its clinical documentation process more efficient; reducing reliance on paper records; providing a consistent, safe, integrated system of care; and improving regulatory compliance. However, the system implemented did not achieve those improvements. The System Has Not Achieved Key Project Objectives of Improving Efficiency and Improving Quality of Care to Veterans The system implemented has not improved the efficiency of the homes’ clinical documentation process or reduced their reliance on paper records because of flaws that staff encountered with the system. The administrators from four of the veterans homes, which use a majority of the system’s modules, including ADL Clinical—a main module of the system that the homes use to manage electronic health records—identified shortcomings with how the system works and indicated that the system is not user‑friendly or intuitive. Specifically, some of the administrators told us that the system is difficult to navigate and results in staff spending unnecessarily long amounts of time locating or entering information multiple times into residents’ patient care records. The chief medical officer at the home in Chula Vista told us that because of concerns staff had about the functionality of the system, given the numerous problems they experienced while using it, he had the staff conduct time studies of some routine tasks in July and early August 2013. Specifically, in a document he provided to CalVet headquarters, he described that staff in four departments performed five different routine tasks and the results revealed that these tasks required approximately twice as much time on average after the system was implemented. For example, a registered nurse determined that it took 105 minutes to conduct a new resident’s initial assessment using the system, compared to the 30 minutes for the same task before the system was implemented. In another example, a pharmacist found that it took seven minutes to process a medication order using the new system, a task that required 3.5 minutes before the system was implemented. Despite these concerns, in late August 2013 CalVet’s then‑undersecretary of Veterans Homes 20 California State Auditor Report 2015-121 June 2016 responded that the time it was taking staff to complete processes with the system would likely decrease over time as they became more comfortable with the system and refined the business processes. In addition to the extra time it was taking them to complete tasks using the system, staff found that they had to enter the same resident information into multiple modules of the system to provide care to the residents. The administrator at one home stated, for example, that when a home receives a new resident, staff must enter the resident’s information into the system’s admission module before they can admit the resident. A nurse then has to reenter that same information into the system’s assessment module before staff can conduct the assessment of the resident. Staff also reported that some system functions would regularly freeze or lock up during use. In one extreme instance, the entire system was down for more than a week at the end of August and beginning of September 2013. Because of continuing concerns, in mid‑2014 CalVet conducted an assessment of the system to determine if it was functioning properly and if it met the quality and needs of CalVet. In its response to CalVet’s assessment, the system contractor indicated that a majority of the more than 500 items identified required setup and training, and many other items could be resolved through enhancements. Further, the system contractor stated there were a dozen identified bugs or issues and that it needed clarification on about 100 items. CalVet’s final project manager stated that the system problems the homes experienced—for example, system slowness and freeze‑ups— were a result of the system’s poor architecture and configuration. Administrators of the homes Administrators of the homes indicated that at times, they resorted indicated that at times, they to manual processes to manage patient care because of the problems resorted to manual processes to they encountered with the system. Two of the five homes that use manage patient care because of the the system’s ADL Clinical functionality—Barstow and Ventura— problems they encountered with indicated that staff reverted back to paper processes when the system the system. freezes or crashes. In fact, the management of the Ventura home stated that staff use paper documentation as a backup because they do not trust the system to work properly. Two other homes that use the system’s ADL Clinical module—the Chula Vista and Lancaster homes—also indicated that staff use paper documentation in addition to the electronic system for assurance. For the remaining three homes—those that use only a minimal number of the ADL Clinical module’s components—two stated that they continue to use paper records for clinical documentation while the third home continues to use the old system—Meditech—and other workarounds. When we asked CalVet about how these functional problems have affected the quality of care veterans receive, its deputy secretary of Veterans Homes stated that the quality of care has not changed and that the homes continue to pass surveys performed by the United States Department of Veterans Affairs and the California California State Auditor Report 2015-121 21 June 2016 Department of Public Health with the system in place. However, he added that the system is not maximizing efficiency because nurses are having to spend more time entering information in the system, limiting the time they are able to spend with residents. Therefore, we believe the overall quality of care provided to the veterans is negatively affected by these inefficiencies because of the increase in time that staff spend using the system. Additionally, staff have encountered serious problems while using the order entry component—the system component that staff use to place a resident’s medication order—that have created a risk of harm to the residents. Specifically, staff reported problems that Staff reported serious problems included the system recording incorrect medication dosages on with the system’s medication prescriptions, failing to notify staff that a resident was allergic order entry module that to a medication, and failing to note verification of whether the included the system recording pharmacy had received prescription orders, resulting in the system incorrect medication dosages on not placing medication orders. For example, in March 2014, staff prescriptions and failing to notify in one home placed on hold a resident’s medication to lower her staff that a resident was allergic to cholesterol, and the system automatically expired the medication a medication. order after seven days because staff did not release the hold. Staff reported to the chief medical officer that the system did not alert the nursing staff that the medication order had expired, and as a result, the resident did not receive her medication for five months. In another example, the administrator at the Ventura home stated that although a veteran’s allergies are recorded in one module of the system, the system does not alert the nurse if a medication to which a resident is allergic is entered in the medication order module. In August 2014, staff at another home reported that a resident’s order for eye drops to treat glaucoma dropped from the system and the patient did not receive the drops for two months; the staff at the home were not aware that the resident was missing the eye drop medication until his son notified them. In response to these types of problems, CalVet suspended the use of the medication order entry module of the system in September 2014. As CalVet identified during its assessment of the system in 2014, staff have to enter patient information more than once and they have encountered various problems in the medication order entry functionality. According to the final project manager, these problems occurred because the system contractor did not configure the system correctly. In April 2015, ADL—the software provider—also communicated to CalVet that based on its work to resolve outstanding trouble tickets over the past three months, it had concerns with the system’s configuration during installation and with the instruction of CalVet staff on the proper setup and implementation of the software. Our information technology (IT) expert reviewed the system’s draft configuration document and determined that it was incomplete and could not be reviewed for quality or correctness. He also noted that the draft architecture design document was incomplete and insufficient and would likely 22 California State Auditor Report 2015-121 June 2016 be a source of implementation problems. However, as we describe later in this report, it was CalVet’s responsibility to monitor its system contractor’s performance and to ensure that the system contractor provided deliverables consistent with the terms of the contract. The Veterans Homes’ Use of the System’s Modules Varies The system has also failed to serve as the integrated system of care that CalVet had planned and expected. CalVet envisioned its integrated system of care as enabling any veterans home to operate seamlessly with any other home, providing access to care information should a veteran move between homes. Because of the problems with the system, however, CalVet implemented only a portion of the system at the last three homes to receive it. According to the final project manager, headquarters and all of the homes received all modules of the system software but some did not receive training on all modules. He stated that when some of the planned tasks, such as training, are not completed, the module is not considered fully implemented. Specifically, three homes—Fresno, Redding, and Yountville—confirmed that although CalVet installed the ADL Clinical module, their staff either did not receive training on it or received training on only a minimal number of the module’s components. Additionally, according to CalVet’s deputy secretary of Veterans Homes, because of the problems the homes have encountered with the system, CalVet allowed individual homes to use the system’s modules in ways that worked best for each one. Because CalVet did not overcome CalVet’s decision to halt the training and limit the use of a system the problems with the system’s module that was not fully working and to allow each home to use the functionality, it failed to achieve modules that worked best for that particular home is understandable its goal of having a fully integrated given the significance of the problems the homes were experiencing. system of care in which the However, because CalVet did not overcome the problems with the homes’ use of the system is system’s functionality, it failed to achieve one of its primary goals: a standardized and veterans receive fully integrated system of care in which the homes’ use of the system consistent care. is standardized and veterans receive consistent care. For example, as we discussed in the Introduction, CalVet’s FSR stated that under the previous system, when a resident transferred to another home, the staff at the new facility were unable to access the individual’s medical history electronically, hampering their ability to provide optimal care and increasing costs. In addition, the FSR indicated that increasing the amount of clinical documentation that the homes recorded electronically would improve CalVet’s ability to evaluate the outcome of services and procedures. It also indicated that improved cost reporting would enable CalVet to monitor and manage the costs of services across different facilities. However, because the homes’ use of the system’s modules varies and because the ADL Clinical module was not fully implemented at all of the homes, the staff at the homes cannot communicate with each other’s systems to share any necessary resident information. Table 3 shows when CalVet implemented the California State Auditor Report 2015-121 23 June 2016 system at its headquarters and each of the eight homes as well as the extent of the system modules in use at each of those locations. Without a fully integrated system that is used in a standardized way, the homes cannot operate seamlessly with other homes and electronically share care information as the planned integrated system envisioned. Table 3 Enterprise-Wide Veterans Home Information System Module Use by Location and Date of Implementation WEST HEADQUARTERS, BARSTOW, CHULA VISTA, FRESNO, LANCASTER, REDDING, VENTURA, LOS ANGELES, YOUNTVILLE, MAY 2012 MAY 2012 JUNE 2013 NOVEMBER 2013 MAY 2012 NOVEMBER 2013 MAY 2012 MAY 2012 NOVEMBER 2013 Multiple Component Modules ADL Clinical (23 components) Software designed to address critical functions of providing long-term care by capturing and managing patient information. Fully (F) 1 14 9 1 11 2 13 11 3 Partially (P) 0 5 5 0 4 0 3 6 1 Not used (N) 2 3 8 0 3 0 2 5 0 Not Implemented (NI) 0 0 0 22 0 21 0 0 19 Not applicable (NA) 20 1 1 0 5 0 5 1 0 ADL Financial (36 components) Software used to support third-party billing, Medi-Cal, Medicare, accounts receivable, transaction history, withdrawals and charges; making, tracking, and controlling financial events. Fully 12 30 29 23 13 24 26 21 29 Partially 7 3 4 5 11 5 2 7 2 Not used 1 3 3 6 10 5 6 8 2 Not Implemented 0 0 0 2 0 2 0 0 3 Not applicable 16 0 0 0 2 0 2 0 0 Single Component Modules Dynamics Financial accounting and business management software that automates creation and management of accounting data and workflow. Used to augment ADL Financial to support financial management. The specific component used is for purchasing and inventory. N F P F F F F F F Gerimenu Software used for resident meal planning to help facilities run their nutrition departments more efficiently. NA P P F N NI N F P Documentum An enterprise content management system used to house and manage electronic documents. N F N NI N NI F F NI Framework LTC Pharmacy management software designed for long-term care and institutional facilities. NA F F NI N NI F F F Sources: California Department of Veterans Affairs’ (CalVet) system use chart provided by system final project manager, and confirmed or updated by staff at headquarters and the eight homes, draft system architecture design document, and the CBORD Group Inc. Gerimenu website, and contractor’s response to the request for proposals. Note: A complete table of system use by components is located at Table B in Appendix B. F = The veterans home indicated it fully used this module. P = The veterans home indicated it partially used this module. N = Although CalVet implemented the module, headquarters or the veterans home indicated it did not use it. NI=CalVet did not implement the module at the veterans home. NA= Although these modules were available for use, staff at headquarters or the homes indicated that components marked NA were generally not applicable for their operations. 24 California State Auditor Report 2015-121 June 2016 CalVet’s Assessment Results Overstated System Compliance Concerns In a June 2015 response to questions it received from the Legislature, CalVet included the results of its assessment of the system and identified several areas in which its system was not complying with the provisions of certain federal and state laws and regulations. However, we found that CalVet is compliant with the laws and regulations it identified or, in one instance, can take steps to avoid reductions to federal reimbursements. Although CalVet indicated that the Although CalVet’s assessment results indicated that the system system does not comply with HIPAA does not comply with the federal Health Insurance Portability and regulations related to its ability to Accountability Act (HIPAA) regulations related to its ability to audit audit record activity, we found that record activity, we found that it is compliant. The HIPAA privacy it is compliant. and security regulations require covered entities to implement appropriate administrative safeguards to ensure the protection of electronic personal health information. The regulations require, among other things, the implementation of audit controls, such as audit logs that record and examine activity in information systems that contain or use protected electronic health information. CalVet indicated that the system’s limited ability to audit health record activity makes it noncompliant with HIPAA. Specifically, according to CalVet’s agency information security officer, rather than having the ability to access audit logs directly from its system, CalVet must request audit logs from the software provider. Additionally, CalVet’s chief information officer stated that based on her experience, the time it takes the software provider to produce the logs for CalVet is unreasonable. However, the assistant director of the State of California Office of Health Information Integrity—which has statutory oversight of state entities’ HIPAA compliance—stated that as long as CalVet can request and receive audit logs, it is compliant with that requirement of HIPAA. Additionally, CalVet indicated that the system does not comply with federal and state laws that prohibit the use of color coding in federal and state agency information systems as the only means of conveying information. For example, a form that requires a user to complete only the areas in red would be noncompliant. CalVet identified a concern that critical on‑screen indicators within the system are coded based on color, and these indicators do not work for individuals who are color blind. However, in each example CalVet provided to us from the system, we found that the color coding was in addition to another means of conveying the information. For example, a patient’s discharge date is listed in red to indicate that the patient has been discharged; however, the existence of a discharge date would notify a staff member that the patient has been discharged; thus, the system does not California State Auditor Report 2015-121 25 June 2016 rely solely on color coding to convey that information. As a result, CalVet could not demonstrate that the system inappropriately relied on color coding. Further, a key provision of the American Recovery and Reinvestment Act of 2009 (ARRA) went into effect in January 2014, requiring public and private health care providers and other eligible professionals to adopt and demonstrate meaningful use of electronic medical records to maintain existing Medicaid and Medicare reimbursement levels. As part of ARRA, Congress mandated a reduction to Medicare Part B reimbursements for eligible professionals’ services if the eligible professionals performing the service did not demonstrate meaningful use of certified electronic health records. CalVet indicated in its assessment results that ADL—the system software vendor—is not certified. To determine the extent of these reductions in reimbursements for eligible professionals, we asked CalVet how many of its professionals were affected by the reductions. The staff services manager of CalVet’s Medical Cost Recovery and Support Unit indicated that although her staff reviewed accounting records and identified some eligible professionals who received the Medicare reimbursement reductions in 2016, the staff have not reviewed all facilities’ accounting records to determine all of the individuals affected. As of early May 2016, CalVet did not know how many of the homes’ eligible professionals will be impacted by the Medicare reimbursement reductions. However, in our review we noted that eligible professionals can apply for hardship exceptions through the federal Centers for Medicare and Medicaid Services in certain categories to avoid this payment reduction. One of the exemption categories is extreme or CalVet’s eligible professionals can uncontrollable circumstances and includes, as an example, that the apply for a hardship exemption eligible professional’s electronic health record vendor was unable as an option to avoid Medicare to obtain certification. Therefore, CalVet’s eligible professionals can Part B reductions until ADL apply for a hardship exemption as an option to avoid the reductions receives its electronic health until ADL receives its electronic health records certification. records certification. Although it is possible for CalVet to take reasonable steps to remain compliant with these laws and regulations, the need for such additional work contradicts the intent of the new system—to improve its operational efficiency. CalVet also intended for the system to improve its regulatory compliance; thus, the need for additional steps, as reasonable as they may be, indicates the system has not improved regulatory compliance. Project Management Failed to Promptly Recognize the Severity of System Problems In its governance plan for the project to implement the system, CalVet outlined the governance bodies that steered, controlled, and managed the project. The plan includes descriptions of the 26 California State Auditor Report 2015-121 June 2016 two main bodies—the project management team (management team) and the executive steering committee (steering committee)— responsible for overseeing the project, their roles and responsibilities, and the structure of each. The management team, led by the project manager (a role filled by CalVet’s deputy secretary for Veterans Homes Information Management), was responsible for managing the day‑to‑day operations of the project—such as the scope, schedule, and resources for the project—to ensure that the project achieved outcomes as planned. The steering committee was an advisory body that supported and provided perspective to the project executive (a role filled California Department of Veterans by CalVet’s undersecretary of Veterans Homes) Affairs’ Enterprise-Wide Veterans Home that held total decision‑making authority. Information System Project: According to the governance plan, the steering Selected Key Roles and Responsibilities committee was to meet monthly to discuss major issues and risks related to the project and to make Project Management Team decisions. The steering committee was not directly • Responsible for addressing issues related to system responsible for managing project activities, but it design, development, and implementation. Resolves was supposed to provide support and guidance to project issues as they arise. those who did. The text box shows selected key Project Manager—California Department of Veterans roles and responsibilities as described in the Affairs (CalVet) deputy secretary for Veterans Homes project’s governance plan. Information Management • Plans, directs, and oversees the daily activities of As early as June 2012, staff began reporting the project. problems with the system’s pharmacy component • Has decision‑making authority within the current to the project executive and the former project project scope to address project changes, issues, and manager, but they did not address the problems. the direction of assigned resources. The steering committee meeting minutes show • Responsible for monitoring system contractor progress. that staff began reporting these problems in • Responsible for regular scheduled project reporting June 2012 and continued to note problems to the project executive. with pharmacy implementation in the monthly • Escalates problems not resolved at the project meetings through the beginning of 2013. manager level or below. Specifically, in July 2012, the pharmacist • Leads the project management team. implementation coordinator noted that the pharmacy team had developed a problem Executive Steering Committee log and would meet weekly to discuss concerns • Responsible for reviewing, resolving, and deciding on problems escalated from the project and attempt to identify a resolution. Again, in management team. the August and October 2012 meetings, the pharmacist implementation coordinator noted Project Executive— CalVet’s undersecretary of the pharmacy team was identifying and tracking Veterans Homes problems. The minutes from the October 2012 • Responsible for the ultimate success of the project. steering committee meeting also showed that • Responsible for reviewing and resolving project the project executive had noted that the system problems that cannot be resolved at lower levels. contractor had done a good job on the pharmacy • Highest level of project review within CalVet and problems and that it had arranged regular meetings chairs the executive steering committee. with software vendors to address problems. Source: CalVet’s governance plan for its Enterprise-Wide However, in November 2012 the pharmacist Veterans Home Information System project. implementation coordinator noted that the pharmacy component was still experiencing California State Auditor Report 2015-121 27 June 2016 problems receiving orders from the order entry component in ADL Clinical. For example, in January and February 2013, he noted that one home was experiencing medication orders missing from the system and as a result, the system was not generating some orders for the pharmacy to fill. Given the severity and recurrence of the problems being reported— and the fact that they were clearly communicated in the steering committee minutes—we expected the meeting minutes to indicate that the project executive or project manager had discussed resolutions or actions to be taken to resolve the problems. However, in the minutes from February 2013, the project executive noted that the home administrators were keeping her updated on how they felt about the system and that it seemed many problems were being resolved through direct communication with the project team even though correspondence between one of the homes and headquarters showed otherwise as the problems continued. Specifically, correspondence to CalVet from the chief medical officer at the Chula Vista home in August 2013 indicated that significant problems with the pharmacy component remained and were increasing the risk of medication error. The chief medical officer noted that no verification was occurring that all medication orders were successfully transmitted, and some orders were not processed and fell into an error queue without notifying staff that the order had not been processed. In fact, CalVet eventually suspended the use of the medication order component at all eight homes. Because that project executive and the project manager during that time are no longer at CalVet, we were unable to obtain their perspectives as to why they did not address the concerns with the pharmacy component raised in the steering committee meetings. According to the manager of the project management office (PMO manager) in CalVet’s Information Services Division (ISD), who was present at the time, the steering committee meetings were only status updates to the committee and resolutions were never discussed. Further, she stated that management originally thought the problems with the order entry functionality were mainly a result of user error and that staff needed more training. CalVet’s project management did not begin to take steps to address CalVet did not begin to take steps reported problems until late 2013, having spent nearly $6 million to address reported problems until from the time staff began reporting problems to the steering late 2013, having spent nearly committee in mid‑2012. According to the final project manager, when $6 million from the time staff began he stepped into his role in November 2013, he learned of numerous reporting problems to the steering problems with the project from CalVet leadership and staff and from committee in mid-2012. his review of project communications, documents, and invoices. In the same month, a staff member from CalVet’s Veterans Homes Division visited the five homes that had implemented the system’s ADL Clinical module—Barstow, Chula Vista, Lancaster, Ventura, 28 California State Auditor Report 2015-121 June 2016 and West Los Angeles—and noted the functionality problems the staff were reporting. Notes from those visits indicate that staff were experiencing several problems, including system modules that would freeze, information showing correctly in the system but appearing inaccurate when a report was generated, and their need to use paper documents or other workarounds as backup or in lieu of the system. According to the final project manager, he, the agency information officer, and the former deputy secretary of administrative services reported critical concerns to the California Department of Technology (Technology Department) in December 2013. In December 2013 CalVet decided to CalVet then decided to pause the project to resolve critical system pause the project to resolve critical problems. The Technology Department’s director at the time system problems. Additionally, the indicated that the Technology Department directed CalVet to follow Technology Department directed the dispute resolution process outlined in the contract with the CalVet to follow the contract’s system contractor, which was to issue a cure letter, a formal written dispute resolution process with the request to provide specific deliverable documents and meet other system contractor. requirements according to the terms of its contract. In January 2014, CalVet sent the system contractor a cure letter requesting the documents for the system architecture and system configuration deliverables, which provide the organizational structure of the system and the arrangement of a computer system’s components. The letter also required the system contractor to meet the identified requirements by February 13, 2014. In early 2014, after the system contractor disputed CalVet’s claims in the cure letter, the Technology Department began facilitating discussions between CalVet and the system contractor. CalVet’s post‑implementation evaluation report (PIER) stated that the Technology Department facilitated these discussions from February through March 2014.3 Although Technology Department staff told us it facilitated a series of meetings to try and resolve the dispute between CalVet and the system contractor, the Technology Department could only provide one document summarizing outstanding problems as of January 28, 2014. That summary identifies CalVet’s and the system contractor’s position on 16 problems. The summary also provides the Technology Department’s comments on most of the problems. Although the document provides a summary of the disputed issues, it does not include any specific action plans to resolve differences or any final resolutions. Under the direction of the Technology Department, CalVet conducted an assessment of the system from May to August 2014 to determine if the system was functioning properly and was meeting its needs. The Technology Department’s former director told us that it required CalVet to conduct a series of assessments to figure out what was causing the problems. He also stated that the Technology Department 3 The PIER is a report submitted to the Technology Department in support of an agency’s request to consider the project complete and to terminate project reporting. California State Auditor Report 2015-121 29 June 2016 lent a staff member—an enterprise architect—to CalVet to perform the assessment. However, the only documentation that the Technology Department provided from its enterprise architect’s review of the system is a one‑page summary that lacks details of his assessment and a conclusion as to whether the system design was functioning properly. CalVet staff at the homes continued to experience serious functionality problems with the pharmacy component, such as errors in medication orders. As a result, in September 2014—just over a year and a half after it should have reasonably recognized the severity of the problems— CalVet suspended the use of that component of the system in all of the homes. In December 2014, after unsuccessful negotiations, CalVet, the Technology Department, and the system contractor signed a settlement agreement to terminate the contract. We asked the Technology Department whether it performed an assessment to ensure that the settlement agreement was in the State’s best interest. The former director stated that the Technology Department performed an assessment of the deliverables listed in the cure letter and found that the system contractor had submitted some of the deliverables but identified others that it had not submitted, which were used as part of the settlement with the system contractor. However, the Technology Department was The Technology Department was unable to provide us with documentation to support its claim that unable to provide documentation it conducted such an assessment of the deliverables to ensure that it conducted an assessment that the settlement amount was in the State’s best interest. The of the deliverables to ensure that former Technology Department director stated that a negotiator the settlement with the system in its Statewide Technology Procurement Division (technology contractor was in the State’s procurement division) negotiated the settlement agreement and best interest. that the negotiations were meant to determine what CalVet needed and what the system contractor was owed. Although we requested documentation from the Technology Department to demonstrate its efforts during the negotiations to ensure that the agreement reached was in the State’s best interest, it did not provide such documentation. According to the deputy director of the technology procurement division, the individual who led the negotiations and his manager have left state service and no documents on the negotiations are in the file they left behind. CalVet’s documentation indicates that the system contractor had outstanding invoices to CalVet for over $1.9 million for full implementation and maintenance of the system in the homes in Fresno, Redding, and Yountville as well as for the remaining deliverables it indicated it had already completed. However, the staff in those three homes did not receive training on and are not using the ADL Clinical module, a large portion of the system that the homes would have used to manage their electronic health records. As such, CalVet valued the portion of the system that the system contractor did implement and agreed to a lower settlement 30 California State Auditor Report 2015-121 June 2016 amount of $350,000. At that point, CalVet had spent $26.2 million implementing the system. In January 2015, CalVet and the Technology Department mutually agreed to close out the project. As of January 2015, CalVet discontinued its efforts to address the functionality problems it had identified and decided to use the functionality available from the system to the best of its ability while it explored options for a replacement solution. According to minutes of the steering committee meeting, CalVet’s acting secretary gave direction in January 2015 to cease any work to As of November 2015, CalVet restore the order entry functionality, to prepare the system’s PIER, continues to use the functionality and to initiate efforts to replace the system. As of November 2015, available from the system CalVet continues to use the functionality available from its system. while it explores options for a According to the current deputy secretary of Veterans Homes, replacement system. CalVet is in the process of looking for a replacement system. Failure to Follow Project Management Oversight Plans Contributed to CalVet’s Unsuccessful Implementation of Its System Because it did not establish and follow all of the key activities of the project management plan that the Technology Department’s California Project Management Methodology (project management methodology) requires, CalVet missed opportunities to detect deficiencies and take corrective actions earlier. The planning stage of that project management methodology describes the development of a project management plan that includes a series of 10 plans that serve as a customized, orchestrated project management workflow for IT projects. The project management methodology document states that the purpose of project management is to ensure that the delivered product, service, or result meets the customer’s requirements and is delivered on time and within budget. Further, it states that a project management methodology improves the quality of project planning, communication, and control of executive and closure processes and thus improves the quality of the deliverables. The completion of the project management plan is the main objective of the planning stage of the project management methodology, which CalVet specified that it would comply with in its FSR. Although CalVet hired a consultant to assist it in creating the series of 10 plans that make up the required project management plan, it did not develop one of those plans and four others were incomplete. Further, CalVet did not complete one of two additional required plans separate from the series of project management plans and did not consistently follow the plans it did develop. Of the six complete plans, CalVet fully followed only four. Table 4 shows the 12 required plans and indicates which ones CalVet developed and followed. California State Auditor Report 2015-121 31 June 2016 Table 4 California Department of Veterans Affairs’ Adherence to Required Project Plans IF DEVELOPED, DID WAS THE PLAN CALVET FOLLOW PURPOSE DEVELOPED? THE PLAN? Project Management Plan Scope Management Plan Define and document what is and is not within the project boundaries, and ensure that the project includes all the work required, and only the No NA work required, to complete the project successfully. Configuration Change Control Document, control, and manage changes to key project components Yes No Management Plan and deliverables throughout the project life cycle. Human Resources Management Plan Identify how and when labor needs will be met to ensure the project has sufficient staff with appropriate skill sets and experience. Partially Partially Additionally, identify and document project roles, responsibilities, and reporting relationships. Communication Management Plan Determine information and communications needs of project stakeholders: who they are, when they will need information, what Partially Partially information they will need, and how it will be given to them. Risk Management Plan Decide how to approach, plan, and execute risk management activities for the project. Establish agreed-upon basis for evaluating risks and Yes Yes ensuring that sufficient resources and time are allocated for risk management activities. Cost Management Plan Plan, estimate, and control costs so that the project can be completed Yes Yes within the approved budget. Quality Management Plan Identify which quality standards are relevant to the project and Yes Partially determine how to satisfy them. Schedule Management Plan Provide for the timely completion of the project. The plan establishes how the project schedule will be managed and controlled and includes Partially Yes estimating the duration of activities. Procurement Management Plan Determine which project needs can best be met by purchasing products or services outside the project organization, and which project needs Yes Yes can be accomplished by the project team. Contract Management Plan Document the products, services, and results requirements needed to Partially No meet the project’s objectives. Other Required Plans Organizational Change Assess stakeholders’ awareness and influence, determine any resistance Management Plan or concerns, and identify optimum communication and actions to Yes Yes be taken. Maintenance and Operations Document how the project will be transitioned to the operational Transition Plan team that will own the new system. Also includes a bridge between No NA the team executing development, the transition team, and the operations team. Sources: California Department of Technology’s Statewide Information Management Manual, interviews with project staff at the California Department of Veterans Affairs (CalVet), and California State Auditor’s review of CalVet’s project management and other required plans and related documentation for its Enterprise-Wide Veterans Home Information System. For example, CalVet did not develop a scope management plan, which defines and documents what is and is not within the project boundaries. That plan describes how a project’s scope is defined, verified, and controlled. If developed and followed, it helps a project’s decision makers determine whether the benefits of a proposed 32 California State Auditor Report 2015-121 June 2016 change in scope are worth the change in costs. This question needs One aspect of the change plan to be answered every time someone on a project requests a change procedures that CalVet did not in scope. Because CalVet did not develop this plan, it did not have follow required the change control a process to ensure that proposed changes to the project included manager to assign an analyst to only the work required to complete the project successfully and to conduct an impact analysis of remain within the boundaries of the defined scope. each change request the project team received. Additionally, as shown in Table 4, although CalVet had a complete quality management plan, it only partially followed that plan. The quality management plan identifies quality control activities as a key component of overall quality management, essential to CalVet’s quality management ensuring a successful project. CalVet’s quality management plan plan stated that it would conduct stated that the project team would conduct structured reviews to structured reviews to determine determine whether project baselines were being maintained and whether project baselines were to confirm the implementation of change requests and corrective being maintained, but it never actions, among other tasks. These reviews were to evaluate project conducted these reviews. progress against project objectives for quality, as defined in the SPR or project charter, and to determine whether shortfalls and causes of problems had been addressed, including compliance with organizational and project policies, processes, and procedures. CalVet’s PMO manager stated that CalVet never conducted these structured reviews. As a result, it did not recognize that it incorrectly reported project progress in its project status reports. As part of its oversight, in August 2012 the Technology Department identified that CalVet’s July 2012 project status report contained costs and milestones that were not aligned with the last approved governance document. As discussed later in this report, CalVet’s project status reports gave the impression that the project was still progressing according to the approved schedule when it was in reality nearly seven months behind. Had CalVet been conducting structured reviews to determine whether the project baseline was being maintained, it would have been able to identify this issue itself. Further, CalVet did not follow its configuration change control management plan (change plan). This plan describes the process the project team will follow to document, control, and manage changes to key project components and deliverables throughout the project. For example, CalVet’s change plan specified that changes to technical or functional requirements, deliverables, or the project schedule were subject to the formal processes specified in the plan. Because CalVet did not follow the change plan, it was unable to demonstrate that it properly understood the impact of different change requests on the project’s costs, scope, and timeline. For example, one aspect of the change plan procedures that CalVet did not follow required the change control manager to assign an analyst to conduct an impact analysis of each change request the project team received. According to the change California State Auditor Report 2015-121 33 June 2016 change in scope are worth the change in costs. This question needs plan, the analyst was to complete an assessment of the proposed One aspect of the change plan to be answered every time someone on a project requests a change change and the change control manager would indicate whether procedures that CalVet did not in scope. Because CalVet did not develop this plan, it did not have the impact analysis affected the project’s scope, cost, schedule, follow required the change control a process to ensure that proposed changes to the project included or resources. The change plan also specified that the analyst manager to assign an analyst to only the work required to complete the project successfully and to would document the methodology followed and the steps taken conduct an impact analysis of remain within the boundaries of the defined scope. to perform the analysis, clearly describing all assumptions and each change request the project constraints. The change plan also stated that the analyst should team received. Additionally, as shown in Table 4, although CalVet had a complete summarize all final recommendations in a succinct fashion for quality management plan, it only partially followed that plan. subsequent review by the project manager, change control board, The quality management plan identifies quality control activities and project executive, including whether the change would have an as a key component of overall quality management, essential to impact on the project’s scope, cost, schedule, or resources. CalVet’s quality management ensuring a successful project. CalVet’s quality management plan plan stated that it would conduct stated that the project team would conduct structured reviews to Despite these requirements, CalVet made a significant change structured reviews to determine determine whether project baselines were being maintained and without following its formal change process. Specifically, a whether project baselines were to confirm the implementation of change requests and corrective change request the system contractor submitted in September 2011 being maintained, but it never actions, among other tasks. These reviews were to evaluate project requested a modification to the conditions under which user conducted these reviews. progress against project objectives for quality, as defined in the acceptance testing (UAT) could proceed. UAT tests what the SPR or project charter, and to determine whether shortfalls and system contractor delivers to determine whether the deliverables causes of problems had been addressed, including compliance with conform to contract requirements. The change request asked to organizational and project policies, processes, and procedures. modify the language in the contract’s statement of work related CalVet’s PMO manager stated that CalVet never conducted to the commencement of UAT. The original language required these structured reviews. As a result, it did not recognize that it CalVet to approve the system contractor’s certification of successful incorrectly reported project progress in its project status reports. completion of system integration testing before proceeding to UAT, As part of its oversight, in August 2012 the Technology Department and the change requested that CalVet and the system contractor identified that CalVet’s July 2012 project status report contained be allowed to approve acceptable progress in system integration costs and milestones that were not aligned with the last approved testing before proceeding to UAT. The change request justification governance document. As discussed later in this report, CalVet’s states that delays in the system environment readiness and in project status reports gave the impression that the project was still the development of certain modules would prevent CalVet from progressing according to the approved schedule when it was in completing verification of all requirements before beginning UAT. reality nearly seven months behind. Had CalVet been conducting A review of this change request indicates that it was approved by structured reviews to determine whether the project baseline CalVet without completion of the impact analysis that its change was being maintained, it would have been able to identify this plan required. issue itself. Because CalVet did not conduct an impact analysis for this change Further, CalVet did not follow its configuration change control request, it cannot demonstrate that it properly considered the management plan (change plan). This plan describes the process effects this proposed change would have on the project. As a the project team will follow to document, control, and manage result of this approved change request, CalVet began UAT before changes to key project components and deliverables throughout the system integration testing was completed, according to its the project. For example, CalVet’s change plan specified that project schedule. Therefore, CalVet could not test parts of the changes to technical or functional requirements, deliverables, or system for which development was delayed and it could not ensure the project schedule were subject to the formal processes specified that what the system contractor delivered conformed to contract in the plan. Because CalVet did not follow the change plan, it was requirements. As we described in an earlier section, CalVet’s final unable to demonstrate that it properly understood the impact project manager indicated that some of the problems identified of different change requests on the project’s costs, scope, and during its assessment of the system, such as the need to enter timeline. For example, one aspect of the change plan procedures patient information more than once and problems using the that CalVet did not follow required the change control manager medication order entry functionality, were the result of incorrect to assign an analyst to conduct an impact analysis of each change system configuration by the system contractor. Had an impact request the project team received. According to the change analysis been conducted for this change request, it would have 34 California State Auditor Report 2015-121 June 2016 considered the negative impacts of beginning UAT testing before system integration testing was completed, allowing CalVet’s project management to make an informed decision. By not fully developing and following all of the plans that made up the required project management plan, CalVet limited its ability to detect deficiencies earlier in the development of the system. According to the final project manager, based on this review of project documents, the project management plans were not followed because oversight was lacking and because problems with the plans were not being escalated to the project executive. He stated that if project management plans were not being followed during the project, the individual assigned ownership for the plan would have been responsible to escalate the problem to the project manager. He further stated that if the project manager could not resolve the problem, it should have been elevated to the project executive, who could also move the problem on to the steering Our review of steering committee committee. CalVet’s governance plan states that the project meeting minutes from the executive provides oversight as needed and has responsibility to project repository did not show resolve project problems that cannot be resolved at lower levels. that problems regarding the Our review of steering committee meeting minutes from the plans were ever escalated to project repository did not show that problems regarding the plans the project executive. were ever escalated to the project executive. CalVet hired a contractor to perform both independent verification and validation (IV&V) services and independent project oversight (IPO) services for its system project. We refer to this contractor as the oversight contractor. CalVet’s contract management plan stated that the oversight contractor, in its IPO role, was responsible for monitoring the project plans and processes, assessing the project’s adherence to required project management processes and methodologies, and providing recommendations for improvement related to the project management effort. The final project manager stated that the oversight contractor should have independently been reporting on and identifying any nonadherence to project management plans and that this would have ensured that even if the problems were not being escalated up the chain of command, decision makers were being made aware of the problems. However, as we describe in the next section, the IPO reports the oversight contractor created did not identify the critical problems with the project. CalVet could not explain why the problems regarding adherence to the project management plan were not escalated to the project executive. Had the problems with the plans been so escalated or had the oversight contractor detected these problems, many of the problems experienced with the system project, such as incomplete project management plans, could have been addressed at an early stage. California State Auditor Report 2015-121 35 June 2016 Inadequate Independent Oversight of the System Project Left Stakeholders Without the Information Necessary to Ensure a Successful Implementation The services that the oversight contractor provided were We noted missing critical inadequate. Specifically, we noted missing critical deliverables deliverables we expected the we expected it would have provided through its contracted oversight contractor would IV&V services. Additionally, our IT expert stated that in his have provided through its opinion, the IPO was largely ineffective. As discussed in the contracted IV&V services. Introduction, IV&V services provide a client with technically proficient “eyes and ears” to oversee a system vendor while an IT system is being developed and implemented, and these services also provide an early warning of process and technical discrepancies and problems that might not otherwise be detected until late in the project life cycle. IPO services provide an independent review and analysis of project management practices to determine whether the project is being well managed. CalVet’s oversight contractor provided both IPO and IV&V services. CalVet’s contract required the oversight contractor to perform tasks and activities in accordance with applicable Institute of Electrical and Electronic Engineers (IEEE) standards, which are industry standards also recommended in state policy. According to our IT expert, the relevant IEEE standard focusing on system, software, and hardware verification and validation processes does not require an organization that is performing traditional technical IV&V services to be organizationally independent from one that is performing optional project management oversight support. However, he stated that separation of IV&V and IPO duties is important and provides a number of advantages to the project and to the State. For example, IPO should assess whether appropriate IV&V services have been procured for the project. It is also within the purview of IPO activities to assess whether the IV&V contractor is doing its job well, such as whether IV&V services are effective and timely. If a contractor is performing both activities, there is also a potential conflict of interest because IPO might recommend that more of the IV&V services it is providing be used. Our IT expert reviewed the IPO and IV&V reports the oversight contractor provided to CalVet and noted that he did not find evidence that the oversight contractor prepared certain key IV&V reports that the contract required. These reports should have provided insights into the technical quality of the work being performed. Some of the more critical missing IV&V deliverables included the following: • Requirements traceability reports monitoring the tracing of project requirements throughout the project life cycle to ensure that the system meets specified requirements. 36 California State Auditor Report 2015-121 June 2016 • System and software verification report assessing whether the work products satisfied the conditions established at the start of the development phase and fully addressed the requirements. • System and acceptance report and testing report assessing whether the delivered products had been thoroughly tested to confirm that they were functional and met contractual requirements. According to our IT expert, these IV&V reports are all critical. However, CalVet’s final project manager stated that the oversight contractor did not provide these deliverables to CalVet and that poor contract management by CalVet was the reason it did not identify the oversight contractor’s failure to submit the reports and require that it do so. By failing to obtain the technical assessments from these reports, CalVet neglected its responsibility to ensure that it was providing stakeholders with information about whether its project to implement a new system fulfilled all technical requirements and was functioning as expected. In addition to missing critical IV&V deliverables, our IT expert concluded that the IPO was largely ineffective. He stated that IPO is there to make sure that the project manager is accurately reporting project status and that IV&V is providing independent technical analysis. His review of IPO and IV&V reports found almost no technical assessment; in addition, although the reports indicated what happened, they offered minimal analysis, with no explanation of root causes or trends. For example, one IV&V report referred to a checklist for delivered training materials indicating that there were problems, but it did not identify the problems and stated that the IV&V review could not be completed. In this case, according to our IT expert, IPO should have raised concerns that IV&V could not complete its work. Similarly, effective IPO should have raised concerns that IV&V was not managing requirements traceability, as mentioned earlier. Although the oversight contractor’s IPO Although the oversight contractor’s reports should have identified these types of deficiencies in the IPO reports should have IV&V work—performed by the same contractor—they did not. identified deficiencies in the IV&V Because the IPO reports did not identify that IV&V was failing work—performed by the same to perform critical work, neither CalVet nor the Technology contractor—they did not. Department received an accurate assessment of whether the processes the system contractor was using were effective and whether the system reflected the agreed‑upon quality and solution. The IPO reports also missed critical information about a variation from the approved project schedule, a variation later identified by the Technology Department. State policy requires that IPO reports assess the expected completion of tasks and milestones compared to the approved project schedule contained within the most recent SPR. However, in its August 2012 IPO report, the oversight California State Auditor Report 2015-121 37 June 2016 contractor acknowledged that previous reports had looked at past project milestone dates and their actual completion dates rather than dates in the most recent approved SPR as it should have to determine whether the project was on schedule. Further, the IPO report stated that beginning with the August 2012 report, future tasks and milestones would be compared to the most recent project schedule approved in the SPR to determine whether the project Because the oversight contractor was on schedule. Because the oversight contractor did not use the did not use the correct dates to correct dates to measure project progress, the variation from measure project progress, it gave the approved schedule was not identified, giving the impression the impression that the project was that the project was still progressing according to the approved still progressing according to the schedule when it was not. approved schedule when it was not. CalVet’s final project manager, who identified the variance while he was employed at the Technology Department, informed us he met with CalVet’s former project manager and the oversight contractor to discuss the variance. As a result, the oversight contractor adjusted its tracking in the August 2012 IPO report to correctly reflect the schedule contained in the approved SPR. In that August IPO report, the oversight contractor reflected that the project was nearly seven months behind the approved schedule. However, if the oversight contractor had been using the correct dates to measure the project’s progress, it would have better informed project stakeholders on the true status of the project’s progress. Further, both CalVet and the project’s decision makers would have been aware earlier that the project was behind schedule and could have taken steps to address variances. CalVet’s final project manager could not explain why his predecessor or the oversight contractor did not use the approved schedule to measure performance, but in his opinion, it was likely that they were familiar with the requirements. As we discussed earlier, our IT expert believes that by using the same contractor to perform both IPO and IV&V functions, CalVet did not ensure it had effective oversight for the project. The deputy director of the Technology Department’s IT Project Oversight Division stated that starting in July 2013, the Technology Department began assigning its own staff for IPO on reportable IT projects it deemed to be of medium or high complexity and that agencies would have to obtain explicit approval to contract for IPO services; this approval would only be granted if the Technology Department did not have enough staff to provide the services itself. She also stated that the Technology Department does not have a policy requiring agencies to obtain IPO and IV&V services from separate contractors; however, it is the department’s expectation that agencies receiving permission to use outside IPO services will obtain IV&V from separate contractors and she agreed this would be a good policy to formalize. 38 California State Auditor Report 2015-121 June 2016 The Technology Department Did Not Adequately Fulfill Its Responsibilities in the Oversight of CalVet’s System The Technology Department’s review of IPO reports for CalVet’s system project was inadequate, and as a result, it did not identify critical concerns and take timely action. The Technology The Technology Department Department received oversight authority after CalVet’s system received oversight authority after project was already under way, and it made the decision to CalVet’s system project was already allow CalVet to continue with its contracted oversight rather under way, and it made the decision than perform the IPO role itself. According to the deputy to allow CalVet to continue with its director of the Technology Department’s IT Oversight Division, contracted oversight rather than because its oversight consisted of a review of project reports perform the IPO role itself. that CalVet’s oversight contractor was producing, it would have had no way of consistently identifying critical concerns related to the project if no significant problems were reported in these documents, as was the case with the CalVet project. However, our IT expert’s assessment of the IPO reports indicated that the fact that these reports did not contain critical information about the project or offer analysis of project progress or vendor performance should have triggered closer review and inspection. State law transferred IT oversight authority from Finance to the Technology Department in January 2008.4 The Technology Department’s Statewide Information Management Manual defines project oversight as independent review and analysis of specific project activities and documentation to determine if the project is on track to be completed within the estimated schedule and cost and if it will provide the functionality the sponsoring entity requires. The oversight responsibilities include requiring agencies to provide the Technology Department with periodic reporting that describes the degree to which an IT project is within approved scope, cost, and schedule; project issues, risks, and corresponding mitigation efforts; and the current estimated schedule and costs for project completion. According to the Technology Department’s former director, before December 2013, its primary role in the oversight of CalVet’s project was reviewing IPO and IV&V reports that CalVet’s oversight contractor prepared. The FSR for CalVet’s system, which was approved in 2007 by Finance—before state law transferred oversight responsibility for IT projects to the Technology Department— specified that CalVet would retain a contractor to perform IPO and IV&V for the overall project. In December 2007, CalVet entered into a contract, totaling just over $1.8 million, for these services. Although the former director added that he had regular 4 Until July 2013, the California Department of Technology was known as the California Technology Agency, and before that it was the Office of the Chief Information Officer. California State Auditor Report 2015-121 39 June 2016 verbal briefings—referred to as portfolio reviews—with both CalVet and the oversight contractor to discuss the status of the project, the frequency of these meetings could vary, ranging from monthly to quarterly. He told us no documentation exists from these meetings. Although we previously described one instance in August 2012 when the Technology Department identified concerns with information in the IPO report and took action to address its concerns by meeting with the oversight contractor, overall we found that the Technology Department’s oversight of the IPO reports for CalVet’s system project was inadequate. Specifically, The Technology Department’s its oversight was not rigorous enough to detect concerns reported oversight was not rigorous enough in the IPO reports or to question why the reports the oversight to detect concerns reported in the contractor prepared did not contain critical information about IPO reports or to question why the the project. As previously noted, our IT expert stated that the reports the oversight contractor IPO reports the oversight contractor prepared did not offer an prepared did not contain critical analysis of project progress or vendor performance and that information about the project. should have triggered the Technology Department’s closer review and inspection. For example, the IPO report for February 2011 indicated that the project was using about half of the allocated resources for the fiscal year but did not offer an analysis of the cause or the implications this would have for the project. The same IPO report identified a risk that the system contractor might not be able to perform or meet the requirements according to its bid for the project; however, the IPO report did not include any significant analysis, thus missing an opportunity for IPO to provide an explanation of what it was seeing. In another example, our IT expert noted that in comparing the IPO reports for March 2011 and April 2011, he found a two‑month delay in the projected end date for the design, configure and development phase; however, there was no comment explaining the sudden two‑month slippage in the schedule. Our IT expert also noted that in the IPO report for June 2011, the rating in “quality for architecture/system performance” was suddenly listed as inadequately defined even though in previous reports the notation was not applicable. He stated that this is a huge red flag, and he noted that the corresponding analysis said that the current draft of the system architecture document did not adequately and accurately reflect the planned system. However, no further analysis or context was provided about the impact of this change. Our IT expert stated that the lack of analysis and general comments should have been a red flag to anyone familiar with the IPO report format. He explained that there is a difference between tracking the project—what is happening, and oversight — reporting what is happening and providing context for people to understand what is important, what is not, and the implication 40 California State Auditor Report 2015-121 June 2016 of what is being observed. He stated that CalVet’s project IPO reports all fell into the tracking category as they rarely offered analysis or perspective on what was happening; they simply reported schedule slips and risks. He added that any review of these IPO reports should have resulted in the Technology Department’s further inquiry of the oversight contractor and perhaps coaching to improve the quality of the reports. Although the IPO reports generally lacked this analysis of events and risks reported, we found no evidence that the Technology Department raised concerns about the quality of the reports with either the oversight contractor or with CalVet. The Technology Department In fact, the Technology Department could not demonstrate that could not demonstrate that it it had even reviewed IPO reports before August 2012, when, had reviewed IPO reports before as discussed earlier, it raised an issue about the information August 2012. At that point, in the oversight contractor’s report. At that point, CalVet had CalVet had spent $15.8 million on spent $15.8 million on the project. Additionally, the Technology the project. Department could not locate any IV&V reports for CalVet’s project. Had the Technology Department’s oversight of the system been more rigorous, it should have identified and raised concerns about the deficiencies of the IPO reports and missing IV&V reports, which likely would have resulted in the oversight contractor informing CalVet earlier of problems with the system’s implementation. The deputy director of the Technology Department’s IT Oversight Division stated that division staff were reviewing IPO and IV&V reports from CalVet’s oversight contractor, but because the reports were not indicating critical errors with the project, the Technology Department did not raise any concerns. However, we believe it should have. As we noted earlier, our IT expert stated that the lack of critical information about the project or analysis of project progress or system contractor performance in the IPO reports should have been sufficient cause for concern. According to that deputy director, starting in July 2013 the Technology Department began assigning its own staff to perform project oversight. However, the former director told us that it devoted its limited staff resources to other troubled projects and only provided a portfolio review of CalVet’s relatively small system project when compared to those other troubled projects going on at the same time. Further, the deputy director indicated that because the Technology Department had an insufficient number of qualified staff to replace vendors on existing projects, CalVet was allowed to continue to use its oversight contractor. She stated that the Technology Department’s practice as of July 2013 has been California State Auditor Report 2015-121 41 June 2016 that any IT projects deemed to be of medium or high complexity As of July 2013, any IT projects must obtain explicit approval from the Technology Department deemed to be of medium or high to contract for their own IPO services and approval is granted complexity must obtain explicit only if the Technology Department does not have enough staff to approval from the Technology provide the IPO services itself. Department to contract for their own IPO services. In a previous audit report issued by our office, we identified concerns with the Technology Department’s limited resources and with its poor documentation of its oversight efforts on IT projects.5 The problems that we noted in that report were occurring at the time the Technology Department was providing oversight to CalVet’s implementation of its system. Specifically, our previous report noted that the Technology Department’s oversight and consulting division hired additional IPO analysts between fiscal years 2011–12 and 2013–14; however, we found it was unclear whether the division had enough positions at that time to effectively oversee the State’s IT projects. As a result, we recommended that the Technology Department conduct a workload assessment to determine the level of staffing and expertise required for the projects it oversees and using that workload assessment, it should make decisions to assign its staff to oversee each IT project. In March 2016, the Technology Department indicated in its one‑year response to our earlier report that it had established processes to capture and evaluate workload information. If it follows these new processes, the Technology Department can better identify the level of oversight it can provide to state IT projects. That previous audit report also noted that although the Technology Department is generally able to hire personnel to fill its IPO analyst positions, the job classification it uses may not attract applicants with the most relevant skills and experience required for IT project oversight. To ensure that it attracts and retains employees with appropriate experience and qualifications to perform IT project oversight, we recommended that the department continue its efforts to gain approval to use the project manager classification for its IPO analyst role. In its one‑year response in March 2016, the Technology Department reported that it determined in October 2015 that the project manager classification was not suitable for use by the IT project oversight division. The Technology Department also indicated that the California Department of Human Resources (CalHR) is leading a significant effort to modernize a variety of state classifications, including IT classifications and the Technology Department is an active participant in CalHR’s reform initiative. Therefore, the Technology Department stated that until better options become available, it 5 High Risk Update—California Department of Technology: Lack of Guidance, Potentially Conflicting Roles, and Staffing Issues Continue to Make Oversight of State Information Technology Projects High Risk, Report 2014-602, March 2015. 42 California State Auditor Report 2015-121 June 2016 will continue to use the current classification—data processing manager—to fill its project oversight role. Nevertheless, we believe that by fully implementing the recommendations from our prior report and fully implementing the additional recommendations we present in this report, the Technology Department can ensure that it fulfills its responsibility of providing oversight to state IT projects. CalVet Did Not Maintain Adequate Documentation for Its Selection of the System Contractor and for Some Key Deliverables Because it did not maintain required documentation for selection of the contractor to implement its system, CalVet cannot demonstrate it complied with contracting requirements, and this lack of documentation raises questions about the prudence of its decisions. As of July 2010, the State Contracting Manual requires state entities to use an RFP for acquisitions of IT projects exceeding $1 million. State law in 2010 required agencies to award the contract to the bidder that achieves the highest score on its proposal. Regarding record retention, the State Contracting Manual states that departments are responsible for maintaining records in sufficient detail to allow anyone to review that documentation and understand how the procurement was requested, conducted, awarded, and administered. The State Contracting Manual further states that record retention varies depending on document type and can vary by department, depending on its internal retention schedule. Generally, these procurement documents should be retained for seven years from the end of the fiscal year in which the contract amount is liquidated—either through completion or termination. CalVet could not provide some of the documentation it should have maintained related to its award of the contract for its system. CalVet’s contract records show that CalVet’s contract records show that in June 2010 it received in June 2010 it received proposals proposals from seven vendors in response to its RFP. However, from seven vendors in response to according to its final evaluation and selection report (evaluation its RFP. However, it reviewed only report), it reviewed six proposals and CalVet’s selection committee six proposals and deemed five of deemed five of them nonresponsive because of material deviations. them nonresponsive because For example, according to the evaluation report, the selection of material deviations. committee found that the documents submitted for one proposal did not substantiate that the project manager met the experience requirements outlined in the RFP. The evaluation report noted that other proposals did not provide supporting references or documentation verifying that the vendor could carry out the RFP requirements. In addition, one of the seven proposals included on the receipt log was not included in the evaluation report as having been evaluated. When we asked CalVet why it was not included, the PMO manager could not answer the question as she was not part of the procurement; further, she stated that there is no one California State Auditor Report 2015-121 43 June 2016 remaining at CalVet that had worked on the procurement. CalVet’s final evaluation report indicated that CalVet scored and ultimately awarded the contract to the only responsive bidder. However, CalVet could not provide CalVet could not provide the proposals for six of the seven the proposals for six of the responding vendors, nor could it provide its evaluation documents seven responding vendors, nor for three proposals, including the winning proposal. When CalVet could it provide its evaluation does not ensure that it maintains this documentation, it cannot documents for three proposals, demonstrate that it made a prudent decision or that it complied including the winning proposal. with state contracting requirements when selecting the vendor for its project. CalVet could not adequately explain why it did not follow contracting requirements. The PMO manager agreed that poor record keeping by the previous contract manager contributed to the difficulty in locating the required contract documentation. Further, she stated that CalVet does not have a policy to periodically verify whether its staff are following contract documentation requirements. Although CalVet’s policy states that each division must provide a records retention schedule approval form annually to its Office of Procurement and Contracting (OPC), according to an OPC manager, OPC does not have a record retention schedule from CalVet’s ISD, the division responsible for CalVet’s IT contracts. Additionally, the final project manager could not provide a records retention schedule for ISD. CalVet also could not demonstrate that it received sufficient documentation for some key system deliverables even though it approved payments for them. Specifically, CalVet’s project files indicate that it accepted and approved payments totaling $733,000 for three key deliverables—system design, system configuration, and UAT—even though it could not provide adequate documentation of receiving these final deliverables. The State Contracting Manual requires state agencies to verify that the goods and services they receive are satisfactory before approving payment for them. Further, CalVet’s contract for implementation of the system required that the system contractor submit all deliverables to CalVet for acceptance and then approval. The system architecture design and configuration deliverables provide the organizational structure of the system and the arrangement of the computer system’s components. Although the contract manager at the time accepted the system configuration and architect design deliverables and approved the payment of $104,500, CalVet could not provide the final documentation. Instead, it could only provide us with a draft version of those documents. CalVet’s final project manager confirmed that no one has been able to provide the final documents for these two deliverables. He stated that many of the system problems the homes encountered were generally related to deficiencies 44 California State Auditor Report 2015-121 June 2016 in the system’s configuration, design, and training. For example, as discussed earlier, some homes indicated that staff have to enter the same resident information into multiple modules within the system, which the final project manager indicates is a result of poor system configuration. Communication between CalVet and executives at ADL—the system’s software provider—indicated that based on its work with CalVet to resolve outstanding trouble tickets over three months in early 2015, ADL was concerned with the system’s configuration during installation and with the instructions to CalVet staff on the proper setup and implementation of the software. In June 2014, in response to CalVet’s system assessment of functionality problems, the system contractor acknowledged there In June 2014, the system contractor were more than 500 outstanding items of which about 240 would acknowledged there were more require further system configuration and additional training of than 500 outstanding items subject matter experts. Because the contract manager at the time of which 240 would require CalVet accepted the system design and configuration deliverables is further system configuration and also no longer at CalVet, it is unclear why she signed the acceptance additional training of subject letter and approved payment for those deliverables as CalVet does matter experts. not have proof it received the final deliverables. The UAT deliverable documents the testing that verifies that the system meets contract requirements and performs at a satisfactory level. For deliverables related to UAT, the contract required the system contractor, among other things, to develop a UAT report that included a description of the defects CalVet identified during UAT, the business processes and system functions or interfaces impacted by each defect that could not be resolved, and a corrective action plan for defects that could not be resolved. CalVet’s final project manager indicated that after he started at CalVet in November 2013, he looked for the UAT documentation but there were no tangible test results or tracking of specific requirements. He stated that for UAT, test cases are usually developed that include the test steps and the success criteria used to evaluate whether the given test case or scenario passed or failed. He added that there should be documentation for each test case completed by each tester. Although the contract required specifics on the testing and results, the deliverable from the contractor for the UAT that CalVet provided to us included only a summary of the tests completed. That summary report did not include information about the specific requirements tested, who conducted the tests, the outcome of each test, the defects identified during the testing, or corrective action plans for each defect that could not be resolved. CalVet’s test manager confirmed that CalVet does not have the completed scripts that indicate who conducted the tests and what the outcomes were. As a result, CalVet cannot demonstrate that UAT was completed even though it approved payment of three invoices totaling $628,527 for UAT. In fact, CalVet’s oversight California State Auditor Report 2015-121 45 June 2016 contractor reported in February 2014 that during UAT, CalVet had not accurately captured the results from the testers, and as a result, there was no empirical evidence to assess to determine whether the developed system met CalVet’s business needs. Because CalVet accepted deliverables and approved payment without sufficient documentation to demonstrate that the system contractor had satisfactorily completed them, CalVet failed in its contract management responsibilities to verify that the system it paid for was properly designed and configured, met contract requirements, and performed satisfactorily. A number of factors contributed to CalVet’s failure to assess the adequacy of UAT deliverables. The test manager stated that because she did not have experience as a test manager in 2011, she could not determine whether the documentation the contractor submitted for testing was sufficient. She further explained that when she voiced her concern that she had never been a test manager to the project manager at the time, the previous contract manager, and the previous agency information officer, they told her it was okay and that they just needed someone to sign off on the testing. She further indicated that CalVet did not complete UAT before the system contractor implemented the system in the homes. Because CalVet did not complete UAT, it missed its opportunity to identify functionality issues before implementing the system in the homes. The test manager indicated that because UAT took longer than expected, the project team received outside pressure from the former project executive, CalVet’s former secretary, and the Technology Department to move forward with implementation to keep the project on schedule. Therefore, CalVet did not complete UAT before implementing the system. Additionally, despite the lack of evidence for certain key IV&V Despite the lack of evidence for deliverables, CalVet approved invoices from its oversight contractor certain key IV&V deliverables, for those deliverables. As we described earlier, our IT expert CalVet approved invoices from reviewed IV&V deliverables and did not find evidence that the its oversight contractor for those oversight contractor prepared requirements traceability matrix deliverables. reports—reports monitoring the tracing of project requirements throughout the project life cycle to ensure that the system meets specified contract requirements. In our review of invoices from the oversight contractor, we identified invoices totaling just over $12,000 for monitoring requirements traceability, but the contract records did not contain any evidence of project manager review or approval of those invoices. Further, CalVet was unable to provide us with documentation that its oversight contractor actually completed the review of requirements traceability. Therefore, it is unclear why the previous contract manager approved payment of the oversight contractor’s invoices. Additionally, according to a 2014 report, the oversight contractor stated that CalVet did not accurately capture the results from the UAT testers and as a result, 46 California State Auditor Report 2015-121 June 2016 there is no empirical evidence to assess whether the developed solution met the users’ business needs. The final project manager cited poor contract management as the reason why CalVet did not identify deficiencies and require the oversight contractor to fulfill the deliverables. However, he indicated that the IV&V contract CalVet signed with its oversight contractor in September 2014 now requires the oversight contractor to submit deliverable acceptance documents signed by the system’s project manager and contract manager along with invoices to ensure that only accepted deliverables are paid. CalVet Identified Some Lessons Learned but Rarely Used Them to Improve Future Phases of Implementation Although CalVet conducted lessons‑learned sessions at points throughout the project from initial procurement through implementation at the Chula Vista home, it generally cannot demonstrate that it used those lessons learned to make improvements during later implementation phases. Additionally, CalVet did not conduct sessions to identify lessons learned during the final implementation phase at the Fresno, Redding, and Yountville homes, preventing it from gaining the full benefit of lessons learned: the ability during future similar projects to duplicate the successes and avoid the shortfalls experienced on earlier projects. CalVet’s implementation plan stated that its PMO would lead lessons‑learned sessions throughout the implementation of the system, to document any lessons that could be usefully applied to implementation at the next site. CalVet implemented the system in three phases: the pilot phase, which included headquarters and the CalVet conducted 14 lessons-learned homes in Barstow, West Los Angeles, Lancaster, and Ventura; sessions from its initial procurement the second phase, which included Chula Vista; and the final phase, efforts in March 2009 through which included Fresno, Redding, and Yountville. Documentation implementation at Chula Vista summarizing the lessons‑learned sessions held shows that CalVet in August 2013, but it generally conducted 14 of these sessions, from its initial procurement cannot demonstrate that it used efforts in March 2009 through implementation at Chula Vista in those lessons learned to make August 2013. According to CalVet’s summary of these sessions, they improvements during later focused on what went well, what did not go as planned, and areas implementation phases. for improvement. Given that CalVet implemented the system in phases, it could have used the lessons it learned during the pilot implementation to improve the Chula Vista implementation, and it could have used the lessons learned from both the pilot implementation and at Chula Vista to improve the final phase of its implementation. Specifically, CalVet’s summary from its lessons‑learned session after the pilot implementation stated that staff indicated that there California State Auditor Report 2015-121 47 June 2016 was a lack of on‑site support during evening shifts during the implementation. Following the implementation at Chula Vista, CalVet’s lessons‑learned session notes again identify that evening support was not consistent. Additionally, after the pilot implementation, CalVet’s lessons‑learned summary identified concerns with training and noted as an improvement going forward the need to maintain binders with the training materials and processes in a central location. However, CalVet’s summary from its lessons‑learned session for the implementation at the home in Chula Vista again notes the need to maintain materials in a central location. These examples demonstrate that CalVet repeated mistakes and therefore did not effectively use the lessons it learned. In fact, CalVet’s PMO manager identified only one change that CalVet did make as a result of lessons learned. Specifically, CalVet increased the time allotted for training at Chula Vista. She stated that CalVet did not go back and document how it implemented changes based on lessons learned, so she is unsure whether any other lessons learned were implemented. CalVet did not conduct lessons‑learned sessions following CalVet did not conduct the final implementation phase at the Fresno, Redding, and lessons-learned sessions following Yountville homes, limiting its ability to identify problems that the final implementation phase, it could correct or avoid and successes it could repeat in future limiting its ability to identify similar projects. According to the PMO manager, CalVet did not problems that it could correct or conduct those lessons‑learned sessions because the system was avoid and successes it could repeat not fully implemented at those three homes. However, CalVet’s in future similar projects. implementation plan specified that the PMO would conduct lessons‑learned sessions throughout the implementation, and since the homes still implemented modules of the system, we believe there were opportunities to document lessons learned for future projects. When we pointed out these missed opportunities to the PMO manager, she did not remember whether project management considered that lessons could be learned from its partial implementation at the three homes. However, she stated that it is reasonable to think so. According to its PIER, CalVet captured final lessons learned during interviews with key management and executive staff. The report states that participants were given the opportunity to look back and identify the most significant problems and successes of the project that would benefit future CalVet project efforts. CalVet summarized these final lessons learned into eight categories, such as scope management, requirements or change management, contract management, and project governance, and it described findings and recommendations for each category. For example, one finding noted that CalVet did not actively manage the system contractor, resulting in the system contractor not always delivering services according to its contract, and agreed‑upon deliverables were not reflected in contract amendments. The corresponding 48 California State Auditor Report 2015-121 June 2016 recommendation was to make sure the right resource is assigned to not only manage the contract but also to enforce the terms of the contract. In another example, CalVet’s finding stated that the requirements were loosely managed throughout the project life cycle, referring to its early identification of too many requirements, the removal of many requirements in the subsequent RFP, the significant gap between the two RFPs, and the large number of change requests. The related recommendation states that CalVet should establish a requirements management plan that defines the process of scheduling, coordinating, and documenting the requirements engineering activities including elicitation, analysis, specification, and verification. CalVet’s incorporation of these final lessons learned into its preparation for its planned future project is critical to ensure that the next system implementation is successful. Recommendations CalVet To ensure that its project management of IT projects promptly identifies potential problems and develops resolutions, by September 2016 CalVet should define the project executive’s and project manager’s responsibilities to ensure that the individuals who fill those positions take an active role in each project. To ensure that it adequately identifies and monitors problems in its future IT projects, by September 2016 CalVet should establish a formal process for its project executive to verify that the project team prepares all of the required project management and other required plans. This formal process should also include a process to periodically verify that the project team is adhering to all these plans. To ensure accountability and independence between the provision of IPO and IV&V services on future IT projects, by September 2016 CalVet should establish a policy requiring it to use separate contractors for IPO and IV&V services when IPO services are not provided directly by the Technology Department. To ensure that it complies with state contracting laws and can demonstrate the basis for its decisions when awarding contracts, by September 2016 CalVet should establish a process to periodically verify that its staff follow state contracting requirements and maintain all required contract documentation. California State Auditor Report 2015-121 49 June 2016 To ensure it maintains all documentation related to its IT contracts, CalVet should, by September 2016, establish a process to verify that all divisions comply with its policy requiring each division to submit a records retention schedule to its Office of Procurement and Contracting. To ensure that it only accepts deliverables and approves payment for deliverables that are complete and meet contract requirements, by September 2016 CalVet should establish processes to do the following: • Ensure that the project executive verifies that individuals assigned to project roles are adequately qualified and experienced. • Verify and maintain documentation of receipt of all contract deliverables before approving payment. • Strengthen its contract management on all future projects by requiring the project manager to sign off on invoices along with the contract manager before approving payment. To ensure that it maximizes its opportunity to successfully implement future IT projects, including its plan to replace its current system, CalVet should, by September 2016, establish a formal process to do the following: • Document the changes it makes as a result of the lessons‑learned sessions it conducts. • Verify that its staff conducts lessons‑learned sessions for all key phases of the next project. • Incorporate the recommendations identified in its PIER. Technology Department To ensure that it can demonstrate that it is acting in the best interest of the State, the Technology Department should, by December 2016, create a formal process to summarize its involvement and document key actions taken and decisions reached during agencies’ contract disputes and negotiations for the termination of a contract and maintain those documents according to its record retention schedule. 50 California State Auditor Report 2015-121 June 2016 To ensure accountability and independence between the provision of IPO and IV&V services, the Technology Department should, by December 2016, establish a written policy requiring departments that request and receive approval to contract for IPO services to use a different contractor than the one providing IV&V services. Although the Technology Department indicated that its intent is not to outsource its statutory responsibility for IPO, in any instances where its staff conduct a portfolio review of a project’s IPO, the Technology Department should, by December 2016, establish a process for its review of documents created by the agency’s IPO contractor that includes verifying whether these reports include critical analysis of project progress and vendor performance so it can intervene when necessary. We conducted this audit under the authority vested in the California State Auditor by Section 8543 et seq. of the California Government Code and according to generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives specified in the Scope and Methodology section of the report. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Respectfully submitted, ELAINE M. HOWLE, CPA State Auditor Date: June 16, 2016 Staff: Tammy Lozano, CPA, CGFM, Audit Principal Richard D. Power, MBA, MPP Fahad Ali, CFE Brian D. Boone, CIA, CFE Karen Jenks, MBA IT Expert: Catalysis Group Legal Counsel: J. Christopher Dawson, Senior Staff Counsel For questions regarding the contents of this report, please contact Margarita Fernández, Chief of Public Affairs, at 916.445.0255. California State Auditor Report 2015-121 51 June 2016 Appendix A COMPARISON OF THE CALIFORNIA DEPARTMENT OF VETERANS AFFAIRS’ ESTIMATED AND ACTUAL COSTS FOR THE ENTERPRISE‑WIDE VETERANS HOME INFORMATION SYSTEM The Joint Legislative Audit Committee asked us to determine the estimated and actual costs, as well as the estimated and actual timelines, for the California Department of Veterans Affairs’ (CalVet) Enterprise‑Wide Veterans Home Information System (system) project. Table A.1 on the following page presents CalVet’s estimated and actual costs for implementing this system. As shown in Table A.1, CalVet initially estimated the costs to implement the system in its feasibility study report (FSR) at just under $34 million. CalVet increased its estimate as the project progressed. However, the actual costs of implementation, as of the completion of its post‑implementation evaluation report in June 2015, were lower than its estimate, at roughly $27.9 million. Although the cost of the project was nearly $9 million less than the last approved special project report (SPR), CalVet implemented much less functionality than it initially planned for and that was approved in its FSR and subsequent SPRs. Specifically, as discussed on page 22, because of the functionality problems that staff encountered during implementation of the system at the first few veterans homes, CalVet indicated that staff either did not receive training or received training on a minimal number of components for the ADL Clinical module functionalities at the Fresno, Redding, and Yountville homes; the ADL Clinical module is a main module of the system through which the homes manage electronic health records. According to their administrators, two of the homes continue to use paper processes to document resident medical information, while a third home uses the previous system, Meditech. Therefore, CalVet spent $27.9 million for implementation of a system that has not been fully implemented at all of the homes, and it did not fulfill one of its primary goals for the project—a fully integrated system of care in which the system used at any home could seamlessly operate with the other homes and in which veterans would receive consistent care throughout the homes—and therefore, does not meet the needs of the veterans homes. 52 California State Auditor Report 2015-121 June 2016 Table A.1 Estimated and Actual Costs for Implementation of the California Department of Veterans Affairs’ Enterprise-Wide Veterans Home Information System DOCUMENT PROJECT DOCUMENT DATE ESTIMATED COSTS ESTIMATED COMPLETION DATE Feasibility Study Report December 2006 $33,982,315 December 2010 Special Project Report #1 August 2008 35,469,331 June 2012 Special Project Report #2 November 2010 37,311,098 October 2013 Special Project Report #3 October 2012 36,744,638 April 2014 ACTUAL COSTS ACTUAL COMPLETION DATE California State Accounting and Reporting System NA 27,914,733* June 2015† Reports (CalSTARS) Sources: Documents referenced in the table column titled Project Document and the respective approval letters, if applicable, for these documents. NA = Not applicable. * Costs recorded in CalSTARS are from July 1, 2007, through June 30, 2015. Costs include CalVet’s and the system contractor’s settlement agreement amount of $350,000. † In June 2015 the California Department of Veterans Affairs (CalVet) submitted its post-implementation evaluation report to the California Department of Technology in support of its request to consider the project complete and to terminate project reporting. Comparison of CalVet’s Estimated and Actual Timelines to Implement the System Table A.2 presents CalVet’s estimated and actual dates for completing major project milestones to implement the system. CalVet initially estimated the project schedule in the FSR and updated the estimated dates in the subsequent SPRs. CalVet extended the overall project schedule in each of the SPRs. Specifically, although CalVet originally expected to complete the system (final acceptance) by December 2010, the system was still not complete in January 2015, when, as we describe on page 30, CalVet and the California Department of Technology agreed to close the project. Several factors contributed to project delays. For instance, CalVet’s initial request for proposal (RFP) was delayed because of changes that were made to the project scope, such as the decision to implement the system at the Fresno and Redding homes. In addition, CalVet issued a second RFP with updated system requirements after its initial RFP failed to generate any responsive proposals. CalVet also delayed the implementation of the system at the Chula Vista home to provide additional support to some of the pilot homes that were struggling with certain complex functionalities of the system. Finally, CalVet postponed the implementation of the system at the Fresno and Redding homes because of delays in the opening of those homes. California State Auditor Report 2015-121 53 June 2016 Table A.2 California Department of Veterans Affairs’ Enterprise-Wide Veterans Home Information System Timeline Comparison ESTIMATED COMPLETION DATES FEASIBILITY STUDY SPECIAL PROJECT SPECIAL PROJECT SPECIAL PROJECT REPORT REPORT #1 REPORT #2 REPORT #3 ACTUAL MAJOR MILESTONES DECEMBER 2006 AUGUST 2008 NOVEMBER 2010 OCTOBER 2012 COMPLETION DATES Solution procurement March 2008 June 2009 December 2010 January 2011 December 2010* User acceptance testing March 2009 April 2010 September 2011 June 2012 June 2013* Implementation at headquarters NI NI January 2012† August 2012† May 2012* Implementation at the Veterans Homes: Barstow September 2010 March 2012 January 2012† West Los Angeles April 2010 August 2012 August 2012† May 2012* Lancaster April 2009 June 2010† May 2012 Ventura April 2009 July 2012 Chula Vista July 2010 January 2012 March 2012 March 2013 June 2013‡ Redding NI April 2011 July 2013 November 2013 Fresno NI July 2011 September 2013 November 2013 November 2013§ Yountville December 2010 June 2012 April 2013 March 2014 Final acceptance December 2010 June 2012 October 2013 April 2014 January 2015ll Post-implementation evaluation report (PIER) NI June 2013 October 2014 October 2014 June 2015 Sources: California Department of Veterans Affairs’ (CalVet) feasibility study report, special project reports (SPR), the PIER, Enterprise-Wide Veterans Home Information System Implementation Contract (implementation contract), and Enterprise Wide Veterans Home Information System deliverable acceptance documents. NI = Not indicated. * Dates differ from those presented in CalVet’s PIER. The date for solution procurement differs because CalVet used the effective date of its implementation contract, as noted in SPR 3, and we used the date that the Department of General Services approved the contract. The date for user acceptance testing differs because CalVet used the date the system contractor submitted the corresponding deliverable, as noted in SPR 3, and we used the date that CalVet accepted the deliverable. The date for the implementation differs because CalVet was unable to provide supporting documentation for the dates it used. Therefore, we relied on the date that CalVet accepted the deliverable. † In SPR 1, CalVet determined it would implement the system in the homes located in West Los Angeles, Lancaster, and Ventura as a pilot implementation to validate the system before statewide roll out. In SPR 2, the pilot implementation was reduced to include CalVet headquarters and the home located in Barstow. In SPR 3, the pilot implementation was expanded to include headquarters and the homes located in Barstow, West Los Angeles, Lancaster, and Ventura. ‡ Because CalVet did not have a deliverable acceptance document for the implementation in the home in Chula Vista, we relied on the dates CalVet listed in its PIER. § According to the PIER, the implementation at the homes located in Fresno, Redding, and Yountville was not fully completed. Specifically, while the financial modules were implemented, the clinical modules—by which the homes manage electronic health records—were never implemented at these homes. ll In January 2015, CalVet and the California Department of Technology mutually agreed to close the Enterprise-Wide Veterans Home Information System project. 54 California State Auditor Report 2015-121 June 2016 Blank page inserted for reproduction purposes only. California State Auditor Report 2015-121 55 June 2016 Appendix B AN EXPANDED VIEW OF ENTERPRISE‑WIDE VETERANS HOME INFORMATION SYSTEM USE BY THE CALIFORNIA DEPARTMENT OF VETERANS AFFAIRS’ VETERANS HOMES The Joint Legislative Audit Committee asked us to determine the level of Enterprise‑Wide Veterans Home Information System (system) use within the California Department of Veterans Affairs’ (CalVet) veterans homes. Table 3 on page 23 provides an overview of when CalVet implemented the system at its headquarters and each of the eight veterans homes as well as the extent of the system modules in use at each of those locations. Table B provides an expanded view of those data. Table B Enterprise-Wide Veterans Home Information System Module Use by Location and Date of Implementation CHULA WEST HEADQUARTERS, BARSTOW, VISTA, FRESNO, LANCASTER, REDDING, VENTURA, LOS ANGELES, YOUNTVILLE, COMPONENTS MAY 2012 MAY 2012 JUNE 2013 NOVEMBER 2013 MAY 2012 NOVEMBER 2013 MAY 2012 MAY 2012 NOVEMBER 2013 ADL Clinical Software designed to address critical functions of providing long-term care by capturing and managing patient information. 1. Rehabilitation (Rehab) Touch NA F F NI NA F NA F F 2. Rehab Editor NA F F NI NA NI NA F F 3. Care Plans NA F F NI F NI F F NI 4. Virtual Body NA P P NI F NI P N NI 5. Certified Nursing Assistant (CNA) NA F N NI F NI F F NI 6. CNA Reports NA F N NI F NI P P NI 7. CNA Assignments NA F P NI F NI F F NI 8. Progress Notes NA F F NI F NI F F NI 9. Shift Reports NA N N NI F NI F N NI 10. Assessments NA P P NI F NI F P NI 11. Rehab Assessment NA F N NI NA NI NA F NI 12. Vitals NA F N NI F NI F F NI 13. Order Entry NA P N NI N NI N N NI 14. Electronic Medication Administration Record NA N N NI N NI N N NI (eMAR)/ Electronic Treatment Authorization Request (eTAR) 15. Minimum Data Set (MDS) NA F F F NA F NA F F 16. Clinical Dashboards N P N NI P NI P N NI 17. Quality Assurance NA NA NA NI NA NI NA NA NI 18. Infection Control NA N F NI F NI F P NI 19. Allergy Management NA F F NI P NI F F NI 20. Diagnosis Management NA F F NI F NI F F P 21. Facility Reports— F P P NI P NI F P NI Clinical/Order Reports 22. Precautions NA F F NI N NI F P NI 23. Forms N F P NI P NI F P NI continued on next page . . . 56 California State Auditor Report 2015-121 June 2016 CHULA WEST HEADQUARTERS, BARSTOW, VISTA, FRESNO, LANCASTER, REDDING, VENTURA, LOS ANGELES, YOUNTVILLE, COMPONENTS MAY 2012 MAY 2012 JUNE 2013 NOVEMBER 2013 MAY 2012 NOVEMBER 2013 MAY 2012 MAY 2012 NOVEMBER 2013 ADL Financial Software used to support third-party billing, Medi-Cal, Medicare, accounts receivable, transaction history, withdrawals and charges; making, tracking, and controlling financial events. 24. Admissions, Discharge, NA F F F F F F F F Transfers (ADT) Registration 25. ADT History F F F F P F P F F 26. Census Actions NA F F F P F F F F 27. Census Control NA F F F P F F F F 28. ADT Corrections NA F F F N F F F F 29. Trust Funds NA F F F F F F F F 30. Cash Receipts NA F F F P F F F F 31. Cash Receipts— P P P P P F P P F Veterans Affairs (VA) Per Diem 32. Online Receipts NA F F F F F F N F 33. Batch Receipts NA F F F F F F F F 34. Rating Calculation NA F F F F F F F F 35. Paysources F F F P P P F P F 36. Medicare NA F F P P P F P F 37. Insurances F F F F N F F F F 38. Rehab Module NA F F NI N NI N N F 39. Ancillary Charges NA F F P N P F P F 40. Professional F F F N N N N N F 41. Wait List NA F F F P F F F F 42. Billing F F F N P N F N F 43. Pre-Billing P P P F N F F P P 44. Off Hours Admissions by Nurses NA F F F F F F F F 45. Forms NA F F F F F F F NI 46. Dashboards F P P F N P N P NI 47. Immunizations NA F P NI F NI F F NI 48. VA Per Diem Charges P F F F P F F F F 49. VA Per Diem Cash Receipts F N N N N F N N F 50. Veterans P F F F P F F F F 51. Ledgers F F F F F F F F F 52. Medical Diagnosis F F F F F F F F F 53. Reports F F F F F F F F F 54. MDS Data P F F P NA P NA P F 55. Recurring Charges N N N N N N N N N 56. Errors Tracking P F F F F F F F F 57. TAR Authorizations- P F F N NA N NA N P Medi-Cal TARs 58. Expected Income and Budget F N N N N N N N N 59. End Period Process F F F F F F F F F California State Auditor Report 2015-121 57 June 2016 CHULA WEST HEADQUARTERS, BARSTOW, VISTA, FRESNO, LANCASTER, REDDING, VENTURA, LOS ANGELES, YOUNTVILLE, COMPONENTS MAY 2012 MAY 2012 JUNE 2013 NOVEMBER 2013 MAY 2012 NOVEMBER 2013 MAY 2012 MAY 2012 NOVEMBER 2013 Dynamics Financial accounting and business management software that automates creation and management of accounting data and workflow. Used to augment ADL Financial to support financial management. The specific component used is for purchasing and inventory. 60. Purchasing and Inventory N F P F F F F F F GeriMenu Software used for resident meal planning to help facilities run their nutrition departments more efficiently. 61. Resident Meal Planning NA P P F N NI N F P Documentum An enterprise content management system used to house and manage electronic documents. 62. Document Management N F N NI N NI F F NI Framework LTC Pharmacy management software designed for long-term care and institutional facilities. 63. Pharmacy Management NA F F NI N NI F F F Sources: California Department of Veterans Affairs’ (CalVet) system use chart provided by the system final project manager, and confirmed or updated by staff at headquarters and the eight homes, draft system architecture design document, and the CBORD Group Inc. Gerimenu website, and system contractor’s response to the request for proposals. F = The veterans home indicated it fully used this module. P = The veterans home indicated it partially used this module. N = Although CalVet implemented the module, headquarters or the veterans home indicated it did not use it. NI=CalVet did not implement the module at the veterans home. NA= Although these modules were available for use, staff at headquarters or the homes indicated the components marked NA were generally not applicable for their operations. 58 California State Auditor Report 2015-121 June 2016 Blank page inserted for reproduction purposes only. California State Auditor Report 2015-121 59 June 2016 STATE OF CALIFORNIA Edmund G. Brown Jr., Governor DEPARTMENT OF VETERANS AFFAIRS OFFICE OF THE SECRETARY POST OFFICE BOX 942895 SACRAMENTO, CALIFORNIA 94295-0001 Telephone: (916) 653-2158 Fax: (916) 653-2456 June 9, 2016 Elaine M. Howle, State Auditor Bureau of State Audits 621 Capitol Mall, Suite 1200 Sacramento, CA 95814 Dear Ms. Howle: Thank you for the opportunity to review and comment on the redacted copy of the California State Auditor (CSA) draft report titled, “The State Paid Nearly $28 Million for a Flawed System That Fails to Meet the Needs of Veterans Homes.” At the request of the Joint Legislative Audit Committee, the CSA conducted an audit of the California Department of Veterans Affairs (CalVet) Enterprise-wide Veteran Home Information System (Ew-VHIS) Project. The project was initiated in 2007 via an approved Feasibility Study Report (FSR) by the California Department of Technology (CDT) and concluded in 2015 with the goal to implement an Electronic Health Record System for its eight veteran homes. As further explained below, CalVet generally agrees with allseven recommendations in the State Auditor’s report, which serve to introduce departmental processes and policies related to information technology projects at CalVet.CalVet also agrees with the Report’s references to costs and scope. We take our fiduciary responsibilities very seriously and remaincommitted to the safeguarding of public resources. In retrospect, we could have moved more quickly or acted with greater urgency. However, CalVet took steps to address our concerns about the project. In December 2013, we reported critical concerns about the project to CDT and then, in partnership with CDT, took the initiative toimplement stronger IT project management processes and practices, in alignment with industry best practices and standards, to prevent a similar situation in the future. Some of the new processesinclude the implementation of the following plans for current and future projects: • Project Governance Plan • Project Change Management Plan • Project Requirements Management Plan • Project Risk and Issue Management Plan • Project Test Management Plan CalVet is also under new leadership, includingmyself as SecretaryandRussell Atterberry as Undersecretary, and we believe that the Auditor’s recommendations will further HONORING CALIFORNIA’S VETERANS 60 California State Auditor Report 2015-121 June 2016 Elaine M. Howle June 9, 2016 Page 2 strengthen the department in areas of project oversight, project management, and contract managementgoing forward. Furthermore, I believe CalVet has already made great strides towards implementing the resolutions. CalVetrecently began the process to find a replacement system. On May 31, 2016, the department successfully completedthe first of four stages of the contracting process. CalVet aims for implementation of the replacement system in 2017. The following are CalVet’sspecific responses to each recommendation provided within the report. Recommendation #1 To ensure that its project management of IT projects promptly identifies problems and develops resolutions, by September 2016, CalVet should define the project executive and project manager responsibilities to ensure that the individuals who fill those positions take an active role on each project. CalVet Response #1 CalVet understands and agrees with this recommendation. Existing IT projects at CalVet have documented roles and responsibilities for the project executive, project manager, and project team. A governance plan is created for each IT project to define the escalation process and ensure effective risk and issue management. Recommendation #2 To ensure that it adequately identifies and monitors problems in its future IT projects, by September 2016, CalVet should establish a formal process for its project executive to verify that the project team prepares all of the required project management and other required plans. This formal process should also include a process to periodically verify that the project team adheres to all these plans. CalVet Response #2 CalVet understands and agrees with this recommendation. Existing IT projects at CalVet have documented formal processes for ensuring required project management plans are created, adhered to, and kept up to date, in accordance with the policies of our control agencies. CalVet is currently working on its IT governance structure, which will be rolled out in a phased approach. By September 2016, the IT governance structure will implement a formal process for ensuring a project executive is identified for each IT project. This project executive will verify that the project team prepares required plans and ensures the plans are followed. Recommendation #3 To ensure accountability and independence between the provision of Independent Project Oversight [IPO] and Independent Verification and Validation [IV&V] on future IT projects, by September 2016, CalVet should establish a policy requiring it to use separate contractors for IPO and IV&V services when IPO services are not provided directly by CDT. HONORING CALIFORNIA’S VETERANS California State Auditor Report 2015-121 61 June 2016 Elaine M. Howle June 9, 2016 Page 3 CalVet Response #3 CalVet understands and agrees with this recommendation. In late 2013, CalVet reported this issue to the CDT relative to the Ew-VHIS project. While previously IPO and IV&V services were provided by the same contractor, CalVet ensured the CDT provided IPO services moving forward. CalVet is currently drafting a department-wide IT policy to ensure IPO and IV&V services on future IT projects use separate contractors for each service. CalVet expects to have this policy implemented by no later than September 2016. Recommendation #4 To ensure that it complies with state contracting laws and can demonstrate the basis for its decisions when awarding contracts, by September 2016, CalVet should establish a process to periodically verify that its staff follows state contracting requirements and maintainsall required contract documentation. CalVet Response #4 CalVet understands and agrees with this recommendation. By September 2016, a formal process will be implemented to periodically verify that state contracting requirements are followed and contract documentation is maintained and kept current. Recommendation #5 To ensure it maintains all documentation related to its IT contracts, CalVet should,by September 2016, establish a process to verify that all divisions comply with its policy requiring each division to submit a records retention schedule to its Office of Procurement and Contracts. CalVet Response #5 CalVet understands and agrees with this recommendation. By September 2016, a formal process will be implemented to ensure that each division submits a records retention schedule to its Office of Procurement and Contracts as required by departmental policy. Recommendation #6 To ensure that it only accepts and pays for deliverables that are complete and meet the contract requirements, by September 2016, CalVet should establish a process to verify and maintain documentation of all contract deliverables before approving payment. CalVet Response #6 CalVet understands and agrees with this recommendation. This is an existing practice at CalVet for acceptance and payments related to all IT contracts and associated deliverables. By September 2016, a formal process will be created to reinforce this expectation of all contract managers. HONORING CALIFORNIA’S VETERANS 62 California State Auditor Report 2015-121 June 2016 Elaine M. Howle June 9, 2016 Page 4 Recommendation #7 To ensure that it maximizes its opportunity to successfully implement future IT projects, by September 2016, CalVet should establish a formal process to verify that its staff conducts lessons learned sessions for all key phases of future projects and incorporate the recommendations identified in its post implementation evaluation. CalVet Response #7 CalVet understands and agrees with this recommendation. Currently, CalVet is applying this process to existing IT projects. By September 2016, the management expects to conduct lessons learned sessions for all key phases of future projects and incorporate the recommendations identified in its post implementation evaluation report in future projects and/or project phases via a formalized process. Sincerely, Vito Imbasciani MD Secretary HONORING CALIFORNIA’S VETERANS California State Auditor Report 2015-121 63 June 2016 May 24, 2016 Elaine M. Howle, State Auditor California State Auditor 621 Capitol Mall, Suite 1200 Sacramento, CA 95814 Re: CALIFORNIA STATE AUDITOR’S REPORT NO. 2015-121 Pursuant to the above audit report, enclosed are the Department of Technology’s comments pertaining to the results of the audit. The Government Operations Agency would like to thank the state auditor for its comprehensive review. The results provide us with the opportunity to better serve our clients and protect the public. Sincerely, Marybel Batjer, Secretary Government Operations Agency Enc 64 California State Auditor Report 2015-121 June 2016 CALIFORNIA DEPARTMENT OF TECHNOLOGY State of California Amy Tong, Interim Director Edmund G. Brown Jr., Governor Memorandum To: Marybel Batjer, Secretary Date: May 24, 2016 Government Operations Agency J4 From: Amy Tong, Interim Director California Department of Technology Subject: Response to California State Auditor's Draft Report No. 2015-121 We are providing for your review the draft California State Auditor's Report No. 2015-121 concerning CalVet's Enterprise Wide Veterans Home Information System (EW­VHIS). The following responses address the California State Auditor's recommendations regarding the California Department of Technology's (CDT) operations. RECOMMENDATIONS RECOMMENDATION #1: To ensure it can demonstrate that it acts in the best interest of the State, the Technology Department should, by December 2016, create a formal process to summarize its involvement and document key actions taken and decisions reached during agencies' contract disputes and negotiations for the termination of a contact and maintain those documents according to its record retention schedule. Department of Technology's Response #1: The Department of Technology agrees with this recommendation. CDT will develop internal procedures for documenting the formal process associated with contract dispute or contract termination negotiations. The procedures would include retaining a summary of negotiation outcomes which will be maintained with applicable law and CDT's records retention schedule. RECOMMENDATION #2: To ensure accountability and independence between the provisions of /PO and IV&V services, the Technology Department should, by December 2016, establish a written policy requiring departments that request and receive approval to contract for /PO services to use a different contractor than the ones providing IV& V services. California State Auditor Report 2015-121 65 June 2016