Summary

March 2015 Los Angeles Department of Water and Power Consequences Linked to Its Premature Launch of Its Customer Information System May Push Total Costs Beyond $200 Million Report 2014-105 COMMITMENT INTEGRITY LEADERSHIP The first five copies of each California State Auditor report are free. Additional copies are $3 each, payable by check or money order. You can obtain reports by contacting the California State Auditor’s Office at the following address: California State Auditor 621 Capitol Mall, Suite 1200 Sacramento, California 95814 916.445.0255 or TTY 916.445.0033 OR This report is also available on our Web site at www.auditor.ca.gov. The California State Auditor is pleased to announce the availability of an online subscription service. For information on how to subscribe, visit our Web site at www.auditor.ca.gov. Alternate format reports available upon request. Permission is granted to reproduce reports. For questions regarding the contents of this report, please contact Margarita Fernández, Chief of Public Affairs, at 916.445.0255. For complaints of state employee misconduct, contact the California State Auditor’s Whistleblower Hotline: 1.800.952.5665. Elaine M. Howle State Auditor Doug Cordiner Chief Deputy March 10, 2015 2014‑105 The Governor of California President pro Tempore of the Senate Speaker of the Assembly State Capitol Sacramento, California 95814 Dear Governor and Legislative Leaders: As requested by the Joint Legislative Audit Committee, the California State Auditor (state auditor) presents this audit report concerning the Los Angeles Department of Water and Power (department) and its implementation of its customer information system (CIS). This report concludes that the premature launch of CIS may cause the department to spend in excess of $200 million on the project, which is significantly more than the department’s original budget of roughly $87 million and its more recent November 2014 budget of nearly $181 million. The growth in actual project costs over budgeted costs results from higher than expected spending on consultants, materials, and the department’s own employees who worked on CIS, as well as the estimated costs associated with outstanding customer accounts that may now be too old (greater than 470 days past due) to collect. More than a year after launching CIS in September 2013, the department is still struggling to normalize important business practices and to collect unpaid accounts. Its customers owe the department more than $681 million as of November 2014—an increase of more than $245 million under its prior billing system—and the department’s efforts to collect on these accounts have been tempered by a desire to manage call wait times in its call centers. The department attempts to limit call wait times, in part, by managing how many of its customers enter its collections process and then call the department to discuss their accounts. The department’s executive management was well aware of the significant problems associated with CIS and yet made the questionable decision to launch the new system. In October 2011— nearly two years before the department launched CIS—the department’s quality assurance consultant rated the CIS project as needing “immediate attention” and warned that no aspect of the project was ready. In fact, the quality assurance expert reported that the project’s scope, quality, and schedule were all at the lowest possible rating. The department’s own reports also showed numerous defects that remained unresolved, both before and after launch. Finally, we found that the department’s executive management provided little to no specific information to the Los Angeles Board of Water and Power Commissioners (board) about the CIS project, effectively denying the board the opportunity to delay CIS’s implementation. Respectfully submitted, ELAINE M. HOWLE, CPA State Auditor 621 Capitol Mall, Suite 1200 Sacramento, CA 95814 916.445.0255 916.327.0019 fax www.auditor.ca.gov Blank page inserted for reproduction purposes only. California State Auditor Report 2014-105 v March 2015 Contents Summary 1 Introduction 5 Audit Results The Los Angeles Department of Water and Power Was Aware of Significant Issues With Its New Customer Information System Yet Still Chose to Launch It 17 The Department’s Decision to Minimize Customer Call Wait Times Following CIS’s Launch May Cost It Millions of Dollars Related to Past‑Due Accounts 22 CIS Cost Far More to Implement Than the Department Originally Budgeted Because of the Project’s Complexity and the Need for Significant Involvement From the Department’s Personnel 31 The Department Is Still Attempting to Resolve Defects That Existed During CIS’s Development 38 The Department Generally Met Proposal and Contracting Requirements but Needs to Strengthen Its Policy Regarding Conflicts of Interest 42 Recommendations 45 Appendix Estimated and Delayed Bills Following Implementation of the Customer Information System 47 Response to the Audit Los Angeles Department of Water and Power 51 California State Auditor’s Comments on the Response From the Los Angeles Department of Water and Power 53 vi California State Auditor Report 2014-105 March 2015 Blank page inserted for reproduction purposes only. California State Auditor Report 2014-105 1 March 2015 Summary Results in Brief Audit Highlights . . . In early September 2013 the Los Angeles Department of Water Our audit of the Los Angeles Department of and Power (department) launched its new customer information Water and Power’s (department) customer system (CIS) to provide billing and customer service functions for information system (CIS) highlighted its roughly 3.8 million customers. Following more than three years the following: of integration and testing, CIS replaced a 40‑year‑old, highly » Over time, the department more customized system that was technologically outdated. However, than doubled its original budget for numerous and profound problems with CIS arose immediately implementing CIS—from $87 million to after its implementation, reasonably causing the public to question nearly $181 million. whether the system was actually ready for everyday use. In fact, both independent quality assurance reports and the department’s » The department minimized or ignored the own assessment of the system’s readiness demonstrate that the severity of issues raised regarding CIS’s department minimized or ignored the severity of the issues readiness before launching it. that existed at the time it made the decision to launch CIS. Further, in the months leading up to the launch, the department • The department’s quality assurance consistently failed to disclose the issues with CIS’s development expert warned that no aspect of the when presenting the status of the Customer Information System project was ready. Connection project (project)1 to the Los Angeles Board of Water and Power Commissioners (board), effectively denying the board • The department shared little to the opportunity to delay the system’s implementation. no specific information with the Los Angeles Board of Water and Power The department originally budgeted $87 million for implementing Commissioners (board) about the CIS; however, it more than doubled that budget to nearly severity of CIS’s issues, limiting the $181 million over time. Nonetheless, immediately after CIS’s launch, board’s oversight ability. it became clear that the system was not yet ready and that the » A recent department‑sanctioned analysis department’s decision to implement it was questionable at best. confirmed that the department’s decision Consequently, the department’s customers began complaining of to launch CIS was premature and its late utility bills, unwarranted shut‑off notices, and excessive wait workforce unprepared. times to speak with customer service representatives. » The department is still struggling— Although the department had received feedback about CIS’s more than a year later—to normalize significant problems before its launch, the department chose important business practices and to not to heed this information. Reports from the department’s collect unpaid accounts. quality assurance expert warned that no aspect of the project was ready; in fact, the quality assurance expert reported that » Although the department has improved the project’s scope, quality, and schedule were all at the lowest its ability to issue timely bills based possible rating and needed immediate attention. However, on actual meter reading, its customers the department shared little to no specific information with the owe more than $681 million as of board about the severity of CIS’s issues, limiting the board’s November 2014 for past‑due bills. oversight ability. A recent department‑sanctioned analysis of the causes of CIS’s failures confirmed that the department’s decision 1 While the new system was under development, it was referred to as the Customer Information Service Connection project. Since its launch, it has been called simply CIS. In this report, we use the term “project” to refer to the system during its development and “CIS” to refer to it after it was launched. 2 California State Auditor Report 2014-105 March 2015 to launch CIS in September 2013 was premature and that the department’s workforce was not prepared to ensure the system’s proper operations. More than a year later, the department is still struggling to normalize important business practices and to collect unpaid accounts. Although the department has improved its ability to issue timely bills based on actual meter readings, its customers owe more than $681 million as of November 2014 for past‑due bills, an increase of over $245 million from July 2013, two months before CIS was launched. This increase in past‑due bills is linked to the department’s attempts to control its long call wait times. Specifically, the department reduced the number of past‑due accounts it placed in collections after the launch of CIS because customers whose accounts are in collections frequently call to resolve their payment issues. In other words, the department limited its efforts to collect past‑due bills in order to decrease its call volume and thus reduce its long call wait times. By September 2014 the department had already spent $187 million on implementing and stabilizing CIS. If it is unable to resolve and ultimately collect on its past‑due accounts, these uncollectible debts, linked in part to the department’s launching the system prematurely, could add in excess of $40 million to CIS’s overall price tag. In the year following CIS’s launch, the department was also ineffective at addressing system issues. Specifically, it has yet to resolve a number of defects—issues that cause a system to not perform as expected by negatively affecting its operations—in CIS that existed before it was launched. Some of these continuing defects are severity level 1 defects, the most significant because they interrupt or make a system’s normal operations impossible. The department’s recent steps to address CIS’s remaining defects and other system issues potentially involve its entering into another multimillion dollar contract. In November 2014 the department issued a request for proposal for work valued at between $13 million and $15 million over three years. The department has asserted that the request for proposal is for new work, but we question this characterization. Our IT expert noted that the request for proposal includes an assessment of the current state of CIS—which has been in use for less than two years—and that it suggests that the original implementation was incomplete and incorrect and requires remediation, which the request for proposal intends to address. Despite its collections difficulties, the department’s current financial situation appears to be sound. In fact, in December 2014, the department’s board approved a transfer of $265.6 million to the Reserve Fund of the city of Los Angeles, which is consistent with the amount it has transferred in previous years. Nonetheless, California State Auditor Report 2014-105 3 March 2015 the department’s poor decision making and poor communications with its board regarding the launch of CIS may unnecessarily cost it millions of dollars from unpaid customer bills—costs that it will ultimately need to either absorb or pass on to its customers in the form of rate increases. Recommendations To ensure that the board can more effectively exercise oversight for the department’s significant information technology projects, the board should take the following actions: • Establish a standing committee composed of board members to oversee and critically evaluate the status of the department’s various information technology projects. • Develop reporting standards for the department’s management to follow when discussing the status of information technology projects. Such reporting standards should, at a minimum, specify the frequency with which the department’s management makes such reports and require the following disclosures about each information technology project: – The amount of project growth, in terms of both budget and scope of work, from initial project estimates through current projections. – The results from system testing and a listing of the critical defects that exist and must be fixed prior to system use. – The concerns the quality assurance contractor has raised and how the department is addressing them. • Develop a process for the board to designate certain information technology projects as having a potentially significant effect on business operations or customer relations, and require that department managers first obtain the board’s approval before launching such critical new systems. Agency Comments In its response to the audit, the board stated that it and the department agreed with the recommendations; however, it stated that it disagreed with the basis on which we made them. The board asserted that the department had not misled it about the project, but, rather an ongoing independent investigation has preliminarily found that a vendor hired to assist in implementing CIS 4 California State Auditor Report 2014-105 March 2015 intentionally misled the department. The board noted that it has made progress in addressing concerns with CIS and will continue to work with all interested parties to ensure issues this audit raised are addressed. California State Auditor Report 2014-105 5 March 2015 Introduction Background In 1902 the city of Los Angeles (city) established the Los Angeles Department of Water and Power (department). With roughly 8,800 employees and an annual budget of more than $7 billion, the department provides water and electricity to roughly 3.8 million residents and businesses and is among the nation’s larger municipal utilities. Its service area encompasses customers in both the city and Owens Valley, as depicted in Figure 1 on the following page. The Los Angeles Board of Water and Power Commissioners (board) oversees the department, with each commissioner appointed to a five‑year term by the city mayor. Subject to approval by the city mayor and the Los Angeles City Council (city council), the board appoints a general manager to oversee the department’s day‑to‑day operations. In September 2013 the department launched a new customer information system (CIS) to assist it in managing some of its core business operations, including billing for power and water usage and managing customer service functions.2 Not long after CIS’s launch, the media reported that the department’s customers were experiencing late or inaccurate bills. Further, some media reports told of customers who lost significant amounts of money from their bank accounts because they were using the department’s automatic bill payment service. Public frustration deepened when the department could not promptly answer its large call volumes stemming from customer inquiries about bills. In response to customer outcry, the city council requested a moratorium on service disconnections and required the department to provide it with updates every 30 days on the status of its corrective actions. The department also created a Web page to provide reports on its progress in fixing CIS and the challenges that it still needs to address. The Web page includes a Mayor’s Dashboard, a weekly report that identifies key performance metrics, including total customer calls, call wait times, and various billing metrics. The Department’s Customer Billing and Collections Processes for Water and Power Services During a three‑day period known within the department as the billing‑cycle window, the department aims to read a customer’s power and water meters and generate a bill, as demonstrated in Figure 2 on page 7. According to the director of customer operations, 2 The department also provides billing services for the city’s Department of Public Works, Bureau of Sanitation. These charges include city sewer service and trash fees. We focus this report on the water and power services the department provides. 6 California State Auditor Report 2014-105 March 2015 Figure 1 Los Angeles Department of Water and Power’s Service Area Source: Service area maps obtained from the Los Angeles Department of Water and Power. California State Auditor Report 2014-105 7 March 2015 Figure 2 Summary of the Water and Power Billing Process Sources: California State Auditor’s analysis of Customer Care and Billing operations manual and rate schedules, and information Los Angeles Department of Water and Power (department) staff provided. * Causes of an abnormal meter reading include human error and meter deficiencies. † Causes of no meter reading include a lack of access to the meter. ‡ If a bill has been estimated three times in a row, or a rate cannot be estimated then the bill is delayed until the issue causing the delay is resolved. § Account issues such as pending field activities, meter changes, and installing solar meters require department staff intervention and may result in a delayed bill. the department reads the majority of its meters manually through in‑person inspections. This process typically starts with a department employee reading a meter and entering the consumption data into a handheld device, which then 8 California State Auditor Report 2014-105 March 2015 uploads the data to the department’s system. Once the system acquires the data, it compares the information to the customer’s previous meter readings to ensure that the current meter readings are accurate and reasonable. If the comparison is within acceptable parameters, the department produces a bill and sends it to the customer. However, if the comparison results in a variance outside of a range specified by the department—known as the quality control parameter—a customer service representative (representative) must then review the abnormal meter reading. If the representative approves the current meter reading, the department will produce a customer bill based on it. Alternatively, if the representative rejects the meter reading or does not have time to review it, the system may generate an estimated bill. The system may also generate estimated bills if the department cannot access a meter, finds problems with the meter, or determines that an account has outstanding issues. When generating an estimated bill, the department’s system follows a hierarchy of estimation techniques based on the available data. First, the system attempts to estimate the customer’s current water or power usage based on that same customer’s usage from the same period in the prior year. For example, if the system needed to estimate a customer’s water bill for the two‑month period of March and April 2014, the system would first consider the customer’s water consumption in March and April of 2013. If these data were unavailable, the system would next try to estimate the customer’s water bill based on consumption in the prior billing period, from January and February 2014. Finally, if the customer’s account‑specific information was unavailable, the system would estimate the customer’s bill based on an average of water or power usage by similar customers. The system generally will produce three estimated bills in a row for a customer, after which it will cease producing bills until the department addresses the problem. Once the department obtains an accurate meter reading, it will reconcile the customer’s actual usage with the estimated usage and either bill or credit the customer for the difference. If the department does not produce a bill on time, it refers to it as delayed. According to the director of customer operations, all utilities issue some estimated and delayed bills. According to the director of customer operations, the department’s focus after issuing a bill is on providing customer service and collecting payments. Customer service includes supplying customers with information about their accounts and responding to their concerns. The department provides information to its customers through call centers, Web site services, and an interactive phone system. When an account goes unpaid for a certain period of time California State Auditor Report 2014-105 9 March 2015 and the amount owed reaches a predetermined level, the department begins the collections process, which Figure 3 illustrates. This process includes mailing a past‑due notice, making two automated phone calls, and then mailing a final disconnect notice. If the account is still not paid, the department terminates the customer’s service. According to the director of customer operations, service termination usually requires that a department employee, known as a field collector, visit the service location to disconnect the service manually; field collectors may give account holders a final opportunity to pay, consistent with department policies, to avoid service termination. Figure 3 Summary of the Collections and Service Termination Processes for Typical Residential and Commercial Customers Bill issued DAYS Customer expected to make payment 1–19 to customer Bill becomes delinquent DAY 20 Residential, Past-due commercial Phone call reminder past-due notice sent Bill is delinquent DAY 30 because customer Residential has not paid it DAY 31 Second phone call reminder Commercial DAY 32 Residential Disconnect notice sent DAY 33 warning Commercial DAY 44 Service termination Residential process initiated DAY 46 Commercial By The field collector DAY 59 is dispatched to take Residential final meter reading and turn off meter DAY 61 Commercial By DAY 61 Service Residential Closing bill issued* terminated DAY 63 Commercial Sources: Los Angeles Electric Rates, July 1, 2008; Los Angeles Water Rates (amended March 19, 2012); Rules Governing Water and Electric Service, October 2008; California State Auditor’s analysis of the Los Angeles Department of Water and Power’s (department) residential and commercial collections process and severance process, and information department staff provided. * The department issues a final notice for the closing bill if the bill is not paid and may refer the debt to a collection agency. 10 California State Auditor Report 2014-105 March 2015 In addition to providing water and power to its customers, the department is also a significant revenue source for the city. The city charter authorizes the department to transfer any surplus funds from its power revenue to the city’s Reserve Fund at the end of each fiscal year, following a specific transfer approval process that includes the board’s and the city council’s authorization. The Customer Information System Connection Project Before the department’s first use of the billing portion of CIS in September 2013, the department used an older system called TRES to support the majority of its customers (roughly 1.4 million electric and 640,000 water customers).3 The department described TRES as a nearly 40‑year‑old legacy computer system that was built on outdated mainframe code and relied on manual processes that did not provide the department with agility in changing rate models. According to the department, TRES also lacked the flexibility needed to generate increasingly complex bills and to integrate with the department’s various other data systems. When the department requested proposals in 2009 from vendors to assist with replacing its billing systems, the department stated that this effort was among its highest priorities, citing the risk associated with trying to support TRES. Before launching CIS, the department made three failed attempts to implement new systems. According to the assistant director of the department’s information technology services division, the first three attempts at system replacement were unsuccessful for a variety of reasons, including high cost projections, uncertain funding, and concerns over the long‑term viability of the vendor that would ultimately help to support a new system. The Customer Information System Connection project (project)—the department’s fourth attempt—began in August 2009. At the core of the project was a billing system called Customer Care and Billing (CC&B), which the department selected with the help of its systems integrator. CC&B is a ready‑made, off‑the‑shelf Oracle software product, which the department stated would significantly improve its flexibility, reliability, data management, and ability to integrate the program with other programs. As shown in Figure 4, CC&B is now the core of a large network of systems—all referred to as CIS since the launch in September 2013—that allows the department to transfer and communicate information among its many business processes, which include meter reading, billing, customer relations, collections, service terminations, field maintenance, and water and power outage management.4 3 The department had a second system, Banner, which it used to bill more than 11,000 customers located in Owens Valley and certain customers receiving nonmetered service in Los Angeles. Because Banner supported a relatively small number of the department’s customers, we focused on TRES. 4 While the new system was under development, it was referred to as the Customer Information Service Connection project. Since its launch, it has been called simply CIS. In this report, we use the term “project” to refer to the system during its development and “CIS” to refer to it after it was launched. California State Auditor Report 2014-105 11 March 2015 Figure 4 Customer Information System Connection Project’s Interfaces Summarized Existing systems integrated New or upgraded system that went live in September 2013 PAYMENT RELATED INTERFACES New system that went live after September 2013 Large batch payments Electronic bill BILLING RELATED INTERFACES pay, auto pay, Credit card Web services, automated and payment processing customer Mailing cancellations telephone system, Bill retrieval Bill printing u a p d d d a re te ss s and kiosks and bill reporting management Electronic billing for select customers ACCOUNTING AND REPORTING RELATED INTERFACES CUSTOMER RELATIONS RELATED INTERFACES Accounting Environmental functions Protection Agency Address i A n t c e u f u o l t e s o r t p m m o h m a a o t t n e i e o r e d n m re C a l n u a a t s i t g o o e n m m sh e e i r p n t a c n e D p d r r t i s o a ifi c p g o c p r a u a r t n m o io t v n a l D L e o p s a r A t n m g e e n le t s o f co r n e h s p i u s o m t r o t p r in y t g ion verification system Building and Safety, premises Collections information activities Customer web services such as Customer Care account inquiries, and Billing Refund payment history, System (CC&B)* checks usage data, and e-notifications Field CC&B interfaces with maintenance 28 other systems creating the information Customer Information System Meter Connection project (project) reading Field activity uploads updates and completion information Meter Meter Meter do re w a n d l i o n a g d s m Sm an a a r g t m em e e te n r t M hi e st t o e r r y i , n a fo n r d m b a i t ll i i o n n g , re e a ffi de ci r e r n o c u y te i a n n v d e n te to st r s y determinate information METER WORK MANAGEMENT RELATED INTERFACES METER DATA MANAGEMENT RELATED INTERFACES Meter inventories Customer Field Field activity and premises Water outage notifications Electricity maintenance updates, information management outage information cancellation, management and completion information OUTAGE MANAGEMENT RELATED INTERFACES UTILITY WORK MANAGEMENT RELATED INTERFACES Sources: California State Auditor’s analysis and adaptation of the project interfaces graphic and supporting documentation regarding various software. Notes: Interfaces are points where systems meet and interact. The arrows represent the direction in which information travels among systems. * Billing, credit and collections, and service terminations are some of the processes that run through CC&B. Others include Bureau of Sanitation billing, starting and stopping utility services, solar and electric vehicle programs, and write‑offs. This summary does not illustrate the many business functions that run through the customer information system. 12 California State Auditor Report 2014-105 March 2015 At the beginning of the project, one of the department’s first steps was to hire a consultant to perform quality assurance activities. In August 2009 the department entered into an agreement with Five Point Partners, LLC (Five Point) to provide the department’s project management with monthly written assessments of the risks and issues that might impede the project’s success, as well as a final assessment of departmental readiness before CIS’s launch. The department’s original agreement with Five Point was for three years and was not to exceed $3.5 million. As we discuss in the Audit Results of this report, this contract grew over time to $9.7 million and was extended through October 2013. A year after entering into its initial agreement with Five Point, the department used a competitive bid process to select PricewaterhouseCoopers as the project’s systems integrator. As such, PricewaterhouseCoopers’ responsibilities included assisting the department in selecting the software product (CC&B was chosen), helping the department identify business processes, mapping processes and requirements into CC&B, configuring CC&B to meet the department’s specific needs, and converting data from TRES for use in CC&B. The original PricewaterhouseCoopers contract had a three‑year term and a cost of $57.2 million, although the department later extended its duration and increased its cost, as we discuss in the Audit Results. As Figure 4 demonstrates, the project was large and complex. According to the assistant director of the information technology services division, the department needed to connect 28 of its systems in order to ready CC&B for operation. Consequently, the department and its consultants shared many important project responsibilities. For example, PricewaterhouseCoopers was responsible for developing the preproduction test plan, while the department was responsible for developing the test scenarios to use and for approving the results of the testing. Further, the project’s data conversion team consisted of both department employees and contractors, as did the project’s change management team and development and integration teams. Moreover, the department was generally responsible for accepting its contractors’ various deliverables and for procuring the necessary software licenses and the hardware for the project. The department initially planned to launch CIS in November 2012, but as Figure 5 shows, CIS’s actual launch date was postponed several times before the actual launch occurred in September 2013. California State Auditor Report 2014-105 13 March 2015 Figure 5 Timeline of Key Events During the Customer Information System Connection Project August 2009 Through September 2013 November 26, 2012 Proposed launch (postponed) April 22, 2013 Proposed launch (postponed) August 4, 2009 Five Point Partners, LLC contract August 5, 2013 Proposed launch (postponed) August 3, 2010 PricewaterhouseCoopers contract September 3, 2013 Actual launch 2009 2010 2011 2012 2013 Sources: Los Angeles Department of Water and Power (department) contracts, press releases, and task assignment agreement. Note: According to the department, the launch date was postponed due to a prolonged system testing period. Scope and Methodology The Joint Legislative Audit Committee (audit committee) directed the California State Auditor to review the department’s development and implementation of CIS. The audit committee approved six objectives. Table 1 lists the objectives that the audit committee approved and the methods we used to address them. Table 1 Audit Objectives and the Methods Used to Address Them AUDIT OBJECTIVE METHOD 1 Review and evaluate the laws, rules, We reviewed relevant state laws and regulations, relevant portions of the city of Los Angeles (city) and regulations significant to the administrative code, the Los Angeles City Charter, the executive directives the city mayor issued, and audit objectives. other background materials. 2 Review and evaluate the Los Angeles • We reviewed relevant state laws and regulations, pertinent city requirements, and department policies Department of Water and Power’s and procedures. (department) procurement and • We interviewed key officials. oversight of contracts to design • Using contract lists, budget data, and cost data the department supplied, we identified contracts and implement its new customer significant to the Customer Information System Connection project (project)—the name given to the information system (CIS), including project to design and implement CIS. We selected three contracts for further review. determining the following: • For the three selected contracts, we reviewed the department’s documentation related to developing a. Whether the department followed the requests for proposal, evaluating bids received, and awarding contracts to determine whether the applicable procurement policies department followed pertinent requirements, including those related to potential conflicts of interest. and procedures related to contract bidding and approval. • We assessed the extent to which the department heeded information its quality assurance contractor—Five Point Partners, LLC.—provided and whether this same contractor delivered certain key b. Whether the department contract requirements. adequately monitored the contractors’ implementation of CIS. • We analyzed the department’s contractually defined acceptance criteria required to launch to determine whether the department’s systems integrator—PricewaterhouseCoopers—and the department documented the project’s progress and whether both signed off on significant contract deliverables. • See also the methods we used to address audit objective 6. continued on next page . . . 14 California State Auditor Report 2014-105 March 2015 AUDIT OBJECTIVE METHOD 3 Review and evaluate CIS’s impact on • We reviewed pertinent city requirements, department policies and procedures, and other documents. the department’s monthly revenues • We interviewed key officials. since its September 2013 launch • We obtained and reviewed accounting reports of the department’s monthly revenues for the period date and determine the reason for from September 2012 through September 2014 and identified no significant revenue fluctuations significant or unusual fluctuations or trends. or trends. In addition, determine the impact of any significant revenue • We obtained and reviewed accounting reports of monthly cash collections for the period from fluctuations, including changes to the September 2013 through September 2014 and identified cash collections fluctuations or trends. amounts the department transferred • We determined whether the Los Angeles Board of Water and Power Commissioners (board) approved a to the city’s general fund. transfer to the city’s Reserve Fund for fiscal year 2013–14 and the transfer amount. • We also obtained and reviewed financial status reports the Office of the City Administrative Officer prepared. Those reports acknowledge shortfalls in revenue for external agencies but do not cite any significant impacts. 4 Identify the extent of customer • We reviewed pertinent department billing policies and procedures, billing processes, and problems resulting from CIS’s other documents. implementation by determining, at • We interviewed key officials and viewed video testimony from the Los Angeles City Council (city council) a minimum, the following: and subcommittee meetings. a. The number and proportion of • We determined that the department refers to late bills as delayed bills. Further, we determined that the customers experiencing late bills. department does not keep metrics on inaccurate bills; therefore, we assessed bills the department issued b. The number and proportion based on estimated usage. of customers experiencing • Using billing data that the department supplied, we summarized for various months, beginning with inaccurate bills. September 2013, the number of bills the department delayed issuing and the number of estimated bills c. The number and proportion the department issued. of customers experiencing • We assessed the steps the department has taken to reduce the number of delayed and estimated bills unwarranted shut‑off notices it issues. or service terminations before • We determined that the department does not track unwarranted shut‑off notices; therefore, we the department imposed a reviewed its collections process, its collections efforts, and its collections goals since launching CIS in moratorium on such terminations. September 2013. d. The number and proportion • We reviewed changes the department made to its collections thresholds for commercial and residential of customers for which the customers from September 2013 through September 2014, and summarized data the department department has resolved billing provided of the number of customers entering collections both before and after threshold changes. and service problems. • We determined that the department does not track unwarranted service terminations; therefore, we reviewed the department’s service termination process and its service termination efforts since launching CIS in September 2013. • We identified how the department responded to the city council’s request for a moratorium on service terminations. Using data the department supplied, we summarized the effect of these changes on the number of terminations the department has completed. • We reviewed and summarized past‑due accounts data—which the department terms bills awaiting payment—that the department supplied for July 2013 through December 2014. We did not audit the accuracy and completeness of the department’s bills awaiting payment data because we are presenting these data in Figure 8 on page 24 for informational purposes and they do not support any recommendations in our audit report. Figure 8 omits data immediately following CIS’s launch in September 2013 because the assistant director of revenue and credit management disclosed to us that he did not start receiving the reports until March 2014. Data from March through September 2014 are not directly comparable to data from October to December 2014 because the department started using a new data query in October that an operations and statistical research analyst asserted is more accurate. These data will not match the department’s financial reports because the data are not based on the same query used to generate the financial reports. These data do not include all money owed by customers; for example, debt owed by customers whose services the department has terminated is not included. California State Auditor Report 2014-105 15 March 2015 AUDIT OBJECTIVE METHOD 5 Identify the initial approved budget • We reviewed pertinent department policies, procedures, and other documents. and final actual cost of CIS and • We interviewed key officials and viewed video testimony from board meetings. determine the following: • We obtained and reviewed project budget information for fiscal years 2009–10 though 2014–15 and a. Whether and to what extent the project cost information for fiscal year 2009–10 through September 2014. project experienced significant • We compared the budget to the cost data the department provided us, identified significant differences, cost increases. obtained and reviewed departmental reports summarizing budget differences, and asked the b. The timing of when the department about the reasons for the differences. We also compared allowable contract amounts to the department became aware of costs the department provided for selected contracts. significant cost increases. • We identified and reviewed the department’s processes for informing its management and its board of c. The reasons for significant project costs and differences from budgeted amounts. cost increases, including costs incurred to address errors after the department implemented CIS in September 2013. 6 Review and assess any other issues • We reviewed pertinent contracts and contract deliverables, interviewed key officials, viewed video that are significant to the audit. testimony from city council and board meetings, and reviewed other documents. • We consulted with an information technology expert to ensure the accuracy of our understanding of information technology project management, defects, pertinent quality assurance reports, and contract scopes of work. • We assessed the department’s actions after launching CIS to address customer service issues, and we analyzed customer call wait time data the department provided to determine how these data correlated to the department’s actions. • We reviewed selected department defect tracking data from August 2013 through September 2014 to determine trends in the identification of defects during the project and after CIS’s launch. • We assessed the department’s decision to launch by analyzing its adherence to go‑live criteria outlined in its contract with PricewaterhouseCoopers and by reviewing the department’s final go‑no‑go criteria spreadsheet. • We obtained and reviewed a report TMG Consulting, Inc. prepared titled Los Angeles Department of Water and Power Approach for CCB/MWM Stabilization Root Cause Analysis (TMG report) and other pertinent documents the department sanctioned to further its understanding of the shortcomings in the project and to determine how to stabilize CIS. CC&B and MWM refer to two systems that are part of CIS. • We obtained and reviewed a request for proposal the department issued on November 20, 2014, titled Customer Information System Support and Upgrade Services. • We asked the department’s former senior assistant general manager, power system, how the department intends to address the TMG report’s recommendations and how the scope of work outlined in the subsequent request for proposal differed from the work the department contracted with and paid PricewaterhouseCoopers to perform. Sources: California State Auditor’s analysis of Joint Legislative Audit Committee audit request number 2014‑105, and information and documentation identified in the table column titled Method. 16 California State Auditor Report 2014-105 March 2015 Blank page inserted for reproduction purposes only. California State Auditor Report 2014-105 17 March 2015 Audit Results The Los Angeles Department of Water and Power Was Aware of Significant Issues With Its New Customer Information System Yet Still Chose to Launch It In early September 2013 the Los Angeles Department of Water and Power (department) launched its new customer information system (CIS). However, numerous and profound problems with CIS arose immediately after its implementation, reasonably causing the public to question whether the system was actually ready for everyday use. In fact, both independent quality assurance reports and the department’s own assessment of the system’s readiness demonstrated that the department minimized or ignored the severity of the issues that existed at the time it made the decision to launch CIS. Further, in the months leading up to the launch, the department consistently failed to disclose the issues with CIS’s development when presenting the status of the Customer Information System Connection project (project) to the Los Angeles Board of Water and Power Commissioners (board), effectively denying the board the opportunity to delay the system’s implementation. The Department Ignored Obvious Warning Signs That CIS Was Not Ready to Launch The department’s management was fully aware that CIS had persistent problems throughout its development and immediately preceding its launch. At the project’s onset, the department hired Five Point Partners, LLC (Five Point) to serve as a quality assurance contractor to monitor and report on the quality of the work performed by the department and the systems integrator, PricewaterhouseCoopers. Additionally, the department worked with PricewaterhouseCoopers to track the project’s readiness. Combined, these activities provided the department with the information necessary to understand the project’s status and to recognize the strong likelihood that launching CIS would result in problems. In a press release issued in November 2014 related to a department‑sanctioned analysis of CIS’s failures, the department acknowledged the existence of obvious warning signs that the new system was not ready for launch. The department spent at least $3.5 million for quality assurance services from Five Point; consequently, from February 2011 to June 2013, Five Point provided the department with written assessments of the project’s budget, schedule, scope, quality, resources, and risk. According to our information technology expert (IT expert), the department made a reasonable decision to hire a quality assurance contractor given the project’s size, 18 California State Auditor Report 2014-105 March 2015 complexity, and financial impact. However, the department disregarded Five Point’s significant and repeated warnings about the quality of the new system, its compressed deployment schedule, and other issues indicating that it was not ready for deployment. For example, in October 2011—nearly two years before the department launched CIS—Five Point rated the project’s overall health as “needs immediate attention.” Five Point never again rated the project’s overall health as reaching “fair,” as shown in Figure 6. Reports from the department’s quality assurance expert warned that no aspect of the project was ready; in fact, the quality assurance expert reported that the project’s scope, quality, and schedule were all at the lowest possible rating and needed immediate attention. Figure 6 Customer Information System Connection Project’s Overall Quality Assurance Ratings February 2011 Through June 2013 gnitaR llarevO ecnarussA ytilauQ 2 Excellent 2 1 Good 0 Fair 1 Needs improvement 2 Needs immediate attention 1 0 -1 -2 BEF RAM RPA YAM NUJ LUJ GUA PES TCO *VON *CED NAJ †BEF RAM RPA YAM NUJ LUJ GUA PES TCO VON CED NAJ BEF RAM RPA YAM NUJ 2011 2012 2013 Source: California State Auditor’s analysis of data obtained from Customer Information System Connection project quality assurance reports for February 2011 through June 2013. * The Los Angeles Department of Water and Power’s (department) management requested that no quality assurance reports be produced for November and December 2011 because the project was not moving forward during that time. † The department was unable to provide a quality assurance report for February 2012. California State Auditor Report 2014-105 19 March 2015 The department’s contract also required Five Point to provide a written assessment of the department’s preparedness for change from the current system to CIS, including identifying issues that needed resolution related to potential technical, financial, security, and operational problems. However, Five Point never provided this deliverable, for reasons that are somewhat unclear. Department staff confirmed that formal documentation canceling the deliverable does not exist, and a contract administrator stated that Five Point did not provide the assessment because of uncertainty between the department and Five Point about its provision. However, the manager of customer information communication and technology stated that the department’s project sponsors and a Five Point employee indicated that Five Point did not provide the assessment because the contract ran out of money. In addition to the concerns Five Point raised, the department’s In addition to the concerns own internal analysis of go‑live readiness demonstrated Five Point raised, the department’s that it was not prepared to launch CIS. The department and own internal analysis of go‑live PricewaterhouseCoopers used a spreadsheet to track 87 criteria readiness demonstrated that it was in eight categories against set measurements required before not prepared to launch CIS. launching the system. Specifically, they depicted each criterion’s status with a green, yellow, or red dot; all but two criteria required a green dot for launch. Figure 7 on the following page illustrates each category and the proportion of the associated criteria the department rated green, yellow, or red in August 2013. The figure demonstrates that roughly two weeks before launch the project did not satisfy the prerequisites the department had set in order to launch CIS, meaning that the criteria showed the new system was not ready. For example, the department required that CIS have no severity level 1 defects—the most critical classification of defects— prior to launch, yet the readiness criteria reflect that defects were still present in the system. The term defect refers to an issue that causes a system to not perform as expected by negatively affecting its operations. An example might be if a mailing address is needed for a bill and the system failed to print the correct address. The chief information officer asserted that the department had agreed to work‑arounds for all the severity level 1 defects before launch. In testimony before the Los Angeles City Council (city council) in November 2013, three months after the launch, the former general manager seemed unable to soundly defend the department’s decision to launch CIS, stating only, “There comes a point—after dress rehearsals and testing—that you simply have to go live to know in a real working environment exactly how the system works.” It was not until about one year later that the department finally acknowledged how poorly prepared it had been to launch CIS. In a press release about the cause of its troublesome launch of CIS, the department admitted that it had overlooked serious planning and implementation challenges, reduced or eliminated much of the 20 California State Auditor Report 2014-105 March 2015 testing, and left inadequate time to properly prepare and train the staff who would use CIS, resulting in a rushed implementation that caused customer service problems. Figure 7 Status of the Customer Information Connection Project’s Go‑No‑Go Criteria as of August 20, 2013 Source: California State Auditor’s analysis of Customer Information System Connection project (project) go‑no‑go criteria tracking spreadsheet obtained from the Los Angeles Department of Water and Power (department). Note: These data are taken from the department’s assessment of go‑no‑go readiness with respect to launching its customer information system. The data used for this figure were dated roughly two weeks before the launch in early September 2013. Department staff who were significantly involved in the project could not locate any subsequent assessments. The Department Did Not Communicate the Severity of CIS’s Issues to Its Board Given the importance of CIS to the department’s operations and the potential negative effects its improper functioning could have on customers, we question the department’s commitment to transparent communication with its board in the months leading up to the launch. In February 2013 one board commissioner stated that as the department approached integration, he wanted it to keep California State Auditor Report 2014-105 21 March 2015 him informed of the project’s progress so that the board could use its authority to ensure a smooth, successful transition. Although the department was anticipating an April launch at the time the commissioner made this statement, we assume his statement held true regardless of the launch date. However, when we reviewed various department officials’ testimony before the board between February 12, 2013, and June 19, 2013, we found that staff shared little to no specific information about the severity of the project’s defects, the testing results, or the quality assurance contractor’s ratings. In fact, we found a number of instances in which the department appeared to downplay or misrepresent to the board the project’s status at the time. For example, before the board on February 12, 2013, the assistant general manager of the customer services division stated that the project’s scope, schedule, and budget were “tracking appropriately.” In contrast, Five Point’s most recent quality assurance report rated the scope and schedule as needing immediate attention—the lowest possible rating—and the budget was rated as fair. In June 2013 the department’s former general manager informed the board that he was receiving briefings from the project team nearly every week about the systems that they tested, the items that either passed or failed, and what he referred to as the “critical path.” He stated that the project team was working through issues and had whittled down the list of problems materially. Yet, in contrast to the former general manager’s statements, the quality assurance report dated June 7, 2013, showed that testing and go‑live readiness were behind schedule; that the system’s implementation had yet to stabilize, meaning the system had not reached the point at which it was unlikely to substantially change or fail and was running smoothly; and that the system still had a high number of open defects. The department’s failure to The department’s failure to accurately inform the board of the In addition to the concerns accurately inform the board of project’s progress impeded the board’s ability to understand Five Point raised, the department’s the project’s progress impeded the and address potential problems before CIS launched. Had the own internal analysis of go‑live board’s ability to understand and department conveyed the true scope and nature of the project’s readiness also demonstrated that it address potential problems before issues, especially in the context of potential for harm to customers, was not prepared to launch CIS. CIS launched. the board might have chosen to delay the CIS implementation. As a governing body, the board has a fiduciary duty to the department’s customers to help ensure that the department’s decisions and actions are sound. In fulfilling that duty the board is dependent upon the department to be forthright with information. Overall, the department’s lack of transparency put the board and customers at a disadvantage in understanding the causes for the problems involved in launching CIS, and their repercussions, and may have damaged the board’s and department’s reputations. 22 California State Auditor Report 2014-105 March 2015 The Department’s Decision to Minimize Customer Call Wait Times Following CIS’s Launch May Cost It Millions of Dollars Related to Past‑Due Accounts As a consequence of the department’s decision to launch CIS before resolving all of the system’s issues, many customers complained of late and inaccurate bills following its implementation. These customers generally contacted the department by phone to resolve their billing issues, resulting in excessively long call wait times. The long call wait times in turn created additional customer complaints. Under pressure from the public and the city council about customer service and the accuracy of its bills, the department changed its parameters for collecting past‑due accounts, significantly reducing the number of customers with past‑due bills that would otherwise be subject to its collections process. Although this action successfully reduced the department’s call wait times, the reduction came at a cost: Because the department referred fewer customers to collections, it may ultimately have to write off millions of dollars of past‑due bills as bad debt. These unpaid bills are an indirect consequence of the department’s decision to launch CIS before the system was ready. Following Its Launch of CIS, the Department Limited Its Collections Efforts to Shorten Call Wait Times The Joint Legislative Audit Committee (audit committee) asked us to determine the number of customers who have experienced inaccurate or late bills since the department’s launch of CIS. According to the department’s director of customer operations, the department does not knowingly send out inaccurate bills and therefore does not track or publish such statistics. However, the department does track the number of bills it bases on estimates of consumption instead of actual meter readings, and it also tracks the number of delayed bills, which it defines as bills it does not mail to customers on time because of billing issues. The department maintained that it eventually reconciles estimated bills with actual customer consumption, stating that it charges or credits the customer for any difference between its estimates of usage and the actual We were unable to quantify the meter readings. The audit committee also asked us to quantify the number of unwarranted disconnect number of unwarranted disconnect notices and service terminations notices and service terminations associated with the implementation of CIS, but we were unable to do associated with the implementation so because the department does not track this information. According of CIS because the department does to the assistant general manager of the customer services division, the not track this information. department had no record of unwarranted service terminations. Following its launch of CIS, the number of estimated and delayed bills exceeded the department’s performance targets. The department’s performance goal for CIS is to estimate no more than 5 percent of the bills it issues each month, or roughly 42,000 of 840,000 total California State Auditor Report 2014-105 23 March 2015 bills per month. However, in the month following its launch of CIS in September 2013, the department issued more than 137,000 estimated bills, or more than three times its goal. Similarly, the department failed to meet its performance goal for delayed bills. Its data show that it issued an average of more than $100 million in delayed bills for the month of January 2014, whereas the department stated in a presentation to the city council’s energy and environment committee that its delayed bills should represent less than $60 million on a week‑to‑week basis, meaning on average. In recent months, the department has been able to reduce the number of estimated and delayed bills it generates. According to its data, it issued nearly 48,000 estimated bills in September 2014, which is close to its goal of no more than 42,000. Its data also show that it significantly reduced the number of delayed bills it issued to less than $33 million in unbilled revenue for the month of September 2014. Thus, in the department’s view, it is now meeting expectations with respect to delayed bills. However, two class action lawsuits have been filed in the Los Angeles Superior Court by ratepayers, which challenge the manner in which the department attempted to charge for these late bills. The Appendix provides more information about estimated and delayed bills. Yet, despite its progress in reducing the number of estimated and delayed bills it issues, the department has struggled in its efforts to seek The department has struggled in payment from customers who have not paid their bills. According to its effort to seek payment from the department’s data, its customers owed more than $681 million as of customers who have not paid November 2014, an increase of more than $245 million since July 2013, their bills—its customers owed two months before the department launched CIS. The older this debt more than $681 million as of becomes, the less likely the department is to collect it. As Figure 8 on the November 2014, an increase of more following page shows, roughly one‑third of the department’s customer than $245 million since July 2013. debt as of December 2014 was older than 90 days. This represents a significant increase from before the department launched CIS, when customer debt over 90 days old was just under 10 percent of all its customer debt. Between July 2013 and November 2014, debt older than 90 days represented the fastest growing segment in bills awaiting payment, increasing by 425 percent, or $180 million. According to the assistant director of the department’s information technology services division, the department wrote off unrecovered debt that it had classified as bad debt before it launched CIS. As a result, a significant portion of the department’s current debt appears to be directly related to the troubled launch of CIS. This increase in the department’s level of customer debt over 90 days old appears to be due in part to the conflicting relationship between collecting customer debt and reducing call wait times. Call wait times and collecting on past‑due accounts are correlated because customers who receive delinquency notices about their failure to pay often call the department, thus increasing the volume of calls the department’s customer service center receives. The department can control how 24 California State Auditor Report 2014-105 March 2015 many customers enter the collections process by changing certain parameters (such as the amount of debt owed and the number of days a customer must be delinquent) before it issues past‑due notices. The department defines the values of these parameters as collections thresholds. By increasing the collections thresholds, the department puts fewer customers into the collections process, resulting in fewer customers calling to discuss their bills. Figure 8 Customer Bills Awaiting Payment to the Los Angeles Department of Water and Power in Selected Months 91+ days 61–90 days 31–60 days 0–30 days $800 700 600 $205 $202 $42 500 $223 $19 $169 $182 $65 $68 $241 $77 400 $59 $63 $95 $74 $106 $50 $91 $91 $353 $127 $101 300 $325 $334 $286 $284 $255 $264 200 100 0 JULY 2013 MARCH 2014 JUNE 2014 SEPTEMBER 2014 OCTOBER 2014 NOVEMBER 2014 DECEMBER 2014 snoilliM ni sralloD Source: California State Auditor’s analysis of bills awaiting payment data obtained from the Los Angeles Department of Water and Power, unaudited. Since launching CIS, the department has frequently modified its collections thresholds, both in an effort to manage call wait times and in response to the city council. In the three months following the launch, customers’ call wait times averaged approximately 24 minutes, almost eight times the department’s goal of three minutes or less. In late November 2013 the city council requested a moratorium on the department’s service termination activities until the department could demonstrate that it had addressed its billing issues. The department ceased collections activities at that time as well. Consequently, during December 2013 California State Auditor Report 2014-105 25 March 2015 and January 2014, average call wait times dropped to roughly 16 to 19 minutes as compared to November 2013, as shown in Figure 9 on the following page. The department frequently revised its collections thresholds in the months that followed. From February through September 2014, the department changed commercial or residential collections thresholds numerous times in an effort to reduce or increase the number of customers in collections and thereby control call volume. The moratorium on the department’s collections activities lasted until late February 2014, when the department resumed collections activities on its commercial customers. It resumed service terminations for those customers two months later, in April 2014. In June 2014 the department began to gradually resume collections and service termination activities on certain residential customers. Before doing so, the department informed the city council in May 2014 that it would slow down its collections efforts, stating that it would not be fair to shut off services if customers cannot contact the department. As Figure 9 demonstrates, the department’s resumption of residential collections coincided with increases in call wait times, from 14 minutes in May 2014 to nearly 39 minutes during August 2014. The department asserted that a portion of the call wait time increase is, in part, attributable to its “busy season,” when many customers move and request that their power be turned on or off; in addition, these are months with warmer temperatures and higher utility bills. The department’s efforts to manage call wait times by modifying its collections thresholds continued in the fall of 2014. To reduce call volume, the department increased the residential collections thresholds in early September 2014, raising the past‑due amount from at least $1,000, the level it had been in July 2014, to at least $10,000 before initiating collections activities. Coinciding with this one change, the department’s data show that its average call wait times fell by almost 25 minutes. Further, Figure 10 on page 27 illustrates that when the department raised the commercial collections threshold in September 2014 from an amount greater than $250 and 19 days past due to an amount greater than $10,000 and more than 60 days past due, the number of commercial customers entering the collections process plummeted from 2,660 on September 8 to just seven over the following two days. The assistant director of revenue and credit management acknowledged the correlation between the department’s collections efforts and its desire to keep call wait times short. The assistant director stated that customers tried to contact The way the department uses the department through its customer contact center (call center), collections to control call volume but because of the excessively large call volumes, they were not able comes at a price: an increasing to discuss their bills with the department in a reasonable amount of number of past‑due accounts and an time. Nevertheless, as noted earlier, the department’s modifying its increasing level of customers debt. collections thresholds has come at a price: an increasing number of past‑due accounts and an increasing level of customer debt. 26 California State Auditor Report 2014-105 March 2015 Figure 9 Los Angeles Department of Water and Power’s Monthly Average Call Wait Times During 2012 Through 2014 and Changes to Its Collections Thresholds in 2014 AVERAGE HOLD TIME FOR CALLERS 2012 2013 2014 )setuniM nI( sllaC ecivreS remotsuC rof emiT tiaW llaC egarevA segnahC dlohserhT 40 35 30 25 20 15 10 5 Goal: 3 minutes 0 JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC February 25, 2014 September 4, 2014 Resume commercial collections Residential >60 days and $10,000 >90 days and $10,000 September 9, 2014 Commercial >60 days and $10,000 March 26, 2014 Commercial >90 days and $3,000 August 21, 2014 Residential >60 days and $2,500 April 17, 2014 Commercial >60 days and $10,000 July 18, 2014 Residential >60 days and $1,000 April 22, 2014 Commercial >60 days and $5,000 June 2, 2014 April 29, 2014 Resume residential collections Commercial >60 days and $3,000 >270 days and $250 May 16, 2014 Commercial >19 days and $250 Sources: Los Angeles Department of Water and Power’s (department) analysis of call wait times and the California State Auditor’s analysis of collections thresholds changes. Note: Collections thresholds are parameters the department sets (such as the debt owed and the number of days a customer must be delinquent) to control how many customers enter into the collections process. California State Auditor Report 2014-105 27 March 2015 Figure 10 Los Angeles Department of Water and Power’s September 2014 Changes to Its Collections Thresholds for Commercial Customers Debt > $250 and older than 19 days Debt > $10,000 and older than 60 days. THRESHOLDS (collections thresholds goal). CHANGE Reducing call wait times is the priority. 2,660 New Accounts Entering Collections 5 2 September 8, 2014 September 9, 2014 September 10, 2014 Amounts Owed $4,233,736 $131,807 $51,462 Source: California State Auditor’s analysis of collections data and collections thresholds changes information obtained from the Los Angeles Department of Water and Power. The department has taken other steps to address long call wait times. Specifically, it reported that it has hired 118 customer service representatives (representatives) to answer phones at the call center. Thirty‑five representatives started in late July 2014 and 83 started in September 2014. According to the department’s general manager, the representatives receive 10 to 12 weeks of training before they begin work in the call center. The new representatives brought the department’s total number of call center representatives to 420 as of early November 2014. With these additional call center staff, the department hopes to accommodate the expected influx of customer 28 California State Auditor Report 2014-105 March 2015 calls as more people enter the collections process. For November and December 2014, the department reported average call wait times of 3.1 and 5.4 minutes, respectively. The department has also introduced new service features to reduce customer wait times. In late November 2013 the department implemented a virtual hold feature on its phone system that allows customers to arrange to be called back rather than wait on hold. The department’s data indicate that call wait times fell about 13.4 minutes in the first month after it implemented this feature; however, this was also the same period the moratorium on service disconnects began. In addition, in early November 2014 the department introduced a Web‑based program that allows customers to enroll in payment plans. By offering customers a Web‑based resource to address their issues, the department can potentially redirect customers from the call center, thereby reducing call volume and wait times. The Department’s Current High Levels of Customer Debt May Significantly Increase the Overall Cost of Implementing CIS Implementing CIS has already Implementing CIS has already cost the department in excess of cost the department in excess of $187 million, and its customer debt that remains uncollected may $187 million, and its customer debt ultimately add tens of millions of dollars to this price tag. Specifically, that remains uncollected may we calculate that of the $238 million that was more than 60 days ultimately add tens of millions of past due as of November 30, 2014, between roughly $11 million dollars to this price tag. (4.5 percent) and $43 million (18 percent) could be bad debt. We based our calculation on the incremental difference in bills awaiting payment between July 2013 and November 2014 (as shown in Figure 8 on page 24) and conservatively assumed that only a portion of the total will prove uncollectable. The bad debt figures are based on the department’s allowance for doubtful accounts—a reduction in its accounts receivable to reflect an estimate of the debts that customers will not pay—for 2013 and 2014, respectively, in its fiscal year 2013–14 Power Revenue Fund (power fund) financial statements.5 Our estimate seems reasonable given the fact that the department’s data showed that $32.8 million of its debt as of December 2014 was older than 470 days, which represents receivables dating back to early September 2013, just after the department launched CIS and debt this old is unlikely to be recovered. The amount of the department’s bad debt may actually be higher. The department significantly increased its allowance for doubtful accounts in the power fund for fiscal year 2013–14— the year in which it began using CIS—from $18.9 million in 5 The Los Angeles City Charter authorizes the department to maintain two funds—the Water Revenue Fund and the power fund. California State Auditor Report 2014-105 29 March 2015 fiscal year 2012–13 to $90 million in fiscal year 2013–14. In notes it prepared for an August 2014 presentation before the city council’s energy and environment committee, department staff stated that the department would maintain the increase until it had a process to write off its bad debts. As a result, the department’s total cost for The department’s total cost for implementing CIS could range from $198 million to $231 million implementing CIS could range from after including our estimates of bad debt, and these costs could $198 million to $231 million after ultimately prove to be even higher if the department’s estimate of including our estimates of bad $90 million as an allowance for doubtful accounts holds true. The debt, or higher if the department’s department has not clarified whether it will pass these additional estimate of $90 million as an costs onto its customers in the form of higher utility rates or absorb allowance for doubtful accounts them through its existing financial resources. holds true. The department has developed a long‑term plan to gradually collect on outstanding customer debt while also minimizing its call wait times. In its October 2014 update to its revenue collection action plan (update document), the department stated that during November and December 2014 it planned to focus on pursuing collections from residential customers who owed more than $5,000 for at least 60 days and commercial customers that owed more than $1,000. Over time, the department plans to gradually change its collections thresholds so that by July 2015 it can focus on residential and commercial customers that have owed more than $250 for at least 19 days, which were its collections thresholds before launching CIS. The department’s ability to realistically hold to its timeline will depend on a variety of factors, such as whether it effectively deploys additional call center staff and whether it revises its collections thresholds as currently planned. The department’s update document also outlines how it will help customers become current on their bills. From November 2014 to March 2015 the department plans to hold Customer Service Saturdays to help customers set up payment plans and learn about their options for paying their past‑due bills. For example, the update document states that from January to March 2015 the department will provide customers who have debt the option of a level payment plan that will allow them to pay a set amount each month until they become current on their bills. The department also intends to implement and invest in tools that incentivize customers to pay, including resuming late payment charges and contracting with collections agencies. It will also begin credit reporting. Nevertheless, the department’s efforts to collect from customers could be compromised if it does not follow through by terminating service to those customers who have repeatedly failed to pay after receiving disconnect notices. The collections and service termination processes are depicted in Figure 3 on page 9. Even though the department has resumed service terminations for its commercial and residential customers, the number of service terminations remains far below the levels before CIS was launched. 30 California State Auditor Report 2014-105 March 2015 In July 2013, two months before launching CIS, the department shut off service to 3,537 accounts and collected more than $1.2 million from customers in the field. As Figure 11 demonstrates, in September 2014 the department terminated service to fewer than 1,000 accounts and collected slightly more than $930,000 because it had suspended service terminations for 11 days that month in an effort to reduce call wait times. According to the department’s update document, it had more than 21,000 accounts active in the service termination process in October 2014. These accounts have more than $30 million of uncollected revenue associated with them. Figure 11 Service Terminations and Field Collections for Selected Months Moratorium on service terminations from Field collections made November 22, 2013, through April 21, 2014 Services terminated Amounts collected $1,543,799 $1,228,337 5,000 $938,599 4,000 1,017 $790,016 $641,384 3,537 482 3,000 3,100 778 2,000 2,204 304 1,000 146 938 176 0 July 2013 June 2014* July 2014 August 2014 September 2014† noitanimreT ecivreS rof elbigilE snoitacoL reteM fo rebmuN Source: California State Auditor’s analysis of collections data and service termination information obtained from the Los Angeles Department of Water and Power (department). Notes: The department reinstated its service termination process in April 2014 and began collecting data on May 19, 2014. June 2014 is the first full month of available severance data after the launch of the customer information system. * According to the department, collections numbers are low in June because of many vacancies in field collections and implementation of a new work management system, and because the department temporarily suspended field collections to reduce call wait times. † The department did not perform field collections or service terminations for a portion of September in an effort to reduce call wait times. California State Auditor Report 2014-105 31 March 2015 Despite the department’s challenges with both collecting on past‑due accounts and minimizing customer call wait times, it has informed the city council that its finances remain strong. In fact, in In December 2014 the board December 2014 the department obtained the board’s approval to approved a transfer of $265.6 million transfer $265.6 million from the power fund to the Reserve Fund of from the power fund to the city’s the city of Los Angeles (city), based on the department’s power fund Reserve Fund. audited financial statements for fiscal year 2013–14.6 For context, the department transferred $246.5 million and $253 million to the city’s reserve fund in fiscal years 2012–13 and 2013–14, respectively. The department budgets the transfer amount as 8 percent of its accrued revenue for the prior fiscal year. We noted that the department’s power fund audited financial statements for fiscal year 2013–14 demonstrate that the department has substantial cash balances: It had nearly $776 million in unrestricted cash and cash equivalents as of June 30, 2014. This amount is equivalent to roughly 27 percent of the department’s power fund operating expenses for fiscal year 2013–14. CIS Cost Far More to Implement Than the Department Originally Budgeted Because of the Project’s Complexity and the Need for Significant Involvement From the Department’s Personnel In late August 2014, a consultant for the department issued a report analyzing the root causes behind the department’s troubled launch of CIS. The consultant specifically observed that the project’s scope was “far too ambitious” with “multiple large scale applications” that put successful implementation at risk from the start. This complexity, which required more involvement from department personnel than it originally budgeted, contributed to the project’s significant cost increases. In fiscal year 2009–10, the department budgeted $86.7 million for the project over a five‑year period ending in fiscal year 2013–14. Of this total, the department budgeted $13.3 million for its own personnel costs. However, as shown in Table 2 on the following page, by the end of fiscal year 2013–14, the department had spent more than $174 million (roughly double the original $86.7 million total budget), of which $52.8 million was for its personnel costs (nearly 400 percent of the original $13.3 million budget for labor). In addition, the department spent significantly more on external costs, such as outside professional services (contractors and consultants), materials, and supplies. 6 According to the Los Angeles City Charter, the reserve fund includes funding for unanticipated expenditures and revenue shortfalls in the city’s general fund. 32 California State Auditor Report 2014-105 March 2015 Table 2 Budget and Actual Costs for the Customer Information System Connection Project Fiscal Years 2009–10 Through 2014–15 (In Thousands) FISCAL YEARS 2009–10 THROUGH 2013–14 FISCAL YEAR 2014–15 ACTUAL COST THROUGH ACTUAL SEPTEMBER BUDGET COST DIFFERENCE BUDGET 2014 DIFFERENCE Department Resources: Budget and Actual Costs Labor (including regular, overtime, and employee benefits) $36,177 $52,757 ($16,580) $0 $1,808 ($1,808) Equipment/facilities/overhead/other 14,938 18,577 (3,639) 0 410 (410) Subtotals $51,115 $71,334 ($20,219) $0 $2,218 ($2,218) Contracts: Budget and Actual Costs Professional services (including PricewaterhouseCoopers and $102,818 $76,199 $26,619 $10,353 $11,142 ($789) Five Point Partners, LLC contracts) Materials and supplies (primarily software and hardware purchases) 16,010 25,926 (9,916) 0 9 (9) Construction, utility, and other contract‑related services 546 710 (164) 0 8 (8) Subtotals $119,374 $102,835 $16,539 $10,353 $11,159 ($806) Grand Totals $170,489 $174,169 ($3,680) $10,353 $13,377 ($3,024) Sources: California State Auditor’s analysis of the Los Angeles Department of Water and Power’s (department) Customer Information System Connection project’s (project) internal budget documents and accounting reports. Note: Although not shown above, the department prepared an initial project budget spanning multiple years, updating the budgeted and projected costs for the entire project each year. Figure 12 presents the project’s initial total budget, the total of the revised project budgets, and total project costs. Further, similar to its lack of transparency discussed earlier regarding the project’s schedule and readiness for launch, the department was not transparent with its board regarding cost overruns on the project. By September 2014 the department had spent $187.5 million on the project, overrunning both its initial and revised budgets, as Figure 12 shows. Board members expressed concern in June 2013 about escalating costs, but the former general manager responded by stating that the project was well within an expected budget and time frame for such a complex project. In addition, several months earlier in February 2013, the department’s assistant general manager of customer services said that the project’s budget was “tracking appropriately.” Through fiscal year 2013–14 the department’s own labor costs on the project had the highest budget overage in terms of dollars spent. The fact that the department incurred most of these labor costs before September 2013, when it launched CIS, reflects its significant responsibilities toward the project’s implementation. The department’s contract with its systems integrator, PricewaterhouseCoopers, identified that the preferred California State Auditor Report 2014-105 33 March 2015 approach to the interfaces between Oracle’s Customer Care and Billing (CC&B) system software and the department’s numerous other systems was middleware, or software that connects various other data systems.7 In its contract with PricewaterhouseCoopers, the department took responsibility for interfacing its existing systems to the middleware, while PricewaterhouseCoopers was responsible for interfacing CC&B and certain other new systems to the middleware. Further, the contract stated that the department would develop the middleware with design and technical support from PricewaterhouseCoopers. Figure 12 Customer Information System Connection Project’s Budgets and Costs for Fiscal Years 2009–10 Through 2014–15 $200 $187.5 $180.8 180 160 140 120 100 $86.7 80 60 40 20 0 Total of Initial budget* revised budgets† Total costs‡ snoilliM nI sralloD Sources: Los Angeles Department of Water and Power’s (department) internal budget documents and accounting reports for the Customer Information System Connection project. Note: We did not audit the accuracy and completeness of the department’s cost information; instead we compared the total cost to the city of Los Angeles’ budgets for reasonableness. * The initial budget includes fiscal years 2009–10 through 2013–14 as of August 10, 2010. † The total of revised budgets includes fiscal years 2009–10 through 2014–15 as of November 6, 2014. ‡ The total costs includes those incurred from July 2009 through September 2014. 7 Figure 4 on page 11 provides an overview of the project and the various systems that had to be successfully interfaced with CC&B. 34 California State Auditor Report 2014-105 March 2015 The degree of investment of its own personnel in the project appears to have been greater than the department originally anticipated. The department provided us with a list of nearly 140 department employees who worked on the project—nearly double the more than 70 individuals who worked for PricewaterhouseCoopers. In June 2013 the department’s former general manager testified before the board explaining the need to extend its contract with PricewaterhouseCoopers through a $12 million contract amendment. Specifically, the former general manager explained the key role the department played, stating, “[Things weren’t] happening as quickly on our end. . . . I’ll call it what it is. . . . These are massive types of changes. . . . Our goal and our plan was to get a lot of things done a lot earlier than they were . . . and to be straight up with you, we didn’t get it done. . . . In terms of running the business, we’ve been understaffed on customer service. . . . I’ve approved staffing. It takes us nine to twelve months to bring a staff person on in the department and working through city personnel. . . . As a result, we’ve been working understaffed on trying to get some of this done, and we’ve been working overtime on getting this done, and our consultants can only move as fast as we partner with them.” The department originally budgeted Aside from its growing personnel costs, the department also $64.7 million in spending through experienced significant cost increases for outside professional fiscal year 2013–14 for outside services, materials, and supplies. The department originally professional services, materials, budgeted $64.7 million in spending through fiscal year 2013–14 and supplies but actually spent but actually spent $102.1 million (an increase of 58 percent). For $102.1 million. example, the department spent $18.5 million more for materials and supplies than it had originally budgeted. It also amended its contracts with its quality assurance consultant, Five Point, and with PricewaterhouseCoopers, thus increasing its costs by a total of $18.2 million. Table 3 summarizes the department’s primary contracts pertaining to the project. The department amended its contract with Five Point four times, increasing the total value from $3.5 million to $9.7 million. The cost increases accompanied additional scope of work items that went beyond providing quality assurance services. For example, in December 2010 the department added $1.45 million to its contract for Five Point to provide additional support to department managers involved with the project and to ensure that the department met its business objectives related to interfaces, conversion, testing, and reporting activities. In November 2011 the department executed a second amendment, adding $2.8 million to the contract for training support. The department’s third amendment did not increase the contract’s value but extended it for roughly eight months because the timeline for launching CIS had slipped. Finally, in March 2013, the department amended the Five Point contract a fourth time, adding six months and $2 million without adding to Five Point’s scope of work. When explaining California State Auditor Report 2014-105 35 March 2015 the need for the fourth amendment in its letter to the board, the department stated that it wanted to ensure that Five Point’s quality assurance, training, and functional support services were available through the post‑implementation stabilization period. Table 3 Contract and Amendment Amounts for the Customer Information System Connection Project’s Systems Integrator and Quality Assurance Consultant (In Thousands) ORIGINAL ACTUAL COST CONTRACT CONTRACT THROUGH AMOUNT AMENDED TOTAL SEPTEMBER 2014 DIFFERENCE PricewaterhouseCoopers $57,200 $12,000 $69,200 $68,239 $961 (Systems Integrator) Five Point Partners, LLC (Quality Assurance 3,500 6,209 9,709 9,684 25 Consultant) Totals $60,700 $18,209 $78,909 $77,923 $986 Sources: Los Angeles Department of Water and Power’s (department) contracts and amendments with PricewaterhouseCoopers and Five Point Partners, LLC, and cost information provided by the department’s assistant chief financial officer. In addition to increasing the value of its contract with Five Point, the department increased PricewaterhouseCoopers contract by $12 million in June 2013, when it extended the contract term by two years. When explaining the need for the amendment to its board, the department asserted that it needed extended support for significant deferred issues and tasks, unanticipated system problems, and additional adjustments to system processes after CIS’s launch. For example, the department anticipated that it would want to adjust the system after its launch from bimonthly billing to monthly billing. The chief information officer informed the board that the department and PricewaterhouseCoopers had worked to set up the system for monthly billing but needed more time to test and implement it. The former general manager stated that the department had not originally included the transition to monthly billing in the scope of work for the project but hoped to add it. Before voting to approve the amendment, the president of the board stated that he was concerned about the $12 million price tag but agreed that the department needed monthly billing. He then warned the department against submitting any additional amendments for the PricewaterhouseCoopers contract. As with the project’s schedule and the cost overruns, the board’s ability to question department management about its contracting activity was limited by the poor information the department’s executive 36 California State Auditor Report 2014-105 March 2015 management provided to it. As Figure 13 shows, the board approved the department’s amendments and contracts valued at $100,000 or more throughout the project. Nevertheless, in our opinion, the department’s management was not forthcoming with the board regarding the project’s budget. For example, in a February 12, 2013, board meeting, the assistant general manager of the customer services division stated that the budget was “tracking appropriately” for the Five Point and PricewaterhouseCoopers contracts. In another instance, during the June 19, 2013, board meeting, one commissioner expressed that it seemed like the project was out of control and that he did not see an end. The former general manager responded that although the project was taking longer than expected, it was on a path to be well within an expected level of budget. We question whether the former general manager would have made such comments had he or his management team provided the board with the budget and actual cost data showing the continued pattern of increased project costs. Other factors may have also contributed to the board’s inability to adequately monitor the project’s budget. Specifically, by September 2013, the city’s mayor had appointed four new commissioners (out of a total of five) with the approval of the city council, substantially reducing the board’s institutional knowledge The changing makeup of the board, on the evolution of the project. The changing makeup of the coupled with poor project status board, coupled with poor project status information, did not information, did not place the place the board in a strong position to hold department managers board in a strong position to hold accountable for a long‑term information technology project. To department managers accountable ensure appropriate oversight, department management needed for a long‑term information to consistently disclose to board members project cost information, technology project. communicate project schedule changes, and highlight risks that could ultimately harm business operations if not properly and promptly addressed. We noted that the department is planning to replace other computer systems, such as its financial information system, in the future. This presents the board with an opportunity to establish reporting practices that ensure greater disclosure of the financial and project status of the department’s large‑scale, multiyear information technology projects. Had board members been properly informed on the project, they would have had an opportunity to challenge the department’s now publicly stated rationale for launching CIS when it did. The department now claims its legacy billing system had become so fragile that there was a sense of urgency to implement CIS. However, had the board been made aware of the project’s rapidly increasing costs and outstanding defects, it might have challenged the department by asking why it was not better to continue using TRES—its existing 40‑year‑old billing system—for another six to 12 months, until the department could mitigate the risks of using CIS. We believe the board should have had an opportunity to at least ask this question and more fully understand the risks associated with the project. California State Auditor Report 2014-105 37 March 2015 Figure 13 Los Angeles Board of Water and Power Commissioners’ Approval Dates of Contracts and Amendments With PricewaterhouseCoopers and Five Point Partners, LLC 2009 August 4, 2009 The Los Angeles Board of Water and Power Commissioners (board) approved the Los Angeles Department of Water and Power’s (department) contract with Five Point Partners, LLC (Five Point) 2010 to provide quality assurance services for the Customer Information System Connection project (project) for $3.5 million. Specifically, the quality assurance consultant was responsible for monitoring and reporting on the quality of the work done by the department’s staff and the systems integrator contractor. August 3, 2010 The board approved the department’s contract with PricewaterhouseCoopers to provide professional services to assist the department in selecting and December 7, 2010 2011 implementing a new customer information system The board approved the department’s first (CIS) for $57.2 million over three years. amendment to its Five Point contract—increasing the contract amount by $1.5 million—to provide functional expertise, help build and implement a training program, and support the management of the client side of activities, including process design, configuration, and testing for CIS implementation. November 1, 2011 2012 The board approved the department’s second amendment to its Five Point contract—increasing the contract amount by $2.8 million—to further support the creation and implementation of the department’s training program. September 5, 2012 The department amended its contract with Five Point for the third time, extending the contract duration March 19, 2013 to provide consistency in the quality assurance 2013 services through CIS’s launch. This amendment did The board approved the department’s fourth not increase the contract’s monetary value. amendment to its Five Point contract, increasing the contract amount by $2 million and extending the contract through October 31, 2013, to ensure that quality assurance, training, and functional support June 19, 2013 services were available throughout and after CIS’s launch. The board approved an amendment to the PricewaterhouseCoopers contract, increasing the contract amount by $12 million and extending the September 2013—Launch contract through August 12, 2015, because the department needed support for significant deferred items, unanticipated system problems, and/or additional modifications to configuration. Sources: The department’s contracts and amendments with PricewaterhouseCoopers and Five Point. 38 California State Auditor Report 2014-105 March 2015 As it currently stands, the department has spent in Root Causes of the Customer Information System’s Instability excess of $187 million, and its costs will continue to accumulate as it attempts to resolve system According to TMG Consulting, Inc.’s (TMG) root cause defects and collect on bills that its customers may analysis, the following are the eight causes of instability ultimately never pay. As of September 30, 2014— in the Los Angeles Department of Water and Power’s only three months into fiscal year 2014–15—the (department) customer information system (CIS): department had already exceeded its project budget 1. Absence of engaged project management: for the year by $3 million. Although the department The department decided to begin using CIS despite did not budget for its own resources to be overwhelming evidence that neither the system participating in the project, the information nor the department was prepared. The department technology services utility administrator indicated did not have a detailed project plan to manage and that department staff have continued to charge track project status. their time to the project in an effort to fix issues 2. Project scope too extensive: The project’s with CIS. Further, the department nearly exhausted original project scope was far too ambitious and its contract with PricewaterhouseCoopers within used several newer, untried technologies. The the first three months of fiscal year 2014–15, even department also made problematic scope changes though that contract was to provide the department with respect to implementing monthly billing. with necessary technical support until August 2015. 3. Lack of experienced resources: Neither In fact, only 1 percent of the contract’s total value of PricewaterhouseCoopers’ staff nor the department’s $69.2 million remained after September 2014. The staff had ever managed a project this big or complex. department recently released a request for proposal for CIS support and upgrade services, which it 4. Critical project deliverables never produced: valued at between $13 million and $15 million and The department allowed the project to advance despite incomplete project tasks. It did not ensure which contains elements that appear to be that it followed and managed contract terms. substantially similar to the scope of work in PricewaterhouseCoopers’ contract. 5. Data conversion validation never executed: Its poor data conversion efforts resulted in a high volume of notices requiring staff intervention and, when not The Department Is Still Attempting to Resolve addressed, caused a high volume of estimated bills. Defects That Existed During CIS’s Development 6. Mandatory code freeze never enforced: The department and its contractors introduced In November 2014 the department made public weekly multiple code releases right up to the system a report by TMG Consulting, Inc. (TMG), a launch, without testing to assess the impact of the utility‑focused information technology advisory fixes on previous testing. firm. The report represents TMG’s root cause 7. Minimal financial testing performed: analysis of what went wrong with the project.8 Its The department tested only one of its 21 billing analysis confirmed that the department’s decision cycles before launch. The lack of testing caused to launch CIS at the time it did was a poor one. incorrect bills and lost revenue. The text box summarizes the eight causes TMG 8. Workforce unprepared for transition after going identified as contributing to the launch’s problems. live: The department never advanced its training The report also identifies the department’s beyond the introductory level, largely because late ineffectiveness at addressing root causes during code delivery and testing complications inhibited CIS’s first year of operation as a reason that the preparation of training materials. “collections have hit unmanageable levels,” and it Source: TMG’s report titled Los Angeles Department of Water and describes CIS as in a state of “continuing instability,” Power Approach for CCB/MWM Stabilization Root Cause Analysis, which it largely attributes to defects. dated August 25, 2014. 8 Five Point acquired TMG in February 2009, and TMG regained independence from Five Point in May 2014. Five Point was the department’s quality assurance consultant for the project. California State Auditor Report 2014-105 39 March 2015 The department categorizes defects by severity level, with severity level 1 being the most critical. The report concludes that “[t]he number of open defects remains extremely high, with no quantifiable progress in reduction during the nearly [one] year since system [launch]. Dozens of severity level 1 defects still exist, indicating that the fundamental problems at [launch] (as demonstrated by open severity level 1 defects at that time) have not been resolved.” According to TMG’s report, “the number of defects at [launch was] a clear indication [CIS] was not close to ready to move into production.” TMG’s observations about the number of lingering severity level 1 defects mirror our own analysis. Figure 14 summarizes the number of defects the department was tracking in its defect database from two weeks before it launched CIS through one year after. Overall, the figure shows that the number of defects dipped on August 29, 2013, the last date for which the department has data available before CIS launched, according to the senior systems analyst. The number of defects then increased after launch. Figure 14 Customer Information System’s Unresolved Defects Before and After Its Launch Severity Level 1* Severity Level 2† September 3, 2013 Severity Level 3‡ Launch Unclassified 600 2 113 500 12 64 400 108 82 315 300 12 343 189 35 77 309 200 73 184 31 100 111 43 104 98 29 63 29 33 0 AUGUST 14, 2013 AUGUST 29, 2013 SEPTEMBER 12, 2013 OCTOBER 9, 2013 MARJCUHN 1220,1 24014 SEPTEMBER 15, 2014 stcefeD fo rebmuN Source: California State Auditor’s analysis of the Los Angeles Department of Water and Power’s (department) defect data. * Severity level 1: A programmatic or configuration defect that renders a major component of the system unusable or inoperable. This error is a loss of capability to perform a critical business function. A short‑term solution does not exist. † Severity level 2: Same as severity level 1; however, there is a temporary work‑around or solution. ‡ Severity level 3: Program or configuration errors limit the capability of the application or function, but there is a practical work‑around or the defect does not affect the department’s operation of the application in any significant respect. 40 California State Auditor Report 2014-105 March 2015 The department has not used consistent definitions when identifying defects, which causes difficulty when comparing the types and number of defects over time. The senior systems analyst explained that the department began applying a stricter definition to severity level 1 defects closer to CIS’s launch in an effort to meet go‑live acceptance criteria; he asserted that the current number of severity level 1 defects would be lower if the department had continued to apply the more stringent definition of severity level 1 defects in the PricewaterhouseCoopers contract. According to TMG’s report, its “review of the [department’s] approach to defect classification identified inconsistencies in the application of severity level (1, 2, etc.). This combined with the varied use of priority (critical, high, medium, etc.) tends to cloud the true significance of the defect.” Although limitations exist in the defect data we received, the department’s defect tracking documents show instances in which certain severity level 1 defects that existed just before the launch continued to be tracked following the launch. As shown in Figure 14 on the previous page, there were 82 severity level 1 defects on August 14, 2013, less than a month before the launch. By August 29, 2013, the department identified only 21 of these 82 defects as severity level 1; it had downgraded another 18 to severity level 2 and presumably closed the other 43. Further, the day before the launch, the department’s defect system reported no open severity level 1 defects, and five high‑ranking department officials—including the project directors and two project sponsors—signed certifications that PricewaterhouseCoopers had completed testing of the new system and had developed resolutions for all severity level 1 and 2 defects. This action seems consistent with the chief information officer’s statement that the department Defect Status had agreed to work‑arounds for all the severity 1 defects before launch as noted earlier. Before implementing software code changes necessary to address a defect in its customer information system Nevertheless, by October 2013, the department had (CIS), the Los Angeles Department of Water and Power resumed tracking 12 of the 21 defects it had listed (department) subjects those changes to a review process and uses different statuses to indicate the defect’s progress as severity level 1 defects on August 29, 2013. Its through the review process. defect tracking extract lists eight of the 12 defects as fixed and the remaining four as downgraded Open: Indicates a test team has begun working to resolve to severity level 2. Two of these were listed as the defect. retested, and the remaining two were listed as fixed Fixed: The defect has passed the test team’s initial testing. and reopened. The text box describes the different Retested: The test team has reviewed the defect at defect statuses the department used. The status least twice. of fixed did not necessarily mean that the department had successfully resolved a defect. Reopened: The defect is unresolved and needs additional The department’s process for addressing defects software code to fix it. and deploying corrections requires the functional Source: The department’s CIS Deployment of Defect Fixes for CCB/ test team to verify the change and then obtain MWM/Reports Process Flow to Production (PRD) v2.7. approval from the department’s Release Approval Committee. However, it is unclear whether California State Auditor Report 2014-105 41 March 2015 the department consistently followed this process. Regardless, three of the four severity level 2 defects continued through March 12, 2014, and two of these defects remained outstanding as of September 15, 2014. Our review also found that the department later changed the statuses of many of the 18 severity level 1 defects it downgraded to severity level 2 between August 14 and August 29, 2013. By September 12, 2013, just after the CIS launch, 17 of these 18 defects were presumably closed. However, by October 9, the department had changed the status of 16 of these 17 defects to retested, reopened, open, and fixed. The department’s defect system data show that four of these 17 continuing defects remained outstanding nearly a year later, in September 2014. The department’s recent steps to address CIS’s remaining defects The department’s recent steps to and other system issues potentially involve its entering into another address CIS’s remaining defects multimillion‑dollar contract. Specifically, in November 2014— and other system issues potentially immediately on the heels of presenting the TMG report to the involve its entering into another board—the department issued a request for proposal for “assessing, multimillion‑dollar contract— prioritizing, planning, implementing, and documenting changes to valued at between $13 million and various customer information systems, subsystems, and associated $15 million over three years. administrative functions, in order to bring about optimizations in a prompt and timely manner.” The department valued the work outlined in the request for proposal at between $13 million and $15 million over three years. We inquired with the department’s former senior assistant general manager of the power system about the extent to which the request for proposal related to the TMG report and whether its scope of work duplicated the project work the department had already contracted with PricewaterhouseCoopers to complete. According to the former senior assistant general manager, no direct relationship exists between the TMG report and the request for proposal. In addition, he stated that the request for proposal was for new work, asserting that department management signed off on PricewaterhouseCoopers’ work after determining that the contractor had completed its obligations. The former senior assistant general manager stated that the department had a number of planned CIS enhancements it wished to complete and ongoing maintenance issues that it wished to address, and that it believed having outside professional support services would be prudent. He cited three projects, including a new financial and human resources system, as new work the department will proceed with as part of the request for proposal. However, we question the department’s characterization of its request for proposal as new work. Our IT expert reviewed the PricewaterhouseCoopers contract and associated request 42 California State Auditor Report 2014-105 March 2015 for proposal, the TMG report, and the most recent request for proposal. Our IT expert noted that the request for proposal includes an assessment of the current state of CIS—which has been in use for less than two years—and asks for recommended process improvements to bring the business in line with industry best practices and city mandates. The request for proposal also tasks the winning bidder with implementing necessary optimizations that support, among other things, an effective collections process that generates timely and accurate notices and collections information that supports the creation of customer bills that have accurate, up‑to‑date rates and meter readings. He stated that the request for proposal suggests that the original implementation was incomplete and incorrect and that it requires remediation, which the request for proposal intends to address. The Department Generally Met Proposal and Contracting Requirements but Needs to Strengthen Its Policy Regarding Conflicts of Interest A significant portion of the department’s spending on contractors for the project pertained to just three vendors: the department’s systems integrator, PricewaterhouseCoopers; the department’s quality assurance consultant, Five Point; and the department’s software vendor, CompuCom Systems, Inc. The department generally followed the city’s and its own contracting requirements for these three contracts. However, the department could not demonstrate that it followed its conflict‑of‑interest policy to fully screen all employees involved in the contracting process for potential conflicts of interest.9 In general, the department adhered to reasonable procurement Out of the $110 million the practices for its three largest contracts. Out of the $110 million department paid to contractors for the department paid to contractors for the project through the project through September 2014, September 2014, $93.3 million (or roughly 85 percent) were roughly 85 percent were payments payments to PricewaterhouseCoopers, Five Point, and CompuCom for its three largest contracts. Systems, Inc. The department advertised and sought proposals for all three of these contracts, obtained proposals from multiple vendors, and based its vendor selection decisions on its scoring of the proposals it received. We also noted that the department obtained approval to execute these three agreements from both the board and the city council when policies required it to do so. In particular, we noted that the department informed the city council 9 The department’s conflict‑of‑interest policy is separate from the conflict‑of‑interest code it is required to adopt pursuant to the State’s Political Reform Act of 1974 (act). That conflict‑of‑interest code designates which department employees must report their interests under the act. California State Auditor Report 2014-105 43 March 2015 of the amended and total values of its revised contracts with both PricewaterhouseCoopers and Five Point before it launched CIS in September 2013. However, the department did not consistently follow the board’s The department did not consistently policy regarding conflicts of interest. Specifically, in 2008 the board follow the board’s policy regarding adopted a policy requiring employees to self‑certify whether they conflicts of interest. have potential conflicts of interest regarding specific contracts. The board’s policy requires employees who report a potential conflict of interest to recuse themselves from participating in that contract process. A 2008 manager’s bulletin (2008 bulletin)— which operationalizes the board’s policy—requires both employees who are close to the contracting process and their supervisors to participate in this screening procedure. The 2008 bulletin identifies three stages in the contracting process during which employees should make their certifications: proposal development, bid evaluation, and contract award. The department does not appear to have fully implemented the board’s policy. Although it was able to provide copies of the certifications for some employees who were involved in evaluating and ultimately selecting the winning vendors, the department had no record of certifications for the employees who played significant roles in drafting the requests for proposal or contract award. Specifically, the department’s contract administrator for the PricewaterhouseCoopers and Five Point contracts had to recollect—several years after the fact—which employees played what roles in those contracting stages. Without a clearly defined universe of the department employees who played a significant role in the contracting process, the department could not fully demonstrate, and we could not fully assess, whether it had adhered to the board’s conflict‑of‑interest policy. The department’s former ethics liaison indicated that she was typically unaware of the contracts the department was developing until it scheduled them for board presentation and approval. She explained that the various units within the department involved with a contract and the supply chain services unit—the unit that oversees and processes the department’s contracting and procurement activities—would identify the individuals who needed to prepare conflict‑of‑interest certifications and then submit the completed certifications to her. The former ethics liaison also asserted that once she received the certifications staff would compare them to any outside employment forms on file and to the voluntary and mandatory financial disclosure statements specified in state law. 44 California State Auditor Report 2014-105 March 2015 In January 2015 the director of supply chain services released standard operating procedures (procedures) for requests for proposal. The procedures outline the tasks staff are to perform during the three contracting stages of proposal development, bid evaluation, and contract award and include steps in each stage related to conflicts of interest. However, the procedures fall short of fulfilling the requirements outlined in the 2008 bulletin discussed previously in that they do not require conflict‑of‑interest certifications from all necessary employees. For example, the 2008 bulletin requires a conflict‑of‑interest certification from the manager or supervisor over the proposal requestor/contract administrator, and the newly updated procedures are silent in this regard. As a result, although the new procedures add clarity to the conflict‑of‑interest process and specify that some employees must complete certifications at all three contracting stages, the department still lacks a process that meets the board’s expectations for guarding against potential conflicts of interest. To better identify potential conflicts, given the department’s limited implementation of its board’s policy, we reviewed the financial disclosure statements filed by certain department employees and board members who played an important role in approving the department’s contract with PricewaterhouseCoopers. State law requires public agencies to adopt a conflict‑of‑interest code and to ensure that certain public officials, including those involved in contract negotiation and formation, file financial disclosure statements in which they report their economic interests (interest statements). Reportable economic interests include investments, real property, sources of income, and business positions. State law prohibits public officials from making, participating in making, or attempting to use their official positions to influence governmental decisions in which they have an economic interest. When we reviewed the interest statements of department employees and board members involved in the negotiation and formation of the PricewaterhouseCoopers contract, we found that all but one filed their interest statements in a timely manner. The individual who did not file an interest statement had signed the transmittal letter when the contract was sent to the board. Given that he did not file an interest statement for the period when this contract was approved, we could not determine whether he may have had any financial interests that were affected by the approval of the contract. In general, the interest statements revealed no material financial interests that raised a concern under applicable conflict‑of‑interest laws. California State Auditor Report 2014-105 45 March 2015 Recommendations To ensure that the board can more effectively exercise oversight for the department’s significant information technology projects, the board should take the following actions: • Establish a standing committee comprised of board members to oversee and critically evaluate the status of the department’s various information technology projects. Given the limited tenure of board members and the potential for multiyear and high‑cost information technology projects, the board president should consider appointing as many committee members as practicable in order to promote continuity of oversight. • Develop reporting standards for the department’s management to follow when discussing the status of information technology projects with the standing committee or the board. Such reporting standards should, at a minimum, specify the frequency with which the department’s management makes such reports and require the following disclosures about each information technology project: – The amount of project growth, in terms of both budget and scope of work, from initial project estimates through current projections. – The results from system testing and a listing of the critical defects that exist and must be fixed prior to system use. – The concerns the quality assurance contractor has raised and how the department is addressing them. • Develop a process for the board to designate certain information technology projects as having a potentially significant effect on business operations or customer relations, and require that department managers first obtain the board’s approval before launching such critical new systems. To ensure that the department can demonstrate compliance with the board’s conflict‑of‑interest policy, the department should develop and implement a process by June 30, 2015, that accomplishes the following for each contract: • Results in a centralized listing of all employees participating in each stage of the contracting process (proposal development, bid evaluation, and contract award), and who are required to submit conflict‑of‑interest disclosure forms per the board’s policy. • Results in a central depository of the conflict‑of‑interest certifications submitted by each employee. 46 California State Auditor Report 2014-105 March 2015 We conducted this audit under the authority vested in the California State Auditor by Section 8543 et seq. of the California Government Code and according to generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives specified in the scope section of the report. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Respectfully submitted, ELAINE M. HOWLE, CPA State Auditor Date: March 10, 2015 Staff: Grant Parks, Audit Principal Sharon L. Fuller, CPA Kathryn Cardenas, MPPA Brianna J. Carlson Jordan Christenson, MPP Michelle J. Sanders IT Expert: Catalysis Group, Inc. Legal Counsel: J. Christopher Dawson, Sr. Staff Counsel For questions regarding the contents of this report, please contact Margarita Fernández, Chief of Public Affairs, at 916.445.0255 California State Auditor Report 2014-105 47 March 2015 Appendix ESTIMATED AND DELAYED BILLS FOLLOWING IMPLEMENTATION OF THE CUSTOMER INFORMATION SYSTEM The Joint Legislative Audit Committee directed the California State Auditor to determine the number of customers that experienced inaccurate or late bills as the result of the Los Angeles Department of Water and Power’s (department) implementation of its new customer information system (CIS). We were able to obtain information on late bills (which the department refers to as delayed bills). However, we were unable to quantify the number of inaccurate bills because the department does not track or publish such statistics; the department asserted that it does not knowingly send out inaccurate bills. Instead, the department monitors the number of bills it bases on estimates of customers’ water and power usage because it is unable to obtain accurate meter readings. We describe the department’s process for estimating bills in the Introduction, and Figure 2 on page 7 provides an overview of the department’s billing process, including meter reading. The department sets certain performance goals related to how often it issues estimated or delayed bills. Further, it tracks these numbers as overall measures of the quality of its billing system. Estimated Bills When the department launched CIS in September 2013, it issued a high number of estimated bills and exceeded its performance goal. However, as the department has worked to address the problems in CIS, its data suggest that the number of estimated bills it issues has decreased. Figure A.1 on the following page shows that at CIS’s launch in September 2013, the department issued more than 135,000 estimated bills; one year later, that number was less than 50,000. According to the director of customer operations, the department has largely resolved the issues causing the high number of estimated bills. The director stated that the department’s goal is to issue no more than 5 percent of its bills based on estimates, or roughly 42,000 bills. The September 2014 data in Figure A.1 show that if the department’s data are correct, it is nearing its goal. According to the director of customer operations, three issues caused the increase in estimated bills: incorrect meter data configurations, new meter reading routes, and narrow quality control parameters. First, the director explained that incorrect meter configurations lead to meter reading exceptions and estimated bills. According to the director, the department does not use a standard meter, and configuring all the different meter types in CIS was very complex. Second, the department takes most meter readings by hand and must plan its routes to optimize its staff’s efforts. When the department launched CIS, new routing software also launched that did not contain the history that the old routing system did—and 48 California State Auditor Report 2014-105 March 2015 the missing information made routes inefficient. Finally, the director explained that each meter reading is subject to a quality control check within CIS, as depicted in Figure 2 on page 7. At launch, the department made the quality control parameters too narrow, causing CIS to generate high numbers of meter readings for customer service representatives to verify. If customer service representatives could not verify the readings, estimated bills were sent. Figure A.1 Number of Actual and Estimated Bills the Los Angeles Department of Water and Power Issued in Selected Months Bills based on actual meter readings 1,000,000 Bills based on estimated meter readings Percentage of estimated bills 900,000 800,000 700,000 600,000 500,000 400,000 300,000 200,000 746,383 137,516 737,490 100,000 816,898 83,717 47,663 0 slliB fo rebmuN Data collection methodology changed. September 2013 March 2014 September 2014 15.6% 10.2% Estimated bills goal is 5 percent 5.6% Source: California State Auditor’s analysis of billing data obtained from the Los Angeles Department of Water and Power (department), unaudited. Notes: We did not audit the accuracy and completeness of the department’s counts of estimated bills or bills based on actual meter readings because we are presenting these data for informational purposes and they do not support any key findings or recommendations in our audit report. We noted that the department changed its methodology for measuring its estimated billing performance in May 2014; thus, the department’s data shown in the figure for September 2013 and March 2014 are not directly comparable to the data shown for September 2014. California State Auditor Report 2014-105 49 March 2015 The director of customer operations asserted that the department has changed its method of estimating bills in order to make them more precise. The director indicated that at launch CIS could estimate bills based on usage trends, as described in the Introduction. In these cases the system averaged usage across all accounts of a specific customer type (residential or commercial) as the basis for estimated bills, with the result that the bills were too high or too low because of the range of usage among customers in Los Angeles. According to the director, the department changed its estimation calculation to limit the average to the geographic region in which an account is located, resulting in more accurate estimated bills. Delayed Bills The number of delayed bills also exceeded the department’s performance goal after the launch of CIS; however, these bills too have decreased significantly in recent months. Figure A.2 on the following page shows that in January 2014—the first month for which an operations and statistics research analyst asserted that the department’s data became accurate and reliable—the department delayed issuing over 60,000 bills. Eight months later, in September 2014, it had reduced that number to fewer than 15,000. The new system cannot issue a bill that is incomplete, meaning a bill that does not reflect charges for one or more of a customer’s services and thus, these bills are delayed. In contrast, the department’s previous business practice was to issue incomplete bills if it did not have all of the necessary information and thus, far fewer bills were delayed. For example, it would issue a bill that included a customer’s charges for water but not for power if necessary and therefore, avoid delaying the bill. To properly use the billing portion of CIS, the department changed its business practice and now issues only complete bills. In November 2014 the department reported that it was close to reaching its goal of delaying less than 1.5 percent of its bills. 50 California State Auditor Report 2014-105 March 2015 Figure A.2 Los Angeles Department of Water and Power’s Monthly Average Delayed Bills and Approximate Delayed Monthly Revenue in Selected Months Delayed residential bills Delayed commercial bills Approximate delayed revenue* 80,000 70,000 60,000 50,000 40,000 30,000 44,093 20,000 60,326 18,292 10,000 10,954 11,811 3,598 0 January 2014 April 2014 September 2014 slliB deyaleD fo rebmuN $124,397,065 $62,407,307 $32,628,400 Source: California State Auditor’s analysis of billing data obtained from the Los Angeles Department of Water and Power (department), unaudited. Notes: We did not audit the accuracy and completeness of the department’s counts of delayed bills and related revenue because we are presenting these data for informational purposes and they do not support any key findings or recommendations in our audit report. We limited our review to comparing the raw data the department provided to information it had reported publicly and assessing the comparison for reasonableness. The figure omits data immediately following the first use of the customer information system (CIS) in September 2013 because an operations and statistics research analyst disclosed to us that the department was unable to query its billing data from September and October 2013 and the data for November and December 2013 were unreliable. * According to an operations and statistics research analyst, CIS estimates the outstanding revenue associated with delayed bills based on factors such as rates for service and customer classes. California State Auditor Report 2014-105 51 March 2015 * 1 2 3 3 * California State Auditor’s comments appear on page 53. 52 California State Auditor Report 2014-105 March 2015 California State Auditor Report 2014-105 53 March 2015 Comments CALIFORNIA STATE AUDITOR’S COMMENTS ON THE RESPONSE FROM THE LOS ANGELES DEPARTMENT OF WATER AND POWER To provide clarity and perspective, we are commenting on the response from the Los Angeles Board of Water and Power Commissioners (board) on behalf of the Los Angeles Department of Water and Power (department). The numbers below correspond to the numbers we have placed in the margin of the board’s response. 1 We obtained sufficient and appropriate evidence during the audit to justify our report’s conclusions. We stand behind our report’s findings. 2 We stand by our conclusions in the audit report. The board states that it agrees with our recommendations but disagrees with our basis for making them. In particular, the board asserts that its members were not misled or kept in the dark about the Customer Information System Connection project’s (project) status. In drawing our conclusions, we viewed pertinent board meetings available on the department’s Web site. Those meetings reflect that board members seemed surprised and concerned in June 2013—only a few months before system launch—about the need to increase the value of the department’s contract with PricewaterhouseCoopers by $12 million. Further, as we discuss on pages 21 and 32 of the report, testimony from high ranking department staff with intimate knowledge of the project provided board members with information that conflicted with internal documents. We further note that the current board membership is significantly different from the board membership that existed when the department made its board presentations in February and June 2013. The city of Los Angeles’ (city) mayor replaced four of the five board members in September 2013, which is the same month in which the department launched its customer information system. 3 To provide clarity and context, neither the department nor the board informed us of an ongoing investigation or the potential for litigation until the formal response to the audit dated February 20, 2015. Further, the city’s request for a “brief” extension was actually for one month since, according to the deputy city attorney making the request, the board needed more time to make its response to our audit report. We found the city’s request unreasonable given that we had shared the report and its recommendations with the department in late January 2015.