Summary

April 2015 California Public Utilities Commission It Needs to Improve the Quality of Its Consumer Complaint Data and the Controls Over Its Information Systems Report 2014-120 COMMITMENT INTEGRITY LEADERSHIP The first five copies of each California State Auditor report are free. Additional copies are $3 each, payable by check or money order. You can obtain reports by contacting the California State Auditor’s Office at the following address: California State Auditor 621 Capitol Mall, Suite 1200 Sacramento, California 95814 916.445.0255 or TTY 916.445.0033 OR This report is also available on our website at www.auditor.ca.gov. The California State Auditor is pleased to announce the availability of an online subscription service. For information on how to subscribe, visit our website at www.auditor.ca.gov. Alternate format reports available upon request. Permission is granted to reproduce reports. For questions regarding the contents of this report, please contact Margarita Fernández, Chief of Public Affairs, at 916.445.0255. For complaints of state employee misconduct, contact the California State Auditor’s Whistleblower Hotline: 1.800.952.5665. Elaine M. Howle State Auditor Doug Cordiner Chief Deputy April 9, 2015 2014-120 The Governor of California President pro Tempore of the Senate Speaker of the Assembly State Capitol Sacramento, California 95814 Dear Governor and Legislative Leaders: As requested by the Joint Legislative Audit Committee, the California State Auditor presents this audit report concerning the quality of the California Public Utilities Commission’s (commission) consumer complaint data and the controls it has established over its information systems. This report concludes that, despite the need for accurate data on the complaints the Consumer Affairs Branch (branch) receives, the quality of that data remains questionable. In 17 of the 45 complaints the branch received in fiscal years 2011–12 through 2013–14 that we selected and reviewed for accuracy, the branch did not correctly categorize the complaints in its Consumer Information Management System (CIMS) database. We found gaps in the training the branch has provided for its staff in categorizing complaints and that the branch has not systematically reviewed staff’s categorization of complaints. As a result, the branch is not consistently capturing the true nature of complaints it receives and is therefore providing users of CIMS data with inaccurate information. We also found that the branch did not provide appropriate information to complainants or did not forward their complaints to a utility, as required, in nine of 12 complaints we reviewed related to Voice over Internet Protocol (VoIP) service. As a result, some consumers with VoIP-related complaints did not receive information that might have helped them resolve their complaints. Further, although we found that the commission generally processes external data requests effectively, it could make the information it posts on its website more robust and easily accessible to consumers and other stakeholders. Finally, we identified pervasive weaknesses in the general controls the commission has implemented over its information systems. Although the commission had certified to the California Department of Technology that it complied with all policy requirements in Chapter 5300 of the State Administrative Manual, we found that key information security documents were nonexistent or lacked critical components. Specifically, the commission’s inventory of its information assets is incomplete, and it has not assessed the risks to its assets. Further, it has not developed an information security plan or an incident response plan. Finally, the commission’s technology recovery plan lacks key elements. Respectfully submitted, ELAINE M. HOWLE, CPA State Auditor 621 Capitol Mall, Suite 1200 Sacramento, CA 95814 916.445.0255 916.327.0019 fax www.auditor.ca.gov Blank page inserted for reproduction purposes only. California State Auditor Report 2014-120 v April 2015 Contents Summary 1 Introduction 7 Chapter 1 The Usefulness of the California Public Utilities Commission’s Consumer Complaint Data Is Limited Because of Problems With Accuracy and Completeness 17 Recommendations 32 Chapter 2 The California Public Utilities Commission Has Poor General Controls Over Its Information Systems, Compromising the Confidentiality, Integrity, and Availability of Its Information 35 Recommendations 43 Response to the Audit California Public Utilities Commission 45 California State Auditor’s Comment on the Response From the California Public Utilities Commission 51 vi California State Auditor Report 2014-120 April 2015 Blank page inserted for reproduction purposes only. California State Auditor Report 2014-120 1 April 2015 Summary Results in Brief Audit Highlights . . . The telecommunications industry has undergone a profound Our audit of the California Public Utilities transformation in recent years with the advent of new technologies Commission’s (commission) consumer such as cable-based Voice over Internet Protocol (VoIP) telephone complaint data and the controls over services. While federal law specifies that the Federal Communications its information systems highlighted Commission (FCC) maintains regulatory jurisdiction over interstate the following: and international telecommunications, it generally gives the states » The commission’s Consumer Affairs jurisdiction over their intrastate telecommunications. With certain Branch (branch) is not capturing the restrictions, California has designated responsibility for regulating true nature of complaints it receives and its intrastate telecommunication services to the California Public is therefore providing data users with Utilities Commission (commission). inaccurate data. The mission of the commission is to protect consumers and ensure • In 17 of the 45 complaints we that California utility customers have safe, reliable utility service at selected and reviewed for accuracy, reasonable rates. It is also responsible for helping consumers resolve the branch did not correctly issues with the industries it regulates. The commission’s Consumer categorize the complaints in the Affairs Branch (branch) supports the commission’s mission by Consumer Information Management helping consumers resolve disputes or informal complaints with System (CIMS). certain utilities. The branch also provides the commission and • Gaps in training and ineffective other entities, such as the Legislature, with information about oversight have resulted in the complaints it receives from consumers regarding utilities. problems with staff accurately Branch staff enter data they receive from consumers into the classifying complaints. Consumer Information Management System (CIMS), a database that contains, among other data, complaint information that is » For nine Voice over Internet organized by category of complaint. Protocol (VolP) complaints, the branch either did not provide appropriate Despite the need for reliable data on the nature of the complaints information to complainants or did not the branch receives, the quality of the commission’s complaint data forward their complaints to a utility is questionable. In 17 of the 45 complaints we selected and reviewed as required. for accuracy, we found that the branch did not correctly categorize the complaints in CIMS.1 Although the branch has provided » Commission staff generally responded training to its staff in classifying complaints, we identified gaps in to external data requests quickly that training and also noted that the branch has not systematically and effectively. reviewed its staff’s classification of complaints. As a result, the » The commission could make complaint branch’s complaint data do not accurately reflect the complaints it data on its website more robust receives, and the branch is providing users with inaccurate data. and easily accessible to consumers and Further, although the branch has made improvements to CIMS other stakeholders. that enhance the quality of certain complaint data elements, these improvements are not effective if complaint data are entered » The commission’s controls over incorrectly when the branch first receives the complaint. its information systems need improvement—key information security documents either were nonexistent or lacked critical components. 1 We tested 30 complaints selected from all telecommunication complaints received in fiscal years 2011–12 through 2013–14 (general complaints). We also reviewed 15 VoIP‑related complaints received by the branch between January 1, 2013, and June 30, 2014. 2 California State Auditor Report 2014-120 April 2015 In September 2014, the branch initiated a quality management team program to increase data quality by, among other things, reviewing the accuracy of how staff classify informal complaints. However, the branch has yet to implement tools to measure this program’s effectiveness. We believe continuing this program and developing such measurement tools will help the branch improve the accuracy of the data and improve its value to users. We also found that the branch could do more to capture the complete nature of complaints consumers have reported. In addition to categorizing complaints, CIMS also allows staff to enter additional information about complaints in the form of attributes. For example, if a consumer complained to the commission that sometimes he or she did not have phone service and was also frustrated with the representative of the phone company, branch staff could categorize one of these issues as the primary reason for the complaint and the other as an attribute. We found 16 of 30 general telecommunication complaints we reviewed had information that could have been included in CIMS as attributes. However, branch staff added attributes for only four of those complaints, thereby omitting descriptive complaint information from the other 12. When the branch does not take advantage of opportunities to record complaint details, it is providing incomplete information to other commission divisions that use the branch’s complaint data, and it is also missing opportunities to provide the commission’s divisions and stakeholders with richer, more useful information. The commission’s ability to identify VoIP complaints is limited because the California Public Utilities Code, Section 710 is ambiguous about whether VoIP providers must provide information to the commission that would assist it in responding informally to VoIP complaints. Not all VoIP providers are required to register with the commission and report information regarding their VoIP customers, and the commission staff do not believe they have the legal authority to compel VoIP providers to report this information. According to an October 2014 branch report, the inability to connect some complaints with VoIP providers in CIMS is a key challenge in producing reports and assisting California consumers because the branch’s ability to process and report on VoIP complaints is directly tied to the quality of information in CIMS about VoIP providers. Further, in nine of the 12 VoIP-related complaints we reviewed that were submitted to the branch after it issued guidance in May 2013 for processing such complaints, the branch either did not provide appropriate information to complainants or did not forward the complaint to the utility as required. As a result, some consumers California State Auditor Report 2014-120 3 April 2015 with VoIP-related complaints did not receive information that might have helped them resolve their complaints, such as contact information for the FCC. Members of the public and entities, such as the Legislature and the FCC, may request data related to consumer complaints from the commission by making an external data request, meaning a request that comes from outside the commission. They may submit such requests to one of the commission’s divisions, such as the Office of Governmental Affairs, which then typically contacts the branch to fulfill the request. In addition, commission entities, such as the Communications Division and the Safety and Enforcement Division, use branch consumer complaint data to develop policy and to identify trends, among other functions. Our review of selected external data requests indicated that the commission generally processes the requests quickly and effectively. We did, however, identify two related requests in which a miscommunication between the requestor and commission staff may have resulted in the requestor’s expectations not being fully met. To address this issue, the branch has proposed, but not formally adopted, modifications of its procedures for processing requests that we believe, if followed, could help the commission avoid miscommunications in the future. We also determined that the commission could make information about contacts and informal complaints from telecommunications consumers (complaint data) on its website more robust and easily accessible to consumers and other stakeholders. The branch began posting complaint data to the commission’s website in 2012 in order to assist consumers, and in December 2013 it developed a plan for improving the quality of the data posted online. The branch’s plan envisioned including online counts of complaints organized by utility company and category of complaint by January 2014, a measure of consumer satisfaction by April 2014, and possibly a separate report of VoIP-related complaints by July 2014. As of January 2015, the branch had posted only the utility company and category-specific data. Further, the complaint data the commission posts on its website can be difficult to find because consumers must follow three nonintuitive links to navigate from the commission’s home page to the location where complaint data are posted. Finally, as part of our assessment of the validity and reliability of the CIMS data, we used the policy requirements in Chapter 5300 of the State Administrative Manual (SAM) as a benchmark for evaluating the controls the commission has implemented over its information systems because the commission acknowledges that these requirements are good business practices. We expected that the commission would have developed adequate 4 California State Auditor Report 2014-120 April 2015 plans, policies, and procedures to provide for the proper protection of its information assets and to ensure its ability to sustain and recover critical information technology services should an unexpected human-made or natural disaster jeopardize its information assets. However, despite certifying to the California Department of Technology (CalTech) that it had complied with all policy requirements in Chapter 5300 of SAM, we found that key information security documents either were nonexistent or lacked critical components. Specifically, the commission has yet to inventory all of its information assets, assess the risks to those assets, and develop an information security plan that provides a strategy for mitigating those risks. Further, the commission does not have an incident response plan that provides for a timely response to, and recovery from, an information security incident, such as a malicious cyber attack. Finally, although the commission has a current technology recovery plan, we question its usefulness because the plan fails to consistently identify critical applications, establish acceptable outage time frames for these applications, and develop strategies for recovery. Until the commission improves the controls it has implemented over its information systems, the confidentiality, integrity, and availability of its information systems will continue to be at risk. Recommendations Legislative To ensure that the commission has the information it needs to better report on VoIP-related complaints, the Legislature should give the commission the authority to collect information from providers regarding their VoIP customers and require VoIP providers to furnish this information to the commission. Commission To ensure that policy makers, enforcement officials, and the general public have access to accurate consumer complaint data in CIMS, the branch should do the following: • Update and provide further training to its staff on properly classifying complaints by September 30, 2015. • Continue to implement its quality management team program component focused on reviewing the categorization of complaints and correcting identified errors. • Develop and implement tools to measure the quality management team program’s effectiveness by September 30, 2015. California State Auditor Report 2014-120 5 April 2015 To ensure that policy makers, enforcement officials, and the general public have access to more complete and meaningful consumer complaint data, the branch should, to the fullest extent possible, include the attributes of each complaint in the data it records in CIMS. To ensure that branch staff provide the appropriate assistance to consumers with VoIP-related complaints, the branch should, by September 30, 2015, further train its staff on providing correspondence to complainants as required by its guidelines. To ensure that consumers have access to the complaint data that will enhance their ability to make informed choices about their telecommunication services, the branch should, by June 30, 2015, create an updated plan that specifies the types of data the branch intends to post online and a timeline for fully implementing that plan. To ensure that the public can easily locate customer complaint data the branch publishes on its website, the commission should make navigating to its customer complaint data more intuitive and direct. The commission should ensure that it complies with all policy requirements in SAM Chapter 5300 no later than April 2016. Specifically, the commission should do the following: • Complete and maintain an inventory of all its information assets. • Conduct an assessment of the risks facing its information assets. • Develop, implement, and maintain an information security plan. • Develop, disseminate, and maintain an incident response plan. • Revise its existing technology recovery plan to include a list of critical applications, their maximum acceptable outage time frames, and detailed recovery strategies for each application. • Ensure that any certifications it submits to CalTech accurately represent its information security environment. Agency Comments The commission generally agreed with our findings and stated its goal was to address all of our recommendations, but indicated that the implementation of the recommendations is dependent upon available resources. 6 California State Auditor Report 2014-120 April 2015 Blank page inserted for reproduction purposes only. California State Auditor Report 2014-120 7 April 2015 Introduction Background The telecommunications industry has undergone a profound transformation in recent years. The U.S. Government Accountability Office defines telecommunications as voice, video, or data transmissions that travel over a variety of wired and wireless networks. Telephone networks primarily transmit voice conversations over a combination of copper wire and fiber-optic cable, connecting the caller and receiver through a system of switches. Similarly, cellular telephone networks use a combination of wired and wireless technology to connect their users. According to a California Public Utilities Commission (commission) decision, by December 2004 the number of wireless subscribers in the United States (U.S.) surpassed the number of traditional wire line subscribers. Between 2000 and 2004, according to the commission, major cable companies began offering cable-based voice telephone services and high-speed advanced Internet service became accessible to 95 percent of U.S. households. During this same period, the first Internet-based Voice over Internet Protocol (VoIP) telephone companies made their appearance. VoIP technology enables telephone communications to take place digitally over the Internet instead of over traditional analog telephone systems. VoIP services may be interconnected, meaning that communications on these systems connect to the public telephone network, or noninterconnected, meaning that communications using these services do not connect to the public telephone network. Some utilities, such as AT&T and Comcast, provide interconnected VoIP services. Skype is an example of a company that provides noninterconnected VoIP service. While federal law specifies that the Federal Communications Commission (FCC) maintains regulatory jurisdiction over interstate and international telecommunications, it generally gives the states jurisdiction over their intrastate telecommunications. With certain restrictions, California state law puts public utilities that provide telecommunication services under the jurisdiction and regulation of the commission. The California Public Utilities Commission The mission of the commission is to protect consumers and ensure that California utility customers have safe, reliable utility service and infrastructure at reasonable rates. It also includes a commitment to environmental enhancement and a 8 California State Auditor Report 2014-120 April 2015 healthy California. In meeting its mission, the commission regulates privately owned utilities, such as those that provide energy, water, and telecommunication services. It also helps consumers resolve issues with the industries it regulates. State law requires the commission to publish annually a workplan that includes information on how members of the public can gain access to the commission’s rate-making process or contact the office of the public advisor. In the last few years, the commission has included counts of consumer complaints to the Consumer Affairs Branch (branch) in this annual workplan. The Commission’s Consumer Affairs Branch The commission’s Consumer Service and Information Division (division) is responsible for collecting, analyzing, and addressing customer comments and complaints concerning the utilities the commission regulates, as well as acting as a conduit of information between the public, regulated utilities, and commission decision makers. As part of the division, the branch supports the commission’s mission by helping consumers resolve disputes with certain utilities as well as by providing the commission with research and analysis of trends in utility customer complaints. Branch staff document consumer complaints through the Consumer Information Management System (CIMS), as we discuss later. The branch also helps customers understand their utility services and bills. The branch maintains two offices—one in San Francisco and one in Los Angeles. Each office employs consumer representatives to answer questions and help consumers resolve complaints about utility bills and services. In addition, the branch employs analysts who respond to data requests by extracting the requested information from CIMS. Evolution of the Branch’s Responsibilities According to the branch’s program manager, helping consumers understand their utility services and bills and assisting consumers in resolving disputes with their utility companies have always been the branch’s primary purpose. However, over time the branch has gained certain additional responsibilities, such as resolving appeals of eligibility for Lifeline, a program that enables low-income citizens to receive discounted telecommunications service. Further, a 2008 commission decision and corresponding resolution mandated that the branch post on the commission’s website data from consumers whose primary language is not English and who contact the branch with an California State Auditor Report 2014-120 9 April 2015 inquiry or complaint. The commission also stated in these documents that the branch should be reasonably confident that the CIMS data it publishes are accurate, reliable, and consistent. As we discuss later in this report, the branch is also responsible for processing requests for consumer complaint data from internal and external stakeholders. In addition, the branch is now tracking and reporting on complaints regarding VoIP services. Beginning in 2013, California Public Utilities Code, Section 710 (Section 710) generally prohibits the commission from regulating VoIP services. However, it allows the commission to track, monitor, and report on consumer complaints about VoIP service, and to informally assist complainants with their VoIP-related issues. Section 710 does not require or specify a manner in which the commission is to track and respond informally to VoIP complaints, nor does it indicate whether VoIP providers must provide information to the commission that would assist it in responding informally to VoIP complaints. The Consumer Information Management System As part of its approach to implementing a 2006 decision referred to as the Consumer Bill of Rights (CBOR), the commission undertook to update the branch’s database. At the time of the CBOR decision, the branch had a database that was designed to track individual complaints rather than to provide management with information to assess particular trends in utility or consumer issues. The commission recognized that the database needed to be upgraded to current standards to accomplish the initiatives in the CBOR. In 2008 the branch installed CIMS to assist staff in processing consumer inquiries and complaints, to increase the commission’s effectiveness and efficiency in processing and resolving complaints, to improve the quality of the services it provides to the public, and to facilitate data analysis throughout the commission. The branch has taken several steps to update and improve the quality of data in CIMS since its 2008 implementation. For example, it has added capabilities to allow it to better characterize the nature of complaints. According to branch guidelines for entering complaint information into CIMS, branch staff are to catalog each complaint based on its characteristics, category, or high-level description; its subcategory, which is a more specific description of the complaint; and its attributes, which provide branch staff an opportunity to further describe the complaint. In 2013 in one of the most significant updates to CIMS, the branch installed the Business Rules Manager software (BRM). The project charter for the CIMS upgrade project that created the BRM noted that the new software would allow the branch to improve the 10 California State Auditor Report 2014-120 April 2015 quality of consumer complaint data it captured at the point of entry. Another BRM planning document indicated BRM would implement logic controls over key elements of the data entry process, such that it validates written case records against business rules before a complaint can be closed in CIMS. For instance, BRM would prevent a staff member from closing a complaint record after he or she had selected a subcategory such as smart meter, which falls under the energy category, and attempted to incorrectly assign it to the telecommunications category. According to the branch’s program manager, after the branch implemented CIMS in 2008, the branch and the commission’s information technology department submitted change orders to improve CIMS’s functionality and design. He further stated that, between 2008 and 2013, the branch had worked on improving its complaint resolution processes and categorization of its complaints and it had determined that implementing the BRM upgrade to CIMS was necessary. The Complaint Process Consumers who have a dispute with their utility providers, including telecommunication utilities, may contact the branch in a variety of ways, including by phone, fax, email, the Internet, or through the U.S. Postal Service, to try to address their dispute. Most disputes the branch receives involve utility billing, utility service, and payment arrangements. When a consumer contacts the branch about a dispute by phone—referred to as a phone contact—branch staff log information related to the call into CIMS, connect the consumer directly to the executive office of the utility in question, and explain the situation on the consumer’s behalf. According to the branch’s program manager, the consumer and utility resolve the majority of phone contacts regarding disputes through this direct connection. Consumers who are dissatisfied with their utility may also file a written complaint with the branch, known as an informal complaint. If branch staff receive adequate information, they will assign a case number, process the informal complaint, and transmit it to the utility for investigation and response. Once the utility responds, branch staff analyze the response and, among other actions, determine whether all issues raised in the informal complaint have been addressed. The Figure illustrates the branch’s consumer phone contact and informal complaint resolution process. The commission’s website notes that this resolution process does not involve judicial review by the commission, so it is quicker and easier than filing a more formal complaint with the commission. California State Auditor Report 2014-120 11 April 2015 Figure California Public Utilities Commission’s Consumer Affairs Branch Consumer Phone Contact and Informal Complaint Resolution Process Web form, letter, fax, Consumer contacts the Consumer email, or Affairs Branch (branch) by: walk-in CPUC CONSUMER PHONE CONTACT California Public Utilities INFORMAL COMPLAINT* Commission’s (commission) branch Branch staff determine whether the consumer is trying to The branch receives a consumer’s informal contact the branch or another entity. If the branch is the complaint by U.S mail, fax, or email; through appropriate contact, the representative creates a case in the commission website; or by a walk-in the Consumer Information Management System (CIMS). to the branch. Branch staff review the informal complaint Consumer Consumer and categorize it in CIMS. has an inquiry has an issue Branch staff send complaint information to the appropriate utility with instructions for response. Branch representative Branch representative provides information in assists the consumer in Utility reviews the complaint and takes action response to the inquiry. resolving his or her issue.† it deems necessary to appropriately address the consumer’s complaint and the commission’s instructions and responds to the branch. Branch staff determine whether to: Branch staff review the utility’s response to Instruct the Refer the determine whether it is complete. consumer to consumer’s issue submit the issue in to the utility’s writing (informal executive office comlaint) Response is Response is incomplete complete Branch staff close case in CIMS. If omission of If omission of information is simple, branch staff information is complex, verbally request additional information from the branch staff send a utility and once deemed sufficient, the response supplemental informal is considered complete. Is the consumer complaint to the utility. satisfied with the utility’s response to the complaint? Utility responds to If branch staff determine the utility was in compliance with the the supplemental commission’s instructions, staff send a closing letter to the consumer informal complaint. regarding the outcome of the informal complaint. YES NO Branch staff close case in CIMS. Sources: The 2008 branch Procedure Manual and the California State Auditor’s analysis of selected informal complaints. * The commission has a formal complaint process that involves its Administrative Law Judge Division. The branch’s informal complaint process provides a final opportunity to resolve an issue informally before a consumer decides to file a formal complaint. † If the branch is not the appropriate contact, branch staff record certain information regarding the nature of the call in CIMS and the consumer is referred to the appropriate entity, which could include another governmental agency or another division of the commission. 12 California State Auditor Report 2014-120 April 2015 Consumers may also file a formal complaint with the commission. The processing of these complaints is administered by the commission’s Administrative Law Judge Division and does not involve the branch. In formal complaints, the consumer has the burden to present his or her case and prove that the utility has violated the utility’s tariff rules, a commission order, or the law. The commission can order a regulated utility to take corrective action on a variety of formal complaints, including making an adjustment to a consumer’s bill. This audit focuses on the commission’s processes for handling consumer phone contacts and informal complaints. It does not address the commission’s formal complaint process. Public Access to Commission Information State law generally allows any member of the public access to the public records of a state or local agency through a California Public Records Act (public records act) request. Public records include any writing containing information relating to the conduct of the peoples’ business. The public records act does not require the disclosure of certain public records, such as records detailing an individual’s medical condition. After an agency receives a request, state law generally requires the agency to determine and notify the requestor within 10 days whether the agency has disclosable public records related to the request. The agency must also state the estimated date that it will make the records available to the requestor. The commission’s legal division receives public records act requests and coordinates its efforts with other commission divisions as necessary to fulfill the requests. Commission records show that from July 1, 2011, to June 30, 2014, the majority of public records act requests submitted by individuals interested in commission telecommunication complaint data were related to the outcome of a single complaint. Members of the public and entities such as the Legislature and FCC may also request data on Sample California Public Records Act request: customer phone contacts and informal complaints Provide a copy of the customer complaint investigation of case from the commission through an external data number 163136. request. The text box provides examples of a public records act request and an external data Sample external data request: request. Our review of a judgmental selection of Provide number of written complaints of customers external data requests determined that the public complaining about being moved from traditional telephone/ submits a request either directly to the branch or copper wire-based service to fiber-optic telephone service by to another commission entity, such as the Office Verizon from January 2012 to present. of Governmental Affairs (OGA) or the News and Source: California Public Utilities Commission records regarding Public Information Office, which then typically California Public Records Act requests and external data requests. contacts the branch to fulfill these requests. The branch’s internal procedures state that it must California State Auditor Report 2014-120 13 April 2015 clarify the request with the requestor within two business days and set a deadline for when the branch will provide the requested data. According to the program manager and the project supervisor of the branch’s analysis section, the time required for the branch to process external data requests varies, based on differences in the nature and scope of the requests. However, the branch’s internal procedures do not prioritize external data requests based on who submits them. Branch procedures indicate that the commission’s legal division often reviews the information assembled to respond to a request before the branch sends it out. Other commission entities, such as OGA, sometimes review the information as well. Commission records show that government organizations, such as the FCC or the Legislature, submit the majority of external data requests for commission complaint data. Scope and Methodology The Joint Legislative Audit Committee (audit committee) directed the California State Auditor to perform an audit of the commission’s practices for compiling and disclosing consumer complaint data regarding telecommunications service. Table 1 outlines the audit committee’s objectives and our methods for addressing them. Table 1 Audit Objectives and the Methods Used to Address Them AUDIT OBJECTIVE METHOD 1 Review and evaluate the laws, rules, Reviewed relevant laws, regulations, policies, and other background materials pertaining to the and regulations significant to the California Public Utilities Commission’s (commission) Consumer Affairs Branch (branch). audit objectives. 2 Evaluate the effectiveness of the • Reviewed the branch’s internal and external data request policies. commission’s policies and procedures • Obtained a list of all external data requests for fiscal years 2011–12 through 2013–14. for responding to requests from • Reviewed all available documentation related to a judgmental selection of 12 data requests and external stakeholders for customer evaluated these requests in relation to the areas specified in the audit objective. complaint data, including, but not limited to, a review of the following • Conducted interviews with branch management to determine the reasons for exceptions areas for a selection of requests over to the data request procedures and any improvements the branch is planning to address the past three fiscal years: these exceptions. a. Intake process and • We determined for item e that data requests are public documents and are therefore covered by customer assistance. provisions relating to the California Public Records Act. Thus, they are not confidential. b. Tracking of requests. c. Timing of responses to request. d. Internal review of requests and approval of responses. e. Confidentiality of requests. f. Protection of customer privacy. g. Any special processing performed or priority given for certain requestors, such as the Office of Ratepayer Advocates, the Legislature, or others. continued on next page . . . 14 California State Auditor Report 2014-120 April 2015 AUDIT OBJECTIVE METHOD 3 Determine how the branch ensures • Conducted interviews to determine how information is entered into CIMS using preset complaint that the Consumer Information categories and how it is used by the branch staff and commission employees, and how branch Management System (CIMS) generates leadership intends to use CIMS data in the future. valid and reliable data for identifying • Reviewed manuals, existing reports, and other written materials to determine relevant processes emerging customer service issues. As for managing consumer complaints and to identify relevant data trends using keyword searches part of this determination, assess the use, and inquiry‑related functionalities. validity, and reliability of CIMS’s preset • Selected 30 informal consumer complaint records from fiscal years 2011–12 through 2013–14 and complaint categories, keyword searches, examined them to determine how branch staff recorded consumer complaints. existing reports, and inquiry‑related functionalities and whether CIMS has the • Evaluated branch database information to determine error rates for selected fields from fiscal capability to track, compile, and report on years 2011–12 through 2013–14. complaint activity to enable data‑driven • Reviewed a selection of general controls the commission has implemented over its information decision making by the commission, systems, such as information security and contingency planning. the Legislature, and the Federal Communications Commission. 4 Determine whether the branch has • Reviewed relevant laws, branch procedures, and other background materials applicable to CIMS. procedures to effectively track and • Interviewed managers and branch staff and analyzed documents to identify the policies and report complaints related to the procedures the branch maintains related to identifying, tracking, and providing reports concerning provision of basic service through VoIP complaints and the provision of basic services through digital technologies. digital technologies and Voice over • Selected 30 complaints—10 each for fiscal year 2011–12 through fiscal year 2013–14—related to Internet Protocol (VoIP). VoIP and digital technologies. – Attempted to compare how the branch addressed the 15 complaints it received before the implementation of California Public Utility Code, Section 710 (Section 710) on January 1, 2013, to how it addressed the 15 it received after that date. – We found no basis of comparison for this test because the branch’s methods for addressing complaints were not consistent either before or after the law became effective. • For each of the 15 complaints the branch received after Section 710 became effective, we performed the following steps: – Reviewed whether and how the branch determined if a complaint was related to VoIP and how the branch responded to the complaint. – Determined whether the branch handled the complaint in accordance with the policies and procedures in place for complaint resolution. • Evaluated the branch’s recent Business Rules Manager software upgrade to CIMS by reviewing documentation, conducting interviews with branch staff, and reviewing our selection of VoIP complaints to determine if the CIMS upgrade functions as intended by preventing the data entry errors the branch planned to prevent. • Evaluated the guidance for capturing data and tracking progress for complaints related to VoIP and other comparable consumer service issues that the branch provided to staff, including policies and procedures, emails, and training materials related to VoIP. • Researched how nine other states’ commissions are tracking and reporting information related to VoIP. 5 Evaluate the commission’s process • Reviewed the branch’s procedures for posting complaint and contact data online. for posting customer complaint and • Conducted interviews with branch management to determine how the branch decided which contact data on its website, including information to post online and how the branch assesses the value of the data. decisions on the nature of information • Reviewed other states’ models to identify best practices for posting complaint data online. to be disclosed—such as whether a complaint was resolved by the service • Evaluated the commission’s website to determine the accessibility of complaint data. provider—and ensuring the accuracy and reliability of these data. Further, determine whether the commission has taken steps to evaluate whether data are accessible, user‑friendly, and valuable to customers, telecommunications carriers, and other stakeholders. Sources: California State Auditor’s analysis of the Joint Legislative Audit Committee’s audit request number 2014‑120, and analysis of information and documentation identified in the table column titled Method. California State Auditor Report 2014-120 15 April 2015 Assessment of Data Reliability In performing this audit, we obtained electronic data files extracted from CIMS. The U.S. Government Accountability Office, whose standards we are statutorily required to follow, requires us to assess the sufficiency and appropriateness of computer-processed information that we use to support our findings, conclusions, or recommendations. To accomplish this, we performed data-set verification procedures and electronic testing of key data elements and identified significant issues. We also conducted accuracy testing for a selection of 45 complaints that the commission received during the period from July 1, 2011, through June 30, 2014, and found 17 errors in the complaint category. In addition, we reviewed selected information system controls the commission implemented over CIMS. Given the government’s increased use of information technology, the State has a compelling need to ensure that its information assets are adequately protected from known and anticipated threats. For many state entities, program operations would effectively cease in the absence of key computer systems. Furthermore, the unauthorized acquisition, access, modification, deletion, or disclosure of information included in state entity files and databases can compromise the integrity of state programs, violate individual right to privacy, and constitute a criminal act. Accordingly, Chapter 5300 of the State Administrative Manual (SAM) provides a framework that many state entities must follow to protect their information assets from a wide spectrum of threats and risks. As part of our assessment of CIMS, we examined whether the commission implemented selected components of SAM Chapter 5300. The results of our review indicate that the commission has pervasive weaknesses in the general controls associated with a large segment of its information systems. Due to these deficiencies, we did not proceed with performing exhaustive testing of the general controls the commission has implemented over all of its information systems or those controls that are specific to the CIMS application. Consequently, there may be additional weaknesses that exist over the CIMS data that we did not identify during our review. As a result of the general control weaknesses and the issues we identified in our electronic and accuracy testing of CIMS, we determined that the CIMS data are not sufficiently reliable for any purpose. For a further discussion of our review of the commission’s information system controls, see Chapter 2. 16 California State Auditor Report 2014-120 April 2015 Blank page inserted for reproduction purposes only. California State Auditor Report 2014-120 17 April 2015 Chapter 1 THE USEFULNESS OF THE CALIFORNIA PUBLIC UTILITIES COMMISSION’S CONSUMER COMPLAINT DATA IS LIMITED BECAUSE OF PROBLEMS WITH ACCURACY AND COMPLETENESS Chapter Summary The quality of the California Public Utilities Commission’s (commission) consumer complaint data remains questionable despite the commission’s efforts at quality improvement. In 17 of the 45 complaints the Consumer Affairs Branch (branch) received during fiscal years 2011–12 through 2013–14 that we selected and reviewed for accuracy, we found that the branch did not correctly categorize the complaint in the Consumer Information Management System (CIMS) database. As a result, the branch is not consistently capturing the true nature of complaints it receives and is therefore providing users of CIMS data with inaccurate information. As we discussed in the Introduction, California Public Utilities Code, Section 710 (Section 710) generally prohibits the commission from regulating Voice over Internet Protocol (VoIP) services. This prohibition has limited the commission’s ability to identify VoIP complaints because not all VoIP providers are required to register with the commission and report information regarding their VoIP customers, and commission staff do not believe it has the legal authority to compel VoIP providers to report this information. In addition to the accuracy errors discussed above, we found that branch staff made other errors in assisting consumers with VoIP-related issues. For example, the branch’s procedures for VoIP-related complaints provide guidance for staff to follow in sending letters to utilities and consumers about the complaints. However, in nine of 12 VoIP-related complaints we reviewed, the branch either did not provide an appropriate closing letter to the complainant or it did not forward the complaint to the utility as required. The Quality of the Commission’s Consumer Complaint Data Remains Questionable Despite Its Efforts at Improvement Although commission entities use consumer complaint data the branch has collected to make policy and identify trends, among other functions, the data are not always accurate. Specifically, 17 out of 45 complaints we reviewed, or 38 percent, were incorrectly classified in CIMS. Based on our review of these data, we believe 18 California State Auditor Report 2014-120 April 2015 We believe there are weaknesses there are weaknesses in the training provided to and oversight in the training provided to and of staff regarding complaint classification in CIMS. We also oversight of staff regarding believe it would enhance the value of the branch’s complaint data complaint classification in CIMS. if branch guidance for processing complaints was aligned with how data are entered into CIMS according to the Business Rules Manager software (BRM) update to CIMS that we describe in the Introduction. The Branch’s Customer Complaint Data Processes Have Improved, But Errors Still Persist Part of the mission of the Consumer Service and Information Division (division) is to collect, analyze, and address consumer comments and complaints concerning the utilities the commission regulates as well as to supply information to commission entities who set policy and manage regulatory enforcement activities. The branch’s program manager stated that as part of the division, the branch shares this mission by helping consumers. According to unaudited branch records between fiscal years 2011–12 and 2013–14, some commission entities, including the Safety and Enforcement Division (Enforcement division), Office of Ratepayer Advocates, and the Administrative Law Judge Division, used branch consumer complaint data to identify utility complaint trends, make policy, and aid in enforcement actions. The branch has established policies and procedures for how complaint data should be categorized in CIMS. According to branch guidelines for processing complaint information into CIMS, branch staff are to classify each complaint based on its characteristics. These guidelines group characteristics into defined categories, which are high-level descriptions of a complaint; subcategories, which are a more specific description of the complaint; and attributes, which provide branch staff an opportunity to capture other information that describes the complaint. According to the branch’s Los Angeles consumer services manager, CIMS automatically populates the category based on the subcategory selected by branch staff. For example, the CIMS data entry guidelines specify that if a consumer has a complaint about his or her telephone service not working, branch staff should record the subcategory for the complaint in CIMS as an outage. Then, because the outage subcategory is associated with the service complaints category, CIMS would automatically populate the category as service. Further, if the complainant in our example also described utility staff as rude when the complainant was attempting to resolve an outage problem with the utility, branch staff could select a customer service attribute California State Auditor Report 2014-120 19 April 2015 in CIMS for the complaint. The data entry guidelines require a category and a subcategory for each complaint; however, attributes, if applicable, are not required. Although the branch has procedures for staff to follow when In our review of 30 general categorizing complaints, we found staff do not always do so. complaints, we found Specifically, we found nine instances in our review of 30 general nine instances in which branch complaints in which branch staff did not choose the correct staff did not choose the correct subcategory for a complaint when entering it into CIMS.2 For subcategory for a complaint when example, the branch incorrectly assigned a 2011 complaint about entering it into CIMS. a high bill to the quality of service subcategory. At the time of the complaint, the branch defined the high bill subcategory as pertaining to instances in which a consumer’s bill is higher than normal but the consumer does not know why, and defined the quality of service subcategory as pertaining to instances in which a consumer has a problem with the performance of a landline or wireless service, such as static, dropped calls, or poor transmission. In this complaint, the complainant alleged that after moving his business and changing the phone numbers for his business telephone lines, his business was charged an additional $565 per month because the telephone provider did not cancel the old phone numbers. The complainant stated that the telephone provider had adjusted the bill but that his business continued to receive excessive billings. Given the branch’s definitions of the high bill and quality of service subcategories, the branch should have assigned this complaint to the high bill subcategory. The branch’s Los Angeles office manager indicated that the subjective nature of categorizing complaints, along with the many issues that could be involved in a complaint, sometimes leads to miscategorized complaints in CIMS. Nonetheless, the branch presents inaccurate information to users of its consumer complaint data when its staff do not correctly enter data into CIMS. We found similar errors in a selection of VoIP-related complaints we reviewed that were submitted between fiscal years 2011–12 and 2013–14. Beginning January 1, 2013, Section 710 allows the commission to track and report on VoIP-related complaints. To this end, branch guidance effective May 31, 2013 requires staff members to assign complaints related to VoIP issues to the nonjurisdictional VoIP subcategory. However, in seven of the 12 VoIP complaints 2 We tested 30 complaints selected from all telecommunication complaints received in fiscal years 2011–12 through 2013–14 (general complaints). We also reviewed 15 VoIP‑related complaints received by the branch between January 1, 2013, and June 30, 2014. Of these 15 VoIP‑related complaints, three were received after January 1, 2013, when Section 710 went into effect, but before May 31, 2013, when the branch issued guidelines for VoIP‑related complaints. We tested the three complaints issued before the branch issued its guidelines using the same criteria as was used for the 30 general complaints. The remaining 12 VoIP‑related complaints were received between June 1, 2013, and June 30, 2014. We describe the results of our testing of these VoIP‑related complaints later in the chapter. 20 California State Auditor Report 2014-120 April 2015 we reviewed that were received after May 2013, the branch did not correctly assign the complaints to the nonjurisdictional VoIP subcategory. For example, the commission assigned a February 2014 complaint to the high-bill complaint subcategory, despite the fact the complainant stated in her complaint that she had a VoIP-related service. Based on branch guidance at that time for processing VoIP-related complaints, the staff member should have assigned the complaint to the nonjurisdictional VoIP complaint subcategory. The branch’s program manager agreed with our assessment for five of the seven incorrectly processed VoIP-related complaints we identified. However, for the other two complaints, the branch did not believe it made errors in its assignments. Specifically, the branch’s Los Angeles consumer services manager does not believe these were VoIP-related complaints because, in one instance, the complaint concerned a credit agency. However, because this Branch guidance indicates complaint resulted from VoIP service charges, we disagree with that if branch staff is not clear the branch’s position. In the other instance, involving a complaint whether a complainant with by a customer with Verizon fiber-optic telephone service, the Verizon fiber‑optic services has branch’s Los Angeles consumer services manager did not think VolP services, the complainant the complaint was VoIP-related, nor did she think branch staff should be contacted to make needed to contact the customer to determine if he or she had that determination. VoIP service. However, branch guidance indicates that if it is not clear whether complainants with Verizon fiber-optic services have VoIP services, the complainant should be contacted to make that determination. Because branch staff did not contact the complainant, the branch did not ensure that the complaint was appropriately classified. Such data entry errors prevent the branch from accurately reporting data about VoIP-related complaints. In another instance among the 12 VoIP-related complaints we reviewed, the branch incorrectly assigned a complaint that was not VoIP-related to the nonjurisdictional VoIP subcategory. This complaint related to a consumer’s charges for Internet service and did not relate to VoIP. The branch’s Los Angeles consumer services manager acknowledged that the complaint should not have been classified as VoIP and that some staff were incorrectly identifying other non-VoIP products as VoIP. To address this issue, a branch consumer services supervisor sent staff an advisory email on the subject in November 2014. Because the branch is not consistently assigning the correct category to complaints it processes, the complaint data in CIMS do not accurately reflect the complaints the branch receives. As a result, the branch is providing users of CIMS data with inaccurate information. Further, as we discuss later in the chapter, because the branch is not consistently assigning VoIP-related complaints California State Auditor Report 2014-120 21 April 2015 to the nonjurisdictional VoIP subcategory, the branch either did not provide contact information for the Federal Communications Commission (FCC) or it did not forward the complaint to the utility as required by its VoIP guidance in seven of 12 VoIP-related complaints we reviewed. Based on our review of a selection of 45 complaints, 30 general complaints and 15 VoIP-related complaints, we believe the branch staff lack sufficient training regarding the accurate classification of complaints. Although branch staff received training with the implementation of the BRM, based on the 38-percent error rate we found in the accuracy of branch staff’s classification of complaints, we concluded there are gaps in the branch’s training efforts. In addition, until recently, the branch did not systematically review its staff’s classification of complaints. Specifically, in September 2014, the branch initiated the quality management team program whose primary objective is to ensure that all work the branch staff produce is reviewed to achieve a higher standard of quality. Among other things, the program staff are charged with reviewing the accuracy of the categories, subcategories, and attributes for all informal complaints. According to the branch’s Los Angeles consumer services manager, program staff are now reviewing complaints for accuracy and forwarding any errors they find to the supervisor of the branch staff who initially processed the complaint. However, the branch has yet to implement tools to measure the program’s effectiveness. We believe continuing this effort will help the branch improve the accuracy of the data and improve its value to users. Although the branch’s implementation of the BRM significantly The branch’s implementation of the reduced the error rates in CIMS data, it has had only a limited effect BRM has had only a limited effect on the overall quality of the data. Using electronic data analysis on the overall quality of the data techniques, we applied the business rules to cases the branch in CIMS. received before the BRM was implemented on November 1, 2013, in addition to cases that were received after the BRM implementation date. For 11 different categories of cases—including complaints and inquiries—we tested a total of seven unique data attributes. Cases received before BRM’s implementation date had error rates as high as 78 percent. Although we found the implementation of the business rules corrected for these types of errors, the benefits of changes such as the BRM upgrade are limited unless branch staff first correctly categorize the cases they enter into CIMS. The Branch Needs Better Guidance for Classifying Complaints Insufficient branch guidance for categorizing nonjurisdictional complaints may be skewing branch data. Specifically, while the branch provides some guidance regarding classification of 22 California State Auditor Report 2014-120 April 2015 nonjurisdictional complaints, following the guidance can result in staff classifying nonjurisdictional complaints under other subcategories, such as billing or service. Although the branch added subcategories to better describe certain nonjurisdictional complaints, such as wiring or customer service issues, depending on the circumstances of the complaint, a nonjurisdictional Of the nine general complaints subcategory may or may not be used. As mentioned earlier, we reviewed where the branch guidance directs staff to choose the subcategory that branch assigned an incorrect best describes the overarching issue related to the complaint subcategory, three involved when entering complaints into CIMS. However, three of the nonjurisdictional complaints that nine general complaints we identified on page 19 where the branch were not categorized by their assigned an incorrect subcategory involved nonjurisdictional overarching issues. complaints that were not categorized by their overarching issues. For example, in 2012 a complainant from New York asked the commission for assistance in resolving a billing dispute with her telecommunications carrier. Because the disputed bill was the overarching issue in her complaint, the guidance directs branch staff to select a billing-related subcategory. However, because this complainant lived in New York and the commission was unable to provide assistance, branch staff correctly ignored the guidance and categorized the complaint as nonjurisdictional. When branch staff follow the guidance to categorize a complaint according to its overarching issue without first determining if the commission has jurisdiction, consumer complaint data could incorrectly report the number and type of complaints that the commission has the ability to resolve. Various Commission Entities Rely on the Branch to Maintain Accurate and Consistently Classified Complaint Data The implementation of the BRM in 2013 further underscores the branch’s need to clarify its guidance for choosing subcategories. As stated in the Introduction, one of the anticipated benefits of the BRM was that it would allow the branch to improve the quality of consumer complaint data it captures at the point of entry. According to branch guidance for the BRM, choosing the correct primary subcategory is important because this selection is a key driver for how a complaint is classified in CIMS. Therefore, assigning the appropriate subcategory when recording a complaint is critical—especially when the complaint falls outside the commission’s regulatory authority. For instance, if staff select the nonjurisdictional VoIP subcategory, branch guidance provides direction about communications with the complainant and the relevant utility about the complaint. California State Auditor Report 2014-120 23 April 2015 As stated previously, we believe it would enhance the value of We believe it would enhance the the branch’s complaint data if branch guidance for processing value of the branch’s complaint complaints were updated to reflect the changes in how data data if branch guidance for are entered in CIMS according to the BRM update to CIMS. processing complaints were Specifically, branch guidance should direct staff to first updated to reflect how data are to identify whether a complaint falls within the jurisdiction of the be entered in CIMS according to the commission’s regulatory authority, and, if it does not, to choose BRM changes. an appropriate nonjurisdictional subcategory. Further, in order to provide more complete consumer complaint data, branch guidance should also direct staff to select at least one attribute, if applicable, for each nonjurisdictional complaint to better describe the nature of the nonjurisdictional complaint. The branch already provides similar guidance regarding attributes for nonjurisdictional complaints but only for those related to VoIP. The program manager indicated that, with increasing requests for branch data, changes in complaint classification guidance is an area worth exploring to help ensure that staff are accurately recording the underlying cause of complaints. As discussed in the Introduction, the branch supports the commission’s mission by helping consumers resolve disputes with certain utilities as well as providing the commission with research and analysis of trends in utility customer complaints. According to unaudited branch records between fiscal years 2011–12 and 2013–14, several commission entities, including the Enforcement division, the Office of Ratepayer Advocates, and the Administrative Law Judge Division used branch consumer complaint data to identify utility complaint trends, make policy, and aid in enforcement actions. The primary entity that used the branch’s data during our audit period was the Communications Division. The Communications Division, which assists the commission in developing and implementing policies to promote competition in all telecommunications markets and addresses regulatory changes that state and federal legislation require, made 15 of the 31 requests for branch consumer complaint data between fiscal years 2011–12 and 2013–14. The Communications Division made the requests to help it make policy, identify trends, and evaluate certain applications during these three fiscal years. For example, in February 2014 the Communications Division requested information regarding consumer complaints made about a cable company between 2011 and 2014 in order to evaluate the cable company’s application to be certified as an eligible telecommunications carrier (eligible carrier). The eligible carrier designation allows telecommunications carriers providing affordable services to certain specified users, such as low-income consumers or rural health care providers, to receive subsidies from a federal fund. The commission asserts jurisdiction 24 California State Auditor Report 2014-120 April 2015 Because commission entities over certifying a carrier as an eligible carrier in California. Because are using branch complaint entities, such as the Communications Division, are using branch information to make decisions complaint information to help them conduct evaluations such as and enforce policy, it is critical that the one just discussed, it is critical that the branch ensure that it is the branch ensure it is recording recording and providing quality consumer complaint data. and providing quality consumer complaint data. Commission divisions and offices that rely on branch consumer complaint data to make decisions and enforce policy need sound data to do so. The program manager stated that consumer complaint data have been used by the Enforcement division for preinvestigation efforts and those data may be used as evidence in formal proceedings. For example, according to the program manager, the Enforcement division may access branch complaint data after receiving a tip about a utility or identifying something of concern during an inspection of a utility. The program manager also stated that when branch data are used in formal commission proceedings, they are only one piece of evidence, and other parties to the proceedings, such as utilities, may present their own complaint data to contradict or challenge the branch’s data. The program manager’s statement underscores the importance of the need for accuracy of the branch’s complaint data. The Branch Could Increase the Quality of Its Data by More Consistently Using Attribute Fields Our review indicates the branch could do more to capture a better understanding of complainant concerns. As discussed in the previous section, complaint attributes provide branch staff an opportunity to capture information about a complaint not already expressed by the category or subcategory of the complaint. We found 16 of the 30 general complaints we reviewed had information that could have been designated as attributes. However, branch staff added only four attributes, thereby omitting information more fully describing the complaints, which, when combined in aggregate with other complaint information, might better help the commission identify consumer complaint trends. For example, the branch received a complaint in February 2013 from a consumer complaining about two issues. Although the branch categorized the complaint as a quality of service issue because of the complainant’s frustration with representatives of her phone company, she also complained that at times she had no phone service, which is an outage issue. However, branch staff did not capture the outage issue in the attribute field for this complaint, even though it is a significant issue. When we asked about the branch’s limited use of the attribute fields, the program manager agreed that the branch could better use those fields to more fully reflect the nature of consumer complaints. California State Auditor Report 2014-120 25 April 2015 The program manager also stated that one of the primary reasons that staff may not use attribute fields for complaints is because attribute fields are not required for them to help consumers resolve their complaints. However, according to the program manager, the branch does provide guidance that calls for staff to use attributes when categorizing VoIP-related complaint information into CIMS because the nonjurisdictional VoIP subcategory does not provide the detail necessary to effectively identify the nature of the complaint. When the branch does not take advantage of opportunities to record more complete complaint details, it is missing opportunities to provide the divisions and stakeholders with richer, more useful information. Ambiguity in State Law Hinders the Branch’s Ability to Track and Report Data Related to VoIP Services Because Section 710 is ambiguous about whether VoIP providers must provide information to the commission that would assist it in responding informally to VoIP complaints—a process discussed in the Introduction—the commission’s ability to identify VoIP complaints is limited. Not all VoIP providers are required to register with the commission and report to it information regarding their VoIP customers, and the commission staff do not believe they have the legal authority to compel VoIP providers to report this information. According to an October 2014 branch report, the inability to connect some complaints with VoIP providers in CIMS is a key challenge in producing reports and assisting California consumers, and the branch’s ability to process and report on VoIP complaints is directly tied to the quality of information in CIMS about VoIP providers. The branch report also noted that the branch has not been able to identify resources at the federal or state level that provide a comprehensive and consistent registry or official list of VoIP providers in California. Although the branch keeps an internal list According to a branch report, the of companies it believes to be VoIP providers, it cannot be sure how branch has not been able to identify complete or accurate this list is, because only VoIP providers whose resources at the federal or state service allows users to send and receive calls through the public level that provide a comprehensive telephone network are required to register with the commission and consistent registry or official list in order to operate in California. As such, the commission’s only of VolP providers in California. means of identifying certain VoIP carriers may be through individual complaints, and even then, complainants do not always know that they are receiving VoIP services. The branch has attempted to identify those entities that offer telecommunication services over a VoIP platform, but it cannot ensure that it knows which complaints regarding some providers relate specifically to VoIP. Some telecommunications providers, 26 California State Auditor Report 2014-120 April 2015 Some telecommunications such as AT&T and Comcast, that offer VoIP services as well as providers, such as AT&T and other telephone services have declined to provide the commission Comcast, that offer VolP services information about their VoIP offerings, including the number of have declined to provide the VoIP customers. According to an email from AT&T’s California commission information about regulatory director (director) to the branch, AT&T objects to their VolP offerings, including the providing the commission with VoIP-related information. The number of VolP customers. director explained that, among other concerns, AT&T believes Section 710 plainly and clearly prohibits the commission from regulating VoIP services, so information requests related to VoIP services cannot, as a matter of law, be rationally related to public utility regulation. A further complicating factor, according to the branch’s program manager, is that many consumers may not know whether their service is VoIP-related. These issues hinder the commission from collecting and reporting precise VoIP-related complaint information to policy-making stakeholders such as the Legislature or the FCC. We reviewed reports and websites for utility customer complaints from nine states—Colorado, Connecticut, Florida, Indiana, Maine, New York, Pennsylvania, Texas, and Vermont—which we selected based on the availability of complaint information in states of varying populations. None of the nine states we reviewed provided information in their reports regarding how they track VoIP-related complaint data, and only Vermont provided data in its utility reports for VoIP-related complaints. We also found that Connecticut reported complaints for utility providers that may be VoIP providers but did not specify that the complaints were VoIP-related. Colorado, Florida, Indiana, Maine, New York, Pennsylvania, and Texas did not address VoIP complaints at all in their reports. Similarly, we did not find any FCC reports that included VoIP-related consumer complaint data. More Staff Training Would Help the Branch Avoid Errors in Identifying and Processing VoIP‑related Complaints and Improve the Reliability of the Branch’s VoIP Data The branch has provided staff with insufficient training related to VoIP. According to its program manager, the branch initially did not provide guidance to employees for identifying and processing VoIP-related complaints because it did not have a large volume of such complaints. After Section 710 became effective in January 2013, the branch began to develop a VoIP “job aid” to clarify how staff should categorize and process VoIP-related complaints to comply with Section 710. The job aid became effective on May 31, 2013. As explained by the commission’s assistant general counsel, after Section 710 was enacted, the commission never issued a formal decision concerning the new law through its rulemaking process. The branch’s program manager explained California State Auditor Report 2014-120 27 April 2015 that the five-month delay occurred because the branch was seeking advice from the commission’s legal and communications divisions on how to process VoIP complaints and it used some of that guidance to create the job aid. In June 2013, the branch trained staff on the new VoIP job aid during its regular monthly staff meetings. According to its program manager, the branch also trained staff on processing VoIP-related complaints in September and October of 2013 during training for the introduction of the BRM. He further stated that the branch has not created additional staff training for two reasons: because the branch has not identified systemic or recurring issues and because of limited staff resources. However, of the seven VoIP-related complaints we describe on pages 19 through 21 that the branch did not correctly assign to the nonjurisdictional VoIP subcategory, five occurred after the branch provided staff with VoIP-related training. Consequently, we believe the training related to VoIP has been insufficient to ensure that the branch appropriately classifies VoIP-related complaints in CIMS. In addition to clarifying how staff should categorize and process VoIP-related complaints, the branch’s VoIP job aid requires staff to forward a nonjurisdictional complaint to a utility when that utility is a regulated utility. The VoIP job aid also requires the branch to send consumers an acknowledgment letter indicating that the branch has received their nonjurisdictional complaint, to send a closing letter that provides details about actions the utility has taken or indicating that the utility refused to respond, and to provide consumers with the FCC’s contact information. However, For nine of the 12 VolP‑related for nine of the 12 VoIP-related complaints we reviewed that complaints we reviewed that were were received after the job aid became effective in May 2013, the received after the job aid became branch either did not provide the FCC’s contact information to effective, the branch either did the complainant or did not forward the complaint to the utility. As not provide the FCC’s contact we describe on pages 20 and 21, in seven of these cases staff either information to the complainant or did not provide the FCC’s contact information or did not forward did not forward the complaint to the complaint to the utility as required because the branch did not the utility. consistently assign VoIP-related complaints to the nonjurisdictional VoIP subcategory. In the other two cases, staff correctly categorized the complaints as nonjurisdictional VoIP, but then failed to follow the job aid instructions for assisting the complainants. For example, the complainant in the February 2014 VoIP services complaint described on page 20 did not receive a letter containing the FCC’s contact information and thus was not provided information about a resource to help resolve her complaint. Branch management, including the program manager and both consumer services managers, agreed with our assessment for five of the nine complaints for which branch staff did not render appropriate assistance. However, for the other four complaints, the branch did not believe it made errors in its assistance. For example, in two instances, the branch’s Los Angeles consumer services 28 California State Auditor Report 2014-120 April 2015 manager indicated that the complaints did not relate to VoIP, and as such, the branch was not required to refer the complaints to the FCC. For another complaint, the consumer services manager stated that a complaint categorized as VoIP was about a billing and collection agency issue rather than a VoIP service issue. She further stated that the utility attempted to resolve the complaint, and therefore the branch did not have to provide the consumer with the FCC’s contact information. However, because these three complaints resulted from charges for VoIP services and the job aid instructs staff on how they should process complaints related to VoIP, we disagree with the branch’s assessment. For the fourth complaint, the branch did not perform its due diligence to verify whether the complaint related to VoIP and thus could not demonstrate that it had obtained all of the necessary information to appropriately assist the consumer. The branch procedures require staff to contact the consumer to request additional information if the complaint is incomplete. A branch reference document for the VoIP job aid also requires staff to verify whether Verizon customers receiving services delivered via fiber-optic technology have a VoIP service as Verizon offers both VoIP and non-VoIP telephone service through that technology. Yet in this complaint related to Verizon fiber-optic service, branch staff did not contact the consumer to determine whether he was receiving any VoIP services. As a result, the branch did not inform the customer of the option of contacting the FCC for assistance. The Commission Generally Processes External Data Requests Effectively As we describe in the Introduction, the commission’s Consumer Service and Information Division (division) is responsible for collecting, analyzing, and addressing customer comments and complaints directed at regulated utilities. As part of the division, the branch supports the commission’s overall mission of serving the public interest by being the first contact for consumers and acting as a conduit of information between the public, regulated utilities, and commission decision makers. We reviewed 12 of the 28 external We reviewed 12 of the 28 external data requests the branch data requests the branch received between July 1, 2011, and June 30, 2014, and we found received between July 1, 2011, that commission staff generally responded to requests quickly and June 30, 2014, and found and effectively. Although the branch did not meet its internal that commission staff generally benchmark for clarifying the request with the requestor within responded to requests quickly two business days in five of the 12 requests we reviewed, we and effectively. identified no harm from these exceptions. The branch gathered the appropriate requested information in all five of these instances. In three of these five instances, the branch contacted the requestor California State Auditor Report 2014-120 29 April 2015 between four and seven business days after it received the request. In the other two instances, branch data show that the branch did not contact the requestor to clarify the parameters of the request at all; however, in these two cases, we found the branch’s delays did not materially affect the commission’s resolution of the data request. The commission responded to all five requests either by the agreed-upon time or within two weeks of receiving the request. However, as part of our testing, we did identify two related requests from the California Senate Committee on Energy, Utilities, and Communication (committee) that stood out from the other data requests we reviewed. Available documentation indicates that the committee’s expectation may not have been fully met, primarily because the committee staff may have expected to receive information that the commission did not initially provide. The commission did, however, produce analysis that it believed was responsive to the committee staff’s request. According to the branch program manager, the branch has proposed, but not formally adopted, modifications of its procedures to allow legislative staff to send data requests simultaneously to the commission’s Office of Governmental Affairs and to the branch. The program manager indicated that this change would reduce the potential for miscommunications regarding data requests. We believe this proposal, if implemented, could help the commission avoid miscommunications in the future. The Commission Does Not Effectively Use Its Website to Make Complaint Data Available Although the commission posts telecommunications consumer contact and informal complaint data (complaint data) on its website, it could make this information more robust and more easily accessible to consumers and other stakeholders. Contacts are communications from consumers directed to the branch in reference to concerns, questions, and complaints related to utility companies. One of California’s telecommunications policies, as described in the California Public Utilities Code, is to encourage the fair treatment of consumers by providing them with sufficient information for making informed choices. In accord with this policy, in 2012 the branch began posting complaint data on the commission website. According to the branch project plan for improving the quality and type of complaint data posted on its website, the branch is to publish complaint data quarterly for each of the prior three months. It publishes these data both in a pie chart and in tables with complaint data broken down by utility company and category of complaint. Branch records also indicated that the branch has separately published, since 2011, a subset of complaint data dating back to 2009 about contacts and informal complaints 30 California State Auditor Report 2014-120 April 2015 from customers who communicate with the branch in a language other than English, known as limited English proficiency (LEP) consumers. When publishing LEP data, the branch follows the guidelines laid out in a 2010 commission resolution that states that the commission must annually post LEP data sorted by utility company. The branch also normalizes the LEP data by reporting it in terms of the number of complaints for each utility per 100,000 customers; this provides consumers with the ability to compare carriers of different sizes. The branch follows the guidance in the resolution and ranks the normalized rates for each utility from highest proportion of complaints and contacts to the lowest. The commission has not yet fully The commission has not yet fully implemented its project plan. implemented its project plan for According to the branch’s analytical unit supervisor, the branch improving the quality and type developed the project plan in December 2013. The project plan of complaint data posted on specifies what complaint data the branch plans to post and its website. outlines the timeline for posting the data. For example, the plan indicates that by January 2014 the branch will include on its website counts of contacts and complaints by company and category and a count of closed informal complaints. The plan also indicates that by April 2014 the branch will post the consumer satisfaction ratio, which is the number of closed informal complaints resolved either in the consumer’s favor or as a compromise between the consumer and utility divided by the total number of closed informal complaints for that utility, and it will consider posting a separate report on VoIP complaints by July 2014. However, as of January 2015, the branch had posted only counts of complaints by utility and category online. The primary reason for the branch’s delay in meeting the milestones in its timeline appears to be a lack of resources. In a June 2014 update to the branch project plan, the branch indicated that it had to indefinitely defer full implementation of the project plan for improving data quality because posting the complaint data was more resource-intensive than expected. By way of comparison, in our review of online reporting of complaint data in four other states, we found that the commission’s counterpart in Vermont has a separate section for complaints against VoIP providers in its online reports. A second state also publishes reports online with more robust data regarding its telecommunications utilities’ responsiveness to consumer complaints than the commission’s website postings. Specifically, the report of the New York Public Service Commission (New York commission) contains metrics about consumer satisfaction, complaint response time, and pending cases. The New York commission also normalizes its complaint statistics similar to the way the California commission normalizes its LEP data. California State Auditor Report 2014-120 31 April 2015 However, the branch’s program manager believes the commission has limited ability to obtain the number of access lines, or consumers, each utility has for normalization purposes. For example, in 2014, Comcast Phone was not receptive to a commission request for this information. A lawyer for Comcast Phone indicated that the company provides wholesale and other business services that do not meet the branch’s definition of access line. Without knowing the number of access lines, the commission cannot normalize its customer complaint data. Further, the branch’s analytical unit supervisor believes it is not feasible to normalize complaint data, given its current schedule for posting data on the commission’s website, because doing so would require more time and resources than the branch currently has. The branch’s analytical unit supervisor indicates the branch is able to normalize LEP complaint data because it involves fewer utilities and less data, and it is published annually rather than quarterly. The branch’s analytical unit supervisor also stated that the commission always envisioned posting complaint data online as an evolving project. However, until the branch improves the quality of the complaint data it posts online, the ability of consumers and other stakeholders to use the data to make informed choices or decisions is limited. Although the commission provides consumers the opportunity to Although the commission provides offer feedback and suggestions on its LEP complaint data on its consumers the opportunity to website, its efforts in this regard have been largely ineffective. A offer feedback and suggestions commission resolution in 2010 required the branch to implement on LEP complaint data on its a mechanism that allows consumers to provide suggestions website, its efforts have been for LEP data. The resolution also stated that the branch should largely ineffective. regularly review this feedback, the data posting process, and the actual published data to identify ways in which it can improve the posted information to make it more easily accessible as well as understandable to consumers. In accordance with this resolution, according to branch records, in June 2011 the branch posted a link to a consumer feedback survey on the commission’s website for LEP data. However, as of November 2014, the commission had received only one comment since it established the link and that comment was not relevant to the data or the website. The branch’s analytical unit supervisor indicated that the branch has never considered soliciting feedback for complaint data, in part because the commission’s consumer survey for LEP data has not produced any usable feedback. In the March 2013 CIMS Upgrade Project Charter, commission staff indicated that accurate and meaningful consumer complaint reports will enable consumers to make better informed decisions about their utility services. However, without any consumer feedback, the commission cannot be sure that the data it posts on its website are meaningful or provide benefit to the public. 32 California State Auditor Report 2014-120 April 2015 Further, the complaint data the commission posts on its website are difficult to find. For example, to find those data, consumers must follow three links to navigate from the commission’s home page to the Consumers section, to the Consumer Affairs Branch section, and finally to the Contacts Data section. This navigation is not intuitive—an individual likely would not be able to find the information without knowing beforehand of its existence. In contrast, consumers can reach the LEP data directly from a link on the home page of the commission website. The branch’s analytical unit supervisor stated that the branch asked that the complaint data be more accessible, which led to the commission’s web master creating a Consumers section on the commission’s home page in December 2013. According to commission staff, the complaint data web page received only 915 unique visitors in 2014— a relatively low view rate given that the branch processed roughly 22,000 telecommunications contacts in fiscal year 2013–14. Although other factors also likely contributed, we believe the relative inaccessibility of the web page may be a factor in the low view rate for the complaint data. The division’s director indicated that the commission has begun an effort to revamp its website and that it will be a completely new site. The commission’s director of administrative services indicated that the target for completing the new website is Fall 2015. As the commission proceeds with its website revision, it should consider ways to make navigating to its consumer complaint data more intuitive and direct. Recommendations Legislative To ensure that the commission has the information it needs to better report on VoIP-related complaints, the Legislature should give the commission the authority to collect information from providers regarding their VoIP customers and require VoIP providers to furnish this information to the commission. Commission To ensure that policy makers, enforcement officials, and the general public have access to accurate consumer complaint data in CIMS, the branch should do the following: • Update and provide further training to its staff on properly classifying complaints by September 30, 2015. • Continue to implement its quality management team program component focused on reviewing the categorization of complaints and correcting identified errors. California State Auditor Report 2014-120 33 April 2015 • Develop and implement tools by September 30, 2015, to measure the quality management team program’s effectiveness. • Update by June 30, 2015, its guidance for categorizing complaints to better integrate with the BRM. For example, the guidance should specify that nonjurisdictional complaints should be classified as such. To ensure that policy makers, enforcement officials, and the general public have access to more complete and meaningful consumer complaints data in CIMS, the branch should, to the fullest extent possible, include the attributes of each complaint in the data it records in CIMS. To ensure that branch staff provide the appropriate assistance to consumers with VoIP-related complaints, the branch should, by September 30, 2015, further train its staff on the requirements of the VoIP job aid and on providing correspondence to complainants as its guidelines require. To ensure that consumers have access to complaint data that will enhance their ability to make informed choices about their telecommunication services, the branch should, by June 30, 2015, create an updated plan that specifies the types of data the branch intends to post online and a timeline for fully implementing that plan. To ensure that it can assess the value to the public of the complaint data it presents on its website, the branch should create a process for those who view its complaint data to provide feedback to the branch including, if necessary, modifying the survey that it uses to collect feedback on LEP data. To ensure that the public can easily locate customer complaint data the branch publishes on its website, the commission should make navigating to its customer complaint data more intuitive and direct. 34 California State Auditor Report 2014-120 April 2015 Blank page inserted for reproduction purposes only. California State Auditor Report 2014-120 35 April 2015 Chapter 2 THE CALIFORNIA PUBLIC UTILITIES COMMISSION HAS POOR GENERAL CONTROLS OVER ITS INFORMATION SYSTEMS, COMPROMISING THE CONFIDENTIALITY, INTEGRITY, AND AVAILABILITY OF ITS INFORMATION Chapter Summary We identified pervasive weaknesses in the general controls the California Public Utilities Commission (commission) has implemented over its information systems. The State’s information assets—including its data processing capabilities, information technology infrastructure, and data (information assets)—are an essential public resource. Implementing appropriate security measures and controls is critical to ensuring state entities’ business continuity and to protecting their information assets. Consequently, we expected that the commission would have well-developed plans, policies, and procedures related to its information systems’ general controls. Chapter 5300 of the State Administrative Manual (SAM) provides a framework that many state entities must follow to protect their information assets from a wide spectrum of threats and risks. Although the commission is not required to comply with the policy requirements in SAM Chapter 5300, its assistant general counsel stated that it complies with these requirements because they are good business practices. Therefore, we used the policy requirements contained in SAM Chapter 5300 as the benchmark against which we evaluated the general controls the commission has implemented over its information systems. Although the commission had certified to the California Department of Technology (CalTech) that it complied with all policy requirements in SAM Chapter 5300, we found that key information security documents either were nonexistent or lacked critical components, as summarized in Table 2 on the following page. Specifically, the commission’s inventory of its information assets is incomplete, and it has not assessed the risks to its assets. Likewise, the commission has not developed an information security plan or an incident response plan. Finally, the commission’s technology recovery plan lacks key elements. The remainder of this chapter details these results and describes how the weaknesses we identified could compromise the confidentiality, integrity, and availability of the information systems the commission currently uses to perform its day-to-day operations. 36 California State Auditor Report 2014-120 April 2015 The Commission Has Not Developed a Program to Address Its Information Security Needs The commission does not have an information security program, which places its information assets at risk of misuse, loss, disruption, or compromise. An entitywide information security program is the foundation of a security control structure and a reflection of senior management’s commitment to addressing security risks. Without a well-designed information security program, security controls may be inadequate; responsibilities may be unclear, misunderstood, or improperly implemented; and controls may be inconsistently applied. The information security program should establish a framework and continuous cycle of activity for assessing risk, developing and implementing effective security procedures, and monitoring the effectiveness of these procedures. However, the commission has yet to complete the three activities that are a prerequisite for developing a meaningful information security program: identifying and understanding the value of its information assets, considering the risks that exist to those assets, and developing a comprehensive information security plan to address those risks. Table 2 Status of Selected Information Security Components Required by the State Administrative Manual INFORMATION SECURITY PROGRAM COMPONENT DESCRIPTION STATUS Inventory of information assets A listing of all programs and information systems identified as collecting, using, maintaining, or sharing information owned by the state entity that identifies their value and required level of protection. Risk assessment The process of identifying risks to operations, assets, individuals, and other organizations arising 5 through the use of information technology. Information security plan An overview of security requirements for the information security program, as well as the controls 5 in place or planned to ensure that requirements are met. CONTINGENCY PLANNING COMPONENT DESCRIPTION STATUS Incident response plan A set of predetermined instructions or procedures to detect, respond to, and limit consequences of 5 malicious cyber attacks against an entity’s information assets. Technology recovery plan A description of the resources, tasks, and data required to manage the entity’s recovery from a disaster or other interruption to its critical information technology systems. Sources: California State Auditor’s assessment of the State Administrative Manual and documentation provided by the California Public Utilities Commission (commission). 5 The commission has not completed this required component. The commission has developed this required component, but it is deficient. California State Auditor Report 2014-120 37 April 2015 Identifying information assets and understanding their value are critical for determining the level of protection those assets require. However, the commission has failed to develop a comprehensive inventory for all of its information assets that identifies whether they are confidential and critical to its business operations. SAM requires state entities to establish and maintain an inventory of all their information assets that, among other things, identifies the owners, custodians, and users of each information asset, in addition to the importance of each asset to the execution of the entity’s mission and program function. Although the commission has established an inventory of certain information assets, such as desktop and laptop computers, physical and virtual servers, and various applications, it has stopped significantly short of identifying all of its information assets. For example, the commission has yet The commission has yet to to inventory the types of data its systems collect and which systems inventory the types of data its contain data that are critical to the commission’s ability to carry out systems collect and which systems its key business functions. contain data that are critical to the commission’s ability to carry out its In addition, SAM requires state entities to categorize the required key business functions. security of their information assets based on the potential impact that a loss of confidentiality, integrity, or availability of such information would have on their operations and assets. We asked the commission to identify for us all of its information systems that contain confidential or sensitive data and those information assets that are critical to its ability to carry out its mission. However, since the commission has yet to inventory all of its information assets, we were not surprised that it was unable to fully identify the types of confidential and sensitive data it maintains and the information systems that are most critical to its business functions. According to the commission’s director of administrative services (administrative services director), it has not managed its information assets in a centralized manner. She explained that because its information systems have different owners, the commission does not maintain consistent documentation that describes the nature of the data these systems contain. By maintaining a comprehensive inventory of its information assets, an entity can perform a meaningful risk assessment to identify the potential threats to its information assets and whether its assets lack sufficient protection from those threats. SAM requires state entities to develop a risk management and privacy program that includes the identification and prioritization of critical information technology applications. Further, SAM requires state entities to conduct a comprehensive risk assessment once every two years to identify, among other things, the threats to its information assets and the points where information assets lack sufficient protection from those threats. However, the commission has yet to develop a risk management and privacy plan and to assess the risks that exist to its information assets. According to the interim chief 38 California State Auditor Report 2014-120 April 2015 information officer (information officer), the commission intends to hire a contractor to create a risk management and privacy program and perform a risk assessment. The administrative services director asserted that the commission has allocated funds for this contract and that it is currently developing the specifications for this work, which it hopes to release to eligible contractors for their consideration in May or June 2015. Until the commission inventories all of its information assets and assesses the risks that currently exist to them, it cannot be certain that it has identified and considered all threats and vulnerabilities to its information systems, that it has addressed the greatest risks, and that it has made appropriate decisions regarding which risks to accept and which to mitigate through security controls. When an entity understands the value of its information assets and the risks that may compromise them, it is then poised to establish appropriate administrative, operational, and technical policies, standards, and procedures to provide for the protection of its information assets. However, the commission does not have an information security plan that fully documents its security policies and procedures. Consequently, the commission cannot ensure that it is providing for the protection of information assets and preventing illegal activity, fraud, waste, and abuse of its information assets. SAM states that state entities shall develop, implement, and maintain an entitywide information security plan that provides, among other things, a description of the state entity’s strategy and approach to prioritizing information security, privacy, and risk management. According to the administrative services director, the commission does not have a comprehensive information security plan. Although the commission was able to provide us with a draft information security plan from January 2012, she stated that this plan was never completed nor was it implemented. Rather, in December 2014, the commission assigned a retired annuitant to assist in developing a comprehensive information security plan that accurately reflects the commission’s operating environment. However, the administrative services director was unable to provide us with a projected date for the completion of this plan. In the absence of an entitywide information security plan, the commission Until the commission completes risks spending too little time and money on controls that protect and implements an entitywide sensitive or critical information assets and it risks overspending on information security plan, it will controls over low-risk resources. Until the commission completes continue to be at risk of misuse, and implements an entitywide information security plan, it will loss, disruption, or compromise of continue to be at risk of misuse, loss, disruption, or compromise state information assets. of state information assets. California State Auditor Report 2014-120 39 April 2015 The Commission Has Poor Controls for Contingency Planning The commission has not developed an incident response plan, nor has it developed a sufficient technology recovery plan to properly respond to unplanned interruptions that could jeopardize the information assets the commission needs to maintain its day-to-day operations. Contingency planning represents a broad scope of activities designed to sustain and recover critical information technology services following unexpected events, such as accidental and deliberate acts by state entity personnel or individuals external to the entity or natural disasters. Losing the capability to process, retrieve, and protect electronically maintained information can significantly affect an entity’s ability to accomplish its mission. If contingency planning controls are inadequate, even relatively minor interruptions can result in lost or incorrectly processed data, which can cause financial losses, expensive recovery efforts, and inaccurate or incomplete information. Given these severe implications, it is critical that an entity have in place procedures for protecting information resources and minimizing the risk of unplanned interruptions and a plan to recover critical operations should interruptions occur. The Commission Lacks an Incident Response Plan The commission has yet to develop documented procedures to The commission has yet to respond to, report on, and recover from an information security develop a comprehensive incident incident, such as a malicious cyber attack against its information response plan to ensure it is assets. Proper incident management includes the formulation poised to respond to, report on, and adoption of a written incident response plan that provides a and recover from an information predetermined set of instructions or procedures to detect, respond security incident, such as a to, and limit the consequences of an incident that actually or malicious cyber attack against potentially jeopardizes an information asset. In addition, incident its information assets. management includes the application of lessons learned from incidents, together with the development and implementation of appropriate corrective actions directed at preventing or mitigating the risk of similar occurrences in the future. Without prompt and appropriate responses to security incidents, violations could continue to occur and cause damage to an entity’s resources indefinitely. Further, violators will not be deterred from continuing to gain inappropriate access, which could result in disclosure of confidential information, financial losses, and embarrassment to the entity. For this reason, SAM requires state entities to develop, disseminate, and maintain a formal, documented incident response plan that provides for the timely assembly of appropriate staff who are capable of developing a response to, appropriately reporting about, and successfully recovering from a variety of incidents. However, 40 California State Auditor Report 2014-120 April 2015 according to the information officer, the commission does not have a plan, policy, or procedure that addresses how to manage and investigate an information security incident. Rather, he stated that the commission intends to include the development of an incident response plan as part of the contract previously discussed on page 38. Until the commission develops a comprehensive incident response plan, it cannot ensure that it is poised to properly respond to and recover from an information security incident. The Commission’s Business Continuity Planning Efforts Are Insufficient The commission’s business continuity efforts have also fallen short of the requirements outlined in SAM. As a result, the commission cannot ensure that critical information assets are protected and available following an interruption or disaster. SAM requires state entities to create a business continuity plan that includes procedures for how the state entity will stay functional in a disastrous situation; those procedures include the creation of a technology recovery plan (recovery plan). Further, California’s Statewide Information Management Manual (SIMM) details the minimum requirements that state entities recovery plans should include. Although the commission was able to provide us with its January 2015 recovery plan, our review of the document noted several deficiencies. According to SIMM requirements, the recovery plan should include a description of critical business functions and their supporting applications, a designation of the maximum acceptable outage time frames for applications supporting critical business functions, and a description of the recovery strategy We found that the commission’s that supports these critical applications. However, we found that recovery plan does not the commission’s recovery plan does not consistently identify consistently identify applications applications supporting critical functions and it does not speak supporting critical functions and to maximum acceptable outage time frames or recovery strategies. it does not speak to maximum Having a clear understanding of what information systems support acceptable outage time frames critical business functions and the amount of time that these or recovery strategies. systems can be unavailable is a prerequisite to determining recovery priorities and developing clear recovery strategies. Without this information, the commission risks spending important time and resources recovering noncritical systems instead of those that are truly critical to its business functions. Moreover, the commission cannot effectively prioritize recovery efforts to minimize cost, system unavailability, and loss of data. Further, the commission’s recovery plan does not comply with SIMM requirements by including the identification and evaluation of alternative recovery strategies. Specifically, the commission’s recovery plan does not specify its process for rebuilding its California State Auditor Report 2014-120 41 April 2015 technology infrastructure at an alternate site should the primary site become unusable. In particular, the commission’s current recovery plan does not contain detailed information on appropriate alternate locations and the types of facilities and equipment that the commission will require at these alternate sites. Moreover, the recovery plan also does not provide detailed procedures that would allow another trained information technology professional to recover the commission’s infrastructure should those with primary responsibility be unavailable during the recovery process. While the administrative services director indicated that the commission would sustain its critical business functions using manual processes until the technology infrastructure could be rebuilt at another location, the commission was unable to provide a documented plan for using these manual processes and it has not conducted an analysis to determine how long it could function using manual processes. Additionally, the commission cannot ensure that its plan to rebuild its infrastructure at another location could be accomplished successfully. As previously mentioned, the commission’s current recovery plan does not identify maximum acceptable outage time frames for applications supporting critical business functions and a description of the recovery strategy that supports these critical applications. Without a clear understanding of alternate site recovery and how quickly key information systems need to be brought back online, the commission risks not having sufficient alternate processing locations available to meet its needs should the primary site become unusable. Finally, despite the requirements outlined in SAM and SIMM, the commission does not conduct regular tests and exercises The commission does not conduct to identify any deficiencies in its recovery plan to further refine regular tests and exercises to the plan. Rather, during our review of the commission’s recovery identify any deficiencies in its plan, we noted that it includes steps to be followed when planned recovery plan to further refine the or unplanned power outages occur. Although the commission’s plan as state policy requires. administrative services director asserts that it successfully recovered from an unplanned power outage in December 2014, the absence of regular testing does not allow the commission to determine the effectiveness of all components of the plan and identify potential weaknesses in the current plan. Further, the commission cannot assess its organizational readiness to execute the plan. The administrative services director stated that the commission intends to address the deficiencies we identified in its next update of the recovery plan in January 2016. However, until the commission creates and implements a recovery plan that meets all requirements in SAM and SIMM, it cannot ensure that critical information assets are protected and available following an interruption or a disaster. 42 California State Auditor Report 2014-120 April 2015 The Commission Repeatedly Misrepresented its Information Security Posture to CalTech Despite the pervasive weaknesses we identified in the general controls that the commission has implemented over its information systems, the commission did not accurately represent its information security posture to CalTech in its annual self-certifications. Specifically, rather than communicating to CalTech the weaknesses that exist in its information system controls, the commission repeatedly certified that it was fully compliant with the requirements prescribed in Chapter 5300 of SAM. By omitting this key information from its certifications, the commission conveyed a false sense of security to CalTech and thus allowed its poor information security posture to persist. CalTech is the primary state government authority responsible for ensuring the confidentiality, integrity, and availability of state information systems and applications and for protecting the State’s information. Accordingly, to safeguard the State’s information assets from a wide spectrum of threats and risks, state law requires many state entities to comply with the information security and privacy policies, standards, and procedures issued by CalTech in Chapter 5300 of SAM. To demonstrate their acknowledgement of these requirements and provide a measure of accountability, CalTech requires state entities under its purview to self-certify their compliance with Chapter 5300’s requirements by January 31 of each year. Although the commission is not expressly required to certify its compliance to CalTech each year, it asserted to CalTech in 2014 and 2015 that it had complied with all policy requirements in SAM Chapter 5300. However, as previously discussed in this chapter, we determined that the commission is significantly out of compliance with these requirements. Further exacerbating the issue, we interviewed the commission’s information officer on January 26, 2015, to learn about the controls the commission has implemented over its information systems. During our interview, the information officer acknowledged that the commission has not created or maintained several key information security documents that SAM Chapter 5300 requires. Nonetheless, three days later, his supervisor—the administrative services director—signed the commission’s 2015 self-certification to CalTech asserting that the commission was fully compliant with the requirements in Chapter 5300. The commission asserts that it recognizes the good business practices in complying with the SAM Chapter 5300 requirements. However, because the commission was disingenuous about its compliance with SAM Chapter 5300 when submitting its self-certifications to CalTech, it has allowed the deficiencies with California State Auditor Report 2014-120 43 April 2015 its information system controls to persist for multiple years. If the commission had been truthful in its self-certifications and acknowledged that it had not implemented all required components By not accurately depicting its of SAM Chapter 5300, the certification form would have prompted information security control it to submit a remediation plan to CalTech identifying the areas environment and failing to of noncompliance along with timelines indicating when it would submit the remediation plans meet the requirements. However, by not accurately depicting its to CalTech, the commission was information security control environment and failing to submit the negligent in its commitment remediation plans to CalTech, the commission was negligent in its to rectify its information commitment to rectify its information security weaknesses. security weaknesses. When we asked the administrative services director about this issue, she stated that the commission did not intend to mislead CalTech as to its information security status. Thus, as a result of our audit, the commission asserted that it submitted an amended certification to CalTech in early March 2015— including a remediation plan—that indicates the commission has not yet implemented all components that SAM Chapter 5300 requires. However, until the commission implements adequate controls, it risks the confidentiality, integrity, and availability of its information systems, which could contain confidential and sensitive information. Recommendations The commission should ensure that it complies with all policy requirements in SAM Chapter 5300 no later than April 2016. Specifically, the commission should do the following: • Develop, implement, and maintain an entitywide information security program. – Complete and maintain an inventory of all its information assets, specifically categorizing the level of required security of the information assets based on the potential impact that a loss of confidentiality, integrity, or availability of such information would have on its operations and assets. – Develop a risk management and privacy plan and conduct an assessment of the risks facing its information assets. – Develop, implement, and maintain an information security plan. • Develop, disseminate, and maintain an incident response plan. 44 California State Auditor Report 2014-120 April 2015 • Revise its existing recovery plan to include the following: – A list of applications supporting critical business functions, their maximum acceptable outage time frames, and detailed recovery strategies for each application. – Detailed procedures for rebuilding its technology infrastructure at an alternate processing site. • Conduct regular tests and exercises to assess the sufficiency of the revised recovery plan and refine the plan when necessary. • Ensure that any certifications it submits to CalTech accurately represent its information security environment. We conducted this audit under the authority vested in the California State Auditor by Section 8543 et seq. of the California Government Code and according to generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives specified in the scope section of the report. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Respectfully submitted, ELAINE M. HOWLE, CPA State Auditor Date: April 9, 2015 Staff: John Billington, Audit Principal Richard Power, MBA, MPP Nisha Chandra Veronica Perez, MPPA Ray Sophie, MPA Legal Counsel: Scott A. Baxter, Sr. Staff Counsel IT Audit Support: Michelle J. Baur, CISA, Audit Principal Ben Ward, CISA, ACDA Sarah Rachael Black, MBA, ACDA Ryan P. Coe, MBA, CISA Gregory D. Martin, CPA For questions regarding the contents of this report, please contact Margarita Fernández, Chief of Public Affairs, at 916.445.0255. California State Auditor Report 2014-120 45 April 2015 * 1 * California State Auditor’s comment appears on page 51. 46 California State Auditor Report 2014-120 April 2015 California State Auditor Report 2014-120 47 April 2015 48 California State Auditor Report 2014-120 April 2015 California State Auditor Report 2014-120 49 April 2015 50 California State Auditor Report 2014-120 April 2015 Blank page inserted for reproduction purposes only. California State Auditor Report 2014-120 51 April 2015 Comment CALIFORNIA STATE AUDITOR’S COMMENT ON THE RESPONSE FROM THE CALIFORNIA PUBLIC UTILITIES COMMISSION To provide clarity and perspective, we are commenting on the California Public Utilities Commission’s (commission) response to our audit. The number below corresponds to the number we placed in the margin of the commission’s response. 1 The commission states that our report implies that it is not assisting consumers. On the contrary, the purpose of this audit was not to conclude on the commission’s overall customer assistance efforts, nor does our report do so. However, based on our testing there were two examples where that appeared to be the case. Specifically, on pages 27 and 28, we noted that staff did not follow commission guidance for assisting consumers who complained about issues relating to Voice over Internet Protocol services in nine of 12 instances we reviewed. As a result, the consumers either did not receive assistance with contacting their utility or did not receive information about resources to help them resolve their complaints. Further, on page 29, we state that the commission could make the consumer contact and complaint data it posts on its website more robust and easily accessible to consumers and other stakeholders.