Summary

Public Utilities Commission: Did Not Effectively Manage Its Contract for Investigating San Francisco’s December 1998 Power Failure May 1999 99117.1 You can obtain reports by contacting the Bureau of State Audits at the following address: California State Auditor Bureau of State Audits 555 Capitol Mall, Suite 300 Sacramento, California 95814 (916) 445-0255 or TDD (916) 445-0255 x 248 OR This report may also be available on the World Wide Web http://www.bsa.ca.gov/bsa/ Permission is granted to reproduce reports. C S A ALIFORNIA TATE UDITOR KURT R. SJOBERG MARIANNE P. EVASHENK STATE AUDITOR CHIEF DEPUTY STATE AUDITOR May 14, 1999 99117.1 Governor of California President pro Tempore of the Senate Speaker of the Assembly State Capitol Sacramento, California 95814 Dear Governor and Legislative Leaders: As requested by the Joint Legislative Audit Committee, the Bureau of State Audits presents its report entitled Public Utilities Commission: Did Not Effectively Manage Its Contract for Investigat- ing San Francisco’s December 1998 Power Failure concerning the Public Utilities Commission’s consultant contract for investigating the power failure. SUMMARY I n contracting with a consultant to investigate the massive power failure that struck the San Francisco Bay Area on December 8, 1998, the California Public Utilities Commission (commission) inadequately monitored its consultant’s contract and failed to ensure that the consultant’s expenditures were reasonable and remained within budgeted amounts. The electrical power outage, caused by a system disturbance at a Pacific Gas and Electric Company (PG & E) substation, left more than one million people in the San Francisco Bay Area without electricity for up to 7.5 hours. Because it did not have adequate technical expertise to explore the causes of the outage and to recommend methods for preventing a recurrence, the commission awarded a $400,000 contract to an outside consulting firm (consultant) that would conduct the investigation. The contract required the consultant to draw expert conclusions and prepare a report suitable for litigation purposes related to the power failure. BUREAU OF STATE AUDITS C A L I F O R N I A S T A T E A U D I T O R 1 660 J Street, Suite 300, Sacramento, California 95814 Telephone: (916) 445-0255 Fax: (916) 327-0019 Even though contracting with the consultant is reasonable, the commission is unable to demonstrate that it evaluated the qualifications of the consultant’s subcontractors. The commission also did not make certain that the consultant’s report contained sufficient detail and analysis to support all of the report’s conclusions. For example, the report has invited criticism because it concludes that “PG & E has an error prone work culture that tends to bypass procedures and work practice requirements.” However, the report does not specify the methodology or detailed analysis that the consultant used to arrive at this conclusion. The commission has agreed to pay an additional amount so that the consultant, which should have submitted a complete analysis, can provide further support for the report’s conclusions. Additionally, the commission based the contract amount on broad estimates that it cannot substantiate, and it has not required the consultant to submit invoices so that the commission can ensure that expenditures for the investigation have been appropriate and within the contract’s budgeted amounts. BACKGROUND The California Public Utilities Commission consists of five commissioners (currently two positions are vacant) appointed by the governor, with Senate approval, for six-year terms. The commission also has a staff of more than 800 employees, including economists, engineers, administrative law judges, accountants, lawyers, administrative and clerical support staff, and safety and transportation specialists. The commissioners make policy, procedural, and program decisions that guide the regulation of energy, telecommunications, water, and transportation utilities. Further, the commissioners initiate investigations and rulemakings, or proceedings that establish policies or rules, to explore broad policy issues, resolve procedural matters, examine allegations of illegal activity at utilities, or respond to legislative requirements. In December 1998, the commission initiated an investigation into the power outage in PG & E’s electrical system in the San Francisco Bay Area that left more than one million people without electricity. The December 8 power failure affected businesses, government agencies, transportation, hospitals, and individuals. 2 C A L I F O R N I A S T A T E A U D I T O R What Caused the Power Outage? Work crew errors at the San Mateo substation initiated PG & E’s system disturbance. In November 1998, a crew de-energized equipment and lines for maintenance work and removed them from service. This process included disconnecting relays used to detect and terminate faults, or short circuits, in the equipment and lines. The crew applied 15 protective grounds as a safety measure and began work at the substation. When the crew completed the work in early December 1998, it failed to remove some of the protective grounds from the system. In addition, the substation operator failed to reengage the protective relays that would have opened the circuit breaker if a fault condition occurred. The protective relay’s purpose is to clear a problem as close to the source as possible by isolating that portion of the system and routing power around it. Because the equipment was energized and had the protective grounds in place and relays disengaged, protective relays at another portion of the system had to operate to clear the fault that occurred on December 8. As a result, this fault took more than one-half second to clear, causing cascading problems within the system. Although one-half second may seem insignificant, the power outage spread to its ultimate size in fewer than 12 seconds, leaving more than one million people throughout most of San Francisco and nearby communities on the San Francisco Peninsula without power for 3 to 7.5 hours. The Commission Ordered an Investigation of the Power Failure Because the power failure’s size and length were so severe, the commission ordered PG & E as well as its own staff to investigate why the outage spread from the source to so many customers and also to determine what measures could prevent a recurrence of a similar electrical failure. In its investigation order, the commission declared its need to be assured, as soon as possible, that similar power outages will not compromise public safety and welfare in the future. The California Independent System Operator (CAISO) and San Francisco’s city attorney also investigated the power outage’s causes. According to the commission, it did not have the staff available to undertake an expert review quickly or to testify as expert witnesses and so it contracted with a consultant for its analysis of the power failure. C A L I F O R N I A S T A T E A U D I T O R 3 The Commission’s Consultant Used Root Cause Analysis The commission’s consultant used a “root cause analysis” to understand the “roots,” or underlying causes, of the power outage and to recommend cost-effective actions to prevent recurrences. The commission chose this approach because it was aware that PG & E was also using a root cause analysis methodology. Interestingly, however, the depth of the commission’s and PG & E’s analyses varied and led to dissimilar conclusions about the fundamental causes for the system failure. For instance, although both entities examined the operators’ errors that led to the system failure, the commission’s consultant went further by reviewing PG & E’s management controls and processes. Whereas PG & E states that the underlying cause of the power outage was operator errors, the commission’s consultant reported that the basic cause was the extent to which PG & E’s procedures and processes were prone to error. In other words, the consultant does not simply attribute the outage to operator errors. Although the consultant’s investigation is under criticism for focusing on management controls and processes, the commission defends its position that the investigation needed to include an evaluation of PG & E’s management controls to ensure that a similar outage will not recur. Jurisdictional Confusion Occurred During the Investigative Process The California Legislature restructured the State’s public utilities in 1996 and established CAISO to maintain centralized control of the electrical transmission grid and to ensure efficient use and reliable operation of this transmission system. Statutes direct CAISO to review each major outage and therefore to address the cause of an outage like the one that occurred in December 1998, to evaluate the utility’s response time and effectiveness, and to assess whether the owner’s or operator’s practices enhanced or undermined the facility’s ability to restore service efficiently. Under the restructuring, the commission remained responsible for ensuring reliability of the distribution systems of investor- owned electric utilities but not the transmission systems they may operate. PG & E’s power failure on December 8 was the first major outage since the Legislature restructured the State’s public utilities. The commission, CAISO, and other entities appear to agree that the power failure began with the transmission system and ultimately affected the distribution system. Because the power outage affected both the transmission and distribution systems, 4 C A L I F O R N I A S T A T E A U D I T O R the lines of jurisdiction between CAISO and the commission became unclear. Several entities have raised concerns about the confusion over jurisdiction and the need for additional direction. CAISO believes that it was appropriate for the commission to ask questions related to the transmission system to ensure reliability of the distribution system, which is under the commission’s jurisdiction. However, CAISO also believes that the commission’s further investigation into transmission- related problems was unnecessary and beyond the commission’s jurisdiction. The commission believes that PG & E did not go far enough in its investigation, and this limitation resulted in the commission’s need to perform additional work. San Francisco’s city attorney issued a report on the outage and believes the depth of the commission’s investigation related to the outage’s root causes was appropriate. Additionally, San Francisco’s report raises concerns about the current regulatory scheme, stating that if the commission had not initiated its investigation, electric utility consumers and other interested parties, such as the city, may not have had the opportunity to be represented or heard. Clearly, parties have taken positions on both sides of the jurisdictional debate. Recently, both the Electricity Oversight Board (board), which oversees CAISO, and the commission have taken steps to correct the problems experienced during this event. Specifically, the board’s executive director has recommended that CAISO coordinate several of the investigative reports and related comments and that CAISO also develop a corrective action plan for PG & E. In addition, the commission is proposing that it and CAISO develop formal investigation protocols and memorialize these protocols in a Memorandum of Understanding. SCOPE AND METHODOLOGY The Joint Legislative Audit Committee requested the Bureau of State Audits to review the commission’s contract with its consultant, Performance Improvement International, related to the December 1998 power outage. We focused on whether the consultant’s work and report conformed with the five-member commission’s direction, and we reviewed the commission’s order of investigation, the contract, related work orders, and the resulting report. To understand the reasons for multiple investigations of the San Francisco Bay Area’s power failure, we interviewed commission staff and also staff of PG & E, CAISO, and the board. C A L I F O R N I A S T A T E A U D I T O R 5 To assess whether the contract amount and consultant’s costs were appropriate, we inquired about the commission’s methodology for determining the amounts of the contract and work orders. We asked to review invoices paid for this contract, but there were none. Therefore, we asked the commission to obtain an invoice from the consultant for the expenditures of the investigation so that we could assess the reasonableness of those expenditures; however, the commission did not receive the invoice during our audit. To evaluate whether the commission effectively monitored the contract and ensured the quality of the product, we compared the monitoring efforts of the commission’s contract managers to the responsibilities for a contract manager listed in the State Contracting Manual. We also reviewed the contract’s compliance with the contracting requirements in the Public Contract Code and the State Contracting Manual. Further, we assessed the consultant’s investigative report of the power outage to determine whether the report’s analysis supported certain conclusions. However, we did not review the technical accuracy of the report. We also interviewed the commission’s project manager and other appropriate parties to determine whether the commission could use the report, as written, for its intended purpose. THE COMMISSION POORLY MONITORED ITS CONTRACT AND CANNOT SUBSTANTIATE THE COST OF THE INVESTIGATION INTO THE BAY AREA’S MASSIVE POWER FAILURE Although it had a reasonable basis for contracting with a consultant that could evaluate the December 1998 power outage in the San Francisco Bay Area, the commission provided inadequate oversight of the contract. In fact, the commission failed to review the qualifications of subcontractors that performed the investigation, ensure the quality of the resulting investigative report, or substantiate the cost of the investigation for which the commission contracted. As a result, the commission cannot demonstrate that it has protected the public’s money. Because it did not have employees available with the expertise required to perform an investigation into an electrical system failure, the commission believed that it needed to contract with 6 C A L I F O R N I A S T A T E A U D I T O R an expert consultant. The commission also believed that determining the cause of the power outage constituted an emergency because another power failure could imperil life and property, affect public safety, and interfere with the delivery of essential public services. For these reasons, the commission exempted the contract from normal advertising and bidding requirements by using an emergency contracting provision. Instead, the commission performed what it called a “market survey” to select a consultant. This procedure entailed the commission’s contacting four different consulting firms; reviewing their previous work experience; comparing their fees; and noting their strengths, weaknesses, and potential conflicts of interest. The commission then selected the consultant it believed held the expertise necessary for conducting the investigation and awarded this firm a $400,000 contract. The commission’s contract required the consultant to determine the cause of the outage and make recommendations for preventing a recurrence. Further, the contract directed the consultant to draw expert conclusions and then to prepare a report format suitable for advancing it in litigation. The Commission Cannot Demonstrate That It Reviewed Its Experts’ Qualifications Although it appears that the lead investigator of the consultant chosen by the commission had adequate investigative expertise, the commission cannot demonstrate that it made certain that the entire investigative team was qualified to analyze and conclude on the causes of PG & E’s electrical system failure. Moreover, the commission could not provide evidence that it had given the consultant written approval to use any subcontractors, as the contract requires. According to the commission, the consultant did not have electrical engineering expertise; therefore, the consultant used the services of 13 subcontractors to perform analyses for the investigation. However, the commission was unable to provide documentation that it had reviewed 12 of the 13 subcontractors’ previous work experience to determine whether the subcontractors were qualified to be on the investigative team. Additionally, when asked for the subcontractors’ resumes, the commission provided only 7 of 13, having first to obtain them from the consultant. The fact that the commission did not have copies of the resumes further suggests that the commission did not review the subcontractors’ qualifications. C A L I F O R N I A S T A T E A U D I T O R 7 The commission stated that it relied upon the opinion of its consultant’s lead investigator to determine which subcontractors were necessary to complete the investigation. Additionally, the commission stated that it gave the consultant’s lead investigator verbal approval to include the subcontractors on the investigative team even though the contract requires this approval in writing. The subcontractors may have had the necessary expertise; however, the commission could not demonstrate that experts conducted the investigation because it did not document that it reviewed the qualifications of the subcontractors who analyzed the electrical system’s failure. The Consultant’s Report Offered Inadequate Support for Some Conclusions Not only did it fail to verify the qualifications of its investi- gative team, but the commission also did not ensure that the consultant prepared a report that is suitable for litigation, as required by the contract. Specifically, the consultant’s report does not contain sufficient detail or analysis to support some conclusions reached. Consequently, the commission is incurring additional, unnecessary costs to obtain the required support. This support is important for the commission so that it may refute any challenges. In addition, according to PG & E, this documentation is important so that it can prepare a response to the report’s conclusions. Despite the commission’s and PG & E’s need for detailed analyses, the consultant’s report does not include the under- lying bases for all of its conclusions. For example, the consultant’s report states that PG & E has an “error prone work culture that tends to bypass procedures and work practice requirements.” The report lists four examples of inconsistencies in procedural use and adherence occurring shortly before the December 8, 1998, power failure. However, based on these examples, we had difficulty arriving at the same conclusion that the report presents. Specifically, it appears that the consultant based the conclusion that PG & E’s entire work force is error prone on a few examples related to one instance—the Bay Area power outage. To further our understanding, we asked the commission to provide us with more detailed supporting analysis that convinced it that the conclusion was appropriate. Only after receiving additional details and analyses not included in the report were we able to understand more clearly how the consultant reached its conclusion. 8 C A L I F O R N I A S T A T E A U D I T O R PG & E has criticized the validity of the commission’s report conclusions and has not prepared a response because it believes that the report’s analyses do not support some conclusions. Because the consultant’s report does not contain sufficient detail to substantiate all the conclusions it presents, the commission prepared a second work order asking the consultant to provide support, and the commission agreed to pay up to $6,255 for the consultant’s additional work. Further, an administrative law judge ordered the commission to hold a “workshop” so that the consultant could explain its conclusions, for which the consultant is likely to bill. The commission held a workshop on May 6, 1999, for all interested parties to discuss in more detail the methodology and support for conclusions reached. Currently, the commission is waiting for further instructions from the administrative law judge. According to the commission, the report does not contain details to support the conclusions reached because the commission directed the consultant not to include them. However, this direction seems inappropriate given that the contract requires the report to be suitable for litigation purposes. As the need for subsequent evidence gathering and the workshop demonstrates, the report’s details are important. Had the commission not given this direction to the consultant but instead made certain that the report met the contract’s specifications, the commission might have avoided additional costs. The Peer Reviewer of the Consultant’s Report May Not Have Been Independent The commission stated that it thought the report was adequate and an independent peer reviewer had ensured its quality. However, we determined that the consultant, not the commission, subcontracted with the reviewer. The consultant also named the reviewer as a general investigative team member, a situation that has called the reviewer’s independence into question. Moreover, according to the reviewer, he reported to and was paid by the consultant, not the commission. The reviewer claims that he was independent; however, because the consultant subcontracted directly with him, their relationship gave the appearance that the reviewer was not independent. C A L I F O R N I A S T A T E A U D I T O R 9 The Commission Has Not Verified That the Consultant’s Costs Were Reasonable In addition to its inadequate monitoring of the consultant and the resulting report on the causes of the Bay Area power failure, the commission did not ensure that the amount of the consultant’s contract was appropriate. The commission also did not make certain that actual expenditures incurred by the consultant were within budgeted amounts. The Commission Cannot Substantiate the Contract Amount Although the commission awarded the consultant a contract, not to exceed $400,000, to perform an investigation into PG & E’s system failure, the commission cannot demonstrate that this amount is reasonable. Specifically, the commission stated that it did not use a formula or specific methodology to derive the contract amount. Thus, we are unable to determine whether the $400,000 was appropriate. Nevertheless, we can conclude that the commission’s approach for establishing the contract amount is unreasonable. The commission should have at least estimated the number of hours the consultant would need to perform each part of the investigation and then multiplied that number by the consultant’s hourly rate. With such a calculation, the commission could have accounted for the costs of the contract and also determined whether the consultant was working within budgeted amounts. Additionally, general contracting provisions in the Public Contract Code require the commission to obtain from the consultant a detailed analysis of the costs to perform a specific job. This type of analysis would have given the commission a sound basis for establishing the contract amount for the investigation of the power failure. The commission’s failure to obtain this analysis further prevented the commission from substantiating the contract amount and holding the consultant accountable for the costs of work performed. Not only did the commission fail to justify the contract amount, but it also did not substantiate the amounts approved for the consultant’s work orders. For instance, the first work order under the contract includes all the work required to complete the contract. However, the total authorized for the work order is $175,000, or $225,000 less than the total contract amount of $400,000. The commission could not explain the disparity between the contract total and this work order amount; rather, 10 C A L I F O R N I A S T A T E A U D I T O R the commission stated that it did not base the amounts for the contract or work order on the number of hours the commission estimated the consultant would need because the commission did not know the amount of work involved. We question the commission’s decision to enter the contract for the investigation of the massive power outage without even estimating the number of hours a consultant would need to complete the work. The Absence of Invoices Has Prevented the Commission From Monitoring Its Consultant’s Costs The commission issued the consultant’s report on March 31, 1999; however, as of May 10, 1999, the consultant had not invoiced the commission for any services rendered. Thus, we were unable to validate the total cost of the investigation into the power outage, and the commission has been unable to monitor the costs incurred by the consultant to determine their appropriateness. The contract for the investigation requires the consultant to invoice the commission each month. Because the consultant began investigating on February 1, 1999, the commission should have received two invoices—one each for February and March 1999. Nonetheless, the commission has not yet received any invoices and has stated that it did not compel the consultant to submit bills during these months because of the urgent need to complete the report. The commission also stated that on several occasions, it has asked the consultant to provide an invoice. While we understand that the consultant needed to complete its investigation rapidly, the commission knew that the investigation needed to be completed within a short period of time when it wrote and approved the contract. Thus, the commission should have held the consultant to contractual requirements. Because it did not do so, the commission cannot determine whether the consultant’s expenses during the investigation were appropriate. In fact, the commission stated that according to the consultant, the costs were $297,000 for work performed under the first work order, or $122,000 more than the amount approved for the consultant to complete the work order of $175,000. We asked the commission if it was aware that the consultant’s costs were over budget or if it was concerned by the amount. The commission stated that it was aware of the costs throughout C A L I F O R N I A S T A T E A U D I T O R 11 the investigation because the consultant communicated its costs verbally. Further, the commission stated that it is not concerned that the consultant was over budget because it is confident that the consultant spent the money on appropriate tasks. Although the commission is not concerned by the consultant’s expenditures, we are unsure about the appropriateness of the costs incurred during the investigation because there were no invoices documenting the consultant’s work. RECOMMENDATIONS To ensure that they address jurisdictional issues, the commission and CAISO should continue their discussions regarding oversight as it relates to the recent PG & E power failure in the San Francisco Bay Area and to any future outages of this type. To ensure that it has received a work product that conformed with contract requirements from the consultant that investigated the outage, the commission should conduct another review of the investigative report before paying the consultant for services rendered. This review would also allow the commission to determine whether the report complies with the contract’s specifications. In addition, before paying the consultant’s invoices, the commission should audit all charges to determine their appropriateness and compliance with contract provisions. We will look into the commission’s contract management issues in our follow-on audit and assess whether the problems with this contract are unique or whether they reflect systemic weaknesses throughout the commission. Our next report will offer more wide-ranging recommendations, as appropriate. 12 C A L I F O R N I A S T A T E A U D I T O R We conducted this review under the authority vested in the California State Auditor by Section 8543 et seq. of the California Government Code and according to generally accepted governmental auditing standards. We limited our review to those areas specified in the audit scope section of this report. Respectfully submitted, KURT R. SJOBERG State Auditor Date: May 14, 1999 Staff: Mary Noble, Deputy State Auditor Tammy Lozano, CPA Bryan Beyer Jennifer Harris Dianna Scott C A L I F O R N I A S T A T E A U D I T O R 13 This page left blank intentionally. 14 C A L I F O R N I A S T A T E A U D I T O R Agencies response provided as text only: May 12, 1999 Mr. Kurt R. Sjoberg State Auditor 555 Capitol Mall, Suite 300 Sacramento, Ca. 95814 Attn: Mary P. Noble, Deputy State Auditor Dear Mr. Sjoberg: Thank you for the opportunity to respond to the draft report, “Public Utilities Commission: Did not effectively Manage its Contract for Investigating San Francisco’s 1998 Power Failure ’’ dated May 11, 1999. Although the limited comment time does not allow for a formal Commission response, attached we offer comments on several of the issues raised in the draft. I do not dispute the criticisms contained in the draft, and recognize the need for this agency to make improvements in this area. I concur with your recommendations and we have already begun to implement them. On April 22, 1999, we initiated discussions with the California Independent System Operator (CAISO), in conjunction with the electricity Oversight Board, to establish formal investigation protocols that clearly delineate the jurisdictional responsibilities between the commission and the CAISO on future outages. We hope to have these in place by June. With regard to your second recommendation on the outage contract, there will be a full review of the investigator’s report and an audit of the invoices prior to any payments being made. Finally, I will ask the Department of General Services to present a training course in the duties and responsibilities of a project manager under state contracting guidelines, to be given in the very near future. Very truly yours, (Signed by:) Wesley M. Franklin Executive Director Attachment Commission Response to California State Auditor Management of Investigative Contract into PG&E Power Outage Approval of Subcontractors: The report correctly concludes that written approval was not obtained for all sub-contractors employed in the investigation. As substantial issues were identified or evolved during the course of the investigation Dr. Chiu the lead investigator would raise with project management staff the need for additional subject matter expertise and propose the use of specific individuals. Their qualifications were discussed verbally and many were brought in for a face- to-face review discussion with Commission staff. Dr. Chiu was then given the verbal approval to proceed. Support for Conclusions: The original staff report was criticized for not containing all of the documents and analyses to support some of PII’s conclusions. Several reasons exist for the lack of detailed documentation. A significant amount of data was received one week before the report’s due date, and while analyzed, it would have been difficult and time consuming to properly reference the documents within the time allotted. It would also have detracted from the teams primary role of writing the report. Secondly, this was a rule making, not an adjudicatory proceeding, and the emphasis was on recommendations for improvements, not past mistakes. We did not want to belabor the negative, but hopefully, emphasize positive steps that could be taken to prevent future outages. The documentation was here, just not referenced in the report. Lastly, virtually all of the documents used in the study were PG&E source documents, provided by the company. They were also consulted on a regular basis by PII as old issues were settled or new ones arose. Again, rulemakings are not usually confrontational in nature, and we consistently affirmed that at the conclusion of all of the individual reports, we would sit down with all the affected parties, expert-to expert, to resolve conflicting recommendations. Independence of Reviewer: While the report questions the independence of the reviewer, it appears that the criticism is really directed at the Commission staff for not structuring a stand-alone contract for the reviewers services; a criticism which we accept. Although not required we felt the PII investigative report, and the PUC staffs review of it, would benefit from an independent review by a qualified electric industry management expert. Mr. McCarthy, a former Senior Vice President of Southern California Edison, known not only for his expertise but also his objectivity and critical approach to problem solving was selected and subsequently performed that critical independent role, he reported to Mr. Schulte directly on substantive issues on a fairly regular basis. participated in meetings with the CAISO and Energy Oversight Board, and independently reviewed the progress of the investigation and the final report. Mr. McCarthy provided critical feedback to me as well as to Dr. Chiu. Mr. McCarthys input to us resulted in definitive changes to the draft report, both additions and deletions. Mr. McCarthy did not report to Dr. Chiu on investigative issues, only on the number of hours he expended. Lack of detailed Cost Estimate: The report correctly identifies the problem, and accurately describes the uncertainty in the process. The original contract amount was derived after a review of the proposals submitted during our market survey, although no one could accurately define all of the issues, and therefore the costs to address them, since PG&E did not issue its report until January 25th well after the commencement of the contract. It was only after we discovered the limitations of their investigation that the scope of our needs became clearer. In addition, several major issues arose during the course of our study that dictated further investigation. As described, the scope and projected costs were discussed verbally as issues arose.