Recommendations

May 2017 K–12 High Speed Network Improved Budgeting, Greater Transparency, and Increased Oversight Are Needed to Ensure That the Network Is Providing Reliable Services at the Lowest Cost to the State Report 2016-129 COMMITMENT INTEGRITY LEADERSHIP CALIFORNIA STATE AUDITOR 621 Capitol Mall, Suite 1200 | Sacramento | CA | 95814 916.445.0255 | TTY 916.445.0033 For complaints of state employee misconduct, contact us through the Whistleblower Hotline: 1.800.952.5665 Don’t want to miss any of our reports? Subscribe to our email list at auditor.ca.gov For questions regarding the contents of this report, please contact Margarita Fernández, Chief of Public Affairs, at 916.445.0255 This report is also available online at www.auditor.ca.gov | Alternate format reports available upon request | Permission is granted to reproduce reports Elaine M. Howle State Auditor Doug Cordiner Chief Deputy May 25, 2017 2016-129 The Governor of California President pro Tempore of the Senate Speaker of the Assembly State Capitol Sacramento, California 95814 Dear Governor and Legislative Leaders: As requested by the Joint Legislative Audit Committee, the California State Auditor presents this audit report concerning the K–12 High Speed Network (K12HSN) program’s operations and funding. This report concludes that by the end of fiscal year 2014–15 the Imperial County Office of Education (ICOE) had accumulated operating reserves for the K12HSN program that totaled nearly $15 million. These reserves appear to be a product of inadequate budgeting by ICOE and a lack of oversight at the state level. At the end of fiscal year 2015–16, ICOE’s operating reserve for the program declined to approximately $5.7 million primarily as a result of it not receiving state funding in that fiscal year. ICOE has taken some steps to improve its budgeting practices for K12HSN; however, concerns remain about its accuracy and transparency. In addition, ICOE needs to improve its planning processes in order to manage network development at the lowest possible cost to the State. Specifically, ICOE has increased or is planning to increase capacity for some portions of the network at significant cost without justifying the need for those increases based on the actual usage of the network. ICOE currently lacks a detailed methodology for determining the appropriate timing and magnitude of capacity increases, and as a result has defaulted to large capacity increases that involve substantial upfront and recurring costs. The recommendations we make to ICOE in these areas are intended to improve the quality of its network development process and to provide state decision makers with better information with which to determine appropriate funding levels for the program. ICOE can also do more to measure and report on the K12HSN program’s effectiveness. State law sets forth specific responsibilities and goals for ICOE in administering the K12HSN program, but ICOE has not reported on some of these areas, such as the network’s reliability and cost-effectiveness, and the California Department of Education (Education) has not required ICOE to do so. Without this information, the State cannot fully evaluate the benefits of the network’s current structure and administration. We therefore recommend that Education, which the law assigns responsibility for measuring K12HSN’s success, direct ICOE to report annually on specific performance measures we identified. Respectfully submitted, ELAINE M. HOWLE, CPA State Auditor 621 Capitol Mall, Suite 1200 Sacramento, CA 95814 916.445.0255 916.327.0019 fax www.auditor.ca.gov iv Report 2016-129 | CALIFORNIA STATE AUDITOR May 2017 Selected Abbreviations Used in This Report BIIG Broadband Infrastructure Improvement Grants CAASPP California Assessment of Student Performance and Progress CalREN California Research and Education Network CENIC Corporation for Education Network Initiatives in California E-Rate Discounted telecommunications and Internet services for schools and libraries EDC 11800 Education Code section 11800 Education California Department of Education ICOE Imperial County Office of Education K12HSN K–12 High Speed Network USAC Universal Service Administrative Company CALIFORNIA STATE AUDITOR | Report 2016-129 v May 2017 Contents Summary 1 Introduction 3 Excessive K12HSN Operating Reserves Have Recently Been Reduced, but Questions Remain About What a Prudent Reserve Should Be 9 ICOE Needs to Improve Its Planning Processes in Order to Manage Network Development at the Lowest Possible Cost to the State 19 Measurement of the Program’s Effectiveness Has Omitted Key Information, and Oversight Has Been Inconsistent 31 Other Areas We Reviewed 37 Scope and Methodology 41 Responses to the Audit Imperial County Office of Education 47 California State Auditor’s Comments on the Response From the Imperial County Office of Education 53 California Department of Education 55 vi Report 2016-129 | CALIFORNIA STATE AUDITOR May 2017 Blank page inserted for reproduction purposes only. CALIFORNIA STATE AUDITOR | Report 2016-129 1 May 2017 SUMMARY The K–12 High Speed Network (K12HSN) is a state‑funded program that was established to enrich pupil educational experiences and improve academic performance by providing high‑speed, high‑capacity Internet connectivity to California’s public school system. Since 2004 the Imperial County Office of Education (ICOE) has had primary responsibility for the design and management of the program. For this audit, we reviewed the ICOE’s processes for managing the network as well as the role of state entities in overseeing the program. This report draws the following conclusions: Excessive K12HSN operating reserves have recently been reduced, but questions remain about what a prudent reserve should be. Page 9 By the end of fiscal year 2014–15, ICOE had accumulated operating reserves for the K12HSN program that totaled nearly $15 million. According to our review, these reserves appear to be a product of inadequate budgeting by ICOE and a lack of oversight at the state level. ICOE has taken some steps to improve its budgeting process, but concerns remain about its accuracy and transparency. ICOE has also spent most of its operating reserve. As a result, to avoid deficits, projected fiscal year 2017–18 expenditures for the K12HSN program will require a higher level of state funding than ICOE has historically received. ICOE needs to improve its planning processes in order to manage network development at the lowest possible cost to the State. Page 19 ICOE lacks a detailed methodology for determining when and by how much it should increase network bandwidth (capacity). Currently, ICOE is pursuing expensive capacity increases to the network’s circuits—the individual connections between network sites or those sites and the rest of the network—even though less expensive options have been available. Our review of those circuits’ usage levels and ICOE’s process for determining necessary levels for circuit capacity increases found that ICOE cannot justify the costs associated with some of these increases. Measurement of the program’s effectiveness has omitted key information, and oversight has been inconsistent. Page 31 State law sets forth specific responsibilities and goals for ICOE in administering the K12HSN program and assigns responsibility for measuring the program’s success to the California Department of Education (Education). However, ICOE has not reported on some key measures associated with the network’s performance, such as its reliability, and Education has not required ICOE to do so. As a result, some aspects of the network’s performance and cost‑effectiveness remain unclear. 2 Report 2016-129 | CALIFORNIA STATE AUDITOR May 2017 In addition, we reviewed ICOE’s contract with the Corporation for Education Network Initiatives in California (CENIC), individual agreements between CENIC and Internet service providers on behalf of K12HSN, and staffing and compensation for the K12HSN program. In some of these areas, we found that ICOE could improve its processes, and we have made recommendations discussed in the Other Areas We Reviewed section of this report. Summary of Recommendations Legislature To help ensure continuous network operations while preserving state resources, the Legislature should appropriate to the K12HSN program an amount that does not exceed $10.4 million for fiscal year 2017–18. If the Legislature wishes to appropriate a lower amount for the program, it should direct ICOE to modify one or more of the planned network upgrades we highlight in this report, either by delaying the upgrade to a subsequent fiscal year or by pursuing a less expensive option. ICOE To better inform decision makers at the state level about the amount of funding necessary to operate and maintain the network, ICOE should formally amend its annual budget documents by November 2017 to specify multiple potential levels of network expenditures for the coming year, and it should detail the specific network upgrades and project costs included in each scenario. To better guarantee that network upgrades are necessary and are achieved at the lowest possible cost to the State, ICOE should develop a formal methodology for reviewing circuit capacity needs. Education To increase transparency in the K12HSN program and help ensure that the State has sufficient information to measure the program’s effectiveness, Education should direct ICOE to report annually on specific performance measures. These performance measures should include cost, network bandwidth, and the frequency and duration of network outages and interruptions. Agency Comments ICOE and Education agreed with our recommendations and have identified actions they are currently taking or plan to take to implement them. CALIFORNIA STATE AUDITOR | Report 2016-129 3 May 2017 INTRODUCTION Background Until 2000 California’s kindergarten through 12th‑grade (K–12) schools, school districts, and county offices of education were individually responsible for obtaining access to the Internet and for connecting with other educational entities and resources as needed. In 2000 the program that would later be known as the California K–12 High Speed Network (K12HSN) was established to connect the State’s public school system to the high‑speed network created for use by California’s universities and community colleges. Between fiscal years 2000–01 and 2003–04, the University of California (UC) received more than $93 million in state appropriations to expand the universities’ network infrastructure to K–12 schools and county offices of education for K12HSN. In July 2004, the Legislature shifted funding from UC to the California Department of Education (Education) and established the K12HSN program to enrich pupil educational experiences and improve academic performance by providing high‑speed, high‑bandwidth Internet connectivity to the public school system. It also required that a lead county office of education be selected to administer the K12HSN program. Subsequently, in September 2004, Education selected the Imperial County Office of Education (ICOE) via a grant application process to administer the program on the State’s behalf. In 2006 the Legislature amended the Education Code to add section 11800 (EDC 11800), which statutorily established the K12HSN program and its administration by a local educational agency. ICOE is currently continuing in its role as the program’s administrator. EDC 11800 assigns the state superintendent of public instruction (superintendent), who oversees Education, the responsibility for measuring the success of K12HSN and ensuring that its benefits are maximized to the extent possible. Finally, EDC 11800 assigns the superintendent the responsibility of establishing a K12HSN advisory board composed of 12 members—a majority of whom are county and school district superintendents—to meet quarterly and to provide policy direction and guidance. When ICOE assumed administration of K12HSN from UC, it continued UC’s practice of contracting with the Corporation for Education Network Initiatives in California (CENIC), a nonprofit organization created to operate and maintain the California Research and Education Network (CalREN). According to CENIC, CalREN is configured with a backbone—a high‑capacity network designed to meet the unique requirements of more than 4 Report 2016-129 | CALIFORNIA STATE AUDITOR May 2017 20 million users, including the State’s K–12 system, public libraries, California Community Colleges, the California State University system, the UC system, and certain private universities in the State. The network ICOE oversees consists of 86 network sites, most of which are located at county offices of education, as shown in Figure 1. Schools’ and districts’ network traffic flows through these sites via individual connections, known as circuits, to the backbone. School districts connect to the network via one or more of the network sites, and districts contract with local Internet service providers (service providers) for those connections. According to statistics from K12HSN, its 86 network sites currently provide connectivity for 83 percent of California public schools, 85 percent of school districts, and all county offices of education, serving nearly 4.8 million students overall. In addition to ICOE’s primary function of overseeing the physical network of the sites and their connections to the backbone, recent budget acts have made ICOE responsible for distributing grant funding to schools with insufficient connectivity for computer‑based testing of students. According to Education’s fact sheet on the California Assessment of Student Performance and Progress (CAASPP) system of computer‑based assessments, in January 2014, CAASPP replaced the former paper‑based assessments. To address the needs of schools and school districts that do not have sufficient network capacity to conduct computer‑based online testing, the Legislature appropriated to Education approximately $26.7 million for fiscal year 2014–15 and an additional $50 million for fiscal year 2015–16 to help schools and school districts obtain the network capacity needed to administer computer‑based assessments. These funds are known as Broadband Infrastructure Improvement Grants (BIIG), and state law directs ICOE to distribute the funds in consultation with Education and the State Board of Education based on an assessment of need. The law also requires the sites that receive the BIIG funds to pay the ongoing costs associated with the improved Internet infrastructure. As a result, ICOE’s involvement with those schools’ and districts’ Internet connectivity is temporary and exists outside of its core function of operating K12HSN. Federal and State Subsidies for K12HSN Activities The Universal Service Administrative Company (USAC)— a not‑for‑profit organization designated by the Federal Communications Commission—administers a federal subsidy program that provides discounts for telecommunications and Internet services used by schools and libraries. This program has become known as the education rate or E‑Rate program (E‑Rate). E‑Rate is supported by the Universal Service Fund, which in turn CALIFORNIA STATE AUDITOR | Report 2016-129 5 May 2017 Figure 1 Structure of the K12HSN Program’s Network ! ! ! ! LEGEND " Hub Sites—connect network sites to the backbone ! ! ! Backbone—operated by CENIC and leased to K12HSN ! ! Network Sites—usually county offices of education ! K12HSN—responsible for all costs associated with overseeing ! connections to the backbone for all 86 network sites " ! School Districts—responsible for their own Internet connections ! ! to K12HSN and associated costs ! ! ! !! ! ! !!!! ! ! !! !! ! ! ! ! !"! ! !!! ! ! ! " ! !! !! !! !" ! !! ! ! "! !!" ! ! ! ! !! ! ! !" ! " !! !! !" !!" ! ! ! ! ! ! "! ! ! !" ! ! " !" "" ! " " ! Sources: California State Auditor’s analysis of CENIC’s contract with ICOE, Education’s grant award letters, and K12HSN network maps. is funded by required contributions from telecommunications carriers, including wireline, wireless, and Voice Over Internet Protocol providers. E‑Rate subsidizes 20 percent to 90 percent of the cost of Internet access and telecommunications services for educational providers, based on the percentage of students eligible for free or reduced‑price lunch under the National School Lunch 6 Report 2016-129 | CALIFORNIA STATE AUDITOR May 2017 Program and whether the school is rural or urban. Through an external consultant, CENIC applies for E‑Rate funding for K12HSN for Internet access costs and upgrades. As shown in Figure 2, there are two methods through which K12HSN may ultimately collect the subsidy. Under one method, the service provider reduces the amount charged to CENIC by the value of the anticipated subsidy and then collects the subsidy from USAC. Under the other method, which is more commonly used, the service provider charges CENIC, which in turn charges K12HSN for the entire cost of the service, and then the provider reimburses K12HSN through CENIC when it receives the subsidy. Because of the extensive duration of the E‑Rate process, many of the subsidies pertaining to a given fiscal year are not credited to K12HSN until the following year. In those cases, K12HSN accrues the amount of the anticipated subsidies and in the meantime uses other available funds to cover its costs. Another Internet subsidy known as the California Teleconnect Fund (Teleconnect Fund) is administered by the California Public Utilities Commission and credits K12HSN for half of the Internet access costs not covered by E‑Rate. This subsidy also applies to the fixed fee that K12HSN pays annually to CENIC in exchange for transmitting network traffic through the backbone to other parts of the network. In most cases, K12HSN Internet access costs eligible for the Teleconnect Fund subsidy are discounted up front rather than through a subsequent reimbursement process. Major Program Revenue and Expenditures Apart from the state and federal subsidies just described, all other major program revenue that ICOE receives is state funding. This funding amount fluctuated early in ICOE’s tenure as the program administrator, but it held steady at about $8.3 million annually for several years until recently, when the State raised questions about the appropriate funding amount for the program. We discuss these issues in the first section of this report. K12HSN expenditures consist of ICOE’s internal costs for overseeing the program, such as personnel, and its contracted agreements with external parties. ICOE’s capital equipment expenditures for K12HSN include both equipment purchased for ICOE’s use in administering the program as well as equipment purchased and placed into service at the network sites. One of ICOE’s most significant agreements with an outside entity is a formal memorandum of understanding with the Butte County Office of Education to pay for a specialist in obtaining E‑Rate and Teleconnect Fund subsidies to assist California schools and school CALIFORNIA STATE AUDITOR | Report 2016-129 7 May 2017 Figure 2 Process Used by CENIC to Obtain Federal Broadband Subsidies for the K12HSN Program APPLYING FOR DISCOUNTS SERVICES START % APPROVED 28 DAYS CENIC requests services CENIC and ICOE CENIC seeks CENIC receives CENIC notifies on behalf of K12HSN by evaluate proposals discounts for a Funding USAC that services The competitive submitting a description of received and select services by Commitment have started. bidding process services requested to USAC, service providers submitting an Decision Letter must be open for a not-for-profit corporation with whom CENIC application for from USAC. at least 28 days. designated by the Federal signs contracts funding to USAC Communications Commission for services. within the to administer the Universal filing window. Service Fund, which funds the E-Rate program. INVOICING AND PAYMENT* Invoicing Method #1 OR Invoicing Method #2 • The service provider reduces the • The service provider invoices amount charged by the value of CENIC $ and receives payment from CENIC $ the anticipated E-Rate subsidy Payable to: Provider CENIC for the full amount of Payable to: Provider and only bills CENIC for the service costs. nondiscounted share of costs. Discounted Invoice Signature Full Invoice Signature • The service provider invoices • CENIC then invoices and and receives reimbursement USAC receives payment from K12HSN, K12HSN $ $ from USAC for the E-Rate Payable to: Provider which accrues the anticipated Payable to: CENIC subsidy amount not charged E-Rate subsidy amount in its to CENIC. K12HSN Subsidy Signature financial records . Full Invoice Signature • The service provider invoices U di n sc d o e u r n th te is d m p e o t r h ti o o d n , o th f e se K r 1 v 2 ic H e S s N co p v r e o r g e r d a m by n t e h v e e E r - p R a a y te s f s o u r b t s h id e y . and receives payment from USAC $ USAC for the E-Rate subsidy Payable to: Provider amount previously paid by CENIC. K12HSN Subsidy Signature • The service provider reimburses ISP CENIC for the E-Rate subsidy $ amount paid by USAC. Payable to: CENIC K12HSN Subsidy Signature • CENIC deposits the subsidy CENIC for amount to the K12HSN K12HSN bank account. Under this method, the K12HSN program pays for services at the time of billing and must wait until the invoicing process is complete before being reimbursed for the E-Rate subsidy. B DI S COMPETITIVE BIDDING VENDOR Sources: USAC E-Rate program guidelines and K12HSN’s contracts with CENIC. * Before 2007 CENIC could request and receive reimbursement directly from USAC for services paid in full. 8 Report 2016-129 | CALIFORNIA STATE AUDITOR May 2017 districts with applying for those funds, which schools and districts can use to offset the costs of their own Internet connections. ICOE also has an agreement with a private consulting firm for educational agencies to provide consulting services related to legislative activities at the state level. However, the majority of K12HSN’s expenditures pertain to its contract with CENIC for the operation and administration of the physical high‑speed network. As the administrator of the network, CENIC is responsible for entering into and servicing contracts with service providers throughout the State and for purchasing network equipment on behalf of K12HSN. Table 1 provides more detail about the program’s revenue and expenditures from fiscal years 2013–14 through 2015–16. Table 1 K12HSN Program Revenue and Expenditures Fiscal Years 2013–14 Through 2015–16 FISCAL YEAR 2013–14 FISCAL YEAR 2014–15 FISCAL YEAR 2015–16 AMOUNT PERCENTAGE AMOUNT PERCENTAGE AMOUNT PERCENTAGE EXPENDITURES Operations* $547,477 5% $743,441 6% $855,149 6% Personnel 1,297,777 12 1,425,313 12 1,413,641 11 Equipment† 92,620 1 499,496 4 739,314 6 CENIC 8,846,453 80 8,871,463 74 9,463,998 71 Other Agreements 296,716 2 423,936 4 759,648 6 Total Expenditures $11,081,043 $11,963,649 $13,231,750 REVENUE State Appropriation $8,340,000 67% $8,340,000 69% $0‡ 0% Other Revenue§ 128,359 1 93,441 1 33,159 1 E-Rate and Teleconnect Fund 4,006,237 32 3,663,457 30 3,585,764 99 Subsidies (Reimbursements) Total Revenues $12,474,596 $12,096,898 $3,618,923 Year‑End Surplus (Deficit) $1,393,553 $133,249 ($9,612,827) Source: California State Auditor’s analysis of ICOE accounting data. Note: The revenue and expenditures in this table represent the money ICOE collected and spent during each fiscal year. * Costs identified as Operations include communications and supplies, debt service, and some external consulting that is not included in the Other Agreements category. † According to the K12HSN program’s chief executive officer, in fiscal year 2014–15 ICOE began classifying the equipment installed at network sites as capital expenditures. As a result, from fiscal year 2014–15 on, its accounting data identify those costs as equipment expenditures, whereas previously it had included those costs in the CENIC category. ‡ Because K12HSN did not receive any funding from state appropriations for fiscal year 2015–16, ICOE used K12HSN’s operating reserve to pay for the portion of program expenditures not covered by Internet subsidies and other funding sources. § Other Revenue includes funds ICOE collected in exchange for the delivery of videoconferencing services to higher education entities. CALIFORNIA STATE AUDITOR | Report 2016-129 9 May 2017 Excessive K12HSN Operating Reserves Have Recently Been Reduced, but Questions Remain About What a Prudent Reserve Should Be Key Points • Historically, K12HSN’s annual expenditures have been considerably less than the state allocation that ICOE requested and received, which led to the program’s accumulating excessive operating reserves. • Although ICOE has taken some steps to improve the K12HSN budgeting process, ICOE must increase the accuracy and detail of the information it provides to state oversight agencies to help the State make well‑informed and appropriate funding decisions. • K12HSN faces cash‑flow challenges, and this situation emphasizes the need for some level of operating reserve. However, ICOE must take further steps to demonstrate that the reserves it claims the program needs are warranted. Inadequate Budgeting and a Lack of Oversight at the State Level Contributed to ICOE’s Excessive Operating Reserves The annual expenditures ICOE incurred for the K12HSN program have historically been considerably lower than the amount of the state appropriation for the program, which led to the program’s accumulating large operating reserves. According to ICOE’s fiscal year 2015–16 audited financial statements, the amount of its state appropriation that ICOE spent annually averaged about $7 million from fiscal years 2010–11 through 2014–15, while the state appropriation remained steady at about $8.3 million per fiscal year—the amount that ICOE continued to request despite accumulating a substantial reserve. As Figure 3 on the following page illustrates, the program’s operating reserve grew from $8.7 million at the end of fiscal year 2010–11 to $14.7 million by the end of fiscal year 2014–15. For fiscal year 2015–16, the State withheld the appropriation to the program because of this large reserve. As a result, ICOE used most of its reserve to operate the program. At the end of fiscal year 2015–16, ICOE’s operating reserve for the program totaled approximately $5.7 million. Inadequate budgeting and a lack of oversight at the state level contributed to ICOE’s excessive operating reserve balance. ICOE’s approach in submitting its budget during this time was to present information to the State at a summary level without any detail to support its projected costs. Specifically, the budget documents ICOE provided to the State included only a single line item combining all projected expenditures pertaining to CENIC, such as equipment, maintenance, network services, and management fees. In contrast, for its internal expenditures, ICOE identified amounts individually by type, even though the total amount of those expenditures was less than a quarter of the CENIC expenditure line item. In addition, ICOE continued to request the same appropriation amount of $8.3 million from the State each year over this period, even though K12HSN’s operating reserves were more than $8 million and continuing to increase. 10 Report 2016-129 | CALIFORNIA STATE AUDITOR May 2017 Figure 3 K12HSN Year‑End Operating Reserves Fiscal Years 2010–11 Through 2015–16 $15 $14.7 $14.3 $13.2 12 $10.9 9 $8.7 6 $5.7* 3 0 2010–11 2011–12 2012–13 2013–14 2014–15 2015–16 Fiscal Year )snoilliM ni( tnuomA evreseR Source: ICOE audited financial statements. * Because K12HSN received no state funding in fiscal year 2015–16, ICOE used the K12HSN operating reserve to pay for the portion of program expenditures not covered by federal Internet subsidies or other funding sources, thereby reducing the reserve amount by approximately $9 million. The K12HSN program’s chief executive officer explained that it has been difficult historically for ICOE to anticipate network service costs and that during this period ICOE took a conservative approach by projecting higher cost estimates. The program’s chief operating officer indicated that ICOE did not intend to generate the large reserve. Nonetheless, we expected that ICOE would reduce its annual budget requests in response to its consistently spending less than the State provided. According to the chief operating officer, the uncertainty about the timing of E‑Rate and Teleconnect Fund payments made requesting a lower state appropriation risky. Nevertheless, the reserve had grown well beyond the amount needed to mitigate those concerns. The consistent amount of the State’s appropriation leading up to fiscal year 2015–16 compared to the actual expenditures incurred leads us to conclude that there was an apparent lack of fiscal oversight of the program at the state level. The former director of the Educational Data Management Division at Education, who reviewed the budget requests for fiscal years 2013–14 and 2014–15, CALIFORNIA STATE AUDITOR | Report 2016-129 11 May 2017 informed us that the division performed a detailed review of network sites that were lacking capacity and the cost for upgrading that capacity as part of ensuring the network could support statewide student testing. However, the formal budget documents ICOE provided to Education in those years were presented at a summary level and do not identify the costs of individual network site upgrades. Further, Education’s budgetary review did not result in adjustments to the state appropriation amount from previous fiscal years, even as the K12HSN program’s operating reserves grew. The division’s current director explained that Education does not play a direct role in decision making for individual K12HSN projects and that his unit is not currently staffed to conduct a more detailed review. ICOE has made efforts in recent years to improve its budgeting practices. For example, ICOE has begun preparing detailed internal projections for specific costs contained in the CENIC portion of its budget. These projections consist of estimated Internet service and equipment costs for each of the network sites. According to the program’s chief executive officer, ICOE first began using this method during its fiscal year 2015–16 budgeting process. Additionally, starting with the fiscal year 2016–17 budget, ICOE modified the format of the budget proposals it provides to the State to include additional detail regarding the types of costs that make up the total projected CENIC expenditures, such as administrative services and total equipment costs. Despite Its Recent Efforts, ICOE Needs to Improve Its Budgeting Process Although ICOE has recently taken steps to include more information during the K12HSN budgeting process, we identified issues with the accuracy and detail of ICOE’s current budget documents. Since the beginning of our audit in November 2016, and partly because of questions we asked, ICOE has reduced projected K12HSN net expenditures for fiscal year 2017–18 by more than $3 million, from $20.3 million to $16.9 million. The reduced projection for program expenditures has resulted in ICOE’s requesting a lower state appropriation, which was initially established at $14 million and has subsequently been revised to $10.4 million. Table 2 on the following page summarizes the size, timing, and cause of the reductions that ICOE has made during the period of our audit to its fiscal year 2017–18 budgeted program expenditures. Some of these reductions pertain to expensive equipment upgrades that ICOE has decided not to fund but that were still included in its planning document supporting the budget. When we reviewed the detailed planning document that supports the CENIC line item in ICOE’s fiscal year 2017–18 budget for K12HSN, we found 12 Report 2016-129 | CALIFORNIA STATE AUDITOR May 2017 numerous instances in which the document’s cost projections were inconsistent with information contained in other K12HSN planning documents. Because of these inconsistencies, we concluded that ICOE was requesting funding for network projects it was no longer pursuing, was overestimating the anticipated costs of other planned projects, and was omitting in some cases the costs that it plans to incur in fiscal year 2017–18. For example, revisions ICOE made to its network maps after we began our audit indicate that ICOE is no longer planning a network upgrade between two network sites in Fresno County. However, the detailed planning document still included the project as a planned cost for fiscal year 2017–18, and this cost was included in ICOE’s budget request documents. As a result, the budget request was overstating K12HSN’s projected costs by about $100,000 for this project alone. When we raised this issue with the program’s chief executive officer, he confirmed the inconsistencies, and ICOE corrected its planning document and the resulting budget request. In total, these corrections resulted in a $344,000 reduction in ICOE’s budget request. We discuss these changes to ICOE’s plans for the network in greater detail in the following section of this report. ICOE’s projected expenditures also double‑counted a portion of its budgeted equipment costs as both capital expenditures for ICOE and as part of the CENIC expenditures in the K12HSN budget. When we called this discrepancy to the attention of the chief executive officer, he acknowledged the error and reduced K12HSN’s projected equipment costs for fiscal year 2017–18 by $350,000 to correct for it. Table 2 Recap of K12HSN Budget Projections for Fiscal Year 2017–18 DATE PROVIDED TO REQUESTED STATE PROJECTED NET CALIFORNIA STATE AUDITOR APPROPRIATION EXPENDITURES* NOTES December 2, 2016 $14,000,000 $20,265,589 This projection represents the initial budget that ICOE provided to us, for which it had intended to seek approval. February 23, 2017 13,434,120 17,605,142 ICOE generated this version of its budget to illustrate its request if the program were to maintain a $2 million operating reserve for equipment. ICOE reduced projected expenditures as a result of deciding not to pursue some network projects in fiscal year 2017–18 that it had planned when we began our audit. February 23, 2017 11,434,120 17,605,142 ICOE generated this projection to illustrate its request if the program were not to maintain the $2 million operating reserve for equipment. February 27, 2017 10,734,120 17,255,142 ICOE reduced projected expenditures based on our inquiry about double-counting equipment expenditures when we reviewed the previous versions of its budget. March 27, 2017 10,390,581 16,911,603 ICOE made revisions based on discrepancies we identified when reviewing the circuit planning tool used to determine the projected CENIC expense. Sources: California State Auditor’s analysis of K12HSN’s fiscal year 2017–18 budget worksheets and documentation pertaining to the budget assumptions. Note: Reductions in ICOE’s requested appropriation do not always equal reductions in projected net expenditures because some expenditure reductions also affected projected Internet subsidies or resulted in other adjustments to revenue attributable to cost savings carried over from fiscal year 2016–17. * For the purpose of identifying net expenditures, we excluded the federal and state subsidies advanced to K12HSN. CALIFORNIA STATE AUDITOR | Report 2016-129 13 May 2017 Given these accuracy issues, there is a clear need for ICOE to provide additional transparency at the state level about the specific network decisions that affect its costs. Our review of the budget documents ICOE submitted to Education for fiscal years 2016–17 and 2017–18 indicates that ICOE is still presenting CENIC‑related network expenditures without identifying the specific network circuit upgrades associated with those costs. Considering how costly a single network upgrade can be, it is essential that ICOE provide a greater level of detail during the budgeting process to explain its rationale for those projects and whether more cost‑effective options are possible. ICOE’s fiscal year 2017–18 budget documents include multiple scenarios for how various levels of state funding would affect the K12HSN program’s finances given ICOE’s expenditure projections. However, these documents do not include or propose any adjustments to anticipated expenditures to account for higher or lower funding levels. A more meaningful approach would be for ICOE to offer a range of possible expenditures to provide the State with options for adjusting its appropriation amount without adversely affecting the program. For example, ICOE could submit budget documents that include or exclude certain expensive planned network upgrade projects for the next fiscal year, along with information about specific upgrades that could be postponed and the postponements’ impact on the network. It is essential that ICOE provide a greater level of detail during the budgeting process. Alternatively, ICOE could provide the State with the range of possible network capacities that would result from proposed upgrades, which could have very different cost implications. By including a detailed list of potential projects and assigning them priority levels for the upcoming year, along with those projects’ impact on K12HSN expenditures, ICOE would better inform the State about the need for the funds it is requesting, as well as the potential benefits or risks associated with various funding levels for the program. Uncertainty Remains Regarding the Cash Flow and Sustainability of the K12HSN Program The K12HSN program’s cash flow and the sustainability of its program expenditures have implications for the size of a prudent operating reserve. ICOE has expressed concerns in this area because of the timing of reimbursements for Internet subsidies that 14 Report 2016-129 | CALIFORNIA STATE AUDITOR May 2017 support the operation of the network. Typically, these reimbursements make up about 30 percent of K12HSN’s revenue, as was shown earlier in Table 1 on page 8, and thus they represent a significant portion of its revenue. As we explain in the Introduction, there are two general processes by which K12HSN can recognize the financial benefit of Internet subsidies: the service provider can either credit the program up front through an advance, or the provider can bill K12HSN and refund the subsidy later. Service providers generally advance the value of the Teleconnect Fund subsidy for qualifying network services. In particular, CENIC advances the entire value of the subsidy to K12HSN for the portion of the program’s backbone costs that the subsidy covers. In fiscal year 2015–16, 98 percent of Teleconnect Fund subsidies were provided to the program as upfront discounts to ICOE. In contrast, service providers advanced only 28 percent of about $5.7 million in federal E‑Rate subsidies for that same year. ICOE had to therefore use other available funds to cover the remaining 72 percent of those costs until the federal government pledged to reimburse the service providers and those providers refunded the subsidies to K12HSN. These reimbursements have arrived historically in the fiscal year following the year in which ICOE initially incurred the expenditures. According to the K12HSN program’s chief operating officer, ICOE felt the need to maintain a substantial operating reserve for K12HSN in case reimbursements were received later than expected. Figure 4 shows the timing of E‑Rate and Teleconnect Fund reimbursements for fiscal years 2013–14 through 2015–16. The majority of the reimbursements were associated with the E‑Rate program and arrived late in the fiscal year. This delay was especially evident in fiscal year 2015–16. In that year, ICOE had received only 22 percent of the total subsidy for the year as of April 30, 2016. By that same month in fiscal years 2013–14 and 2014–15, ICOE had already collected all or nearly all of its E‑Rate funds. Consequently, ICOE had to support the cost of operations for the first three quarters of fiscal year 2015–16 using the program’s operating reserve. Further, two reimbursements representing 8 percent of the E‑Rate reimbursements ICOE was owed did not arrive until October 2016, after the close of the 2015–16 fiscal year. ICOE has expressed concern that these delays could become even longer in future years, placing additional pressure on K12HSN’s cash position. This concern has arisen because of changes the Federal Communications Commission made to the E‑Rate program in 2014, which include adding new technologies to those qualifying for the subsidy and updating the electronic application submission process. According to ICOE’s E‑Rate consultant, these changes lengthen the process for receiving reimbursements because of extensions to deadlines for filing funding applications. Although ICOE has historically had sufficient funds to support operations while waiting for E‑Rate reimbursements, the trend shown in Figure 4 indicates that the delays associated with those reimbursements pose cash‑flow challenges for the K12HSN program. CALIFORNIA STATE AUDITOR | Report 2016-129 15 May 2017 Figure 4 Cumulative Percentages of Internet Subsidy Revenue Received by ICOE Fiscal Years 2013–14 Through 2015–16 100% 80 60 40 20 0 July August Septe mber October Nove mber Dece mber January February March April May June ydisbuS tenretnI fo egatnecreP evitalumuC EOCI yb devieceR euneveR 100% 80 60 40 20 0 July August Septe mber October Nove mber Dece mber January February March April May June FISCAL YEAR 2015–16* 92% 83% 22% 22% 22% 0% 0% 0% 3% 11% 11% 14% ydisbuS tenretnI fo egatnecreP evitalumuC EOCI yb devieceR euneveR 100% 80 60 40 20 0 July August Septe mber October Nove mber Dece mber January February March April May June FISCAL YEAR 2014–15 99% 99% 99% 100% 42% 35% 35% 35% 26% 5% 5% 6% ydisbuS tenretnI fo egatnecreP evitalumuC EOCI yb devieceR euneveR FISCAL YEAR 2013–14 100% 100% 100% 100% 32% 32% 4% 7% 16% 20% 0% 0% Sources: California State Auditor’s analysis of statements for the bank account held on behalf of the K12HSN program by CENIC and ICOE’s financial records. Note: For fiscal years 2013–14 through 2015–16, 96 percent of ICOE’s accrued subsidy revenue was for the federal E-Rate subsidy and only 4 percent was for the Teleconnect Fund subsidy. This figure addresses only subsidy revenue owed to ICOE in the form of reimbursements because the service providers otherwise advance the subsidies and ICOE does not pay for those portions of its Internet services. * As of the end of fiscal year 2015–16, ICOE had received only 92 percent of the total subsidy reimbursement amount it was owed for fiscal year 2014–15. ICOE received the remaining 8 percent in October 2016. 16 Report 2016-129 | CALIFORNIA STATE AUDITOR May 2017 The State has taken steps to address ICOE’s concerns about the timing of E‑Rate reimbursements. Before 2016 Education provided the annual state appropriation to ICOE in two installments. For example, Education paid 90 percent in March and January of fiscal years 2013–14 and 2014–15, respectively, and it paid the remaining 10 percent in December of the following fiscal year in both instances. In 2016 the Legislature amended EDC 11800 to direct Education to pay 75 percent of the total state K12HSN annual funding amount by August 31 of each fiscal year and the remaining 25 percent by January 31. This new funding schedule, which took effect in fiscal year 2016–17, addresses some of K12HSN’s cash‑flow challenges by providing a substantial portion of the state appropriation early in the fiscal year. However, as we discuss in greater detail later, the adjustment to the funding schedule alone may not be enough to sustain the program throughout the year, given ICOE’s planned expenditures. This new funding schedule addresses some of K12HSN’s cash‑flow challenges but may not be enough. In addition to maintaining an operating reserve to ensure that it can meet planned expenditures, ICOE’s proposed budget had designated portions of the K12HSN program’s excess cash for emergency replacement of equipment. ICOE initially informed us that its target operating reserve for this purpose was $2.5 million and that this amount was based on the statewide cost to replace failed equipment across network sites in 2013. However, since 2013 ICOE has already replaced the equipment for many of the network sites, staggering the useful lives of current network equipment and reducing the likelihood of a comprehensive failure. ICOE agreed with our concerns regarding its methodology, but it has not yet developed a new way to determine a prudent equipment reserve. Developing this methodology will help the State determine whether a reserve level above the minimum necessary to ensure regular network operations is justified. Using the K12HSN budget for fiscal year 2016–17 and the state appropriation amount of $8 million, we estimate that the program’s operating reserves at the end of the fiscal year will total almost $600,000 in addition to another $6.4 million expected from E‑Rate reimbursements in the following year. The fiscal year 2017–18 CALIFORNIA STATE AUDITOR | Report 2016-129 17 May 2017 budget that ICOE submitted to the State in March 2017 requested $10.4 million. The budget identified $16.9 million in net expenditures for the year, some of which will be reimbursed as subsidy payments after the close of fiscal year 2017–18. Using these budgeted expenditures and the timing of past E‑Rate reimbursements, we projected the K12HSN program’s cash levels for fiscal year 2017–18 under two possible funding scenarios—the $8 million in the Governor’s proposed budget and the $10.4 million that ICOE submitted as its request. Our analysis indicates that to avoid program deficits, ICOE will require an increase in the State’s historical appropriation in fiscal year 2017–18. We estimated that under the $8 million scenario, the program would begin the 2017–18 fiscal year with an operating reserve of nearly $600,000 and end the year with a deficit of $2.2 million. In contrast, with a $10.4 million appropriation, the program would end the year with a reserve of slightly more than $200,000. However, depending on the timing of E‑Rate reimbursements and how quickly ICOE implements some of its planned network upgrades, the K12HSN program could deplete its reserve at some point during fiscal year 2017–18, even with this larger appropriation. In past years, ICOE has addressed the issue of potentially problematic delays in subsidy reimbursements by carrying a large cash reserve for the program. Although it is aware of this issue, because the K12HSN program no longer holds a large reserve and because program revenue and expenditures have not aligned historically, it is important that ICOE closely monitor the program’s cash levels and adjust the timing of its expenditures as needed. In the next section, we discuss our concerns with aspects of the network planning process that have driven projections of the program’s future costs, including planned costs for some expensive network upgrades that have contributed to ICOE’s request for a larger state appropriation in fiscal year 2017–18. Recommendations Legislature To help ensure continuous network operations while preserving state resources, the Legislature should appropriate to the K12HSN program an amount that does not exceed $10.4 million for fiscal year 2017–18. If the Legislature wishes to appropriate a lower amount for the program, it should direct ICOE to modify one or more of the planned network upgrades we highlight in the following section, either by delaying the upgrade to a subsequent fiscal year or by pursuing a less expensive option. 18 Report 2016-129 | CALIFORNIA STATE AUDITOR May 2017 ICOE To better inform decision makers at the state level about the amount of funding necessary to operate and maintain the network, ICOE should formally amend its annual budget documents by November 2017 to specify multiple potential levels of network expenditures for the coming year, and it should detail the specific network upgrades and project costs included in each scenario. As part of this process, ICOE should also provide information about how these upgrades will affect the network’s functionality. To ensure that its projected program costs are as accurate as possible, ICOE should institute by November 2017 a formal practice for reviewing its budget planning document against its current network design plans and correct any inaccuracies before finalizing and submitting its budget. To help ensure that the K12HSN program maintains the necessary amount of state funds in reserve, ICOE should prepare a formal methodology for a proposed equipment reserve that is based on the actual likelihood of equipment failure and the costs associated with replacing that equipment. To ensure that it is able to continue critical network services with reduced operating reserves, ICOE should establish procedures to routinely monitor the K12HSN program’s cash balance and to evaluate upcoming costs. If at any point ICOE determines that it will be unable to fund its costs due to delayed subsidy payments, it should notify Education regarding the size and timing of the anticipated shortfall and postpone significant discretionary expenditures, such as upgrading network site equipment, until ICOE collects the subsidies it is owed. CALIFORNIA STATE AUDITOR | Report 2016-129 19 May 2017 ICOE Needs to Improve Its Planning Processes in Order to Manage Network Development at the Lowest Possible Cost to the State Key Points • ICOE chose network upgrades that were not the most cost‑effective options available. • ICOE could not justify some expensive upgrade decisions because it had incomplete information. • The network’s structure of backup routes has increased the costs of circuit upgrades, but the benefits of these backup routes are unclear. • ICOE has generally claimed available subsidies for network costs, but it has not done so in some cases. ICOE Chose Network Upgrades That Were Not the Most Cost‑Effective Options Available ICOE’s current process for determining the capacity needs for the network it manages does not sufficiently account for cost differences among available options. The cost of increases to network capacity depends on the service provider’s pricing, the location of the circuits being upgraded, and the magnitude of the capacity increase—the quantity of additional capacity that will be provided. When ICOE decides to increase the capacity of a circuit, it generally has the option to double, triple, quadruple, or provide a tenfold increase in the circuit’s capacity as measured in gigabits per second (gigs). In some circumstances, ICOE may choose to upgrade capacity using dark fiber, which is a type of circuit that can provide capacities up to and exceeding 100 gigs. Generally speaking, increasing a circuit’s capacity tenfold is more expensive than other options and also requires the acquisition and installation of new equipment.1 In some cases, when upfront installation costs are low—as they were for one of the circuits that ICOE procured for the network site at Santa Clara—procuring dark fiber may be less expensive than other available options. The program’s chief executive officer explained that ICOE has opted for tenfold capacity increases generally because it has had difficulty predicting the future use of a circuit and would rather err on the side of providing too much network capacity than too little. By consistently pursuing large increases in capacity, ICOE has chosen costly upgrade options even though less expensive options were available for more intermediate capacity increases. We reviewed ICOE’s decisions to upgrade the capacities of 20 K12HSN circuits, as well as its decisions to upgrade another 10 circuits connected to those circuits for a total of 30 circuits. We included the additional 10 circuits in our review because of ICOE’s practice of upgrading some circuits to provide backup routes to the backbone for network sites to use when their 1 Doubling, tripling, or quadrupling a circuit’s capacity may in some circumstances be more expensive than increasing capacity tenfold, as when dark fiber upfront costs are low. Additionally, according to the K12HSN program’s chief executive officer, depending on the equipment already in use at a network site, the site may also require new equipment for these incremental increases. 20 Report 2016-129 | CALIFORNIA STATE AUDITOR May 2017 primary circuits fail. We found that ICOE consistently selected the least expensive option among various service providers at a given capacity level. For example, when choosing a service provider for the 100‑gig circuit connecting its Chaffey and San Bernardino network sites, ICOE chose the least costly provider, which ICOE’s bid documents indicate was $164,000 less in annual service costs than the competing bids. However, on multiple occasions, ICOE decided to pursue capacity levels that were not cost‑effective. Specifically, in nine of 30 instances, ICOE chose a more expensive tenfold capacity upgrade option even though it received bids for other less costly incremental upgrade options. For example, ICOE chose to upgrade one of its Contra Costa circuits from 10 gigs of capacity to 100 gigs. ICOE also requested and received bids from service providers for doubling the circuit’s capacity to 20 gigs. According to its planning documents, the decision to select the 100‑gig upgrade will cost the K12HSN program an additional $129,000 in one‑time costs and $30,000 more in annual costs. In another instance, ICOE chose to increase the capacity of one of its Alameda site circuits from 10 gigs to 100 gigs. ICOE also had the option of tripling the circuit’s capacity to 30 gigs. According to ICOE’s planning documents, selecting the 100‑gig option will cost K12HSN an additional $129,000 in one‑time costs and another $87,600 in annual costs compared to the cost of the upgrade to 30 gigs. Table 3 provides equivalent information and cost implications for the nine upgrade decisions for which we identified less expensive options. Among the nine upgrades we could compare to less expensive options, ICOE’s planned upgrades will cost a total of $206,056 in additional annually recurring costs and $2 million in additional one‑time costs. These additional costs result from ICOE’s practice of pursuing tenfold upgrades when increasing circuits’ capacities even when service providers have offered less expensive incremental increases. Moreover, we identified some instances in which ICOE did not even consider less expensive upgrade options for the upcoming 2017–18 fiscal year. In four of the decisions we reviewed, which are also listed in Table 3, ICOE chose to upgrade circuits from 10 gigs to 100 gigs or dark fiber without requesting that service providers submit pricing details for 20‑gig, 30‑gig, or 40‑gig alternative options. ICOE thus cannot demonstrate that the results of its decisions will be cost‑effective, and we could not determine how much ICOE could have saved with an alternative option. Furthermore, federal regulations require competitive bids for all services supported by E‑Rate subsidies; agencies applying for these subsidies are directed to post their requests for those services on USAC’s website. Consequently, ICOE will not qualify for federal E‑Rate subsidies if it procures these alternative capacity options in the future without first competitively bidding them. Instead, its options are limited to moving ahead with the planned expensive upgrade or waiting to rebid the circuit in the following year. According to the program’s chief executive officer, ICOE did not CALIFORNIA STATE AUDITOR | Report 2016-129 21 May 2017 request bids for the 20‑gig or 40‑gig capacity options for fiscal year 2017–18 because the circuits were already operating at 10 gigs, and ICOE’s general rule is to upgrade circuit capacity tenfold, so it decided to upgrade the circuits to 100 gigs or dark fiber. However, our review of these circuits’ actual usage determined that ICOE cannot justify this universal and costly approach when less‑expensive options exist. Table 3 Comparison of Costs Between ICOE’s Circuit Upgrades and Alternative Options ADDITIONAL COSTS: DIFFERENCE BETWEEN PLANNED UPGRADE INITIAL FISCAL ICOE’S PLANNED UPGRADE COSTS AND LEAST COSTLY ALTERNATIVE YEAR OF OPERATION CAPACITY ADDITIONAL ADDITIONAL OR PLANNED REASON FOR UPGRADE ONE‑TIME ANNUAL COST ONE‑TIME OPERATION UPGRADE LEVEL CIRCUIT REVIEWED ANNUAL COST UPFRONT COST (SAVINGS) UPFRONT COST 2015–16 Capacity 1 gig to Mendocino County Office $119,040 $47,640 $63,840 $47,640 10 gigs of Education (COE) to CalREN Sunnyvale 2015–16 Backup route 10 gigs to Los Angeles COE to 88,800 364,000 (24,479) 363,000 dark fiber CalREN Los Angeles 2016–17 Capacity 10 gigs to Chaffey Joint Unified High 42,000 633,000 (72,072) 484,979 dark fiber School District (JUHSD) to CalREN Los Angeles 2016–17 Capacity 10 gigs to Alameda COE to 193,200 129,000 87,600 129,000 100 gigs CalREN Oakland 2016–17 Capacity 10 gigs to Contra Costa COE to 84,000 129,000 30,000 129,000 100 gigs CalREN Oakland 2016–17 Backup route 10 gigs to Chaffey JUHSD to 100,800 438,000 22,728 204,767 dark fiber San Bernardino County Superintendent of Schools (CSS) 2016–17 Backup route 10 gigs to Santa Clara COE to 16,697 542,954 (12,103) 469,034 dark fiber CalREN Oakland 2016–17 Backup route 10 gigs to Fresno COE 1 to 149,940 129,983 80,052 128,983 100 gigs CalREN Fresno 2017–18 Lower network 1 gig to Lassen COE to 107,940 50,000 30,490 50,000 costs* 10 gigs CalREN Sacramento 2017–18 Capacity 10 gigs to Fresno COE 2 to 96,000 202,500 In December 2016, when 100 gigs CalREN Fergus ICOE put these circuits out 2017–18 Capacity 10 gigs to Riverside COE at Calhoun to 147,516 200,000 to bid for upgrade, it did not 100 gigs CalREN Palm Desert request bids for upgrades at capacity levels lower than 2017–18 Capacity 10 gigs to San Bernardino CSS to 78,000 400,000 100 gigs/dark fiber. As a dark fiber CalREN University of result, we could not compare California Riverside the costs of ICOE’s chosen 2017–18 Backup route 10 gigs to Sacramento COE to 60,000 0 upgrades to those of lower 100 gigs CalREN West Sacramento capacity upgrades. Totals $1,283,933 $3,266,077 $206,056 $2,006,403 Source: California State Auditor’s analysis of ICOE planning documents. Note: The information in this table is based on cost estimates used in the budget projections ICOE submitted to the State on March 29, 2017. * ICOE upgraded this circuit from 1 gig to 10 gigs even though neither this circuit’s usage nor that of any other circuit partnered with it met ICOE’s criteria for an upgrade. ICOE was able to acquire the greater capacity at a lower cost than the cost for the original 1-gig capacity. However, ICOE could have selected a lower-capacity option of 2 gigs and reduced network costs even further. 22 Report 2016-129 | CALIFORNIA STATE AUDITOR May 2017 Because It Had Incomplete Information, ICOE Could Not Justify the Costs of Its Most Expensive Upgrade Decisions ICOE’s current methodology for determining the timing and magnitude of increases to network capacity does not ensure that those increases are justifiable. The K12HSN program’s chief executive officer explained that ICOE generally considers increasing a circuit’s capacity when it experiences usage peaks that approach or exceed 50 percent of total capacity. Although ICOE has not performed a formal analysis to support the rationale for its practice, the program’s chief executive officer explained that the 50 percent threshold is based on historical patterns of use at certain network sites, and it is intended to allow time for ICOE to upgrade the circuit before it reaches full capacity and the speed of data transmission is negatively affected. Although ICOE chose to upgrade specific circuits identified in Table 3 because they were operating near or above 50 percent of their capacity, we found that ICOE’s available usage data did not demonstrate a need for the level of upgrade it had chosen. For example, Figure 5 shows usage data for a circuit that ICOE decided to upgrade because its usage reached 50 percent of its capacity. However, the usage data ICOE retained did not demonstrate an inherent need for a tenfold increase in capacity, as is ICOE’s general practice. By upgrading to a 20‑gig circuit instead, ICOE could have saved $129,000 in one‑time upfront costs and $30,000 annually at this site while likely meeting the site’s needs for at least the short term. We believe ICOE could do more to improve its ability to predict circuit use, thereby avoiding costs that are higher than necessary for providing capacity. ICOE has not possessed sufficient data on historical network use to justify its decisions to pursue expensive capacity upgrades instead of more cost‑effective alternatives. The primary factor that ICOE considers to support its upgrade decisions is the use of its circuits. ICOE reviews usage data collected by CENIC’s monitoring software, Cricket, and by its own network monitoring software, SolarWinds Orion (Orion). These programs provide measurements of a circuit’s usage over time, expressed in terms of gigs. Cricket retains these data for roughly one year, and Orion retains the data for about two years. Both systems purge any data beyond their retention periods. ICOE could archive Orion data before the retention period expires but has not done so in the past. Although this two‑year period may be sufficient to enable ICOE to determine whether a circuit requires an upgrade—for example, if a circuit is being used at maximum capacity for multiple consecutive months—the period is not long enough to demonstrate the level of new capacity required. ICOE intends to use Orion to monitor the use of all K12HSN circuits against their capacity. May 2017 CALIFORNIA STATE AUDITOR | Report 2016-129 23 Figure 5 Capacity Levels and Costs for a Specific Upgrade to the K12HSN Program’s Network January F 2 e 0 b 1 r 5 uary 201 M 5 arch 2015 April 2015 May 2015 June 2015 July 201 A 5 ugu S st e p 20 te 1 m 5 ber 2 O 01 c 5 tobe N r 2 o 0 v 1 e 5 mber D 20 e 1 ce 5 mber 2 J 0 a 1 n 5 uary 2 F 0 e 1 b 6 ruary 201 M 6 arch 2016 April 2016 May 2016 June 2016 July 20 A 1 u 6 gus S t e 2 p 0 t 1 e 6 mber 20 O 1 c 6 tobe N r o 2 v 0 e 1 m 6 ber D 2 e 0 c 1 e 6 mber 20 J 1 a 6 nuary 20 Fe 1 b 7 ruary 20 M 17 arch 2017 yaD hcaE desU sgiG mumixaM Cost of planned upgrade: Planned upgrade—100 gigs (tenfold current circuit capacity) 100 • $84,000 annual cost • $129,000 one-time cost 90 80 70 Because this circuit's usage reached 50 percent of its capacity during the highlighted period, it broadly fits within ICOE's guidelines for initiating capacity upgrades. ICOE plans to upgrade this circuit to tenfold its current capacity (from 10 gigs to 100 gigs) rather than doubling or tripling its capacity. However, the circuit's usage since that time has not warranted the cost 60 associated with the chosen upgrade when less expensive options are available. 50 40 Cost of alternative upgrade #2: Alternative upgrade #2—30 gigs (triple current circuit capacity) 30 • $81,000 annual cost • No one-time cost Cost of alternative upgrade #1: Alternative upgrade #1—20 gigs (double current circuit capacity) 20 • $54,000 annual cost • No one-time cost Current circuit capacity—10 gigs 10 50% of current circuit capacity 0 Sources: California State Auditor’s analysis of daily maximum usage data from Orion for the K12HSN Contra Costa County Office of Education network site and ICOE’s planning documents for K12HSN. 24 Report 2016-129 | CALIFORNIA STATE AUDITOR May 2017 However, as part of our overall review of K12HSN circuit usage that included the 30 circuits discussed previously, we found eight instances in which ICOE either did not monitor a circuit or began monitoring it only after a significant amount of time had passed since its installation, thereby limiting any review of the circuit’s usage at the time of or for the period following its last upgrade. According to the program’s chief executive officer, any unmonitored circuits are the result of administrative error. When ICOE does not monitor a circuit with Orion, its only option for systematically evaluating historical usage is to rely on Cricket’s limited one‑year view. We found eight instances in which ICOE either did not monitor a circuit or began monitoring it only after a significant amount of time had passed. Access to additional historical data may be useful in assessing a circuit’s needs as its usage increases. For example, if ICOE decides to upgrade a circuit’s capacity in a given year based on recent usage, by the time it revisits that circuit for another potential upgrade in two years, the original data that it used to determine the initial upgrade will have been purged. This information would have allowed ICOE to compare rates of growth in usage before and after it last upgraded the circuit in order to evaluate whether growth rates have been constant or are accelerating, as well as to account for any historical jumps in usage that might be followed by long, stable periods. Additionally, retaining historical data pertaining to the circuits serving high‑use network sites could help ICOE predict how circuits serving sites that currently have lower capacity needs may ultimately behave. If it retains these data by periodically downloading them from Orion, ICOE could maintain indefinite access to circuits’ historical usage at minimal cost. The K12HSN program’s chief executive officer questioned whether retaining data for more than two years would add additional value but agreed that it would be easy to do. We believe that doing so can only improve ICOE’s ability to project the long‑term capacity needs of individual circuits, which in turn would help ensure that the costs of network upgrades are necessary and prudent. CALIFORNIA STATE AUDITOR | Report 2016-129 25 May 2017 In addition, ICOE has not systematically collected information from network sites about possible future use trends when deciding whether to increase those circuits’ capacities. ICOE maintains a database that stores information about the network capacities of school district circuits that connect to the network sites. According to the program’s chief executive officer, ICOE uses the information in the database to gain insight into the maximum possible usage it can anticipate from any district and the maximum possible usage at a network site. However, the program’s chief operating officer informed us that this information is incomplete because it is updated voluntarily by districts and not verified by ICOE. Furthermore, we question whether this information would be of any value in determining network sites’ actual capacity needs because of the minimal likelihood that all districts would be using their maximum capacities at the same time. Rather than focusing on maximum capacity, ICOE could improve its current practices by regularly collecting information about districts’ actual expected usage. Ideally, this practice would include formal communications with staff members at network sites— where districts connect to the network—and focus on those sites’ knowledge of the network usage plans of their connecting districts. With a greater understanding of network sites’ future usage plans, ICOE would be in a better position to identify those upgrade capacities that are truly needed and how soon those needs would materialize. The program’s chief executive officer explained that ICOE currently discusses anticipated capacity needs with representatives of network sites, but it does so informally on a case‑by‑case basis and generally when the network site initiates the conversation. He also stated that ICOE is interested in further monitoring the use and anticipated use of the network by districts and believes this effort would add value to its decision process, although it has not yet implemented this monitoring. ICOE’s Practice of Providing Equivalent Capacity Backup Routes Results in Additional Costs When ICOE Increases the Capacity of a Primary Circuit To ensure continuous access to the backbone for network users, particularly when dealing with power outages or other network interruptions, ICOE has structured the network to allow at least two routes between most locations and the backbone: a primary circuit and at least one alternative route. Although this strategy minimizes the likelihood that users will lose access to the backbone, it also means that in most cases an increase to a primary circuit’s capacity necessitates a commensurate increase to one or more other circuits to accommodate any growth in traffic expected for the primary circuit if that circuit fails. These increases would 26 Report 2016-129 | CALIFORNIA STATE AUDITOR May 2017 be necessary regardless of the level of traffic those other backup circuits incur for their regular dedicated purposes. In most cases, a site’s backup route involves a circuit to a neighboring network site and then using that site’s connection to the backbone. This structure has significant cost implications in situations where neighboring sites have different levels of usage. For example, according to the program’s chief executive officer, usage levels for one of the circuits at the Los Angeles Unified School District (LAUSD) network site prompted ICOE in fiscal year 2015–16 to plan to install a new dark fiber circuit connecting the site to its neighboring network site at the Los Angeles County Office of Education, as well as to upgrade one of the county office site’s circuits from 10 gigs of capacity to dark fiber to ensure an effective backup route. As of March 2017, the LAUSD site’s circuit was operating at roughly 17 gigs, while the county office circuit was operating only at around 7 gigs. Upgrading the lower‑usage county office circuit to dark fiber—which represents the equivalent of 100 gigs in terms of potential capacity—in order to provide a backup route is expensive. According to ICOE’s planning documents, the combined cost for both circuit upgrades in their first year of operation will be greater than $1 million. In other instances, as an alternative means to provide backup routes, ICOE chose to add new circuits to the network rather than to upgrade existing circuits. This approach is an alternative to ICOE’s frequent practice of pairing network sites and their circuits to provide backup routes to the backbone. Under this approach, once ICOE has decided to upgrade a network site’s primary circuit, it also installs a circuit between the site and another part of the backbone to provide a new backup route. This backup route structure has significant cost implications in situations where neighboring sites have different levels of usage. ICOE plans to add new circuits to provide backup routes for some network sites whose existing circuits ICOE has already upgraded or plans to upgrade. This practice can be more efficient than pairing network sites with different usage levels because doing so means the decision to upgrade an existing circuit to a higher capacity requires only a single additional circuit upgrade for redundancy rather than two upgrades. However, given the costs associated with high‑capacity upgrades, this practice still has significant cost CALIFORNIA STATE AUDITOR | Report 2016-129 27 May 2017 implications for the K12HSN program. For example, ICOE planned to upgrade the existing circuit connecting the Santa Clara network site to the network backbone because of this circuit’s current usage. To provide a backup route for the upgraded circuit, ICOE planned to disconnect Santa Clara from its current network site partner and install a second circuit directly to the backbone. ICOE’s planning documents indicate that this additional circuit will cost roughly $550,000 in its first year. Although its practice of providing backup routes adds significant costs for operating the network, ICOE cannot demonstrate the actual benefit this approach provides. We found that nearly all 86 network sites have backup routes for their circuits. Further, in five of the instances we reviewed wherein ICOE selected upgrades to 100 gigs or dark fiber at significant additional cost, it did so to provide backup routes consisting of completely new circuits or as upgrades to existing circuits whose usage was significantly less than the circuits with which they were paired. Table 3, presented earlier on page 21, lists these circuits and their costs. As we discuss in the following major section of this report, ICOE has not systematically tracked or reported on the frequency or duration of network outages; therefore, it is unclear how often the network actually relies on backup routes to keep schools and districts connected. As a result, the benefit achieved by the significant costs of the current backup route structure is also unclear. Missed Internet Subsidies Resulted in Some Extra Costs for the State ICOE has not ensured that it applies for the E‑Rate subsidy for all of its network upgrades. As the Introduction describes, CENIC manages all service provider contracts on behalf of ICOE and the K12HSN program. Although CENIC is the party of record in the E‑Rate application process, which it administers with the assistance of a third‑party contractor, ICOE is ultimately responsible as the K12HSN lead agency for ensuring that CENIC secures the E‑Rate subsidy whenever possible. We reviewed documentation for 20 circuit upgrades to determine whether ICOE ensured that CENIC applied for the E‑Rate subsidy and, if so, whether CENIC submitted the subsidy applications within program deadlines. During this process, we identified one instance in which ICOE decided to proceed with a circuit upgrade that did not qualify for an E‑Rate subsidy because the circuit was procured without competitive bidding. Specifically, ICOE upgraded one of its circuits at the network site at Santa Maria Joint Unified High School District from 1 gig to 10 gigs of capacity. The K12HSN program’s chief executive officer told us that to meet the E‑Rate requirement, ICOE had initially sought competitive bids 28 Report 2016-129 | CALIFORNIA STATE AUDITOR May 2017 for the upgrade. However, the only bid ICOE received was from a service provider that would have difficulties installing the upgrade because of its limited infrastructure in the area. We identified one instance in which ICOE decided to proceed with a circuit upgrade that did not qualify for an E‑Rate subsidy, and CENIC informed us of another instance. According to the program’s chief executive officer, ICOE asked its existing provider to install the upgrade after determining that the circuit’s peak use was at its capacity and ICOE anticipated that the provider could install the upgrade quickly. Because this procurement with the existing service provider occurred without competitive bidding, the circuit is not eligible for E‑Rate for the duration of that contract. We estimate that nearly $100,000 of the circuit’s annual costs would have been reimbursed by E‑Rate subsidies. After the circuit was installed in July 2016, ICOE again sought competitive bids for the same circuit in December 2016 as part of the most recent E‑Rate application period, and it received a bid that it plans to implement in fiscal year 2017–18. At this time, ICOE does not have a planned date to disconnect the circuit, which continues to incur unsubsidized costs. In addition, CENIC informed us of one other instance beyond the 20 circuits we reviewed in which ICOE did not secure an E‑Rate subsidy for a circuit it procured. The K12HSN program’s chief executive officer clarified that the circuit was procured outside of E‑Rate as a temporary solution because the existing 1‑gig circuit was operating at capacity and ICOE’s planned upgrade was not occurring promptly. This circuit was in use between May 2015 and December 2016, and we estimate that E‑Rate could have reimbursed $32,000 of the circuit’s costs to the State in its first full year of service. For the other 19 circuits we reviewed, we used application identification numbers provided to us by CENIC to track those applications through the various phases of the E‑Rate process for which CENIC is responsible. Two of the circuits—although physically part of the network—are owned and paid for by county offices of education, meaning that ICOE in its role as the K12HSN program administrator is not responsible for securing subsidies for those circuits. For the remaining 17 circuits, CENIC appears to have complied with deadlines for posting circuit projects for CALIFORNIA STATE AUDITOR | Report 2016-129 29 May 2017 competitive bidding, submitting funding applications, and verifying to USAC that Internet services had begun. Because USAC does not require applicants to identify the individual broadband products for which they are seeking subsidies, we could not determine definitively whether specific circuits were included in the E‑Rate application materials we reviewed. Nevertheless, we were able to link a majority of the circuits we reviewed to the application forms using contract numbers. Recommendations ICOE To better guarantee that network upgrades are necessary and are achieved at the lowest possible cost to the State, ICOE should develop a formal methodology for reviewing circuit capacity needs. This methodology should include consideration of multiyear trends in network traffic and the implications prospective upgrades may have for other parts of the network. Doing so would not only assist ICOE when determining the magnitude of circuit upgrades relative to the cost involved, but would also help ICOE determine whether it can delay upgrades until it can establish those upgrades’ eligibility for E‑Rate subsidies. To provide as many options for network upgrades as possible and to help ICOE provide the most cost‑effective upgrade options without risking its eligibility for subsidies, ICOE should adopt the practice of requesting bids at all feasible levels of capacity upgrades as opposed to only those levels that represent a tenfold increase in circuit capacity. To help facilitate the review of circuit capacity needs, ICOE should maintain historical data for network traffic as long as technically feasible. It should also ensure that its monitoring software includes all network sites. To reduce the risk of having to react to large increases in network traffic, ICOE should formalize a process to include input from network site administrators during network upgrade planning. 30 Report 2016-129 | CALIFORNIA STATE AUDITOR May 2017 Blank page inserted for reproduction purposes only. CALIFORNIA STATE AUDITOR | Report 2016-129 31 May 2017 Measurement of the Program’s Effectiveness Has Omitted Key Information, and Oversight Has Been Inconsistent Key Points • ICOE’s reporting on the network’s performance has not addressed some of the goals and responsibilities named in state law. • In its annual reports and updates to the K12HSN advisory board, ICOE has not reported on other performance metrics in a consistent manner. Tracking and Reporting of the Network’s Performance Is Not Comprehensive ICOE’s reporting on the network’s performance has not included performance measures for some of the goals and services established for the K12HSN program in state law. EDC 11800 establishes primary responsibilities for ICOE as the lead educational agency responsible for administering the K12HSN program. In addition, EDC 11800 assigns the superintendent, who oversees Education, responsibility for measuring K12HSN’s primary responsibilities include the success of the K12HSN program, and makes the following: the K12HSN advisory board responsible for recommending policy direction and providing • Reliable and cost-effective Internet service. broad operational guidance to the superintendent • Videoconferencing. and ICOE. As shown in the text box, EDC 11800 • Reliable and secure interconnectivity. requires K12HSN to provide critical services, including reliable and cost‑effective Internet service Source: EDC 11800. and interconnectivity among public school entities. To report on the network’s performance, ICOE and CENIC—at the direction of ICOE—provide network capacity statistics to the K12HSN advisory board and Education through annual reports and presentations during the advisory board’s quarterly meetings. For example, ICOE provided information in its annual reports describing the number of network sites and their circuits’ capacity, as well as the percentages of schools and districts connected to the network. More recently, ICOE has also reported on the connectivity and capacity of individual districts and schools. However, neither the annual reports nor the information ICOE has provided at the advisory board meetings addresses the reliability of the network. In its annual reports, ICOE stated that network operators at CENIC are able to control latency, a measurement of data delay, and packet loss, a measurement of data sent compared to data received. Videoconference users, for example, experience latency as a loss of video and audio synchronization, which may result in participants’ talking over each other because of the delay. With even a small amount of packet loss, a videoconference may periodically lose audio, and images may freeze. Industry standards support both measures as appropriate methods to gauge a network’s quality of service. However, ICOE has not provided statistics in 32 Report 2016-129 | CALIFORNIA STATE AUDITOR May 2017 its annual reports regarding the latency or packet loss experienced over the network. Additionally, none of the materials provided to the advisory board during the period from February 2012 through November 2016 contains any information on packet loss or latency over the portions of the network funded by the K12HSN program. Although we identified a reference to latency in a document from a meeting in May 2014, that document did not provide information on latency between CalREN and the network itself. Instead, it provided some measurements of latency between the backbone and the testing servers in Chicago as part of efforts to prepare for the California Assessment of Student Performance and Progress computer‑based testing described in the Introduction. Moreover, ICOE has not reported to Education or the advisory board on the network’s overall reliability—including the frequency, duration, location, or cause of network service outages or interruptions. Doing so would help those responsible for overseeing ICOE’s administration of the K12HSN program determine whether network reliability is increasing or decreasing, and it would also ensure that ICOE effectively addresses any potential reliability issues associated with specific service providers or geographical regions. ICOE has not reported to Education or the advisory board on the network’s overall reliability—including the frequency, duration, location, or cause of network service outages or interruptions. According to the program’s chief operating officer, CENIC formally notifies ICOE when a network connection is not functioning, and ICOE monitors the outages in real time. However, at this time, ICOE tracks these outages informally, and the program’s chief executive officer stated that ICOE does not formally report them to Education. The chief operating officer also stated that ICOE attempts to ensure the network’s reliability by providing multiple connections to diverse sites using different service providers, so that if one provider’s service is not operating at full capacity, others may continue to function without interruption. However, because ICOE does not report on the frequency, duration, location, or cause of network outages, nor on the latency and packet loss experienced by the network, it cannot demonstrate the effectiveness of its efforts to monitor reliability. CALIFORNIA STATE AUDITOR | Report 2016-129 33 May 2017 ICOE has also not reported on another critical aspect of managing the network: the cost‑effectiveness of the Internet service it provides through CENIC on behalf of the State. Specifically, ICOE has not reported the costs of providing Internet service to its users in a manner that allows stakeholders to compare annually whether ICOE is doing better or worse in providing network connectivity at the lowest possible cost. Additionally, as we explain earlier in this report, ICOE has not provided sufficiently detailed information in its budgeting documents about the need for and cost implications of specific circuit upgrades. ICOE has noted in its annual reports that contracting with CENIC results in deep discounts for the K12HSN program because of the volume of CENIC’s purchases of Internet services from service providers. However, ICOE has not quantified these savings or measured them against costs associated with CENIC’s administration of the network. As a result, ICOE has not established whether and why the decisions it has made for K12HSN are the most cost‑effective. When we asked about the lack of reporting, the program’s chief operating officer explained that ICOE could provide data on the historical cost per unit of network connectivity, but Education has not asked for this type of information in ICOE’s annual reports about K12HSN. Nevertheless, she acknowledged that ICOE could do more to demonstrate the cost‑effectiveness of its Internet service. We believe that ICOE could accomplish this objective by monitoring and reporting on network costs not only in terms of available capacity, but also per unit of network capacity actually used. ICOE has not reported on the cost‑effectiveness of the Internet service it provides through CENIC on behalf of the State. As mentioned earlier, the superintendent who oversees Education is primarily responsible for measuring the success of the K12HSN program. However, Education does not conduct comprehensive monitoring of network reliability, including frequency and duration of network interruptions or other performance measures. We spoke with the current and former directors of Education’s Educational Data Management Division, who informed us that there is no State General Fund allocation provided to Education for the work associated with overseeing K12HSN. The current director explained that Education monitors the network’s performance by reviewing outage reports from ICOE and testing summary reports from the 34 Report 2016-129 | CALIFORNIA STATE AUDITOR May 2017 private contractor responsible for the State’s online testing. We found that the testing summary reports are focused primarily on the number of tests started and completed during a given day and do not identify or categorize network outages or delays. The outage report we reviewed demonstrates the percentage of time per month that individual network sites are operating, but this report does not identify the date, duration, frequency, or cause of any outages affecting network availability. Although the current director provided us with the outage report for March 2017, he was unable to provide these reports from previous periods to demonstrate Education’s review. The former director stated that ICOE is not required to report—and EDC 11800 does not allow Education to compel ICOE to report—any specific performance measures or to use any specific format or frequency, and the current director agrees with this position. We disagree with Education’s position that it is not allowed to require ICOE to report on specific performance measures. State law directs Education to measure the success of the program, which is not possible without collecting performance data. In addition, Education’s original grant award notice, which was signed and accepted by both Education and ICOE, identifies ICOE’s responsibilities related to performance measurement as the lead agency for K12HSN. This grant agreement was fairly robust and required ICOE, as a condition of receipt of the grant funds, to cooperate with Education’s evaluation of the program and comply with Education’s reporting requirements, including providing all requested documentation to Education in a timely manner. However, subsequent grant awards do not include this language; instead, they state in general terms that ICOE’s responsibilities include technical oversight of the project, and that funds from the grant award are to be used to facilitate collaboration among existing projects and services. We disagree that Education is not allowed to require ICOE to report on specific performance measures. ICOE can use its relationship with CENIC to help it provide the types of information that will assist Education to measure the success of K12HSN as EDC 11800 requires. Currently, ICOE’s contract with CENIC does not require CENIC to report on any specific performance measures. Because CENIC’s own service CALIFORNIA STATE AUDITOR | Report 2016-129 35 May 2017 expectations document states that it provides periodic reports to its board of directors on backbone outages, latency, and packet loss, we expect that it could report this same information to ICOE. If CENIC routinely shared performance reliability measures, ICOE could then report this information to users, Education, and other stakeholders to help demonstrate the program’s quality and effectiveness. ICOE Has Reported Other Measures Inconsistently In contrast to the necessary measures of network performance and reliability that ICOE has omitted from its reporting, ICOE has demonstrated the ability to report on other measures and services required by EDC 11800. However, it has not shared this information on a consistent basis in its annual reports or in its quarterly presentations to the advisory board. As required by EDC 11800, K12HSN’s advisory board issued a report in 2007 that identified recommendations for measuring the success of the network. Despite those recommendations, ICOE has not consistently collected and reported on some of the metrics this report recommended. For example, the report recommended quantifying the number of videoconferencing services that K12HSN provided since the inception of the program as well as establishing a baseline against which ICOE could then monitor the goal of increasing videoconferencing use each year. Nonetheless, ICOE has not reported these statistics consistently. ICOE informed us that Education has not asked for this information. In May 2014, ICOE presented a report to the advisory board that contained statistics on the number and types of videoconferences K12HSN supported with its video application over the preceding eight years. ICOE did not include or update these statistics in its 2015 annual report, and we identified that it presented only one subsequent update to its advisory board in August 2016. Further, we noted inconsistencies when comparing some of the statistics reported in this update with statistics previously provided in May 2014. In response to our questions about these discrepancies, the K12HSN program’s chief operating officer informed us that ICOE changed its process for compiling this information, resulting in a different presentation of the statistics in 2016. When we asked ICOE about its inconsistent reporting on videoconferences, the program’s chief operating officer confirmed that this was a failure on ICOE’s part, but she said that ICOE would have reported these data had Education indicated that such information would be helpful. She indicated that ICOE is able to report on several aspects of videoconferencing use and provided us with some recently compiled statistics, which ICOE plans to include in its 2016 annual report. 36 Report 2016-129 | CALIFORNIA STATE AUDITOR May 2017 Recommendations Education To increase transparency in the K12HSN program and help ensure that the State has sufficient information to measure the program’s effectiveness, Education should direct ICOE to report annually on specific performance measures. These performance measures should include the following metrics: • Cost per unit of capacity used. • Network bandwidth. • Frequency, duration, cause, and location of network outages or interruptions. • Latency and packet loss on network circuits. Education should stipulate that the receipt of grant funds is conditional based on the recipient’s agreement to provide these measures and other information deemed necessary by Education, either on request or at regular intervals determined by Education. If Education believes that it does not currently have legal authority to direct ICOE to report on this information, it should seek legislative change to obtain that authority. ICOE To better support future reporting efforts for the K12HSN program, ICOE should amend its contract with CENIC to require CENIC to report on specific network performance measures, including the frequency, cause, location, and duration of network outages or interruptions. CALIFORNIA STATE AUDITOR | Report 2016-129 37 May 2017 OTHER AREAS WE REVIEWED To address the audit objectives approved by the Joint Legislative Audit Committee (Audit Committee), we reviewed the subject areas shown in Table 4. The table indicates the results of our review and any associated recommendations we made that are not already discussed in the other sections of this report. Table 4 Other Areas Reviewed as Part of This Audit CENIC’s Agreements With Service Providers We reviewed 16 service-level agreements, one from each service provider currently serving one or more network sites. All of the agreements contain specific protections for CENIC in the event of a service outage or interruption. The agreements generally require CENIC to raise issues with the service provider before the provider will consider crediting CENIC’s account. CENIC’s chief financial officer informed us that service providers have issued credits to K12HSN and provided two examples from fiscal year 2016–17 of credits approximating $1,600 and $1,800. However, ICOE was unable to provide us with the frequency with which it received credits because of service outages or interruptions and had to seek this information from CENIC. Recommendation To ensure that the K12HSN program receives all of the service credits to which it is entitled, ICOE should amend its contract with CENIC to clarify CENIC’s responsibilities in this area, including reporting to ICOE about network outages or interruptions and requests for credits to service providers, along with the outcomes of those requests. E‑Rate Deposits We reviewed 15 cash receipts pertaining to nearly $9 million of E-Rate subsidies received for the period from fiscal years 2013–14 through 2015–16. CENIC deposited these amounts—received from various service providers—in the bank account it holds for K12HSN. For each receipt, we verified that the amount of the check matched the amount CENIC deposited to the bank account. In addition, we determined that CENIC deposited each check promptly after indicating that it had received payment. continued on next page . . . 38 Report 2016-129 | CALIFORNIA STATE AUDITOR May 2017 K12HSN Staffing and Administrative Costs • Because ICOE’s role as the lead educational agency for the K12HSN program is to provide administrative services and project oversight and not to provide or maintain Internet access— responsibilities that are assigned to the service providers and CENIC—we considered the entire amount ICOE spends on K12HSN’s operations and personnel as administrative costs. We summarize the major categories of these expenditures in Table 1 on page 8. • Administrative costs, primarily staff compensation, have been relatively stable. K12HSN has maintained consistent staffing levels, with minimal fluctuation in overall salary costs during fiscal years 2013–14 through 2015–16. We calculated that the K12HSN program’s total staffing increased only minimally during our review period, from 13.7 full time equivalent (FTE) employees at ICOE during fiscal year 2013–14 to 14.4 FTEs during fiscal year 2015–16. Total salary and other monetary compensation for those employees equaled $1.05 million and $1.07 million in fiscal years 2013–14 and 2015–16, respectively. • Because K12HSN staff members are ICOE employees, we compared their compensation against ICOE’s salary schedules and found that program salaries were all within the appropriate salary ranges. We noted that most management employees are compensated at the higher steps of their pay ranges, whereas classified employees are predominantly compensated at the lower steps of their pay ranges. Therefore, we also reviewed K12HSN employees’ salary histories along with salary schedules and pay policies to ensure that ICOE complied with its policies in determining salary levels. We found that ICOE complied with its policies and compensated employees at the appropriate ranges and steps from fiscal years 2013–14 through 2015–16. Further, our review identified that several of the management employees had longer tenures with ICOE, meaning that they are at or near the top of their respective salary schedules. In addition, the management salary schedule has fewer steps, resulting in employees’ reaching the upper ranges in a shorter amount of time. ICOE’s Memorandum of Understanding With Butte County Office of Education • ICOE also has a memorandum of understanding with the Butte County Office of Education (Butte) for the services of an E-Rate administrator, who works in collaboration with ICOE and Education to provide E-Rate and Teleconnect Fund expertise and training to K12HSN, county offices of education, and school districts throughout the State. The scope of services originally involved multiple individuals performing Butte’s functions, but has since been reduced to the current individual E-Rate administrator position. • For fiscal year 2016–17, in addition to the budgeted amounts of $150,000 in salary and benefits and $32,534 for direct expenditures such as materials and supplies, travel, conferences, and dues and memberships, the memorandum of understanding requires ICOE to reimburse Butte for $23,551 in other overhead and indirect costs. However, we noted that the E-Rate administrator does not appear to rely on Butte’s office resources in performing her duties, as her work involves a combination of telecommuting and traveling throughout the State. Therefore, we question the appropriateness of ICOE paying some of these overhead and indirect costs when it could consider adding this position to its complement of existing staff and contractors, resulting in a lower total cost to the State. Recommendation To ensure efficient use of state funds, ICOE should conduct a cost and benefit analysis of its memorandum of understanding with Butte to determine whether it represents the most cost‑effective approach to providing the program activities it covers. CALIFORNIA STATE AUDITOR | Report 2016-129 39 May 2017 Ongoing Use of ICOE • The current and former directors of the Educational Data Management Division at Education confirmed that Education has never rebid the K12HSN grant since awarding it to ICOE in 2004, nor has it sought proposals from other local educational agencies. The former director informed us that during her tenure she considered reopening the bid, but very few counties expressed interest. In addition, the directors and their staff explained that the risk of interruptions to the network that would result from changing local agencies outweigh any potential benefit, and they questioned the need to make any changes, in large part because they believe ICOE has been running the program effectively. • Nevertheless, Education cannot justify its position because of the lack of consistent reporting on the performance of the network, which we discuss in the section starting on page 31. Our recommendation related to improved performance monitoring for the K12HSN program will assist Education in assessing the potential risks and benefits of rebidding the contract. BIIG Funding • Our review of legislatively mandated reports on the planning and progress of the BIIG implementation leads us to conclude that ICOE’s process for administering the grants and its methodology for identifying grant recipients has generally been consistent with the requirements the Legislature implemented as part of the BIIG program’s funding. • According to the December 2016 connectivity report prepared by ICOE, 182 sites had been funded with BIIG 1.0 funds, with about $19 million of the $26.7 million in total funds spent. In February 2017, the program’s chief executive officer reported to the advisory board that a total of 363 active projects pertained to BIIG 1.0 and 2.0. He reported that approximately $21 million in BIIG 1.0 funds and about $6.9 million of the total $50 million in BIIG 2.0 funds had been spent thus far. 40 Report 2016-129 | CALIFORNIA STATE AUDITOR May 2017 Blank page inserted for reproduction purposes only. CALIFORNIA STATE AUDITOR | Report 2016-129 41 May 2017 SCOPE AND METHODOLOGY The Audit Committee directed the California State Auditor to conduct an audit of ICOE’s operation of the K12HSN program and of the funding for the program. The audit analysis that the Audit Committee approved contains 11 objectives. Table 5 lists the objectives and the methods we used to address them. Table 5 Audit Objectives and the Methods Used to Address Them AUDIT OBJECTIVE METHOD 1 Review and evaluate the laws, rules, Reviewed relevant laws, rules, regulations, and procedures. and regulations significant to the audit objectives. 2 Identify and evaluate the roles and • Identified and evaluated the roles and responsibilities of Education, ICOE, CENIC, and the K12HSN responsibilities of Education, ICOE, advisory board as prescribed in the Education Code. We also reviewed roles and responsibilities CENIC, the K12HSN advisory board, defined in contracts and other relevant agreements between ICOE and third parties. and any other state or local agencies • Reviewed the meeting frequency and attendance of the K12HSN advisory board over the past or contractors that may be involved in five years. managing or administering K12HSN. • Determined that the K12HSN advisory board’s meeting frequency over the past five years complied with state law but that meetings consistently included fewer than the advisory board’s 12 members. The K12HSN program’s chief operating officer told us that the district superintendents on the advisory board often have other commitments that prevent their attending advisory board meetings. 3 Determine what services K12HSN • Reviewed ICOE’s master contract with CENIC, as well as all amendments and addenda to that contracts for with CENIC or others. contract. We also determined whether ICOE’s contracts with CENIC contained any specific Ascertain what performance performance measures for contracted services CENIC provides to ICOE. measures Education and ICOE have • Interviewed staff at ICOE and Education about the performance measures they review to ensure developed for ensuring that K12HSN’s that K12HSN’s benefits are maximized through the agreement with CENIC. Although ICOE has benefits are maximized, including reported on certain measures of network performance and has broadly referenced the financial whether contracting with CENIC benefit of contracting with CENIC, it has not quantified the benefits or measured them against represents the most cost-effective costs of the contract. option for the State. • Reviewed a selection of CENIC’s service-level agreements with individual Internet service providers. We also identified whether those agreements contain protections in the event of service outages or interruptions. • Determined that the provisions in the contract satisfy requirements in state law with respect to intellectual property rights and interest earned on state funds. 4 Assess how ICOE ensures that Internet • For a judgmental selection of 20 upgrade projects from fiscal years 2012–13 through 2015–16, subsidies are maximized and fully and reviewed E-Rate application information to ensure that CENIC claimed E-Rate discounts for properly credited to K12HSN. the projects. • Reviewed the projects’ application forms and determined whether the applications complied with USAC’s deadlines. • For a selection of 26 school districts served by the upgraded circuits, confirmed that ICOE complied with E-Rate subsidy requirements to obtain authorization from the school districts included in ICOE’s E-Rate applications for those projects. • For each of the past three fiscal years, reviewed a judgmental selection of five E-Rate deposits to ensure that funds were properly credited to the K12HSN program. continued on next page . . . 42 Report 2016-129 | CALIFORNIA STATE AUDITOR May 2017 AUDIT OBJECTIVE METHOD 5 Identify all of K12HSN’s funding sources for the most recent three fiscal years and do the following: a Determine K12HSN’s major • Obtained and reviewed K12HSN’s audited financial statements and accounting data for fiscal categories of expenditures and years 2013–14 through 2015–16. In addition, we determined categories of revenue and evaluate whether its spending expenditures for each of the past three fiscal years and identified any trends. practices are reasonable, • Calculated and compared the proportion of E-Rate and Teleconnect Fund subsidies advanced to cost-effective, and consistent K12HSN for fiscal years 2013–14, 2014–15, and 2015–16. with state policies. • For a judgmental selection of 20 network circuit upgrades and 10 additional upgrades to related circuits ICOE approved for fiscal years 2015–16 through 2017–18, reviewed and determined whether the selected service provider and selected level of service or equipment were the most cost-effective options available. • Interviewed ICOE staff regarding rationales for upgrade selection decisions. b Assess whether ICOE’s From our review of expenditures, identified any major administrative costs besides staffing costs. We administrative costs for K12HSN also determined whether the activities underlying those costs were reasonable and necessary for are reasonable and in accordance K12HSN’s mission. with its grant agreement. 6 Review ICOE’s and Education’s • Identified the Government Finance Officers Association’s Recommended Budget Practices: A budgetary planning practices Framework for Improved State and Local Government Budgeting as a source for best practices. for K12HSN, including how they • Reviewed ICOE’s budget documents for fiscal years 2013–14 through 2016–17 and proposed determine the need for capital budget documents for fiscal year 2017–18. Further, we reviewed both internal planning investments and operating documents and the budget documents that were submitted to the State. We determined whether reserves. Assess whether the these documents and ICOE’s process for creating them conformed to best practices. entities’ budgetary planning • Interviewed relevant individuals to determine the State’s processes for reviewing ICOE’s budget activities follow best practices submissions and for setting state funding levels for the program. and whether the entities use consistent methodologies. • Identified ICOE’s methodology for identifying network circuits that require capacity upgrades and determined the reasonableness of that methodology. • Determined whether ICOE has developed a long-term plan for the network structure and whether that plan is reasonable. • For the selection of 20 upgraded network circuits and 10 other upgraded circuits connected to them reviewed under Objective 5, documented available maximum usage data for those circuits. We estimated the new costs these upgrades will incur for the K12HSN program as compared to the costs of less expensive alternatives ICOE did not pursue. We also determined whether network usage at those circuits warranted the upgrades ICOE selected. • Interviewed ICOE’s staff to determine their reasons for approving upgrades. • Reviewed the capacity of the surrounding network sites at the time of the upgrade and identified how they were affected by the upgrade. • Reviewed K12HSN advisory board meeting minutes and determined the extent to which ICOE reviewed the upgrade with the K12HSN advisory board, as well as the timing of any such review and the board’s input on the upgrade. We concluded that any discussion of network upgrades with the advisory board occurs at a general level and does not include specific costs of and justifications for those upgrades. • Identified any network circuits with maximum traffic levels substantially higher than the 50 percent threshold ICOE set for pursuing capacity upgrades during ICOE’s project planning stages for fiscal year 2016–17 and determined that ICOE has consistently taken steps toward upgrading those circuits. CALIFORNIA STATE AUDITOR | Report 2016-129 43 May 2017 AUDIT OBJECTIVE METHOD 7 Determine K12HSN’s operating Reviewed the program’s audited financial statements to identify operating reserve levels for the reserves for the most recent three K12HSN program for the period from fiscal years 2010–11 through 2015–16. fiscal years. To the extent possible, do the following: a Calculate K12HSN’s operating • Using those financial statements and ICOE’s accounting data for the K12HSN program, as well as reserve at the end of fiscal its projected expenditures for fiscal year 2016–17, estimated the program’s operating reserve as year 2016–17. of June 30, 2017. Using that information and ICOE’s projected expenditures for fiscal year 2017–18, we estimated the program’s operating reserve level as of June 30, 2018, under different funding scenarios. • Reviewed CENIC’s records to identify the timing of E-Rate subsidy reimbursements from service providers during the past three fiscal years. We also determined when K12HSN received those funds. b Determine what a prudent annual • Interviewed staff and documented ICOE’s methodology or rationale for maintaining its level of operating reserve should be. operating reserves. • Interviewed responsible parties at Education to obtain their perspective about what they believe a prudent reserve should be. • Using the above procedures, determined whether ICOE’s proposed reserves for K12HSN are necessary and prudent. 8 Determine the core duties of K12HSN staff. Further, determine K12HSN staffing levels and costs for the most recent three fiscal years and perform the following: a Ascertain whether staffing Reviewed K12HSN personnel and payroll information for the past three fiscal years and determined levels have increased and the staffing levels and compensation for each year. We also determined the reasons for any increases or reasons why. decreases in compensation and documented any changes in staffing levels. b Review staffing plans and assess Reviewed current budgetary staffing documents and job descriptions documenting the whether these plans are adequate. responsibilities of K12HSN staff for the past three fiscal years. c Determine how staff • Compared actual compensation for K12HSN staff to ICOE’s salary schedules for the past compensation rates are set three fiscal years. and whether those rates • Determined ICOE’s methodology for setting compensation levels, including setting compensation are commensurate with for ICOE positions unique to the K12HSN program. Further, we determined whether ICOE followed their responsibilities. those policies when setting compensation for K12HSN employees. 9 Review the grant agreement Determined from interviews with ICOE and Education staff that no overarching grant agreement or between Education and ICOE and specific program requirements exist outside of the Education Code. do the following: a Determine what process Obtained grant award letters issued to ICOE and interviewed staff at Education about its selection Education follows in awarding process and whether it has considered other potential grantees since first awarding the grant to ICOE. the K12HSN grant. We then evaluated the reasonableness of Education’s decision. b Establish how Education Interviewed staff at Education to determine the type of information it reviewed each year for fiscal calculates the annual grant award years 2013–14 through 2015–16 before providing the grant and whether that information included and whether that calculation consideration of ICOE’s operating reserve. takes into consideration any operating reserves. c Ascertain whether Education has • Obtained and reviewed K12HSN’s annual reports, advisory board meeting minutes and measured ICOE’s performance presentation documents, and program updates to determine the extent to which ICOE against the grant agreement. reported to Education and the K12HSN advisory board on performance measures outlined in Determine whether opportunities the Education Code. exist for Education to improve • Determined whether more specific guidance from Education is needed to improve the consistency its monitoring. and detail of reporting on the K12HSN program’s performance and cost-effectiveness. continued on next page . . . 44 Report 2016-129 | CALIFORNIA STATE AUDITOR May 2017 AUDIT OBJECTIVE METHOD 10 Determine how ICOE and/ • Interviewed staff at ICOE and Education to determine what service metrics they monitor with or Education monitor CENIC’s respect to CENIC’s performance in managing K12HSN’s physical network. performance and whether • Determined whether ICOE’s contract with CENIC should be amended to require reporting on these entities are providing all specific measures related to the performance of the network, such as the network’s reliability. necessary oversight. 11 Review and assess any other issues • Reviewed relevant state laws that established the funding for BIIG 1.0 and 2.0 to identify roles and that are significant to the audit. responsibilities for administering the grants. • Interviewed ICOE staff regarding ICOE’s methodology for awarding the grant funds to California public schools and school districts. • Obtained and reviewed legislatively mandated reports related to these grants. We determined that ICOE’s reported process for administering the grants and its methodology for identifying recipients were in line with funding requirements. Sources: California State Auditor’s analysis of the Audit Committee’s audit request number 2016-129 and analysis of information and documentation identified in the table column titled Method. CALIFORNIA STATE AUDITOR | Report 2016-129 45 May 2017 Assessment of Data Reliability In performing this audit, we relied on various electronic data files that we obtained from the entities listed in Table 6. The U.S. Government Accountability Office, whose standards we are statutorily required to follow, requires us to assess the sufficiency and appropriateness of computer‑processed information that we use to support our findings, conclusions, or recommendations. Table 6 describes the analyses we conducted using data from these systems, our methodology for testing them, and the limitations we identified in the data. Although we recognize that these limitations may affect the precision of the numbers we present, in total there is sufficient evidence to support our audit findings, conclusions, and recommendations. Table 6 Methods Used to Assess Data Reliability INFORMATION SYSTEM PURPOSE METHOD AND RESULT CONCLUSION ICOE’s Orion Software To determine the usage We performed reliability testing by comparing usage We found the Orion data to rates for the K12HSN outputs from the Orion software to an independent be sufficiently reliable for program’s circuits monitoring system used by CENIC known as Cricket. We these purposes. monitored by Orion as compared equivalent time frames for the same parts of the part of audit procedures network and found that the two systems were consistent examining ICOE’s network with one another. upgrade decisions. ICOE’s Imperial County To identify and calculate We reconciled major categories of revenue and We found the ICSIS data to Schools Information major sources of revenue expenditures from ICSIS with ICOE’s external audited be sufficiently reliable for System (ICSIS) and expenditures for financial statements for fiscal years 2014–15 and 2015–16, these purposes. the K12HSN program which include a separate opinion on compliance for fiscal years 2013–14 requirements for the K12HSN program. through 2015–16. The fiscal year 2013–14 ICOE audited financial statements do not include specific information for K12HSN. As a result, we interviewed the external auditor to determine the extent and detail of audit testing performed for K12HSN in that year, as well as the subsequent two fiscal years, and determined that level of testing supports the detail with which we present the ICSIS information in our report. To determine the value Fiscal year 2015–16 was the first year ICSIS tracked the Undetermined reliability. of Internet subsidies the accrued E-Rate and Teleconnect Fund subsidies specifically K12HSN program accrued for the K12HSN program. K12HSN’s audited financial Although this determination in fiscal year 2015–16 in statements for that fiscal year indicate that the subsidy may affect the precision of order to project future amounts were unaudited. Because the K12HSN program has the numbers we present, program revenue. not yet fully received the reimbursements associated with there is sufficient evidence the accrued subsidy amounts, we cannot determine the to support our audit preciseness of those amounts. findings, conclusions, and recommendations. Sources: California State Auditor’s analysis of various documents, interviews, and data from ICOE. 46 Report 2016-129 | CALIFORNIA STATE AUDITOR May 2017 We conducted this audit under the authority vested in the California State Auditor by Section 8543 et seq. of the California Government Code and according to generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives specified in the Scope and Methodology section of the report. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Respectfully submitted, ELAINE M. HOWLE, CPA State Auditor Date: May 25, 2017 Staff: Linus A. Li, CPA, CMA, Audit Principal Mark Reinardy, MPP Ryan T. Canady Sean D. McCobb, MBA Bridget Peri, MBA Karen Wells Legal Counsel: Stephanie Ramirez‑Ridgeway, Assistant Chief Counsel CALIFORNIA STATE AUDITOR | Report 2016-129 47 May 2017 * 1 * California State Auditor’s comments begin on page 53. 48 Report 2016-129 | CALIFORNIA STATE AUDITOR May 2017 ICOE Management’s Responses Recommendation #1: • To better inform decision makers at the state level about the amount of funding necessary to operate and maintain the network, by November 2017 ICOE should formally amend its annual budget documents to specify multiple potential levels of network expenditures for the coming year, and it should detail the specific network upgrades and project costs included in each scenario. As part of this process, ICOE should also provide information about how these upgrades will affect the network’s functionality. Response: • ICOE acknowledges and agrees with this recommendation and has taken steps and will take further steps to revise the planning and procurement processes to meet the November timeline. This will afford ICOE the opportunity of reporting its budget year requirements at an acceptably early point in time to better align with the State’s budgeting process. ICOE has plans to work with CENIC to open the procurement process in the June time frame. This will result in the ability to consider bids and make decisions on the 2018-19 network changes in early fall. • In addition, annual budget documents developed for presentation to CDE staff members and K12HSN Advisory Board members will include multiple iterations of possible network changes to meet node site demand and include their respective costs. This will enable our liaisons at CDE and our board members to provide guidance on these decisions. Recommendation #2: • To ensure its projected program costs are as accurate as possible, by November 2017 ICOE should institute a formal practice for reviewing its budget planning document against its current network design plans and correct any inaccuracies before finalizing and submitting its budget. Response: • ICOE agrees with this recommendation and has taken steps to ensure that a final review is conducted before submitting budget documents to CDE or other State agencies. ICOE will implement a new process at the end of the new bidding cycle as it prepares budget proposals 2 for the State in the fall. In the 2015-16 school year accounting guidelines changed with regard to how discounts provided by the state and federal telecommunications subsidy programs are to be reported. These changes have been implemented by ICOE’s accounting team. One result of these changes is that year-on-year comparisons require footnoting and explanations as to why both expenditures and revenues are inflated by sums that were formerly not reported since they were taken immediately off invoice as a realized discount. CALIFORNIA STATE AUDITOR | Report 2016-129 49 May 2017 Recommendation #3: • To help ensure that the K12HSN program maintains the necessary amount of state funds in reserve, ICOE should prepare a formal methodology for a proposed equipment reserve that is based on the actual likelihood of equipment failure and the costs associated with replacing that equipment. Response: • ICOE agrees with the recommendation and will work with the K12HSN Network Implementation Committee and CENIC to develop a formal methodology for determining an adequate equipment reserve that takes into consideration the current fleet of equipment, useful service life, and the likelihood that the equipment may fail prematurely. These steps will also include monitoring end-of-life announcements from equipment providers and the development of an asset inventory system that permits K12HSN to easily ascertain which node site equipment has not been replaced due to circuit upgrades and therefore needs replacement only when it is end-of-life. Recommendation #4: • ICOE should establish procedures to routinely monitor the K12HSN program’s cash balance and evaluate upcoming costs. If at any point ICOE determines that it will be unable to fund costs due to delayed subsidy payment, it should notify Education regarding the size and timing of the anticipated shortfall and postpone significant discretionary expenditures, such as upgrading network equipment at sites, until ICOE collects the subsidies it is owed. Response: • ICOE agrees with the recommendation and notes that while previously there was less risk of deficit spending due to the reserve available, monitoring its cash position and planning expenditures in order to remain solvent is already part of the process for the internal accounting team. Nevertheless, ICOE will incorporate a process for reviewing the program’s cash position in alignment with existing ICOE fiscal procedures. • In addition, ICOE agrees to follow the recommendation that it keep CDE fully apprised of any anticipated revenue shortfalls and plans to mitigate the risk. Recommendation #5: • To better guarantee that network upgrades are necessary and are achieved at the lowest possible cost to the State, ICOE should develop a formal methodology for reviewing circuit capacity needs. This methodology should include consideration of multi-year trends in network traffic and the implications prospective upgrades may have for other parts of the network. Response: • ICOE agrees with the recommendation and plans to engage with its Network Implementation Committee to develop a tool and/or template to formally collect information on current usage and anticipated usage over the next 6, 12 and 18 month periods. The plan is to work with 3 node sites to conduct data collection from the districts that they currently serve. These efforts may also include collecting information on district-level initiatives and other drivers of bandwidth usage and a link to the URL at which usage may be monitored, if any exists. • In addition, ICOE will continue to use its monitoring tools to look at historical perspectives that can provide insight on growth trends and compile this information into public reports. 50 Report 2016-129 | CALIFORNIA STATE AUDITOR May 2017 Recommendation #6: • To provide as many options for network upgrades as possible and to help ICOE provide the most cost-effective upgrade options without risking its eligibility for subsidies, ICOE should adopt the practice of requesting bids at all feasible levels of capacity upgrades as opposed to only those levels that represent a tenfold increase in circuit capacity. Response: • ICOE agrees with the recommendation and will seek bids at different capacities within the acceptable range of CENIC’s and node site technical parameters. These factors may include but not limited to equipment specifications, rack space, power and environmental limitations. In addition, ICOE will continue its practice of increasing capacity ten-fold in cases in those cases for which it yields additional savings to the State. Recommendation #7: • To help facilitate the review of circuit capacity needs, ICOE should maintain historical data for the K12HSN traffic as long as is technically feasible. It should also ensure that its monitoring software includes all K12HSN sites. Response: • While ICOE agrees with the recommendation, it is our position that past network 4 performance data points beyond a two-year period do not contribute meaningful data to the decision-making process for future capacity increases. Nevertheless, keeping historical performance data beyond the two-year period is not difficult and it will be factored into the decision-making process. • ICOE will revisit the process for updating monitoring tools upon notification from CENIC that equipment has been installed or replaced. Recommendation #8: • To reduce the risk of having to react to large increases in network traffic, ICOE should formalize a process to include input from network site administrators during network upgrade planning. Response: • ICOE agrees with these recommendations. They align well with the process ICOE has used in previous years to gather input from node site administrators. While ICOE already has a process to meet with node site administrators on an annual basis, this work can be improved with a formal data collection template that allows ICOE, along with node site administrators and CENIC to develop recommended changes to the network in order to better support school districts in their respective service areas. Recommendation #9: • To better support future reporting efforts for the K12HSN program, ICOE should amend its contract with CENIC to require CENIC to report on specific network performance measures, including the frequency, cause, location and duration of the network outages or interruptions. CALIFORNIA STATE AUDITOR | Report 2016-129 51 May 2017 Response: • ICOE agrees with the recommendation and plans to work with CENIC to develop a mechanism to enhance the reporting of network performance and reliability. ICOE will begin using its own monitoring system to compile outage reports and utilize those as a cross- reference against new reports to be provided by CENIC. When service credits are earned as a result of lengthy outages, ICOE will track the CENIC process of recovering those credits and ensure their receipt. The contract between ICOE and CENIC will be amended as needed to provide for these additional functions. Other Recommendations #1: • To ensure that the K12HSN program receives all of the service credits to which it is entitled, ICOE should amend its contract with CENIC to clarify CENIC’s responsibilities in this area, including reporting to ICOE about network outages or interruptions and requests for credits to service providers, along with the outcomes of those requests. Response: • ICOE agrees with the recommendations and will work with CENIC to amend the existing agreement and develop a procedure for reporting network outages and include any potential service credits from service providers. Other Recommendations #2: • ICOE should conduct a cost and benefit analysis of its memorandum of understanding with Butte to determine whether it represents the most cost-effective approach to providing the program activities it covers. Response: • The relationship between ICOE and Butte County Office of Education (BCOE) dates back to when we jointly drafted the response to the Request for Proposal staged by CDE in summer of 2004. Their role has been instrumental in the delivery of services for the program and advising CDE on the federal and state broadband subsidy programs as well as special projects such as Broadband grants and the professional development program. Over time there have been many changes to the role of BCOE in the program, both expansions and contractions of services dedicated to supporting California school districts in their efforts to secure a maximum level of federal support for their telecommunications services. Recently, two shifts simultaneously adjusted the workload again. One factor was that the CDE determined it would assume a greater role in Erate conversations in Washington DC and California Teleconnect Fund conversations in San Francisco, reducing BCOE’s workload. In addition, the Broadband Infrastructure Improvement Grants created a new need for K12HSN to support awardee school districts in transitioning contracts from CENIC’s Erate application to the individual district’s Erate application, increasing BCOE’s workload. In addition, BCOE is assisting ICOE in the development of procurement best practices to complement the Technical Assistance and Professional Development (TAPD) program. • ICOE agrees that a cost-benefit analysis for the existing agreement with BCOE as it pertains to statewide E-rate support will provide valuable guidance in making any prudent and reasonable adjustments to the annual agreement. 52 Report 2016-129 | CALIFORNIA STATE AUDITOR May 2017 Blank page inserted for reproduction purposes only. CALIFORNIA STATE AUDITOR | Report 2016-129 53 May 2017 Comments CALIFORNIA STATE AUDITOR’S COMMENTS ON THE RESPONSE FROM THE IMPERIAL COUNTY OFFICE OF EDUCATION To provide clarity and perspective, we are commenting on the response to our audit from ICOE. The numbers below correspond to the numbers we have placed in the margin of ICOE’s response. Although ICOE states that it has managed to reduce operational 1 costs, financial information from its general ledger shows that costs pertaining to K12HSN program operations have increased in recent years. As we illustrate in Table 1 on page 8, ICOE’s costs for activities other than the contracted services provided by CENIC increased from $2.2 million in fiscal year 2013–14 to $3.7 million in 2015–16. Over the same period, the cost of services provided by CENIC also increased from $8.8 million to $9.5 million. ICOE’s explanation regarding recent changes to accounting 2 guidelines is not relevant to this issue. Our audit findings related to ICOE’s budgeting process, which we discuss in the section beginning on page 11, pertain to errors and inaccuracies in the projected expenditures ICOE planned to submit to the State as support for ICOE’s requested level of funding for fiscal year 2017–18. These findings are unrelated to the accounting change pertaining to the way ICOE reports the value of the K12HSN program’s Internet subsidies at the close of each fiscal year. Although ICOE plans to develop tools to improve its ability 3 to project future usage for network circuits, we encourage it to establish goals for its projections beyond the 18‑month timeframe referenced in its response. Contracts pertaining to network circuit upgrades frequently encompass three or more years, so projecting usage as far into the future as possible would benefit the K12HSN program. We are confused by ICOE’s assertion that more than two years 4 of historical usage data is not meaningful for decisions about capacity increases. Our review indicates that ICOE has not attempted to store or utilize such data beyond two years, so it is unclear how ICOE could determine that this data would not add value to projecting future usage. As we explain on page 24, retaining historical data will allow ICOE to compare rates of growth in usage before and after it last upgraded a circuit in order to determine whether growth rates have been constant or are accelerating. We also state on page 24 that retaining historical data pertaining to the circuits serving high‑use network sites could help ICOE predict how circuits serving sites that currently have 54 Report 2016-129 | CALIFORNIA STATE AUDITOR May 2017 lower capacity needs may ultimately behave. Given the significant cost differences we identified among various levels of capacity increases, as shown in Table 3 on page 21, and the value of being able to project future usage for multiple years, retaining additional historical data and using that data when making upgrade decisions will help ensure the network is able to provide reliable services at the lowest cost to the State. CALIFORNIA STATE AUDITOR | Report 2016-129 55 May 2017 56 Report 2016-129 | CALIFORNIA STATE AUDITOR May 2017