Summary

Enterprise Licensing Agreement: The State Failed to Exercise Due Diligence When Contracting With Oracle, Potentially Costing Taxpayers Millions of Dollars April 2002 2001-128 rotiduA etatS ainrofilaC S T I D U A E T A T S F O U A E R U B The first five copies of each California State Auditor report are free. Additional copies are $3 each, payable by check or money order. You can obtain reports by contacting the Bureau of State Audits at the following address: California State Auditor Bureau of State Audits 555 Capitol Mall, Suite 300 Sacramento, California 95814 (916) 445-0255 or TDD (916) 445-0255 x 216 OR This report may also be available on the World Wide Web http://www.bsa.ca.gov/bsa/ The California State Auditor is pleased to announce the availability of an online subscription service. For information on how to subscribe, please contact David Madrigal at (916) 445-0255, ext. 201, or visit our Web site at www.bsa.ca.gov/bsa. Alternate format reports available upon request. Permission is granted to reproduce reports. � � � ��������� ���� ������ ������������� ������������������� ������������ ����������������������� April 16, 2002 2001-128 The Governor of California President pro Tempore of the Senate Speaker of the Assembly State Capitol Sacramento, California 95814 Dear Governor and Legislative Leaders: As requested by the Joint Legislative Audit Committee, the Bureau of State Audits presents its audit report concerning our review of the State’s contracting practices in entering into an enterprise licensing agreement (ELA) with the Oracle Corporation (Oracle). This report concludes that the State executed the ELA contract worth almost $95 million despite evidence suggesting the need for Oracle database licenses was limited. Additionally, the departments of Information Technology, General Services, and Finance approved the ELA without validating the cost savings projections presented by Logicon Inc. (Logicon), who helped the State negotiate the ELA with Oracle. Our analysis of Logicon’s data indicates that rather than saving $111 million by entering into the ELA, as purported by Logicon, the State could spend from $6 million to $41 million more on Oracle database licenses and maintenance than it would if there was no contract. Furthermore, it appears Logicon stands to make more than $28 million from the ELA, a fact the State may not have been apprised of. We also found that as of March 20, 2002, nearly 10 months after the ELA was approved, no state departments have acquired new licenses under the ELA. Moreover, by June 2002, when the Department of Finance expects to complete the method for charging the ELA’s costs to departments, the State will have accumulated more than $17 million in ELA costs and interest charges. Furthermore, the Department of General Services used an inexperienced negotiating team and it limited the involvement of legal counsel in the ELA contract and, as a result, many contract terms and conditions necessary to protect the State are vague or missing altogether. Finally, our legal consultant has advised that a court might conclude that the ELA contract is not enforceable as a valid state contract because it may not fall within an exception to the State’s competitive bidding requirements. Respectfully submitted, ELAINE M. HOWLE State Auditor ������������������� �������������������������������������������������� ���������������������������������������� CONTENTS Summary 1 Introduction 7 Chapter 1 Failing to Assess the Need for Database Licenses and Lacking Guidance From the Appropriate Departments, the State Committed Millions of Taxpayer Dollars for an Enterprise Software License It Has Yet to Use 17 Recommendations 43 Chapter 2 The State’s Inexperienced Negotiating Team Allowed Oracle and Logicon to Obtain a Long-Term Contract With Major Risks for the State 45 Recommendations 67 Appendix A Chronology of Key Events Occurring Before and After the Enterprise Licensing Agreement 69 Appendix B Projected Cost of the Enterprise Licensing Agreement 73 Appendix C The State’s Enterprise Licensing Agreement and Logicon’s Related Side Agreements With Oracle 77 Responses to the Audit Department of Information Technology 81 California State Auditor’s Comments on the Response From the Department of Information Technology 89 State and Consumer Services Agency, Department of General Services 93 California State Auditor’s Comments on the Response From the Department of General Services 101 Department of Finance 103 1 SUMMARY RESULTS IN BRIEF On May 31, 2001, the State entered into a six-year enter- prise licensing agreement (ELA), a contract worth almost $95 million, to authorize up to 270,000 state Audit Highlights . . . employees to use Oracle Enterprise Edition 8i database software (enterprise database licensure) and to provide maintenance sup- On May 31, 2001, the State port. By broadly licensing software, an entity comprised of many entered into a six-year users can potentially achieve significant volume discounts and enterprise licensing agreement (ELA), a contract worth reduce its overall administrative costs. However, a preliminary almost $95 million, to survey by the Department of Information Technology (DOIT) of authorize up to 270,000 127 state government departments two months earlier strongly state employees to use Oracle suggested that relatively few state workers might need or want database software and to provide maintenance support. any new Oracle Corporation (Oracle) products. Although only 21 departments responded to this survey, DOIT made no fur- Our audit of this acquisition ther efforts to assess the State’s need for Oracle software. The revealed the following: other 2 state departments normally charged with oversight of þ By broadly licensing large information technology (IT) projects—the Department software, a buyer that of General Services (General Services) and the Department of has many users, such as the State, can achieve Finance (Finance)—also failed to assess the State’s actual need significant volume for the contract. discounts and reduce its overall administrative costs. Further, DOIT and Finance could have reviewed the ELA þ However, the State proposal1 and perhaps saved the State from making a flawed proceeded with this decision, but neither did so, citing a lack of specific procedures procurement even though and inadequate time. Also, General Services’ unprepared a survey of departments disclosed limited demand negotiating team agreed to a contract that left the State for new Oracle products. unprotected against numerous risks. In its proposal, Logicon Inc. (Logicon), Oracle’s reseller, presented data showing the þ In spite of such limited interest, the Department ELA would save the State millions of dollars over the life of the of Information Technology contract.2 General Services, DOIT, and Finance approved the ELA made no further efforts to without taking the time to validate Logicon’s data, which our assess the State’s need for review shows to be significantly overstated. Lacking an in-depth Oracle software. continued . . . 1The ELA proposal as it is used here and throughout the report consists of the projected costs and savings amounts that Logicon Inc. provided to the State. The proposal includes Logicon’s assumptions and was accompanied as of May 17, 2001, by past state purchase order data for Oracle products and support. 2Logicon Inc. recently changed its name to Northrop Grumman Information Technology. 1 þ The decision to support understanding of whether the ELA might fill a legitimate need the ELA was likely swayed for state departments, and without knowing the true costs by Logicon’s projections and benefits of the contract, the State committed millions of that the State could save taxpayer dollars to a questionable technology purchase. about $111 million. þ The departments In November 2001, five months after the ELA took effect, of General Services, Finance sent out its own survey, this time asking all Information Technology, and Finance approved the departments to identify their needs for the newly purchased ELA without validating enterprise database licensure. Preliminary results of this Logicon’s cost savings survey indicate that many of the State’s largest departments projections; unfortunately need database software licenses for less than a fourth of these projections proved to be significantly their authorized positions. For example, the Department of overstated. Corrections has almost 24,000 custody staff whose jobs likely þ Logicon apparently stands do not require them to have their own computer station or to receive more than database software. Although the State’s actual demand for $28 million as a result the software license is uncertain, the contract could establish of the ELA, a fact the the perception that Oracle Enterprise Edition 8i database State may not have been is the de facto standard throughout state departments in apprised of. spite of a statewide policy that agencies should adopt the þ Nearly 10 months after best technical solution for their particular needs. The sheer the ELA was approved, volume of the ELA purchase may create the perception no state departments had acquired new licenses among departments that the Oracle database is the standard under the ELA, which may software and their most cost-effective choice. Unfortunately, be due to the fact that departments’ perception of a de facto standard may reduce General Services had not innovation and flexibility in state IT projects. issued instructions to state departments on how to do so. Besides not knowing the actual need for statewide enterprise þ By June 2002, when the database licensure, the State entered the ELA without any formal Department of Finance evaluation of the contract’s technical or business advantages. expects to complete the The State had never evaluated an ELA proposal before and method for charging lacked specific procedures to do so. Nonetheless, DOIT and the ELA’s costs to state Finance routinely evaluate IT projects and possess the expertise departments, the State will have accumulated needed to evaluate the ELA proposal—DOIT the need to license more than $17 million 270,000 users and Finance the proposal’s cost projections. How- in ELA costs and ever, neither validated these aspects of the ELA proposal. To its interest charges. credit, Finance’s Technology Investment Review Unit (TIRU), þ General Services used an which is responsible for reviewing IT proposals and ensuring IT inexperienced negotiating expenditures represent a prudent investment of resources and team and limited the meet state needs, raised concerns about the ELA proposal and involvement of legal counsel in the ELA recommended postponing it until the next year. However, its contract, and as a result, concerns and recommendation went unheeded. As a result, the many contract terms and State committed almost $95 million in taxpayers’ money for conditions necessary to protect the State are vague software that could affect IT decisions across all departments or missing altogether. without knowing if the ELA was an appropriate procurement of technology or if its costs and benefits were justified. continued . . . 2 3 þ Finally, our legal Furthermore, although it had been in effect for nearly consultant has advised 10 months, as of March 20, 2002, no state department had us that a court might acquired new licenses under the ELA. This may be due to the conclude that the ELA fact that General Services had not issued instructions to state contract with Oracle is not enforceable as a valid departments on how to do so. In addition, although Finance is state contract because working on a cost allocation model to charge state departments it may not fall within an for licenses they acquire under the ELA, it is not yet complete. exception to the State’s competitive bidding Until it is complete, state departments will not know the exact requirements. cost of acquiring licenses under the ELA. Furthermore, by June 2002, when Finance expects to complete the cost allocation model, the State will have accumulated more than $17 million in ELA costs and interest charges, and will likely have little ben- efit to show for it. By broadly licensing software in an organization, an ELA has potential benefits for both buyer and seller: it can reduce a large consumer’s administrative costs and give the seller a firm commitment to purchase in volume. However, General Services’ negotiating team was inexperienced and unprepared, with no expertise in software contracts and no in-depth knowledge of Oracle’s business and contracting practices. Moreover, General Services limited the involvement of its legal counsel in the ELA contract to a few hours of review just before it was signed, and in general, limits its legal counsel’s role in all IT contracts. Therefore, many of the ELA contract terms and conditions necessary to protect the State’s interests are vague or missing altogether. Also, the six-year term of the contract, with an option for four more years, deviates from the standard industry practice of limiting contracts of this nature to between three and five years because of the rapidly changing technology field. By entering into a long-term contract that lacks legal safeguards, the State faces considerable financial risk over many years. For example, the ELA gives the State no protection against risks such as Oracle’s lowering prices, software upgrades not being included in the purchase price, and a declining need for the licenses. In short, the State had never before negotiated an ELA and let Oracle and its reseller, Logicon, use common vendor negotiating tactics to push through a largely one-sided contract. The decision to support the ELA was likely swayed by Logicon’s estimates that the State would save about $111 million over and above the contract’s cost if it exercised its option for an added four years of maintenance. However, our review of Logicon’s proposal and data indicates that rather than saving money by entering into the ELA, the State stands to spend almost $6 million more on Oracle database licenses and maintenance 2 3 than it would without the contract if it exercises its four- year option, and almost $41 million more if it terminates the contract after its normal six-year term. Although Logicon was responsible for initiating the sales presentations that resulted in the ELA, none of the three departments thoroughly validated the data in Logicon’s proposal, a small effort that might have saved the State millions of taxpayer dollars. Furthermore, it appears that Logicon stands to make more than $28 million as a result of the ELA, a fact that the State may not have been apprised of. Our legal consultant has advised us that a court might conclude that the ELA contract with Oracle is not enforceable as a valid state contract because it may not fall within an exception to competitive bidding requirements, as claimed by General Services. Logicon’s apparent undisclosed role, actions, and compensation raise additional questions about the validity of the ELA contract. However, a finding that the Oracle contract is unenforceable because it failed to comply with competitive bidding requirements would raise questions about the impact on the State’s best interests. For example, our legal consultant cautioned that even if a court determined that the ELA contract is void, additional questions are raised by the financing provisions of the ELA contract, in which Logicon assigned a $52.3 million loan to Koch Financial Corporation (Koch Financial). Because Koch Financial apparently acted in good faith and the State accepted the database license and maintenance support on May 31, 2001, Koch Financial will likely assert that the State is obligated to repay the loan. If that position is correct, the State’s recourse for recovering the $52.3 million may be to recover the money from Oracle and Logicon. Also, the State has agreed under the ELA contract that if the Legislature does not appropriate funds for the financing provisions or the State does not otherwise make payment and the ELA contract is terminated, the State will not replace the Oracle license with substantially similar database licenses for a period of one year from the termination date. Successful enforcement of this provision could effectively shut down many departments’ operations. Further legal analysis is required to understand the impact of these provisions on the contract and to make a determination as to whether the contract is void or otherwise unenforceable. 4 5 RECOMMENDATIONS Before pursuing future enterprise licensing agreements, the State should take the following actions: • DOIT, Finance, and General Services should seek legislation establishing the authority to enter into an ELA that protects the State’s interests and defines each department’s respective role and responsibility in the ELA process. • DOIT and Finance should develop policies and procedures on how to evaluate future ELAs. To be effective, one state department needs to take responsibility for developing and justifying the ELA proposal. • Finance should complete its survey and develop a method to allocate the ELA’s cost to state departments. • General Services should ensure its negotiating team possesses all the types of expertise necessary to protect the State’s interests. To identify the legal measures to take to protect the State’s interests, we recommend the following: • General Services should further study the ELA contract’s validity in light of the wide disparities we identified in Logicon’s projections of costs and savings, and consult with the attorney general on how to protect the State’s best interests. • General Services should work closely with the attorney general in further analyzing the ELA contract; all amendments, includ- ing any and all documents pertaining to the side agreements between Oracle and Logicon; and the laws and policies relating to the ELA, including the potential legal issues that this audit has identified. If it is determined that the ELA is enforceable, General Services should renegotiate its contract with Oracle to add and clarify the terms and conditions necessary to better protect the State. The Legislature should consider requiring that all IT contracts above a specified dollar amount receive a legal review by General Services. 4 5 AGENCY COMMENTS General Services, DOIT, and Finance agree with our recom- mendations. However, General Services disagrees with our conclusion that the ELA may not meet the requirements for a sole-source contract, and DOIT provides a list of changes it requested to an earlier draft of the report. All three depart- ments also discuss some of the steps they are taking to improve and implement the ELA or to develop a process for future ELAs. n 6 7 INTRODUCTION BACKGROUND On May 31, 2001, the Department of General Services (General Services) signed an enterprise licensing agreement (ELA) with the Oracle Corporation (Oracle) to license up to 270,000 users of Oracle Enterprise Edition 8i database software (enterprise database licensure), obtain 100,000 universal power units to allow for internet access, and receive six years of maintenance support services, with an option for four additional years of maintenance support. An ELA is an agreement between a software vendor and a consumer that licenses the buyer for multiple users of specific software on a large scale (enterprise software). The ELA between Oracle and the State, the first such agreement the State has entered into, allows all state departments and employees to use the enterprise database and also allows both state and local governments to buy other Oracle products at a 50 percent discount from list prices through September 2006. General Services will pay $94.6 million for the license and six years of maintenance in 10 installment payments (5 payments for principal and interest and 5 payments for maintenance) beginning in September 2002. On an annual basis beginning June 1, 2007, and continuing through June 1, 2010, the State can extend the maintenance support services at an annual cost of approximately $7 million. If the State extends support services for all four years, the ELA will cost $122.6 million. An ELA’s firm commitment to purchase an agreed-upon amount of goods or services to cover anticipated demand potentially allows a buyer to reduce overall costs by negotiating more favor- able terms. The primary sources of ELA savings to a consumer are the following: • Lower license costs from making a volume purchase. • Lower annual maintenance costs from negotiating a fixed, multi-year rate. • Lower administrative costs to acquire, review, track, and report on license usage than if state departments separately purchased licenses. 6 7 Unlike ELAs, volume purchase agreements, currently avail- able to all state departments, establish discounts for future purchases based on anticipated demand. However, because these agreements do not involve a firm commitment to pur- chase anything, vendors may not offer discounts as deep as those given through an ELA. INFORMATION TECHNOLOGY Information technology (IT) involves using computer technology to create, store, exchange, and use information such as data, voice conversations, still images, motion pictures, and multimedia presentations. Simply put, IT means information processed and managed by computers using hardware and software. Software is a set of instructions that gives a computer the capacity to perform different functions or applications. Unlike computer hardware that a buyer can resell, loan, or give away without prior permission from the manufacturing company, computer software is copyrighted intellectual property that buyers agree to use as instructed by its manufacturer. This agreement, called a license, is what one actually buys; the manufacturer retains ownership of the software itself. There are two major categories of software—system software and application software. System software consists of control programs such as operating software and database management systems. The Oracle Enterprise Edition 8i database is one example of a database management system. Application software is any program that processes data for a specific purpose. Inventory and payroll programs are examples of application software. Database software is designed to store data in individual records composed of designated fields of information. The software makes it easy to access, manage, and update this stored information. Also, the software’s ability to sort and arrange information fields allows a user to discover raw data’s less obvious significance and produce customized reports. EVENTS CULMINATING IN THE ELA According to its director, beginning in June 2000, representa- tives of the Department of Information Technology (DOIT) 8 9 attended meetings at which state chief information officers (CIOs) expressed concern with how much their respective departments were paying to license and support software. Because of these concerns, in that same month, DOIT contracted with Logicon Inc. (Logicon), a software reseller and provider of IT sys- tems and support services, to review industry best practices for enterprisewide software licensing and provide a report delineating alternative licensing strategies for the State to consider. Although DOIT received a draft, Logicon never completed the report, and DOIT ultimately cancelled the contract on November 30, 2001. Between February and May 2001, Logicon made a series of sales presentations for representatives of DOIT, General Services, and the Department of Finance (Finance). Included in at least one of these presentations was a document in which Logicon suggested the State employ it to negotiate an ELA with Oracle. Furthermore, in January 2001, an enterprise software work group—consisting of members from DOIT, General Services, and representatives from the State’s data centers—was formed to determine how to capitalize on the State’s large number of yearly software acquisitions. The enterprise software work group agreed that the State should leverage its purchasing power with Oracle by buying at a volume discount rate. According to the manager of General Services’ acquisitions branch, during these meetings and various phone conversations, DOIT verbally recommended that General Services use the ELA as a means of leveraging the State’s buying power with Oracle. However, according to DOIT, while it verbally recommended to General Services that the State needed a means for leveraging its purchases, it did not recommend an ELA or specify Oracle as the vendor. In late May, a consensus was reached among DOIT, Finance, and General Services to contract directly with Oracle rather than through Logicon. On May 31, 2001, DOIT, General Services, and Finance agreed that the State should enter into a sole-source ELA contract with Oracle. (For a further description of the activities that led up to the execution of the ELA and that have occurred subsequently, see Appendix A.) THE STATE’S ROLE IN IT PROJECTS Three departments—DOIT, General Services, and Finance—guide and monitor IT projects or procurements within the State. As Figure 1 on the following page shows, DOIT is responsible for the review, approval, and oversight of the technical aspects of 8 9 these projects—General Services for procurement of most IT goods and services—and Finance for evaluating the ben- efits of an investment in IT and approving the funding for proposed projects. FIGURE 1 The State’s Information Technology Development Process �������� �������� ����������� �������������� ���������� ���������������� ��������������� ��������������������� ���������������� ������������� ����������������������� ��������������������� ��������������������� ����������������� �������������� ������������������� ������������������� ������������ ������������������� �������������� ������������� ������������������ ����������������� ����������������� ������������������ ��������������� ����������������������� ������������������ ������������� ���������������� ��������������������� �������������������� �������������������� ��������������������� ������������������� �������������������� ������������������� ���������������������� ������������������� ����������������� ���������������������� ������������������ ������������������ ����������������� ����������������� ������������������� ���������������� ��������������������� ����������� ����������� ����������������� ������������������ ����������� ���������������� ������������������ ����������������� ������������������ ����������������������� ������������������ ���������������� ������������������� ������������������� ��������������������� �������������������� ������������� ������������� ���������������� ��������������������� ����������� ������������������� ����������������������� Source: Bureau of State Audits. Department of Information Technology DOIT’s responsibilities are to guide agencies statewide on acquiring, managing, and appropriately using IT. Also, DOIT is to oversee state departments’ management of specific IT projects and develop policies for acquiring new IT software or hardware. DOIT communicates IT policies through the State Administrative 10 11 Manual and management memos sent to all state departments. A department seeking approval for an IT project is generally required to submit a feasibility study report (FSR) to DOIT. The FSR gives the business and technical reasons to justify investing state resources in the project, why the proposed project is needed, the means for ensuring its success, and a comprehensive analysis of its benefits and costs. DOIT’s review focuses primarily on the project’s technical viability, justification, proposed management risks, and alignment with the State’s mission. DOIT is also responsible for monitoring projects to ensure compliance with statewide strategies, policies, and standards. Finally, DOIT develops best practices and guidance on acquiring, managing, and using IT, as well as directing how the State can use IT to reduce the cost of government. Department of General Services Depending on the procurement method used, General Services and the state departments each have different responsibilities. Among other things, General Services develops and administers authorized procurement methods, reviews and approves contracts that are above established limits, supervises purchase orders that exceed specified dollar limits, reviews requests for exemptions from competitive bidding requirements, and when needed, guides departments in acquiring goods or services. Generally, departments are responsible for identifying the appropriate goods or services that best meet their needs and for obtaining them in a fair manner and at the best value. Departments must acquire IT goods and services through competitive means unless they can show General Services that an emergency requires immediate procurement or that no other supplier can meet the need. Also, General Services has the final authority for the general procurement procedures relating to IT acquisitions. General Services currently uses various leveraged procurement methods—including volume purchase agreements, master service agreements, and the California Multiple Award Schedules (CMAS) program—that give state departments lower prices by negotiating bulk discounts based on the entire state’s needs. Volume purchase agreements give state departments vendor discounts that increase with the volume purchased. Master service agreements streamline the contracting process by having General Services prescreen vendors so state departments can buy according to one master service agreement rather than through numerous, individually negotiated contracts. The 10 11 CMAS program lists preapproved suppliers of goods and services that vendors agree to sell under the same terms, conditions, and prices as those in existing multiple-award-schedule contracts (base contracts). The base contract must be awarded either by the General Services Administration of the United States or by some other entity—typically a state or county. Using any of these procurement methods, any state department can purchase goods and services from participating vendors without going through a bidding process. Department of Finance Finance is responsible for approving the funding for IT projects described in the state departments’ FSRs. Finance’s evaluation focuses on a project’s business outcomes, investment value, and proposed source of funding. Finance also analyzes the FSR’s business case assessment to determine whether the department has adequately justified the business need for the proposed IT project. Although it has the authority to approve the expenditures budgeted for an IT project, Finance is not a part of the contracting process. SOFTWARE AND SUPPORT PRICING Pricing strategies for business software and support are unique to the software industry and different from typical commodities. The costs to produce most commodity items are driven by considerable fixed expenses for raw materials, manufacturing, labor, distribution, and storage. Pricing is established at cost plus a profit margin. In addition, following the purchase, the consumer generally does not require ongoing assistance to use most commodities. This typically leaves little room to negotiate the price for a commodity, because all negotiation comes directly from the producer’s profit from the sale. According to our technical consultant, developers of enterprise- wide software like Oracle have a very different business model. Software development has substantial up-front costs for research and development, but once the software has been developed and is ready for distribution, manufacturing and distribution costs are minor—no more than making and mailing duplicate copies of compact discs to customers or providing them with the ability to download the software from the Internet. This makes the incremental cost of producing the second and subsequent copies of a software product negligible. 12 13 Our technical consultant also advises that software sales do not typically end with the licensure of the product. In fact, software vendors get most of their revenue from annual maintenance fees, a recurring source of revenue for the duration of the software usage. Maintenance prices are usually based upon the current (undiscounted) list price of the software. For instance, Oracle’s current maintenance fees are 22 percent of list prices. Software maintenance refers to the personal assistance vendors provide to users of their software. Users who have software maintenance agreements with a vendor may contact the vendor’s support personnel for assistance in installing a piece of software or using a certain function. Software maintenance also includes periodic updates, such as adding new functions, correcting identified defects, and adapting software to new hardware and software configurations. Our technical consultant also advises that maintenance revenue provides enterprise software vendors multiple opportunities to realize income from the same client for the same software, first when the software is initially licensed, and later, on a recurring basis, when clients pay for maintenance. The opportunity to establish an ongoing revenue stream and the low incremental cost of producing copies of software combine to furnish software companies like Oracle a great deal of flexibility to discount list prices for initial licensing of software without sacrificing profitability. Realizing the future expense and difficulty customers will face if they switch software, vendors use discounts to gain the high levels of commitment buyers make in selecting software. Consumers incur significant costs to implement a major software product like a database—costs such as installation, training, and the integration of the product into the user’s environment. Should an organization later decide to switch to another vendor’s product, it can expect to pay all of the start-up costs associated with the new software, as well as potentially substantial conversion costs of reworking existing systems. Software vendors commonly discount software licenses to make them as attractive as possible for the initial purchase, because they know that once a product has been implemented, changing to a different product is usually not cost-effective. Vendors assume the long-term revenue stream from maintenance will offset any loss in revenue from discounting the software. For example, Oracle is widely known to offer discounts of 75 percent or more from its list price for 12 13 large purchases. In fact, the Department of Justice reported a price reduction of 95 percent from the Oracle CMAS rates for a purchase made in fiscal year 2000–01. SCOPE AND METHODOLOGY The Joint Legislative Audit Committee (audit committee) requested the Bureau of State Audits to examine the State’s contracting practices in entering into the ELA with Oracle. We were asked to review a number of specific areas including the following: • The justification for using a sole-source contract for the ELA. • The roles of DOIT, General Services, and Finance in develop- ing and executing the ELA and whether these roles conflict with their statutory responsibilities. • The methods used to justify the technical and business need for the ELA. • The terms of the agreement and whether they are in the best interests of the State. • The funding sources that will pay for, and the number of state departments that will participate in, the ELA. We were also asked to identify the ELA’s fixed and variable costs and to evaluate the reasonableness of the projected savings from the ELA. Lastly, the audit committee requested we obtain a legal opinion on whether the contract is null and void if it was executed in violation of state law. To understand the State’s contracting practices for IT, we employed a legal consultant with expertise in public contracting to research laws and regulations governing that area. We identified circumstances in which a state department can justify a sole-source contract for IT. We also reviewed a sample of competitively bid IT contracts approved by General Services to determine the nature of these procurements. Finally, we interviewed staff at General Services and reviewed documents related to selected contracts, including the Oracle ELA. To determine DOIT’s statutory role and responsibilities regarding contracts and IT policies and procedures, we reviewed state laws and regulations, as well as DOIT’s policies and procedures on reviewing IT projects. To understand DOIT’s role related to the 14 15 Oracle contract, we interviewed key staff and reviewed various documents generated during the ELA’s development. We also assessed the roles and responsibilities of General Services and Finance by reviewing their policies and procedures on IT projects and contracts, interviewing key personnel, and reviewing documents pertaining to the ELA. To assess the methods used to justify the technical and business need for the Oracle contract and to determine if it was in the State’s best interest, we employed an IT consultant (technical consultant) to review General Services’ justification for the ELA, along with other pertinent documents explaining the rationale for acquiring the enterprise database licensure. We also interviewed staff at General Services, DOIT, and Finance to determine whether they had adequately researched the necessity of enterprise database licensure for 270,000 state employees. Finally, we reviewed the results of a statewide Oracle survey conducted by DOIT in March 2001. To evaluate the reasonableness of the projected savings from the Oracle contract, we reviewed General Services’ calculation, which it prepared following Logicon’s instructions and using Logicon’s assumptions, of estimated savings. (Due to General Ser- vices’ clerical role in this process, we refer to these as Logicon’s projections throughout the report.) We also reviewed for math- ematical accuracy and propriety, Logicon’s compilation of three past fiscal years of state spending on Oracle databases. To iden- tify the purchases covered by the ELA, we reviewed the historical purchases with General Services’ staff. Using the same assump- tions that Logicon used, we then recalculated the projections. To determine whether the negotiated terms and conditions in the ELA are in the State’s best interest, our technical consultant identified industry best practices for negotiating IT agreements. He also reviewed the types of terms and conditions found in other software licensing agreements and compared them with those in the ELA. To ascertain the funding sources used to pay for the ELA and the number of participating state departments, we reviewed the ongoing effort of Finance to identify current and future users of the Oracle products contained in the ELA and its efforts to develop a cost allocation mechanism to pay for the contract. Finally, our legal consultant researched case law to identify legal precedents governing the validity of public contracts and deter- mining the consequences of potential invalidity. n 14 15 This page left blank intentionally for reproduction purposes only. 16 17 CHAPTER 1 Failing to Assess the Need for Database Licenses and Lacking Guidance From the Appropriate Departments, the State Committed Millions of Taxpayer Dollars for an Enterprise Software License It Has Yet to Use CHAPTER SUMMARY The State signed the Oracle Corporation (Oracle) enterprise licensing agreement (ELA) without knowing if there was sufficient need to justify committing $94.6 million in state funds for up to 270,000 state employees and consultants (users) to use Oracle Enterprise Edition 8i database software (enterprise database licensure). None of the three departments directly involved in approving the ELA—the Department of Information Technology (DOIT), the Department of General Services (General Services), or the Department of Finance (Finance)—conducted a comprehensive analysis to gauge or confirm the level of statewide interest in the ELA license. However, at least two months before the ELA was executed, DOIT had preliminary survey data showing a limited interest in additional Oracle products—data that should have led it to question the size of the proposed purchase. Before signing the ELA, the State also lacked informed guidance on the proposal’s merits. The State had never evaluated an ELA proposal before and lacked specific procedures to do so. Nonetheless, DOIT and Finance routinely evaluate information technology (IT) projects and possess the expertise needed to evaluate the ELA proposal—DOIT the need to license 270,000 users and Finance the proposal’s cost projections. However, neither did so, citing a lack of suitable procedures and inadequate time. To its credit, Finance’s Technology Investment Review Unit (TIRU), which is responsible for reviewing IT proposals and ensuring IT expenditures represent a prudent investment of resources while meeting the State’s needs, identified specific concerns with the ELA proposal, and on May 10, 2001, communicated these concerns to the directors of DOIT and Finance. It also recommended postponing the ELA proposal until the next year to give the State a chance to develop appropriate 16 17 policy. However, its concerns and recommendation were not heeded. As a result, the State committed almost $95 million in taxpayers’ money without knowing if the ELA was an appropriate procurement of technology or if its costs and benefits were justified. In contrast to the possible $16 million savings at the end of the six-year term of the ELA represented by Logicon Inc. (Logicon), an Oracle reseller that benefited significantly from the contract, our estimates (using the same data and assumptions Logicon used in its proposal, but correcting for the errors we found) show the ELA may cost the State approximately $41 million more than it would have paid without the ELA. A review of the data (state purchase orders) supporting Logicon’s savings analysis shows substantial costs that should have been excluded. Although basing their approval of the ELA partly on Logicon’s estimate of how much the State would save, none of the three departments thoroughly validated the figures in that estimate or the underlying questionable assumptions. Besides failing to properly analyze the ELA proposal’s projected costs and savings, Finance did not develop a cost allocation method that would allow General Services to bill state departments that acquire enterprise database licenses using the ELA. Although Finance is working on a cost allocation method, it is not yet complete. Finance and General Services both contend that state departments can acquire database licenses before the cost allocation model is complete. However, neither Finance nor General Services has formally notified departments of this or provided them with ordering and pricing information. Thus, it is not surprising that as of March 20, 2002, no state departments have acquired new licenses under the ELA. By June 2002, when the cost allocation model is expected to be ready, the State will have accrued over $17 million in ELA and interest costs and likely received little benefit in exchange. DOIT IGNORED CRITICAL DATA SHOWING A LIMITED NEED FOR ADDITIONAL ORACLE DATABASE LICENSES In approving the ELA proposal, DOIT ignored data suggesting the need for additional Oracle database licenses was limited and by doing so allowed the State to commit millions of taxpayer dollars on excess software license capacity and maintenance. The results of a preliminary survey sent to information and executive officers of 127 state entities by DOIT in spring 2001 strongly 18 19 suggested that most had no immediate need for additional Oracle products, including database licenses. Either the entities did not use the Oracle database software or expressed no inter- Of the 127 surveys est in a consolidated purchase of Oracle products. In its survey, sent to state entities, DOIT instructed only those government entities interested in DOIT received only consolidating their purchases of Oracle licenses to respond. 21 responses, 5 of which Of the 127 surveys sent to state entities, DOIT received only indicated an interest in 21 responses, 5 of which indicated a possible interest in purchasing purchasing additional any additional Oracle products under a consolidated agreement Oracle products under a in the near future. Five months after the State approved the ELA, consolidated agreement Finance sent out another survey that will be used in develop- in the near future. ing a method to allocate the ELA’s costs to state departments. Early results of this survey reveal that many of the State’s largest employers need database licenses for only a small percentage of their employees. Unfortunately, it appears the State may have overbought Oracle’s enterprise database licensure and support maintenance in a long-term, six-year contract. DOIT’s Survey Suggested Few State Departments Wanted to Buy More Oracle Software In early March 2001, DOIT sent an electronic mail survey to 127 state entities to identify those interested in making a consolidated purchase of Oracle software licenses within the next six months. Response to this survey showed slight interest among existing Oracle software users and indicated that many departments did not use Oracle software at all. Two months after the survey, without any follow-up on the respondents, DOIT approved the proposal to purchase an enterprise license for up to 270,000 users of the Oracle database. By not accurately gaug- ing the State’s need for the enterprise database licensure, DOIT allowed millions of dollars in state resources to be committed for a highly uncertain use. According to DOIT’s instructions, the survey’s purpose was to identify a guaranteed volume of licenses that departments would be willing to purchase as a group through a California Multiple Award Schedules (CMAS) program with Oracle so they would receive a better volume discount than otherwise. The instructions described the cooperative purchase price as based on a guaranteed volume of licenses to be purchased, unlike traditional volume agreements with license prices based on anticipated spending. DOIT believed that such a guaranteed group purchase would yield a larger discount than either a traditional volume agree- ment or than smaller, individual department purchases. The survey asked for information on the existing use and future 18 19 need of Oracle software, including database licenses. Specifi- cally, the survey asked what Oracle products the departments owned and the numbers and types of licenses applicable for each product. DOIT’s survey also asked each entity to identify its immediate needs for additional Oracle products, its needs over the next two to three years, and whether it had budgeted for current and future Oracle software needs. The survey results showed little interest in a consolidated purchase of Oracle database licenses. In fact, about two months before DOIT approved the ELA proposal authorizing an enterprise license for up to 270,000 users, the survey showed that departments with a total of 99,000 authorized positions might not need or want any such purchase. Twelve responding entities, representing 21 percent of the State’s total workforce of 234,000 authorized positions as of January 2002, said they currently used Oracle products. However, only five of those entities, representing 12 percent of the State’s workforce, expressed an interest in participating in a consolidated purchase of additional Oracle products. Conversely, the Employment Development Department and the Department of Consumer Affairs (Consumer Affairs)—who together represent more than 16,000 authorized positions—stated that they would not participate in a consolidated purchase of Oracle products. Also, the Department of Developmental Services with over 10,000 positions and the Board of Equalization with almost 4,000 positions stated that they do not use Oracle products. Finally, assuming that departments having no short-term interest in a consolidated Oracle purchase followed DOIT’s instructions to not respond to the survey, several large departments, including the California Highway Patrol, Department of Corrections (Corrections), and Department of Motor Vehicles, with a combined workforce of more than 69,000 authorized positions, did not need Oracle products or want to participate in a consolidated purchase of Oracle licenses within the next six months. Although well-planned Although well-planned and well-negotiated volume purchases and well-negotiated can save money, buyers must invest considerable time and effort volume purchases can to determine actual need and feasibility before making any save money, buyers must commitments. Unfortunately, although the March 2001 survey invest considerable effort gave several indications of the limited interest in a consolidated to determine need and purchase of any type of Oracle software licenses—including the feasibility before making enterprise database licensure later purchased through the ELA— any commitments. DOIT ignored these signs. First, DOIT ignored the lack of interest suggested by the low survey response. Of 127 surveys sent, 20 21 DOIT had evidence of only 21 responses. According to DOIT’s directions for completing the survey, that could have meant that 106 state entities were not interested in a volume purchase of Oracle licenses or chose not to respond for other reasons. However, we found no evidence that DOIT followed up with the nonresponding entities to confirm their reasons. A second indication of low demand for Oracle database licenses was the relatively small number of respondents, only 12, saying they used any type of Oracle products. Assuming that every employee of these 12 entities might want to use some type of Oracle database, the demand would equal roughly 50,000 database users, far less than the 270,000-user capacity licensed under the ELA. Lastly, only 5 state entities responded that they had a need DOIT ignored strong for any additional Oracle products within the next six-month to evidence that state three-year period. entities had expressed little or no need for DOIT ignored this strong evidence that state entities had little additional Oracle or no need for additional Oracle database software. Rather than database software. follow up with the March 2001 survey respondents or adminis- ter a new survey designed to better measure the State’s need for the Oracle enterprise database licensure, DOIT instead approved the ELA in May 2001, less than two months later. Finance’s Survey Indicates That State Departments Need Database Licenses for Only a Fraction of Their Employees In November 2001, five months after the ELA was approved, Finance sent out another survey to assess the need for Oracle enterprise database licensure and establish a basis to allocate the ELA costs to departments. This survey explicitly required all departments to respond—regardless of whether they used or wanted Oracle licenses. Preliminary survey results suggest that nine of the State’s largest departments that use Oracle database software need licenses for only a fraction, sometimes as few as one-tenth, of their employees. Unfortunately, by the time Finance initiated its survey, the State had already committed taxpayers’ money to buy both licensure and maintenance for all state workers. Finance administered the survey as a preliminary step to appropriately allocate the ELA’s cost among the various departments and will use information on the current and planned use of the Oracle enterprise database licensure to develop a cost allocation model. However, as of April 2002, 10 months after the ELA was approved, the analysis of the survey is incomplete. Finance’s budget manager of 20 21 administration says that analyzing the survey data has been delayed because data provided by some departments appeared to be incomplete and requires interpretation. In cases where the data varies considerably from corresponding data provided Until it completes by Oracle, Finance is going directly to Oracle in an attempt a survey of state to clarify the data. The budget manager hopes to have departments begun the analysis substantially completed by mid-May. Until it in November 2001, completes its analysis of the survey, Finance cannot develop Finance cannot develop an appropriate model for allocating the ELA’s cost among an appropriate model state entities. Finance states that departments can acquire the for allocating the cost database licenses before the allocation model is complete. of the ELA among However, without such a model, state departments do not know state departments. the precise cost of doing so and, therefore, cannot adequately evaluate or compare the license costs for pending IT projects. Further, state departments have not yet been informed of how to acquire new licenses using the ELA. Thus, it is not surprising that as of March 20, 2002, no state departments have acquired new licenses under the ELA. Our review of the survey results as of March 7, 2002, for the 12 state departments with the largest numbers of authorized positions, revealed that 11 use Oracle database products to some extent. However, 2 of the departments—the Departments of Justice (Justice) and Consumer Affairs, which together represent over 11,000 authorized positions—indicated their existing multiyear contracts with Oracle leave them no interest in participating in the ELA. In fact, Justice has specifically requested to be exempted from the ELA to allow it to continue using its existing Oracle contract. Our review of the 9 other large departments using Oracle products suggests the majority of their employees likely did not need enterprise database licensure because they presently have few existing Oracle database licenses. According to the survey responses, these 9 departments use a total of 30,000 licenses although they represent almost 132,000 authorized positions. For example, the Department of Health Services, with over 6,200 positions, has contracts with Oracle for only 1,350 database licenses. Similarly, Corrections currently has 4,710 database licenses, comprising less than 10 percent of its workforce of more than 48,000 positions. This data raises questions about whether departments such as Corrections, with about 50 percent of its employees working as custody staff, would ever need database licenses for each of its employees. Although the ELA will cover up to 270,000 users—more than the total number of state positions authorized as of January 2002— according to the survey, 113,000 of those positions will not use 22 23 Oracle database software in just these 11 departments alone. Such overbuying could be costly because the ELA shows that the State purchased not only a license for up to 270,000 users of Oracle database software but also maintenance services for all those users. DOIT AND FINANCE DID NOT ADEQUATELY EVALUATE THE ELA PROPOSAL’S MERITS The State negotiated and ultimately approved the ELA proposal without sufficient technical guidance, assessment of need, or verification of projected benefits. According to officials at DOIT, General Services, and Finance, the State had never before considered a statewide software purchase, nor did it have any Although both Finance specific guidance in identifying the extent of the need for the and DOIT possess the software and in negotiating the key provisions to include in expertise needed to the contract. The State lacked guidance despite the fact that evaluate key aspects of DOIT had looked at the concept of statewide software licensing the ELA proposal, neither as early as June 2000, when it hired Logicon to research and did so, citing lack of present information on enterprise licensing. Nevertheless, suitable procedures and DOIT and Finance routinely evaluate IT proposals, including inadequate time. those involving software purchases, and possess the expertise needed to evaluate aspects of the ELA proposal—DOIT the need to license 270,000 users and Finance the cost projections. However, neither did so, citing a lack of suitable procedures and inadequate time. To its credit, Finance’s TIRU identified specific concerns with the ELA proposal, and on May 10, 2001, communicated these concerns to the directors of Finance and DOIT. It also recommended that the proposal be postponed until the following year, giving the State a chance to develop appropriate policy. However, TIRU’s concerns and recommendation were not heeded. As a result, the State committed almost $95 million without knowing whether the costs and benefits of the ELA were justified. DOIT Took No Responsibility for Evaluating the ELA Proposal, Yet It Apparently Helped Facilitate Its Development Despite having considered the concept of statewide enterprise software licensing as early as June 2000, DOIT did not establish policies or provide useful guidance to the departments involved with the ELA proposal. Furthermore, DOIT did not evaluate or assess if the need for the level of demand envisioned in the 22 23 ELA proposal was realistic. DOIT stated that it did not evaluate the ELA proposal because the Oracle contract was statewide and therefore not an IT project that required its review. While it was doing little Nevertheless, even without policies and procedures specific to a to evaluate the ELA statewide ELA proposal, DOIT, through its routine evaluations proposal, DOIT actually of IT proposals, had the necessary expertise to determine helped move the ELA whether there was sufficient demand to support the proposal’s proposal forward by assumption that all state employees would use Oracle’s facilitating meetings, database software. Had it done so, DOIT could have used that providing input, information to guide General Services in negotiating the ELA and recommending contract. Considering that the concept of statewide enterprise its approval. agreements was uncharted territory for the State, DOIT’s lack of technological guidance and limited participation in analyzing the ELA proposal’s merits seems inconsistent with its statutory responsibility. By its inaction, DOIT failed to alert the State of the risks associated with such agreements and therefore failed to protect taxpayer resources. Moreover, while it was doing little to evaluate the ELA proposal, DOIT actually helped move the ELA proposal forward by facilitating meetings, providing input, and recommending its approval. DOIT Had Explored the Idea of Statewide Software Licensing In June 2000 DOIT contracted with Logicon to identify the practices of large-scale software manufacturers, review best practices for enterprise licensing, and deliver a “white paper” outlining alternative licensing strategies. However, DOIT’s direc- tor said that Logicon did not complete, and was not paid for, the required work. The director also stated that Logicon never delivered any documents related to the contract. Yet the former deputy director of acquisitions and policy said DOIT received a draft copy of the white paper in late 2000 or early 2001. In fact, in an e-mail to the current deputy director of acquisitions and policy, the former deputy director indicated that she left three copies of the white paper draft (white paper) in her workspace when she left DOIT’s employment. At our request, Logicon gave us a draft copy of the white paper on March 5, 2002. Nine days later, DOIT was able to find the white paper and sent us a copy. It is unfortunate that DOIT was unable to locate its copy before it recommended approval of the ELA because the white paper provided valuable insight on licensing agreements with software vendors. For example, Logicon advised that software users were now looking at the implications of the entire software contract life cycle and not just the obvious financial provisions. Logicon said that every manufacturer, vendor, or reseller has “suggested” 24 25 pricing strategies, yet nearly all of them negotiate within those strategies. According to Logicon, the vendor license agreement is, at most, a starting point and the inclusion of any specific provision is dependent on the circumstances of the negotiation. Logicon advised that the State should carefully review contracts and add whatever provisions necessary to minimize risk and maximize the chances of meeting its business objectives. More- over, the paper stated that most software vendors would exploit increasingly restrictive terms and conditions to increase revenue. Because of this, Logicon advised the State, as a potential enter- prise license user, to negotiate maintenance entitlements during the initial license negotiations and lock them in for multiple years, because failing to do so may result in both increased maintenance fees and additional license fees. Most importantly, the white paper noted that asset management (tracking and managing numbers and types of software licenses in use) is definitely one of the keys to lowering software licensing costs in the near term. Referring to California’s State Administrative Manual, which says departments should perform inventories at least every three years, Logicon advised that these inventories should include software, allowing the State to perform basic asset management. Asset management would enable the State to look at high usage software licenses and begin to negotiate alternative licensing pricing strategies. DOIT had further dealings with Logicon beginning in February 2001, when Logicon made the first of several sales In February 2001 presentations to DOIT and other state agencies regarding Logicon made the the relative merits of various software acquisition methods, first of several sales including ELAs. However, Logicon’s sales presentations lacked presentations to any information contained in its white paper on how the State DOIT and other state could minimize its risk and maximize its benefits in negotiating departments on the an ELA contract. Although the previously discussed March 2001 merits of various software survey of 127 state entities revealed little interest in a consoli- acquisition methods, dated purchase of Oracle database licenses, DOIT did not take including ELAs. that result as a signal to further inventory its software assets statewide. Without any validated information on the demand for or benefits of a statewide purchase of Oracle enterprise database licenses, DOIT recommended approving the Oracle ELA proposal in May. 24 25 DOIT Did Not Evaluate the ELA Proposal DOIT stated that one of the reasons it did not evaluate the ELA proposal was because this was uncharted territory for the State and it did not have any procedures that would apply. While that is true, DOIT does have expertise in certain areas that could have been used to assist in the evaluation of the ELA proposal and provide needed guidance. For example, the staff at DOIT DOIT could have regularly evaluate the IT proposals state departments submit as determined if there part of their feasibility study reports. Part of DOIT’s evaluation was sufficient demand is to determine if the need for proposed technology has been statewide to justify the adequately established. DOIT could have used this type of assumption in the ELA expertise to determine if there was sufficient demand statewide proposal that all state to justify the ELA proposal’s assumption that all state employees employees be covered should be covered under the agreement. Had it performed such under the agreement. an evaluation of the ELA proposal, DOIT could have provided guidance to General Services in its negotiations with Oracle. However, because neither it nor the other departments involved in the ELA proposal had fully researched the need to license up to 270,000 users, DOIT approved the ELA proposal without questioning General Services’ purchase of what appears to be substantial and costly overcapacity of Oracle database software. In approving the ELA proposal without knowing the extent of database software use among state departments, DOIT also did not follow the advice it gave state departments in one of its management memos concerning IT acquisition planning and enterprise asset management policies. On May 1, 2001, one month before the ELA contract was approved, DOIT issued a management memo to all state agencies requiring them to develop plans to catalog and report all their department-wide software. According to the memo, this information would let DOIT and General Services know which shared or identical IT solution strategies exist statewide, enabling them to develop enterprise volume purchase agreements that could leverage the State’s buying power. However, DOIT did not specify a time when state entities would be required to actually report on the amount of department-wide software they possess. In June 2001 DOIT issued another management memo that specified that state departments must submit a report on the number and types of software they use by January 31, 2003. In the absence of such a software inventory, in November 2001, 5 months after the ELA contract was executed, the State began to partially initiate this process. At that time, Finance sent out a survey to state departments requesting that they inventory and report what Oracle software they currently use and what licenses they have. 26 27 As of April 2002, this process was still ongoing. Consequently, 10 months after executing the ELA, the State still does not know the extent of Oracle software use among state departments. Further, when it approved the ELA, DOIT was also aware that neither General Services nor Finance had validated the projected costs and savings in the ELA proposal. However, DOIT believed that it lacked the authority to do so and that validating the proposed monetary benefits was Finance’s responsibility. Consequently, even after Finance requested that it do so, DOIT did not verify the projections in Logicon’s ELA proposal. Nevertheless, when subsequently asked by General Services to help it review some data provided by Logicon, DOIT did provide limited assistance. Specifically, to support the projected costs included in its ELA proposal, Logicon provided copies of Oracle purchase orders to General Services covering the past three fiscal years ending with fiscal year 1999–2000. Being short of time, General Services said it requested DOIT to determine whether the amounts shown in the purchase orders furnished by Logicon for each fiscal year accurately reflected Oracle database purchases. DOIT confirms that its efforts consisted of tracing the purchase order amounts to summary documents prepared by Logicon and adding up the amounts to calculate the total state spending for Oracle database products for fiscal years 1997–98 through 1999–2000. DOIT did no further analysis to verify the accuracy or completeness of the purchase orders. According to DOIT, General Services did not ask it to verify that the purchase orders contained only state-related acquisitions of Oracle database products. However, DOIT did review the purchase orders, concluded they were either from or connected with DOIT facilitated meetings state departments, and verbally informed General Services that attended at various times Logicon’s compilation included only Oracle database purchases. by Logicon, General Services, Finance, and DOIT Helped Facilitate the ELA Proposal CIOs of other state departments regarding Although its director said DOIT participated in developing the ELA proposal initiated the method and approach to be used for the ELA, it had no by Logicon. role in negotiating or executing the Oracle ELA contract. However, documents DOIT provided us show it was involved in various steps leading up to the contract’s execution. For example, DOIT facilitated meetings attended at various times by Logicon, General Services, Finance, and chief information officers (CIOs) from other state departments regarding the ELA proposal initiated by Logicon. On May 24, 2001, DOIT, along with General Services and Finance, reached a consensus that the State would need to execute a sole-source contract with Oracle rather 26 27 than entering into a contract with Logicon. On May 27, 2001, DOIT’s deputy director of acquisitions and policy requested information from General Services and Logicon so that it could better justify its recommendation to proceed with the ELA proposal. Finally, on May 28, 2001, DOIT and General Services formally recommended that Finance prepare an analysis of the ELA proposal. (See Appendix A for a more detailed description of the roles played by DOIT, General Services, and Finance in developing and executing the ELA.) In Approving the ELA Proposal, Finance Ignored Its TIRU’s Unresolved Concerns Own Warning Signs • Annual projected costs and Finance approved the ELA proposal even though its assumptions underlying savings were TIRU had unresolved concerns with the proposal’s not validated. merits. In addition, TIRU realized no one had analyzed • A contract that provides Oracle or validated the ELA proposal, leaving the State at enterprise database licensure for risk for unforeseen costs. Although responsible for all state employees might create a evaluating the business outcomes and investment value de facto standard. of proposed IT expenditures, Finance did not conduct • A funding mechanism had not been identified. such an evaluation for the Oracle ELA proposal. TIRU’s chief says it does not review proposals submitted • Roles and responsibilities of DOIT, General Services, and Finance had not directly from vendors. The chief maintained that the been delineated to show who would department that solicited the proposal or is otherwise take ownership of the project. accountable for the subject area of the proposal is • Using a federal contract model to responsible for analyzing and documenting a business execute the State’s ELA might not case that justifies the State’s funding. Then, Finance be appropriate. would typically validate the business case that the • The State had not established a department presented. Finance expected DOIT to take process on how to execute such a proposal. the lead on the Oracle ELA proposal. On May 10, 2001, TIRU e-mailed the directors of DOIT and Finance a list of concerns, including the lack of an identified funding mechanism to pay for the ELA and a lack of verification of assumptions underlying the annual costs and projected savings from the agreement. DOIT ultimately forwarded these concerns to Logicon, a beneficiary of the contract, to address. TIRU repeated its concerns to the director of Finance in an internal memorandum dated May 30, one day before the contract was signed. TIRU informed the director that neither General Services nor DOIT had validated or analyzed the proposal. It also warned that the State was executing the agreement without legislative review. Finally, TIRU advised that should the State decide to enter into the contract, it should be on a policy basis with the understanding that an evaluation 28 29 of the ELA proposal’s merits was not completed. Nonetheless, while not recommending it, Finance’s director approved the ELA the next day. According to Finance’s budget manager of administration, the director believed that the State had limited risk based on assurances from the director of General Services Finance’s TIRU, in an that this was a “reasonable proposal” and his understanding that internal memorandum the contract included a provision allowing the State to terminate dated one day before the ELA for convenience. Unfortunately, Finance’s director was the contract was signed, unaware that the contract included another clause nullifying repeated its unresolved this provision. concerns to the director of the department. Finance Did Not Use Its Expertise to Identify Flaws in the ELA Proposal Although responsible for reviewing and controlling IT-related expenditures, Finance did not analyze and validate the ELA proposal because it viewed the ELA as a commodity purchase and not a normal IT project. Additionally, Finance believed the State was not going to give further consideration to the ELA proposal until the subsequent fiscal year and, therefore, did not consider an immediate analysis of the proposal’s merits to be necessary. Consequently, it did not confirm the benefits and costs of the ELA proposal, though using its normal analytical procedures would have exposed the ELA proposal’s weaknesses. As mentioned, TIRU had legitimate concerns about the ELA pro- posal; however, by not validating the cost savings analysis in the proposal prepared by Logicon, Finance missed an opportunity to help the State avoid committing millions of taxpayers’ dollars. Contrary to Finance’s position on the ELA, software products should not be viewed the same as other commodities. According to the IT expert for the Office of the Auditor General of Canada: “Software products differ from other goods. They are not interchangeable, are continually upgraded, and pricing is based on the number of users rather than the number of physical items . . . The market is also extremely com- petitive and involves many players. Software can also have far-reaching business implications . . . It has a direct impact on users, and there are high costs associated with changes (such as training needs and lost productivity). Software procurement involves a life-cycle commitment compared to a readily replaceable commodity. In signifi- cant cases it should be handled as a large IT project, and be subjected to the same rigor and discipline.” 28 29 If the ELA’s enterprise database licensure was a mere com- modity, as Finance claimed, it would not be subject to the manufacturer’s control through continuous upgrades and costly maintenance contracts. Also, departments would not have to spend time integrating database software with existing IT technology to make it function. Holding it responsible for helping control state expenditures, state law gives Finance the authority to approve proposed expenditures for IT projects. To meet this responsibility, Finance reviews such projects proposed by state agencies. Although Finance had no procedures specifically designed to evaluate an ELA proposal, it had ample expertise to analyze cost projec- tions such as those included in the ELA proposal. Unfortunately, Finance did not do so. According to the budget manager of administration, Finance was not aware until late May that the ELA proposal was moving The chief of Finance’s forward and by then it lacked time to perform a proper due TIRU recommended to diligence analysis. On May 10, 2001, when the chief of TIRU the directors of Finance informed DOIT of her concerns regarding the ELA proposal, and DOIT that the ELA she also recommended that DOIT take the lead on the project, proposal be shelved until performing the analysis necessary to justify the project 2002 to allow a proper and developing the related policy. She also states that she analysis of the proposal. recommended the proposal be shelved until the following year to allow time for such actions to occur. Finance believed that DOIT concurred with its recommendations and therefore saw no urgency in analyzing the cost projections itself when, on May 17, 2001, Logicon provided it with the purchase orders underlying the cost projections. It was not until May 21, 2001, that Finance learned from General Services that the proposal was moving forward. The budget manager of administration stated that she did not know whether Finance would have had enough time to complete an analysis or, although hopeful, whether such an analysis would have changed the outcome given that TIRU’s earlier warnings had gone unheeded. Further it was not until May 28, 2001, that DOIT and General Services formally recommended that Finance prepare an analysis. Although it is questionable whether Finance could have com- pleted a thorough review had it begun the process when it learned the proposal was moving forward, it may have been able to identify many of the errors we discuss in the following sec- tion. As it was, the State committed almost $95 million without knowing if the projected costs and benefits were justified. 30 31 CONTRARY TO LOGICON’S FLAWED ESTIMATES, THE ORACLE ELA COULD COST THE STATE ADDED MILLIONS IN TAXPAYER RESOURCES Not only has the State potentially spent money for enterprise database capacity and support maintenance that may not be Logicon, whose only used, the Oracle ELA could cost as much as $41 million above role according to the what the two would have cost in the absence of the contract. contract was as the This is because the State did not validate the projections of costs designated lender, and and savings prepared by Logicon, who, acting in an undisclosed who apparently stands capacity as an Oracle reseller or licensing agent, will benefit to make more than significantly from the contract. Specifically, Logicon, whose $28 million as a result only role according to the contract was as the designated lender, of the ELA, developed and who apparently stands to make more than $28 million as the cost savings analysis a result of the ELA, developed the business case analysis Gen- used to justify the State’s eral Services used to justify the State’s decision to contract with decision to contract Oracle. However, based on our review of the supporting data and with Oracle. using its assumptions, Logicon’s analysis was seriously flawed: it was based on costs that should have been excluded because they were outside the ELA’s coverage or did not follow the analysis’ stated methodology. We also found numerous other errors in Logicon’s calculations, and many of its assumptions were questionable. Logicon’s Projected Savings From the ELA Are Overstated DOIT, General Services, and Finance all approved the ELA based in part on the $111 million in projected savings—yet no one thoroughly validated Logicon’s calculations. Logicon was responsible for initiating the sales presentations that resulted in the ELA proposal. Consequently, one would expect the State to closely scrutinize Logicon’s information if the State was using that data to justify the commitment of resources, especially one as significant as the Oracle ELA. Furthermore, Logicon has apparently been well compensated for the software product Oracle sold to the State, although it appears that the State may not have been apprised of that fact. In developing its business case for the Oracle ELA, Logicon used a methodology of reviewing state purchase orders obtained from Oracle for fiscal years 1997–98 through 1999–2000 to determine the past spending trend and to project how much the State would spend annually, lacking the ELA, on Oracle database licenses over the life of the contract, including the four option years. Logicon then used a set percentage—22 percent—of the projected annual cost of new licenses to estimate the yearly cost of maintenance to support the new licenses. Logicon also used 30 31 purchase orders obtained from Oracle for fiscal year 1999–2000 to determine the annual amount spent on renewals for database maintenance support. Based on the assumption that the State would continue to spend funds for database licenses and maintenance for renewed licenses in amounts equal to those spent in fiscal year 1999–2000— Logicon determined according to its projections, $7.8 million and $3.6 million, that the State would respectively—Logicon determined that the State would save save about $16 million about $16 million during the first six years of the ELA. More during the first six years significantly, Logicon estimated the State would save a total of the ELA. Conversely, of $95 million more if it exercised the option to receive four rather than save money, additional years of maintenance after the contract’s six-year our calculations indicate term was complete and the ELA payments became much lower. that the State could However, our calculations, using the same assumptions and data spend approximately as Logicon but correcting for any errors we found, show that the $41 million more than ELA would not produce the savings projected. In fact, as shown it would have absent in Table 1, the State could spend approximately $41 million the ELA. more during the first six years of the agreement than it would have without the ELA. However, if the State elects to exercise the four option years at the end of the contract, the added expense may decrease to $5.6 million. (Appendix B provides more detail regarding our calculations and Logicon’s projected cost savings and assumptions.) Logicon’s Calculations Include Numerous Errors Logicon’s calculation of savings from the ELA was based partly on the State’s history of spending for Oracle software products. We reviewed the purchase orders compiled by Logicon and found numerous errors in its calculations. The errors—which included adding in costs for products not covered by the ELA, costs that were included more than once, and costs that were not incurred in fiscal year 1999–2000, the last year of Logicon’s compilation—resulted in database license costs being overstated by $3.2 million for fiscal year 1999–2000. Logicon asserted that it screened the compilation to ensure the cost savings projection considered only past expenditures for Oracle products that would be covered by the ELA. However, when we reviewed Logicon’s compilation, we identified nine purchase orders totaling $2.2 million for non-database products purchased in fiscal year 1999–2000. Although these costs should not have been included in its compilation, Logicon used them in determining the total projected cost savings. 32 33 32 33 1 ELBAT ALE eht morF sgnivaS laitnetoP detatsrevO ylssorG nocigoL )snoilliM nI( †rotiduA etatS *nocigoL lanoitpO lautcartnoC lanoitpO lautcartnoC ecnanetniaM doireP ecnanetniaM doireP ot ,7002 ,13 yaM ot ,1002 ,13 yaM ot ,7002 ,13 yaM ot ,1002 ,13 yaM slatoT 1102 ,03 yaM 7002 ,03 yaM slatoT 1102 ,03 yaM 7002 ,03 yaM gnidnepS detcejorP 0.64 $ 4.81 $ 6.72 $ 0.87 $ 2.13 $ 8.64 $ sesnecil weN 0.01 0.4 0.6 0.71 8.6 2.01 ecnanetniam weN 0.16 6.04 4.02 6.831 0.58 6.35 ecnanetniam deweneR 0.711 0.36 0.45 6.332 0.321 6.011 detcejorp gnidneps latoT )6.221( )0.82( )6.49( )6.221( )0.82( )6.49( stnemyap ALE )6.5( $ 0.53$ )6.04( $ 0.111 $ 0.59$ 0.61 $ )tsoc(/sgnivas evitalumuC .noisiviD tnemerucorP ,secivreS lareneG fo tnemtrapeD :ecruoS did ti sa slevel gnidneps emas eht ta sraey 01 txen eht rof yllaunna ecnanetniam dna sesnecil esabatad elcarO esahcrup ot eunitnoc lliw etatS eht taht semussa noitcejorp s’nocigoL * .0002–9991 raey lacsfi ni nocigoL yb delipmoc atad redro esahcrup elbaliava fo noitcerroc dna weiver ruo no desab era srebmun ruo ,revewoH .nocigoL yb edam esoht sa snoitpmussa emas eht desu eW † .ALE elcarO eht gnitucexe ot roirp etatS eht ot dedivorp dna Additionally, Logicon improperly included about $440,000 in Our review of Logicon’s costs related to fiscal year 2000–01 purchase orders, when its compilation of cost stated methodology called for only including costs through savings from the ELA fiscal year 1999–2000 in its compilation. We also identified revealed that millions another cost that, although for database products, is expressly of dollars of costs excluded from the ELA and, therefore, should not have been were included in the included when determining the cost savings. In May 2000 compilation in error. Justice purchased database licenses from Oracle totaling $4.1 million for local law enforcement users. According to the terms of the ELA, only licenses used by state employees and state contractors are covered by the agreement. Moreover, the type of database license purchased by Justice differed from those covered by the ELA. General Services’ staff also confirmed our belief that 75 products should not have been included in Logicon’s projections because the items were not covered by the ELA. These items have a total value of $9.5 million. Logicon concluded that the State spent $6.9 million on Oracle licensed products covered by the ELA in fiscal year 1997–98, $7.1 million in fiscal year 1998–99, and $7.8 million in fiscal year 1999–2000. Logicon indicated that, because the State’s spending trend for Oracle software licenses was increasing each fiscal year, it was reasonable to project that the State’s spending level for new Oracle licenses would be at least $7.8 million each fiscal year over the life of the ELA con- tract. In fact, the purchase orders show that during those same fiscal years the State actually spent $3.9 million, $1.7 million, and $4.6 million, respectively, on products covered by the ELA. Logicon’s compilation of how much the State spent in fiscal year 1999–2000 on Oracle renewed maintenance support was also overstated. Logicon’s methodology showed the State had spent $3.6 million on renewed maintenance—maintenance services carried over from the prior fiscal year—for Oracle database products during that fiscal year. Logicon then used the same figure, plus the cost of new license maintenance purchased in the current fiscal year and increased by an annual cost escala- tion factor of 5 percent, in projecting how much the State would spend without the ELA on recurring maintenance renewals over the contract period. However, as depicted in Figure 2, Logicon included in its projection purchase orders for items other than maintenance, purchase orders that were never executed, and amounts for maintenance that will not migrate to the ELA. Furthermore, Logicon combined purchase orders with dates occurring between April 1997 and May 2001—a span of five fiscal years—as if they all related to the maintenance costs for 34 35 fiscal year 1999–2000. As a result, the $3.6 million that Logicon projected as representing the cost of annual maintenance renew- als for Oracle products was significantly overstated. FIGURE 2 Logicon’s Compilation of $3.6 Million in Renewed Maintenance Support Costs for Fiscal Year 1999–2000 Was Overstated by at Least $3.2 Million �������� ��������� ������������� ������������������������ ����������� ����������� �������������������� ������������������� ����������� ���������������� �������������� �������������������� �������� ��������������������������� ��������� ������������������� Source: Department of General Services, Procurement Division. * Maintenance costs falling outside of fiscal year 1999–2000 included the following: • $99,879 for fiscal year 1998–99 • $665,606 for fiscal year 2000–01 • $364,756 for fiscal year 2001–02 To figure the amount of renewed maintenance costs for fiscal year 1999–2000, we first had to identify from the purchase orders Logicon provided which ones included Oracle maintenance that occurred in whole or in part during fiscal year 1999–2000. Then we prorated any multiyear maintenance agreements to separate out only the costs associated with fiscal year 1999–2000. Using Logicon’s methodology, and after compensating for the previously described errors and overstatements, the data showed that the State spent about $480,000 on Oracle maintenance renewal costs in fiscal year 1999–2000—$3.2 million per year less than that claimed by Logicon. The multiple errors and resulting wide disparity 34 35 between our numbers and the numbers Logicon presented to the State to consider in its decision to enter the ELA raises the question that Logicon may have misled the State. The fact that Logicon appears to benefit by as much as $28.5 million from its role in the ELA makes these disparities even more troubling. As we discuss at greater length in Chapter 2, General Services should consult with the attorney general’s office on whether such vastly different results on cost information and projections call into question the legal validity of the ELA. Underlying Assumptions for Logicon’s Cost Savings Model Are Questionable In claiming the State would save $111 million, Logicon not only used inaccurate calculations but also made several shaky assumptions that place the State in a tenuous position. Because the ELA may be in effect for 6 to 10 years, any false assumption will have significant financial consequences for the State. For instance, in the proposal, the cost savings model assumes the State would continue to make annual purchases of Oracle database software licenses in an amount equal to that purchased in fiscal year 1999–2000—$7.8 million. However, as we discussed earlier, the amount projected by Logicon as representing the State’s annual purchase of database licenses was grossly overstated. Additionally, given its current budget crisis, the State may not be in a position to approve many discretionary IT projects in the near future. The cost savings model also assumes that prices for Oracle database products will increase or remain constant. However, if the past business practices of Oracle are any indication, this assumption is unlikely to hold true. As recently as June 2001, Oracle’s chief executive officer stated that the company lowers its prices every year. Further, the database management software market is highly competitive, with Oracle’s chief competitors—IBM and Microsoft— recently capturing some of its market share. Another Logicon’s Questionable assumption—not overtly stated but implied—that Assumptions Oracle’s products will be technologically superior to its • Past purchase trends will hold for the competitors and continue to meet the State’s needs for next 10 years. another decade, may be overly optimistic in the ever- • Prices for Oracle database products will either remain constant or increase. changing and competitive environment of the database market. Lastly, as previously shown, the assumption • Oracle’s products will continue to meet the State’s needs. that all state employees require database software is • All state employees need to access an almost certainly untrue and purchasing such software electronic database. and maintenance for all state employees is a waste of the State’s resources. 36 37 The State Did Not Thoroughly Review or Validate Logicon’s Calculations DOIT, General Services, and Finance approved the ELA proposal based in part on the $111 million that Logicon estimated the State would save over 10 years (the 6-year contract term plus DOIT, General Services, the 4-year optional maintenance term). Yet, as we previously and Finance approved discussed, neither DOIT nor Finance attempted to validate Logicon’s the ELA proposal based in calculations as we did. Likewise, General Services stated that it did part on the $111 million not in any way review or evaluate the purchase orders compiled that Logicon estimated by Logicon. All three departments either asserted that validating the State would save the proposal was not their responsibility or claimed they did over 10 years. Yet, none not have sufficient staff or time to properly assess Logicon’s of them attempted proposed cost savings because of Oracle’s May 31 deadline for to validate Logicon’s executing the contract. calculations as we did. We realize the time to perform a proper analysis was limited. However, the time constraint was largely a conscious choice by those representing the State to accept the May deadline Oracle imposed rather than risk negotiating for an extension. Furthermore, we believe the departments’ concerns about lacking sufficient time for a proper analysis may be overstated. Although it took us additional time to gain an understanding of what was covered under the ELA and to confirm our results, in two and one-half days using two audit staff, we were able to identify sufficient errors in Logicon’s projections to determine that the data includes millions of dollars in erroneous costs, nullifying the purported savings. Had the State begun analyzing the data when Logicon gave it the purchase orders on May 17, 2001, we believe there was enough time to do the same type of analysis that we did. Logicon Will Apparently Receive Significant Compensation for Its Role Relating to the ELA It appears that Logicon stands to receive revenue from the ELA totaling $28.5 million as a result of a complex arrangement used to finance a significant portion of the amount owed under the ELA and related side agreements. As later discussed in Chapter 2 and Appendix C, General Services states it was unaware of the existence of these side agreements. Logicon was designated as the lender under the terms of a financing agreement used to pay $52.3 million—the cost of the software licenses and one year of maintenance—of the total ELA costs. However, under the terms of a separate side agreement between Logicon and 36 37 Oracle that was executed on May 31, 2001, Oracle Logicon Will Assist identifies Logicon as the “preferred leasing agent” in a Variety of Tasks and contemplates that instead of making annual • Processing orders and assisting with maintenance payments to Oracle for five years at ordering requirements. $6.3 million a year, General Services will now make • Assisting in database configuration. those payments to Logicon. • Tracking of purchases. The effect of the side agreement is that, in • Administering volume purchase agreements to state, city, and county apparent consideration for acting as the leasing personnel. agent and for providing ELA support services, Logicon will get to keep just over $1 million out of each of the five annual maintenance payments to be made by General Services—a total of $5.2 million over five years. Furthermore, Oracle executed another agreement on August 31, 2001, with Koch Financial Corporation (Koch Financial)—the lender assigned to take Logicon’s place in the financing agreement—and Logicon. In this agreement, Oracle directs Koch Financial to pay Logicon $52.7 million in loan proceeds, including interest, and directs Logicon to pay Oracle the sums due under the May 2001 side agreement between Oracle and Logicon. The apparent combined effect of these agreements is that Logicon received the $52.7 million from Koch Financial and remitted only $36.5 million to Oracle, keeping the remaining $16.2 million. Under the ELA contract, General Services will repay Koch Financial the loan amount of $52.3 million plus interest of $10.9 million. Table 2 shows a depiction of these agreements and their financial effect. Finally, if the State exercises its option and receives an added four years of maintenance services under the ELA, Logicon stands to make an additional $7.1 million—the difference in the $28 million General Services would pay Logicon over the four years and the $20.9 million Logicon would pay Oracle. Logicon will therefore receive just under $22 million—about 23 percent of the contract’s costs—for its services over the six-year term of the ELA and another $7.1 million if the State exercises its option. Because of the magnitude of Logicon’s stake in the ELA, and given Logicon’s and Oracle’s apparent failure to disclose that stake to the State at the time of the negotiation, we have serious doubts as to whether the State was able to negotiate Oracle’s best price for the ELA or that it was in the State’s best interest. 38 39 TABLE 2 The Oracle ELA: Financing and Schedule of Payments (In Millions) Koch Financial Logicon Oracle Date Action General Services (Assignee) (Lender) (Contractor) August 31, 2001 Logicon assigned its rights as lender under the ELA to Koch Financial. Koch Financial paid Logicon $52.7 million including interest. $(52.7) $52.7 August 31, 2001 Logicon paid $36.5 million to Oracle under a side agreement for the costs of the assets (software licenses) covered by the ELA and first year of maintenance. (36.5) $36.5 September 1, 2002, The State will to make five September 1, 2006 payments to Koch Financial totaling $63.2 million, which includes $10.9 million in interest. $(63.2) 63.2 The State will make five payments to Logicon totaling $31.4 million for annual maintenance. (31.4) 31.4 Logicon will forward $26.2 million of the maintenance payments to Oracle. (26.2) 26.2 Net (paid) received over contract term (94.6) 10.5 21.4 62.7 September 1, 2007, If the State exercises to the option for four September 1, 2011 additional years of maintenance. (28.0) 7.1 20.9 Totals, including four-year option $(122.6) $10.5 $28.5 $83.6 38 39 THE STATE HAS YET TO DEVELOP A COST ALLOCATION MODEL OR TO CAPITALIZE ON THE ELA Finance has yet to develop a mechanism to charge departments for use of the Oracle database license created by the ELA. Finance plans to develop such a model after it completes its statewide survey of departments’ needs, begun in November 2001. Officials at Finance intend to complete their analysis of the survey by mid-May; however, they stated the allocation model may not be ready until June 2002. By then the State will have accumulated over $17 million in accrued Finance stated that the interest charges and fixed contract costs. Figure 3 illustrates allocation model to the accumulating interest charges and allocated fixed costs charge state departments of not yet having a cost allocation model for the ELA. By for the ELA may not be September 1, 2002, these accumulating costs will total about ready until June 2002. $20 million; at the same time, General Services’ first By then the State will payments for the ELA’s financed costs and maintenance, have accumulated over totaling $14.1 million, will be due. $17 million in accrued interest charges and Having Little Ability to Use the ELA Reduces Its Utility contract costs. In addition to data on the current and planned usage of Oracle database products by all state departments, Finance’s statewide survey will also provide necessary information about whether state departments have purchased any Oracle database licenses or entered into any maintenance contracts since the ELA was signed. The absence of an allocation model along with the lack of any specific pricing information or ordering instructions informing departments how to purchase the database licenses through the agreement may further reduce any cost savings or utility from the ELA. This is because state departments needing additional Oracle database software licenses, unaware of how to take advantage of the ELA, will either have to wait to order licenses or incur uncertain future charges for additional Oracle database users. In reviewing the preliminary results of the November 2001 survey, we identified 12 state departments that have entered into their own maintenance contracts with Oracle—totaling $1.1 million for products covered by the ELA—since it was signed on May 31, 2001. For example, in July 2001, the California Highway Patrol executed a maintenance contract with Oracle for approximately $216,000 for 56 database software licenses. This maintenance would have been covered by the ELA had a cost allocation model been in place. Of the 12 state departments that entered into maintenance agreements with 40 41 40 41 3 ERUGIF tseretnI gnidulcnI *htnoM hcaE stsoC ALE ni noilliM 3.1$ gniurccA sI etatS ehT ����������� ����������� ����������� ���������� ���������� �������� �� ����������� ����������� ����������� ���������� ������������ ����������� ���������� ���������� ����������� ���������������� ������������� ��������������� ��������������� ��������������� ���������������� ������������ ��������������� ��������������������� ������������� ������������� ���������������� ���������������� ����������������� ���������������� �������������������� �������������������� ����������������� ���������� ����������������� ���������������� ������������ ������������������� ���������������� ��������������� ���������������� ������������ ��������������� ����������� ������������������� ������������� ����������������� ��������������� ������� ����������� ���������������� ��������������������� �������������������� ��������������� ����������� ������������������ ������������� ������������������ ��������������� �������������� ��������������� ����������� ����������� ��������� �������������� �������������������� �������������������� ����������� �������������� ��������������� ��������������� ���������������� ������������������ �������������� ������������������� ������������������� ����������������� ������������� ��������������� ���������������� �������������� �������������������� ��������������� ������� ����������������� ������������� ���������������������� ����������������� ��������������������� ����������� ���������������������� �������������� ������������������ ������������� ��������������������� �������������� ���������������������� ��������� ������������������ ���������������������� .ecnaniF dna ,TIOD ,secivreS lareneG yb dedivorp noitatnemucod ;1002 ,13 yaM no detucexe ,noitaroproC elcarO dna etatS eht neewteb tcartnoC :ecruoS .tseretni sulp decnanfi stessa eht fo tnuoma eht sedulcni hcihw ,)mret raey-xis eht revo noillim 6.49$( tcartnoc eht fo tnuoma dexfi eht gnitacolla yb detaluclac erew htnom rep stsoC * Oracle after the effective date of the ELA, 11 executed the agreements before Management Memo 01-19 was issued on September 17, 2001, informing all state departments of the ELA and advising them not to enter into new maintenance agreements. Potential Allocation Models All Have Benefits and Drawbacks To allocate the costs of the ELA, Finance could eventually choose from among many possibilities, each with perceived benefits and concerns. What follows is our analysis of three such possible choices. One possibility would be to allocate the costs to all state departments irrespective of their use of Oracle databases and types of licenses. This model would be easy to compute and implement. However, by charging all state departments without regard to their individual usage, the State would implicitly be establishing the Oracle enterprise database licensure as a standard product within state government. Departments would be paying for Oracle licenses whether using them or not, while database products coming from other vendors would still have software and maintenance costs associated with them. Economic considerations could influence a state department’s decision to choose the Oracle database, an IT decision that should be based on technical merit and suitability for the task. A second alternative would be to allocate ELA costs among all departments that use Oracle products based on the number of employees at each department. Again, this method would be simple to compute and implement, and somewhat more equitable than the first. The problem with this method is that it may result in the larger departments paying a disproportionate share of the costs even if their respective use of covered Oracle products is less than other smaller departments, or they do not use the specific Oracle products covered by the ELA. A third possibility would be to allocate the ELA costs only to those departments that add Oracle database users through the ELA, with the proportionate share to be paid by each department determined by the number of new users. This alternative seems reasonable although more difficult to administer, but it too has certain drawbacks. By distributing the cost of the Oracle ELA only among departments currently using the enterprise database licenses, it is likely that those departments would have to bear a greater proportion of the costs than with the other alternatives discussed. Though the allocated cost is fixed, it would be spread among a smaller number of departments. If these departments are required to pay a disproportionately greater share of the 42 43 cost, they may not be able to afford to be among the first to add users through the ELA. This may deter state departments from selecting Oracle when choosing database software. Alternatively, the State might mitigate this effect by setting a fixed price that all existing and prospective users of Oracle database software would have to pay for a license and maintenance. However, the State would be subject to criticism from competing database vendors if the fixed price is artificially low, or this solution might encourage departments to establish separate agreements with Oracle or other vendors if the price is higher than one offered in the marketplace. RECOMMENDATIONS Before pursuing future enterprise licensing agreements, the State should take the following actions: • DOIT, Finance, and General Services should seek legislation establishing the authority to enter into an ELA that protects the State’s interests and clarifies each department’s respective role and responsibility in the ELA process. • DOIT and Finance should develop policies and procedures on how to evaluate future ELAs. To be effective, one state department needs to take responsibility for developing and justifying the ELA proposal. • Finance should complete its survey and develop a method to allocate the ELA’s cost to departments. • Finance should notify the Legislature at least 30 days in advance of any state department executing any future ELAs. • DOIT needs to continue its efforts to create a statewide IT inventory, including software. n 42 43 This page left blank intentionally for reproduction purposes only. 44 45 CHAPTER 2 The State’s Inexperienced Negotiating Team Allowed Oracle and Logicon to Obtain a Long-Term Contract With Major Risks for the State CHAPTER SUMMARY The State faces increased risks because the Department of General Services (General Services) entered into an unusually long software contract with the Oracle Corporation (Oracle) while failing to mitigate the risks of such a contract by negotiating for more protective terms and conditions. The six-year contract, with an option for four more years, is an enterprise licensing agreement (ELA) to cover up to 270,000 state users of Oracle’s Enterprise Edition 8i database software (enterprise database licensure). A lack of safeguard provisions in the ELA puts the State at risk for such problems as future software upgrades not covered in the contract and a reduced demand for licenses. Also, by agreeing to purchase enough capacity to license every state worker, the State may have created the perception that Oracle is its de facto standard for a database, reducing both competition and flexibility in information technology (IT) projects. General Services—whose negotiating team lacked expertise in software licensing contracts and knowledge of Oracle’s past business practices—was poorly prepared for its negotiation of the ELA with Oracle. Oracle is known for using sales tactics to close long-term, large-scale contracts that “lock in” customers who will find it costly to ever switch to a competitor. Not surprisingly, the State representatives’ lack of experience in negotiating contracts of this type allowed Oracle and its reseller, Logicon Inc. (Logicon), to dictate contract terms that favored Oracle but jeopardized the State. Further, the only role that General Services’ legal counsel played during the negotiations was limited to a few hours’ review of the contract’s terms and conditions occurring the day before and the day it was signed. In fact, we found that it is General Services’ policy to limit legal review of all IT contracts, not just the ELA. Our legal consultant advised us that a court may find the contract with Oracle is not enforceable because it does not appear to meet the statutory requirements for a sole-source 44 45 contract. In addition, Logicon’s undisclosed role in, and compensation from, the ELA raises troubling questions about the ELA contract’s validity. Nevertheless, a finding that the Oracle contract is unenforceable because it failed to comply with competitive bidding requirements would raise additional questions concerning the impact on the State. For example, even if a court determined the ELA contract is void, additional questions are raised by the financing arrangements in which Logicon assigned a $52.3 million loan to Koch Financial Corporation (Koch Financial). Because Koch Financial apparently acted in good faith and the State has already received the database license and maintenance support funded by the loan, Koch Financial is likely to assert that the State is obligated to repay the loan. Also, under the contract the State has agreed that if the Legislature does not appropriate funds for the financing arrangements or the State does not otherwise make payment and the ELA contract is terminated, the State will not replace the Oracle license with substantially similar database licenses for one year from the termination date. Further legal analysis is required to understand the impact of these provisions on the contract as a whole and to make a determination as to whether the contract is void or otherwise unenforceable. THE STATE DID LITTLE TO PROTECT ITSELF AGAINST RISKS ASSOCIATED WITH LONG-TERM SOFTWARE CONTRACTS Although long-term contracts for software licenses and maintenance support are typically three to five years long, the State’s contract with Oracle runs for six years with a maintenance option for four more years. The State believed the Long-term software longer contract term would maximize its cost savings. In fact, license contracts General Services’ analysis, based on Logicon’s assumptions and carry substantial risks calculations, shows that to reap the projected $111 million associated with rapid savings from the ELA, the State not only has to realize an changes in technology, estimated annual demand of $7.8 million for new users of vendor performance, the database license, it also has to exercise its option for the and upgrades. added four years of maintenance support. (See Table B.1 in Appendix B for additional detail.) However, long-term software license contracts carry substantial risks associated with the rapid changes in technology, vendor performance, and upgrades. Unfortunately, the State rushed into the Oracle ELA without 46 47 negotiating strong provisions to guard against such risks, so the longer contract increases the chances that savings envisioned over the life of the contract will not materialize. The length of software service contracts generally ranges between three and five years, partly because of the rapidly changing nature of the software industry. Our technical consultant observes that by entering into such a large long-term contract, the State increased risks such as the following: • The vendor going out of business, being purchased, or other- wise becoming unable to perform. • Technology changes that leave the State with a prepaid, long- term contract for a product that has diminishing value. • Future software upgrades that are not supported under the contract. • Lack of funding to make all future payments required under the contract. • Demand for the software licenses not meeting expectations. Unfortunately, the State’s hastily negotiated contract with Oracle lacks adequate provisions to minimize these risks. If the If the State finds it State finds it overestimated demand for the licenses, it has no overestimated demand, contractual remedy because the licenses are already bought and it has no contractual paid for; if Oracle lowers its prices, the agreement contains no remedy because the price provisions requiring the company to lower the contract license covering up price; if Oracle’s software fails to perform or it fails to maintain to 270,000 users has its products, the State has little recourse because the agreement already been bought and lacks provisions for binding arbitration if the two parties paid for. disagree; and if Oracle goes out of business, the State has limited protection, again because the contract does not provide for arbitration or an exit strategy. To protect against such risks, buyers normally try to negotiate mitigating safeguards as part of the terms and conditions of a contract. For example, a buyer would normally want to ensure that contract terms clearly define the support level the vendor will provide, including how many upgrades and subsequent versions of the software will be furnished at no additional cost. The Oracle ELA has a section in the technical support plan (an attachment to the contract) entitled “Oracle 46 47 Updates Subscription Service” that states the service includes the base level for product updates. To understand what “base level” means, we asked Oracle for clarification. The company’s legal representative said the State would be entitled to a subsequent release of a software program at no added cost for its use, excluding those new features that were licensed separately. Further, the State could purchase such separately licensed features and the maintenance support for them using the volume purchase agreement included in the ELA contract. Finally, Oracle’s legal representative directed us to the program updates provision in the “Oracle Technical Support Policies” section of the contract. However, the program updates provision does not clearly define Oracle’s technical support policies. Although these policies are located on Oracle’s Web site, and the contract could have easily included them by reference, it does The State left itself not. Because General Services did not ensure that the technical vulnerable to rising costs support policies on the Web site as of May 31, 2001, were made for software upgrades part of the contract, the State runs the risk that Oracle might because it failed to assert that it can unilaterally update or modify its support negotiate adequate policies. In fact, Oracle last updated the technical support contract provisions. policies on its Web site on December 3, 2001. The State left itself vulnerable to rising costs for software updates because it failed to negotiate adequate contract provisions. Other state agencies have attained better protection against these risks in their negotiations with Oracle. For example, the Department of Justice (Justice) contracted with Oracle in May 2000 for software licenses and maintenance support for five years. Unlike the State’s contract with Oracle, Justice’s contract includes a provision allowing it to receive a credit for technical support for terminated licenses. Also, the Regents for the University of California (regents), in their May 2000 five- year contract, stipulated that if Oracle is unable to substantially remedy instances of not adhering to the contract’s warranty provisions, the regents have the right to stop using the licenses and other services and recover the fees they paid Oracle. Unfortunately, the State has no such provisions in its contract. 48 49 EXECUTION OF THE ELA CONTRACT MAY HAVE CREATED THE PERCEPTION OF A DE FACTO STANDARD THAT MAY LIMIT INFORMATION TECHNOLOGY INNOVATION IN THE STATE The ELA contract threatens to establish the perception that Oracle enterprise database software is the de facto standard, steering future technology decisions toward Oracle products and reducing innovation and flexibility within state IT projects. This perception might cause departments to elect to use the Oracle enterprise database software simply because another database software might involve additional costs. Having a “standard” database software chosen for cost avoidance rather than technical merits does not encourage sound technology decisions. The de facto standard might encourage short-term choices that look effective but turn out to have technical disadvantages. According to the statewide policy sent out in Management Memo 01-19, issued September 17, 2001, the State did not intend the Oracle contract to set its standard for database software. The management memo instructs state agencies to continue selecting the database products most suitable for their particular needs. Nevertheless, our technical consultant explains that a de facto standard is created by people believing something to be the standard. He also said that reasonable people both in state service and in the external vendor community might interpret the ELA as establishing Oracle as the standard database for the State. Further confirmation of this viewpoint came from an analyst with the Gartner Group, an internationally known information technology research and consulting firm. In January 2002 our technical expert interviewed the Gartner Group analyst, who questioned the wisdom of establishing Oracle as a standard for the entire state. The analyst was unaware that the State did not intend the ELA procurement to make Oracle its standard database software. However, creating a standard was the perceived effect in that analyst’s mind (and presumably in others) of purchasing an enterprise license authorizing up to 270,000 users of Oracle database software. Similarly, the Department of Finance’s (Finance) Technology Investment Review Unit’s (TIRU) chief also expressed concern that the ELA may create a de facto standard for Oracle database software for state agencies. 48 49 As yet, the State has no written policy on how it will evaluate IT proposals with database software costs now that the ELA To date, the State has extends the enterprise database licensure to all state employees. no written policy on According to its acquisitions manager, General Services backs out how it will evaluate IT database costs from IT proposals so that Oracle is not a de facto proposals with database standard—but at the time of this audit, General Services had software costs now that not formalized this process nor disseminated this information the enterprise database to state departments. Our technical consultant observes that licensure covers all until the creation, publication, and wide distribution of a policy state employees. explaining how the State will evaluate database costs in future proposals, most of the software vendor community is likely to assume that Oracle database licenses are “free” or will be evaluated more favorably because of the ELA contract. Likewise, until Finance completes and publishes the cost allocation model explaining how the ELA’s costs will be distributed and paid, state agencies can reasonably assume that Oracle database software is either free or costs them whether they use it or not. Although Finance issued a Budget Memo on February 27, 2002, directing departments not to spend money for software acquisition or maintenance support so that General Services could recoup current-year budgeted amounts, the memo stated that completing the allocation model could take until June 2002. THE STATE’S NEGOTIATING TEAM HAD LITTLE SPECIALIZED KNOWLEDGE Because General Services failed to properly prepare for contract negotiations with Oracle, the State faces numerous increased risks over the next 6 to 10 years. For example, General Services did not include on its negotiating team anyone with expertise in the area of software licensing agreements or anyone with an in-depth knowledge of Oracle’s past business practices. More- over, General Services’ legal counsel’s role in the negotiations was limited to a few hours’ review of the contract’s terms and conditions occurring the day before and the day it was signed. Consequently, the State lacks crucial protections against many uncertainties, including financial risks related to Oracle’s future pricing and upgrades. The Negotiating Team Lacked Expertise in ELA and Software Contracts None of the members of General Services’ negotiating team had expertise in ELA contracts or software licenses. Thus, the team did not represent the State’s interests as effectively as it 50 51 could have. If the State determines that it has overpurchased capacity, it has no contractual remedy. If Oracle lowers prices, the State lacks price protection in its contract. The State’s negotiating team appears to have succumbed to common The State’s negotiators vendor negotiating tactics that rushed it into a largely one-sided appear to have contract without balanced protection for both parties’ interests. succumbed to vendor The team accepted a contract with Oracle containing terms not negotiating tactics that tailored to fit a long-term contract for the purchase of database rushed it into a largely software licenses. Further, the team accepted terms that Oracle’s one-sided contract. representative and reseller, Logicon, proposed for the contract; however, these terms place the State at a disadvantage and Oracle in a position of control. The terms of a contract valued at nearly $95 million should pertain to the specific nature of the agreement and mitigate risks to both parties. However, with little substantive modification, the State accepted an ELA contract based on many of the same terms and conditions as an earlier California Multiple Award Schedules (CMAS) agreement with Oracle, an agreement with a significantly different purpose from that of the ELA. According to General Services, Oracle’s CMAS contract was used only as a starting point for the terms and conditions of the ELA contract. Still, the CMAS contract, though not designed to address issues unique to an ELA, was used as a template with few modifications, resulting in terms that favored Oracle rather than the State. For example, because the CMAS terms and conditions were not tailored to meet the specific needs of an ELA, there is no provision explaining how or if state departments with existing database software license contracts with Oracle could, or are required to, migrate to the new ELA. General Services maintains that departments having existing software license contracts with Oracle can transition to the statewide ELA at the end of their existing contracts. According to its September 17, 2001, management memo, General Services directed departments to acquire all new or additional Oracle database licenses and support through the ELA. However, the provisions of the ELA are silent regarding this issue. General Services further stated that Logicon made the original ELA proposal to the State and also acted as a liaison and representative for Oracle. General Services said that Logicon provided input for the contract’s statement-of-work section and proposed the inclusion of enough licenses for all state employees, and that the number of power units and the duration of the contract are based on the initial Logicon proposal. However, Logicon was not just an Oracle 50 51 representative. In June 2000, prior to submitting its original ELA proposal and acting as Oracle’s liaison, Logicon was retained to advise the Department of Information Technology Prior to acting as a (DOIT) on the merits of enterprise-wide software licensing and liaison and representative alternative licensing strategies. It is possible that during the for Oracle, Logicon course of its work, Logicon may have realized that the State was retained to advise was a likely customer for a statewide Oracle purchase. As stated DOIT on the merits of in Chapter 1, while Logicon provided DOIT with a draft white enterprise-wide software paper, according to DOIT, a final product was not delivered and licensing strategies. Logicon was never paid under that contract. Ironically, in that draft Logicon advised the State to carefully review contracts to minimize risk because software vendors would exploit contract terms and conditions to increase profits. Unfortunately the negotiating team did not receive this advice, and relied on Logicon to suggest certain contract terms even though it is clear that by this time Logicon was no longer acting as the State’s consultant, but rather as Oracle’s representative. What is not clear is whether DOIT, General Services, or Finance was aware of how Logicon would benefit from the ELA, except in its role as the designated lender. Our technical consultant observes that for an agreement of the ELA’s magnitude, common sense and sound business practices dictate that the State use diligence to ensure the contract protects its interests. “Standard” terms and conditions are generally those terms and conditions found in all state contracts. To protect against risks specific to activities contemplated by a particular contract, a buyer must negotiate protections into the contract. In normal business practice, the consultant said, a buyer would assemble, prior to the negotiation, a knowledgeable team of experts to identify issues relevant to the particular purchase. In the Oracle contract, the State’s negotiating team would have focused on the recourse available if the vendor fails to perform, refunds or other considerations to be granted if the State were unable to use all the licenses, and warranties sufficient for a pro- curement of this size. However, as discussed earlier, the ELA does not adequately address these points. In fact, General Services neglected to enlist the help of its legal counsel when it drafted the ELA. The importance of a negotiating team having the appropri- ate expertise is generally acknowledged. For example, citing the Gartner Group, the Texas comptroller of public accounts, in a performance review of software management, made the following observations: 52 53 “Managers of software portfolios should have varied expertise, including the ability to negotiate contracts. Key aspects to any successful negotiation include: written negotiation objectives and strategies; a clear understanding of the organization; a negotiating team that represents the end-user, legal, finance, purchasing, and management; an explanation of participant roles; a written understanding of what must be achieved in the negotiation and what can be given up; executive management buy-in; and finally, a pre-negotiation run- through of how the negotiation will be managed.” When we inquired about the experience of the members of General Services’ negotiating team, we were informed that none has expertise in ELA contracts or software licenses in general, and none had any experience with or knowledge about dealing with Oracle. As a result, it is likely that the interests of the State were not as effectively represented in the negotiation as they could have been. The Negotiating Team Lacked Knowledge of Oracle’s Business Practices In the contract negotiations, the State lacked someone with an in-depth knowledge of Oracle’s past business practices. Such an expert might have recognized and countered Oracle’s tactics to better protect the State’s interests. Without understanding Oracle’s practices of discounting, lowering prices every year, and using aggressive sales tactics, the negotiating team accepted contract terms that do not adequately protect the State. Our technical consultant observes that for a contract as large as the ELA, good business practices dictate that a negotiating team study the vendor it is doing business with. How would that vendor be likely to approach the negotiations? What were the Without understanding experiences of others negotiating with that vendor? Again citing Oracle’s practices of the Gartner Group, the Texas comptroller of public accounts’ discounting, lowering performance review of software management said that “at a prices every year, and minimum, agencies should perform an analysis of the vendor using aggressive sales that includes . . . the competitive strengths of each product and tactics, the negotiating vendor; the importance of the product in the vendor’s portfolio; team accepted terms that the vendor’s position in the industry; and the vendor’s business do not adequately protect practices.” The review comments that “the principle reason the State. for the analysis would be to assess the risks of acquiring and implementing the software and working with the vendor on a long-term basis.” 52 53 However, the State’s negotiating team performed no such analysis; was likely unaware of Oracle’s marketing, negotiating The negotiating team and pricing strategies; and so was not prepared to face Oracle performed no vendor in the negotiations. However, if anyone on the negotiating analysis; was unaware team had reviewed industry and other publications, the team of Oracle’s marketing, would have known that Oracle’s business strategies include negotiating, and pricing the following: strategies; and so was not prepared to negotiate. • Offering substantial discounts (approaching 80 percent) from list prices to close large deals. • Aggressively selling to the highest levels of an organization by basing its arguments on the “positive impact on the custom- er’s business” rather than the technical details of the database and competition. • Using high pressure sales tactics to close long-term, high-value deals quickly—saying, for example, “If you buy databases for ten computers this year, and promise to buy databases for fifty computers over the next five years, we’ll give them to you at the special rate we have now. Prices are going up; it’s the fourth quarter and we are ready to deal.” • Practicing the Oracle maxim, “lock customers in and lock competitors out,” by getting from customers a long-term com- mitment that encourages migration to Oracle products and helps establish Oracle as an organizational standard. The high cost of later transition away from Oracle products to those of competitors discourages future competition. • Oracle has a history of changing the way that it licenses products, making long-term commitments problematic. For example, the most recent version of the Oracle enterprise database software, Version 9i—released two weeks after the State executed the ELA—has certain features that are separately licensed. If the State wanted to upgrade from the 8i to the 9i version and also wanted these special features, it would have to pay a separate license and maintenance fee to Oracle for them. According to our technical consultant, other Oracle business developments that were occurring shortly before the contract was finalized included pricing pressure from its customers. In 2000 and the first half of 2001, Oracle’s users and prospective clients were putting tremendous pressure on the company to lower prices. Oracle’s database market share was being threat- ened on the high end (large complex systems) by IBM and on 54 55 the low end (small departmental systems) by Microsoft. Oracle responded with a price reduction on June 14, 2001. The next day, in response to questions about Oracle’s pricing, Oracle CEO Larry Ellison said, “Every year, we lower our prices.” Legal Counsel Had Little Time to Review the Contract Although naming its legal counsel as part of the negotiating team, General Services did not include legal counsel in its negotiations with Oracle, nor did counsel review the entire contract until the afternoon of May 31, 2001, the day the General Services did not contract was executed—hardly enough time to conduct a include legal counsel in its thorough review. Both the chief counsel and staff counsel say negotiations with Oracle, that, unlike other types of procurement contracts, General nor did counsel review the Services’ Office of Legal Services (legal services) is not required entire contract until the by law or policy to review IT procurement contracts. However, afternoon of the day it the purpose of having state legal counsel review a contract is to was executed. assure the contract effectively represents the agreement between the parties and to assure that the terms and conditions protect the interests of the State. A more complete legal review might have brought to light missing provisions needed to safeguard the State or provisions that put the State at risk. Lacking such a legal review, General Services did not take the necessary steps to protect the State’s interests. Staff counsel for General Services said she was given the general terms and conditions of the CMAS boilerplate, the template the ELA was based on, the day before the contract was executed. However, the boilerplate did not contain any of the language specific to the purchase. Staff counsel said she did not receive the entire contract for review until the afternoon of the day it was executed, May 31, and therefore was unable to thoroughly review it. Three months later, in August 2001, the staff counsel prepared an opinion for the benefit of the lender stating that the contract is legal and binding on the State. The opinion did not include counsel’s legal analysis. Not only was legal counsel’s review of the contract for the ELA limited, legal services generally does not review contracts for the procurement of IT goods and services. As a result, there is less assurance that the State’s interests are protected for these types of contracts even though many can involve millions of dollars. State law requires General Services to review and approve all non-IT contracts that are not otherwise exempted. Additionally, state law requires General Services to be a party to or super- vise all contracts for the acquisition of IT goods and services. 54 55 According to General Services’ chief counsel, its procurement division has sole responsibility for executing or supervising General Services’ legal IT contracts while legal services has oversight responsibility for counsel generally does all non-IT services and consultant services contracts. The chief not review contracts for counsel also stated that legal services’ staff are assigned to assist the procurement of the procurement division when necessary. However, according IT goods and services. to the chief counsel, legal services’ assistance is generally not required or used for IT contracts. For example, as previously dis- cussed, legal services only received parts of the Oracle ELA contract for review the day before and the entire contract the day it was executed. Because of time constraints, it could only give a cursory review of the terms and conditions of the contract. THE STATE’S CONTRACT WITH ORACLE MAY NOT BE ENFORCEABLE After reviewing supporting documents, our legal consultant advised us that a court might find that the ELA is not enforceable as a valid state contract because it may not fall within an exception to competitive bidding requirements, as claimed by General Services. However, further analysis is required to understand the impact on the State of a finding that the Oracle contract is unenforceable. For example, our legal consultant cautioned that even if a court found that the ELA contract is void for failure to comply with competitive bidding requirements, additional questions are raised by the financing arrangements for the $52.3 million dollar loan under which Logicon assigned its rights to Koch Financial. Because Koch Financial apparently acted in good faith and the State has received the full consideration for the loan—the enterprise database licensure and one year of maintenance support—under the financing provisions, Koch Financial is likely to assert that the State is obligated to repay the loan. Also, the State has agreed to stop using the ELA’s enterprise database licensure if the Legislature does not appropriate funds for the loan payments or the State does not otherwise make payment and the ELA contract is terminated. More importantly, under the ELA contract the State also agreed not to replace the Oracle license with substantially similar database licenses for one year from the termination date. Moreover, Logicon’s role, actions, and compensation from the ELA raise troubling questions about the validity of the ELA contract. 56 57 With very narrow exceptions, state law requires competitive procurement of IT goods and services. In the opinion of our legal consultant, these exceptions do not seem to allow state agencies to take advantage of the new ELA trend in software licensing, including the Oracle ELA. Moreover, General Services did not make the determinations required by state laws and policies to justify the ELA’s innovative but noncompetitive procurement on a sole-source basis. Finally, contrary to state policy, General Services did not obtain formal prior authorization from the cabinet-level agency secretary to enter into this sole-source contract. In the opinion of our legal consultant, a court might conclude General Services failed to comply with statutory competitive General Services did bidding requirements for contracting when entering into the not make the required ELA contract. Under current California law, an agreement made determinations to justify in disregard of these requirements is void and unenforceable. the ELA’s innovative Thus, the ELA may not be an enforceable contract. California but noncompetitive courts have found that a state department’s legal mode of procurement on a sole- contracting is the measure of its power to contract; thus, a source basis. contract made in disregard of the established mode is invalid. Competitive bidding is the mode of contracting, which state law generally requires departments to use. The purpose of competitive bidding, according to Domar Electric, Inc. v. City of Los Angeles, is to “guard against favoritism, improvidence, extravagance, fraud and corruption; to prevent the waste of public funds; and to obtain the best economic result for the public” and “to stimulate advantageous marketplace competition.” In the absence of legislation and corresponding policies and procedures designed to protect the public while gaining the advantages of large-scale licensing, ELAs do not fulfill the policy goals of competitive bidding. Also, as discussed in Chapter 1, we found that the amount of compensation Logicon is receiving for its disclosed role in the ELA is too much to be merely compensation for being a lender and for the limited software support services it will provide. Since Logicon has apparently already received $16.2 million for acting as a reseller and will apparently receive another $5.2 million for acting as the leasing agent and providing ELA support services, for a total of $21.4 million (almost 23 percent of the total con- tract price) during the fixed six-year term of the contract, it appears that this was not a sole-source contract. Logicon’s undisclosed role as a leasing agent or reseller of Oracle products and as a provider of maintenance services raises additional questions 56 57 about the sole-source justification for the ELA. Oracle is not, in fact, the sole source, and other entities could fulfill the undis- closed roles assigned to Logicon. The ELA May Not Meet Statutory Requirements for a Sole-Source Exemption From Competitive Bidding Requirements After reviewing the ELA documents General Services provided us, our legal consultant advised us that a court might find that the ELA does not meet the legal requirements for a sole-source To justify a sole-source exemption from the competitive bidding requirements for purchase of IT goods and purchases of IT goods and services. Specifically, Public Contract services, the director Code, Section 12102, requires that General Services’ director make must either determine one of two determinations to justify a sole-source purchase of IT that the goods and goods and services. The director must either determine that the services are the only goods and services proposed for acquisition are the only goods ones that can meet and services that can meet the State’s need or that the goods the State’s need or that and services are needed in cases of emergency where immediate they are needed in cases acquisition is necessary for the protection of the public health, of emergency where welfare, or safety. General Services made neither of the required immediate acquisition determinations to justify the ELA as a sole-source contract. is necessary to protect the public. Also, Executive Order W-103-94 (executive order) directs state departments to obtain written approval from the responsible cabinet-level agency secretary or the highest-ranking full- time employee of the organization to enter any sole-source contract. General Services’ normal administrative practice is to obtain that approval prior to execution of a sole-source contract. However, General Services had already approved the ELA contract with Oracle on May 31, 2001, when it presented the State and Consumer Services Agency’s secretary (agency secretary) with the formal request for sole-source contract approval on July 18, 2001. The agency secretary’s designee signed the requested approval on August 21, 2001, nearly three months after General Services accepted the assets and incurred the financial liability under the ELA contract for which it sought approval. The July 2001 sole-source justification and request for approval General Services submitted focused exclusively on the economic benefits of entering into a sole-source contract with Oracle. For example, in justifying why the acquisition was confined to a single supplier, General Services described the Oracle proprietary software package as including “help desk” services such as distributing software and updates, and tracking software 58 59 license usage for better asset control, thus reducing the reporting burden for state employees. General Services also touted the administrative benefit of being able to eliminate annual contract negotiations by individual departments and allowing for software version control by a statewide release of updates. To justify choosing Oracle, General Services stated that Oracle’s ELA proposal would allow the State to leverage its purchasing power as a single entity, receive desirable database maintenance pricing, and be eligible to buy other Oracle products at a 50 percent discount over a five-year period. However, none of the documents our legal consultant reviewed showed that General Services’ director had found that the goods and services it proposed to acquire were the only ones that could meet the State’s needs or that an emergency required the procurement. Nor did the documents disclose that not only Oracle but also Logicon would be providing the goods and services of which Oracle was supposedly the only source. While it has developed a sole-source request form and a procedure for seeking prior authorization from the agency secretary, General Services did not follow those procedures when it sought authorization for the Oracle ELA. General Services’ description of the software package being proprietary suggests that it may have been relying on an example included in the State Administrative Manual (SAM) for when IT sole-source procurements might be justified. The SAM example covers when the lease or purchase of proprietary software is available Although it may be only from a single source. According to General Services’ legal necessary to enter a counsel, in relying on this SAM example, General Services sole-source contract with met the requirement that the director find that the goods and Oracle for an enterprise- services it proposed to acquire from Oracle were the only ones wide license for all that could meet the State’s needs. However, because all software existing Oracle database is proprietary, the SAM also recognizes that competition exists licenses being used by if multiple distributors can provide the software. For example, the State, in the absence manufacturers and resellers (such as Logicon) offer commercial of a statewide standard off-the-shelf software through CMAS or through one of the establishing Oracle as State’s three master agreements for IT products. Additionally, the the State’s database enterprise license agreement attempts to anticipate the State’s software, such sole- future need for database software, a need that may very well be source justification would met by other vendors or similar software products. Although it not hold true for future may be necessary to enter a sole-source contract with Oracle unknown needs. for an enterprise-wide license for all existing Oracle database licenses being used by the State, in the absence of a statewide standard establishing Oracle as the State’s database software, such sole-source justification would not hold true for future unknown needs. 58 59 In sum, while General Services’ sole-source justification explains its view of enterprise licensing benefits, our legal consultant advised us that the explanation does not appear to fit the General Services’ current restrictions on sole-source procurement. Also, General justification for this Services’ justification for this sole-source procurement reflects procurement indicates its current lack of procedures to achieve the policy goals of that it relied on competitive bidding in such a negotiated procurement. For information from Oracle example, its justification for this procurement reflects that for the State’s estimate of General Services relied on information from Oracle, both for the future needs for database State’s estimate of future needs for Oracle’s enterprise database licensure and for its licensure and for its estimate of future savings. estimate of future savings. General Services Lacked Authority to Create an ELA Sole-Source Procurement According to its counsel, General Services believes the Public Contract Code gives statutory authority to create ELA contracts. General Services believes that if an ELA contract is obtained consistent with competitive bidding requirements, including any exceptions to those requirements, then General Services has authority to enter the ELA contract. However, our legal consultant found no specific statutory authority for ELAs. Public Contract Code, Section 12101.5, authorizes acquisition methods compatible with the State’s short- and long-term fiscal needs, including multiple awards, master service agreements, and procurements with vendors having multiple award schedules. Except for certain multiple awards authorized by the above statute, all the requirements of competitive bidding apply to these alternative acquisition methods. The Public Contract Code provisions that General Services’ counsel pointed to as authorizing an ELA all refer back to acquisitions that require competitive bidding. Our legal consultant advised that these sections might authorize General Services or DOIT to consolidate existing, competitively awarded contracts that require annual negotiation of maintenance contracts and to enter into a volume purchase agreement with Oracle for those services. However, General Services’ sole-source justification memo to the agency secretary states that Oracle did not offer acceptable terms to enter into a volume purchase agreement. According to our legal consultant, since the sections General Services’ counsel pointed to assume compliance with Public Contract Code provisions that require competitive bidding, these sections do not appear to authorize the State to 60 61 enter a sole-source contract for an enterprise license for a given edition of proprietary software for every employee of the State, even if the contract would save the State money. Moreover, our legal consultant found no corresponding policy or procedure that would describe how an assumption of projected need or savings would accomplish the purposes of competitive bidding. Our legal consultant advises that although courts give great weight to the interpretation of a statute by officials charged with its administration, final responsibility for interpreting the law rests with the courts. Thus, if a court determines that a state department’s administrative action, including entering contracts, is not authorized by statute, the courts will find that action void. Given the lack of any specific statutory authorization or framework for the State to enter into an ELA, our legal consultant concluded that it is uncertain that General Services’ interpretation of the Public Contract Code would be upheld in court. Further Analysis of the ELA Contract Is Necessary to Determine the Impact on the State if the Contract Is Found Void If, indeed, General Services did not comply with statutory requirements for sole-source contracts, a court might find that its contract for the ELA is void. California courts have If General Services did not determined that public contracts executed in violation of comply with statutory statutes or regulations requiring competitive bidding are requirements for sole- void. Our legal consultant advises that California law appears source contracts, a court to currently prohibit payments on void contracts. might find that the ELA contract is void. According to our legal consultant, even when the State acts in the good faith belief that the contract is exempt from the competitive bidding requirement and the contractor performs in good faith, it is debatable whether the contractor may recover any amount on some equitable basis for performing the contract. No California decision permits such recovery, but commentators have advocated it and argued that the question is still open. However, on February 4, 2002, in Amelco Electric v. City of Thousand Oaks, the California Supreme Court suggested its approval of prior cases not permitting contractors to recover when contracts fail to meet competitive bidding requirements. Although California law may prohibit payments on void contracts, a finding that the contract for the ELA is void would raise additional questions about the impact on the best interests 60 61 of the State. Further legal analysis of other provisions of the ELA and the financing arrangement with Oracle is required to make that determination. For example, even if the courts find that the contract for the ELA is void, it is likely that Koch Financial would assert that the State still owes it the $52.3 million incurred under the financing arrangements with Oracle.3 (Appendix C gives an in-depth description of the financing arrangements.) Logicon, the State’s designated lender, immediately assigned its rights under the financing provisions of the contract to Koch Financial, and Koch Financial was to pay Oracle for the capacity to license up to 270,000 users of database software, one year of maintenance support, and sales tax. Also, while the contract stipulates that the State has no obligation to pay for any portion of the assets before it accepts them, the August 31, 2001, amendments state that the financed assets—the database license and one year’s support—have been delivered to and accepted by the State as of May 31, 2001. Moreover, the State should anticipate that Koch Financial would claim that it has already incurred the full cost of the database license and maintenance support. In addition, General Services’ senior staff counsel provided a written opinion to Logicon and Koch Financial stating the contract is valid and binding on the State. According to our legal consultant, if a court agrees with this opinion, the State apparently owes the $52.3 million that Koch Financial financed. In view of these provisions, the State should anticipate that Koch Financial would assert that it has independent rights to payment and therefore the State is obligated to it for the entire $52.3 million, even if the contract with Oracle is void and unenforceable. If this position were valid, the State might have to recover from Oracle and Logicon the $52.3 million that Koch Financial loaned to finance the purchase. According to the terms of the financing provisions, if the Legislature does not appropriate funds in a future fiscal year, the State is obligated to pay only for charges incurred through the end of the preceding fiscal year. However, our legal counsel advises that if a court found the State breached the financing provisions by not paying Koch Financial, a court might encumber state funds to repay that loan, even if the Legislature refuses to appropriate funds to repay the loan and the State 3 Our legal consultant did not give an opinion on the validity of General Services’ financing arrangements with Logicon, and then Koch Financial, which ultimately financed the acquisition of the Oracle enterprise database licensure. 62 63 terminates the contract. Additionally, the failure to appropriate funds to repay the loan might have a negative impact on the State’s credit rating. Also, if the Legislature does not appropriate funds to repay the loan or the ELA contract is not otherwise funded by the State and it is, as a result, terminated, certain terms and conditions of The State agrees not to the ELA contract amendment could seriously affect the State’s replace the Oracle license ability to use the Oracle software license. When General Services with substantially similar and Oracle amended the contract on August 31, 2001, they database licensure for revised the standard financing provisions to expressly state that one year if the State if the State exercises its right to terminate the contract because exercises its right to the Legislature does not appropriate funds for the ELA or the terminate the contract. ELA contract is not otherwise funded by the State, the State agrees to stop using the ELA’s enterprise database licensure. More importantly, the State also agrees not to replace the Oracle license with substantially similar database licensure for one year from the termination date to the extent the law permits. Successful enforcement of this provision could effectively shut down many departments’ operations. Curiously, in the August 31, 2001 amendment, the State appears to agree that if funds are not appropriated for the financing for other separate license contracts Oracle has with Justice and the Department of Consumer Affairs, the State will not permit those agencies to use the ELA licenses for a period of one year following that event. While we identify these provisions as problematic, further legal analysis is required to understand the impact of these provisions and the contract as a whole on a finding that the contract is void. LOGICON’S ROLE RAISES ADDITIONAL QUESTIONS ABOUT THE VALIDITY OF THE ELA CONTRACT The amount of compensation Logicon apparently has received and will continue to receive under side agreements with Oracle raises questions about its actual role in the ELA and the impact of that role on the validity of the sole-source justification. (Appendix C gives an in-depth description of the side agreements.) The total compensation appears to go well beyond what would be reasonable for Logicon’s disclosed role as lender. In fact, the percentage of the total contract price that Logicon will apparently receive suggests Logicon is acting as a reseller of Oracle products and services. Because Logicon is not 62 63 the only reseller of these products and services, it is unlikely that a state agency could have justified a sole-source procurement. In fact, by definition, at least two sources were available—Oracle and Logicon. Since the ELA was executed as a sole-source contract between General Services and Oracle with Logicon named as the contract’s lender—a role it assigned to Koch Financial—it is confusing to find that Logicon had a significant role and compensation apart from that of a lender. Oracle’s apparent assignment to Logicon of rights to payments under the ELA contract is confusing because any assignment under the contract is subject to General Services’ prior written consent. We have seen no such written authorization, and according to the counsel for General Services, none was given. Nor is there documentation approving Logicon as a subcontractor or leasing agent. Yet it appears that despite the “noncollusion Despite the express affidavit” in the ELA, which prohibits Oracle from offering a provision in the ELA price to benefit an undisclosed entity, the price Oracle gave for contract that prohibits the enterprise database license was in part for Logicon’s benefit. Oracle from offering a price to benefit an According to correspondence we received from Oracle, “Oracle undisclosed entity, the did not pay Logicon for its role in the Contract.” However, price Oracle gave was in an examination of the side agreements reveals that Koch part for Logicon’s benefit. Financial apparently did pay Logicon at Oracle’s direction—as the result of a confidential agreement between Oracle and Logicon on disposing the loan proceeds. Moreover, one of the side agreements contemplates that General Services will make maintenance payments to Logicon, which will in turn pay Oracle its designated share. The ELA contract between General Services and Oracle states that the contract cannot be assignable in whole or in part without the State’s written consent. Further, the noncollusion affidavit states that Oracle certifies any quotation provided is not made in the interest of or on behalf of any undisclosed person, partnership, company, association, organization, or corporation. As originally drafted, the ELA contract incorporated the terms of a standard loan agreement for installment purchases made by the State and designated Logicon as the State’s lender. The lender is entitled to all payments owed in the payment schedule in consideration of the lender’s paying the assets’ costs directly to the supplier (Oracle). The payment paragraph goes on to state that “through a third party assignment between Logicon, Inc. and Koch Financial Corporation, [Lender] has assigned all its 64 65 rights, title and interest in, to and under this Agreement,” including the right to receive payments, to Koch Financial as the assignee. In contrast to the side agreement between Oracle and Logicon, the ELA contract itself represents that Logicon is the lender and does not disclose any other role for this entity. Documents by and The contemporary documents by and between the State’s between the State’s representatives do not show the State was aware of Logicon’s representatives do not expanded role or the magnitude of Logicon’s compensation. show that the State According to information from Oracle, Logicon is to perform was aware of Logicon’s the enterprise license support desk duties but otherwise has expanded role or the acted only as the State’s “Lender.” Moreover, Oracle represented magnitude of Logicon’s to us that it had not paid Logicon for its role in the ELA. Yet compensation. in a May 31, 2001, Payor Addendum, and an August 31, 2001, Assignment of Payments/Payment Direction, Oracle effectively assigned significant consideration to Logicon that it would have otherwise received from Koch Financial. The State’s records relating to the ELA sole-source purchase suggest that General Services had no knowledge of this greater role played by Logicon. General Services’ July 2001 request for sole-source approval (approved in August 2001) makes no mention of Logicon. On July 16, 2001, the Legislative Analyst’s Office (LAO) asked General Services several questions, including why Logicon was selected to be the State’s designated lender for this contract. Oracle suggested in an e-mail that General Services respond as follows: “Because of Logicon’s experience with large enterprise license transactions with the federal government, and the financial backing of their $15 billion parent company, Northrop Grumman, Logicon was the natural choice for this contract.” General Services repeated Oracle’s suggestion verbatim in its answer to the LAO, except to replace “Logicon was the natural choice for this contract” with “Logicon was selected by the State and Oracle to be the lender of this contract.” General Services gave no other explanation of Logicon’s role, except as lender. Indeed, the first mention of Logicon’s role in connection with maintenance appears in General Services’ July 2001 draft enterprise license ordering instructions. The draft instructions list Logicon as the contractor to contact for ordering Oracle database products and gives Logicon’s address, phone, and e-mail for the enterprise license support desk. 64 65 A management memo dated September 17, 2001, informing state departments about the ELA between the State and Oracle, does not refer to Logicon. Rather, it directs users to an enterprise license support desk and gives a phone number. Oracle has told us in correspondence that Logicon will perform the duties of the enterprise license support desk. Although the State may have been unaware of these side agree- ments, a question arises whether the State and Oracle could legitimately enter a sole-source agreement if Oracle was not, in fact, the sole source of the contract’s license. Logicon’s Erroneous Savings Projections May Make the Contract Voidable As discussed in Chapter 1, we arrived at vastly different numbers in reviewing the data that supports the costs and projections that Logicon presented to the State in an effort to convince it to sign the Oracle ELA contract. For example, although Logicon projected that the State would save as much as $16 million Although Logicon during the first six years of the contract, using Logicon’s data projected the State and assumptions, we project that the State could spend as would save as much much as $41 million more than it would have without the as $16 million during ELA. This wide disparity demonstrates at a minimum a lack the first six years of the of diligence by Logicon in preparing its representations about contract, using Logicon’s cost savings to General Services. Moreover, the State accepted data and assumptions, Logicon’s representations at face value and relied on them when we project the State deciding to enter the contract. If the flaws we found in Logicon’s may spend as much as projections were made with knowledge or without enough $41 million more than knowledge of the subject matter to support the representations it would have without that it made, the State may have a basis under state law for the ELA. challenging the ELA’s enforceability. For example, if the flaws in the representations Logicon made to the State on Oracle’s behalf were found to rise to the level of civil fraud, the ELA contract may be voidable. In cases of intentional misrepresentation, the State may have additional remedies, including those under the False Claims Act, which imposes civil penalties for false claims made for public moneys. The fact that Logicon, the contract’s named lender, prepared the costs and savings projections for General Services and did not disclose its additional roles under the ELA, may also call into question the validity of the financing provisions. These issues require additional study of the facts and complex legal analysis to arrive at definitive conclusions. 66 67 RECOMMENDATIONS Before negotiating any future enterprise licensing agreements, General Services should assemble a negotiating team that pos- sesses all the types of expertise necessary to protect the State’s interests. Ultimately, only the courts can resolve the legal issues we have identified. Nonetheless, for the various legal issues we have identified, the appropriate legal authorities must carefully analyze the impact of these issues on each other and then decide the course of action that protects the State’s best interests. To identify the legal measures to take to protect the State’s interests, we recommend that General Services do the following: • Continue to study the ELA contract’s validity in light of the wide disparities we identified in Logicon’ projections of costs and savings, and consult with the attorney general on how to protect the State’s best interests. • Work closely with the attorney general in further analyzing the ELA contract; all amendments, including any and all documents pertaining to the side agreements between Oracle and Logicon; and the laws and policies relating to the ELA, including the potential legal issues that this audit has identified. If the contract is determined to be enforceable, General Services should renegotiate to ensure it includes adequate protections for the State. The Legislature should consider requiring all IT contracts over a specified dollar amount to receive a legal review by General Services. 66 67 We conducted this review under the authority vested in the California State Auditor by Section 8543 et seq. of the California Government Code and according to generally accepted government auditing standards. We limited our review to those areas specified in the audit scope section of this report. Respectfully submitted, ELAINE M. HOWLE State Auditor Date: April 16, 2002 Staff: Doug Cordiner, Audit Principal Steven A. Cummins, CPA Gayatri Patel Amari B. Watkins Nicki Ruszczycky Consultants: Riegels Campos & Kenyon LLP Catalysis Group 68 69 APPENDIX A Chronology of Key Events Occurring Before and After the Enterprise Licensing Agreement The Department of General Services (General Services) entered into an enterprise licensing agreement (ELA) with Oracle Corporation (Oracle) on May 31, 2001. The ELA represented a culmination of events involving General Services, the Department of Information Technology (DOIT), and the Department of Finance (Finance), among others. These entities were attempting to use the State’s purchasing power to obtain large discounts on database software licenses from Oracle. Figure A.1 on the following pages is a chronology of key events that led up to the ELA and that have occurred since, based on documents and written statements we obtained from the three departments. 68 69 70 71 1.A ERUGIF ALE eht htiW detaicossA stnevE yeK fo ygolonorhC ������������ ������������������������������� ��������������������������� ���������� �������������������������������������� ���������������������������� ���������������������������� ����������������� ���������������������������� �������������������� ����������������������� ����������������������������� ������������� ������������������������� ���������������������������� ������������������������������� ����������������������������� ������������������������������ ���������������� ����������������������������� �������������������������������� ������������������������ ����������������� ��������������������������������� �������������������������� ���������������������������� �������������������������� ��������� ��������������������������������� �������������������� ��������������������������� ������������������������� �������������������������� ��������������������������������� ����������������������������� ���������� ���������������� ��������������������������������� ������������������������������ ��������������������������� ��������������� �������������������������������� ����������������������������� ������������������������ �������������������������� ��������������������������� ������������������������� ��������������������������������� �������������������������� ������������������������������� ����������������������������� ��������������������������� ������������������������� ��������������������������� �������������������������������� ������������������������������� ����������������������������� ������������������� ������������������������ �������������������� ������������������� ������������������������ ���������� ��������������������������� ���������������������������� ������������������������������� ������������ ��������������������������������� ��������������������������� ����������������������������� ����������������������������� ������������������������������� ������������������������ �������������������������� ��������������������������� ����������������� ����������������������������� ��������������������������� �������������������������������� �������������������������������� ������������������������������ ������������������������������� ������������������������������� ������������������������� ����������������������������� ��������������������������� ��������������������������������� �������������������������������� ������������������������ ����������������������� �������������� �������������������� ������������������������� ��������������������������� ������������������������������ �������������� ��������������� ������� ������������������������� ����������������������������� �������������������������������� ���������������������������� ��������������������� �������������������������� ������������������������������ ���������������������������� ������������������������� �������������������������������� ������������ ������������������������� ���������� ���������������������������� �������������������������������� ������������������������� ������������������������� ���������������������������� ������������������� ����������������������������������� ������������������������ ������������������������������� ������������������������������� ����������������������������� ����������������������������� ���������������������������� ���������������������������� �������������������������� ������������ ��������������� ���������������������������� ���������������� ���������� ���������� �������� ���������� 70 71 DEUNITNOC–1.A ERUGIF ���������� ���������������������������������� ���������������������������������� ���������� ������������������������������ ������������������������������� ������������������������������� ����������������������������� ���������� ���������������������������� ������������������������� ������������������������������� ������������������������ ������������������������������� ���������������������������� �������������������������������� ������������������������������� ������������������������������ ������������������������������ �������������������������� ����������������������� ��������������������������������� ����������������������������� ���������������������� ������������������������� �������������������������� �������������������������������� ������������������������� ����������������������������� ��������������������������������� ���������� ����������������������������� ������������������������������ ���������� ����������������������������� ��������������������������������� ���������������������������� ����������������������������� �������������������������������� ������������������������� ������������������������������� ������������������������������ ������������������������������ ������������������������������ �������������������������������� ������������������������� ������������������������� ��������������������������� ������� ������������������������������� �������������������������������� ����������������� �������������������������� ��������������������������� �������������������������� ���������������������� ������������������������ ��������������������������� ������������������������������� ���������������������������� ��������� ������������������������������ ������������������������������ ����������������������� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � �� �� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � �� � � � � � � � � � � � � � � � � � � � � � �������������������������� ���������� ���������� ����������������������������������� ����������������������������� ���������� ������������������������������ ��������������������������� ������������������������������ ��������������������������� ����������������������������� �������������������������� ������������������������� ���������������������������� ������������������������������ �������������������������������� ������������������������������� ����������������������������� ������������������������������� ��������������������������� ����������������������� ��������������������������� ����������������������� ������������������������������� �������������������������������� ������������������������������� ����������������������������� ������������������������������� ��������������������������� ������������������������������� ��������������������������� �������������������������� ���������������������������� ��������������������������� ��������������������������������� ������������������������������� ��������������� �������������������������� ��������� ��������� ������������ ���������� egap txen no deunitnoc 72 73 DEUNITNOC–1.A ERUGIF ������������� ������������������������������������ ����������������������������� �������������������������������� ����������������������������������� ��������������������������������� ��������������������������������� �������������������������������� ������������������������������������ ��������������������������� ������������������������������������� ���������� ����������������������������� ������������������������������ ��������������������������������������� �������������������������������� ���������������������������������� ����������������������������� ����������������������������������� ����������������������������� ������������������������������ ����������������������� ����������������������������������� ���������������������������� ��������������������������������� ��������������������������������� ������������������������������ ���������������� ����������������������������� ���������������������������������� ������������������������������ �������������������������������� ����������������������������� ������������������������� �������������������������������� ��������������������������� ������������������������������������ ����������������������������� ����������� ������������������������������� �������������������������� ���������������������������������� ���������������������������� ������� �������������������������� ��������������������� ���������������������������� ����������� ����������� �������������� ���������������� ��������������������������������� ����������������������������� �������������������������������� ������������������������������� ����������� ������������������������������� ���������������������������� ��������������������������������� ������������������������������� ����������������������������� ������������������������������ ��������������������������������� ����������������������������� ���������������������������������� ������������������� ����������������������� ������������������������ �������� ���������� ���������� APPENDIX B Projected Cost of the Enterprise Licensing Agreement According to projections prepared by Logicon Inc. (Logicon), the State will save approximately $111 million if it extends the six-year term of its ELA contract with Oracle Corporation (Oracle) to include four additional years of maintenance service. The $94.6 million ELA contract provides an enterprise license authorizing up to 270,000 users of Oracle database software plus maintenance support. However, as is shown in Tables B.1 and B.2 on the following pages, Logicon significantly overstated the savings the State can expect to receive over the life of the ELA contract. Table B.1 shows Logicon’s projected savings. However, as shown in Table B.2, rather than realize these savings, the State could spend approximately $41 million more than it would by the end of the first six years if the ELA did not exist and almost $6 million more if it elects to receive the four years of maintenance options. 72 73 74 75 1.B ELBAT sgnivaS detcejorP s’nocigoL )snoilliM nI( sraeY ecnanetniaM lanoitpO doireP tcartnoC slatoT slatotbuS 11–01 01–90 90–80 80–70 slatotbuS 70–60 60–50 50–40 40–30 30–20 20–10 raeY lacsiF :gnidneps detcejorP weN 0.87 $ 2.13 $ 8.7 $ 8.7 $ 8.7 $ 8.7 $ 8.64 $ 8.7 $ 8.7 $ 8.7 $ 8.7 $ 8.7 $ 8.7 $ *sesnecil weN 0.71 8.6 7.1 7.1 7.1 7.1 2.01 7.1 7.1 7.1 7.1 7.1 7.1 †ecnanetniam deweneR 6.831 0.58 5.52 6.22 8.91 1.71 6.35 6.41 2.21 9.9 7.7 6.5 6.3 ‡ecnanetniam latoT gnidneps 6.332 0.321 0.53 1.23 3.92 6.62 6.011 1.42 7.12 4.91 2.71 1.51 1.31 detcejorp ALE )6.221( )0.82( )0.7( )0.7( )0.7( )0.7( )6.49( )1.42( )4.12( )7.81( )3.61( )1.41( — stnemyap launnA 0.111 $ 0.59 $ 0.82$ 1.52$ 3.22$ 6.91$ 0.61 $ — 3.0 $ 7.0 $ 9.0 $ 0.1 $ 1.31$ sgnivas .noisiviD tnemerucorP ,secivreS lareneG fo tnemtrapeD :ecruoS nocigoL .0002–9991 raey lacsfi ni sa etar emas eht ta sraey 01 txen eht rof yllaunna sesnecil esabatad elcarO esahcrup ot eunitnoc lliw etatS eht taht semussa noitcejorp s’nocigoL * .stcudorp elcarO rof sredro esahcrup latnemtraped morf atad desu .)noillim 8.7$( seef esnecil fo tnecrep 22 ot tnuoma yllaunna desahcrup sesnecil no ecnanetniam raey-tsriF † ,raey lacsfi tnerruc eht ni desahcrup sesnecil wen no ecnanetniam sulp ,raey lacsfi tsap eht fo 03 enuJ fo sa gnitsixe esab esnecil eht no ecnanetniam sevlovni ecnanetniam deweneR‡ .tnecrep 5 fo rotcaf noitalacse ylraey a yb desaercni dna 74 75 2.B ELBAT gnidnepS etatS detcejorP fo noitalipmoC s’nocigoL fo noitaluclaC desiveR s’rotiduA )snoilliM nI( sraeY ecnanetniaM lanoitpO doireP tcartnoC slatoT slatotbuS 11–01 01–90 90–80 80–70 slatotbuS 70–60 60–50 50–40 40–30 30–20 20–10 raeY lacsiF :gnidneps detcejorP weN 0.64 $ 4.81 $ 6.4 $ 6.4 $ 6.4 $ 6.4 $ 6.72 $ 6.4 $ 6.4 $ 6.4 $ 6.4 $ 6.4 $ 6.4 $ sesnecil weN 0.01 0.4 0.1 0.1 0.1 0.1 0.6 0.1 0.1 0.1 0.1 0.1 0.1 ecnanetniam deweneR 0.16 6.04 5.21 9.01 3.9 9.7 4.02 5.6 2.5 9.3 7.2 6.1 5.0 ecnanetniam latoT gnidneps 0.711 0.36 1.81 5.61 9.41 5.31 0.45 1.21 8.01 5.9 3.8 2.7 1.6 detcejorp ALE )6.221( )0.82( )0.7( )0.7( )0.7( )0.7( )6.49( )1.42( )4.12( )7.81( )3.61( )1.41( — stnemyap launnA sgnivas )6.5( $ 0.53 $ 1.11$ 5.9 $ 9.7 $ 5.6 $ )6.04($ )0.21($ )6.01($ )2.9( $ )0.8( $ )9.6( $ 1.6$ )tsoc dedda( .srorre rof detcerroc ,snoitpmussa dna snoitatupmoc s’nocigoL gnisu snoitaluclac rotiduA :ecruoS redro esahcrup eht fo noitcerroc dna weiver ruo no desab era srebmun ruo ,revewoH .snoitcejorp sti ni desu nocigoL taht ygolodohtem dna snoitpmussa emas eht desu eW :etoN .noitucexe s’ALE eht ot roirp etatS eht ot dedivorp dna nocigoL yb delipmoc atad 76 77 APPENDIX C The State’s Enterprise Licensing Agreement and Logicon’s Related Side Agreements With Oracle On May 31, 2001, the Department of General Services (General Services) and Oracle Corporation (Oracle) executed an enterprise licensing agreement (ELA), establishing contractual terms for purchasing Oracle Enterprise Edition 8i database software licenses (enterprise database licensure) and related maintenance and terms for the financing of both for the contract’s first year. Under the ELA, the State agrees to pay $94.6 million for enterprise database licensure for up to 270,000 users (state employees and contractors) and a specified level of annual technical support and maintenance services for the period May 31, 2001, through May 31, 2007. In addition to the enterprise database licensure, the State acquired 100,000 universal power units for Internet use. The ELA contract also grants state and local government agencies a five-year special discount of 50 percent on additional Oracle products through a volume purchase-pricing clause. The ELA contract gives the State an option to extend the term of the contract, for maintenance and technical support only, for an additional four years to May 30, 2011. Including this optional extended maintenance, the ELA contract is potentially worth $122.6 million to Oracle. The financing vehicle for purchasing the enterprise license and first-year maintenance costs in the amount of $52.3 million is reflected in the underlying financing provisions. The State’s designated lender is Logicon Inc. (Logicon), and the financing provisions acknowledge that Logicon will assign all its rights to the loan proceeds to Koch Financial Corporation (Koch Financial). According to the financing provisions as amended on August 31, 2001, the rights and obligations of Oracle and the State under the ELA contract are separate and independent of the rights and obligations of the State and Koch Financial under the financing provisions. Unlike the California Multiple Award Schedules (CMAS) program provisions that the ELA incorporates, the financing provisions state that the ELA contract may not be terminated for convenience. Those provisions also say that the 76 77 State may not elect to prepay any portion of the loan’s unpaid balance, which is a modification to the standard language used in the State’s financing plan for installment purchases. According to the financing provisions, if funds are not appropriated by the Legislature in a future fiscal year, the State does not incur further obligation and is only obligated to pay all charges “incurred” through the end of the preceding fiscal year. The financing provisions stipulate that the State has no obligation to pay for any portion of the assets before it accepts them. The financing provisions further provide that the financed assets, the “license of software and . . . the acquisition of capitalizable support functions,” have been delivered to and accepted by the State as of May 31, 2001. The State acknowledges that it has directed Logicon—its primary lender— to pay Oracle for the assets and agrees to pay all sums due to the lender as set forth in the payment schedule. On August 31, 2001, the parties amended the ELA contract’s special provisions, which, by the contract’s terms, take precedence over the general terms and conditions. According to General Services’ counsel, the intent of the amendment was, among other things, to make the enterprise database licensure perpetual upon Oracle’s receipt of the loan proceeds. The amendment to the ELA contract also clarifies that, after the first year, the maintenance part of the State’s payments are to be made to Oracle, while the asset part of the payment, in other words the loan payment, is to be paid directly to Koch Financial. It also restates that Logicon, as the lender, has assigned all its rights, title, and interest in the ELA contract to Koch Financial. Side Agreements Between Oracle, Logicon, and Koch Financial Oracle and Logicon entered into agreements at the time of and subsequent to the ELA’s execution. These side agreements relate to performance of the ELA contract, affecting who receives public funds financed through the financing provisions and the splitting of payments made under the ELA contract beginning in the second and subsequent years of its term. On May 31, 2001, Oracle and Logicon entered into two separate but related agreements. The first is a service provider agreement by which Logicon would provide “expert onsite support services” and other services as an independent 78 79 contractor of Oracle. The agreement obligates Logicon to immediately deliver to General Services all Oracle enterprise database licenses delivered to it under a Logicon purchase order to Oracle dated May 31, 2001. Consideration is “acknowledged as received by Logicon,” as indicated in the assignment of payment obligations set forth in the ELA contract. The second agreement between Logicon and Oracle that apparently took effect on May 31, 2001, is a payor addendum between Oracle and Logicon “pursuant to terms and conditions identical to the ELA.” This addendum is marked as containing “confidential financial information protected from disclosure.” The addendum identifies Logicon as having been “designated the preferred leasing agent for use under General Services Order,” stating that General Services has consented to the assignment by Oracle and General Services of all future payment obligations to Logicon. The addendum recognizes that Logicon will have a separate financing agreement with Koch Financial as the assignee of Logicon’s rights and duties as designated lender. Also, the addendum sets out the amounts Logicon will pay to Oracle; the difference between the State’s payments to Logicon and Logicon’s payments to Oracle is Logicon’s apparent consideration for acting as leasing agent and for providing onsite support services. Finally, the addendum includes an agreement between Logicon and Oracle that they disclose the payment terms only to General Services. In addition, on August 31, 2001 pursuant to a letter agreement signed by Oracle, Koch Financial, and Logicon, the parties entered into an assignment agreement concerning the ELA contract’s assignment of payments. That letter agreement also addresses payment direction and Logicon’s rights to proceeds under the financing agreement. The assignment agreement directs Koch Financial to pay Logicon the $52.7 million in loan proceeds and accrued interest in exchange for Logicon’s rights, title, and interest in the financing provisions of the ELA contract with the State and the financed assets covered by the ELA. In the letter agreement, Oracle directs Logicon to pay from the loan proceeds the sums due Oracle under the May 31, 2001, payor addendum for the capitalized assets and accrued interest amounting to $36.5 million. According to our legal consultant, the side agreements would require the written consent of the State to be binding on the State. According to its counsel, General Services had neither seen any of these side agreements nor has Oracle or Logicon sought the consent of General Services to these apparent modifications to the ELA contract. 78 79 This page left blank intentionally for reproduction purposes only. 80 81 Agency comments provided as text only. Department of Information Technology 801 K Street, Suite 2100 Sacramento, CA 95814 April 5, 2002 Elaine M. Howle* State Auditor Bureau of State Audits 555 Capitol Mall, Suite 300 Sacramento, CA 95814 Dear Ms. Howle: Attached is the Department of Information Technology’s (DOIT) response to your April 2, 2002 draft audit report on the State’s contracting practices for enterprise licensing agreements. As our response indicates, DOIT concurs with the findings and recommendations in your report. I am thankful for your review, and for the opportunity to respond to the draft report. The BSA recommendations will be very helpful in improving the management of current and future software volume purchases. If you need additional information, please do not hesitate to contact me or Robert Dresser, Chief Counsel, at (916) 445-3050. Sincerely, (Signed by: Elias S. Cortez) ELIAS S. CORTEZ Chief Information Officer State of California *California State Auditor’s comments begin on page 89. 80 81 On April 4, 2002 the Department of Information Technology (DOIT) provided a written response to the Bureau of State Audits (BSA) pursuant to a discussion held during the April 2, 2002 exit inter- view for the Contracting Practices Audit. We would appreciate your incorporating by reference the April 4, 2002 response as though it were fully set forth in this document. (Exhibit 1) DOIT very much appreciates that the BSA Auditing team listened carefully and attentively to DOIT’s comments in the exit interview and as a result made certain changes to the draft report. We apologize in advance if our current response covers materials that have already been changed as a 1 result of the exit interview. Recommendations of BSA We fully concur with those recommendations set forth on page 49 of the draft report. Specifically, we concur that legislation is needed to clarify the roles of DOIT, the Department of Finance (DOF) and the Department of General Services (DGS) with respect to each of the department’s functions as they relate to the development of an Enterprise Licensing Agreement (ELA). We have also pro- posed administratively that the ELA process be clarified so that it applies only to a Statewide ELA (SELA). We are confident that a clearer definition of each department’s role in developing, negoti- ating and executing a SELA will result in a vehicle which will be sure to bring substantial savings to the State by using a statewide licensing agreement to leverage the State’s purchasing power. DOIT has sent proposed legislative reforms to Assembly Member Diaz for utilization in delibera- tions regarding Assembly Bill 1559 (AB 1559). Some of these reforms deal with the process of clarifying the respective roles and responsibilities of DOIT and DGS related to procurement. We anticipate working closely with various legislators to bring about these positive changes during the legislative session later this year. DOIT is working together with DGS and DOF to establish admin- istratively an ELA or SELA process that protects the State’s interests, discussion between the three departments are on-going and further collaboration is needed with the Governor’s Office before final agreement is reached on the creation of a new Statewide ELA process. DOIT agrees that the State should have conducted additional analysis and validation in the devel- opment of the Oracle ELA. We do not, however, believe that then-existing DOIT procedures used 1 to evaluate the Information Technology (IT) proposals submitted by individual State departments were the appropriate vehicles for evaluating the Oracle ELA or future ELAs or SELAs. Our Deputy Director for the Project Review and Oversight Division, Roy McBrayer, made the following points in this regard during the exit interview: • The successful development of a SELA for software is a fairly complex task. The State has little experience in this area and is continuing its research to determine how to best manage the process of developing a SELA. • While the execution of the Oracle Statewide ELA might have been better, it is erroneous 1 to suggest that DOIT could use the IT Project Review and Oversight process it currently has in place to evaluate potential SELAs. 1 • The IT Project Review and Oversight process is specifically tailored to the implementa- tion of IT systems, not to the acquisition of an enterprise license agreement. 82 83 Based on our experience with the Oracle ELA, DOIT is working jointly with DGS and DOF to develop a process for negotiating a Statewide ELA. It is a new domain for the State and is worthy of deeper analytic study before any decision is made about the ultimate process and structure. After thorough study and development, any proposed Statewide ELA process should be formalized either legislatively or administratively before the State enters into any future Statewide ELAs. DOIT’s Legislative Mandate As we discussed during the Exit interview, DOIT’s legislative mandate set forth in Government Code Sections 11700, 11701 and 11710 includes, in part, that DOIT provide guidance and leader- ship to State agencies in identifying, designing and implementing IT applications and, where fea- sible, promote phased implementation and funding of large and complex projects. In bringing the 2 idea of the Oracle ELA to the State, DOIT sincerely believed that it was complying with its legisla- tive mandate and performing its appropriate role. A Statewide IT Inventory DOIT also fully concurs with BSA’s recommendation that DOIT needs to continue its efforts to establish a statewide IT inventory, which includes software. In that regard, and in response to com- ments on page 34 of the draft BSA report, DOIT respectfully wishes to bring to your attention the following: • The statement that the State did not begin to initiate the process to obtain an inventory 3 of software until November 2001, is erroneous. • Executive Order D-10-99, issued in June 2001, was written to promote the legal use of licensed software and best practices in software management. This executive order detailed several requirements that were phased in to implement a controlled software management program within the State. A DOIT Management Memo, No. 01-10, was issued in June 2001, which required departments to comply with Executive Order D- 10-99. • The first phase required that by January 31, 2002, departments submit to DOIT a Soft- ware Management Plan substantiating compliance with DOIT’s software management policies. • The second phase required that departments submit to DOIT a report by January 31, 2003, detailing how the Software Management Plan was implemented, and to include a baseline inventory of all software as evidence of effective software management. DOIT deemed it reasonable to allow departments one year to implement their Software Man- agement Plans and complete their software inventories. • The third phase required departments to certify annual compliance and update software inventories. • This phased process will ensure implementation of an effective statewide software management program within reasonable timeframes. DOIT respectfully suggests that 3 the BSA change its observation to acknowledge DOIT’s initiation of the process within six months of the Executive Order. 82 83 Logicon Report Addendum During our presentation at the exit interview, BSA agreed that DOIT never received the Addendum to the Logicon report of July 14, 2000. DOIT respectfully suggests that DOIT, therefore, should not 1 be held responsible for failing to follow or take into consideration certain findings or recommenda- tions contained in the Addendum. We addressed this matter in the April 4, 2002 written response to BSA’s request and have attached a copy of that as Exhibit 1. Comments on Specific Pages of BSA’s Draft Report Our preliminary comment is that the draft report contains a number of speculative comments such 4 as the one appearing on the title page where it is stated that the Oracle ELA could “possibly” cost millions of taxpayers dollars. In fact, efforts are currently underway to identify all pre-existing State Oracle contracts with the intention of folding those contracts in the ELA, obtaining credit for these contracts, and ultimately realizing the anticipated value of the ELA. Suggested Additional Clarifications The following comments refer to areas where DOIT believes that further modification or clarification should be considered. (Here, the referenced pages tie to the pagination of the draft report shared with DOIT prior to the exit interview.) Again, DOIT apologizes in advance if our current response covers materials that have already been changed as a result of the exit interview. Page 3 The DGS/DOIT Survey 5 The preliminary survey referred to in the BSA draft report was jointly conducted by DGS and DOIT. Since several State departments affirmatively responded to the survey indicating that they would 6 need new Oracle products during the next six months, we believe that the term “relatively few State workers” might be misleading. 7 We do not agree with the comment on page 3 that DOIT made no other efforts to assess the State’s need for Oracle software. We discussed during the exit interview the fact that the Enterprise Workgroup, consisting of State Chief Information Officers (CIOs), had brought to DOIT’s attention the need for the State to leverage its purchasing power when buying IT. The Enterprise Workgroup firmly believed that independent purchases of IT by individual State departments failed to utilize the great potential for leveraging purchasing on a statewide basis. The only additional software evaluation required prior to establishing a Statewide ELA pertains to the current and continuing need for the product, the present and future value, how quickly the prod- uct will become obsolete, variations in license types and support services and whether a reduced price can be obtained by consolidating existing contracts into a single agreement. 7 DOIT respectfully suggests that BSA modify its characterization of DOIT’s technological evaluation of the Oracle ELA and note what this evaluation consisted of and when it was performed. 84 85 Page 4 The Flexibility of State IT Projects In the discussion during the exit interview, we questioned the conditional statement on Page 4 that 8 there is a perception that there exists a de facto standard, which “may” reduce innovation and flex- ibility in State IT projects. We do not believe that there is sufficient evidence to conclude that such a perception exists on a widespread basis nor that if such a perception does exist that it “may” reduce the flexibility of State IT projects. The State currently uses a number of different software systems produced by a variety of manufacturers. The purpose of the Oracle ELA was to consolidate pur- chases by existing and future users of Oracle products, and to utilize the potential for the State to pool its purchases of Oracle products and receive a better price. Page 5 Appropriateness of a Review Process 1 For reasons stated above, DOIT does not believe that the then-existing review process for individual departmental IT acquisitions was the appropriate process for evaluating the Oracle ELA. 9 We question whether the Statement on page 5 beginning “without any apparent benefit” is accurate, in light of the current efforts to consolidate Oracle contracts as described above. Page 7 Additional Savings to the State Made Possible by the ELA 0 DOIT believes that potential savings to cities and counties, attributable to the Oracle ELA, may prove to be substantial and should be considered in assessing the ultimate value of the Oracle ELA. Page 8 1 We request that the bottom part of page 8 be modified to reflect that the draft report dated July 21, 2000 (different from the July 14, 2000 report received by BSA from Logicon) did not contain the Addendum. Page 19 Migration to the Oracle ELA 9 Page 19 does not take into consideration the proposed migration of other departments with Oracle contracts to the ELA, as discussed above. Departmental Interest in Oracle Products 6 DOIT does not believe that the phrase “limited interest in additional products” accurately represents the fact that several State departments had indicated a substantial interest in ordering at least $19 q million of Oracle products in the next fiscal year. Appropriate Protocol for Approving an ELA In light of the discussion during the exit interview, it is our understanding that the draft report will be 1 modified at the bottom of page 19 to reflect that DOIT does not agree that its IT Project Approval Process was the appropriate vehicle for approval of the ELA. Page 20 4 We disagree with the conclusion on page 20 that the State did not know that the ELA was an appropriate procurement of technology. For reasons discussed above, DOIT believes that the Oracle ELA was an appropriate purchase of software licenses. 84 85 Page 21 6 DOIT believes that several State departments as well as one county responded to the survey affir- matively, indicating a substantial interest in purchasing Oracle products in the next six months. Page 24 DOIT respectfully disagrees with the conclusion on page 24 that “DOIT ignored the signs.” The State 6q departments positively responding to our survey indicating an interest in purchasing $19 million worth of Oracle products and licenses in the next six months constituted evidence that there was a real interest in Oracle products, thus justifying an effort to pool State purchasing power in an ELA. Page 26 Migration to the Oracle ELA 9 We would hope that BSA might modify its findings on page 26 to reflect the substantial possibility that additional State departments will soon migrate to the Oracle ELA. Page 27 Evaluation Process for IT Proposals 1 For reasons previously discussed, we do not agree that the-then existing DOIT evaluation process for individual departmental IT proposals was applicable to assessing the need for an ELA. We therefore request that BSA delete from page 28 the statement that indicates that DOIT was inactive and failed to perform its role in properly assessing the need for the Oracle ELA. Page 29 1 We request that the use of the word “unfortunate” be deleted as it suggests a certain amount of blame on the part of DOIT regarding not using an Addendum which DOIT never received. In addi- tion, for reasons set forth in the April 4, 2002 DOIT written response, we believe the language on pages 29 and 30 should be modified or deleted. Page 31 Technological Appraisal For reasons previously set forth, DOIT requests that BSA modify page 31 to reflect the reasons why 1 DOIT did not use its then existing procedures to determine if there was a sufficient need to justify an ELA. It is DOIT’s position that DOIT did in fact do the necessary technological appraisal and w needs assessment to justify going forward with the Oracle ELA proposal. Page 32 DOIT’s Efforts to Pool the State’s Purchasing Power 7w We do not agree that DOIT asserted that it had no responsibility to intercede with respect to the Oracle ELA. DOIT acknowledges its substantial role in helping to develop the effort to pool the State’s purchasing power, which resulted in the creation of the Oracle ELA. This is precisely DOIT’s statutory mandate. Page 33 Technological Assessment 7 For the reasons discussed above, DOIT believes that it did properly assess the technological merit of the proposed ELA including an assessment of the Statewide need for this technological solution. Although more validation and assessment could have been performed, we believe that clarification by legislation or administrative action in this area will be beneficial to the State. 86 87 Page 34 Departmental Software Inventory 3 We have previously set forth DOIT’s substantial efforts to obtain from each department a software inventory. We request that BSA amend page 34 to reflect the process described above. DOIT, DGS and DOF Were Partners in Developing the Oracle ELA w During the exit interview, we attempted to clarify that DOIT had a substantial role in developing the idea of the State pooling its purchasing power to achieve substantial savings. Page 50.1 Appendix A We would appreciate BSA modifying page 50.1 to reflect the fact that DOIT never received the 1 Addendum to the Logicon report. 5 We would also appreciate BSA amending the last box in the right hand column to reflect that the survey was a joint DOIT and DGS survey. e In addition, we would request that BSA amend the top box in the right hand column (beginning January, 2001) to reflect that several vendors were involved in discussions regarding the leveraging of the State’s purchasing power. On May 24, 2001, it was decided that in order to do a sole source contract the contract must be entered into directly with Oracle and that Logicon could not be a party. Director Keene and Director Cortez were asked to convey that message to Oracle and Logicon, which they did that same day. Soon thereafter, Director Cortez attended a meeting involving Koch Financial, but it was Director Cortez’ understanding as of May 31, 2001 when the Oracle ELA was approved, that Logicon would not be a party to the contract. Conclusion Although BSA correctly points out that the respective statutory roles of DOIT, DGS and DOF in the development, negotiation and execution of a Statewide ELA need clarification, it is DOIT’s belief that each participant responsible for creation of the Oracle ELA made a sincere effort to reduce the cost of IT to the taxpayers of California and did the best job they could under the circumstances especially considering that this was a pioneering, unprecedented effort. We would also respectfully submit that it might be somewhat premature to conclude that the Oracle ELA will result in a loss to the State. 86 87 This page left blank intentionally for reproduction purposes only. 88 89 COMMENTS California State Auditor’s Comments on the Response From the Department of Information Technology To provide clarity and perspective, we are commenting on the Department of Information Technology’s (DOIT) response to our audit report. The numbers correspond to the numbers we have placed in DOIT’s response. 1 Subsequent to the exit conference, we made revisions to the draft report and shared those revisions with DOIT. Unfortunately, DOIT apparently did not read the revisions we made to the draft. Had it done so, DOIT would have known that we revised page 26 of the report and reflected those changes in our summaries to say that it need not have used its existing procedures in assessing the assumption of need contained in Logicon’s proposal, rather it could have used the skills gained through its routine evaluations to assess whether such need existed. We also modified pages 24 and 25 of the draft and related summaries to remove all reference to and information from an addendum to Logicon’s white paper that was sent to us, but, according to DOIT, never sent to it. 2 The director’s statement that DOIT brought the idea for the Oracle ELA to the State is puzzling because it contradicts the assertion DOIT made on page 9 of the report. There, DOIT asserted that while it verbally recommended to General Services that the State needed a means to leverage its purchases, it did not recommend an ELA or specify Oracle as the vendor. 3 There is a difference between planning to take an inventory of the State’s software and actually taking such an inventory, as we state on page 26 of the report. DOIT has thus far sent two management memos to state departments and agencies: one requiring that they plan the inventory and another requiring them to report on their software inventory by January 31, 2003. The survey that Finance sent to all state departments in November 2001 requesting that they each report on all the Oracle software they had at that time, was the first step toward actually having the data necessary to create a statewide information technology inventory. 88 89 4 All the conclusions in the report, including its title, are based on the evidence we reviewed and analyzed, and we stand by them. 5 The director is mistaken. According to its instructions for the March 2001 survey, DOIT was responsible for the survey, and if enough interest was indicated, General Services would negoti- ate a volume purchase through a California Multiple Award Schedules agreement that the State had with Oracle. In fact, the director of General Services stated that he and his department were not aware that such a survey had been conducted. 6 We disagree. As we state on page 20 of the report, only five state departments responded to the survey stating that they were interested in purchasing additional Oracle products. Further, these departments represent only 12 percent of the State’s work- force. Finally, it is unlikely that these five departments would need access to Oracle’s database for every authorized position. 7 We do recognize the activities the director speaks of on pages 8 and 9 of the report. However, as the director correctly notes, one of the aspects that required additional evaluation prior to establishing the ELA pertains to the current and future need for the product. As we discuss on pages 21 and 27 of the report, DOIT made no further efforts to assess the State’s need for Oracle software beyond holding meetings and conducting the survey in March 2001. Furthermore, the survey suggested the need for additional Oracle products was limited. 8 As we state on page 49 of the report, the ELA threatens to establish the perception that the Oracle enterprise database software is the State’s de facto standard. We stand by our statement. Furthermore, as we state on page 28, TIRU raised this same concern in its May 10, 2001, memo to the directors of DOIT and Finance. 9 While, according to the director, efforts may be underway to identify and migrate all existing state Oracle contracts to the ELA, that effort will still fall well short of the 270,000 users and associated maintenance support the State is paying for through the ELA. 0 The director’s belief that the potential savings to cities and counties attributable to the ELA should be considered in assessing its value is not relevant to Logicon’s claim that the State would save $111 million if it exercised a four-year maintenance option to the contract. Logicon’s analysis relates 90 91 to Oracle database licenses and related maintenance available only to state employees and state contractors. If any benefit accrues to cities and counties, it would be through the volume purchase agreement portion of the ELA contract, which was not included in Logicon’s analysis. As we state on page 3, our review of Logicon’s proposal indicates that the State will spend millions more with the ELA than it would without it. q The director’s statement is inaccurate. Of the $19 million in additional Oracle products the director claims state departments expressed an interest in purchasing over the next fiscal year, DOIT could not provide documentation in the form of a survey response for over $14 million. Furthermore, another $3 million represented purchases that the departments indicated they would not be interested in making within the next fiscal year. w We believe we do accurately characterize DOIT’s role throughout our report and particularly as it relates to developing a method to leverage state purchases of software described on pages 8, 9, 26, and 27. e The information we reflect in Appendix A in the box dated “Beginning January 2001” is accurate according to the evidence DOIT provided. 90 91 This page left blank intentionally for reproduction purposes only. 92 93 Agency’s response to the report provided as text only. State and Consumer Services Agency Office of the Secretary 915 Capitol Mall, Suite 200 Sacramento, CA 95814 April 5, 2002 Elaine Howle, State Auditor* Bureau of State Audits 555 Capitol Mall, Suite 300 Sacramento, California 95814 Dear Ms. Howle: Enclosed is our response prepared by the Department of General Services to the Bureau of State Audits’ Report No. 2001-128 entitled, Enterprise Licensing Agreement: The State Failed to Exercise Due Diligence When Contracting With Oracle Possibly Costing Millions of Taxpayers Dollars. A copy of the response is also included on the enclosed diskette. I want to assure you that this Agency is fully committed to assist in the implementation of your rec- ommendations. In fact, as noted in the report the actions recommended are already underway. If you have any questions or need additional information, please contact me at 653-2636. Sincerely, (Signed by: Clothilde V. Hewlett) Clothilde V. Hewlett Undersecretary Enclosures *California State Auditor’s comments appear on page 101. 92 93 Date: April 5, 2002 File No.: 2001-128 To: Aileen Adams, Secretary State and Consumer Services Agency 915 Capitol Mall, Room 200 Sacramento, CA 95814 From: Department of General Services Executive Office Subject: RESPONSE TO BUREAU OF STATE AUDITS’ REPORT NO. 2001-128– “ENTER- PRISE LICENSING AGREEMENT: THE STATE FAILED TO EXERCISE DUE DILI- GENCE WHEN CONTRACTING WITH ORACLE POSSIBLY COSTING MILLIONS OF TAXPAYER DOLLARS” Thank you for the opportunity to respond to the Bureau of State Audits’ (BSA) Report No. 2001-128 which addresses recommendations to the Department of General Services (DGS). The following response addresses each of the recommendations pertaining to the activities of the DGS related to the Enterprise License Agreement (ELA) with the Oracle Corporation. Based on the request of the BSA, we will not comment on the recommendations directed to the Department of Finance (DOF) and the Department of Information Technology (DOIT). OVERVIEW OF THE REPORT The DGS has reviewed the findings, conclusions and recommendations presented in Report No. 2001-128. The DGS will take appropriate actions to address the recommendations. Although raising a number of valid issues that need to be addressed, the BSA recognizes that an entity comprised of many database users, such as the state, can potentially achieve significant volume discounts and reduce its overall administrative costs through the use of ELAs. In its analy- sis of the 2002/03 Budget Bill, the Legislative Analyst’s Office also recognizes the benefits of ELAs and provides a number of suggestions and recommendations that can help the Legislature ensure that future agreements are more cost effective and beneficial to the state. The DGS shares the views expressed and agrees that the process used to enter into an ELA must be conducted in a manner that protects the best interests of the state. The use of ELAs to acquire database software and maintenance services, such as was done with Oracle, is a relatively new acquisition method that allows the state to leverage its purchasing power. As noted below, the state is working to ensure that a model process is used in developing and implementing future ELAs. An ELA has significant financial benefit by allowing the state to consolidate its statewide need for proprietary software products to obtain the best price in a single contract. As noted in the BSA’s report, the primary sources of ELA savings are lower license costs from making a volume purchase; lower annual maintenance costs from negotiating a fixed, multi- year rate; and, lower administrative costs to acquire, track, and report license usage than if state departments separately purchased licenses and maintenance. As to the Oracle ELA, which is the 94 95 Aileen Adams -2- April 5, 2002 first agreement of its kind utilized by the state, the DGS recently had an independent analysis per- formed of this agreement. The analysis identified a number of actions that could be taken to ensure that the value of the Oracle agreement is maximized. The DGS is actively pursuing the recom- mended actions to ensure that the state receives the full realization of benefits available under this leveraged procurement. In its report, the BSA raises a number of valid issues concerning the process used to develop and imple- ment the Oracle ELA. Prior to the audit, the DGS was aware of many of these issues and had begun taking actions to ensure that best practices are followed in any future ELA procurements. These actions include working with DOIT, DOF and the state’s data centers to develop a model process to be followed in acquiring future ELAs. The model process will clearly identify the roles and responsibilities of each of the state agencies involved in the development and implementation of an ELA. The proposed draft model includes components which provide that each ELA have: (1) a detailed needs assessment prepared; (2) a report submitted for control agency review and approval that contains analyses of such key issues as projections of use and anticipated benefits; and, (3) an acquisition process conducted that includes an independent third party review of the proposed agreement. In addition, the DGS, in conjunction with the DOF, is currently taking a number of steps to ensure that the state is maximizing the value of the Oracle ELA. Specifically, as recommended by the previously discussed independent analysis, a baseline for use of the agreement is being developed. This process includes identifying and validating data related to existing and potential use. Although not yet complete, we expect the results to be of significant value in ensuring that savings are maxi- mized under the agreement. If fully utilized, the ELA represents a discount of more than 80% off Oracle list prices for data- base licenses. In addition, the Volume Purchase Agreement (VPA) component of the ELA offers a discount of 50% off list price for a variety of Oracle products. Our review of a sample of contracts established prior to the ELA indicates that discounts offered by Oracle were significantly smaller. Specifically, we found examples where discounts offered by Oracle ranged from 19% to 34% off Oracle list price. These examples also demonstrated that prior to the ELA, Oracle calculated maintenance pricing based upon the list price rather than the discounted price of a product. This resulted in maintenance rates that ranged from 24% to 33% of the net license price. Under the ELA, the maintenance rate does not exceed 17% of the net license price. The DGS realizes that all licenses must be deployed to achieve these discount levels, and we are working proactively to achieve this goal. In addition to the potential for savings at the state level, the Oracle ELA offers the opportunity for significant savings to local government entities. Through the VPA component, local governments receive the same discount offered to state entities, 50% off of Oracle list price. In a recent example, a local district indicated that use of the VPA resulted in savings of $196,560 on an Oracle purchase originally offered at $393,120. Prior to the ELA, at least thirty state departments independently negotiated multiple contracts annually for Oracle products and maintenance. According to the Department of Transportation, a complex negotiation could take four to five months to complete. The ELA offers state departments 94 95 Aileen Adams -3- April 5, 2002 an administrative benefit in that they individually will no longer be required to conduct the annual negotiation for Oracle products and services covered by the agreement. During its audit the BSA also developed a concern that the Oracle ELA could result in the percep- tion by state agencies that the database covered by the agreement is a de facto standard. The DGS shares this concern and has a firm commitment to ensuring that a de facto standard does not result. The DGS’ position was clearly disseminated in Management Memo 01-19, which was issued on September 17, 2001. This statewide policy memorandum provides that the Oracle ELA is intended to control costs associated with the state’s current and anticipated demand for Oracle database, products and services. It is not the intent to standardize the use of Oracle database and product. State users are free to select the technology solution that best meets their needs. The DGS is also in the process of developing a comprehensive communication plan for the agreement. In addition to addressing change management and outreach-related communications, the plan will address how to communicate to prevent any unintended perceptions that may suppress database competition. Additionally, the DGS is working with the DOF and DOIT to ensure that information technology projects that provide for the use of the Oracle ELA are clearly supported by the propos- ing entity’s business requirements. The BSA also expresses its opinion that DGS’ alleged failure to fully comply with state statutes gov- erning sole source contracting could affect the validity of the Oracle ELA. BSA correctly notes DGS’ disagreement on this issue. In view of the substantial impact that the ultimate resolution of these matters could have on the rights and obligations of the parties, including any efforts to re-negoti- ate the agreement, we are reluctant to further discuss our differences regarding this issue in the limiting context of this report. We would note, however, that newly enacted sections of the Public Contract Code (PCC), Sections 10298 and 10299, enacted in 2000, provide legislative authority for information technology procurements that leverage the state’s buying power. The DGS believes that the ELA is clearly such an agreement. It is the DGS’ position that the Oracle ELA was fully justified as an appropriately authorized sole source contract. The DGS was not establishing and acquir- ing a standard statewide database, in which case the department would have been obligated to competitively assess the merits of competing products. The Oracle database software was already 1 being used by many state agencies and was likely to continue to be used for a substantial period of time. The Oracle product, therefore, met the state’s needs and thus represented an exception to the otherwise required competitive bidding under PCC Section 12102 (a) (1) and State Administra- tive Manual (SAM) Section 5209. PCC Section 12102 (a) (1) specifically provides for a competi- tive acquisition exception when the “the goods and services proposed for acquisition are the only goods and services which can meet the state’s need.” SAM Section 5209, which makes practical interpretations of this section, specifically allows for the acquisition of proprietary software without competitive bidding. The Oracle database software and the license authorizing use of that software are such proprietary software. The DGS will re-examine its policies regarding sole-source information technology procurements and confer with the Attorney General regarding any necessary legal clarification. In summary, the DGS is fully receptive to recommendations proposed by the BSA and other inter- ested parties. The DGS recognizes that additional actions need to be taken to ensure that the values offered by the Oracle ELA are fully realized. Further, the DGS recognizes that process 96 97 Aileen Adams -4- April 5, 2002 improvements need to be made prior to entering into future ELAs. The DGS is working closely with the DOF, DOIT and other stakeholders to ensure maximum value from the current agreement and the implementation of a model process to be followed in developing any future ELAs. Overall, the DGS believes that the concept of leveraged procurement that led to the Oracle ELA, the first such agreement entered into by the state, are of significant value when properly evaluated and imple- mented. The following response only addresses the recommendations that are addressed to the DGS. In general, the actions recommended by the BSA have merit and will be promptly addressed. RECOMMENDATIONS CHAPTER 1 RECOMMENDATION # 1: DOIT, Finance and General Services should seek legislation establishing the authority to enter into an ELA that protects the State’s interests and clarifies each department’s respec- tive role and responsibility in the ELA process. DGS RESPONSE # 1: A model process for developing and implementing an ELA is currently in draft form. This pro- cess includes best practices in software acquisition and defines the roles and responsibilities of DOIT, DOF and DGS. The process contains four phases: needs assessment, proposal devel- opment, acquisition and final approval. For each of the phases, a responsible lead department is assigned and actions and work products identified. We are available to discuss a model process with the Legislature. RECOMMENDATION # 2: DOIT and Finance should develop policies and proce- dures on how to evaluate future ELAs. To be effective, one state department needs to take responsibility for developing and justifying the ELA proposal. DGS RESPONSE #2 As discussed under the previous recommendation, the proposed model process will include the assignment of a responsible lead department for the development and justification of ELA proposals. 96 97 Aileen Adams -5- April 5, 2002 CHAPTER 2 RECOMMENDATION # 1: Before negotiating any future enterprise licensing agree- ments, General Services should assemble a negotiating team that possesses all the types of expertise necessary to protect the State’s interests. DGS RESPONSE # 1 The DGS will ensure that sufficient resources and expertise are assigned to any future ELA propos- als. The previously discussed model process identifies the detailed actions that must be performed prior to entering into an ELA. Included in these actions is a step that provides for independent third party review of each proposed agreement by experts in the field. The model process, including the independent review activity, will ensure that only best practices are used in procuring future ELAs. RECOMMENDATION # 2: To identify the legal measures to take to protect the State’s interests, we recommend that General Services do the fol- lowing: · Continue to study the ELA contract’s validity in light of the wide disparities we identified in Logicon’s projec- tions of cost and savings and consult with the Attorney General on how to protect the State’s best interests. · Work closely with the Attorney General in further ana- lyzing the ELA contract; all amendments, including any and all documents pertaining to the side agreements between Oracle and Logicon; and the laws and policies relating to the ELA, including the potential legal issues that this audit has identified. DGS RESPONSE # 2 The DGS has developed a relationship with the Attorney General related to the Oracle ELA. This relationship includes consulting on the issues raised in the BSA’s report. 98 99 Aileen Adams -6- April 5, 2002 RECOMMENDATION # 3: If the contract is determined to be enforceable, General Ser- vices should renegotiate the contract to ensure it includes adequate protections for the State. DGS RESPONSE # 3 Amendment of the contract is one of the issues currently being discussed with the Attorney General. RECOMMENDATION # 4 The Legislature should consider requiring all information technology contracts over a specified dollar amount to receive a legal review by General Services. DGS RESPONSE # 4 The DGS takes this recommendation very seriously and will immediately review current practices to determine when legal review of information technology contracts is warranted. The DGS’ staff will be available to discuss the results of this review with the Legislature. CONCLUSION The DGS is firmly committed to effectively and efficiently controlling the state’s procurement pro- cess. As part of its continuing efforts to improve this process, the DGS will take appropriate actions to address the issues presented in the report. If you need further information or assistance on this issue, please call me at 376-5012. (Signed by: Dennis Dunne for Barry D. Keene) Barry D. Keene, Director Department of General Services 98 99 This page left blank intentionally for reproduction purposes only. 100 101 COMMENTS California State Auditor’s Comment on the Response From the Department of General Services To provide clarity and perspective, we are commenting on the Department of General Services’ (General Services) response to our audit report. The number corresponds to the number we have placed in General Services’ response. 1 We disagree with General Services’ statement that the Oracle product met state needs. As we acknowledge on page 59, for those state employees currently using the Oracle database soft- ware, it may have been necessary to sole source with Oracle. However, we fail to see how General Services can contend that Oracle has the only database software able to meet the State’s future, and as yet unknown needs. 100 101 This page left blank intentionally for reproduction purposes only. 102 103 Agency’s comments provided as text only. April 5, 2002 Department of Finance Office of the Director State Capitol, Room 1145 Sacramento, CA 95814-4998 Ms. Elaine Howle, State Auditor Bureau of State Audits 555 Capitol Mall, Suite 300 Sacramento, CA 95814 Dear Ms. Howle: Thank you for the opportunity to respond to the draft report, “Enterprise Licensing Agreement: The State Failed to Exercise Due Diligence When Contracting with Oracle Possibly Costing Millions of Taxpayer Dollars,” received March 28, 2002. I concur with your recommendation that an enterprise licensing agreement (ELA) process needs to be established that protects the State’s interests and clarifies each department’s respective role and responsibility in the ELA process. The Administration will shortly propose a process to the Legislature that achieves those goals. It is my expectation that each of the three control agencies that has an interest in ensuring sound investment in information technology-the Departments of Finance, General Services, and Information Technology-will develop policies and procedures to implement that proposed process. Given the apparent potential for major savings to the State, the promise of a termination-for-con- venience clause that was intended to mitigate the State’s financial risk, and the accelerated deci- sion time-frame imposed by the vendor, I made a decision to support the Department of General Service’s procurement proposal. In hindsight, it may have been more prudent to delay execution of the agreement, and first complete a thorough validation of the invoice data provided by Oracle, as my staff recommended at the time. The Administration is currently in discussions with Oracle to refine the provisions of the existing contract. We believe that with those refinements, it will have value to the State. When those dis- cussions are concluded, the Department of Finance will complete the methodology for assigning contract costs to departments. Sincerely, (Signed by: B. Timothy Gage) B. TIMOTHY GAGE Director 102 103 cc: Members of the Legislature Office of the Lieutenant Governor Milton Marks Commission on California State Government Organization and Economy Department of Finance Attorney General State Controller State Treasurer Legislative Analyst Senate Office of Research California Research Bureau Capitol Press 104