AI and California State Government

Artificial Intelligence and California State Government Report #284 | December 2024 Milton Marks Commission on California State Government Organization and Economy www.lhc.ca.gov LITTLE HOOVER COMMISSION Dedicated to Promoting Economy and Pedro Nava Efficiency in California State Government Chair The Little Hoover Commission, formally known as the Milton Anthony Cannella Marks “Little Hoover” Commission on California State Government Vice Chair Organization and Economy, is an independent state oversight agency. Dion Aroner David Beier* By statute, the Commission is a bipartisan board composed of five public members appointed by the governor, four public Asm. Phillip Chen members appointed by the Legislature, two senators and Gil Garcetti two assemblymembers. José Atilio Hernández Jason Johnson* In creating the Commission in 1962, the Legislature declared Sen. Dave Min its purpose: Asm. Liz Ortega ...to secure assistance for the Governor and itself in Janna Sidley promoting economy, efficiency and improved services in the Sen. Scott Wilk transaction of the public business in the various departments, *Served on study subcommittee agencies and instrumentalities of the executive branch of the state government, and in making the operation of all state departments, agencies and instrumentalities, and COMMISSION STAFF all expenditures of public funds, more directly responsive to the wishes of the people as expressed by their elected Ethan Rarick representatives... Executive Director Tamar Foster The Commission fulfills this charge by listening to the public, Deputy Executive Director consulting with the experts and conferring with the wise. In the course of its investigations, the Commission typically empanels Krystal Beckham advisory committees, conducts public hearings and visits Daniel Harris-McCoy government operations in action. Ashley Hurley Shara McAlister Its conclusions are submitted to the Governor and the Legislature Allie Powell for their consideration. Recommendations often take the form of legislation, which the Commission supports through the legislative process. Cover Image: Created by DALL-E Contacting the Commission All correspondence should be addressed to the Commission Office: Little Hoover Commission 925 L Street, Suite 805, Sacramento, CA 95814 (916) 445-2125 | LittleHoover@lhc.ca.gov This report is available from the Commission’s website at www.lhc.ca.gov. Letter from the Chair December 9, 2024 The Honorable Gavin Newsom Governor of California The Honorable Mike McGuire The Honorable Brian Jones President pro Tempore of the Senate Senate Minority Leader and members of the Senate The Honorable Robert Rivas The Honorable James Gallagher Speaker of the Assembly Assembly Minority Leader and members of the Assembly DEAR GOVERNOR AND MEMBERS OF THE LEGISLATURE: Artificial intelligence has brought about a seismic societal shift in which computers can mimic human thinking and creativity. Governments must quickly learn how to use this new technology for the common good while protecting against associated risks. For this reason, the Little Hoover Commission held a series of hearings earlier this year designed to identify best practices relating to the implementation of artificial intelligence in California state government. Drawing on the knowledge of AI experts from academia, the tech industry, labor, and government, these hearings explored how AI can help make government service more effective and efficient. They also addressed well-founded concerns relating to AI’s potential negative impacts on labor, privacy, and historically marginalized populations. In this report, we offer a number of recommendations that build upon the AI policy efforts of the current administration. They include the construction of a state-owned computing center to support AI for the public good, providing all state workers with access to general-purpose AI tools, and pioneering approaches to AI training, procurement, and oversight. What binds these recommendations is the belief that AI innovation cannot be a top-down enterprise, but should instead draw on the collective passion, creativity, and diversity of California’s state workers and, indeed, all Californians seeking to make our state a better place to live. The Commission respectfully submits this work and stands ready to assist as we boldly embrace this new era of technological progress together. Sincerely, Pedro Nava, Chair Little Hoover Commission ARTIFICIAL INTELLIGENCE AND CALIFORNIA STATE GOVERNMENT | 1 Table of Contents EXECUTIVE SUMMARY..........................................................................3 INTRODUCTION .....................................................................................5 BACKGROUND: KEY CONCEPTS IN ARTIFICIAL INTELLIGENCE......7 RECOMMENDATIONS...........................................................................11 Build a California AI Computing Center..........................................................11 Create a California AI Council..........................................................................14 Empower State Workers...................................................................................15 Make GenAI Implementation Easier...............................................................20 Improve GenAI Procurement and Risk Assessment......................................24 Strengthen Human Worker Voices..................................................................27 Educating the Public about GenAI...................................................................29 Legislative Oversight.........................................................................................32 California Must Think Beyond GenAI..............................................................35 GLOSSARY OF AI TERMS.....................................................................39 APPENDIX A: EXECUTIVE ORDER N-12-23 (SYNOPSIS)..................40 APPENDIX B: GENAI CAPABILITIES ...................................................41 APPENDIX C: TECHNICAL IMPLEMENTATION..................................42 NOTES....................................................................................................45 2 | LITTLE HOOVER COMMISSION Executive Summary Artificial intelligence is a powerful force that has AI tools along with robust training and other the potential to fundamentally transform the way forms of support, these workers will spearhead AI we live. But, in the absence of safeguards, AI can solutions across state government. At the same also expose us to risks of bias, misinformation, and time, improving procurement practices will ensure AI significant economic dislocation. These rewards and systems are acquired efficiently and with appropriate risks are exacerbated by the development of a new safeguards to manage associated risks. kind of artificial intelligence — generative artificial California has a longstanding commitment to intelligence — which can imitate human thinking and ensuring the safe and responsible adoption of produce novel content including text, images, audio, artificial intelligence in state government. In 2018, and video. this Commission released the report, “Artificial California is a global home to technological Intelligence: A Roadmap for California”, which innovation, and Gov. Gavin Newsom is a long-time outlined key steps to support AI research, harness AI advocate of the responsible use of technology in to enhance public services, and address the evolving the public sector. And yet, time and again, state nature of work, all while safeguarding privacy, government has struggled to implement and utilize transparency, and accountability. new technology. Avoiding a repetition of these past With Gov. Gavin Newsom’s 2023 executive order mistakes is critical. calling for the broader integration of generative AI The unifying aim of this report’s recommendations in state operations, there is now an urgent need for is to empower state workers, researchers, and further guidance to ensure its effective and secure others dedicated to serving the public good, such as implementation. This report seeks to provide some non-profit workers, to harness AI responsibly and of that critical guidance, with the overarching goal of effectively, driving innovation within state operations improving governance and delivering better services and beyond. to Californians. To this end, we recommend that the state take the following actions: STATE-OWNED AI COMPUTING CENTER One major recommendation is the establishment RECOMMENDATION 1: BUILD A CALIFORNIA of a state-owned AI computing facility, which would AI COMPUTING CENTER not only enhance government operations but also California should develop its own state-run, cloud- support academic institutions, nonprofits, and start- based AI computing center to provide affordable, ups in advancing their efforts. scalable access to compute power, AI tools, and datasets for state agencies, academic institutions, AI COUNCIL non-profits, and startups. We also recommend the creation of a state AI Council to oversee the implementation of AI in state RECOMMENDATION 2: CREATE A CALIFORNIA AI COUNCIL government, including the recommendations in this A California AI Council should be established to report. oversee the implementation of AI in state operations. TRAINING AND PROCUREMENT This includes the development of the proposed AI This report also focuses on two areas crucial to AI computing center and other recommendations found adoption: worker training and procurement. By in this report. giving all state workers access to general-purpose ARTIFICIAL INTELLIGENCE AND CALIFORNIA STATE GOVERNMENT | 3 RECOMMENDATION 3: EMPOWER STATE should also engage in self-education relating to AI/ WORKERS GenAI technology and policy. California should provide all state employees with RECOMMENDATION 9: CALIFORNIA MUST access to secure general-purpose GenAI tools along THINK BEYOND AI with mandatory basic GenAI training. The state California’s current focus on GenAI is too narrow. should also facilitate peer-to-peer and self-directed The state should expand its GenAI procurement GenAI trainings tailored to the unique needs of its and risk-mitigation policies and forms to include all individual workers and programs. technologies that have significant risk, not just GenAI. RECOMMENDATION 4: MAKE IMPLEMENTATION EASIER The state should facilitate GenAI innovation without increasing agency workloads by clarifying GenAI use policies, supporting inter-agency and inter- governmental collaboration, identifying agency needs, and piloting and scaling GenAI applications across agencies. RECOMMENDATION 5: IMPROVE PROCUREMENT AND RISK ASSESSMENT The state’s current procurement and risk-mitigation processes for GenAI should be streamlined as much as possible without compromising safety. RECOMMENDATION 6: STRENGTHEN HUMAN WORKER VOICES The state must balance leveraging GenAI while maintaining a strong human presence in its operations. It can achieve this, in part, through regular forums, agency position statements, and the creation of an AI ombudsperson. RECOMMENDATION 7: EDUCATING THE PUBLIC To convince Californians of the value, safety, and harmlessness of GenAI in government operations, the state should take steps to educate the public about how it intends to implement GenAI and protect against GenAI harm. RECOMMENDATION 8: LEGISLATIVE OVERSIGHT The California Legislature should establish a “Select Committee on California Public Sector AI” to oversee AI/GenAI implementation in state operations. It 4 | LITTLE HOOVER COMMISSION Introduction Artificial intelligence is a powerful and potentially Yet the history of state government’s efforts to adopt transformative technology that is reshaping the way new technologies does not breed an abundance we live. It is both a useful tool and—very likely—a of confidence. Again and again, California’s public disruptive force. There are countless examples of agencies have struggled to implement and use how this bold technology can change human life, innovations that should have produced better from better health outcomes to improved traffic service. management. But, in the absence of adequate safeguards, artificial intelligence can also expose To some degree, the very structure of state us to risks of bias, misinformation, and significant government reflects these past problems. More economic disruption. than a decade ago, then-Gov. Jerry Brown proposed a restructuring of state government that included These risks and rewards are exacerbated by the the creation of the California Department of development of a new kind of artificial intelligence— Technology. In endorsing that plan, our Commission generative artificial intelligence—which can imitate noted the state’s history of “difficulties in getting human thinking and produce novel content. This large IT projects up and running.”1 A more recent kind of artificial intelligence is still an emerging example is the years-long effort to create a technology, is evolving at an extraordinary pace, and centralized accounting system for state government, is at the cutting edge of both regulation and use. the Financial Information System for California, commonly known as FI$Cal. In a series of reports, The potential impact of this new technology is as the State Auditor has repeatedly noted that the significant—if not more so—in government as in development and implementation of the FI$Cal any other area of society. When harnessed safely system has included missed deadlines and other and effectively, AI could enable more efficient and problems.2 cost-effective service delivery. It could also support state employees by freeing them from repetitive, Avoiding a repetition of these past mistakes is critical, routine tasks, allowing them to focus on substantive and this report makes recommendations designed programmatic work, which they are uniquely to do exactly that. In short, the state should put qualified to perform. Regardless of how they interact artificial intelligence into the hands of rank-and-file with state government, whether as service recipients state workers and let them drive innovation. These or taxpayers, Californians could and should benefit. workers understand their agencies’ needs and workflows best, and they should play a leading role in There are reasons for optimism. California is not determining how AI is used. only a global hub for technological innovation, but Gov. Gavin Newsom has long been a proponent of responsibly using technology in the public sector. His “The potential impact of this new executive order on generative artificial intelligence, technology is as significant—if discussed in detail in this report, has already yielded positive results. Additionally, in 2014 he authored a not more so—in government as in book, Citizenville: How to Take the Town Square Digital any other area of society.” and Reinvent Government, which explores the role of technology in transforming government. ARTIFICIAL INTELLIGENCE AND CALIFORNIA STATE GOVERNMENT | 5 This means creating an ecosystem that will Our Commission believes in the remarkable potential foster bottom-up transformation, and our of AI to enhance good governance. However, its recommendations are the building blocks of adoption must be grounded in the values that that ecosystem. Agencies will need access to the have long guided California’s most effective efforts: necessary computing power and AI tools. Workers safety, transparency, and accountability. This report must be empowered to use AI safely and creatively. outlines concrete steps, based on these principles, The state must ensure that AI products can be that can maximize the benefits of this extraordinary procured efficiently and with reduced exposure to technology while safeguarding against potential potential harms. Taken together, these steps will lead harms. to better services for Californians. First, we recommend the creation of a state-owned AI computing center that we believe will facilitate the responsible and transparent development and use of AI in California. This center will be available to both state agencies and other entities that will use AI for the good of the public, including academic institutions, nonprofits, and startups. In particular, it will provide access to compute power and AI development tools, software, and datasets that might otherwise be insufficiently secure and/or cost prohibitive. Second, we offer several recommendations for how California can train and support state workers in the responsible and proficient use of AI, as well as how agencies can efficiently procure AI tools while minimizing risk. These are only two of the many topics related to artificial intelligence that must be addressed by the state, but we believe that they hold particular promise. By providing all state workers with access to safe and secure general-purpose AI tools along with training and a variety of mechanisms for fostering AI innovation, California’s public servants will become the main driving force behind AI innovation in the state. And, by improving the state’s current GenAI procurement process, which also involves risk management, we believe California can adopt these tools more efficiently and with a minimum of risk. 6 | LITTLE HOOVER COMMISSION Background: Key Concepts in Artificial Intelligence What is artificial intelligence? The definition of AI “Weak” AI systems are designed to handle specific found in the National Artificial Intelligence Initiative tasks, such as navigating traffic or recognizing Act of 2020 has frequently been reused in California images. In contrast, “strong” AI, or artificial general policy documents and legislation:3 intelligence, would be capable of performing any intellectual task that a human can do, though such The term “artificial intelligence” means a systems do not yet exist. machine-based system that can, for a given set of human-defined objectives, make predictions, Foundational models, like ChatGPT, represent an recommendations or decisions influencing real evolution of weak AI. These models are trained or virtual environments. Artificial intelligence on vast, diverse datasets and can be fine-tuned systems use machine and human-based inputs for various specialized tasks, making them highly to: (A) perceive real and virtual environments; versatile within a wide range of applications. ChatGPT (B) abstract such perceptions into models is a prime example of a foundational model. Since through analysis in an automated manner; and it is trained to understand and generate human (C) use model inference to formulate options for language, it is also classified as a “large language information or action. model.” Additionally, because ChatGPT can produce novel text based on the data it has processed, it falls In simpler terms, AI seems “intelligent” because, under the category of “generative AI” or GenAI. like humans, it can make sense of large amounts of complex information, draw conclusions, and pursue GenAI has been perceived as transformative because goals. Al also has the potential to act autonomously it can imitate—and arguably improve upon— or semi-autonomously and to “learn” over time. AI’s higher-order human thinking skills such as analysis, capacity to independently improve its data analysis summarization, organization, comparison, judgment, and generate better outputs is known as “machine and the creation of new content in a variety of learning”.4 media and genres. When ChatGPT was released on November 30, 2022, the public gained access to a AI is not new. Indeed, most people are so familiar technology that could quickly generate high-quality with task-specific AI applications—deemed “narrow” writing, summarize long documents, and write or “weak” AI—that they are unaware that they are computer code. By January of 2023, ChatGPT had actually using AI technology. Common examples attracted 100 million active monthly users, making it include search tools (Google), recommenders the fastest-growing software application in history.7 (Netflix), and navigation apps (Apple Maps).5 Such And the current interest in GenAI has led to broader systems are also used in a variety of government inquiries about how all forms of AI can support operations in California.6 For example, CAL FIRE is operations across a wide variety of sectors. using AI algorithms to spot wildfires remotely and reduce response times. ARTIFICIAL INTELLIGENCE AND CALIFORNIA STATE GOVERNMENT | 7 AI and State Government tech industry; and several measures to help ensure that GenAI does not negatively impact vulnerable California has a history—a long one, given the pace communities, state labor, and Californians in of technological change—of working to ensure the general.9 (For a synopsis of the order, see Appendix responsible adoption of artificial intelligence in A.) state government. In 2018, this Commission issued Moreover, through its Request for Innovative Ideas a report titled “Artificial Intelligence: A Roadmap for initiative, the state is testing several GenAI pilot California”, which recommended a series of steps to solutions that will make California’s healthcare support AI research, take advantage of AI to enhance facilities safer, make it easier for non-English public services, create a structure of education to speakers to access health and social services, and address inevitable changes in the nature of work, allow Californians to navigate the state’s roadways and protect core values of privacy, transparency, more quickly and securely.10 and accountability. In this report, we called for the Governor and Legislature to respond with action The California Legislature has likewise become that would be “big, bold, and scalable.” Among other very active in proposing the regulation of artificial steps, we recommended: intelligence both inside and outside of government. ◊ The creation of a cabinet-level special advisor to In the 2023-24 legislative session alone, the the Governor on AI. Legislature passed and the Governor signed several bills to regulate the use of AI across privacy, ◊ The designation of a chief AI officer for every transparency, child safety, election integrity, and state agency. education among other topics.11 ◊ Strategic plans for the use of AI within state agencies. Among the measures passed are a bill clarifying that ◊ Training so that workers have the skills necessary personal information under the California Consumer to thrive in an AI world. Privacy Act applies to AI-stored data (AB 1008) and another requiring AI developers to disclose the data ◊ The creation of an AI Commission. used for training their systems (AB 2013). A uniform Governor Newsom has long recognized the value of definition of AI has also been established in state technology in civil service and has recently taken a law (AB 2885), and a separate provision mandates leading role in advancing the deployment of GenAI that AI developers include provenance disclosures within California state government. In September of in content produced by their systems (SB 942), 2023, he issued an executive order that encouraged enhancing transparency. California’s state agencies to explore GenAI adoption In child protection, two bills expand existing statutes and set in motion a multifaceted process to support on child pornography to include AI-generated content GenAI implementation and mitigate risk.8 (AB 1831, SB 1381). Several pieces of legislation also Among other deliverables, this order has already tackle AI’s potential to manipulate elections, requiring resulted in guidelines for purchasing GenAI disclosures for political ads or election materials applications and safeguarding against possible harms altered by AI (AB 2355, AB 2655, AB 2839). In such as inaccuracy, algorithmic bias, or breaches in healthcare, new rules ensure that AI’s use in patient data security; a GenAI training curriculum for state communications is transparent and that human workers; formal partnerships with academia and the judgment remains central in medical decisions (AB 3030, SB 1120). Furthermore, California is enhancing 8 | LITTLE HOOVER COMMISSION California’s AI Ecosystem In industry, academia, and government policy, California is a global leader in AI innovation.12 This vibrant tech ecosystem drives AI development across a range of industries, contributing to groundbreaking research, technological advancements, and practical deployment. The state’s unique environment makes California a critical player in AI’s ongoing evolution, setting the pace for global progress in the field. The tech industry plays a central role in California’s AI ecosystem, spanning a variety of functions. Companies that develop essential hardware provide the computing “backbone” for AI, while cloud providers supply the vast computing resources necessary to train and deploy AI models at scale. Other firms focus on creating foundational AI models, which can be fine- tuned for a wide range of applications. From large established companies to nimble startups, the tech sector in California is a driving force for innovation. In addition to the tech industry, California’s universities and research institutes, such as Stanford and UC Berkeley, are key players in advancing AI through cutting-edge research and training the workforce of the future. Venture capital plays a critical role in scaling AI startups, funding innovations, and pushing new technologies to market. Advocacy groups and think tanks contribute to the ecosystem by addressing ethical concerns, promoting transparency, and ensuring AI serves the public good. Government agencies and the California Legislature work to craft policies that encourage responsible AI use while addressing concerns relating to bias and privacy. Labor representatives work to ensure that AI’s integration into industries supports workers’ rights and facilitates equitable transitions. Together, these diverse stakeholders form an interconnected ecosystem that fosters the responsible, inclusive growth of AI in California. AI literacy through bills that promote its inclusion catastrophic harm.” Just as importantly, the Governor in school curriculums (AB 2876, SB 1288). Together, expressed his openness to future AI legislation: “To these initiatives aim to promote a responsible and those who say there’s no problem here to solve … I ethical use of AI across the state. disagree.”13 However, Gov. Newsom vetoed the most far-reaching Our Commission shares this view—problems need of these bills (SB 1047), which would have, according addressing. We anticipate numerous new bills in to his veto message, “require[d] developers of large the Legislature on the topic of AI, and this report artificial intelligence (Al) models, and those providing aims to offer suggestions to policymakers on how the computing power to train such models, to put California can better navigate this extraordinary certain safeguards and policies in place to prevent transformation. ARTIFICIAL INTELLIGENCE AND CALIFORNIA STATE GOVERNMENT | 9 The Promise of Artificial Intelligence Though revolutionary in many ways, the public- sector use of artificial intelligence holds at its core a promise that is fundamental to government: the provision of better services to Californians. The Commission believes that the greatest benefits of this new technology will be tangible to citizens in simple yet crucial ways—shorter lines at the DMV, faster responses to unemployment claims, more efficient healthcare and social services, or smoother processes for obtaining licenses and permits. Beyond these improvements, there will undoubtedly be breakthroughs we cannot yet foresee, for one of AI’s most profound impacts will be on the state workforce. AI as the potential to relieve state workers—the government’s frontline service providers—of the burden of routine, repetitive tasks. This shift will free up their time and talents, enabling them to focus on more meaningful, complex, and impactful work within the programs they know best, thus driving substantive improvements in the services they deliver. This is the lens through which our Commission views AI’s potential. While gains in efficiency, cost- effectiveness, and speed are certain, we believe the most significant outcome of AI’s widespread adoption in state government will be its ability to transform the way Californians experience public services. At its heart, this transformation is about enhancing service quality, ensuring that Californians can access government services more easily, more quickly, and with fewer frustrations. 10 | LITTLE HOOVER COMMISSION Recommendations Recommendation 1: Build a To facilitate public-sector AI and AI for the public California AI Computing Center good, California should invest in developing its own cloud-based AI computing center, which would be California should develop its own state-run, cloud- owned and operated by the state and would offer based AI computing center to provide affordable, wide-ranging access to a variety of AI development scalable access to compute power, AI tools, and tools and applications. Such an initiative would not datasets for state agencies, academic institutions, only meet the state’s own AI needs but also serve non-profits, and startups. This initiative would academic institutions, non-profits, and startups, while increase efficiency, foster innovation, and generate offering the potential for generate income by selling revenue by also offering access to for-profit and access to other governments and private entities. out-of-state entities at higher costs, while ensuring A CENTRAL PORTAL FOR ACCESSING transparency, ethical AI development, and oversight. COMPUTE POWER AND AI TOOLS To implement this recommendation, the state Generally speaking, the best model for California is should: the National Artificial Intelligence Research Resource (NAIRR) pilot—a federal program led by the National ◊ Allocate initial funding to build the necessary Science Foundation—which aims to democratize infrastructure and ensure long-term financial access to AI for research and educational purposes.16 sustainability. The NAIRR will provide not only computational power ◊ Create an accessible online portal that provides but also curated AI datasets, models, software, easy centralized access to compute power, AI and training resources. It will achieve this through tools, datasets, and training resources. cross-sector collaboration between several federal ◊ Establish strong data security, privacy, and agencies, national and academic computing centers, compliance protocols that align with California’s and tech companies like Google, AWS, and OpenAI legal and ethical standards. among many others. ◊ Develop pricing models that offer free or low-cost The expressed goals of the NAIRR are to “spur access for in-state organizations, while charging innovation, increase diversity of talent, improve out-of-state and for-profit entities higher fees, capacity, and advance safe, secure, and trustworthy encouraging broad participation while ensuring AI in research and society.”17 We recommend that revenue generation. California adopt these goals in guiding its own efforts BACKGROUND to expand access to compute and AI technology. Cloud computing has become critical to innovation One of the notable features of the NAIRR is the in a wide range of sectors insofar as it offers development of an integrated portal aimed at remote access to computational resources.14 For lowering barriers to entry and increasing the diversity AI development, in particular, cloud computing is of users.18 This portal will not only be easily navigable essential because it provides access to the processing but will also include access a wide range of tools and power required to handle resource-intensive tasks resources tailored to users of various experience like AI model training and analysis of large datasets. levels, from beginners to advanced professionals. Cloud platforms allow users to flexibly “rent” compute resources without the need for costly infrastructure investments, making innovation more accessible.15 ARTIFICIAL INTELLIGENCE AND CALIFORNIA STATE GOVERNMENT | 11 California should develop a user-friendly portal, A state-operated AI computing center would also modeled after the NAIRR, as part of its computing provide the California Legislature with clearer center. This portal should provide integrated, one- oversight and budgetary review capabilities. stop access to compute power, AI tools, datasets, Legislators would have a more direct line and training materials, making it easy for state to monitor and evaluate how resources are agencies, academic institutions, non-profits, and being used, allowing them to adjust funding startups to utilize the available resources. By or regulations in real time.21 Currently, when focusing on accessibility, clear navigation, and governments rely on private cloud vendors, inclusivity, the portal would ensure that users from budget reviews can be limited to reviewing diverse backgrounds can effectively engage with AI contracts and expenditures, which lack granular technology and drive innovation across sectors. details about operations, security, and data management.22 With a state-run center, the WHY THE AI COMPUTING CENTER SHOULD Legislature could evaluate not only financial BE STATE-RUN efficiency but also ethical considerations, data Governments have several models at their disposal protection practices, and the overall effectiveness when facilitating research, often relying on a blend of AI deployments across state agencies. This of public and private resources to meet their needs. transparency would result in more informed For instance, the NAIRR draws on a mix of public and decision-making and better budget allocations private computing power, combining resources from for AI projects. national labs and the private sector. However, we believe California should take a different approach 2. Ethical, Legal, and Social Implications (ELSI) by creating its own state-operated cloud-based Considerations computing center to support public-interest AI.19 This A government-run cloud center is far more is for several reasons: likely to prioritize funding for Ethical, Legal, and Social Implications (ELSI) research. Historically, 1. Transparency and Accountability government initiatives like the federal Human Developing a state-owned computing center Genome Project have allocated funds specifically will allow for direct oversight by the Governor, for ELSI programs, ensuring that ethical ensuring transparency and accountability. The considerations keep pace with technological center would likely be operated by the California advances. Over decades, the National Institutes Department of Technology, giving the state of Health set aside 3% to 5% of its research full control over AI initiatives and guaranteeing dollars for this purpose, which proved crucial that they align with ethical standards.20 Unlike in addressing public concerns about genetic private cloud providers, which have their research.23 By adopting a similar model for AI, own commercial interests, a state-run center California can ensure that its cloud computing eliminates potential conflicts of interest. infrastructure not only promotes innovation This approach avoids the risk of companies but also supports research into the broader prioritizing their own products or steering societal impacts of AI. This commitment to ethical research in ways that could disadvantage governance is something private companies, competitors. Furthermore, with the cloud center driven by profit, may not prioritize in the same under the control of a public agency, the state way. can enforce adherence to AI principles, ensuring ethical AI deployment across state projects. 12 | LITTLE HOOVER COMMISSION 3. Flexibility in Collaborating with Nonprofits, By owning its own infrastructure, California could Academia, and Private Industry remain agile, easily adopt new AI technologies, A state-owned AI computing center would and avoid being locked into unfavorable terms, provide California with the flexibility to create higher fees, or limited support from any single partnerships with a wide array of organizations, vendor. Long term, the state would have full including nonprofits, universities, and private control over the evolution of its AI infrastructure, companies. This flexibility is crucial for fostering allowing for greater adaptability and innovation innovation and ensuring that AI research and in AI deployments across all state agencies. applications benefit from diverse perspectives.24 WHO WILL BENEFIT FROM A STATE-RUN AI If the center were privately run or operated in COMPUTING CENTER? collaboration with a single vendor, partnership California Government agreements could be more complex, limiting the range of collaborations the state could pursue. The Commission believes state government will With full control over its own infrastructure, benefit significantly from having access to AI tools California could form partnerships based on and these benefits are described throughout this merit, expertise, and public interest, without the report. In brief, having its own compute center constraints or conflicts that might arise from and AI tools would enable the state to operate private sector involvement. more efficiently, reduce costs, and enhance its ability to innovate in service delivery. A centralized 4. Financial and Technological Independence compute center would provide secure, scalable, One of the most compelling reasons for and cost-effective infrastructure for AI-driven California to build its own AI computing center is projects, allowing state agencies to process large to gain financial and technological independence. amounts of data and deploy AI solutions without While subscription-based cloud services offer relying on expensive external providers. Ultimately, convenience, they come with significant long- this infrastructure would improve public services, term costs that scale with usage. Over time, enhance transparency, and make California a leader these recurring expenses become a burden, in using AI to solve complex challenges. particularly as demand for AI and data processing Academic Researchers, Educators, Non-Profits increases across state agencies. By investing in its own cloud infrastructure, California could Access to low-cost compute power and AI tools mitigate these ongoing expenses, retaining more would enable academic researchers, educators, and control over its budget and long-term financial non-profits to significantly expand their impact and planning. accelerate innovation. For academic institutions, affordable computing resources would allow for Moreover, building a state-run AI computing larger-scale research projects, more advanced data center would free California from vendor lock- analysis, and the ability to experiment with complex in—a common issue with commercial providers. models that are otherwise limited to industry.26 This Vendor lock-in occurs when changing to a new would not only enhance research output but also provider becomes prohibitively expensive or provide students with practical experience using complex, effectively trapping the state within cutting-edge technology. a particular tech ecosystem.25 This stifles innovation, inflates costs, and prevents the adoption of newer, more efficient technologies. ARTIFICIAL INTELLIGENCE AND CALIFORNIA STATE GOVERNMENT | 13 With affordable access to AI tools, non-profit and non-profit sectors to keep pace with rapidly organizations could improve data-driven decision- evolving technological advancements. making and develop customized AI solutions to Other Governments optimize their operations and maximize impact.27 Doing so would positively affect a range of Many state and local governments lack the resources sectors, including education, healthcare, economic to build their own computing center and offer access development, and the promotion of human rights. to AI tools. By providing affordable, scalable compute Democratizing access to computing power and AI and AI applications to other governments, California tools would enable these organizations to compete could help them modernize their operations and on a more level playing field with industry. adopt AI solutions. This could include services for public safety, social services, healthcare, and However, access to advanced compute power environmental management.30 and tools is a significant challenge for academic researchers, educators, and non-profits due to high Startups and For-Profit Entities costs and limited infrastructure.28 Many of these For-profit entities, particularly startups and smaller organizations operate on tight budgets and cannot enterprises, could also be drawn to California’s afford the expensive hardware or cloud-based cloud computing center for its competitive pricing resources needed for large-scale computational and ethical focus. Many businesses, especially tasks. Moreover, the infrastructure necessary to those in industries like healthcare, legal tech, and support advanced computing, such as data centers financial tech, require secure and compliant cloud or high-performance machines, requires substantial infrastructure but may find commercial offerings investment, which many institutions are unable to prohibitively expensive. By offering a government- maintain. Additionally, complex licensing agreements operated alternative, California could attract and access restrictions often place essential software companies that prioritize transparency, ethical AI and datasets behind paywalls, making it difficult to development, and regulatory compliance. access the latest technologies and innovations in fields like AI and data science. Recommendation 2: Create a California AI Council Another key challenge is the lack of expertise required to leverage high-performance computing A California AI Council should be created to oversee systems effectively. Many academic institutions and the safe and effective implementation of AI in state non-profits lack the specialized personnel needed to operations. configure and optimize these tools. This problem is compounded by the competitive environment, where To implement this recommendation, the state well-funded industry players can more easily secure should: cutting-edge resources, leaving academics and non- ◊ Create in statute an AI Council that would include profits with fewer opportunities.29 Furthermore, even broad representation from state government, when external compute power is available, privacy academic experts, and key stakeholder groups. and security concerns, especially when handling ◊ Grant the AI Council authority to oversee sensitive data, may prevent these organizations from the implementation of AI in California state using cloud-based AI solutions. Combined with the government, including the development of limited availability of grants that cover computational the proposed AI computing center and other needs, these barriers inhibit the ability of academic recommendations found in this report. 14 | LITTLE HOOVER COMMISSION BACKGROUND public education campaign and an effort to move California must ensure that all of its AI reform beyond a narrow focus only on GenAI, will also efforts adhere to the core principles of transparency, benefit from the coordination and leadership of accountability, and safety. Additionally, the state a broadly representative council. Lastly, given must ensure that there are reasonable pathways California’s outsized global role in the development and timelines toward its goals. The Commission of AI and other new technologies, the Council can believes that the best way to ensure the success fulfill important roles of liaison and leadership with of the reforms outlined in this report — including respect to other states, to localities, and even to the creation of a state-owned cloud computing other countries. center, the provision of universal training for state Recommendation 3: Empower employees, and the streamlining of the process State Workers for acquiring and adopting AI tools — is a broad- based leadership council. Although the exact The state can most efficiently reap the rewards of membership or name of such an organization GenAI by enabling its rank-and-file workers to use are not critical, it should bring together a broad GenAI tools in their daily tasks. California should spectrum of constituencies that will be involved in therefore provide all state employees with access the implementation of AI into state government, to secure general-purpose GenAI tools along with including most particularly administration officials, mandatory basic GenAI training. The state should academic experts, and key stakeholder groups, such also facilitate peer-to-peer and self-directed GenAI as organized labor and the tech industry. trainings tailored to the unique needs of its individual workers and programs. Initially, to develop the state-run AI computing center, the council should create a comprehensive roadmap To implement this recommendation, the state that addresses technical requirements, funding should: models, and potential partnerships, ensuring that the center’s infrastructure is scalable, secure, and ◊ Give all state workers access to secure general- accessible. Early stages should focus on identifying purpose GenAI tools. potential locations for data centers, securing initial ◊ Make basic GenAI training mandatory for all state funding, and drafting a governance framework workers. specific to the cloud computing center that aligns ◊ Recruit GenAI “peer instructors” from within with California’s legal and ethical standards. individual agencies to help facilitate hands-on, peer-to-peer trainings that address agency- Beyond that, the council should provide a framework specific needs. and schedule for the additional reforms identified in this report, focusing especially on universal training ◊ Organize self-directed GenAI training to help for state workers and bureaucratic streamlining workers identify how GenAI tools can help with for procurement processes. Given the size and their particular workflows. complexity of California government, the reforms ◊ Incentivize GenAI adoption and innovation we identify are massive undertakings, and while through prizes and awards. the day-to-day implementation of these efforts will be left to specific state agencies, general oversight requires a more multidisciplinary structure. The other reforms identified in this report, such as a ARTIFICIAL INTELLIGENCE AND CALIFORNIA STATE GOVERNMENT | 15 BACKGROUND This type of “bottom-up” scaling is a common method General-purpose GenAI applications like ChatGPT, of generating, testing, and sharing innovative ideas in Claude, and Gemini will enable state workers with many industries and we think the state should adopt little to no IT experience complete a variety of tasks this approach as well.33 But this needs to be done in much more quickly and efficiently. Indeed, the sheer a secure and structured manner, with appropriate range of what GenAI tools can do is remarkable (for a IT security and training, in order to maximize GenAI list of GenAI capabilities, see Appendix B). benefits and minimize GenAI risks. General-use GenAI applications have been shown Most state workers won’t know whether a given to improve job performance significantly. A study publicly-available GenAI platform will offer an of 435 college-educated professionals found that acceptable level of data security. Major tech having access to ChatGPT reduced the time needed companies have historically misused customer data for a set of writing tasks by 40% and increased and stopped only when faced with scandal and quality by 18%, with especially notable improvements litigation. And GenAI companies are currently being among lower-skilled workers.31 And, in a study of accused of knowingly violating copyright laws for the over 700 employees from the Boston Consulting sake of training their AI models.34 In fact, the state’s Group, workers who used GenAI for a range of own “Responsible AI” course is hesitant to make any typical consulting tasks completed them 25.1% more firm statements about data privacy, saying only that quickly and their products were deemed 40% higher it differs based on the GenAI model used and level of in quality than the control group.32 We believe that subscription. the California state workforce could experience This is not reassuring, and the state’s own similar benefits. In what follows, we offer several (appropriate) anxieties concerning data privacy, recommendations for how to achieve this goal. accuracy, and freedom from algorithmic bias may GIVE ALL STATE WORKERS ACCESS deter employees from benefitting from GenAI tools. The state has tended to promote the kind of GenAI According to a 2024 survey, data privacy and data pilot projects that are being developed through security were the top GenAI-related concerns among its Request for Innovative Ideas initiative. While public sector employees.35 Given all of this, California impactful, these GenAI applications are being must provide its workers with a secure environment custom-built to serve a specialized purpose and will to test and use GenAI tools without fear of causing therefore not be relevant to most state employees. accidental harm including—but not limited to— violations of data privacy. The easiest and most efficient way to reap the rewards of GenAI is to put general-purpose GenAI Two large California institutions have created secure tools into the hands of as many state workers as general-use GenAI environments and could serve as possible and allow them to drive GenAI innovation potential models for the state. UC San Diego piloted in California government. Doing so will enable these TritonGPT in October 2023 and made it available workers to experiment with how GenAI tools can to its 37,000 employees in May 2024. TritonGPT is assist with their particular tasks. Especially helpful powered by the San Diego Supercomputer Center and creative GenAI solutions can then be shared and uses Meta’s open-source Llama large language with colleagues and, in some cases, may be formally model, ensuring that user data remains private. adopted as a best practice. 16 | LITTLE HOOVER COMMISSION Around the same time, UC Irvine deployed its ZotGPT that GenAI tools are safely integrated into state Chat, which it developed in conjunction with Amazon operations. Web Services and Microsoft Azure AI with the privacy A more ambitious solution would be for California of its over 57,000 students, faculty, and staff again at to develop a publicly owned AI computing center, top of mind.36 as recommended above. Doing so would give the California might also look to Singapore’s “Pair” state greater control over its data and cybersecurity chatbot assistant. Pair possesses ChatGPT-like and would have the additional benefit of providing functionality but was built expressly for public California researchers and non-profits with access to sector employees. Pair is trained on government the computing power needed to run AI applications. data, is accessible only through government-issued MAKE BASIC TRAINING MANDATORY devices, and is secure enough to handle confidential The state has released a basic GenAI training called information. Pair has proven highly popular. “Responsible AI for Public Professionals”.40 This According to its website, it was accessed by 11,000 course is available to all state employees—with users from over 100 agencies in its first two months supervisor permission—via the CalLearns website. It and currently has over 4,500 weekly users.37 was created in collaboration with InnovateUS, which California is, of course, aware of its security develops tech-related trainings for public sector standards and has already created an infrastructure employees. for GenAI testing that meets these requirements. The This course covers the fundamentals of GenAI: what California Department of Technology was tasked with it is, how it can help public employees do their work, building a “sandbox” to test pilot GenAI tools by no and how to avoid GenAI-related risks like inaccuracy later than March 2024.38 or bias. It is well-made, relatively brief, and provides In the world of tech, a sandbox is a controlled, several opportunities for hands-on practice using a isolated testing space where developers and users variety of GenAI applications. Short quizzes are given can experiment with models, tools, or applications at the end of each video to drive home the most without affecting real-world data or systems.39 It important ideas. The course is self-paced and takes allows users to safely test, fine-tune, and validate about three hours total to complete. GenAI functionalities, simulate scenarios, and The state has repeatedly highlighted the explore new ideas without the risk of unintended significant potential benefits and harms of consequences. It serves as an important step in the using this technology. For example, the State of development and deployment process, allowing for California Benefits and Risks of Generative Artificial experimentation, troubleshooting, and optimization Intelligence Report claims: “When used ethically and before moving to a production environment. transparently, GenAI has the potential to dramatically A version of the current sandbox should be scaled improve service delivery outcomes and increase up, providing all state workers with on-demand access to and utilization of government programs” (p. access to a secure GenAI environment or, much 2). And Gov. Newsom’s Executive Order declares that less preferably, access by application, where groups GenAI “can enhance human potential and creativity seeking to test a GenAI use case would be allowed but must be deployed and regulated carefully to to use the sandbox for a limited time. This secure mitigate and guard against a new generation of infrastructure would serve as a foundation for risks.” promoting innovation and efficiency while ensuring ARTIFICIAL INTELLIGENCE AND CALIFORNIA STATE GOVERNMENT | 17 The Commission agrees and, given the potential RECRUIT UNIT-LEVEL INSTRUCTORS FOR impact (positive and negative) of GenAI, believes that PEER-TO-PEER TRAINING all state workers should be required to complete the In addition to the “Responsible AI” training, state introductory “Responsible AI for Public Professionals” workers should have access to hands-on end-user course. Doing so will have a double benefit. GenAI trainings tailored to their particular tasks and needs. These trainings should be led by “peer First, it will help ensure that California’s public instructors” who are experts not just in GenAI, but workers understand how to protect against GenAI in the day-to-day work of their colleague-pupils. This risk. This is crucial given the near impossibility of kind of training will help bridge the gap between the monitoring employee use of publicly-available GenAI general GenAI training provided in the “Responsible applications, which are readily available for free over AI” course and the very diverse range of tasks the Internet. Self-monitoring and clear policies about performed by California’s state workforce.42 acceptable and unacceptable GenAI use are the best option. Indeed, the chief information officer of For example, an epidemiologist-GenAI peer instructor a major California city, who is also a national leader from within the Department of Public Health might in AI governance, told Commission staff that it was work with fellow epidemiologists as they discover impossible to monitor and control all GenAI use and ways to use GenAI to write or translate code used that policy and education were needed instead. to analyze health data. Or a GenAI peer instructor from the Department of Human Resources could Second, completing the “Responsible AI” course may help colleagues learn to use GenAI to streamline the encourage state employees who have not previously state’s hiring process. used GenAI to experiment with the technology and how it can help them work better and more GenAI peer instructors could be identified in several efficiently. ways. For example, the Office of Data and Innovation has developed a five-course Foundations of GenAI Indeed, research has shown that there is series (Completed Staff Work, Critical Thinking Tools, considerable room for GenAI growth in government. Data Strategies, “GenAI for Public Professionals”, and According to a recent survey, only 36% of public Human Centered Design for Public Professionals).43 sector employees reported that they understood State workers who complete all five courses could GenAI and its “potential impacts on business automatically be granted certification as a GenAI peer processes” “well” or “completely”. And only 22% instructor and begin to work with their supervisors used GenAI applications on a daily basis. This lags to organize GenAI trainings in their agency. The state behind the all-industry average of 52% and 28%, has also offered free Oracle Cloud Infrastructure respectively.41 certification, and this could be a second potential path to becoming a GenAI peer instructor. This should be seen as an opportunity. Most public employees lack experience using GenAI. By By recruiting GenAI peer instructors through its mandating that all of its state employees complete training and certification programs, California the “Responsible AI” training, California can quickly can foster a culture of innovation and practical close this knowledge gap and begin taking advantage application of GenAI across its agencies. These peer of GenAI. Furthermore, it can achieve all of this instructors will play a crucial role in translating GenAI immediately and at little added cost—a win for potential into tangible benefits, ensuring that all units everyone involved. are equipped to leverage this technology effectively. 18 | LITTLE HOOVER COMMISSION Federal Policies and Guidelines: Training and Procurement The federal government has adopted a proactive approach to mitigating AI risk through procurement, while also supporting workforce initiatives to leverage AI for the public good. These efforts could serve as models for state governments. Procurement and Risk Management: Last year, the federal government purchased more than $100 billion in IT products and services. Federal leaders are leveraging this significant purchasing power as part of the government’s proactive efforts to promote the responsible deployment and use of artificial intelligence.44 In January 2023, the National Institute of Standards and Technology (NIST) released the Artificial Intelligence Risk Management Framework, which provides voluntary guidance for addressing risks in the design, development, use, and evaluation of AI systems.45 NIST also developed a series of companion resources to support implementation of the framework. These include a playbook, a “living” resource with suggestions for navigating and using of the framework to achieve outcomes, and a profile focused on risk management specifically for generative AI.46 Building on this work, the Biden Administration later issued an executive order establishing standards for AI safety and security within federal government.47 This was followed by guidance for federal agencies to advance innovation and manage risks as they use48 and acquire AI.49 Workforce: Recruiting future workers with familiarity with AI technology for public employment is crucial for leveraging AI in government work. These technologically adept employees can drive innovation and efficiency in government operations, from automating routine tasks to analyzing big data for better policy decisions. Strategies to attract and retain workers skilled in AI technology might include competitive salaries, professional development opportunities, and innovative project roles. Earlier this year, the federal government launched an initiative— the National AI Talent Surge—to recruit workers with AI expertise into a number of programs including the newly created Department of Homeland Security AI Corps.50 Equally important is the need to cultivate a pipeline of future talent by encouraging AI-related education from an early age. Integrating AI training programs within K-12 curriculums and expanding AI courses and degrees at universities can ensure a steady flow of knowledgeable professionals ready to meet the state’s needs. The US National Science Foundation’s EducateAI initiative aims to help develop the future AI workforce,51 and has provided $8 million in funding for five projects to expand access to AI educational opportunities to prepare K-12 and undergraduate students.52 ARTIFICIAL INTELLIGENCE AND CALIFORNIA STATE GOVERNMENT | 19 FACILITATE SELF-DIRECTED TRAINING For example, the Department of Technology or AI All that said, it is simply not possible to provide Community could organize prizes and awards for GenAI trainings that are customized to the individual GenAI innovation. A “GenAI Innovation” category tasks and needs of the state’s nearly quarter-million could also be added to GovTech’s Best of California employees. Asynchronous web-based trainings such Awards, which honors “high-achieving state IT as the “Responsible AI” course and use of GenAI peer officials and significant recent IT projects”.54 instructors to lead agency-specific trainings will help. Recommendation 4: Make GenAI But, in the end, it will be up to each state worker to teach themselves how GenAI can benefit them. The Implementation Easier state must therefore play an active role in promoting State agencies are busy and often lack the time, self-directed GenAI training. resources, and/or expertise to implement GenAI Self-directed training will let state workers take in their operations. The state should therefore control of their professional development, learning facilitate GenAI innovation without increasing agency to tailor GenAI’s many capabilities to their individual workloads by clarifying GenAI use policies, supporting work contexts and experiment with GenAI tools at inter-agency and inter-governmental collaboration, their own pace, discovering applications that are identifying agency needs, and piloting and scaling most relevant to their roles.53 GenAI applications across agencies. State agencies should help workers identify how To implement this recommendation, the state GenAI can assist them through a process in which should: workers can list their daily responsibilities, identify ◊ Empower individual state workers to experiment pain points, and learn about GenAI capabilities. with general-use GenAI applications by clarifying Supervisors should seek to create a clear match use policies. between worker needs and available GenAI tools. ◊ Encourage interagency collaboration through INCENTIVIZE INNOVATION: PRIZES AND communities of practice. AWARDS ◊ Distribute a survey to determine what support Finally, to encourage state workers and teams to agencies need to implement GenAI. learn about GenAI and use it creatively in their ◊ Create mechanisms for piloting and scaling GenAI work, the state should also introduce a series of applications to facilitate implementation and incentives. Recognition awards, promotions, and lower costs. financial bonuses can serve as powerful motivators for employees. Alternatively, by establishing annual ◊ Collaborate with city, county, and state governments to share GenAI expertise and awards or contests to honor those who excel in defray costs. GenAI adoption and innovation, the state can foster a culture of creativity and excellence in this space. BACKGROUND Technological innovation in government is too often “The state should put artificial stuck in a Catch-22. Adopting a new technology intelligence into the hands of like GenAI can ultimately save an agency time and resources. But change is inevitably disruptive rank-and-file state workers and and, ironically, requires surplus time, funding, and let them drive innovation.” expertise up front. 20 | LITTLE HOOVER COMMISSION Such resources are typically in short supply, meaning EMPOWER WORKERS BY CLARIFYING USE agencies will struggle to adopt new technologies or POLICIES will need to hire costly consultants to implement As noted above, the state can and will benefit from IT solutions on their behalf.55 These challenges will GenAI tools almost immediately once its workers be particularly acute for smaller agencies that lack start using general-purpose GenAI applications in substantial IT budgets and personnel. their daily work and the Commission recommends that every state employee be granted access to a For example, Nolan Sullivan, Director of Health and secure environment in which to use these tools. Human Services for Yolo County, told Commission staff that artificial intelligence could help streamline However, uptake of general-purpose GenAI tools like the application process for services like CalFresh, ChatGPT, Claude, and Gemini is currently muddled allowing his understaffed office to focus on difficult by a lack of clear policy regarding their use. The cases and outreach to especially vulnerable Department of Technology should be tasked with communities. But, Sullivan continued, because producing official guidelines which agency leaders setting up an AI system would entail unacceptable can use to shape their own internal policies and levels of cost and disruption, AI adoption will remain practices. The guidelines should address topics such impossible for the foreseeable future. as whether it is ever appropriate for state employees to use publicly available general-purpose GenAI tools, This is a tragedy, and it is widespread. the level of privacy that can be expected from these tools, and the kinds of information that is safe to A 2024 survey of 300 government “decision makers” enter into them. revealed that 91% of state and local leaders felt their agency would benefit from IT modernization and By having clear guidance on these and other issues, 94% had “concrete plans to invest in or enhance agency leaders and workers will feel confident that technology in the next five years.” But only 40% they are purchasing and using these tools correctly, viewed IT innovation as an immediate priority, and will be able to take advantage of the benefits of primarily due to incompatibility with outdated readily-available general-purpose GenAI relatively “legacy” IT infrastructure, inadequate staff expertise quickly. and inter-agency collaboration, and the convoluted nature of the government grant system.56 ENCOURAGE COLLABORATION In addition to empowering individual workers and California’s long-term challenge is to remove units to use GenAI tools, the state should establish these barriers through IT and data modernization mechanisms that will allow multiple agencies and hiring of staff who are proficient in AI. In the to share and discuss ideas and best practices. meantime, we offer recommendations for GenAI Institutional siloes are perceived as barriers to implementation that require relatively few resources technological innovation.57 And, in any case, better but will nevertheless facilitate GenAI adoption and collaboration across agencies will not only lead to innovation. a more robust conversation around GenAI but may have additional ancillary benefits. The state’s Artificial Intelligence Community — a “community of practice” in the California Department of Technology — is already helping achieve this, and we recommend that it be expanded. ARTIFICIAL INTELLIGENCE AND CALIFORNIA STATE GOVERNMENT | 21 This community currently hosts well-attended not only streamline the procurement process but webinars every other month, and we believe it is well- will also encourage vendors to provide high-quality positioned to disseminate its benefits more broadly. information from the start. The GovAI Coalition has Moreover, we believe that larger agencies should adopted this practice through its Vendor Registry, create agency-specific AI Communities, open to all and vendors are allowed to update or withdraw their employees, to discuss AI-related ideas, best practices, materials from this repository at any time. and concerns relating to their particular scope of SURVEY ON BARRIERS TO work. We provide technical details as to how this IMPLEMENTATION might be done in Appendix C. To determine what kinds of support agencies will require to effectively implement GenAI, we Multi-layered networks of interconnected state recommend distributing a survey to agencies asking workers who are focused on the adoption and use about perceived barriers to GenAI implementation. of AI would produce compound benefits, creating Information collected through this survey will help a robust ecosystem of GenAI knowledge and guide the allocation of GenAI-related resources and innovation, and accelerating the adoption of GenAI shape GenAI strategy going forward. To guarantee across California state government. a sufficient number of responses, the survey will CALIFORNIA LEARNING HUB require a mandate from the Governor’s Office or the The Commission also recommends that the state California Legislature. create a central California GenAI Learning Hub dedicated to sharing GenAI-related resources, The survey would be designed to gather insights from trainings, documents, and announcements. The state agencies on the barriers (and opportunities) Learning Hub would also function as the online home related to the implementation of GenAI. The survey of the general and agency-specific AI Communities could also probe the potential benefits of GenAI mentioned above. In this role, the Hub would serve across various scales of application—from small- as a vibrant platform for interagency discussion scale uses like drafting memos to large-scale projects relating to GenAI blue-sky ideas and best practices. such as automating customer service. Respondents would be asked to share any existing GenAI initiatives A possible model for the California GenAI Learning within their agencies and identify the resources Hub is the federal AI Community of Practice, needed to expand these efforts. which was launched by the U.S. General Services CENTRALIZING AND SCALING APPS Administration and Federal Chief Information In the sections above, we strongly recommended that Officer in 2019.58 This federal effort brings together California provide all state employees with access to more than 4,400 local, state, and federal public general-purpose GenAI tools like ChatGPT, Claude, sector employees for AI trainings, hackathon-style or Gemini to maximize their practical benefit. That challenges, and information and idea sharing. said, more specialized GenAI tools will sometimes be In addition to its knowledge-sharing function, the required. GenAI Learning Hub could also serve a practical In an ideal scenario, all GenAI projects within state role in streamlining GenAI procurement by serving government—regardless of type and agency of as a repository for technical and safety information origin—would begin as pilot programs, created with about GenAI tools that other agencies might wish to the intention of sharing with other agencies. This purchase.59 By archiving GenAI product information would be done without having to develop an entirely and making it available for review, the state will 22 | LITTLE HOOVER COMMISSION new GenAI application with the assistance of costly Sharing and fine-tuning may, however, be difficult vendors and consultants. both for fiscal and contractual reasons, and because of the specialized nature of the GenAI applications For example, as noted above, the state has themselves. The state should therefore develop a commissioned several GenAI pilot solutions process for making GenAI solutions available to other through its Request for Innovative Ideas initiative. agencies with a minimum of additional effort and Although built for particular agencies to serve expense. Exactly how it achieves this should be left specific purposes, these pilots were intelligently to those with the relevant technical, fiscal, and legal chosen because they represent uses—for example, expertise. That said, we offer some specific options in customer service, document translation, or resource Appendix C. management—that are theoretically transferable to other agencies. AI and State Healthcare Delivery In 2022, healthcare spending in the U.S. reached $4.5 trillion, accounting for 17.3% of the GDP.60 State governments play a critical role in this sector, managing public health initiatives and safety net programs like Medicaid (called Medi-Cal in California). Across both the public and private sectors, AI holds significant promise for improving efficiency, detecting fraud, optimizing resource allocation, and enhancing disease surveillance. These advancements can lead to more effective and equitable healthcare administration. Here are some key ways AI can transform how states manage and regulate healthcare:61 ◊ Healthcare Delivery: Enhance diagnostics, optimize patient scheduling, streamline workflows, and provide real-time patient support. ◊ Healthcare Facility Inspections: Simplify survey documentation and report writing, ensuring consistent, accurate findings aligned with regulations while saving time and reducing subjectivity. ◊ Public Health Oversight: Monitor health trends and analyze data to inform policy decisions and allocate resources efficiently. ◊ Safety Net Services: Identify individuals eligible for programs like Medi-Cal by analyzing data across systems. ◊ Healthcare Workforce Licensing: Automate application reviews, streamlining the licensing process for completeness and compliance. ◊ Fraud Detection: Detect fraud by analyzing patterns in claims data, preventing improper payments and inefficiencies. ARTIFICIAL INTELLIGENCE AND CALIFORNIA STATE GOVERNMENT | 23 PARTNER WITH OTHER GOVERNMENTS Recommendation 5: Improve California should also formally join the GovAI GenAI Procurement and Risk Coalition. According to its website “[t]he GovAI Assessment Coalition is composed of over 600 public servants from over 250 local, county, and state governments Agencies should be able to assess GenAI risk and that represent over 150 million Americans across the purchase GenAI technology without unnecessary nation united in [its] mission to promote responsible bureaucratic hurdles. The current procurement and and purposeful AI in the public sector.”62 risk-mitigation processes should be streamlined as much as possible without compromising safety. By pooling the collective knowledge and resources of its members, the GovAI Coaltion has quickly created To implement this recommendation, the state a number of high-quality deliverables. This includes should: model templates for an AI governance policy, an AI ◊ Streamline the GenAI procurement process and incident response plan that could form the basis for create a mechanism for state agencies to share a boilerplate risk-mitigation plan for the state, an AI relevant information. factsheet, and a vendor agreement. It has also built a portal for a vendor registry where factsheets can be ◊ Improve procurement training so that agencies are better able to navigate the process. submitted for prior review by interested agencies, a practice that the Commission recommends the state ◊ Improve the guidelines used by vendors to adopt as well.63 describe their products. ◊ Implement a verification process to ensure By participating in the Coalition, the State of transparency and accuracy, reducing the risk of California or its agencies could benefit from the incomplete or misleading disclosures. collaborative efforts of its many members. And, while the state already has considerable leverage due to BACKGROUND its purchasing power, this could be increased still The procurement process—how departments go further—and extended to its smaller partners—by about purchasing items—is crucial to the discussion joining a group of governments that are collectively of GenAI in state operations. This is because the in the process of purchasing GenAI tools. procurement process that California has created for GenAI tools mandates that agencies evaluate the risks inherent in the GenAI tools they are thinking about buying and, in some cases, develop a plan for “Artificial intelligence is a mitigating this risk prior to adoption. powerful and potentially Done well, this has the potential to streamline GenAI transformative technology that implementation by reducing barriers to adoption in is reshaping the way we live. It the case of less risky applications while identifying is both a useful tool and—very and anticipating potential risk in advance. This contrasts with the “after-the-fact” litigation-based likely—a disruptive force.” approach to combatting AI risk adopted in some recent California legislation relating to AI.64 24 | LITTLE HOOVER COMMISSION The current procurement and risk-assessment IMPROVE GUIDELINES AND MODEL processes are hindered by gaps in guidance, the RESPONSES absence of a reliable method for verifying vendor The effectiveness of the procurement process responses, and redundancies that waste resources. could be enhanced through the provision of Streamlining these processes will not only simplify model responses to guide vendors. Vendors are the implementation of GenAI tools but also enhance currently required to disclose the presence of GenAI their safety and effectiveness. technology in products they intend to sell using a form that contains technical questions about a REDUCE REDUNDANCY IN PROCUREMENT product’s purpose, training data, and performance The GenAI procurement process should be metrics.66 streamlined to allow state agencies to acquire AI products more efficiently. There are several While the form itself is sound, there is a need opportunities to eliminate redundancy in the GenAI for clearer guidelines on the quality, length, and procurement process. First, it is currently required technical detail expected from vendor responses. that all written solicitations include mandatory The state should provide formal guidance. Doing so contractual language and a detailed form to be filled would improve vendor response quality and reduce out by the vendor, even for non-IT purchases. These the back-and-forth questioning between the state requirements should be limited to IT purchases and vendors, making the procurement process more only.65 efficient. Additionally, California should establish a method Additionally, California should provide examples of through which agencies can share information completed vendor responses, such as the model regarding products that have already been evaluated “audio classifier” factsheet provided by IBM, which and approved. By sharing this information among could serve as a benchmark.67 This would help multiple agencies, the state can avoid requiring vendors understand the state’s expectations and vendors to complete identical forms for subsequent ensure that vendor responses are evaluated properly purchases of the same product, thus streamlining by state reviewers. the procurement process. This practice is already employed by the GovAI Coalition, a consortium that Finally, California should ensure consistency in largely consists of local governments, and it would risk-mitigation plans for GenAI applications. While benefit both vendors and the state by reducing current materials outline four key risk management repetitive tasks. principles, there is no guidance on what a risk- mitigation plan should look like. Offering model IMPROVE PROCUREMENT TRAINING plans could provide a clear structure for agencies, The state should improve its procurement training especially those that are smaller or less experienced to better support agencies in navigating the in AI procurement.68 bureaucracy. Training exists now, but it largely duplicates previously published guidelines and does not sufficiently address common vendor questions or provide visuals and guidance on how to complete the required forms. An improved, interactive training, complete with an example procurement form and a final exam to confirm mastery, would make the process smoother and more efficient for agency staff. ARTIFICIAL INTELLIGENCE AND CALIFORNIA STATE GOVERNMENT | 25 The Commission’s 2018 AI Roadmap In 2018, the Commission published its report, Artificial Intelligence: A Roadmap for California, which explored how the state could adopt AI in ways that promote benefits such as economic growth and enhanced service delivery. In many ways, that earlier report anticipated the themes of this one. Together, the two documents reflect the Commission’s dedication to analyzing this evolving issue. Thus the 2018 report outlined key principles to shape the state’s approach to AI: autonomy, responsibility, privacy, transparency, and accountability. Furthermore, it emphasized the importance of equipping policymakers with a clear understanding of AI’s potential uses, risks, and necessary safeguards—foreshadowing the focus of this new report. The report laid out concrete goals for the state to pursue with respect to AI: ◊ Grow the state’s economy. ◊ Enhance services to Californians. ◊ Empower educational institutions to provide training. ◊ Improve data collection while protecting privacy. ◊ Promote privacy, transparency, and accountability. To achieve those goals, the report laid out an “agenda for change” – 10 steps recommended by the Commission: ◊ Create a cabinet-level special advisory on AI. ◊ Designate of a chief AI officer in each state agency. ◊ Encourage the creation of a chief AI officer in independent offices, such as those of elected constitutional officers. ◊ Require each state agency to develop strategic plans for AI. ◊ Create a strategic plan to ensure that future workers would be prepared for a world with AI. ◊ Improve data collection, and widely share data within state government while protecting privacy. ◊ Create an AI Commission. ◊ Ensure an environment of sustained investment, research and development of AI. ◊ Improve data collection regarding at-risk jobs. ◊ Promote apprenticeships and other training for workers whose jobs might be affected by AI. Finally, the report also called for a campaign of broad public engagement about AI – an effort that in the Commission’s words needed to be “structured to listen as much as inform.” 26 | LITTLE HOOVER COMMISSION VERIFICATION AND EVALUATION OF BACKGROUND VENDOR RESPONSES Gov. Newsom has ordered the state to “establish The state should implement a formal verification criteria to evaluate the impact of GenAI to the state process for the information submitted by vendors. government workforce, and provide guidelines on There is a risk that vendors may obscure details in how State agencies and departments can support their responses due to competitive pressures, which state government employees to use these tools could undermine transparency and trust. Robust effectively and respond to these technological verification guidelines are therefore needed, and advancements”.70 This report is due no later than the “software documentation” and “communication January 1, 2025. quality” standards developed by IBM Research could serve as a good intellectual foundation and starting But there is a fine line between assistance, point.69 automation, and worker redundancy. A recent article in Nature provides an excellent overview of Recommendation 6: Strengthen scholarship on the impacts of AI and automation on Human Worker Voices employment. And, it will be noted, many of these predictions border on the apocalyptic.71 To give just State employees may be anxious that California’s one example, it cites a “classic study” that “suggests rapid adoption of technology, including GenAI, will that AI could replace 47% of the 702 job types in cause them to lose their jobs, erode their dignity as the United States within 20 years”.72 On the whole, workers, diminish the quality of service that their current scholarship suggests that AI’s impact on agencies provide, or all of the above. The state must employment will be complex, with both utopian therefore balance leveraging GenAI while maintaining and dystopian possibilities coexisting. While AI has a strong human presence in its operations. What this the potential to create high-quality jobs through means should be a topic of long-term discussion. human-machine cooperation, it also poses serious potential risks for job displacement across a range of To implement this recommendation, the state industries. should: ◊ Organize annual forums with state employees, AI could potentially automate many tasks that labor, policymakers, and the public to discuss the are currently performed by human state workers. impacts of tech implementation in government Indeed, the state’s own Benefits and Risks Report operations. declares that GenAI could assist with a wide range of tasks that are commonly performed by government ◊ Require larger state agencies to publish position employees including content generation, customer statements on the respective roles of humans service, data analysis, code generation, and various and technology in their operations. research tasks (pp. 11-13). ◊ Create an independent GenAI Ombudsperson to field state worker concerns. ARTIFICIAL INTELLIGENCE AND CALIFORNIA STATE GOVERNMENT | 27 This anxiety has been anticipated by public and A key focus was on equity and inclusivity. Panelists private sector entities seeking to promote AI. Phrases emphasized the importance of ensuring that like “augment not replace” and “human in the economic opportunities in the tech sector are loop” have been repeated to the point of becoming accessible to all Californians and safeguarding the clichés. The names of GenAI applications—for data of vulnerable groups from misuse. Additionally, example, Google’s Gemini (“twins”) and Microsoft state labor representatives highlighted their Copilot—convey an impression of a healthy constituents’ worries about the impact of GenAI partnership between humans and machines. And on the quality of work life and the state’s overall company names like OpenAI and Anthropic imply a employment landscape. transparent, human-centered corporate culture. These are important topics, and the conversation But the long-term impact of GenAI on the state’s needs to continue. To this end, the state should workforce—and society more generally—is unclear. organize an annual forum on “GenAI, State Workers, For example, while leadership has insisted that and State Service” that brings together state workers, California’s state employees with not lose their labor, policymakers, and the public. This event could jobs to GenAI, these employees may simply not be include sessions on topics such as the impact of replaced after leaving their agencies if a suitable GenAI on state employment, the daily experiences GenAI replacement is available. In academia, the of state workers in using GenAI, and the apparent sharp decline of tenure-track faculty positions impacts on service delivery to Californians. through attrition—and their replacement with low- AGENCY POSITION STATEMENTS ON paid part-time positions—offers a clear and chilling HUMANS AND TECHNOLOGY parallel.73 Larger California state agencies—for example, the seven cabinet-level “superagencies”—should also Given the transformative nature of GenAI, California be required to publish position statements that must have open and honest conversations with its clarify the respective roles of human workers and employees about the ongoing roles of humans and technology in their operations. In Appendix C, we technology in state operations. The Commission offer an extensive list of the key features of such a offers the following recommendations to facilitate position statement. them. As the representative of state workers, organized labor must also be recognized as a critical By engaging in mindful reflection on the respective partner in these discussions. roles and benefits of humans and technology in their ANNUAL FORUM: “AI, STATE WORKERS, operations, state agencies will get a clearer sense of AND STATE SERVICE” their priorities and values, allowing them to navigate On May 29, 2024, the Joint California Summit on a future characterized by potentially dizzying Generative AI was held in San Francisco.74 One innovation more easily. panel, titled “The Future of the California Workforce,” The development of these position statements featured representatives from labor, education, the should involve a range of stakeholders—a broad social justice movement, and state government. spectrum of leadership and staff at all levels—so The panelists discussed the GenAI-related needs that the perspectives of all relevant parties are taken and concerns of current state employees and the into account. Once developed, these statements cultivation of a GenAI-savvy work force through should be distributed internally and to the public, California’s schools and, in particular, its community with periodic review to ensure that they reflect colleges. 28 | LITTLE HOOVER COMMISSION technological advancements and evolving public Recommendation 7: Educating expectations. the Public about GenAI INDEPENDENT OMBUDSPERSON The citizens of California will need to be convinced To ensure that potential GenAI issues are addressed of the value, safety, and harmlessness of GenAI in promptly and worker complaints are heard, the state state operations. The state should therefore take should create an independent GenAI ombudsperson. steps to educate the public about how it intends to This ombudsperson would serve as a confidential implement GenAI and protect against GenAI harm. resource for state workers to report any problems or concerns they encounter with GenAI systems.75 To implement this recommendation, the state should: While it is important for workers to feel comfortable contacting their leadership and IT professionals ◊ Create a public information campaign describing directly, there may be situations where anonymity how the state plans to use GenAI, how GenAI will is necessary for addressing more sensitive issues benefit all Californians, and the steps the state such unsanctioned use of GenAI, unreported GenAI will take to ensure GenAI safety. This campaign inaccuracy or bias, or a reduction in work quality could include the use of GenAI as a mechanism or service due to GenAI implementation. The of explanation. ombudsperson would provide a secure channel for ◊ Organize GenAI-themed hackathons to elicit whistleblowers, ensuring that their voices can be innovative ideas and build rapport with the heard without fear of retribution. public. A number of California state departments have ◊ Commit to the successful implementation of a an ombudsperson position including Health Care useful, high-profile GenAI application to convince Services, Insurance, Aging (Long-Term Care), and the public of GenAI’s value in state operations. Corrections and Rehabilitation. We recommend ◊ Publish an easily accessible online list of all that the GenAI ombudsperson be housed within automated decision systems used in state the Department of Technology both because of its operations that is regularly updated as these familiarity with the technical aspects of GenAI and its tools are adopted or disbanded. central role in the risk-mitigation process. BACKGROUND By offering an independent and impartial Gov. Newsom has directed all California state perspective, the ombudsperson would be able to agencies to explore the use of GenAI technology investigate complaints thoroughly and recommend to enhance operations and public service, while corrective actions where necessary. This approach advocating for responsible GenAI development. But will help build trust among state workers and ensure public sentiment toward AI has grown increasingly that problems arising from GenAI implementation negative, especially following ChatGPT’s release in are addressed in a timely manner. late-2022. ARTIFICIAL INTELLIGENCE AND CALIFORNIA STATE GOVERNMENT | 29 While opinions may change, multiple surveys from This chatbot would be designed to answer queries 2023 have shown that Americans are concerned about California’s use of GenAI, providing clear and about the impact of AI. According to an Ipsos poll, 43% concise explanations about what the technology is, of Americans have an unfavorable view of AI (39% its capabilities, and its current and future role in both view AI favorably); 36% think AI will “fundamentally internal and public-facing operations. Furthermore, change American society”; and 75% have “little to no it would inform users about the state’s commitment trust” in the companies that develop AI systems. And to privacy and safety, detailing the measures taken to a substantial majority of Americans are concerned protect sensitive data. about the impact of AI on “jobs and society” (71%) and Fine-tuned on the GenAI deliverables the state has AI-driven misinformation and deepfakes (76%).76 already produced, the chatbot would not only serve Looking at California in particular, a study by the as a practical tool for public education but also as Public Policy Institute of California has shown that a pioneering example of GenAI in action within 56% of Californians view AI negatively and 29% of government services. This initiative could help build Californians—a significant percentage—are “very public trust and understanding by demonstrating worried” or “somewhat worried” about their jobs transparency and responsiveness through a real- being eliminated by automation or new technologies time, interactive platform. including AI.77 Cumulatively, this suggests that a HACKATHONS TO BOOST PUBLIC variety of steps should be taken to assure the public ENGAGEMENT that California will use this technology in a safe, At least portions of the public are eager to see transparent manner. the state embrace GenAI, have expertise and PUBLIC EDUCATION CAMPAIGN perspectives to share, and would like to be part of Given the considerable anxiety over AI amongst that process. Hackathons are a great way to leverage Americans and Californians in particular, a strong this interest. public education campaign about why GenAI is trustworthy and how it serves the public good will be A hackathon is a short-term event in which a essential to build trust in its GenAI endeavors. This variety of stakeholders collaborate on a software or campaign should be simple and direct in tone, and technology-related project, often designed to address designed to educate members of the public who may a defined technical or social problem.78 The goal is to have little or no knowledge or background in dealing create a working prototype or solution by the end of with technology. Components of this public-education the session. effort could include a media campaign, online As part of this study, Little Hoover Commission staff educational content, feedback platforms to allow attended or became familiar with hackathons relating Californians to share their reactions and concerns, and to social justice-related issues. These included community partnerships with educational institutions, “Housing Stabilization Hackathon” held in Cincinnati nonprofits, and similar organizations. and a hackathon organized by UC Davis’ AI Institute A PUBLIC RELATIONS CHATBOT for Next Generation Food Systems, at which students The Commission also recommends fusing the GenAI worked to improve Digital Green’s Farmer.chat medium and message through the development of a chatbot to support small farmers.79 GenAI chatbot to educate the public about the state’s GenAI efforts. 30 | LITTLE HOOVER COMMISSION California state agencies have hosted similar events— California may be particularly interested in for example, relating to water management—using Singapore’s OneService chatbot, which its citizens the data housed in the California Open Data Portal can use to make inquiries or submit a complaint. with apparent success.80 We recommend that the An AI tool is used to process the submission, state organize a regular, even frequent series of classifying and routing it to the appropriate agency. hackathons relating the application of GenAI to California could develop a similar GenAI tool, issues of public concern. Possible broad topics enabling its residents to access government services include the optimization of public services, education without having to navigate its sometimes complex and workforce development, and the provision of bureaucracy. services for vulnerable populations. REPORT HIGH-RISK AUTOMATED DECISION SYSTEMS TO THE PUBLIC By leveraging hackathons, the state can tap into the enthusiasm and expertise of the public, fostering Californians have a right to know when their lives are a collaborative environment where innovative being affected by any “automated decision system”, solutions to pressing issues can emerge. These which has become legal shorthand for AI broadly events can serve as a powerful tool to build public conceived.84 The Commission therefore recommends interest and support for the government’s use of that Californians have easy access to a list of high-risk GenAI, showing its commitment to transparency and automated decision systems in use by the state. This community engagement. list should be updated on a regular basis to ensure that it remains current. A USEFUL PUBLIC-FACING APPLICATION The GenAI pilot projects announced thus far will California policymakers generally seem to support support internal operations and logistics, meaning the principle of AI “transparency”. For example, the the public will largely be unaware of the impact state’s GenAI risk-assessment form asks how a GenAI GenAI is having on the delivery of state services. applications will indicate to the user that they are At one level, this makes since given the riskiness interacting with GenAI rather than a human. of a “bad” interaction between a GenAI application And legislation introduced last year would have and a member of the public, which could prove required “a deployer to, prior to an automated embarrassing and potentially harmful to the user or decision tool making a consequential decision… the state.81 notify any natural person that is subject to the That said, a high-profile, public-facing GenAI consequential decision that an automated decision application that delivers clear, tangible benefits will tool is being used…” It also would have required the enhance the state’s ability to convince Californians opportunity to correct any incorrect personal data of the value of GenAI.82 Singapore has once again and, if feasible, “not be subject to the automated been a leader in public-facing artificial intelligence decision tool and to instead be subject to an applications and could serve as a model for alternative selection process or accommodation, as California’s efforts. It has used AI tools to personalize prescribed.”85 learning in public schools, streamline immigration clearance, and help citizens locate new job The Commission supports the thinking behind opportunities.83 this bill, but also believes that the public should not have to wait until they have been affected by an automated decision system to know what high-impact AI systems are currently being used ARTIFICIAL INTELLIGENCE AND CALIFORNIA STATE GOVERNMENT | 31 by the state. We therefore recommend that the BACKGROUND “comprehensive inventory of all high-risk automated In the realm of government operations, maintaining decision systems” mandated by AB 302 (Ward) a sustained focus on AI and GenAI implementation be posted online along with the completed Risk- poses a challenge due to short political attention Assessment forms for all GenAI procurements spans, the enormity and complexity of the topic, and deemed “high-risk” through the state’s risk the constantly changing nature of AI. This may be determination process.86 further exacerbated by the technology “hype cycle”, where initial excitement and high expectations are By making this information available to the followed by a period of disillusionment and diverted public, the state can increase trust in government interest towards the “next big thing”.87 operations, allowing citizens and advocacy groups to provide informed feedback. This will, in turn, The state has expressed its commitment to the promote ethical AI usage and allow the public to widespread implementation of GenAI across its serve as a watchdog. Moreover, California could set a agencies and the California Legislature should make precedent for other jurisdictions, acting as a national sure that this happens by exercising its oversight leader in transparent governance concerning AI and role. automated decision-making tools. ESTABLISH A SELECT COMMITTEE ON Recommendation 8: Legislative CALIFORNIA PUBLIC SECTOR AI Oversight The Commission strongly recommends that the Legislature establish a “Select Committee on Longer-term oversight will be necessary to ensure California Public Sector AI” for the 2025-6 legislative that California’s implementation of GenAI in state session. The primary advantage of a select operations is successful. The California Legislature committee is that it can provide focused attention should establish a “Select Committee on California on almost any topic of interest or concern. For Public Sector AI” to oversee AI/GenAI implementation example, there are currently select committees in the in state operations. It should also engage in self- California Assembly on “Biodiversity”, “Retail Theft”, education relating to AI technology and policy. and the “2028 Olympic and Paralympic Games”.88 To implement this recommendation, the state The need for a Select Committee on Public Sector AI should: is clear. This committee would make sure the state’s AI and GenAI policies and uses are implemented ◊ Form a “Select Committee on California Public properly and reflect best practices. It would also Sector AI” to oversee the state’s implementation respond to whistleblower complaints and hold bad of AI and GenAI in its operations. actors to account. ◊ Require larger state agencies to submit progress reports outlining AI/GenAI initiatives, including A less preferable alternative would be to have an objectives, performance metrics, and risk established standing committee (or committees) assessments. hold informational and oversight hearings on AI/ ◊ Require the Legislative Analyst’s Office to provide GenAI.89 In the Assembly, the majority of AI bills are regular briefings and recommendations on AI- referred to the Committee on Privacy and Consumer related developments, legislative proposals, and Protection. In the Senate, AI bills go to various the impact of AI on state operations. committees, perhaps most typically to the Committee on Government Organization. 32 | LITTLE HOOVER COMMISSION One or both of these committees could commit to This will not be especially difficult to implement once a the formal oversight of AI/GenAI implementation and reporting mechanism has been established. In fact, a policy within the state. But this approach will almost relatively narrow version of this is already in place. certainly be less effective because such committees The recently chaptered AB 302 (Ward) requires that may well be distracted by competing duties and lack the California Department of Technology compile a a focused mandate. A select committee dedicated “comprehensive inventory of all high-risk automated to AI/GenAI implementation in state operations will decision systems” that includes “categories of data and provide more consistent and thorough oversight. We personal information the automated decision system therefore urge the California Legislature to establish uses to make its decisions”. This inventory must be a Select Committee on California Public Sector AI at submitted to the Assembly Committee on Privacy its earliest possible convenience. and Consumer Protection and Senate Committee on REPORTING AI USE TO THE LEGISLATURE Governmental Organization on an annual basis starting To ensure the responsible and transparent in 2025. deployment of AI and GenAI in California state AI BRIEFINGS government, it is also imperative to establish a Following the public release of ChatGPT, there has structured reporting mechanism that can inform been a rapid increase in the number of AI-related bills legislative oversight and guide future policymaking. in state legislatures. In California, prior to the 2017-18 This should be restricted to larger agencies—for legislative session, few if any bills relating to AI had been example, the super-agencies or agencies with more introduced. AI legislation then began to accelerate, with than a certain number of employees—to avoid five bills appearing in 2017-18, 16 in 2019-20, and 13 in imposing an undue bureaucratic burden on smaller 2021-22.90 ChatGPT was launched in late-2022 and, in units. 2023-24, a remarkable 67 bills on AI were proposed in The Commission therefore recommends that the California legislature.91 these larger state agencies be required to And yet legislators and some senior staff may not submit AI/GenAI implementation reports. These possess the knowledge they need to assess AI-related reports should outline the agency’s AI and GenAI legislation effectively given its newness as a topic initiatives over the past calendar year, detailing of public concern, the intricacies of the technology their objectives, performance metrics used to itself, and the breadth and complexity of the AI gauge their effectiveness, and risk assessments policy landscape.92 Stanford’s Institute for Human- used to determine their safety both pre- and post- Centered AI has attempted to remedy this through deployment. its annual Congressional Boot Camp on AI.93 And at By requiring regular submissions of these reports, the federal level, one lawmaker is actually pursuing a the Legislature will be better equipped to monitor the graduate degree in machine learning in order to better progress of AI implementations, ensure compliance understand the topic.94 with state policies, and make informed decisions We therefore recommend that the Legislative Analyst’s about future investments in AI technologies. It will Office produce a series of briefings designed to inform also enhance AI transparency and foster a culture of state lawmakers and senior staff about AI and AI-related accountability and continuous improvement within policy issues.95 These sessions could focus on topics state agencies. such as privacy, security and ethical considerations; the economic impacts of AI; and the effect on the provision of services. ARTIFICIAL INTELLIGENCE AND CALIFORNIA STATE GOVERNMENT | 33 European Union AI Act The European Union Artificial Intelligence Act of 2024 represents a regulatory model that California policymakers may wish to consider. It establishes a classification system for AI applications based on their level of risk:96 Unacceptable risk systems: ◊ Completely banned because they can cause serious harm. ◊ Includes uses like social scoring and manipulative AI. High-risk systems: ◊ Used in important areas like healthcare, law enforcement, immigration, and critical infrastructure. ◊ Must follow strict rules. Limited risk systems: ◊ Includes things like chatbots and deepfakes. ◊ Need to be transparent about how they work. Minimal risk systems: ◊ Video games, spam filters, etc. ◊ No special regulations. Developers of high-risk AI systems have the most responsibilities, including managing risks at every stage, keeping technical records, ensuring the system works correctly, and enabling human oversight. The organizations that use these systems also have duties, but they are less extensive. General Purpose AI providers need to follow specific rules, such as documenting their models and respecting copyright laws. Those with higher risks face extra requirements. The EU will set up an AI Office to oversee these models, check for major risks, and handle complaints. California’s efforts to regulate generative AI resemble the EU AI Act insofar as both have adopted a risk-based approach. In California’s case, the state categorizes GenAI applications (for use in government operations only) based on the risks they pose, particularly in terms of data breaches or the potential for biased or misleading outputs. And, based on risk level, agencies are required to develop mitigation plans to protect against GenAI-related harms. Like the EU’s regulations for high-risk AI systems, prior to deployment in state operations, California requires developers to submit detailed technical information during the procurement process to ensure that GenAI tools meet security and ethical standards. This approach emphasizes transparency and accountability, aligning California’s framework with the EU’s focus on managing risk and safeguarding public trust. 34 | LITTLE HOOVER COMMISSION Recommendation 9: California WHY CALIFORNIA’S POLICIES ARE TOO NARROW Must Think Beyond GenAI Despite the recent frenzy surrounding ChatGPT California’s current focus on generative AI (GenAI) is and GenAI more generally, AI is far from new. In too narrow. The state should therefore expand its fact, most people are so familiar with task-specific GenAI procurement and risk-mitigation policies and AI applications—deemed “narrow”, “weak”, or forms to include all technologies that have significant “traditional AI” or “machine learning”—that they are risk, not just GenAI. totally unaware that they are using AI technology.97 These kinds of AI systems are likewise used in a To implement this recommendation, the state variety of consequential government operations in should: California such as the aforementioned AI system ◊ Develop a comprehensive, “future-proof” used by CAL FIRE to spot wildfires remotely and framework for identifying technologies reduce response times.98 that require in-depth procurement and risk But California also uses the COMPAS recidivism risk management procedures based on category of predictor, which has been accused of algorithmic risk rather than particular types of technology. bias against defendants, Black people, and young ◊ Revise the current GenAI and IT procurement and people, as well as bias favoring imprisonment over risk-mitigation guidelines and trainings so they release.99 And, as a recent CalMatters article reports, focus on these risk categories rather than specific a faulty fraud-detection algorithm led to the wrongful technologies. denial of over 600,000 unemployment claims by the BACKGROUND California Employment Development Department.100 The Governor’s executive order and its deliverables The CalMatters article also notes that this fraud- pertain solely to GenAI. This is not surprising given detection system is an example of AI but is not the transformative nature of the technology and the covered by the same safety protocols as GenAI extraordinary level of public interest and concern. applications: “The [GenAI] guidelines will not protect But its exclusive focus on GenAI is too narrow and people from other forms of the technology that have should be expanded to include all technologies— already proven harmful to Californians.” those that already exist and those of the future—that carry significant risk. This can be accomplished by Dr. Brandie Nonnecke, Director of UC Berkeley’s building IT policies that focus on categories of risk CITRIS Policy Lab, testified at a Commission hearing rather than particular types of technologies. that California’s GenAI policies should be extended “to all forms of machine learning”, from linear and logistical regression models to complex neural “Adoption must be grounded in networks. the values that have long guided “These technologies, even the simpler forms California’s most effective of machine learning, are making consequential decisions that impact California residents’ lives efforts: safety, transparency, profoundly. We must ensure that our procurement and accountability.” and risk mitigation guidelines encompass the full spectrum of AI technologies.” ARTIFICIAL INTELLIGENCE AND CALIFORNIA STATE GOVERNMENT | 35 The Little Hoover Commission agrees, and While there may be understandable resistance to recommends that the state take various measures updating IT policies given the significant effort that to protect Californians from all technologies, not just has been invested in developing GenAI procurement GenAI. and risk mitigation guidelines, it is important to recognize that California has already laid much of EXTEND GUIDELINES TO CURRENT AND the necessary groundwork. By building on existing FUTURE TECH frameworks and expanding them to encompass a California also needs to consider how it can broader set of technologies, California can create a create IT procurement policies that are more comprehensive policy that not only strengthens its comprehensive, encompassing both a broader set of leadership in GenAI governance but also prepares current technologies as well as unanticipated future the state to navigate and regulate whatever technologies like agentic AI or quantum computing technologies the future may hold. that pose similar or even greater risks. To achieve this, the state should revise its risk-mitigation strategies so that they focus on categories of risk rather than specific technologies like GenAI. California is well positioned to do this because it has already identified the appropriate categories of risk in the governance materials it created for GenAI: “data” and “use” (GenAI Toolkit, p. 29). In other words, a system’s risk is determined based on the consequences, first, of a data breach and, second, inaccurate or biased output. A data breach affecting up to 19,000 users of the BenefitsCal system, through which hackers gained access to users’ personal information through login credentials stolen from other websites, is a clear example of the potential consequences of “data”- related risk.101 And the incorrect denial of 600,000 unemployment claims mentioned above is, in turn, an example of the consequences of “use”-related risk. Risk criteria relating to “data” and “use” could form the basis of a comprehensive IT governance strategy. But California may also want to consider a more holistic approach to procurement and risk-mitigation that takes into account an even broader set of tech- related risk categories. In Appendix C, we propose a specific set of “future proof” criteria for evaluating technologies. 36 | LITTLE HOOVER COMMISSION AI’s Environmental Footprint As artificial intelligence has become more prominent, increasing attention has been placed on the significant environmental impact of these systems. However, this impact is often difficult to measure, due in part to a lack of industry data and transparency. In the absence of reporting requirements, companies can choose what information they share about the environmental impacts of AI models and data centers. For instance, some prominent model developers including OpenAI, Google, Anthropic, and Mistral do not report emissions from training, while Meta does.102 And only one-third to one-half of data center operators report water- use metrics.103 However, researchers have developed some estimates about the amount of resources consumed by AI, which primarily focus on energy consumption, greenhouse gas emissions, and water use.104 AI MODELS One study found that the energy needed to train an OpenAI model was equal to the amount one would consume while streaming over 1.6 million hours of Netflix programming.105 The greenhouse gasses emitted during training can also be substantial. In general, larger models produce more emissions, but emissions on smaller models operated on less efficient energy sources can also be high.106 Another study found that generating a text query using AI would take 10 times the amount of electricity as a regular Google internet search.107 AI systems also must be cooled to avoid overheating, which is typically done by circulating water. As a result, these systems often consume significant amounts of water. For example, one AI model consumed 5.4 million liters of water when trained. By 2027, some researchers estimate that global AI water demand may equate to half of the total annual water withdrawal of the UK.108 DATA CENTERS Researchers are also examining AI’s environmental footprint by looking at the data centers that power AI systems. However, as data centers do not just store information for AI models, but everything uploaded on the internet, it is difficult to parse out the energy demands of AI specifically. Nevertheless, as data center usage has increased due to the growth of AI, this is an emerging area of concern. The US is home to one-third of the world’s data centers, and 80 percent of the energy consumed by the nation’s data centers are in 15 states — including California.109 Across the state, there are nearly 280 data centers, with high concentrations in Los Angeles and the Silicon Valley.110 The Electric Power Research Institute estimates that California data centers could consume almost nine percent of that state’s electricity by 2030, up from nearly four percent in 2023. Estimates for US data centers follow similar projections.111 Most data centers run on fossil fuels and even those operating on renewable energy sources rely heavily on carbon-intensive diesel back-up systems ARTIFICIAL INTELLIGENCE AND CALIFORNIA STATE GOVERNMENT | 37 for when their supply fluctuates or the grid goes down.112 This is likely to remain true for some time, as analysts predict that much of the US data center load growth in the medium-term will be supported by natural gas.113 As such, Goldman Sachs analysts estimate that carbon dioxide emissions from data centers may more than double between 2022 and 2030.114 HOW ARE COMPANIES RESPONDING? Several large tech companies set bold targets to reduce their environmental footprints, but these efforts, they say, have been challenged by the expansion of AI. Google, which set a goal to achieve net-zero emissions and 24/7 carbon-free energy by the end of the decade,115 revealed in July that its emissions increased by nearly 50 percent in the past five years.116 Its data centers also consumed 17 percent more water in 2023 than the year prior.117 Microsoft, which pledged to be carbon-negative and water-positive by 2030,118 announced in May that its emissions were 29 percent higher than in 2020. Meta and Amazon also saw their total emissions rise after setting ambitious climate targets.119 HOW ARE POLICYMAKERS RESPONDING? Estimates of the environmental impact of AI are often cited in the state, national, or even global context. However, data center development is heavily concentrated in a few counties and cities, often in places where electricity is cheapest — regardless of its source — and where water supplies are scarce.120 It is in these communities that the burden of AI is most visible. Data Center Alley — a cluster of nearly 300 data centers in northern Virginia — processes nearly 70 percent of the nation’s digital traffic.121 The high energy consumption from data centers in this region has kept some coal burning power stations online that were originally planned to shut down.122 In The Dalles, Oregon — located within a dry region in the midst of a multiyear drought — data centers consumed more than a quarter of all water used in the city last year. Here in California, 50 data centers in Santa Clara now consume 60 percent of the city’s electricity.123 Policymakers in these communities have put forward proposals to incentivize companies to meet energy efficiency standards,124 require reporting of estimated or current energy or water consumption,125 mandate sustainability targets,126 and prohibit the development of data centers in certain areas.127 Additionally, some experts have called for clear reporting standards or rating uniform rating systems to enable more accurate assessment and comparison of the environmental impacts of AI models.128 In recognition of the need for standardization, earlier this year Congressman Ed Markey introduced legislation to require the federal government to assess AI’s current environmental footprint and create a voluntary framework to report environmental impacts.129 However, some researchers are skeptical that companies will follow voluntary disclosures. California’s response to these issues is beyond the scope of the recommendations of this report, but the Commission feels it is important to note that critical environmental policy choices await the state’s leaders as they address the future use of artificial intelligence. 38 | LITTLE HOOVER COMMISSION Glossary of AI Terms 1. Artificial Intelligence (AI): Technology that imitates cognitive processes typically performed by humans such as decision-making or the pursuit of goals. 2. Generative AI (GenAI): A type of AI capable of “generating” novel content like text, images, audio, or video. 3. Machine Learning (ML): A type of AI capable of self-improvement by “learning” from its data. 4. Chatbot: An AI tool that can converse with humans through text or speech. ChatGPT is a well known example of a chatbot. 5. Algorithm: A set of rules or instructions that guide an AI application as it processes data and makes decisions. 6. Neural Network: A computational model patterned after the structure of the human brain. 7. Deep Learning: A multi-layered neural network capable of advanced processing. 8. Reinforcement Learning: A type of machine learning that improves itself through external feedback (positive or negative) based on its actions. 9. Unsupervised Learning: A type of machine learning capable of finding patterns in data without external feedback. 10. Automated Decision Systems (ADS): AI systems that make decisions without human supervision. 11. Ethics in AI: The moral dimensions of AI, addressing concerns relating to transparency, fairness, privacy, etc. 12. AI Governance: IT policy intended to ensure that AI systems are used safely and ethically. 13. Bias in AI: Systematically skewed AI outcomes due to problems in design, training data, or use. 14. Explainable AI (XAI): An AI system whose decisions can be understood by humans and explained to others. 15. Cloud Computing: Use of remote servers for computing applications rather than local servers or personal computers. 16. Quantum Computing: Computing based on quantum theory. Capable of exponentially faster and more complex processing. ARTIFICIAL INTELLIGENCE AND CALIFORNIA STATE GOVERNMENT | 39 Appendix A: Executive Order N-12-13 (Synopsis) Section Action Agencies Date 1 Benefit-risk report on GenAI deployment by the State GovOps, Tech, Data, Nov. 21, 2023 GOBiz 2 Confidential risk assessment re: GenAI impact on energy infrastructures, other Cyber, Threat, Tech, Mar., 2024 critical infrastructures, and mass casualty events Military, CHP 3a Guidelines for procurement, use, and training re: GenAI in the public sector GovOps, Tech, DGS, Cyber Mar., 2024 3b Guidelines for individual agency analyses of GenAI use on vulnerable GovOps, Tech, ODI Jul., 2024 communities/ensuring equitable outcomes 3c Updates to State’s project, approval, and contract terms GovOps, DGS, Tech Jan., 2025 3d Inventory of high-risk GenAI uses by all agencies/depts. subject to the Governor Administered by Tech Within 60 days of EO 3e All agencies/depts. subject to the Governor shall consider procurement and N/A None potential enterprise use opportunities of GenAI to improve efficiency 3f Create a “sandbox” infrastructure to safely pilot GenAI projects Tech Mar., 2024 3g All agencies/depts. subject to the Governor shall consider GenAI pilot projects All subject to Governor Jul., 2024 to improve employee performance and public experience 4 Consult with stakeholders (Legislature, labor, vulnerable and marginalized GovOps, HR, DGS, Tech, N/A communities) re: all EO actions ODI, Cyber 5a Provide training to state workers to ensure high quality, equitable use of GenAI GovOps, Tech, HR, Labor Jul., 2024 5b Develop criteria to evaluate impact of GenAI on state government workforce GovOps, HR, Labor Jan. 1, 2024 and organizations 6 Establish a formal partnership and hold a GenAI summit with UC Berkeley’s GOBiz, GovOps Held May 29, 2024 College of Computing, Data Science, and Society and Stanford’s Institute for Human-Centered Artificial Intelligence 7 Evaluate potential impact of GenAI on regulatory issues Legal counsel (Periodic) Abbreviations: GovOps = Government Operations Agency Tech = Department of Technology DGS = Department of General Services Cyber = Cybersecurity Integration Center Data = Office of Data & Innovation CHP = Highway Patrol Threat = Threat Assessment Center Military = Military Department HR = Department of Human Resources Labor = Labor and Workforce Development Agency GOBiz = Gov.’s Office of Business & Economic Development 40 | LITTLE HOOVER COMMISSION Appendix B: GenAI Capabilities The U.S. Office of Personnel Management has created the following list of GenAI capabilities, which gives a snapshot of how it can assist in a variety of workflow tasks:130 Automation & Efficiency Increasing efficiency of manual or repetitive tasks Accelerating software development by generating, explaining, or debugging code, or by translating code from one programming language to another Enhancing customer experience by creating customized responses to inquiries Content Creation & Drafting and designing communication products, including generating outlines or text and images for Communication reports, presentations, emails, web, or social media Improving and refining writing, including providing grammar, spelling, and style suggestions and transforming complex concepts into plain language Translating products into other languages Accessing Information Synthesizing information, such as summarizing meeting notes or transcripts, or extracting the main idea or key elements from documents Improving accessibility through features such as audio and video transcription Creativity and Analysis Brainstorming and ideating Analyzing and deriving insights from datasets and generating graphs ARTIFICIAL INTELLIGENCE AND CALIFORNIA STATE GOVERNMENT | 41 Appendix C: Technical Implementation AI Communities: The mission of the state’s existing “Artificial Intelligence Community” is “to bring state experts together in shaping the future of ethical, transparent, and trustworthy Artificial Intelligence” and to “foster awareness on AI and to share best practices, use cases, resources, and lessons learned”.131 Through its webinars, this community provides updates on AI-related efforts within the state and features talks by guest speakers on various aspects of AI. Opportunities for feedback and questions from the community are also provided. The community plays a crucial role and its function should be expanded for several reasons. First, there is already considerable buy-in. This is evidenced by the high attendance at these events, which often draw around 200 participants. Second, these participants represent a wide range of state agencies but are united by their enthusiasm for how AI can enhance state service. The culture is remarkable, characterized by genuine passion and excitement for this technology. Given its strong and diverse participation, this community of practice is particularly well-positioned to disseminate its benefits efficiently throughout state agencies. To achieve this, its role as a hub for AI- related brainstorming and idea-sharing should be expanded beyond its current webinar series. This can be accomplished through online programs, or asynchronous online discussions in which participants can contribute to ongoing threads. Additionally, specialized AI Communities should be established within larger state agencies. These agency-based AI Communities, which must be open to all agency employees, would organize discussions around GenAI-related topics tailored to the specific needs and challenges of that agency. By creating such communities, agencies will cultivate specialized expertise, foster collaboration, and generate innovative solutions that are directly relevant to their operations. Furthermore, these agency-level AI Communities could collaborate with one another and the central AIC to share insights, lessons learned, and best practices. Sharing GenAI Apps: There are many specific procedures the state might use to share GenAI applications among agencies. Here, we offer some options: First, California might consider adopting a shared services model, where broadly useful GenAI tools are maintained and made available through a central entity.132 This would allow agencies to purchase and fine-tune to the tool while reducing the need for additional independent contracts with external companies. The U.S. General Services Administration maintains a robust shared services portfolio, which agencies can then purchase for a competitive price.133 California may wish to emulate this model—ideally facilitated by the AI computing center proposed in our initial recommendation—as it seeks to expand the use of GenAI in its operations. Open-source offers another powerful solution for scaling GenAI tools efficiently and cost-effectively.134 By eliminating the need for expensive licensing fees, open-source GenAI significantly reduces the financial burden of deploying GenAI tools on a wide scale. Additionally, the flexibility of open-source code allows agencies to customize the GenAI tools to meet their specific needs without relying on external vendors for every modification.135 This adaptability ensures that each agency can tailor their AI tools to their unique requirements quickly and efficiently. 42 | LITTLE HOOVER COMMISSION And, finally, open-source AI fosters a collaborative environment where state agencies can work together to improve and enhance the GenAI tools they use and avoid duplicating efforts. The transparency of open-source GenAI will also help ensure that these tools meet the state’s security and privacy standards.136 Agency Position Statements: Key features of such a position statement might include: ◊ Clarification of Roles: Clearly define the specific roles of human workers versus technology in the agency’s operations.137 ◊ Commitment to Human-Centered Values: Emphasize the agency’s commitment to preserving human judgment, empathy, and creativity in carrying out their mission.138 ◊ Integration of Technology: Outline where and how technology will be integrated to support human efforts, specifying tasks that will be augmented or automated by technology. ◊ Transparency in Decision-Making: Highlight the agency’s commitment to transparency in how decisions regarding the use of technology are made, including stakeholder involvement and alignment with broader state objectives. ◊ Operational Efficiency: Describe the expected improvements in operational efficiency through the use of technology, ensuring that these efficiencies are in service of enhancing overall mission and service delivery. ◊ Balance between Efficiency and Human Touch: Stress the importance of balancing technological efficiencies with maintaining a human-centered approach, especially in public-facing services where human interaction is critical.139 ◊ Alignment with State Objectives: Ensure the position statement aligns with the broader goals and objectives of the state of California. ◊ Commitment to Ongoing Evaluation: Include a commitment to regularly evaluate the impact of technology on both workers and service delivery. ◊ Public Communication and Engagement: Plan for regular communication with the public about how technology is being used with opportunities for feedback and engagement. ARTIFICIAL INTELLIGENCE AND CALIFORNIA STATE GOVERNMENT | 43 Future Proofing Beyond GenAI: Thus the Commission pro1p. oses the following “future proof” criteria for evaluating technologies being considered for adoption: Criterion Description Quality of Output The technology’s results are accurate and free from algorithmic bias. Data Security and Privacy The technology can protect data from breaches and guarantee individual privacy. Ethical Considerations The technology aligns with societal values and serves the public good. Legal and Regulatory Compliance The technology complies with laws and regulations. Transparency and Explicatbility The technology’s processes are understandable and can be explained to others. User Accessibility and Inclusivity The technology is accessible to all potential users. Controllability and Human The technology remains under human oversight and can be stopped if Oversight necessary. Interoperability and Integration The technology can be integrated with existing systems. Scalability and Adaptability The technology can be scaled and can adapt to new situations and conditions. 44 | LITTLE HOOVER COMMISSION Notes 1. Little Hoover Commission, A Review of Government 15. Oracle Cloud Infrastructure: “What is Cloud Computing?” Reorganization Plan No. 2 [Report #211] (May, 2012). website: https://www.oracle.com/cloud/what-is-cloud- computing/. 2. Auditor of the State of California, FI$Cal Status: 16. NAIRR Pilot website: https://nairrpilot.org/. Twin bills California’s New, Centralized Fiscal System Will Miss Its (CREATE AI Act, S.2714 and H.R.5077, 118th Cong. Completion Target Again While Agencies Still Struggle to (2023)) are currently proceeding through the House Use the New Technology (Jan. 4, 2022). and Senate. For an extensive range of commentary on the NAIRR, see the Stanford HAI National AI Research 3. Policy documents deriving from Gov. Newsom’s Resource website: https://hai.stanford.edu/policy/na- executive order on GenAI use variations of this tional-ai-research-resource. definition, as do a significant proportion of AI-related 17. Quoted from the About NAIRR Pilot website: https:// bills. Still, a need has been perceived for an official legal nairrpilot.org/about. definition of AI (see AB 2885 (Bauer-Kahan) and SB 721 (Becker)) and there is concern that lack of clarity about 18. National Artificial Intelligence Research Resource Task the meaning of AI could hinder regulatory efforts. L. Force, “Strengthening and Democratizing the U.S. Arti- Korte, J. White, and D. Gardiner, “The Bumpy Path Ahead ficial Intelligence Innovation Ecosystem: An Implemen- tation Plan for a National Artificial Intelligence Research for AI”, Politico California Playbook (Apr. 17, 2024). Resource” (Jan., 2023). 4. A clear and helpful glossary of concepts relating to AI has 19. As of now, a state-run computing center to support been created by Stanford HAI. See C. Manning, “Artificial public-interest AI would be unprecedented. Gov. Kathy Intelligence Definitions”, Stanford University Human-Cen- Hochul has announced the formation of the Empire AI tered Artificial Intelligence (2020). Consortium to explore the creation of an AI computing center in Upstate New York: Press Release, “Governor 5. The GeeksforGeeks website provides a helpful overview Hochul Unveils Fifth Proposal of 2024 State of the State: of the many fields to which AI has been applied, includ- Empire AI Consortium to Make New York the National ing e-commerce, education, robotics, navigation, health- Leader in AI Research and Innovation”, Office of Gov. care, vehicles, agriculture, human resources, and more: Kathy Hochul (Jan. 8, 2024). https://www.geeksforgeeks.org/applications-of-ai/ 20. The California Department of Technology already oper- 6. See nn. 98 and 99 for two contrasting examples. ates large data centers in Rancho Cordova and Vacaville. 7. L. Hu, “ChatGPT Sets Record for Fastest-Growing User See the CDT Data Center Overview website: https://cdt. Base”, Reuters (Feb. 2, 2023) ca.gov/services/data-center/. 8. Cal. Exec. Order N-12-23 (Sep. 6, 2023). 21. U.S. Government Accountability Office, “Cloud Comput- ing: Agencies Need to Address Key OMB Procurement 9. For a running list of GenAI deliverables stemming from Requirements” (Sep. 20, 2024). EO N-12-23, see the California Department of Technolo- gy’s GenAI Executive Order website: https://cdt.ca.gov/ 22. T.C. Irwin, S. Mazraani, and S. Saxena, “How to Control technology-innovation/artificial-intelligence-community/ the Fiscal Costs of Public-Private Partnerships”, Interna- genai-executive-order/. tional Monetary Fund (2018). 10. E. Eidam, “California Taps Vendors to Explore Generative 23. For an overview of the ELSI Program, see: https:// AI Use Cases”, GovTech (May 9, 2024). www.genome.gov/Funded-Programs-Projects/ELSI-Re- search-Program-ethical-legal-social-implications. 11. Press Release: “Governor Newsom Announces New Initiatives to Advance Safe and Responsible AI, Protect 24. “Diversity in AI is Essential to our Future”, Pivotal Ven- Californians”, Office of the Governor (Sep. 29, 2024). tures (Dec. 7, 2023). 12. Gov. Newsom’s executive order on generative artificial 25. Joon Solutions Global, “What is Vendor Lock-In and intelligence (N-12-23) contains similar praise of Califor- How Can You Avoid It?”, Medium (May 25, 2023). Quixy nia’s role in the development of AI, as does the GovOps Editorial Team, “What is Vendor Lock-In? Tips to Avoid Benefits and Risks of GenAI Report (p. 6). It” (Apr. 3, 2024). “Vendor Lock-In in Cloud Computing”, Geeks for Geeks (Jan. 27, 2023). 13. SB 1047 (Wiener) Veto Message, Office of the Governor (Sep. 29, 2024). 26. C. Anderson et al., “Science at Extreme Scales: Where Big Data Meets Large-Scale Computing”, IPAM Long Program 14. For a helpful introduction, see S. Susnjara, “Public Cloud White Paper (Fall 2018). Cf. N. Ahmed and N.C. Thomp- Use Cases: 10 Ways Organizations are Leveraging Public son, “What Should be Done About the Growing Influence Cloud”, IBM (Mar. 20, 2024). of Industry in AI Research?”, Brookings (Dec. 5, 2023). ARTIFICIAL INTELLIGENCE AND CALIFORNIA STATE GOVERNMENT | 45 27. OpenAI, “Introducing OpenAI for Nonprofits” (May 30, 37. GovTechSingapore “Pair” website: https://www.tech.gov. 2024). S. Di Troia et al., “Inspiring Action: Identifying the sg/products-and-services/for-government-agencies/pro- Social Sector AI Opportunity Gap”, Stanford HAI [working ductivity-and-marketing/pair/. paper] (Feb., 2024). 38. Cal. Exec. Order N-12-23 (Sep. 6, 2023), 3.e. Cf. S. 28. J. Wang and M. Muro, “How the National Artificial Intelli- Fox-Sowell, “California Announces 6-Month Trial of Gen- gence Research Resource can Pilot Inclusive AI”, Brook- erative AI Tools Inside State Government”, StateScoop ings (Jul. 9, 2024). (May 10, 2024). 29. B. Freed, “State Workforce Trends Show ‘Mixed Bag’ for 39. Editorial Staff, “Data Sandbox”, DevX (Oct. 16, 2023). For CIOs as Turnover Increases”, StateScoop (Jul. 6, 2023). a useful overview of how sandboxes can help mitigate GenAI risk, see GovTech/Center for Digital Government, 30. Later in this report, we recommend that the state join Using a Sandbox to Explore Generative AI (2023). the GovAI Coalition, a consortium of governments (large- ly city and county) that are working together to develop 40. To avoid any confusion, the Commission is recommend- AI policy best practices. In developing an AI compute ing that all state workers take the introductory course center, California could both draw on the collective titled “Responsible AI for Public Professionals 24-25” on expertise of this Coalition and, once built, share its re- the CalLearns website. On the InnovateUS site, the same sources with its members. course is titled “Responsible AI for Public Professionals: Using Generative AI at Work” and is part of a larger “Re- 31. S. Noy and W. Zhang, “Experimental Evidence on the sponsible AI for Public Professionals” series. Productivity Effects of Generative Artificial Intelligence”, Science 381.6654 (Jul. 13, 2023): 187-192. Cf. E. Bryn- 41. See SAS, Generative AI Global Research Report: Strat- jolfsson, D. Li, and L. Raymond, “Generative AI at Work”, egies for a Competitive Advantage (2024) and data National Bureau of Economic Research (Nov., 2023), dashboard. For an overview, see N. Davidson, “What which found that GenAI increases productivity by 14% Government Can Learn from the Private Sector About overall and by 34% in the case of novice and low-skilled AI”, GovTech (Jul. 10, 2024). workers. 42. On the many benefits of peer-to-peer training, see K. 32. K. Lakhani et al., “Navigating the Jagged Technological Palmer and D. Blake, “How to Help Your Employees Frontier: Field Experimental Evidence of the Effects of AI Learn from Each Other”, Harvard Business Review (Nov. on Knowledge Worker Productivity and Quality”, Harvard 8, 2018). Business School Technology & Operations Mgt. Unit Working Paper [No. 24-013] (2023). For a reader-friendly 43. GovOps/CalHR/ODI/CDT, State of California: Building a synopsis, see D. Berreby, “ChatGPT Helps, and Worries, GenAI Ready Workforce (Jul., 2024), 9. Business Consultants, Study Finds”, New York Times (Dec. 44. Executive Office of the President, “Fact Sheet: OMB 28, 2023). Issues Guidance to Advance the Responsible Acquisition 33. See, for example, D. Rigby, J. Sutherland, and A. Noble, of AI in Government” (Oct. 3, 2024). “Agile at Scale”, Harvard Business Review (May-June, 45. National Institute of Standards and Technology (NIST), AI 2018). Risk Management Framework (Jul. 26, 2024). 34. C. Kang, C. Metz, and S. Thompson, “Four Takeaways 46. National Institute of Standards and Technology (NIST), on the Race to Amass Data for A.I.”, New York Times Artificial Intelligence Risk Management Framework: Gen- (Apr. 6, 2024) and S. Morrison, “The Tricky Truth About erative Artificial Intelligence Profile (Jul., 2024). how Generative AI Uses Your Data”, Vox (Jul. 27, 2023). ChatGPT allows users to not have their data used to train 47. Executive Office of the President, Executive Order on the future models, but only on an opt-out basis. Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence (Oct. 30, 2023). 35. See also C. Teale, “What is Generative AI? Most of the Public Sector Workforce Doesn’t Know”, Route Fifty (Jul. 48. Executive Office of the President, Advancing Gover- 17, 2024). nance, Innovation, and Risk Management for Agency Use of Artificial Intelligence (Mar. 28, 2024). 36. Here are the links to the TritonGPT and ZotGPT web- pages. For an overview of their development, see M. 49. Executive Office of the President, Memo: Advancing the Attridge, “Campus-GPT: How 2 University of California Responsible Acquisition of Artificial Intelligence in Gov- Campuses Are Designing Their Own Specialized AI Tools”, ernment (Sep. 24, 2024). Best Colleges (Apr. 24, 2024). 50. AI.gov., Join the National AI Talent Surge website: https://ai.gov/apply/. 46 | LITTLE HOOVER COMMISSION 51. The U.S. National Science Foundation, “NSF launches Differences”, Hertie School for Digital Governance Blog EducateAI initiative” (Dec. 5, 2023). (Apr. 15, 2024). 52. U.S. National Science Foundation, “NSF Investing Nearly 65. It is currently required that all written solicitations $8M in EducateAI Awards to Develop Next Generation of include “Mandatory Disclosure Language” and the GenAI Well-Trained AI Workforce” (Sep. 4, 2024). Reporting and Factsheet, even for non-IT purchases. Generative AI Toolkit: https://genai.cdt.ca.gov/procure- 53. For a helpful review of the benefits of and best prac- ment/contract-disclosure-and-special-provisions.html. tices in self-directed learning in professional contexts, see S. Park, “Self-Directed Learning in the Workplace”, 66. This information is disclosed in the GenAI Reporting and Academy of Human Resource Development Conference Factsheet (STD 1000) form. Proceedings (2008). 67. IBM Research AI Factsheets 360 Audio Classifier: https:// 54. For a list of recent Best of California award winners, see aifs360.res.ibm.com/examples/max_audio_classifier. D. Noone, “IT Leaders, State Projects Recognized as ‘Best of California’ for 2024”, GovTech (Aug. 20, 2024). 68. An example is the GovAI Coalition’s AI Incident Response Plan: https://www.sanjoseca.gov/home/showpublished- 55. See, for example, T. Mickle, “The A.I. Boom Has an Un- document/109732/638458752836100000; cf. the NIST likely Early Winner: Wonky Consultants”, New York Times Computer Security Incident Handling Guide: https://nvl- (Jun. 26, 2024). California’s initial five GenAI RFI2 pilot pubs.nist.gov/nistpubs/specialpublications/nist.sp.800- projects will be developed in conjunction with consul- 61r2.pdf. tancies like Deloitte, Accenture, and Ignyte Group. See also “State Announces GenAI Pilot Partnerships with Five 69. D. Piorkowski et al., “Towards Evaluating and Eliciting Vendors”, GovReport (May 10, 2024). High-Quality Documentation for Intelligent Systems”, IBM Research [Working Paper] (Nov., 2020). 56. EY Center for Government Modernization, EY 2024 Fed- eral, State and Local Trends Report: Key Findings (2024). 70. Cal. Exec. Order N-12-23, 5b. 57. EY Center for Government Modernization, Trends Report, 71. Y. Shen and X. Zhang, “The Impact of Artificial Intelli- op. cit. (2024). gence on Employment: The Role of Virtual Agglomera- tion”, Nature (Jan. 17, 2024). 58. The AI Community of Practice website can be found at: https://coe.gsa.gov/communities/ai.html. 72. Shen and Zhang (2024): p. 4. The “classic study” is C.B. Frey and M.A. Osborne, “The Future of Employment: 59. For example, the Reporting and Factsheet (STD 1000) How Susceptible are Jobs to Computerisation?”, Tech- forms required for all GenAI purchases. As it stands, nological Forecasting and Social Change 114 (2017): these must be solicited on a purchase-by-purchase basis. 254–280. 60. Centers for Medicare and Medicaid Services, National 73. See C. Flaherty, “Tracking the Evolution (and Erosion) of Health Expenditures Data (Historical) website: https:// Tenure”, Inside Higher Ed (May 17, 2022). www.cms.gov/data-research/statistics-trends-and-re- ports/national-health-expenditure-data/historical 74. The event was co-organized by UC Berkeley’s College of Computing, Data Science, and Society, Stanford’s Insti- 61. Fuller discussions of many of these items can be found in tute for Human-Centered AI, the Governor’s Office of N. Mittal and I. Saif, “The Government & Public Services Business and Economic Development (GO-Biz), and the AI Dossier” and “The Life Sciences & Health Care AI Dos- California Government Operations Agency (GovOps). sier”, Deloitte AI Institute (2021). 75. For a detailed report on the value of ombudspeople 62. Cf. S. Rispens, “San Jose, Calif., Forms Nationwide Gov- in government, see C. Houk et al., “A Reappraisal: The ernment AI Coalition”, StateScoop (Mar. 21, 2024). Nature and Value of Ombudsmen in Federal Agencies”, Administrative Conference of the United States (Nov. 14, 63. GovAI Coalition Deliverables: https://www.sanjoseca. 2016). gov/your-government/departments-offices/informa- tion-technology/artificial-intelligence-inventory/gov- 76. “Americans Hold Mixed Opinions on AI and Fear its ai-coalition/templates-resources Potential to Disrupt Society, Drive Misinformation”, Ipsos News (May 4, 2023). Cf. M. Faverio and A. Tyson, “What 64. E.g., AB 2839, AB 1831, and AB 1836. For a comparison the Data Says about Americans’ Views of Artificial Intel- of the risk-based approaches to AI governance found in ligence”, Pew Research (Nov. 21, 2023), which also notes the EU AI Act and President Biden’s Executive Order on that public concern over AI is increasing. the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence, see T. Jelinek, “AI Governance: 77. M. Baldassare et al., “PPIC Statewide Survey: Californians EU and US Converge on Risk-Based Approach Amid Stark and their Economic Well-Being”, PPIC (Nov., 2023). ARTIFICIAL INTELLIGENCE AND CALIFORNIA STATE GOVERNMENT | 47 78. N. Conneely, “What is a Hackathon? (Ultimate Guide)”, the Digital Era” in June of 2024. For a write up of the Fliplet (Jan. 17, 2024). hearing, see A. Sheeler, “Is AI a Threat to Our Elections? California Lawmakers Warn Public is ‘Ill-Prepared’”, Sac- 79. Cincinnati Housing Stabilization Hackathon: https:// ramento Bee (Jun. 5, 2024). www.flywheelcincinnati.org/housing-stabilization-hack- athon. AIFS hackathon: https://aifs.ucdavis.edu/news- 90. These bills were identified using the Legislative Informa- and-events/2024-ag-extension-advisor-hackathon tion System text search “artificial intelligence”. In 2013- 14, SB 860 (Budget) provided support for the BRAIN 80. For State of California data-related hackathons, see: initiative, which, among other things, would support AI https://data.ca.gov/about. research by mapping the human brain. 81. New York Times columnist Kevin Roose’s disturbing con- 91. A helpful search engine for federal and state AI leg- versations with Microsoft’s chatbot, then called Bing, is islation from 2023 onward has been created by the an infamous example (“A Conversation with Bing’s Chat- CITRIS Policy Lab at UC Berkeley and can be accessed at: bot Left Me Deeply Unsettled”), New York Times (Feb. https://citrispolicylab.org/ailegislation/. An older spread- 16, 2023)). Air Canada’s GenAI chatbot gave a passenger sheet version of this site, which covered bills introduced misleading information about its refund policy. When prior to 2023, was created but currently seems to be the customer took Air Canada to court, the airline was unavailable. forced to honor the incorrect statements made by the chatbot (L. Cecco, “Air Canada Ordered to Pay Customer 92. C. Kang and A. Satariano, “As A.I. Booms, Lawmakers who was Misled by Airline’s Chatbot”), The Guardian Struggle to Understand the Technology”, New York Times (Feb. 16, 2024)). (Mar. 3, 2023). The U.S. Supreme Court has also admit- ted its general lack of knowledge about technology; see 82. Cf. K. Colton et al., “Great Expectations: How US gov- B. Ortutay, “Why the Supreme Court Tiptoeing Past a ernment Agencies Can Meet Public Demand for Better Key Social Media Shield Helps Big Tech”, Associated Press Service”, McKinsey & Company (Dec., 2023). (May 18, 2023). 83. S.M. Miller, “Singapore’s AI Applications in the Public 93. For the Stanford HAI Congressional Boot Camp Sector: Six Examples”, Management and Business Review on AI, see: https://hai.stanford.edu/congressio- (2023). nal-boot-camp-ai. 84. According to Cal. Government Code §11546.45.5, “Auto- 94. On Rep. Don Beyer’s educational pursuits, see D. Klep- mated decision system” means a computational process per, “A Congressman Wanted to Understand AI. So He derived from machine learning, statistical modeling, data Went Back to a College Classroom to Learn”, Associated analytics, or artificial intelligence that issues simplified Press (Apr. 11, 2024). output, including a score, classification, or recommen- dation, that is used to assist or replace human discre- 95. It is notable that the LAO’s federal counterpart, the Con- tionary decision-making and materially impacts natural gressional Research Service, has produced briefings on persons. AI for its constituents: https://crsreports.congress.gov/ product/pdf/R/R47644. 85. AB 2930 (Bauer-Kahan). Cf. SB 942 (Becker), SB 896 (Dodd), AB 2885 (Bauer-Kahan). 96. This synopsis is based on the EU Artificial Intelligence Act, “High-Level Summary of the AI Act” (Feb. 27, 2024). 86. AB 302 (Ward) mandates that this inventory be provided to the Assembly Committee on Privacy and Consumer 97. Cf. n. 5. Protection and Senate Committee on Governmental Organization only. 98. UC San Diego Press Release, “ALERTCalifornia and CAL FIRE’s Fire Detection AI Program Named One of TIME’s 87. For a concise history of the ebbs and flows of AI, see Best Inventions of 2023” (Oct. 24, 2023). cdteliot, “A Historical Overview of AI Winter Cycles”, Perplexity (Jul. 23, 2024). 99. C. Engel, L. Linhardt, and M. Schubert, “Code is Law: How COMPAS Affects the Way the Judiciary Handles the 88. For a full list of Assembly select committees, see: https:// Risk of Recidivism”, Artificial Intelligence and Law (2024). www.assembly.ca.gov/committees#Select-Committees. For the Senate, see: https://www.senate.ca.gov/commit- 100. K. Johnson, “If California Government Wants to Use tees. AI, it Will Have to Follow these New Rules”, CalMatters (Mar. 21, 2024). 89. For example, the Senate Elections and Constitutional Amendments Committee and Assembly Elections Com- 101. A. Sheeler, “Hackers Used Stolen Passwords to Access mittee held a joint informational hearing on “Artificial Thousands of BenefitsCal Accounts”, Sacramento Bee Intelligence and Elections: Protecting Democracy in (Apr. 11, 2024). 48 | LITTLE HOOVER COMMISSION 102. N. Maslej et al., Artificial Intelligence Index Report 2024, 117. M. Peterson, “How Much More Water and Power Does AI Index Steering Committee, Institute for Human-Cen- AI Computing Demand? Tech Firms Don’t Want You to tered AI, Stanford University (Apr., 2024). Know”, Los Angeles Times (Aug. 26, 2024). 103. Organisation for Economic Co-Operation and Devel- 118. B. Smith, “Microsoft Will be Carbon Negative by 2030”, opment, Working Party on Artificial Intelligence Gover- Microsoft (Jan. 16, 2020). Also, B. Smith, “Microsoft will nance, Measuring the Environmental Impact of AI Com- Replenish More Water Than it Consumes by 2030”, Mic- pute and Applications: The AI Footprint (Nov. 17, 2022). rosoft (Sep. 21, 2020). 104. Little attention has been paid to the impacts that occur 119. A. Rathi and D. Bass, “Microsoft’s AI Push Imperils Cli- at other points of the AI lifecycle, including production, mate Goal as Carbon Emissions Jump 30%”, Bloomberg transport, and end-of-life. These impacts, such as the (May 15, 2024). soil and air contamination that can occur when mining earth materials used to create high-powered AI process- 120. One-fifth of data centers rely on water from moderately ing chips or hazardous waste from the disposal of these to highly stressed watersheds. Source: M.A.B. Siddik chips, currently lack studies focusing specifically on AI-re- et al., “The Environmental Footprint of Data Centers in lated impacts. the United States”, Environmental Research Letters 16.6 (May, 2021). 105. J. Vincent. “How Much Electricity Does AI Consume?”, The Verge (Feb. 16, 2024). 121. A. Olivo, “Internet Data Centers are Fueling Drive to Old Power Source: Coal”, Washington Post (Apr. 17, 2024). 106. Stanford University Human-Centered Artificial Intelli- gence, Artificial Intelligence Index Report 2024. 122. Science Friday, “Understanding and Curbing Generative AI’s Energy Consumption” (Feb. 16, 2024). 107. M. Peterson, “How Much More Water and Power Does AI Computing Demand? Tech Firms Don’t Want You to 123. M. Peterson, “Power-Hungry AI Data Centers are Raising Know”, Los Angeles Times (Aug. 26, 2024). Electric Bills and Blackout Risk”, Los Angeles Times (Aug. 12, 2024). 108. S. Ren. “How Much Water Does AI Consume? The Public Deserves to Know”, OECD.AI (Nov. 30, 2023). Also, P. Li et 124. VA HB 116. al., “Making AI Less ‘Thirsty:’ Uncovering and Addressing 125. VA HB 910 & VA SB 285 the Secret Water Footprint of AI Models”, arXiv.org (Oct. 29, 2023). 126. OR HB 2816. 109. B. Daigle, “Data Centers Around the World: A Quick 127. VA HB338 & VA HB1010. Look”, United States International Trade Commission (May 2021). 128. N. Bashir et al., “The Climate and Sustainability Implica- tions of Generative AI”, An MIT Exploration of Generative 110. Data Center Map. California Data Centers. https://www. AI (March 27, 2024). Also, J.B. Keller et al., “The US Must datacentermap.com/usa/california/. Balance Climate Justice Challenges in the Era of Artificial Intelligence”, Brookings (Jan. 29, 2024). Also, J. Vincent, 111. U.S. data centers consumed four percent of the nation’s “How Much Electricity Does AI Consume?”, The Verge electricity in 2023 and could consume over 9 percent by (Feb. 16, 2024). 2030. Source: Electric Power Research Institute, Power- ing Intelligence: Analyzing Artificial Intelligence and Data 129. Office of Ed Markey, United States Senator, Massachu- Center Energy Consumptionz (May 28, 2024). setts, “Markey, Heinrich, Eshoo, Beyer Introduce Legis- lation to Investigate, Measure Environmental Impacts of 112. G. Hering and K. Fogarty, “With Cleaner Options Avail- Artificial Intelligence” (Feb. 1, 2024). able, Data Centers Double Down on Diesel”, S&P Global (May 20, 2020). 130. U.S. Office of Personnel Management, “Responsible Use of Generative Artificial Intelligence for the Federal 113. B. Geman, “AI Data Center Demand Poised to Juice U.S. Workforce”: https://www.opm.gov/data/resources/ Power Needs”, Axios (May 30, 2024). ai-guidance/. GenAI capabilities are quoted verbatim. 114. Goldman Sachs, “AI is Poised to Drive 160% Increase in The categories (e.g., “Automation & Efficiency”) were Data Center Power Demand” (May 14, 2024). created by Little Hoover Commission staff. 115. Google, Net-Zero Carbon websitge: https://sustainabili- 131. See the AIC homepage with link to past and future ty.google/operating-sustainably/net-zero-carbon/. meetings: (https://cdt.ca.gov/technology-innovation/ artificial-intelligence-community/). 116. D. Milmo, “Google’s Emissions Climb Nearly 50% in Five Years Due to AI Energy Demand”, The Guardian (Jul. 2, 132. “Generative AI in Shared Services: An Introductory 2024). Guide”, SSON (May 15, 2023). ARTIFICIAL INTELLIGENCE AND CALIFORNIA STATE GOVERNMENT | 49 133. U.S. General Services “Shared Services” website: https:// www.gsa.gov/buy-through-us/shared-services. 134. A. Smith and D. Farris, “The Case for Open-Source Gen- erative AI in Government”, Booz Allen Hamilton Perspec- tives (n.d.). See also M. Zuckerberg, “Open Source AI is the Path Forward”, Meta Newsroom (Jul. 23, 2024). Meta has been a leader in open-source GenAI tools and has ar- gued strongly for its proliferation. Shane Witnov, Privacy and Policy Director at Meta, made this case at the May 25, 2024 Little Hoover Commission hearing. 135. “The Rise of Customized Generative AI Models in Enter- prises”, Kaizen Institute (n.d.). 136. As mentioned above, UC San Diego used Meta’s open- source Llama LLM in the development of TritonGPT in part because open source GenAI tools provide users with greater control over security issues. 137. J. Hreha, “Balancing AI, Automation, and Human Over- sight in the Workplace”, Persona (Jun. 10, 2023). 138. H. Mann, “The New AI: Artificial Integrity”, Duke Corpo- rate Education (Jun., 2024). 139. P. Kiernan, “Balancing Tech and Humanity: Innovate Without Losing Touch, Part 2”, ThoughtLab (Sep. 12, 2024). 50 | LITTLE HOOVER COMMISSION Little Hoover Commission Members CHAIR PEDRO NAVA | Santa Barbara JOSÉ ATILIO HERNÁNDEZ | Burbank Appointed to the Commission by Speaker of the Assembly Appointed by Speaker of the Assembly Anthony Rendon John Pérez in April 2013 and reappointed by Speaker in April 2023. Founder and CEO of IDEATE California, a of the Assembly Anthony Rendon in 2017 and again public relations and policy management firm. Also, founder in 2021. Government relations advisor. Former State and Board Chairman of ideateLABS. Former Director for Assemblymember from 2004 to 2010, civil litigator, External Affairs and Community Relations for ConnectEd: deputy district attorney and member of the state Coastal The California Center for College and Career. Commission. Elected chair of the Commission in March 2014. JASON JOHNSON | Napa Appointed by Governor Newsom in June 2023. Member VICE CHAIR ANTHONY CANNELLA | Ceres of the Land Trust of Napa County Board of Trustees and Appointed to the Commission by the Senate Rules Horary Commander of Travis Air Force Base. Former Committee in March 2022. Civil engineer and principal with Managing Partner at Founders Den. Founder and former Northstar Engineering Group. Former State Senator from CEO at August Home Inc. 2010 to 2018. Previously served on the Ceres City Council and was twice elected mayor of that city. Elected Vice Chair SEN. DAVE MIN | Irvine of the Commission in July 2023. Appointed to the Commission by the Senate Rules Committee in September 2021. Elected in November 2020 DION ARONER | Berkeley to represent the 37th Senate District. Represents Anaheim Appointed to the Commission by the Senate Rules Hills, Costa Mesa, Huntington Beach, Irvine, Laguna Beach, Committee in April 2019. Partner for Aroner, Jewel, and Laguna Woods, Lake Forest, Newport Beach, Orange, Ellis. Former State Assemblymember from 1996 to 2002, Tustin, and Villa Park. chief of staff for Assemblymember Tom Bates, social worker for Alameda County, and the first female president ASM. LIZ ORTEGA | San Leandro of Service Employees International Union 535. Appointed to the Commission by Speaker of the Assembly Anthony Rendon in March 2023. Elected in November DAVID BEIER | San Francisco 2022 to represent the 20th Assembly District. Represents Appointed to the Commission by Governor Edmund G. Hayward, San Leandro, most of Union City, portions Brown Jr. in June 2014 and reappointed in January 2018. of Dublin and Pleasanton, and several unincorporated Managing director of Bay City Capital. Former senior officer communities. of Genentech and Amgen, and counsel to the U.S. House of Representatives Committee on the Judiciary. JANNA SIDLEY | Los Angeles Appointed to the Commission by Governor Edmund G. ASM. PHILLIP CHEN | Yorba Linda Brown Jr. in April 2016 and reappointed in February 2020. Appointed to the Commission by Speaker of the Assembly Partner at Ichor Strategies and appointed to the Board Anthony Rendon in October 2021. Elected in November of the Los Angeles City Employee Retirement System 2016 to represent 55th District. Represents portions of Los (“LACERS”). Former general counsel at the Port of Los Angeles, Orange and San Bernardino counties and the Angeles and city attorney at the Los Angeles City Attorney’s cities of Brea, Chino Hills, Diamond Bar, La Habra, Industry, Office. Placentia, Rowland Heights, Walnut, West Covina and Yorba Linda. SEN. SCOTT WILK | Santa Clarita Appointed to the Commission by the Senate Rules GIL GARCETTI | Los Angeles Committee in April 2023. Elected in November 2016 to Appointed to the Commission by Governor Gavin Newsom represent the 21st Senate District. Represents communities in November 2021. Professional photographer and author in the Antelope, Santa Clarita, and Victor Valleys. of ten books. Former Los Angeles County District Attorney, Full biographies are available on the Commission’s teaching Fellow at Harvard University’s Kennedy School, website at www.lhc.ca.gov. and president of the California Science Center Foundation’s Board of Trustees. ARTIFICIAL INTELLIGENCE AND CALIFORNIA STATE GOVERNMENT “DEMOCRACY ITSELF IS A PROCESS OF CHANGE, AND SATISFACTION AND COMPLACENCY ARE ENEMIES OF GOOD GOVERNMENT.” By Governor Edmund G. “Pat” Brown, addressing the inaugural meeting of the Little Hoover Commission, April 24,1962, Sacramento, California Milton Marks Commission on California State Government Organization and Economy www.lhc.ca.gov