Recommendations

July 2018 Department of Rehabilitation Its Inadequate Guidance and Oversight of the Grant Process Led to Inconsistencies and Perceived Bias in Its Evaluations and Awards of Some Grants Report 2017‑129 COMMITMENT INTEGRITY LEADERSHIP CALIFORNIA STATE AUDITOR 621 Capitol Mall, Suite 1200 | Sacramento | CA | 95814 916.445.0255 | TTY 916.445.0033 For complaints of state employee misconduct, contact us through the Whistleblower Hotline: 1.800.952.5665 Don’t want to miss any of our reports? Subscribe to our email list at auditor.ca.gov For questions regarding the contents of this report, please contact Margarita Fernández, Chief of Public Affairs, at 916.445.0255 This report is also available online at www.auditor.ca.gov | Alternate format reports available upon request | Permission is granted to reproduce reports Elaine M. Howle State Auditor July 12, 2018 2017‑129 The Governor of California President pro Tempore of the Senate Speaker of the Assembly State Capitol Sacramento, California 95814 Dear Governor and Legislative Leaders: As requested by the Joint Legislative Audit Committee, the California State Auditor presents this audit report of the Department of Rehabilitation’s (Rehabilitation) grant process. Rehabilitation works in cooperation with nonprofit agencies to provide services, such as career education and training, assistive technology, and independent living skills training to individuals with disabilities. This report describes the results of our examination of Rehabilitation’s process to solicit grant applications, evaluate applications and award grants, and review any appeals, for a selection of grants awarded to these agencies during fiscal years 2014–15 through 2017–18. This report concludes that Rehabilitation’s inadequate guidance and oversight of the grant process led to inconsistencies and, in certain cases, perceived bias in its evaluations and awards of some grants. Although required to have procedures in place, Rehabilitation failed to formalize a process for soliciting and awarding grants, which resulted in numerous shortcomings and inconsistencies. For instance, Rehabilitation did not clearly define the roles and responsibilities of staff involved in the grant process to ensure they knew how to carry out the process. It also did not adequately solicit stakeholder feedback in developing the requests for applications (RFAs) for most of the grants we reviewed, and the RFAs did not always include adequate scoring criteria, or descriptions of the evaluation, award, and appeals processes. Additionally, Rehabilitation’s poor records management and it not ensuring that staff complied with its records retention policy contributed to its failure to appropriately respond to some public records requests. As it relates to the evaluation of applications, we found that Rehabilitation failed to publish solicitations for individuals evaluating the grant applications (evaluators) and ensure they were free from bias and, for one grant, selected evaluators with previous ties to one of the applicants—creating at least the appearance of potential bias. Rehabilitation also failed to demonstrate that it provided adequate training or written instructions to its evaluators to ensure they understood the evaluation process, including how to score grant applications. This contributed to some scoring inconsistencies, and Rehabilitation sometimes convened new evaluation panels to rescore applications without rectifying the issues that caused the inconsistencies in the original scoring. Without adequate oversight of the grant process, Rehabilitation lacks assurance that staff and evaluators adhered to its grant process, and that it can demonstrate the process was followed as intended. We found that Rehabilitation’s grant review committees responsible for reviewing appeals did not adequately review each appeal, such as for potential evaluator prejudice and whether evaluators’ scores were supported by evidence, as suggested by its grant manual. Further, regarding an appeal of one grant award, a grant review committee made recommendations to Rehabilitation upon identifying deficiencies in the grant process; however, Rehabilitation chose not to address these recommendations and allowed some errors to persist through the subsequent evaluation and award. Respectfully submitted, ELAINE M. HOWLE, CPA State Auditor 621 Capitol Mall, Suite 1200 Sacramento, CA 95814 916.445.0255 916.327.0019 fax www.auditor.ca.gov iv Report 2017-129 | CALIFORNIA STATE AUDITOR July 2018 Blank page inserted for reproduction purposes only. CALIFORNIA STATE AUDITOR | Report 2017-129 v July 2018 CONTENTS Summary 1 Introduction 7 Rehabilitation Failed to Formalize Procedures for Soliciting and Awarding Grants, Resulting in Inconsistencies and Inadequacies in the Grant Process 13 Significant Gaps in the Evaluation of Grants Raised Questions About the Adequacy and Fairness of the Process 29 Rehabilitation Did Not Always Follow Its Appeals Process as Suggested in Its Grant Manual, and Its Review Committees Did Not Always Ensure Fair Evaluations 45 Scope and Methodology 57 Response to the Audit Department of Rehabilitation 61 vi Report 2017-129 | CALIFORNIA STATE AUDITOR July 2018 Blank page inserted for reproduction purposes only. CALIFORNIA STATE AUDITOR | Report 2017-129 1 July 2018 SUMMARY Results in Brief Audit Highlights . . . The Department of Rehabilitation (Rehabilitation) provides services Our audit of Rehabilitation’s grant and advocacy to Californians with disabilities. Rehabilitation application and review process offers the majority of its services directly to individuals with highlighted the following: disabilities through the vocational rehabilitation program, which » Although required to have procedures provides services to disabled individuals to assist them in obtaining in place for soliciting and awarding competitive employment. It also works in cooperation with grants, Rehabilitation failed to formalize 28 independent living centers, which are nonresidential, nonprofit, such procedures, which resulted in community‑based agencies designed and operated within a inconsistencies and inadequacies in the local community by individuals with disabilities, to provide services grant process. and advocacy. Rehabilitation supports these services through a combination of federal and state funding. Recently, its process for » Rehabilitation had some deviations from soliciting and evaluating grant applications and awarding grant funds or gaps in its grant process that raised (grant process) has come under scrutiny, particularly as it relates questions about its fairness. to the 2017 Systems Change Network Hub grant (Systems Change grant). In our review of this grant and three others that Rehabilitation • It inappropriately accepted certain awarded from fiscal years 2014–15 through 2017–18, we found that information from some grant Rehabilitation would significantly benefit from improvements in a applicants after deadlines stipulated in variety of areas to ensure that it consistently and fairly conducts the its requests for applications had passed. grant process, and to help it defend its award decisions. • It limited its pool of prospective Rehabilitation took varying approaches to the grant process, some evaluators and did not always ensure of them questionable, for each of the four grants we reviewed, that they were free from conflicts of largely because it did not have formalized written procedures for the interest or bias before selecting them. grant process. Although state and federal regulations require that » Rehabilitation did not always follow Rehabilitation have procedures and a format in place, and despite its appeals process, and the review developing a draft grant solicitation manual (grant manual) to committees did not always conduct document its process in 2015, Rehabilitation never finalized the draft comprehensive reviews to determine or instructed staff to follow the details set forth in the grant manual. whether errors or omissions occurred, Rehabilitation officials stated that Rehabilitation did not finalize the evaluator biases affected the scoring process, grant manual because it does not award many grants and has had or evaluators supported their scores. higher priorities. We found that the grant manual largely contained procedures that were reasonable and useful and, in the absence of formalized procedures or other sufficient direction provided to its program management and staff (program staff), we used it as the basis for our assessment of Rehabilitation’s grant process. In our review of four grants that Rehabilitation awarded from fiscal years 2014–15 through 2017–18, we found that it did not adequately plan the grant process and that it failed to clearly define and assign staff roles and responsibilities at its outset. Rehabilitation also could not demonstrate that key staff were free from conflicts of interest; received the required ethics training; and understood confidentiality procedures before developing requests for applications (RFAs), the initial step in the grant process. By not undertaking these critical steps, Rehabilitation did not adequately ensure a consistent, complete, and fair grant process. 2 Report 2017-129 | CALIFORNIA STATE AUDITOR July 2018 Additionally, RFAs should include clear scoring criteria; application deadlines; and descriptions of the evaluation, award, and appeals processes. Rehabilitation did not adequately disclose in its RFAs all necessary information to ensure a transparent competitive process and, in some cases, disclosed inaccurate information. For instance, the scoring process described in the RFAs did not always align with the actual process that the individuals evaluating the grant applications (evaluators) followed when scoring grant applications, and Rehabilitation generally did not disclose in advance its methods for awarding grants in the event of a tie. Further, when Rehabilitation initiated its grant process, we found that it did not designate a centralized location to maintain grant‑related documents and, in some cases, destroyed key documents. This contributed to Rehabilitation failing to respond fully to some requests for public records related to the grant process. Although Rehabilitation included deadlines in its RFAs for submitting documentation, we found that it inappropriately accepted certain information from some grant applicants after these deadlines. For example, Rehabilitation accepted an entire revised section of an application from one grant applicant after the submission deadline. Instead of disqualifying the applicant as specified in the RFA, it continued to process the application and ultimately awarded one of the grant awards to this applicant—possibly preventing another qualified applicant from receiving grant funding. Further, rather than publishing a solicitation for evaluators of grant applications and seeking the disability community’s participation, which is suggested in the grant manual, Rehabilitation selected evaluators without issuing a solicitation for three of the four grants we reviewed. As a result, it limited its pool of prospective evaluators and missed the opportunity to ensure that it obtained the most qualified evaluators possible from the larger disability community. Rehabilitation also did not always ensure that prospective evaluators were free from conflicts of interest or bias before selecting them, and in fact it selected some evaluators for one grant who had held leadership positions in an organization that had a known relationship with one grant applicant, which created at least the potential for perceived bias. Additionally, the grant manual requires that Rehabilitation appoint a technical review team whose responsibilities include training evaluators on the grant process and program requirements before the evaluation process begins, answering evaluators’ questions related to the program or evaluation process, and ensuring after evaluations are complete that evaluators followed instructions during the evaluation process. Rehabilitation did not always appoint technical review teams to oversee the evaluation of each grant; sometimes staff fulfilled the responsibilities of the teams. However, CALIFORNIA STATE AUDITOR | Report 2017-129 3 July 2018 the technical review teams and staff failed to ensure evaluators followed the evaluation process, which contributed to procedural errors in the evaluation of applications for each of the grants we reviewed. In addition, the technical review teams and staff did not adequately review evaluators’ scores and comments for each grant to ensure that evaluators followed instructions, likely resulting in some applicants appealing certain grant awards. When Rehabilitation identifies errors in the evaluation process, the grant manual indicates that staff should begin a new RFA process to rectify errors and ensure that the grant process is fair. Although it may be necessary for Rehabilitation to restart the grant process to remedy the errors, we believe that under certain circumstances Rehabilitation can correct the issues and have evaluators rescore applications without restarting the grant process. Instead, it sometimes asked evaluators to rescore applications without correcting the issues. The four grants we reviewed for this audit resulted in nine appeals, and Rehabilitation did not consistently adhere to the appeals process contemplated in its grant manual. State regulations require Rehabilitation's chief deputy director (chief deputy) to appoint a grant review committee (review committee) to review the appeal. In addition, the grant manual gives the chief deputy the option to acknowledge receipt of the appeal in writing and to notify the appellant of the qualifications of the review committee members. Although we found that Rehabilitation consistently acknowledged its receipt of appeals, it did not always notify appellants of the review committee members’ qualifications; thus, appellants were not fully informed about those who would conduct a review of their appeal. State regulations provide limited direction regarding how Rehabilitation should review and process appeals, requiring only that applicants submit appeals within 30 days of the notice of the intent to award, and that a review committee review the request and notify the appellant in writing of its decision within 30 days of the request. The chief deputy said that she followed state regulations in addressing appeals, but that she was unfamiliar with the additional steps provided in the grant manual. Further, we found that the review committees did not always conduct comprehensive reviews to determine whether procedural errors or omissions had occurred, evaluator prejudice affected the scoring process, and evaluators supported their scores with evidence from the relevant applications, as the grant manual suggests. For the 2017 Systems Change grant, when the review committee found procedural errors in the grant process, Rehabilitation did not resolve these errors before awarding the grant. Further, although the grant manual suggests that the review committees should determine whether issues occurred in the grant 4 Report 2017-129 | CALIFORNIA STATE AUDITOR July 2018 process, such as procedural errors, given the numerous issues we found, we believe Rehabilitation should designate staff, separate from those who are responsible for developing RFAs, creating scoring criteria, and selecting evaluators, to conduct such reviews of each grant before Rehabilitation makes a final decision as to the grant recipients. This oversight of the grant process will provide Rehabilitation with additional assurance that program staff and evaluators adhered to its grant process, and help it demonstrate that the process was followed. Summary of Recommendations To comply with federal and state requirements, and to ensure consistency and fairness in its grant process, Rehabilitation should do the following: • Issue regulations describing its grant process from RFA development through appeals. It should submit its proposed regulations to the Office of Administrative Law no later than December 2018. • Revise and formalize the policies and procedures in its grant manual to incorporate the rules adopted by regulation and to address the recommendations in this report. The grant manual should specify that any deviations from the required grant process must be for good cause and be documented. Rehabilitation should clarify the roles and responsibilities of program staff involved in the grant process and ensure that those staff are free from conflicts of interest, receive the required ethics training, and understand confidentiality procedures. To provide grant applicants a full understanding of the grant process, Rehabilitation should disclose in its RFAs clear scoring criteria and descriptions of the evaluation, award, and appeals processes. Rehabilitation should also ensure that it maintains all relevant grant documentation in a centralized location and responds fully to all requests for public records related to grants. Rehabilitation should issue a public solicitation for evaluators for each grant and should train evaluators on conflicts of interest, including a discussion of bias, or the appearance of bias, before selecting them as evaluators. Further, to ensure that it equips evaluators with the information necessary to conduct sufficient evaluations of applications, Rehabilitation should develop evaluator training that can be tailored to each grant and includes instruction on how to evaluate applications. CALIFORNIA STATE AUDITOR | Report 2017-129 5 July 2018 Rehabilitation should also resolve issues before it rescores applications when it identifies procedural errors. Further, to ensure that Rehabilitation has appropriate oversight of its grant process and can demonstrate that it was followed, it should designate staff, separate from those involved in the respective grant process, to conduct a review for procedural errors, evaluator prejudice, and whether evaluators supported their scores with evidence from the relevant applications before it awards grants. Agency Comments Rehabilitation agreed with our recommendations and indicated that it plans to implement them. We look forward to Rehabilitation’s 60‑day response to our recommendations to learn more about its progress. 6 Report 2017-129 | CALIFORNIA STATE AUDITOR July 2018 Blank page inserted for reproduction purposes only. CALIFORNIA STATE AUDITOR | Report 2017-129 7 July 2018 INTRODUCTION Background The Department of Rehabilitation (Rehabilitation) works in partnership with the disability community, including disabled persons, caretakers, and other stakeholders, to provide services and advocacy that result in employment, independent living, and equality for individuals with disabilities, including those living with traumatic brain injuries. To carry out its mission, Rehabilitation employs more than 1,800 people. Although Rehabilitation offers the majority of its services directly to individuals with disabilities through the vocational rehabilitation program, such as career assessments and counseling, career education and training, access to assistive technology, and independent living skills training, it also works in cooperation with other agencies to provide these services to individuals with disabilities. Specifically, Rehabilitation works with 28 nonresidential, nonprofit, community‑based agencies—known as independent living centers—which are operated by individuals with disabilities and located throughout California. In addition to independent living centers, Rehabilitation also works with other specialized community‑based nonprofit agencies that provide services to people with blindness, survivors of traumatic brain injuries, and other disabled individuals who need assistive technology. Each independent living center provides services to people with a variety of disabilities. Funding for Independent Living Centers Rehabilitation provides financial support to independent living centers through a combination of federal and state funding. For a state to be eligible to receive federal assistance for its independent living programs, federal law requires it to establish and maintain a Statewide Independent Living Council (State Council) appointed by the governor. The majority of a State Council’s members must be made up of individuals with disabilities who are not employed by any state agency or independent living center. Federal law also permits parents or guardians of individuals with disabilities, advocates, and private business representatives to serve on the State Council. California’s State Council consists of 18 members, including disabled persons who are consumers of independent living services, advocates for people with disabilities, representatives of the business community, and the director of Rehabilitation as an ex officio, nonvoting member. The State Council’s duties include developing the State Plan for Independent Living (State Plan) and submitting it to the federal government for approval at least once every three years. The State Council develops the State Plan in conjunction with Rehabilitation, which is California’s designated state entity 8 Report 2017-129 | CALIFORNIA STATE AUDITOR July 2018 responsible for carrying out the State Plan. Consistent with the State Plan, Rehabilitation used a competitive process to award all of the grants it administered during our audit period. Table 1 shows the eight grants Rehabilitation awarded during our audit period of fiscal years 2014–15 through 2017–18, and highlights the four grants that we selected for our review. Table 1 Grants Rehabilitation Awarded Fiscal Years 2014–15 Through 2017–18 YEAR NAME OF GRANT AWARDED FUNDING AND REQUEST (MOST AVAILABLE DURATION FOR APPLICATIONS RECENT TO PER GRANT OF GRANT NUMBER OF NUMBER OF (RFA) NUMBER PURPOSE OF GRANT OLDEST) PERIOD PERIOD APPLICANTS AWARDEES California Assistive To establish an effective and efficient 2018 $3.4 million 3.25 years 2 1 Technology Program program for coordination and delivery total of statewide assistive technology AT-18-01 services. Systems Change To maintain, coordinate, and 2018 Up to 2.5 years 2 1 Network Hub continue development of an existing $937,500 (Systems Change) Systems Change network focused on total independent living issues affecting IL-17-01 persons with disabilities. Youth Transition To design and implement a Youth 2017 $200,000 2 years 12 6 Transition Program within California’s per awardee IL-17-02 independent living network for youth with disabilities, which can serve as a model for service delivery. Independent Living Services To intensify efforts to identify and 2017 $9.5 million 3 years 23 19 for Older Individuals Who reach underserved ethnically diverse total Are Blind (OIB) populations of older persons with visual Awards vary impairments in California. based on OIB-17-01 county square miles and population Technical Assistance, • To achieve greater long-term stability 2015 $132,600 1 year 11 7 Leadership Development, for independent living networks. total and Capacity Building • To establish a project-demonstrated Awards vary leadership development model that based on IL-16-01 is based on peer review and peer service that the awardee mentoring and is designed by and for will provide the independent living community. • To build capacity by undertaking or developing revenue-generating ventures. Traumatic Brain Injury To provide five core services, as 2015 $420,000 3 years 13 7 (TBI) identified in Welfare and Institutions per awardee Code Section 4357, for individuals with 02-24-2015 traumatic brain injury and their families. CALIFORNIA STATE AUDITOR | Report 2017-129 9 July 2018 YEAR NAME OF GRANT AWARDED FUNDING AND REQUEST (MOST AVAILABLE DURATION FOR APPLICATIONS RECENT TO PER GRANT OF GRANT NUMBER OF NUMBER OF (RFA) NUMBER PURPOSE OF GRANT OLDEST) PERIOD PERIOD APPLICANTS AWARDEES Technical Assistance, • To achieve greater long-term stability 2014 $209,900 1 year 20 10 Leadership Development, for independent living networks. total and Capacity Building • To establish a project-demonstrated Awards vary leadership development model that based on IL-15-01 is based on peer review and peer service that the awardee mentoring and is designed by and for will provide the independent living community. • To build capacity by undertaking or developing revenue-generating ventures. OIB To intensify efforts to identify and 2014 $9.3 million 3 years 23 22 reach underserved ethnically diverse total OIB-14-01 populations of older persons with visual Awards impairments in California. vary based on county population Source: Analysis of RFAs and supporting documentation for grants Rehabilitation awarded in fiscal years 2014–15 through 2017–18. n = We selected these grants for review to determine whether Rehabilitation followed its grant review process as required by law or in accordance with its draft grant solicitation manual (grant manual). We selected two OIB grants because, in part, these grants are greater in award value than others. Rehabilitation’s Organizational Approach to the Grant Process For the purposes of this report, the process for soliciting and evaluating grant applications from the independent living centers and awarding grant funds to them (grant process) comprises three general phases: the solicitation of grant applications, the evaluation of applications and the awarding of grants, and the appeals process. Figure 1 on the following page summarizes these phases. Rehabilitation has two primary divisions that oversee the grant process, and additional executive management and review teams are responsible for certain procedural controls. The Specialized Services, Blind and Visually Impaired and Deaf and Hard of Hearing Division administers the grant process for the OIB grant, and the Independent Living and Community Access Division administers the Systems Change grant, TBI grant, Youth Transition grant, and Technical Assistance, Leadership Development, and Capacity Building grant. 10 Report 2017-129 | CALIFORNIA STATE AUDITOR July 2018 Figure 1 Summary of Rehabilitation’s Intended Grant Process and Responsible Parties REHABILITATION’S INTENDED GRANT PROCESS RESPONSIBLE PARTIES Solicit Applications • Consider holding a kickoff meeting to start the grant process and educate participants on the process and their roles. • Obtain stakeholder feedback before and during RFA development. Program management and staff (program staff) • Include in the RFA information on how to apply, how Rehabilitation will select grantees, and how applicants can appeal the grant award decision. • Post the RFA to Rehabilitation’s website. Evaluate Applications and Award the Grant • Establish an evaluation panel by: - Issuing a solicitation. - Reviewing candidate résumés. - Providing training on conflict-of-interest and confidentiality procedures. - Selecting qualified evaluators. • Train evaluators on: - The purpose of the grant program. Technical review team - Regulatory requirements of the grant. - How to evaluate applications. • Ensure that grant applicants submit complete applications  Administrative review team by the deadline. • Answer evaluators’ questions during the evaluation. • Verify that evaluators followed all evaluation instructions. • Approve and issue the notice of intent to award. Director and chief deputy director Evaluate Appeals • Acknowledge receipt of any appeals. • Appoint a grant review committee (review committee).* Chief deputy director (chief deputy) • Notify all potentially affected applicants of the appeal. • Evaluate each appeal and relevant records. • Notify the appellant in writing of the decision within Review committee 30 days from the date of the request.* • Notify affected applicants of the appeal decision. Source: Analysis of the California Code of Regulations, Title 9, Section 7334, and Rehabilitation’s grant manual. * Required by state regulations. CALIFORNIA STATE AUDITOR | Report 2017-129 11 July 2018 In summary, according to Rehabilitation’s grant manual, each grant begins with the solicitation phase, during which program staff plan the grant process, assign employees’ roles and responsibilities, obtain feedback from stakeholders to inform the development of the RFAs, and draft the RFAs. The RFAs should explain what information and documentation applicants should include in their application to demonstrate that they meet the requirements to apply for the grant. It should also include a detailed description of the criteria evaluators will use to score applications, such as the applicant’s experience and effectiveness in providing certain services. For the four grants we reviewed, Rehabilitation posted the final RFAs to its website. The grant manual also indicates, in the evaluation and award phase, that program staff are responsible for identifying essential and desired qualifications for those who will evaluate the applications (evaluators), and program management is responsible for selecting evaluators with those qualifications. Evaluators may be state employees, including Rehabilitation employees, or outside subject‑matter experts. Program staff also coordinate the logistics of evaluations and appoint a technical review team. This team consists of subject‑matter experts in program policy and the grant process, and its purpose is to train evaluators on the evaluation process and program requirements before the evaluation, answer evaluators’ questions during the evaluation, and follow up after the evaluation process is complete to ensure that evaluators followed the intended steps. This team then summarizes the evaluation process and recommends grant awards in a memorandum to the director and chief deputy for approval. Once the director and chief deputy approve the grant awards, Rehabilitation posts notice of the awards on its website. If the director and chief deputy do not approve the award, the grant manual specifies that the grant process should start over. The appeals phase ensues in cases where an applicant appeals an award decision. Upon receipt of an appeal, a review committee appointed by the chief deputy evaluates the appeal’s merit, and the grant manual suggests that the review committee should determine whether there were any procedural errors or omissions, whether there was evidence that evaluator prejudice affected the scoring process, and whether evaluators’ scores are supported by evidence in the relevant applications. After completing its review, the review committee notifies the appellant of its decision, and the grant manual suggests that the chief deputy should notify other affected parties of the appeal decision. 12 Report 2017-129 | CALIFORNIA STATE AUDITOR July 2018 Blank page inserted for reproduction purposes only. CALIFORNIA STATE AUDITOR | Report 2017-129 13 July 2018 Rehabilitation Failed to Formalize Procedures for Soliciting and Awarding Grants, Resulting in Inconsistencies and Inadequacies in the Grant Process Key Points • Although federal regulations require Rehabilitation to have procedures and state regulations require it to have a format in place for its grant process, during our audit period of fiscal years 2014–15 through 2017–18, Rehabilitation had developed only a grant manual that contains procedures it does not require staff to follow. Instead, it had provided its staff with discretion in how they solicit, evaluate, and award grants. This lack of established procedures and level of discretion were key factors contributing to the many inconsistencies and shortcomings we identified in Rehabilitation’s grant process. Further, without formalized procedures, Rehabilitation has potentially violated federal and state regulations and does not know whether its staff and evaluators follow a consistent and fair grant process. • In our review of the grant manual, we found that the procedures were largely useful and reasonable, and in the absence of any other sufficient direction provided by Rehabilitation to its staff, we used the grant manual and the minimal state regulations that pertain to Rehabilitation’s appeals process to perform our assessment. • For the four grants we reviewed, we found that in developing its RFAs, Rehabilitation failed to take steps that are critical to ensuring a consistent, complete, and fair grant process. For example, in most instances, it did not clearly define staff responsibilities, obtain stakeholder input and feedback on the content Requirements for Formalized Procedures of RFAs, and disclose all information pertinent and a Format for the Grant Process to the grant process in its RFAs. Code of Federal Regulations Since at least 1992, “Each state shall have procedures for • Further, in part because of its haphazard reviewing and approving applications for subgrants and and disorganized approach to managing amendments to those applications, for providing technical documentation of the grant process, assistance, for evaluating projects, and for performing other Rehabilitation did not fully respond to administrative responsibilities . . .” several requests it received for public records California Code of Regulations regarding certain grants. Since at least 2001, “Rehabilitation must establish and maintain a format to evaluate, prioritize, and award applications for grants . . .” Failure to Develop Formalized Procedures for the Grant Process State Plan for Independent Living Since at least 2008, “Rehabilitation will comply with all applicable federal statutes and regulations in effect with Although Rehabilitation is aware that it is required respect to the three‑year period it receives funding under to have procedures in place for the grant process, the state plan.” it has developed only draft procedures and has not Source: State and federal regulations, as well as state plans for required staff to follow these procedures. Federal the last 10 years. law requires states to satisfy certain conditions to receive federal aid and, as shown in the text box, 14 Report 2017-129 | CALIFORNIA STATE AUDITOR July 2018 for more than 25 years, the Code of Federal Regulations has required each state to have procedures for reviewing and approving applications for the grants awarded to service providers. Further, state regulations require Rehabilitation to establish and maintain a format to evaluate, prioritize, and award applications for grants. However, current state regulations do not prescribe procedures or establish a particular format for awarding grants and offer only minimal direction for processing appeals. Additionally, the State Plan requires Rehabilitation to comply with all federal legal requirements and to be consistent with state law. As a result of these requirements, we expected Rehabilitation to have developed uniform written procedures, approved by its management, that sufficiently describe the steps in its grant process—from development of the RFA to the final decision on any appeals. The chief deputy acknowledged that Rehabilitation is aware of the requirements that it have procedures and that it should have finalized the grant manual; however, she explained that she was not aware that Rehabilitation had not finalized the grant manual until around the time the Joint Legislative Audit Committee (Audit Committee) approved this audit. Upon learning this, she stated that the former chief of the Contracts and Procurement Division (former contracts chief) advised her that the division had not finalized the grant manual because Rehabilitation does not award many grants and the Contracts and Procurement Division had other, higher priorities.1 The chief deputy stated that, had she known that Rehabilitation was lacking such critical procedures, she would have expedited completion of the grant manual. Without formalized written procedures, Rehabilitation has potentially violated federal and state regulations. Further, it lacks assurance that its staff and evaluators follow a consistent and fair grant process and that the most qualified applicants receive awards. The chief deputy acknowledged that Rehabilitation should formalize its procedures as a matter of good practice, but stated that she believes Rehabilitation has met the requirement to have procedures through sharing practices by word of mouth, institutional knowledge, and reference to its previous RFAs to describe the grant process. We disagree that word of mouth, We disagree that word of mouth, institutional knowledge, and past institutional knowledge, and past practices are a substitute for having formalized procedures, which practices are a substitute for having we would expect to be written and approved by Rehabilitation’s formalized procedures. management. Further, we found that the RFAs we reviewed are not uniform, consistent, or comprehensive. We also recognize that the California Administrative Procedure Act (administrative procedure act) generally prohibits state agencies from issuing or using any 1 Several months after we began our audit, Rehabilitation posted a revised version of its grant manual on its website. For the purpose of our review, we focused on the grant manual that Rehabilitation developed around 2015 and disseminated to certain staff. CALIFORNIA STATE AUDITOR | Report 2017-129 15 July 2018 guideline, manual, or instruction that meets the definition of a regulation without first engaging in the administrative procedure act’s rulemaking process.2 Because the procedures in the grant manual are intended to describe how Rehabilitation solicits, evaluates, and awards competitive grants, as well as to supplement its scant regulations governing the appeals process, we find that if finalized, the grant manual likely meets the administrative procedure act’s definition of a regulation. Consequently, we believe Rehabilitation should engage in the rulemaking process. This is particularly true because Rehabilitation’s RFAs, as described later in this report, omit key steps and do not adequately describe Rehabilitation’s comprehensive grant process. Therefore, we believe that to ensure To ensure transparency, fairness, transparency, fairness, and compliance with both state and federal and compliance with both state and requirements, Rehabilitation should promulgate regulations federal requirements, Rehabilitation to govern its grant process. Once Rehabilitation promulgates should promulgate regulations to regulations, it could further develop its grant manual to provide govern its grant process. instruction to staff on how to meet the regulatory requirements. Although the grant manual has been available to staff since approximately 2015, and Rehabilitation has provided the grant manual in response to two requests under the California Public Records Act (public records act) for grant policies and procedures, as well as supporting documents, Rehabilitation does not require staff to follow the current grant manual. Before Rehabilitation received the first appeal of the 2017 Systems Change grant award, which we describe beginning on page 49, Southern California Resource Services (SCRS), an applicant that was not awarded the grant, requested Rehabilitation's policies and procedures for conducting the grant process, and Rehabilitation provided SCRS with the grant manual. In doing so, it did not state to SCRS that the manual was a draft and did not represent its formal policies and procedures. Similarly, in response to a subsequent request from SCRS for Rehabilitation’s adopted grant solicitation manual, Rehabilitation again provided its draft grant manual. Although it included the word “draft” in the title of the electronic file that contained the grant manual, Rehabilitation did not explain that it does not require staff to follow the manual during the grant process. According to the former contracts chief, because the grant manual is in draft form, Rehabilitation does not consider it compulsory or enforceable— meaning that Rehabilitation does not require staff to follow it during the grant process. However, we find it troubling that Rehabilitation does not require staff to follow the grant manual but chose to present the manual to SCRS—a grant applicant—as its procedures. As of June 2018, the chief deputy advised us that Rehabilitation is working to promulgate regulations and finalize the grant manual. 2 The administrative procedure act defines a regulation as a rule, regulation, order, or standard of general application or the amendment, supplement, or revision of any rules, regulations, orders, or standards adopted by any state agency to implement, interpret, or make specific the law enforced or administered by it, or to govern its procedure. 16 Report 2017-129 | CALIFORNIA STATE AUDITOR July 2018 In the absence of formalized procedures or other sufficient direction to Rehabilitation’s staff, and given that we found that the grant manual largely contains reasonable and useful procedures, we used the grant manual to assess the adequacy of Rehabilitation’s grant process. In general, the grant manual clearly defines each stage of the grant process and expectations of staff and evaluators. Further, the grant manual outlines process controls, which include the roles of staff and management who are responsible for overseeing aspects of the grant process to ensure that Rehabilitation carries it out appropriately. Nevertheless, we did identify multiple areas for improvement in the draft manual, as certain key procedures provide too much flexibility and discretion to program staff. For example, the grant manual states that before program staff begin developing the RFA, they should do some planning, including making decisions regarding how much weight each component of the scoring criteria, such as the applicant’s experience and effectiveness in providing relevant services and its financial position, will receive in the total score. However, by not explicitly requiring program staff to establish a weight for each component and to include that weight in each RFA, Rehabilitation risks not fully disclosing how it scores applications. Similarly, the grant manual states that staff should consider including information in the RFA about whether correspondence regarding an appeal can be made by email, how Rehabilitation will acknowledge receipt of an appeal, and who will send the acknowledgment. If this were a requirement, Rehabilitation would have greater assurance that applicants will have the information necessary for filing an appeal. Lack of Clearly Defined Roles and Responsibilities in the Grant Process In its grant manual, Rehabilitation suggests that, at the start of the development of each RFA, management and staff participating in the grant process conduct a kickoff meeting, wherein they are educated on the steps in the grant process and their roles and responsibilities in the process. The grant manual also suggests that participants in the kickoff meeting document the expected deliverables during the grant process and which staff are responsible for each deliverable. If conducted appropriately, kickoff meetings ensure that those involved in the grant process understand Rehabilitation’s expectations of them, document these expectations, and know how to carry out the grant process adequately as planned. Rehabilitation could not demonstrate However, Rehabilitation could not demonstrate that it held these that it held kickoff meetings or kickoff meetings or documented participants’ roles and responsibilities documented participants’ roles and for the four grants we reviewed, as shown in Table 2 on page 18. For responsibilities for the four grants example, for the 2017 Systems Change grant, Rehabilitation did not we reviewed. schedule a kickoff meeting before the grant process began, which officials stated began in November 2016, and instead held the meeting CALIFORNIA STATE AUDITOR | Report 2017-129 17 July 2018 roughly two months later in January, just one week before it released the RFA—a key document in the grant process that requires significant coordination on the part of staff to prepare. Rehabilitation also did not document deliverables or staff roles and responsibilities during the grant process. According to the former chief of independent living, assistive technology, and traumatic brain injury programs (chief of independent living), who has since assumed another position at Rehabilitation but at the time was responsible for administering the 2017 Systems Change grant process, due to scheduling conflicts, it was not feasible to hold this meeting before starting the grant process without delaying the RFA. Further, the independent living deputy director, who was responsible for overseeing the entire process for the 2017 Systems Change grant, was not involved in the meeting when it was eventually held. As another example, for the 2014 OIB grant, the former OIB program manager, who has since retired, did not recall holding a kickoff meeting or otherwise defining staff roles. Rehabilitation’s failure to ensure that it clearly defined and documented participants’ roles and responsibilities in the grant process, and to ensure that key management responsible for overseeing the grant process were present at kickoff meetings, created an environment in which the shortcomings and inefficiencies in its grant process that we describe throughout this report were able to occur. Various Rehabilitation staff also did not receive ethics training, which Various Rehabilitation staff also includes conflict‑of‑interest training, before becoming involved in did not receive ethics training, the grant process, as shown in Table 2 on the following page. The which includes conflict‑of‑interest grant manual specifies that Rehabilitation employees in positions training, before becoming involved that involve awarding grants or contracts must receive ethics in the grant process. training, which includes conflict‑of‑interest training, every other year. Further, state law requires certain state employees involved in making governmental decisions—including most of those involved in the grant process—to attend training on relevant ethics laws and regulations within six months of assuming their position and every two years thereafter. These trainings include topics to help employees recognize any personal or private interests that may affect their ability to perform their job fairly and impartially. Because staff involved in the grant process may make decisions regarding grant awards, this training is of particular importance. Despite these requirements, Rehabilitation could not demonstrate that five key staff members involved in the 2017 Systems Change grant—the independent living deputy director, chief deputy, former contracts chief, deputy director of specialized services, and an attorney—had completed the required ethics training within two years of the start of the grant process. Similarly, Rehabilitation could not demonstrate that key staff who worked on the 2017 OIB grant, including the chief deputy, former contracts chief, and deputy director of specialized services, had completed the required ethics training within two years of the start of the grant process. 18 Report 2017-129 | CALIFORNIA STATE AUDITOR July 2018 According to the chief of Rehabilitation’s human resources branch, a unit within her branch notifies managers of staff who need to complete ethics training; however, she explained that Rehabilitation expects managers to be responsible for ensuring that their staff receive the training. The administrative services deputy director, who oversees the human resources branch, stated Rehabilitation also expects managers, such as a deputy director or chief deputy, to ensure that they themselves receive the proper and required amount of training. By not ensuring that its employees receive ethics training as required by state law, Rehabilitation failed to take advantage of a resource that would increase the likelihood that its staff involved in the grant process will be fair and impartial. Table 2 Rehabilitation Consistently Failed to Adequately Develop RFAs NAME OF GRANT AND YEAR AWARDED SYSTEMS OIB TBI OIB CHANGE RFA DEVELOPMENT AND SOLICITATION PROCESS 2014 2015 2017 2017 Hold a kickoff meeting to ensure that management and staff are aware of their X X X X roles and responsibilities and that these expectations are documented Ensure that management and staff receive ethics training, which includes X X X X conflict-of-interest training X X X X Develop and train staff on confidentiality procedures Ensure that management and staff sign the confidentiality and conflict-of-interest X X X X form specific to the grant X  t X Solicit stakeholder feedback for RFA development Develop RFA, including adequate: t t t t Scoring criteria t t t t Evaluation and award process t t t t Appeals process     Application deadlines X X X X Create a single archive location for all documents developed during the grant process Source: California State Auditor’s review of four selected RFAs and supporting documentation from fiscal years 2014–15 through 2017–18, and Rehabilitation’s grant manual.  = Followed t = Partially followed X = Did not follow CALIFORNIA STATE AUDITOR | Report 2017-129 19 July 2018 Rehabilitation also did not develop procedures or otherwise Rehabilitation did not develop ensure that it informed staff at the beginning of the grant process procedures or otherwise ensure that of its expectations for handling confidential information or it informed staff at the beginning of documentation, as shown in Table 2. According to the grant manual, the grant process of its expectations confidentiality procedures should ensure that staff develop and store for handling confidential documents in a way that controls how, when, and to whom they information or documentation. are circulated; describe how staff identify, maintain, and dispose of draft documents containing confidential information; and designate specific communication channels for simultaneous disclosure of information to ensure that all interested persons have the same access. Such procedures are important to maintain the integrity of the process and to prevent accidental disclosure of documents before Rehabilitation issues the final award. Given the importance of this issue, we would expect Rehabilitation to have developed confidentiality procedures to help ensure that staff working on grants know how to handle documentation during the grant process. Rather, according to the former contracts chief, Rehabilitation typically provides verbal instruction to staff at the beginning of the grant process on how to handle confidential information. For the four grants we reviewed, program managers and the deputy director of specialized services, who provided perspective on the 2014 OIB grant in the absence of the retired program manager, explained that they did not provide any written procedures to staff regarding how to handle confidential information. In one case, the program manager recalled verbally discussing it, and in other instances program managers indicated that they had not received instruction to develop confidentiality procedures. This informal approach is concerning because Rehabilitation lacks assurance that staff receive complete and consistent direction on how to handle confidential information at the beginning of each grant process, if they receive direction at all, and risks staff disclosure of confidential information. The former contracts chief acknowledged that developing standardized confidentiality procedures would be beneficial going forward. Rehabilitation also did not ensure that staff certified that they were free from conflicts of interest or that they understood confidentiality expectations. Specifically, the grant manual states that all Rehabilitation staff involved in the grant process must certify that they have no conflicts of interest with any of the organizations that are competing for the grant and that they understand how to keep the grant process confidential. Staff are to certify this by signing a conflict‑of‑interest and confidentiality form. The grant manual also states that for staff to have the knowledge necessary to sign this form, Rehabilitation must first provide training so that they understand what they are signing, what is expected, and the penalties they could suffer if they breach confidentiality or have a conflict of interest. However, we found 20 Report 2017-129 | CALIFORNIA STATE AUDITOR July 2018 that only one of the 26 employees involved in the four grants we reviewed signed a conflict‑of‑interest and confidentiality form. In addition, none of these employees received training on conflict of interest and confidentiality specific to the individual grant, as shown in Table 2. The program managers for these grants generally stated that they did not provide staff involved in the grant process with training or have them sign conflict‑of‑interest and confidentiality forms because they were not aware that it is Rehabilitation’s practice to do so. By not having procedures for managers to follow to ensure that staff receive the required training, are free from conflicts of interest, and understand confidentiality procedures, Rehabilitation risks providing an unfair advantage to certain applicants if a staff member has a connection to an applicant or inadvertently releases confidential information before the award is made public. Failure to Solicit Stakeholder Feedback Rehabilitation did not always solicit stakeholder input and feedback to inform its development of RFAs, missing the opportunity to receive valuable insights from subject‑matter experts and the disability community. However, the policy of inclusion and full participation of individuals receiving public funds for specified rehabilitation services is articulated in federal law, while state law requires Rehabilitation both to be consistent with federal policy and to attempt to ensure the regular and meaningful involvement of the disability community in the policy development and implementation of independent living programs. Further, the grant manual states that Rehabilitation should reach out to stakeholders, including the potential pool of applicants, before and during the development of an RFA, in part to increase stakeholder and applicant satisfaction with the RFA and the evaluation process, as well as with the selection results. Rehabilitation did not adequately However, as shown in Table 2 on page 18, Rehabilitation did not solicit stakeholder feedback in adequately solicit stakeholder feedback in developing the RFAs for developing the RFAs for three of the three of the four grants we reviewed. For example, the former chief four grants we reviewed. of independent living explained that Rehabilitation did not solicit stakeholder feedback for the 2017 Systems Change grant because it had a short time frame in which to develop the RFA, as the State Council did not approve the operations plan for the State Plan until November 2016, due to new requirements and procedures from the Workforce Innovation and Opportunity Act. In addition, he stated that he relied on staff from Rehabilitation’s Contracts and Procurement Division for guidance, and they did not advise him to solicit stakeholder feedback. Further, because Rehabilitation had already issued the RFA for the Systems Change grant in past years, and the independent living community and stakeholders CALIFORNIA STATE AUDITOR | Report 2017-129 21 July 2018 were already familiar with the grant, he did not think it necessary to solicit feedback on procedural components of the RFA. Although we understand the delay and shortened time frame to complete the RFA, we believe it would have been beneficial for Rehabilitation to seek feedback from the disability community and subject‑matter experts to inform the development of its RFAs and address any stakeholder concerns. Rehabilitation sufficiently solicited and obtained stakeholder input and feedback for just one of the four grants we reviewed. Specifically, for the 2015 TBI grant, Rehabilitation published a survey on its website requesting stakeholder input and feedback on the number of grants it should award and asking them to rate the importance of scoring criteria. In contrast, Rehabilitation did not solicit stakeholder input and feedback when developing the RFA for the 2014 OIB grant. For the 2017 OIB grant, Rehabilitation chose to solicit input and feedback only from certain stakeholders. In particular, to notify stakeholders of its meeting regarding the RFA, Rehabilitation sent an email to the program’s mailing list that, according to the OIB program manager, included representatives from the grantees for the previous award. However, to ensure that To ensure that it offered all interested it offered all interested stakeholders the opportunity to participate stakeholders the opportunity to in the meeting, we would have expected Rehabilitation to at least participate in the meeting, we post a notice about the meeting on its website. By not attempting would have expected Rehabilitation to solicit stakeholder input and feedback from the greater disability to at least post a notice about the community to inform the development of its RFAs, which are the meeting on its website. foundation of the funding decisions for each grant, Rehabilitation risks that the final RFA could be missing information or could contain inaccurate information about the grant or the evaluation and award process. Lack of Information in RFAs About the Evaluation, Award, and Appeals Process None of the RFAs we reviewed included specific information about how evaluators would assign points to each of the scoring components when reviewing applications, which we found to be a best practice. To identify best practices for disclosing to applicants the process evaluators would use to score applications, we selected and reviewed two RFAs for grants of federal funds administered by two other state agencies. We found that both RFAs included detailed descriptions of how evaluators would score applications. For example, one agency described in its RFA the process evaluators would use to score applications, from 0 for inadequate to 4 for excellent or outstanding. It also explained the basis for each point assignment, such as that applications receiving a score of 4 demonstrate the applicant’s ability and intent to exceed the requirements, provide evidence of the applicant’s current ability 22 Report 2017-129 | CALIFORNIA STATE AUDITOR July 2018 to comply with the grant requirements, and propose detailed plans or methodologies that further describe how the applicant will exceed requirements. However, Rehabilitation did not include similar detailed descriptions in its RFAs. For example, out of a total of 10 points possible for a scoring component, we would expect the RFA to disclose that a well‑qualified score would be between 7 and 10 points, a qualified score would be between 4 and 6 points, an unqualified score would be between 1 and 3 points, and a component that was missing would receive a score of 0. Instead, for the four RFAs we reviewed, Rehabilitation disclosed only the total number of possible points for each component—such as 21 points for the Plan of Operation component. Without a description of the range of points for each level of quality, applicants may not fully understand the type of information and level of detail necessary to achieve high scores, the specific criteria that evaluators are using to assess their applications, or the significance of the scores their applications receive. Further, Rehabilitation provided scoring documents to evaluators that lacked adequate guidance on how to score applications. The grant manual indicates that, in conjunction with the scoring criteria in its RFAs, Rehabilitation should develop one benchmark for each scoring component—such as Organizational Structure and Capacity, Core Services, and Accessibility and Populations Served. These benchmarks should provide evaluators with guidance on how to score applications by providing a detailed description of the content and level of quality needed for an applicant to receive a score of well‑qualified, qualified, or unqualified. For the 2017 Systems Change grant, we found that the benchmarks Rehabilitation provided to the third evaluation panel did not contain the numeric scoring range it expected for each level of quality, which may have given evaluators too much discretion in scoring applications.3 The former chief of independent living attributed the omission of numeric ranges from the scoring benchmarks to his general lack of experience in reviewing benchmarks. For two grants, Rehabilitation Additionally, for the 2017 Systems Change grant and the 2017 OIB did not ensure that the scoring grant, Rehabilitation did not ensure that the scoring benchmarks benchmarks it provided evaluators it provided evaluators aligned with the evaluation criteria in the aligned with the evaluation criteria corresponding RFAs. For example, for the 2017 Systems Change in the corresponding RFAs. grant, Rehabilitation provided the first evaluation panel with two sets of benchmarks, neither of which aligned with the scoring criteria in the RFA. Specifically, one set of benchmarks was incomplete because it did not contain portions of two scoring components, and the other 3 As shown in Figure 2 on page 40, the 2017 Systems Change grant involved five rounds of scoring by three evaluation panels. CALIFORNIA STATE AUDITOR | Report 2017-129 23 July 2018 included two scoring components that were not disclosed in the RFA, and excluded another four components that were disclosed in the RFA. Rehabilitation did not rectify these inconsistencies and Rehabilitation did not rectify several provided each of the three evaluation panels inaccurate scoring inconsistencies and provided each benchmarks through subsequent rounds of scoring. Further, for the of the three evaluation panels for 2017 OIB grant, the scoring benchmarks indicated that applicants the 2017 Systems Change grant were required to submit letters of support, whereas the RFA did not inaccurate scoring benchmarks. specifically require these letters. Similar to the issues with the benchmarks, we found that the score sheets Rehabilitation provided to evaluators did not align with the criteria in the RFA for two of the four grants we reviewed. In particular, the score sheets for the 2017 Systems Change grant included an evaluation component that was not listed in the RFA. For the 2015 TBI grant, the score sheets indicated that applicants were required to describe how they improved and developed needed services other than TBI, while the RFA required applicants to discuss only TBI services. When Rehabilitation does not ensure that its scoring materials align with the criteria in its RFAs, it risks that evaluators will perform unfair evaluations based on criteria of which applicants are unaware; use inappropriate criteria for scoring applications; and score applications inconsistently, which could result in scores that inaccurately reflect the quality of the applications. Rehabilitation also did not fully disclose certain steps in its evaluation, award, or suggested appeals processes in the four RFAs we reviewed. For example, in each of the four grants we reviewed, we found that Rehabilitation did not adequately disclose important details regarding its evaluation process, such as the number of evaluators it planned to use, how it would select evaluators, and the type of oversight it would perform to ensure that evaluators conducted the evaluation as expected and fairly. Further, Rehabilitation did not disclose how it would announce the grant award decision in two of the four grants we reviewed. The grant manual also suggests that RFAs include detailed descriptions of the appeals process, such as how Rehabilitation will acknowledge an appeal, the evidence Rehabilitation will review, and the types of determinations it should reach based on its review of that information. However, the four RFAs we reviewed did not fully disclose the information regarding the appeals process that is set forth in regulations, such as that the review committee, which Rehabilitation's chief deputy appoints to review appeals, must deliver its decision within 30 days of the appeal date. Further, Rehabilitation did not disclose in the RFAs for the four grants we reviewed the additional steps suggested by its grant manual. By failing to include these descriptions in its RFAs, Rehabilitation has not adequately informed the public and potential applicants about regulatory requirements and its suggested appeals process, which could lead appellants to question the thoroughness or fairness of the process. 24 Report 2017-129 | CALIFORNIA STATE AUDITOR July 2018 Rehabilitation did not disclose, in Rehabilitation also did not disclose, in three of the four RFAs we three of the four RFAs we reviewed, reviewed, how it would address tied scores between applicants. For how it would address tied scores the RFA for the 2015 TBI grant, Rehabilitation included a tiebreaker between applicants. rule specifying that it would give greater priority to certain components in the event of a tie and would refer back to the highest scores of those components sequentially until it broke the tie. For this grant, there was a tie between two applicants, and Rehabilitation used the methodology disclosed in its RFA to break the tie. In contrast, we also found two ties between applicants for the 2014 OIB grant, and Rehabilitation did not explain a tiebreaker rule in its RFA for the grant. In this instance, Rehabilitation selected the applicants that were in closest proximity to the geographic area the grant would serve. Although we find this approach reasonable, we would expect Rehabilitation to disclose the methodology in the RFA. The grant manual discusses who is responsible for applying a tiebreaker rule; however, it does not prescribe a method for addressing applications that receive the same score, or require staff to include one in RFAs. Therefore, we also believe that Rehabilitation should consider establishing some standardized options for breaking ties that program staff can customize, such as using the highest‑priority components of each grant or the geographic location of the applicant and the county in which services would be provided. The former contracts chief agreed with our conclusion and stressed the importance of disclosing the chosen methodology in the RFA. By not disclosing a tiebreaker rule in its RFAs, Rehabilitation is missing the opportunity to increase transparency by informing applicants about how it will award grants in the event of a tie. Failure to Enforce Application Filing Deadlines Rehabilitation’s grant manual states that it should decide in advance of posting the RFA whether to apply a hard or soft deadline for applicants to submit their applications. The grant manual defines a hard deadline as one by which applicants must submit applications that are complete and ready for scoring, and any wrong or missing information, approvals, or certifications are grounds for rejection or a scoring penalty. The grant manual defines a soft deadline as one by which applicants must submit applications but can be given extra time to submit incomplete or incorrect information that is immaterial, as requested by the administrative review team. We found that Rehabilitation included hard deadlines in the RFAs for each of the four grants we reviewed by specifying that it would disqualify, or not consider for funding, any application packages received after the deadline or any that did not contain all required items by the specified deadline. However, as we describe on page 30, Rehabilitation accepted some required items after the application deadlines for three of the four grants we reviewed. CALIFORNIA STATE AUDITOR | Report 2017-129 25 July 2018 Poor Document Management Practices and Responses to Requests for Public Records For the four grants we reviewed, Rehabilitation could not demonstrate having responded completely or within the required time frame to 15 of the 29 public records requests it received from up to four applicants we selected for review from each grant. According to state law, within 10 days of receipt of the request for public records, Rehabilitation must respond in writing with an estimate of when it will provide copies of those records. Rehabilitation must also promptly make public records, with certain exceptions, available to any person that requests a copy of the record. State law also specifies that upon request, Rehabilitation must provide an exact copy of the requested record, unless it is impracticable to do so. However, for the 2017 Systems Change grant, Rehabilitation failed to fully respond to a request from SCRS by not providing it with a complete copy of an application that was requested because it omitted the application letter. In addition, across the four grants we reviewed, Rehabilitation did not fully respond to various other requests for documents, such as scoring criteria, evaluator selection notes, and evaluators’ score sheets. Rehabilitation also failed to respond within the required or stated Rehabilitation failed to respond time frame to five requests, including three requests regarding within the required or stated time the 2017 Systems Change grant, one for the 2015 TBI grant, and frame to five public records requests. another for the 2017 OIB grant. For example, Rehabilitation took 13 days to respond to one of SCRS' requests and 31 days to respond to the request regarding the 2015 TBI grant. Further, for another request from SCRS related to the 2017 Systems Change grant, although Rehabilitation provided an initial response within 10 days, it delivered the remaining requested documents in multiple stages, starting three weeks after the date of the request and five days after it originally indicated it would provide the documents. Moreover, Rehabilitation provided the last of the documents nearly 40 days after the original request. Because of these delays, SCRS received relevant information past its deadline for submitting an appeal, and therefore filed an addendum to its appeal. Rehabilitation provided differing reasons for not fully responding to the requests for public records or not responding in a timely manner. According to Rehabilitation's legislation and communications analyst, when Rehabilitation receives a request for public records, its Office of Legislation and Communications submits a document request and the corresponding request letter to relevant program staff. He and other Rehabilitation staff offered varying reasons for not fully responding or not responding in a timely manner, including that it was an oversight, that staff and management had to search their email for correspondence and save the documents to a centralized location, that program staff did not provide the 26 Report 2017-129 | CALIFORNIA STATE AUDITOR July 2018 requested records on time, and that the Office of Legislation and Communications inadvertently did not remind program staff to provide such records. Rehabilitation’s poor records Rehabilitation’s poor records management and the fact that it did management and the fact that it not ensure that staff complied with its records retention policy did not ensure that staff complied contributed to its failure to respond to public records requests with its records retention policy completely or in a timely manner. The grant manual advises contributed to its failure to Rehabilitation to consider creating a single location to archive respond to public records requests all documents developed during the grant process. Further, completely or in a timely manner. Rehabilitation’s records retention policy generally states that it must maintain all contract documents for seven years. According to the former contracts chief, this also includes agreements required by its federal agencies, such as those related to the grant process. However, Rehabilitation did not designate a centralized location to archive documents for any of the four grants we reviewed. Specifically, we found that some staff destroyed documentation; unnecessarily kept documents at their desks; did not save email correspondence in a centralized location; and deleted email correspondence, including key decisions and documents regarding the grant process. Staff, including one deputy director, attributed the failure to archive materials in a centralized location to a lack of instruction on how and where to store documents, particularly email correspondence. Staff's failure to retain all grant documents also indicates that staff were not familiar with Rehabilitation’s records retention policy. In addition, we found that the legislation and communications analyst could confirm the completeness of the public records act tracking log only beginning in September 2016, when he began working at Rehabilitation. Staff who created the log and tracked requests before that time have since left Rehabilitation. Thus, we cannot be certain that Rehabilitation provided us with all of the public records requests it received during our audit period. Rehabilitation’s failure to ensure that staff were familiar with its records retention policy, and to designate a centralized location for storing documents, sometimes rendered its responses to records requests related to grants incomplete and contributed to it not responding fully or in a timely manner to requests for public records. CALIFORNIA STATE AUDITOR | Report 2017-129 27 July 2018 Recommendations To comply with federal and state requirements, and to ensure consistency and fairness in its grant process, Rehabilitation should do the following: • Issue regulations describing its grant process from RFA development through appeals. It should submit its proposed regulations to the Office of Administrative Law no later than December 2018. • Revise and formalize the policies and procedures in its grant manual to incorporate the rules adopted by regulation and to address the recommendations in this report. The grant manual should specify that any deviations from the required grant process must be for good cause and be documented. To ensure that management and staff involved in the grant process are sufficiently informed about the process and their responsibilities, Rehabilitation should require these employees to attend a kickoff meeting before the development of each RFA in which participants discuss the key stages of the grant review process, each individual’s roles and responsibilities, and requirements surrounding conflicts of interest and confidentiality. Further, it should record these discussions in meeting minutes to ensure that expectations of employees are clearly defined and documented. To comply with state laws and regulations and help ensure that staff involved in making governmental decisions during the grant process are impartial, Rehabilitation should ensure that they receive ethics training, which includes conflict‑of‑interest training, at least every two years. To help ensure that staff involved in the grant process adequately protect confidential information, Rehabilitation should develop confidentiality procedures for each grant. Further, it should ensure that staff involved in the grant process sign the conflict‑of‑interest and confidentiality forms before the development of the RFA for each grant. To ensure that it has received sufficient input and feedback from the disability community to inform the development of RFAs, Rehabilitation should solicit and document stakeholder input and feedback before and during the development of each RFA. 28 Report 2017-129 | CALIFORNIA STATE AUDITOR July 2018 To increase transparency and ensure that applicants have the information necessary to understand the grant process, Rehabilitation should include in its RFAs clear scoring criteria and descriptions of the evaluation, award, and appeals processes, including the process it will use to address applications that receive tied scores. To ensure that Rehabilitation maintains all relevant grant documentation and responds fully to requests for public records, it should immediately adhere to its records retention policy and save all grant‑related documents, including email correspondence and attachments, to a centralized location. CALIFORNIA STATE AUDITOR | Report 2017-129 29 July 2018 Significant Gaps in the Evaluation of Grants Raised Questions About the Adequacy and Fairness of the Process Key Points • Administrative review teams accepted certain documents from some applicants after the application deadlines disclosed in the RFAs for three grants. In addition, by not publishing a solicitation for evaluators, as suggested by its grant manual, Rehabilitation limited its pool of prospective evaluators to select individuals and missed the opportunity to ensure that it obtained the most qualified evaluators possible from the larger disability community. Further, Rehabilitation did not adequately consider bias or the appearance of bias when selecting evaluators, or ensure that evaluators were free from conflicts of interest and were aware of confidentiality procedures before they scored applications. • Rehabilitation could not demonstrate that it followed its grant manual by providing adequate training or written instructions to evaluators regarding the purpose of the grants, relevant regulatory requirements, or how to evaluate and score applications. Without adequate training, Rehabilitation’s evaluators did not have the knowledge they needed to score the applications appropriately, creating some delays in the grant process due to rescoring. • The technical review teams, which are generally responsible for overseeing the evaluation process, did not always adequately review the evaluators’ scores and comments to ensure that evaluators followed the evaluation process and that the process was consistent and fair for the applicants for each of the grants we reviewed. Acceptance of Grant Documents After the Filing Deadlines Before Rehabilitation provides the grant applications to evaluators for scoring, the grant manual states that an administrative review team, which Rehabilitation assembles for each grant and which includes staff from the Contracts and Procurement Division, is responsible for reviewing all applications for completeness and ensuring that applicants submit all required documents. The grant manual also indicates that the administrative review team is responsible for determining whether applicants submit their applications by the deadline in the RFA. As we discuss on page 24, Rehabilitation included hard deadlines in each of the four RFAs we reviewed. Further, the grant manual states that in no case should Rehabilitation allow an applicant to change the content of an application narrative or other application component after the filing deadline. However, according to the former contracts chief, if the administrative review team identifies missing documents before the deadline, staff may ask the applicant to submit the documents, although Rehabilitation would accept only “immaterial” documents after the deadline. To explain what the administrative review team considers immaterial, the former contracts chief provided 30 Report 2017-129 | CALIFORNIA STATE AUDITOR July 2018 an example of an applicant submitting a required form without a necessary signature before the deadline, and explained that Rehabilitation would consider the missing signature immaterial and ask the applicant to sign and return the form after the deadline. Rehabilitation’s practice of accepting some documents after the submission deadline is similar to what the grant manual describes as a soft deadline. The administrative review teams accepted certain documents after the hard deadline specified in the RFAs for three of the four grants we reviewed, as shown in Table 3, and thus may have provided some applicants with an unfair advantage. For example, for the 2015 TBI grant, we found that a member of the administrative review team requested additional documentation from the Independent Living Center of Southern California (ILCSC) after the submission deadline. One of the documents ILCSC submitted after the deadline fit the former contracts chief’s definition of immaterial documents; however, the other document was an entire revised section of ILCSC’s application narrative. The administrative review team accepted the required documents late and did not disqualify ILCSC, although under the terms of the RFA it should have done so. Subsequently, Rehabilitation granted one of the seven awards for the 2015 TBI grant to ILCSC, possibly preventing another qualified applicant from receiving grant funding. Rehabilitation also accepted late documents from some of the four applicants we selected to review for the 2014 OIB grant and for one of the four applicants we selected for review for the 2017 OIB grant. When we asked the chief deputy if the practice of accepting documents after the deadlines published The chief deputy was unaware that in the RFAs was appropriate, she explained that she was unaware the administrative review teams that the administrative review teams had accepted documents after had accepted documents after the the deadline, and that she did not believe this was an acceptable deadline, and she did not believe practice. Because Rehabilitation did not enforce its deadlines, it this was an acceptable practice. lacked a process control to ensure that the administrative review team did not accept portions of, or entire applications, after the deadlines. Failure to Publish Solicitations for Evaluators and Reasonably Ensure That Evaluators Are Bias Free The grant manual states that Rehabilitation should publish a solicitation for evaluators on its website that includes the essential and desirable qualifications for evaluators. However, Rehabilitation could not demonstrate that it issued a solicitation for evaluators with a list of qualifications for three of the four grants we reviewed, as shown in Table 3. For the 2017 Systems Change grant, the chief of independent living at the time of the grant process stated that he did not publish a solicitation for evaluators for the first evaluation panel CALIFORNIA STATE AUDITOR | Report 2017-129 31 July 2018 because he already had potential evaluators in mind, and for the second evaluation panel he was facing significant time constraints and wanted evaluators who would be seen as highly credible by the independent living community. As the grant manual indicates, the best evaluators are those with a breadth of knowledge that is relevant to the specific services provided under the respective grant. By not issuing a solicitation for evaluators, Rehabilitation limited its pool of prospective evaluators to selected individuals and missed the opportunity to ensure that it obtained the most qualified evaluators possible from the larger disability community. Table 3 Rehabilitation Generally Failed to Follow Its Administrative Review and Evaluator Selection Processes NAME OF GRANT AND YEAR AWARDED SYSTEMS OIB TBI OIB CHANGE ADMINISTRATIVE REVIEW AND EVALUATOR SELECTION PROCESS 2014 2015 2017 2017 Administrative review team ensures that applicants met the deadline X X X  and included all required documentation per the RFA Issue a public solicitation for evaluators that includes essential X  X X evaluator qualifications Certify that evaluators are free from conflicts of interest before X X X X selecting evaluators Source: California State Auditor's review of Rehabilitation’s administrative review and evaluator selection processes for four grants awarded from fiscal years 2014–15 through 2017–18, and Rehabilitation's grant manual.  = Followed X = Did not follow For one of the three grants for which Rehabilitation did not issue a solicitation for evaluators, it selected two evaluators with previous ties to one of the applicants, creating at least the appearance of potential bias. Consistent with state law, Rehabilitation’s grant manual states that evaluators must be free of financial interests in any of the applicant organizations. Although the grant manual states that evaluators must be free of personal relationships with any of the applicants’ principals or employees, state law is largely silent on prohibiting or disclosing personal bias in grant decision makers. We did not identify any evaluators with financial conflicts of interest; however, we found one grant in which Rehabilitation selected evaluators who had held leadership positions in an organization that had a known affiliation to the grant applicant that ultimately received the grant award. Specifically, we found that the two evaluators who made up the second evaluation panel for the 2017 Systems Change grant were former executive directors of FREED Center for Independent Living (FREED), which is one of 21 member organizations of the awardee, the California Foundation for Independent Living Centers (CFILC). 32 Report 2017-129 | CALIFORNIA STATE AUDITOR July 2018 One of these evaluators had served as the executive director of FREED from 1994 through 2001, whereas the other served in this capacity from 2001 through 2007. These evaluators gave CFILC the higher score of the two applicants, and Rehabilitation subsequently awarded the grant to CFILC. The independent living deputy director indicated that Rehabilitation reviewed the evaluators only for financial conflicts of interest and did not discuss bias or the perception of bias when selecting them. However, by not carefully considering the effect its selection of evaluators could have on the integrity of the process, Rehabilitation created the potential for perceived bias when it selected evaluators with former leadership positions in an organization that was a member of one of the two applicants. The independent living deputy director explained that the independent living community is small, making Rehabilitation further limited the it difficult to find evaluators who are subject‑matter experts without pool of potential evaluators by any connections to the applicants. Although this may be true, choosing not to issue a solicitation Rehabilitation further limited the pool of potential evaluators by for evaluators. choosing not to issue a solicitation for evaluators. Rehabilitation also did not ensure that prospective evaluators were free of conflicts of interest before selecting them for any of the four grants we reviewed, as Table 3 shows. The grant manual indicates that after Rehabilitation provides evaluators with conflict‑of‑interest and confidentiality training, evaluators must sign a form certifying that they are free from any conflicts of interest with any of the organizations that are competing for the grant and that they understand how to keep the grant process confidential.4 For the 2014 and 2017 OIB grants, Rehabilitation had some evaluators sign conflict‑of‑interest and confidentiality forms on the day of the evaluation and could not demonstrate that it provided all evaluators with the list of applicants before they signed the form or scored the applications. By not following the procedure described in its grant manual, Rehabilitation risks facing delays in the grant process if it needs to find a replacement for an evaluator who identifies on the day of the evaluation, or after evaluations begin, that he or she has a conflict with an applicant. For the 2017 Systems Change grant, Rehabilitation did not obtain conflict‑of‑interest and confidentiality forms until the day of the evaluation for some evaluators and, in one instance, could not demonstrate that evaluators signed such forms until months after they scored applications and after the grant was awarded. Specifically, several days after the evaluation concluded, an independent living manager sent the three evaluators on the first panel an email asking them to sign conflict‑of‑interest and confidentiality forms. More than three months later, she 4 Although similar, the discussion on pages 16 through 20 refers to Rehabilitation not providing employees involved in the grant process with training on conflicts of interest and confidentiality. The discussion in this section specifically relates to evaluators. CALIFORNIA STATE AUDITOR | Report 2017-129 33 July 2018 emailed two of the three evaluators, asking them to sign the conflict‑of‑interest and confidentiality forms again because she could not locate them. In another instance, Rehabilitation allowed an evaluator on the third evaluation panel for the 2017 Systems Change grant to submit her conflict‑of‑interest and confidentiality form along with her score sheet when she completed the evaluation—once again failing to ensure that an evaluator was free from conflicts of interest before scoring applications. The chief of independent living at the time stated that he was not concerned with the evaluator submitting the form late because he had a telephone conversation with her in which she declared that she did not have any conflicts of interest. However, we do not consider a verbal discussion between Rehabilitation staff and an evaluator sufficient to determine whether a conflict of interest exists. In our review of another state agency’s conflict‑of‑interest policies for best practices in selecting evaluators, we found that they are more comprehensive than Rehabilitation’s. For example, the agency can also elect to use subject‑matter experts as evaluators; however, it requires evaluators to agree not to score an application from an agency for which they have worked, not to score an application on which they have consulted, and not to score an application from an agency with which they have had any connection that is, or might appear to be, a conflict. Further, the agency’s sample conflict‑of‑interest and confidentiality agreement requires evaluators to disqualify themselves if they have a personal or professional relationship with an applicant that could affect their objectivity. In contrast, Rehabilitation’s conflict‑of‑interest and confidentiality forms for three of the four grants we reviewed required evaluators to certify that they have no personal or financial interest, and no present or past employment or activity, that would be incompatible with their participation in the grant process. However, Rehabilitation’s forms did not define what it considers “incompatible” with the grant process. We also found that the conflict‑of‑interest and confidentiality forms used for the 2017 Systems Change grant process did not prohibit evaluators with past employment with applicants from participating in evaluations. To reduce potential bias or the perception of bias, we believe Rehabilitation should expand its disclosure on its conflict‑of‑interest and confidentiality forms to include past and current employment, past and current relationships, and activities that may constitute bias. As described on page 19, Rehabilitation did not establish procedures or inform staff of how to keep information and documentation regarding its grant process confidential. Similarly, it did not provide Rehabilitation did not provide adequate training to evaluators to ensure that they understood adequate training to evaluators to how to keep the process confidential before having them sign ensure that they understood how to conflict‑of‑interest and confidentiality forms, as shown in Table 4 keep the process confidential. on the following page. For the 2017 Systems Change grant, email 34 Report 2017-129 | CALIFORNIA STATE AUDITOR July 2018 correspondence indicates that Rehabilitation received notice of a possible confidentiality breach when an evaluator, who was not a Rehabilitation employee, requested a stipend from her superior for an additional two days to rescore applications due to irregularities with the grant review process. Although we did not consider this disclosure of information to be significant enough to adversely affect the outcome of the 2017 Systems Change grant, the independent living deputy director stated that it would have been irresponsible to disregard the confidentiality breach and explained that it was one of the reasons she convened a second evaluation panel. By not providing evaluators training on its confidentiality procedures, Rehabilitation exposes the grant process to confidentiality breaches that are significant. Such breaches could include providing an applicant with information that could give it an unfair advantage, or leaking the results of an evaluation before Rehabilitation publishes the award. Table 4 Rehabilitation Did Not Provide Adequate Training and Instructions to Evaluators NAME OF GRANT AND YEAR AWARDED SYSTEMS OIB TBI OIB CHANGE EVALUATOR TRAINING AND INSTRUCTIONS 2014 2015 2017 2017 Provide evaluators training and written instructions on: X X X X Confidentiality procedures t X X t The purpose of the grant program X X X X Regulatory requirements of the grant t t t X How to evaluate applications Source: California State Auditor's review of Rehabilitation’s evaluator training process for four grants awarded from fiscal years 2014–15 through 2017–18, and Rehabilitation's grant manual. t = Partially followed X = Did not follow Inadequate Evaluator Training and Instructions Rehabilitation could not demonstrate that it provided adequate training or written instructions to its evaluators to ensure that they understood the grant evaluation process, including how to score grant applications, for the four grants we reviewed. In its grant manual, Rehabilitation states that evaluators are to receive training and instructions on the purpose of the grant, relevant regulatory requirements, and the process for evaluating and scoring applications. CALIFORNIA STATE AUDITOR | Report 2017-129 35 July 2018 Further, Rehabilitation acknowledges in the grant manual that evaluators may not have sufficient financial knowledge and experience to understand financial documents submitted by an applicant. The most egregious and frequent errors that, in part, resulted from Rehabilitation’s failure to provide evaluators with adequate training and written instructions occurred in the 2017 Systems Change grant, causing the panel to have to rescore the applications. In the first round of scoring for the grant, the three evaluators provided identical scores and similar comments for each applicant. Although the grant manual states that evaluators should discuss their assessments of each scoring component among themselves, it also states that they do not need to reach a consensus on the scores they assign to the applicant. However, Rehabilitation cannot demonstrate that it provided this instruction to the evaluators. In addition, each evaluator provided SCRS, one of the two applicants, a score of 0 for one component, which was valued at up to 15 points per evaluator, for a potential total of 45 points. The scoring benchmarks stated that evaluators should assign a score of 0 when an applicant does not answer a question or does not address any of the components of a question on the application. After the evaluation, however, the chief of independent living at the time indicated that program staff found that SCRS’ application was responsive to the component that the evaluators had given a score of 0. When we asked two of the three evaluators how the identical scores occurred, they explained that because they were not instructed that they should not have identical scores, the group read a section of an application together, discussed the section, and agreed on a score and corresponding comment. This approach indicates that the evaluators did not conduct individual assessments as specified by the grant manual. In fact, the two evaluators indicated that program staff did not clearly explain to them that they could award points to an applicant for a component that the applicant did not address in the section for that component but did address in another section of the application. The program manager stated that after noticing these errors, program staff provided evaluators with additional verbal instruction that they provide a score of 0 only if the applicant did not address the component in any section of an application. However, the program manager did not document what was included in the additional verbal instruction, and Rehabilitation still did not provide written instructions to the evaluators before asking them to rescore the applications a second time. In its second round of scoring, the same evaluation panel scored some components more than 3 points apart, a large enough difference that Rehabilitation grew concerned that this indicated that the evaluators were not using the same criteria to review the applications. The new scores also resulted in a different grant 36 Report 2017-129 | CALIFORNIA STATE AUDITOR July 2018 recipient. Specifically, without adequate instruction on how Rehabilitation expected them to score the second round, for a component with a possible score of 15 points, one evaluator provided SCRS with a score of 14 points, another provided a score of 11 points, and the remaining evaluator scored it as 1 point. The evaluator who provided SCRS with 1 point explained that she believed SCRS had not addressed the question and that a low score for this component was entirely appropriate. The chief of independent living at the time emailed the evaluators to ask them how they would resolve the misalignment in the few areas where they had more than a 3‑point difference between the highest and lowest scores. This email indicates that Rehabilitation expected the evaluators to score each component within a 3‑point differential, although it had not documented this expectation in the grant manual or, based on available documentation, provided them with this instruction before the second round of scoring. Upon identifying the differences among the scores, Rehabilitation asked the evaluation panel to score the applications a third time to resolve those areas where they had scored the applicants more than 3 points apart. The program manager stated that she provided the evaluators with additional verbal instruction that the component scores had to be within 3 points, but again did not provide them with written instructions before they rescored the applications a third time. Without providing evaluators with sufficient training and written instructions to reference during the evaluation process, Rehabilitation cannot ensure that evaluators understand how to score applications consistently and adequately. Failure of the Technical Review Team to Fulfill All Responsibilities According to the grant manual, Rehabilitation must appoint a technical review team for each grant consisting of subject‑matter experts on program policy, such as the program manager of the division responsible for awarding the grant, and on the RFA process, such as the program manager of contracts. The team is responsible for providing the evaluators with orientation information; meeting with evaluators to answer program or process questions; ensuring that evaluators follow instructions; applying the tiebreaker rule if necessary; and summarizing the evaluation process and results, including an awardee recommendation, in a memorandum to the director and chief deputy. Although Rehabilitation did not always appoint technical review teams to oversee the evaluation for each grant, as shown in Table 5, we found that staff sometimes fulfilled the responsibilities of these teams. However, the grant manual does not describe procedures sufficient to guide a technical review team’s assessment of the evaluation process to ensure a consistent and fair process for all grant applicants. CALIFORNIA STATE AUDITOR | Report 2017-129 37 July 2018 Table 5 Rehabilitation’s Technical Review Teams Did Not Always Fulfill Their Responsibilities NAME OF GRANT AND YEAR AWARDED SYSTEMS OIB TBI OIB CHANGE TECHNICAL REVIEWS OF THE EVALUATION PROCESS 2014 2015 2017 2017 X  t t Appoint technical review team t t t t Technical review team ensures that evaluators follow the evaluation process Technical review team summarizes the evaluation process, resulting in a t  t X memorandum to the director and chief deputy Source: California State Auditor's review of Rehabilitation’s technical review team selection and review process for four grants awarded from fiscal years 2014–15 through 2017–18, and Rehabilitation's grant manual.  = Followed t = Partially followed X = Did not follow Rehabilitation’s technical review teams and staff failed to adequately ensure that evaluators followed the evaluation process for each grant we reviewed, as shown in Table 5. Technical review teams and staff generally stated that they reviewed evaluators’ scores and comments. However, because these teams and staff could not demonstrate that they provided adequate training and instructions to evaluators on how to evaluate applications, as described previously, and because Rehabilitation does not have procedures to guide assessments of the evaluation process, we found that the technical review teams and staff did not adequately ensure that evaluators followed the evaluation process. For example, the evaluators’ scoring sheets for two of the grants included checklists, and in both instances the evaluators either did not consistently use the checklists to support their scores or did not use the checklists at all. In particular, an evaluator for the 2015 TBI grant gave an applicant a score of 5 out of a total of 7 points for one component by checking all of the boxes on the checklist, thus apparently indicating that the applicant addressed all of the criteria for the component, which could have warranted a score of 7. However, this evaluator did not provide comments, and therefore it is unclear why the evaluator deducted points from this component. In another instance, the evaluators gave an applicant identical scores for a component but disagreed on how much of the component the applicant addressed in their checklists, with one evaluator indicating that the component was not addressed at all and another indicating that it was fully addressed. The program manager for this grant could not explain why there was a discrepancy in how evaluators used the checklist, and could only speculate as to why this was the case. In addition, Rehabilitation included a checklist on the score sheets program staff provided to the first and second evaluation panels for the 2017 Systems Change 38 Report 2017-129 | CALIFORNIA STATE AUDITOR July 2018 grant. However, none of the evaluators from the first evaluation panel used the checklist to support their scores. Moreover, evaluators’ comments for two of the four grants we reviewed did not always align with their scores or the scoring criteria in the RFA. For example, for the 2017 OIB grant, evaluators noted that an applicant did not include Had the technical review teams braille as a service it provides; however, the RFA scoring criterion performed a more thorough review, for that component did not require applicants to specifically provide they could have identified and braille services. Had the technical review teams performed a more rectified issues. thorough review, they could have identified and rectified these issues. In addition, although the grant manual indicates that evaluators should provide evidence, such as comments, to support their scores, Rehabilitation did not always ensure that they did so for the four grants we reviewed. Specifically, the first evaluation panel for the 2017 OIB grant generally did not include comments to justify their scores for three of the six applications we reviewed.5 The deputy director of the Administrative Services Division, who is primarily responsible for the grant manual, stated that she did not know why Rehabilitation does not require evaluators to include comments to support their scores. We question how the technical review teams can reasonably conclude that evaluators followed the evaluation process if evaluators do not include comments to justify their scores. Further, according to the grant manual, the technical review team should submit a memorandum summarizing the evaluation process and results to the director and chief deputy and request approval for its grant selection recommendation. After the director and chief deputy have reviewed the memorandum, the grant manual states that they have two options: accept the recommended applicant and issue the award or reject the recommended applicant and restart the grant process. Three of the four grants we reviewed did not fully follow this procedure. For example, for the 2017 Systems Change grant, the technical review team did not issue a memorandum to the director and chief deputy. According to the chief of independent living at the time, who was also a member of the Systems Change technical review team, he did not submit a memorandum because he was not instructed to create a memorandum and he expected his supervisor, the independent living deputy director, to communicate the evaluation process and results to the director and chief deputy. Because of this failure to follow the process outlined in the grant manual, and because Rehabilitation staff did not identify any errors in the evaluation process, the independent living deputy director made the decision to award the grant to CFILC, based on the recommendations of the evaluation panel but without input from the director and chief deputy. Although the grant manual states that the director and chief deputy should 5 As we discuss on pages 55 and 56, the 2017 OIB grant included two evaluation panels and two rounds of scoring. CALIFORNIA STATE AUDITOR | Report 2017-129 39 July 2018 consider and decide whether to accept the technical review team’s recommendations to award grants, the chief deputy stated that, when she assumed her position in 2016, it was Rehabilitation's practice to have program deputy directors award grants. Therefore, she explained that she did not consider modifying the practice. However, the chief deputy was unable to provide any documentation to support that the director or chief deputy had delegated the authority to award grants. The chief deputy ultimately agreed that Rehabilitation should clarify the roles of the deputy director in the grant manual. Therefore, if Rehabilitation determines that the director and chief deputy can delegate this authority to program deputy directors, we believe Rehabilitation’s policies and procedures should reflect this. Questionable Rescoring of Applications Although the grant manual does not specifically allow Rehabilitation to rescore applications, we found that when program staff identified scoring inconsistencies in the evaluation of applications, Rehabilitation sometimes convened new evaluation panels to rescore applications. In cases where an RFA has omissions or errors in its disclosed scoring criteria, it may be necessary for Rehabilitation to restart the grant process to remedy those issues. However, when issues that caused the scoring inconsistencies relate to internal processes, such as errors in the benchmarks or score sheets Rehabilitation provides to evaluators, we believe it may be reasonable for Rehabilitation to rescore applications without restarting the grant process, but only if it corrects the issues that caused those inconsistencies. For example, if Rehabilitation identifies a discrepancy between the scoring criteria outlined in the benchmarks and the RFA, Rehabilitation could amend its benchmarks to align with the RFA to remedy the discrepancy. In such a case, Rehabilitation could then ask evaluators to rescore applications with the correct benchmarks as guidance. However, for two of the four grants we reviewed, Rehabilitation chose For two of the four grants we to rescore the applications before it issued an initial grant award, reviewed, Rehabilitation chose to without sufficiently addressing the issues that caused concern. For rescore the applications before example, for the 2017 Systems Change grant, Rehabilitation created it issued an initial grant award, three separate evaluation panels for a total of five rounds of scoring, without sufficiently addressing the four of which occurred before the grant award and one of which issues that caused concern. occurred after an appeal of its decision to award the grant to CFILC, as illustrated in Figure 2 on the following page. As we discussed on pages 35 and 36, Rehabilitation identified inconsistencies in the first two rounds of scoring, specifically related to inadequate and inaccurate instruction provided to evaluators. However, rather 40 Report 2017-129 | CALIFORNIA STATE AUDITOR July 2018 Figure 2 Rehabilitation Convened New Evaluation Panels to Rescore Applications for the Systems Change Grant and Deviated From Its Intended Grant Process Summary of the Intended Grant Process per the Grant Manual Process for the Systems Change Grant Rehabilitation develops Second Third and issues the RFA. First Evaluation Panel Evaluation Panel Evaluation Panel Rehabilitation issues a Rehabilitation selected Evaluators rescored Evaluators rescored Rehabilitation selected Rehabilitation selected solicitation for evaluators, three evaluators without applications. applications. two evaluators without three evaluators without provides conflict-of-interest issuing a solicitation issuing a solicitation issuing a solicitation training, and selects or providing or providing or providing evaluators. conflict-of-interest training. conflict-of-interest training. conflict-of-interest training. Rehabilitation provides Rehabilitation did not Rehabilitation did not Rehabilitation did not training to evaluators on provide adequate training provide adequate training provide adequate training the grant process. on the grant process. on the grant process. on the grant process. Evaluators review and Evaluators completed Evaluators submitted Evaluators submitted Evaluators submitted Evaluators conducted score applications. their evaluation of their second evaluation their third evaluation on their evaluation on their evaluation on two applications on on March 3, 2017, and March 10, 2017, and April 11, 2017, and November 13, 2017, and February 24, 2017, and scored SCRS the highest. scored SCRS the highest. scored CFILC the highest. scored CFILC the highest. scored CFILC the highest. Rehabilitation issued a Rehabilitation issued a notice of intent to award notice of intent to award the grant to CFILC. the grant to CFILC. Technical review team Program management Program management Program management verifies that evaluators noted that evaluators’ noted that evaluators had set aside the scores, followed instructions in scores and comments scores with more than a noting that they had SCRS appealed the SCRS appealed the evaluating applications. were identical and 3-point difference changed dramatically grant award decision. grant award decision. that evaluators gave between the highest between the second SCRS scores of 0 and lowest scores for and third round of Technical review team Rehabilitation selected Rehabilitation selected in some categories. four scoring components. scoring, and selected summarizes the a review committee to a review committee to new evaluators. evaluation process and review the appeal. review the appeal. recommends award(s) to the director and The review committee The review committee chief deputy for approval. found discrepancies in the upheld the decision to scoring materials that award the grant to CFILC. invalidated the scoring Director and chief deputy Director and chief deputy OR process for both evaluation approve recommended deny recommended award(s). panels. It recommended CFILC award(s). that Rehabilitation form a $ new evaluation panel to Rehabilitation issues a rescore applications. notice of intent to award. Source: California State Auditor’s review of the Systems Change RFA, SCRS’ appeals, review committee decisions, and documentation provided by and interviews with Rehabilitation staff, and Rehabilitation’s grant manual. Indicates that Rehabilitation deviated from its intended grant process. CALIFORNIA STATE AUDITOR | Report 2017-129 41 July 2018 than developing clear written instructions for evaluators to follow, Rehabilitation asked the same evaluators to rescore the applications again, resulting in the third round of scoring. After the third round of scoring, in which the panel scored SCRS higher than CFILC, Rehabilitation chose to convene a second evaluation panel to rescore the applications for a fourth time because, according to the review committee’s decision regarding SCRS’ first appeal, the total scores had “changed dramatically” between the second and third rounds of reviews by the first evaluation panel. Once Rehabilitation established a second evaluation panel, one of the evaluators informed Rehabilitation that the scoring benchmarks the evaluators received did not align with the score sheets. Despite learning of this discrepancy, Rehabilitation did not remedy it before the second evaluation panel completed its scoring. Further, Rehabilitation could not demonstrate that it provided evaluators with direction to address the difference in the scoring criteria. Because of this failure to resolve the discrepancies between the benchmarks and score sheets, a grant review committee invalidated the evaluation and ultimately recommended that a third panel evaluate the applications, as we discuss later in the report. If Rehabilitation had provided evaluators with additional guidance on how to resolve the discrepancies, it could have resolved any confusion about the evaluation process and potentially prevented the circumstances that led to a fifth round of scoring of the applications. The other grant for which Rehabilitation chose to rescore the applications before issuing the initial award was the 2014 OIB grant. Before it awarded this grant, Rehabilitation convened a new evaluation panel to rescore the applications. The deputy director of specialized services explained that after the first evaluation panel finished scoring the applications, she became concerned with the panel’s scoring and understanding of the OIB program and its services. Specifically, she felt that the first panel did not fully understand how to properly evaluate the grant applications because, contrary to her expectations, a disproportionately large number of agencies that had not provided services in the past received higher scores than some previous grantees for the OIB program. Because of this issue, and her concern about the impact that different awardees could have on the quality of services provided to OIB consumers, she and the program manager selected a second panel of subject‑matter experts to rescore the applications. However, we question the necessity of a second evaluation panel. As we discussed previously, Rehabilitation did not issue a solicitation for evaluators to ensure that it had a broad pool of potential candidates, or provide adequate training and written instructions to evaluators on the evaluation process. Given that it is Rehabilitation’s responsibility to select qualified evaluators and provide them with the necessary information to adequately conduct their scoring of applications, we 42 Report 2017-129 | CALIFORNIA STATE AUDITOR July 2018 question why Rehabilitation did not initially take the necessary steps to select and adequately prepare evaluators. If it had done so, it may have been able to avoid the circumstances that led to it selecting a second evaluation panel. Further, one of the evaluators Rehabilitation selected for the 2014 OIB grant’s second evaluation panel was a program analyst for the OIB program who had participated in the development of the RFA and had overseen the initial evaluation panel. Specifically, we found emails indicating that she assisted in the development of the RFA and compiled the scores from the first evaluation panel into a spreadsheet. Further, she also conducted the administrative review for some applications for this grant. According to the program manager for the grant, it was not Rehabilitation’s practice to have program staff act as evaluators for grant applications, and it is generally inappropriate for program staff to do so. In this case, however, he stated that he felt comfortable with the program analyst serving on the second evaluation panel because she was very familiar with applicants who provide OIB services and is an extremely objective person. Regardless of his belief of the analyst’s Rehabilitation risked introducing ability to be objective, Rehabilitation risked introducing bias into bias into the evaluation when the evaluation when it selected an individual who participated it selected an individual who in the development of the RFA and reviewed scores from the participated in the development of first evaluation panel. Because the grant manual indicates that the RFA and reviewed scores from program staff are responsible for developing RFAs, we believe it is the first evaluation panel. inappropriate for them to participate as evaluators for grants their program administers. In addition, this program analyst had already participated in conducting extensive analyses of several applications and made recommendations regarding which applicants Rehabilitation should award funding to before she joined the second evaluation panel. The analyses included a review of the types of services the applicants provide and the number of consumers applicants proposed to serve. In some cases, the analysis included recommendations that Rehabilitation award funds to applicants even if they did not score the highest in the evaluation, indicating, for instance, that the applicant that scored higher had no experience or staff to provide OIB services, or that the applicant did not provide services in the area at that time. We question why Rehabilitation chose to place this individual on the second evaluation panel, given that her perspective may have influenced the evaluation and awards to certain applicants. We also found that the evaluators from this second panel did not complete individual score sheets as specified in the grant manual, and there was not a clear separation of duties. When we spoke to the program analyst and one of the other evaluators, both generally explained that since the program analyst was the only CALIFORNIA STATE AUDITOR | Report 2017-129 43 July 2018 sighted member of the evaluation panel, she read the applications to the other evaluators and recorded the panel’s scores. They both stated that the panel submitted one combined score sheet because of time constraints. However, the evaluator elaborated that the panel scored the applications together and reached a consensus on the score or averaged the evaluators’ scores to give applicants a single score. Although we appreciate Rehabilitation’s attempt to accommodate both sighted and blind evaluators, the grant manual indicates that each evaluator should use an individual score sheet, and that the evaluation coordinator should subsequently enter the individual scores into a spreadsheet. However, in this instance, the program analyst served both as an evaluator and as the coordinator who recorded the scores. Further, neither the program manager nor one of the evaluators recalled anyone other than the program analyst reading the scores to the evaluation panel to verify their accuracy. We believe that keeping these roles separate is a critical process control to ensure that the scores accurately reflect each evaluator’s assessment. Furthermore, in this case, because two of the evaluators were blind, Rehabilitation lacks assurance that the sighted evaluator appropriately and accurately recorded their scores. Recommendations Legislature To avoid bias or the perception of bias, the Legislature should enact legislation that prohibits state agencies from selecting as an evaluator of grant applications a representative, former member, or former staff of any organization or person that is applying to receive grant funding from the state agency. Rehabilitation To ensure consistency and fairness in the evaluation process, Rehabilitation should make sure that it accepts only complete applications submitted before the deadline, unless otherwise specified in the RFA. If the RFA specifies a hard deadline and applicants submit incomplete applications, Rehabilitation should not accept any portions of applications submitted after the deadline and should assess the penalty for incomplete applications specified in the RFA. To help ensure that evaluators adequately protect confidential information and that the evaluation process is fair, Rehabilitation should develop standardized evaluator training for confidentiality procedures and conflicts of interest, including a discussion of bias 44 Report 2017-129 | CALIFORNIA STATE AUDITOR July 2018 or the appearance of bias. Rehabilitation should also ensure that the candidates receive this training and sign conflict‑of‑interest and confidentiality forms before it selects evaluators. Further, it should prohibit program staff who participate in the development of an RFA from acting as evaluators for the applications Rehabilitation receives in response to that RFA. To increase the transparency of its selection process and to ensure that it receives the most qualified evaluators possible, Rehabilitation should issue a public solicitation for evaluators for each grant that includes a description of essential and desirable qualifications. To ensure that evaluators have the information necessary to sufficiently and fairly assess and score applications, Rehabilitation should develop training by December 2018 that can be tailored to each grant and includes at a minimum the following topics: • The purpose and relevant regulatory requirements for the grant. • Instructions on how to score applications, including an applicant’s financial information, and direction that they must provide comments to support their scores. Rehabilitation should provide this training to evaluators before allowing them to score applications. To ensure that it provides sufficient oversight of the grant process, Rehabilitation should ensure that the technical review teams it assigns to grants provide the director and chief deputy with a memorandum summarizing the evaluation process and the evaluators’ recommended grant awardees. Rehabilitation should also designate an individual responsible for reviewing and approving the memorandum and recommended awardees before it publishes its notice of intent to award. If it finds errors in an evaluation that merit restarting the grant process, rescoring of applications, or convening a new evaluation panel, Rehabilitation should resolve any issues before it begins the rescoring process. It should also notify applicants to ensure that they are aware of any changes to the process due to the errors. Further, it should consider promulgating regulations and amending its grant manual to permit staff to request evaluators to rescore applications or convene a new panel when it finds issues with an evaluation. CALIFORNIA STATE AUDITOR | Report 2017-129 45 July 2018 Rehabilitation Did Not Always Follow Its Appeals Process as Suggested in Its Grant Manual, and Its Review Committees Did Not Always Ensure Fair Evaluations Key Points • Although Rehabilitation followed the appeals process set forth in state regulations, it did not always follow the appeals process contemplated in its grant manual. For example, the chief deputy did not always notify the intended grantees of appeal requests that could affect their grant awards, a notification that the grant manual provides as an option but that we believe should be required. The chief deputy stated that she adhered to state regulations when addressing appeals but was unfamiliar with the additional steps provided in the grant manual. • The review committee responsible for reviewing appeals and reaching a final determination on the outcome did not adequately review each appeal and missed key areas, such as potential evaluator prejudice and whether scores were supported by evidence. Further, for the 2017 Systems Change grant, the review committee made recommendations to Rehabilitation upon identifying deficiencies in the grant process; however, Rehabilitation chose not to implement these recommendations and allowed some errors to persist through the subsequent evaluation and award. • Rehabilitation received nine appeals for the four grants we reviewed, and appellants cited various reasons for their appeals, including evaluator bias, that evaluators’ scores were not justified, and that the evaluation process lacked clear and appropriate scoring criteria. For Most Grants, Rehabilitation Generally Followed Its Process When Receiving Appeals Rehabilitation followed its appeals process as required by regulations, and generally followed the best practices identified in its grant manual, when receiving appeals and appointing review committees for the nine appeals it received, as shown in Table 6 on the following page. State regulations provide minimal direction regarding how Rehabilitation should review and process appeals, while its grant manual describes optional procedures that are significantly more detailed. State regulations require applicants to submit appeal requests within 30 days of the date Rehabilitation notified the public of its intent to award the grant. We found that all appellants submitted their appeal requests within 30 days of the date of the award notice. The grant manual gives the chief deputy the option, upon receipt of an appeal, to acknowledge receiving the appeal in writing and to notify all intended awardees that could be affected by the decision for the appeal, a procedure we find to be a best practice and that we think should be required by Rehabilitation. Although Rehabilitation’s chief deputy acknowledged all appeal requests in writing, she generally did not notify all intended awardees of the appeals. However, we found in our review that other Rehabilitation staff, such as program managers, notified most intended awardees of the appeals. 46 Report 2017-129 | CALIFORNIA STATE AUDITOR July 2018 Table 6 The Chief Deputy Complied With State Regulations and Generally Followed Best Practices When Receiving Appeals NAME OF GRANT AND YEAR AWARDED SYSTEMS OIB TBI OIB CHANGE PROCESS FOR RECEIVING APPEALS 2014 2015 2017 2017 Number of appeals 1 1 5 2 Rehabilitation accepts appeal requests within 30 days of the     notice of intent to award, as required by state regulations     Chief deputy acknowledges receipt of the appeal t t t t Chief deputy notifies the intended awardee of the appeal Chief deputy appoints a review committee, as required by     state regulations Chief deputy notifies appellant of the individuals appointed to   X X serve on the review committee and their qualifications Source: California State Auditor’s review of the appeals process for selected grants awarded from fiscal years 2014–15 through 2017–18, state regulations, and Rehabilitation’s grant manual.  = Followed t = Partially followed X = Did not follow State regulations require the chief deputy to appoint a review committee of up to three Rehabilitation employees to review each appeal. In addition, the grant manual gives the chief deputy the option to notify the appellant regarding the members appointed to serve on the review committee and their qualifications. For seven of the nine appeals, the chief deputy provided the appellant with job titles of the members of the review committee, but she did not provide information regarding the members’ qualifications as suggested by the grant manual. The chief deputy explained that she followed state regulations when administering appeals and was unfamiliar with the additional guidance in the grant manual, since she had not reviewed it and Rehabilitation had not yet adopted it. We would expect her to be familiar with the guidance Rehabilitation has made available to its staff, and she acknowledged that she should have been familiar with the grant manual, even in its draft form. Inadequate Assessment and Response to Appeals For each appeal, the grant manual authorizes Rehabilitation to consider adding language to the RFA requiring the review committee to review records relevant to the evaluation and scoring of the appellant’s application, including the applications of intended grantees whose grant award the appeal could affect. It also suggests that the review committee should determine whether the evaluation CALIFORNIA STATE AUDITOR | Report 2017-129 47 July 2018 panel followed the requirements for evaluation described in the RFA and, if it discovers any procedural errors or omissions, to determine if the error or omission had a substantial effect on the outcome of the overall scoring. Further, the grant manual allows the review committee to determine whether there is evidence that evaluator prejudice affected the scoring process, and whether evaluators supported their scoring with evidence from the relevant applications. Because the grant manual suggests that the review committees should determine whether the potential for evaluator prejudice existed as part of their appeal reviews, we assessed whether they performed such a review. However, given the numerous issues we described previously regarding the evaluation of applications, including the potential for perceived evaluator bias, we believe that having the review committee determine whether these issues occurred after Rehabilitation awards the grant is too late in the process. Specifically, we believe that before Rehabilitation awards the grant it should designate staff, separate from those who are responsible for developing RFAs, scoring criteria, and selecting evaluators, to conduct such reviews. This oversight of the grant process will provide Rehabilitation with additional assurance that program staff and evaluators adhered to its grant process, and that it can demonstrate the process was followed as intended. Nevertheless, we found that the review committee did not always The review committee did not always review all relevant applications and generally did not conduct review all relevant applications comprehensive reviews of the evaluation and scoring process and generally did not conduct to identify procedural errors, evaluator prejudice, or whether comprehensive reviews of the evaluators supported their scores with evidence in the relevant evaluation and scoring process. applications, as shown in Table 7 on the following page. If the review committee identifies any of these issues, the grant manual allows it the option of rescoring the applications affected by the appeal and delivering a decision to the chief deputy within 30 days of the date of the appeal request. Although we identified procedural errors in all four grants we reviewed, the review committee identified errors in only two of the four grants. When the review committee identified these errors, it did not rescore the applications itself. Rather, for these two grants, the review committee required Rehabilitation to convene new evaluation panels to rescore the applications. Finally, state regulations require the review committee to notify the appellant in writing of its decision within 30 days of the date of the appeal request, and the grant manual suggests that the chief deputy notify all affected parties of the final decision as soon as practical. However, the review committee and the chief deputy did not always notify appellants or other applicants. As we discussed previously, the chief deputy was unfamiliar with the additional steps described in the grant manual for notifying applicants of appeals that may affect them. 48 Report 2017-129 | CALIFORNIA STATE AUDITOR July 2018 Table 7 The Review Committee Did Not Always Comply With State Regulations or Follow Best Practices When Reviewing and Responding to Appeals NAME OF GRANT AND YEAR AWARDED SYSTEMS OIB TBI OIB CHANGE PROCESS FOR REVIEWING AND RESPONDING TO APPEALS 2014 2015 2017 2017 Number of appeals 1 1 5 2 Review committee reviews evaluation and scoring of the appellant's  t t t application and the applications of the intended awardees Review committee adequately determines whether: Procedural errors or omissions occurred in the evaluation and X X t t scoring process X  t X Evidence exists that evaluator prejudice affected the scoring process  X t X Scoring is supported by evidence in the relevant applications If the review committee identified errors or omissions, it shall rescore X X t X the applications Review committee notifies appellant of its decision within 30 days of the X  t  date of the request, as required by state regulations Chief deputy notifies all intended awardees of the review committee's t X t t final appeal decision Source: California State Auditor’s review of the appeals process for selected grants awarded from fiscal years 2014–15 through 2017–18, state regulations, Rehabilitation’s grant manual, and the review committees’ decisions.  = Followed t = Partially followed X = Did not follow Further, although we found that the review committees sometimes contacted appellants or program staff for additional information regarding appeals, regulations do not specify a process for committees to do so. State regulations require appellants to clearly identify all issues in dispute, include in their appeal a full statement of their position with respect to each issue, and submit pertinent facts and reasons in support of the applicant’s position, but they do not provide a process for Rehabilitation to contact an appellant or program staff to obtain additional information that it deems necessary. Although we found that in some cases the review committee identified the need for additional information and allowed appellants to submit it, appellants may not always have the opportunity to clarify or further augment the assertions in their appeals if Rehabilitation does not specifically outline a process in regulation for the review committee to obtain additional information. Further, regulations also state that a review committee’s decision is final, so if it does not give an appellant an opportunity to clarify information submitted in its appeal, a review committee may make an irrevocable decision based on insufficient CALIFORNIA STATE AUDITOR | Report 2017-129 49 July 2018 information. By outlining a process in regulations for appellants to submit additional information or documentation the review committee deems necessary, Rehabilitation can better ensure that its review committees thoroughly address appellants’ concerns. If Rehabilitation does not believe it can accomplish its secondary review of additional information that appellants may submit or that program staff may provide within 30 days, it could consider extending the regulatory time frame to issue its appeal decision from 30 days to 45 days. As discussed throughout this report, Rehabilitation had patterns of errors and inconsistencies that occurred in the grant process, resulting in at least one applicant appealing its decision on each of the grants we reviewed. Although the appeals contained varying allegations, we identified some that were common to many. For example, five of the nine appeals cited discrepancies between the scoring criteria, application content, or evaluators’ scores and comments. Additionally, seven appeals alleged evaluator bias because evaluators had connections to the grant applicants, evaluation panels lacked representation from the disability community, or evaluators provided the appellant low scores. Finally, one appellant contended twice that Rehabilitation failed to follow its policies and procedures by not issuing public solicitations for evaluators. The following subsections describe specific details of the grant process and subsequent appeals for each of the four grants we reviewed. 2017 Systems Change Grant During two appeals of the 2017 Systems Change grant, Rehabilitation did not consistently follow the appeals process as described in the grant manual, which we identified as a best practice, and it did not rectify procedural errors before it awarded the grant. Figure 2 on page 40 illustrates the various evaluation panels, rounds of scoring, and appeals for the 2017 Systems Change grant. In May 2017, SCRS filed an appeal of Rehabilitation’s decision to award the grant to CFILC, alleging that Rehabilitation failed to provide appropriate guidance to the first evaluation panel, which resulted in identical comments, blank scores, and identical scores. It also alleged that Rehabilitation failed to award the grant to SCRS after the first evaluation panel identified SCRS as the grantee, that Rehabilitation provided CFILC with preferential treatment and selected biased evaluators for its second evaluation panel, and that Rehabilitation negligently and intentionally falsified scores in order to select evaluators who would score in favor of CFILC. Although the chief deputy appointed a review committee upon receipt of the first appeal, she did not notify SCRS of the review committee members’ qualifications, as suggested by the grant 50 Report 2017-129 | CALIFORNIA STATE AUDITOR July 2018 manual. Instead, one of the review committee members informed SCRS only of the names of the appointed review committee members and their respective positions. We believe it is important for Rehabilitation to inform the appellant of the review committee members’ qualifications to familiarize the appellant with those who are responsible for conducting a thorough evaluation of the appeal and possibly rescoring applications. In reviewing the first appeal of the 2017 Systems Change grant, the review committee found procedural errors, identified that Rehabilitation needed to rescore the applications, and recommended that Rehabilitation address the errors before it conducted the rescoring. It also found that score sheets used by the first two evaluation panels did not always align with the scoring criteria in the RFA and As a result of SCRS’ first appeal, the 2017 Systems the scoring benchmarks. The grant manual Change grant review committee made the indicates that when Rehabilitation develops an following four recommendations to Rehabilitation RFA, it should also create scoring benchmarks for conducting the rescoring of the applications: and score sheets for evaluators to use when reviewing applications against the scoring criteria 1. “Rehabilitation should provide adequate and accurate written guidance to Rehabilitation staff in advance to outlined in the RFA. In its decision, the review manage and coordinate the evaluation review process committee stated that based on Rehabilitation’s (e.g., timelines, panel selection, panel instructions, decision to have the first evaluation panel rescore scoring and rescoring criteria, and determination of the applications twice after the initial scoring, it award recipient). This is a recommendation because was clear that Rehabilitation found inconsistencies [Rehabilitation] asserted that the Grant Solicitation Manual in the evaluators’ scoring. Further, the review was a draft manual and had not been adopted and was committee stated that an evaluator had pointed not applicable to the RFA. out the discrepancies in an email to Rehabilitation 2. “Rehabilitation should provide adequate and accurate during the second panel’s evaluation. The written guidance to the panel of evaluators regarding review committee concluded that Rehabilitation the scoring process in advance to ensure clear evidently had not remedied the discrepancies in communication of expected scoring protocols (e.g., the scoring materials before giving them to the scoring is to be conducted independently, evaluators second evaluation panel. It stated that the score will be expected to rescore if there is more than a 3 point sheets, scoring benchmarks, and RFA form the spread or range of scores, evaluators will be providing a framework for fair evaluation of the applications, raw score and the coordinator will weigh the scores in and that the discrepancies between these accordance with the Benchmark). documents invalidated the scoring for both the 3. “Rehabilitation should ensure consistency in the scoring first and second evaluation panels. Further, the [components] amongst the Reviewer (scoring) Sheets, review committee compared the score sheets with the Benchmark, and the RFA. the final scores for the first and second evaluation 4. “Rehabilitation should ensure the content contained on the panels and, due to crossed‑out text and the lack of Reviewer (scoring) Sheets is accurately labeled to clearly a date or other identifier, was unable to discern the differentiate different rounds of scores, as appropriate. applicable scores submitted by the first evaluation Rehabilitation should also ensure the reviewers’ scores are panel for its second round of scoring versus its discernable (e.g., free from black‑outs and strike‑outs).” third round of scoring. The review committee Source: The review committee’s June 2017 decision letter to SCRS made four recommendations to Rehabilitation in in response to its appeal of the 2017 Systems Change grant award. advance of convening a third evaluation panel to rescore the applications, as shown in the text box. CALIFORNIA STATE AUDITOR | Report 2017-129 51 July 2018 Although the grant manual authorizes the review committee to determine whether there was evidence that evaluator prejudice affected the scoring process, and SCRS made such allegations in its appeal, the review committee did not make a determination for these allegations, stating that the scoring discrepancies it found were sufficient to set aside the award from both the first and second evaluation panels. However, in our review of the evaluation panels for the 2017 Systems Change grant, we identified that both evaluators appointed to the second evaluation panel had previous ties to CFILC, as we discussed previously. Therefore, because the review committee did not express an opinion as to whether evidence existed to indicate evaluator prejudice, it may not have completed the comprehensive level of review suggested in the grant manual, which we found to be reasonable and a best practice. We did note, however, that in its decision letter the review committee made a recommendation that Rehabilitation provide staff with written guidance in advance on how to manage and coordinate the evaluation review process, including guidance on panel selection. The committee stated that it made this recommendation because it believed the staff could benefit from such instructions. The third evaluation panel scored the applications yet again, with the result that CFILC was again awarded the 2017 Systems Change grant, followed by a second appeal from SCRS in December 2017. SCRS’ second appeal alleged that Rehabilitation did not publish a solicitation for evaluators according to its grant manual, failed to select unbiased evaluators, and neglected to implement the recommendations of the review committee— allegations that we found had some merit. We noted that in response to the review committee’s decision and recommendations after the first SCRS appeal, Rehabilitation did not issue additional written guidance to staff or evaluators. We also found that Rehabilitation did not amend the scoring benchmarks and score Rehabilitation did not amend the sheets used by the third evaluation panel to fully include two of the scoring benchmarks and score eight scoring components described in the RFA, even though sheets used by the third evaluation the review committee identified these errors in its decision. The panel to fully include two of the benchmarks and score sheets for one of these two components— eight scoring components described Organizational Experience—omitted the evaluation of whether the in the RFA, even though the review applicant demonstrated its expertise in and capacity to conduct committee identified these errors in an effective community organizing campaign to support issues its decision. of common interest to persons with disabilities that are related to community‑based living and implementation of the Olmstead Decision.6 The other scoring component—Assessment of Needs— failed to include an evaluation of the applicant’s understanding 6 In the Olmstead v. L.C. decision of June 1999, the U.S. Supreme Court ruled that given certain conditions, states are required to provide community-based treatment for persons with mental disabilities. 52 Report 2017-129 | CALIFORNIA STATE AUDITOR July 2018 of what is necessary for the future direction of the grantee in providing guidance for the Systems Change—a program to provide statewide collaboration of organizations and individuals to solicit participation, develop leaders, and build coalitions in the disability and independent living communities. Nevertheless, the review committee for the second appeal, made up of the same Rehabilitation employees who served on the review committee for the first appeal, not only upheld the decision to award the grant to CFILC, but also stated that Rehabilitation was not required to accept all of the review committee’s previous recommendations or adhere to the grant manual. We find this decision troubling because it contradicts the review committee’s own statement in the decision regarding SCRS’ first appeal that the RFA, scoring benchmarks, and score sheets form the framework for a fair evaluation of the applications, and its recommendation that Rehabilitation should ensure that these scoring materials are consistent. We question why the review We question why the review committee upheld the decision to committee upheld the decision to award the grant to CFILC rather than invalidate the scores when award the grant to CFILC rather Rehabilitation did not implement its recommendations. The review than invalidate the scores when committee made these recommendations in response to its Rehabilitation did not implement review of the issues SCRS provided in its first appeal. Further, given its recommendations. the issues the recommendations addressed—such as adequate and accurate written guidance for staff and evaluators about the grant process, and consistency in scoring components, score sheets, benchmarks, and the RFA—we would expect Rehabilitation to have addressed them before it convened a third evaluation panel. Because Rehabilitation did not rectify these issues, we would have expected the review committee to stand by its previous conclusions, absent evidence showing that the review committee’s previous conclusions were incorrect. Therefore, the review committee not only undermined its previous appeal decision, it also allowed the deficiencies it identified in the process in the first appeal to persist. Further, Rehabilitation’s failure to implement the review committee’s recommendations demonstrated its disregard for ensuring a fair and transparent process. We believe the review committee should have judged the grant process to be inadequate, once again invalidated the scores, and required Rehabilitation to restart the grant process. The grant manual suggests that, if a review committee identifies procedural errors that had a substantial effect on the outcome of the overall scoring, it should rescore the applications itself. Instead, when the review committee for SCRS’ first appeal identified such errors, it required Rehabilitation to convene a third evaluation panel to rescore the applications and, as mentioned earlier, did so without fully resolving the errors the review committee identified regarding the scoring materials. According to the members of the review CALIFORNIA STATE AUDITOR | Report 2017-129 53 July 2018 committee, they did not receive any written guidance on how to conduct an appeal review. They did explain that they referenced the grant manual for informational purposes but did not adhere to its guidance because the manual was a draft. When we asked the review committee members why they did not rescore the applications themselves, they stated that they believed that subject‑matter experts knowledgeable about the grant’s services would be better suited to conduct the rescoring. Although the grant manual does not currently provide an option to convene a new panel to rescore applications, we believe it is reasonable for Rehabilitation to do so when the review committee feels that subject‑matter expertise is necessary. Therefore, we recommend that Rehabilitation revise its process to provide this option. However, regardless of whether Rehabilitation must ensure that it the review committee or a new evaluation panel rescores the remedies procedural errors and other applications, Rehabilitation must ensure that it remedies procedural shortcomings the review committee errors and other shortcomings the review committee identifies in identifies in the grant process before the grant process before any rescoring takes place. any rescoring takes place. In its final decision letter in January 2018, the review committee provided an overview of the four rounds of scoring completed by the first two evaluation panels. We believe this overview mischaracterized the events leading up to SCRS’ appeals. For example, it identified that CFILC received the highest score in the first round, but failed to mention that part of the reason it scored higher was that, as Rehabilitation had identified, evaluators had incorrectly assigned a score of 0 to SCRS for one component. This error accounted for up to 45 points out of a possible total of 300, or 15 percent of the overall score. Because CFILC scored only about 37 points higher than SCRS, if evaluators had scored SCRS for this component, SCRS could have received the highest score for this round. In its decision letter, the review committee also stated that Rehabilitation discovered that, for the second round of scoring, there was more than a 3‑point difference between the highest and lowest scores given by each evaluator in four scoring components. In addition, the changes in the second round of scores were significant in that they resulted in SCRS, not CFILC, being the recipient of the award. Although the second round of scoring resulted in a new awardee, we noted that the total points awarded to each applicant did not change significantly, even though the evaluators’ rescoring correctly gave SCRS points in a category for which it had previously received 0 points. Specifically, CFILC’s total score decreased by about 6 points and SCRS’ total score increased by roughly 40 points out of a possible 300 points—with 25 of those points attributable to the component evaluators originally scored as 0. Finally, the review committee stated in its final decision letter that, given the dramatic total score changes between the second and third rounds, Rehabilitation chose to select a second evaluation panel to perform a fourth round of scoring. However, we found that from the second to third rounds, CFILC’s total score had decreased 54 Report 2017-129 | CALIFORNIA STATE AUDITOR July 2018 by 15 points and SCRS’ score had increased by roughly 10 points out of 300, or just about 7 percent and 5 percent, respectively. Therefore, we believe the review committee overstated the changes in scores when it referred to them as dramatic, and we question the reasonableness of Rehabilitation’s decision to convene the second evaluation panel to rescore the applications. The review committee did not fully Further, the review committee did not fully investigate SCRS’ investigate SCRS’ allegations that allegations that the evaluators for the 2017 Systems Change grant the evaluators for the 2017 Systems were biased. The review committee’s response to the second appeal Change grant were biased. stated that SCRS did not provide convincing evidence of any evaluator’s bias or a close personal relationship with a person who applied for the grant. However, in support of its allegation of bias, SCRS provided screen captures of the social media contacts of one of the evaluators, including evidence that she was “friends” with the former and current executive directors of CFILC and “liked” CFILC’s webpage. Further, SCRS also provided evidence that the evaluator was not “friends” with SCRS management and had not “liked” SCRS’ webpage. When we followed up with the review committee regarding its decision related to SCRS’ allegations of bias, the review committee stated that it had considered SCRS’ allegations regarding the evaluator’s social media, but that it believed SCRS’ evidence regarding this matter was insufficient to prove bias. However, we question why the review committee did not investigate the allegation, as the evidence SCRS provided creates at least the perception of bias or evaluator prejudice in the evaluation process. 2014 OIB Grant The 2014 OIB review committee did not identify a couple of key issues that occurred during the grant process, indicating that its review could have been more thorough. In particular, during the evaluation process for the grant, the second panel of evaluators used a single, combined score sheet, contrary to the grant manual’s direction that evaluators are to score applications individually. Given that this decision was a departure from the evaluation process specified in the grant manual, and that the review committee identified that the second panel of evaluators reached a consensus for each application component, we question why the review committee did not identify this as an error. Further, as we discussed previously, Rehabilitation assigned a program analyst to the second evaluation panel who had participated in the development of the RFA and helped administer the first evaluation of applications. When we attempted to follow up with the review committee to learn why it did not identify these concerns, Rehabilitation informed us that the only member of the committee was a retired annuitant that it no longer employs. CALIFORNIA STATE AUDITOR | Report 2017-129 55 July 2018 2015 TBI Grant We also found an error in the grant process for the 2015 TBI grant that the review committee overlooked. In its decision, the review committee indicated that its one member reviewed the evaluation process, including scoring, and laws governing conflicts of interest and concluded that there were no defects in the evaluation process, nor was there evidence of a conflict of interest. However, as described previously, we found that Rehabilitation accepted application documents for the grant after the hard deadline in the RFA from ILCSC—an applicant whose total score tied that of the Betty Clooney Foundation, the appellant. Had Rehabilitation followed its grant manual and adhered to both the hard deadline and the penalty for missing that deadline specified in the RFA, it should have disqualified ILCSC. We also found that evaluators did not consistently use their checklists to support their scores, which we believe is a procedural error. Had the review committee adequately reviewed the evaluation process and identified the errors we found, we would have expected it to reconsider the original grant award, and instead award the grant to a qualified applicant that had met the deadline. 2017 OIB Grant Rehabilitation received five appeals for the 2017 OIB grant, the most among the four grants we reviewed, as shown in Table 7 on page 48. For this grant, Rehabilitation had received applications from 23 entities—some of which applied for funding in more than one county. The appeals included a variety of allegations, such as evaluator bias, whether evaluation panel members were knowledgeable and qualified, that the evaluation process lacked clear and appropriate scoring criteria, and that evaluators’ scores were sometimes not justified. For instance, one appellant stated that Rehabilitation’s scoring criteria lacked clear and appropriate benchmarks and that when evaluators did provide comments they were cryptic and vague. The appellant also alleged that the evaluation panel lacked members with backgrounds in independent living and that statewide scoring of independent living applicants seemed to show bias against having these types of entities operating OIB grants. Another appellant questioned the evaluators’ scores on some sections of its application. The review committee found various shortcomings in the grant process and recommended that Rehabilitation convene a new evaluation panel to rescore certain components of the affected applications. Some of the issues that the review committee identified in the grant process included inconsistencies between the scoring criteria in the RFA and the benchmarks evaluators used to score the applications, and evaluators’ comments did not always support or align with their scores. 56 Report 2017-129 | CALIFORNIA STATE AUDITOR July 2018 As we stated earlier, while the grant manual does not currently provide an option to convene a new evaluation panel, we believe it is reasonable for Rehabilitation to do so in certain instances. In this instance, the review committee members did not believe they had sufficient time to rescore the applications and release their decision before the 30‑day deadline set forth in regulations. Therefore, they recommended that program staff convene a second evaluation panel to rescore the applications to ensure that they issued their decision within the required 30 days. As discussed previously, Rehabilitation could consider extending the time to issue its appeal decision from 30 days to 45 days, which would allow the additional time, if necessary, for the review committee to rescore applications, rather than issuing a decision to convene a new evaluation panel solely to meet the deadline to respond to the appeal. Recommendations To ensure that it consistently and thoroughly evaluates appeals, Rehabilitation should establish the following in state regulations and its grant manual: • Staff at the appropriate level of authority are to acknowledge all appeal requests, notify intended awardees that could be affected by the appeals, and inform the appellant of the qualifications of the review committee members. Staff at the appropriate level of authority must also notify all affected parties of the review committee’s final decision within the time frame Rehabilitation establishes in regulations. • A process for the review committees to request additional information from appellants or program staff. To allow time for an adequate review of any additional information, Rehabilitation should consider extending the time for review committees to issue their decisions on appeals from 30 days to 45 days. • To be able to rescore applications when necessary, the review committee members should be subject‑matter experts or, if they are not subject‑matter experts, the review committee should have the authority to recommend a new evaluation panel instead of rescoring applications itself when it identifies a reason to invalidate previous evaluations. To ensure that Rehabilitation has appropriate oversight of its grant process and can sufficiently demonstrate that it followed the process, it should designate staff, separate from those involved in the respective grant process, to conduct a review of each grant process for procedural errors, evaluator prejudice, and whether evaluators supported their scores with evidence from the relevant applications before it awards grants. CALIFORNIA STATE AUDITOR | Report 2017-129 57 July 2018 SCOPE AND METHODOLOGY The Audit Committee directed the California State Auditor to evaluate Rehabilitation’s application process for the grant solicitation program. Specifically, we were directed to review Rehabilitation’s policies and procedures for its grant application and review processes to determine whether they comply with relevant laws, rules, and regulations, as well as to review a selection of grants Rehabilitation awarded to determine whether it complied with its own policies and procedures. The Systems Change grant that Rehabilitation solicited in early 2017 was of particular concern to the requesters of the audit. The requesters’ key overall concern was whether Rehabilitation’s grant process is fair and transparent. Table 8 lists the objectives that the Audit Committee approved and summarizes the methods we used to address those objectives. Table 8 Audit Objectives and the Methods Used to Address Them AUDIT OBJECTIVE METHOD 1 Review and evaluate the laws, rules, and We reviewed relevant federal and state laws, rules, and regulations. regulations significant to the audit objectives. 2 Review Rehabilitation’s policies and • Obtained and reviewed Rehabilitation’s grant manual—its only documented procedures for its grant application and nonregulatory procedures regarding the grant process that were available from fiscal review processes and determine whether years 2014–15 through 2017–18. they comply with relevant laws, rules, • Compared the grant manual to relevant laws and regulations to determine whether and regulations. it complied. • Interviewed and obtained perspective from relevant Rehabilitation staff on inconsistencies or necessary improvements to the grant manual. continued on next page . . . 58 Report 2017-129 | CALIFORNIA STATE AUDITOR July 2018 AUDIT OBJECTIVE METHOD 3 For a selection of grants awarded during • Obtained a list of all grants Rehabilitation awarded and the associated applicants during the last three fiscal years, including the fiscal years 2014–15 through 2017–18, and, in addition to the 2017 Systems Change grant, 2017 Systems Change grant, assess selected three additional grants for review. the following: • In doing so, we considered factors such as the total dollar amount of the grants and the a. Whether Rehabilitation followed divisions that administered the grant. applicable policies and procedures during • For each of the four grants, we judgmentally selected four applications for review, with the the grant application and review processes exception of the Systems Change grant, which had only two applicants. and ensured that evaluators reviewing • Compared the legal requirements and Rehabilitation’s grant manual to its practices on each the applications did not have any conflicts grant, including RFA development and solicitation, selection and training of evaluators, of interest. evaluation of applications and awards, and handling of appeals, for each selected grant b. Rehabilitation’s grant application and and the selected applications. relevant documents to determine whether • Obtained and reviewed statements of economic interests and ethics training records there were any inconsistencies in the for state employees involved in each grant process to identify whether employees had review process. disclosed financial conflicts of interest and whether employees received required training. For the four grants we reviewed, we did not identify any financial conflicts of interest between Rehabilitation employees involved in the grant process and the grant applicants. • Obtained and reviewed signed conflict-of-interest and confidentiality forms and, to the extent possible, employment histories for nonstate-employee evaluators for each grant to determine if conflicts of interest, bias, or the perception of bias existed. • Obtained and reviewed email correspondence for key Rehabilitation staff involved in the Systems Change grant and certain other grants to identify any documentation of decisions and to determine whether inappropriate business activities existed. • Obtained and reviewed evaluators’ scores and comments for a selection of applicants for the four grants we reviewed. • Interviewed relevant staff to obtain their perspective on inconsistencies we identified in Rehabilitation’s grant process. 4 To the extent possible, assess Rehabilitation’s We addressed this objective by following the audit procedures we describe under Objective 3. review of the applications for the 2017 Systems Change grant, including the following: a. How Rehabilitation selected its panel of evaluators and whether this process was effective and appropriate. b. The evaluators’ consistency of assessments and comments regarding each applicant. c. Whether any of the evaluators had a conflict of interest. d. Its methodology of scoring applicants in each round. 5 Determine whether a more effective process • Identified two other state agencies that administer federal grants similar to Rehabilitation exists for awarding grants, including using through a competitive application process. staff from another agency to select the panel • Reviewed the selected state agencies’ RFAs and related documentation for selected federal of evaluators. grants to identify any best practices for the grant process. • Obtained and reviewed the two selected state agencies’ policies and procedures to identify any best practices for the grant process. • Reviewed the State Contracting Manual to identify any best practices. • In our review of policies and procedures and RFAs for the two other agencies and the State Contracting Manual, we did not identify as a best practice that awarding agencies use staff from other state agencies to select a panel of evaluators. We believe that if Rehabilitation fully implements our recommendations and adheres to its grant manual, using staff from other state agencies to select evaluators will not be necessary. CALIFORNIA STATE AUDITOR | Report 2017-129 59 July 2018 AUDIT OBJECTIVE METHOD 6 Review and assess any other issues that are • Identified appeals related to the four grants we reviewed. Analyzed Rehabilitation’s significant to the audit. response to the appeals and compared its practices to the regulations, and the grant manual’s optional procedures for appeals. • Identified whether Rehabilitation received any public records act requests from the applicants we selected to review, and determined whether Rehabilitation responded completely and within legally required time frames to these requests. Source: Analysis of the Audit Committee’s audit request number 2017-129, planning documents, and information and documentation identified in the table column titled Method. We conducted this audit under the authority vested in the California State Auditor by Section 8543 et seq. of the California Government Code and according to generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives specified in the Scope and Methodology section of the report. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Respectfully submitted, ELAINE M. HOWLE, CPA State Auditor Date: July 12, 2018 Staff: Laura G. Kearney, Audit Principal Karen Wells Jessica Derebenskiy Lauren A. Taylor, MPP Jasmine Zandian Legal Counsel: Mary Lundeen, Senior Staff Counsel For questions regarding the contents of this report, please contact Margarita Fernández, Chief of Public Affairs, at 916.445.0255. 60 Report 2017-129 | CALIFORNIA STATE AUDITOR July 2018 Blank page inserted for reproduction purposes only. CALIFORNIA STATE AUDITOR | Report 2017-129 61 July 2018