Board of State and Community Corrections

California Violence Intervention and Prevention (CalVIP) Grant Program: Final Evaluation of Cohort 3 Grantees March 29, 2024 Prepared by: Social Policy Research Associates, Inc. Christian Geckeler Lea Folsom Leela Hebbar Andrew Wiegand Prepared for: Board of State and Community Corrections 2590 Venture Oaks Way, Suite 200 Sacramento, CA 95833 Contract number: 60291 Final Evaluation Report Acknowledgements The authors of this report would like to thank staff members of the California Board of State and Community Corrections (BSCC) for providing leadership, input, and guidance. Thanks especially to Kasey Warmuth, Michael Martinez, and Katrina Jackson. Thank you as well to the third cohort of grantees of the California Violence Intervention and Prevention (CalVIP) Grant Program for providing data, contributing to interviews, and allowing us to work with you to conduct this evaluation. See Appendix A for a list of these grantees. This report would not be possible without your contributions. The authors would also like to thank other SPR staff members who contributed to the completion of this report, including Olivia Pham who was critical in the development of the grantee dashboards and mechanisms for storing, compiling, cleaning, and processing the grantee quarterly progress report data. We would also like to thank other staff who supported this project in numerous and varied ways, including: Ava Alvarez, Gabrielle Filip-Crawford, Ivette Gutierez, Krystal Hong, Heather Lewis-Charp, Ben Mahrer, Jessica Munoz, Oliver Stabbe, and Caleb Van Docto. This report does not necessarily reflect the official position of the BSCC. The BSCC makes no guarantees, warranties, or assurances of any kind, express or implied, with respect to the information in this report, including any information on linked sites and including, but not limited to, accuracy of the information or its completeness, timeliness, usefulness, adequacy, continued availability, or ownership. This product is copyrighted by Social Policy Research Associates, Inc. Internal use by an organization and/or personal use by an individual for noncommercial purposes is permissible. All other uses require the prior authorization of the copyright owner. Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report i Table of Contents Executive Summary .............................................................................................................. ES-1 I. Introduction .............................................................................................................................. 1 A. The CalVIP Grant Program and Its Goals ...................................................................... 1 B. The Cohort 3 CalVIP Grantees ........................................................................................ 3 C. Grant Timeline .................................................................................................................... 5 D. CalVIP Logic Model ........................................................................................................... 6 E. Approach to the Evaluation ..............................................................................................7 II. An Overview of Grantee Projects ....................................................................................... 9 A. Types of Services ................................................................................................................ 9 B. Projects and Service Tracks ............................................................................................ 11 C. Service Provider Partners ............................................................................................... 12 D. Examples of CalVIP Projects and the Delivery of Services ...................................... 13 1. Hospital-Based Intervention Services ....................................................................... 13 2. Street Outreach Services and Connecting to Participants .................................. 15 3. Increasing Participants’ Overall Wellbeing .............................................................. 15 4. K-12 Education and School-Based Services ............................................................ 17 E. Summary ............................................................................................................................. 18 III. Funding, Staffing, and Partnerships ................................................................................ 19 A. Cohort 3 CalVIP Funding and Expenditures ............................................................... 19 B. CalVIP Project Staffing ..................................................................................................... 21 C. Building and Strengthening Partnerships .................................................................. 22 D. Summary ............................................................................................................................ 24 IV. Enrollment and Participant Composition ..................................................................... 26 A. Outreach and Recruitment ............................................................................................ 26 B. Enrollment Numbers ....................................................................................................... 27 C. Participant Demographics ............................................................................................. 28 Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report ii D. Summary ............................................................................................................................. 31 V. Service Delivery .................................................................................................................... 32 A. Numbers and Types of Services Delivered ................................................................. 32 B. One-Time Services .......................................................................................................... 34 C. Strategies Used to Engage Participants ..................................................................... 35 D. Summary ............................................................................................................................ 37 VI. Exits and Outcomes ........................................................................................................... 39 A. CalVIP Project Exit Rates and Exited Participant Demographics ......................... 39 B. Participant Outcomes ..................................................................................................... 42 C. Summary ............................................................................................................................ 46 VII. Summary and Conclusion ................................................................................................ 48 A. Summary of Key Findings ............................................................................................... 48 1. Overview of Grantee Projects and Services ............................................................ 48 2. Funding, Staffing, and Partnerships ......................................................................... 49 3. Enrollment and Participant Composition ............................................................... 50 4. Service Delivery ............................................................................................................ 50 5. Exits and Outcomes ..................................................................................................... 52 B. Implications for Future Program Funding .................................................................. 52 C. Lessons for Future Research ......................................................................................... 55 References ................................................................................................................................ R-1 Appendix A: List of Cohort 3 CalVIP Grantees ................................................................. A-1 Appendix B: Evaluation Design Details .............................................................................. B-1 Appendix C: Quarterly Progress Report Guide ................................................................ C-1 Appendix D: Interview Protocol .......................................................................................... D-1 Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report ES-1 Executive Summary The Board of State and Community Corrections (BSCC) administers the California Violence Intervention and Prevention (CalVIP) grant program, which is a state- funded initiative designed to reduce violence in California communities that are disproportionately impacted by violence, including those with the highest rates of homicides, shootings, and aggravated assaults. CalVIP is intended to: 1) fund evidence-based practices for reducing violence in communities; 2) support individuals at the highest risk of either perpetrating or becoming victims of violence; and 3) build and nurture a network of public agencies and community- based organizations (CBOs) that support efforts to reduce violence. In 2020, the BSCC awarded $36,550,083 to 32 grantees as part of its third cohort of CalVIP grants. Grantees included: 14 CBOs, 12 medium- to large-sized city agencies; and six small city agencies in cities with a population of 200,000 or less.1 The projects these grantees implemented with their grants began on October 1, 2020, amid the COVID-19 pandemic. The grant term included two periods: 1) an 11- quarter program operating period (October 1, 2020 to June 30, 2023), in which grantees established their projects, enrolled participants, served and exited them, tracked outcomes, and engaged in reporting, and 2) a two-quarter administrative period (July 1, 2023 to December 31, 2023) in which grantees completed final reporting and financial audits, and finalized their local evaluations. A. Evaluation Approach In 2020, the BSCC also awarded a contract to Social Policy Research Associates (SPR) to conduct an implementation and outcomes evaluation of this third cohort of CalVIP grantees. The evaluation was designed to answer three research questions: 1) What were the characteristics of grantees and their CalVIP projects and to what extent were they able to achieve their project goals? 2) What factors supported or hindered project implementation? 3) What were the outputs and outcomes and how did they vary across projects? 1 A 13th city was awarded a grant but withdrew from the program after the first year and is not included in this evaluation’s findings. Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report ES-2 To answer these questions, the study team gathered and analyzed data from three main sources: 1) quarterly progress reports (QPRs) that the study team developed to standardize data across projects and which grantees completed for the 11 quarters of the operating period of the grant; 2) interviews with 16 purposefully selected grantees to learn additional information about project implementation; and 3) grantee planning documents and communications, including grantees’ proposals, local evaluation plans, local evaluation reports, and other communications used to clarify information about project reporting. B. Key Findings from the Evaluation • Grantees underspent grant funds, but were able to staff their projects, train their staff, and establish partnerships as expected. Grantees also developed strategies to help their service delivery partners better track data and to help law-enforcement and non-law enforcement partners work together more effectively. Managing these various project inputs was critical for grantees in implementing and operating their CalVIP projects. • Grantees exceeded their collective enrollment goal by nearly 50 percent, enrolling 9,523 people into CalVIP projects. They did so by engaging in direct outreach with potential participants and their families, building word- of-mouth referrals in their communities, and working with numerous and wide-ranging types of referral partner organizations. Participants tended to be Hispanic/Latino (52.5 percent) or black/African-American (32.1 percent), 24 years of age and younger (77.6 percent), and male (68.4 percent). • The 32 CalVIP grantees delivered 22 different types of services to thousands of CalVIP participants. Nearly all grantees (30 out of 32) provided case management services, delivering them to 27 percent of all participants. At least half of the grantees delivered another set of services that reached between 15 to 22 percent of participants (depending on the service) and included: engaging participants in terms they understood and responding to their trauma and needs, de-escalating tensions and interrupting violence, supporting participants (e.g., mentoring, supportive services), and teaching participants soft skills (e.g., conflict resolution, life skills). Grantees less frequently offered, and participants less frequently received, services involving hard skills training (e.g., employment, education) and the delivery of Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report ES-3 traditional social services (e.g., health, housing). In delivering their services, 12 grantees offered more than one service track—characterized by different enrollment goals, eligibility criteria, sets of services, and exit criteria—and partnered with an average of eight separate organizations to help provide services. At least two grantees also delivered “one-time” services to large numbers of people who were not formally enrolled in the CalVIP projects. • Grantees exited 5,441 (57.1 percent) of the project participants. Of these, 60.7 percent successfully completed projects while others left on their own or were asked to leave. The 42.9 percent of non-exited participants either stayed enrolled, through other funding grantees obtained, or may have been exited without grantees capturing these data. The demographic profile for exited participants closely resembles the profile for all participants. • Grantees reported that at least 21.1 percent of participants who exited achieved positive outcomes in one or more of the 27 standardized outcome categories. Grantees reported the highest rates of positive outcomes for: three measures of well-being having to do with participants feeling like they were better able to make positive life choices (21.1 percent), feeling safer in their communities (14.7 percent), or having an increased feeling of their basic needs being met (13.2 percent); two skills-based outcomes including having improved conflict resolution/anger management/violence avoidance/social emotional skills (18.8 percent) and improved life/healthy choices/family management skills (13.2 percent); and having reduced arrests and new charges (13.5 percent). Less frequently reported were outcomes related to health, education, housing, employment, and crime-related outcomes involving convictions and incarcerations. Participants may have achieved positive outcomes in multiple outcome categories. C. Discussion and Conclusion Overall, CalVIP grantees accomplished many of the goals of the program, including delivering evidence-based services to those at risk of violence and building a network of partners to do so. Grantees implemented new projects, produced many of the outputs and outcomes they intended, provided services, and observed outcomes for thousands of participants. Lessons learned from the evaluation, itself, include: 1) providing grantees with a standardized QPR template helped summarize Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report ES-4 a large amount of data in concise and relatively consistent ways, and 2) the local evaluations completed by each grantee helped each one better understand that evaluation was a critical aspect of the work, while also contributing knowledge to the field about each projects’ implementation, outputs, and outcomes. There are some areas that the BSCC may wish to consider for future research of violence intervention and prevention programs and future cohorts of CalVIP. These include: 1) examining whether grantees should target participants who have been more substantially involved in the justice system prior to enrolling in CalVIP; 2) considering the relative balance of services offered to participants and whether there are opportunities to increase the delivery of hard skills and more standard social services given the evidence that suggests these services can be useful in limiting justice system involvement, even for the younger people who formed the majority of CalVIP participants; 3) considering whether there should be more standardized project models or service tracks, and sets of services from which grantees can choose, which have clearer and more consistent standards that limit variability and allow for a greater exploration of the impact of certain models on specific types of outcomes; and 4) considering ways to allow for the collection of individual participant data (over aggregate data) given the ways it would allow researchers to link different types of data within projects and participation to outcomes. Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 1 I. Introduction The Board of State and Community Corrections (BSCC) administers the California Violence Intervention and Prevention (CalVIP) grant program, a state funded initiative designed to reduce violence in communities across California. Since 2017, the BSCC has funded four cohorts of CalVIP grantees, and in 2020, the BSCC awarded a contract to Social Policy Research Associates (SPR) to conduct an implementation and outcomes evaluation of the third cohort of CalVIP grantees. This report describes this evaluation. The report is organized into seven chapters. This introductory chapter provides the history and goals of the CalVIP grant program, an overview of funding and eligibility parameters for cohort 3 grants, and an overview of the evaluation. Chapter II provides an overview of the grantees and how their projects were designed to reduce violence in their communities. Chapter III describes the investments that grantees made in setting up and operating their projects, including staffing, training, and partnerships. Chapter IV describes participant recruitment and enrollment, and the demographic characteristics of participants. Chapter V describes the delivery of CalVIP funded services, including the numbers of participants who received different types of services. Chapter VI examines the numbers and demographic characteristics of participants who exited grantee projects and the outcomes that participants achieved. Finally, Chapter VII summarizes the findings from the report and discusses the implications for future cohorts of the CalVIP grant program as well as other violence intervention and prevention programs. A. The CalVIP Grant Program and Its Goals The CalVIP grant program began in Fiscal Year (FY) 2017-2018, replacing the California Gang Reduction, Intervention, and Prevention (CalGRIP) grant program which had been in operation since 2007. Both programs were designed to reduce violence within California cities disproportionately impacted by violence, but CalVIP adapted CalGRIP in several ways, including expanding eligibility to include community-based organizations (CBOs) in addition to cities. With a budget of $9 million per year, in FY 2017-2018 and FY 2018-2019, the state awarded funding to its first two CalVIP grantee cohorts. In FY 2019-2020, the legislature made two additional, substantial changes to the program. First, they provided a sizeable one- Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 2 time appropriation, boosting the annual allocation to $30 million. Second, the signing of Assembly Bill (AB) No. 1603, known as the Break the Cycle of Violence Act, codified the CalVIP grant program, establishing its goals, the distribution of funds, and the authority and duties of administering it to the BSCC. The goals of the CalVIP grant program, as outlined in AB 1603, are as follows. 1. CalVIP should fund evidence-based practices for reducing violence in communities. More specifically, funding is to be used by grantees to implement coordinated, community-based strategies for reducing violence in communities heavily impacted by that violence. As indicated in AB 1603, the purpose of CalVIP is to “improve public health and safety by supporting effective violence reduction initiatives in communities that are disproportionately impacted by violence, particularly group-member involved homicides, shootings, and aggravated assaults” (Assembly Bill No. 1603). The legislation also goes on to state that “CalVIP grants shall be used to support, expand and replicate evidence-based violence reduction initiatives, including without limitations, hospital-based violence intervention programs, evidence-based street outreach programs, and focused deterrence strategies, that seek to interrupt cycles of violence and retaliation…” (Assembly Bill No. 1603). 2. CalVIP funding should support people at the highest risk of either perpetrating or becoming victims of violence. In other words, programming need not target only those causing violence but those affected by it as well. Furthermore, the program should be focused on understanding the circumstances and locations in which people live, where violence is particularly acute, and striving to stem these causes of violence for both perpetrators and victims. 3. CalVIP grantees are to use funding to build and nurture a network of public agencies and CBOs that support efforts to reduce violence in communities. As noted above, CalVIP grantees include both “cities that are disproportionately impacted by violence” and “CBOs that serve the residents of those cities” (Assembly Bill No. 1603). Furthermore, the CalVIP grant program requires city grantees to “pass through” at least 50 percent of their grant funds to either CBOs or other city agencies dealing with community safety or violence prevention that are not law enforcement agencies. Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 3 In short, the current CalVIP grant program supports the growth of a broad range of organizations and governmental agencies, through direct and indirect funding, with the goal of reducing violence through wide-ranging, evidence-based services delivered to both potential perpetrators and potential victims. B. The Cohort 3 CalVIP Grantees Under this new mandate, the BSCC awarded its third cohort of CalVIP grants to: 14 CBOs, which included various non-profit, 501(c)(3) organizations; 13 agencies in medium- to large-sized cities; and six agencies in “small cities,” or cities with a population of 200,000 or less. These 33 grantees all operated within cities that were “disproportionately impacted by violence,” which was defined in AB 1603 as a city that met one of three criteria. • “The city experienced 20 or more homicides per calendar year during two or more of the three calendar years immediately preceding the grant application.” • “The city experienced 10 or more homicides per calendar year and a homicide rate that was at least 50 percent higher than the statewide homicide rate during two or more of the three calendar years immediately preceding the grant application.” • An applicant to the CalVIP grant program otherwise demonstrated “a unique and compelling need for additional resources to address the impact of homicide, shootings, and aggravated assaults” in their community, which was further clarified by the grant program’s Executive Steering Committee for cohort 3 to mean a city having experienced seven or more homicides per calendar year during two or more of the three years prior to the grant application and had a homicide rate that was at least 25 percent higher than the statewide homicide rate during two or more of the three calendar years immediately preceding the grant application. Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 4 Funding for cohort 3 included $35,050,000 in competitive grants and $3 million in funding earmarked for the City of Los Angeles2 for a total of $38,050,000.3 However, one grantee (City of Stockton) withdrew from the program within the first year, leaving the 32 grantees shown in Exhibit I-1, funded at a level of $36,550,083.4 Appendix A lists the 32 cohort 3 CalVIP grantees along with their project name and individual grant amount. Exhibit I-1: Cohort 3 CalVIP Grantees, Organized by Type of Grantee CBOs Cities Small Cities •Catholic Charities of the •Bakersfield •Grass Valley* Diocese of Oakland* •Fresno* •Greenfield* •Fresh Lifelines for Youth, Inc. •Long Beach •Gustine •Fresno County Economic •Los Angeles* •King City Opportunities Commission •Oakland •Marysville* •Kitchens for Good* •Oxnard* •Parlier •Los Angeles Brotherhood •Richmond* Crusade •Sacramento* •Lundquist Institute •Salinas* •Safe Passages (Advance •San Bernardino Peace)* •San Francisco •Sierra Health Foundation •San José Center* •Soledad Enrichment Action, Inc. •South Bay Community Services* •Southern California Crossroads •The Regents of UC Davis* •Watts Gang Task Force Council* •Youth ALIVE! * Indicates that the grantee participated in interviews, as is explained below in section E. 2 The California Budget Act of 2021-22 provided an additional $1 million for the City of Los Angeles reflected in a contract amendment that was fully executed in May 2023. 3 The FY 2019-2020 budget calls for the BSCC to set aside 5 percent ($1.5 million) for its own administrative operations. 4 The City of Stockton’s remaining grant funds were not re-distributed since grantees were already in their second year by the time funds were reconciled and relinquished from the city. Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 5 The locations of each of these grantees are shown in Exhibit I-2 and are also included in Appendix A. Exhibit I-2: Cohort 3 CalVIP Grantee Locations, Organized by Type of Grantee As will be described further in the next chapter, grantees adopted a wide range of different strategies and approaches to achieve the goals of the grant program, including hospital-based intervention services, street outreach and intervention services, school-based services, and services that taught participants soft-skills and provided them with supports designed to help them avoid violence. C. Grant Timeline The full cohort 3 CalVIP program lasted 39 months or 13 quarters. Grantees began operating their grants on October 1, 2020, amid the COVID-19 pandemic, and continued operating them up through December 31, 2023. Within this time frame, there were two distinct periods of activity: the operating period and the Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 6 administrative period. The operating period ran from October 1, 2020 to June 30, 2023, during which grantees established their projects, recruited and enrolled participants, delivered services, exited participants, and tracked participant outcomes. Grantees were also responsible for completing their quarterly progress reports (QPRs) for both BSCC oversight and the statewide evaluation during this period (see section E. for additional details on the QPRs). The administrative period ran from July 1, 2023 to December 31, 2023 (the last two quarters of the grant). In this period, grantees engaged in final reporting for the statewide evaluation, completed financial audits, and finalized their local evaluations. D. CalVIP Logic Model The CalVIP logic model presents a framework for identifying how inputs and investments of the CalVIP program led to the development and delivery of project services which, in turn, led to outputs and outcomes and the eventual desired community-level changes, namely the reduction of violence within grantee communities. The logic model shown in Exhibit I-3 synthesizes information initially gleaned from grantee documents which were then updated over the course of the evaluation based on other collected data. Each of the elements in this model is expanded upon in subsequent chapters, including the services in Chapter II, followed by the inputs in Chapter III, and the outputs and outcomes in Chapters IV, V, and VI. Exhibit I-3: CalVIP Logic Model Inputs Services Outputs Outcomes •Grantees (CBOs, cities, •Service design (service •Grantee-level outputs •Reduced criminal justice small cities) tracks, service delivery (partnerships system involvement partners) developed, services •Funding (funding/ •K-12/PSE educational provided, goals expenditures/pass- •Service categories enrollment/gains achieved) through amounts) (case management, •Improved psychosocial conflict resolution, •Enrollment (numbers, •Staffing (number, measures CBT, culturally demographics) structure, training) responsive services, •Improved soft-skills •Services delivered •Partners (referral mentoring, pro-social (types delivered and •Found/retained partners/service skills, life skills, numbers of employment delivery partners) substance abuse participants involved) treatment, vocational •Mental/physical health •Participants skills, work readiness, •Exits (numbers, gains (demographic profiles housing, health, etc.) demographics, reasons and level of risk) •Improved housing for exit) stablity Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 7 E. Approach to the Evaluation The statewide evaluation of the cohort 3 CalVIP program was designed to answer three main research questions, which are shown alongside a more detailed set of research questions in Exhibit I-4. Exhibit I-4: Primary and Detailed Research Questions for the Evaluation Primary Question Detailed Questions 1. What were the • Where were grantees and their projects located? characteristics of the • How were projects funded and how much did grantees spend in operating grantees and to what them? extent were grantees • What types of partnerships were involved? able to achieve their CalVIP project goals? • How did grantees manage and staff their projects? • How many and what types of people did grantee projects enroll and serve? • What types of services did they provide and how many people engaged in them? • How many participants exited and what outcomes did they achieve? • How did these outputs and outcomes align with grantees’ stated goals and objectives? 2. What factors supported • To what extent did partnerships, funding, staffing, training, and other or hindered project contextual factors, such as the COVID-19 pandemic, affect the implementation? implementation of grantee projects? • To what extent were grantees able to pivot or develop innovative strategies when approaches did not work as planned? 3. How did outputs and • How do outputs (i.e., enrollment, service delivery, and exits) and outcomes outcomes vary among vary by populations served and service delivery models? the different types of projects that were implemented? The answers to these research questions came from an analysis of the following three types of data. • Quarterly Progress Reports (QPRs). The study team used the QPR to capture information on grantees’ project activities, outputs, and outcomes over the 11 quarters of the grant operating period (from October 2020 through June 2023). The study team developed the QPR tool (see Appendix C) to standardize service types and outcomes for reporting purposes since these varied considerably across grantees. Grantees completed 10 QPRs, with the first QPR combining the first two quarters. Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 8 • Grantee interviews. To provide additional details around and examples of the features of grantee projects and the factors that supported or hindered their implementation, the study team interviewed staff from 16 purposefully selected grantees. The selection process identified grantees that were diverse in both type (e.g., CBO, city, small city) and location while also having features or reporting experiences, in their QPRs or in other communications, which seemed either innovative or of interest due to their alignment with grant goals. These 60-minute telephone interviews took place between March and April of 2023. • Grantee documents and communications. The study team also analyzed information about grantee projects captured in grantee proposals, grantee local evaluation plans and location evaluation reports (which all grantees produced under the terms of their grants), and other communications with grantees as part of providing support to grantees around data collection. Additional information on data collection and data limitations is in Appendix B. Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 9 II. An Overview of Grantee Projects This chapter provides an overview of the projects that cohort 3 CalVIP grantees implemented. It describes the types of services they delivered, the ways in which some grantees organized services into distinct service tracks, the partners grantees worked with to deliver services, and examples of several grantee projects. A. Types of Services As highlighted in the logic model shown in the prior chapter (Exhibit I-3), services were a critical component of CalVIP projects. These could be many and varied, and specific to each grantee. It was a strength of the program that grantees had the flexibility to design projects and services to meet the needs within their communities, rather than adhere to a one-size-fits-all model. This allowed grantees, who were closest to their communities and the most aware of their communities’ needs, to build projects designed to address those needs. In building these projects, and consistent with AB 1603, grantees also offered support for the services they planned to provide. In their proposals and QPRs, grantees described the ways in which the services were evidence-based, meaning that there was evidence in the research literature to support their use as a strategy for interrupting “cycles of violence and retaliation in order to reduce incidence of homicides, shootings, and aggravated assaults” (Assembly Bill No. 1603). These services did so either directly, by stopping violent activities, or indirectly, by helping participants become better able to commit fewer acts of violence or to take themselves out of harm’s way. For example, some grantees discussed in their planning documents and QPRs using evidence-informed screening tools to identify youth or adults most at-risk for violence (either as perpetrators or victims). Other grantees described delivering services or using curriculums or models such as: trauma-informed care, the Risk-Need-Responsivity Model (Andrews & Bonta, 2007), cognitive behavioral therapy (CBT), motivational interviewing (MI), dialectical behavior therapy (DBT), trauma restorative practices, including circle keeping and restorative inquiry, intensive or strengths-based case management, supportive services, mentoring, life skills classes, social emotional learning (SEL), Strengthening Families (Kumpfer & Magalhães, 2018), the Thinking for a Change curriculum (Bush et al., 1997), positive youth development models, CeaseFire (Skogan et al., 2008), and the Tiny-homes project (Scally et al., 2020). Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 10 Because the services they provided were many and varied, grantees were asked to categorize their services into one of 22 standardized types of services that the study team generated from the services described in grantee documents. Exhibit II- 1 lists these service types and indicates the number of grantees that provided each one. The number of services any one grantee provided ranged from five to 22, and on average grantees provided about 12 different types of services. Most grantees (30 of the 32) provided case management services while half (16) or more provided services that involved supporting participants (e.g., mentoring, supportive services, individual development planning), engaging them in terms they understood and responding to their trauma and needs (e.g., culturally responsive services, trauma informed care, motivational interviewing, mental health services), de- escalation/violence interruption services/intervention services, or teaching them soft skills (e.g., life skills training, conflict resolution training, pro-social skills, individual development planning). Less frequently provided were hard skills training (e.g., vocational skills training and educational services) and more standard social services (e.g., health, housing). Exhibit II-1: Number of Grantees Providing Each Type of CalVIP Service Number of Type of Service Grantees Case Management Services 30 Mentoring Services 25 Life skills/Healthy Choices/Family Management Services 25 Supportive Services (including any supports provided or tracked through the 23 project that aid participants in achieving other goals) Conflict Resolution/Anger Management/Violence Avoidance/Social Emotional 23 Learning (SEL) Training Pro-social/Leadership Skills and Activities 22 De-escalation/Violence Interruption/Intervention Services 22 Culturally Responsive/Culturally Competent Services 22 Individual Development Planning/Life, Education, or Career Planning 21 Health: Mental/Behavioral Services 20 Trauma-Informed/Restorative Justice Practices 19 Motivational Interviewing 18 Work/Career Readiness Training 16 Cognitive Behavioral Therapy (CBT) 14 Vocational Skills Training/Apprenticeships 13 Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 11 Health: Physical Health Services 12 Education: K-12/GED Services 12 Housing Services 10 Diversion Programs (out of justice system services) 8 Substance Abuse Treatment Services 7 Tattoo Removal Services 5 Education: Postsecondary Services 5 Source: Cohort 3 CalVIP QPR Data In addition to the services shown in Exhibit II-1, grantees made considerable use of the ability to “write-in” services in their QPRs (referred to in the QPR as “other” fields), sometimes using multiple write-in fields.5 Some examples of these write-in services include: education-adjacent services like after school programming or tutoring, victim services (especially around domestic violence, sexual assault, and human trafficking), immigration assistance, home visits, healing rituals, field trips to educational providers or employers, assistance with public benefits, funeral services, assistance in reconnecting with families, and various legal services. B. Projects and Service Tracks As noted in the CalVIP logic model (Exhibit I-3), projects included either single or multiple service delivery tracks with a track being defined as having distinct enrollment goals (also noting that participants could not be co-enrolled across service tracks), eligibility criteria, sets of services, exit goals, and definitions of successful completion.6 Collectively, the 32 cohort 3 CalVIP grantees delivered 73 different service tracks: 20 grantees (62.5 percent) operated a single service track and the remaining 12 grantees (37.5 percent) operated between two and 10 tracks. Appendix A lists the number of service tracks operated by each grantee and the box below provides examples of three grantees that operated multiple service tracks. 5 When data provided by grantees in write-in fields overlapped with the 22 types of services, grantees were asked to revise their data. Due to the timing of these revisions or the infrequent use of some of these write-in fields, it was not always possible or appropriate to combine them. 6 Section 3 of the QPR guide (Appendix C) shows the fields grantees reported on for each service track. Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 12 Examples of CalVIP Grantees the Operated Multiple Service Tracks Fresh Lifelines for Youth, a CBO grantee, operated two tracks: 1) the Law Program in which up to 200, 14- to 19-year-olds from both the schools and juvenile detention facilities participated in law-related education classes and coursework designed to build social-emotional learning skills, and 2) the Court-Appointed Friend and Advocate (CAFA) Mentor Program in which up to 60 similarly aged youth in juvenile courts worked with a mentor over a period of up to year, engaging in regular sessions and involving family visits. The City of Los Angeles operated 10 tracks, each serving different neighborhoods and with different community partners, as part of its Gang Reduction and Youth Development (GRYD) intervention services. While there were many tracks there were two main models. The first model aimed to enroll between 77 and 153 (depending on the neighborhood), 10- to 15-year-olds who were assessed as having elevated risk factors for joining a gang. These participants typically received case management, supportive services, and services to support the development of life and pro-social skills, among others. The second aimed to enroll between 60 and 87 (depending on the neighborhood), 14- to 25-year-olds identified as taggers or gang members. These youth received the same services noted above as well as substance abuse treatment services, tattoo removal services, conflict resolution training services, and mentoring, over three to six months. King City operated three tracks. Each track was a separate version of its Road to Success PLUS program, in which youth, meeting different eligibility criteria participated in an 8- to 10-week life skills course in a group setting. The three tracks included: 1) a K-12 educational system program that aimed to enroll 105, 12- to 17-year-olds in the K-12 system, with no criminal history but at risk of expulsion; 2) a Monterey County Juvenile Probation program that aimed to enroll 50, 14- to 19-year olds and their families; and 3) a community-based program operated through the King City Police Department that aimed to enroll 60, 14- to 17-year-olds who had committed a non- violent misdemeanor crimes and were not in the juvenile justice system. C. Service Provider Partners Consistent with the third goal of the CalVIP grant program (i.e., “to build and nurture a network of public agencies and CBOs that support efforts to reduce violence in communities”), grantees built out networks of service provider partner organizations to assist them in the delivery of the services described above, or, in some cases, the management and delivery of entire service tracks. Grantees reported having anywhere from one to 36 different service provider partners, with an average of 7.7 service provider partners each. As shown in Exhibit II-3, the most common types of service provider partners were CBOs/nonprofits (27). This was followed by other public agencies (14). Many grantees also partnered with probation/parole, K-12 education organizations, and health care providers. Less Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 13 than a quarter of grantees partnered with police/sheriff and higher education institutions and only one partnered with a workforce agency. Exhibit II-3: Number of Grantees Working with Different Types of Service Provider Partners r e CBO/Non-profit 27 n t r a Other Public Agency 14 P r e Other 14 d iv o Probation/Parole 10 r P e c K-12 Education 10 iv r e Health Care Provider 10 S f o e Police/Sheriff 8 p y T Higher Education 6 Workforce System Agency 1 Source: Cohort 3 CalVIP QPR Data D. Examples of CalVIP Projects and the Delivery of Services The following examples provide a sense of the services and service tracks that CalVIP grantees delivered as well as how they delivered them and to whom. The first two examples describe projects or tracks specifically identified in the CalVIP legislation, including hospital-based intervention services and street outreach and violence intervention and interruption services. The second two examples describe services or service tracks grantees developed in response to the needs of their communities, including projects designed to improve participants’ overall sense of well-being, and projects housed within K-12 schools, provided education-based services, and other K-12 school-based projects. 1. Hospital-Based Intervention Services Eight grantees (three cities and five CBOs) partnered with local medical centers to provide hospital-based intervention services, which research suggests may be beneficial in reducing violence in communities (Webster et al., 2022).7 These services support victims of community-based violence and are set up to help them 7 This meta-analysis finds neutral to positive results for these programs while acknowledging that additional research is needed, often due to their small number of participants. Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 14 recover and achieve non-violent lifestyles. At the bedside, or soon after being released from the hospital, hospital staff connected project staff with victims of trauma, gang-related incidents, or community violence. All eight hospital intervention projects provided participants with case management services and supportive services. Most also connected participants with peer-to-peer mentoring or peer support groups which enabled these participants to develop positive relationships with adults who shared similar lived experiences and could encourage and support them. Some of these projects connected participants to additional physical or mental health services to treat conditions such as post-traumatic stress disorder or depression while others had licensed mental health staff available on- site. Some helped participants develop individual life or career plans to help support them in leaving a gang lifestyle while others provided work readiness or education services to deter participants from a life of crime. Two specific examples of this type of project are described below. Soledad Enrichment Action, Inc. partnered with the City of Los Angeles and the University of Southern California Medical Centers to conduct their Caught in the Crossfire project, which offered a range of hospital and community-based services to youth and young adults ages 10 to 30 years of age. The project enabled trauma victims to plan for a positive, non-violent lifestyle while they were still in the hospital recovering from their injuries or just after their release. Soledad offered a range of services, including but not limited to case management, conflict resolution/anger management/violence avoidance/social-emotional learning training, housing, life and education planning, substance abuse treatment, tattoo removal, and work readiness training. Peer mentors and peer intervention specialists also supported participants and their families by promoting positive alternatives to violence. The City of San José operated the Trauma to Triumph Program in partnership with the San José Regional Medical Center, the Santa Clara Valley Medical Center, and several CBOs to identify hospital trauma center patients ages 12 to 30 who were victims of community-based and gang-related violence. The grantee viewed these traumatic incidents as an opportunity to help participants turn away from a gang lifestyle by providing case management intervention services, career and educational counseling, connections to K-12 and postsecondary education services, Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 15 and other supportive services. The case management support was provided by CBO staff who had shared experiences similar to those of their participants. 2. Street Outreach Services and Connecting to Participants Also mentioned in the CalVIP legislation are street outreach services, which seek to reduce violence and gang activity by enlisting staff to identify and connect with at- risk people at various locations within their communities. Street outreach services sometimes involve the delivery of violence intervention services, but also often involve guiding people into more focused services in safer settings. For instance, the Cities of Richmond, Fresno, and San Bernardino identified youth or adults who lived in low-income neighborhoods and were at high risk for gang recruitment and gang-related activities and other crimes, including violent crimes and drug use. Staff for these projects provided culturally competent services that aimed to build these community members’ trust and eventually enroll them into other CalVIP services (e.g., case management or mental health services). Other grantees addressed community safety by identifying youth participants and encouraging behavioral changes. For example, the City of Oxnard conducted presentations with hundreds of community members to provide information about diversion services and other community support to steer them away from gang or violent activity. The Cities of Greenfield and King City identified youth participants with misdemeanor offenses or youth on probation for diversion services. And the City of Oxnard and LA Brotherhood relied on referrals from justice partners and sought to connect youth with alternatives to violence and gang involvement by connecting them with de-escalation or violence interruption services, including anger management classes, tattoo removal, and other services, such as education, vocational training, or substance use treatment services. 3. Increasing Participants’ Overall Wellbeing Many grantees provided services that addressed their basic needs and taught them soft skills (e.g., conflict resolution skills or practices learned through cognitive behavioral therapies) designed to improve their ability to avoid compromising situations.8 What often unites these types of services is their ability to affect various 8 Two meta-analyses to consider are Papalia et al., 2020 and McGuire 2008. Both examine a wide range of soft-skills-focused interventions that show, to varying degrees and with varying populations, how these types of services can aid in reducing violence. Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 16 psychosocial measures of well-being. Delivery of these services often involved helping participants improve their lives by developing individualized service plans that outlined short-term goals (e.g., addressing trauma or violent behavior by taking part in counseling or anger management) or long-term goals (e.g., rebuilding family relationships or achieving a high school diploma). Many grantees that delivered these types of services also immersed participants in relationship- or community-building activities or exposed them to pro-social activities. Examples included traveling to a local gym or going on excursions to museums or theme parks. Another key service included life skills training, which addressed participants’ trauma and helped them develop effective communication and coping skills. Below are two examples of projects involving these types of services. Safe Passages/Advance Peace Stockton’s project served youth and young adults who were at the highest risk of perpetrating or being victims of gun violence. Youth entered the project through street outreach services provided by Advance Peace neighborhood change agents, referrals from the City of Stockton’s Office of Violence Prevention, and other partners. Each cohort of participants engaged in one or more types of supportive and planning services, including supportive services designed to address their basic needs and cognitive behavioral therapy, life skills training, mentoring, and trauma awareness services. Each participant created and worked their way through individualized life management plans (LifeMAPs) that outlined their short-term and long-term personal goals and steps for achieving them. Participants were also invited on excursions designed to improve their pro- social skills if they were actively participating in services, had completed a LifeMAP, and maintained healthy relationships with staff. Some of these events included taking college tours, meeting with government officials, attending conferences, or taking part in restorative justice dialogues. The City of Sacramento operated their Youth Peacemaker Fellowship Project, which sought to reduce gun violence by identifying youth, ages 12 to 17, who were at risk of becoming perpetrators and/or victims of gun violence. The project offered them 33 months of intensive case management as well as other services such as the creation of Life MAPS, engaging in group education for transformational opportunities, the delivery of conflict resolution skills training, and the delivery of pro-social and leadership opportunities. Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 17 4. K-12 Education and School-Based Services Several grantees offered K-12 education and school-based services. These services often focused on improving participants’ K-12 educational outcomes, such as improving school attendance, increasing enrollment or the number of credits received, or working towards GED or high school diploma attainment. Grantees that provided these services sought to create positive environments for children and youth and support the development of pro-social and life skills so that participants could focus during school. These projects also provided participants with vital supportive services needed to help them keep up with their education such as providing food, connecting them to physical or mental health services, or providing school supplies or clothing. Two examples are described below. The City of Gustine partnered with local law enforcement agencies and several schools in the Gustine Unified School District (GUSD) to create their Youth Services Bureau program. After seeing a large spike in youth arrest and behavioral issues after the pandemic, the city and its partners sought to target prevention services to all GUSD students and parents by conducting age-appropriate presentations related to justice system diversion, conducting community-oriented activities to foster community trust in the police and offer parent education. They also connected students with comprehensive trauma-informed and culturally responsive mental health counseling and support. King City and the City of Greenfield partnered to further community safety efforts and combat community youth violence in the Salinas Valley. Because neither city has a unified school district, these cities joined forces and partnered with local high school district leaders that governed their respective schools. They also partnered with Sun Street Centers, a CBO, several local educational agencies, and the local probation department to implement a comprehensive approach to addressing community safety through targeted youth engagement. Their Road to Success PLUS Program focused its efforts on addressing the behavioral change of youth at risk of, or involved with, local law enforcement or the juvenile justice system. Staff offered trauma-informed and culturally responsive evidence-based interventions and case management services to encourage youth to stay in school and avoid suspension, expulsion, and/or reincarceration. Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 18 E. Summary In summary, grantees operated complicated projects, involving multiple service tracks and service delivery partners. Grantees delivered 22 different types of services. Nearly all grantees provided case management services and half or more provided services that supported participants in other ways, including addressing their underlying traumas, teaching them soft skills, and engaging them to de- escalate or interrupt violent situations. Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 19 III. Funding, Staffing, and Partnerships To ensure that their CalVIP projects were operating efficiently, it was critical for grantees to effectively manage grant funds and establish and maintain project staffing and partnerships. A. Cohort 3 CalVIP Funding and Expenditures Grantees used their CalVIP funds to build new projects, or to adapt previously existing ones, expanding or altering services to reach new geographic areas, enroll additional participants, or offer new services. To operate their projects, grantees either spent grant funding themselves or “passed through” funding to partner organizations. As noted in the introduction, city grantees were required to pass through no less than 50 percent of their grant funds to CBOs in their communities or to other public agencies or departments that were not law enforcement agencies, but which were primarily dedicated to community safety and prevention. While they were not required to, CBOs could also pass through funds. As shown in Exhibit III-1, 11 out of 12 city grantees and all six small city grantees, as well as one CBO grantee, passed through funds to other organizations. The one city grantee that did not pass through funds was exempted from this requirement.9 Exhibit III-1: Numbers of Grantees Passing Through Grant Funds to Other Organizations for All Grantees and Each Grantee Type Number that Total Passed Through Grantee Type Number Funds CBO grantees 14 1 City grantees 12 11 Small city grantees 6 6 All Grantees 32 18 Source: Cohort 3 CalVIP QPR Data Exhibit III-2 shows the award amount and the amount grantees invoiced, both overall and for each type of grantee. The invoiced amount is then further broken down by the amount passed through to partners and the amount retained by the 9 This grantee met the exemption, spelled out in legislation, that if a city agency was also an agency primarily dedicated to community safety and prevention (e.g., an Office of Violence Prevention or Office of Neighborhood Safety), it was not required to pass through funds. Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 20 grantee. As can be seen in the Exhibit, the 32 CalVIP grantees spent $25,659,939, or 70.2 percent of the $36,550,083 they were awarded. This percentage is comparable across grantees by type: CBO grantees spent 70.2 percent of their funds, city grantees spent 69.7 percent of their funds, and small city grantees spent 72.9 percent of their funds. However, there is considerable variation in the rate of expenditures at the individual grantee-level, which can be seen in the individual grantee data provided in Appendix A. In terms of the 11 city and six small city grantees required to pass through funds (i.e., not including the one city grantee that was exempted from this requirement), 10 city grantees and three small city grantees passed through 50 percent or more of their grant funds, with many passing through close to all their grant funds. One city and three small city grantees fell somewhat or considerably short of this goal. Exhibit III-2: Cohort 3 CalVIP Funding and Invoiced Amounts for All Grantees and Grantees and by Type of Grantee $36,550,083 $25,659,939 $17,046,884 $16,503,199 65.5% $11,961,249 $11,510,824 31% 95% 34.5% 69% $3,000,000 $2,187,866 48.7% 5% 51.3% Grant award Invoiced Grant award Invoiced Grant award Invoiced Grant award Invoiced amount amount amount amount amount amount amount amount All grantees CBO grantees City grantees Small city grantees Invoiced: Passed Through Invoiced: Retained by grantee Source: Cohort 3 CalVIP BSCC Invoice Data Notes: Financial data are based on invoices submitted and reviewed by the BSCC up through February 22, 2024. At the time of this report, the BSCC was still reviewing invoices and waiting to receive updated information from grantees and so amounts could still change. Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 21 While it is clear from these data that grantees underspent grant funds, an important caveat is that expenditure data is based on invoices submitted and reviewed by the BSCC up through February 22, 2024. As of the writing of this report, the BSCC was still reviewing invoices and waiting to receive updated information from grantees. This updated information will likely result in some changes to invoiced amounts, including adjustments to the amount of funding passed through to partners. While these changes may lead to overall increases in expenditures, especially for some grantees, they are unlikely to result in substantial changes to grantee expenditures. There are many reasons why grantees underspent funds. However, the most prominent reason suggested in the data is that some grantees experienced management and implementation challenges, often due to both the COVID-19 pandemic and staff turnover issues (as is described below), at both start-up and on an ongoing basis, leading to delayed enrollment and launch of service delivery and thus the underspending of grant funds. B. CalVIP Project Staffing Staffing CalVIP projects involved two key sets of staff. • Staff who oversaw all aspects of grant and evaluation management, including: an authorized organizational officer, a project director, a financial officer, a fiscal day-to-day contact, and a project day-to-day contact. Some grantees named distinct staff members to each of these five roles and others, especially smaller grantees, tended to have fewer people in these roles, with each one playing more than one role. • Staff who delivered project services. Hiring or securing available staff for these roles was sometimes quite complicated, especially for projects with multiple service tracks or for grantees managing multiple service provider partners. For grantees that operated services directly, had fewer tracks, and fewer partners, staffing for service delivery was simpler. Grantees were generally able to staff their projects successfully. Within the first three quarters of the grant, over half (51.6 percent) of grantees had fully completed project staffing and nearly all grantees were able to complete this process by the end of the operating period, if not well before then. As some staff indicated in interviews, the opportunity to hire new staff improved the quality of the Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 22 programming they were able to offer. A respondent from one grantee explained how before CalVIP, their hospital-based intervention project was unable to serve a sizeable portion of injured youth during the weekends due to limited staffing, but cohort 3 funding enabled them to hire an additional staff member that allowed them to work with participants on the weekends. A few grantees experienced staffing challenges. The representative for one grantee explained that CalVIP was their organization’s first grant, and they did not sufficiently fund the project director position. An important lesson this respondent took away was the need to hire/appoint a full-time, dedicated staff person for project management. Interview respondents for other grantees noted how the COVID-19 pandemic put a particular strain on their ability to staff and operate their projects due to staff illness, increased turnover, and staff burnout (sometimes caused by the exposure to trauma inherent in this type of work). Grantees explained how this highlighted a need for them to support their staff in dealing with caregiver trauma by providing access to resources such as mental health services, yoga classes, and healing circles. Finally, related closely to the hiring of staff was the need to train them. In interviews, some grantees indicated that they trained their own staff and partner staff about their project models and service approaches while other grantees indicated that they required staff to be trained in specific roles and have specific certifications. For example, one grantee required its case management staff to possess nationally recognized case management credentials and mental health training. Another grantee trained its outreach staff on trauma-informed practices or cultural competencies to ensure they had the necessary skills to engage with participants on a personal level. While training was important, it took grantees some time to complete it. Nearly all grantees completed staff training by the end of the operating period, but only about one-third had completed it by the end of the third quarter of the project. C. Building and Strengthening Partnerships Consistent with the third goal of the grant, CalVIP grantees were to use their funds to build and nurture a network of public agencies and CBOs to support their efforts in reducing violence in grantee communities. Chapter II discussed the partnerships that grantees relied upon to deliver services, and the next chapter discusses Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 23 partnerships grantees used to aid in participant recruitment and enrollment. To further understand how grantees built and strengthened these partnerships, and to surface lessons for future funding cycles, the evaluation examined grantees’ processes for developing partnerships. According to QPR data, grantees were generally able to establish partnerships quickly and successfully. Within the first three quarters of the grant, over half (51.6 percent) of the grantees had fully completed partnership agreements, memorandums of understanding (MOUs), subcontracts, and other informal arrangements with their partners. Nearly all of the remaining grantees were able to complete this process over the rest of the operating period. One benefit of partnerships that grantee staff discussed in interviews was how much they valued partners as a source of staff who either had the specialized skills (e.g., therapists) needed to deliver specific types of services, skills in delivering culturally competent services, or life experiences similar to participants they planned to serve. These latter two were important for positioning projects to be able to relate to participants and build rapport. Grantee staff noted, however, that it could be an obstacle to implementing projects when such staff could not be replaced, when there were too few staff with these skills or experiences, or when there was a high rate of turnover among such staff. One challenge that some grantees encountered in working with partners was training or conveying to them the importance of quality data reporting. As one grantee staff member noted, these partners were not always “…used to making data-driven decisions…” and could “view it as a chore versus seeing it as valuable information…” making it “harder for them to understand the importance of daily inputting and the integrity of the data.” The main strategy grantees developed for overcoming this challenge was building stronger communication channels, including regularly scheduled meetings and impromptu communication involving texts, emails, and phone calls. As one grantee explained, these regular meetings helped both sides of the partnership understand their roles and services better. Some grantees also used these meetings to provide training or technical assistance on specific program models or to stress the importance of issues like data reporting, regular data tracking, and using data to make data-informed decisions. In terms of tracking service delivery and project participation, grantees also discussed the importance of having centralized recording systems, including Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 24 spreadsheets, such as Excel or Smartsheet, or various real-time databases like Airtable, Salesforce, Civacore, and Efforts to Outcomes. Many CalVIP projects also involved a partnership between law enforcement and non-law enforcement agencies (either CBOs or other types of city agencies). For three projects, law enforcement agencies were the grantees and non-law enforcement partners provided services (as discussed in Chapter II) or aided with referrals (as discussed in the next chapter). For the other projects, non-law enforcement agencies were the grantee and law enforcement agencies were either service provider or referral partners. In interviews, grantees shared that these partnerships could sometimes be challenging, especially when staff from law enforcement and non-law enforcement organizations had differing perspectives on the approaches to take with participants, or when participants had deep-seated mistrust of law enforcement. Grantee staff highlighted how staff members of non- law enforcement organizations and agencies were sometimes distrustful of law enforcement staff due to their focus on arrest and enforcement, and thus did not want to share certain information about participants because they were concerned about contributing to a participant’s involvement or re-involvement with the justice system. Solutions that grantees developed to help bridge these gaps included: 1) utilizing a third-party agency to facilitate conversations across agencies and organizations and to promote collaboration; 2) embedding a non-law enforcement staff member (e.g., a licensed clinical social worker) with a group of law enforcement staff to more effectively respond to and de-escalate emergency assistance calls (e.g., those concerning people experiencing homelessness) while also building rapport between staff; and 3) inviting law enforcement staff to interact with participants to create more positive and restorative perceptions of law enforcement among participants. D. Summary At the time of this report, grantees had spent 70.2 percent of their grant funds. This underspending likely reflects under-reporting due to close-out and reconciliation that was not complete at the time of this report combined with some level of genuine underspending due to various implementation challenges, especially those due to COVID-19. In terms of other inputs, grantees were largely able to staff their projects and establish partnerships as expected, and confronted and overcame partnership challenges involving data collection and participant tracking, and Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 25 cultural differences between law-enforcement partners and their staff and the staff of non-law enforcement organizations and agencies as well as participants. Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 26 IV. Enrollment and Participant Composition This chapter discusses the process by which CalVIP grantees recruited and enrolled participants and the results of those efforts, including the numbers of people they enrolled and the composition of the enrolled participants. A. Outreach and Recruitment Critical to enrolling participants were three broad outreach and recruitment approaches grantees used to make people aware of and become interested in CalVIP services. First, grantees and their service provider partners engaged in direct outreach with potential participants and members of their communities. For grantees that operated their own projects, these recruitment efforts were led by their own staff. When service provider partner organizations operated projects or service tracks, recruitment was typically handled by those organizations’ staff members. Outreach efforts included: advertising and direct messaging via social media and email networks, doing overview presentations at community events or at partner locations where potential participants and their families might regularly be in attendance (e.g., community centers or schools), distributing flyers or other materials within the community, leveraging the work of street outreach teams who often delivered other light touch services but likewise spread the word about projects, and getting the word out to various people within their communities through personal and professional networks. Second, grantees relied on word-of-mouth referrals, when a former or current participant, or one of their friends or family members, shared information about the project with friends or family who trusted them. About one-third of grantees reported in their QPRs (as a write-in category) that they relied on this approach. Others discussed this approach during grantee interviews. Other grantees may have relied on this approach as well but did not think to report it. Third, grantees developed and strengthened relationships with referral partners, or organizations that helped to identify and refer potential participants for enrollment. These types of organizations, shown in Exhibit IV-1, typically reviewed people they were already working with for CalVIP project eligibility. For example, grantees discussed in interviews how the criminal justice system agencies they worked with as referral partners (e.g., police departments and sheriff’s offices and probation departments and parole offices) could often review the people they were Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 27 working with in the courts, in juvenile probation camps, or who were on probation or parole, and identify people who would likely be eligible for their CalVIP projects. Overall, the use of referral partners in the recruitment process was important for most grantees. Grantees reported having between zero and 31 referral partners each, with an average of 7.7 referral partners per grantee. Exhibit IV-1: Numbers of Grantees Working with Different Types of Referral Partner Police/Sheriff 22 CBO/Non-profit 19 r e n t r a Probation/Parole 19 P la r r e K-12 Education 15 f e R f o Other Public Agency 11 e p y T Health Care Provider 9 Workforce System Agency 2 Source: Cohort 3 CalVIP QPR Data B. Enrollment Numbers Cohort 3 CalVIP grantees enrolled participants during all 11 quarters of the operating period of the grant, with enrollment peaking around quarter 7 of the operating period but staying relatively steady throughout. In total, grantees enrolled 9,523 people into CalVIP projects.10 They also re-enrolled a small number of these same people: 368 for a second time and 162 for three or more times. As shown in Exhibit IV-2, grantees exceeded their collective enrollment goal of 6,679 people by nearly 50 percent (142.6 percent).11 As with many other overall counts, 10 One grantee did not provide its final, quarter 11 QPR. This may have resulted in a small undercounting of participants enrolled and slight alterations in the demographic data. 11 This enrollment goal may somewhat undercount the true goal. The enrollment goal numbers were sometimes missing or included numbers that could not fully be verified by grantees. Nevertheless, even if the goal was considerably higher, grantees still would have met their goal. Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 28 there was some variation in this measure by grantee. Grantee level enrollment and enrollment goals are provided in Appendix A. Exhibit IV-2: Cohort 3 CalVIP Participants Enrolled Relative to Enrollment Goals Participants Enrolled 9,523 Enrollment Goal 6,679 Source: Cohort 3 CalVIP QPR Data C. Participant Demographics Exhibit IV-3 shows the composition of cohort 3 CalVIP participants across several different demographic categories.12 Exhibit IV-3: Demographic Data for Cohort 3 CalVIP Participants Race/Ethnicity Hispanic/Latino 52.5% Black/African American 32.1% White 4.8% Multiple Race or Ethnicity 5.9% Asian 1.2% Other 1.6% Native Hawaiian/Pacific Islander 0.6% American Indian/Alaska Native 0.1% Middle Eastern/North African 0.1% Declined to state 0.9% Age at Enrollment 12 and under 6.9% 13-17 47.8% 12 The percentages shown in the Exhibit are based on a denominator of participants for whom data were known. As is discussed in Appendix B and described in the Exhibit, grantees indicated that data were either unknown or not collected for a considerable number of participants with a rate that varied for each demographic category. To simplify the presentation, these unknown data were removed from these demographic calculations. Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 29 18-24 22.9% 25-34 12.1% 35 and older 10.4% Gender Male 68.4% Female 28.6% Other/non-binary/3rd gender 2.8% Prefer not to state 0.2% Participation Status Mandated 6.7% Voluntary 93.3% Point of Diversion/System Involvement No contact with law enforcement 64.3% Informal contact with law enforcement 15.1% Pre-adjudication 0.6% Post-adjudication, in community, no supervision 2.0% Post-adjudication, in community, under supervision 7.7% Post-adjudication, in-facility 10.3% Assessed Risk Group at Enrollment Low 9.8% Medium 14.1% Medium-high 19.1% High 42.2% Other 14.9% Education Status at Enrollment Enrolled in elementary or middle school 16.2% Enrolled in high school 49.6% Enrolled in training program 3.8% Enrolled in 2/4-year college 0.7% Not Enrolled/Highest completed: Elementary or middle school 2.0% Not Enrolled/Highest completed: Some high school 10.9% Not Enrolled/Highest completed: High school diploma 11.9% Not Enrolled/Highest completed: GED 2.4% Not Enrolled/Highest completed: Some college or vocational training 1.9% Not Enrolled/Highest completed: Certificate or college degree 0.4% Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 30 Employment Status at Enrollment Employed: Not looking for other/additional employment 12.9% Employed: looking for other/additional employment 4.4% Unemployed: Looking for employment 26.8% Unemployed: Not looking for employment/student 48.1% Unemployed: Not looking for employment/other reason 7.7% Housing Status at Enrollment Living independently (own or rent) 9.3% Living with parent/s 63.7% Living with relatives (not foster care) 9.7% Living with a friend 2.7% Living in foster care 1.8% Living in out-of-home care through child welfare or probation 2.2% Living in unstable housing (car, street, abandoned building, tent, etc.) 6.7% Other 4.0% Source: Cohort 3 CalVIP QPR Data Notes: Percentages in this table were calculated using data on only the enrolled participants for whom demographic data were known. Of the 9,523 total enrolled participants, demographic data were unknown or not collected for between 15.3 percent for age and 51.3 percent for employment status. See Appendix B for additional details on this limitation to the data. These data suggest the following about the composition of CalVIP participants. • Most participants were either Hispanic/Latino (52.5 percent) or Black/African American (32.1 percent), and 68.4 percent were male. This composition of race and gender is not surprising based on demographic data provided by grantees in their proposals for the communities they served (including data on cities, specific neighborhoods they served, and referral partners with whom they worked). There is also the fact that these groups are over- represented within the justice system (Hetey & Eberhardt, 2018; Lofstrom et al., 2024), which is both a referral source for projects and reflects the people grantees want to work with so that they can keep them from entering the justice system in the first place. • Over three-quarters of participants (77.6 percent) were children, youth, and young adults: 47.8 percent were between the ages 13 and 17, 22.9 percent were between 18 and 24, and 6.9 percent were 12 years of age or younger. Given these age ranges, it is also not surprising to see that 65.8 percent of Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 31 participants were enrolled in elementary, middle, or high school, that 63.7 percent indicated that their housing status was living with parents, and relatively few participants were employed. • Grantees addressed the mandate of the program to serve people at the highest levels of risk. About three-quarters of participants were assessed by grantees as being medium- to high-risk of perpetrating or being a victim of violence: 42.2 percent were assessed as high risk, 19.1 percent were assessed as medium-high risk, and 14.1 percent were assessed as medium risk. • About two-thirds of participants (64.3 percent) had no contact with the justice system. While this percentage seems high, given the general focus of the program on reducing violence, it is likely due to the fact that almost half of CalVIP participants were 17 or younger, and many projects were focused on preventing them from becoming perpetrators or victims of violence, and thereby preventing them from becoming involved in or having contact with the justice system in the first place. • Other notable characteristics are that participants voluntarily participated in CalVIP projects (93.3 percent), and they were frequently unemployed (82.6 percent), with many (48.1 percent) not looking for work. Both are consistent with earlier findings. The former, i.e., that they were not mandated to participate, is consistent with participants’ more limited level of involvement in the justice system, where such mandates typically originate. The latter is, again, consistent with the younger age span of the participants. D. Summary Grantees exceeded their collective enrollment goal by nearly 50 percent, enrolling 9,523 people into CalVIP projects. They were able to achieve this through three main types of outreach and recruitment, including direct outreach with potential participants, building word-of-mouth referrals, and working with referral partners. Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 32 V. Service Delivery This chapter discusses CalVIP service delivery, including the numbers of participants who received each type of service, special one-time services, and the strategies grantees used to keep participants engaged in services. A. Numbers and Types of Services Delivered Grantees recorded the number of people who participated in each of the 22 types of services included in the QPR. As already discussed in Chapter II, grantees provided all 22 types of services. Exhibit V-1 shows the numbers of participants receiving each type of service and the percentage of these participants out of the total number enrolled (9,523).13 Exhibit V-1: Numbers and Percentages of CalVIP Participants Receiving Each Service Type Case Management 2,571 (27.0%) Culturally Responsive/Culturally Competent Srvs 2,093 (22.0%) De-escalation/Violence Interruption/Interv Srvs 2,084 (21.9%) Mentoring 2,005 (21.1%) Supportive Services* 2,001 (21.0%) Conflict Res/Anger Mgmnt/Violence Avoid/SEL Trng 1,962 (20.6%) Trauma-Informed/Restorative Justice Practices 1,961 (20.6%) Pro-social/Leadership Skills 1,824 (19.2%) Motivational Interviewing 1,462 (15.4%) Cognitive Behavioral Therapy 1,458 (15.3%) Life skills/Healthy Choices/Family Mgmnt 1,451 (15.2%) Ind Dev Planning/Life, Ed, or Career Planning 1,129 (11.9%) Health: Mental/Behavioral 925 (9.7%) Work/Career Readiness Trng 793 (8.3%) Education: K-12/GED 708 (7.4%) Health: Physical Health 585 (6.1%) Voc Skills Trng/Apprenticeships 546 (5.7%) Diversion 491 (5.2%) Housing 196 (2.1%) Substance Abuse Treatment 191 (2.0%) Tattoo Removal 68 (0.7%) Education: Postsecondary 67 (0.7%) 13 As noted in Chapter IV, one grantee did not provide its final, quarter 11 QPR. This may have resulted in a small amount of under-reporting of service receipt data. Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 33 Source: Cohort 3 CalVIP QPR Data Notes: The Exhibit shows the number of unique people who participated in each type of service and the percentage of these people out of the number of participants (9,523). Since participants could receive more than one type of service, the same participants may appear more than once in the Exhibit. Thus, the sum of all participants shown receiving services is more than the total number participants enrolled. Based on the data presented in Exhibit V-1 and some additional QPR data, the following are some notable features of project service delivery. • The number of grantees providing a type of service (see Exhibit II-1) was mostly ranked in the same order as the number of participants who received that type of service. In other words, the services provided to the most participants were typically also offered by the most grantees and the reverse was also true. • Case management was the most widely delivered service, with about a quarter (27.0 percent) of all participants receiving it. Consistent with the point made in the previous bullet, case management was also the most widely provided service, with 30 of the 32 grantees providing it. • The services delivered to at least half or more of the participants receiving case management (i.e., the next 10 services shown in the Exhibit which were delivered to 15.2 to 22 percent of participants), included services that involved engaging participants in terms they understood and responding to their trauma and needs (e.g., culturally responsive services, trauma informed care, motivational interviewing, and cognitive behavioral therapy), de- escalation/violence interruption services/intervention services, supporting participants (e.g., mentoring, supportive services), and teaching participants soft skills (e.g., conflict resolution training, pro-social skills, life skills training). • The services provided to the fewest participants (between .7 and 11.9 percent) frequently involved the delivery of hard skills and traditional services like educational services, vocational training services, substance use treatment services, mental and physical health services, and housing services. While grantees did not provide any reasons for providing these services less often than others—nor was there any requirement to do things differently—one reason may have been that grantees initially were concerned that these services were too costly or might require even greater partner investment. It also may have been the case that these services simply did not reflect the Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 34 needs of their participants whether due to their, on average, younger age, or for other reasons. • On average, participants received 3.1 different services.14 B. One-Time Services Two grantees provided certain types of “one-time” services that were unlike other service tracks in that they only engaged people at a single point in time and did not enroll them in a CalVIP project. At the same time, these services reached large numbers of people and served as a way for grantees to further recruit people that they might later enroll. Fresno Police Department managed eight service tracks, mostly led by various service delivery partners. Two of these tracks, delivered by a single partner, consisted of these one-time services, which involved training programs, developed in multiple languages (English, Spanish, and Punjabi), and which were designed to educate and increase awareness around issues of domestic violence and human trafficking. The first track involved hour-long trainings held for youth, their families, and staff at various organizations and providers in Fresno County, including consulates, universities, social service providers, schools, and workforce system agencies, sometimes in person and sometimes remote. The grantee’s subcontracted service provider estimated that that they reached as many as 3,000 people for these trainings. The other track involved disseminating information from these trainings through an interview format on stations like Univision, which the grantee estimated to have reached 18,000 viewers. Youth Alive! operated seven service tracks, one of which involved a violence interrupter team that conducted mediations in the community, resolving disputes between parties. These staff members, who were often state-certified prevention workers, were trained in various techniques designed to help them understand trauma and its symptoms, mediate conflicts, and de-escalate violent situations. Through this service, the grantee estimated that it reached approximately 1,200 people through street outreach mediations. 14 The number of services any one person may have received could have been significantly higher or lower than this average. However, the aggregate nature of the data does not allow the data to be parsed in a way that shows this range. For instance, the same 2,571 people who received case management could have also received all the other services. Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 35 C. Strategies Used to Engage Participants During grantee interviews, project staff noted two major obstacles in implementing services and in keeping participants engaged. They also noted several strategies for overcoming these obstacles. The first obstacle was the pandemic which was about six months in at the start of the cohort 3 CalVIP grant period. Staff from at least seven grantees noted how COVID-19 delayed the delivery of planned programming while they adjusted their procedures to comply with new social distancing requirements, adapted programming to carry out remote services, and experienced delays with partner referrals and supportive services. In some cases, grantees also needed to adapt their originally planned set of services to address more immediate needs of their participants, such as providing food or other basic services. They also had to adapt in other ways such as delivering supplies that participants would need (e.g., art supplies or workbooks) when meeting with staff virtually or by sending staff into the field to locate participants wherever they were. Interview respondents spoke appreciatively of the BSCC for its flexibility in making these adaptations. Most grantees pivoted to virtual programming, but many grantees described how engaging participants virtually could be challenging compared to in-person services because it was difficult to build rapport, and as the pandemic continued, participants and staff became increasingly fatigued by the challenges of virtual interactions. Interview respondents also identified how the pandemic highlighted digital literacy needs and inequities among participants. Staff at six grantees, for example, described in their interviews how many of their participants lacked working computers or the stable internet connections needed for virtual programming. The second obstacle to delivering services that grantee staff shared during interviews was how the trauma faced by many of their participants often impeded their ability to avoid violent situations and to complete CalVIP programming. These staff members wanted to emphasize how personal or community violence disproportionately impacted CalVIP participants and how participants often lacked effective coping methods and were thus experiencing profound trauma or personal crises. Moreover, respondents shared that their participants often lacked adequate support or access to critical needs like housing, transportation, substance abuse treatment, mental health care, a stable income, or even trusting and supportive Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 36 adults or caregivers. While many of these conditions are to be expected given the purpose of the grant and the populations they were designed to serve, staff members wanted to emphasize just how much of a barrier these conditions could be, especially when it came to staying engaged with a project over time, and just how difficult they could be to overcome. Grantees engaged in several practices for overcoming these challenges. First, in terms of addressing the needs of participants, as already discussed in Chapter II, grantees built their projects to deliver services based on evidence-based practices, thus making sure they could address the particular needs of participants, especially when trauma was a barrier to service delivery. Second, grantees discussed in interviews the importance of building rapport to help connect with participants and build their trust. Some discussed the need to provide participants with a trusting, reliable, caring adult, which involved showing up for participants, creating a safe and comfortable space, and being caring and supportive of them to build consistency and trust. Another strategy grantee staff members highlighted in interviews that builds upon this idea was hiring or engaging staff members with lived experiences similar to those of their participant populations. Doing so established credibility with participants since they knew that staff had experienced and overcame similar situations in their own lives. A staff member for one grantee explained that sharing meals with participants can be a “powerful tool for building rapport when working with youth.” Sharing food helped their participants create connections, build trust, and better engage. “They come with distrust, abandonment issues, and anybody who’s ever done something for them has had ulterior motives….What we are doing is we’re creating a family for them so that they know with the support, care, love, and resources – they can accomplish anything they want….I am going to give you life skills, I am going to give an allowance, I am going to give you discipline – I want to make sure you are involved in whatever you want to be in. We created that type of environment for these young people who have, for the most part, never felt any of those things.” –Grantee Staff Member Third, grantees used one of many other approaches for connecting with participants. A staff member for one grantee discussed in an interview how Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 37 preventing school expulsion, having students’ criminal justice records cleared, or working with the truancy court or probation officers on their behalf were all big motivators for the youth with whom they worked. Some grantee staff members also described how engaging the whole family ensured that their youth were successful by helping to provide a support network for that participant to stay engaged in services. One grantee further found that soliciting caregiver feedback on service delivery helped them better engage less involved or initially hesitant caregivers, thus learning from the process. Staff members from other grantees discussed how providing participants with coping mechanisms (e.g., the use of sensory/tactile items) was an effective tool to help them manage their traumatic stress or anxiety and keep them from becoming too distracted. While many of these services are important on their own, staff wanted to point out how they had the added benefit of keeping participants engaged so that they could receive additional services. Finally, grantee staff members shared how offering various types of incentives emerged as a promising practice when it came to keeping participants engaged. In some cases, these involved financial incentives. Some examples of financial incentives that grantee staff members discussed included: monthly financial allowances made as participants progressed through their individual service plans, daily or weekly stipends to encourage attendance at apprenticeships or internships designed to offer training and job experience, and targeted incentives (e.g., gift cards) designed to encourage completion of services such as mental health or individual counseling. That said, a staff member for one grantee explained that incentives do not necessarily need to be cash to be effective. For example, their participants could be motivated to continue services for access to additional resources, such as tattoo removal, or by offering trips, such as a field trip to the zoo after a period of completing services. D. Summary Grantees delivered services to thousands of participants, with each participant receiving about three services, on average. Twenty-seven percent of participants received case management services, and between 15 and 22 percent of participants received services that used specific delivery approaches, such as trauma- or culturally-informed care, in responding to their trauma and needs, de-escalating tensions and interrupting violence, supporting participants, and teaching participants soft skills. Fewer participants (between 1 and 12 percent) received more Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 38 standard social services or hard skills training services. A few grantees also provided one-time services to large groups of people who did not enroll (e.g., public information campaigns or community events). The evaluation also surfaced several strategies that grantees used to keep participants engaged, including developing evidenced-based services for dealing with trauma, building rapport by meeting with participants to build their trust, enlisting staff with lived experience, connecting with participants by teaching them coping skills and connecting with their families, and providing incentives to encourage the completion of services. Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 39 VI. Exits and Outcomes This chapter examines the participants who exited from CalVIP projects and their outcomes and discusses these findings. A. CalVIP Project Exit Rates and Exited Participant Demographics Over the operating period of the grant (October 2020 through June 2023) grantees reported that 5,441 out of 9,523 total participants (57.1 percent) exited CalVIP projects for the first time. During this same period, grantees reported that a small number of participants exited more than one time: 102 participants exited a second time and 139 participants exited three or more times.15 Exhibit VI-1 shows the reasons grantees indicated for a participant’s (first time) exit. Of the 5,441 participants who exited, 60.7 percent (2,825) completed their projects successfully, with each grantee defining successful completion somewhat differently for each service track that it operated.16 While the definition connected to each service track or project varied, some of the more common ways grantees defined successful completion included: completing a pre-defined number of classes or program hours; completing the goals they established or addressing identified needs as per an individual development plan; having been assessed as a lower level of risk of justice involvement than they were upon entering; avoiding initial or further criminal justice system involvement or victimization for a set period of time; and improving a rating on some type of assessment delivered previously. Exhibit VI-1 also shows the other reasons participants exited CalVIP projects. The other, more common reasons included 22.4 percent (1,043) who either dropped out or were participants with whom the project lost contact and 9.8 percent (455) for whom grantees indicated there was some other (unstated) reason for their exit. Far less frequent were participants who exited because the grantee asked them to leave for non-compliance (3.1 percent), they were arrested or incarcerated (2.1 percent), 15 As noted in Chapter IV, one grantee did not provide its final, quarter 11 QPR. This may have resulted in a small amount of under-reporting of exit and outcome data. 16 786 participants (14.4 percent of all participants who exited) had “unknown/did not collect” as an exit reason. Following the approach introduced in Chapter IV and discussed in Appendix B, these participants were removed from the analysis of exit reasons, meaning that percentages shown in Exhibit VI-1 are based on 4,655 participants rather than the full 5,441 who exited. Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 40 or the grantees removed them due to a determination that services were not appropriate for the participant (2.0 percent). Exhibit VI-1: Percentage of Exited Participants with Each Exit Reason Successful completion 60.7% Dropped out/lost contact 22.4% n o Non-compliant (asked to leave) 3.1% s a e R t ix Arrest/incarceration 2.1% E Services determined not appropriate 2.0% Other 9.8% Source: Cohort 3 CalVIP QPR Data Notes: Although 5,441 participants exited projects, the denominator used for calculating percentages was 4,655, after removing the 786 participants for whom the exit reason was “unknown/did not collect.” See Appendix B for additional details on this limitation to the data. Conversely, 4,082 (42.9 percent) had not exited their CalVIP projects by the end of the grant operating period. At the grantee level, the rate of exit ranged from one grantee that did not exit any participants to six that exited all their participants. Anecdotal information gathered from grantees as part of the final reporting process, along with interview data, suggest two possible explanations for what happened with participants who did not exit. 1. Grantees continued to serve at least some participants under other funding. While cohort 3 CalVIP grants were critical to supporting the work of these grantees, many of the projects that grantees operated already existed in some form before cohort 3 CalVIP funding and continued after, sometimes with alterations to the model. Prior or subsequent funding may have involved grants from other cohorts of CalVIP (since some cohort 3 grantees were awarded cohort 4 grants) or come from other sources. 2. Some grantees may have lacked the capacity to collect better exit data and thus underreported exits that actually occurred. As discussed previously, some Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 41 grantees struggled to have their own service delivery partners collect quality data. Several grantees had to revise data provided to them by their subcontracted partners as they discovered issues (as pointed out in the evaluator’s data quality review). Some grantees, and their sub-contracted service delivery partners, may therefore have been unable to track some participant information and were unable to provide better exit data. In terms of the participants who exited CalVIP projects and for whom data was known, the study collected race/ethnicity, age, and gender data.17 Exhibit VI-2 shows these data. The composition of participants exiting cohort 3 CalVIP projects is similar to those who enrolled (see Exhibit IV-3) and many of the findings discussed around the composition of participants in Chapter IV mirrors what is shown below for participants who exited. This need not be the case given that 57.1 percent of participants exited. Exhibit VI-2: Demographics for All Exited Participants Race/Ethnicity Hispanic/Latino 55.3% Black/African American 28.5% White 4.4% Multiple Race 7.3% Asian 1.4% Other 1.8% Native Hawaiian/Pacific Islander 0.7% American Indian/Alaska native 0.2% Middle Eastern/North African 0.1% Declined to state/Prefer not to state 0.3% Age at Enrollment* 12 and under 4.7% 13-17 54.0% 18-24 24.0% 25-34 9.6% 35 and older 7.8% Gender 17 See Appendix B for a discussion and details around data that was unknown/not collected. Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 42 Male 67.9% Female 31.4% Other/Non-binary/3rd gender 0.5% Prefer not to state 0.3% Source: Cohort 3 CalVIP QPR Data Notes: Percentages in this table are calculated using only data on exited participants for whom demographic data were known. Of the 5,441 exited participants, demographic data were unknown or not collected for between 16.9 percent for gender and 19.3 percent for race/ethnicity. See Appendix B for additional details on this limitation to the data. *The QPR instructed grantees to provide for anyone exiting, their age at enrollment since the age at exit may not have been easy to determine. B. Participant Outcomes The QPR instructed each grantee to record the number of participants who both exited its project and demonstrated a positive outcome, or a change that happened to participants compared to some prior state, which might have included a participant’s status prior to enrollment, at enrollment, or at some point while enrolled. However, consistent with the variation in projects that grantees offered, participant outcomes were also wide-ranging. In their QPRs, grantees described an extensive list of outcomes that they captured, including: • improvements in health, well-being, or feelings of connectedness with others; • improved connections to health or other social services; • an exhibition of increased resilience or prosocial tendencies; • a sense that basic needs were being met; • reduced arrests, convictions, incarcerations or other involvement with law enforcement; • reduced involvement with gang activity and reduced retaliation or violence towards other groups; • fewer altercations in school or at home; • decreased truancy or increased attendance or engagement at school; • reduced substance use; • completion of educational milestones like credits or a degree; • indications of improved living conditions; and Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 43 • placement in employment. To help standardize these outcomes, the QPRs asked grantees to match each of their outcomes to one of 29 outcome categories that the study team developed, based on a review of grantee documents, designed to standardize the outcomes grantees recorded for participants. Exhibit VI-3 displays the total number of participants with positive outcomes (and the percentage of these participants out of all exited participants) for the 27 outcome categories in which there were outcomes. Exhibit VI-3: Number and Percentage of Exited Participants with Positive Outcomes Reported by Outcome Category for All Grantees Psychosocial 4: Life Choices 1,368 (21.1%) Skills 2: Conflict/Anger Management/SES 1,025 (18.8%) Psychosocial 3: Safer in the Community 801 (14.7%) Crime 2: Arrests/Charges 733 (13.5%) Psychosocial 5: Basic Needs Met 722 (13.3%) Skills 3: Life/Health/Family Choices 716 (13.2%) Crime 1: Violence/Gang Activity 521 (9.6%) Psychosocial 1: Engaged with Society 378 (6.9%) Health 4: Victimization 371 (6.8%) Employment 1: Employment 364 (6.7%) s Skills 4: Work/Career Readiness 362 (6.7%) e ir o Psychosocial 2: Connected to Others 349 (6.4%) g e Health 1: Mental Health 342 (6.3%) t a C K-12 Ed 3: Attendance 340 (6.2%) e m K-12 Ed 1: Enrollment/Credits 282 (5.2%) o c Crime 4: Incarceration 246 (4.5%) t u O Health 3: Drug/Alcohol Use 243 (4.5%) K-12 Ed 4: Disciplinary Actions 189 (3.5%) Housing 1: Stable Housing 182 (3.3%) K-12 Ed 2: Diploma/GED 176 (3.2%) Crime 3: Convictions/Petitions 141 (2.6%) Postsecondary 1: Enrollment/Credit 114 (2.1%) Health 2: Physical Health 66 (1.2%) Employment 3: Earnings 47 (0.9%) Employment 2: Retention/Promotion 37 (0.7%) Skills 1: Justice System Awareness 24 (0.4%) Postsecondary 2: Certificates 1 (0.0%) Source: Cohort 3 CalVIP QPR Data Notes: The exhibit shows the number of unique people who exited and demonstrated a positive change to the outcome category shown, as well as the percentage of people who experienced that outcome, out of the number of people who exited (5,441). Grantees could record a participant as having achieved more than one positive outcome. Thus, the sum of all participants shown exceeds the total number exited. Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 44 Based on the data in Exhibit VI-3, and some additional analysis of participant outcome data, a few general trends are apparent. • Grantees captured positive outcomes for a considerable percentage of participants. While these data have some limitations (as discussed below) which make it impossible to calculate the total number of participants who obtained a positive outcome, it was at least 1,368 (the largest number for any one category) and is therefore likely much larger. Furthermore, the QPR data show that grantees captured positive outcomes in about five outcome categories, on average. This breadth and number of outcomes is considerable. • The groups of outcome categories that participants achieved most frequently are the “psychosocial” and “skills” outcomes, followed by the “crime” outcomes. Chapter V noted that some of the most frequently delivered services were those that focused on affecting psychosocial and short-term outcomes like supportive services, CBT, pro-social services, and developing life plans. While the study cannot examine any direct correlation between these services and outcomes, the similarity between these services and outcomes is notable. • The two specific outcome categories achieved most frequently (by a considerable margin) involve participants making better life choices or feeling better about them (Psychosocial 4) and reporting improvements around conflict and anger management and social emotional skills (Skills 2). These are followed by four others involving participants feeling safer in their communities (Psychosocial 3), experiencing new skills around life skills, health, and family choices (Skills 3), having reduced arrests/charges (Crime 2), and feeling that one’s basic needs are being met (Psychosocial 5). All six of these outcomes also fit the pattern highlighted in the bullet immediately above as being outcomes closely associated with the delivery of the services provided to the most participants and by the most grantees. These outcomes also speak to the general approach grantees took with their programs in terms of improving participants’ capacity to navigate the violence within the communities they lived and to face these environments with an increased capacity for avoiding violence. • Grantees tracked outcomes in most, but not all (27 out of 29), of the outcome categories available to them. Grantees did not report any outcomes around job satisfaction (Employment 4) or degrees in higher education (Postsecondary 3) Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 45 and only one positive outcome was reported for the receipt of a higher education certificate (Postsecondary 2). Given that the data show several hundred outcomes in the other employment outcome categories, the lack of data on job satisfaction is likely a result of grantees’ inability to collect this information. While the lack of data on post-secondary certificate or degree attainment may also reflect a difficulty in collecting this information, it also likely reflects the fact that such activity was minimal among exited participants or that these outcomes are longer-term in nature and grantees were not able to capture them during the grant operating period. COLLECTING DATA ON OUTCOMES In their QPRs, grantees described the ways they collected data on outcomes. Many grantees used qualitative data collection approaches, including interviews and focus groups (which were sometimes participant-led) and surveys. These assessed participants’ accomplishments, sentiments, and differences in knowledge and personal growth. In interviews, grantees noted that collecting survey data could sometimes be challenging because they found it difficult to identify/develop reliable survey tools, making them accessible and written at the right level and using the right terminology for their populations. They also found it difficult to identify strategies for tracking and following participants over time. To a lesser extent, grantees also relied on administrative data, which they indicated worked best when they were able to establish a data-use agreement (DUA) with partners. However, grantees were not always able to do this, especially for criminal justice system data (e.g., arrests, gun charges, parole/probation information, convictions, or incarcerations) because partners were sometimes unwilling or unable to share these data. For some grantees, it was challenging to collect quality data from service delivery partners, especially when navigating many different partners with oversight over their own service delivery tracks. Although these are notable accomplishments, it is important to discuss some limitations to these outcome data in terms of understanding what they report and to help shed insight on future evaluation efforts. • These outcomes, by definition, only describe positive changes to participants. This means that outcomes only describe reductions in negative activities like crime or substance use, or increases or improvements in other measures like Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 46 education, employment, or skills. The study lacks the ability to account for any neutral or negative changes alongside these positive ones. • Outcomes are limited to participants who exited their projects. This is important since a participant could have experienced positive changes (or negative ones considering the point above) while still enrolled, prior to exiting. In theory, grantees should still have reported these outcomes at the point of exit. However, given the relatively high percentage of participants who had not exited at the end of the operating period, it is possible that data under-count outcomes that participants experienced. • While the evaluation can broadly describe whether there were positive outcomes, the specific type and nature of the gains is difficult to capture. For example, a group of grantees might all have reported improvements in conflict or anger management. However, for one grantee, this could have meant improvements on a skills assessment related to a class taken while, for another grantee, it involved a positive change involving fewer conflicts with friends or associates as measured on a self-assessment. Thus, while the evaluation recorded positive outcomes across a range of broad categories, because these categories mask important differences in the specific outcomes achieved, the ability to summarize outcomes was necessarily limited to the more general summary. • While the study observed the number of participants achieving any one outcome, it could not determine the total number of participants achieving outcomes. Because the study collected aggregate participant data, it was not possible to determine individual counts or conduct additional analyses such as how many multiple positive outcomes individual participants achieved. C. Summary Grantees reported that large numbers of participants exited grant-funded projects, exited these projects successfully, and had positive outcomes. Out of the 9,523 cohort 3 CalVIP project participants, grantees exited 5,441 (57.1 percent) people. Of these, 60.7 percent successfully completed projects while others left on their own or were asked to leave. The 42.9 percent of non-exited participants either stayed enrolled, through other funding grantees obtained, or may have been exited without Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 47 grantees capturing these data. The demographic profile for exited participants, closely resembles the demographic profile for all participants. The categories with the most positive outcomes among participants included: three measures of well-being having to do with participants feeling like they were making better life choices (21.1 percent), feeling safer in their communities (14.7 percent), or having their basic needs met (13.2 percent); two skills-based outcomes including improved conflict/anger management/social emotional skills (18.8 percent) and improved ability to make life/health/family choices (13.2 percent); and having reduced arrests and new charges (13.5 percent). Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 48 VII. Summary and Conclusion This report summarizes the evaluation of the third cohort of grantees, funded under the CalVIP grant program, a state funded initiative administered by the BSCC. The CalVIP program is designed to reduce violence in some of the most violent communities across California. This chapter summarizes the key findings from the evaluation and then discuss the implications of this evaluation and its findings, first for the consideration of future funding decisions, and subsequently for future research on programs designed to reduce violence in communities. A. Summary of Key Findings The BSCC awarded $36,550,083 to 32 cohort 3 grantees who began operating their CalVIP projects in October 2020, amid the COVID-19 pandemic, which had various implications for implementation.18 These grantees recruited and enrolled participants, delivered services, exited participants, and tracked their outcomes for an 11-quarter operating period that lasted through June 30, 2023. At that point they continued to manage their grants for another two quarters as part of an administrative period that included additional grant and state-wide evaluation reporting, financial activities, and the completion of a local evaluation. The sections below highlight key findings from the statewide evaluation of the CalVIP program. 1. Overview of Grantee Projects and Services The 32 CalVIP grantees used their grants funds to implement a wide array of projects and services. • Grantees delivered at least five and as many as 22 different types of services. Case management services were offered most frequently (30 grantees) and at least two-thirds of grantees also provided another eight types of services that included offering various other types of support, improving participant psychosocial skills, teaching soft skills, and providing direct conflict intervention services. Less frequently offered were those associated with hard skills and traditional service delivery systems like 18 The City of Stockton was awarded a grant, not included here, but withdrew from the CalVIP program after the first year. Neither its project, nor its funding, are included in report findings. Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 49 educational services, vocational training services, substance use treatment services, mental and physical health services, and housing services. • Twenty grantees provided a single service track and 12 grantees provided three to 10 service tracks. These service tracks were distinct lines of service into which participants could not co-enroll and which had separate enrollment goals, eligibility criteria, services that they delivered, and exit criteria. 2. Funding, Staffing, and Partnerships Managing the implementation of these three inputs was critical for grantees in effectively operating their CalVIP projects, including ensuring they had the capacity to recruit, enroll, and serve participants. • Cohort 3 CalVIP grantees spent 70.2 percent of their grant awards. While overall, grantees underspent funds, there was considerable variation in expenditures at the individual grantee level meaning that some grantees expended all their funds while others underspent. Due to invoices still being processed, this evaluation may not include all expenditures, but given COVID-19-related operational delays, it is not likely to change substantially. • Grantees staffed their projects, trained their staff, and established partnerships as expected, without significant delays. Most grantees had completed these implementation processes early, but several continued to work on them throughout the program operating period. • In alignment with the goals of the grant program, grantees developed from one to 36 partners (an average of 7.7) that helped deliver services or operated service tracks. Grantees most frequently utilized CBOs for these roles but also relied on various types of local government and law enforcement agencies, education providers, health care providers, and workforce system agencies. • Grantees found that partners often had limited capacity to accurately track and process participant data, but they found ways to overcome this challenge. Strategies included setting up regular communication channels for training and learning and sharing data collection tools and systems with partners to increase their capacity for accurate data reporting. Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 50 • Partnerships between law-enforcement and non-law enforcement organizations posed challenges, but also opportunities. Law enforcement agencies (police, sheriff, probation, and parole) and non-law enforcement agencies and organizations (CBOs and other types of city agencies) frequently needed to work together, but could approach problems with different, sometimes conflicting perspectives or face resistance or hesitation from participants reluctant to work with law enforcement. Strategies for overcoming these issues involved relying on a third-party to facilitate conversations, having staff from both organizations work closely together, and having law enforcement meet participants in safe settings to build trust and collaboration. 3. Enrollment and Participant Composition Recruiting and enrolling participants were critical to grantees’ success. Key findings include the following. • Grantees used three main recruitment strategies: engaging in direct outreach with potential participants, building word-of-mouth referrals, and working with referral partners. Grantees were able to engage in these strategies quickly and established a large and diverse set of referral partners. • Grantees enrolled 9,523 people into their CalVIP projects, exceeding their own collective enrollment goals by nearly 50 percent. While this success rate varied at the individual grantee level (i.e., some grantees fell short of their goals while others exceeded them), collectively grantees’ recruitment and enrollment efforts were effective. • Participants tended to be Hispanic/Latino (52.5 percent) or black/African- American (32.1 percent), 24 years of age or younger (77.6 percent), and male (68.4 percent). The racial and gender breakdown of participants is not surprising given the demographics of the neighborhoods served, as described in grant proposals. The age of participants aligns with the high rate of participants who were still in school, most of whom were living at home, and not employed. 4. Service Delivery The services provided to participants formed the core of the program in that these were the means by which grantees aimed to reduce violence in their communities, Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 51 either by directly intervening to prevent violence from occurring, or more commonly, by providing participants with the opportunities, skills, and supports needed to avoid violence. Key findings include: • Services provided to the most participants were also provided by the most grantees, and the reverse was also generally true. Accordingly, case management was delivered to more participants than any other service (27 percent) and was also the service offered by the most (30 of the 32) grantees. • The services provided to the largest numbers of participants (excluding case management) tended to focus on supporting participants in other ways, building soft-skills, intervening in difficult situations, and generally improving psychosocial and other short-term outcomes of participants. In contrast, services that were provided to fewer participants were those focused on the delivery of hard skills and traditional social services, such as vocational training skills, education, health, and housing services. • Two grantees provided one-time services, delivered at a single time to large groups of people who were not enrolled participants. These included videos and presentations designed to educate and increase awareness around domestic violence and human trafficking and community mediations designed to mitigate conflicts. According to grantees, these services reached several thousands of people and may have contributed to recruitment and enrollment efforts. • Grantees used different strategies for keeping participants engaged. Major obstacles grantees faced that led participants to disengage from services included the use of remote services and other service reductions brought on by the pandemic, and the high level of trauma experienced by many participants, which made them resistant to receiving help. Solutions grantees relied upon included developing evidenced-based services for dealing with trauma, building rapport by meeting with participants to build their trust, enlisting staff with lived experience, connecting with participants by teaching them coping skills and connecting with their families, and by providing incentives to encourage the completion of services and activities. Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 52 5. Exits and Outcomes Each grantee defined exit criteria for its project or service tracks and then followed participants to see if they exited, the reasons for their exits, and whether people who exited, experienced positive outcomes. Key findings on exits and outcomes include the following. • Grantees reported that 5,441 out of 9,523 participants (57.1 percent) exited CalVIP projects. Of those who exited, 60.7 percent successfully completed their project (based on a project-specific definition set by each grantee) while another 22.4 percent dropped out or lost contact and the remainder exited for various other reasons. • The remaining 4,082 participants (42.9 percent) did not formally exit from their projects. These participants either stayed enrolled in services or exited with their data not captured by grantees. • Exited participants closely resembled the overall composition of participants in terms of race, age, and gender. This need not be the case given the sizable number of participants who did not exit. • Participants most frequently achieved positive outcomes that were focused on psychosocial measures and other skills, followed by those involving reduced crime. The six outcome categories with the largest percentage of positive outcomes (out of exited participants) included: improved life choices (21.1 percent); improved conflict resolution and anger management skills (18.8 percent); improved feelings of safety in their own communities (14.7 percent); having reduced arrests or charges (13.5 percent); feelings that basic needs were being met (13.3 percent); and developing new life skills, health skills, and family choice skills (13.2 percent). These outcomes have a clear relationship to the services provided by many grantees which focused on influencing participants’ thinking and decision-making as well as providing them various types of support. B. Implications for Future Program Funding These findings both summarize the efforts of cohort 3 CalVIP grantees, and also point to some implications for future funding. Although this report does not presume to provide wide-ranging recommendations regarding the design of future Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 53 funding, the findings from this evaluation do raise some issues for potential consideration. Despite the stated intent of the CalVIP program being to reduce violence in California communities, the projects and services grantees implemented were often engaged with school-aged youth, many of whom had relatively little justice system contact or criminal histories. This may be highly appropriate, as reaching people before they first engage in violence or have contact with the justice system may be a suitable way to reduce crime in the long run. Further, grantees may have made informed decisions about who they felt they could most effectively serve, based on the needs of their communities and the resources available to them. That said, as future funding is contemplated, it is worth considering whether participants who have already been involved more substantially with the justice system should be targeted more heavily. Many of the cohort 3 projects focused heavily on influencing psychosocial skills and were designed to affect participants’ thinking and decision-making. There is substantial literature suggesting these types of services are influential and can have meaningful and lasting effects on participants, so grantees reliance on them is appropriate, especially given the emphasis the CalVIP program places on the use of evidence-based practices.19 However, there is also evidence that people who are at- risk of violence, even younger individuals who made up the majority of CalVIP participants, need access to hard skills and more tangible activities, including job training and skill development and education (Lacoe and Betesh, 2019). The CalVIP cohort 3 grantees were somewhat less focused on these areas as well as services like housing and health care, and offered services designed to address them far less frequently. The low incidence of services like housing and health care is not unsurprising given that so many participants were younger and living with families. With regards to education and vocational training, there may have been funding constraints around providing such services, which often can be more expensive and/or involve a greater degree of external partner involvement. Even though many grantees underspent funds and may have been able to afford these services in retrospect, they may not have envisioned them as being feasible based on their initial spending plans. Future funding decisions may wish to consider the relative 19 See the references to various literature in Chapter II. Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 54 balance of services offered and perhaps encourage grantees (or some portion of grantees) to include more services targeting these hard skills. One clear strength of the CalVIP program is that it allows individual grantees to develop their own projects and tailor them to the needs of their communities. This allows organizations closest to the situation and most aware of the unique needs in their communities to build projects designed to address those needs. Furthermore, given the partnerships and multiple service tracks that these grantees can develop, this responsibility can lie across a partner network and not just with the grantees themselves. A drawback to this highly customized approach, however, is that there is great variability across and even within grantee projects in terms of who they are enrolling, the services they are offering, and the types of outcomes they are trying to achieve. In many ways, there is no single CalVIP program; rather, CalVIP includes multiple projects and many different service tracks funded under the CalVIP funding umbrella. This, too, may be entirely appropriate, but it does make it difficult to establish consistent standards around service delivery and output and outcome reporting. Given this, going forward it may be of interest to consider whether there should be distinct project models or service tracks from which grantees can choose and which have clear and consistent standards across grantees. This more limited set of options would also have the advantage of providing grantees with their own cohort of similar projects, targeting support and technical assistance around specific service types, ensuring that summaries of their efforts would not mask the variability inherent in the projects, and providing more uniform models that could be more cleanly analyzed and evaluated. Finally, this report has demonstrated that grantees made substantial progress in implementing their grants and achieving the desired outcomes. They implemented an array of projects targeted to meet the unique needs of their communities, enrolled 9,523 people, provided a large number of services, exited more than 5,000 participants, and recorded a wide array of meaningful outcomes. Despite this, much more could be learned and documented about their accomplishments if grantees had been more successful in capturing or recording data that would both help them manage their projects and participants, and document the demographic characteristics of those participants, the services they received, or their outputs and outcomes. And some grantees also struggled to gather useful information from their partners or service providers, which left them less able to manage their Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 55 projects overall. Additional attention on the part of grantees and/or additional technical assistance on the part of the research team may lead to improvements in the ability to monitor and assist grantees as well as evaluate their efforts and successes. C. Lessons for Future Research In addition to potential implications for future funding, this evaluation also considers implications for future research on programs like CalVIP that are designed to reduce violence within communities. The first of these is to collect individual-level data, rather than aggregate data. Doing so would enable researchers to link data across multiple data sets, track participants and their progress over time, meaningfully assess how much and how many different services a participant received, and link those services to specific outcomes. Although this limitation was known throughout the life of this evaluation, the use of individual-level data was infeasible for this cohort of grantees. Future evaluation efforts would be greatly enhanced if they had access to such data, though clearly this would require additional funds and a heavier emphasis on consistent data collection across grantees. That said, accessing individual-level data is not simple and may have additional costs. Doing so requires a data system that is (at least somewhat) standardized across grantees who are doing somewhat disparate things, with varying information technology infrastructures. This would undoubtedly create an additional burden to grantees in collecting standardized information, especially when there is a need to keep these data private and secure. However, the utility of having such data is so great that it is certainly worth considering whether future programming should include funding for a tracking or data system across all grantees. This would likely benefit grantees by providing a uniform tool for data collection and perhaps in helping better manage service delivery. It would also substantially improve the ability of evaluators to draw meaningful and insightful conclusions about the program and its outcomes by examining each participant’s service history and linking those services to outcomes. That area for improvement aside, this evaluation notably highlights at least two promising practices that contributed significantly to this current evaluation and seem important for subsequent evaluations, whether of CalVIP or other violence Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 56 intervention and prevention programs. The first was the knowledge gained by providing grantees with a way to summarize their efforts in a standardized way, in this case through the QPR. Although this tool was not perfect and included challenges in standardizing data across a wide variety of efforts, it also provided a means of summarizing a large volume of data in concise and relatively consistent ways. The QPR enabled the evaluators to present and assess meaningful aggregate data on the characteristics of participants served, the services they received, and the outcomes they obtained. Second, the BSCC funded not only an overall evaluation of this cohort of grantees, but also required each grantee to use some portion of its funds to conduct a local evaluation. These local evaluations are available for public review and, while they vary in quality and comprehensiveness, their requirement made clear to grantees that evaluation was a critical aspect of their efforts. While some grantees included more extensive formative evaluations as part of their implementation, incorporating local evaluation feedback in a continuous loop of testing, learning, and improving parts of their programs, the fact that each grantee was required to conduct a local evaluation meant that they were attuned to learning from their efforts and assessing what was working and what was not. This approach is extraordinarily beneficial for future research to consider. Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report R-1 References Andrews, D.A. and Bonta, J. (2007) The Risk-Need-Responsivity Model of Assessment and Human Service in Prevention and Corrections: Crime-Prevention Jurisprudence. The Canadian Journal of Criminology and Criminal Justice, 49, 439- 464. 2007. https://doi.org/10.3138/cjccj.49.4.439 Assembly Bill No. 1603, Chapter 735. https://www.bscc.ca.gov/wp- content/uploads/AB-1603-Signed101819.pdf Bush, J., Glick, B., and Taymans, J. (1997). Thinking for a Change: Integrated Cognitive Behavior Change Program. Prepared for the National Institute of Corrections, U.S. Department of Justice. 1997. (The current version of the Thinking for a Change curriculum can be found at: https://nicic.gov/resources/nic- library/web-items/thinking-change-40) Hetey, R. C. and Eberhardt, J. L. (2018). The Numbers Don’t Speak for Themselves: Racial Disparities and the Persistence of Inequality in the Criminal Justice System. Current Directions in Psychological Science, 27(3), 183-187. 2018. https://doi.org/10.1177/0963721418763931 Kumpfer, KL., and Magalhães, C. (2018). Strengthening Families Program: An Evidence-Based Family Intervention for Parents of High-Risk Children and Adolescents. Journal of Child Adolescent Substance Abuse. Volume 27, 2018 – Issue 3, Online 26 Mar 2018. Lacoe, J. and Betesh, H. (2019). “Supporting Reentry Employment and Success: A Summary of the Evidence for Adults and Young Adults.” Mathematica. Prepared for the U.S. Department of Labor. 2019. https://wdr.doleta.gov/research/FullText_Documents/ETAOP2019- 11%20REO%20Supporting%20Reentry%20Employment%20RB090319.pdf Lofstrom, M., Martin, B., and Skelton, A. (2024). Arrests in California. Public Policy Institute of California. January 2024. https://www.ppic.org/wp- content/uploads/arrests-in-california.pdf McGuire J. (2008). A review of effective interventions for reducing aggression and violence. Philos Trans R Soc Lond B Biol Sci. 2008 Aug 12;363(1503):2577-97. doi: 10.1098/rstb.2008.0035. PMID: 18467276; PMCID: PMC2606715. Evaluation of the Cohort 3 CalVIP Grant Program Final Evaluation Report 2 Papalia, N., Spivak, B., Daffern, M., & Ogloff, J. R. P. (2020). Are Psychological Treatments for Adults With Histories of Violent Offending Associated With Change in Dynamic Risk Factors? A Meta-Analysis of Intermediate Treatment Outcomes. Criminal Justice and Behavior, 47(12), 1585-1608. https://doi.org/10.1177/0093854820956377 Skogan, W., Hartnett, S., Bump, S., and Dubois, J. (2008). “Evaluation of CeaseFire- Chicago.” Prepared for the National Institute of Justice (NIJ). March 2008. https://www.ojp.gov/ncjrs/virtual-library/abstracts/evaluation-ceasefire- chicago Scally, C. P., Gold, A., Oliver, W., and Salerno, C. (2020). “Implementing Tiny Homes and Permanent Supportive Housing: Early Lessons from Housing First Village in Bozeman, Montana.” Urban Institute. August 2020. https://www.urban.org/sites/default/files/publication/102715/implementing- tiny-homes-as-permanent-supportive-housing.pdf Webster, D. W., Richardson, J., Meyerson, N., Vil, C., and Topazian, R. (2022). Research on the Effects of Hospital-Based Violence Intervention Programs: Observations and Recommendations. The ANNALS of the American Academy of Political and Social Science, 704(1), 137-157. https://doi.org/10.1177/00027162231173323 Evaluation of the Cohort 3 CalVIP Grant Program Appendix A: List of Cohort 3 CalVIP Grantees A-1 Appendix A: List of Cohort 3 CalVIP Grantees Number of Project Award Invoiced Project Grantee Name Project Name Service Enrollment Amount Amount Enrollment Tracks Goal CBO Grantees (Location) Catholic Charities TRUE Academy 1 $ 1,062,957 $ 443,457 150 549 (Oakland) FLY's Violence Prevention and Fresh Lifelines for Youth Intervention Programs for Oakland 2 $ 1,033,839 $ 1,005,058 260 1107 (Oakland/San José) and San Jose Fresno County Economic Opportunities Commission Project Peace 1 $ 925,000 $ 849,191 40 67 (Fresno) Kitchens for Good (San Project Launch 1 $ 1,000,000 $ 916,420 432 226 Diego) Los Angeles Brotherhood Proud to be Me Trauma Informed Crusade, Black United Young Gang Violence Prevention 1 $ 1,500,000 $ 690,378 267 311 Fund, Inc. (Los Angeles) Program Safe Passages (Advance Safe Passages 1 $ 1,342,386 $ 1,342,386 100 123 Peace) (Oakland) Sierra Health Foundation Healing the Hood Initiative 1 $ 1,500,000 $ 784,699 70 116 (Sacramento) Soledad Enrichment Caught in the Crossfire 1 $ 1,500,000 $ 823,424 480 129 Action, Inc. (Los Angeles) South Bay Community Increasing Resiliency in High-Risk 1 $ 1,500,000 $ 1,236,745 188 158 Services (Chula Vista) Youth to Reduce Involvement in Crime Southern California Crossroads Hospital-Based Violence 1 $ 216,654 $ 193,654 100 441 Crossroads (Lynwood) Intervention Lundquist Institute/Harbor UCLA The Lundquist Institute at Medical Center Hospital Based 1 $ 1,499,999 $ 695,449 270 156 Harbor – UCLA (Torrance) Violence Intervention Program-Safe Harbor Evaluation of the Cohort 3 CalVIP Program Appendix A: List of Cohort 3 CalVIP Grantees A-2 Number of Project Award Invoiced Project Grantee Name Project Name Service Enrollment Amount Amount Enrollment Tracks Goal The Regents of the UCDMC Wraparound Hospital-Based University of California - UC 1 $ 966,049 $ 648,129 65 39 Violence Intervention Program Davis (Davis) Watts Gang Task Force One Watts Initiative 1 $ 1,500,000 $ 894,149 300 326 Council (Los Angeles) Youth Alive! (Oakland) From Incident to Intervention 7 $ 1,500,000 $ 1,438,110 375 457 All CBO Grantees 21 $ 17,046,884 $ 11,961,249 3,097 4,205 City Grantees Bakersfield Police and Community City of Bakersfield 1 $ 1,500,000 $ 891,492 50 63 Partnership Reducing Violence Through City of Fresno, Police Intervention, Prevention, and 8 $ 1,263,368 $ 926,328 736 1304 Department Education City of Long Beach, Long Beach Activating Safe Department of Health and 2 $ 1,102,698 $ 740,236 105 56 Communities Human Services City of Los Angeles GRYD 10 $ 3,000,000 $ 2,873,484 864 1393 City of Oakland, Department of Violence Oakland Cease Fire 1 $ 1,499,672 $ 1,071,618 75 51 Prevention City of Oxnard, Police Operation Peace Works and offer 1 $ 500,000 $ 370,914 90 87 Department Peace Through Collaboration City of Richmond, Office of Beloved Community: Next Level 2 $ 1,500,000 $ 1,415,453 170 396 Neighborhood Safety Collaborative City of Sacramento Youth Peacemaker Fellowship 1 $ 1,500,000 $ 14,535 60 79 R.E.A.D.Y Outreach; Intervention & Re- City of Salinas 2 $ 1,007,121 $ 870,474 270 363 entry + The Tiny Homes Program Focused Deterrence, Community- City of San Bernardino 4 $ 1,500,000 $ 1,285,396 125 257 based Street Outreach City of San Francisco, CONNECT-You 1 $ 1,500,000 $ 659,435 25 60 Police Department Evaluation of the Cohort 3 CalVIP Program Appendix A: List of Cohort 3 CalVIP Grantees A-3 Number of Project Award Invoiced Project Grantee Name Project Name Service Enrollment Amount Amount Enrollment Tracks Goal City of San José, Mayor's Trauma to Triumph Hospital-Based Gang Prevention Task 1 $ 630,340 $ 391,459 185 66 Intervention Program Force All City Grantees 34 $ 16,503,199 $ 11,510,824 2,755 4,175 Small City Grantees City of Grass Valley - Homeless City of Grass Valley 1 $ 574,695 $ 499,836 90 174 Violence Reduction Initiative City of Greenfield: Road to Success City of Greenfield 3 $ 484,170 $ 403,585 215 123 Plus Gustine Mental Health and Programs City of Gustine 1 $ 593,487 $ 241,214 75 186 in Schools King City King City: Road to Success PLUS 3 $ 461,171 $ 422,268 215 90 City of Marysville Youth Gang Reduction Task Force 2 $ 286,477 $ 185,403 20 41 City of Parlier Communities That Cure Violence 8 $ 600,000 $ 435,560 622 529 All Small City Grantees 18 $ 3,000,000 $ 2,187,866 1,237 1,143 All Grantees 73 $ 36,550,083 $ 25,659,939 6,679 9,523 Source: Grantee Documents and Cohort 3 CalVIP QPR Data Evaluation of the Cohort 3 CalVIP Program Appendix B: Evaluation Design Details B-1 Appendix B: Evaluation Design Details This appendix provides additional details on the approach taken in the evaluation, including a description of how the data were collected and their limitations. A. Additional Details on Data Collected There were three main data sources, which are described below. 1. Quarterly Progress Reports The Quarterly progress reports (QPRs), which grantees used to capture information on their grant activities over the operating period (from October 2020 through June 30, 2023), were the primary data source for the evaluation. The QPRs were designed to standardize information across grantees given that they often operated very different types of programs and delivered different types of services from one another. The study team developed the QPR tool based on information in grantee proposals and planning documents as well as other requirements the BSCC had for grantee reporting for monitoring purposes. As can be seen in the QPR guide (Appendix C), which the study team developed to aid grantees in completing the online QPR tool (completed using Alchemer), the QPR asked grantees to complete questions organized into five main sections. • Section 1 asked for basic contact information and information designed to alert the BSCC as to whether grantees needed additional support. • Section 2 asked for information on each grantee’s current operational status, including the status of its expenditures, progress towards completing its self-defined goals and objectives, and progress towards implementing key inputs such as partnerships, staffing, and data collection efforts. • Section 3 asked for information on the numbers of distinct programs, or service delivery tracks, that each grantee operated and information about each of those tracks, including eligibility criteria and enrollment goals, the services provided, and criteria used by grantees in determining when participants had successfully completed each track. • Section 4 asked for numbers, both quarterly and cumulatively, of participants who enrolled, received each type of service, and exited, as well as demographic information for those participants. Evaluation of the Cohort 3 CalVIP Grant Program Appendix B: Evaluation Design Details B-2 • Section 5 asked for a description of the outcomes that grantees captured and the numbers of participants who achieved those outcomes, both quarterly and over the lifetime of the grant. The schedule for completing QPRs included 10 submissions, encompassing the 11- quarter operating period. The first QPR combined the first and second of these reporting periods, including activities from the start of the grant in October 2020 up through March 2021. The last QPR covered the 11th reporting quarter, which lasted from April to June of 2023. Grantees were to submit QPRs approximately 75 days after the end of each quarter. The study team collected and reviewed the submitted QPRs on a regular basis. It also worked closely with the grantees to request revisions and clarifications to the data with the goal of creating as complete, consistent, and accurate a dataset as possible. 2. Grantee Interviews To explore and provide rich data around the features of grantee programs and the factors that supported or hindered project implementation, the study team interviewed one to two lead staff members from 16 purposefully selected (i.e., neither random nor fully representative) grantees. The selection criteria sought to identify a group of grantees that was as representative as possible while still identifying features of interest to the evaluation. These criteria included: 1) grantee type (i.e., city, CBO, small city); 2) geographic diversity; 3) notable practices highlighted in the QPRs and grantee documents, such as the use of certain evidence-based practices, ways of scaling or adapting projects, working with specific populations, or the use of specific program models; and 4) noteworthy implementation experiences identified in QPRs, including challenges grantees experienced and strategies used to overcome them.20 20 This last selection criteria obviously biases any results in discovering challenges and solutions, but it offered the opportunity to explore challenges and success that the study team had previously noted in interactions with grantees and thus worth additional attention as to the nature of the issues or the solutions provided by grantees. Evaluation of the Cohort 3 CalVIP Grant Program Appendix B: Evaluation Design Details B-3 The study team conducted these 60-minute interviews with grantee staff by video conference or telephone between March and April of 2023 (The 16 grantees are indicated with an asterisk in Exhibit I-1, above). Using a customized interview protocol (Appendix D), study team staff members asked grantee representatives to react to, expand upon, and offer additional examples around information they provided in their QPRs. Interviewers also asked grantee staff members to discuss the overall implementation process and successes and challenges they encountered, as well as clarifying information around the partnerships they developed, the effect of the COVID-19 pandemic on their project implementation, and their enrollment, delivery of project services, and process for capturing participant outcomes. These interviews captured examples and useful quotations that highlight specific approaches and practices that grantees engaged in and help to illustrate and expand on the multiple choice or short answer information frequently provided in the QPR data. 3. Grantee Documents and Communications The study team also analyzed information about grantee projects captured in the ways outlined below. These data were sometimes used directly, but were more often used indirectly, in shaping the QPR tool and interview protocols or in helping to further interpret the data collected in these tools. • Grantee Proposals. At the start of the evaluation, the study team reviewed each grantee’s proposal to identify initial program plans and to start building a comprehensive picture of the work in which grantees planned to engage. • Grantee Local Evaluation Plans (LEPs) and Local Evaluation Reports (LERs). Under the grant program, each grantee was required to develop an LEP and an LER, which described various features of their projects and project outputs and outcomes as well as information including the methods of data collection that grantees used or challenges and successes they encountered while implementing their projects. • Grantee Communications. Part of the study team’s role was to provide support to grantees in completing the QPRs, to provide a quality review of their QPR data, and to hold grantee convenings to support grantees in the evaluation of their own projects. These conversations were often process- focused or focused on interim or in-process data, but they helped to inform Evaluation of the Cohort 3 CalVIP Grant Program Appendix B: Evaluation Design Details B-4 the study team’s data collection process, and sometimes yielded examples which helped in understanding grantees’ QPR data. C. Data Limitations to the QPR Data There are three limitations to the QPR data that affect its analysis and interpretation in addressing the evaluation research questions. 1. Missing Data Enrollment, service delivery, exit, and outcome data may be slightly under- reported due to missing quarter 11 data from one grantee. This level of under- reporting, if it exits, is likely to be small based on this grantee’s numbers relative to the overall number of CalVIP participants. 2. Aggregate Nature of the Data Because the QPR collected aggregate, rather than individual-level data, it was not possible to follow a given participant’s progress or view the entirety of the data collected for a given person. For instance, grantees reported on the number of people receiving a given type of service during each quarter as well as the demographic profile of the people enrolled. However, because these data were only reported in aggregate for each measure, it was not possible to link these data to examine, for example, the demographic profile of the people who received a given service. Likewise, the aggregate nature of the data limited the ability to review data for quality issues related to inconsistent reporting for a given person, either within a quarter or over time, potentially increasing the likelihood of certain types of reporting errors. 3. Data that were Unknown/Not Collected For a considerable percentage of people who enrolled (Chapter IV) and who exited (Chapter VI) CalVIP projects, grantees indicated that data were either unknown or not collected. Additional details on these unknown data are described below. • The percentage of participants for whom data were unknown/not collected varied by demographic category and ranged from 13.1 (gender, for those who enrolled) to around 50 percent (“point of diversion,” “assessed risk group,” and “employment status,” for those who enrolled). • This issue is particularly concentrated in the data for some grantees. For four enrollment demographic categories (“point of diversion,” “assessed risk Evaluation of the Cohort 3 CalVIP Grant Program Appendix B: Evaluation Design Details B-5 group,” “employment,” “housing”) there are at least eight grantees who list more than 50 percent of their participants as having “unknown” data for this category and several of these have a rate close to 100 percent. • The problem of unknown or uncollected data is widespread among grantees. For every enrollment demographic category, there at least three grantees that report more than 10 percent of participants as having “unknown” data. While these high rates of unknown/not collected data were addressed with the grantees during the QPR quality assurance process, and some grantees were able to adjust for this shortcoming, collecting these data remained a persistent issue throughout the evaluation. Either grantees did not deem it appropriate to collect these data given the particular circumstances of the participants with whom they were working (e.g., they wanted to protect privacy or build rapport), or they lacked the capacity to for various reasons. Regardless of the reason, when levels of unknown data are high, it is more difficult to draw meaningful conclusions as any unknown data could mask information that could substantially alter what is conveyed by the known data. For example, if a grantee knew demographic data on “employment at enrollment” for only 50 percent of participants, and these data showed that 75 percent of those participants were employed (37.5 percent overall), should the percentages in the unknown 50 percent be reversed, then this would lead to a much different understanding of the employment status of these participants: only 50 percent employed compared to 75 percent. One option to address this would have been to have report the percentages of any demographic data, inclusive of the rates of participants for whom data were unknown/not collected. However, this can be confusing given that the rates for some demographic categories are often very low. Instead, the study team removed these unknown participants from the demographic analysis and examined demographic data only for the participants for whom these data were known. Exhibit B-1 shows these adjustments for the enrollment demographic data reported in Chapter IV and Exhibit B-2 shows these adjustments for the exit demographic data reported in Chapter VI. For each demographic category, it lists the percentage of participants for whom data is “unknown/did not collect” along with the percentage and number of participants for whom these data are known. These adjusted enrollment numbers were the numbers then used as the denominator for Evaluation of the Cohort 3 CalVIP Grant Program Appendix B: Evaluation Design Details B-6 the percentages of participants shown for each demographic category in the report. Exhibit B-1: Percentage and Number of Participants at Enrollment with Demographic Data that were Unknown and Known Number of Percentage of Participants Participants at Enrollment at Enrollment Total Number of Enrolled Participants 9,523 100% Race/Ethnicity Unknown/not collected data 1,788 18.7% Known data 7,735 81.3% Age at Enrollment Unknown/not collected data 1,460 15.3% Known data 8,063 84.7% Gender Unknown/not collected data 1,247 24.9% Known data 8,276 75.1% Participation Status Unknown/not collected data 2,748 28.9% Known data 6,775 71.1% Point of Diversion/System Involvement Unknown/not collected data 4,678 49.1% Known data 4,845 50.9% Assessed Risk Group at Enrollment Unknown/not collected data 4,846 50.9% Known data 4,677 49.1% Education Status at Enrollment Unknown/not collected data 3,340 35.1% Known data 6,183 64.9% Employment Status at Enrollment Unknown/not collected data 4,882 51.3% Known data 4,641 48.7% Housing Status at Enrollment Unknown/not collected data 3,524 37.0% Known data 5,999 63.0% Source: Cohort 3 CalVIP QPR Data Evaluation of the Cohort 3 CalVIP Grant Program Appendix B: Evaluation Design Details B-7 Exhibit B-2: Percentage and Number of Participants at Exit with Demographic Data that were Unknown and Known Number of Exited Percentage of Exited Participants Participants Total Exited 5,441 100% Race/Ethnicity Percentage with unknown data 1,051 19.3% Percentage with known data 4,390 80.7% Age at Enrollment* Percentage with unknown data 939 17.3% Percentage with known data 4,502 82.7% Gender Percentage with unknown data 920 16.9% Percentage with known data 4,521 83.1% Source: Cohort 3 CalVIP QPR Data Notes: *The QPR instructed grantees to provide for anyone exiting, their age at enrollment since the age at exit may not have been easy to determine. Evaluation of the Cohort 3 CalVIP Grant Program Appendix C: Quarterly Progress Report Guide C-1 Appendix C: Quarterly Progress Report Guide CalVIP (Cohort 3) Quarterly Progress Report (QPR) This document serves as your guide to the CalVIP (Cohort 3) Quarterly Progress Report (QPR). To start your QPR, please access the online tool at one of the following locations. • To start the QPR for the Quarter 1/Quarter 2 report, please navigate to: [URL] • To start a QPR for any of the Quarter 3 through 11 reports, please navigate to: [URL] These links will take you to a login page. Please log in with your username and password, which will be provided to you by BSCC or evaluation team staff. You can also contact your study team liaison for your username and password. These usernames and passwords help customize your reporting experience and minimize the information you need to fill out in future reports. Since the information in the QPR is not sensitive, these usernames and passwords are not secret. Feel free to share them with other staff at your organization who may need to complete your QPR. We realize that you may need to complete your QPR over multiple sessions. To do so, you will be provided a customized web address you can use to return to a given QPR that you have begun. At the bottom of the first full page (once you log in), you will find this link. It will activate after you move past the first page. You will also be emailed this link based on the email address you provide on the first page of the QPR. The email will come from [evaluation email]. Please be sure to check your junk mail folder if you do not receive it. Also, we recommend that you copy and paste this link before advancing past the first page just in case you have difficulties receiving the email. Please remember that whenever you start a new QPR for a subsequent quarter, always start with the links in the bullet points above. The link within the QPR that you are emailed only allows you to edit an existing report, not start a new one. The rest of this document lays out the questions as they appear in the QPR. While the QPR itself includes most instructions, this document shows you all questions in a single place, helps explain question skip patterns, lists out answer choices for dropdown menus, and explains when certain fields will be auto-filled based on prior questions or past QPRs. Throughout, you will see items highlighted in green if they only appear based on your selection of certain responses such as a follow-up question that only appears if you select “yes” to a question. Items highlighted in blue are those that change based on the section of the report you are in. For example, if your project has multiple goals, then you will see the goal number highlighted in blue. These do not show in the actual QPR but are illustrated here to help guide you. For additional instructions on using the online tool, please view the recorded training under the Cohort 3 tab on the BSCC’s website at: http://www.bscc.ca.gov/s_cpgpcalvipgrant/ Evaluation of the Cohort 3 CalVIP Grant Program Appendix C: Quarterly Progress Report Guide C-2 If you have further questions on how to complete the report or difficulty accessing it, please contact your liaison at: [evaluation email]. Section 1: Grantee and Reporting Information 1.1.1. Please provide the following information about your project and you. a. Grantee: [auto-filled grantee name] b. Project Title: [auto-filled from prior QPR] c. BSCC Grant Award Number: [auto-filled from prior QPR] d. Your name: [text] e. Your title: [text] f. Phone: [phone] g. Email: [email] 1.1.2. Please select the reporting period for this QPR. Quarter 1/2: October 2020 - March 2021 Quarter 3: April - June 2021 Quarter 4: July - September 2021 Quarter 5: October - December 2021 Quarter 6: January - March 2022 Quarter 7: April - June 2022 Quarter 8: July - September 2022 Quarter 9: October - December 2022 Quarter 10: January - March 2023 Quarter 11: April - June 2023 1.1.3. Would you like any Technical Assistance? Yes No [If yes] a. Please describe the type and nature of the assistance you would like. Evaluation of the Cohort 3 CalVIP Grant Program Appendix C: Quarterly Progress Report Guide C-3 [text] Section 2: Quarterly Grant Status 2.1. Expenditure Status 2.1.1. Please report on the status of your grant and project expenditures as of the end of the reporting period. [auto-filled based on grantee a. Grant award amount: name selected in 1.1.1 a] b. Grant amount invoiced-to-date (Sum of quarterly invoices): [currency] [c. and d. will only be visible for city and small city grantees based on your selection in 1.1.1.a.] [auto-filled based on grantee c. Amount of grant budgeted for pass-through funds: name selected in 1.1.1 a] d. Amount of pass-through funds expended (record $0 if you have no [currency] pass-through funds): 2.1.2. Are grant funds being expended as planned and on schedule? Yes No [If no] a. Please describe why not and what your plans are for adjusting future spending. [text] 2.2. Status of Grant Agreement Goals and Objectives 2.2.1. How many goals does your project have? 1 2 3 4 5 6 7 8 2.2.2. Are any of these goals new since your grant agreement? Yes Evaluation of the Cohort 3 CalVIP Grant Program Appendix C: Quarterly Progress Report Guide C-4 No [If yes] a. Please write out the goals that are new since your grant agreement. [text] Goal [1] [2.2.3. a. and b. will repeat based on the number of goals selected 2.2.1.] 2.2.3. Please provide the following information about Goal [1] and each of its objectives. a. Please write out Goal [1] and each of its objectives. For each objective, please indicate your plans for measuring completion of that objective. Leave any unused objective fields blank. Goal 1 [All fields in this table will autofill starting in the Q3 reporting period, based upon the answers provided in the prior reporting period.] Objective 1a [text] Please describe your plans for measuring completion [text] of Objective [1]a. Objective 1b Progress [text] Please describe your plans for measuring completion [text] of Objective [1]b. Objective 1c Progress [text] Please describe your plans for measuring completion [text] of Objective [1]c. Objective 1d Progress [text] Please describe your plans for measuring completion [text] of Objective [1]d. Objective 1e Progress [text] Please describe your plans for measuring completion [text] of Objective [1]e. Objective 1f Progress [text] Please describe your plans for measuring completion [text] of Objective [1]f. Evaluation of the Cohort 3 CalVIP Grant Program Appendix C: Quarterly Progress Report Guide C-5 b. Please provide progress updates for Goal [1] and each of the objectives for this goal that you filled in immediately above. Goal 1 Progress [text] Objective [1]a Progress [text] Objective [1]b Progress [text] Objective [1]c Progress [text] Objective [1]d Progress [text] Objective [1]e Progress [text] Objective [1]f Progress [text] Describe overall progress towards the stated goal [text] during the reporting period. Describe challenges in reaching the stated goal and all [text] associated objectives during the reporting period. If applicable, what steps did you implement to address [text] any challenges? Section 2.3: Project Inputs and Implementation 2.3.1. List all organizations and agencies that refer participants to your project. Also indicate the type of each organization. Self-referrals or referrals from family or friends will be captured further below. Name of Organizations that Refer to the Project Organization Type [All fields in this table will autofill starting in the Q3 [Org type dropdowns include: reporting period, based upon the answers provided CBO/Non-profit, K-12 Education, in the prior reporting period.] Higher Education, Health Care Provider, Police/Sheriff, Probation/Parole, Workforce System Agency, Other Public Agency, Other] [text] [drop down] [text] [drop down] [text] [drop down] [text] [drop down] [text] [drop down] [text] [drop down] Evaluation of the Cohort 3 CalVIP Grant Program Appendix C: Quarterly Progress Report Guide C-6 [text] [drop down] [text] [drop down] [text] [drop down] 2.3.2. In addition to the organizations listed above, list all organizations that have an agreement or formal commitment to support your project’s goals. Also, indicate the type of each organization and its role in the project. Partner Organization Name Organization Type Project Role [All fields in this table will autofill starting in the Q3 [Org type [Project role reporting period, based upon the answers provided dropdowns dropdowns in the prior reporting period.] include: CBO/Non- include: provides profit, K-12 services, advisor, Education, Higher evaluator, Education, Health technical Care Provider, assistance, other.] Police/Sheriff, Probation/Parole, Workforce System Agency, Other Public Agency, Other] [text] [drop down] [drop down] [text] [drop down] [drop down] [text] [drop down] [drop down] [text] [drop down] [drop down] [text] [drop down] [drop down] [text] [drop down] [drop down] [text] [drop down] [drop down] [text] [drop down] [drop down] [text] [drop down] [drop down] [text] [drop down] [drop down] [text] [drop down] [drop down] [text] [drop down] [drop down] [text] [drop down] [drop down] Evaluation of the Cohort 3 CalVIP Grant Program Appendix C: Quarterly Progress Report Guide C-7 2.3.3. For the reporting period, please indicate the implementation status of the following project activities using the scale provided. Also provide a description of your progress, noting any accomplishments, challenges, and plans for overcoming these challenges. • Not Started = the project has not yet focused on implementing this project component. • Planning Phase = the project has started preparations and plans to but has not yet begun to implement this component. • Implementation Started = the project has initiated implementing this component, but it is not fully developed or completed, or is need of refinement. • Complete/Established = the project component is fully in place/completed and supporting project goals or objectives. • N/A = does not apply to the project. a. Partnerships. What is the status of partnership agreements, including MOUs, subcontracts, etc., as well as any informal arrangements needed to deliver project services and complete project goals? Not Started Planning Phase Implementation Started Completed/Established N/A Describe progress: [text] b. Staffing and/or Volunteers. What is the status of hiring or securing arrangements for people for positions needed to deliver project services and complete project goals? Not Started Planning Phase Implementation Started Completed/Established N/A Describe progress: [text] c. Staff/Partner Training. What is the status of providing training to staff and/or partners needed to provide project services and complete project goals? Not Started Planning Phase Implementation Started Evaluation of the Cohort 3 CalVIP Grant Program Appendix C: Quarterly Progress Report Guide C-8 Completed/Established N/A Describe progress: [text] d. Identification, Outreach and Enrollment. What is the status of identifying, conducting outreach to, and enrolling participants into project services? Not Started Planning Phase Implementation Started Completed/Established N/A Describe progress: [text] e. Evidence-based practices. What is the status of identifying and implementing project interventions based on strategies known to achieve positive participant outcomes? Not Started Planning Phase Implementation Started Completed/Established N/A Describe progress: [text] f. Data Collection/Evaluation. What is the status of identifying and conducting systematic and ongoing data collection to measure participation and evaluation measures? Not Started Planning Phase Implementation Started Completed/Established N/A Describe progress: [text] g. Please describe any ways in which the COVID-19 pandemic has affected implementation. Evaluation of the Cohort 3 CalVIP Grant Program Appendix C: Quarterly Progress Report Guide C-9 [text] Section 3: Description of Grantee Services In this section you will clarify who can participate in the services provided by your grant and the nature of those services. For most grantees, the information in this section will stay the same throughout your grant. For the following questions, please describe each “program” that is funded by your CalVIP grant, also referred to as your CalVIP project. The definition of a program is a distinct service track within your project that has its own unduplicated enrollment and may have distinct eligibility criteria and/or sets of services that it provides to participants that are different from other programs. In other words, for a project to have multiple programs, participants must not cross between them and programs may be different from one another in other ways. Some grantees offer only one program as part of their projects. Those that offer multiple programs may do so in different ways. Grantees with multiple programs may offer them from within one or across many different organizations. Furthermore, this report is concerned with tracking information about programs where a participant is enrolled and expected to receive multiple services. In contrast, this report is not concerned with tracking participant information for services that are singular in nature such as one-time workshops or classes, street outreach services (on their own and not connected to more intensive services), or other types of limited contact where enrollment information typically is not available and there is no expectation of additional contact. While these limited involvement services may be important, this report is not tracking them. Given the definitions above, how many programs are part of your CalVIP-funded project? [Dropdown] 1 2 3 4 5 6 7 8 9 10 Evaluation of the Cohort 3 CalVIP Grant Program Appendix C: Quarterly Progress Report Guide C-10 Section 3.1: Program [1] [This section will replicate with questions for Program 2, Program 3 and so forth, depending on the number of programs selected immediately above.] 3.1.1. For Program [1] of your CalVIP-funded project, please list the name of the program, the organization that is primarily responsible for operating it (this organization should be listed in the partner section above or be your organization), and the enrollment goal (i.e., the number of participants you plan to enroll) for this program over the lifetime of the project. Name of Program Lead Operating Organization Enrollment Goal [text] [text] [number] 3.1.2. For Program [1] of your CalVIP-funded project, please provide the following information about program eligibility. a. What is the minimum age a person can be to [number] participate? Min Age (if none, enter “N/A”): b. What is the maximum age a person can be to [number] participate? Max Age (if none, enter “N/A”): c. Describe the process used to determine a [text] participant’s eligibility. d. Describe any other eligibility requirements [text] participants must meet to be in this program. 3.1.3. For Program [1] of your CalVIP-funded project, are participants formally assessed? Yes No [If yes] a. What tool and how is it used? [text] 3.1.4. Does Program [1] of your CalVIP-funded project use a validated risk assessment tool? Yes No [If yes] a. What tool and how is it used? [text] Evaluation of the Cohort 3 CalVIP Grant Program Appendix C: Quarterly Progress Report Guide C-11 3.1.5. Please select the types of services provided to participants as part of Program [1] (select all that apply). We recognize that grantees have many ways of describing service components. Please select the services from the list that best describe those provided by Program [1] of your project. Do not include services that participants may receive from other organizations or programs that are made through “soft” referrals and not tracked as part of your program. Please note that you will need to report on participant activity in each category selected (for the project as a whole) in question 4.3.1. Case Management Life skills/Healthy Choices/Family Management Conflict Resolution/Anger Mentoring Management/Violence Avoidance/Social Motivational Interviewing Emotional Learning Training Pro-social/Leadership Skills and Activities Cognitive Behavioral Therapy Substance Abuse Treatment Culturally Responsive/Culturally Competent Supportive Services (including any supports Services provided or tracked through the program that aid De-escalation/Violence participants in achieving other program goals) Interruption/Intervention Services Tattoo Removal Diversion Trauma-Informed/Restorative Justice Practices Education: K-12/GED Vocational Skills Training/Apprenticeships Education: Postsecondary Work/Career Readiness Training Health: Mental/Behavioral Other: [text] Health: Physical Health Other: [text] Housing Other: [text] Individual Development Planning/Life, Other: [text] Education, or Career Planning 3.1.6. Describe the measurable milestone(s) Program [1] uses to determine when a participant has successfully completed services? Please describe what criteria you use for determining completion? Note that you will use this definition for identifying participants who “successfully complete” your project in Section 4. [Text] Section 4: Enrollments, Services Received, and Exits This section is for you to report how many unique/unduplicated individuals enrolled in your project, received services, and/or exited during the reporting period. In this section, you should record this information for your project overall and not for individual programs. Evaluation of the Cohort 3 CalVIP Grant Program Appendix C: Quarterly Progress Report Guide C-12 Section 4.1: Enrollment and Reentry 4.1.1. For the reporting period, please record the total number of participants who enrolled in the project and the number of participants who enrolled in the project in each enrollment category listed below. The “1st Time Enrolled” column should include the number of participants who enrolled in project services during the reporting period and who had never previously enrolled in project services since the start of the grant. The “2nd Time Enrolled” and “3(+) Time Enrolled” columns should include the number of participants who enrolled a second or third (or greater) time during the reporting period, after having previously been exited at any time during the grant. In other words, to be included in the “2nd Time Enrolled” column, a participant must have previously been recorded once in the “Exit” section (Section 4.4) at any point during the grant period. In order to be included in the “3(+) Time Entry” column, a participant previously must have been recorded in the “Exit” section two or more times at any point during the grant period. It is possible for a person to exit and reenter in the same reporting period. The total of the numbers you enter in each column for items b., c., and d. should equal the value you have listed in each respective column of item a. For example, if you list “10” in the “1st time enrolled” column in item a., the total number of people in the “1st time enrolled” column in item c. should also equal 10. a. Total Number of Participants Enrolled 1st Time 2nd Time 3(+) Time Enrolled Enrolled Enrolled Total Enrollment b. Please Indicate the Referral Source and Number of Participants Enrolled by Referral Source Referral Source Name 1st Time 2nd Time 3(+) Time Enrolled Enrolled Enrolled [Please write in any referral agencies based on the list provided in 2.3.1. This section will pre-populated starting in Quarter 3. ] [text] [text] [text] [text] Evaluation of the Cohort 3 CalVIP Grant Program Appendix C: Quarterly Progress Report Guide C-13 [text] [text] Family/Friend Unknown/Did not collect c. Number of Participants Enrolled by Participation Status 1st Time 2nd Time 3(+) Time Enrolled Enrolled Enrolled Mandated Voluntary Unknown/Did not collect d. Number of Participants Enrolled by Point of Diversion/System Involvement 1st Time 2nd Time 3(+) Time Enrolled Enrolled Enrolled No contact with law enforcement Informal contact with law enforcement Pre-adjudication Post-adjudication, in community, no supervision Post-adjudication, in community, under supervision Post-adjudication, in-facility Unknown/Did not collect Section 4.2: Participant Demographics at Enrollment 4.2.1. For the reporting period, please record the total number of participants who enrolled and the number of participants in each demographic category at the point of enrollment. The total of the numbers you enter in each column for items b., c., and d. should equal the value you have listed in each respective column of item a. For example, if you list “10” in the “1st time enrolled” column in item a., the total number of people in the “1st time enrolled” column in item c. should also equal 10. a. Total Number of Participants Enrolled Evaluation of the Cohort 3 CalVIP Grant Program Appendix C: Quarterly Progress Report Guide C-14 1st Time 2nd Time 3(+) Time Enrolled Enrolled Enrolled Total Enrollment b. Number of Participants by Age at Enrollment 1st Time 2nd Time 3(+) Time Enrolled Enrolled Enrolled 12 years or younger 13-17 years old 18-24 years old 25-34 years old 35 years or older Unknown/Did not collect c. Number of Participants by Gender at Enrollment 1st Time 2nd Time 3(+) Time Enrolled Enrolled Enrolled Female Male Non-binary/3rd Gender Other Prefer not to state Unknown/Did not collect d. Number of Participants by Race/Ethnicity at Enrollment Please list multi-racial/ethnic participants under the multiple race or ethnicity row. 1st Time 2nd Time 3(+) Time Enrolled Enrolled Enrolled American Indian/Alaska Native Asian: Chinese Asian: Japanese Asian: Filipino Evaluation of the Cohort 3 CalVIP Grant Program Appendix C: Quarterly Progress Report Guide C-15 Asian: Korean Asian: Vietnamese Asian: Indian Asian: Laotian Asian: Cambodian Asian: Other Black or African American Hispanic or Latino Middle Eastern/North African NH/PI: Native Hawaiian NH/PI: Guamanian NH/PI: Samoan NH/PI: Other White Other Multiple race or ethnicity Declined to state Unknown/Did not collect e. Number of Participants in Assessed Risk Groups at Enrollment 1st Time 2nd Time 3(+) Time Enrolled Enrolled Enrolled Low Medium Medium-high High Other: [text] Unknown/Did not collect f. Number of Participants of Different Education Statuses at Enrollment Evaluation of the Cohort 3 CalVIP Grant Program Appendix C: Quarterly Progress Report Guide C-16 1st Time 2nd Time 3(+) Time Enrolled Enrolled Enrolled Enrolled in elementary or middle school Enrolled in high school Enrolled in training program Enrolled in 2/4-year college Not Enrolled/Highest Completed: Elementary or Middle school Not Enrolled/Highest Completed: Some high school Not Enrolled/Highest Completed: High school diploma Not Enrolled/Highest Completed: GED Not Enrolled/Highest Completed: Some college or vocational training Not Enrolled/Highest Completed: Certificate or college degree Unknown/Did not collect g. Number of Participants with Different Employment Statuses at Enrollment 1st Time 2nd Time 3(+) Time Enrolled Enrolled Enrolled Employed: Not looking for other/additional employment Employed: Looking for other/additional employment Unemployed: Looking for employment Unemployed: Not looking for employment/student Unemployed: Not looking for employment/other reason Unknown/Did not collect h. Number of Participants with Different Housing Statuses at Enrollment 1st Time 2nd Time 3(+) Time Enrolled Enrolled Enrolled Evaluation of the Cohort 3 CalVIP Grant Program Appendix C: Quarterly Progress Report Guide C-17 Living independently (own or rent) Living with parent/s Living with relatives (not foster care) Living with a friend Living in foster care Living in out-of-home care through child welfare or probation Living in unstable housing (car, street, abandoned building, tent, etc.) Other Unknown/Did not collect Section 4.3: Participation in Services 4.3.1. Please record the number of unduplicated individuals that participated in each type of service offered across the project during the reporting period. In the following table, for each service category provided under the grant, please indicate the total number of unique individuals that received those services during the reporting period. Participants exiting during the reporting period should be included in the total so long as they received the selected services at any time during the reporting period. The same participant can be reported across different services but should only be counted one time for any given service. The purpose of this table is to provide a ‘snap-shot’ of participation by service type during the reporting period. [The following list of services will prepopulate based on the answers selected in 3.1.5] Unduplicated Participant Count [Case Management] [Conflict Resolution/Anger Management/Violence Avoidance/Social Emotional Learning Training] [Cognitive Behavioral Therapy] [Culturally Responsive/Culturally Competent Services] [De-escalation/Violence Interruption/Intervention Services] [Diversion] [Education: K-12/GED] [Education: Postsecondary] Evaluation of the Cohort 3 CalVIP Grant Program Appendix C: Quarterly Progress Report Guide C-18 [Health: Mental/Behavioral] [Health: Physical Health] [Housing] [Individual Development Planning/Life, Education, or Career Planning] [Life skills/Healthy Choices/Family Management] [Mentoring] [Motivational Interviewing] [Pro-social/Leadership Skills and Activities] [Substance Abuse Treatment] [Supportive Services] [Tattoo Removal] [Trauma-Informed/Restorative Justice Practices] [Vocational Skills Training/Apprenticeships] [Work/Career Readiness training] Other: [text] Other: [text] Other: [text] Other: [text] 4.3.2. Please record the number of individuals that participated in each type of service offered across the project over the lifetime of the grant. In the following table, for each service category provided under the grant, please indicate the total number of unique individuals that received those services over the lifetime of the grant. The same participant can be reported across different services but should only be counted one time for any given service. For example, if a participant received Service X in quarter 3 and quarter 5, then this individual should only be counted once for Service X in this table. [The following list of services will prepopulate based on the answers selected in 3.1.5] Unduplicated Participant Count [Case Management] [Conflict Resolution/Anger Management/Violence Avoidance/Social Emotional Learning Training] [Cognitive Behavioral Therapy] [Culturally Responsive/Culturally Competent Services] Evaluation of the Cohort 3 CalVIP Grant Program Appendix C: Quarterly Progress Report Guide C-19 [De-escalation/Violence Interruption/Intervention Services] [Diversion] [Education: K-12/GED] [Education: Postsecondary] [Health: Mental/Behavioral] [Health: Physical Health] [Housing] [Individual Development Planning/Life, Education, or Career Planning] [Life skills/Healthy Choices/Family Management] [Mentoring] [Motivational Interviewing] [Pro-social/Leadership Skills and Activities] [Substance Abuse Treatment] [Supportive Services] [Tattoo Removal] [Trauma-Informed/Restorative Justice Practices] [Vocational Skills Training/Apprenticeships] [Work/Career Readiness training] Other: [text] Other: [text] Other: [text] Other: [text] Section 4.4: Participant Exits 4.4.1. For the reporting period, please record the total number of participants who exited the project and their reason for exit. Similar to what was previously described for enrollment, the “1st Time Exits” column should include the number of participants who exited the project during the reporting period and who had never previously exited the project, since the start of the grant. The “2nd Time Exits” and “3(+) Time Exits” columns should include the number of participants who exited a second or third (or greater) time during the reporting period, after having previously exited at any time during the grant. In other words, to be included in the “2nd Time Exits” column, a participant must have previously exited once at any point during the grant period. In order to be included Evaluation of the Cohort 3 CalVIP Grant Program Appendix C: Quarterly Progress Report Guide C-20 in the “3(+) Time Exit” column, a participant previously must have exited two or more times at any point during the grant period. It is possible for a person to exit multiple times in the same reporting period. The total of the numbers you enter in each column for items b., c., and d. should equal the value you have listed in each respective column of item a. For example, if you list “10” in the “1st time exits” column in item a., the total number of people in the “1st time exits” column in item c. should also equal 10. a. Total Number of Participants at Exit 1st Time 2nd Time 3(+) Time Exits Exits Exits Total Participants Exited b. Primary Reason for Exit [For the “successful completion” field, please refer to the definition provided in 3.1.6.] 1st Time 2nd Time 3(+) Time Exits Exits Exits Successful completion Dropped out/lost contact Non-compliant (asked to leave) Arrest/incarceration Services determined not appropriate Other Unknown/Did not collect Section 4.5: Exiting Participant Demographics 4.5.1. For the reporting period, please record the demographic categories for participants exiting the project. For exiting participants, please categorize individuals based on the demographic information collected at enrollment. For example, if a participant enrolled into project services when they were 17 years old, and exited when they were 20 years old, you would still count them in the “13-17 years old” line. Also, the total of the numbers you enter in each column for items b., c., and d. should equal the value you have listed in each respective column of item a. For example, if you list “10” in the Evaluation of the Cohort 3 CalVIP Grant Program Appendix C: Quarterly Progress Report Guide C-21 “1st time exits” column in item a., the total number of people in the “1st time exits” column in item c. should also equal 10. a. Total Number of Participants at Exit 1st Time 2nd Time 3(+) Time Exits Exits Exits Total Participants Exited b. Number of Exiting Participants by Age 1st Time 2nd Time 3(+) Time Exits Exits Exits 12 years or younger 13-17 years old 18-24 years old 25-34 years old 35 years or older Unknown/Did not collect c. Number of Exiting Participants by Gender 1st Time 2nd Time 3(+) Time Exits Exits Exits Female Male Non-binary/3rd Gender Other Prefer not to state Unknown/Did not collect d. Number of Exiting Participants by Race/Ethnicity Please list multi-racial/ethnic participants under the multiple race or ethnicity row. 1st Time 2nd Time 3(+) Time Exits Exits Exits American Indian/Alaska Native Asian: Chinese Asian: Japanese Asian: Filipino Asian: Korean Evaluation of the Cohort 3 CalVIP Grant Program Appendix C: Quarterly Progress Report Guide C-22 Asian: Vietnamese Asian: Indian Asian: Laotian Asian: Cambodian Asian: Other Black or African American Hispanic or Latino Middle Eastern/North African NH/PI: Native Hawaiian NH/PI: Guamanian NH/PI: Samoan NH/PI: Other White Other Multiple race or ethnicity Declined to state Unknown/Did not collect Section 5: Participant Outcomes 5.1.1. In the following table, please list each outcome your project measures, how you would categorize this outcome based on the list provided, and the method by which you measure whether a positive change has occurred. An outcome refers to a change that happens to participants as compared to some prior state. The BSCC is interested, in particular, in measurable, positive changes that occur to participants. The prior state in question could be the participant’s status prior to enrollment, at enrollment, or at some point while receiving services (note that “positive” changes for justice system involvement typically involve lower scores, and fewer incidents rather than increases). Some examples of positive outcomes that your program may try to capture might include placement in employment, increased earnings, fewer altercations at school, completion of educational credit or a certificate or degree, indications of improved physical or mental health, more stable housing, improved connections to trusting adults, etc. Please note that you should only include measurable changes. Some possible ways to measure changes involve using a validated (or non-validated) assessment tool, examining health, education, or criminal justice system records, observing official documents (e.g., a pay stubs, Evaluation of the Cohort 3 CalVIP Grant Program Appendix C: Quarterly Progress Report Guide C-23 transcripts, expert observations), or gathering self-reported information through a participant survey, questionnaire, or follow-up calls. For the Outcome Categories you should select, the full description of each one is as follows (abridged versions appear in the dropdown menu): Crime 1: Reduced Violent Activity/Gang Involvement Crime 2: Reduced Arrests/Charges Crime 3: Reduced Convictions/Sustained Petitions Crime 4: Reduced Incarceration Employment 1: Found Employment Employment 2: Retained Job/Promoted Employment 3: Increased Earnings Employment 4: Improved Job Satisfaction K-12 Education 1: Enrollment/Credit Completion K-12 Education 2: Diploma/GED Completion K-12 Education 3: Improved Attendance K-12 Education 4: Reduced Disciplinary Actions Postsecondary Education 1: Enrollment/Credit Completion Postsecondary Education 2: Certificate Completion Postsecondary Education 3: Degree Completion Health 1: Improved Mental Health Health 2: Improved Physical Health Health 3: Reduced Drug/Alcohol Use Health 4: Reduced Victimization Housing 1: More Stably Housed Psychosocial 1: Feeling More Engaged with Society Psychosocial 2: Feeling More Connected to Others Psychosocial 3: Feeling Safer in the Community Psychosocial 4: Feeling Better Able to Make Positive Life Choices Psychosocial 5: Increased Feeling of Basic Needs Being Met Skills/Knowledge 1: Improved Justice System Awareness Skills/Knowledge 2: Improved Conflict Resolution/Anger Management/Violence Avoidance/ Social Emotional Skills Skills/Knowledge 3: Improved Life/Healthy Choices/Family Management Skills Evaluation of the Cohort 3 CalVIP Grant Program Appendix C: Quarterly Progress Report Guide C-24 Skills/Knowledge 4: Improved Work/Career Readiness Project Outcome Outcome Category Method of Measuring Positive Change [Text] [Dropdown list from [For this field, please list each outcome categories shown above] the project measures as the project describes them.] [Text] [Dropdown list] [Text] [Text] [Dropdown list] [Text] [Text] [Dropdown list] [Text] [Text] [Dropdown list] [Text] [Text] [Dropdown list] [Text] [Text] [Dropdown list] [Text] [Text] [Dropdown list] [Text] [Text] [Dropdown list] [Text] [Text] [Dropdown list] [Text] 5.1.2. In the following table, for each outcome your project measures, please report on the number of participants who exited the project during this reporting period AND demonstrated a positive change for the given outcome. The second column should include an unduplicated count of individuals who exited your project during the reporting period, but only if they experienced positive changes. An individual should appear no more than once in any given row, but may appear in more than one row. In other words, the total possible number of individuals who could be included in each row of the table would be the total number of individuals who exited the project during this reporting period. Outcome Number of Individuals Exiting with a Positive Outcome for this Reporting Period [This column will auto-populate [number] based on the outcomes listed in 5.1.1 starting in quarter 3.] [Text] [number] [Text] [number] [Text] [number] [Text] [number] [Text] [number] [Text] [number] Evaluation of the Cohort 3 CalVIP Grant Program Appendix C: Quarterly Progress Report Guide C-25 [Text] [number] [Text] [number] [Text] [number] 5.1.3. In the following table, for each outcome your project measures, please report on the number of participants who exited the project at any point of the grant AND demonstrated a positive change for the given outcome. The second column should include an unduplicated count of individuals who exited your project at any point during the lifetime of the grant, but only if they experienced positive changes. An individual should appear no more than once in any given row, but may appear in more than one row. In other words, by quarter 11, the total possible number of individuals who could be included in each row of the table would be the total number of individuals ever exited from the project over the lifetime of the grant. Outcome Number of Individuals Exiting with a Positive Outcome During the Grant Period [This column will auto-populate [number] based on the outcomes listed in 5.1.1 starting in quarter 3.] [Text] [number] [Text] [number] [Text] [number] [Text] [number] [Text] [number] [Text] [number] [Text] [number] [Text] [number] [Text] [number] Evaluation of the Cohort 3 CalVIP Grant Program Appendix D: Interview Protocol D-1 Appendix D: Interview Protocol Introductory information Grantee: Name of respondent(s): Date of interview: Interviewer: Notetaker: Thank you for taking the time to speak to us today. My name is [name] and my colleague is [name]. We are from Social Policy Research Associates, the organization conducting the statewide evaluation of the CalVIP cohort 3 program. As you may know, we’ve been learning about grantee programs through our various evaluation activities, including reviewing your LEPs and QPR data and through our collaborations with you or your staff at the CalVIP convenings and Project Director Meetings. However, these interviews we’re conducting give us a chance to learn about your program and many other CalVIP cohort 3 programs in greater detail. This call should take about an hour. We’ll be asking you a number of questions about the program you’ve put in place under your CalVIP cohort 3 grant and the success and challenges you’ve encountered in implementing it. We will use the information gathered in these interviews alongside the other data we are gathering for the evaluation to more effectively tell the story of the programs cohort 3 has put in place. While we may use quotes and examples you share in this interview, we won’t link them to your name or the name of your organization within our report. We’d like you to feel comfortable sharing challenges as well as successes since that will help to better convey the story of how these programs evolved and changed and potentially share lessons with others implementing similar programs. Do you have any questions before we begin? Is it okay if I record the conversation for my notes? These notes or the recording won’t be shared with anyone outside of the SPR study team. Evaluation of the Cohort 3 CalVIP Grant Program Appendix D: Interview Protocol D-2 I. Introductions I want to start by asking you about your program and how you designed and planned it. 1.1. Please introduce yourself and your role on your CalVIP cohort 3 funded program. 1.2 After reading your program materials and reports to the BSCC, I understand your CalVIP cohort 3 program focuses on [provide a brief summary of their program from program materials]. Does that sound about right? Is there anything you’d like to clarify? II. Partnerships I’d now like to ask you about your partners. I see from your QPRs that you have the following partner organizations (read the partner organizations). Partner Organization Organization Project Role Notes Name Type 2.1. What have been the main challenges in working with these partners and why? You don’t need to name the partner specifically. 2.2. What have been successful approaches to working with partners? III. The COVID-19 Pandemic Impact I next want to ask you about how the COVID-19 pandemic affected your program. 3.1 Did you have to make changes to your originally proposed program design due to the pandemic? 3.1.1 If so, what changes did you have to make and why? 3.2 Will you incorporate any of these changes into your work permanently? If so, which ones and why? IV. Enrollment and Service Delivery Let’s transition to talking about your enrollment and the services you deliver. We had noted a few specific implementation challenges and solutions in your reports, Appendix D: Interview Protocol Appendix D: Interview Protocol D-3 related to enrollment and services that interested us. Can you tell me more about that? Challenges Related to Solutions Enrollment and Service Delivery Notes 4.1 From your QPRs, you seem to be somewhat behind/mostly on target/exceeding your enrollment goal of <X> participants. 4.1.1 (If behind) Why is that and how are you trying to address this issue? 4.1.2 (If on target/exceeding) What strategies or approaches have helped you meet/exceed this goal? 4.2 What has been the most challenging thing about engaging your participants in program services? 4.3 What has been the most effective ways to engage and keep participants engaged in services? 4.4. A few elements of your program that seemed particularly interesting to us in reviewing your program included [fill in elements here]? How did you come to include these and why? What has worked well about these program elements and what has been challenging to implement? Notable Approaches/Promising Notes Practices V. Participant Outcomes Let’s move on to discussing your participant outcomes. From your QPRs, I see that your program is hoping to achieve the following outcomes. Appendix D: Interview Protocol Appendix D: Interview Protocol D-4 Method of Measuring Positive Outcome Outcome Category Change 1 2 3 4 5 Notable Approaches to Notes Measuring Outcomes e.g., Partnership with outside agency to collect data 5.1. Thinking about your outcomes, are there outcomes you are finding it particularly easy or difficult for your participants to meet? What is it about the participants or the program that may be contributing to this? What would you do differently next time? 5.2. What types of data have made it easy to measure these outcomes? What types of data or other evaluation support have been difficult to get? V. Lessons Learned Finally, let’s close out our meeting with some broader reflections. 6.1. Thinking about implementing your program, what have been the biggest successes? Probe: What about the program contributed to this. 6.2. What have been the biggest challenges in setting up and operating your program? Probes: establishing partners, budgeting, contracting issues, establishing program models, conducting data collection or reporting, etc. 6.2. What would you recommend to other organizations implementing similar programs? Appendix D: Interview Protocol