BSCC
Board of State and Community Corrections
Read the report at Board of State and Community Corrections ↗
Transition-Age Youth
Pilot Program
Evaluation Report #2
September 2021
Report prepared by Evident Change for the Board of State and
Community Corrections pursuant to Penal Code sec. 1000.7
CONTENTS
Executive Summary ......................................................................................................... i
Introduction ..................................................................................................................... 1
Evaluation Design ........................................................................................................... 2
Findings ........................................................................................................................... 6
Conclusions and Recommendations ............................................................................. 21
Questions or comments regarding this report can be directed to Dr. Erin Espinosa,
Director of Research, at eespinosa@evidentchange.org
EXECUTIVE SUMMARY
Introduction
Senate Bill (SB) 1004 (Ch. 865, Statutes of 2016) and SB 1106 (Ch. 1007, Statutes of
2018)1 provided six counties (Alameda, Butte, Napa, Nevada, Santa Clara, and Ventura)
the opportunity to implement a transition-age youth (TAY) pilot program with deferred
entry of judgment in juvenile halls for young adult offenders. 2 Behavioral and
psychoneurological research indicating that young adults (between the ages of 18 and
24) may developmentally benefit from rehabilitative rather than punitive approaches to
corrections3 initiated the legislation.4
To be eligible for the TAY program, potential participants must meet statutory criteria,
including being in the age range (between the ages of 18 and 24 at the time of the
qualifying offense), offense type (charged with a felony offense, other than a violent,
serious, or sexual felony), prior record (no prior conviction for a violent, serious, or sexual
felony offense), suitability for the program, and otherwise would have served time in
custody in a county jail. In addition, candidates must consent to participate in the program
and agree to waive their right to a trial or hearing, plead guilty to the charge(s), and waive
time for the pronouncement of the judgment. TAY program participants engage in
services such as cognitive behavioral therapy and age-appropriate educational and
vocational programming and participate in community supervision. Upon a participant’s
successful completion of the program, the court will dismiss the participant’s criminal
charge(s) associated with this sentence.
1 SB 1106 amended SB 1004 and extended the date of authorization to establish a pilot program to
January 1, 2022, and expanded the scope of the pilot to include Ventura County, which ultimately chose
not to participate in the TAY program and is not included in this or the previous report.
2 A third law related to the TAY program, AB 1390, was enacted in July 2019. AB 1390 expanded the
program eligibility criteria to include young adults who were between the ages of 21 and 24 at the time of
their arresting offense. Program participation by an individual in this age group must be approved locally
by the jurisdiction’s multidisciplinary team established for this project.
3 Cauffman, E. (2012). Aligning justice system processing with developmental science. Criminology and
Public Policy, 11(4), 751–758. doi:10.1111/j.1745-9133.2012. 00847.x; Farrington, D. P., Loeber, R., &
Howell, J. C. (2012). Young adult offenders: The need for more effective legislative options and justice
processing. Criminology and Public Policy, 11(4), 729–750. doi: 10.1111/j.1745-9133.2012. 00842.x;
Scott, E., Bonnie, R. J., & Steinberg, L. (2016). Young adulthood as a transitional legal category: Science,
social change, and justice policy. Fordham Law Review, 85(2), 641–666; Steinberg, L. (2012). Should the
science of adolescent brain development inform public policy? Issues in Science and Technology, 28(3),
70–76.
4 Senate Committee on Public Safety. (2016). SB 1004 young adults: deferred entry of judgment pilot
program. Retrieved from
http://leginfo.legislature.ca.gov/faces/billAnalysisClient.xhtml?bill_id=201520160SB1004.
i
Evaluation Design
The legislation authorizing TAY programs requires the Board of State and Community
Corrections (BSCC) to conduct an evaluation of the programs’ impact and effectiveness
and to submit a comprehensive evaluation report to the Assembly and Senate
Committees on Public Safety. In May 2020, the BSCC contracted with Evident Change
to conduct this evaluation.
Building off the original evaluation plan developed by the BSCC, the Evident Change
evaluation team used a mixed-method and participatory process–based approach to
examine the TAY programs’ impact and effect in three primary areas (outlined in SB 1004):
(1) sentencing, especially opportunities for community supervision; (2) presence of the
program in the juvenile facility; and (3) program completion, skills improvements, and
recidivism. The team also conducted a qualitative process evaluation that explored how
TAY programs were structured, implemented, and operated; program challenges and
successes; and program staff’s knowledge, perceptions, and recommendations for
improvement.
This report is the second of two reports submitted by Evident Change to the BSCC. The
previous report summarized findings based on available data through mid-December
2020, with an emphasis on qualitative and descriptive findings. This report provides an
overview of the quantitative analytics using data collected from the California Department
of Justice (DOJ) to specifically target findings and conclusions related to sentencing
(Evaluation Question One; EQ1) and recidivism (Evaluation Question Three; EQ3).
Limitations of This Report
The first limitation to the evaluation design comes in the identification and development
of comparison groups to the TAY participant group. The evaluation team recommends
the use of propensity score matching (PSM) when working with administrative data for
program evaluation that includes outcome analyses. PSM is a sampling approach that
results in statistically equivalent comparison groups (e.g., treatment and non-treatment
groups, participant and non-participant groups) and could be used in evaluations such as
this one in lieu of randomization. However, TAY participants from Santa Clara County
accounted for a larger proportion of the individuals in that evaluation group. For PSM to
be reliably used in this evaluation, additional TAY participants from counties other than
Santa Clara County are necessary.5 Due to these factors, the evaluation team chose not
to use PSM for the evaluation design and instead utilized a nonequivalent control group
5 Andrilon, A., Piracchio, R., & Cheveret, S. (2020). Performance of propensity matching to estimate causal
effects in small samples. Statistical Methods in Medical Research, 29(3), 644–658.
https://doi.org/10.1177/0962280219887196; Cenzer, I., Boscardin, J., & Berger, K. (2020). Performance of
matching methods in studies of rare diseases: A simulation study. Intractable Rare Diseases Research,
9(2) 79–88. doi: 10.5582/irdr.2020.01016
ii
design that examined similarities and differences between the TAY participant group and
two comparison groups (pre- and post-TAY comparison groups).
Another limitation to the evaluation design relates to the time point for determining
recidivism-based outcomes. During the evaluation period, the average length of
enrollment in the TAY pilot across all counties was 12.9 months. Due to the variability in
the length of operations of the TAY pilot across all counties, variability in the average
length of enrollment, and the relatively short timeframe of operation of the TAY pilot, the
examination of differences in recidivism between the TAY participant group and the
comparison groups was limited to six months from program exit.
Therefore, the results included in this report should be reviewed with caution, as the data
collection and analytics were limited due to (1) limited variables in the DOJ data set, and
(2) limited timeframe for sample selection. Further analysis should include a longitudinal
study design that incorporates both case processing (e.g., arrest records, petitions) and
sentencing outcome data collection and analysis.
Findings
For the ease of understanding the evaluation approach, the findings (in both this
Executive Summary and in the full report) are presented with EQ3 first followed by EQ1,
and then a short overview of the overlap between these findings and the qualitative
information included in the first report. The key findings of Report #2 of the TAY evaluation
are as follows:
Recidivism-EQ3:
Evaluation Question: What is the program’s effectiveness with respect to program
participants and a comparison group?
Data from the California DOJ allowed for the creation of two comparison groups resulting
in three distinct evaluation groups: 1.TAY participant, 2. Post-TAY participant comparison
group, and 3. Pre-TAY comparison group (see full report for description of group
development). The TAY participant group had slightly fewer previous arrests than the two
comparison groups and significantly fewer prior arrests for felonies than the comparison
groups. Specifically, Latinx/Hispanic people in the TAY participant group had a lower
average for prior arrests than both comparison groups. While previous arrest rates were
also higher for other races/ethnicities in the comparison groups than the TAY participant
group, the difference between groups was not statistically significant. Other than those
differences, the groups were comparably similar by race/ethnicity, gender, and age.
There were statistically significant differences between the evaluation groups. Specifically,
new arrests within six months of program exit accounted for a smaller proportion of
outcomes for TAY participants than for the pre-TAY comparison group. For new violations,
iii
there were significant differences between TAY participants and the post-TAY
comparison group, with TAY participants experiencing 13% fewer violations within six
months of exit than the post-TAY comparison group. In addition, there were statistically
significant differences in violations between the post- and pre-TAY comparison groups,
with the post-TAY group experiencing a larger proportion of violations within six months.
There were no statistically significant differences between groups for new convictions. In
addition, a stepwise binomial regression model was developed to examine the influence
of being a TAY participant on future arrests in comparison to other predictor variables
(e.g., race/ethnicity). The final model indicated that being a participant in the TAY pilot
reduced the likelihood of rearrest in six months (p. <. 01), while being White increased
the likelihood of rearrest in six months.
An examination of TAY program completion (e.g., successful versus unsuccessful)
revealed a statistically significant difference between new arrests, new convictions, and
new violations for people who were successful in completing TAY programming
compared with people who were identified as unsuccessful. Over 60% of those who did
not successfully complete the program were rearrested within six months compared to
only 9% of those who successfully completed it.
Sentencing - EQ1:
Evaluation Question: What is the TAY program’s impact on sentencing, especially
opportunities for community supervision?
Due to the variability in how individuals were identified and enrolled in TAY across the
participating counties (as detailed in the process evaluation results), the best fit for the
study design to evaluate the impact of the TAY pilot on sentencing was the creation of a
TAY participant sample and population-based comparison groups anchored in two time
points: prior to each county’s implementation of TAY and after TAY implementation. The
process included the identification and examination of both the individuals and the arrest
dispositions related to an arrest or arrest event. This process resulted in the development
of three distinct evaluation groups (see full report for a description of the groups): 1. TAY
participant, 2. after TAY, and 3. before TAY.
Sentencing differences were statistically significant between the TAY participant group
and the “after TAY” group for jail or prison only, jail and probation, diversion/deferral, and
acquitted/dismissed. When comparing the participant group to the “before TAY” group,
differences in sentence proportions were statistically significant for jail and probation,
diversion/deferral, and acquitted/dismissed. Specifically, the TAY participant group
experienced a significantly smaller proportion of jail and probation sentences (7.9%) than
both the “before” (72.4%) and “after” (62.8%) TAY comparison groups. It should be noted,
however, while not being statistically significant, TAY participants who received a jail
sentence had longer sentences than individuals in the comparison groups.
iv
Connection to Process Evaluation Outcomes:
Net Widening
The quantitative data analysis for EQ 1 suggests that staff reports during structured
interviews related to the variation in enrollment and eligibility criteria can be preliminarily
confirmed. While the data do not indicate a definitive impact of net widening due to the
TAY legislation (as seen in the variation in sentence dispositions), the data indicate that
discretion within the counties as described in staff interviews and surveys allowed for
enrollment of participants who may not have been arrested for an eligible offense.
Considerations for Juvenile Hall Component
In interviews, some staff indicated that the juvenile hall component of the TAY pilot was
no longer in use. Staff indicated that ending that portion of the program was due to factors
such as the realignment of the Department of Juvenile Justice and/or COVID-19
responses, while others indicated it should remain and that the length of time in juvenile
hall for TAY participants should be extended.
The outcome analysis for EQ3 found no statistical significance related to successful
completion of the TAY program for those who spent time in juvenile hall compared to
participants who did not spend time in juvenile hall. It should be noted that a larger
proportion of participants who did not receive the in-custody portion of the program had
successful TAY program completions compared with those who spent some time in
juvenile hall (e.g., 63.5% of those who experienced an in-custody stay were successful
compared with 75.0% of those without a juvenile hall stay).
Future Evaluation Considerations
To examine the impact of the TAY pilot more extensively, future evaluations should
incorporate an integrated cross-sectional and longitudinal study design. The qualitative
data collection should incorporate case processing, participant perspectives, stakeholder
perspectives, and other social artifacts (e.g., petitions, written disposition
recommendations). This approach would allow for a more comprehensive examination of
both the formal and informal influences of TAY programming inclusive of community
contextual considerations (e.g., other local diversion programs or services) that may be
correlated to sentencing outcomes related to TAY programming.
Future evaluations may also consider the benefits of a larger TAY participant group. A
larger TAY participant group will increase the statistical influence of counties with less
TAY participation than others and ultimately allow for the use of PSM to identify and
develop comparison groups more accurately. It should be noted that one of the greatest
challenges with evaluation research is in identifying equivalent comparison groups. In the
development of evidence-based programs and interventions, primarily from healthcare
v
research, randomized controlled trials (RCTs) have been considered the gold standard in
controlling for sample variability between the study or treatment group and comparison or
control groups. 6 Future research should consider the development of statistically
equivalent groups via PSM, as 1. randomization is extremely difficult for studies of
treatment services, and 2. supports of individuals involved with court systems are
extremely expensive to conduct and often difficult to complete in social service systems.7
6 Grossman, J., & Mackenzie, F. (2005). The randomized controlled trial: Gold standard, or merely standard?
Perspectives in Biology and Medicine, 48(4), 516–534.
7 Mezey, G., Robinson, F., Campbell, R., Gillard, S., Macdonald, G., Meyer, D., Bonell, C., & White, S.,
(2015). Challenges to undertaking randomised trials with looked after children in social care settings. Trials,
16(206). https://doi.org/10.1186/s13063-015-0708-z
vi
INTRODUCTION
Senate Bill (SB) 1004 (Ch. 865, Statutes of 2016) and SB 1106 (Ch. 1007, Statutes of
2018)8 provided six counties (Alameda, Butte, Napa, Nevada, Santa Clara, and Ventura)
the opportunity to implement a transition-age youth (TAY) pilot program with deferred
entry of judgment in juvenile halls for young adult offenders. 9 Behavioral and
psychoneurological research indicating that young adults (between the ages of 18 and
24) may developmentally benefit from rehabilitative rather than punitive approaches to
corrections10 initiated the legislation.11
To be eligible for the TAY program, potential participants must meet the following
requirements, as stated in SB 1004.
1. Must be between the ages of 18 and 20 at the time of the offense (note: AB 1390,
enacted in July 2019, expanded the age range to include young adults between the
ages of 21 and 24 at the time of the offense).
2. Must be found suitable for the program using a risk assessment instrument.
3. Must be found able to benefit from services generally reserved for delinquents.
4. Must meet the rules of the juvenile hall developed in accordance with applicable
regulations set forth in Title 15 of the California Code of Regulations.
5. Must be charged with a felony offense, other than a violent, serious, or sexual felony
offense.
6. Cannot have a prior conviction for a violent, serious, or sexual felony offense.
7. Cannot be required to register as a sex offender pursuant to Chapter 5.5 of Title 9,
Part 1.
8 SB 1106 amended SB 1004 and extended the date of authorization to establish a pilot program to
January 1, 2022, and expanded the scope of the pilot to include Ventura County, which ultimately chose
not to participate in the TAY program and is not included in this or the previous report.
9 A third law related to the TAY program, AB 1390, was enacted in July 2019. AB 1390 expanded the
program eligibility criteria to include young adults who were between the age of 21 and 24 at the time of
their arresting offense. Program participation by an individual in this age group must be approved locally
by the jurisdiction’s multidisciplinary team established for this project.
10 Cauffman, E. (2012). Aligning justice system processing with developmental science. Criminology and
Public Policy, 11(4), 751–758. doi:10.1111/j.1745-9133.2012. 00847.x; Farrington, D. P., Loeber, R., &
Howell, J. C. (2012). Young adult offenders: The need for more effective legislative options and justice
processing. Criminology and Public Policy, 11(4), 729–750. doi: 10.1111/j.1745-9133.2012. 00842.x;
Scott, E., Bonnie, R. J., & Steinberg, L. (2016). Young adulthood as a transitional legal category: Science,
social change, and justice policy. Fordham Law Review, 85(2), 641–666; Steinberg, L. (2012). Should the
science of adolescent brain development inform public policy? Issues in Science and Technology, 28(3),
70–76.
11 Senate Committee on Public Safety. (2016). SB 1004 young adults: deferred entry of judgment pilot
program. Retrieved from
http://leginfo.legislature.ca.gov/faces/billAnalysisClient.xhtml?bill_id=201520160SB1004.
1
8. Would have otherwise served time in custody in a county jail.
9. Must consent to participate in the program and agree to waive their right to a speedy
trial or preliminary hearing, plead guilty to the charge or charges, and waive time for
the pronouncement of the judgment.
For an individual who is determined to be eligible and suitable for, and who consents to
participate in, the TAY program, the court enters a deferred entry of judgment. An
individual may then be enrolled into a TAY program, where they can serve up to one year
in a juvenile hall. During TAY program participation, individuals may receive supports and
services such as age-appropriate educational, vocational, and supervision services
and/or mental health services. If a participant successfully completes the program, the
court will dismiss the individual’s charge(s) associated with their TAY enrollment.
However, if the individual is found to perform unsatisfactorily in the program,12 the
probation department may file a motion of entry of judgement. Once it receives the motion,
the court conducts a hearing to establish whether a judgment should be entered. If the
court determines that an individual was not benefiting from the services and supports
included in the program or is performing unsatisfactorily in the program, the court may
render a verdict of guilty to the charge(s) and schedule a sentencing hearing.
The legislation also requires the Board of State and Community Corrections (BSCC) to
conduct an evaluation of the program and to submit a comprehensive evaluation report
to the Assembly and Senate Committees on Public Safety by December 31, 2023. The
evaluation must address the following areas: (1) the impact of the TAY program on
sentencing, especially related to opportunities for community supervision; (2) the impact
of the TAY program on minors in juvenile facilities; and (3) the effect of the TAY program
on participants compared with the results for young adults sentenced for similar crimes
who did not participate in the TAY program. In May 2020, BSCC contracted with Evident
Change to conduct the evaluation.13
EVALUATION DESIGN
The evaluation was framed within three primary (as outlined in SB 1004) and four
secondary evaluation questions (EQs).
Primary Evaluation Questions
12 While the definition of successful completion varied across programs (see qualitative section of report
#1), in general, participants could be found to have an unsatisfactory program completion if they
committed a new offense or violated conditions of the program.
13 Evident Change (formerly the National Council on Crime and Delinquency) is a nonprofit that supports
the improvement of social services systems through research and data analytics.
2
1. What is the TAY program’s impact on sentencing, especially opportunities for
community supervision?
2. What is the impact of the presence of the program on minors in the juvenile facility?
3. What is the program’s effectiveness with respect to program participants and a
comparison group?
Secondary Evaluation Questions
4. How is the program structured?
5. How is the program implemented and operated?
6. What challenges and successes did the program experience?
7. What are program staff’s knowledge, perceptions, and recommendations related to
improving TAY programs?
This is the second of two reports Evident Change submitted to the BSCC regarding the
TAY pilot program. Due to limitations in accessing and processing quantitative data from
the California Department of Justice (DOJ), Evident Change requested a no-cost contract
extension to allow for the examination of the evaluation questions requiring official arrest
data from DOJ. In December 2020, Evident Change submitted the first report to the BSCC,
which includes the process evaluation components with an overview of each TAY
program, findings for EQs 2 and 4–7, and partial findings for EQs 1 and 3. Please see the
first report, titled Transition-Age Youth (TAY) Pilot Program Evaluation Report (December
2020), for more information on the initial findings. The current report focuses on
answering EQs 1 and 3, connecting the quantitative findings to the process outcomes,
and providing overall recommendations for future exploration.
While the Evident Change evaluation team assessed the quality of the quantitative data
as part of preparing (i.e., cleaning) the data for analysis, it was determined that
comparison groups would need to be developed differently than originally planned to
appropriately examine EQs 1 and 3. To streamline the discussion and interpretation of
the comparison group development, this report begins with an examination of EQ 3
methods and findings, followed by EQ 1 methods and findings and concluding with the
connection between the first report’s quantitative analysis and the process evaluation.
EQ 3 Methods
Following the original evaluation design developed by the BSCC to assess the impact of
the TAY program on both EQ 1 (sentencing) and EQ 3 (program effectiveness), the
evaluation team examined differences between two primary groups: (1) A comparison
group, based on data provided by DOJ and consisting of individuals sentenced from
counties participating in the TAY pilot before TAY was a sentencing option (specifically
between April 1, 2015, and the date that participant data identified as the TAY program
start date, with program start dates varying by individual counties), known as the pre-TAY
3
program comparison group;14 and (2) the TAY program participant group (individuals
served by the TAY program through official program enrollment).
For EQ 3, the data quality and integrity assessment revealed two potential additional
groups for comparison to the TAY participant group, again based on data provided by the
DOJ: (1) a post-TAY comparison group (consisting of TAY-eligible individuals who did not
participate in the program after the date of TAY implementation) and (2) an opt-out group
(consisting of TAY-eligible individuals who chose not to participate in the program;
reported by Santa Clara and Butte counties only). To ensure all results of the TAY
evaluation are reliably interpreted as an aggregate outcome project, the evaluation team
excluded analysis that included the opt-out group from this report; results for the opt-out
group are available by request. Table 1 provides an overview of the number of TAY
participants by county.
Table 1
Distinct TAY Program Participants by Participating County
County # %
Alameda 9 3.9%
Butte 64 28.1%
Napa 3 1.3%
Nevada 18 7.9%
Santa Clara 134 58.8%
Total 228 100.0%
Propensity score matching (PSM) is a sampling approach that results in statistically
equivalent comparison groups (e.g., treatment and non-treatment groups) and could be
used in evaluations such as this one in lieu of randomization. However, a larger number
14 Pre-TAY and post-TAY program group members were selected for inclusion in these cohorts based on
a single conviction using the following criteria.
1. Must be within either the pre-TAY or post-TAY program time periods.
2. Must have at least one TAY program–eligible felony.
3. Must not have any prior convictions with ineligible offenses.
4. Must not have a prior sex offense.
5. Must be within one of the five counties participating in the TAY program.
6. Must be within the age range for the TAY program.
7. Must have a disposition of probation/jail (this is one single disposition type, not either/or; the data
include date of disposition and do not include the time order of jail before probation or jail after probation
dispositions).
8. Must have a jail exit date that allows for a follow-up period of at least six months, based on calendar
time as well as the data extract cutoffs.
4
of TAY participants outside of Santa Clara County (ideally at least a sample of 200
participants in addition to those in the Santa Clara group) is necessary for a reliable
application of PSM in generating comparison groups that would provide a valid statistical
inference to the larger population.15 Due to these factors, the evaluation team chose not
to use PSM for the evaluation design and instead implemented a nonequivalent control
group design that examined comparisons between the TAY participant group and two
comparison groups (pre- and post-TAY comparison groups).
Study Groups Development and Demographics
Each participating county provided internal tracking documents to the evaluation team for
TAY program participants. This information included individuals enrolled in the TAY
program, those who opted out of program participation (opt-outs), cases closed with no
TAY program ordered, individuals pending enrollment in TAY, and those designated as
“unsuitable”16 for the program (for Santa Clara County). Due to significant issues with
missing data (e.g., some variables were only available for one department, some cases
lacked identification numbers to facilitate matching across county and DOJ data sets), for
this evaluation report, only individuals who were identified as being enrolled in the TAY
program were included from the county data while both comparison groups were
developed from data collected from DOJ.
The following is a summary of the final three groups included in the study design.
1. TAY participant group: In addition to the internal tracking documents, each county
provided a file of DOJ data matched to the county’s TAY program participants. The
TAY participant group includes individuals from all five counties who participated in
the TAY program. Individuals who had an enrollment date or variable indicating an
enrollment status within the data provided by the participating counties were included
in this sample. Of the data that Evident Change received from the counties, four
participants who were not matched to the DOJ data sets were excluded from the
analysis, resulting in a final TAY participant group sample size of 228.
2. Pre-TAY comparison group: This group consists of individuals who received a
conviction after April 1, 2015, and before the implementation of the TAY program in
their respective counties as identified within the DOJ data sets, and who met TAY
eligibility criteria, resulting in a sample size of 455.
15 Andrilon, A., Piracchio, R., & Cheveret, S. (2020). Performance of propensity matching to estimate causal
effects in small samples. Statistical Methods in Medical Research, 29(3), 644–658.
https://doi.org/10.1177/0962280219887196; Cenzer, I., Boscardin, J., & Berger, K. (2020). Performance of
matching methods in studies of rare diseases: A simulation study. Intractable Rare Diseases Research,
9(2) 79–88. doi: 10.5582/irdr.2020.01016
16 Santa Clara County is the only county that submitted data on individuals determined to be unsuitable for
the TAY program.
5
3. Post-TAY comparison group: This group consists of individuals who were TAY
program eligible following TAY implementation in the participating counties and who
were not enrolled in the TAY program, as identified in both the DOJ and county data
sets, resulting in a sample size of 144. It should be noted that this group includes
individuals who chose not to participate in the program voluntarily and those who were
not enrolled due to other programmatic processing or local decisions.
FINDINGS
Demographics of Evaluation Groups
The evaluation team used bivariate statistics to examine differences between the
evaluation groups. More specifically, Fisher’s exact test (F test) was used to determine if
differences between groups were statistically significant. An F test allows for the
comparison of proportions between groups with small sample sizes (e.g., at least one
expected frequency is less than 5) to determine whether the differences between the
groups are statistically significant. If the result is significant, then post-hoc testing is
completed to determine which pairs of variables are significant.
In each evaluation group, the largest proportion of participants was Latinx/Hispanic.
However, the pre-TAY comparison group had a statistically significant higher proportion
of Black/African American individuals than the post-TAY comparison and participant
groups. In addition, White individuals accounted for a significantly larger proportion of
participants in the TAY participant group and the pre-TAY comparison group (Table 2).
Table 2
Race/Ethnicity by Evaluation Group
Post-TAY Pre-TAY
TAY Participant Comparison Comparison
Race/Ethnicity
Group Group
n % n % n %
Asian/Pacific Islander 10 4.4% 11 7.6% 23 5.1%
Black/African American 28 2.3%* 27 18.8%* 154 33.8%*
Latinx/Hispanic 98 43.0% 70 48.6% 174 38.2%
White 71 31.1%* 33 22.9% 95 20.9%*
Multiple/Other 21 9.2% 3 2.1% 9 2.0%
Total 228 100.0% 144 100.0% 455 100.0%
*p < .05
Possibly due to small sample sizes, there were no statistically significant differences by
gender between evaluation groups. It should be noted, however, that a slightly smaller
6
proportion of individuals in the TAY participant group were categorized as female than
the two study groups (Table 3).
Table 3
Gender Composition by Evaluation Group
Post-TAY Pre-TAY
TAY Participant Comparison Comparison
Gender
Group Group
n % n % n %
Female 28 12.3% 24 16.7% 61 13.4%
Male 197 86.4% 120 83.3% 394 86.6%
Multiple 3 1.3% 0 0.0% 0 0.0%
Total 228 100.0% 144 100.0% 455 100.0%
Note: The “Multiple” category was developed to rectify the gender identification differences
between local county data and DOJ data. For example, one of these individuals had a gender
identification of male in the local county data and female in the DOJ data. Rather than exclude
these three individuals from the analysis, we developed the multiple category.
The TAY participant group included a small proportion of people who were ages 21 to 24
(n=13) at the time of the arrest that brought them into the TAY program, while the
comparison groups did not include any people in this age range. The average age for all
three groups was 19. However, there were no statistically significant differences related
to age between the groups (Table 4).
Table 4
Age Composition by Evaluation Group.
Post-TAY Pre-TAY
Age at TAY Participant
Comparison Group Comparison Group
Arrest
n % n % n %
18 82 36.0% 49 34.0% 156 34.3%
19 87 38.2% 60 41.7% 173 38.0%
20 46 20.2% 35 24.3% 126 27.7%
21 12 5.3% 0 0.0% 0 0.0%
22 0 0.0% 0 0.0% 0 0.0%
23 1 0.4% 0 0.0% 0 0.0%
Total 228 100.0% 144 100.0% 455 100.0%
7
Offense History of Evaluation Groups
Data collected on offense history of individuals from each of the evaluation groups was
examined from the data extract pulled from DOJ.17 It includes all arrests and convictions
for individuals included in the study design. On average, TAY participants had a lower
rate of prior arrests than the comparison groups. Table 5 provides an overview of the
average number of prior arrests by group.
Table 5
Average Number of Prior Arrests by Evaluation Group
Group Average Prior Arrests
Post-TAY Comparison Group 7.38
Pre-TAY Comparison Group 7.36
TAY Participant 5.02
The evaluation team conducted additional bivariate analyses to examine the differences
in the severity of offense history by evaluation group. For this examination, the team used
analysis of variance (ANOVA) to test for statistical differences between the groups. If
differences were significant, the team conducted post-hoc testing using Tukey (Honest
Significance Difference) HSD to determine which pairings of variables between the
groups were significantly different. On average, TAY participants had more prior arrests
for violations and infractions than the comparison groups, although the differences
between groups were not statistically significant. However, there was a statistically
significant difference in the average number of previous felony arrests, with both
comparison groups having a higher average of prior felony arrests than the TAY
participant group. Table 6 provides an overview of offense history by severity and by
group.
Table 6
Offense History by Severity by Evaluation Group
Post-TAY Pre-TAY
Arrest Offense Severity TAY Participant Comparison Comparison
Group Group
Felony 2.34** 4.05** 3.43**
Misdemeanor 3.84 4.46 4.54
Infraction 1.46 1.30 1.38
Violation 3.44 2.52 2.09
**p < .01
17 Data retrieved from DOJ includes all offense or arrest history available to the DOJ.
8
When examining offense history by race/ethnicity, the differences between
Latinx/Hispanic individuals by group were statistically significant, which may be because
more Latinx/Hispanic individuals were represented in all groups than other
races/ethnicities. Specifically, Latinx/Hispanic people in the TAY participant group had a
lower average for prior arrests than both comparison groups. While previous arrest rates
were also higher for other races/ethnicities in the comparison groups than the TAY
participant group, the difference between groups was not statistically significant. However,
the average number of prior arrests for White TAY participants was nearly twice that of
the TAY participants of other races/ethnicities and more similar to the averages for the
comparison groups (excluding Asian/Pacific Islander). Table 7 provides an overview of
the average arrest history between groups by race/ethnicity.
Table 7
Arrest History Averages by Race/Ethnicity and Evaluation Group
Post-TAY Pre-TAY
Race/Ethnicity TAY Participant Comparison Comparison
Group Group
Asian/Pacific Islander 3.33 3.67 3.53
Black/African American 3.87 8.04 7.16
Latinx/Hispanic 3.53* 6.64** 5.56*
White 6.81 6.04 6.99
Multiple/Other 3.60 9.00 6.67
*p < .05; **p < .01
Recidivism Outcome Sample Selection
Due to a range of factors, counties participating in the TAY pilot began enrolling
participants over a span of approximately two years (Table 8).
Table 8
Start Date of TAY Program Enrollment by County
County Start Date of TAY Program Enrollment
Alameda June 2019
Butte March 2017
Napa April 2018
Nevada May 2017
Santa Clara October 2017
During the evaluation period, the average length of enrollment in the TAY pilot across all
counties was 12.9 months (ranging from 10.9 months to 17.5 months). Due to the
9
variability in the length of operations of the TAY pilot across all counties, variability in the
average length of enrollment, and the relatively short timeframe of operation of the TAY
pilot across all counties, the evaluation team limited the examination of differences in
recidivism between the TAY participant group and the comparison groups to six months
from program exit.
For the TAY participant group, 72 participants were excluded from the sample for
outcome analysis as they were still active in the program on the date of the last data
extract. An additional 35 participants were excluded due to being less than six months
from the program end date (e.g., participant was only out of the program for two months
upon data extract). The data collected included descriptive data for 19 participants on
discharge reason (e.g., successful or unsuccessful) without a corresponding end date.
However, data for 14 of these 19 people did include a facility exit date. Subsequently, the
evaluation team used the facility exit date as a proxy for program discharge in the analysis.
The remaining five individuals with missing discharge or program end dates were
excluded from the TAY participant group, resulting in a final outcome sample of 116 TAY
participants.18
For the pre- and post-TAY comparison groups, the evaluation team created a proxy end
date to approximate an end date similar to the TAY participant group program end date.
The comparison groups included individuals with multiple dispositions (e.g., probation or
jail), sentence types, locations, and durations. The date of exit from a jail facility was used
as the proxy beginning date for the outcome evaluation. The evaluation team conducted
descriptive statistics on demographics by outcome group with no significant variance
between the outcome subsample and the original comparison groups. Table 9 provides
an overview of the final sample size for the outcome analysis by evaluation group. Table
10 provides a breakdown of the outcome sample by participating county.
Table 9
Outcome Sample Size by Evaluation Group
Group Total
TAY Participant 116
Post-TAY Comparison Group 144
Pre-TAY Comparison Group 455
18 The last date of court data included in the data sets was November 2020. In addition, the data sets
included only two TAY participants who exited the program after shelter-in-place orders were implemented
(March 19, 2020) and had a six-month follow-up period within the data collected. Therefore, analytics
comparing outcomes before and during COVID-19 could not be conducted between groups.
10
Table 10
Outcome Sample Size by Participating County
Post-TAY Pre-TAY
TAY Participant Comparison Comparison
County
Group Group
n % n % n %
Alameda 2 1.7% 6 4.2% 237 52.1%
Butte 40 34.5% 28 19.4% 45 9.9%
Napa 1 0.9% 1 0.7% 17 3.7%
Nevada 10 8.6% 10 6.9% 8 1.8%
Santa Clara 63 54.3% 99 68.8% 148 32.5%
Total 116 100.0% 144 100.0% 455 100.0%
Arrest and Court Outcomes
Each county provided data related to the offense that led to a participant’s TAY program
enrollment. However, the data varied from county to county. In some cases, an arrest
date and offense description were provided, while in some cases no arrest date was
provided with several offense descriptions. Subsequently, these data did not directly
match the DOJ data sets and were used to conduct individual case checks to confirm the
correct arrest in the DOJ data as the index (i.e., initiating) arrest leading to TAY program
enrollment. In most cases, the first arrest prior to TAY enrollment was selected as the
index event. However, some individuals had additional arrests between their TAY index
arrest and TAY start date. For the outcome analyses, only arrests for offenses that
occurred after an individual’s TAY enrollment date were included.
New arrest: Includes any new arrest event occurring within the six-month period
following the program end date (for TAY participants) or proxy program end date (for
comparison groups).
New violation: Includes probation violations and revocations.19
New conviction: Includes offenses with a disposition of convicted, petition sustained,
jail, etc. In addition, conviction events associated with the TAY enrollment arrest may
be included in the recidivism analysis if the conviction occurred during the six-month
outcome period after exiting the TAY program.
19 Probation violation and revocation data did not include an event date (as the arrest and court action data
included) but only a disposition date.
11
Differences Between Groups for Arrest, Conviction, and Violations
The evaluation team conducted bivariate analyses to examine the differences in new
arrests, convictions, and violations by evaluation group. For this portion of the evaluation,
the team used ANOVA to test for statistical differences between the groups. If differences
were significant, the team conducted post-hoc testing using Tukey HSD to determine
which variable pairings between the groups were significantly different.
There were statistically significant differences between groups with new arrests within six
months of program exit, accounting for a smaller proportion of outcomes for TAY
participants than for the pre-TAY comparison group. For new violations, there were
significant differences between TAY participants and the post-TAY comparison group,
with TAY participants experiencing 13% fewer violations within six months than the post-
TAY comparison group. In addition, there were statistically significant differences in
violations between the post- and pre-TAY comparison groups, with the post-TAY group
experiencing a larger proportion of violations within six months. There were no statistically
significant differences between groups for new convictions (Table 11).
Table 11
Outcomes by Type and Evaluation Group
Post-TAY Pre-TAY
TAY Participant Comparison Comparison
Outcome
Group Group
n % n % n %
New Arrest 33 28.4%* 56 38.9% 196 43.1%*
New Felony Arrest 24 20.7% 47 32.6% 144 31.6%
New Misdemeanor Arrest 29 25.0% 37 25.7% 139 30.5%
New Conviction 25 21.6% 24 16.7% 74 16.3%
New Violation20 9 7.8%** 30 20.8%* 59 13.0%*
Total 116 100.0% 144 100.0% 455 100.0%
Note: The column total is the sample size; *p < .05; **p < .01; new arrest may include multiple
offenses, and so the n for new arrest is not the total of the subset of new felony and new
misdemeanor arrests.
Next, the evaluation team examined differences in types of new offenses (e.g., person
versus weapons arrests). The data set included well over 100 types of offenses. To
simplify the analysis, the evaluation team recategorized those offense types into six
groups: person (e.g., assault), sex offense (e.g., sexual assault), property (e.g., graffiti),
20 When testing for significance between groups at the county level, the evaluation team found a significant
difference for one county between all three evaluation groups for new violations.
12
weapons (e.g., gun use in vehicle theft), drugs (e.g., possession of cocaine), and other
(e.g., violation of probation). The only significant difference between groups was for the
“other”21 arrest category. Specifically, the TAY participant group experienced fewer new
arrests for “other” offenses than the pre-TAY comparison group (Table 12).
Table 12
New Offense Types by Evaluation Group
Post-TAY Pre-TAY
TAY Participant Comparison Comparison
Outcome
Group Group
n % n % n %
New Person Arrest 9 7.8% 20 13.9% 46 10.1%
New Sex Offense Arrest 1 0.9% 0 0.0% 1 0.2%
New Property Arrest 22 19.0% 36 25.0% 103 22.6%
New Weapons Arrest 5 4.3% 16 11.1% 43 9.5%
New Drug Arrest 13 11.2% 14 9.7% 64 14.1%
New Other Arrest 24 20.7%* 33 22.9% 132 29.0%*
Total 116 100.0% 144 100.0% 455 100.0%
*p < .05; Note: “New Other Arrest” includes all arrests not included in the previous categories.
New Arrests by Race/Ethnicity and Gender
An examination of new arrests by race/ethnicity by evaluation group did not find any
statistically significant differences between groups. However, White participants
accounted for a larger proportion of new arrests for the TAY participant group than the
post-TAY comparison group (Table 13).
21 “Other” offenses included violations, infractions, and other offenses not categorized in the table.
13
Table 13
New Arrests by Race/Ethnicity and Evaluation Group
Post-TAY Pre-TAY
TAY Participant Comparison Comparison
Race/Ethnicity
Group Group
n % n % n %
Asian/Pacific Islander 1 14.3% 2 18.2% 6 26.1%
Black/African American 2 20.0% 14 51.9% 63 40.9%
Latinx/Hispanic 13 26.0% 23 32.9% 70 40.2%
White 16 43.2% 14 42.4% 55 57.9%
Multiple/Other 1 8.3% 3 100.0% 2 22.2%
Total 33 28.4% 56 38.9% 196 43.1%
An examination of differences in new arrests by gender across evaluation groups showed
no statistically significant differences between groups (Table 14).
Table 14
New Arrests by Gender and Evaluation Group
Post-TAY Pre-TAY
TAY Participant Comparison Comparison
Gender
Group Group
n % n % n %
Female 3 15.8% 7 29.2% 24 39.3%
Male 30 31.6% 49 40.8% 172 43.7%
Total 33 28.4% 56 38.9% 196 43.1%
Recidivism Outcomes for TAY Participants by Program Completion
The evaluation team conducted additional analysis to examine recidivism-related
outcomes by TAY participant completion type (successful versus unsuccessful
completion as defined by the counties). Using an F test with the creation of 2X2
contingency tables, the analysis revealed a statistically significant difference between
new arrests, new convictions, and new violations for people who were successful in
completing TAY programming compared with people who were identified as unsuccessful.
Over 60% of those who did not successfully complete the program were rearrested within
six months compared with only 9% of those who successfully completed it. In addition,
almost 60% of those identified as unsuccessful and almost 3% of successful individuals
were convicted within six months (Table 15).
14
Table 15
Outcome Comparisons Between Successful and Unsuccessful TAY Participants
Successful (n=76) Unsuccessful (n=40)
Outcome
n % n %
New Arrest 7 9.2%** 26 65.0%**
Felony 2 2.6% 22 55.0%
Misdemeanor 6 7.9% 23 57.5%
New Conviction 2 2.6%** 23 57.5%**
Felony 0 0.0% 17 42.5%
Misdemeanor 2 2.6% 12 30.0%
New Violation 0 0.0%** 9 22.5%**
**p < .01; new arrest, new conviction, and new violation all represent TAY participants; the data
included multiple offenses per arrest and conviction.
Influence of Program Participation on Six-Month Recidivism
A stepwise binomial regression was run to examine the influence of all predictor variables
(e.g., race/ethnicity, gender, evaluation group) on new arrests within six months of
program completion. With this approach, an initial model was run with all available
variables. That model was not statistically significant. Therefore, additional models were
run after removing (i.e., deleting) the variable with the lowest level of significance. The
Hosmer-Lemeshow test showed that the final model shown in Table 16 fit the data well.
The model indicated that being a participant in the TAY pilot reduced the likelihood of
rearrest in six months (p. <. 01) while being White increased the likelihood of rearrest in
six months (p. <.001).
Table 16
Regression Model for New Arrest Within Six Months of TAY Program Completion
Standard
Variable Estimate z value P value
Error
Constant -0.431 0.138 -3.122 0.0018**
Race
Asian/Pacific Islander -0.697 0.398 -1.752 0.0798
Black/African American 0.136 0.194 0.700 0.4837
White 0.673 0.200 3.366 0.0008***
Multiple/Other -0.336 0.496 -0.678 0.4975
TAY Participant -0.676 0.237 -2.852 0.0043**
Post-TAY Comparison -0.155 0.200 -0.776 0.4376
**p < .01, *** p < .001; Null deviance: 961.6 on 714 degrees of freedom; Residual deviance:
933.6 on 708 degrees of freedom; Race is regressed on new arrest by race.
15
Recidivism Outcomes for TAY Participants by Facility Time
The evaluation team conducted additional analysis using the F test to examine outcomes
for TAY participants who spent some time (defined as any time, no matter the length) in
juvenile hall during the program compared with those who did not. While the difference
between these groups was not statistically significant, of those who spent some time in
juvenile hall (n=96), 63.5% successfully completed the program versus 75% of those who
successfully completed the program without a juvenile hall stay.
EQ 1 Methods
For EQ 1, the evaluation team sought to examine whether the proportion of community
supervision sentences was different between the three study groups. To operationalize
sentencing options by (1) eligible individuals and (2) eligible offenses, the evaluation team
used both a listwise and pairwise process to align court actions by comparable
TAY-eligible participants and TAY-eligible offenses.22 Cases where court actions did not
include a TAY-eligible offense and where the individual had a prior conviction for a
TAY-ineligible offense (i.e., too severe) were excluded. This resulted in a DOJ sample of
pre- and post-TAY comparison groups consisting of 2,870 distinct individuals.
As a next step, the remaining individuals who met the following criteria were included in
the comparison groups.
Must have been ages 18 to 20 at the time of the court action.
Court action/offenses must be from one of the five participating counties.
Must have at least one felony offense on the court action.
Court action must be after April 1, 2015.
After excluding records that did not meet the above criteria, the final sample included
1,854 distinct individuals.
Operational Definitions for Sentencing Variables
The DOJ data set included two different variable fields related to court-directed
sentencing. Within those fields were 228 categories related to disposition. One field
included a description of the disposition with information on the type and magnitude of
the individual’s disposition, consisting of 200 variables (e.g., acquitted, diverted,
probation, jail). The other field included the location (e.g., prison or jail) of the disposition,
consisting of 28 variables (e.g., jail, probation, fines, restitution, work programs).
22 Newman, D. (2014). Missing data: Five practical guidelines. Organizational Research Methods, 17(4),
372–411.
16
Therefore, for each court action, which could include multiple offenses, the disposition
description and location fields were combined to create the four following sentence
categories by individual and date of the court action.
Jail or prison only: Disposition and/or sentence location only includes jail or prison
(does not include probation).
Jail and probation: Disposition is probation and includes prison or jail or the sentence
location indicates jail or prison.23
Probation only: Dispositions and sentence locations do not include jail or prison.
Diversion/deferral: The most severe court action was a diversion or deferral (e.g., no
disposition included probation or jail/prison).
Acquitted/dismissed: All dispositions must be acquittals and/or dismissals.
Sentencing Sample Criteria
Due to the variability in how individuals were identified and enrolled in TAY across the
participating counties (as detailed in the process evaluation results), the evaluation team
determined the best fit for the study design would be the creation of both a TAY participant
sample and population-based comparison groups anchored in two time points: prior to
each county’s implementation of TAY and after TAY implementation. The process
included the identification and examination of both the individuals and the arrest
dispositions related to an arrest or arrest event. This process resulted in the development
of the following three distinct evaluation groups.
TAY participant (n=164 individuals): This group includes TAY participant dispositions
that occurred after TAY was implemented in their respective county. This participant
sample is not identical to the list of participants presented in the outcome analysis due
to filtering techniques used to ensure each sample resembled each other as much as
possible. For example, this cohort was filtered to only include 18- to 20-year-olds due
to the fact the pre- and post-TAY samples used in the outcome analysis did not include
21-year-olds.
After TAY (n=574 individuals): This group includes individual dispositions for non-TAY
participants that occurred after TAY implementation. This sample does not include
post-TAY dispositions for TAY participants.
Before TAY (n=1,116 individuals): This group includes all dispositions that occurred
prior to TAY implementation.
23 Note that it is possible to receive a disposition of “jail/probation,” and these records comprise most of this
category.
17
Proportional Differences in Sentencing Dispositions
The results of the sentencing analysis are presented at the disposition or sentence level,
not at the individual level. If an individual has more than one disposition or sentence
during any time included in the data set, all sentences are included (aggregated by court
action date).
Table 17 provides an overview of the sentencing proportions by type and by evaluation
group. The evaluation team conducted an F test to examine the differences in sentence
proportions by evaluation group. Differences were statistically significant between the
TAY participant group and the “after TAY” group for jail or prison only, jail and probation,
diversion/deferral, and acquitted/dismissed. When comparing the participant group to the
“before TAY” group, differences in sentence proportions were statistically significant for
jail and probation, diversion/deferral, and acquitted/dismissed. Specifically, the participant
group experienced a significantly smaller proportion of jail and probation sentences (7.9%)
than both the before (72.4%) and after (62.8%) TAY comparison groups. These results
indicate that proportional differences between these sentencing options are correlated
with the implementation of the TAY pilot program.
Table 17
Sentencing Proportions by Type and Evaluation Group
After TAY Before TAY
TAY Participant Comparison Comparison
Sentence
Group Group
n % n % n %
Jail or Prison Only 12 4.8%* 91 12.6%* 114 8.0%
Jail and Probation 20 7.9%* 453 62.8%* 1,037 72.4%*
Probation Only 12 4.8% 36 5.0% 53 3.7%
Diversion/Deferral 158 62.7%* 49 6.8%* 14 1.0%*
Acquitted/Dismissed 47 18.7%* 76 10.5%* 181 12.6%*
Convicted/Fined/Other 3 1.2% 16 2.2% 33 2.3%
Total 252 100.0% 721 100.0% 1,432 100.0%
*p < .05
To examine the differences in sentence length by evaluation group by sentence type, the
team conducted a T-test. None of the differences were statistically significant. However,
TAY participants who received a jail sentence had longer sentences than individuals in
the comparison groups (Table 18).
18
Table 18
Average Sentence Length by Evaluation Group and Sentence Type
Jail Prison Probation
Group
# Days # Days # Days
TAY Participant 29 189.52 3 771.67 29 1,081.00
After TAY 494 142.84 62 1,040.82 479 1,005.26
Before TAY 1,086 127.95 97 883.73 1,084 1,151.58
Note: Counts will not match Table 17 due to the exclusion of suspended sentences from the
sentence length analysis. Those sentences were excluded due to the lack of verification of the
actual sentence being served (e.g., time in jail) or not.
Due to the statistically significant decrease in jail and probation sentences across all three
evaluation groups and the non-significant differences in sentence lengths between groups,
the data suggest that the TAY pilot program may not have resulted in net widening for jail
and probation sentences. In addition, the increased proportion of diversions/deferrals
across the “before TAY” and “after TAY” timepoints suggests that participating probation
departments increased the use of diversion programming during TAY implementation.
These results should be reviewed with caution as the data collection and analytics were
limited to correlational analyses and causality due to (1) limited variables in the DOJ data
set, and (2) limited timeframe for sample selection. Further analysis should include a
longitudinal study design that incorporates both case processing (e.g., arrest records,
petitions) and sentencing outcome data collection and analysis.
In addition to the sentencing evaluation, post-hoc analysis of the index (i.e., initiating)
offenses resulting in TAY program enrollment suggests that some individuals were
enrolled without having been arrested for a felony offense (which was one of the criteria
for TAY eligibility). Of the 228 TAY participants included in the study sample, 212 had an
index arrest that included a felony, indicating that 16 who were arrested for misdemeanor
or infraction offenses were also enrolled in the pilot program.
Connection to Process Evaluation Outcomes
Net Widening
During data collection for the process evaluation, TAY program staff indicated that there
were differing understandings among decision makers regarding who is suitable or
eligible for TAY program involvement. Staff indicated they would like to see more people
enrolled in TAY programs and to have a common understanding of and application for
TAY program eligibility criteria among system partners. For example, most respondents
to a survey administered for the process evaluation reported that they would like
clarification about whether an individual’s juvenile justice history (70.2% of respondents)
or adult justice history (78.7%) are factors in determining program eligibility. In addition,
some interview participants suggested the program’s eligibility criteria should be
19
expanded to include misdemeanors, as did close to two thirds (61.7%) of survey
respondents.
The quantitative data analysis for EQ 1 suggests that staff reports related to the variation
in enrollment and eligibility criteria can be preliminarily confirmed. While the data do not
indicate a definitive impact of net widening due to the TAY legislation (as seen in the
variation in sentence dispositions), the data indicate that discretion within the counties as
described in staff interviews and surveys allowed for enrollment of participants who may
not have been arrested for an eligible offense.
Considerations for Juvenile Hall Component
Each county customized their TAY program to respond to unique community needs.
While the initiating legislation intended for each program to have a juvenile hall in-custody
component, in practice, some counties either did not implement an in-custody component
or that piece of the program was not consistently applied to all participants, for various
reasons (e.g., local expectations, response to COVID-19). Table 19 provides an overview
of the expected duration of TAY program components by county.
Table 19
Expected Duration of TAY Program Components by County
Included In- Expected
Expected
Custody Duration of Expected
Duration of In-
County Component for Community Duration of
Custody
All or Some Supervision Entire Program
Component
Participants Component
Alameda Yes, some 30–45 days 8–11 months About 12 months
Butte Yes, all About 90 days About 9 months About 12 months
No, did not use
Napa N/A 12 months 12 months
this component
Nevada Yes, some Varied Varied 12–18 months
Santa Clara Yes, all 30–60 days 6–9 months About 12 months
When asked whether the juvenile hall component should be retained as part of the TAY
program model, some staff indicated that it was no longer in use as part of the program
due to factors such as the Department of Juvenile Justice’s upcoming realignment and/or
COVID-19 responses, while others indicated it should remain and that the length of time
in juvenile hall for TAY participants should be extended.
The outcome analysis included in this report for E3 found that there was no statistical
significance related to successful completion of the TAY program for those who spent
some time in juvenile hall compared to participants who did not spend time in juvenile hall.
It should be noted that a larger proportion of participants who did not receive the in-
custody portion of the program had successful TAY program completions compared with
20
those who spent some time in juvenile hall (e.g., 63.5% of those who experienced an in-
custody stay were successful compared with 75% of those without a juvenile hall stay
being successful). Therefore, while staff perceptions of the potential impact and need for
the juvenile hall component were mixed, the preliminary findings, while not predictive,
suggest that the juvenile hall component may not impact successful program completion.
In addition, the results of the E1 and E3 quantitative data analytics suggest that the
strongest predictor of lower rates of recidivism (based on rearrest, conviction, or receiving
a violation within six months of program completion) for people who meet TAY program
eligibility is successful TAY program completion.
Staff/Stakeholder Perceptions of TAY Program Benefits
The evaluation team found through the process evaluation that TAY staff and
stakeholders felt programs are beneficial and made recommendations for improvement.
County representatives generally believed the TAY pilot program is worthy of the effort
and provides positive services, supports, and opportunities for young adults. Most
indicated that (1) the TAY program is beneficial to young adults in their community,
(2) they would like to see an increase in agreement and consistency in the
operationalization of eligibility and suitability between stakeholders in the decision making
for their community, (3) they would like to see a standard operationalization and
agreement on defining successful and unsuccessful completions, and (4) funding for
program support and enhancement is necessary to move the project beyond a pilot status.
CONCLUSIONS AND RECOMMENDATIONS
Based on the process, sentencing, and outcome findings of this preliminary evaluation,
future implementation of the TAY program should consider the role of the juvenile hall
component of the program model. The outcome analysis suggests that the juvenile hall
component may not contribute to greater levels of success than for individuals who
participated in only the community component of the approach. In addition, the sentencing
data suggest that once the TAY pilot was implemented, there was an increase in the
proportion of individuals who were referred to diversion programs, indicating that the TAY
pilot could be contributing to an increased usage of diversion programs in lieu of
in-custody approaches to adult offending in the participating communities.
Evaluation Limitations
Some limitations to this evaluation of the TAY pilot project may be attributed to, and not
limited to, the following areas: (1) limited number of interviews and survey responses for
qualitative data collection (see first report), (2) varied approaches toward determining
eligibility, suitability, and enrollment of participants in the TAY pilot by participating
21
counties (see first report for description of implementation approaches), (3) variances in
the number of participants enrolled in each participating county (e.g., Santa Clara and
Butte County’s participants together accounted for 87% of the entire participant sample
size) resulting in potential sample bias, and (4) the time limited scope of quantitative data
collection through administrative data due to the relatively short time frame of TAY
implementation by each project site.
The qualitative data collected was reliant on a small number of interviews with TAY county
representatives and BSCC staff, responses to an online survey administered to TAY staff
and stakeholders, and TAY source documents such as the county applications and
internal reports. Therefore, the process evaluation results presented in the first report and
referenced in the current report were limited to the views of those who chose to participate
in an interview and/or respond to the survey. In addition, the varied approaches toward
determining eligibility, suitability, and enrollment across each county implementing TAY
challenged the internal reliability of cross-site evaluation of the impact on sentencing by
the implementation of the TAY pilot as well as outcome evaluation. Specifically, the
quantitative data analysis indicated that some individuals enrolled in the program were
not arrested for a TAY eligible felony offense, thereby highlighting individual counties’
discretion in enrollment. These data suggest that staff perceptions of program variability
may be accurate and hint that the implementation of the TAY pilot may have included
opportunities for net widening. However, the relatively limited scope of data collection and
length of program implementation hindered the evaluation team’s ability to examine the
correlation of arrest offense significantly and comprehensively to program enrollment and
any potential causal predictors of future recidivism based on that enrollment.
Future Evaluation Considerations
Should the TAY program continue to operate in the five original counties and/or expand
to additional counties, the evaluation team recommends ongoing evaluation to support
the continuous quality improvement of the program and to support and inform the BSCC
in providing technical assistance to key stakeholders associated with the implementation.
To examine the impact of the TAY pilot more extensively on sentencing, future
evaluations should incorporate an integrated cross-sectional (data collected at a targeted
point or targeted points in time) and longitudinal (over the course of TAY pilot
implementation) study design. The qualitative data collection should incorporate case
processing (e.g., arrest records, petitions, etc.), participant perspective (through
interviews or focus groups), stakeholder perspectives (expanded beyond the limited
capacity of this evaluation to include and not limited to defense attorneys, prosecutors,
key probation staff, and other community stakeholders), and other social artifacts
(e.g., petitions, written disposition recommendations) related to sentencing decisions.
This approach would allow for a more comprehensive examination of both the formal and
informal influences of TAY programming inclusive of community contextual
considerations (e.g., other local diversion programs or services) that may be correlated
to sentencing outcomes related to TAY programming.
22
Future evaluations may also consider the benefits of a larger TAY participant group for
counties other than Santa Clara and Butte (due to extended program enrollment). A larger
participant group, across all participating counties, will increase the statistical influence of
counties with less TAY participation than others and ultimately allow for the use of PSM
to identify and develop comparison groups that are more accurately reflective of both the
overall TAY participant group and county-level distinctions than could be assumed in this
evaluation. It should be noted that one of the greatest challenges with evaluation research
is in identifying equivalent comparison groups. In the development of evidence-based
programs and interventions, primarily from healthcare research, randomized controlled
trials (RCTs) have been considered the gold standard for controlling for sample variability
between the study or treatment group and comparison or control groups.24 Given that
randomization is extremely difficult for studies of treatment services and supports of
individuals involved with court systems, are extremely expensive to conduct, and often
difficult to complete in social service systems, future research should consider the use of
nearest neighbor matching to create statistically equivalent groups via PSM.25
As another consideration toward the enhancement of future examinations of both the
impact of sentencing and outcomes of the TAY pilot program, quantitative data collection
and analysis should include predictive modeling techniques and survival analysis. More
specifically, predictive modeling such as binomial regression can be used for predicting
the odds of experiencing or seeing an event, given the influence of predictor variables.
The binomial regression model is part of a family of generalized linear models (GLMs).
GLMs are typically used to model the relationship between an expected value in response
to a combination of predictor variables.26 For example, the binomial regression model
could be used to predict the odds of an individual experiencing an event such as a
thunderstorm on a particular day based on the (1) current temperature, (2) barometric
pressure, (3) time of year, (4) humidity, (5) geography/location, and (6) altitude.
Expanding the analyses to include time-to-event outcomes, including convictions for
individuals participating in TAY compared to comparison groups, would allow for the
integration of time to event as an outcome, and expand the markers of program impact
from simple recidivism. For example, nonparametric Cox survival analysis models can be
created to estimate the prediction of a time-to-event experience or outcome. Using these
models, an expanded project evaluation can examine the impact of program dosage on
the length of time to the occurrence of new outcomes, rather than relying on arrest and
court conviction data. For example, violations are the most significant predictor of future
24 Grossman, J., & Mackenzie, F. (2005). The randomized controlled trial: Gold standard, or merely
standard? Perspectives in Biology and Medicine, 48(4), 516–534.
25 Mezey, G., Robinson, F., Campbell, R., Gillard, S., Macdonald, G., Meyer, D., Bonell, C., & White, S.,
(2015). Challenges to undertaking randomised trials with looked after children in social care settings. Trials,
16(206). https://doi.org/10.1186/s13063-015-0708-z
26 Scheick, T., Zhang, M., & Gerds, T. (2008). Predicting cumulative incidence probability by direct binomial
regression. Biometrika, 95 (1), 205–220.
23
out-of-home placements for both juvenile and adult justice system involved individuals.27
Including time-to-event analysis that examines the difference in time to violations between
TAY participants and a comparison group based on PSM would allow leadership
implementing the TAY pilot to explore local policies, staffing procedures, and other
contextual elements associated with TAY program implementation that may contribute to
or enhance outcomes related to arrests for violations.
27 Guo, S., & Metcalfe, C. (2020). The long road to probation completion: A longitudinal analysis of the
effect of life events on re-arrest among probationers. Deviant Behavior, DOI:
10.1080/01639625.2020.1841587; Dir, A. L., Magee, L. A., Clifton, R. L., Ouyang, F., Tu, W., Wiehe, S. E.,
& Aalsma, M. C. (2021). The point of diminishing returns in juvenile probation: Probation requirements and
risk of technical probation violations among first-time probation-involved youth. Psychology, Public Policy,
and Law, 27(2), 283–291. https://doi.org/10.1037/law0000282
24