CSA
Summary
Read the report at California State Auditor ↗
California
Commission
on Teacher
Credentialing:
It Could Better Manage Its Credentialing
Responsibilities
November 2004
2004-108
rotiduA
etatS
ainrofilaC
S
T
I
D
U
A
E
T
A
T
S
F
O
U
A
E
R
U
B
The first five copies of each California State Auditor report are free.
Additional copies are $3 each, payable by check or money order.
You can obtain reports by contacting the Bureau of State Audits
at the following address:
California State Auditor
Bureau of State Audits
555 Capitol Mall, Suite 300
Sacramento, California 95814
(916) 445-0255 or TTY (916) 445-0033
OR
This report is also available
on the World Wide Web
http://www.bsa.ca.gov/bsa/
The California State Auditor is pleased to announce
the availability of an on-line subscription service.
For information on how to subscribe, please contact
the Information Technology Unit at (916) 445-0255, ext. 456,
or visit our Web site at www.bsa.ca.gov/bsa
Alternate format reports available upon request.
Permission is granted to reproduce reports.
� � �
��������� ���� ������
������������� �������������������
������������ �����������������������
November 9, 2004 2004-108
The Governor of California
President pro Tempore of the Senate
Speaker of the Assembly
State Capitol
Sacramento, California 95814
Dear Governor and Legislative Leaders:
As requested by the Joint Legislative Audit Committee, the Bureau of State Audits presents its audit
report concerning the efficiency and effectiveness of the teacher credentialing process administered by
the California Commission on Teacher Credentialing (commission).
This report concludes that the commission could increase its ability to measure the effectiveness of its
teacher development programs, the efficiency of the teacher credentialing process, and the performance
of its internal operations. In addition, by focusing its customer service, better managing its workload,
and taking full advantage of a new automated processing system, the commission could improve its
credential application process. Further, we identified several areas in the commission’s process for
developing program standards, which college and universities follow when preparing prospective
teachers, that lack structure and could be improved. Finally, in December 2002 the commission
suspended its continuing accreditation reviews of most colleges and universities, which limits its
ability to ensure that they operate teacher preparation programs in accordance with the commission’s
standards. The commission is evaluating its accreditation policy and it does not plan to propose a
revised policy to its governing body until August 2005.
Respectfully submitted,
ELAINE M. HOWLE
State Auditor
�������������������
�������������������������������������������������� ���������������������������������������� ������������������
Blank page inserted for reproduction purposes only.
CONTENTS
Summary 1
Introduction 5
Chapter 1
The Commission Could Better Evaluate the
Effectiveness of Its Efforts and Better Measure the
Performance of the Teacher Credentialing Process 13
Recommendations 32
Chapter 2
By Better Managing Its Customer Service,
Workload, and Technology, the Commission
Could Improve Application Processing 35
Recommendations 57
Chapter 3
The Commission Should Refine Its Process for
Developing Program Standards and Resume
Its Continuing Accreditation Reviews of Colleges
and Universities 59
Recommendations 73
Agency Response
California Commission on Teacher Credentialing 75
California State Auditor’s Comments on the
Response From the California Commission on
Teacher Credentialing 109
Blank page inserted for reproduction purposes only.
California State Auditor Report 2004-108 11
SUMMARY
RESULTS IN BRIEF
The California Commission on Teacher Credentialing
(commission) was created in 1970 with the responsibility
Audit Highlights . . . of ensuring excellence in education by establishing high
standards for the preparation and licensing of public school
Our review of the
educators. The commission also issues licenses and permits for
credentialing process
school administrators and educators working in specialized
administered by the
California Commission teaching areas. In fiscal year 2003–04 the commission granted
on Teacher Credentialing approximately 239,000 teacher and administrator licenses
(commission) revealed the
and renewals. In addition to its licensing responsibility, the
following:
commission develops program standards to address the quality
þ The commission could of the programs that accredited colleges and universities provide
better evaluate the
to prospective teachers. The commission’s other duties include
effectiveness of the
adopting credential exams, accrediting colleges and universities
programs it oversees
and better measure the that meet program standards, operating teacher development
performance of the teacher programs designed to help prospective teachers complete the
credentialing process.
requirements needed for a credential, and reviewing allegations
þ The commission could of misconduct against credential holders or applicants. Our
take additional steps to review found that the commission could make improvements
improve its processing of
to better evaluate the programs it oversees and its internal
credential applications,
operations, more effectively manage its application processing,
including focusing its
customer service activities. and refine how it updates program standards. In addition, the
commission should resume its continuing accreditation reviews
þ Several areas of the
of colleges and universities.
commission’s process
for developing program
standards lack structure The commission could increase its ability to measure the
and could be improved.
effectiveness of its teacher development programs, the
þ The commission efficiency of the teacher-credentialing process (process), and
suspended its continuing the performance of its internal operations. By doing so, the
accreditation reviews
commission would be able to streamline and improve its efforts.
in December 2002
For example, its teacher development programs provide funding
and is evaluating its
accreditation policy, and for individuals who do not yet meet the requirements for a
it does not expect to teaching credential, yet the commission has not sufficiently
present a revised policy to
evaluated and accurately reported on two of its three teacher
its governing body until
development programs. As part of its oversight of the process
August 2005.
in California, the commission has some measures of the overall
health of the process. However, it could improve its analysis of
those measures and could develop further measures to better
track the performance of the process and of individual teacher
preparation programs.
California State Auditor Report 2004-108 11
Despite the importance of strategic planning, the commission
has lacked specific performance measures to guide and evaluate
its efforts. Further, the commission’s February 2001 strategic
plan is outdated and lacks performance measures. In addition,
the commission does not annually track its progress
in completing the tasks it described in the strategic plan.
Subsequent to our fieldwork, the commission updated the tasks
in its strategic plan.
The commission has implemented some reforms of the process
and is contemplating others. It has also worked to reduce the
barriers to becoming a California teacher. In addition to these
efforts, the commission is considering whether to consolidate
the examinations that it requires prospective teachers to pass.
By focusing its customer service, better managing its workload,
and taking full advantage of a new automated application-
processing system, the commission could improve its processing
of applications. Facing a significant volume of contacts, the
commission has not taken sufficient steps to focus its customer
service activities. Proper management of customer service is
necessary because the large volume of telephone calls and
e-mails that the commission receives takes staff away from the
task of processing credential applications.
Although the commission typically processes applications
for credentials in less than its regulatory processing time of
75 business days, applications go unprocessed for a significant
amount of this time because staff members are busy with other
duties. The commission has taken some steps to improve its process,
including automating certain functions as part of its Teacher
Credentialing Service Improvement Project (TCSIP), which
is a new automated application processing system that the
commission plans to implement in late October 2004. However,
the commission has not performed sufficient data analysis to
make informed staffing decisions. TCSIP offers tangible time-
saving benefits, such as allowing colleges and universities
to submit applications electronically and automating the
commission’s review of online renewals, but the commission
does not plan to use either function to its full potential in the
foreseeable future.
Although online renewals offer the benefit of faster and more
efficient processing, the commission has not sufficiently
publicized this benefit. The commission could do more to
inform teachers about the benefits of online renewal by
22 California State Auditor Report 2004-108 California State Auditor Report 2004-108 33
performing the data analysis necessary to determine where
the commission needs to do additional outreach and by better
highlighting online renewal’s availability and faster processing time.
The commission is in the midst of a 10-year process of
developing program standards that comply with the
requirements of Senate Bill 2042, Chapter 548, Statutes of 1998
(act). The commission does not have an overall plan to guide
its efforts to finish implementing program standards or its
ongoing standard-setting activities. Further, the commission’s
recent experiences developing program standards to meet
the act’s requirements offer an opportunity to evaluate how
to better manage its future efforts. Our review of five sets
of recently developed program standards identified areas in
the commission’s process for developing program standards
that lack structure and could be improved. Among other
issues, the commission does not use a methodical approach
to form advisory panels of education professionals that assist
it in developing program standards; neither does it always
put in perspective the results of its field-review surveys to
the commission’s governing body (commissioners) when
recommending standards for adoption.
Finally, the commission suspended its continuing accreditation
reviews of colleges and universities in December 2002.
Continuing accreditation reviews are an important component
of the commission’s accreditation system and help ensure that
colleges and universities operate teacher preparation programs
that meet the commission’s standards. The commission
indicated that it suspended continuing accreditation reviews
to allow colleges and universities time to implement the
commission’s new standards and for it to evaluate its
accreditation policy. Although the commission has been
working with representatives from colleges and universities to
evaluate its accreditation policy, it does not plan to propose a
revision to the commissioners until August 2005.
RECOMMENDATIONS
To determine their success, the commission should establish
performance measures for each of its teacher development
programs.
22 California State Auditor Report 2004-108 California State Auditor Report 2004-108 33
To better plan and evaluate its efforts, the commission should
regularly update its strategic plan and when appropriate
quantify performance measures for tasks, in terms of the results
it aims to achieve.
The commission should continue to consider ways to streamline
the process.
The commission should improve application processing
by better focusing its customer service efforts, analyzing
application-processing data, requiring institutional customers
to submit applications electronically to the extent that it is
economically feasible, and encouraging more educators to renew
their credentials online.
To improve the process by which it develops program standards
for college and university teacher preparation programs, the
commission should develop an overall plan to guide its efforts
to fully implement the act’s requirements. This plan should
describe the commission’s process for developing standards
and should provide more structure for that process. Further,
to ensure that colleges and universities meet these program
standards, the commission should promptly resume its
continuing accreditation reviews.
AGENCY COMMENTS
The commission concurs with many of our recommendations,
but believes that it will need changes in its statutory authority
or additional funding and staffing to implement them.
Moreover, the commission believes the report has significant
omissions, errors, and misinterpretations. We carefully analyzed
the commission’s response and, although we made some
minor modifications to the report text, we stand by our audit
conclusions and recommendations. n
44 California State Auditor Report 2004-108 California State Auditor Report 2004-108 55
INTRODUCTION
BACKGROUND
The California Commission on Teacher Credentialing
(commission) was created in 1970, with the responsibility
to ensure excellence in education by establishing
high standards for the preparation and licensing of public
school educators. Previously, the California Department of
Education (Education) issued licenses, known as credentials, to
teachers, but the Legislature believed that the public would be
better served by having a separate state entity issue teaching
credentials. The commission also issues licenses and permits
for school administrators and educators working in specialized
teaching areas.
The commission’s governing body (commissioners) is a group of
19 individuals, of whom 15 are voting members. The governor,
with the advice and consent of the State Senate, appoints 14 of
the voting members; the superintendent of public instruction
or a designee is the remaining voting member. The Regents
of the University of California, the Trustees of the California
State University, the California Postsecondary Education
Commission, and an association representing independent
colleges and universities each provide one of the nonvoting
members. State law requires the commissioners to meet at
least once each month in no fewer than 10 months per year to
conduct commission business. The commissioners also appoint
an executive director who is responsible for the commission’s
daily operations. As of September 2004 the commission had
165 positions. Figure 1 on the following page shows the
organization of the commission.
The certification, assignments, and waivers division evaluates
and processes teacher and administrator applications and
renewals, which totaled approximately 239,000 in fiscal
year 2003–04. It also provides customer service and deposits
fees, among other functions.
The professional services division develops program standards
that college and university teacher preparation programs must
adhere to in order for the commission to accredit them. These
program standards address the quality of the programs that
accredited colleges and universities provide to prospective
44 California State Auditor Report 2004-108 California State Auditor Report 2004-108 55
teachers, including curriculum, field experiences, knowledge,
and skills. In addition, this division implements credential
examinations, accredits colleges and universities that meet
program standards, and administers teacher development
programs designed to help prospective teachers complete the
requirements they need for a credential.
FIGURE 1
Organization of the California Commission on Teacher Credentialing
������������
���������
����
���������������
���������
��������
�����������
��������������
������������ ���������� �������� ���������� ��������
������������
�������� ������������ ������������ ���������� �����
��� �������
�������� ��������� ��������� ���������� ���������
��������
��������
Source: California Commission on Teacher Credentialing.
The division of professional practices helps the commission’s
committee of credentials (whose members the commissioners
appoint) review allegations of misconduct against credential
holders or applicants. The committee of credentials also makes
recommendations to the commissioners as to whether probable
cause exists to take adverse action against those individuals.
In situations in which a credential holder or applicant has
been convicted of a serious crime, the commission revokes or
mandatorily denies the credential or application. According to
its records, the commission resolved approximately 4,900 cases
in fiscal year 2003–04.
License fees and license examination fees support the commission.
The fiscal year 2004–05 license fee is $55 for all credential
types and for renewals; this fee funds all of the commission’s
66 California State Auditor Report 2004-108 California State Auditor Report 2004-108 77
operating expenses. The State establishes the fee amount
annually in its budget act. The last fee change was in fi scal
year 2000–01, when the State reduced the fee from $60 to
$55. Examination fees can range from $41 to $226, with the
commission receiving a portion of these fees and the companies
that administer the examinations receiving the rest. The
commission’s portion of examination fees is designated for the
development, maintenance, and administration of tests and
other assessments. In fi scal year 2003–04 the commission’s
operating budget was $26.6 million.
OVERVIEW OF THE TEACHER-CREDENTIALING PROCESS
California law requires that the commission issue licenses,
known as teaching credentials, to individuals teaching
in California’s public schools. Many types of teaching
and administrative credentials, permits, and
certifi cates exist. As the text box shows, among
the most common are single-subject credentials
Type and Number of Credentials
for those teaching in grades 7 through 12 and
Granted During Fiscal Year 2003–04
multiple-subject credentials for those teaching in
grades kindergarten through 6. In addition, the
Single subject 31,453
commission issues emergency permits to those
Multiple subject 59,698
who have completed most of a credential program
Emergency and 30-day
substitute permits 72,617 or hold a credential in another subject, and it also
Administrative services 10,400 issues 30-day substitute permits to those that serve
All others 64,571 as a day-to-day substitute teacher in a classroom.
Total 238,739
Typically, to earn a teaching credential in
California, a prospective teacher must earn
Source: California Commission on Teacher
Credentialing. a college degree at an accredited college or
university. The individual must demonstrate
academic preparation in the subject he or she
wishes to teach by either completing an approved
course of study in the subject or passing an examination on
the subject (prospective teachers seeking a multiple subject
credential must take and pass the examination to demonstrate
academic preparation). In addition, the prospective teacher
must complete a teacher preparation program and take several
examinations, depending on the credential he or she seeks. After
the prospective teacher’s college work is complete, the college or
university that the individual attended submits an application
to the commission, which the commission evaluates against the
credential requirements. If the prospective teacher meets the
requirements and passes a background check, the commission
will grant a preliminary credential. The preliminary credential
66 California State Auditor Report 2004-108 California State Auditor Report 2004-108 77
allows the individual to teach for up to five years. To continue
teaching after that time, the individual must complete a fifth
year of study at an accredited California college or university, or
obtain National Board Certification, and complete an induction
program (a program of support and assessment). Once these
requirements are completed, the individual or the individual’s
sponsor applies to the commission, which grants the individual
a professional clear credential if all items are in order. After this
time, the teacher needs to renew the credential every five years
by passing a background check, meeting professional growth
and service requirements, and paying the renewal fee.
Because the demand for teachers often exceeds the supply of
credentialed teachers available, schools have several options to
fill teacher positions. The most common of these is the use of
an emergency permit. The commission often issues such permits
to persons who have completed most of a credential program
or who hold a credential in a subject area different from the
one they will be teaching. Emergency permits allow individuals
to teach for one year, after which they may reapply up to four
times, apply for a credential if they now meet the requirements,
or discontinue teaching. The commission also administers several
programs designed to help prospective teachers complete the
requirements for a teaching credential with the assistance of a
school district, county office of education, college, or university.
OTHER ENTITIES INVOLVED IN THE TEACHER-
CREDENTIALING PROCESS
Although the commission has a lead role in the teacher-
credentialing process (process), other entities play a supporting
role. Colleges and universities prepare prospective teachers by
offering teacher preparation programs as a course of study. To
provide a teacher preparation program, a college or university
must have its program accredited by the commission as meeting
the commission’s standards. As of August 2004 the commission
had accredited more than 80 California colleges and universities
and eight school districts to offer teacher preparation programs.
Of these, the California State University system prepared the
most teacher candidates, almost 13,000 in fiscal year 2002–03.
The commission is required to align the program standards that
colleges and universities follow in their teacher preparation
programs with the State Board of Education’s (state board)
academic content standards. Under the direction of the
state board and the superintendent of public instruction,
88 California State Auditor Report 2004-108 California State Auditor Report 2004-108 99
Education provides education policy direction to local school
districts. The state board is the governing and policy-making
body for Education and sets policy for academic content in grades
kindergarten through 12. School districts and county offices of
education hire credentialed teachers, but they also coordinate
with the commission to administer several teacher development
programs and to hire teachers who do not have a credential.
STATE AND FEDERAL POLICY DECISIONS AFFECT THE
PROCESS
Recent changes in state and federal law have significantly
affected the process by increasing the need for teachers and
raising the standards of quality for the teachers that public
schools hire.
The State’s class-size reduction programs are intended to
improve educational achievement for all students by reducing
the number of students per teacher. School districts that choose
to participate in these programs receive additional funding for
each student enrolled in classes of about 20 students. The State
implemented the most recent program, for grades kindergarten
through 3, in the 1996–97 school year. Although this program
provided students the benefit of closer instruction, it created
an immediate need for credentialed teachers to achieve the smaller
class sizes. Because not enough credentialed teachers were available
to fill this need when the law became effective, many school
districts needed to hire teachers by using emergency permits.
Another law affecting the process is the federal No Child Left
Behind Act of 2001 (federal act). A key goal of the federal act is
to ensure that all students are taught by highly qualified teachers
by the end of the 2005–06 school year. To be highly qualified, a
teacher of core academic subjects must meet three requirements:
He or she (1) must have a bachelor’s degree, (2) must either have
a state teaching credential or hold intern status for no more
than three years, and (3) must demonstrate competence in the
core academic subject matter. To implement these requirements,
the State must align its existing credentialing and professional
development practices with the federal act’s goals. The federal
act has caused the commission to study how to replace
emergency permits, because holders of emergency permits have
not demonstrated competence in the core academic subject
matter and thus do not meet the federal act’s definition of
highly qualified teachers. To ensure that the State meets the
teacher requirement goal for core academic subjects, Education
88 California State Auditor Report 2004-108 California State Auditor Report 2004-108 99
has advised schools receiving Title I funds (a federal program
that provides funding to schools with children at risk of failing
to meet the State’s academic requirements) to hire only highly
qualified teachers beginning with the 2002–03 school year.
Education has further stated that all other public school teachers
who are teaching core academic subjects should meet the federal
act’s requirements by the end of the 2005–06 school year.
SCOPE AND METHODOLOGY
The Joint Legislative Audit Committee (audit committee)
asked the Bureau of State Audits to study the effectiveness and
efficiency of the process, including the relevant laws, rules, and
regulations. The audit committee asked us to take a number of
steps to study the effectiveness of the commission’s activities.
Specifically, the audit committee asked us to determine whether
the commission evaluates the effectiveness of its training,
development, and certification practices, including whether
it has established and met meaningful performance measures.
The audit committee also requested that we determine whether
any barriers prevent qualified individuals from being certified as
teachers. In addition, the audit committee asked us to review the
commission’s process for handling applications to identify any
backlog, average processing times, and any causes or contributors
to delayed processing. Further, the audit committee asked us to
examine the process for establishing and evaluating standards
for college and university programs that prepare prospective
teachers and administrators. Finally, the audit committee asked
us to examine whether the commission and other entities
involved in the process overlap in roles and activities. After
reviewing the roles of the commission and Education, the
California State University, the University of California, the
Board of Governors of the California Community Colleges,
private universities, and the California Postsecondary Education
Commission, we found no significant duplication of efforts
among these entities.
Good management practices suggest that organizations use
performance measures as a way of determining whether their
efforts are productive and successful. Thus, we reviewed whether
the commission evaluates the effectiveness of the teacher
development programs it administers. Further, we evaluated
the commission’s February 2001 strategic plan, which the
commission partially updated after the end of our fieldwork,
to determine whether the commission has established and met
its own performance measures and goals. In the areas of the
1100 California State Auditor Report 2004-108 California State Auditor Report 2004-108 1111
commission’s key responsibilities of application processing,
standards development, and teacher discipline, we examined
whether the commission has established performance measures,
goals, or expectations to guide its staff’s efforts toward successful
outcomes. Because the commission both collects data on
activities related to the process and has an oversight role, we also
explored whether these data could provide performance measures
that the commission could use to gauge the relative success of its
efforts and the overall health of the process. Finally, we examined
the commission’s efforts to remove barriers that prevent viable
teacher candidates from receiving teaching credentials.
Processing of credential applications and renewals is one of the
commission’s primary responsibilities, and thus we reviewed
how it manages this workload. Because a significant workload
of applications and renewals exists, we reviewed whether
the commission has taken appropriate steps to evaluate this
workload to determine how to make the process more efficient.
Because customer service activities take staff away from
processing applications and renewals, we evaluated how the
commission uses the customer service data it gathers, such as the
reasons for the large number of e-mail and telephone inquiries it
receives, to make its customer assistance activities more efficient.
In late October 2004 the commission plans to implement the
third phase of the Teacher Credentialing Service Improvement
Project (TCSIP) to replace the existing system for processing
applications and renewals. We assessed the efficiencies that
the commission will gain with TCSIP and whether additional
efficiencies would be possible by automating more functions. We
also reviewed whether the option to renew credentials online,
which became available in July 2002, has made the process more
efficient for the commission and its customers.
To examine how the commission establishes and evaluates
teacher and administrator program standards, we reviewed
five sets of recently developed program standards that colleges
and universities either have implemented or are in the process
of implementing. The commission is in the midst of developing
new program standards as a result of Senate Bill 2042,
Chapter 548, Statutes of 1998. Because the commission used
an advisory panel of education professionals to assist in the
development of each set of program standards we reviewed,
we determined whether it selected panel members objectively
based on their qualifications. Further, we examined how the
commission ensures that the content of the program standards
is appropriate, is considered important by the California
1100 California State Auditor Report 2004-108 California State Auditor Report 2004-108 1111
education community, and is without bias. The development
of the program standards is taking place over at least a 10-year
period, so we assessed whether the commission has a plan to
ensure that it develops program standards on a timely basis.
We also reviewed the commission’s accreditation process,
because accreditation of teacher preparation programs at colleges
and universities is critical to ensuring that these programs
follow the commission’s standards. The commission suspended
continuing accreditation reviews of colleges and universities
in December 2002 and is evaluating the accreditation
process. Thus, we reviewed the status of the revisions to the
commission’s accreditation process. n
1122 California State Auditor Report 2004-108 California State Auditor Report 2004-108 1133
CHAPTER 1
The Commission Could Better
Evaluate the Effectiveness of
Its Efforts and Better Measure
the Performance of the Teacher
Credentialing Process
CHAPTER SUMMARY
The California Commission on Teacher Credentialing
(commission) could increase its ability to measure the
effectiveness of its teacher development programs,
the teacher-credentialing process (process), and its internal
operations. By doing so, the commission would be able to
streamline and improve its efforts and the process.
The commission’s teacher development programs provide
funding for individuals who do not yet meet the requirements
for a teacher credential. However, the commission has neither
sufficiently evaluated nor accurately reported on two of its three
teacher development programs. For example, the commission
did not have the effectiveness of the California School
Paraprofessional Teacher Training Program (paraprofessional
program) independently evaluated, as state law requires, and has
overstated the benefits of the Pre-Internship Teaching Program
(pre-intern program) in a report to the Legislature.
In its oversight role of the process in California, the commission
has some measures of the overall health of the process. However,
it could improve its analysis of those measures and could
develop further measures to better track the effectiveness of
the process and of individual teacher preparation programs
(preparation programs) that colleges, universities, and school
districts offer. Further, if the commission and other entities
involved worked to resolve funding and technical issues, the
commission could use the results of the teaching performance
assessment, annual data on retention of teachers, and
administrator surveys that are currently under development
to better measure various aspects of the process and the
preparation programs.
1122 California State Auditor Report 2004-108 California State Auditor Report 2004-108 1133
The commission has lacked specific performance measures
to guide and evaluate its efforts. Further, the commission’s
February 2001 strategic plan is outdated and lacks performance
measures, and the commission does not annually track its
progress in completing the tasks its plan described. Subsequent
to our fieldwork, the commission updated the tasks in its
strategic plan.
The commission has implemented some reforms of the process
and is contemplating others. For instance, it has reduced the
barriers to becoming a California teacher for individuals who
received their teacher training in other states. It has also developed
tests that allow teacher candidates who pass them not to take
otherwise required preparation courses. In addition to these
efforts, the commission is considering whether to consolidate the
examinations it requires of prospective teachers.
THE COMMISSION HAS NEITHER FULLY EVALUATED
NOR ACCURATELY REPORTED THE RESULTS OF TWO OF
ITS THREE TEACHER DEVELOPMENT PROGRAMS
The commission’s teacher development programs—the
paraprofessional, pre-intern, and the California Internship
For two teacher Teacher Preparation (intern) programs—provide funding to help
development programs, individuals meet the requirements for a teaching credential
state policy makers do or, in some cases, to meet the requirements for entrance to
not have the information a college, university, or school district preparation program.
they need to determine Because the commission’s evaluation of its paraprofessional
whether the programs and pre-intern programs has been limited and, in some
have been successful. cases, inaccurately reported, state policy makers do not have the
information they need to determine whether these programs have
been successful and should be continued. However, the commission
has collected information that indicates that the intern program has
been successful in meeting its objectives.
The Commission Has Not Adequately Evaluated the
Performance of the Paraprofessional Program
Despite spending more than $34 million on the paraprofessional
program since fiscal year 1994–95, the commission has not had
a third party independently evaluate the program as statute
requires, nor has it established relevant and valid measures
of the program’s performance. The primary purpose of the
paraprofessional program is to help school paraprofessionals,
such as teachers’ assistants, library media aides, and
instructional assistants, become certificated classroom teachers
1144 California State Auditor Report 2004-108 California State Auditor Report 2004-108 1155
in public schools. The law requires the program to focus on
recruiting to meet the demand for certain kinds of teachers,
such as cross-cultural/bilingual, special education, kindergarten
through third grade, and local education agencies’ own specifi c
teacher needs. The program provides scholarships of about
$3,000 per year to defray participants’ costs for tuition, books,
and fees. Participants continue to serve as paraprofessionals
while completing their education. According to the
commission, because participants typically go to school only
part-time and because most enter the program having completed
relatively few college courses, completing their education may
take them as long as seven years. However, commission staff
members stated that they have not limited the number of years
that a participant may receive a paraprofessional scholarship.
For every year that they receive a scholarship, participants
must commit to teach for one school year after receiving a
preliminary credential in the districts or county offi ces of
education through which they received assistance. Therefore, in
return for administering the paraprofessional program, school
districts and county offi ces of education receive a number of
years of service from a fully qualifi ed teacher who
is already accustomed to a particular school district’s
environment. According to the commission, other
Data Requirements for
Independent Evaluation of the anticipated benefi ts include improving the instruction
Paraprofessional Program that paraprofessionals provide while in the program
and diversifying the teaching profession.
1. Total cost per program graduate.
2. Economic status of program participants. Although the state law that authorized the
paraprofessional program requires it to do so,
3. Description of the other resources made
available to participants. the commission has not contracted with an
independent evaluator to determine the success
4. Comparison of performance on standardized
tests between pupils taught by program of the paraprofessional program. The law requires
graduates and pupils taught by others with an annual evaluation that includes the seven data
equivalent experience.
requirements described in the text box. However,
5. Improvements in pupil dropout rates and commission documents indicate that the cost of this
other measures of delinquency in classes
evaluation is a concern; and the commission has
taught by program graduates.
never contracted for a third party to conduct the
6. Extent to which program graduates remain
independent evaluation.
in the communities in which they reside
and in which they teach.
In its December 1997 meeting, the commission’s
7. Attrition rate of program graduates.
governing body (commissioners) decided to
postpone the independent evaluation until
Source: California Education Code, Section 44393.
an unspecifi ed future date. In the agenda item
associated with this meeting, commission staff
estimated that an evaluation that fully addressed
1144 California State Auditor Report 2004-108 California State Auditor Report 2004-108 1155
all of the statutory requirements would cost at least $1 million,
although the commission had not solicited bids for this
evaluation. At that time, the paraprofessional program was
funded at nearly $1.5 million per year. Part of the reason the
commission staff gave for its high estimate of the evaluation’s
cost was that two of the requirements (requirements 4 and
5 in the text box) would be difficult to measure because the
independent evaluator would have to compile data about
a multitude of factors that cause fluctuations in student
performance on standardized tests and that contribute to the
dropout rate. Commission staff also suggested other options,
such as proceeding with the evaluation but excluding the
two problematic requirements and submitting a proposal
to the Legislature to delete or substantially change the two
requirements. According to the commission’s director of
governmental relations, after December 1997 the commission
verbally informed the Legislature and the Governor’s Office
about the lack of funding for the evaluation. Although the
commission provided us with copies of several budget requests
for $33,000 (the amount it believed the evaluation would cost
before the Legislature added the two problematic evaluation
requirements to the law in 1997) that it made before December
1997, the commission could not provide us documentation of
budget requests that it made after that date.
In addition to requiring an independent evaluation, state law
requires the commission to provide the Legislature an annual
status report that includes, among other elements, the number
of paraprofessionals in the program who are subsequently
employed as teachers in the public schools. This retention rate
is one of the more critical measures of the program’s success in
meeting its objectives, but two problems weaken its value as a
program performance measure. First, the local program directors
report the retention rate themselves, and neither they nor the
commission track program graduates after they fulfill their years-
of-service obligation. Second, the commission could not provide
supporting documentation for the 823 program graduates that
it reported as of the end of fiscal year 2002–03 who were still
serving in the classroom, and this number was inconsistent with
a table in the report indicating that local programs reported
only 616 program graduates still serving in the classroom.
Commission staff could not provide us with supporting
documentation for the table either. This inconsistency and lack
of supporting documentation casts doubt on the validity of
1166 California State Auditor Report 2004-108 California State Auditor Report 2004-108 1177
the data the commission used in its fiscal year 2002–03 annual
report and makes it difficult for policy makers to determine the
program’s success.
Although the commission collects data from local program
directors on retention and other matters involving the
Because the commission paraprofessional program, it does not use these data as
did not develop ways performance measures to evaluate the success of individual
to measure and local programs. Further, commission staff explained that when
monitor local program the commission developed the paraprofessional program, it did
performance, nearly not establish a way to monitor whether local programs were
70 participants of verifying participants’ academic progress and did not develop
programs whose a set of consequences for underperformance by local programs
participation was or by individual program participants. The commission is now
scheduled to end by trying to determine what to do about nearly 70 participants
December 2003 have in local programs who were to have completed their credential
not completed credential requirements by December 2003 or sooner but who have not
requirements. completed credential requirements. If the commission had
developed ways to measure and monitor local program performance
and developed specific consequences for underperformance, it
might have been able to avoid this dilemma.
The Commission Has Not Demonstrated That the Pre-Intern
Program Has Accomplished Its Objectives
In 1997, when the number of teachers serving with emergency
permits was increasing dramatically, legislation created the
pre-intern program to help retain and prepare emergency-
permit teachers to become credentialed teachers. However, after
seven years of program operation and $80 million in costs, the
commission has not shown that the pre-intern program has
achieved this and other program goals. The pre-intern program
is ending after the 2004–05 school year because pre-interns do
not meet the requirements of the federal No Child Left Behind
Act of 2001 (federal act), which requires a highly qualified teacher
in every classroom after June 2006. Although the commission
is discontinuing the pre-intern program and has no current
plans to replace it with a similar funded effort, our review of the
commission’s process of evaluating the program offers insight
into how the commission can improve its practices.
One purpose of the pre-intern program is to decrease the
number of teachers holding emergency permits by providing
individuals who have not yet completed the subject matter
requirements for entry into a preparation program with education
in the subject matter that they are assigned to teach. Another
1166 California State Auditor Report 2004-108 California State Auditor Report 2004-108 1177
purpose is to improve participants’ teaching performance by
providing training in classroom management, pupil discipline,
and teaching strategies. To achieve these goals, the commission
annually awards to competitively selected counties and local
school districts (local programs) grants of $2,000 to provide
training and support for each pre-intern participant. Individuals
can participate in the program for two or, in some cases, three
years. The requirements for an individual to participate are
the same as those for an emergency permit: Both require the
individual to pass the California Basic Educational Skills Test and
to have a bachelor’s degree with a minimum number of units in
the subject of the teaching assignment.
In October 2001, after three years of program operation, the
commission submitted a final program report to the Legislature,
as state law required, providing information on the success
of the pre-intern program and including recommendations
on whether the Legislature should continue, modify, or
discontinue the program. In the report, the commission
recommended that the Legislature continue the program with
increased funding, based on the following measures of program
performance that the commission provided:
• In the first two years of the pre-intern program, almost
90 percent of all pre-interns were retained for a second year,
compared to around 65 percent of first-year emergency-
permit teachers.
• In the first two years, nearly 60 percent of pre-interns passed
their subject matter examinations. These pass rates are similar
to those of all test takers, despite the fact that pre-interns are
largely members of groups that tend to pass at lower rates
than the general population.
• Sixty-one percent of principals responding to a commission
survey believed that pre-intern teachers performed “better”
or “much better” than other teachers with a similar amount
of experience.
Although these three measures indicate some level of program
success and may support a recommendation to continue the
program, we found problems with them. For instance, to support
the claim that only 65 percent of first-year emergency-permit
holders are retained for a second year, the commission provided
us with a 1996 agenda item describing a then-recent commission
study of a sample of individuals holding multiple-subject
1188 California State Auditor Report 2004-108 California State Auditor Report 2004-108 1199
emergency permits. To obtain a more current view of retention, we
used commission data for fiscal years 1998–99 and 1999–2000 and
calculated that 80 percent of first-year emergency-permit holders
were retained in those years. This compares more favorably to the
90 percent retention rate for pre-interns during the same period.
Further, the commission could not provide supporting
documentation for the statement that pre-interns are
largely members of groups that tend to pass subject matter
Given the problems examinations at lower rates than the general population.
we found with the Without these data, it is difficult for the commission to defend
commission’s evaluation, its position that the rate at which pre-interns pass these
we believe that it examinations, which the commission says is similar to the pass
may have incorrectly rates of all test takers, is an indication of program success. In
concluded that the pre- addition, the commission could not provide us with supporting
intern program should documentation for the results of the principal survey cited
be continued without in its report. Although we could not examine the statement’s
substantial modification. validity, the results of the principal survey alone would not
indicate program success. Given the problems we found with the
commission’s evaluation, we believe that the commission may
have incorrectly concluded that the pre-intern program should
be continued without substantial modification.
Indeed, subsequent to this final program report, the commission
conducted a review of the pre-intern program that cast some
doubt on the program’s success in helping participants reach
credential status. The commission randomly selected 248 pre-
interns from the 5,800 program participants in fiscal year
1999–2000 and determined their credential status as of fiscal
year 2002–03. Three years after taking part in the two-year
pre-intern program, only 45 percent of participants in the
commission’s study had received either an intern position
or a preliminary or professional clear credential. Because the
legislative intent of the program was to help pre-interns progress
into an internship as quickly as possible, we believe these initial
findings should have caused the commission to determine the
reason for these lower-than-expected results and implement
corrective action.
No Requirement Exists to Formally Evaluate the Intern
Program, but Commission Data Points to Program Success
Although the Legislature has not required the commission to
report the results of the intern program, the commission has
collected information that indicates the program is meeting its
objectives. A teaching internship allows an individual to serve as
1188 California State Auditor Report 2004-108 California State Auditor Report 2004-108 1199
a fully paid teacher of record while simultaneously participating
in a one- or two-year preparation program through a university
or school district. The State has authorized university and school
district teaching internships since 1967 and 1983, respectively.
Therefore, in 1993, when legislation established a local
assistance program to provide funding for teaching internships,
the internship model of teacher preparation was not a new
concept. At that time, the Legislature did not build any annual
reporting requirements or independent evaluations into the
intern program, as it did with the commission’s other teacher
development programs.
Despite this lack of external reporting requirements, the
Commission data commission has collected information that indicates the program
indicates that the intern has been successful in bringing underrepresented groups and
program has been individuals qualified to teach in hard-to-staff subject areas
successful in retaining into the education workforce. In addition, commission data
teachers and advancing indicate that the intern program has been successful in retaining
them toward full teachers and advancing them toward full credential status.
credential status. Based on information the commission collected from reports of
local intern programs, 85 percent of fiscal year 1998–99 interns
were retained after five years; according to the commission’s
credential database as of August 2004, approximately 90 percent
of all individuals who participated in the intern program before
fiscal year 2002–03 have obtained full credential status.
THE COMMISSION COULD IMPROVE ITS ABILITY
TO MEASURE THE PERFORMANCE OF PREPARATION
PROGRAMS AND THE PROCESS
In its oversight role of the process in California, the commission
has some measures of the overall effectiveness of the process;
however, the commission could improve its analysis of those
measures and could develop further measures to better track
the performance of the process and of college, university,
and school district preparation programs. In particular, the
commission annually reports on the number of California
teaching credentials it issues and the number of emergency
permits and credential waivers it grants. However, it provides
this information with limited, if any, analysis of the trends
associated with these numbers and does not account for external
factors that could affect these statistics. In addition, if the
commission and the other entities involved worked to remove
current obstacles, the commission could use the results of the
teaching performance assessment, annual data on retention
2200 California State Auditor Report 2004-108 California State Auditor Report 2004-108 2211
of teachers, and administrator surveys that are currently in
development to better measure various aspects of the process
and the preparation programs.
The Commission Is Required to Provide Oversight of the
Process and Preparation Programs
Statute requires the commission to establish professional
standards, assessments, and examinations for entry and
advancement into the education profession. Although
legislation provides the basic framework for standards,
assessments, and examinations, the law recognizes that the
commission will exercise its prerogative to determine the
details of those requirements. In addition, the law requires the
commission to streamline the credentialing system to ensure
competence in the subject areas while allowing flexibility in
staffing local schools. Our legal counsel believes that the law
allows the commission to exercise professional judgment in
developing credential requirements and then authorizes the
commission to oversee the credentialing system, looking to
improve teacher quality while maintaining an adequate quantity
of teachers to staff local schools. In addition, as the entity that
accredits and establishes standards for preparation programs, the
commission has the responsibility to monitor the performance
of these programs to ensure that they follow through on the
plans they submitted during the accreditation process and to
determine whether the preparation program standards result in
quality teachers.
The Commission Has Some Performance Measures of the
Process but Could Improve Its Analysis of Those Measures
Because of its oversight role, the commission has some
performance measures of the process but it could improve its
analysis of those measures. For instance, state law requires the
Although the commission commission to report annually the number of individuals
annually reports recommended for credentials by colleges and universities and
statistics on the teacher by school districts operating internship programs and to report
credentialing process, it the number of individuals receiving a credential from a training
provides limited, if any, program completed outside of California. The same state law
analysis of the trends declares that the number of teaching credentials the commission
associated with these issues is a measure of whether preparation programs are meeting
numbers. the challenge of preparing increasing numbers of new teachers.
Figure 2 on the following page shows a general increase in the
number of teaching credentials that the commission issued
since fiscal year 1991–92. However, in its annual reports, the
2200 California State Auditor Report 2004-108 California State Auditor Report 2004-108 2211
commission does not explain why the number has changed or
what efforts have been particularly successful in increasing the
number of credentialed teachers.
FIGURE 2
Number of California Teacher Credentials Issued by Year
Fiscal Years 1991–92 Through 2002–03
������
������
������
������
������
�����
�
� � � � � � � � � � � �
� � � � � � � � � � � �
� � � � � � � � � � � �
� � � � � � � � � � � �
� � � � � � � � � � � �
� � � � � � � � � � � �
� � � � � � � � � � � �
�
�
����������
��������������������������������
��������������������������
���������������������������������
��������������������������������������������������
Source: California Commission on Teacher Credentialing.
Another statutorily required measure of the capacity of the
process to meet the demand for teachers is the number of
emergency permits and credential waivers (permits and
waivers) the commission issues. When a school cannot recruit
a credentialed teacher for a vacancy, these permits and waivers
allow schools to hire individuals who do not have a teaching
credential or do not hold the applicable teaching credential.
2222 California State Auditor Report 2004-108 California State Auditor Report 2004-108 2233
Therefore, the number of permits and waivers that the
commission issues is a strong indicator of whether the supply of
The commission has not credentialed teachers currently meets the demand. Although the
used important factors commission has presented the overall numbers of permits and
to determine and explain waivers in its annual reports, it has not used important factors, such
why the number of as the unemployment rate, to determine and explain why these
emergency permits has numbers have decreased in recent years.
decreased in recent years.
A number of factors independent of the process could affect
the number of teaching credentials, permits, and waivers issued
each year, and the commission should account for these factors
when analyzing this type of data. For instance, the relative
attractiveness of the teaching profession—salary, benefits,
and job conditions compared to other available employment
opportunities—affects an individual’s desire to enter and stay in
the teaching profession.
The unemployment rate is a measure of the other employment
opportunities available to prospective or established teachers.
According to information contained in the Little Hoover
Commission’s 2001 report titled Teach Our Children Well, during
times of high unemployment, the proportion of teachers with
emergency permits decreases; in turn, during times of low
unemployment, the proportion of teachers with emergency
permits increases. Figure 3 on the following page shows that
from fiscal years 1999–2000 to 2002–03, when unemployment
rates in California increased, permits and waivers decreased.
Analyzing the factors we have just discussed would be useful,
but the state law that requires the commission to report these
statistics does not instruct the commission to include an
analysis of trends. However, because the commission is the state
entity providing oversight of the process and of preparation
programs, it seems reasonable that the commission would formally
analyze and discuss these indicators of the process and programs’
performance annually. When these statistics appear without any
analysis or discussion, they hold little value, and users of the reports,
such as legislators and educators, must draw their own conclusions.
2222 California State Auditor Report 2004-108 California State Auditor Report 2004-108 2233
FIGURE 3
Number of Emergency Permits and Waivers Compared to the Unemployment Rate
Fiscal Years 1992–93 Through 2002–03
��� ������
������
��
������
� ������
������
�
������
������
�
If Fully Implemented, the Teaching Performance Assessment
Could Be Used to Measure the Performance of Teacher
Preparation Programs
In addition to better analyzing the data it already collects, the
commission could also annually collect data related to the quality
of preparation programs. The primary tool the commission
currently uses to evaluate these programs’ quality is its
continuing accreditation reviews, which include, as we discuss
in Chapter 3, on-site reviews of each preparation program
��������������������������
�������������������������
����������������
�����������������
� � � � � � � � � � �
� � � � � � � � � � �
� �
��
� �
��
� �
��
� �
��
� �
��
� �
��
� �
��
�� �
�
� �
��
� �
��
� �
��
� � � � � � � � � � �
�
�
����������
Sources: Permits and waivers data are from the Certification, Assignments, and Waivers Division of the California Commission on
Teacher Credentialing. Unemployment data are from the Employment Development Department.
2244 California State Auditor Report 2004-108 California State Auditor Report 2004-108 2255
every five to seven years. However, the commission can also
use data from the teaching performance assessment, a partially
implemented assessment of prospective teachers’ ability to
teach, to evaluate preparation programs and reform efforts.
Although Senate Bill 2042, Chapter 548, Statutes of 1998 (act),
requires all preparation programs to use a teaching performance
assessment, it makes this assessment subject to the availability
of funds in the annual budget act. In March 2003, before the
commission implemented the teaching performance assessment,
a state senator and the then-secretary for education wrote to the
commission on behalf of the California State University (CSU),
the University of California (UC), and independent colleges
and universities, stating that given the significant budget cuts
to every area of education, funding was not available for the
colleges and universities to meet the new requirement within
the time frame that the commission adopted. They then
In April 2003 the requested the commission revise its implementation schedule
commissioners delayed according to the availability of state funding. As a result, in
implementing the April 2003 the commissioners voted to delay implementing
teaching performance the teaching performance assessment until the State’s budget
assessment until the improves, but they encouraged preparation programs to
State’s budget improves, implement the assessment on a voluntary basis. According to the
but encouraged programs commission, of the more than 80 preparation programs, at least
to voluntarily implement 18, including three CSU campuses, voluntarily implemented the
the assessment. commission’s teaching performance assessment in the 2003–04
school year. Six other preparation programs reported that they
would implement it in the 2004–05 school year.
The act also allows preparation programs to develop their
own teaching performance assessment, which must have
commission approval, rather than use the commission’s model.
In fact, despite the availability of the commission’s model and
the budget concerns these entities expressed, a consortium of
California universities, including Stanford University, eight
UC campuses, and two CSU campuses, created its own teaching
performance assessment, which the commission indicates
the consortium has not submitted for review and approval.
Although this assessment and any others must be based on the
same performance standards for teacher candidates, the scoring
systems are not likely to align with one another or with the
commission’s model. These differences could limit the commission’s
ability to use the assessment data to compare the performance of
preparation programs.
2244 California State Auditor Report 2004-108 California State Auditor Report 2004-108 2255
Because of Recent Efforts to Combine Its Data With California
Department of Education Data, the Commission May
Be Able to Use Teacher Retention as a Measure of
Preparation Programs
Education research indicates that well-prepared teachers are
more likely to stay in the teaching profession; thus, the retention
rate of teachers is an important measure of the performance of
Although the retention preparation programs. Although the State does not currently
rate of teachers is an have a way to determine teacher retention accurately and
important measure of the efficiently, recent efforts by the commission and the California
performance of teacher Department of Education (Education) may help overcome some
preparation programs, of the technical obstacles to obtaining this data. In January 2004
the commission does the commission and Education entered into an agreement to
not have a process for develop a database that the commission could potentially use
following credential to study teacher retention. The database, which will satisfy the
recipients into the federal act’s reporting requirements, links the commission’s
workforce to see how data with data that Education’s California Basic Education Data
long they are retained. System (CBEDS) maintains. The commission has data on the
types of credentials it issued, the dates of issuance, and the
colleges and universities that recommended the credentials.
However, according to a report that a commission consultant
wrote, the commission does not have a process for following
credential recipients into the workforce to see how long they
remain in teaching, which makes it difficult for the commission
to provide policy makers with this information.
On the other hand, the commission’s consultant explained that
the CBEDS database holds a vast amount of data on currently
employed teachers, their schools, their years of teaching
experience, and their teaching assignments. The consultant
also explained that the CBEDS database contains information
about teachers’ credential status, but because teachers report this
information themselves, it is not the official record of credential
status. Thus, the agreement between the commission and
Education calls for replacing the CBEDS credential information
with the official credential status from the commission’s records.
This will allow the State to meet federal reporting requirements
and allow the commission to monitor whether teachers’
assignments match their credential status. In addition, the
commission will be able to use CBEDS data to determine which
of the teachers to whom it has issued credentials remain in
the teaching profession. The commission could then calculate
a retention rate of teachers in the aggregate and also by other
categories, including school district, the preparation program
2266 California State Auditor Report 2004-108 California State Auditor Report 2004-108 2277
that recommended the candidate for a credential, and credential
type. Such information could provide valuable insights into the
California teacher workforce.
Although combining commission and CBEDS data provides
better data for understanding the teacher workforce, the director
Combining its data with of the commission’s certification, assignments, and waivers
Education’s data might division (certification division) categorized its current efforts as
allow the commission preliminary and exploratory. He explained that the commission
to determine which of still faces some technical obstacles, one of which is that most
the teachers to whom records of the credential information before 1989 are on
it has issued credentials microfiche and not in the commission’s database. Despite these
have remained in the obstacles, the director believes that the commission may be able
teaching profession. (perhaps within the next five years) to use the shared data to
study the retention of teachers.
Another potential obstacle to obtaining data on the retention
of teachers is that the commission believes it has no specific
policy directive to do so. The commission explained that
various departments, including the Employment Development
Department and Education, collect data on teachers but that the
major barriers to combining these data sources is the lack of a
common teacher identifier and the absence of a policy directive
to bring the systems together. The commission and Education
were able to overcome the lack of a common teacher identifier
by asking school districts to include each teacher’s credential
number in their annual CBEDS reports. However, the language
of the agreement between the commission and Education
shows that the primary motivation for the current data-sharing
arrangement is the federal act’s reporting requirements. Without
a clear policy directive, the commission may decide not to use
its resources to overcome the technical obstacles necessary to
analyze and report on the retention of teachers.
The Commission Could Use the Results of an Administrator
Survey as One Measure of the Performance of Preparation
Programs
Along with its use of other measures, such as retention and
the teaching performance assessment results, the commission
could measure the performance of preparation programs by
soliciting feedback from the supervisors of beginning teachers
through an administrator survey. Commission staff agreed
that an administrator survey could be a valuable source of data
for evaluating the performance of preparation programs and
said that the commission could possibly modify the current
2266 California State Auditor Report 2004-108 California State Auditor Report 2004-108 2277
surveys as part of the commission’s continuing accreditation
visits to standardize them and quantify the results. Data from
an administrator survey would allow the commission to
compare the classroom performance of graduates of the various
preparation programs throughout the State. The CSU system,
which educated nearly half of the newly credentialed teachers
in fiscal year 2002–03, is currently developing an administrator
survey to evaluate the success of its preparation programs. CSU
staff explained that the survey still needs more development
but said that it could eventually be useful statewide. However,
CSU staff explained, and best business practices suggest, that
evaluators should not focus on only one measure of program
performance, such as a single survey’s results, to make decisions
that affect preparation programs.
THE COMMISSION HAS NOT ESTABLISHED SPECIFIC
PERFORMANCE MEASURES FOR ITS DIVISIONS
The commission’s February 2001 strategic plan (2001 plan),
which the commission partially updated just after we completed
our fieldwork, was outdated and did not establish the specific
performance measures the commission needed to evaluate the
results of its current efforts. In addition, the commission does
not systematically track whether it is successfully completing
the tasks it outlined in the 2001 plan. As a result of inadequate
strategic planning, the commission has lacked specific
performance measures to guide, evaluate, and improve its efforts.
The Department of Finance (Finance) explains in its strategic
planning guidelines that performance measures, which are
the quantified results an agency seeks to achieve, are one
component of an effective strategic plan. The guidelines further
state that performance measures provide a basis for assessing the
successful achievement of the agency’s mission, vision, goals,
and objectives by focusing on attainment of the objectives.
The commission did not In a report issued in February 2000, a consultant that the
establish performance Legislative Analyst’s Office (legislative analyst) hired to conduct
measures for 18 of a comprehensive management study of the commission’s
56 tasks outlined in its credential process said that the commission’s 1999 strategic
February 2001 strategic plan did not conform to Finance’s guidelines. Specifically, the
plan, even though consultant criticized the plan for not quantifying objectives
the results that the or establishing specific performance measures. Despite these
commission wanted to criticisms of the 1999 strategic plan, the commission’s 2001 plan
achieve in these areas did not establish performance measures for 18 of the 56 tasks
are quantifiable. it outlined, even though the results the commission wanted to
achieve in these areas are quantifiable.
2288 California State Auditor Report 2004-108 California State Auditor Report 2004-108 2299
One example of a task for which the commission did not
quantify its goals is the expansion of teacher development
programs. The performance measures that the 2001 plan
describes are increased numbers of credential candidates
entering teaching through alternative routes and a reduction
in the number of individuals serving on emergency permits.
However, the commission did not quantify the increase or
reduction that it wanted to achieve. For the task of expanding
teacher development programs, the commission could have
specified, based on a cost-benefit analysis, a certain percentage
increase in the annual number of credential candidates entering
teaching through alternative routes as a performance measure. It
could then have used that measure to evaluate the success of its
efforts and adjust its activities to achieve a successful outcome.
Without specific performance measures, the commission cannot
determine whether its expansion of teacher development
programs was a success or whether it should modify or
discontinue it.
We found that six of the 2001 plan’s 56 tasks did quantify the
results the commission wanted to achieve. For example, the 2001
plan states that the commission’s certification division will
use statistical reports to ensure that it processes credential
applications within 75 business days and credential renewals
within 10 business days, with an acceptable deviation of no
more than one business day. As we describe in Chapter 2, the
commission’s certification division tracks application processing
times, which it uses to shift resources among processing teams.
The 2001 plan also contains 32 tasks for which no adequate
measure of the commission’s performance exists or for which
the only performance measure is a particular completion date.
For example, one task that the 2001 plan outlined is to complete
an independent evaluation of the beginning teacher support and
assessment program by December 2001. In this case, the only
applicable performance measure is the completion of the task
by the stated date. Although this task and its corresponding
completion date are wholly appropriate, the completion
date, like all the other specific completion dates in the plan,
had already passed. Therefore, these tasks, assuming that the
commission actually completed them, are outdated, which
highlighted the need for the commission to update its 2001 plan
to reflect its current or future efforts.
2288 California State Auditor Report 2004-108 California State Auditor Report 2004-108 2299
In March 2001, a month after the commission issued its 2001
plan, the legislative analyst published a follow-up report to the
consultant’s study that again criticized the commission for not
clearly identifying how it will assess and track the impact of its
reform efforts and for establishing few quantifiable performance
measures. Despite this criticism and the fact that all completion
dates outlined in the 2001 plan had passed, the commission
did not update the tasks in its 2001 plan until September 2004,
shortly after we pointed out that the plan’s tasks were obsolete.
According to its executive director, the commission did not
update the 2001 plan tasks due to several events during the past
two years that caused a shift in staff assignments, including
implementation of the federal act, budget and personnel
cuts, and development of the Teacher Credentialing Service
Improvement Project. In addition, the executive director
explained that the commission has had four leadership changes
among the commissioners since December 2002 and has not
had a full complement of commissioners since 1999. The
executive director further stated that, at the commissioners’
direction, the commission has postponed long-range strategic
planning until commission vacancies are filled. Although
we acknowledge that the commission has had significant
challenges, proper planning—including following a current
strategic plan—is a generally accepted practice for achieving
success in challenging times. In September 2004, at the request
of the chair of the commissioners, commission staff presented
updated strategic plan tasks to the commissioners. However,
because the commission updated the tasks after we completed
our fieldwork, we were not able to evaluate the tasks and
associated performance measures.
In addition, the commission does not systematically track
The commission does whether it is successfully completing the tasks outlined in the
not systematically track 2001 plan. The executive director prepares an annual list of
whether it is successfully accomplishments that are aligned with the 2001 plan’s goals to
completing the tasks present to the commissioners but does not link the list to the
outlined in the strategic tasks that the 2001 plan described. When we asked the directors
plan. of the commission’s three program divisions—certification,
professional services, and professional practices—whether they
use the 2001 plan or other performance measures to monitor
their divisions’ success, each director was aware of the 2001
plan, but only the director of the certification division was
actively monitoring whether his division was meeting the
performance measures in it. Without focused follow-up and
3300 California State Auditor Report 2004-108 California State Auditor Report 2004-108 3311
monitoring of the performance measures the commission
established in its 2001 plan, a strategic plan holds little value for
the commission.
THE COMMISSION HAS MADE EFFORTS TO STREAMLINE
AND REMOVE BARRIERS FROM THE PROCESS
In exercising its oversight of the process, the commission has
implemented some reforms and is contemplating others. Although
state law mandates the framework of the process, the commission
has the responsibility to analyze the process periodically and
report to the Legislature if particular requirements are no longer
necessary or need adjustment. In addition to proactively seeking
improvements in the process, the commission also must carry
out legislative mandates that streamline the process but that
were developed independently of the commission.
In recent years, the commission has implemented reforms
of both types. For instance, the commission administers the
paraprofessional, pre-intern, and intern programs, which
assist teacher applicants in meeting credential requirements.
In addition, because of recent legislation, the commission has
adopted tests that allow teacher candidates who pass them not
to take otherwise required courses in preparation programs.
The commission has also reduced the barriers to becoming a
The commission California teacher for individuals who received their teacher
administers three training in other states. Specifically, since 1998 state law
programs, which assist has allowed the commission to grant preliminary teaching
teacher applicants in credentials to individuals with at least three years’ public school
meeting credential teaching experience in other states. For individuals without
requirements, and it has three years’ experience, state law since 1998 has allowed
adopted tests that, if the commission to issue preliminary teaching credentials
passed, allow teacher to individuals who graduated from out-of-state preparation
candidates not to take programs that the commission deemed equivalent to California
otherwise required teacher preparation programs. However, the commission has not
preparation courses. analyzed how many additional teachers it has certified as a result
of these two reforms.
To streamline credential requirements, the commission is
planning to hold public meetings with stakeholders to discuss
the current structure of teacher examinations. The commission
plans to address issues, such as the appropriate way to assess
basic skills and overlapping content among examinations. In
April 2005 the commission expects to decide on possible actions
to take on the current teacher examination structure.
3300 California State Auditor Report 2004-108 California State Auditor Report 2004-108 3311
RECOMMENDATIONS
To determine the success of its paraprofessional, pre-intern, and
intern programs, the commission should establish performance
measures for each of these teacher development programs. In
addition, the commission should do the following:
• Ensure that the statistics it presents in its program reports
are consistent and that it maintains the supporting
documentation for these statistics.
• Monitor how local teacher development programs verify the
academic progress of participants and establish consequences
for underperformance.
• Resume requests for budget increases to fund an independent
evaluation of its paraprofessional program that assesses all the
requirements in the applicable statute or seek to amend those
parts of the law that it believes would be too costly to implement.
To provide context to education professionals and policy makers
for why the number of credentials, permits, and waivers it issues
has changed, the commission should include an analysis with
the statistics it publishes in its annual reports.
The commission should collaborate with colleges and
universities to determine what funding is necessary to activate
and maintain the teaching performance assessment as the
enabling legislation envisions it. It should then request the
Legislature and Governor’s Office to authorize this function in
future budget acts.
The Legislature may wish to consider giving the commission a
specific policy directive to obtain and use data on teacher retention
to measure the performance of the process and preparation
programs and provide this information in its annual reports.
To aid it in developing performance measures for preparation
programs, the commission should keep itself informed of
surveys and reports that other entities prepare, such as the
administrator survey the CSU is developing.
To better plan and evaluate its efforts, the commission
should regularly update its strategic plan and should quantify
performance measures when appropriate in terms of the results
the commission wants to achieve. Also, the commission should
3322 California State Auditor Report 2004-108 California State Auditor Report 2004-108 3333
present the commissioners with an annual status report on how
the commission has achieved the goals and tasks it outlined in
its strategic plan.
The commission should continue to consider ways to streamline
the process, such as consolidating examinations it requires
of credential candidates. If the commission determines that
specific credential requirements are no longer necessary, it
should seek legislative changes to the applicable statutes. n
3322 California State Auditor Report 2004-108 California State Auditor Report 2004-108 3333
Blank page inserted for reproduction purposes only.
3344 California State Auditor Report 2004-108 California State Auditor Report 2004-108 3355
CHAPTER 2
By Better Managing Its Customer
Service, Workload, and Technology,
the Commission Could Improve
Application Processing
CHAPTER SUMMARY
By better handling customer service, better managing its
workload, and taking full advantage of a new automated
application-processing system, the certification,
assignments, and waivers division (certification division)
within the California Commission on Teacher Credentialing
(commission) could improve its application processing. The
certification unit (unit) within the certification division is
primarily responsible for customer service and application
processing. Facing a significant volume of contacts, the
certification division has not taken sufficient steps to manage its
customer service activities more effectively. Proper management
of customer service is necessary because the large volume of
telephone calls and e-mails that the unit receives takes staff time
away from credential processing.
Although the certification division typically processes
applications for credentials in less than its regulatory processing
time of 75 business days, applications go unprocessed for a
significant amount of this time because staff members are busy
with other duties. The certification division has taken some steps
to improve its process, including automating certain functions
as part of its Teacher Credentialing Service Improvement Project
(TCSIP). However, the certification division has not sufficiently
analyzed existing data to make informed decisions as to the
types and numbers of credentials it should be processing at any
given time. TCSIP was intended in part to improve credential
processing times, although now, as the project is about to be
fully implemented, the certification division is unsure of how
much efficiency TCSIP will bring to the application approval
process. TCSIP offers tangible time-saving benefits, such as
the electronic submission of applications from colleges and
universities that choose to use this function and automated
review of online renewals. However, the certification division does
not currently plan to use either function to its full potential.
3344 California State Auditor Report 2004-108 California State Auditor Report 2004-108 3355
Although online renewals offer the benefit of faster and more
efficient processing, the certification division has not sufficiently
publicized this benefit. The certification division could better
inform teachers who do not use online renewals about its
benefits by performing the data analysis necessary to determine
where to target additional outreach efforts and by better
highlighting the availability of online renewal and the benefit of
faster processing.
THE CERTIFICATION DIVISION COULD BETTER MANAGE
CUSTOMER SERVICE
Facing a significant volume of contacts, the certification division
has not taken sufficient steps to better manage its customer
service activities, which consume much of the unit’s time. The
commission has a variety of customers, including teachers,
school administrators, institutions, and the general public.
Proper management of customer service is necessary because
the large volume of telephone calls and e-mails that the unit
receives takes staff away from credential processing. Although
the certification division collects data on the reasons that people
contact the unit and could take steps to collect additional data,
it has not sufficiently analyzed these data to address ways that
it could help customers find answers to their questions on its
Web site, in its publications and forms, and through its other
outreach efforts. The certification division could also improve
its customer service by eliminating some inefficiencies and by
making its Web site more informative.
Activities Other Than Processing Applications Consume Much
of the Unit’s Time
The unit faces a significant number of applications to process,
a large volume of incoming telephone calls, and a volume of
One of the major reasons e-mails that has increased steadily since 2001. One of the major
that applications require reasons that the unit takes nearly 75 business days to process
nearly 75 business days applications is that unit employees must spend much of their
to be processed is that time tending to other responsibilities, including customer
employees must spend service, training and development, and other tasks. Because we
much of their time found that the certification division’s timekeeping summaries
tending to responsibilities were not entirely reliable, we estimated the time that unit staff
other than processing spend on various activities by compiling the time sheets that
applications. staff prepared, whose results we present in Figure 4. As the
figure shows, during fiscal year 2003–04, based on the unit’s
self-reported timekeeping reports, unit employees spent 19,000
hours (39 percent of their time) processing applications. They
3366 California State Auditor Report 2004-108 California State Auditor Report 2004-108 3377
spent an additional 18,800 hours (39 percent of their time) in
customer service activities such as answering telephones and
responding to e-mails.
FIGURE 4
Use of Time by Certification Unit Staff
���������������������� �����
�������������� ��������������
����������������������
���������������
������������������
����������������
���������������
����������������
Source: California Commission on Teacher Credentialing time reports, fiscal year 2003–04.
In July 2003, concerned that its workload had reached the
point where it would no longer be able to meet its 75-business
day regulatory limit for processing applications, the
certification division cut back on the unit’s customer service
operations, reducing the hours that staff answer telephones
from about 450 hours a week to approximately 119 hours, a
74 percent reduction. The certification division also abandoned
its goal of keeping customers on hold for no longer than
five minutes and reduced the number of staff members who
answer the telephones from 10 to five. The certification
division also increased the time for responding to customer
e-mails from three business days to between three and five
business days as of July 2004.
Despite these cutbacks, the unit’s significant volume of
customer service contacts has continued into 2004. Figure 5
on the following page shows the volume of telephone calls
and e-mails the unit received during the past three fiscal
years. During fiscal year 2003–04 customers sent nearly 64,000
3366 California State Auditor Report 2004-108 California State Auditor Report 2004-108 3377
e-mails, which the certification division’s records show took
about 8,900 hours to review and answer. During this same
period, the unit received nearly 224,000 calls. Of these, callers
sought to speak directly with someone in the unit 80,000 times,
which took staff 6,000 hours to field, whereas callers made
nearly 144,000 calls to the commission’s automated telephone
system.
FIGURE 5
Annual Number of Calls and E-Mails to Certification Unit Staff
Fiscal Years 2001–02 Through 2003–04
��������������������������������
������
����������������������
�������
���������������������������
�������
�������
�������
�������
�������
������
�
������� ������� �������
����������
Source: California Commission on Teacher Credentialing telephone and e-mail reports.
If processing applications is to be the unit’s priority, as the
certification division states it is, the commission must identify
additional ways to focus the unit’s efforts on credential
processing. The certification division must seek more innovative
ways to handle its workload and provide customers the
information they need.
3388 California State Auditor Report 2004-108 California State Auditor Report 2004-108 3399
The Certification Division Can Better Analyze the Types
of Questions Customers Ask to Focus Its Customer Service
Activities
Despite the significant number of customer contacts, the
certification division has not sufficiently analyzed the types
of questions customers ask so that it can better focus the unit’s
customer service activities. For example, although the certification
division collects data on customers’ reasons for calling, it
has not sufficiently analyzed the data to identify the types of
questions customers ask in telephone calls and to reduce the
number of calls by making the information that customers
seek available on the commission’s Web site. The certification
division’s telephone system generates a report that summarizes
the reason for each incoming call. The categories in the report
include the number of questions about examinations, about
how out-of-state teachers can obtain a credential to teach in
California, and about specific types of credentials. Thus, this
report provides valuable insight into the number of calls that
the unit receives for different questions and types of credentials.
The certification division’s management uses the report to help
unit staff get an idea of the types of questions callers ask, so that
they will be better prepared to answer these types of questions.
The certification division also uses the report to update the
general information about the commission and credentials that
the commission’s automated telephone response system provides
to callers. The information that the automated telephone
response system provides is limited and most call-routing paths
require the caller either to leave his or her name and address to
receive more information, to visit the commission’s and other
related Web sites, or to speak to certification division staff for
more information.
The manager of the certification division indicated that many
customers call for information that is already available on the
By better analyzing commission’s Web site, such as test dates. The manager does
customer data, the not believe that it is possible to control the number of these
commission could identify calls. However, using the information that the telephone system
areas where customers report provides, the certification division could identify areas in
need more guidance which customers need more guidance or a clearer explanation
and make appropriate of the certification division’s requirements. It could do this by
adjustments to the public performing an analysis, such as ranking the different types of
information it provides. calls it receives, and by consistently reviewing this information
and adjusting the information that it provides to the public
on its Web site and in leaflets to handle some of those more
frequently asked questions.
3388 California State Auditor Report 2004-108 California State Auditor Report 2004-108 3399
Further, the certification division does not collect data on the
types of questions that customers ask in e-mails. Although it
tracks the monthly volume of e-mails that the unit receives,
the certification division would understand its inflow of e-mails
better if it categorized the different types of e-mails to identify
areas in which it could improve customer service. It could do
this by sampling e-mails regularly. Although the manager of the
certification division indicated that she believed that gathering
this data would be time-consuming, she occasionally looks at
some of the incoming e-mails to get a sense for the questions
applicants ask, and she sometimes requests a change to the
Web site and leaflets based on her review.
After our inquiries, the manager studied e-mails over a one-week
period in July 2004. She reviewed 1,144 e-mails and concluded
that the types of questions customers asked were similar to those
they asked in telephone calls. This is a useful first step, but the
certification division should be analyzing this data routinely.
Further, because the certification division knows that customers
who are sending e-mails have access to the Internet, it should
pay particular attention to the types of questions they ask to
determine how it could address those questions more readily on
its Web site. The certification division could also set quantifiable
goals for reducing the volume and types of questions customers
ask in telephone calls and e-mails in order to measure the
success of its efforts to minimize those questions.
The Certification Division Wastes Time by Manually
Responding to and Routing E-Mails
The certification division would also benefit from automating
how it responds to and forwards e-mails to the staff members
who reply to questions in e-mails. A single staff member is
primarily responsible for processing e-mails that customers
send to the commission’s customer service address; that address
received a total of nearly 64,000 e-mails in fiscal year 2003–04.
When he receives e-mails, he responds to the customer if he
can answer the question and has time; mostly he responds by
manually copying into the e-mail a generic message informing
A staff member reported the customer that the unit will respond to the question within
spending 500 hours three to five business days. He then looks for keywords to
in fiscal year 2003–04 determine where he should send the e-mail and manually
manually processing forwards it to one of the four unit teams that are responsible
e-mails. for replying. This staff member reported spending 500 hours in
fiscal year 2003–04 manually processing e-mails.
4400 California State Auditor Report 2004-108 California State Auditor Report 2004-108 4411
The director of the certification division does not believe that
this manual process is efficient and has informally asked the
commission’s information technology and support management
division for a better way to respond to and route e-mails.
Although the information technology and support management
division has not yet come up with a solution, the certification
division’s e-mail program can easily be set to issue an automated
response and also to perform an automated scan of e-mails for
keywords to forward the e-mails to the appropriate unit teams.
The Certification Division Could Do More to Encourage
Customers to Use the Commission’s Web Site
To be more responsive to customers and potentially reduce
the volume of telephone calls and e-mails it receives, the
commission should modify its Web site to help customers find
their answers online more easily and to be more accurate. Using
data that the certification division routinely collects on the
types of questions customers ask by telephone, we searched
for information on the Web site that would likely answer the
customers’ questions. However, we not only had difficulty
The section containing finding the commission’s “frequently asked questions” page,
answers to “frequently but discovered that detailed information relating to the most
asked questions” is common reasons that people call the unit is buried within the
difficult to find on the Web site. For example, our analysis showed that among the most
commission’s Web site. frequent inquiries were questions from out-of-state applicants.
Looking at the opening page of the Web site, however, we found
no link targeting out-of-state applicants, but rather we had to
search through several links to find this information. Although
many customers may be patient enough to perform a detailed
search, others may not be and will instead call or e-mail the
unit. Further, the commission’s Web site does not group similar
links together but has links to various topics on either side of
the opening page, with no apparent organization. In addition,
although the certification division would like customers to
renew their credentials online, it could better highlight the link
it uses to take a customer to this function. The commission has
indicated that it is currently redesigning its Web site.
Finally, the Web site indicates that applicants should anticipate
an eight- to 12-week turnaround for applications that they
have submitted. However, the certification division’s data
show that from December 2002 to August 2004, staff took
an average of 12 to 14 weeks to process applications. Because
the certification division created an expectation that it would
process applications faster than its actual processing times,
4400 California State Auditor Report 2004-108 California State Auditor Report 2004-108 4411
customers who had not heard from the certification division
within 12 weeks may have been more inclined to call or e-mail
to inquire about their applications’ status. In August 2004
average processing times fell to seven to eight weeks, but the
certification division should regularly compare actual processing
times to the turnaround times it claims on its publications and
adjust those times as needed.
BETTER WORKLOAD ANALYSIS COULD HELP REDUCE
PROCESSING TIMES
The certification division must take appropriate measures to
better manage its application workload and reduce its processing
times. Although the certification division has generally been
able to meet its regulatory processing time of 75 business days,
credential applications go unprocessed for a significant amount
of time. The certification division has taken some steps to
improve the process, including automating certain functions
and developing a new automated application-processing system.
However, it has not gathered or analyzed sufficient data on the
time it takes to process credentials so that it can make informed
staffing decisions or ensure that it consistently completes all
applications within 75 business days.
The Certification Division Generally Meets Its Regulatory
Processing Time of 75 Business Days
The commission established the 75-business-day limit in 1985
in response to a law requiring state licensing agencies to put
in regulation the maximum allowable times for processing
permits. As a penalty for exceeding the 75-business-day limit,
the commission must refund the applicant’s credential filing
fees at the applicant’s request, except under certain conditions,
From December 2002 to such as when the background check requires extra time. Figure 6
July 2004, the average shows the weekly average processing times for applications
times for processing from December 2002 to July 2004, broken down by the four
applications by the teams that process credentials, based on a report that unit staff
certification unit’s four created. From December 2002 to July 2004 the reported average
teams ranged from 61 to processing times for the unit’s four processing teams ranged
70 business days. from 61 to 70 business days, which shows that the unit’s normal
operation generally comes very close to but does not exceed
the 75-business-day limit. However, the individual teams do
exceed the 75-business-day limit at various times, as the figure
shows. The average processing times dropped significantly in
July 2004 for two reasons. To prepare to implement TCSIP, which
will disrupt application processing, the staff of the certification
4422 California State Auditor Report 2004-108 California State Auditor Report 2004-108 4433
division worked 450 hours of overtime. In addition, the director
of the certification division indicates that the number of
applications the commission received through July 2004 has
decreased from the same period during 2003.
FIGURE 6
Average Processing Time for Applications
December 2002 Through July 2004
�����
�����
��� �����
�����
��������������������
��
��
��
��������������
���� ���� ���� ���� ���� ��� ���� ���� ���� ����� ���� ���� ���� ���� ���� ���� ���� ��� ���� ����
���� ���� ����
Source: California Commission on Teacher Credentialing reports of average processing time.
The certification division does not track the number of
applications that take longer than 75 business days to process;
instead, it relies on applicants to request a refund when they
believe that processing of their applications has exceeded that
limit. However, routinely tracking this information would be a
useful indicator of the status of application-processing activities
4422 California State Auditor Report 2004-108 California State Auditor Report 2004-108 4433
and would also identify problem applications that staff have
misplaced or that need management’s attention. In July 2004
the director of the certification division began creating a weekly
report that identifies old and lost applications that have deviated
from the normal flow. The director provides the report to each
unit team to determine why staff have not processed these
applications. Although producing this report is a useful first step
in identifying applications that may exceed the 75-business-day
limit, the certification division should have TCSIP routinely
track applications that are taking longer than 75 business days
to process.
Processing applications takes several steps. Three units within
the commission’s certification division are primarily responsible
for processing applications. Figure 7 illustrates this workflow.
Figure 7 shows that the cashiering unit’s work is the first step
in processing an application. On average, applications typically
wait an average of three business days before the cashiering
unit processes them. It opens the mail, sorts the applications,
enters data from the applications into the certification division’s
automated system, and deposits applicants’ checks before
routing the applications to the certification unit.
The certification unit (unit) is responsible for determining
whether to grant or deny an application, the second step. The
unit consists of four teams, each responsible for reviewing and
processing certain types of applications. Depending on the type
of credential for which the individual applied, this review can
range from the simple—verifying that the applicant completed
the application properly, passed the required examinations, and
received a background clearance—to a more extensive review
of the applicant’s college transcript to verify that he or she met
specific course requirements. The certification division does
not track the time that reviewing different types of applications
takes but instead tracks only the amount of time that blocks of
applications stay in the unit. On average, applications await a
team’s review for 54 to 63 business days, comprising a workload
on hand for a week that ranged from a minimum of around
22,000 applications to a maximum of 57,000 applications from
December 2002 to July 2004.
4444 California State Auditor Report 2004-108 California State Auditor Report 2004-108 4455
FIGURE 7
Units Responsible for Processing Applications
�����������
R
R
R
R
R
�������������� ����������������� ���������������������������
���������� ���������������������������������� �����������������������������
������������������ ������������������� ����������������������������
����������� �����������������������������
��������������� ����������������������������
�����������
Source: California Commission on Teacher Credentialing.
According to the director of the certification division, this
wait is not due to external factors, such as waiting to receive
applicants’ examination scores or fingerprint clearance, because
the commission generally receives both electronically. He
believes that the certification division could reduce this wait
time if more staff were available to process these applications;
yet, as we discuss in the next section, the certification division
has not performed the analysis necessary to balance its current
staff allocation. Also, by better managing customer service and
further automating its process, the certification division might
be able to free unit staff to work on applications and reduce this
wait time.
4444 California State Auditor Report 2004-108 California State Auditor Report 2004-108 4455
The work of the document mail preparation unit (mail unit)
is the final step in the processing of applications. Mail unit
staff perform a final quality control review of the approved
credentials and then mail copies to applicants and to the
counties in which applicants teach. Applications wait an average
of four business days before the mail unit processes them.
There are exceptions to this process, most notably for applicants
who require additional background checks and for online
renewals. When a background check issue is raised, the
unit forwards the application to the commission’s division
of professional practices for follow up. The director of the
certification division indicated that this follow up can take up to
several months to resolve, depending on the severity of the issue
and the length of time it takes the commission to receive the
documents it needs to complete its review. On the other hand,
applicants who renew their credentials online reap the benefit
of having their applications processed within 10 business days.
Because the commission receives information and payments from
these applicants electronically, their renewals avoid the cashiering
unit and receive priority in the certification and mail units.
Better Data Analysis Could Help the Certification Division
More Effectively Manage Its Workload
The certification division does not sufficiently analyze data to
effectively manage its application processing workload, nor does
it set adequate processing goals. It collects a great deal of data
on its activities related to processing applications, including
information on the number of different types of credentials and
renewals that the commission grants and rejects over a period of
The certification division time and individual production statistics showing the number
has only a general sense of applications that individual staff members process and the
of the time it takes to frequency of staff errors. Although the certification division
process different types of generates a weekly report that shows the number of applications
applications and has not or renewals waiting for staff to review or mail, the report does
conducted time studies not break down these applications by unit team. Further, the
on the effort needed to certification division does not have a goal for the maximum
process them. number of applications it can have in its workload, although the
director indicated that he prefers to have a workload of fewer
than 20,000 applications and renewals.
The certification division began manually generating a new
report in June 2004 that shows the unit’s daily workload by
team. This report enables the unit to identify when a team can
anticipate a large number of incoming credential applications.
4466 California State Auditor Report 2004-108 California State Auditor Report 2004-108 4477
Further, the certification division has only a general sense of
the time it takes to process different types of applications. For
instance, it knows that less complex applications, such as those
for single-subject or emergency permits, do not take staff as
long to process as do cross-cultural language and development
permits. However, the certification division has not conducted
time studies on the effort needed to process different types of
applications to know the time needed on average to process
each type. Without such data analysis, the certification division
does not know whether the credential types it assigned to each
of the four teams to process are optimal or if each team has
sufficient staff.
Finally, the certification division’s report of average processing
time does not provide sufficiently detailed information on the
status of applications. The certification division creates this
report manually rather than generating it from the automated
application-processing system. This report provides the difference
in business days between the current date and the dates that
the commission received the applications currently with the
unit. It uses a similar calculation to determine the length of
time that applications wait for processing by the cashiering
unit and the mail unit. This report is the primary tool that the
certification division uses to monitor whether it is meeting its
75-business-day limit and to make staffing decisions among
The average processing the teams. However, the report does not provide information
time report does not on the number of applications that the certification division
provide information on has not processed within the 75-business-day limit nor on
the number or types of applications that are nearing the limit. This information would
applications that have provide the certification division more useful information with
not been processed which to make decisions to move staff among the unit’s four
within the 75-business- teams. The director indicated that the certification division
day limit or that are prepares this report manually because the current application-
nearing the limit. processing system cannot produce it. As is common when designing
a new automated system, we would have expected the commission
to have had the opportunity to replace existing manual reports,
such as this one, with an automated version when it designed
TCSIP. However, the commission states that it did not have such an
opportunity, and TCSIP will generate only the same reports that the
previous computer system produced. Nonetheless, the certification
division indicated that TCSIP will allow staff to create new reports
and the certification division is considering creating an automated
report of average processing time.
4466 California State Auditor Report 2004-108 California State Auditor Report 2004-108 4477
A New Fingerprint Interface and the Use of Overtime Have
Helped the Certification Division Manage Its Workload
The certification division’s new fingerprint interface with the
Department of Justice (Justice) has reduced the effort required to
process applications for credentials. As part of the background
check, an applicant must pass a fingerprint check that Justice
administers. Prior to 2002 applicants had to send paper copies
of their fingerprints to the certification division’s fingerprint
unit, which then mailed them to Justice for review. In 2002 the
certification division set up an electronic interface with Justice’s
Live-Scan system, which allows applicants to register their
fingerprints electronically in their county with Justice rather than
sending paper fingerprint cards to the certification division. Justice
performs the background check and then notifies the certification
division electronically of the results, which has significantly reduced
the effort that processing fingerprints requires.
The use of overtime hours has also helped the certification
division better manage its application backlog. From
Use of overtime in November 2003 through July 2004 the certification division
June and July 2004 staff worked approximately 1,000 hours of overtime, at a cost
reduced the average of roughly $31,000, to process applications. For example, to
processing days. prepare to implement the third phase of TCSIP (planned for
late October 2004), staff worked 450 hours of overtime in June
and July 2004, processing credentials to reduce the number of
applications in its workload. Due to this effort, the certification
division reduced its average processing time from between 52 to
74 business days to between 44 and 51 business days.
THE CERTIFICATION DIVISION MUST SEEK WAYS
TO FULLY USE ITS NEW AUTOMATED APPLICATION-
PROCESSING SYSTEM
The certification division is unsure of the benefit that TCSIP will
ultimately have on processing times, although it is intended
partially to streamline the processing of applications. TCSIP
is a three-phase computer project designed to provide Web-
based lookup of application status, provide teachers the ability
to renew credentials over the Internet, and replace obsolete
hardware. The first two phases, implemented in October 2001
and June 2002, reside in a computer system separate from the
new credential database and hardware to be developed in the
third phase, which the commission will implement in late
October 2004.
4488 California State Auditor Report 2004-108 California State Auditor Report 2004-108 4499
The potential benefits of TCSIP will have no effect on much
of the certification division’s caseload, including all applications
from school districts. Several of TCSIP’s functions, including the
electronic submission of applications from colleges and universities
and the automated review of renewals and applications, have
promise for improving credential processing if they work properly
and can be expanded to other processing areas.
Although TCSIP Is Designed Partly to Reduce Processing
Times, the Certification Division Is Now Unsure Whether It
Will Provide Significant Efficiencies
Although the commission gave the Department of Finance
(Finance) and the Joint Legislative Budget Committee (budget
committee) the impression that TCSIP, which will cost the
commission approximately $8.9 million, would provide
efficiencies that would likely improve its application processing
In its May 2001 times, the certification division is now uncertain of the specific
feasibility report, the efficiencies it expects to receive. In the TCSIP feasibility report
commission stated that that the commission provided to Finance and the now-defunct
the Teacher Credentialing Department of Information Technology in May 2001, the
Service Improvement commission stated that it undertook TCSIP to address critical
Project was undertaken business problems, including unacceptably high application
to address critical processing times. The budget committee was also under the
business problems, impression that TCSIP would help address what it termed as the
including unacceptably commission’s lengthy processing times. In a letter to Finance in
high credential June 2001 supporting continued funding for TCSIP, the chair of
processing times. the budget committee discussed TCSIP’s potential to streamline
the credentialing process and reduce processing times.
In actuality, the certification division does not know what
efficiencies to expect from the third phase or from TCSIP as a
whole. The director of the certification division has stated that
the commission does not know what overall effects TCSIP will
have on processing time and that he does not know whether
the new system will be more or less complicated than the
current system. Moreover, with the implementation of TCSIP,
the certification division is automating parts of its processing of
applications but is not revising its processing method. That said,
TCSIP does offer some benefits. The commission implemented
the first phase in October 2001, enabling anyone to view the
status of a teacher’s credentials online. This appears to be a
success, as calls to the unit for this information have decreased
substantially. The second phase, implemented in June 2002,
enabled credential holders to renew their credentials online;
this provides some efficiencies to the certification division
4488 California State Auditor Report 2004-108 California State Auditor Report 2004-108 4499
because these applications bypass the cashiering unit. However,
the certification division has not sufficiently encouraged
customers to use this online renewal function. Additionally, two
functions that will be installed as part of the third phase, namely
the electronic submission of documents from colleges and
universities and the virtual credential officer, which will be used
for online renewals, should provide additional savings. However,
decreased staffing in the unit may also affect the efficiencies
from TCSIP. As of July 2001, shortly after the commission
proposed TCSIP, the unit had 40 positions. In September 2004
unit staffing had decreased to 33.8 positions. Having less staff
to process applications will likely negate some of the efficiencies
that may result from automation.
Requiring Institutional Customers to Submit Documents
Electronically Could Free Up Cashiering Unit Resources
To increase the efficiencies that it could potentially realize from
electronic submission of prospective teachers’ applications, the
certification division should require all colleges and universities
to use this feature. Electronic submission of applications should
provide savings to the commission because such applications
bypass the cashiering unit, free up cashiering unit staff to work
on other activities, and reduce the chance that the commission
will reject an application that an institutional customer
completed incorrectly. According to the commission, TCSIP
will feature edit checks for electronically submitted applications
that will prevent a customer from submitting a document that
lacks mandatory data. Based upon our review of the certification
division’s study of rejected applications and our discussions with
certification division management, we determined that online
edit checks might easily have caught and corrected at least
1,000 of the 2,203 applications that colleges and universities
submitted from July 2003 to March 2004 that the commission
later rejected.
Unless the commission requires all colleges and universities to
Unless the commission submit applications electronically, the electronic submission
requires all colleges and capability will not provide the certification division nearly
universities to submit as much efficiency as it could. The certification division does
applications electronically, not know how many colleges and universities will actually
the electronic submission take advantage of this feature, nor does it plan to require all
capability will not provide institutional customers to do so. Also, if an applicant does
as much efficiency as it not have a credit card, the application cannot be submitted
could. electronically. The director asserts that not all colleges and
universities may feel comfortable with the electronic process.
5500 California State Auditor Report 2004-108 California State Auditor Report 2004-108 5511
However, it is reasonable to assume that most are sophisticated
enough to do so. The commission granted approximately
22,000 credentials that colleges and universities submitted
during fiscal year 2002–03; receiving all of these applications in
electronic form would have created significant efficiencies.
Further, the certification division did not design this feature to
accept applications from school districts, which submit paper
applications to the unit for substitute teachers. Processing
these applications makes up a significant amount of the unit’s
workload, with approximately 21,000 applications granted
in fiscal year 2003–04, or 18 percent of the nonrenewal
applications granted for the year. The director stated that he was
unsure whether TCSIP could be designed to include electronic
submission from school districts, because the certification
division never formally studied the complexity or cost of this
potential feature. To increase the efficiency of application
processing, the certification division should strive to get as many
applications in electronic form as is economically feasible.
Expanding the Use of TCSIP’s Virtual Credential Officer
Function Could Free Up Certification Resources
The virtual credential officer function, which the commission
plans to implement in late October 2004 as part of TCSIP’s third
phase, allows for the automatic approval of online renewals
if the customer completes the form properly, passes the
background check, and pays the renewal fee. Thus, those
applications that the virtual credential officer function can
process eliminate the need for a manual application review,
allowing the certification division to reallocate unit staff
resources elsewhere.
However, the certification division has no current plans to use
The certification division the virtual credential officer function to process applications
has no current plans to for the more routine credential types, such as multiple- and
use the virtual credential single-subject credentials or permits for 30-day substitutes. The
officer function to process director of the certification division cited technology concerns
applications for the more and the cost of the additional programming that would be
routine credential types. required as the reasons for not developing these functions to
work with these credential types. However, the commission never
solicited a quote from the vendor that is developing TCSIP to
determine the cost of adding these functions, nor did it formally
analyze the benefits.
5500 California State Auditor Report 2004-108 California State Auditor Report 2004-108 5511
Expanding the virtual credential officer function to process
applications for multiple- and single-subject credentials
and for 30-day substitute permits would be logical because
these applications are relatively straightforward to process
compared to some other types. For multiple- and single-subject
applications that colleges and universities submit, unit staff must
perform a background check, review test scores, verify transcript
information, and confirm that an authorized representative at
the university has recommended the application. Applications
for 30-day substitute permits require only that unit staff perform
a background check and verify that the applicant has passed the
California Basic Educational Skills Test and earned a bachelor’s
degree. On the other hand, applications for child-center permits
require unit staff to verify work experience with an employer
and perform a detailed review of a college transcript to confirm
that the individual completed core course requirements.
In fiscal year 2003–04 the unit rejected 11.2 percent of the
child-center permit applications it received, whereas it rejected
only 4 percent of the first-time multiple- and single-subject
applications and 4.4 percent of the first-time 30-day substitute
permits. The renewal applications for which the certification
division plans to use the virtual credential officer had a
rejection rate of 3.5 percent in fiscal year 2003–04. Because
rejection rates seem to indicate the complexity of an application
and thus the likelihood of error on the customer’s part, the
certification division should at a minimum consider using the
virtual credential officer to process multiple- and single-subject
applications and 30-day substitute permits in the near future.
In discussing the technology concerns that prevent the
certification division from expanding the virtual credential
officer, the director of the certification division indicated that
he did not believe it would be feasible for the virtual credential
officer to approve automatically an application that requires
transcript submission. The certification division requires all
first-time applicants that colleges and universities recommend
to submit a paper transcript so that the unit staff can verify that
the individual has earned a bachelor’s degree. Further, the unit
staff also review transcripts for specific course requirements on
the applications submitted by college and university staff who
did not attend the training academy that the commission held
in 2001.
5522 California State Auditor Report 2004-108 California State Auditor Report 2004-108 5533
The certification division could remove this potential obstacle
by no longer requiring applications that colleges and
universities submit to include transcripts and instead relying
on authorized representatives of the colleges and universities
to verify that applicants have met credential requirements.
Such representatives already perform this function by signing
and completing a checklist certifying that they have reviewed
the applications and that the applicants have met all of the
course requirements for multiple- and single-subject credentials,
such as the requirement for a U.S. Constitution course. When
we consider this review and the fact that the commission
accredits the teacher preparation programs of these colleges
and universities, a separate review by unit staff does not appear
necessary for every application that a college or university
submits. To save staff resources in the unit and to make
applications easier to use with the virtual credential officer,
the certification division should consider reviewing transcripts
only for colleges and universities that have not met reasonable
quality standards that the commission establishes.
Similarly, the commission’s transcript requirement for 30-day
substitute permits may not always be necessary if a school
district submits the application. The director of the certification
division indicated that the Los Angeles Unified School District
has several staff members who are responsible for submitting
applications to the certification division and that it does very
well at submitting them. The certification division should
consider using the virtual credential officer for school districts
from which the unit receives a low percentage of erroneously
submitted applications. If the certification division successfully
automates these functions, it could eliminate the need for
manual review of a significant number of applications.
THE CERTIFICATION DIVISION’S MANAGEMENT OF
ONLINE RENEWALS NEEDS IMPROVEMENT
Although online renewals offer the benefit of faster and more
efficient processing, the certification division has not sufficiently
publicized this benefit. The certification division also has not
taken sufficient measures to encourage applicants to renew their
credentials online, nor has it identified the number of renewals
it receives online versus through the mail. Without knowing the
percentage and types of teachers who choose to renew online,
the certification division cannot gauge its progress.
5522 California State Auditor Report 2004-108 California State Auditor Report 2004-108 5533
The Virtual Credential Officer Will Streamline Online Renewals
The certification division has not fully realized the potential of
renewing credentials online, an option that became available in
June 2002 and cost the commission almost $1.2 million as part
of TCSIP. When a teacher’s credential comes up for renewal, the
teacher may choose to renew it either through the commission’s
Web site or by mailing a paper renewal. Those who choose
Teachers who choose to to renew online are rewarded with a turnaround time of
renew their credentials seven to 10 business days, instead of the up to 75 business
online are rewarded with days that the certification division takes to process all other
a turnaround time of applications, including renewals that arrive by mail. Although
seven to 10 business days. the certification division receives an average of 10,000 renewals
a month, most require very little human interaction. Generally,
unless the application is subject to an audit, the certification
division requires only that the individual pay the appropriate fee
and properly complete the renewal form. Thus, the certification
division’s role in this process is to record the payment, update
the credential status on its automated application processing
system, and mail the new credential to the teacher and the
appropriate county office of education. For the certification
division, the main benefit of online renewals is that they bypass
the entire cashiering process, which eliminates the need to input
renewal data manually and deposit checks, freeing up cashiering
staff for other activities.
As we mentioned previously, with the implementation of
the third phase of TCSIP, planned for late October 2004, the
certification division plans to streamline the online renewal
process further by implementing the virtual credential officer
function. If successful, this function will eliminate much of the
human intervention needed to grant renewals by performing
a variety of edits on an online renewal to take the place of the
current manual review process. However, even with the virtual
credential officer function, the certification division intends to
continue to print out a copy of the paper application to scan for
its microfiche records.
The Certification Division Could Better Inform Customers
About Online Renewals and Lacks Crucial Data Needed to
Target Its Outreach Efforts
The certification division could better inform teachers about
online renewals by highlighting its availability and the benefit of
faster processing. The certification division currently informs the
teaching community about the availability of online renewals at
workshops and conferences. The certification division also told
5544 California State Auditor Report 2004-108 California State Auditor Report 2004-108 5555
us that it communicates the availability of online renewals to
applicants who call about renewals. In addition, the certification
division indicated that it includes leaflets promoting online
renewals when it mails 30-day substitute permits, because they
are on a one-year cycle for renewal. However, the commission’s
Web site makes the online renewal option less obvious than
the paper renewal application. To urge credential holders to
use the online renewal function, the commission should
structure its Web site to highlight this option.
The certification division has done little to communicate the
benefit of faster processing of online renewals to teachers. Not
The commission has only has it not included this information on the commission’s
done little to publicize Web site, but it also does not convey it in information leaflets
the benefit of faster that are specifically about online renewals. The director of the
processing of online certification division acknowledged that most teachers probably do
renewals to teachers. not know that the commission processes online renewals in seven
to 10 business days until after they have submitted an application.
The certification division also lacks crucial information about
online renewals that it could use to increase the number of
teachers who use this function. Currently, the certification
division knows how many online renewal applications
it receives per month, and it should want to know what
percentage of total renewals these represent. This percentage
would provide the certification division with a fairly good
measure of whether it has convinced applicants to renew online.
Knowing this percentage would also allow the certification
division to compare it to a target percentage or percentages for
different credential types and measure whether it was meeting
these targets. However, the certification division cannot
calculate this percentage because it currently does not know
the total number of renewals that applicants could perform
online. It does know that the total number of renewals and
credential upgrades from preliminary to professional clear status
is approximately 10,000 per month, but it can use that figure
only for a rough comparison. Without this percentage, the
certification division will have difficulty gauging the success of
the online renewal function. As Figure 8 on the following page
shows, the number of online renewals has increased steadily
since the commission introduced this option in June 2002.
5544 California State Auditor Report 2004-108 California State Auditor Report 2004-108 5555
FIGURE 8
Online Renewals
July 2002 Through April 2004
�����
�����
�����
�����
�����
�
���� ��������������������� ���� �������� ���� ��� ���� ���� ��������������������� �������� ���� ����
���� ���� ����
����������������������
Source: California Commission on Teacher Credentialing cashiering reports and estimates.
In addition, because there are different types of credentials, a
critical first step that could help the certification division is to
know which types of credentials teachers commonly renew
online. This information would be useful to determine whether
some segments of credential holders are not renewing online.
For example, if the certification division noticed that only a
low percentage of teachers with multiple-subject credentials,
which are for teaching in elementary schools, renewed online,
it could focus its outreach efforts on elementary school teachers.
Until the certification division collects and analyzes crucial
data on the number and type of online renewals versus paper
renewals it receives, it will not be able to gauge the impact of the
online renewal option.
5566 California State Auditor Report 2004-108 California State Auditor Report 2004-108 5577
RECOMMENDATIONS
To ensure that the public information it provides meets its
customers’ needs, the certification division should routinely update
its Web site and leaflets based on an analysis of customer data.
Specifically, the certification division should do the following:
• Use the telephone system report to its full potential by
identifying specific areas in which the certification division
can improve the information it communicates to the general
public on the Web site and in leaflets.
• Gather meaningful data about the types of e-mail questions
the certification division receives and use the data to improve
the information it communicates to the general public on the
Web site and in leaflets.
• Ensure that the information on the Web site is accurate and
easy for customers to use.
To improve the efficiency of e-mail processing, the certification
division should automate its response to and routing of e-mails.
To ensure the effective management of the unit’s application
workload, the certification division should routinely monitor
the composition of the applications that it has not yet processed
and collect and analyze data on the average review times for
different types of applications.
To ensure that it continues to meet the 75-business-day limit,
the certification division should routinely have TCSIP create
automated reports to track the average processing times
and regularly list all applications that are taking more than
75 business days to process.
To ensure that it optimizes the time-saving benefits of TCSIP,
the certification division should use automated processes rather
than manual ones where possible. Specifically, the certification
division should do the following:
• Require colleges and universities to submit credential
applications to the commission electronically to the extent
that is economically feasible.
• Consider expanding TCSIP to allow school districts to submit
applications electronically and to allow the virtual credential
officer function to process routine applications.
5566 California State Auditor Report 2004-108 California State Auditor Report 2004-108 5577
To encourage more customers to renew their credentials online
and to determine whether additional outreach efforts may be
necessary, the certification division should gather data on and
study the percentage of renewals it receives online for different
types of credentials. Also, the certification division should do
the following:
• Publicize the fact that it processes online renewals faster than
paper renewals.
• Make the link to online renewals more obvious on the
commission’s Web site. n
5588 California State Auditor Report 2004-108 California State Auditor Report 2004-108 5599
CHAPTER 3
The Commission Should Refine Its
Process for Developing Program
Standards and Resume Its Continuing
Accreditation Reviews of Colleges
and Universities
CHAPTER SUMMARY
The California Commission on Teacher Credentialing
(commission) is responsible for setting program standards
for colleges and universities that prepare prospective
teachers. The commission’s recent experiences developing
program standards to meet the requirements of the legislative
mandate in Senate Bill 2042, Chapter 548, Statutes of 1998 (act),
offer an opportunity to evaluate how to manage these efforts
better in the future.
Our review of five sets of recently developed program
standards identified several areas in the commission’s process
for developing program standards that lack structure and
could be improved. For example, the commission does
not use a methodical approach to form advisory panels of
education professionals that assist it in the development of
program standards. Using a methodical approach would help
the commission ensure that it chooses the most qualified
individuals as panel members and that its selections are
objective. Also, the commission does not always present in
proper perspective the results of its field-review surveys of
education professionals on draft program standards so that
the commission’s governing body (commissioners) could
understand their relevance.
In addition, the commission is missing opportunities to
ensure further that its standards are free from bias toward any
particular group, because its process excludes the views of some
groups whose total number of responses to the survey is less
than the number that would otherwise trigger a review. In
addition, the commission does not have an overall plan to guide
its efforts to finish implementing some program standards that the
act required and to guide its ongoing standard-setting activities.
5588 California State Auditor Report 2004-108 California State Auditor Report 2004-108 5599
Planning would help the commission identify the steps
necessary to periodically update program standards and the
resources it plans to use.
In December 2002, the commission suspended its continuing
accreditation reviews of colleges and universities. Continuing
accreditation reviews are an important component of the
commission’s accreditation system and are intended to
ensure that colleges and universities are operating teacher
preparation programs that meet the commission’s standards.
The commission indicated that it suspended the continuing
accreditation reviews in order to allow colleges and universities
time to implement the commission’s new standards and for it to
evaluate its accreditation policy. Although the commission has
been working with representatives from colleges and universities
to evaluate its accreditation policy, it does not plan to propose a
revised policy to the commissioners until August 2005.
LEGISLATION REFORMED THE STATE’S TEACHER
PREPARATION, DEVELOPMENT, AND CREDENTIALING
REQUIREMENTS
The act created substantial reforms in the way
California teachers are prepared and licensed. The
act implements new standards to govern all aspects
Key Changes in Teacher Preparation and
of teacher preparation, including a standards-
Credentialing Requirements
based teacher performance assessment in teacher
Prospective teachers must: preparation programs, and it requires all new
teachers to complete a beginning-teacher support
• Complete a subject matter program,
or examination equivalent, that the program. The act’s requirements are based on the
commission has approved and that
recommendations of an advisory panel composed
is aligned with the state content and
performance standards for pupils adopted of education professionals that the commission
by the State Board of Education. formed in response to a legislative mandate for a
• Complete a program of professional review of the commission’s teacher preparation
teacher preparation that includes a teacher and credentialing policies. The text box presents
performance assessment prior to earning a
some of the key changes in teacher preparation
preliminary teaching credential.
and credentialing resulting from the act.
• Complete an induction program of
support and formative assessment
during the fi rst two years of teaching as In response to the act’s requirements, the
a requirement for earning a professional commission began developing three new
clear credential.
interrelated sets of standards for program quality
and effectiveness in subject matter preparation,
Source: California Commission on Teacher
teacher preparation, and induction. The
Credentialing.
commission indicated that these three program
standards create a learning-to-teach continuum
that rigorously prepares prospective teachers in
6600 California State Auditor Report 2004-108 California State Auditor Report 2004-108 6611
the subject they will be authorized to teach, as
well as in the methods of teaching and classroom
Teacher Preparation Program Standards
management. The text box describes the intent of
Subject matter standards require prospective these three standards.
teachers to demonstrate that they possess
suffi cient knowledge in the subject they will
be authorized to teach. To develop program standards that are effective in
preparing teachers, the commission coordinates
Professional preparation standards require
its efforts with education professionals and
prospective teachers to demonstrate that
they possess the ability to teach pupils and independent contractors. To assist it in developing
manage classrooms.
standards, the commission forms advisory panels
Teacher induction program standards require staffed with education professionals who have
prospective teachers to complete a two-year
expertise in the subject. In addition, for the subject
support program to refi ne their knowledge,
skills, and abilities to teach pupils. matter standards, the commission’s contractor
develops an examination that prospective
teachers can take in lieu of completing a teacher
Source: California Education Code, Section 44225.
preparation program. As part of this process, the
commission forms a bias review committee—
also staffed with education professionals—to
identify whether the standards contain any biased content or
language that the commission should remove. The contractor
also conducts a content validity study, surveying education
professionals to determine the knowledge, skills, and abilities
that entry-level teachers need for effective job performance.
The survey requests respondents to rate each requirement
and also solicits written comments to allow respondents to
indicate whether any content requirements that are necessary
for effective entry-level job performance are missing. After its
advisory panel analyzes the survey’s results, commission staff
propose the subject matter requirements to the commissioners
for approval. Once the commissioners approve the requirements,
the commission’s advisory panels develop subject matter
standards for teacher preparation programs, and the contractor
develops the subject matter content examination.
Before fi nalizing teacher preparation program standards, the
commission sends draft copies of the program standards to
education professionals throughout California for review.
The commission also conducts this review, known as a fi eld
survey, for subject matter program standards and other
standards that the commission develops, such as standards for
induction programs and administrative services. The review
is intended to determine whether the program standards that
colleges and universities will implement will prepare effective
teachers. The survey asks respondents to rate the importance
of each program standard and also solicits written comments
to allow respondents to indicate whether they believe critical
6600 California State Auditor Report 2004-108 California State Auditor Report 2004-108 6611
characteristics of program quality that are necessary for
preparing effective teachers are missing from the standards.
After its advisory panels analyze the results of the field surveys,
commission staff recommend the draft program standards for
the commissioners’ approval.
Once the commissioners adopt a new program standard, they
also adopt an implementation plan, which typically provides
colleges and universities a two-year transition period in which
to revise their teacher preparation programs to conform to the
newly adopted program standards. In addition, it allows the
commission time to review and approve their new program
proposals. The implementation plan includes a timeline for
colleges and universities to phase out the teacher preparation
programs they administered under the commission’s previous
standards. This phase-out period is also necessary so that
students who began their coursework under the commission’s
previous teacher preparation standards can complete their
course of study.
THE COMMISSION DEVELOPED PROGRAM STANDARDS
IN A REASONABLE ORDER, BUT IT SHOULD DEVELOP
AN OVERALL PLAN TO FINISH IMPLEMENTING THE
ACT’S REFORMS
The commission has been developing standards in four
phases. The order of these phases appears reasonable because
the commission focused first on developing standards that
reflect key credentialing changes that the act required.
Specifically, the commission began by developing new multiple
subject teaching credential standards, professional teacher
preparation, and induction standards, which it believed had
the broadest impact. The three subsequent phases focused on
developing 13 new single-subject standards. The order in which
the commission chose to develop the subject standards appears
reasonable because it first developed core curriculum subjects
for pupils—English, mathematics, science, and social science.
The remaining phases focused on developing other subjects,
such as art and agriculture. Figure 9 provides an overview of the
commission’s progress in developing program standards, along
with the dates that the commissioners adopted the standards
and the implementation periods for each standard.
6622 California State Auditor Report 2004-108 California State Auditor Report 2004-108 6633
9
ERUGIF
deriuqeR
tcA
eht
tahT
sdradnatS
gnipoleveD
ni
ssergorP
s’noissimmoC
ehT
�����������������
�����������������������
����
����
����
����
����
����
����
����
����
����
����������������
����������������������������������
���������������������
����������������������������������
���������������������������������������
��������������������
�����������
�����������������������������������
��������������������
�������������������
������������������������������
�������������
�������������������
�������������������
������������������������������
��������������������
�����������
������������������������������
������������������
������������������
�������������
�����������������
����������������������������
��������������������
��������
�����������
��������������������
�������������
�������������������
�����������������������
��������������������
�����������
����������������������������
������������������������
�������������
���������������
�����������������������
��������������������
�����������
������������������������������
������������������������
�����������������������
����������������������������
�����������������������
�����������������������������
���������������������������
������������������������
�����������
���������������������� �����������������������
���������
.gnilaitnederC
rehcaeT
no
noissimmoC
ainrofilaC
:ecruoS
6622 California State Auditor Report 2004-108 California State Auditor Report 2004-108 6633
As the figure indicates, the commission has been developing
Some program program standards since 1998. However, some are still under
standards are still under development, and several others, including the program
development, and not all standards for core curriculum subjects, have not yet been
colleges and universities implemented by all colleges and universities. For example, the
have implemented several English standard was adopted in January 2003. The commission
others, including the indicates that as of September 30, 2004, it has approved two
program standards for of the 11 revised program proposals that it has received; and
core curriculum subjects. it estimates that about 43 more colleges and universities
will submit revised program proposals for approval around
March 2005. The commission began developing the final phase
of program standards covering five subjects in April 2004.
Because developing a program standard takes about a year and
a half, the commission staff plan to propose these five standards
to the commissioners in July 2005. If the commissioners adopt
a two-year implementation period, as they have previously,
colleges and universities will not finish implementing these
program standards until mid-2007, nearly nine years after the
act became law.
Although the commission created an approach to develop
standards as the act required, it did not use an overall plan that
included timelines for developing standards and the resources
it planned to use. Such a plan would also provide a framework
for describing how the commission would finish implementing
the program standards that the act required. For example, in
addition to the standards still undeveloped, the commission
has not fully implemented all aspects of its teacher preparation
program standards, including the teaching performance
assessment and assessment quality standards. These standards
are critical components of the learning-to-teach continuum
that the act intended to put in place to strengthen teacher
preparation. As we noted in Chapter 1, although the act requires
all teacher preparation programs to use a teaching performance
assessment, it makes this subject to the availability of funding in
the annual budget act. Nevertheless, the commission is working
with colleges and universities to implement this requirement
on a voluntary basis. In addition, the commission indicates that
it has initiated a study on the potential consolidation of the
teaching performance assessment with other exams.
An overall plan would also guide the commission’s efforts
to perform all of its other regular standard-setting activities,
including periodically updating program standards and
validating examinations. The commission believes that program
standards and examinations should be reviewed periodically
6644 California State Auditor Report 2004-108 California State Auditor Report 2004-108 6655
and rewritten as job requirements and expectations change over
An overall plan time. These standard-setting activities can be costly and require
would provide a coordination with education professionals and independent
framework for how contractors. Thus, such a plan would benefit the commission by
the commission would allowing it to plan for the resources it needs to meet its ongoing
finish implementing standard-setting responsibilities.
the program standards
required by the act Although the commission recently developed a planning
and also guide the calendar to guide its professional services division in reporting
commission’s efforts to the commissioners, it does not include estimates of staff time.
to perform all of its Also, in June 2004 commission staff prepared a status report for
other regular standard- the commissioners that included a description of its progress in
setting activities. implementing the act’s requirements and its efforts in assisting
colleges and universities to implement new programs. However,
the report does not include a timeline or a strategy for the
commission to complete its planned tasks, nor does it list the
resources the commission plans to use. Although these two
reports are useful, they do not represent an overall plan that the
commission could use to guide its efforts to implement the act’s
requirements fully.
The commission recognized the need for planning and required
prospective contractors to prepare a management plan in their
response to the commission’s request for proposals to develop
examinations related to program standards. The process of
developing examinations takes time, resources, and significant
coordination with education professionals. Therefore, the
contractor who won the bid created management plans that
described an approach for managing its responsibilities in the
phases of developing examinations. For each of the phases,
the contractor also developed detailed project management
plans that described how the contractor planned to accomplish
key activities. For example, the management plans included
monitoring elements, such as a timeline that schedules the
estimated dates for starting and completing key activities.
This type of planning would also be helpful to the commission
to guide its efforts and focus future activities. Although the
commission has made progress over the past six years, not all
of the intended benefits of the act have yet been realized. An
overall plan could help the commission determine what it has
yet to complete and the resources it will need to put in place the
learning-to-teach continuum that the act intended to achieve.
6644 California State Auditor Report 2004-108 California State Auditor Report 2004-108 6655
THE COMMISSION’S RECENT EXPERIENCES IN
DEVELOPING PROGRAM STANDARDS PROVIDE
AN OPPORTUNITY TO REFINE ITS STANDARDS
DEVELOPMENT PROCESS
The commission’s recent experiences in developing standards
in response to the teacher preparation reforms that the act
required provide it with an opportunity to refi ne the process
it uses to establish program standards in order to improve its
future efforts. The commission’s staff coordinate and monitor
the development of each standard, and they informally share
with one another the lessons they learned from developing each
standard. Although this informal process is helpful, it does not
ensure that the commission rigorously evaluates what went well
or poorly during the development of a program standard and
does not ensure that it makes future improvements. Further, a
formal process is valuable should staff turnover occur between
standard revision periods.
During our review of the fi ve sets of standards identifi ed in
the text box, we identifi ed several areas in the commission’s
development of standards that lack structure and that it could
improve, including how it forms advisory panels and how it
conducts and analyzes survey results. Further, the
commission has an inadequate policy for ensuring
that staff maintain important documents relating to
The fi ve sets of standards we reviewed:
the development of program standards.
1. English
2. Mathematics The commission’s policy requires that the
3. Art commission retain in offi ce areas all records
that it produces or maintains for as long as they
4. Induction
serve the immediate purposes for which they
5. Administrative services
were created. The policy also requires that staff
retain advisory panel research records in offi ce
areas for at least one year and then transfer
them to the state records center for 15 years. However, the
commission’s policy does not include guidelines for retaining
specifi c documents created during the development of program
standards. Consequently, the commission staff assigned to a
program standard decide which documents to keep and the
length of time to keep them; individual staff members store
these documents in their workspace. Letting staff decide how
to maintain documents is a poor idea, because people have
different views about what documents are important, how
to store them, and how long to keep them. Further, if some staff
members leave the commission, others may not be able to locate
6666 California State Auditor Report 2004-108 California State Auditor Report 2004-108 6677
these documents. As a result of this inadequate practice of retaining
records, we were not able to review some of the key documents
relating to how the commission developed program standards.
The Commission Does Not Use a Methodical Approach to
Form Advisory Panels
Our review of the selection process for the five advisory
Lacking a methodical panels found that the commission does not use a methodical
approach, the approach to evaluate candidates for panels. Although we did
commission may have not determine that the panel members whom the commission
difficulty ensuring that its selected were unqualified, without a way to evaluate or rate
panel member selections candidates’ qualifications, the commission does not know
are considered objective whether other candidates are more qualified. Further, it may
by outside parties. have difficulty ensuring that outside parties consider its
selections of panel members to be objective.
For four of the panels we reviewed—English, mathematics,
art, and administrative services—the commission required
candidates to submit an application and in most cases a résumé
to show their interest in serving on the panel. In addition, for
the English, mathematics, and art panels, the commission asked
candidates to demonstrate that they met at least one of the
commission’s qualifications, which were similar among the three
standards. For example, candidates for the English advisory panel
were to meet at least one of the qualifications outlined in the
text box on the next page. For the administrative services panel,
the commission’s qualifications were that the candidate hold a
current position as an administrator, educator, or school board
member or be a parent. For all four standards, the commission
solicited the availability of panel members to attend meetings
to develop standards. Commission staff evaluated the candidate
applications and proposed a panel to the executive director, who
made the final decision to appoint panel members.
However, the commission’s applications for the English,
mathematics, and art panels asked the candidates only for
a general description of how they met the commission’s
qualifications rather than asking them to address specifically
how they met the qualifications. Lacking a more structured
application to solicit explicitly how candidates met the
qualifications, the commission may not have received sufficient
information to be able to evaluate the candidates. For the
administrative services panel, the application appropriately asked
for the candidate’s position.
6666 California State Auditor Report 2004-108 California State Auditor Report 2004-108 6677
For four of the panels we reviewed, the commission
did not consistently use a ranking process to ensure
The commission advertised that it
would select English panel members that it appointed the most qualifi ed or desired
who possessed one or more of the candidates to serve. For example, when forming
following qualifi cations: the English and mathematics advisory panels, the
commission assigned each candidate a rank of 1,
1. Teacher in grades 7 to 12 with
2, or 3, with the score based on the candidates’
outstanding programs in English.
qualifi cations and availability for meetings.
2. Active in current efforts to improve
However, the commission could not show us how
curriculum, instruction, and teacher
preparation. it arrived at these rankings and explained that
the rankings were based on what the commission
3. Recent experience teaching English in
grades 7 to 12. called a “holistic” approach of evaluating candidate
qualifi cations. According to the commission, the
4. Leadership role in English at the statewide
level or at the local level. holistic approach evaluated candidates based
on their educational qualifi cations, experience,
5. Experience providing professional
and knowledge of academic content standards
development to teachers or prospective
teachers of English. for pupils. The commission also indicated
that members of the State Board of Education
6. Experience teaching students from diverse
linguistic and cultural backgrounds. (state board) who were particularly interested
in assuring the alignment of the subject matter
7. Experience developing, administering, or
scoring constructed-response test items. standards for teachers with academic standards for
pupils, participated in reviewing the panel member
8. Experience as mentor teacher or
supervising teacher of English. qualifi cations and selecting panel members. However,
the commission did not rank 13 of the 61 candidates
9. Distinguished college or university
who applied for the English panel and 19 of the
faculty member who teaches English to
prospective teachers. 103 who applied for the mathematics panel.
10. Knowledgeable specialist in school
curricula who focuses primarily in English. The commission did not use a ranking process
when evaluating candidates to serve on the art
11. Highly competent school principals and
assistant principals who have expertise in and administrative services panels. According
teaching English. to the commission, it selected panel members
based on their educational qualifi cations and
Source: California Commission on Teacher
experience. However, the commission stated that
Credentialing.
the art standards, although important, were not
as important as the English and mathematics
standards; therefore, the review process did
not involve the same level of rigor. When forming the
administrative services panel, the commission indicated that
it focused on selecting panel members who possessed specifi c
knowledge of the administrative services credential area.
For all four panels, the commission did not use a checklist or
other review tool to ensure that the proposed panel collectively
met all the commission’s qualifi cations. Without a tool that
showed how the commission methodically selected the candidates,
we could not ascertain how it determined that the candidates it
6688 California State Auditor Report 2004-108 California State Auditor Report 2004-108 6699
appointed were the best qualified. Although the panel members
appear to possess the qualifications that the commission required,
a methodical approach to making selections would help ensure
that the commission selects the best qualified candidates
and would better establish that the commission selects panel
members objectively.
In the documents soliciting candidates to apply for these four
panels, the commission indicated that its panel selection process
would also focus on other factors, such as whether the candidate
The role of ethnic was a teacher or an administrator, and that it would emphasize
diversity and other ethnic diversity in making its selections. Although the
factors was unclear in commission sometimes gathered this information, we found no
advisory panel selections, evidence of how the commission evaluated these factors when
despite the significance selecting members for these four panels. Thus, the role of these
that the commission factors was unclear, despite the significance that the commission
placed on them in its placed on them in its documents requesting candidates to apply.
documents requesting
candidates to apply. For the fifth standard we reviewed, induction programs,
the commission used a different process when forming the
advisory panel. It did not solicit nominations, nor did it require
interested candidates to submit an application. Instead, the
commission indicated that it selected panel members that
were actively working in beginning-teacher support programs
and individuals from the California Department of Education.
Although the commission stated that panel members have
experience in administering Beginning Teacher Support and
Assessment programs, it does not know whether other interested
candidates might be more qualified than those it selected.
The Commission Could Do More to Report Its Survey Results
and Consider the Views of Minority Groups
When presenting the results of its field-review survey to the
commissioners as a basis for recommending the standards
to adopt, commission staff did not always put the results
in perspective in such a way that the commissioners could
understand their relevance. For example, when commission
staff proposed the four core subject matter standards to the
commissioners for adoption, it reported in its agenda item
that it had received 432 responses to the field-review survey.
However, only 23 of the responses the commission received were
for English and 21 responses were for mathematics. Although
the commission received 33 responses for social science, it
could not locate any of the remaining responses related to the
science standard and thus could not tell us the composition of
6688 California State Auditor Report 2004-108 California State Auditor Report 2004-108 6699
the remaining 355 responses. Similarly, commission staff also
Commission staff did reported 115 responses for the art and three other single subject
not always put the matter standards rather than reporting the results for each
field survey results in standard. Disclosing the actual responses for each standard, as
perspective so that commission staff did for the induction and administrative services
the commissioners standards, would have provided the commissioners with a better
could understand their perspective on the actual results of the field-review survey.
relevance.
Also, the commission is missing an opportunity to ensure
further that its subject matter requirements are free from
biased language and content because its process excludes
the views of some groups. As we noted previously, the
commission’s contractor conducts a content validity study.
The commission’s contractor analyzes the results of its survey
and presents the results to the commission and the advisory
panels in a tabular format that aggregates the responses from
population groups including African-American, Asian, Hispanic,
and other target groups of education professionals. The
contractor flags for the advisory panel’s review subject matter
requirements that receive a lower rating for necessity at entry or
importance for effective job performance. However, our review
of the criteria for flagging an item found that in some instances
the contractor did not flag subject matter requirements to which
minority groups gave low ratings because the number of people
responding in these groups was less than the number that
would trigger a review flag. Although these groups represent
a low percentage of the total responses, when the majority of
respondents in one group expresses a concern, the commission
could benefit from providing the results to its advisory panels
and bias review committee to ensure further that its subject
matter requirements are free from biased language or content.
The Commission Needs to Better Document Its Process for
Ensuring That Program Standards Align With State Academic
Content Standards
In response to the act’s credentialing requirements, the
commission developed new program standards and
examinations that align with the academic content standards
for students that the state board adopted. This alignment
requirement is applicable to three of the five standards we
reviewed—English, mathematics, and art. However, the
commission did not always adequately document how it
ensured the alignment of standards.
7700 California State Auditor Report 2004-108 California State Auditor Report 2004-108 7711
The commission asked its advisory panels to develop subject
matter standards that align with the student academic content
standards. According to the commission, it secured a contractor
to independently review the English and mathematics subject
matter standards to address alignment and congruence concerns
that members of the state board and a stakeholder had raised.
However, for these two standards, the commission was not able
to provide us with all of the relevant documents that would
demonstrate that the contractor performed an adequate review.
For example, for the English standards, the commission was
not able to provide us with the recommendations developed
from the review or how the advisory panel addressed any
related recommendations. Further, commission staff were not
able to answer our questions about the contractor’s method
of performing the independent reviews or about other
inconsistencies on the documents that it provided. Because the
commission could not demonstrate to us that its contractor
performed the studies appropriately, we question whether
the commission could defend that these two subject matter
requirements were appropriately aligned. The commission was
able to provide sufficient documentation for us to conclude that
the alignment study of art standards was performed.
THE COMMISSION HAS SUSPENDED CONTINUING
ACCREDITATION REVIEWS, AND IT IS EVALUATING
ITS ACCREDITATION POLICIES
Continuing accreditation reviews are an important component
of the commission’s accreditation system because they
help ensure that colleges and universities operate teacher
preparation programs that meet the commission’s standards.
The commission performs continuing accreditation reviews
primarily through on-site reviews. However, in December 2002
the commission suspended its continuing accreditation
reviews of all colleges and universities, except for the few
seeking national accreditation, to allow them time to
implement new teacher preparation programs and to evaluate
its own accreditation polices and procedures. Since then, the
Suspending continuing commission has limited its accreditation activities to performing
accreditation reviews initial accreditation reviews, which are primarily desk reviews of
limits the commission’s college and university teacher preparation program proposals.
ability to ensure that In March 2004 the commission extended this suspension,
colleges and universities indicating that doing so would provide financial relief to
are operating programs the commission and to colleges and universities. However,
that meet its standards. suspending continuing accreditation reviews limits the
commission’s ability to ensure that colleges and universities that
7700 California State Auditor Report 2004-108 California State Auditor Report 2004-108 7711
recommend prospective teachers for credentials are operating
programs that meet the commission’s eight common standards
and specific teacher preparation program standards. For
example, one of the eight common standards, resources, requires
colleges and universities to demonstrate that they consistently
allocate resources to credential programs.
The California Education Code requires the commission to
adopt and implement an accreditation system to ensure that
colleges and universities operate teacher preparation programs
that meet the commission’s program standards. To comply with
this requirement, the commission adopted an accreditation
policy that includes initial reviews of teacher preparation
program proposals combined with a continuing accreditation
process that consists primarily of on-site reviews of how colleges
and universities operate their teacher preparation programs.
Following a continuing accreditation review, the commission
may do one of three things: (1) grant colleges and universities
full accreditation, (2) grant them accreditation with stipulations,
or (3) deny accreditation. Once it has granted a college or
university full accreditation, the commission does not conduct
another continuing accreditation review for five to seven years.
However, if it grants accreditation with technical or substantive
stipulations, the college or university must take corrective action
within one year, and the commission may schedule another
on-site review. Finally, if the commission denies accreditation,
the college or university must phase out its teacher preparation
program by the end of the semester or quarter of that
accreditation denial.
In its December 2003 agenda item, the commission reported
In the four and a half that in the four and a half years before it suspended continuing
years before it suspended accreditation reviews, it found that colleges and universities
continuing accreditation did not always operate their programs in accordance with its
reviews, the commission standards. For example, of the 73 continuing accreditation
granted half of the reviews it conducted between fiscal years 1997–98 and 2002–03,
colleges and universities the commission accredited 37 of the colleges and universities
it reviewed accreditation with substantive or technical stipulations; it granted full
with stipulations because accreditation to the remaining 36. The commission indicated
they did not operate that the 36 colleges and universities have since taken corrective
their programs fully action to clear the stipulations, and the remaining one is in the
in accordance with final process of clearing the stipulations.
standards.
Although the commission planned to resume continuing
accreditation reviews in July 2004, it instead extended the
suspension in March 2004 to June 2005, to give it time to focus
7722 California State Auditor Report 2004-108 California State Auditor Report 2004-108 7733
on reviewing its accreditation policy. Until the commission
resumes the continuing accreditation reviews, it cannot ensure
that colleges and universities operate teacher preparation
programs that meet its program standards. Further, because the
commission conducts on-site reviews of colleges and universities
every five to seven years as part of the continuing accreditation
policy, when the commission resumes its continuing accreditation
review activities in July 2005, up to 25 colleges and universities
could be overdue for a continuing accreditation review.
The commission believes that it should evaluate its current
accreditation policy, which it adopted in 1993, to address
current state and national trends that call for greater
accountability in education using performance measures for
teachers and pupils. However, before making significant changes
to its policy, the commission is required to consult with an
independent evaluator and with representatives from colleges,
universities, and other educational organizations. To comply
with this requirement, it secured an independent contractor
in fiscal year 1999–2000 to examine the policy, including the
process for conducting continuing accreditation reviews, over a
three-year period. It asked the contractor to determine whether
the continuing accreditation review process provides a fair
and productive review that results in college and universities
making improvements. However, the commission did not ask
the contractor to recommend alternatives that would result in
meaningful oversight of colleges and universities in light of
state and federal requirements for greater accountability and
performance measures. Consequently, when the commission
received the contractor’s final report in March 2003, it found
that the recommendations had limited use for this purpose.
Further, although the commission has been working with
representatives from colleges and universities to evaluate the
accreditation policy, under its current plans, it will not propose a
revised policy to the commissioners until August 2005.
RECOMMENDATIONS
The commission should develop an overall plan to guide its
efforts to finish implementing program standards that the act
required and to guide its ongoing standard-setting activities to
include timelines and the resources it plans to use.
7722 California State Auditor Report 2004-108 California State Auditor Report 2004-108 7733
To ensure that it objectively appoints education professionals
to its advisory panels, the commission should develop a
methodical approach that includes evaluating candidates’
qualifications against the qualifications the commission seeks in
panel members.
To provide the commissioners with a better perspective on the
results of field-review surveys, commission staff should present
the actual results for each standard.
The commission should follow its policy for retaining records
to ensure that it maintains important documents for a specified
time in case it needs them later for general information,
research, or legal proceedings. The commission should also
implement a policy to designate the specific standard-setting
records that its staff should retain.
To ensure that colleges and universities operate programs that
meet the commission’s standards, the commission should
promptly resume continuing accreditation reviews. Further, it
should take steps to complete the evaluation and revision of its
accreditation policy promptly.
We conducted this review under the authority vested in the California State Auditor by
Section 8543 et seq. of the California Government Code and according to generally accepted
government auditing standards. We limited our review to those areas specified in the audit
scope section of this report.
Respectfully submitted,
ELAINE M. HOWLE
State Auditor
Date: November 9, 2004
Staff: Nancy C. Woodward, CPA, Audit Principal
John Baier, CPA
Benjamin Miles Belnap, CIA
Ana Clark
Paul Philip Zahka
7744 California State Auditor Report 2004-108 California State Auditor Report 2004-108 7755
Agency’s comments provided as text only.
Commission on Teacher Credentialing
1900 Capitol Avenue
Sacramento, California 95814
October 20, 2004
Ms. Elaine M. Howle*
State Auditor
555 Capitol Mall, Suite 300
Sacramento, California 95814
Dear Ms. Howle:
All state agencies in California need to be ever mindful of opportunities to review and revise their
policies and procedures to be more responsive to the public and their elected representatives.
The Commission welcomes objective reviews of its programs and operations as well as resulting
commendations and recommendations for improvement. We appreciate the Legislature’s interest
in determining whether the Commission can continue to pursue efficiencies while maintaining
effectiveness.
On behalf of the Commission on Teacher Credentialing I want you to know that we have reviewed
the draft audit findings entitled “California Commission on Teacher Credentialing: It Could Better
Manage Its Credentialing Responsibilities” received October 14, 2004. Given the significance of
this review in relation to comprehensive studies of state government it is important to note the
conclusion that the work of the Commission is not duplicative of any other entity. We appreciate the
commendations regarding effectiveness of teacher intern programs and efforts by the Legislature
1
and the Commission to streamline credentialing for teachers prepared in other states. The
Commission has serious concerns, however, about what we believe are significant omissions,
errors and misinterpretations.
I have included a detailed summary of our concerns, which fall into the following categories:
2
1. Omissions. The audit charge may have constrained the scope of this particular review
with respect to a workload analysis. For example, teacher credential candidates,
through their fees, funded an extensive workload analysis conducted by an independent
contractor supervised by the Office of the Legislative Analyst. This workload analysis
showed that the Commission was significantly understaffed. The Legislature corrected
this understaffing, however, subsequent to that Legislative effort the Commission staff has
been cut by 22% percent while the workload has increased by 6% percent.
We believe it is important to at least acknowledge circumstances created by the ongoing
3
loss of staff, increasing workload, across the board program reductions and the critical
revenue deficiencies that have resulted in further staff and program reductions is
negligent in that these factors have a direct impact on the Commission’s capacity to
* California State Auditor’s comments begin on page 109.
7744 California State Auditor Report 2004-108 California State Auditor Report 2004-108 7755
Ms. Elaine M. Howle
October 20, 2004
Page 2
address its core mission. Further, it may be useful to describe procedures put into
4
place over the past three years to modify business practices and reassign workload to
reflect ever-increasing workload with less staff resources in an effort to address critical
credentialing priorities, such as credential suspensions and revocations.
2. Proposals that Would Necessitate Changes in Statutes. Some recommendations,
5
although welcome, would necessitate amendments to the Education Code since they
propose responsibilities that exceed the Commission’s current legislative mandates.
3. Proposals that Would Require Additional Funding. Other recommendations would
require substantial new funding. For example, the Commission was funded to replace
outdated technology to avoid a collapse of the credentialing issuance system. The level
of funding provided was intended only to replace worn equipment and to process the
6
additional workload anticipated due to mandates such as class size reduction. Adding
technology to substantially reduce processing times would result in significant costs that
the Commission does not currently have revenue to support.
4. Proposals that Would Require Additional Staffing. Other recommendations would require
7
additional staffing. For example, conducting ongoing analyses of unemployment trends
would require additional staff positions, as this is outside of the Commissions current
scope of work.
In addition, the attachment summarizes what we believe are inaccuracies and misinterpretations
8
of the data. For example, to conclude that the Commission has suspended accreditation of
preparation programs is not accurate. Indeed, the Commission, with the assistance of a large
number of classroom teachers, site administrators, parents and educators has reviewed and
revised four sets of preparation standards—for teacher preparation, induction, subject matter and
advanced preparation. To date, all teacher preparation that have submitted proposals for these
programs have been reviewed against these standards. This is the first step in the mandated
restructuring of standards governing teacher preparation designed to align teacher preparation
standards with standards for pupil performance adopted by the State Board of Education, as
mandated by Senate Bill 2042 (Alpert).
Again, we welcome the opportunity to discuss Commission operations with representatives of the
public and state policymakers as we all strive to improve our services in the most efficient and
effective manner possible.
Sincerely,
(Signed by: Lawrence H. Madkins, Jr.)
Lawrence H. Madkins, Jr.
Commission Chair
Attachment
7766 California State Auditor Report 2004-108 California State Auditor Report 2004-108 7777
COMMISSION RESPONSES TO BSA AUDIT REPORT RECOMMENDATIONS
Audit Report Recommendations Commission Responses
CHAPTER 1
To determine their success, the The Commission has historically evaluated its teacher
Commission should establish development programs based on compliance with the
performance measures for each of its requirements for grant funds that are set forth in the
teacher development programs. In Request for Applications. The Commission agrees
addition, the Commission should do the it could adopt additional performance measures that
following: address the effectiveness of programs in meeting
• Ensure that the statistics it presents in statutory objectives. The consent form process,
its program reports are consistent and which the Commission implemented in 2001 to
that it maintains the support for these track candidate enrollment in each of its teacher
statistics. development program will help the Commission
• Monitor how local teacher monitor the effectiveness of programs in helping
development programs assess the candidates achieve a credential.
academic progress of participants
and establish consequences for
underperformance.
• Resume requests for budget
augmentations to fund an independent
evaluation of its paraprofessional
program that assesses all the
requirements in the applicable statute
or seek to amend those parts of the
law that it believes would be too costly
to implement.
To provide context to education The Commission agrees that a thoughtful analysis of
professionals and policy makers for why teacher supply and demand data is helpful to policy
credential, permit, and waiver numbers makers at all levels however such an analysis would
7
have changed, the Commission should require additional resources and information that are
include an analysis with the statistics it not currently available to the Commission. Such an
publishes in its annual reports. analysis could be at odds with state policy directives
or increase the State’s exposure to litigation.
Historically, this type of analysis has been undertaken
by independent entities that specialize in conducting
research and analysis on important educational
issues. The Commission provides data upon request
to independent bodies that conduct such analyses.
7766 California State Auditor Report 2004-108 California State Auditor Report 2004-108 7777
COMMISSION RESPONSES TO BSA AUDIT REPORT RECOMMENDATIONS
The Commission should collaborate with The Commission agrees that implementation of the
colleges and universities to determine Teaching Performance Assessment (TPA) would help
what funding is necessary to activate complete the vision of SB 2042, and the Commission
and maintain the teaching performance has endeavored to work with institutions to implement
assessment, as envisioned by the the TPA. The Commission looks forward to direction
enabling legislation. They should then from state policy makers in resolving the funding
request the Legislature and governor’s issues that have prevented the full implementation of
office to authorize the function in future the assessment. Commission staff continues to work
budget acts. with colleges and universities to implement the TPA
on a voluntary basis.
The Legislature may wish to consider The Commission agrees that teacher retention data
giving the Commission a specific would be helpful information for policy makers to
policy directive to obtain and use have. Such data could be useful analyzing the effect
teacher retention data to measure the teacher preparation programs, induction programs,
performance of the teaching credential and employment conditions have on teacher
9
process and of teacher preparation retention. However, the collection of such data would
programs and provide this information in require an integrated data collection system and
its annual reports. coordination with other state entities and could have
significant costs to both state and local agencies.
To aid it in developing performance The Commission is aware of, and has commended,
measures for teacher preparation the efforts of the CSU in developing its annual
programs, the Commission should keep employer survey. Data collected from such a survey
itself informed of surveys and reports could provide useful information about the quality
that are prepared by other entities such of teacher preparation programs. The systematic
as the administrator survey the CSU is collection of valid and reliable data gathered through
developing. surveys and performance assessments is under
consideration as part of the Commission’s review of
its accreditation system.
The Commission’s executive office The Executive Director annually prepares a list of
0
should present the Commissioners with accomplishments directly linked to the strategic goals
an annual status report on how the that are read at a Commission meeting and in the
Commission has achieved the goals and Official Minutes of the meeting. Also, the agenda
tasks outlined in the strategic plan. items provide a status report on the goals and tasks
at each meeting.
The Commission should continue to The Commission concurs and has been exploring the
consider ways to streamline the teacher streamlining of exams for the past year.
credentialing process, such as the
consolidation of examinations required of
credential candidates. If the Commission
determines that specific requirements
are no longer necessary credential
requirements, it should seek legislative
changes to applicable statutes.
7788 California State Auditor Report 2004-108 California State Auditor Report 2004-108 7799
COMMISSION RESPONSES TO BSA AUDIT REPORT RECOMMENDATIONS
CHAPTER 2
To ensure that the public information The Commission concurs with this recommendation
it provides meets the needs of its and has already implemented a method to gather
customers, the certification division data on e-mails. Also, the Web site is currently being
should routinely update its Web site and redesigned to reflect the questions that are being
leaflets based on an analysis of customer asked on the phones and in e-mails. The new Web
data. Specifically, the certification division site will be unveiled on January 3, 2005.
should do the following:
• Use the phone system report to its
full potentials by identifying specific
areas where the certification division
can improve the information it
communicates to the general public
on the web site and in leaflets.
• Gather meaningful data about
the types of e-mail questions the
certification division receives and use
the data to improve the information it
communicates to the general public
on the Web site and in leaflets.
• Ensure that the information presented
on the Web site is accurate and easy
for customers to use.
To improve the efficiency of e-mail The Commission concurs and has already started a
processing, the certification division pilot project.
should automate its response to and
routing of e-mails.
To ensure the effective management The Commission does not believe that this
q
of the unit’s application workload, recommendation is feasible due to the way the
the certification division should applications are entered into the database. The
routinely monitor the composition of specific type of application is entered at the time it is
the applications that are waiting to be evaluated, which is at the end of the processing time.
processed, and collect and analyze data To redesign the system would be cost prohibitive
on the average review times for different because it would require costly design modifications
types of applications. to a system that has not been completely tested
and implemented as well as reclassification of staff.
There is no cost benefit analysis in this report that
addresses any added value of having this data. The
division already monitors the number of applications
that each team processes and effectively manages
staff based on that report.
7788 California State Auditor Report 2004-108 California State Auditor Report 2004-108 7799
COMMISSION RESPONSES TO BSA AUDIT REPORT RECOMMENDATIONS
To ensure that the certification division The Commission concurs and expects to develop an
continues to meet the 75-business-day automated report once TCSIP is implemented.
regulatory limit, the certification division
should routinely create automated
reports from TCSIP to track the average
processing times and regularly list all
applications that have taken more than
75 business days to process.
To ensure that it optimizes the timesaving Although the Commission concurs that electronic
benefits of TCSIP, the certification submission is the more efficient method and should
w
division should use automated processes be encouraged, instituting such a requirement would
rather than manual ones were possible. require regulatory change. The recommendation is
Specifically, the certification division silent as to the effect such a requirement would have
should do the following: on candidates who do not have the requisite credit
card necessary to accomplish an on-line transaction.
• Require colleges and universities The issue of enhancing the on-line process by
to submit credential applications to expanding TCSIP is dependent on resources.
the commission electronically to the
extent that is economically feasible.
• Consider expanding TCSIP to allow
school districts to submit applications
electronically and to allow the virtual
credential officer to process routine
applications.
To encourage more customers to renew The Commission concurs and is already in the
their credentials online, the certification process of making changes to the Web site.
division should gather data on and study
the percentage of renewals it received
online for different types of credentials to
identify areas where additional outreach
efforts may be necessary. Also, the
certification division should do the
following:
• Publicize the fact that online renewals
are given a processing priority over
paper renewals.
• Make the link to online renewals more
obvious on the Commission’s Web
site.
8800 California State Auditor Report 2004-108 California State Auditor Report 2004-108 8811
COMMISSION RESPONSES TO BSA AUDIT REPORT RECOMMENDATIONS
CHAPTER 3
The Commission should develop an The Commission has finished its work related to
e
overall plan to guide its efforts to finish the development and implementation of program
implementing program standards by the standards pursuant to SB 2042 with the exception
act and to guide its ongoing standard to the implementation of the Teaching Performance
setting activities that include timelines and Assessment. Program sponsors are submitting
resources it plans to use. responses to the standards according to the timelines
adopted by the Commission. Staff look forward
to further direction from state policy makers with
regard to the availability of funding for the Teaching
Performance Assessment. The Commission agrees
that a long-range plan with associated timelines for
reviewing and updating future program standards
would be a helpful planning tool.
To ensure that it objectively appoints The Commission staff uses a methodical approach
r
education professionals to its advisory to the appointment of advisory panels. The type of
panels, the Commission should develop work to be done and balancing factors relative to
a methodical approach that includes staff, stakeholders, agencies, and political context are
evaluating candidate’s qualifications considered when evaluating applicant qualifications.
against the Commissions qualifications. The Commission staff strives to assemble panels
that are balanced, not only in terms of the individual
qualifications of each member, but also relative to
geography, gender, race and ethnicity, urban/rural,
size of institution or local education agency, and
constituent representation. While such an approach
does not lend itself to a “checklist” type of evaluation
of applicants relative to qualifications and standards
adopted by the Commission. This has resulted
in consensus by the field with regard to both the
process and product.
t
To provide the Commissioners with a This has not been an issue raised by the Commission
better perspective on the results of the or stakeholders.
field review surveys, commission staff
should present the actual results for each
standard.
8800 California State Auditor Report 2004-108 California State Auditor Report 2004-108 8811
The Commission should follow its record The Commission agrees that improving the existing
retention policy to ensure that important records management policy is a worthwhile goal.
documents are maintained for a specified
time in case they are needed later for
general information, research, or legal
proceedings. The Commission should
also implement a policy to designate the
specific standard setting records that
should be retained, the length of time to
retain them, and the locations and format
– paper or electronic – in which they
should be stored.
To ensure that colleges and universities The Commission continues to be fully engaged in the
operate programs that meet the evaluation and revision of its accreditation framework.
Commission’s standards, the commission All programs have undergone in-depth, rigorous initial
should promptly resume continuing accreditation review within the past two years for one
8
accreditation reviews. Further, it should or more educator preparation programs. To resume
take steps to promptly complete the site visits during the 2004-2005 fiscal year would be
evaluation and revision of its accreditation both premature and unproductive. Institutions would
policy. not have sufficient time to prepare for accreditation
visits and would face significant unanticipated costs
in planning and organizing a visit. Without sufficient
lead-time to prepare for a visit, the quality of findings
could be compromised. Given the 24-month lead-
time institutions need (and have requested) to plan,
budget, and prepare for a visit, the earliest practical
date that the Commission could initiate site visits
would be the fiscal year 2006-2007. The Commission
looks forward to working with the state policy makers
to ensure that the Commission’s funding is adequate
to support this important responsibility.
EXECUTIVE SUMMARY
To better plan and evaluate its efforts, the The Commission regularly updates its strategic
y
commission should regularly update its plan and quantifies performance measures where
strategic plan and quantify performance appropriate.
measures in terms of the results to be
achieved of all tasks where appropriate.
8822 California State Auditor Report 2004-108 California State Auditor Report 2004-108 8833
Commission on Teacher Credentialing
1900 Capitol Avenue
Sacramento, California 95814
u
CONFIDENTIAL
October 20, 2004
Ms. Elaine M. Howle
State Auditor
555 Capitol Mall, Suite 300
Sacramento, California 95814
Re: “California Commission on Teacher Credentialing: It Could Better Manage Its Credentialing
Responsibilities”
Dear Ms. Howle:
The Commission always welcomes reviews of its policies, programs and procedures as such
reviews allow us to be more responsive to the public and their representatives. We appreciate the
Legislature’s interest in determining whether the Commission can continue to pursue efficiencies
while maintaining effectiveness. Be assured that the Commission takes the audit process and the
impact of this forthcoming report very seriously.
However, given the significance of this endeavor, I have serious concerns that the audit report and
resulting recommendations did not take into consideration the statutory, resource and fiscal reality
within which the Commission works. This is significant to understanding both why current business
practices are the way they are, as well as the Commission’s capacity to implement the audit
recommendations given current legislative authority and fiscal constraints. We have what I consider
to be a very serious obligation to the school children of California and we strive every day to fulfill
this mission given fiscal, staffing and workload constraints.
The attached documents are provided as a response to the recommendations in the draft audit
report entitled “California Commission on Teacher Credentialing: It Could Better Manage Its
Credentialing Responsibilities” received October 14, 2004. The attachment includes a detailed
summary of our concerns, which fall into the following categories:
2
1. Omissions. The audit charge may have constrained the scope of this particular review
with respect to a workload analysis. For example, teacher credential candidates, through
their fees, funded an extensive workload analysis conducted by an independent contractor
supervised by the Office of the Legislative Analyst. This workload analysis showed that the
Commission was significantly understaffed. The Legislature corrected this understaffing,
however, subsequent to that Legislative effort the Commission staff has been cut by 22%
percent while the workload has increased by 6% percent.
8822 California State Auditor Report 2004-108 California State Auditor Report 2004-108 8833
Ms. Elaine M. Howle Confidential
October 20, 2004
Page 2
2. Proposals that Would Necessitate Changes in Statutes. Some recommendations, although
5
welcome, would necessitate amendments to the Education Code since they propose
responsibilities that exceed the Commission’s current legislative mandates.
3. Proposals that Would Require Additional Funding. Other recommendations would require
substantial new funding. For example, the Commission was funded to replace outdated
technology to avoid a collapse of the credentialing issuance system. The level of funding
provided was intended by the Legislature only to replace worn equipment and to process
the additional workload anticipated due to mandates such as class size reduction. Adding
6
technology to substantially reduce processing times would result in significant costs that the
Commission does not currently have revenue to support.
4. Proposals that Would Require Additional Staffing. Other recommendations would require
7
additional staffing. For example, conducting ongoing analyses of unemployment trends
would require additional staff positions.
I would also like to take this opportunity to point out two issues that are of serious concern with
regard to the content and tone of the draft document.
While I understand that the auditors assigned to this task may have felt constrained by what
was prescribed/limited by the scope of the audit, I find it inexplicable that the audit report and
recommendations do not address the state fiscal, staffing and political framework that effects
the Commission’s capacity to address its core mission, the licensure and discipline of educators
in California. Not acknowledging circumstances created by the ongoing loss of staff, increasing
3
workload, across the board program reductions and the critical revenue deficiencies neglects
factors having a direct impact on the Commission’s capacity to address its core mission. Further,
the audit report does not, by any significant measure, acknowledge the fact that, as a result of
4
these circumstances, Commission has modified its business practices and reassigned workload
over the past three years to assure that strategic priorities, credential issuance and credential
revocations, are addressed first.
The resulting dilemma for legislators as they review this work is that audit findings and
recommendations are presented without reference to the reality that the Commission has lost both
a significant amount of personnel and fiscal resources over the last three fiscal years, while at the
same time workload continues to grow. Further, the audit recommendations effectively assign
35
responsibilities without the fiscal or personnel resources or legislative authority to complete these
tasks. Absent the presentation of this type of information, and the attendant analysis with regard to
the impact on the Commission’s operations, there is no way that the Commission can be effective in
implementing a significant portion of the audit recommendations. Given this situation, I find myself
asking, what is the point of presenting the findings and recommendations if the other side of the
analysis isn’t also presented for discussion and consideration?
8844 California State Auditor Report 2004-108 California State Auditor Report 2004-108 8855
Ms. Elaine M. Howle Confidential
October 20, 2004
Page 3
1
Finally, after reviewing the draft I feel that it is important to share the impression that the draft
document appears to reflect an excessively negative tone, inconsistent with what was actually
reported in the audit findings and recommendations. This is of significant concern as the language
of the document, specifically the chapter and subchapter headings appear to be misleading
generalizations where in fact, there is scant evidence supported only by broad assumptions. This
is further confused by recommendations that, while they may be useful (in fact, the Commission
has already implemented several of them), are relatively minor in impact. Such recommendations
are made to appear as if they will result in significant changes in the way the Commission does
business when in fact, they are simply making minor and untested changes to the process.
We hope that this letter, and the longer document attached can assist you and your staff review as
u
you consider revisions in the draft audit report. The Commission will receive the larger document at
their November 30/December 1 meeting should the draft audit be finalized as is. In any case, we
request that the shorter letter and summary constitute our official response to the audit report, as
provided by law. If you would like to address any of these matters, please do not hesitate to contact
me at the number provided above.
Sincerely,
(Signed by: Lawrence H. Madkins, Jr.)
Mr. Lawrence H. Madkins, Jr.
Chair, California Commission on Teacher Credentialing
Attachments
8844 California State Auditor Report 2004-108 California State Auditor Report 2004-108 8855
CCTC Clarifications to BSA Audit Report
Page 5 (last paragraph)
Response: Overall, the report does not make a distinction between the process of developing and
i
implementing standards for teacher preparation programs pursuant to SB 2042 and other standards
development activities, such as single subject, pupil personnel services, and administrative
services. Further, the report confuses program standards with test specifications and requirements.
Although the goal is to align the program standards and tests, the two are not comparable or
interchangeable in consideration of the overall development process. This confusion has resulted in
erroneous and misleading conclusions and recommendations.
Page 6 (2nd paragraph)
o
The report states that the “Commission suspended visits in fiscal year 2001-2002.”
Response: To clarify, the Commission continues to be fully engaged in the evaluation and revision
of its accreditation framework. All programs have undergone in-depth, rigorous initial accreditation
review within the past two years for one or more educator preparation programs. To resume site
visits during the 2004-2005 fiscal year would be both premature and unproductive. Institutions
would not have sufficient time to prepare for accreditation visions and would fact significant
unanticipated costs in planning and organizing a visit. Without sufficient lead-time to prepare for
a visit, the quality of findings could be compromised. Given the 24-month lead-time institutions
need (and have requested) to plan, budget, and prepare for a visit, the earliest practical date that
the Commission could site visits would be the fiscal year 2007-2007. The Commission looks
forward to working with state policy makers to ensure that the Commission’s funding is adequate to
support this important responsibility. The report also indicates that the Commission was evaluating
p
its accreditation policy because it “believes it is outdated.” It should be noted that the Commission
has not said that it believes the system to be outdated, only that they want to review the system
to see if it is outdated. The Commission’s Committee on Accreditation has been working with
representatives from colleges and universities and K-12 education on a monthly basis to review
its accreditation policies. The recommendations for revised polices will be presented to the
Commission in May 2005.
In addition, we note that the report generally uses “colleges and universities” when talking about
a
accreditation, and does not appear to take into consideration that the Commission also accredits
school districts that operate professional preparation programs.
Page 8
“State law requires the commissioners to meet at least 10 times per year to conduct commission
business.”
s
Response: This statement is not accurate. Education Code Section 44219 requires the
Commission to meet at least once each month in no fewer than ten months each year. The
Commission is meeting this requirement with only eight meetings in 2004 and seven meetings in
2005 by scheduling two day meetings which span two months whenever possible.
Page 11
d
Response: Inclusion of 30-day sub permits and emergency permit data in the chart is misleading.
1
8866 California State Auditor Report 2004-108 California State Auditor Report 2004-108 8877
Type and Number of Credentials Granted (Processed) During FY 2003-04
(This is workload data, not based on Issue Date, includes First/New types and Renewals)
Source: CCTC (2003-04Workload Report)
Auditor Report Totals CCTC Workload Report Totals
Single Subject 30,704 Single Subject 31,453
Multiple Subject 58,538 Multiple Subject 59,698
Emergency Permits 72,617 Emergency Permits 14,261
Day-to-day Sub Permits (all types) 58,356
Other permits 845
Cross-cultural / Bilingual 10,647 Cross-cultural / Bilingual 16,049
Administrative Service 9,899 Administrative Service 10,403
All Others 56,334 All Others 47,674
Total 238,739 238,739
Page 11
The report indicates that to continue teaching after the five-year preliminary credential, an individual
must complete an induction program, fifth year of study, or obtain national board certification.
f
Response: Attainment of National Board Certification does not waive the induction requirement for
beginning teachers who are prepared in state.
Page 11
The report states that candidates “demonstrate academic preparation in the subject he or she
wishes to teach by either completing an approved course of study in the subject or passing an
examination on the subject.”
Response: The statement is true for single subject credential candidates only. The statement is
g
no longer true for multiple subject credential candidates, who can only demonstrate academic
preparation by passage of a commission-approved examination.
Page 13 (1st paragraph)
The report states that “Under the direction of the State Board of Education, the superintendent of
public instruction, Education provides education policy direction to local school districts.”
h
Response: The word “Education” that follows superintendent of public instruction should be
deleted.
INTRODUCTION
2
8866 California State Auditor Report 2004-108 California State Auditor Report 2004-108 8877
Page 16 (last paragraph)
The report indicates that the Commission is in the midst of developing new program standards as a
result of SB 2042.
Response: This statement is incorrect. The Commission has completed developing new program
j
standards pursuant to SB 2042, and has adopted all new program standards pursuant to SB 2042.
Page 17 (1st paragraph)
The report indicates that the Commission uses a panel of education professionals to develop each
program standard.
Response: A more accurate statement would be to indicate that the Commission seeks the advice
k
from and participation of education professionals in the development of each set of program
standards. (Each set of program standards typically includes several individual standards). The
Commission has used advisory panels, task forces, design teams, focus groups, review teams, and
work groups depending on the scope of work to be done.
Page 17 (last paragraph, next to last sentence)
The report states that “The Commission suspended continuing accreditation reviews of colleges
and universities in December 2002 and is updating the accreditation process to meet the
requirements of the act.”
Response: It is unclear what is meant by this statement. The Commission suspended accreditation
site visits in December 2002 in order to allow programs to respond and go through the initial
l
accreditation review for 2042 programs and to focus its material and human resources on
conducting those in-depth initial reviews. The Commission’s review of its accreditation policies is
not related to the requirements of SB 2042.
Chapter 1
i
Overall, the report does not make a distinction between the process of developing and
implementing standards pursuant to SB 2042 and other standards development activities, such as
single subject, pupil personnel services, and administrative services.
Page 18 (2nd paragraph)
The report indicates that the Commission has responsibility for three teacher development
programs.
Response: The Commission’s co-administration of the Beginning Teacher Support and
;
Assessment (BTSA) program actually gives the Commission responsibility for four teacher
development programs.
Page 19
The report notes that the Commission has not fully evaluated and accurately reported results from
two of its teacher development programs.
Paraprofessional Teacher Training Program
3
8888 California State Auditor Report 2004-108 California State Auditor Report 2004-108 8899
Response: In Chapter 1, “The Commission Could Better Evaluate the Effectiveness of Its Efforts
and Better Measure the Performance of the Teacher Credentialing Process”, the report addresses
the California School Paraprofessional Teacher Training Program (paraprofessional program).
Two major issues are discussed. The first is in relation to the implementation of an external,
independent evaluation of the paraprofessional program. The report states that “the commission
has not contracted with an independent evaluator to determine the success of the paraprofessional
program.”
In response, it is important to clarify that steps were taken to pursue funding for an independent
evaluation but funding for this purpose has not been included in the annual budget act.
Commission staff submitted Budget Change Proposals in September, 1991 and again in
September, 1992. In June, 2000 a letter was written from the CCTC to the Budget Conference
Committee stating that the CCTC did not receive funding support for the independent evaluation
and requested $100,000.00 and two positions for administrative support. Again, funding was not
provided. Given that financial support was not provided, staff could not proceed with securing a
contractor to conduct the independent evaluation. Staff has, however, collected information from
paraprofessional programs annually and reports this data in its annual legislative report.
The second issue that surfaces in the audit report is focused on program quality and success.
Commission staff are currently working to improve the quality of program level data collection as
well as the focusing the process of the analysis of this data. Staff are working with districts who
hold paraprofessional grants to document candidates annual progress and to identify candidates
that have not made adequate progress.
Pre-Intern
Chapter 1 also examines the condition of the pre-intern program administered by CCTC staff. It
is important to clarify the purpose of the pre-intern program. The purpose of the program was
to provide funding to districts who would in turn, support candidates to increase their capacity to
provide instruction and classroom management and to assist these candidates to pass a subject
matter exam (in some single subject candidate cases, coursework). The “survival pedagogy”
and the test preparation and goal of passing the exam was in order to advance these pre-intern
candidates to an Intern program. In 2000-2001 the issue of assisting candidates advancement to an
Intern program surfaced among pre-intern program Directors. At that point in time, the Commission
added requirements to both pre-intern and Intern programs grant award process requiring pre-
intern programs to be directly linked to Intern programs in order to receive funding, thus ensuring
the intended ongoing relationship between the two programs. Specifically, program applicants were
directed to explain how they would assist candidates in progressing through teacher development
and gain preliminary credentials.
In addition in an attempt to measure the success of the pre-intern program the audit report attempts
z
to compare pre-intern candidates with emergency permit holders. Pre-interns and emergency
permit holders are not easily comparable. The pre-intern program purposefully selects candidates
who need support and test preparation. Often, emergency permit holders have already passed
their subject matter examination. In order to truly compare these two groups, much more data
and analysis is necessary than is currently provided in the audit report. The goal is to assure that
all teachers are credentialed. The pre-intern program assisted many toward reaching this goal by
providing a pathway into an Intern program.
4
8888 California State Auditor Report 2004-108 California State Auditor Report 2004-108 8899
Pages 22-23
The report implies that inconsistency and lack of support casts doubt on the validity of the data the
Commission uses in its annual reports.
Response: Notwithstanding the error noted in the specific report cited in the report, supporting
documentation is on file in PSD for prior years. As was explained to the auditors, there were
extenuating circumstances surrounding the 2002-03 report. These conditions were unique and will
x
not be repeated in the future. To call into question the validity of all annual reports based on the
error contained in a single report is misleading. We note the availability of the following supporting
documentation:
1995 – 1997- Program and participant data was included in paper project summaries which
describe the status of the program as well as participant’s academic progress. The summaries are
on file in PSD.
1997 – 2001-02 – A paper Annual Report Form that includes the evaluation elements identified
in Education Code Section 69619.1 (re-authorized as 44393) was devised to collect program and
participant data. Clarification, if necessary, was obtained by e-mail and phone. The reports are on
file in PSD.
2002-2003 – Participant and program data was collected through an electronic consent form. With
the exception of evaluation elements 4 and 5, the form includes evaluation elements identified in
Education Code Section 44393.
2003-04 – Program, participant and recent graduate data was collected through the electronic
consent form with clarification obtained by e-mail and phone. To ensure that all graduate data was
collected, programs were asked to complete a paper annual report form. The reports are on file in
PSD.
Page 23 (1st full paragraph)
The report states that “As a possible result, the commission is now trying to determine what to do
about nearly 70 individuals who are participating in local program that were scheduled to end in
December 2003 or sooner, but who have not completed credential requirements.”
Response: It should be noted that prior to the audit, the Commission had already taken steps to
determine the appropriate way to handle these cases. A letter sent on May 12, 2004 was sent to
program directors asking for explanations why the identified individuals who were participating in
local programs scheduled to end in December 2003 had not completed program requirements.
On 9/22/04 and 9/27/04, letters directed programs to request reimbursement from remaining
participants, as necessary, and a status report on funds reimbursed to date.
Page 23
The report states that the Commission has not demonstrated that the Pre-Intern Program has been
successful in accomplishing its objectives.
5
9900 California State Auditor Report 2004-108 California State Auditor Report 2004-108 9911
Response: One of the primary purposes of the Pre-Intern Program is to provide support to
individuals who would otherwise be teaching on an Emergency Permit that enables them to meet
the subject matter requirement necessary for entry into an internship or teacher preparation
program. In other words, the program helps individuals reach the first step on the road to a
preliminary credential. Prior to the establishment of the program, it was common for some
individuals serving on an Emergency Permit to receive little support or guidance in attaining subject
matter competency.
Although we understand the point made by the report, we believe that the use of Commission data
for 1998-99 and 1999-2000 does not provide an appropriate data set for analyzing the program.
z
The data collected in 1998-99 and 1999-2000 that showed the retention rate of 80% was based
on emergency permits holders after thousands of pre-interns had been removed. Most of those
permit holder who remained were, as noted on pages 10 and 12 of the report were those who had
completed most of their preparation coursework or held another credential. The pre-interns were
those who had the farthest to go and based on the 1996 study were far more likely to drop out.
That they stayed at a high rate was remarkable and was therefore noted in the legislative report.
As was pointed out in the Report to the Legislature, pre-interns as a whole had taken the required
subject matter exam at least once. Therefore, it seemed reasonable that the comparison group
should be those who had taken and failed the subject matter exam. Second time test takers have a
c
pass rate of less than 30%. These data were provided to the auditors, and have been provided to
the CCTC in agenda items.
Page 36
“In addition, the commission’s executive office does not systematically track whether the
commission is successfully completing the tasks outlined in the strategic plan. As a result of
inadequate strategic planning, the commission lacks specific goals and performance measures to
guide, evaluate, and improve its efforts.”
Response: The commission agenda items are linked to the strategic goals of the Commission.
Also, the Executive Director meets weekly with all Sr. Managers to monitor the status of all projects
as they relate to the Commission’s goals. In addition, the Executive Director annually prepares a
0
list of accomplishments that are directly linked to the strategic goals of the Commission.
Page 37
The audit report states that “One example of a task for which the Commission did not quantify the
results to be achieved, but could have, is the task to expand its teacher development programs.”
Response: For ten straight years the Commission expanded its alternative certification program.
Both the intern and pre-intern programs were designed to be per capita, market sensitive programs.
For all ten of the years of the intern program, each intern for which there was a request and a
verified intern who became a teacher of record received funding. We did not set targets or quotas
because this is a program that was designed to respond to the needs of school districts. We did
seek fiscal flexibility so that we could move funds from the intern program to the pre-intern program
as an effort to make both programs more responsive to the participating districts. Our task was to
respond to demand not set predetermined, quantified targets. The quantified target was set by the
funds available. Staff has seen its task as responding to local needs not setting quantifiable targets
as the priority.
6
9900 California State Auditor Report 2004-108 California State Auditor Report 2004-108 9911
Page 39
“In addition, the commission’s executive office does not systematically track whether the
commission is successfully completing the tasks outlined in the strategic plan.”
Response: The commission agenda items are linked to the strategic goals of the Commission.
Also, the Executive Director meets weekly with all Sr. Managers to monitor the status of all projects
0
as they relate to the Commission’s goals. In addition, the Executive Director annually prepares a
list of accomplishments that are directly linked to the strategic goals of the Commission.
Page 41
“. . . maintains the support for these statistics.”
v
Response: It would seem that agenda items would be sufficient. It does not seem reasonable that
original data would need to be kept in perpetuity or in the case of the pre-intern program for 9 years.
Page 42
“Also, the commission’s executive office should present the commissioners with an annual status
report on how the commission has achieved the goals and tasks outlined in the strategic plan.”
Response: The Executive Director annually prepares a list of accomplishments directly linked to the
0
strategic goals that are read at a Commission meeting and in the Official Minutes of the meeting.
Also, the agenda items provide a status report on the goals and tasks at each meeting.
Page 43
Chapter Summary
Response: The text in Chapter 2 is provided without the backdrop of budget cuts, staff reductions
3
and increasing workload. Without this context the findings are often misleading and sometimes
inaccurate. While many of the findings center on collecting data and analyzing it, the performance
of these activities requires resources and those resources have to be balanced against the gain
realized from analyzing the data. None of the findings or recommendations in the report provides
a cost-benefit analysis. In a time of diminished resources that type of analysis would have been
helpful in weighing the usefulness of the recommendations.
It is important to know that the certification unit has suffered a decrease in staff of 22% over
the past five years while experiencing a 6% increase in workload. The workload did drop by
approximately 6% from 2002-03 to 2003-04 which has helped to relieve some of the tension
between processing applications and providing customer service. This information is necessary to
understand that even when/if all of the recommendations are in place there may not be sufficient
staff to provide customer service and process applications. Additionally, there is no mention that
MGT of America conducted a $250,000 management study in 1999-2000 of the Commission,
in particular the certification division, and within that report is a workload study. The BSA report
b
suggests that the division does not have any workload analysis, however, an analysis does in fact
exist. It would not be cost effective to devote staff time to do another workload analysis particularly
when a new technology system is very close to implementation.
7
9922 California State Auditor Report 2004-108 California State Auditor Report 2004-108 9933
As rightly noted in the report, the certification unit can work to provide information on the web in a
user-friendlier format and it is working with the Commission’s technology staff to redesign the web
site with a target unveiling of January 3, 2005.
b
The certification unit is fully aware of its processing workload and uses a biweekly report to make
staff staffing decisions. The BSA report states that the certification unit is not aware of the types of
applications that are pending. The report is accurate, however it does not state that the application
process does not allow for this type of data because the type of application is not known until it
is processed by the certification staff. To change that would require a complete redesign of the
processing system, yet there is no indication in the report with regard to the benefits, if any, of
knowing this information. The report acknowledges that the processing time has been within the
regulatory limit and within the timeframe that the past administration deemed appropriate per a veto
n
message in the 2000-01 budget. There is no mention of that veto message in this report, yet it was
provided to the auditors.
m
The report also mischaracterizes the Teacher Credentialing Service Improvement Project (TCSIP)
and takes out of context the comments of the certification director. The report does not provide
enough detail about the project to provide the necessary context of the project. While the report
,
states that TCSIP is a three-phase project, it does not state that the efficiencies anticipated from
the project at the time the project was initiated have already been mostly realized in the first two
phases, online credential lookup and application status and online credential renewals. The report
accurately addresses in particular the efficiencies of the online renewal process. However, the
.
report does not state that the third phase of the project has not been implemented and is still in
the testing phase and at the time the auditors were conducting their research the project was
not ready for testing. Within that context, the certification director could not make comments
about the possible efficiencies of phase three because the system had not been tested. The
director did mention in those fact finding sessions that there would be efficiencies realized from
the college and university online credential recommendation submission process, that the edit
process would reduce errors that create double work and the efficiencies of the virtual credentialing
/
officer, comments that are not included in this report. Also the report characterizes the division
as not planning to fully utilize the automated process and that we chose not to move forward with
it. As with any project, it is important to have one phase functioning prior to starting a second
or third phase and with any new technology system it takes from six months to a year to have it
fully operational. Also without the context of budget constraints and reduced staff it would not be
6
a sound management decision to enhance a system before it is even operational or is it sound
management decision to start an enhancement when there are no financial resources to fund the
project. Once the systems are fully operational and funding is available, of course the certification
division will want to fully utilize all automation that can be provided so it is misleading to say the
division does not plan to use the full potential in the foreseeable future given the ongoing fiscal
restraints. It is important to note that the Commission suggested that the auditors consider deferral
of recommendations relative to the TCSIP until after the final phase of the project had been
completed, as any assumptions based on current practice or conjecture would be premature.
Lastly, the Chapter 2 summary states that the certification division could better inform teachers
!
about the online credential renewal process. However there is no acknowledgement in this report
that when the online process first became available the certification division in cooperation with
California State University, Chico, developed a color leaflet explaining the process. Those leaflets
8
9922 California State Auditor Report 2004-108 California State Auditor Report 2004-108 9933
were sent to every county office of education to include with teachers payroll checks. Can we do
more? Yes, we can and the redesigned web page will emphasize the online credential renewal
option.
Page 44
Response: The third sentence under this heading includes information that is somewhat of a
surprise since the auditors informed us that our publications and forms did not need changing and
that they thoroughly explained the credential or permit requirements. In fact all of the information
leaflets were redesigned through a contract with California State University, Chico following a
recommendation of the MGT of America management study of the Commission in 1999-2000.
Page 44
Response: The same recommendation was made in the MGT of America Management study, yet
there is no reference to the fact that in the MGT study the certification unit devoted 31% of its time
processing credentials, and since that it has decreased to 39% and at the same time there has
been a 6% increase in the number of applications and a 22% reduction in staff.
Page 45 (2nd Paragraph)
Response: The content of the paragraph states that the certification division reduced customer
4
service operations, yet it does not state that there was an increase in credential applications and a
reduction in staff. Without that data, it appears that this was an arbitrary decision not based in any
data analysis of which there was.
Page 47 (2nd Paragraph)
At the September/October 2004 Commission meeting, the certification division established a
performance goal of reducing phone calls and e-mails by 5%.
Page 49
Response: The Commission is actively redesigning its web site with a target date of January 3,
2005. When the web site is unveiled, users will find application processing data that is up-to-date
and the online renewal process will be featured with an explanation of the advantages to online
renewal.
Page 50
Response: The Commission strongly disagrees with the finding that the division does not gather
b
or analyze data on the time it takes to process credentials. The certification unit staff prepares
a report twice a week that provides the processing time of all applications. In addition there is
a weekly report that identifies every pending application and which team it has been assigned.
Nowhere in the report does it address the ebb and flow of applications. The report states that
generally the division meets the regulatory 75-working day processing time, however it does not
state that during the months of July through October the Commission receives 38% of its workload
and it is processing applications received during this period that may exceed the 75-day limit.
9
9944 California State Auditor Report 2004-108 California State Auditor Report 2004-108 9955
Page 54
Response: The Commission strongly disagrees with the statement that the division does not
b
perform sufficient data analysis to effectively manage its application processing workload. After this
statement the report proceeds to outline all of the data reports and analysis the division performs
on its workload. The general recommendation is to monitor the type of applications that are
pending. The report does not state that the current or future process does not provide this type of
data nor does this report provide any analysis why that data would be helpful. In order to capture
that data, a reclassification of the cashiering staff and a complete change in the process would be
necessary. While it is possible to do this there is no analysis in the report that shows the costly
reengineering of the process would reduce the processing time.
The report states that the certification division has not conducted time studies on the effort
needed to process different types of applications. The division has not done that because MGT of
America conducted a management study in 1999-2000 that included a time study for processing
applications. Why, after an investment of $250,000 would the certification division replicate the
@
MGT time study? Also, why would the division complete a time study when a new technology
system is to be implemented?
In the third sentence in the paragraph at the top of page 56, there is a statement that the division
had the opportunity to include an automated workload report when TCSIP was designed. That is
not an accurate statement. Without the proper context of the TCSIP project and the fact that almost
#
all of the current reports were considered out-of-scope, the division did not have a choice as stated
in the report to request this report. We do hope to develop it once the system is operational.
Page 58
m
Response: The findings in this section are misleading and the comments of the certification
.
director are taken out of context. Nowhere in this report does it state that phase three of TCSIP is
has not yet been implemented and during the gathering of the data for this report the system was in
the very early stages of testing. In fact staff informed the auditors at the beginning of the audit that
the Commission is in the process of changing data systems, however the audit was not delayed.
The comments the certification director made were in reference to phase three of the project, not
the complete project, and at the time the system had not been tested at that time so the efficiencies
were still unknown.
Also not included in this report was the statement by the Certification Director, that there would
/
be benefits from college and universities submitting credential applications online, that there was
a enhanced edit process that should eliminate having to redo processing errors and the virtual
credentialing officer. The comments about the new system centered on the collection of data that
was significantly different than the current system. As the report accurately states, much of the
efficiencies have already been realized by the first two phases of the project, the third phase is
the data collection phase where there are efficiencies but not as great as those when the first two
phases were was implemented.
Page 62
Response: The Virtual Credentialing Officer (VCO) is one of the best aspects of TCSIP when it
comes to efficiency and reducing staff time to process the online renewals. At the time the data for
this report was collected, the VCO had not been fully designed nor had it been tested. At the time
the auditors met with the certification director, he stated that the VCO needed to be operational
10
9944 California State Auditor Report 2004-108 California State Auditor Report 2004-108 9955
before any further enhancements could be planned. The report accuses the certification division
of not soliciting a quote to enhance the VCO, however why would we solicit a quote when we do
6
not have any financial resources to pay for it. The report does not include any statements about
the lack of financial recourses available to the Commission and that it would be unwise to take this
action without funding.
Page 64
1
Response: The heading is very misleading. The VCO was developed prior to the arrival of the audit
team and this heading makes it sound like this is a new idea. The VCO was included in the project
to streamline the online renewals.
$
Also the last sentence under this heading states that the Commission is going to print a copy of the
application, as if that was an inefficient process. The Commission required by law to maintain a
copy of the application and keep if for a period of the lifetime of the credential holder.
Page 65
%
Response: The text in this section does not reflect the heading. The certification staff does not
need data to increase the users of the online renewal process. We concur with the report that the
web site can and will be enhanced to highlight the online credential renewal process. As stated in
the report a flyer is included in every 30-day substitute permit outlining the online renewal process.
However, data is not needed to increase the numbers of users.
The text under this section states that data is needed to determine success of the process. There
are different methods to determine success. This report would like the division to determine
the total number of renewals, which of those renewals are offered online and then determine
the percentage who are using the online process. By setting a percentage we could determine
success. As the auditors know through discussions, because of the database design from 15 years
ago, it is almost impossible for us to determine the total number of renewals and how many can be
renewed on the web.
Another measure of success, which is just as valuable, is the number of teachers using the online
process and how it has continued to increase. Figure 8 in the report demonstrates that the online
process is a success. There has been over a 600% increase in the use of the online process from
July 2002 to April 2004. By any measure, a 600% increase would be determined a success.
Response: The Commission expects to develop an automated report once TCSIP is implemented.
Page 70
The audit report notes that the Commission should revamp its process for developing program
standards. And also notes that “In addition, the commission does not have an overall plan to
guide its efforts to finish implementing some program standards required by the act and to guide
its ongoing standard-setting activities. Planning would help the commission identify the steps
necessary to periodically update program standards and the resources it plans to use.” Finally, the
report says “Our review of five recently developed program standards…”
11
9966 California State Auditor Report 2004-108 California State Auditor Report 2004-108 9977
^
Response: Throughout the document, the writers seemed to mix two different topics and treat
them as one. One topic is the implementation of SB 2042 and the other is the general process that
the Commission uses to develop standards. Both topics certainly are in need of review, but the
presentation gave a misleading impression.
&
The Commission adopted a plan to implement the SB 2042 standards developed by the Advisory
Committee. It is attached.
*
To clarify, it was not five standards, but five sets of standards.
Page 71
Implementation of SB 2042
Response: The report made a number of references to the Commission’s implementation of the
legislative mandates of Senate Bill 2042. Numerous times in the report the comment was made
that the Commission did not have a plan that included timelines for developing and implementing
&
the standards. That particular charge is hard to understand, given that there was a plan and the
reform has taken place according to the plan.
The chart developed by the audit team clearly shows the results of the plan. Shortly after the
passage of SB 2042 in 1998, the Commission began the process of developing three inter-related
sets of standards of program quality and effectiveness for Elementary Subject Matter Preparation
Programs, Professional Teacher Preparation Programs and Professional Teacher Induction
Programs. In addition, the Commission adopted a set of Assessment Quality Standards to define
the Teaching Performance Assessment. All of these standards were adopted in the 2001-2002
fiscal year. Taken together, they represent standards that create a learning-to-teach continuum
that was a significant reform in teacher preparation. At the time of the adoption of each of the
sets of standards, the Commission also adopted an implementation plan/schedule. Following the
adoption of these sets of standards, the Commission began a 3-phase project for development
and implementation of new standards for each of the 13 single subject areas. Again, when the
standards are adopted, the Commission also adopts an implementation plan. The first set of
standards were adopted and are being implemented. The second set of standards were adopted
and are just beginning the implementation phase. The third group of standards are about half
way through the development phase. Taken as a whole, this represents a massive reform in the
teacher preparation system. The total length of the entire process from the very beginning to the
final implementation of the last group of standards will have been about nine years. However,
each individual group of standards have been implemented in a two year time period after they are
adopted.
The only implementation problem in the standards is that the Secretary for Education and the chair
of the Senate Appropriations Committee requested the Commission to delay the implementation
of the Teaching Performance Assessment, due to the state fiscal crisis. The state fiscal crisis
continues, as does the delay in implementation of the TPA. The Commission will reconsider the
implementation of the TPA in the spring of 2005. It is significant to note that since the beginning
of the SB 2042 standards development process in 1998, the Commission has faced some
e
very serious financial constraints that have required some adjustments in how the work was
accomplished and how it was resourced. However, the project has gone forward, and is now
nearing completion.
12
9966 California State Auditor Report 2004-108 California State Auditor Report 2004-108 9977
Page 74
The audit report states that the Commission developed program standards in a reasonable order,
but it should develop an overall plan to finish implementing the act’s reforms.
Response: The report made a number of references to the Commission’s implementation of the
legislative mandates of Senate Bill 2042. Numerous times in the report the comment was made
that the Commission did not have a plan that included timelines for developing and implementing
&
the standards. That particular charge is hard to understand, given that there was a plan and the
reform has taken place according to the plan.
The chart developed by the audit team clearly shows the results of the plan. Shortly after the
passage of SB 2042 in 1998, the Commission began the process of developing three inter-related
sets of standards of program quality and effectiveness for Elementary Subject Matter Preparation
Programs, Professional Teacher Preparation Programs and Professional Teacher Induction
Programs. In addition, the Commission adopted a set of Assessment Quality Standards to define
the Teaching Performance Assessment. All of these standards were adopted in the 2001-2002
fiscal year. Taken together, they represent standards that create a learning-to-teach continuum
that was a significant reform in teacher preparation. At the time of the adoption of each of the
sets of standards, the Commission also adopted an implementation plan/schedule. Following the
adoption of these sets of standards, the Commission began a 3-phase project for development
and implementation of new standards for each of the 13 single subject areas. Again, when the
standards are adopted, the Commission also adopts an implementation plan. The first set of
standards were adopted and are being implemented. The second set of standards were adopted
and are just beginning the implementation phase. The third group of standards are about half
way through the development phase. Taken as a whole, this represents a massive reform in the
teacher preparation system. The total length of the entire process from the very beginning to the
final implementation of the last group of standards will have been about nine years. However,
each individual group of standards have been implemented in a two year time period after they are
adopted.
The only implementation problem in the standards was the request to delay the implementation
of the Teaching Performance Assessment, due to the state fiscal crisis. The state fiscal crisis
continues, as does the delay in implementation of the TPA. The Commission will reconsider the
implementation of the TPA in the spring of 2005. It is significant to note that since the beginning
of the SB 2042 standards development process in 1998, the Commission has faced some
very serious financial constraints that have required some adjustments in how the work was
e
accomplished and how it was resourced. However, the project has gone forward, and is now
nearing completion.
Page 74 – 75
The report states that: “The order in which the commission chose to develop the subject standards
appears reasonable...” and “An overall plan would also guide the commission’s efforts...program
standards and examinations should be reviewed periodically...”
Response: The single subject standards development followed the order and coincided with the
timetable already established for the regular orderly updating of single subject exams (and the
program standards to which they relate). The schedule was adopted by the Commission in 1999
(agenda items of Jan. 20 and June 23, 1999).
13
9988 California State Auditor Report 2004-108 California State Auditor Report 2004-108 9999
Page 75
“If the Commissioners adopt a two-year implementation period, as they have previously, colleges
and universities will not finish implementing these program standards until mid-2007, nearly nine
years after the act became law.”
Response: AB 1307(Goldberg) passed in October 2001 requires that changes in education
regulations allow stakeholders (candidates) two years to make the transition, upholding old
regulations during that period. The Commission cannot require the “implementation” of new
program requirements of candidates in less than two years. Commission staff has stood ready
to conduct program reviews since the adoption of each set of standards. In addition, submission
for program approval is not a required of colleges and universities who may submit at will but is
an institutional option. CTC staff currently has dozens of single subject programs under review
and dozens more scheduled for review over the next year. These reviews are rigorous and time-
intensive. The chief obstacle to completing those reviews at this time is a lack of funds to train and
coordinate expert field reviewers.
Page 75
“Although the commission developed an approach to develop standards required by the act, it
did not use an overall plan that included timelines for developing standards and the resources it
planned to use.”
Response: This appears to confuse the timeline and plan for the implementation of SB 2042 with
&
the timeline and plan for test development. As we have noted, there was a detailed timeline for
SB 2042 standard development with regard to test development. CTC released a Request for
Proposals (RFP) developing single subject exams and standards that required bidders to submit a
detailed plan structure and resources for the project. The response to that RFP from NES, Inc. (as
described in the audit report) described specifically how and when they would accomplish the CTC
plan within the contract budget. That project is near completion on schedule within the budget and
requirements of the awarded contract.
Page 75 (2nd paragraph)
Response: The paragraph indicates that there was not a plan to develop the standards. The
&
Commission had a plan and the standards were developed and adopted along with an
implementation plan. The TPA was part of the implementation plan that was delayed after adoption
because of the state fiscal crisis.
Development of Program Standards
Page 77
“Currently, the Commission informally evaluates the lessons learned from developing standards.”
Response: CTC is required by law (Ed Code Section 44225 i and 44288) to conduct standards
development in the manner that it does, using advisory panels of experts. It further validates the
findings of those experts with the same processes (field reviews, bias reviews, alignment and
congruence reviews) that it used for the last development of standards in the late 1980’s. All
(
standards in the 2042 reform were developed using the same process for consistency. The fact that
this process is encoded in law suggests that it is formal.
14
9988 California State Auditor Report 2004-108 California State Auditor Report 2004-108 9999
Page 78
r
Response: The report states that the Commission does not use a methodical approach to form
advisory panels. A “checklist” model could make the standard-setting process more challenging.
The selection of panel members does require a review of prospective panel member qualifications
and a ranking. However, the ranking is into general categories, rather than a straight serial
ranking. From that point, the Commission must consider a number of other balancing factors that
will affect panel member selection. They are, in no particular order, size of the panel, task of the
panel, size of the pool of applicants, employment of applicants, constituency group represented,
)
ethnicity, geographical distribution, affiliation with K-12 education or institutions of higher education,
particular expertise, availability, etc. Once these factors come into the process, it is not possible to
Q
have a process that “panel members are selected objectively.”
Further there is considerable variation in the task that various panels are given that will greatly
affect how panel members are selected. The Commission employs multiple strategies to utilize
advice from experts in the field in the development of its policies. This includes formal advisory
panels, appointed task forces, focused work groups, and design teams.
Page 78, (1st paragraph, 2nd sentence)
“…documents created during the standard setting process.”
W
Should be “…documents created during the program standards field review process.”
Page 79
“Lacking a more structured application to explicitly solicit how candidates met the qualifications,
the commission may not have received sufficient information to be able to evaluate candidates’
qualifications.”
E
Response: The audit report seems to suggest that the Commission should rank the qualifications
of applicants. However, the Education Codes cited above do not place, for instance, administrators
higher than teachers or higher education faculty higher than K-12 faculty. Further, no representative
R
from the Commission stated that one set of standards is more “important” than any other. The
r
Commission staff uses a methodical approach to the appointment of advisory panels. The type of
work to be done and balancing factors relative to staff, stakeholders, agencies, and political context
are considered when evaluating applicant qualifications. The Commission staff strives to assemble
panels that are balanced, not only in terms of the individual qualifications of each member, but also
relative to geography, gender, race and ethnicity, urban/rural, size of institution or local education
)
agency, and constituent representation. While such an approach does not lend itself to a “checklist”
type of evaluation of applicants relative to qualifications and standards adopted by the Commission.
This has resulted in consensus by the field with regard to both the process and product. Using
this process, the Commission complies with the law. The auditors were provided with the selection
chart and a description of the process.
Page 81
“For the fifth standard we reviewed – induction programs- the commission used a different process
when forming the advisory panel. It did not solicit nominations, nor did it require interested
candidates to submit an application. Instead, the commission indicated that it selected panel
members who were active....”
15
110000 California State Auditor Report 2004-108 California State Auditor Report 2004-108 110011
Response: The induction standards were developed by specialists from induction programs and
were under the overall guidance of the larger SB 2042 panel. The management of induction
programs was a shared function between the Commission and the Department of Education, as
required in the Education Code.
The SB 2042 Advisory Panel was already charged with the development of Induction program
standards, and thus a new panel was not required to be selected. A subcommittee of existing
SB 2042 Advisory Panel members was established and then augmented by individuals from the
BTSA community who, by virtue of their background and experiences, could provide specialized
background knowledge to the current SB 2042 advisory panel members on the subcommittee.
Page 82
The report states that “The Commission could do more to report its survey results and consider the
view of minority groups.”
Response: The report confuses the methods used for various surveys without regard to the
distinction of the purpose of the survey leading to a serious charge that the Commission does
not consider the views of minority groups. The content validity study is conducted in an objective
and statistical manner for a particular reason. The bias review study is charged to ensure that
standards and examination items are free from biased language or content. The field survey of the
T
draft standards is to gain further comments for experts in the field about the work that has been
proposed for adoption by the Commission at a public meeting.
Page 82
“For example, when commission staff proposed the four core subject area standards to the
commission for adoption, it reported…”
Y
Response: Staff reported to auditors that some of the surveys submitted represented the response
of many more than one individual, such as a whole department. This was also noted on the
surveys and confirmed with the academic departments by Staff. In fact one of the Commissioners
who works for the CSU system assisted staff with distributing those surveys to the CSU campuses
and can confirm these facts. This explains the difference in the numbers of surveys versus the
numbers of responses reported to the panels and the Commissioners.
Page 82
“Also the Commission is missing an opportunity to further ensure that its subject matter
requirements are free from biased language and content because its process excludes the views of
some groups.”
Response: Although the standards field surveys are “color-blind,” the subject matter requirements,
upon which the standards and exams are based, are subjected to a separate bias review by a
committee made up of individuals from underrepresented groups. In addition, the content validity
study is submitted to a select sampling of individuals also from underrepresented groups to insure
their inclusion in the process. Though items are flagged by numbers of responses, the panels were
also presented with a complete set of data to consider, including minority responses. Auditors were
given this information from both contractors for single subjects. Finally, the standards field surveys
are sent to every education institution across the state, so without collecting specific data on the
16
110000 California State Auditor Report 2004-108 California State Auditor Report 2004-108 110011
source of the responses, every group had equal opportunity to have their responses considered
T
equally with everyone else. Using these different methods insures fair and equitable participation of
all interested groups.
Page 83
“However, the Commission did not always adequately document how it ensured that the standards
were aligned.”
Response: The auditors were told how the Commission ensures alignment to the K-12 standards as
follows:
1. The contractor drafts an alignment.
2. The Commission invites the curriculum consultants from CDE to participate in the panels’
work to insure alignment.
3. The panels compare the present subject matter standards to the K-12 standards.
4. The panels compare the contractor’s alignment to the K-12 standards.
5. The alignment and congruence panel compares the subject matter panel’s alignment.
6. The field review asks responders familiar with the K-12 standards whether the draft subject
matter requirements and standards are what beginning teachers need to be able to know and
teach.
7. The correspondences with the K-12 standards are noted in the subject matter requirements.
This same process was followed for developing all 13 areas of single subject standards. In effect
a system of checks and balances was set up to insure no major areas of the k-12 standards were
left out of the subject matter standards. The audit reports states that the Commission was not able
to provide the results of the independent review. However, those documents are available from
the contractor and staff upon request in both the contractor’s alignment and the alignment and
U
congruence committee report to the contractor. The auditors did not ask for this information.
Page 84
Commission’s “ . . efforts to revise it policies have stalled . . .”
Response: The Commission has worked continuously during the past year to review its
accreditation policies through the Committee on Accreditation and a workgroup of stakeholders.
These groups have met regularly and are on pace with the established timeline for this project.
Activities in the past 10 months have increased with the Commission assigning its Committee
on Accreditation to oversee the review and utilize an Accreditation Study Work Group to assist in
bringing recommendations to the Commission next spring. There is a report on the progress of the
I
accreditation review on the agenda of each Commission meeting. Rather than stalling, the efforts
have actually increased.
Resumption of Continuing Accreditation Visits
17
110022 California State Auditor Report 2004-108 California State Auditor Report 2004-108 110033
Page 84
Response: Throughout the report, there is an inconsistent reporting of the date that the
Commission suspended its continuing accreditation site visits. On p. 84 the date listed is
O
December 2002. Earlier in the chapter (p.71.) the date is listed as fiscal year 2001-2002. The
correct date is December 2002 (fiscal year 2002-2003).
Page 84
“However, suspending continuing accreditation limits the commission’s ability to ensure that
colleges and universities....are operating programs that meet the commission’s eight common
standards and specific teacher preparation program standards.”
Response: All of California’s teacher preparation programs had to respond to the Preconditions,
Common Standards and Program Standards established pursuant to SB 2042. Therefore, all of
California’s teacher preparation programs have been reviewed and granted initial accreditation
8
within the last two years, and the majority within the last year. The Commission can be reasonably
assured, therefore, that these programs are presently meeting Commission standards even in the
absence of site visits during 2003-04.
Page 86
The report states that the Commission believes that its accreditation policy is outdated.
p
Response: It should be noted that the Commission has never stated that. The purpose of the
accreditation review is for the Commission to gain enough information so that it can make an
informed decision about its accreditation policies. At the conclusion of the review, the Commission
may make that determination, but it has not been made yet.
Page 86
Response: The report indicates that the Commission did not ask the contractor to recommend
alternatives to the present accreditation system. In fact, the Commission did not formally ask the
contractor for the Accreditation Framework review to recommend alternatives. By statute, the
Commission was required to conduct a review of the Framework (its policies) using an external
contractor. The review called for was about the Commission’s policies and their implementation.
The statute required that the Commission could not make substantive changes in its policies
until the Framework study was completed. The study was completed in Spring 2003. Once the
study was completed, the Committee on Accreditation and Commission began its review of the
findings, eventually leading to the decision by the Commission to authorize and Accreditation Study
Workgroup.
About one and one-half years before the completion of the study, Commission staff asked the
contractor to recommend alternatives to the present system. All parties recognized that this was
not in the “scope of work” of the contract and a revision of the contract would be necessary, if the
scope of work were to be expanded. The contractor did, however, agree to voluntarily provide
some background information about other accreditation systems without a formal change in the
contract. Using the background of the results of the independent study on the implementation of
the Commission’s accreditation policies, the Accreditation Study Work Group and the Committee on
Accreditation are reviewing alternatives to the present system as part of its charge.
18
110022 California State Auditor Report 2004-108 California State Auditor Report 2004-108 110033
ATTACHMENT
INTRODUCTION TO THE RESPONSE TO SB2042 IMPLEMENTATION ISSUES
FROM THE AUDITORS’ REPORT
SB 2042 Implementation plans and timelines
There were two primary SB 2042 implementation plans. The first plan covered the work of the
&
SB 2042 Advisory Panel as this group worked on the development of program standards for teacher
preparation programs, elementary subject matter programs, and induction programs. The SB 2042
Advisory Panel provided regular updates to the Commission as to the progress being made in
accomplishing the charge to the Panel to develop these three sets of program standards. The larger
panel worked on the development of the teacher preparation and the elementary subject matter
preparation program standards; a subcommittee of the SB 2042 panel worked on the development
of the Induction program standards in order to take advantage of the specialized knowledge of
representatives from the BTSA community. The regular updates to the Commission included plans
and timelines for accomplishing the charge. See the attached timeline of Commission updates and
the supporting documentation referenced within the attached timeline.
The second SB 2042 implementation plan was adopted by the Commission in September 2001,
at the same time as the Commission adopted the professional teacher preparation, elementary
subject matter, and induction program standards completed by the SB 2042 Advisory Panel. This
plan governed the implementation in the field for the three sets of program standards adopted by
the Commission (see supporting document “I” for the SB 2042 standards field implementation plan).
Regular updates were subsequently provided to the Commission as to the progress in carrying out
the adopted SB 2042 standards field implementation plan. See attached timeline of Commission
updates and supporting documentation referenced with the attached timeline.
19
110044 California State Auditor Report 2004-108 California State Auditor Report 2004-108 110055
2042 Implementation Plan Chronology
DATE EVENT REFERENCE DOC.
1997 SB 1422 Advisory Panel makes final A
recommendations for changes to the
requirements for earning and renewing
teaching credentials
Sept. 17, 1998 SB 2042 signed A
Sept 24-25, 1998 First meeting of Commission- appointed new A
Advisory Panel to develop teacher preparation
standards. Panel charge based on SB 2042
legislation includes teacher preparation,
elementary subject matter preparation,
induction and teaching performance
assessment standards development.
March 1999 Progress report provided to the Commission A
on SB 2042 Advisory Committee activities.
Plan approved for 1999 timeline for the
SB 2042 Advisory Committee’s work,
including teacher preparation, elementary
subject matter preparation, and teaching
performance assessment standards
development. Single subject matter content
also reviewed with a later timeline for single
subject matter standards development.
May 1999 Progress report provided to the Commission B
on SB 2042 Advisory Committee activities.
Updated timeline presented for
Commission review.
July 1999 The Commission approved a plan and C
timeline for conducting single subject
matter test validity studies for single
subject matter tests. This process routinely
takes place every five years in order to keep
subject matter examinations current with
student content standards.
September 1999 Progress report provided to the Commission D
on SB 2042 Advisory Committee activities.
Updated timeline information presented for
Commission review.
December 1999 Progress report provided to the Commission D
on SB 2042 Advisory Committee activities.
Updated timeline information presented
for Commission review. Joint discussions
held regarding induction (Level II) with BTSA
Leadership Team.
110044 California State Auditor Report 2004-108 California State Auditor Report 2004-108 110055
February 2000 The Commission approved a plan and C
timeline for revised single subject matter
test content specifications for single
subject matter tests. This process would
include development of new single subject
matter standards pursuant to SB 2042,
in a parallel process to the other program
standards being developed by the Advisory
Committee.
June 2000 Progress report provided to the P
Commission on SB 2042 Advisory
Committee activities, including next steps
in the implementation process.
September 2000 Progress report provided to the Q
Commission on SB 2042 Advisory
Committee activities, including updated
timelines of implementation activities.
November 2000 Progress report provided to the Commission F
on SB 2042 Advisory Committee activities,
including development of program standards
and induction standards. The Commission
reviewed the initial plan for the field review
of the new draft standards.
December 2000 Progress report provided to the Commission G
on SB 2042 Advisory Committee activities.
Further information on the plan for field
review of the new draft standards provided.
Progress report provided to the Commission H
on SB 2042 Advisory Committee activities.
Results of the field review were provided
January 2001 Implementation plan for field review of R
teacher preparation and subject matter
preparation standards presented to the
Commission.
April 2001 Implementation plan for single subject
matter examinations and program standards
development approved by the Commission
September 2001 New program standards adopted by the I, J
Commission. Implementation plan for new
elementary subject matter preparation,
teacher preparation, and induction program
standards approved by the Commission.
The plan includes activities and dates. Plan
activities are funded in part by a federal HEA
Title II Teacher Quality Enhancement State
Grant.
110066 California State Auditor Report 2004-108 California State Auditor Report 2004-108 110077
September 2001 Implementation Plan Drafted for O
Transitioning BTSA programs to Induction
programs
February 2002 Update on progress in implementing the K
SB 2042 Implementation Plan provided to
the Commission’s Executive Director
March 2002 Update on the development of single T
subject matter program standards and
examinations provided to the Commission
April 2002 Updated progress report on the SB 2042 L
Implementation Plan provided to the
Commission
November 2002 Updated progress report on the SB M
2042 Implementation Plan provided
to the Commission. Updates includes
dates, locations, and attendees at various
implementation activities.
May 2004 Updated progress report on the SB N
2042 implementation presented to the
Commission. The report includes next steps
to complete the implementation process
for teacher preparation program standards,
induction standards, and elementary subject
matter standards.
110066 California State Auditor Report 2004-108 California State Auditor Report 2004-108 110077
Blank page inserted for reproduction purposes only.
110088 California State Auditor Report 2004-108 California State Auditor Report 2004-108 110099
COMMENTS
California State Auditor’s Comments
on the Response From the California
Commission on Teacher Credentialing
To provide clarity and perspective, we are commenting
on the response to our audit report from the California
Commission on Teacher Credentialing (commission). The
numbers below correspond with the numbers we have placed in
the margin of the commission’s response.
1
As part of our quality control process, our standard practice is to
provide agencies five working days—an agency review period—
to review and comment on a draft copy of the report. During
this time, we encourage agencies to discuss with us any concerns
with the report, including any factual issues or word choices
they may identify. In keeping with this practice, we provided
copies of the draft report for commission staff to read at the exit
conference that occurred on October 4, 2004. Further, during
the agency response period, we contacted the commission’s
audit liaison and other staff on at least four occasions and
offered to meet or discuss any concerns that the commission
may have had; yet the commission rejected our offers. The
commission’s audit liaison did provide us with suggested title
and heading changes—most of which we did not use because
the evidence provided to us during the audit did not support
the commission’s suggested changes. The audit liaison also
provided us suggested changes to the text box on page 7 in the
Introduction as we explain in note 23.
2
The commission is correct that the audit request did not ask for
a workload study. As we describe in the Scope and Methodology
section beginning on page 10 of the report, the Joint Legislative
Audit Committee asked us to study the effectiveness and
efficiency of the credentialing process. One of the areas we
studied was how the commission manages its workload. Because
a significant workload of applications and renewals exists, we
reviewed whether the commission has taken appropriate steps
to evaluate its workload to determine how it can make the
process more efficient. Our conclusion as presented in Chapter 2
is that there are additional improvements the commission can
implement to make its credentialing process more efficient.
110088 California State Auditor Report 2004-108 California State Auditor Report 2004-108 110099
3
The State’s fiscal condition has challenged all state agencies to
operate their programs with fewer resources. Our audit approach
was premised on the idea that the commission is not likely
to receive significant increases in funding. Thus, most of our
recommendations are focused on improvements the commission
can make with its existing staff and funding.
4
The commission may have overlooked our discussion on
pages 36 through 38 where we describe the workload volumes
the commission has experienced and how the commission
modified its practices to respond to them. Our recommendations
focus on additional ways that the commission can use existing
staff time more efficiently.
5
If the commission believes a change in statute is necessary to
implement certain recommendations, it has the ability to initiate
the process of asking the Legislature to consider the change.
6
Although we agree that our recommendations related to
enhancements of its automated application processing
system are not ones that can be immediately implemented,
we believe the commission should thoughtfully consider
these enhancements. As noted on page 51 of the report, the
commission has not taken the step of assessing the feasibility
or costs of these enhancements. This would be a step to be
undertaken before any funding requests could be made.
7
We disagree that developing an analysis of factors affecting
the numbers of credentials, permits, and waivers that the
commission grants would necessitate an increase in staffing
of the commission. Much of the information is readily
available, such as unemployment data published by the
Employment Development Department, or the issues are
ones the commission staff know about. For example, when
the commission issued its fiscal year 2002–03 teacher supply
report in May 2004, several of the commissioners asked why
the application numbers were higher and the commission staff
noted the effects of the State’s incentives to create an interest in
teaching as a career and the expiration of a fee waiver incentive
in 2002.
8
The commission mischaracterizes our report. We state on
pages 1, 3, 12, 60, and 71 of the report, that as of December 2002
the commission suspended its continuing accreditation reviews,
except for those few who are seeking accreditation through a
national organization. Since December 2002, the commission’s
111100 California State Auditor Report 2004-108 California State Auditor Report 2004-108 111111
efforts have been primarily initial accreditation reviews, which
are desk reviews of teacher preparation program proposals
that colleges and universities submitted in response to the
commission’s recently adopted standards. However, the colleges
and universities are continuing to operate teacher preparation
programs under the previous standards and these programs are
not subject to any continuing accreditation review.
9
The commission mischaracterizes our recommendation. As
we discuss on page 26 of the report, the commission has
already begun coordinating with the California Department of
Education (Education) to create an integrated data collection
system to meet the reporting requirements of the federal
No Child Left Behind Act of 2001. We also discuss in that
section some of the analyses of teacher retention that are
possible when the database is fully functional.
0
The commission overstates the breadth and scope of the list of
accomplishments that the executive director presents to the
commission’s governing body (commissioners). As we point out
on page 30 of the report, this list of accomplishments does not
track the progress of strategic plan tasks, which are the means by
which the commission works toward its goals.
q
This is an issue that we would have expected the commission
to let us know about during the agency review period. We have
reassessed our analysis based on the commission’s response
and have modified the text of the report on pages 46 and
47 to remove references to an analysis of applications in the
commission’s workload by credential type. However, this text
change did not affect the wording of our recommendation.
w
If the commission believes a change in regulation is necessary
to implement this recommendation, it has the ability to initiate
the change. Moreover, we recognize that not all applications
may be able to be submitted electronically, which is why we
phrased our recommendation to implement this to the extent
that is economically feasible.
e
The commission is incorrect that it is finished developing and
implementing the various program standards. Figure 9 on page
63 shows the status of the commission’s work and that five sets
of single-subject standards are still in the development stage.
Moreover, during the implementation phase, the commission
continues to have an active role in performing the initial
accreditation review of proposals submitted by the colleges and
111100 California State Auditor Report 2004-108 California State Auditor Report 2004-108 111111
universities. Thus, we stand by our recommendation that the
commission needs an overall plan, which includes timelines
and the resources it plans to use, to guide its efforts to finish
implementing the standards.
r
We disagree with the commission’s assertion that its approach is
methodical. We describe the lack of structure in the commission’s
current process on page 67 of the report. Moreover, we disagree
with the commission’s statement in its response on page 100,
where it asserts that because of all the balancing factors it considers
in making panel member selections, “it is not possible to have a
process that panel members are selected objectively.”
t
The commission misses the point. Even though the
commissioners have not expressed concerns with the lack of
perspective that commission staff provide, the commission staff
are not presenting the field survey results in the most factual
and straightforward manner. Consequently, the commissioners
and stakeholders would not know if there were any problems.
y
The commission’s actions contradict its assertion that it
regularly updates its strategic plan. As we note on page 28 of
the report, the February 2001 strategic plan (2001 plan) was the
most current plan at the time of our audit and was outdated.
We also acknowledge that the commission updated the tasks
in the plan in September 2004—three and a half years after it
prepared the 2001 plan. In its guidelines for strategic planning,
the Department of Finance indicates that one characteristic
of a successful strategic planning process is that both the plan
and the planning process are reviewed and modified regularly
(usually annually).
u
The commission’s designation of this document as confidential
is not supported by any legal authority of which we are aware.
On October 21, 2004, we notified the executive director of the
commission that we planned to publish both of the commission’s
responses. Consequently, we have included this document in its
entirety and have performed the same review and analysis of the
commission’s comments as we perform on any audit response.
Furthermore, the page numbers that the commission uses in this
document are for a draft copy of our report and thus do not match
the page numbers in the final version of our report.
i
We disagree with the commission’s statement that the report
causes confusion resulting in “erroneous and misleading
conclusions and recommendations.” Senate Bill 2042,
Chapter 548, Statutes of 1998 (act), mandated a comprehensive
111122 California State Auditor Report 2004-108 California State Auditor Report 2004-108 111133
update of all program standards, including subject matter
standards that it required the commission to ensure were aligned
with the content of the student academic standards that the
State Board of Education (state board) had adopted. Consistent
with the discussion in its response on page 99, the commission
asserted to us that it used a process similar to the one that
it uses when it performs periodic updates of its standards.
Therefore, we saw no need to make an artificial distinction
based on what prompted the commission to review program
standards. On page 61 of the report we provide a description
of the commission’s process for developing and implementing
standards. Further, Figure 9 on page 63 shows the standards
under development since the act was passed and the adopted
implementation periods. Pages 60 and 61 of the report provide
text boxes to define the commission’s standards and key changes
in teacher preparation and the credentialing requirements of the
act. Moreover, we disagree that our report “confuses program
standards with test specifications and requirements.” To provide
context for how the commission develops program standards, we
provide information on page 61 regarding how the commission’s
contractor develops an examination that prospective teachers can
take in lieu of completing a teacher preparation program, including
the contractor’s use of a content validation study. We then state
on page 70 that the commission could benefit from providing the
results of the contractor’s content validation study to its advisory
panels and bias review committee to ensure further that its subject
matter requirements are free from biased language and content.
Finally, as stated in note 1 on page 109, we offered to meet with
the commission to discuss any concerns with the report during the
agency response period and the commission rejected our offers.
o
The commission misquotes our report. On page 3 of the
report we state that “the commission suspended its continuing
accreditation reviews.” Further, as we describe in note 8
on page 110, the commission mischaracterizes its current
accreditation activities.
p
This is an issue that we would have expected the commission
to let us know about during the agency review period. We have
eliminated the phrase “which it believes is outdated” on pages 3
and 60 of the report. Also, the commission’s September 2004
update to its strategic plan indicates that commission staff will
present the revised accreditation policy to the commissioners
in August 2005, not May 2005, as the commission stated in its
response. We note the August 2005 date on pages 1, 3, 60, and
73 of the report.
111122 California State Auditor Report 2004-108 California State Auditor Report 2004-108 111133
a
As we state in the Scope and Methodology of report on
page 12, our review focused on the accreditation of college
and university teacher preparation programs. We have
modified the text on page 8 to add the eight school districts
it has accredited in addition to the more than 80 colleges and
universities that the commission has accredited to operate
teacher preparation programs.
s
This is an issue that we would have expected the commission
to let us know about during the agency review period. We have
modified the text on page 5 to more closely conform to the text
of the state law.
d
The commission informed us of its concern with the text box
on page 7 during the agency response period, but its response
does not reflect the changes we agreed to make. To the extent we
could verify the commission’s methodology, we made changes.
Specifically, we accepted the commission’s revised numbers for
single- and multiple-subject credentials, expanded the heading
of the third line to “emergency and 30-day substitute permits,”
and slightly modified the commission’s count for administrative
services credentials. We also moved the “cross-cultural/bilingual”
numbers to the “all others” category.
f
This is an issue that we would have expected the commission to let
us know about during the agency review period. We have modified
the text on page 8 to incorporate the commission’s comments.
g
This is an issue that we would have expected the commission
to let us know about during the agency review period. We have
added text on page 7 indicating that prospective teachers for a
multiple-subject credential must take and pass an examination
to demonstrate academic preparation.
h
The commission misquotes our report. The correct quote is
“Under the direction of the state board [of Education] and
the superintendent of public instruction, Education provides
education policy direction to local school districts.” Therefore,
we have not changed the text as the commission suggested.
j
We disagree that the commission has completed the
development and adoption of all program standards pursuant
to the act. The act mandated a comprehensive update of all
program standards, including subject matter standards that
it required the commission to ensure were aligned with the
content of the student academic standards that the state board
111144 California State Auditor Report 2004-108 California State Auditor Report 2004-108 111155
had adopted. Figure 9 on page 63 of the report shows the status
of the commission’s work and that five sets of single-subject
standards are still in the development stage.
k
This is an issue that we would have expected the commission to
let us know about during the agency review period. Because the
commission used advisory panels of education professionals for
each of the five sets of standards we reviewed, we have changed
the text on page 11 to reflect this.
l
This is an issue that we would have expected the commission
to let us know about during the agency review period. We have
changed the text on page 12 from “updating” to “evaluating”
and deleted the text “to meet the requirements of the act.”
;
The commission misunderstands the focus of the section
beginning on page 14 of the report. Our focus is on the three
commission programs that help individuals meet the requirements
for a teaching credential or, in some cases, to meet the requirements
for entrance to a college, university, or school district preparation
program. We recognize the commission and Education co-
administer the Beginning Teacher Support and Assessment
program; however, it is an induction program for teachers that
have already earned a preliminary credential.
z
The commission incorrectly implies that we initiated the
comparison of the pre-intern program to emergency permit
holders. As we discuss page 18 of our report, the commission
made the comparison between emergency-permit holders and
pre-interns in its final program report to the Legislature. We only
pointed out that the commission could not support the data it
used in its comparison and that when we obtained current data
from the commission’s database, the difference in the retention
rate between the two groups for the same time period was not
as profound as the commission originally reported. Moreover,
the commission’s assertion that the retention of emergency
permit holders increased once “thousands of pre-interns had
been removed” from their ranks is not supported by its own
data. In fiscal year 1998–99, which was the first year of the
pre-intern program and when it had only 957 participants,
the retention rate of first-time emergency permit holders was
81 percent. In fiscal year 1999–2000, when the number of pre-
intern participants rose to 5,800, the retention rate of first-time
emergency permit holders decreased to 78 percent. Therefore,
we stand by our analysis and conclusions.
111144 California State Auditor Report 2004-108 California State Auditor Report 2004-108 111155
x
This is an issue that we would have expected the commission to let
us know about during the agency review period. We have changed
the text on page 17 to be specific to the fiscal year 2002–03 annual
report for the paraprofessional program.
c
A comparison between the pass rates of pre-intern test takers
and the pass rates of the general population of test takers would
be valid but, as we point out on page 19 of our report, the
commission could not support the statement that pre-interns
are largely members of groups that tend to pass subject matter
examinations at lower rates than the general population.
Likewise, the commission could not provide us with data on the
percentage of pre-interns that had taken the required subject
matter exam at least once. Therefore, we could not validate the
commission’s comparison of the test results of pre-interns to
those of “all test takers” that it made in its October 2001 report
to the Legislature (as shown in the second bullet on page 18) or
the comparison that the commission makes in its response to
our report on page 91.
v
We believe that the documentation supporting the statistics the
commission reports in its annual reports should be maintained
for the length of time designated in its document retention
policy. In view of the fact that this documentation is useful for
follow-up analysis and the assessment of trends, the commission
may wish to evaluate whether its policy specifies a reasonable
length of time to retain this supporting documentation.
b
The commission misses the point. Its current reports do not
provide it with sufficient data to monitor its workload. For
example, the commission’s weekly report, which is discussed
on page 46, does not break down the application workload by
unit team, which would provide a rough estimate of the relative
complexity of the workload waiting to be processed. Also, as
is discussed on page 43, the commission does not track the
number of applications that take longer than 75 business days
to process. Further, the commission’s claim that its biweekly
report provides the processing time of all applications is
misleading. As we discuss on page 47, the commission’s report
shows the difference in business days between the current date
and the dates that the commission received the applications.
This is not processing time, but wait time. Monitoring the
makeup of its workload and knowing the specific amount of
time that different types of applications take to process would
be beneficial to the commission because it would provide the
111166 California State Auditor Report 2004-108 California State Auditor Report 2004-108 111177
commission greater opportunities to make informed staffing
decisions and to ensure that it consistently completes all
applications within a 75-business-day time frame.
n
We did not include a discussion of the governor’s veto message
because we do not take exception to the 75-business-day
processing time requirement.
m
We disagree that we quoted the director of the certification
division out of context. The director has consistently asserted
to us that he did not know if the third phase of the Teacher
Credentialing Service Improvement Project (TCSIP) would
improve application processing times. In fact, the commission
states later in this paragraph of its response that the director
could not make comments about the possible efficiencies of the
third phase because the system had not been tested. Moreover,
on page 50 of the report, we note that the units’ processing staff
had decreased by 6.2 positions between the time it proposed
TCSIP and September 2004. We also acknowledge that having
less staff to process applications will likely negate some of the
efficiencies that may result from automation.
,
Contrary to the commission’s statement, on pages 49 and 50 of
the report we discuss the efficiencies from the online credential
look-up and the online renewals. Moreover, this comment
conflicts with the commission’s statement in its response on
page 95 which says, “As the report accurately states, much of the
efficiencies have already been realized by the first two phases of
the project . . .”.
.
Contrary to the commission’s statement, we note on pages 2,
11, 48, 51, and 54 of the report that the commission planned to
implement the third phase of TCSIP in late October 2004.
/
The commission’s assertion that we do not acknowledge the
efficiencies from the third phase of TCSIP is inaccurate. We
state on page 50 of the report that the electronic submission
of applications from colleges and universities will provide
efficiencies and on page 51 we state that the virtual credential
officer will provide efficiencies. However, we believe that the
commission could achieve additional efficiencies from the third
phase of TCSIP if it requires all institutional customers—colleges,
universities, and school districts—to submit applications
electronically, to the extent that it is economically feasible.
111166 California State Auditor Report 2004-108 California State Auditor Report 2004-108 111177
Similarly, we believe that expanding the virtual credential officer
to process more routine credential types could also provide
additional efficiencies.
!
The commission may have overlooked the discussion on page 55
of the report where we discuss this leaflet.
@
The commission mischaracterizes the full cost of the
management study performed by MGT of America (MGT) as the
cost of what it calls a time study. In fact, the MGT management
study report included a staffing chapter in which MGT
performed a high-level analysis that resulted in an average
number of credentials processed per hour. MGT did not
determine how long it took commission staff to process different
types of credentials. Moreover, the MGT report highlighted that
it based its estimate of processing capacity on the commission’s
then-current allocation of resources and method of processing
credentials. It also noted that changing the way staff are used
would affect this calculation. Since the commission has modified
its allocation of workload among its staff during the four years
since the MGT report and is implementing the third phase of TCSIP,
it can no longer rely on the analysis that MGT performed. We stand
by our analysis and recommendation that the commission needs to
routinely monitor the composition of the applications that it has
not yet processed and collect and analyze data on the average
review times for different types of applications.
#
We have modified the text on page 47 of the report to
include the commission’s comment that it did not have an
opportunity to request this report when it designed TCSIP.
$
We do not take issue with the fact that the commission
maintains a microfiche copy of the paper applications it
receives. Our point is that even with the virtual credential
officer, the commission intends to print paper copies of the
electronic applications it receives to manually convert them into
a microfiche record. This is less than optimal. The commission
could more efficiently store these applications electronically.
%
We have modified the heading on page 54 to more closely reflect
the text of the section.
^
We disagree that we “mix two different topics and treat them as
one.” The act mandated a comprehensive update of all program
standards, including subject matter standards that it required
the commission to ensure were aligned with the content of the
111188 California State Auditor Report 2004-108 California State Auditor Report 2004-108 111199
student academic standards that the state board had adopted.
Consistent with the commission’s discussion in its response on
page 99, the commission asserted to us that it used a process
similar to the one it uses when it performs periodic updates of
its standards. Therefore, we believe the commission can use our
recommendations to improve its future efforts in developing and
implementing standards for teacher preparation in California.
&
The commission misunderstands our recommendation to
develop an overall plan. As we stated on page 64, an overall
plan would include monitoring elements such as timelines
for developing standards and the resources it needs to meet
those timelines. Also, on page 64, we acknowledge that the
commission developed an approach to developing standards;
however, the approach did not include timelines for the standard
development activities that were linked to the resources
the commission planned to use. Further, the “SB 2042
implementation plan” the commission attached to its response
and the other items that the commission describes are a series of
updates, reports, agenda items, and other documents. Although
such documents could be supporting elements to an overall
plan, they do not represent an overall plan that could be used
to efficiently manage, guide, track, and monitor the various
activities needed to implement the act’s requirements.
*
We have modified the text to clarify that we reviewed five sets
of standards.
(
The commission misunderstands the need for a formal
evaluation of the processes it uses to conduct standards
development. Although the framework is set forth in statute,
the commission has some discretion in the tasks it uses to meet the
requirements of the framework. It is these tasks that the commission
should evaluate to determine whether it could make further
improvements in how it carries out its statutory responsibilities.
)
As we state on page 69 of the report, the commission could
not provide us with evidence of how it considered whether
a candidate was a teacher or an administrator, or how it
emphasized ethnic diversity in its selections, despite the
significance that the commission placed these factors in its
documents requesting candidates to apply.
Q
We disagree with the commission that it is not possible for panel
members to be selected objectively. If done consistently and
with a checklist or other review tool as we state on page 68, the
111188 California State Auditor Report 2004-108 California State Auditor Report 2004-108 111199
commission could ensure that the proposed panel collectively
met the commission’s qualifications. The commission’s current
process for panel member appointments is inconsistent, lacks
objectivity, and may not ensure that it selects the best-qualified
panel members available.
W
This is an issue that we would have expected the commission
to let us know about during the agency review period.
To clarify and be precise about where we found that the
commission’s record retention practices were weak, we have
changed the text on page 66 to “documents created during the
development of program standards.”
E
Our report does not suggest or recommend that the
commission use a ranking process. The ranking process we
describe on page 68 is the process the commission used, albeit
inconsistently, when selecting the English and mathematics
advisory panels. To address weaknesses in the commission’s
selection of these two panels and the art and administrative services
panels, we recommended that it develop a methodical approach
that includes evaluating all candidates’ qualifications against the
qualifications the commission seeks in panel members.
R
The commission’s director of the professional services division
made this statement to us. In an e-mail from this director to us
on September 21, 2004, in which she explained the process the
commission used to select art panel members, she stated “that
the art standards, although important, did not have the same
level of criticality as the English and math standards therefore,
the process did not involve the same level of rigor.” We asked
the director to clarify the term “criticality,” however, she did
not respond. Therefore, we referred to the American Heritage
Dictionary of the English Language definition of criticality,
which it defines as “being of the highest importance.”
T
The commission misses our point. The contractor’s content
validation studies provide a great deal of information about the
viewpoints of education professionals. However, as we note on
page 70 of the report, for most minority groups responding to
the surveys, the number of people responding to the surveys was
less than the number that would trigger a review flag. This does
not mean that the views of minority groups were unimportant,
and we believe the commission could benefit from having the
advisory panels and bias review committee look at these results.
Doing so could provide additional assurance that program
standards are free from biased language and content.
112200 California State Auditor Report 2004-108 California State Auditor Report 2004-108 112211
Y
The commission’s response suggests that it can place more
weight on a survey from an institution than from an individual.
We disagree. Based on the surveys the commission received,
there exists no valid method to place more weight on one survey
versus another.
U
Contrary to the commission’s statement, the audit team
repeatedly asked for documentation of how the commission
ensured that its program standards aligned with the student
academic standards the state board adopted. With reference
to the report the commission describes, we followed up verbal
requests for it in August and September 2004 with e-mail
requests on September 2, September 29, and October 5, 2004.
On October 8, 2004, commission staff provided us with the
same documents they had given us in July, but the staff did not
provide the report.
I
This is an issue that we would have expected the commission
to let us know about during the agency review period. We have
changed the wording on page 71 to state that the commission is
evaluating its accreditation policy.
O
This is an issue that we would have expected the commission
to let us know about during the agency review period. We have
changed the date on page 60 to December 2002.
112200 California State Auditor Report 2004-108 California State Auditor Report 2004-108 112211
cc: Members of the Legislature
Office of the Lieutenant Governor
Milton Marks Commission on California State
Government Organization and Economy
Department of Finance
Attorney General
State Controller
State Treasurer
Legislative Analyst
Senate Office of Research
California Research Bureau
Capitol Press
112222 California State Auditor Report 2004-108