Los Angeles County Vsap 3.0 Consultant’s Volume Test Report

Los Angeles’ County Voting Solutions for All People (VSAP) 3.0 Voting System Volume and Reliability Test Report for California Secretary of State CAF-21009-VRTR-01 County of Los Vendor Name Angeles Vendor System VSAP 3.0 Prepared by: 4720 Independence St. Wheat Ridge, CO 80033 303-422-1566 www.SLICompliance.com Accredited by the U.S. Election Assistance Commission (EAC) for Selected Voting System Test Methods or Services Los Angeles County VSAP 3.0 California Certification Volume and Reliability Test Report v3.0 Revision History Date Release Author Revision Summary January 25th, 2022 1.0 B. Watters, Initial Release M. Santos February 9th, 2022 2.0 B.Watters, Corrections made per CA SOS request February 10th, 2022 3.0 T.Mapps Corrections made per CA SOS request Disclaimer The information reported herein must not be used by the client to claim product certification, approval, or endorsement by NVLAP, NIST, or any agency of the Federal Government. Trademarks • SLI Compliance is a registered trademark of Gaming Laboratories International, LLC. • All products and company names are used for identification purposes only and may be trademarks of their respective owners. Copyright 2022 by SLI ComplianceSM , a division of Gaming Laboratories International, LLC. California Certification Volume and Reliability Test Report Page 2 of 9 Report No. CAF-21009-VRTR-01 Los Angeles County VSAP 3.0 California Certification Volume and Reliability Test Report v3.0 TABLE OF CONTENTS INTRODUCTION ................................................................................................................................. 4 TESTING RESPONSIBILITIES ................................................................................................................ 4 EQUIPMENT TESTED ........................................................................................................................... 5 TEST PREPARATIONS ......................................................................................................................... 5 TEST ACTIVITIES............................................................................................................................... 6 TEST SCHEDULES .............................................................................................................................. 6 EVALUATION OF TESTING .............................................................................................................. 7 California Certification Volume and Reliability Test Report Page 3 of 9 Report No. CAF-21009-VRTR-01 Los Angeles County VSAP 3.0 California Certification Volume and Reliability Test Report v3.0 INTRODUCTION This Volume and Reliability Test Report details the work performed for the Volume and Reliability testing on the County of Los Angeles’s Voting Solutions for All People 3.0 (VSAP 3.0) voting system against the California Voting System Standards (CVSS). The purpose of this test was to verify that voting equipment will operate reliably in conditions approximating normal use by voters in a polling place on Election Day. Testing Responsibilities All testing was conducted under the guidance of personnel verified by California Secretary of State (CASOS) to be qualified to perform the testing. Responsibility Assignments CASOS utilized third party testers during the performance of portions of the examination. Temporary contract personnel were utilized in the Volume testing. All testing was witnessed or executed by CASOS personnel, with assistance from SLI Compliance personnel. Staffing CASOS: • NaKesha Robinson • Danielle Dawson • Temporary employees to vote test ballots SLI: • Brooke Watters Location The test was conducted in Los Angeles, California, at the VSAP Operations Center, 12100 Rivera Road, Whittier, California. Los Angeles County was responsible for security of the site and equipment throughout the duration of the test. Facility: Unless otherwise specified herein, examination and review were performed at the following standard ambient conditions and tolerances: • Temperature: 68–75° F (±3.6°F) • Relative Humidity: Local Site Humidity • Atmospheric Pressure: Local Site Pressure California Certification Volume and Reliability Test Report Page 4 of 9 Report No. CAF-21009-VRTR-01 Los Angeles County VSAP 3.0 California Certification Volume and Reliability Test Report v3.0 • Time Allowable Tolerance: ±5% Schedule Testing was conducted January 18–19, 2022. Los Angeles County delivered and set up the equipment on January 14, 2022. Equipment Tested The equipment used included 50 Ballot Marking Devices (BMD). Ballot Marking Device The BMD is the primary touchpoint for the voter and hub of the voting system, guiding users with screen prompts and symbols. The BMD features a touchscreen, an audio-tactile interface (controller and headphones), paper handler (scanner and printer), QR code scanner, and dual-switch input which voters use to generate, verify, and cast paper ballots. Completed ballots are transferred to the integrated ballot box, which can be detached for unloading. Test Preparations The election definition utilized was the same Primary Election (March 2020) used for the VSAP 3.0 Functional Test. One hundred ballots were marked, printed, and cast on each BMD. Each device was prepared as follows: • Each device was assigned a unique test number. • A label with the unique number was placed on the front of the device. • Beginning with “1,” the numbers progressed sequentially and were at least three inches high in order to allow the cameras to capture the number when any incident resolutions were filmed. An inventory of all devices was taken. The inventory included: • The type of device • Device serial number • Hardware version • Software and/or firmware versions • The test number assigned to the machine • A column to record the number of ballots processed on the machine during the test The machines were arranged to allow sufficient space around the machines to provide operators with space to stack un-voted ballots, as well as personal space for health considerations (minimum six feet distancing). California Certification Volume and Reliability Test Report Page 5 of 9 Report No. CAF-21009-VRTR-01 Los Angeles County VSAP 3.0 California Certification Volume and Reliability Test Report v3.0 TEST ACTIVITIES All testing was accomplished in three phases: • Environment setup • Logging of devices, hashing of devices, and election loading • Voting of ballots on BMD On Friday, January 14th, Los Angeles County set up all devices to be used for the scheduled testing. On Tuesday, January 18th, with Los Angeles County representatives present, CASOS and SLI Compliance representatives inventoried all devices as listed in the “Equipment Tested” section above. All devices were verified to have the correct firmware by checking hash codes of the installed firmware against the hash codes of record for the validated firmware. The election to be utilized was deployed to each device, and zero reports were run. Temporary employees were utilized to perform the testing. Temporary employees processed 100 ballots through each BMD. As temporary employees finished their ballot marking, they were transferred to new devices. All BMD’s were exercised with 100 ballots being processed through each over the three-day schedule for Volume testing. A total of 5,000 ballots were voted during testing. Test Schedules All testing was accomplished according to the schedule provided in the Tables below. Table 1 – VSAP 3.0 Preparation Activities Task Task # 1 All machines were delivered to testing site. Setup machines. LA County was responsible for security of the venue during 2 testing. 3 Testing team kickoff meeting. Reviewed the test plan and rules. 4 Setup video camera(s) and adjusted for best coverage. Verified the machines. Checked machines to make sure no damage occurred in 5 transit. Devices were delivered with VSAP 3.0 Trusted Build already installed. 6 Each machine was labeled with test number. Each machine was validated against Trusted Build HASHes to be running 7 Trusted Build VSAP 3.0. 8 A log of machine serial numbers information was captured. California Certification Volume and Reliability Test Report Page 6 of 9 Report No. CAF-21009-VRTR-01 Los Angeles County VSAP 3.0 California Certification Volume and Reliability Test Report v3.0 Task Task # 9 Photograph each machine being tested with its serial number and test number. 10 Test decks were distributed for each machine. Table 2 – VSAP 3.0 Ballot Marking and Reading Testing Activities Task Task # 1 Checked-in temporary employees. Briefed temporary employees on the test objectives for the day, their roles and 2 responsibilities, and provided them with the instructions a voter would be given on how to mark ballots using a BMD. The voters were instructed to signal any unexpected condition on a machine by 3 stepping back from the machine and raising their hand. CASOS representative documented any errors on the error form, entered it in 4 the error log, and ensured the error and any corrective actions were videotaped. SLI staff interviewed the voter, analyzed the event, took any steps necessary to 5 continue marking ballots, and completed an incident report. Incidents were categorized either as one of the substantive failures described in 6 the Protocol or as a non-substantive failure such as voter error or a defective ballot. 7 Opened Polls on machines as needed. 8 Voters marked 100 ballots on each BMD. Any unexpected results were resolved and then documented and categorized as 9 either ballot defect, testing error, or tabulation error. 10 Closed polls on machines as voting was completed. 11 Repeated this process for all machines. 12 Secured and retained all media and artifacts. All media from all devices and all boxed ballots brought to the CASOS. Chain of 13 custody for ballots and media maintained by CASOS. The logs from each ballot marking device were loaded into BMG and the logs 14 provided to the CASOS. EVALUATION OF TESTING The following were noted during the Volume Test, and each was subsequently mitigated: 1. Timid Feeds – Out of the 50 ballot marking devices used during testing, two experienced timid feeds or Ballot Page Metadata (BPM) errors warranting the ballot to be re-fed into the device. On units #5 and #48, voters on these devices reported having to re-insert the ballot multiple times before being California Certification Volume and Reliability Test Report Page 7 of 9 Report No. CAF-21009-VRTR-01 Los Angeles County VSAP 3.0 California Certification Volume and Reliability Test Report v3.0 accepted for each voting session. Other voters were witnessed experiencing a similar issue on other devices, though they did not report it due to being successful on their second attempt to load the ballot. The BMD device by design gently pulls the ballot into the device in a timid, push/pull manner. Testers would sometimes be too gentle triggering the rollers, not allowing the ballot to catch. Testers would also sometimes attempt to force feed the ballot or would pull the ballot back after the device began to gently pull the ballot into the device. When the tester would perform one of these three actions, the BMD would appear to pull the ballot in, but immediately eject it and indicate to the voter the ballot would need to be re-inserted. The scanner on device #48 was cleaned which assisted in reducing the number of instances a ballot was rejected due to timid feed though the issue persisted. Clear instructions to voters and poll workers regarding the timid feed will mitigate the chances of this occurring. 2. QR Code Errors • Two BMDs reported an “Unable to Read Ballot” error when the ballot being cast was initially inserted into the ballot box. In all instances, the ballots were re-inserted on the same device and successfully loaded. SLI inspected the BMDs and found no obvious cause for the error and rejection of the ballot. • One BMD unit, #24, rejected both paper and electronic ballots when attempting to obtain the Zero Report during the Opening of the Polls. The machine was rebooted and accepted the ballot for the open polls report. The BMD did not experience any other issues during Volume testing voting sessions. • Several ballots provided to voters were missing a QR code with precinct and ballot style information. The voter was displayed with a “Ballot Is Empty” onscreen message. These ballots were able to be manually activated by using the election credentials and entering the precinct number manually. 3. Paper Jam – Five devices experienced a paper jam during testing. Each device experienced only a single paper jam during testing as follows: • Unit #1 experienced a paper jam during the initial insertion of a blank ballot to begin the voting session. When removed from the scanner, the ballot was undamaged with no noticeable markings or bends. • Unit #9 reported a paper jam to the voter; however, the ballot had not yet been feed into the BMD. The error displayed without any prompt of input from the voter between voting sessions. • Unit #15 experienced a paper jam immediately after inserting the first ballot after polls were opened. When removed, the ballot was noticeably damaged along its edge and replaced with a new ballot. California Certification Volume and Reliability Test Report Page 8 of 9 Report No. CAF-21009-VRTR-01 Los Angeles County VSAP 3.0 California Certification Volume and Reliability Test Report v3.0 • Unit #32 experienced a paper jam during the casting of the ballot. When removed from the scanner, the ballot was undamaged with no noticeable marking or bends. It was re-cast without further issues. • Unit #33 presented the Error Code 203 to the voter with instructions that a paper jam needed to be cleared. When removed from the scanner, the ballot was undamaged with no noticeable markings or bends. In all cases once the ballot was clear from the scanner/printer and the screen cleared using election worker credentials, jamming did not occur with additional attempts. 4. Error Code – Two devices, #10 and #45, experienced error code 901. This occurred without any interaction from the voter. Error Code 901 states that “There is an issue that requires this unit to be rebooted.” Once rebooted the error was cleared and voting resumed without further issues. 5. Language – While casting a ballot in Indonesian, a voter pointed out that the election definition did not contain a translation screen for the “Party- Nominated Offices” instructions page and instead had to display it in English. 6. Frozen Screen – A single device, unit #40, was reported by the voter to get hung up on the “Print Ballot” page for too long and buttons would be unresponsive. Waiting an additional moment, pressing the “Print” button an additional time at the bottom of the screen more firmly and after a brief delay the BMD would move on. This could possibly be due to user error, causing the device to get hung up. As directed by the California Secretary of State, this report does not include any recommendation as to whether or not the system should be approved. End of Volume and Reliability Test Report California Certification Volume and Reliability Test Report Page 9 of 9 Report No. CAF-21009-VRTR-01