County of Los Angeles Vsap 2.1 Consultant's Volume Testing Report

Los Angeles County VSAP Tally 2.1 Voting System Volume and Reliability Test Report for California Secretary of State CAF-20007-VRTR-01 Vendor Name Los Angeles County Vendor System VSAP Tally 2.1 Prepared by: 4720 Independence St. Wheat Ridge, CO 80033 303-422-1566 www.SLICompliance.com Accredited by the U.S. Election Assistance Commission (EAC) for Selected Voting System Test Methods or Services Los Angeles County VSAP Tally 2.1 California Certification Volume and Reliability Test Report v3.0 Revision History Date Release Author Revision Summary July 28, 2020 1.0 M. Santos Initial Release August 10, 2020 2.0 M. Santos Updates for CASOS comments August 10, 2020 3.0 M. Santos Update for CASOS comment Disclaimer The information reported herein must not be used by the client to claim product certification, approval, or endorsement by NVLAP, NIST, or any agency of the Federal Government. Trademarks • SLI Compliance is a registered trademark of Gaming Laboratories International, LLC. • All products and company names are used for identification purposes only and may be trademarks of their respective owners. Copyright 2020 by SLI ComplianceSM, a division of Gaming Laboratories International, LLC. California Certification Volume and Reliability Test Report Page 2 of 8 Report No. CAF-20007-VRTR-01, v3.0 Los Angeles County VSAP Tally 2.1 California Certification Volume and Reliability Test Report v3.0 TABLE OF CONTENTS INTRODUCTION ................................................................................................................................. 4 TESTING RESPONSIBILITIES ................................................................................................................ 4 EQUIPMENT TESTED ........................................................................................................................... 5 TEST PREPARATIONS ......................................................................................................................... 5 TEST ACTIVITIES............................................................................................................................... 6 TEST SCHEDULES .............................................................................................................................. 6 EVALUATION OF TESTING .............................................................................................................. 8 California Certification Volume and Reliability Test Report Page 3 of 8 Report No. CAF-20007-VRTR-01, v3.0 Los Angeles County VSAP Tally 2.1 California Certification Volume and Reliability Test Report v3.0 INTRODUCTION This Volume and Reliability Test Report details the work performed for the Volume and Reliability testing on the Los Angeles County’s Voting Solutions for All People Tally 2.1 (VSAP Tally 2.1) voting system against the California Voting System Standards (CVSS). The purpose of this test was to verify that voting equipment will operate reliably in conditions approximating normal use by voters in a polling place on Election Day. Testing Responsibilities All testing was conducted under the guidance of personnel verified by California Secretary of State (CASOS) to be qualified to perform the testing. Responsibility Assignments CASOS utilized third party testers during the performance of portions of the examination. Temporary contract personnel were utilized in the Volume testing. All testing was witnessed or executed by CASOS personnel, with assistance from SLI Compliance personnel. Staffing SOS: • NaKesha Robinson • Danielle Rodriguez • Rodney Rodriguez • Temporary employees to vote test ballots SLI: • Alex Nestico • Steven Ramage Location The test was conducted in Los Angeles, California, at the VSAP Operations Center. Los Angeles County was responsible for security of the site and equipment throughout the duration of the test. Facility: Unless otherwise specified herein, examination and review were performed at the following standard ambient conditions and tolerances: • Temperature: 68–75° F (±3.6°F) • Relative Humidity: Local Site Humidity California Certification Volume and Reliability Test Report Page 4 of 8 Report No. CAF-20007-VRTR-01, v3.0 Los Angeles County VSAP Tally 2.1 California Certification Volume and Reliability Test Report v3.0 • Atmospheric Pressure: Local Site Pressure • Time Allowable Tolerance: ±5% Schedule Testing was conducted July 21–23, 2020. Los Angeles County delivered and set up the equipment on July 20, 2020. Equipment Tested The equipment used included: • 50 Ballot Marking Devices (BMD) • 1 BMD Manager (BMG) implementation – Needed for log data Ballot Marking Device The BMD is the primary touchpoint for the voter and hub of the voting system, guiding users with screen prompts and symbols. The BMD features a touchscreen, an audio-tactile interface (controller and headphones), paper handler (scanner and printer), QR code scanner, and dual-switch input which voters use to generate, verify, and cast paper ballots. Completed ballots are transferred to the integrated ballot box, which can be detached for unloading. Test Preparations The election definition was the same Primary Election (March 2020) used for the VSAP Tally 2.1 Functional Test. One hundred ballots were marked, printed, and cast on each BMD. The three BMD’s used for the VSAP Tally 2.1 Environmental Hardware Test were also utilized during this Volume testing. Each device was prepared as follows: • Each device was assigned a unique test number. • A label with the unique number was placed on the front of the device. • Beginning with “1,” the numbers progressed sequentially and were at least three inches high in order to allow the cameras to capture the number when any incident resolutions were filmed. An inventory of all devices was taken. The inventory included: • The type of device • Device serial number • Hardware version • Software and/or firmware versions California Certification Volume and Reliability Test Report Page 5 of 8 Report No. CAF-20007-VRTR-01, v3.0 Los Angeles County VSAP Tally 2.1 California Certification Volume and Reliability Test Report v3.0 • The test number assigned to the machine • A column to record the number of ballots processed on the machine during the test The machines were arranged to allow sufficient space around the machines to provide operators with space to stack un-voted ballots, as well as personal space for health considerations (minimum six feet distancing). One BMD was equipped and operated on battery power for two hours. TEST ACTIVITIES All testing was accomplished in three phases: • Environment setup • Logging of devices, hashing of devices, and election loading • Voting of ballots on BMD On Monday, July 20th, Los Angeles County set up all devices to be used for the scheduled testing. On Tuesday, July 21st, with Los Angeles County representatives present, CASOS and SLI Compliance representatives inventoried all devices as listed in the “Equipment Tested” section above. All devices were verified to have the correct firmware by checking hash codes of the installed firmware against the hash codes of record for the validated firmware. The election to be utilized was deployed to each device, and zero reports were run. Temporary employees were utilized to perform the testing. Temporary employees processed 100 ballots through each BMD. As temporary employees finished their ballot marking, they were transferred to new devices. One device was removed during the volume test and replaced with a unit labeled as “50 A”. All BMD’s were exercised with 100 ballots being processed through each over the three-day schedule for Volume testing. A total of 5,000 ballots were voted during testing. Test Schedules All testing was accomplished according to the schedule provided in the Tables below. Table 1 – VSAP Tally 2.1 Preparation Activities Task Task # 1 All machines were delivered to testing site. California Certification Volume and Reliability Test Report Page 6 of 8 Report No. CAF-20007-VRTR-01, v3.0 Los Angeles County VSAP Tally 2.1 California Certification Volume and Reliability Test Report v3.0 Task Task # Setup machines. LA County was responsible for security of the venue during 2 testing. 3 Testing team kickoff meeting. Reviewed the test plan and rules. 4 Setup video camera(s) and adjusted for best coverage. Verified the machines. Checked machines to make sure no damage occurred in 5 transit. Devices were delivered with VSAP Tally 2.1 Trusted Build already installed. 6 Each machine was labeled with test number. Each machine was validated against Trusted Build HASHes to be running Trusted 7 Build VSAP Tally 2.1. 8 A log of machine serial numbers information was captured. 9 Photograph each machine being tested with its serial number and test number. 10 Test decks were distributed for each machine. Table 2 – VSAP Tally 2.1 Ballot Marking and Reading Testing Activities Task Task # 1 Checked-in temporary employees. Briefed temporary employees on the test objectives for the day, their roles and 2 responsibilities, and provided them with the instructions a voter would be given on how to mark ballots using a BMD. The voters were instructed to signal any unexpected condition on a machine by 3 stepping back from the machine and raising their hand. CASOS representative documented any errors on the error form, entered it in the 4 error log, and ensured the error and any corrective actions were videotaped. SLI staff interviewed the voter, analyzed the event, took any steps necessary to 5 continue marking ballots, and completed an incident report. Incidents were categorized either as one of the substantive failures described in the 6 Protocol or as a non-substantive failure such as voter error or a defective ballot. 7 Opened Polls on machines as needed. 8 Voters marked 100 ballots on each BMD. Any unexpected results were resolved and then documented and categorized as 9 either ballot defect, testing error, or tabulation error. 10 Closed polls on machines as voting was completed. 11 Repeated this process for all machines. 12 Secured and retained all media and artifacts. All media from all devices and all boxed ballots brought to the CASOS. Chain of 13 custody for ballots and media maintained by CASOS. The logs from each ballot marking device were loaded into BMG and the logs 14 provided to the CASOS. California Certification Volume and Reliability Test Report Page 7 of 8 Report No. CAF-20007-VRTR-01, v3.0 Los Angeles County VSAP Tally 2.1 California Certification Volume and Reliability Test Report v3.0 EVALUATION OF TESTING The following were noted during the Volume Test, and each was subsequently mitigated: 1. Timid Feeds – Out of the 5,000 ballots fed into the ballot marking device units during testing 149 experienced timid feeds or Ballot Page Metadata (BPM) errors warranting the ballot to be refed into the device. The 149 timid feeds occurred in 23 of the 50 units. The BMD device by design gently pulls the ballot into the device, in a timid, push/pull manner. Testers would sometimes attempt to force feed the ballot or would pull the ballot back after the device began to gently pull the ballot into the device. When the tester did one of those two things, the BMD would appear to pull the ballot in, but immediately eject it. Clear instructions to voters and poll workers regarding the timid feed will mitigate the chances of this occurring. 2. QR Code Errors – a. Five BMDs encountered a “QR Code not Read” error upon initial insertion into the ballot box. In all instances, the ballots were tried on another device, and successfully read. Further examination of the ballots determined that the QR codes printed on the ballots, used to activate the voting session were slightly skewed when printed, thus causing random anomalies in some machines, but not others. b. One BMD unit, #40, would intermittently reject seemingly valid QR codes. When inserting the ballot, the feeder would start to take the ballot in, and then reject it. The BMD was restarted, but the device continued to reject the ballots. Preventative maintenance was also performed on the unit, but the unit continued to reject ballots. The device was removed from the test and replaced with a new unit which was labeled “50A.” 3. Blank Screen & Paper Jam – One device encountered a blank screen and paper jam after the tester inserted the ballot. The ballot was removed, following protocol, but the screen remained blank. After a hard reboot of the system, the tester was able to proceed without further incident. As directed by the California Secretary of State, this report does not include any recommendation as to whether or not the system should be approved. End of Volume and Reliability Test Report California Certification Volume and Reliability Test Report Page 8 of 8 Report No. CAF-20007-VRTR-01, v3.0