Hart Verity Voting 3.2 Voting System Volume Testing Report

Hart InterCivic Verity Voting 3.2 Voting System Volume and Reliability Test Report for California Secretary of State CAF-22009-FTR-04 Vendor Name Hart InterCivic Vendor System Verity Voting 3.2 Prepared by: 4720 Independence St. Wheat Ridge, CO 80033 303-422-1566 www.SLICompliance.com Accredited by the U.S. Election Assistance Commission (EAC) for Selected Voting System Test Methods or Services California Secretary of State Hart InterCivic Verity Voting 3.2 California Certification Volume & Reliability Test Report v4.0 Revision History Date Release Author Revision Summary June 12, 2023 1.0 A. Nestico Initial Release June 21, 2023 2.0 A. Nestico Update for typos June 23, 2023 3.0 A. Nestico Update for CASOS comments July 14, 2023 4.0 A. Nestico, Update for CASOS comments M. Santos Disclaimer The information reported herein must not be used by the client to claim product certification, approval, or endorsement by NVLAP, NIST, or any agency of the Federal Government. Trademarks • SLI Compliance is a registered trademark of Gaming Laboratories International, LLC. • All products and company names are used for identification purposes only and may be trademarks of their respective owners. Copyright 2023 by SLI ComplianceSM, a division of Gaming Laboratories International, LLC. Volume and Reliability Test Report v4.0 Page 2 of 9 Document No. CAF-22009-FTR-04 California Secretary of State Hart InterCivic Verity Voting 3.2 California Certification Volume & Reliability Test Report v4.0 TABLE OF CONTENTS INTRODUCTION ................................................................................................................................. 4 TESTING RESPONSIBILITIES ................................................................................................................ 4 EQUIPMENT USED .............................................................................................................................. 4 TEST PREPARATIONS ......................................................................................................................... 5 TEST ACTIVITIES............................................................................................................................... 6 EVALUATION OF TESTING .............................................................................................................. 8 Volume and Reliability Test Report v4.0 Page 3 of 9 Document No. CAF-22009-FTR-04 California Secretary of State Hart InterCivic Verity Voting 3.2 California Certification Volume & ReliabilityTest Report v4.0 INTRODUCTION This Volume and Reliability Test Report details the work performed for the Volume and Reliability testing on the Hart InterCivic Verity Voting 3.2 (Hart Verity Voting 3.2) voting system against the California Voting System Standards (CVSS). The purpose of this test was to verify that voting equipment will operate reliably in conditions approximating normal use by voters in a polling place on Election Day. Testing Responsibilities Staffing SOS: • Todd Ross • Deborah Ledsinger • Twenty temporary employees to vote test ballots. Hart: • Tyson Gosch SLI: • Alex Nestico Location The test was conducted in Woodland, California, at a facility provided by Yolo County, California. Facility Yolo County Administration Building, 625 Court St Woodland, CA 95695 Equipment Used The equipment included: • 10 Verity Touch Writers • 6 Verity Readers • 20 Verity Scans Verity Touch Writer Verity Touch Writer is a touch-screen Ballot Marking Device (BMD) that prints voter-marked ballots to a commercial-off-the-shelf (COTS) printer. Voters use the electronic touch display interface to privately and independently make their selections on the ballot. Voters can also make selections with Verity Volume and Reliability Test Report v4.0 Page 4 of 9 Document No. CAF-22009-FTR-04 California Secretary of State Hart InterCivic Verity Voting 3.2 California Certification Volume & Reliability Test Report v4.0 Access, an Audio-Tactile interface (ATI) component with three tactile buttons, one audio port (for headphones), and one port for external two-switch devices. When voters finish making their selections, they print the marked ballot. Verity Reader Verity Reader is an optional paper ballot review device suitable for use by all voters, including non-disabled voters and voters with disabilities. Voters can insert their marked paper ballot to visually verify how their ballot will be counted when the ballot is cast in the Verity system, and/or hear audio read-back of their ballot choices. (For voters with disabilities, Verity Reader offers the same accessibility features as the Verity Touch Writer ballot marking device.) Verity Scan Verity Scan is Verity’s precinct tabulator scanning solution for paper ballots. Verity Scan is paired with a purpose-built ballot box to ensure accurate, secure, and private ballot scanning and vote casting. When opening the polls, authorized users activate the Verity Scan device to prepare it to receive marked paper ballots. Verity Scan indicates when it is appropriate to insert ballots, and when ballots have been successfully cast. Verity Scan records Cast Vote Records (CVRs) and audit log data in redundant, secure storage locations, including the Verity vDrive. The Verity vDrive storage is portable flash memory and allows the CVRs to be transferred to the Verity Count tabulation and reporting system. Test Preparations Election Provisions The election definition was the same general election used for functional testing. • Approximately 2,000 single sheet ballots were scanned on each Verity Scan. • Approximately 100 blank ballots were marked and printed on each Touch Writer. • Approximately 800 ballots were verified on each Reader. Hart supplied: • Identical test decks of ballots, pre-marked in a pattern approved by the Secretary of State . • Additional blank ballots to replace any ballots that were damaged during the test. • Sufficient quantities of blank ballots or ballot paper to mark on the Touch Writer machines. Volume and Reliability Test Report v4.0 Page 5 of 9 Document No. CAF-22009-FTR-04 California Secretary of State Hart InterCivic Verity Voting 3.2 California Certification Volume & Reliability Test Report v4.0 Each device was prepared as follows: • Each device was assigned a unique number. • A label with that number was placed on the front of the device. • Beginning with “1,” the numbers progressed sequentially and were three inches high in order to allow the cameras to capture the number when incident resolutions were filmed. The machines were arranged to allow the Verity Scan units to be tested first, followed by the Verity Touch Writers and then the Verity Readers. Test Activities All testing was accomplished in five phases: • Environment setup • Logging of devices, hashing of devices, and election loading • Scanning of ballots on Verity Scan • Voting of ballots on Verity Touch Writer • Reading of Verity Touch Writer produced ballots on Verity Reader On day 1, Hart set up all devices to be used for the scheduled. On day 2 and 3, with Hart representatives present, California Secretary of State (CASOS) and SLI Compliance representatives inventoried all devices as listed in the “Equipment Used” section above. This included recording all serial numbers. Next, all devices were verified to have the correct firmware by checking hash codes of the installed firmware against the hash codes of record for the validated firmware. Twenty temporary employees were utilized to perform the testing. Initially, 20 Verity Scans were utilized, with 1,000 ballots scanned through each. As temporary employees finished their scanning, they were transferred to the Verity Touch Writers where they voted and printed 100 ballots. As each temporary employee finished creating their 100 Verity Touch Writer Ballots and the preprinted ballots used in the Scan, they took those printed ballots and verified them on Verity Readers. Lastly, temporary employees then cast another 1,000 ballots through the Verity Scan. Volume and Reliability Test Report v4.0 Page 6 of 9 Document No. CAF-22009-FTR-04 California Secretary of State Hart InterCivic Verity Voting 3.2 California Certification Volume & Reliability Test Report v4.0 The table below provides a chronological view of the activities. Table 1 – Hart Verity Voting 3.2 Volume Testing Activities D D a a y y 2 1 -3 a nd 2 Hashes were taken from every Hart Verity device and validated against the 1 Trusted Build hashes to be running Trusted Build 3.2 A log of machine serial numbers and Physical Configuration Audit (PCA) 2 information was captured. 3 Photographed each Hart Verity device being tested 4 Checked-in temporary employees Briefed temporary employees on the test objectives, their roles and responsibilities, and provided them with the instructions a voter would be given on 5 how to scan ballots on the Verity Scan Device and how to mark ballots on the Verity Touch Writer machine. 6 Temporary employees were briefed on how to operate the Scan machine. The voters were instructed to signal any unexpected condition on a machine by 7 raising their hand. 8 SOS staff interviewed the voter and analyzed the event. 9 Took any steps necessary to continue voting and completed an incident report. Incidents were categorized either as one of the substantive failures described in 10 the Protocol or as a non-substantive failure, such as voter error or a defective ballot. 2 Hour Battery Test started on Verity Touch Writer Device #7 (W1500065606) with Brother Printer HL-L6400DW (U64185D2N572273) using the Duracell 11 PowerSource 660 UPS (X22). The test started at 9:40 am and ended at 1:15 pm with 27 ballots voted and approximately 100% battery power remaining. Approximately 1,000 ballots were scanned on each Verity Scan machine over the 12 course of testing. As a voter finished, the polls were closed on that machine and voters moved to a 13 Touch Writer Voters marked approximately 100 ballots on each Verity Touch Writer machine 14 over the course of testing. Voters read approximately 800 ballots on each Verity Reader machine over the 15 course of testing. Approximately 1,000 additional ballots were scanned on each Verity Scan 16 machine over the course of testing. 17 Totals printed. Volume and Reliability Test Report v4.0 Page 7 of 9 Document No. CAF-22009-FTR-04 California Secretary of State Hart InterCivic Verity Voting 3.2 California Certification Volume & Reliability Test Report v4.0 Media was removed from all Verity Scan machines as they were completed and 18 retained. Ballots were removed from the ballot box and straightened. Ballots were boxed for 19 return to CASOS. The total number of ballots processed on the device was entered on the inventory 20 sheet and marked with the label bearing the machine number indicating that machine was complete. Incident reports were logged and the number of machines experiencing each type 21 of substantive failure was recorded. 22 Closed polls on machines as necessary. 23 Secured and retained all media and artifacts. All media from all devices and boxed ballots will be brought to the CASOS. Chain 24 of custody for ballots and media will be maintained by CASOS. 25 Testing was concluded by end of day June 7th, 2023. Evaluation of Testing Five distinct issues and issue types were noted during the testing. 1. Verity Reader Device #15 would report an error that “One or more landmarks could not be located”. Restarting the Verity Reader device allowed the scanning of two more ballots before the same issue was reported. Reloading the election definition did not correct the issue. Next step in troubleshooting would be calibration; however, the calibration kit is currently not available. Verity Reader Device #15 was removed from testing. To date, no proof of concept has been demonstrated to verify the calibration kit would resolve the issue. 2. On Verity Scan, a paper jam message that displayed likely as a result of the “voter” holding the ballot too long once the scanner started to pull it in. This message was seen one time out of the 40,000 ballots cast on all Verity Scan devices. 3. On Verity Scan, a “Your choices may not be properly marked” warning message that displayed regardless of the choices being properly marked. This message was seen two times out of the 40,000 ballots cast on all Verity Scan devices. Second attempts at scanning the ballots were successful for the two ballots that encountered this situation. The error rate for this issue was “.00005”. This message can be caused by the ballot being inputted in a skewed manner. Re-insertion of the ballot will correct the issue. 4. On Verity Scan, a “Could not read one or more barcodes” error message that displayed regardless of the choices being properly marked. This message was seen 21 times out of the 40,000 ballots cast on all Verity Scan devices. Second attempts at scanning the ballots were successful for the 21 Volume and Reliability Test Report v4.0 Page 8 of 9 Document No. CAF-22009-FTR-04 California Secretary of State Hart InterCivic Verity Voting 3.2 California Certification Volume & ReliabilityTest Report v4.0 ballots that encountered this situation. The error rate for this issue was “.000525”. This message can be caused by the ballot being inputted in a skewed manner. Re-insertion of the ballot will correct the issue. 5. On Verity Scan, a “Ballot is not the correct length” error message that displayed regardless of the choices being properly marked. This message was seen three times out of the 40,000 ballots cast on all Verity Scan devices. Second attempts at scanning the ballots were successful for the three ballots that encountered this situation. The error rate for this issue was “.000075”. As directed by the California Secretary of State, this report does not include any recommendation as to whether or not the system should be approved. End of Volume and Reliability Test Report Volume and Reliability Test Report v4.0 Page 9 of 9 Document No. CAF-22009-FTR-04