Hart Verity Voting 3.1 Voting System Volume Testing Report

Hart InterCivic Verity Voting 3.1 Voting System Volume and Reliability Test Report for California Secretary of State CAF-19048-VRTR-01 Vendor Name Hart InterCivic Vendor System Verity Voting 3.1 Prepared by: 4720 Independence St. Wheat Ridge, CO 80033 303-422-1566 www.SLICompliance.com Accredited by the U.S. Election Assistance Commission (EAC) for Selected Voting System Test Methods or Services California Secretary of State Hart InterCivic Verity Voting 3.1 Voume and Reliability Test Report Revision History Date Release Author Revision Summary November 4, 2019 1.0 M. Santos Initial Release Disclaimer The information reported herein must not be used by the client to claim product certification, approval, or endorsement by NVLAP, NIST, or any agency of the Federal Government. Trademarks SLI Compliance is a registered trademark of Gaming Laboratories International, LLC. All products and company names are used for identification purposes only and may be trademarks of their respective owners. Copyright 2019 by SLI ComplianceSM, a division of Gaming Laboratories International, LLC. California Certification Volume and Page 2 of 9 Reliability Test Report Document No. CAF-19048-VRTR-01, v1.0 California Secretary of State Hart InterCivic Verity Voting 3.1 Voume and Reliability Test Report TABLE OF CONTENTS INTRODUCTION ........................................................................................................................................... 4 TESTING RESPONSIBILITIES .......................................................................................................................... 4 EQUIPMENT TO BE TESTED ............................................................................................................................ 4 TEST PREPARATIONS ................................................................................................................................... 5 TEST ACTIVITIES......................................................................................................................................... 6 EVALUATION OF TESTING ........................................................................................................................ 9 California Certification Volume and Page 3 of 9 Reliability Test Report Document No. CAF-19048-VRTR-01, v1.0 California Secretary of State Hart InterCivic Verity Voting 3.1 Voume and Reliability Test Report INTRODUCTION This Volume and Reliability Test Report details the work performed for the Volume and Reliability testing on the Hart InterCivic Verity Voting 3.1 (Hart Verity Voting 3.1) voting system against the California Voting System Standards (CVSS). The purpose of this test was to verify that voting equipment will operate reliably in conditions approximating normal use by voters in a polling place on Election Day. Testing Responsibilities Staffing SOS: Todd Ross Fourteen temporary employees to vote test ballots Hart: Tyson Gosch SLI: Mike Santos Alex Nestico Location The test was conducted in Austin Texas, at a facility provided by Hart. Facility: Wingate by Wyndham & Williamson Conference Center 1209 N IH 35 Round Rock TX 78664 Equipment to be tested The equipment included: 10 Verity Touch Writers 10 Verity Readers 20 Verity Scans Verity Touch Writer Verity Touch Writer is a touch-screen Ballot Marking Device (BMD) that prints voter-marked ballots to a commercial-off-the-shelf (COTS) printer. California Certification Volume and Page 4 of 9 Reliability Test Report Document No. CAF-19048-VRTR-01, v1.0 California Secretary of State Hart InterCivic Verity Voting 3.1 Voume and Reliability Test Report Voters use the electronic touch display interface to privately and independently make their selections on the ballot. Voters can also make selections with Verity Access, an Audio-Tactile interface (ATI) component with three tactile buttons, one audio port (for headphones), and one port for external two-switch devices. When voters finish making their selections, they print the marked ballot. Verity Reader Verity Reader is an optional paper ballot review device suitable for use by all voters, including non-disabled voters and voters with disabilities. Voters can insert their marked paper ballot to visually verify how their ballot will be counted when the ballot is cast in the Verity system, and/or hear audio read-back of their ballot choices. (For voters with disabilities, Verity Reader offers the same accessibility features as the Verity Touch Writer ballot marking device.) Verity Scan Verity Scan is Verity’s digital scanning solution for paper ballots. Verity Scan is paired with a purpose-built ballot box to ensure accurate, secure, and private ballot scanning and vote casting. When opening the polls, authorized users activate the Verity Scan device to prepare it to receive marked paper ballots. Verity Scan indicates when it is appropriate to insert ballots, and when ballots have been successfully cast. Verity Scan records Cast Vote Records (CVRs) and audit log data in redundant, secure storage locations, including the Verity vDrive. vDrive storage is portable flash memory and allows the CVRs to be transferred to the Verity Count tabulation and reporting system. Test Preparations Election Provisions The election definition was the same general election used for functional testing. Four hundred single sheet ballots were scanned on each Verity Voting Scan; Fifty blank ballots were marked and printed on each Touch Writer; and Fifty ballots were verified on each Reader. Hart supplied: Identical test decks of ballots, pre-marked in a pattern approved by the Secretary of State. Additional blank ballots to replace any ballots that were damaged during the test. California Certification Volume and Page 5 of 9 Reliability Test Report Document No. CAF-19048-VRTR-01, v1.0 California Secretary of State Hart InterCivic Verity Voting 3.1 Voume and Reliability Test Report Sufficient quantities of blank ballots or ballot paper to mark on the Touch Writer machines. Each device was prepared as follows: Each device was assigned a unique number. A label with that number was placed on the front of the device. Beginning with “1,” the numbers progressed sequentially and were three inches high in order to allow the cameras to capture the number when incident resolutions are filmed. The machines were arranged to allow the Verity Scan units to be tested first, followed by Verity Touch Writers and then Verity Readers. Test Activities All testing was accomplished in five phases: Environment setup. Logging of devices, hashing of devices, and election loading. Scanning of ballots on Verity Scan. Voting of ballots on Verity Touch Writer. Reading of Verity Touch Writer produced ballots on Verity Reader. On Sunday, October 27 Hart set up all devices to be used for the scheduled testing. On Monday, October 28, with Hart representatives present, California Secretary of State and SLI Compliance representatives inventoried all devices as listed in the “Equipment to be Tested” section above. This included recording all serial numbers. Next, all devices were verified to have the correct firmware by checking hash codes of the installed firmware against the hash codes of record for the validated firmware. Finally, for the day, the election to be utilized was deployed to each device, and zero reports were run. On Tuesday, October 29, 14 temporary employees were utilized to perform the testing. Initially, 14 Verity Scans were utilized, with 400 ballots scanned through each. As temporary employees finished their scanning, they were transferred to new California Certification Volume and Page 6 of 9 Reliability Test Report Document No. CAF-19048-VRTR-01, v1.0 California Secretary of State Hart InterCivic Verity Voting 3.1 Voume and Reliability Test Report devices. The first six to finish moved on to the remaining Verity Scans, so that all 20 Verity Scan devices were exercised with 400 ballots being scanned. Next, as they finished their scanning on Verity Scan, the remaining temporary employees were moved over to vote and print 50 ballots on Verity Touch Writers. Lastly, as each temporary employee finished with creating their 50 Verity Touch Writer Ballots, they took those printed ballots and verified them on Verity Readers. The table below provides a chronological view of the activities. Table 1 – Hart Verity Voting 3.1 Volume Testing Activities Sunday, October 27th 1 All machines delivered to testing site. 2 Setup machines. Hart was responsible for security of the venue during testing. Monday, October 28th 3 Testing team kickoff meeting. Reviewed the test plan and rules. 4 Setup video cameras and adjusted for best coverage. Verified the machines. Checked machines to make sure no damage occurred in 5 transit. Devices arrived with Verity 3.1 Trusted Build already installed. 7 Each machine was labeled with its test number. Each machine was validated against Trusted Build HASHes to be running 8 Trusted Build 3.1. A log of machine serial numbers and Physical Configuration Audit (PCA) 9 information was captured. 10 Photographed each machine being tested. Test decks were distributed for each machine as necessary. Test decks were 11 supplied by Hart. Tuesday, October 29th Checked-in temporary employees; processed payroll and non-disclosure (NDA) 1 documentation as required by the SOS and the temporary agency. Briefed temporary employees on the test objectives, their roles and 2 responsibilities, and provided them with the instructions a voter would be given on how to operate the machine. 3 Temporary employees were briefed on how to operate the Scan machine. The voters were instructed to signal any unexpected condition on a machine by 4 raising their hand. 5 SOS staff interviewed the voter and analyzed the event. California Certification Volume and Page 7 of 9 Reliability Test Report Document No. CAF-19048-VRTR-01, v1.0 California Secretary of State Hart InterCivic Verity Voting 3.1 Voume and Reliability Test Report 6 Took any steps necessary to continue voting and completed an incident report. Incidents were categorized either as one of the substantive failures described in 7 the Protocol or as a non-substantive failure, such as voter error or a defective ballot. 8 400 ballots were scanned on each Verity Scan machine. 9 As a voter finished, the polls were closed on that machine. 10 Totals printed. Media was removed from all Verity Scan machines as they were 11 completed and retained. Ballots were removed from the ballot box and straightened. Ballots were 12 boxed for return to CA SOS. The total number of ballots processed on the device was entered on the 13 inventory sheet and marked with the label bearing the machine number indicating that machine was complete. Incident reports were logged and the number of machines experiencing 14 each type of substantive failure was recorded. 15 Secured and retained all media and artifacts. Table 2 – Hart Verity Voting 3.1 Ballot Marking and Reading Activities Verity Touch Writer and Verity Reader Testing Activities (Ballot Marking and Reading) Tuesday, October 29 Briefed temporary employees on their roles and responsibilities and 1 provided them with the instructions a voter would be given on how to mark ballots with the Verity Touch Writer machine. The voters were instructed to signal any unexpected condition on a 2 machine by raising their hand. SLI staff interviewed the voter, analyzed the event, took any steps 3 necessary to continue marking ballots, and completed an incident report. Incidents were categorized either as one of substantive failure or as a non- 4 substantive failure, such as voter error. 5 Opened polls on machines as necessary. 6 Voters marked 50 ballots on each Verity Touch Writer machine. 7 Voters read the 50 ballots on each Verity Reader machine. Any unexpected results were resolved and documented as a ballot defect, 8 a testing error, or a tabulation error. California Certification Volume and Page 8 of 9 Reliability Test Report Document No. CAF-19048-VRTR-01, v1.0 California Secretary of State Hart InterCivic Verity Voting 3.1 Voume and Reliability Test Report Verity Touch Writer and Verity Reader Testing Activities (Ballot Marking and Reading) Tuesday, October 29 9 Closed polls on machines as necessary. 10 Secured and retained all media and artifacts. All media from all devices and boxed ballots will be brought to the 11 CASOS. Chain of custody for ballots and media will be maintained by SOS. 12 Testing was concluded by end of day October 29, 2019. Evaluation of Testing Three types of issues were noted during the testing. 1. On Verity Scan, a paper jam message that displayed whenever the “voter” held the ballot too long once the scanner was trying to pull it in. This message was seen 26 times out of 8,000 ballots cast, with 12 of the messages occurring for one ”voter.” Second attempts at input were successful for each ballot that encountered this situation. 2. On Verity Scan, a ballot skew message displayed whenever the “voter” put the ballot in at too much of an angle. This message was 65 times out of 8,000 ballots cast, with 40 of the messages occurring for three of the “voters.” This occurred when the “voter” was at an angle to the input tray and/or using only one hand on either side of the ballot and inserting at an angle. Second attempts at input were successful for each ballot that encountered this situation. 3. On Verity Reader, a ballot skew message displayed whenever the “voter” put the ballot in at too much of an angle. This message was one time out of 500 ballots read. This occurred when the “voter” was at an angle to the input tray and/or using only one hand on either side of the ballot and inserting at an angle. Second attempts at input were successful for each ballot that encountered this situation. As directed by the California Secretary of State, this report does not include any recommendation as to whether or not the system should be approved End of Volume and Reliability Test Plan California Certification Volume and Page 9 of 9 Reliability Test Report Document No. CAF-19048-VRTR-01, v1.0