Hart Verity Voting 3.2 Voting System Accessibility and Usability Testing Report

Hart InterCivic Verity Voting 3.2 Accessibility, Usability, and Privacy Test Report for California Secretary of State CAF-23004-AUPTR-02 Vendor Name Hart InterCivic Vendor System Verity Voting 3.2 Prepared by: 4720 Independence St. Wheat Ridge, CO 80033 303-422-1566 www.SLICompliance.com Accredited by the Election Assistance Commission (EAC) for VSTL status. Hart InterCivic Verity Voting 3.2 California Certification Accessibility, Usability & Privacy Test Report v2.0 Copyright 2023 SLI ComplianceSM, a Division of Gaming Laboratories International, LLC Revision History Date Release Author Revisions 6/7/2023 v1.0 A. Nestico Initial Release; 7/13/2023 V2.0 A.Nestico Updates for CASOS comments Disclaimer The Certification Test results reported herein must not be used by the client to claim product certification, approval, or endorsement by NVLAP, NIST, or any agency of the Federal Government. Results herein relate only to the items tested. Trademarks • SLI is a registered trademark of SLI Compliance, a Division of Gaming Laboratories International, LLC. • All other products and company names are used for identification purposes only and may be trademarks of their respective owners. The tests referenced in this document were performed in a controlled environment using specific systems and data sets and results are related to the specific items tested. Actual results in other environments may vary. Opinions and Interpretations There are no SLI opinions or interpretations included in this report. California Certification AUP Test Report v2.0 Page 2 of 11 Report Number CAF-23004-AUPTR-02 Hart InterCivic Verity Voting 3.2 California Certification Accessibility, Usability & Privacy Test Report v2.0 TABLE OF CONTENTS INTRODUCTION ................................................................................................................................... 4 REFERENCES....................................................................................................................................... 4 SYSTEM OVERVIEW ........................................................................................................................... 4 SCOPE OF THE HART INTERCIVIC VERITY VOTING 3.2 VOTING SYSTEM .................................................. 4 TEST RESULTS SUMMARY ................................................................................................................ 5 TESTING SUMMARY .............................................................................................................................. 5 Volunteer One ................................................................................................................................ 5 Volunteer Two ................................................................................................................................ 7 Volunteer Three ............................................................................................................................. 7 Volunteer Four ............................................................................................................................... 9 Volunteer Five .............................................................................................................................. 10 Volunteer Six ................................................................................................................................ 10 FINAL CONSIDERATIONS ................................................................................................................ 11 California Certification AUP Test Report v2.0 Page 3 of 11 Report Number CAF-23004-AUPTR-02 Hart InterCivic Verity Voting 3.2 California Certification Accessibility, Usability & Privacy Test Report v2.0 Introduction This Accessibility, Usability, and Privacy (AUP) Test Report details the testing performed during AUP testing of the Hart InterCivic Verity Voting 3.2 (Hart Verity Voting 3.2) voting system. References • California Voting System Standards (CVSS) System Overview Scope of the Hart InterCivic Verity Voting 3.2 Voting System This section provides a description of the scope of Hart InterCivic Verity Voting 3.2 voting system components involved in this test: • Two Verity Touch Writer firmware/hardware, polling place (W1500067006, W2014374311) • Verity Reader firmware/hardware, polling place (R2115230606) • Verity Scan firmware/hardware, polling place (S2317536201) The Hart Verity Touch Writer ballot marking platform is used for the creation of paper ballots. These ballots are later scanned and tabulated by the Hart Verity Scan optical ballot counter. The Hart Verity Reader system employs a precinct-level optical scanner which is designed to scan paper ballots, interpret voting marks, and display the ballots onto a viewable tablet such that the voter is able to review and verify their choices. The Hart Verity Scan system employs a precinct-level optical scan ballot counter (tabulator) in conjunction with an external ballot box. This tabulator is designed to scan paper ballots, interpret voting marks, and deposit the ballots into the secure ballot box. Election details in Accessibility, Usability and Privacy testing: • Ballot size: 8.5 x 14 • Election Name: Presidential General • Election Type: General • Language: English, Korean, Chinese, Vietnamese • Election Content: Seven contests and three propositions California Certification AUP Test Report v2.0 Page 4 of 11 Report Number CAF-23004-AUPTR-02 Hart InterCivic Verity Voting 3.2 California Certification Accessibility, Usability & Privacy Test Report v2.0 Test Results Summary Testing Summary Three devices were used for the AUP testing: Verity Touch Writer, Verity Reader, and Verity Scan. The sessions were conducted with Hart InterCivic personnel acting as poll workers, and the volunteers were acting as voters voting on the Hart Verity Voting 3.2 voting system devices. When the volunteers arrived, they were given a quick briefing on the testing and the devices. The sessions were a free form, ad hoc test where the volunteers were able to vote a ballot in any manner that they chose. The volunteers voted on the Verity Touch Writer device, processed the ballot on the Verity Reader device, and scanned the ballot on the Verity Scan device. The volunteers provided feedback in real-time as they were voting, as well as completing an Accessibility Test survey for each device. Additionally, their observations were recorded as each volunteer navigated their way through the process. Volunteer One Volunteer One used the Audio Tactile Interface (ATI) and jelly switches for the Verity Touch Writer and Verity Reader devices. Once they completed voting the ballot and verifying their votes using the Verity Reader, they scanned the ballot through Verity Scan. Afterwards, they completed a written a survey. The survey results are shown in Figure 1. California Certification AUP Test Report v2.0 Page 5 of 11 Report Number CAF-23004-AUPTR-02 Hart InterCivic Verity Voting 3.2 California Certification Accessibility, Usability & Privacy Test Report v2.0 Figure 1 – Volunteer One Survey Volunteer felt the timeframe it took to vote was somewhat acceptable because they were already familiar with the system. Volunteer One Summary Volunteer One was surveyed after the testing and asked if they were satisfied using this system or if they would prefer another. The volunteer said they would prefer another system, either Dominion or Los Angeles County’s Voting System for All People (VSAP) Volunteer One was asked if they had any suggestions, comments, or changes they would like to provide. Volunteer One stated that the scroll wheel was hard to use due to their limited mobility preventing them from making complete turns with the wheel and that the response on screen to the scroll wheel was too fast. They felt from previous experiences that the jelly-switches were more difficult to use this time around. Being able to move the screen closer would be helpful for those with limited reach. Additionally, being able to vote outside from a car would be helpful. No concerns were voiced over privacy issues. California Certification AUP Test Report v2.0 Page 6 of 11 Report Number CAF-23004-AUPTR-02 Hart InterCivic Verity Voting 3.2 California Certification Accessibility, Usability & Privacy Test Report v2.0 Volunteer Two Volunteer Two used the ATI and headphones for the Verity Touch Writer and Verity Reader devices. Once they completed voting the ballot and verifying their votes using the Verity Reader, they scanned the ballot Reader, they scanned the ballot through Verity Scan. Afterwards, they answered a survey. Results are shown in Figure 2. Figure 2 – Volunteer Two Survey Volunteer Two Summary Volunteer Two was surveyed after the testing and responded that they were satisfied with this voting system. Volunteer Two was asked if they had any suggestions, comments, or changes they would like to provide. Volunteer Two said the system was simple to use; however, they did not like using the scroll wheel to get to the “next” button (as in next contest); they would like it if it would automatically go to the “next” button rather than having to scroll to it. No concerns were voiced over privacy issues. Volunteer Three Volunteer Three used the ATI and headphones for the Verity Touch Writer and Verity Reader devices. Once they completed voting the ballot and verifying their votes using the Verity Reader, they scanned the ballot through Verity Scan. California Certification AUP Test Report v2.0 Page 7 of 11 Report Number CAF-23004-AUPTR-02 Hart InterCivic Verity Voting 3.2 California Certification Accessibility, Usability & Privacy Test Report v2.0 Afterwards, they completed a written survey. The survey results are shown in Figure 3. Figure 3 – Volunteer Three Survey Volunteer Three Summary Volunteer Three was surveyed after the testing and responded that they were satisfied with this voting system. Volunteer Three was asked if they had any suggestions, comments, or changes they would like to provide. They said the instructions did not give all the information they should have and that it should be clearer that a counterclockwise scroll is used to get out of the menu. The normal speech speed was too slow, but the fast speed was too fast, there should be increments between the speech speeds. Additional comments Volunteer Three had included: • They found the help button and read out needed more information and additional helpful information. • The move wheel scrolling in a circular fashion was confusing and they would prefer a mouse scroll wheel. • There should be an independent physical “next” button to move to the next contest rather than using the wheel to select the on-screen next button, along with an independent “back” button to go back to previous contests. • A mechanical (refreshable) braille display for reading the contests. California Certification AUP Test Report v2.0 Page 8 of 11 Report Number CAF-23004-AUPTR-02 Hart InterCivic Verity Voting 3.2 California Certification Accessibility, Usability & Privacy Test Report v2.0 • Devices should have more information about propositions at the vote centers. • The ability to focus on bits of a paragraph, such as repeating a sentence, pausing, or rewinding to help break up information for easier understanding. • The Verity Scan did not contain an audible indicator that the ballot was accepted or rejected, though this may have just been disabled during testing. • There should be more high contrasts options, such as black background with yellow contrast. Volunteer Four Volunteer Four used the Audio Tactile Interface (ATI) and headphones for the Verity Touch Writer and Verity Reader devices. Once they completed voting the ballot and verifying their votes using the Verity Reader, they scanned the ballot through Verity Scan. Afterwards, they completed a written a survey. The survey results are shown in Figure 4. Figure 4 – Volunteer Four Survey Volunteer Four Summary Volunteer Four was surveyed after the testing and responded that they preferred this when compared to the pen and paper method of voting; however, they felt that an app on their phone that allowed use of their native accessibility features would be better. Volunteer Four was asked if they had any suggestions, comments, or changes they would like to provide. They thought that the Verity Reader usage was confusing and California Certification AUP Test Report v2.0 Page 9 of 11 Report Number CAF-23004-AUPTR-02 Hart InterCivic Verity Voting 3.2 California Certification Accessibility, Usability & Privacy Test Report v2.0 unnecessary and that all devices need padding for resting one’s arm when using the device. The volunteer did like the low range of pitch of the speech and how easy it was to understand. They also felt the scroll wheel made write-ins a much easier process. Volunteer Five Volunteer Five used the ATI and headphones for the Verity Touch Writer and Verity Reader devices. Once they completed voting the ballot and verifying their votes using the Verity Reader, they scanned the ballot through Verity Scan. Afterwards, they completed a written survey. The survey results are shown in Figure 5. Figure 5 – Volunteer Five Survey Volunteer Five Summary Volunteer Five was surveyed after the testing and responded that they were satisfied using this voting system. Volunteer Five was asked if they had any suggestions, comments, or changes they would like to provide. They said they would like the speech speed to be an incremental increase/decrease rather than slow, normal, and fast. They liked how comfortable the headphones were and liked using the scroll wheel. Volunteer Six Volunteer Six used the ATI and headphones for the Verity Touch Writer and Verity Reader devices. Once they completed voting the ballot and verifying their votes California Certification AUP Test Report v2.0 Page 10 of 11 Report Number CAF-23004-AUPTR-02 Hart InterCivic Verity Voting 3.2 California Certification Accessibility, Usability & Privacy Test Report v2.0 using the Verity Reader, they scanned the ballot through Verity Scan. Afterwards, they completed a written a survey. The survey results are shown in Figure 6. Figure 6 – Volunteer Six Survey Volunteer Six Summary Volunteer Six was surveyed after the testing and responded that they were satisfied using this voting system. Volunteer Six was asked if they had any suggestions, comments, or changes that would like to provide. They said that they really liked using this system and felt that it was great to use. They would like to have a phonetic alphabet for clearer understanding of the letters used, for example selecting V and having it say victor to help distinguish it from B. They liked using the scroll wheel. Final Considerations The general consensus of the volunteers was that they felt the technologies implemented for accessibility and usability improved the experience for voters that are most in need of them. As directed by the California Secretary of State, this Accessibility, Usability, and Privacy Testing Report does not include any recommendation as to whether or not the system should be approved. End of AUP Test Report California Certification AUP Test Report v2.0 Page 11 of 11 Report Number CAF-23004-AUPTR-02