Consultant's Accessibility Report

Election Systems & Software EVS 6.3.0.2 Accessibility, Usability, and Privacy Test Report for California Secretary of State CAF-ESS22006-AUPTR-01 Vendor Name Election Systems and Software (ES&S) Vendor System EVS 6.3.0.2 Prepared by: SLI ComplianceSM 4720 Independence St. Wheat Ridge, CO 80033 303-422-1566 www.SLICompliance.com Accredited by the Election Assistance Commission (EAC) for VSTL status. Election Systems and Software EVS 6.3.0.2 California Certification Accessibility, Usability, & Privacy Test Report Copyright ©2022 SLI Compliance SM, a Division of Gaming Laboratories International, LLC Revision History Date Release Author Revisions 11/30/2022 1.0 M. Santos Initial Release Disclaimer The Certification Test results reported herein must not be used by the client to claim product certification, approval, or endorsement by NVLAP, NIST, or any agency of the Federal Government. Results herein relate only to the items tested. Trademarks • SLI is a registered trademark of SLI Compliance, a Division of Gaming Laboratories International, LLC. • All other products and company names are used for identification purposes only and may be trademarks of their respective owners. The tests referenced in this document were performed in a controlled environment using specific systems and data sets; results are related to the specific items tested. Actual results in other environments may vary. Opinions and Interpretations There are no SLI opinions or interpretations included in this report. Accessibility, Usability, and Privacy Test Report v1.0 Page 2 of 9 Report Number CAF-ESS22006-AUPTR-01 Election Systems and Software EVS 6.3.0.2 California Certification Accessibility, Usability, & Privacy Test Report TABLE OF CONTENTS INTRODUCTION ................................................................................................................................. 4 REFERENCES ..................................................................................................................................... 4 SYSTEM OVERVIEW ......................................................................................................................... 4 VOTING SYSTEM SCOPE ..................................................................................................................... 4 CERTIFICATION TEST RESULTS SUMMARY ................................................................................. 5 ACCESSIBILITY, USABILITY, AND PRIVACY TESTING SUMMARY .............................................................. 5 Volunteer One .............................................................................................................................. 5 Volunteer Two .............................................................................................................................. 6 Volunteer Three ........................................................................................................................... 7 EVALUATION OF TESTING .................................................................................................................... 8 Issues Identified ........................................................................................................................... 8 Suggestions ................................................................................................................................. 9 Final Considerations .................................................................................................................... 9 Accessibility, Usability, and Privacy Test Report v1.0 Page 3 of 9 Report Number CAF-ESS22006-AUPTR-01 Election Systems and Software EVS 6.3.0.2 California Certification Accessibility, Usability, & Privacy Test Report Introduction SLI Compliance is submitting this test report as a summary of the certification testing efforts for the Election Systems & Software EVS 6.3.0.2 (ES&S EVS 6.3.0.2) voting system. The purpose of this document is to provide an overview of the Accessibility, Usability, and Privacy certification testing effort and the findings of the testing effort for the ES&S EVS 6.3.0.2 voting system’s DS300 optical ballot counter and ExpressVote HW2.1 ballot marking device components. References California Voting System Standards (CVSS) System Overview Voting System Scope This section provides a description of the ES&S EVS 6.3.0.2 voting system components utilized for Accessibility, Usability, and Privacy testing: • One ExpressVote HW2.1 ballot marking device, in a kiosk • One ExpressVote HW2.1 ballot marking device, on a table • One DS300 optical ballot counter with ballot container The ES&S EVS 6.3.0.2 ExpressVote HW2.1 is a ballot marking device used to assist voters with marking ballots.. These ballots are later scanned and tabulated by the DS300. The ES&S EVS 6.3.0.2 DS300 system employs a precinct-level optical scan ballot counter (tabulator) in conjunction with an external ballot box. This tabulator is designed to scan paper ballots, interpret voting marks, and deposit the ballots into the secure ballot box. Accessibility, Usability, and Privacy Test Report v1.0 Page 4 of 9 Report Number CAF-ESS22006-AUPTR-01 Election Systems and Software EVS 6.3.0.2 California Certification Accessibility, Usability, & Privacy Test Report Certification Test Results Summary Accessibility, Usability, and Privacy Testing Summary An election was run utilizing: • One DS300 optical ballot counter with ballot container • One ExpressVote HW2.1 in a kiosk • One ExpressVote HW2.1 on a table Three volunteers were used to evaluate each device, each with disabilities ranging from visual impairment to differing levels of fine motor control abilities. • Volunteer One was blind, but otherwise fully functioning • Volunteer Two was a Spanish language specialist with no disabilities • Volunteer Three was blind, but otherwise fully functioning The sessions were conducted with the volunteers voting on the ES&S devices. When the volunteers arrived, they were given a quick briefing on the testing and the devices. The volunteers then voted using the ExpressVote HW2.1, in either a kiosk or table position, to create a ballot. This ballot was tabulated using the DS300 optical ballot counter. Once testing was completed, each voter completed an Accessibility Test survey for each device. Volunteer One ExpressVote HW2.1 Initial Configuration: • One ExpressVote HW2.1 in a kiosk • Front approach voting booth in privacy configuration • One DS300 optical ballot counter with ballot container • Sitting in a chair • Using headphones • Audio on • Blank Screen Privacy option used • Keypad used Observations included: • Used a ballot style with many write-ins and grew frustrated with the write-in process. Accessibility, Usability, and Privacy Test Report v1.0 Page 5 of 9 Report Number CAF-ESS22006-AUPTR-01 Election Systems and Software EVS 6.3.0.2 California Certification Accessibility, Usability, & Privacy Test Report DS300 Initial Configuration: • Device setup in standard configuration Observations included: • No observations of note Volunteer One Summary Overall, the volunteer felt satisfied with the voting system and the speech being clear and of sufficient volume. Issues the volunteer had occurred during the write-in process. The write-in process can feel cumbersome and time-consuming when having to be performed several times and that the audio speed of moving through the alphabet was too slow which only added to the cumbersome feeling. There was confusion over the audio, giving instructions for a single write-in candidate contest as “First choice, write-in, last choice” as a single sentence. Additionally, there was confusion over going back a character during write-in as to whether it required pressing the back key or left arrow key. While reviewing write-ins that were typed in, the volunteer felt having the letters played back individually was confusing. Volunteer Two ExpressVote HW2.1 Initial Configuration: • One ExpressVote HW2.1 in a kiosk • Front approach voting booth privacy configuration • One DS300 optical ballot counter with ballot container • Sitting in a chair • Using headphones • Audio on • Keypad used Observations included: • Was a Spanish language specialist with no disabilities. • Was most concerned with language accuracy • Maxed out the character input for a write-in • Thorough use of the display setting options Accessibility, Usability, and Privacy Test Report v1.0 Page 6 of 9 Report Number CAF-ESS22006-AUPTR-01 Election Systems and Software EVS 6.3.0.2 California Certification Accessibility, Usability, & Privacy Test Report • Voted two ballots, one to verify Spanish language display and the second to verify the Spanish audio DS300 Initial Configuration: • Device setup in standard configuration Observations included: • No observations of note Volunteer Two Summary Volunteer Two was a member of a Language Access group that wanted to evaluate the Spanish language usage of the ExpressVote HW2.1. The volunteer was advised that the accessibility test was intended to evaluate the system for use by those with varying disabilities and not to evaluate the systems language supporting capabilities. The volunteer found the experience to be an average experience overall. However, the volunteer questioned whether the system could be used independently by those using audio to vote. The volunteer noted that they would prefer to use a paper ballot over the device. The volunteers primary concern was that counties would not properly pronounce foreign languages. Again, the volunteer was advised that the accessibility test was intended to evaluate the system for use by those with varying disabilities and not to evaluate the systems language supporting capabilities. The volunteer was also advised that verification of the system’s ability to meet all State and Federal language requirements was verified during functional testing of the system. Volunteer Two stated that they had a better experience when they were using the accessibility controls with large view options. Volunteer Three ExpressVote HW2.1 Initial Configuration: • One ExpressVote HW2.1 in a kiosk • Front approach voting booth privacy configuration • One DS300 optical ballot counter with ballot container • Sitting in a wheelchair • Using headphones • Audio on • Blank Screen Privacy option used • Keypad used Accessibility, Usability, and Privacy Test Report v1.0 Page 7 of 9 Report Number CAF-ESS22006-AUPTR-01 Election Systems and Software EVS 6.3.0.2 California Certification Accessibility, Usability, & Privacy Test Report Observations included: • Some minor inconveniences with trying to get the accessibility panel and cord around the privacy wall. • Asked about settings to change the hardness/softness of the button presses. DS300 Initial Configuration: • Device setup in standard configuration Observations included: • No observations of note Volunteer Three Summary Volunteer Three had an overall good experience with using the voting system. The volunteer liked the speech output and the ease of voting. The only functionality the voter found to be difficult to use was the write-in method. Additionally, the instructions were unclear as to whether the voter should be pressing the center of the keypad or the other buttons when making selections. The volunteer also found the buttons to be hard to press. The volunteer also shared concerns that people with mobility issues would have issues with pressing the buttons. Evaluation of Testing This section provides summary lists of the issues identified by volunteers during testing and any suggestions provided by the volunteers. Issues Identified Issues identified during testing included: • Write-in process can feel cumbersome and time-consuming when having to be performed several times, and the audio speed of moving through the alphabet was too slow which only added to the cumbersome feeling. • Confusion over the audio, giving instructions for a single write-in candidate contest as “First choice, write-in, last choice” as a single sentence. • Confusion over going back a character during write-in as to whether it required pressing the back key or left arrow key. • While reviewing write-ins that were typed in having the letters played back individually was confusing. Accessibility, Usability, and Privacy Test Report v1.0 Page 8 of 9 Report Number CAF-ESS22006-AUPTR-01 Election Systems and Software EVS 6.3.0.2 California Certification Accessibility, Usability, & Privacy Test Report • Concern about being able to use the system independently. Suggestions Several suggestions were made by the volunteers during the sessions as follows: • A volunteer suggested a method of finding what contests were skipped during the ballot review before printing their ballot and the ability to repeat the instructions using a keypad key. • A volunteer would like the keyboard to navigate utilizing left/right keys instead of the up/down keys. • A volunteer suggested options to allow a voter to change the sensitivity of the keypad button presses. Final Considerations The consensus of the volunteers was that they felt the technologies implemented for accessibility and usability were easy to use, but to varying degrees. All volunteers had one device that they preferred using for various reasons. As directed by the California Secretary of State, this accessibility, usability, and privacy testing report does not include any recommendation as to whether or not the system should be approved. End of AUP Test Report Accessibility, Usability, and Privacy Test Report v1.0 Page 9 of 9 Report Number CAF-ESS22006-AUPTR-01