Consultant's Accessibility Report

ES&S EVS 6.0.4.2 Accessibility, Usability, and Privacy Test Report ESS-18S52059-AUPTR-01 Vendor Name Election Systems and Software (ES&S) Vendor System EVS 6.0.4.2 Prepared by: SLI ComplianceSM 4720 Independence St. Wheat Ridge, CO 80033 303-422-1566 www.SLICompliance.com Accredited by the Election Assistance Commission (EAC) for VSTL status. Election Systems and Software EVS 6.0.4.2 California Certification Accessibility, Usability, & Privacy Test Report Copyright 2019 SLI Compliance SM, a Division of Gaming Laboratories International, LLC Revision History Date Release Author Revisions 8/05/2019 1.0 D. George Initial Release Updates based on 8/09/2019 1.1 D. George comments 8/16/2019 1.2 D. George Additional edits 8/20/2019 1.3 D. George Final edits Disclaimer The Certification Test results reported herein must not be used by the client to claim product certification, approval, or endorsement by NVLAP, NIST, or any agency of the Federal Government. Results herein relate only to the items tested. Trademarks SLI is a registered trademark of SLI Compliance, a Division of Gaming Laboratories International, LLC. All other products and company names are used for identification purposes only and may be trademarks of their respective owners. The tests referenced in this document were performed in a controlled environment using specific systems and data sets; results are related to the specific items tested. Actual results in other environments may vary. Opinions and Interpretations There are no SLI opinions or interpretations included in this report. Accessibility, Usability, and Privacy Test Report v1.3 Page 2 of 13 Report Number ESS-18S52059-AUPTR-01 Election Systems and Software EVS 6.0.4.2 California Certification Accessibility, Usability, & Privacy Test Report TABLE OF CONTENTS INTRODUCTION ........................................................................................................................................... 4 REFERENCES ............................................................................................................................................... 4 SYSTEM OVERVIEW ................................................................................................................................... 4 VOTING SYSTEM SCOPE ............................................................................................................................... 4 CERTIFICATION TEST RESULTS SUMMARY ........................................................................................... 5 ACCESSIBILITY, USABILITY, AND PRIVACY TESTING SUMMARY ........................................................................ 5 Volunteer One ........................................................................................................................................ 5 Volunteer Two ........................................................................................................................................ 7 Volunteer Three ..................................................................................................................................... 8 Volunteer Four ..................................................................................................................................... 10 Volunteer Five ...................................................................................................................................... 11 EVALUATION OF TESTING ............................................................................................................................ 13 Issues Identified ................................................................................................................................... 13 Suggestions ......................................................................................................................................... 13 Final Considerations ............................................................................................................................ 13 Accessibility, Usability, and Privacy Test Report v1.3 Page 3 of 13 Report Number ESS-18S52059-AUPTR-01 Election Systems and Software EVS 6.0.4.2 California Certification Accessibility, Usability, & Privacy Test Report Introduction SLI Compliance is submitting this test report as a summary of the certification testing efforts for the ES&S EVS 6.0.4.2 voting system. The purpose of this document is to provide an overview of the Accessibility, Usability, and Privacy certification testing effort and the findings of the testing effort for the ES&S EVS 6.0.4.2 voting system’s DS200, ExpressVote, and ExpressVote XL components. References California Voting System Standards (CVSS) System Overview Voting System Scope This section provides a description of the ES&S EVS 6.0.4.2 voting system components utilized for Accessibility, Usability, and Privacy testing: DS200 HW1.3 / 2.20.0.0 ExpressVote HW 2.1/ 2.5.0.0 ExpressVote XL / 1.2.0.0 The EVS 6.0.4.2 DS200 system employs a precinct-level optical scan ballot counter (tabulator) in conjunction with an external ballot box. This tabulator is designed to scan paper ballots, interpret voting marks, and deposit the ballots into the secure ballot box. The EVS 6.0.4.2 ExpressVote and ExpressVote XL ballot marking platforms are solutions used for creation of paper ballots. These ballots are later scanned and tabulated by the DS200 optical ballot counter. Accessibility, Usability, and Privacy Test Report v1.3 Page 4 of 13 Report Number ESS-18S52059-AUPTR-01 Election Systems and Software EVS 6.0.4.2 California Certification Accessibility, Usability, & Privacy Test Report Certification Test Results Summary Accessibility, Usability, and Privacy Testing Summary An election was run utilizing: One DS200 Device One ExpressVote Device One ExpressVote XL Device Five volunteers were used to evaluate each device, each with disabilities ranging from visual impairment to differing levels of fine motor control abilities. Volunteer One was blind, but otherwise fully functioning Volunteer Two was blind, but otherwise fully functioning Volunteer Three was in a wheelchair, with a limited range of motion when lifting their arms Volunteer Four was blind, but otherwise fully functioning Volunteer Five’s disability was not apparent The sessions were conducted with ES&S personnel acting as poll workers and the volunteers voting on the ES&S devices. When the volunteers arrived, they were given a quick briefing on the testing and the devices. The volunteers then voted using the ExpressVote, in either a kiosk or desk position, to create a ballot. This ballot was tabulated using the DS200. They then voted and created a ballot using the ExpressVote XL device. This ballot was also tabulated using the DS200 device. Once testing completed, each voter completed an Accessibility Test survey for each device. Volunteer One ExpressVote Initial Configuration: Front approach voting booth privacy configuration Sitting in a chair Using headphones Audio on Blank Screen Privacy option used Keypad used Accessibility, Usability, and Privacy Test Report v1.3 Page 5 of 13 Report Number ESS-18S52059-AUPTR-01 Election Systems and Software EVS 6.0.4.2 California Certification Accessibility, Usability, & Privacy Test Report Observations included: Volunteer One observed that the Braille on the ExpressVote device was only on the bottom of the ballot chute. They also noted that the N audio sounds like M, making it hard to distinguish between N and M on the keyboard. Volunteer One did not like having to listen to all the write-in prompts when voting for more than one contest. Volunteer One did like how the keypad felt to use. Volunteer One felt that going back and forth between a previous contest and the current contest would cause the device to not start the audio and selections at the top of the list. Volunteer One felt that the instructions were not enough at the beginning of the election. ExpressVote XL Initial Configuration: Front approach voting booth privacy configuration Sitting in a chair Using headphones Audio on Blank Screen Privacy option used Universal Voting Console (UVC) used Observations included: Volunteer One felt that there should be a divider between the two volume buttons to ensure a clear distinction between the two buttons. Volunteer One noted that there was no braille on the device itself to enable understanding of its functions. Volunteer One noted that when re-prompting the contest header audio, the ExpressVote XL did not notify the voter of the choices already selected for the contest. DS200 Initial Configuration: Device setup in standard configuration Accessibility, Usability, and Privacy Test Report v1.3 Page 6 of 13 Report Number ESS-18S52059-AUPTR-01 Election Systems and Software EVS 6.0.4.2 California Certification Accessibility, Usability, & Privacy Test Report Observations included: Volunteer One found that if a privacy sleeve was available, then the utilization of the DS200 was satisfying without being confusing. Volunteer One Summary Volunteer One found the systems to be satisfying to use, with little to no confusion. Additionally, they felt confident in the privacy and accuracy of their vote experience. Audio speech could be a bit clearer (this can be fixed during the election creation process) but overall was able to understand the ballot. Volunteer Two ExpressVote Initial Configuration: Front approach voting booth privacy configuration Sitting in a chair Using headphones Audio on Blank Screen Privacy option used Keypad used Observations included: Volunteer Two found it difficult to press the buttons on the keypad. Volunteer Two found the review summary screen to be too confusing, with unclear instructions. Volunteer Two enjoyed the speed of the voting method. They specifically highlighted the ability to move between contests. Volunteer Two was unable to find the space bar button on the onscreen keyboard during the write-in process. Volunteer Two felt that the audio could be clearer. ExpressVote XL Initial Configuration: Front approach voting booth privacy configuration Accessibility, Usability, and Privacy Test Report v1.3 Page 7 of 13 Report Number ESS-18S52059-AUPTR-01 Election Systems and Software EVS 6.0.4.2 California Certification Accessibility, Usability, & Privacy Test Report Sitting in a chair Using headphones Audio on Blank Screen Privacy option used Universal Voting Console (UVC) used Observations included: Volunteer Two found the review summary screen to be too confusing and was unsure how to operate the screen. Volunteer Two was unable to find the space bar button on the onscreen keyboard during the write-in process. Volunteer Two did not notice the right and left buttons on the UVC, stating that the labels did not match the audio instructions. Volunteer Two enjoyed the size of the UVC device compared to the keypad and the ability to vote quickly and accurately. Volunteer Two felt that the audio could be clearer. DS200 Initial Configuration: Device setup in standard configuration Observations included: Volunteer Two felt that the DS200 device was satisfying to use, with the insertion slot being easier to find and insert a card compared to the other systems tested. Volunteer Two Summary Volunteer two found the systems to be satisfying to use, especially the ExpressVote XL. The volunteer found the large size of the UVC on the ExpressVote XL to be better than the keypad of the ExpressVote. Overall, they felt that the system ensured privacy while voting, and that they were confident in the accuracy of their vote being recorded all while done in a timely manner. Volunteer Three ExpressVote Initial Configuration: Accessibility, Usability, and Privacy Test Report v1.3 Page 8 of 13 Report Number ESS-18S52059-AUPTR-01 Election Systems and Software EVS 6.0.4.2 California Certification Accessibility, Usability, & Privacy Test Report Front approach voting booth privacy configuration Sitting in a wheelchair Using headphones Audio on Blank Screen Privacy option used Keypad used Observations included: Volunteer Three had trouble understanding how to move to the next page when on the two-page proposition contest due to repeating audio. Volunteer Three was not able to use the zoom, contrast, or other screen options using the keypad. Volunteer Three did not like having to go through the entire alphabet to access the space bars and previous letters when using the keypad. Volunteer Three felt that the privacy blockers on the sides of the device did not provide adequate privacy. User voted on a write-in candidate using the keypad to type in a name ExpressVote XL Initial Configuration: Front approach voting booth privacy configuration Sitting in a wheelchair Using headphones Audio on Blank Screen Privacy option used UVC used Observations included: Volunteer Three found that the ExpressVote XL was too high to use when seated in a wheelchair. Volunteer Three found it difficult to scroll through the entire alphabet using the UVC and onscreen keyboard for the write-in candidates. Volunteer Three liked the fact that the printed ballots ejected from the ballot chute were easily accessible. Accessibility, Usability, and Privacy Test Report v1.3 Page 9 of 13 Report Number ESS-18S52059-AUPTR-01 Election Systems and Software EVS 6.0.4.2 California Certification Accessibility, Usability, & Privacy Test Report Volunteer Three liked having all of the contests and candidates on one screen during the vote summary screen. Volunteer Three found that the screen blackout option was easily visible and easy to comprehend. DS200 Initial Configuration: Device setup in standard configuration Observations included: Volunteer Three found that the DS200 was too high to use when seated in a wheelchair. Volunteer Three enjoyed the feature that ballots can be inserted in any orientation. Volunteer Three Summary While the user felt that their vote was private and accurately recorded, it wasn’t as easy to use independently as the current voting system in Sacramento. Additionally, they felt some confusion at the device operations and instructions that were given. From the wheelchair position, the devices felt too high and too much force was necessary to properly press the buttons on the UVC. Volunteer Four ExpressVote Initial Configuration: Front approach voting booth privacy configuration Sitting in a chair Using headphones Audio on Blank Screen Privacy option used Keypad used Observations included: Volunteer Four utilized both the keypad and headphones attachments. Volunteer Four found that the ExpressVote was very intuitive, easy to use, and did not have any issues with the device. Accessibility, Usability, and Privacy Test Report v1.3 Page 10 of 13 Report Number ESS-18S52059-AUPTR-01 Election Systems and Software EVS 6.0.4.2 California Certification Accessibility, Usability, & Privacy Test Report ExpressVote XL Initial Configuration: Front approach voting booth privacy configuration Sitting in a chair Using headphones Audio on Blank Screen Privacy option used Universal Voting Console (UVC) used Observations included: Volunteer Four utilized both the keypad and headphones attachments. Volunteer Four felt that the UVC was too complicated to use. Volunteer Four felt that the audio speech sounded sluggish when listening to the device. DS200 Initial Configuration: Device setup in standard configuration Observations included: Volunteer Four found it difficult to find the correct ballot insertion slot because the two slots on the device were not clearly marked as to where the activation ballots should be inserted. Volunteer Four Summary Volunteer Four found the system to privately, accurately and easily record their vote, with a preference of the ExpressVote over the ExpressVote XL. The keypad assistive device was found to be satisfying to use, while the UVC was too big and clunky. Audio speech could sound sluggish (this can be corrected during the election creation) Volunteer Five ExpressVote Initial Configuration: Front approach voting booth privacy configuration Accessibility, Usability, and Privacy Test Report v1.3 Page 11 of 13 Report Number ESS-18S52059-AUPTR-01 Election Systems and Software EVS 6.0.4.2 California Certification Accessibility, Usability, & Privacy Test Report Sitting in a chair Audio on Observations included: Volunteer Five did not have any issues with the device and felt the device was easy to use as they had previous experience with this device. ExpressVote XL Initial Configuration: Front approach voting booth privacy configuration Sitting in a chair Using headphones Audio on Observations included: Volunteer Five did not have any issues with the device. Volunteer Five noted that when changing the font size, the candidate names would disappear. They believed it could be confusing when toggling back and forth between the larger and smaller fonts. DS200 Initial Configuration: Device setup in standard configuration Observations included: Volunteer Five felt very secure using the device. They stated that they could hear the ballot drop and believed the case looked secure. Overall, Volunteer Five felt very confident in the device. Volunteer Five Summary Volunteer Five has had previous experience with this voting system before and rated an extremely satisfying experience in the system privately, accurately and easily recording their vote. Some minor technical issues were reported with glare and increase font size causing parties to be placed on a different page requiring the user to toggle back and forth but overall they viewed the system as very secure. Accessibility, Usability, and Privacy Test Report v1.3 Page 12 of 13 Report Number ESS-18S52059-AUPTR-01 Election Systems and Software EVS 6.0.4.2 California Certification Accessibility, Usability, & Privacy Test Report Evaluation of Testing This section provides summary lists of the issues identified by volunteers during testing and any suggestions provided by the volunteers. Issues Identified Issues identified during testing included: When performing a write-in on the ExpressVote, the audio for the N button and the M button sounded similar to one volunteer. For the ExpressVote XL, three volunteers had trouble pressing the buttons on the UVC device. One volunteer found that the DS200 device was too high to use when seated in a wheelchair. Suggestions Several suggestions were made by the volunteers during the sessions as follows: Two volunteers suggested that the audio could be clearer and more concise on both the ExpressVote as well as the ExpressVote XL. One volunteer noted that the review screen for the ExpressVote could be more comprehensive, as they believed that not enough information was present. One volunteer suggested adding the ability to adjust the screen angle to remove glare on the ExpressVote device. Final Considerations The consensus of the volunteers was that they felt the technologies implemented for accessibility and usability were easy to use, but to varying degrees. All volunteers had one device that they preferred using for various reasons. As directed by the California Secretary of State, this accessibility, usability, and privacy testing report does not include any recommendation as to whether or not the system should be approved. End of Certification Test Report Accessibility, Usability, and Privacy Test Report v1.3 Page 13 of 13 Report Number ESS-18S52059-AUPTR-01