Consultant's Accessibility, Usability, and Privacy Test Report

Dominion Democracy Suite 5.10 Accessibility, Usability, and Privacy Test Report CDV-19014-AUPTR-01 Vendor Name Dominion Voting Systems Vendor System Democracy Suite 5.10 Prepared by: SLI ComplianceSM 4720 Independence St. Wheat Ridge, CO 80033 303-422-1566 www.SLICompliance.com Accredited by the Election Assistance Commission (EAC) for Selected Voting System Test Methods or Services Dominion Democracy Suite 5.10 California Certification Accessibility, Usability & Privacy Test Report Copyright 2019 SLI ComplianceSM, a Division of Gaming Laboratories International, LLC Revision History Date Release Author Revisions 8/5/2019 v1.0 M. Santos Initial Release; 8/21/2019 v2.0 M. Santos Updates for CASOS comments Disclaimer The Certification Test results reported herein must not be used by the client to claim product certification, approval, or endorsement by NVLAP, NIST, or any agency of the Federal Government. Results herein relate only to the items tested. Trademarks • SLI is a registered trademark of SLI Compliance, a Division of Gaming Laboratories International, LLC. • All other products and company names are used for identification purposes only and may be trademarks of their respective owners. The tests referenced in this document were performed in a controlled environment using specific systems and data sets and results are related to the specific items tested. Actual results in other environments may vary. Opinions and Interpretations There are no SLI opinions or interpretations included in this report. California Certification AUP Test Report v2.0 Page 2 of 14 Report Number CDV-19014-AUPTR-01 Dominion Democracy Suite 5.10 California Certification Accessibility, Usability & Privacy Test Report TABLE OF CONTENTS INTRODUCTION ................................................................................................................................... 4 REFERENCES....................................................................................................................................... 4 SYSTEM OVERVIEW ........................................................................................................................... 4 SCOPE OF THE DOMINION DEMOCRACY SUITE 5.10 VOTING SYSTEM ..................................................... 4 SYSTEM TOPOLOGY DIAGRAM .............................................................................................................. 5 CERTIFICATION TEST RESULTS SUMMARY ................................................................................... 6 TESTING SUMMARY .............................................................................................................................. 6 Volunteer One ................................................................................................................................ 7 Volunteer Two ................................................................................................................................ 8 Volunteer Three ........................................................................................................................... 10 Volunteer Four ............................................................................................................................. 11 EVALUATION OF TESTING ................................................................................................................... 12 Volunteer Originated Issues ........................................................................................................ 12 Volunteer Comments/Suggestions .............................................................................................. 13 Final Considerations .................................................................................................................... 14 California Certification AUP Test Report v2.0 Page 3 of 14 Report Number CDV-19014-AUPTR-01 Dominion Democracy Suite 5.10 California Certification Accessibility, Usability & Privacy Test Report Introduction SLI Compliance is submitting this test report as a summary of the certification testing efforts for the Dominion Democracy Suite 5.10 voting system. The purpose of this document is to provide an overview of the certification testing and the findings of the testing effort for the Dominion Democracy Suite 5.10 voting system. This effort included Accessibility, Usability, and Privacy testing of the Dominion Democracy Suite 5.10 voting system’s ImageCast Evolution and ImageCast X components. References • California Voting System Standards (CVSS) System Overview Scope of the Dominion Democracy Suite 5.10 Voting System This section provides a description of the scope of Dominion Democracy Suite 5.10 voting system components involved in this test: • ImageCast Evolution (ICE) firmware/hardware, polling place • ImageCast X (ICX) firmware/hardware, polling place The Dominion Democracy Suite ICE system employs a precinct-level optical scan ballot counter (tabulator) in conjunction with an external ballot box. This tabulator is designed to mark and/or scan paper ballots, interpret voting marks, communicate these interpretations back to the voter (either visually through the integrated LCD display or audibly via integrated headphones) and, upon the voter’s acceptance, deposit the ballots into the secure ballot box. In addition to scanning and tabulating marked paper ballots, the ImageCast Evolution is also a ballot-marking device for voters with disabilities. The Democracy Suite ICX ballot marking platform is used for the creation of paper Electronic Mobile Ballots. These ballots are later scanned and tabulated by the ICC optical ballot counter and/or scanned, verified, and cast by the ICE. California Certification AUP Test Report v2.0 Page 4 of 14 Report Number CDV-19014-AUPTR-01 Dominion Democracy Suite 5.10 California Certification Accessibility, Usability & Privacy Test Report System Topology Diagram California Certification AUP Test Report v2.0 Page 5 of 14 Report Number CDV-19014-AUPTR-01 Dominion Democracy Suite 5.10 California Certification Accessibility, Usability & Privacy Test Report Certification Test Results Summary Testing Summary An election was run utilizing: • One ICX Prime polling place device • One ICE polling place device Four volunteers were used to evaluate each device. Their physical abilities were varied, primarily by differing levels of fine motor control abilities. Volunteer One was in a motorized wheelchair and was not able to raise arms above chest. Volunteer Two was partially blind, but otherwise fully functioning. Volunteer Three was blind, but otherwise fully functioning. Volunteer Four was vision impaired, but otherwise fully functioning. The sessions were conducted with Dominion personnel acting as poll workers and the volunteers voting on the Dominion devices. When the volunteers arrived, they were given a quick briefing on the testing and the devices. The sessions were a free form, ad hoc test where the volunteer was able to vote a ballot in any manner that they chose. The volunteers voted on both the ICE device and the ICX Prime device. The volunteer provided feedback both real-time as they were voting, as well as completing an Accessibility Test survey for each device. Additionally, observations were made as each volunteer navigated their way through the process. California Certification AUP Test Report v2.0 Page 6 of 14 Report Number CDV-19014-AUPTR-01 Dominion Democracy Suite 5.10 California Certification Accessibility, Usability & Privacy Test Report Volunteer One ICX Prime Initial Configuration: • Front approach voting booth privacy configuration. • Motorized wheelchair. • Used Accessible Tactile Interface (ATI) and headphones. • Audio on, display on. Observations included: • Volunteer increased the volume and rate of speech. • Volunteer successfully voted a write-in with the ATI. • Volunteer could not get to the "Language, Text Size, Audio, View and More" buttons on his/her own. • To get to the "Language, Text Size, Audio, View and More" button, the contest header must first be displayed. • Volunteer turned off the visual about halfway through marking the ballot. • Volunteer successfully turned off the privacy shade but noticed that they didn't get instruction to deselect the option. • Volunteer commented that volume doesn't work well. • One voice can be adjusted but not all voices can be adjusted. • Volunteer commented that the rate worked well. • Volunteer commented that when adjusting the volume, the instruction started over and that was a little frustrating. • Volunteer commented that it was nicer that both sets of arrow buttons did the same thing throughout most of the ballot. • Volunteer commented that the "Review" button is visually on the left but the right button needs to be selected to get there. ICE Initial Configuration: • Front approach privacy configuration. • Motorized wheelchair. • Used ATI and headphones. • Audio on, display on. Observations included: • Volunteer cannot reach high enough to vote with the touchscreen. California Certification AUP Test Report v2.0 Page 7 of 14 Report Number CDV-19014-AUPTR-01 Dominion Democracy Suite 5.10 California Certification Accessibility, Usability & Privacy Test Report • Volunteer is concerned the touchscreen is higher than the regulation allows. • Volunteer used the ATI with headphones to vote. • Volunteer was able to feed ballot by himself/herself. • Volunteer increased rate. • Volunteer asked if the visual could be turned off during the voting process. • Volunteer successfully voted a write-in with the ATI. • Volunteer commented that while using the write-in screen, the audio did not explain how to accept the entry. • Volunteer commented that there are too many options on every contest. The language that is used is confusing at times. • Volunteer commented that there are too many options on the visual screen as well. • Volunteer commented that when the "Mark" button was selected, instead of returning to marking/selecting candidates as thought, the ballot was cast. • Volunteer started a second voting session with audio only. • Volunteer commented that it was still a lot to listen to but maybe a blind person would be more adapted. • Volunteer commented that audio has too many adjectives for a button (right green paddle). A blind person won't know the color of the button. • Volunteer commented that it's manageable. • Volunteer commented that there are too many audio words on the review screen. • Volunteer commented that this machine is very frustrating. • Volunteer commented that changing a selection from the review screen was complicated. Once selection was changed and viewer was returned to the review screen, the screen returned to the starting contest, not the contest that they had been currently reviewing. Volunteer One Summary Volunteer One was surveyed after the testing and responded that while there were issues with navigation, overall they felt the usability, accessibility, and privacy features of the system met expectations for being able to cast a ballot. Volunteer Two ICX Prime Initial Configuration: • Front approach voting booth privacy configuration. • Sitting in a chair. California Certification AUP Test Report v2.0 Page 8 of 14 Report Number CDV-19014-AUPTR-01 Dominion Democracy Suite 5.10 California Certification Accessibility, Usability & Privacy Test Report • Using headphones. • Audio on, display on. Observations included: • Volunteer would like to be able to drag the volume and rate with their finger. • Volunteer had a hard time changing their choice. • Volunteer wanted to listen to other candidates after selecting a candidate but was unable to. • Volunteer commented that the volume level was different for the reading of the measure. It was too quiet. • Volunteer commented that the audio for the measure is a different voice than the rest of the ballot. • Volunteer commented that they couldn't read the write-in option. It was too small even with "Big" text size. • Volunteer commented that there needed to be more physical privacy for a partially blind voter. ICE Initial Configuration: • Front approach privacy configuration. • Sitting in a chair. • Using headphones. • Audio on, display on. Observations included: • Volunteer commented that the instructions were not read to them. • Volunteer commented that the "Scroll Down" was too small to read for partially blind even on "Big" text size. • Volunteer commented that he/she didn't know a selection was made because the selection scrolled off screen. • Volunteer had a hard time scrolling up the candidate list because the scroll down in the contest header looked to be the same on-screen button. • Volunteer tried to do a write-in but didn't double tap the letters so never entered anything. Volunteer understood that the write-in was not entered and selected someone else. • Volunteer commented that the articulation was not as clear with the ICE as it was with the ICX. • Volunteer commented that they would like the same hand gestures that you use on your smartphone. Currently it isn't user friendly. California Certification AUP Test Report v2.0 Page 9 of 14 Report Number CDV-19014-AUPTR-01 Dominion Democracy Suite 5.10 California Certification Accessibility, Usability & Privacy Test Report • Volunteer commented that the ICX is more user friendly than the ICE. • Volunteer commented that there is a clicking noise in the audio that might be distracting. • Volunteer tried to do a write-in again after explaining to them that each letter needed to be tapped twice. • Volunteer could not determine how to print/cast ballot. • Volunteer commented that the privacy around the ICE should also be around the ICX. Volunteer Two Summary Volunteer Two was surveyed after the testing and responded that they felt the usability and privacy features of the system met expectations for being able to cast a ballot. Volunteer Two did not feel the ICE met accessibility requirements. Volunteer Three ICX Prime Initial Configuration: • Front approach voting booth privacy configuration. • Sitting in a chair. • Using ATI. • Audio on, display on. Observations included: • Volunteer commented that they moved faster through the ICX due to hearing that the previous volunteer had to double click and they were more familiar with the ATI after using it on the ICE. ICE Initial Configuration: • Front approach privacy configuration. • Sitting in a chair. • Using ATI controller. • Audio on, display on. Observations included: • Volunteer could not get from the audio instructions to the first contest. They felt that the audio instructions were unclear. • Volunteer needed to press the right hand button. California Certification AUP Test Report v2.0 Page 10 of 14 Report Number CDV-19014-AUPTR-01 Dominion Democracy Suite 5.10 California Certification Accessibility, Usability & Privacy Test Report • Volunteer commented that the audio is confusing when the first contest is considered contest two of nine, not contest one of nine. • Volunteer commented that they shouldn't be given an option to review if they hadn't made any choice yet. • Volunteer commented that they don’t want to keep going to the heading. • Volunteer commented that they felt that the help button was part of the ATI and not an actual button. • Volunteer commented that they would have abandoned his/her voting session due to not being able to navigate. • Volunteer did abandon their voting session. Volunteer Three Summary Volunteer Three was surveyed after the testing, and responded that while there were issues with navigation, overall they felt the usability, accessibility, and privacy features of the system met expectations for being able to cast a ballot. Volunteer Four ICX Prime Initial Configuration: • Front approach voting booth privacy configuration. • Sitting in a chair. • Used ATI with headphones • Audio on, display on. Observations included: • Volunteer Four marked and printed one ballot using headphones, ATI, and the touchscreen. No issues were encountered. • Volunteer commented that the audio output was clear. • Volunteer commented that the voting process (speed) was good. ICE Initial Configuration: • Front approach privacy configuration. • Sitting in a chair. • Display on. Observations included: • Volunteer voted one ballot on the touch screen and had some trouble with locating “Vote for 2” instructions due to the small text. California Certification AUP Test Report v2.0 Page 11 of 14 Report Number CDV-19014-AUPTR-01 Dominion Democracy Suite 5.10 California Certification Accessibility, Usability & Privacy Test Report • Volunteer commented that the text was too small, "Vote for 2" instructions hard to read. • Volunteer commented that adding different colors to the screen would be helpful, some colors / contrast were too bright. Volunteer Four Summary Volunteer Four was surveyed after the testing, and responded that they felt the usability, accessibility, and privacy features of the system met expectations for being able to mark, print, and cast a ballot. Evaluation of Testing This section reviews issues encountered as well as comments and suggestions. Summaries of the testing and volunteer responses are provided below. Volunteer Originated Issues Issues encountered by the volunteers during testing included: ICE • The touch screen seemed to be higher than regulation allows. • When performing a write-in, the audio did not tell voter how to accept the entry. • There are too many options for every contest. Given the screen size of the ICE device, too much content was being abbreviated and displayed. This made it very difficult for the voter to navigate the ballot. • Text size too small to read for partially blind even on "Big" text size. • When volunteer voted a contest, moved to next contest, then returned to previous contest, they did not see the selection previously made as that choice was off screen. As a result, they attempted to make a selection of a visible choice but were unable to do so, not realizing that they were attempting an overvote because the selection was scrolled off screen. ICX Prime • The volume level was different for the reading of the measure name versus the content of the measure. Volume was too low for reading of the measure. Volume should remain the same through the course of the voting session, unless the voter explicitly changes it. • The write-in label in the write-in field was too small of a font. Even selecting "Big" text size font, did not change the font size. California Certification AUP Test Report v2.0 Page 12 of 14 Report Number CDV-19014-AUPTR-01 Dominion Democracy Suite 5.10 California Certification Accessibility, Usability & Privacy Test Report Volunteer Comments/Suggestions Several comments and suggestions were made during the sessions. Volunteer Comments ICE • ICX is more user friendly than the ICE. • When voter selected the "Mark" button, they thought it was going to return to marking/selecting candidates but instead the ballot was cast. • There are too many audio words on the review screen. • Voter had a hard time scrolling up the candidate list because the scroll down in the contest header looked to be the same on-screen button. • The articulation was not as clear with the ICE as it was with the ICX. • Voter would like the same hand gestures that you use on your smartphone. Currently it isn't user friendly. • Changing a selection from the review screen was complicated. Once selection was changed and viewer was returned to the review screen, the screen returned to the starting contest, not the contest that they had been currently reviewing. • Concerns that the audio has too many adjectives for a button (right green paddle). A blind person won't know the color of the button. ICX Prime • To get to the "Language, Text Size, Audio, View and More" button, the contest header must first be displayed. • Volume doesn't work well. • One voice can be adjusted but not all voices can be adjusted. • Audio output was clear. • Voting process speed was good. • The "Review" button is visually on the left, but the right button is used to select it. • Voter wanted to listen to other candidates after selecting a candidate but was unable to. • There is a clicking noise in the audio that might be distracting. • Voter moved faster through the ICX due to hearing that the previous volunteer had to double click and he/she was more familiar with the ATI after using it on the ICE. California Certification AUP Test Report v2.0 Page 13 of 14 Report Number CDV-19014-AUPTR-01 Dominion Democracy Suite 5.10 California Certification Accessibility, Usability & Privacy Test Report Volunteer Suggestions ICE • The language that is used for audio instructions is confusing at times. ICX Prime • Adding different colors to the screen would be helpful, some colors / contrast were too bright. • The privacy screen around the ICE should also be around the ICX, in order to increase the voter’s physical privacy. Final Considerations The consensus of the volunteers was that they felt the technologies implemented for accessibility and usability improved the experience for voters that are most in need of them although the ICE could have simpler navigation. From a privacy point of view, one volunteer expressed concern for partially blind voters on the ICX. As directed by the California Secretary of State, this accessibility, usability, and privacy testing report does not include any recommendation as to whether or not the system should be approved. End of AUP Test Report California Certification AUP Test Report v2.0 Page 14 of 14 Report Number CDV-19014-AUPTR-01