Consultant's Accessibility Test Report

Dominion Democracy Suite 5.2 Accessibility, Usability and Privacy Test Report CDV-306-VTR-01 Prepared for: Vendor Name Dominion Voting Systems Vendor System DVS 5.2 Prepared by: SLI ComplianceSM 4720 Independence St. Wheat Ridge, CO 80033 303-422-1566 www.SLICompliance.com Accredited by the Election Assistance Commission (EAC) for Selected Voting System Test Methods or Services Test Report v1.0 Report Number CDV-306-VTR-01 Page 1 of 14 Dominion Voting Democracy Suite 5.2 California Accessibility, Usability & Privacy Test Report Copyright 2017 SLI ComplianceSM Revision History Release Author Revision Summary 1.0 M. Santos Initial Release Disclaimer The Certification Test results reported herein must not be used by the client to claim product certification, approval, or endorsement by NVLAP, NIST, or any agency of the Federal Government. Results herein relate only to the items tested. Trademarks • SLI is a registered trademark of SLI Compliance, a Division of Gaming Laboratories International, LLC. • Intel and Pentium are registered trademarks of Intel Corporation. • Microsoft, MS are registered trademarks and Internet Explorer, Windows, Visual C++, Visual Basic, VBX, ODBC, and MFC are trademarks of Microsoft Corporation. • All other products and company names are used for identification purposes only and may be trademarks of their respective owners. The tests referenced in this document were performed in a controlled environment using specific systems and data sets and results are related to the specific items tested. Actual results in other environments may vary. Opinions and Interpretations There are no SLI opinions or interpretations included in this report. Test Report v1.0 Page 2 of 14 Report Number CDV-306-FTR-01 Dominion Voting Democracy Suite 5.2 California Accessibility, Usability & Privacy Test Report TABLE OF CONTENTS INTRODUCTION ........................................................................................................................................ 4 References ........................................................................................................................................ 4 SYSTEM OVERVIEW ................................................................................................................................. 4 Scope of the Dominion Democracy Suite 5.2 Voting System ........................................................... 4 System Topology Diagram ................................................................................................................ 5 CERTIFICATION TEST RESULTS SUMMARY ................................................................................................ 6 Functional Testing Summary ............................................................................................................. 6 Volunteer One ................................................................................................................................ 7 Volunteer Two ................................................................................................................................ 8 Volunteer Three ........................................................................................................................... 10 Volunteer Four ............................................................................................................................. 11 Volunteer Five .............................................................................................................................. 12 Evaluation of Testing ....................................................................................................................... 13 Issues ........................................................................................................................................... 13 Comments/Suggestions ............................................................................................................... 13 Final Considerations .................................................................................................................... 14 Test Report v1.0 Page 3 of 14 Report Number CDV-306-FTR-01 Dominion Voting Democracy Suite 5.2 California Accessibility, Usability & Privacy Test Report Introduction SLI Compliance is submitting this test report as a summary of the certification testing efforts for the Dominion Democracy Suite 5.2 voting system. The purpose of this document is to provide an overview of the certification testing effort and the findings of the testing effort for the Dominion Democracy Suite 5.2 voting system. This effort included Accessibility, Usability, and Privacy testing of the Dominion Democracy Suite 5.2 voting system’s ICE and ICX components. References • California Voting System Standards (CVSS) System Overview Scope of the Dominion Democracy Suite 5.2 Voting System This section provides a description of the scope of Dominion Democracy Suite 5.2 voting system components: • EMS – Results Tally and Reporting (RTR) application, central count location • ImageCast Central (ICC) application, central count location • ImageCast Evolution (ICE) firmware/hardware, polling place • ImageCast X (ICX) firmware/hardware, polling place The Dominion Democracy Suite 5.2 Election Management System (EMS) represents a set of N-Tier software applications (EMS, RTR, Adjudication) for pre- voting and post-voting election project activities that are applicable to jurisdictions of various sizes and geo-political complexities. The Dominion Democracy Suite ICC system consists of a central high-speed optical scan ballot counter (tabulator) called the ICC Ballot Counter and is used for processing absentee ballots (such as vote by mail). This ballot counter unit is based on commercial off the shelf (COTS) hardware coupled with custom-made ballot processing application software. It is used for high-speed centralized scanning and counting of paper ballots. The Dominion Democracy Suite ICE system employs a precinct-level optical scan ballot counter (tabulator) in conjunction with an external ballot box. This tabulator is designed to mark and/or scan paper ballots, interpret voting marks, communicate these interpretations back to the voter (either visually through the integrated LCD display or audibly via integrated headphones) and, upon the voter’s acceptance, deposit the ballots into the secure ballot box. Test Report v1.0 Page 4 of 14 Report Number CDV-306-FTR-01 Dominion Voting Democracy Suite 5.2 California Accessibility, Usability & Privacy Test Report The Democracy Suite ICX ballot marking platform is a solution used for the creation of paper Electronic Mobile Ballots. These ballots are later scanned and tabulated by the ICC optical ballot counter and/or scanned, verified, and cast by the ICE. System Topology Diagram Test Report v1.0 Page 5 of 14 Report Number CDV-306-FTR-01 Dominion Voting Democracy Suite 5.2 California Accessibility, Usability & Privacy Test Report Certification Test Results Summary Functional Testing Summary An election was run utilizing: • 1 ICX polling place device • 1 ICE polling place device Five volunteers participated in this test to evaluate each device. Their physical concerns were varied, primarily by differing levels of fine motor control abilities. One volunteer had sight issues, which differed significantly from the others. Volunteer One used a motorized wheelchair with movable desktop and had significant fine motor skill impediments. Volunteer Two was blind, with no motor skill impediments. Volunteer Three used a wheelchair, had no use of the left arm, and reasonably good coordination with the right arm. Volunteer Four used a wheelchair with near normal fine motor skills. Volunteer Five used a motorized wheelchair and had some fine motor control impairment, but was able to use the touch screen. The sessions were a free form, ad-hoc test where the volunteer was able to vote a ballot in any manner that they chose. The volunteer provided feedback both real- time as they were voting, as well as completing a survey after the ballot had been cast. Additionally, observations were made as each volunteer navigated their way through the process. Test Report v1.0 Page 6 of 14 Report Number CDV-306-FTR-01 Dominion Voting Democracy Suite 5.2 California Accessibility, Usability & Privacy Test Report Volunteer One ICX Volunteer One used: • Side approach ICX setup • Paddles and headphones Audio and display were on. The paddle was on the table and Volunteer One used arm/hand to press, using a wide sweeping arc to manipulate the paddle. While voting with the paddle, when the left paddle button was pressed and held, letters were input multiple times. Also, the right paddle button would move several places for each press. Volunteer One commented that system should only move one spot on each press, as a voter may not have enough dexterity to remove their hand quickly enough. Volunteer One managed to get into the menu area without realizing it at first and had some trouble getting back to the contest, but did manage to after a minute. Volunteer One suggested that the press time on the paddles be a configurable setting. Some people may have a harder time releasing when they are pressing a button on the paddle. ICE Volunteer One used: • Front approach ICE setup • Paddles and headphones Audio was on; display was off. The paddle was located on Volunteer One’s desktop on the wheelchair. The wire for the headphones was moved out of the way so that Volunteer One could use their chin to press the buttons. Volunteer One found an issue that when performing a write-in, the audio referred them to the display even though the display was off. Volunteer One found an issue that the proposition name was not listed or even identified as a proposition. Volunteer retried the contest and said that it did give the name of the contest but didn’t list it as a proposition. Test Report v1.0 Page 7 of 14 Report Number CDV-306-FTR-01 Dominion Voting Democracy Suite 5.2 California Accessibility, Usability & Privacy Test Report Volunteer One found an issue on the review screen that with a binary device (sip and puff or paddles) no “Cast Ballot” option is presented. It kept cycling through the contests without ever giving an option to cast the ballot. Volunteer One Summary Volunteer One was surveyed after the testing, and responded that they felt the Usability, Accessibility, and Privacy features of the system met expectations for being able to cast a ballot. Volunteer Two ICX Volunteer Two used: • Front approach ICX setup • ATI and headphones Audio and display were on. Volunteer Two liked the reading back of name/spelling for a write-in. Volunteer Two liked the ICX better than the ICE interface. The application shut down unexpectedly about seven minutes into the session and had to be restarted. After the restart, the application shut down a second time about seven minutes into the session. Testing was transferred to second machine, which crashed about seven minutes into the session. Testing was transferred to a third machine, which also crashed about seven minutes into the session. Testing on the ICX stopped. Dominion examined the logs to research this issue and determined that, after a period of time, when using the audio and going through a number of screens, the content being displayed on the screen was conflicting with the audio presentation. This set up a race condition which resulted in the application shutting down. ICE Volunteer Two: • Sat in a chair Test Report v1.0 Page 8 of 14 Report Number CDV-306-FTR-01 Dominion Voting Democracy Suite 5.2 California Accessibility, Usability & Privacy Test Report • Used ATI and headphones Audio only selection was used, so the screen was off. Volunteer Two liked the grooves at the entrance to the paper port as it gives the voter an indication of where the paper will go. Volunteer Two liked the use of the right arrow to delete a character for a write-in. Volunteer Two said that during letter entry, the alphabet seems to jump around, that “Q, A, Z, &,….”, seemed to be jumping around. This was determined to be due to default use of a non-QWERTY keyboard. Volunteer Two suggested that QWERTY be the default, as it is the most commonly used keyboard layout, and many people are familiar with it. Volunteer Two was able to reconfigure the keyboard setting to represent a QWERTY layout. Trying to do a write-in, Volunteer Two ran into issues trying to get to the keyboard. Volunteer Two asked if there is a “repeat” button for the audio, and found it frustrating not to be able to have something repeated instead having to wait until it comes around again. Volunteer Two tended to switch back and forth from using their thumb to press buttons, to using their index finger. Volunteer Two at times felt like they were stuck. Volunteer Two ran into issues trying to delete a character while inputting a write-in. Navigated away from where they intended to be and had some difficulty getting back to the correct location. Volunteer Two commented that they would like to have the ability to skip the instructions that get repeated for every contest. Volunteer Two did like the layout of the ATI. Would like to have the layout of the key selection (QWERTY) explained initially. Likes the QWERTY layout. Volunteer Two commented that it was more intuitive to use the right arrow to go to the next contest, instead of the down arrow. Volunteer Two commented that they thought that the Space bar should be on its own line, not in a row with others. Volunteer Two commented about liking the synthesized voice and felt it was better than a human voice (clearer). Volunteer Two ran into a situation that during a write-in if you have to go into help, when coming back to the keyboard the focus is not on the keyboard. Test Report v1.0 Page 9 of 14 Report Number CDV-306-FTR-01 Dominion Voting Democracy Suite 5.2 California Accessibility, Usability & Privacy Test Report Volunteer Two commented that when in a contest, if a person could use the up arrow to hear the contest name and then the down arrow to hear the candidate list, this would be “awesome”. Volunteer Two, in a vote-for-3 contest, performed a write-in, then navigated to the next write-in option and thought that they had deleted the first write-in. Volunteer Two realized it was a vote-for-3 contest. Volunteer Two Summary Volunteer Two was surveyed after the testing, and responded that while there were issues with navigation and the issue of the ICX crashing, overall they felt the Usability, Accessibility, and Privacy features of the system met expectations for being able to cast a ballot. Volunteer Three ICX Volunteer Three used: • Front approach configuration • ATI The display was on. Volunteer Three did not encounter issues and proceeded through ballot in a matter of a few minutes, printing their ballot. Volunteer Three did tend to double tap on the screen. Volunteer Three had some trouble with the size of the keyboard on screen when doing a write-in. They did not realize that they could increase the size until someone showed them. Volunteer Three commented that even the “Big” size is too small and would like to see bigger fonts available. Volunteer Three commented that the display was too high for comfortable viewing and would like to see the keyboard display lower on the screen. ICE Volunteer Three used: • Front approach configuration • ATI The Display was on, and a table was needed to set the ATI on. Test Report v1.0 Page 10 of 14 Report Number CDV-306-FTR-01 Dominion Voting Democracy Suite 5.2 California Accessibility, Usability & Privacy Test Report Volunteer Three wondered why the ICE device required a double tap whereas the ICX only required a single tap. Note that this is due to a difference in screen technology. Volunteer Three did not encounter issues and proceeded through the ballot in a matter of a few minutes, printing their ballot. Volunteer Three Summary Volunteer Three was surveyed after the testing and responded that they felt the Usability, Accessibility, and Privacy features of the system met expectations for being able to cast a ballot. Volunteer Four ICX Volunteer Four used: • Front approach configuration • ATI The display was on. Volunteer Four voted one ballot using the touchscreen. Volunteer Four voted a second ballot using headphones and ATI. No issues were encountered. ICE Volunteer Four used: • Front approach configuration • ATI The display was on. Volunteer Four voted one ballot on the touch screen and had some trouble with the double tap that ICE needs as compared to the single tap for ICX. Volunteer Four voted a second ballot with headphones and buttons, audio only. Volunteer Four had some trouble during the write-in trying to find “Cancel”. Test Report v1.0 Page 11 of 14 Report Number CDV-306-FTR-01 Dominion Voting Democracy Suite 5.2 California Accessibility, Usability & Privacy Test Report Volunteer Four Summary Volunteer Four was surveyed after the testing, and responded that they felt the Usability, Accessibility, and Privacy features of the system met expectations for being able to cast a ballot. Volunteer Five ICX Volunteer Five used: • Front approach configuration • ATI The display was on. Volunteer Five voted one ballot using the touchscreen. No issues were encountered. Volunteer Five noted that the user can touch anywhere in the row for the choice (ICX), but with ICE had to touch in the box. Having less surface area to touch made it more difficult for people with fine motor control impairment. ICE Volunteer Five used: • Straight in configuration • Button panel on a board The display was on. Volunteer Five switched to using the touch screen. Volunteer Five used their left hand to vote and right hand to move to the next contest. Volunteer Five had some trouble with keeping their touch on the screen long enough for it to register, but worked through it. Volunteer Five felt that the touch screens between ICE and ICX were very different; ICE required much more contact than did ICX. Volunteer Five Summary Volunteer Five was surveyed after the testing and responded that they felt the Usability, Accessibility, and Privacy features of the system met expectations for being able to cast a ballot. Test Report v1.0 Page 12 of 14 Report Number CDV-306-FTR-01 Dominion Voting Democracy Suite 5.2 California Accessibility, Usability & Privacy Test Report Evaluation of Testing This section reviews issues encountered as well as comments and suggestions. Summaries of the testing and volunteer responses are provided below. Issues Issues encountered during testing included: • When performing a write-in, the audio referred the voter to the display even though the display was off. • On the review screen when using a binary device (sip and puff or paddles), no “Cast Ballot” option is presented. The voter kept cycling through the contests without ever being given option to cast the ballot. • During a write-in, if the user goes into Help, when coming back to the keyboard the focus is not on the keyboard. • The ICX application shut down unexpectedly about seven minutes into a session. Comments/Suggestions Several comments and suggestions were made during the sessions. Comments included: • The layout of the ATI was easy to understand and use. • It would be more intuitive to use the right arrow to go to next contest instead of the down arrow. • The ATI could be improved if the space bar was on its own line and not in a row with others. • For the font on the screens, even the “Big” size is too small; would like to see bigger fonts available. Suggestions included: • Use QWERTY layouts on both device types for consistency. • Have a “repeat” button for the audio so the voter can repeat a section of audio immediately rather than having to wait until it comes around again. • Have the ability to skip the instructions that get repeated in every contest. • Have the layout of the key selection (QWERTY) explained initially. Or if a non-QWERTY layout is used, its layout should be described. Test Report v1.0 Page 13 of 14 Report Number CDV-306-FTR-01 Dominion Voting Democracy Suite 5.2 California Accessibility, Usability & Privacy Test Report • When in a contest, if the voter could select “up arrow” to hear contest name and then “down arrow” to hear the candidate list it would improve the voter experience. • The display is too high for comfortable viewing. Placing the keyboard display lower on the screen would improve the voter experience. Final Considerations The consensus of the volunteers was that they felt the technologies implemented for accessibility and usability improved the experience for voters that are most in need of them. From a privacy point of view, all volunteers seemed to feel that their privacy was kept intact and none expressed any issue or concern. As per the direction given by the California Secretary of State, this accessibility, usability and privacy testing report does not include any recommendation as to whether or not the system should be approved. End of Test Report Test Report v1.0 Page 14 of 14 Report Number CDV-306-FTR-01