Consultant's Accessibility, Usability, and Privacy Test Report

Dominion Democracy Suite 5.19 Voting System Accessibility, Usability, and Privacy Test Report v1.0 for California Secretary of State CDV-19014-AUPTR-01 Vendor Name Dominion Voting Systems Vendor System Democracy Suite 5.19 Prepared by: SLI ComplianceSM 4720 Independence St. Wheat Ridge, CO 80033 303-422-1566 www.SLICompliance.com Accredited by the Election Assistance Commission (EAC) for VSTL status. California Secretary of State Dominion Democracy Suite 5.19 California Certification Accessibility, Usability & Privacy Test Report Copyright © 2025 SLI ComplianceSM, a Division of Gaming Laboratories International, LLC Revision History Date Release Author Revisions 7/10/2024 v1.0 E. Bickley Initial Release; Disclaimer The Certification Test results reported herein must not be used by the client to claim product certification, approval, or endorsement by NVLAP, NIST, or any agency of the Federal Government. Results herein relate only to the items tested. Trademarks • SLI is a registered trademark of SLI Compliance, a Division of Gaming Laboratories International, LLC. • All other products and company names are used for identification purposes only and may be trademarks of their respective owners. The tests referenced in this document were performed in a controlled environment using specific systems and data sets and results are related to the specific items tested. Actual results in other environments may vary. Opinions and Interpretations There are no SLI opinions or interpretations included in this report. California Certification AUP Test Report v1.0 Page 2 of 10 Report Number CDV-19014-AUPTR-01 California Secretary of State Dominion Democracy Suite 5.19 California Certification Accessibility, Usability & Privacy Test Report TABLE OF CONTENTS INTRODUCTION ................................................................................................................................... 4 REFERENCES....................................................................................................................................... 4 SYSTEM OVERVIEW ........................................................................................................................... 4 SCOPE OF THE DOMINION DEMOCRACY SUITE 5.19 VOTING SYSTEM ..................................................... 4 CERTIFICATION TEST RESULTS SUMMARY ................................................................................... 5 TESTING SUMMARY .............................................................................................................................. 5 Volunteer One ................................................................................................................................ 5 Volunteer Two ................................................................................................................................ 6 Volunteer Three ............................................................................................................................. 7 Volunteer Four ............................................................................................................................... 7 Volunteer Five ................................................................................................................................ 8 EVALUATION OF TESTING ..................................................................................................................... 9 Volunteer Originated Issues .......................................................................................................... 9 Volunteer Comments/Suggestions ................................................................................................ 9 Final Considerations .................................................................................................................... 10 California Certification AUP Test Report v1.0 Page 3 of 10 Report Number CDV-19014-AUPTR-01 California Secretary of State Dominion Democracy Suite 5.19 California Certification Accessibility, Usability & Privacy Test Report Introduction SLI Compliance is submitting this test report as a summary of the certification testing efforts for the Dominion Democracy Suite 5.19 (Dominion DS 5.19) voting system. The purpose of this document is to provide an overview of the certification testing and the findings of the testing effort for the Dominion DS 5.19 voting system. This effort included Accessibility, Usability, and Privacy testing of the Dominion DS 5.19 voting system’s ImageCast Evolution and ImageCast X components. References • California Voting System Standards (CVSS) System Overview Scope of the Dominion DS 5.19 Voting System This section provides a description of the scope of Dominion DS 5.19 voting system components involved in this test: • ImageCast Evolution (ICE) firmware/hardware, polling place • ImageCast X (ICX) firmware/hardware, polling place The Dominion Democracy Suite ICE system is designed to mark and/or scan paper ballots, interpret voting marks, communicate these interpretations back to the voter (either visually through the integrated LCD display or audibly via integrated headphones) and, upon the voter’s acceptance, deposit the ballots into the secure ballot box. The Dominion Democracy Suite ICX ballot marking platform is used for the creation of paper ballots. These ballots are later scanned and tabulated by the ICC optical ballot counter and/or scanned, verified, and cast by the ICE. The ICX is also a ballot-marking device for voters with disabilities, providing ADA capabilities. California Certification AUP Test Report v1.0 Page 4 of 10 Report Number CDV-19014-AUPTR-01 California Secretary of State Dominion Democracy Suite 5.19 California Certification Accessibility, Usability & Privacy Test Report Certification Test Results Summary Testing Summary An election was run utilizing: • One ICX Prime polling place device • One ICX Classic polling place device • One ICE polling place device Five volunteers were used to evaluate each device. Their physical abilities were varied, primarily by differing levels of fine motor control abilities. Volunteer One was in a motorized wheelchair and was not able to raise arms above chest. Volunteer Two was partially blind, but otherwise fully functioning. Volunteer Three was in a motorized wheelchair and was not able to raise arms above chest. Volunteer Four was vision impaired, but otherwise fully functioning. Volunteer Five was in a motorized wheelchair. When the volunteers arrived, they were given a quick briefing on the testing and the devices. The sessions were a free form, ad hoc test, where the volunteers were able to vote in any manner that they chose. The volunteers provided feedback both in real time as they were voting, as well as completing an Accessibility Test survey for each device. Additionally, observations were made as each volunteer navigated their way through the process. Volunteer One Table 1 – Volunteer One Survey Yes No N/A If used, was the display clear and readable? X If used, what was the headset audio clear and easy to understand? X If used, were the assistive devices easy to reach and use? X Do you feel the voting method was private? X Do you feel this accessible voting device allows you to vote independently? X Do you feel confident that your vote was recorded accurately? X Were the voting instructions clear? X California Certification AUP Test Report v1.0 Page 5 of 10 Report Number CDV-19014-AUPTR-01 California Secretary of State Dominion Democracy Suite 5.19 California Certification Accessibility, Usability & Privacy Test Report Volunteer One Summary Volunteer One was surveyed and expressed satisfaction with the system, noting that voting became easier once they understood how to use the yellow buttons on the accessible tactile interface (ATI) device. They recommended that the text instructions be updated to highlight the yellow button feature prominently within the on-screen guidelines. Additionally, they emphasized the importance of this feature and suggested that poll workers be specifically trained on it to better assist and instruct voters effectively. Volunteer Two Table 2 – Volunteer Two Survey Yes No N/A If used, was the display clear and readable? X If used, what was the headset audio clear and easy to understand? X If used, were the assistive devices easy to reach and use? X Do you feel the voting method was private? X Do you feel this accessible voting device allows you to vote independently? X Do you feel confident that your vote was recorded accurately? X Were the voting instructions clear? X Volunteer Two Summary Volunteer Two, who utilized audio and an ATI device during testing, was surveyed and expressed satisfaction with the system. They noted that adjusting the volume and speed was frequently necessary for optimal audio clarity. Initially, the distinction between the yellow and blue ATI arrows was confusing, but they managed to understand it as they continued using the system. They acknowledged the availability of the privacy screen option. However, the volunteer found the accessibility controls somewhat challenging, particularly due to the absence of an Enter Key; they initially mistook the green bar for this function. They noted it took some time to adapt to making selections without an Enter Key, which led to confusion. Additionally, they expressed that navigating the candidate list was difficult as selections often inadvertently returned them to the header. During a write-in attempt, the volunteer did not double-tap the letters, resulting in no entry. Upon realizing the write-in had not been processed, they opted to select another candidate instead. California Certification AUP Test Report v1.0 Page 6 of 10 Report Number CDV-19014-AUPTR-01 California Secretary of State Dominion Democracy Suite 5.19 California Certification Accessibility, Usability & Privacy Test Report Volunteer Three Summary Table 3 – Volunteer Three Survey Yes No N/A If used, was the display clear and readable? X If used, what was the headset audio clear and easy to understand? X If used, were the assistive devices easy to reach and use? X Do you feel the voting method was private? X Do you feel this accessible voting device allows you to vote independently? X Do you feel confident that your vote was recorded accurately? X Were the voting instructions clear? X Volunteer Three Summary Volunteer Three, who utilized audio and an ATI device during testing, reported an improvement in overall satisfaction compared to previous experiences. They highlighted that the audio clarity has improved from past models, especially for non- text-to-speech components which were clear and distinct. They also noted that clarity diminished when the machine converted text into speech, particularly during contests, and further degraded when the audio was sped up. This made it challenging to understand last names, which vary significantly. The volunteer also found the repeated button presses required to navigate between contests cumbersome and described the initial instructions for write-ins as confusing, specifically the directions to select letters twice. They suggested that write-ins could be simplified and recommended clearer instructions to press the select button upon reaching the desired letters. Additionally, they expressed frustration with the interface returning them to the header, disrupting their navigation experience. Volunteer Four Table 4 – Volunteer Four Survey Yes No N/A If used, was the display clear and readable? X If used, what was the headset audio clear and easy to X understand? If used, were the assistive devices easy to reach and use? X Do you feel the voting method was private? X Do you feel this accessible voting device allows you to vote X independently? Do you feel confident that your vote was recorded accurately? X Were the voting instructions clear? X California Certification AUP Test Report v1.0 Page 7 of 10 Report Number CDV-19014-AUPTR-01 California Secretary of State Dominion Democracy Suite 5.19 California Certification Accessibility, Usability & Privacy Test Report Volunteer Four Summary Volunteer Four, who utilized audio and an ATI device during testing, was surveyed after the testing and expressed satisfaction with the system. The volunteer noted that the audio instructions were louder than the audio for reading contest names, and they found the ATI conveniently located and user-friendly. Changing selections was reported as straightforward. However, the volunteer did not see the utility of the left and right yellow buttons and expressed hesitancy about using the Help button, fearing it might trigger an overload of unrelated audio information. They found the process of selecting letters for write-ins straightforward but noted the absence of a direct option to accept the completed text. Instead, they had to navigate through each letter to find the 'Accept' command at the top of the interface. The volunteer also mentioned that at times, the audio was excessive and repetitive, leading to confusion about certain pop-up messages regarding additional options. Once they realized they could proceed without listening to the complete audio instructions, they found voting to be easier, though the repetitive audio remained a point of concern. Overall, they concluded that the system was easy to use. Volunteer Five Table 1 – Volunteer One Survey Yes No N/A If used, was the display clear and readable? X If used, what was the headset audio clear and easy to X understand? If used, were the assistive devices easy to reach and use? X Do you feel the voting method was private? X Do you feel this accessible voting device allows you to vote X independently? Do you feel confident that your vote was recorded accurately? X Were the voting instructions clear? X Observations included: Volunteer Five Summary Volunteer Five, who utilized the touchscreen interface during testing, was surveyed after the testing and expressed satisfaction with the system, suggesting improvements to the audio and visual prompts could enhance the experience. They noted that making selections required pressing an icon twice, but there was no prompt indicating that a double press was necessary to confirm a choice. They also missed the review button at the bottom of the screen and sought assistance for reviewing their selections. The volunteer recommended that adding clear guidance on how to review the ballot at the end of the voting process would be beneficial. California Certification AUP Test Report v1.0 Page 8 of 10 Report Number CDV-19014-AUPTR-01 California Secretary of State Dominion Democracy Suite 5.19 California Certification Accessibility, Usability & Privacy Test Report Additionally, they highlighted that the method of voting used during the exercise maintained voter privacy. Evaluation of Testing This section reviews issues encountered as well as comments and suggestions. Summaries of the testing and volunteer responses are provided below. Volunteer Originated Issues Issues encountered by the volunteers during testing included: • Volunteer when reaching for the ATI, the ATI device slid and got stuck and caused an error. • There is a noticeable difference in the volume, speed, and clarity between the system instructions and the text to speech reading of the ballot. • The double select required to select a candidate or contest option can be confusing or take getting used to. Volunteer Comments/Suggestions Several comments and suggestions were made during the sessions. Volunteer Comments ICX Classic/Prime (same interface) • Volunteer commented that once they figured out how to use the yellow buttons voting was easier. • Volunteers commented that accessibility controls were a little confusing as there was no Enter Key and initially thought the green bar was the Enter Key. • Volunteer commented that it was clunky to have to press the button repeatedly to get to the next contest. • Volunteer commented that when they first listened to instructions on write- ins, they did not understand the instructions about select letters twice, which were confusing. • Volunteer commented that selections required pressing an icon twice and noted the absence of any prompt to indicate that a double press was necessary to confirm a choice. • Volunteer selections returned them to the header frequently and volunteer commented that they did not want to keep going back up to the header. California Certification AUP Test Report v1.0 Page 9 of 10 Report Number CDV-19014-AUPTR-01 California Secretary of State Dominion Democracy Suite 5.19 California Certification Accessibility, Usability & Privacy Test Report • Volunteer commented the write-ins were easy to select letters, but when they finished there was not an option to accept. Volunteer had to go through each letter to find ‘Accept’ at the top. ICE • All Volunteers felt their ballot had been securely cast. Volunteer Suggestions ICX Classic/Prime (same interface) • The text instructions should be updated to include the yellow button update and should be emphasized in the on-screen instructions and with poll workers. • Guidance on how to review your ballot at the end would be helpful. • Better audio and visual prompts regarding the need to double select letters when during the write-in process would be helpful. • Reduction in repetitive audio instructions would be helpful. • Improvement to the text to speech technology would be helpful. ICE • None Final Considerations The consensus of the volunteers was that they felt the technologies implemented for accessibility and usability improved the experience for voters. As directed by the California Secretary of State, this accessibility, usability, and privacy testing report does not include any recommendation as to whether or not the system should be approved. End of AUP Test Report California Certification AUP Test Report v1.0 Page 10 of 10 Report Number CDV-19014-AUPTR-01