County of Los Angeles Vsap 2.1 Consultant's Accessibility Testing Report

Los Angeles County VSAP Tally 2.1 Voting System Accessibility, Usability, and Privacy Test Report for California Secretary of State CHI-20006-AUPTR-01 Vendor Name County of Los Angeles Vendor System VSAP Tally 2.1 Prepared by: SLI ComplianceSM 4720 Independence St. Wheat Ridge, CO 80033 303-422-1566 www.SLICompliance.com Accredited by the Election Assistance Commission (EAC) for VSTL status. Los Angeles County VSAP Tally 2.1 California Certification Accessibility, Usability, and Privacy Test Report v1.0 Copyright 2020 SLI ComplianceSM, a Division of Gaming Laboratories International, LLC Revision History Date Release Author Revisions 8/3/2020 v1.0 M. Santos Initial Release; Disclaimer The Certification Test results reported herein must not be used by the client to claim product certification, approval, or endorsement by NVLAP, NIST, or any agency of the Federal Government. Results herein relate only to the items tested. Trademarks SLI is a registered trademark of SLI Compliance, a Division of Gaming Laboratories International, LLC. All other products and company names are used for identification purposes only and may be trademarks of their respective owners. The tests referenced in this document were performed in a controlled environment using specific systems and data sets and results are related to the specific items tested. Actual results in other environments may vary. Opinions and Interpretations There are no SLI opinions or interpretations included in this report. California Certification AUP Test Report v1.0 Page 2 of 13 Report Number CA-20006-AUPTR-01 Los Angeles County VSAP Tally 2.1 California Certification Accessibility, Usability, and Privacy Test Report v1.0 TABLE OF CONTENTS INTRODUCTION ................................................................................................................................... 4 REFERENCES ....................................................................................................................................... 4 EQUIPMENT TESTED ............................................................................................................................ 4 CERTIFICATION TEST RESULTS SUMMARY ................................................................................... 4 TESTING SUMMARY .............................................................................................................................. 4 Volunteer One ................................................................................................................................ 5 Volunteer Two ................................................................................................................................ 6 Volunteer Three ............................................................................................................................. 7 Volunteer Four ............................................................................................................................... 8 Volunteer Five ................................................................................................................................ 9 Volunteer Six ................................................................................................................................ 10 Volunteer Seven .......................................................................................................................... 11 Volunteer Eight ............................................................................................................................ 12 FINAL CONSIDERATIONS ................................................................................................................ 13 California Certification AUP Test Report v1.0 Page 3 of 13 Report Number CA-20006-AUPTR-01 Los Angeles County VSAP Tally 2.1 California Certification Accessibility, Usability, and Privacy Test Report v1.0 Introduction SLI Compliance is submitting this test report as a summary of the certification testing efforts for the County of Los Angeles’ Voting Solutions for All People Tally 2.1 (VSAP Tally 2.1) voting system. The purpose of this document is to provide an overview of the certification testing and the findings of the testing effort for the VSAP Tally 2.1 voting system. This effort included accessibility, usability, and privacy testing of the VSAP Tally 2.1 voting system’s Ballot Marking Device (BMD) component. References California Voting System Standards (CVSS) Equipment Tested The VSAP Tally 2.1 voting system component involved in this test included: BMD firmware/hardware, polling place The BMD is the primary touchpoint for the voter and the hub of the voting system, guiding users with screen prompts and symbols. The BMD features a touchscreen, an audio-tactile interface (controller and headphones), paper handler (scanner and printer), QR code scanner, and dual-switch input which voters use to generate, verify, and cast paper ballots. Completed ballots are transferred to the integrated ballot box, which can be detached for unloading. Certification Test Results Summary Testing Summary The sessions were conducted with LA County personnel acting as poll workers and volunteers voting on the VSAP Tally 2.1 BMDs. When the volunteers arrived, they were given a briefing on the testing and the devices. The sessions were a free form, ad hoc test where the volunteers were able to vote a ballot in any manner that they chose. The volunteers provided feedback in real-time as they were voting, as well as completing an Accessibility Test survey for each device. Additionally, all observations were recorded as each volunteer navigated their way through the process. California Certification AUP Test Report v1.0 Page 4 of 13 Report Number CA-20006-AUPTR-01 Los Angeles County VSAP Tally 2.1 California Certification Accessibility, Usability, and Privacy Test Report v1.0 Volunteer One Table 1 – Volunteer One Survey Agree Agree Disagree Disagree strongly somewhat somewhat strongly The voting method was private. X I feel I can use this system to vote X independently. I am confident that my vote was X recorded accurately. The voting instructions were clear X and complete. The voting method was easy to X use. I could read the display easily. X I could understand the speech X The assistive device(s) were easy X to reach and use. See I found the system confusing to use. Summary The timeframe it took to vote was X what I expected. Volunteer One Summary Volunteer One was surveyed after the testing and responded that they would feel comfortable using this system. The tester liked that the voice used in the audio was not a computerized voice. The tester also noted that when there was more than one candidate, the instructions did not repeat when you switched between contests. For example, if they moved forward and then went back to make a change. The tester also noted that the tactile switch seemed too sensitive and wanted to know if there was a way to adjust the sensitivity. The tester also inquired about adjusting the speed of the audio. No concerns were voiced over privacy issues. California Certification AUP Test Report v1.0 Page 5 of 13 Report Number CA-20006-AUPTR-01 Los Angeles County VSAP Tally 2.1 California Certification Accessibility, Usability, and Privacy Test Report v1.0 Volunteer Two Table 2 – Volunteer Two Survey Agree Agree Disagree Disagree strongly somewhat somewhat strongly The voting method was private. X I feel I can use this system to vote X independently. I am confident that my vote was X recorded accurately. The voting instructions were clear X and complete. The voting method was easy to X use. I could read the display easily. N/A I could understand the speech See output. Summary The assistive device(s) were easy X to reach and use. I found the system confusing to See use. Summary The timeframe it took to vote was See Summary what I expected. Volunteer Two Summary Volunteer Two was surveyed after the testing and responded that they would feel comfortable using this system. The tester felt that the down arrow key on the assistive device was too sensitive. The tester felt it repeated too much. The tester was concerned that there was no braille for the selection button. The tester also felt that the process to complete the ballot took longer than expected. The tester felt the latency response was annoying. No concerns were voiced over privacy issues. California Certification AUP Test Report v1.0 Page 6 of 13 Report Number CA-20006-AUPTR-01 Los Angeles County VSAP Tally 2.1 California Certification Accessibility, Usability, and Privacy Test Report v1.0 Volunteer Three Table 3 – Volunteer Three Survey Agree Agree Disagree Disagree strongly somewhat somewhat strongly The voting method was private. X I feel I can use this system to vote X independently. I am confident that my vote was X recorded accurately. The voting instructions were clear X and complete. The voting method was easy to X use. I could read the display easily. X I could understand the speech N/A output. The assistive device(s) were easy N/A to reach and use. I found the system confusing to X use. The timeframe it took to vote was X what I expected. Volunteer Three Summary Volunteer Three was surveyed after the testing and responded that they would feel comfortable using this system. The tester liked the accessibility of BMD. The tester thought it was a good accessibility height and felt it was very easy to maneuver around. No concerns were voiced over privacy issues. California Certification AUP Test Report v1.0 Page 7 of 13 Report Number CA-20006-AUPTR-01 Los Angeles County VSAP Tally 2.1 California Certification Accessibility, Usability, and Privacy Test Report v1.0 Volunteer Four Table 4 – Volunteer Four Survey Agree Agree Disagree Disagree strongly somewhat somewhat strongly The voting method was private. X I feel I can use this system to vote X independently. I am confident that my vote was X recorded accurately. The voting instructions were clear X and complete. The voting method was easy to X I could read the display easily. X I could understand the speech X output The assistive device(s) were easy X to reach and use. See I found the system confusing to use. Summary The timeframe it took to vote was X what I expected. Volunteer Four Summary Volunteer Four was surveyed after the testing and responded that they would feel comfortable using this system. The tester pointed out that the candidate names were not re-read (audio) when changing back and forth between contests. The tester felt that the different voices used in the audio could be confusing. The tester also pointed out that the text of the contest (Yes/No) had to be touched on the screen for the audio to start. Tester felt that the text of the proposition should be read first and then the contest options (Yes/No) after. No concerns were voiced over privacy issues. California Certification AUP Test Report v1.0 Page 8 of 13 Report Number CA-20006-AUPTR-01 Los Angeles County VSAP Tally 2.1 California Certification Accessibility, Usability, and Privacy Test Report v1.0 Volunteer Five Table 5 – Volunteer Five Survey Agree Agree Disagree Disagree strongly somewhat somewhat strongly The voting method was private. X* I feel I can use this system to vote X independently. I am confident that my vote was X recorded accurately. The voting instructions were clear X and complete. The voting method was easy to X use. I could read the display easily. X I could understand the speech N/A output. The assistive device(s) were easy N/A to reach and use. I found the system confusing to X use. The timeframe it took to vote was X what I expected. Volunteer Five Summary Volunteer Five was surveyed after the testing and responded that they would feel comfortable using this system. The tester liked the accessibility of the device. The tester felt the “More” button on the screen should be brighter. *The voter expressed concerns about privacy, because of the number of observers during the testing process. An explanation was provided to the voter, that the testing observers are not a part of the normal voting process, and the voter will have complete privacy while in a polling place. California Certification AUP Test Report v1.0 Page 9 of 13 Report Number CA-20006-AUPTR-01 Los Angeles County VSAP Tally 2.1 California Certification Accessibility, Usability, and Privacy Test Report v1.0 Volunteer Six Table 6 – Volunteer Six Survey Agree Agree Disagree Disagree strongly somewhat somewhat strongly The voting method was private. X I feel I can use this system to vote X independently. I am confident that my vote was X recorded accurately. The voting instructions were clear See and complete. Summary The voting method was easy to X use. I could read the display easily. X I could understand the speech X – See output. Summary The assistive device(s) were easy X to reach and use. See I found the system confusing to use. Summary The timeframe it took to vote was See Summary what I expected. Volunteer Six Summary Volunteer Six was surveyed after the testing and responded that they would feel comfortable using this system. The tester felt the voting contest options were clear, but the System Help instructions were not. Once the tester knew how to adjust the rate settings, the tester thought it was easier to understand. The tester felt the process took at little longer than expected. The tester felt the system was confusing because the tester had to select each contest option to have it read to them. The tester also pointed out that the select button on the keypad could not be used to exit out of the “More” button screen overlay. No concerns were voiced over privacy issues. California Certification AUP Test Report v1.0 Page 10 of 13 Report Number CA-20006-AUPTR-01 Los Angeles County VSAP Tally 2.1 California Certification Accessibility, Usability, and Privacy Test Report v1.0 Volunteer Seven Table 7 – Volunteer Seven Survey Agree Agree Disagree Disagree strongly somewhat somewhat strongly The voting method was private. X I feel I can use this system to vote X independently. I am confident that my vote was X recorded accurately. The voting instructions were clear X and complete. The voting method was easy to X use. I could read the display easily. x I could understand the speech N/A output. The assistive device(s) were easy N/A to reach and use. I found the system confusing to See use. Summary The timeframe it took to vote was See what I expected. Summary Volunteer Seven Summary Volunteer Seven was surveyed after the testing and responded that they would feel comfortable using this system. The tester found the system very easy to use and thought the process was faster than expected. No concerns were voiced over privacy issues. California Certification AUP Test Report v1.0 Page 11 of 13 Report Number CA-20006-AUPTR-01 Los Angeles County VSAP Tally 2.1 California Certification Accessibility, Usability, and Privacy Test Report v1.0 Volunteer Eight Table 8 – Volunteer Eight Survey Agree Agree Disagree Disagree strongly somewhat somewhat strongly The voting method was private. X I feel I can use this system to vote X independently. I am confident that my vote was X recorded accurately. The voting instructions were clear X and complete. The voting method was easy to X use. I could read the display easily. X I could understand the speech N/A output. The assistive device(s) were easy N/A to reach and use. I found the system confusing to See use. Summary The timeframe it took to vote was X what I expected. Volunteer Eight Summary Volunteer Eight was surveyed after the testing and responded that they would feel comfortable using this system. The tester found the system very easy to use but did note that the tester assists the tester’s elderly parents with voting and that they may find the “More” button confusing. No concerns were voiced over privacy issues. California Certification AUP Test Report v1.0 Page 12 of 13 Report Number CA-20006-AUPTR-01 Los Angeles County VSAP Tally 2.1 California Certification Accessibility, Usability, and Privacy Test Report v1.0 Final Considerations The general consensus of the volunteers was that they felt the technologies implemented for accessibility and usability improved the experience for voters that are most in need of them. No concerns were voiced over privacy issues As directed by the California Secretary of State, this accessibility, usability, and privacy testing report does not include any recommendation as to whether or not the system should be approved. End of AUP Test Report California Certification AUP Test Report v1.0 Page 13 of 13 Report Number CA-20006-AUPTR-01