County of Los Angeles Vsap 2.0 Consultant's Accessibility Testing Report

California Secretary of State Consultant’s Report on: Usability, Accessibility and Privacy Testing of the Los Angeles Voting Solutions for All People (VSAP) 2.0 Ballot Marking Devices (BMD) 1.0 Voting System Prepared for the California Secretary of State by: Paul Craft Kate McGregor Gisela Salas December 24, 2019 FCMG: Accessibility Test Report Revision history Version Change Author(s) Changes to previous version date 1.0 2019-11-27 Gisela Salas Initial draft 1.1 2019-12-15 Gisela Salas 2nd draft; edits and addition of addendums 1.1 2019-12-17 Kate McGregor Review and edits 1.1 2019-12-18 Paul Craft Review, attaching exhibits and formatting. 1.2 2019-12-22 Paul Craft Revised to address client comments Gisela Salas Revised to include comments on use of 1.3 2019-12-23 sip and puff and dual buttons 1.4 2019-12-24 Paul Craft Revised to address client comments Last update: 2019‐12‐24 Status: Final Version: 1.4 Page 2 of 27 FCMG: Accessibility Test Report Table of Contents1 Summary of System Components ............................................................................................................. 4 Scope of Work and Reporting .................................................................................................................... 4 Description of System Submitted for Certification ................................................................................... 5 Approach to Testing ..................................................................................................................................... 6 Scope Limitations ......................................................................................................................................... 6 Detailed Reporting on the Phases of Testing .......................................................................................... 6 Consultants’ Accessibility Testing Phase .......................................................................................... 7 Volunteer Voters Accessibility Testing Phase .................................................................................. 7 Observations ........................................................................................................................................ 23 Overall Findings .......................................................................................................................................... 24 Conformance to CVSS Requirements .................................................................................................... 25 Last update: 2019‐12‐24 Status: Final Version: 1.4 Page 3 of 27 FCMG: Accessibility Test Report Summary of System Components The Los Angeles Voting Solutions for All People (VSAP) 2.0 Ballot Marking Devices (BMD) 1.0 Voting System submitted for certification testing consisted of the following major components: Software Modules: VSAP BMD software system, consisting of three programs: BMD-Admin, BMD- Diagnostics, and BMD-Vote. Hardware Components: VSAP BMD Hardware including: o Touch Screen o Controller Key Pad o Audio Headset o Printer/Scanner o Ballot Box The specific equipment used in this test were: Serial Number CBGBMDAA0000025 Serial Number CBGBMDAA0000035 Serial Number CBGBMDAA0000036 Serial Number CBGBMDAA0000099 Serial Number CBGBMDAA0000135 Serial Number CBGBMDAA0000174 Scope of Work and Reporting This report covers the work performed to assist the California Secretary of State (SOS) with Accessibility Testing for the VSAP 2.0 Ballot Marking Devices (BMD) 1.0 Voting System. We are not attorneys and do not offer legal advice. The tests were conducted to assist Last update: 2019‐12‐24 Status: Final Version: 1.4 Page 4 of 27 FCMG: Accessibility Test Report the California Secretary of State (SOS) with collection of facts and evidence in order for them to make certification decisions. However, to advise the SOS on the determination of whether the system complies with California’s certification requirements would require an interpretation of law. This report does not provide recommendations or offer any opinion as to whether the system can be certified. The work performed and all findings are strictly limited to the specific serial numbered hardware elements and specific software elements exercised during this test. An inventory of those items is included in the earlier section entitled “Summary of System Components.” The results described in this report should be reliable and repeatable for those specific items. The decision to apply those results to decisions regarding other items is solely at the discretion and risk of the Secretary of State and election officials who purchase the system. FCMG conducted the examinations with the goal of providing the SOS with a basis for evaluating the extent to which the voting system meets applicable standards. CVSS References were reviewed and conformance notes were provided to the SOS. Description of System Submitted for Certification The VSAP 2.0 Ballot Marking Device (BMD) Voting System is a voting system that utilizes a computer touch screen, audio headphones, a controller keypad, a ballot- marking device (BMD) that prints a paper ballot for verification, along with an attached ballot box where the ballot can be cast. The VSAP presents voters with the option to select candidates and propositions using the touch screen or through a controller while listening to an audio ballot. Voters also have the ability to view or listen to their ballot choices prior to printing out the paper ballot. The system also allows voters the ability to preselect their choices and enter a polling code to print and cast their ballot at a voting center. The VSAP can also work with a variety of other voter interface devices including a sip and puff switch and dual button switches. After the voter makes their selections, the BMD prints their choices on a paper ballot. The voter’s selections are displayed on the touch screen and also read through the audio output. Once they are satisfied with their choices the voter can choose to cast their ballot and it will be inserted into the attached ballot box. The BMD does not record the voter’s ballot and does not maintain a record of the voter’s ballot after the voting session has ended. Last update: 2019‐12‐24 Status: Final Version: 1.4 Page 5 of 27 FCMG: Accessibility Test Report Approach to Testing FCMG used the guidelines from Section 5.4.1 of ISO/IEC 25062 “Software engineering – Software product Quality Requirements and Evaluation – Common Industry Format (CIF)” for usability test reports, provides the following: Participants in testing should be assigned to ser groups which are segmented based on key characteristics and capabilities of each group. In order to generate valid summative statistical analyses, it is necessary to test sufficient number of subjects. Eight or more subjects per segment were recommended. Based upon the requirements of section 3.3 of the CVSS the following user groups should have been established. Eight, blind or low eyesight (visual) Eight mobility Eight fine motor skill Eight cognitive/developmental Eight deaf Scope Limitations The testing consisted of a small pool of volunteers who participated in the accessibility test thus resulting in some scope limitations. A total of 18 individuals participated in the test. Three volunteers identified their impairment as visual, three said that they had mobility and dexterity impairments, one stated that they had visual/perceptual and mobility challenges, and 11 chose not to disclose a disability. It should be noted that those with dexterity impairments could also be considered to have mobility and/or fine motor skill impairments. Staff was not able to make a judgment on those who chose not to disclose their disability. Individuals with other disabilities did not test the system. None of the volunteers used sip and puff or dual button switches during the testing. Detailed Reporting on the Phases of Testing Accessibility testing took place at the Los Angeles County Registrar’s Main Office in Norwalk, and two warehouse locations. During the Phase 1 Consultants’ Accessibility Testing Phase members of FCMG tested the VSAP BMD using various functions that would normally be used by voters with disabilities (i.e. audio headsets, various levels of audio, screen off, the controller, and the dual button switches). The Phase 2 Volunteer Voters Accessibility Testing Phase included participant voters with a variety of disabilities. These participants were asked to vote on VSAP BMD Last update: 2019‐12‐24 Status: Final Version: 1.4 Page 6 of 27 FCMG: Accessibility Test Report machines while a tester from FCMG observed their voting sessions. After their voting was complete, the volunteer testers responded to a survey with structured interview questions regarding their voting experience, their opinions, and any recommendations for future improvements. Details on both phases are discussed below. Consultants’ Accessibility Testing Phase Consultants from FCMG conducted accessibility tests on the VSAP BMDs at two of the Los Angeles County Registrar warehouses. Tests were conducted using ballots from the Primary 2016 and General 2016 Elections. The tests consisted of voting numerous ballots using the touch screen, the controller keypad, audio headsets, and the dual button switches. The following is a synopsis of the anomalies found during the test procedures. Logs of some of the voting sequences are provided in Attachment C – Accessibility Testing Logs. Some of the text found on the touch screen ballot did not align with what was heard on the audio ballot. Long periods of silence made it seem as if the voting session was over. Sensitivity issues with the touch screens were noted where ballot selections could not be made because the system would not respond to the voter’s touch. Paper jams or misfeeds. Volunteer Voters Accessibility Testing Phase During the voter participation phase of accessibility test, 18 volunteers were scheduled with the assistance of the California Secretary of State’s Office. When the volunteers arrived, they were provided a brief overview of the test being conducted and of the devices. Most of the volunteer voters appeared to have an interest in, and some experience with, accessible interfaces for voting. Three voters stated that they had previous experience with the system. One voter stated that he had helped to design the system. Volunteers 5, 6, and 13 were visually impaired. Volunteers 9, 10, and 17 had mobility and dexterity impairments. Volunteers 16 had visual/perceptual and mobility impairments. Volunteers 1, 2, 3, 4, 7, 8, 11, 12, 14, 15, and 18 did not disclose their disabilities. The volunteers were then escorted to one of the VSAP BMDs in the room to begin the voting process. Once they completed and cast their ballot, each voter completed an Accessibility Test survey. The average amount of time spent voting was 21.86 minutes. Some of the volunteers voted very quickly (eight to 19 minutes). Others explored the Last update: 2019‐12‐24 Status: Final Version: 1.4 Page 7 of 27 FCMG: Accessibility Test Report functions within the audio and video ballots and exercised the system thoroughly rather than attempt to vote efficiently. In these instances, the time it took the volunteers to cast their ballots (57 and 100 minutes to vote one ballot) was not reflected in the calculations for the amount of time an average voter would require. The ballot selected for the test was a November 8, 2016 General Election ballot with thirty-two contests. Volunteers used the audio headsets and the controller to assist them in voting. The sip and puff interface and button paddlers were not exercised during the accessibility testing by the participants. The volunteer test voters were observed as they exercised the VSAP BMD. As voters completed voting on each machine, a structured interview regarding their experience with the device was conducted. This section contains summaries of the observations and information collected during those interviews. A transcription of each set of observations and interviews is located in Attachment A, the “Accessibility Test Records.” Each voter was asked to select one of five responses to a statement regarding their experience on the devices. The structured responses were Agree Strongly, Agree Somewhat, Disagree Somewhat, Disagree Strongly and Not Applicable or No Opinion. The following pages include a summary of the voters’ responses to each statement: Last update: 2019‐12‐24 Status: Final Version: 1.4 Page 8 of 27 FCMG: Accessibility Test Report Statement 1. “The voting method was private.” Table 1 Vote Privately Responses n % Agree Strongly 15 83 Agree Somewhat 3 17 Disagree Somewhat 0 0 Disagree Strongly 0 0 No Opinion/N/A 0 0 When asked about the BMD’s privacy, 15 voters responded “Agree Strongly” and three responded “Agree Somewhat.” Of those voters with visual impairment, three responded “Agree Strongly”. Among the voters with mobility or dexterity impairments, one responded “Agree Strongly” and two responded “Agree Somewhat.” One voter with visual/perceptual and mobility impairments responded “Agree Strongly.” Of those voters who chose not to disclose their impairment ten responded “Agree Somewhat” and one responded “Agree Somewhat.” Last update: 2019‐12‐24 Status: Final Version: 1.4 Page 9 of 27 FCMG: Accessibility Test Report Statement 2. “I feel I can use this system to vote independently.” Table 2 Vote Independently Responses n % Agree Strongly 17 94 Agree Somewhat 1 6 Disagree Somewhat 0 0 Disagree Strongly 0 0 No Opinion/N/A 0 0 After voting on the BMD, 17 voters said that they “Agree Strongly” and one responded with “Agree Somewhat” when asked about using the system to vote independently. Three volunteers with visual impairment responded “Agree Strongly.” Three voters with visual/perceptual and mobility impairments responded “Agree Strongly.” Of the volunteers with mobility or dexterity disabilities, one answered “Agree Strongly.” Ten voters with undisclosed impairments responded “Agree Strongly” and one voter responded “Agree Somewhat.” Last update: 2019‐12‐24 Status: Final Version: 1.4 Page 10 of 27 FCMG: Accessibility Test Report Statement 3. “I am confident that my vote was recorded accurately.” Table 3 Vote Accurately Responses n % Agree Strongly 16 89 Agree Somewhat 0 0 Disagree Somewhat 2 11 Disagree Strongly 0 0 No Opinion/N/A 0 0 After voting on the VSAP BMD, 16 voters said that they “Agree Strongly” and two responded with “Disagree Somewhat” when asked about whether they were confident their vote was recorded accurately using the system. Three volunteers with visual impairments and three with mobility impairments responded “Agree Strongly.” One voter with visual/perceptual and mobility impairments responded “Agree Strongly.” Of those with undisclosed impairments nine voters responded “Agree Strongly” while two voters responded “Disagree Somewhat.” Last update: 2019‐12‐24 Status: Final Version: 1.4 Page 11 of 27 FCMG: Accessibility Test Report Statement 4. “The voting instructions were clear and complete.” Table 4 Instructions Clear and Complete Responses n % Agree Strongly 9 50 Agree Somewhat 6 33 Disagree Somewhat 3 17 Disagree Strongly 0 0 No Opinion/N/A 0 0 Half of the voters said “Agree Strongly” to this statement regarding the Instructions being clear and complete on the VSAP BMD. Six voters responded “Agree Somewhat” and the remaining three volunteers responded “Disagree Somewhat.” Of those voters with visual impairments one stated “Agree Strongly,” one responded “Agree Somewhat” and one responded “Disagree Somewhat.” Among the voters with mobility and dexterity impairments, one responded “Agree Strongly” and two “Agree Somewhat.” One voter with visual/perceptual and mobility impairments responded “Agree Strongly.” Five voters with undisclosed impairments answered “Agree Strongly,” three responded “Agree Somewhat” and two responded “Disagree Somewhat.” Last update: 2019‐12‐24 Status: Final Version: 1.4 Page 12 of 27 FCMG: Accessibility Test Report Statement 5. “The voting method was easy to use.” Table 5 System easy to use Responses n % Agree Strongly 14 78 Agree Somewhat 4 22 Disagree Somewhat 0 0 Disagree Strongly 0 0 No Opinion/N/A 0 0 Fourteen voters responded “Agree Strongly” and four stated “Agree Somewhat” when asked whether the voting method was easy to use. Two voters with visual impairment responded with “Agree Strongly” and one responded “Agree Somewhat.” Among the voters with mobility or dexterity impairments, two also replied “Agree Strongly” and one “Agree Somewhat.” Seven voters with undisclosed impairments answered “Agree Strongly” and two responded, “Agree Somewhat.” Last update: 2019‐12‐24 Status: Final Version: 1.4 Page 13 of 27 FCMG: Accessibility Test Report Statement 6. “I could read the display easily.” Table 6 Display easy to read Responses n % Agree Strongly 11 61 Agree Somewhat 1 6 Disagree Somewhat 0 0 Disagree Strongly 0 0 No Opinion/N/A 6 33 When asked whether the display was easy to read, the three voters with visual impairments had no opinion and responded “Not Applicable.” Two voters with mobility and dexterity impairments answered “Agree Strongly” and one “Agree Somewhat.” One voter with a visual/perceptual and mobility impairment responded “Agree Strongly.” Seven voters with undisclosed impairments answered “Agree Strongly” and three said it was “Not Applicable.” Last update: 2019‐12‐24 Status: Final Version: 1.4 Page 14 of 27 FCMG: Accessibility Test Report Statement 7. ”I could understand the speech output.” Table 7 Speech output understandable Responses n % Agree Strongly 9 50 Agree Somewhat 3 17 Disagree Somewhat 1 6 Disagree Strongly 0 0 No Opinion/N/A 5 28 Three voters answered, “Agree Strongly,” one “Agree Somewhat” and two “Not Applicable” when asked whether the speech output was easy to understand. One voter with visual impairment responded “Agree Strongly” and one “Agree Somewhat.” Among the voters with mobility or dexterity impairments, one responded with “Agree Strongly” and two “Not Applicable.” The voter with undisclosed impairments also responded “Agree Strongly.” Last update: 2019‐12‐24 Status: Final Version: 1.4 Page 15 of 27 FCMG: Accessibility Test Report Statement 8. ”The assistive device(s) were easy to reach and use.” Table 8 Assistive devices easy to use Responses n % Agree Strongly 15 83 Agree Somewhat 2 11 Disagree Somewhat 0 0 Disagree Strongly 0 0 No Opinion/N/A 1 6 Voters were asked whether the assistive devices on the VSAP BMD were easy to use. Fifteen voters responded with “Agree Strongly,” two “Agree Somewhat” and one stated “No Opinion.” Among the voters with visual impairment, two said “Agree Strongly” and one “Agree Somewhat.” Two voters with mobility or dexterity impairments responded, “Agree Strongly,” and one “Agree Somewhat.” The voter with visual/perceptual and mobility impairments responded “Agree Strongly.” Nine voters with undisclosed impairments responded “Agree Strongly.” Last update: 2019‐12‐24 Status: Final Version: 1.4 Page 16 of 27 FCMG: Accessibility Test Report Statement 9. ”I found the system confusing to use.” Table 9 System confusing Responses n % Agree Strongly 0 0 Agree Somewhat 3 16 Disagree Somewhat 1 6 Disagree Strongly 14 78 No Opinion/N/A 0 0 When asked whether they found the VSAP BMD to be confusing, 14 voters answered “Disagree Strongly,” one responded “Disagree Somewhat” and three responded “Agree Somewhat.” The three voters with visual impairment responded with “Disagree Strongly.” Of those volunteers with mobility or dexterity impairments, one answered “Disagree Somewhat” and two “Disagree Strongly.” The voter with visual/perceptual and mobility impairments responded “Disagree Strongly.” Fourteen voters with undisclosed impairments responded “Disagree Strongly” and three responded “Agree Somewhat.” Last update: 2019‐12‐24 Status: Final Version: 1.4 Page 17 of 27 FCMG: Accessibility Test Report Statement 10. “The timeframe it took to vote was what I expected.” Table 10 Time it took to vote was what I expected Responses n % Agree Strongly 14 78 Agree Somewhat 3 16 Disagree Somewhat 1 6 Disagree Strongly 0 0 No Opinion/N/A 0 0 With regard to whether the time it took to vote was what they expected using the BMD, 14 voters responded “Agree Strongly,” three responded “Agree Somewhat” and one stated “Disagree Somewhat.” One voter with visual impairment responded with “Agree Strongly,” one stated “Agree Somewhat” and one stated “Disagree Somewhat.” All three of the voters with mobility or dexterity impairments answered “Agree Strongly.” The voter with visual/perceptual and mobility impairments said “Agree Strongly.” Eight of the voters with undisclosed impairments said “Agree Strongly” and three stated “Agree Somewhat.” Last update: 2019‐12‐24 Status: Final Version: 1.4 Page 18 of 27 FCMG: Accessibility Test Report Perceptions of Voting System and Suggestions Voters were asked two additional series of questions on their perceptions of the voting system along with their suggestions. Following is a compilation of their responses. “Would you be satisfied using this system to vote in an election or would you rather vote using another method? If you prefer another method, what method would you prefer?” Responses from the voters included: I think it’s amazing. Would be satisfied with this system. Creates a voting experience for those with disabilities – does not create a different experience for those with disabilities. Does not create separation which is huge. Pretty close to capturing the goal of a private independent vote for someone with a disability – a huge step. Huge lead for LA County. Previous system was outdated which strengthened the need for new system. This is good. Usually use mail in ballot. Yes. This method is easier. Would use in an election. Satisfied with this. Yes, would like to use in an election. Yes, Liked the system. Yes, This one. Yes, Would prefer to use this one again. Liked using the system. Good. I like this system. Yes (Sic. But) concerned that all information that is available to voter is available by audio. (Note: missing components in ballot). Happy with this. Yes This one. It was so clear, easy to read, and use with or without headphones. I have visual/perceptual impairment and mobility disability and this was amazing. Quick, clear easy – loved it! Last update: 2019‐12‐24 Status: Final Version: 1.4 Page 19 of 27 FCMG: Accessibility Test Report Each voter was then asked: “Do you have any suggestions for changes on this system and/or any other comments you would like to provide?” Responses from VSAP volunteer voters included: Ballot readers – Make indention higher on the sides of where ballot is inserted (raise the sides of the tray to prevent skewing of paper when inserted to prevent paper jams. Intake people at voting place need to be better trained. There should be visually prompt for intake people. Training needs to be more thorough. Workers need to be trained on all aspects of voting system (Know that machines are not connected to the internet). Voters should be offered a demonstration on how to use the machines when they go into the precinct. Need news blitz. Machines need to be explained on the news. Nope. Loved it. Voter asked if this was changed since last Saturday’s Mock election. Scroll button on Referendums was hard to see. Can see well on this ballot. Much clearer. Touch screen could be a problem for some voters. On some machines on Saturday, voters had to press down hard. Touched 64 but did not select. System is good. Used system in Mock and found instructions repetitious. Just the ones going through {sic. While answering survey). When message – Ballot Printed. Cast your ballot now. I didn’t know what to do. On Measure it reads all the controls before the measure. Put the controls below the detail. No suggestions. Good. Have a tie for the headphone cord to keep it out of the way. The machine should highlight what is being read so you can follow along. Review your selections should just read rather than requiring us to touch each contest, then we should have the ability to stop it. If it could be lowered for voters who do not have the strength to lift arms and those with limited mobility or in low manual wheelchair. Good job! Last update: 2019‐12‐24 Status: Final Version: 1.4 Page 20 of 27 FCMG: Accessibility Test Report That the first page of the system has an explanation of what to do with ballot. Name of company (VSAP) above where you insert the ballot is not visible from sitting angle. Should be in white so it is contrasting. Should have instruction that tells voter to listen to details or first put details and then option to vote. Also, should include option to skip details. If you skip a choice, it should alert voter. Arrows are too far apart. Need to be closer together. Repeat button. Would be helpful to have raised “R” and “L” on arrows. Reviewing title “Judge…” was not clear. Ready to print – pushing round button and not doing nothing. Likes that it tells you the number of options/candidates. But it should include information that write-in is an option. Suggestion to spell out candidate names if repeat button is used. Not real clear but when you get the hang of it with practice it works. Party Preference: Voter wanted to know if text displayed party preference. Audio did not state party preference. Believes it could be the screen reader. Confusing on selecting and deselecting a candidate. Instructions could be more clear especially for someone with developmental or learning disabilities. Deselect is confusing. If you change your mind or want to change your choice would be more clear. It would be helpful to have a repeat button. It is reading html text would be confusing to a person not used to using a screen reader. (Voter is aware that text will be repeated if back button (left arrow) is used. Write-in candidate option - How would you do that? Played with arrows and was able to maneuver through QWERTY keyboard with her limited vision. It would be nice if audio would read out name that was typed. It reads letters one by one. I think repeat button is really necessary. Measure A saying Yes or No. Not reading text. Long wait while we talked. Measure details come after instructions. Good for someone who knows ballot Last update: 2019‐12‐24 Status: Final Version: 1.4 Page 21 of 27 FCMG: Accessibility Test Report question. Might be helpful to have introduction that states first you will hear details. Measure 60 and other Measures audio stopped reading detail at Fiscal Impact and then nothing further although text on screen has fiscal impact description. Delay in speech. More concise. If it could be more like Jaws. Less punctuation. Get rid of repetitive instructions. An audio instruction for those who do not read Braille. Volume could be higher. He found the keypad and navigated through without difficulty. Moved forward and backwards through the alphabet. “You have to be patient the text of the measures.” Voter is listening to the measures. Taking time. We could streamline it a bit. There is a lot of pause. I don't want to hear the instructions on every question. If I could go right to the details. I really appreciate the fact that I can do this. If you are not familiar with the measure you have to be patient. I’m used to using a screen reader. You probably wouldn’t understand it. High Speed Speech. I’ve been doing mail in ballots in those it says a yes vote does this and a no vote does that. This doesn’t say what each vote does. < > Is where it says what it does. I like this. I really do. The measure text was not complete. Not sure if she wanted to vote yes or no. Measure text should be at the top of the screen. I don’t know what I voted on. (State Measure 51) Nope, loved it. I can’t wait to use this in March to be independent in voting privately in person at a vote center!!! Last update: 2019‐12‐24 Status: Final Version: 1.4 Page 22 of 27 FCMG: Accessibility Test Report Try to make the station a bit more private. This should be on video training for the poll workers. I watched the video profile on election. Observations The following observations by FCMG and SOS personnel were annotated on the Accessibility Test Records. Observed a voter that touched their choice on a contest (selection 64) but it did not select. Although voter was warned that volume level was low they did not increase volume until Contest 3. When voter went to cast ballot, ballot not read message appeared on screen; ballot ejected. Went to put in to cast ballot and it appeared that new ballot was initiated since language screen appeared. Voting stopped without getting message that ballot had been cast. When voter left, Device Locked message appeared. Unlocked machine with QR Code and took ballot to another machine to attempt to cast. Inserted ballot and it went to language screen, then cast ballot option appeared. Selected cast ballot and ballot was cast. Party Preference not stated on audio like on the screen. < > signs in text. Long pause before reading. Had no problem with vote for 3. Fiscal impact on Measure 52 was missing or not stated. Missing on all measures. HTML brackets are a problem. Annoying and does not follow well. Had some difficulty in keeping track on where he was in the ballot. Vote Nominated Offices was not read. No instructions to conclude your voting. Press to the right seems to be what you want to do. Did not hear the voice. Had to assist in turning up the volume. Did not list the candidates in Antelope Valley contest. Does not read them in down button. Not highlighting on the screen nor in the Audio. Only reads selected candidates but not available options. Had to do a double tap to get it to select and a double tap to go to the next screen. Found the Vote Nominated Offices doesn’t read. Last update: 2019‐12‐24 Status: Final Version: 1.4 Page 23 of 27 FCMG: Accessibility Test Report The Move button needs to be enlarged for both visually impaired and for those with fine motor skill impairments. Write in on Judge #42 – it is not highlighting or reading the letters selected. Write-in can be done from screen but not on keypad. 3 tries to start voting. 3 tries to go to next highlighting and then not highlighting. Put on headphones for Audio using touchpad. Non-Partisan Offices – touched next, nothing happened. Hit button multiple times. At Voter Nominated Offices voter asked if it was finished. Apparently no audio. Sensitivity or motor skills caused issues in making ballot selections. (Sic. Voter) held onto screen at times. (Sic. Voter) observed touching some selections multiple times. Voter turned off screen. Could not read Braille. Seemed to navigate with no issues. Pause button was not working. Voter stated: “Not telling me what party candidate is.” Noted the Less/Greater than script in the audio file. Voter stated “Once I got the hang of it, it is pretty easy.” Overall Findings Based upon the responses of the voter volunteers, they were generally positive about voting on the BMD. Even with the limited pool of volunteers and types of disabilities, the tests conducted demonstrated that the usability of the system and the accessible interfaces varied from one voter to another depending upon their specific condition and their experience with accessible interfaces. As an example, the time it took to go through the same ballot varied greatly from voter to voter. During the volunteer voter’s phase of the test, voters were able to successfully mark their test ballots and cast their ballots. Overall, the voters that participated in the accessibility testing conducted on October 3, 2019 at the Los Angeles County Registrar’s Office were of the opinion that the VSAP BMD provided them with the ability to vote privately and independently and were satisfied with their voting experiences. During the Consultants’ testing phase numerous anomalies were observed as described in the section above entitled Consultants’ Accessibility Testing Phase. An extraordinary number of printer jams were experienced. Some jams occurred after as few three ballots were cast. In these cases, the ballot paper would jam within the printer since ballots were stacked in the ballot box in a manner that caused the ballot paper to jam Last update: 2019‐12‐24 Status: Final Version: 1.4 Page 24 of 27 FCMG: Accessibility Test Report and did not allow the ballot to properly drop into the ballot box. Some ballots could not be cast at the conclusion of the voting session which would have caused a voter to have to be reissued a ballot and begin the entire voting process from the beginning. In other cases, the ballot would jam when it was initially loaded and would become crushed in the paper feeder path. Although compliance with the the misfeed rate in CVSS is not a part of the Usability, Accessibility and Privacy Testing, misfeeds and paper jams have an adverse impact on usability, accessibility and privacy. In cases where the jam occurred after the ballot was printed the voter’s privacy would be compromised as the poll worker resolved the jam. In several instances, after voting an entire ballot, it could not be cast and had to be replaced.. This would mean that the entire voting session would have to be initiated from the beginning again and the voter would need to go through the entire ballot making all their selections to cast their ballot. For voters using the audio ballot this would have a very detrimental effect on the system’s usability as they would need to repeat the lengthy audio ballot voting process. The text was not read using the audio functionality because of programming during the election configuration. The audio ballot behaved differently when using the touch screen mode vs using the toggle switch to navigate the voting session. For example, the audio ballot did not read all of the text of a measure using the toggle switch, but did so when using the touch screen mode only. This matter was referred to Smartmatic for investigation and follow up. Some voters had challenges in making their selections on the touch screen and had to touch the screen multiple times. In some instances, the voters were observed just touching around the screen until a selection was made. Some voters noted that there were some long delays/pauses in the audio in varying parts of the ballot. This was confusing for the voter. and is also not in conformance with section 3.2.8.b, CVSS standards. Depending on the voter’s impairment this could be detrimental since the pause could be mistaken for the end of the ballot. Some voters found that the instructions and process for selecting and deselecting a candidate was confusing. Conformance to CVSS Requirements With regard to the CVSS requirements applicable to the BMD, the tests yielded expected results with the exception of the following anomalies: Last update: 2019‐12‐24 Status: Final Version: 1.4 Page 25 of 27 FCMG: Accessibility Test Report 3.2.2.1: Notification of Effect of Over voting - If the voter attempts to select more than the allowable number of choices within a contest on a VEBD or PCOS, the voting system shall notify the voter of the effect of this action before the ballot is cast and counted. When a voter attempts to over vote a race the BMD automatically cancels the first choice and accepts the second 3.2.4.1.b: Auditory privacy - During the voting session, the audio interface of the voting system shall be audible only to the voter. The BMD has a second headphone jack that allows a pollworker to listen to the audio ballot for the purpose of assisting a voter using the audio ballot. 3.2.7.a.: No page scrolling - Voting systems shall not require page scrolling by the voter. Long candidate lists require the voter to scroll on BMDs. 3.2.8.b: "Maximum completed system response time for vote confirmation - When the voter performs an action to record a single vote, the completed system response time of the VEBD shall be no greater than one second in the case of a visual response, and no greater than five seconds in the case of an audio response." Some errors noted - Delays or no audio responses observed. 3.2.9.a.ii: Auditability of records for English readers - Any records, including paper ballots and paper verification records, shall have sufficient information to support auditing by poll workers and others who can read only English. Foreign language ballots only print the selection of Yes or No in the chosen language. 3.3.3.b: Audio-tactile interface functionality includes instructions and feedback on initial activation of the ballot (such as insertion of a smart card), if applicable; Some errors noted—instructions missing on audio ballot. Last update: 2019‐12‐24 Status: Final Version: 1.4 Page 26 of 27 FCMG: Accessibility Test Report 3.3.3.b. Audio-tactile interface functionality includes instructions and feedback on confirming and changing votes. Some errors noted—instructions missing on audio ballot. 3.3.3.c.iii: Sanitized headphone or handset - A sanitized headphone or handset shall be made available to each voter. This requirement can be achieved in various ways, including the use of "throwaway" headphones, or of sanitary coverings. Headphones are not disposable and no coverings were provided. 3.3.3.c.iv: Initial volume - The voting system shall set the initial volume for each voting session between 40 and 50 dB SPL. dBs were not measured, but the initial volume was inaudible. 3.3.3.f: Tactile discernibility of controls - Mechanically operated controls or keys on an accessible voting station shall be tactilely discernible without activating those controls or keys. The buttons on the key pad of the BMD do not have the variable resistance that allows touch discernible use. The keys depress smoothly and nothing is felt until they bottom out. This should be further examined in accessibility testing. 3.3.5.c. Visibility of displays and controls - Labels, displays, controls, keys, audio jacks, and any other part of the accessible voting station necessary for the voter to operate the voting system shall be easily legible and visible to a voter in a wheelchair with normal eyesight (no worse than 20/40, corrected) who is in an appropriate position and orientation with respect to the accessible voting station. The label for the headphone jack is slightly raised plastic in the same color as the body of the machine, It is almost invisible in diffused lighting with 20/20 vision. Another connection is a symbol which is indecipherable. Last update: 2019‐12‐24 Status: Final Version: 1.4 Page 27 of 27