County of Los Angeles ISB 1.2 Consultant's Accessibility Testing Report

County of Los Angeles’ VSAP 1.2 Interactive Sample Ballot RAVBMS Usability, Accessibility, and Privacy Test Report CAF-19018-UAPTR-01 Vendor Name County of Los Angeles VSAP 1.2 Interactive Sample Vendor System Ballot RAVBMS Prepared by: 4720 Independence St. Wheat Ridge, CO 80033 303-422-1566 www.SLICompliance.com Accredited by the Election Assistance Commission (EAC) for Selected Voting System Test Methods or Services County of Los Angeles VSAP 1.2 ISB RAVBMS Accessibility, Usability & Privacy Test Report Copyright 2019 by SLI ComplianceSM, a Division of Gaming Laboratories International, LLC Revision History Date Release Author Revision Summary Nov. 18, 2019 1.0 M. Santos Initial Release Dec. 27, 2019 2.0 M. Santos Updated for CASOS comments Disclaimer The information reported herein must not be used by the client to claim product certification, approval, or endorsement by NVLAP, NIST, or any agency of the Federal Government. Trademarks • SLI is a registered trademark of SLI Compliance. • All products and company names are used for identification purposes only and may be trademarks of their respective owners. California Certification Page 2 of 23 AUP Test Report v2.0 Report Number CAF-19018-UAPTR-01 County of Los Angeles VSAP 1.2 ISB RAVBMS Accessibility, Usability & Privacy Test Report TABLE OF CONTENTS OVERVIEW ......................................................................................................................................... 4 PHASE I – DOCUMENTATION REVIEW ........................................................................................... 4 PHASE II – USABILITY AND ACCESSIBILITY TESTING ................................................................ 5 3.2 GENERAL USABILITY REQUIREMENTS ............................................................................................ 5 3.3 ACCESSIBILITY REQUIREMENTS ................................................................................................... 16 PHASE III – PRIVACY TESTING ..................................................................................................... 22 SUMMARY ........................................................................................................................................ 23 California Certification Page 3 of 23 AUP Test Report v2.0 Report Number CAF-19018-UAPTR-01 County of Los Angeles VSAP 1.2 ISB RAVBMS Accessibility, Usability & Privacy Test Report Overview The California Voting Systems Standards (CVSS) were written in such a way to be applicable to a wide variety of voting technology. Therefore, the relevant portions of the CVSS are reviewed as the relate to the Remote Accessible Vote by Mail (RAVBM) system for the purposes of this report. The use of “voting system” shall apply to the RAVBM system. This report provides results for the usability, accessibility, and privacy testing of the County of Los Angeles’ VSAP 1.2 Interactive Sample Ballot Remote Access Vote by Mail System (VSAP 1.2 ISB RAVBMS). Testing was divided into three phases. Phase I included a review of all pertinent documents as an inspection of the manufacturer’s documentation of usability and accessibility testing performed during system development. Phase II included all usability and accessibility testing in accordance with California Voting System Standards (CVSS) requirements. Phase III included privacy testing. This testing verified compliance CVSS requirements and the following California Elections Code (EC) specific requirements: • The RAVBM shall not have the capability, including an optional capability, to use a remote server to mark a voter’s selections transmitted to the server from the voter’s computer via the Internet. [EC19295(a)] • The RAVBM shall not have the capability, including an optional capability, to store any voter identifiable selections on any remote server. [EC19295(b)] • The RAVBM shall not have the capability, including the optional capability, to tabulate votes. [EC19295(c)] Phase I – Documentation Review During Phase I, documentation was reviewed to verify and validate the manufacturer’s documentation of usability and accessibility testing performed during system development. Review of the Technical Data Package (TDP) validated that the requirement was satisfactorily covered. California Certification Page 4 of 23 AUP Test Report v2.0 Report Number CAF-19018-UAPTR-01 County of Los Angeles VSAP 1.2 ISB RAVBMS Accessibility, Usability & Privacy Test Report Phase II – Usability and Accessibility Testing In this phase, functional tests were exercised in order to verify and validate the following in accordance with the applicable CVSS requirements: • 3.2 General Usability Requirements • 3.3 Accessibility Requirements 3.2 General Usability Requirements 3.2.1.1 Overall Performance Metrics The requirements of this section set benchmarks for the usability of the system as a whole. There are three performance requirements that deal with effectiveness and two reporting requirements, one for efficiency and one for satisfaction. The metrics are defined as follows: a. Total Completion Score – the proportion of users who successfully cast a ballot (whether or not the ballot contains erroneous votes). Failure to cast a ballot might involve problems such as a voter simply “giving up” during the voting session because of an inability to operate the system, or a mistaken belief that one has successfully operated the casting mechanism. b. Perfect Ballot Index – the ratio of the number of cast ballots containing no erroneous votes to the number of cast ballots containing one or more errors (either a vote for an unintended choice, or a missing vote). c. Voter Inclusion Index – a measure of both voting accuracy and consistency. It is based on mean accuracy and the associated standard deviation. Accuracy per voter depends on how many “voting opportunities” within each ballot are performed correctly. A low value for the standard deviation of these individual accuracy scores indicates higher consistency of performance across voters. d. Average Voting Session Time – mean time taken per voter to complete the process of activating, filling out, and casting the ballot. e. Average Voter Confidence – mean confidence level expressed by the voters that the system successfully recorded their votes. Because of the statistical nature of the testing, numerical results must be interpreted very carefully. The numbers have meaning only within the context of the Voting Performance Protocol (VPP), a (NIST) approved standard. Note especially that the tests associated with these requirements are designed as repeatable controlled experiments and not as “realistic” measures of voting behavior California Certification Page 5 of 23 AUP Test Report v2.0 Report Number CAF-19018-UAPTR-01 County of Los Angeles VSAP 1.2 ISB RAVBMS Accessibility, Usability & Privacy Test Report a. Total completion performance – The system shall achieve a total completion score of at least 98% as measured by the VPP. b. Perfect ballot performance – The system shall achieve a perfect ballot index of at least 2.33 as measured by the VPP. c. Voter inclusion performance – The system shall achieve a voter inclusion index of at least 0.35 as measured by the VPP. Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered validated that the requirement was satisfactorily covered, with total completion, perfect ballot index and voter inclusion index minimum measurements all being met or surpassed, in a controlled environment– the voter uses their own equipment in the processing of their ballot, which leads to improved usability versus one-time usage of unknown hardware. 3.2.1.2 Additional Usability Metrics a. The test lab shall report all the effectiveness metrics for usability as defined and measured by the VPP. b. The test lab shall report the average voting session time, as measured by the VPP. Note that this requirement does not apply to the audio interface of a system, or to the use of special input devices for voters with dexterity disabilities. c. The test lab shall report the average voter confidence, as measured by the VPP. Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered – the voter uses their own equipment in the processing of their ballot, which leads to improved usability versus one- time usage of unknown hardware 3.2.2 Functional Capabilities The following functional capabilities of the system were evaluated: If the voter attempts to select more than the allowable number of choices within a contest on a VEBD or PCOS, the voting system shall notify the voter of the effect of this action before the ballot is cast and counted. (3.2.2.1) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered – the voter is prevented from overvoting. The voting system shall allow the voter, at the voter’s choice, to submit an undervoted ballot without correction. (3.2.2.2) California Certification Page 6 of 23 AUP Test Report v2.0 Report Number CAF-19018-UAPTR-01 County of Los Angeles VSAP 1.2 ISB RAVBMS Accessibility, Usability & Privacy Test Report Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered – the voter is allowed to vote an undervoted ballot without correction, if they so choose. The voting system shall provide the voter the opportunity to correct the ballot for an undervote before the ballot is cast and counted. (3.2.2.3) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered – the voter is given the opportunity to correct the ballot for an undervoted contest before the ballot is cast. VEBD shall prevent voters from selecting more than the allowable number of choices for each contest. (3.2.2.4) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered – the voter is prevented from overvoting a contest. VEBD shall provide feedback to the voter, before final casting of the ballot that identifies specific contests for which the voter has selected fewer than the allowable number of choices (i.e., undervotes). This feature shall not be disabled. (3.2.2.6) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered – the voter is notified of any contest(s) that are undervoted and that the feature is not able to be disabled. VEBD shall provide the voter the opportunity to correct the ballot before it is cast and counted. This correction process shall not require external assistance. The corrections to be supported include modifying an undervote and changing a vote from one candidate to another. (3.2.2.7) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered – the voter is given the opportunity to correct the ballot for any contest before the ballot is cast. This includes undervoted contests as well as contests that are correctly voted and the voter simply desires to change their choice. The correction process is self-contained and does not require external assistance. VEBD shall allow the voter to change a vote within a contest before advancing to the next contest. (3.2.2.8) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered – the voter is able to change a voted contest prior to advancing to the next contest. VEBD shall provide navigation controls that allow the voter to advance to the next contest or go back to the previous contest before completing a vote California Certification Page 7 of 23 AUP Test Report v2.0 Report Number CAF-19018-UAPTR-01 County of Los Angeles VSAP 1.2 ISB RAVBMS Accessibility, Usability & Privacy Test Report on the contest(s) currently being presented (whether visually or aurally). (3.2.2.9) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered – the voter is able to advance to the next contest, or go back to the previous contests, without voting the current contest, for either visual or aural presentations 3.2.5 Cognitive Issues The features specified in this section are intended to minimize cognitive difficulties for voters. a. Completeness of instructions – The voting system shall provide instructions for all operations inherent to the voting system or that are generated by default. Instructions that are part of a ballot definition are not subject to this requirement. (3.2.5.a) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered. The instructions provided for a RAVBMS implementation are sufficient. b. Availability of assistance from the system – The voting system shall provide a means for the voter to get help directly from the system at any time during the voting session. (3.2.5.b) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered. Assistance available from the system during RVABMS implementation is sufficient. c. Plain language – Operational instructions and voting system help material shall conform to norms and best practices for plain language. (3.2.5.c) i. Clarity of warnings – Warnings and alerts issued by the voting system shall clearly state: o The nature of the problem; (3.2.5.c.i.1) o Whether the voter has performed or attempted an invalid operation or whether the voting equipment itself has malfunctioned in some way; and (3.2.5.c.i.2) o The set of responses available to the voter. (3.2.5.c.i.3) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered. Language and clarity of warnings provided by the system for a RAVBM system implementation are sufficient. California Certification Page 8 of 23 AUP Test Report v2.0 Report Number CAF-19018-UAPTR-01 County of Los Angeles VSAP 1.2 ISB RAVBMS Accessibility, Usability & Privacy Test Report ii. Context before action – When an instruction is based on a condition, the condition shall be stated first, and then the action to be performed. (3.2.5.c.ii) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered – the condition-based instructions state the condition first, then the action to be performed. iii. Start each instruction on a new line – The system shall start the visual presentation of each new instruction on a new line. (3.2.5.c.iii) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered – the visual presentation of each new instruction is presented on a new line. iv. Use of positive – The system shall issue instructions on the correct way to perform actions, rather than telling voters what not to do. (3.2.5.c.iv) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered – the instructions tell voters the correct way to perform actions, not what not to do. v. Use of imperative voice – The system's instructions shall address the voter directly rather than use passive voice constructions. (3.2.5.c.v) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered – the instructions directly address voter with an imperative voice. vi. Gender-based pronouns – The system shall avoid the use of gender- based pronouns. (3.2.5.c.vi) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered – the system does not use gender-based pronouns. d. No bias among choices – Consistent with the California Elections Code, the voting system shall support a process that does not introduce bias for or against any of the contest choices to be presented to the voter. In both visual and aural formats, the choices shall be presented in an equivalent manner. (3.2.5.d) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered – the contest choices are not presented to the voter in a biased manner for any of the contests choices, in either visual or aural format. e. Ballot design – The voting system shall provide the capability to design a ballot with a high level of clarity and comprehensibility. (3.2.5.e) California Certification Page 9 of 23 AUP Test Report v2.0 Report Number CAF-19018-UAPTR-01 County of Los Angeles VSAP 1.2 ISB RAVBMS Accessibility, Usability & Privacy Test Report i. Contests split among pages or columns – The voting system shall visually present a single contest on a single page or column except where the number of choices in a contest makes it impossible. (3.2.5.e.i) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered – the system does visually present a single contest on a single page or column. ii. Indicate maximum number of candidates – The voting system shall require that the ballot clearly indicate the maximum number of candidates for which one can vote within a single contest. (3.2.5.e.ii) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered – the ballot clearly indicates the maximum number of candidates allowed to be voted for in a single contest. iii. Consistent representation of candidate selection – The relationship between the name of a candidate and the mechanism used to vote for that candidate shall be consistent throughout the ballot. (3.2.5.e.iii) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered – the ballot is consistent in presenting candidates and the mechanism for selection of candidate to the voter. f. Conventional use of color – The use of color by the voting system shall agree with common conventions: (a) green, blue or white is used for general information or as a normal status indicator; (b) amber or yellow is used to indicate warnings or a marginal status; (c) red is used to indicate error conditions or a problem requiring immediate attention. (3.2.5.f) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered – the colors used agree with common conventions. g. Icons and language – When an icon is used to convey information, indicate an action, or prompt a response, it shall be accompanied by a corresponding linguistic label. (3.2.5.g) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered – icons are accompanied with a corresponding linguistic label. California Certification Page 10 of 23 AUP Test Report v2.0 Report Number CAF-19018-UAPTR-01 County of Los Angeles VSAP 1.2 ISB RAVBMS Accessibility, Usability & Privacy Test Report 3.2.6 Perceptual Issues The requirements of this section are designed to minimize perceptual difficulties for the voter. e. Accommodation for color blindness – The default color coding shall support correct perception by voters with color blindness. (3.2.6.e) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered – the default color coding supports voters with color blindness. f. No reliance solely on color – Color coding shall not be used as the sole means of conveying information, indicating an action, prompting a response, or distinguishing a visual element. (3.2.6.f) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered – color coding is not used as the sole means of conveying information. 3.2.7 Interaction Issues The requirements of this section are designed to minimize interaction difficulties for the voter. a. No page scrolling – Voting systems shall not require page scrolling by the voter. (3.2.7.a) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered – scrolling is not the required method of navigating a page. b. Unambiguous feedback for voter's selection – The voting system shall provide unambiguous feedback regarding the voter’s selection, such as displaying a checkmark beside the selected option or conspicuously changing its appearance. (3.2.7.b) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered – the voter’s selection is unambiguously displayed as the selected choice. c. Accidental Activation – The location and sensitivity of the input mechanisms shall be designed to minimize accidental activation. i. Size and separation of touch areas (3.2.7.c.i) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered. The voters will be using their own equipment, so they can enlarge the screen as needed, if utilizing a touch screen. Navigation/selection is done per individual environment. California Certification Page 11 of 23 AUP Test Report v2.0 Report Number CAF-19018-UAPTR-01 County of Los Angeles VSAP 1.2 ISB RAVBMS Accessibility, Usability & Privacy Test Report ii. No repeating keys – No key or control on a voting system shall have a repetitive effect as a result of being held in its active position. (3.2.7.c.ii) Review of the requirement showed that navigation keys (up and down arrows, tab) did have a repetitive effect. 3.2.8 Timing Issues These requirements address how long the system and voter wait for each other to interact. a. Maximum initial system response time - The initial system response time of a VEBD shall be no greater than 0.5 seconds. Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered – the system responds within the allotted initial response time period. b. Maximum completed system response time for vote confirmation – When the voter performs an action to record a single vote, the completed system response time of the VEBD shall be no greater than one second in the case of a visual response, and no greater than five seconds in the case of an audio response. (3.2.8.b) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered – the system responds within the allotted time periods for both visual and aural responses. c. Maximum completed system response time for all operations – The completed system response time of a VEBD for visual operations shall be no greater than 10 seconds. (3.2.8.c) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered – the system’s visual completed response time is within the allotted time period. d. System response indicator – If a VEBD has not completed its visual response within one second, it shall present to the voter, within 0.5 seconds of the voter's action, some indication that it is preparing its response. (3.2.8.d) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered – if a visual response is not completed within the prescribed timeframe, an indication of response preparation is given to the voter. e. Voter inactivity time – The VEBD shall detect and warn about lengthy voter inactivity during a voting session. Each system shall have a defined and California Certification Page 12 of 23 AUP Test Report v2.0 Report Number CAF-19018-UAPTR-01 County of Los Angeles VSAP 1.2 ISB RAVBMS Accessibility, Usability & Privacy Test Report documented voter inactivity time, and that time shall be between two and five minutes. (3.2.8.e) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was not satisfactorily covered – after five minutes, no inactivity warning was issued. f. Alert time – Upon expiration of the voter inactivity time, the voting system shall issue an alert and provide a means by which the voter may receive additional time. The alert time shall be between 20 and 45 seconds. If the voter does not respond to the alert within the alert time, the system shall go into an inactive state. (3.2.8.f) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was not satisfactorily covered – after five minutes, no alert was issued, nor did the system convert to an inactive state 3.2.9 Alternative Languages a. General support for alternative languages – The voting system shall be capable of presenting the ballot, contest choices, review screens, vote verification records, and voting instructions in any language that Elections Code section 14201 or the Section 203 of the federal Voting Rights Act requires in any California jurisdiction. (3.2.9.a) i. Voter control of language – A VEBD shall allow the voter to select among the available languages throughout the voting session while preserving the current votes. (3.2.9.a.i) o Review of the VSAP 1.2 ISB RAVBMS showed that English, Spanish, Chinese, Japanese, Korean, Vietnamese, Tagalog, Hindi, Khmer, Thai, Armenian, Farsi, and Russian are supported. The language selection is able to be adjusted at any time during the ballot process. ii. Complete information in alternative language – Information presented to the voter in the typical case of English-literate voters (including instructions, warnings, messages, contest choices, and vote verification information) shall also be presented when an alternative language is being used, whether the language is written or spoken. (3.2.9.a.ii) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered – complete information is provided in alternative languages. iii. Auditability of records for English readers – Any records, including paper ballots and paper verification records, shall have sufficient information to support auditing by poll workers and others who can read only English. (3.2.9.a.iii) California Certification Page 13 of 23 AUP Test Report v2.0 Report Number CAF-19018-UAPTR-01 County of Los Angeles VSAP 1.2 ISB RAVBMS Accessibility, Usability & Privacy Test Report Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered – paper verification records provide sufficient information, in English. iv. Usability testing by S-ATA for alternative languages – The S-ATA shall conduct summative usability tests for each of the system's supported languages, using subjects who are fluent in those languages but not fluent in English and shall report the test results, using the Common Industry Format. (3.2.9.a.iv) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered. Testing was performed on this system by personnel fluent in English. General Guidelines Testing verified that the voting system is designed to achieve high usability for voters in the following ways: Voter Effectiveness • Verify voting devices are designed to allow voters to accurately cast a ballot that will be counted as their selections are intended. Review of the VSAP 1.2 ISB RAVBMS validated that this requirement was satisfactorily covered – the voting application is designed to allow voters to accurately mark their ballot as they intend. • Verify voting devices incorporate best practices for user interface design in order to reduce undervotes. Review of the VSAP 1.2 ISB RAVBMS validated that this requirement was satisfactorily covered – the review screen reduces opportunities for un-intended undervotes. Voter Efficiency • Verify voting device interfaces are designed to support a voter’s ability to accomplish completion of their ballot such that they are confident that his/her ballot will be counted as intended. Review of the VSAP 1.2 ISB RAVBMS validated that this requirement was satisfactorily covered – the voting application interface is designed to allow voters to complete their ballot as they intend. • Verify voting devices are designed to minimize the number of interactions that are required to complete typical actions during a voting session. Review of the VSAP 1.2 ISB RAVBMS validated that this requirement was satisfactorily covered – the number of interactions needed to complete typical tasks is minimized. California Certification Page 14 of 23 AUP Test Report v2.0 Report Number CAF-19018-UAPTR-01 County of Los Angeles VSAP 1.2 ISB RAVBMS Accessibility, Usability & Privacy Test Report Voter Satisfaction • Verify voting systems are designed to support a voting process that is easy and intuitive. Review of the VSAP 1.2 ISB RAVBMS validated that this requirement was satisfactorily covered – the voting process is designed to be easy and intuitive. • Verify voting systems interfaces make it clear to the voter how to begin the voting process. Review of the VSAP 1.2 ISB RAVBMS validated that this requirement was satisfactorily covered – the system interface makes it clear as to how the voter is to begin. • Verify voting systems interfaces make it unequivocally clear to the voter whether a ballot has been successfully cast or not. Review of the VSAP 1.2 ISB RAVBMS validated that this requirement was satisfactorily covered – the system makes it clear when the process is finished, with the printing of the ballot. • Verify voting systems do not overload voters with excessive information at any one time. Review of the VSAP 1.2 ISB RAVBMS validated that this requirement was satisfactorily covered – excessive information is not presented to the voter at any one time. • Verify voting systems offer voters indicators of the overall progress in the voting session, to make it clear what steps have already been completed, and what steps remain before completion. Review of the VSAP 1.2 ISB RAVBMS validated that this requirement was satisfactorily covered – the voter can easily determine their progress in the voting session. • Verify voting systems make it clear when the voter has completed each step or task in the voting process. Review of the VSAP 1.2 ISB RAVBMS validated that this requirement was satisfactorily covered – the system provides the voter clear indications as to when a step or task is complete in the voting process. • Verify voting systems allow users to easily find the races they wish to vote. Review of the VSAP 1.2 ISB RAVBMS validated that this requirement was satisfactorily covered – the system allows voters to easily find races that they wish to vote on. California Certification Page 15 of 23 AUP Test Report v2.0 Report Number CAF-19018-UAPTR-01 County of Los Angeles VSAP 1.2 ISB RAVBMS Accessibility, Usability & Privacy Test Report • Verify voting systems minimize the presentation of extraneous information in the visual or audio interface. Review of the VSAP 1.2 ISB RAVBMS validated that this requirement was satisfactorily covered – the system minimizes presentation of information in the visual interface. Review of the VSAP 1.2 ISB RAVBMS validated that this requirement was satisfactorily covered – the system minimizes presentation of information in the audio interface. 3.3 Accessibility Requirements 3.3.1 General The requirements of this section are relevant to a wide variety of disabilities. a. Accessibility throughout the voting session – A VEBD shall be integrated into the manufacturer’s complete voting system so as to support accessibility for disabled voters throughout the voting session. i. Documentation of Accessibility Procedures – The manufacturer shall supply documentation describing: o Recommended procedures that fully implement accessibility for voters with disabilities; and (3.3.1.a.i.1) o How a VEBD supports those procedures. (3.3.1.a.i.2) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered – with a RAVBM system, the voter is utilizing their personal equipment, which provides them with the level of accessibility that they need for any given disabilities. b. Complete information in alternative formats – When the provision of accessibility involves an alternative format for ballot presentation, then all information presented to non-disabled voters, including instructions, warnings, error and other messages, and contest choices, shall be presented in that alternative format. (3.3.1.b) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered – alternative formats provide complete information. c. Secondary means of voter identification – If a voting system provides for voter identification or authentication by using biometric measures that require a voter to possess particular biological characteristics, then the California Certification Page 16 of 23 AUP Test Report v2.0 Report Number CAF-19018-UAPTR-01 County of Los Angeles VSAP 1.2 ISB RAVBMS Accessibility, Usability & Privacy Test Report system shall provide a secondary means that does not depend on those characteristics. (3.3.1.d) Review of the VSAP 1.2 ISB RAVBMS showed that this requirement is not applicable – no biometric measures are required. d. Accessibility of paper-based vote verification – If a VEBD generates a paper record (or some other durable, human-readable record) for the purpose of allowing voters to verify their votes, then the system shall provide a means to ensure that the verification record is accessible to all voters with disabilities. i. Audio readback for paper-based vote verification – If a VEBD generates a paper record (or some other durable, human-readable record) for the purpose of allowing voters to verify their votes, then the system shall provide a mechanism that can read that record and generate an audio representation of its contents. (3.3.1.e.i) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was not applicable, as voters will implement their own hardware. If a voter has equipment that utilizes either OCR technology, or a QR reader with audio playback, then they would be able to obtain an audio read back of their printed ballot. 3.3.2 Low Vision These requirements specify the features of the accessible voting station designed to assist voters with low vision. a. Usability testing by S-ATA for voters with low vision - The S-ATA shall conduct summative usability tests on the voting system using individuals with low vision and shall report the test results, using the Common Industry Format. (3.3.2.a) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was not applicable, as voters will implement their own hardware. b. Adjustable saturation for color displays - An accessible voting station with a color electronic image display shall allow the voter to adjust the color saturation throughout the voting session while preserving the current votes. At least two options shall be available: a high and a low saturation presentation. (3.3.2.b) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was not applicable, as voters will implement their own hardware. c. Distinctive buttons and controls – Buttons and controls on accessible voting stations shall be distinguishable by both shape and color. This applies to buttons and controls implemented either "on-screen" or in hardware. This California Certification Page 17 of 23 AUP Test Report v2.0 Report Number CAF-19018-UAPTR-01 County of Los Angeles VSAP 1.2 ISB RAVBMS Accessibility, Usability & Privacy Test Report requirement does not apply to sizeable groups of keys, such as a conventional 4x3 telephone keypad or a full alphabetic keyboard. (3.3.2.c) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was not applicable, as voters will implement their own hardware. d. Synchronized audio and video – The voting station shall provide synchronized audio output to convey the same information as that which is displayed on the screen. There shall be a means by which the voter can disable either the audio or the video output, resulting in a video-only or audio-only presentation, respectively. The system shall allow the voter to switch among the three modes (synchronized audio/video, video-only, or audio-only) throughout the voting session while preserving the current votes. (3.3.2.d) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was not applicable, as voters will implement their own hardware. Synchronized audio/video, video-only, or audio only are all obtainable. If the voter environment contains a screen, audio output, and a screen reader, synchronized output is available. The voter can turn off audio to have video only. Likewise the voter can turn off video display to have audio only. 3.3.3 Blindness These requirements specify the features of the accessible voting station designed to assist voters who are blind. a. Usability testing by S-ATA for blind voters - The S-ATA shall conduct summative usability tests on the voting system using individuals who are blind and shall report the test results, using the Common Industry Format. b. Audio-tactile interface – The accessible voting station shall provide an audio-tactile interface (ATI) that supports the full functionality of the visual ballot interface. Full functionality includes at a minimum: o Instructions and feedback on initial activation of the ballot (such as insertion of a smart card), if applicable; (3.3.3.b.1) o Instructions and feedback to the voter on how to operate the accessible voting station, including settings and options (e.g., volume control, repetition); (3.3.3.b.2) o Instructions and feedback for navigation of the ballot; (3.3.3.b.3) o Instructions and feedback for contest choices, including write-in candidates; (3.3.3.b.4) California Certification Page 18 of 23 AUP Test Report v2.0 Report Number CAF-19018-UAPTR-01 County of Los Angeles VSAP 1.2 ISB RAVBMS Accessibility, Usability & Privacy Test Report o Instructions and feedback on confirming and changing votes; (3.3.3.b.5) o Instructions and feedback on final submission of ballot. (3.3.3.b.6) i. Equivalent functionality of ATI – The ATI of the accessible voting station shall provide the same capabilities to vote and cast a ballot as are provided by its visual interface. (3.3.3.b.i) ii. ATI supports repetition – The ATI shall allow the voter to have any information provided by the voting system repeated. (3.3.3.b.ii) iii. ATI supports pause and resume – The ATI shall allow the voter to pause and resume the audio presentation. (3.3.3.b.iii) iv. ATI supports transition to next or previous contest – The ATI shall allow the voter to skip to the next contest or return to previous contests. (3.3.3.b.iv) v. ATI can skip initiative or referendum wording – The ATI shall allow the voter to skip over the reading of an initiative or referendum so as to be able to vote on it immediately. (3.3.3.b.v) Review of the VSAP 1.2 ISB RAVBMS validated that these requirements were not applicable, as voters will implement their own hardware. c. Audio features and characteristics – Voting stations that provide audio presentation of the ballot shall do so in a usable way, as detailed in the following sub-requirements. vii. Intelligible audio – The audio presentation of verbal information by both recorded and synthetic speech shall be readily comprehensible by voters who have normal hearing and are proficient in the language. This includes such characteristics as proper enunciation, normal intonation, appropriate rate of speech, and low background noise. Candidate names shall be pronounced as the candidate intends. This requirement applies to those aspects of the audio content that are inherent to the voting system or that are generated by default. (3.3.3.c.vii) Review of the VSAP 1.2 ISB RAVBMS validated that these requirements were not applicable, as voters will implement their own hardware. d. Ballot activation – If the voting station supports ballot activation for non-blind voters, then it shall also provide features that enable voters who are blind to perform this activation. (3.3.3.d) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered – blind voters are able to activate the ballot using their personal equipment. California Certification Page 19 of 23 AUP Test Report v2.0 Report Number CAF-19018-UAPTR-01 County of Los Angeles VSAP 1.2 ISB RAVBMS Accessibility, Usability & Privacy Test Report e. Ballot submission and vote verification – If the voting station supports ballot submission or vote verification for non-blind voters, then it shall also provide features that enable voters who are blind to perform these actions. (3.3.3.e) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered – the system enables blind voters to verify votes and print their ballot. 3.3.4 Dexterity These requirements specify the features of the accessible voting station designed to assist voters who lack fine motor control or use of their hands. a. Usability testing by S-ATA for voters with dexterity disabilities - The S- ATA shall conduct summative usability tests on the voting system using individuals lacking fine motor control and shall report the test results, using the Common Industry Format. b. Support for non-manual input – The accessible voting station shall provide a mechanism to enable non-manual input that is functionally equivalent to tactile input. All the functionality of the accessible voting station (e.g., straight party voting, write-in candidates) that is available through the conventional forms of input, such as tactile, shall also be available through non-manual input mechanisms such as mouth sticks and "sip and puff" switches. (3.3.4.b) Review of the VSAP 1.2 ISB RAVBMS validated that these requirements were not applicable, as voters will implement their own hardware. 3.3.6 Hearing These requirements specify the features of the accessible voting station designed to assist voters with hearing disabilities. a. Reference to audio requirements – The accessible voting station shall incorporate the features listed under the requirements for voting equipment that provides audio presentation of the ballot. (3.3.6.a) Review of the VSAP 1.2 ISB RAVBMS validated that these requirements were not applicable, as voters will implement their own hardware. b. Visual redundancy for sound cues – If the voting system provides sound cues as a method to alert the voter, the tone shall be accompanied by a visual cue, unless the station is in audio-only mode. (3.3.6.b) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered – sound cues have accompanying visual cues. California Certification Page 20 of 23 AUP Test Report v2.0 Report Number CAF-19018-UAPTR-01 County of Los Angeles VSAP 1.2 ISB RAVBMS Accessibility, Usability & Privacy Test Report 3.3.7 English Proficiency These requirements specify the features of the accessible voting station designed to assist voters who lack proficiency in reading English. Use of ATI- For voters who lack proficiency in reading English, the voting equipment shall provide an audio interface for instructions and ballots. (3.3.7) Review of the VSAP 1.2 ISB RAVBMS validated that this requirement was not applicable, as voters will implement their own hardware. 3.3.8 Speech Speech not to be required by equipment – Voting equipment shall not require voter speech for its operation. (3.3.8) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered – the system does not require speech by the voter for operation. Tools Implemented During Testing The tools listed below were utilized during testing: • Wave Web Accessibility evaluation tool: o A tool used to evaluate accessibility of Websites, to Web Content Accessibility Guidelines (WCAG) 1.2 and section 508 o The manufacturer provided URLs which were run through Multi-guideline Accessibility and Usability Validation Environment (MAUVE) to help determine accessibility • Evaluera website o A tool used to evaluate web applications regarding conformance to the WCAG 1.2. o The manufacturer provided URLs which were run through 508 Checker to help determine compliance. o http://www.evaluera.co.uk/ • ChromeVox o A screen reader that is an extension of Google Chrome and is used to assist visually impaired users. California Certification Page 21 of 23 AUP Test Report v2.0 Report Number CAF-19018-UAPTR-01 County of Los Angeles VSAP 1.2 ISB RAVBMS Accessibility, Usability & Privacy Test Report Phase III – Privacy Testing In this phase, functional tests were exercised in order to verify and validate the following in accordance with the applicable CVSS requirements: • 3.2.4.1 Privacy at the Polls • 3.2.4.2 No Recording of Alternative Format Usage • EC19295 RAVBMS Requirements Privacy at the Polls a. Visual privacy – The ballot, any other visible record containing ballot information, and any input controls shall be visible only to the voter during the voting session and ballot submission. (3.2.4.1.a) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered as the voter will utilize their own hardware in the environment of their choosing. b. Auditory privacy – During the voting session, the audio interface of the voting system shall be audible only to the voter. (3.2.4.1.b) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered as the voter will utilize their own hardware in the environment of their choosing. c. Privacy of warnings – The voting system shall issue all warnings in a way that preserves the privacy of the voter and the confidentiality of the ballot. (3.2.4.1.c) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered as the voter will utilize their own hardware in the environment of their choosing. d. No receipts – The voting system shall not issue a receipt to the voter that would provide proof to another of how the voter voted. (3.2.4.1.d) Review of the VSAP 1.2 ISB RAVBMS validated that this requirement was not applicable as the voter will implement their own hardware, and print their own marked ballot, as a RAVBM system. No Recording of Alternative Format Usage a. No information shall be kept within an electronic Central Voter Registration (CVR) that identifies any alternative language feature(s) used by a voter. (3.2.4.2.a) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered, as CVR’s are not created. California Certification Page 22 of 23 AUP Test Report v2.0 Report Number CAF-19018-UAPTR-01 County of Los Angeles VSAP 1.2 ISB RAVBMS Accessibility, Usability & Privacy Test Report b. No information shall be kept within an electronic CVR that identifies any accessibility feature(s) used by a voter. (3.2.4.2.b) Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered, as CVR’s are not created. 19295 RAVBMS Requirements • The RAVBM shall not have the capability, including an optional capability, to use a remote server to mark a voter’s selections transmitted to the server from the voter’s computer via the Internet. [EC19295(a)] Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered – the system does not use a remote server to mark a voter’s selections. • The RAVBM shall not have the capability, including an optional capability, to store any voter identifiable selections on any remote server. [EC19295(b)] Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered – the system does not have the capability to store any voter identifiable selections on any remote server. • The RAVBM shall not have the capability, including the optional capability, to tabulate votes. [EC19295(c)] Review of the VSAP 1.2 ISB RAVBMS validated that the requirement was satisfactorily covered – the system does not have the capability to tabulate votes. Summary The testing conducted for the VSAP 1.2 ISB RAVBMS included validation of accessibility, usability, and privacy requirements as set forth by the State of California. This report identifies those requirements and the results of the testing performed. Three issues were encountered in meeting the requirements. For requirements 3.2.8.e and 3.2.8.f, there are no inactivity warnings. For requirement 3.2.7.c.ii, navigation keys (up and down arrows, tab) did have a repetitive effect As per the direction given by the California Secretary of State, this security testing report does not include any recommendation as to whether or not the system should be approved. End of Accessibility, Usability, and Privacy Test Report California Certification Page 23 of 23 AUP Test Report v2.0 Report Number CAF-19018-UAPTR-01