Secretary of State Staff Test Report

A P LEX ADILLA | SECRETARY OF STATE | STATE OF CALIFORNIA OFFICE OF VOTING SYSTEMS TECHNOLOGY ASSESSMENT 1500 11th Street | Sacramento, CA 95814 | Tel 916.695.1680 | Fax 916.653.4620 | www.sos.ca.gov Dominion Voting Systems ImageCast Remote 5.2 Remote Accessible Vote by Mail System Staff Report Prepared by: Secretary of State’s Office of Voting Systems Technology Assessment March 29, 2018 Table of Contents I. Introduction ......................................................................................... 1 1. Scope ................................................................................... 1 2. Summary of the Application ................................................. 1 3. Contracting and Outsourcing ............................................... 1 II. Summary of the System .................................................................... 2 III. Testing Information and Results ........................................................ 2 1. Background .......................................................................... 2 2. Functional Testing ................................................................ 2 3. Accessibility, Usability and Privacy Testing .......................... 3 4. Security and Telecommunications Testing ........................... 14 5. Software Review Testing ..................................................... 23 IV. Compliance with State and Federal Laws and Regulations ............... 24 1. Elections Code Review ......................................................... 24 2. Section 508 and WCAG Compliance Review ....................... 25 V. Conclusion .......................................................................................... 25 2 | Page A P LEX ADILLA | SECRETARY OF STATE | STATE OF CALIFORNIA OFFICE OF VOTING SYSTEMS TECHNOLOGY ASSESSMENT 1500 11th Street | Sacramento, CA 95814 | Tel 916.695.1680 | Fax 916.653.4620 | www.sos.ca.gov I. INTRODUCTION 1. Scope This staff report presents the test results for the all phases of the certification test of the Dominion Voting Systems (DVS) ImageCast Remote 5.2 Remote Accessible Vote by Mail (RAVBM) system. The purpose of the testing is to test the compliance of the RAVBM system with California and Federal laws. Testing also uncovers other findings, which do not constitute non-compliance, and those findings are reported to the RAVBM system vendor to address the issues procedurally. The procedures for mitigating any additional findings are made to the documentation, specifically the ImageCast Remote 5.2 Use Procedures. ImageCast Remote 5.2 was evaluated against the applicable portions of the California Voting Systems Standards (CVSS). The CVSS were written in such a way to be applicable to a wide variety of voting technology, therefore the relevant portions of the CVSS were reviewed as it relates to the ImageCast Remote 5.2 RAVBM system for purposes of this report. The use of “voting system” shall apply to RAVBM. 2. Summary of the Application Dominion Voting Systems submitted an application for the ImageCast Remote 5.2 system on September 5, 2017. ImageCast Remote 5.2 is a vendor hosted application, solely for the purposes of ballot marking pursuant to Elections Code sections 303.3 and 19283. Dominion was required to submit the following: 1) the technical documentation package (TDP); 2) hardware for functional testing (Windows and Apple OS systems), including all supported browsers, and screen readers; 3) source code; and 4) the Dominion ImageCast Remote 5.2 Use Procedures. 3. Contracting and Consulting Upon receipt of a complete application, the Secretary of State (SOS) released a Request for Quote (RFQ) for assistance with the Source Code Review, Telecommunications and Security Review, and Usability, Accessibility and Privacy testing. The statement of work (SOW) also had the option for assistance with Functional Testing. That testing was conducted by SOS staff. Through the formal California contracting process, the Secretary of State awarded a contract to SLI Compliance, a division of Gaming Laboratories International, LLC, to assist with testing. II. SUMMARY OF THE SYSTEM Dominion’s ImageCast Remote 5.2 remote accessible vote by mail application is a web-based interface. The voter can optionally use a screen reader to navigate through the ballot. After marking their ballots and reviewing their selections, the voter must print their selections. The printed paper cast record is returned to the local elections official, where it will be remade into a ballot. The printed paper cast record and the ballot are kept together for auditing purposes. III. TESTING INFORMATION AND RESULTS 1. Background The ImageCast Remote 5.2 software platform is a fully hosted service provided by Dominion. The infrastructure consists of Dell servers, Cisco network equipment, Microsoft SQL Servers as well as a complete hosting platform based on Microsoft .NET technology. Accessibility and functional testing of this system was conducted by SOS staff. The functional test was conducted at the Secretary of State’s office in Sacramento, California during the month of November 2017. End user Accessibility testing was conducted in three waves. Wave I User Accessibility testing was performed by nine (9) volunteers throughout the state of California, from November 30 to December 1, 2017. Wave II took place from December 4, to December 8, 2017, concluding with Wave III from December 18 to December 22, 2017 with twenty-nine (29) volunteer testers. The Software Review was performed by SLI, Gaming LLC, during the month of March 2018. Security and Telecommunications testing was also performed by SLI during the month of March 2018. 2. Functional Testing Summary System Configuration: The ImageCast Remote 5.2 Remote Accessible Vote by Mail System is supported by a fully-redundant network and application architecture protecting against a single point of failure. 2 | Pa ge For the functional test, the SOS provided Dominion with three test ballot data sets of California elections as described below: 1. State of California Primary Election – Fictional Jurisdiction 2. State of California General Election - Fictional Jurisdiction 3. Local Jurisdiction - Fictional Rank Choice Election SOS staff used desktop, laptop and tablet computing devices to access, mark and print the paper cast vote record as described in the ImageCast Remote 5.2 Use Procedures. Each test environment exercised undervotes, overvotes, all contest marking, and write-ins. Staff also used keyboard functions, standard USB mice, and the touchpad on the laptops to mark selections. Undervotes The system warned of undervotes, and allowed the tester to proceed to the end of the ballot to print the paper cast record. Overvotes The system warned of overvotes, and allowed the tester to proceed to the end of the ballot and print the paper cast record. Write-Ins The system allowed the tester to make write in selections with a fifty (50) character limit. Printed Paper Cast Record Each paper cast record printed displays the Election Type (General Election, Primary and Rank Choice Voting), the Election Day and date (Day of the Week, Month, Day, Year). The system also produces a QR Code about mid-way on the page, which can be used for ballot duplication at the county elections office. 3. Accessibility, Usability, and Privacy Testing For Accessibility, Usability and Privacy Testing the SOS conducted three waves of End User Usability and Accessibility testing. Additionally, SLI conducted three phases of testing for Accessibility, Usability and Privacy. SOS End User Usability Testing Wave I User Accessibility testing was performed by nine (9) volunteers throughout the state of California, from November 30 to December 1, 2017. Wave II took place from December 4, to December 8, 2017, concluding with Wave III from December 18 to December 22, 2017 with twenty-nine (29) volunteer testers. The testing was conducted by SOS staff. The testers were recruited through several channels including several groups within the 3 | Pa ge accessibility community. Each tester used his/her own technology, including any auxiliary peripheral devices and screen reading software to access, mark, and print their paper cast vote record. The testers were asked to complete pre-test and post-test surveys, documenting such information as demographics, the technology used for testing, and the tester’s post-test experience using the system. The survey results of each are included in Attachment A of this report. Please note that personally identifying information has been redacted. The technical issues identified in all three waves as requiring mitigation or responses are listed below in Table 3A. Table 3A: Tester Technical Issues Issue Response/Mitigation Using Internet Explorer on Using Internet Explorer 11 with JAWS the described Windows 10 with the JAWS situation has not been encountered by Dominion’s screen reader for Windows. internal testers or as a result of Dominion’s external When I access the link you accessibility testing program. Dominion Voting Systems provided and inputted the (DVS) was unable to reproduce the error. credentials everything was fine. The email that was sent to me for the pin is where the problem is at. After activating the link, I asked to type the verification code. As I am a screen reader user, I activated the audio option for the CAPTCHA. I refreshed the code four times and every single code I was given did not work. I am sure I typed the verification code that I was provided via the download correctly within a one minute time span. Error Code: The Verification Text You entered is not valid. Please try again. Testers unable to submit a User should change modes in JAWS from JAWS Virtual write-in candidate as their Cursor mode, to JAWS Forms mode. In the experience voting choice because, after of DVS Quality Assurance and external accessibility checking the checkbox to testers this has proven to resolve navigation issues such enter a write-in candidate, as the one described. The use procedures will be they were not able to locate updated accordingly. the edit field in which to type 4 | Pa ge in the name of the write-in candidate. The focus of his screen DVS was unable to recreate. reader bounced around the HTML ballot when the tester unchecked a selection to make a different selection for a particular contest. Tester encountered this problem on multiple contests. After marking the checkbox User should change modes in JAWS from JAWS Virtual to enter a write-in candidate, Cursor mode, to JAWS Forms mode. In the experience tester was unable to find the of DVS Quality Assurance and external accessibility edit field where the tester testers this has proven to resolve navigation issues such could type in the name of as the one described. the write-in candidate. JAWS did not properly read DVS was unable to recreate. the text associated with each voting option for the ballot measure near the end of the ballot. Instead it simply read "choice" for both options. Tester selected one of the voting choices for this contest; JAWS then began to properly read the text associated with each voting choice textbox. Tester was unable to The suggestion to help make it easier to locate the download and print the downloaded ballot on the voter's device has been noted completed vote cast record. by the DVS Product team, for possible future In particular, tester was able modifications. to activate the buttons used to download the vote cast record. However, tester was unable to locate the file containing the vote cast record after downloading it. When using the Google User should change modes in JAWS from JAWS Virtual Chrome browser - Tester Cursor mode, to JAWS Forms mode. In the experience observed that JAWS did not of DVS Quality Assurance and external accessibility properly read labels for the testers this has proven to resolve navigation issues such edit fields for voter ID, first as the one described. The use procedures will be name, and last name when updated accordingly. 5 | Pa ge navigation the webpage using the JAWS virtual cursor. However, JAWS correctly read the edit field labels for these three fields when using the "tab" key to navigate through them. JAWS did not read the User should change modes in JAWS from JAWS Virtual control used to select the Cursor mode, to JAWS Forms mode. In the experience voter's birthdate. of DVS QA and external accessibility testers this has proven to resolve navigation issues such as the one described. The use procedures will be updated accordingly. The "verify email address" Using the latest Firefox with JAWS 18 the described edit field appears to be situation has not been encountered by the internal missing from the tab order. testers or as a result of our external accessibility testing Tester found that tabbing program. Dominion Voting Systems would need from "email address" edit additional information in order to assess the issue. field appears to skip the "verify email address" edit field. After checking the checkbox To remedy this scenario, to change modes in JAWS from to write in a candidate name, JAWS Virtual Cursor mode, to JAWS Forms mode. The tester was unable to locate use procedures will be updated accordingly. the edit field to type in the make of a write-in candidate. Tester got to the end of the Tester was asked to check bottom of browser window to process and clicked the make certain the download package was not present. "Mark Your Ballot" option, Reset user credentials, and he was able to complete the and the screen processing test using another browser, but the same computer. button just kept spinning. When unchecking his first Regarding Write-in, based on testing done on the CA selection, the screen reader RAVBM instance starts cycling the error (https://demo.intvoting.com/UAT/California/IVApp) message, rather than switching modes (between JAWS 'virtual cursor' mode allowing the voter to and JAWS 'Forms' mode)* enables the Write-in field to proceed. When voter have the user's content keyed into it. The use proceeds to make a write-in procedures will be updated accordingly. selection, the write in messages do not Regarding informing the screen reader user when necessarily agree with allowed number of selections has been exceeded; this screen reader choices. - recommended enhancement is acknowledged as a Likes the steps to complete significant usability improvement and has been added to breadcrumb trail. Does not the next release. tell voter if they have exceed 6 | Pa ge the number of choices. Tells * The same is true for NVDA Browse Mode and Focus user that a checkbox was Mode. not checked, but still lists it as a choice. When using the arrow keys on the list box it was difficult to get out of the box, on the secret question. On the Ballot Localization resources are able to be edited to achieve Confirmation/Review page, the jurisdiction's choice of verbiage in the voting the voter’s selections are application's instructions, and navigation buttons. The presented to him/her for use procedures will be updated accordingly. review and possible editing. I noticed that after the sentence, “Once you have selected the “Cast Ballot” button below, your vote will be cast and you will be unable to make any further changes” there is a button that says “Preferenti”. This button serves the same function as the “Go Back” button at the bottom of the page, but is supposed to say (I’m assuming) “Preferential Choices”. I recommend removing this button altogether, or changing it to say “Go Back”, or at least fixing this button to say “Preferential Choices”. Tester had extensive issues To re-order RCV selections the last (most recent) with reordering Rank Choice ranking must be unranked, then other selections must be selections after deciding the unranked in that order. From this feedback it may be change mind. that the instructions for completing an RCV contest are not clear enough and require enhancements to improve usability. Still says "to make selection, Localization resources are editable in order to achieve click in the box containing the jurisdiction's choice of verbiage in the voting your choice". Not useful. Still application's instructions, and navigation buttons. In can't check write-in addition control over contest descriptions, candidates 7 | Pa ge candidate. No findable place and voting instructions is enabled via data in the election to write name. Could not tell project. if I had checked a write-in in the candidate list. Warning Regarding selection of write-in candidates, switching message tossed me back to between JAWS 'virtual cursor' mode and JAWS 'Forms' the top of the page. When I mode enables the Write-in field to have the user's realized I'd checked an extra content keyed into it. The use procedures will be updated write-in, could not remove it accordingly. because I was told I could only remove top ranking Regarding the re-ranking of selections, to change selection rankings, the most recently ranked choice must be un-ranked first, and so on until the ranking required to be changed has been reached. Regarding better screen reader communication of candidate ranking status, this recommended enhancement is acknowledged as a significant usability improvement and has been added to the next release. Consultant Phase Accessibility, Usability, and Privacy Testing SLI conducted three phases of testing for Accessibility, Usability and Privacy. Phase I was a review of the ImageCast Remote 5.2 documentation of Usability and Accessibility testing performed during system development. Phase II included all Accessibility and Usability testing. Phase III included Privacy testing. SLI used the following tools in Table 3B to during testing to evaluate ImageCast Remote 5.2. Table 3B: Consultant Tools Tool Name Tool Purpose Tool Use Wave Web Tool to evaluate accessibility of The vendor provided url’s Accessibility web sites, to WCAG 2.0 and were run through MAUVE to evaluation tool Section 508. help determine accessibility. Evaluera website Tool to evaluate web The vendor provided url’s applications regarding were run through 508 conformance to WCAG 2.0 checker to help determine standards. compliance. Chromevox – Screen Tool is a screen reader that is Screen reader used for reader extension of an extension of Google Accessibility and Usability Google Chrome. Chrome, to assist visually testing. impaired users. 8 | Pa ge Phase I Documentation Review The documentation was reviewed by SLI to verify and validate the manufacturer’s documentation of usability and accessibility performed during system development. Review of the TDP validated that the requirement was satisfactorily covered. Phase II-A Usability Testing SLI evaluated ImageCast Remote 5.2 against Sections 3.2 (General Usability Requirements and 3.3 (Accessibility Requirements) of the CVSS. This section covers the requirements for usability of a system. SLI found that all but one of the requirements were satisfactorily met. ImageCast Remote 5.2 did not meet one (1) of the requirements. Table 3C: Phase II-A – Usability Testing CVSS Requirement Result Mitigation/Response CVSS (3.2.7.c.ii): No repeating Review of the The use procedures will keys. No key or control on a requirement showed that be updated accordingly. voting system shall have a navigation keys (up and repetitive effect as a result of down arrows, tab) did being held in its active position. have a repetitive effect. Phase II-B Accessibility Testing SLI evaluated ImageCast Remote 5.2 against the applicable portions of the CVSS for Accessibility Testing for compliance. The table below shows each of the applicable CVSS sections, and the result of compliance. Table 3D: Phase II-B – Accessibility Testing CVSS Requirement Result CVSS (3.3.1.a.i.1): Accessibility throughout Review of the requirement validated that the voting session - A VEBD shall be the requirement was satisfactorily integrated into the manufacturer’s complete covered. voting system so as to support accessibility for disabled voters throughout the voting session. Documentation of Accessibility Procedures - The manufacturer shall supply documentation describing: recommended procedures that fully implement accessibility for voters with disabilities; and CVSS (3.3.1.a.i.2): How a VEBD supports those procedures. 9 | Pa ge CVSS (3.3.1.b) : Complete information in Review of the requirement validated that alternative formats - When the provision of the requirement was satisfactorily accessibility involves an alternative format covered. for ballot presentation, then all information presented to non-disabled voters, including instructions, warnings, error and other messages, and contest choices, shall be presented in that alternative format. CVSS (3.3.1.d): Secondary means of voter Review of the requirement showed that no identification - If a voting system provides biometric measures are required, so this for voter identification or authentication by requirement is not applicable. using biometric measures that require a voter to possess particular biological characteristics, then the system shall provide a secondary means that does not depend on those characteristics. CVSS (3.3.1.e.i): Accessibility of paper- Review of the requirement validated that based vote verification - If a VEBD the requirement was not applicable, as a generates a paper record (or some other voter will implement their own hardware. If durable, human-readable record) for the a voter has equipment that utilizes either purpose of allowing voters to verify their OCR technology, or a QR reader with votes, then the system shall provide a audio playback, then they would be able means to ensure that the verification record to obtain an audio read back of their is accessible to all voters with disabilities. printed ballot. i. Audio read back for paper-based vote verification - If a VEBD generates a paper record (or some other durable, human- readable record) for the purpose of allowing voters to verify their votes, then the system shall provide a mechanism that can read that record and generate an audio representation of its contents. CVSS (3.3.2.c): Distinctive buttons and Review of the requirement validated that controls - Buttons and controls on the requirement was not applicable, as a accessible voting stations shall be voter will implement their own hardware. distinguishable by both shape and color. This applies to buttons and controls implemented either "on-screen" or in hardware. This requirement does not apply to sizeable groups of keys, such as a conventional 4x3 telephone keypad or a full alphabetic keyboard. CVSS (3.3.2.d): Synchronized audio and Review of the requirement validated that video - The voting station shall provide the requirement was not applicable, as a synchronized audio output to convey the voter will implement their own hardware. same information as that which is displayed Synchronized audio/video, video-only, or on the screen. There shall be a means by audio only are all obtainable. If voter which the voter can disable either the audio environment contains a screen, audio 10 | Pa ge or the video output, resulting in a video- output and a screen reader, synchronized only or audio-only presentation, output is available. Voter can turn off respectively. The system shall allow the audio to have video only. Likewise voter voter to can turn off video display to have Audio switch among the three modes only. (synchronized audio/video, video-only, or audio-only) throughout the voting session while preserving the current votes. CVSS (3.3.3.b.1): Audio-tactile interface - Review of the requirements validated that The accessible voting station shall provide these requirements were not applicable, an audio-tactile interface (ATI) that as voter will implement their own supports the full functionality of the visual hardware. ballot interface. Full functionality includes at a minimum: o Instructions and feedback on initial activation of the ballot (such as insertion of a smart card), if applicable; CVSS (3.3.3.b.2): Instructions and feedback to the voter on how to operate the accessible voting station, including settings and options (e.g., volume control, repetition); CVSS (3.3.3.b.3): Instructions and feedback for navigation of the ballot; CVSS (3.3.3.b.4): Instructions and feedback for contest choices, including write-in candidates; CVSS (3.3.3.b.5): Instructions and feedback on confirming and changing votes; CVSS (3.3.3.b.6): Instructions and feedback on final submission of ballot. CVSS (3.3.3.c.vii): Audio features and Review of the requirements validated that characteristics - Voting stations that these requirements were not applicable, provide audio presentation of the ballot as voter will implement their own shall do so in a usable way, as detailed in hardware. the following sub-requirements. iv. Intelligible audio - The audio presentation of verbal information by both recorded and synthetic speech shall be readily comprehensible by voters who have 11 | Pa ge normal hearing and are proficient in the language. This includes such characteristics as proper enunciation, normal intonation, appropriate rate of speech, and low background noise. Candidate names shall be pronounced as the candidate intends. This requirement applies to those aspects of the audio content that are inherent to the voting system or that are generated by default. CVSS (3.3.3.d): Ballot activation - If the Review of the requirement validated that voting station supports ballot activation for the requirement was satisfactorily covered nonblind voters, then it shall also provide – blind voters are able to activate the features that enable voters who are blind to ballot using their personal equipment. perform this activation. CVSS (3.3.4.b): Support for non-manual Review of the requirements validated that input - The accessible voting station shall these requirements were not applicable, provide a mechanism to enable non- as a voter will implement their own manual input that is functionally equivalent hardware. to tactile input. All the functionality of the accessible voting station (e.g., straight party voting, write-in candidates) that is available through the conventional forms of input, such as tactile, shall also be available through non-manual input mechanisms such as mouth sticks and "sip and puff" switches. CVSS (3.3.6.a): Reference to audio Review of the requirements validated that requirements - The accessible voting these requirements were not applicable, station shall incorporate the features listed as voter will implement their own under the requirements for voting hardware. equipment that provides audio presentation of the ballot. CVSS (3.3.6.b): Visual redundancy for Review of the requirement validated that sound cues - If the voting system provides the requirement was satisfactorily covered sound cues as a method to alert the voter, – sound cues have accompanying visual the tone shall be accompanied by a visual cues. cue, unless the station is in audio-only mode. CVSS (3.3.7): Use of ATI- For voters who Review of the requirements validated that lack proficiency in reading English, the this requirement was not applicable, as a voting equipment shall provide an audio voter will implement their own hardware. interface for instructions and ballots. CVSS (3.3.8): Speech not to be required by Review of the requirement validated that equipment - Voting equipment shall not the requirement was satisfactorily require voter speech for its operation. covered- the system does not require speech by the voter for operation. 12 | Pa ge Phase III Privacy Testing SLI evaluated ImageCast Remote 5.2, for compliance with California Elections Code requirements for privacy within an RAVBM system, in addition to the applicable portions of the CVSS. The table below shows each of the applicable code sections and CVSS sections, and the result of compliance. Table 3D: Phase III – Privacy Testing Applicable California Elections Code Result Section & CVSS Requirement EC Section 19295(a): The RAVBM shall not Review of the requirement validated have the capability, including an optional that the requirement was satisfactorily capability, to use a remote server to mark a covered. voter’s selections transmitted to the server from the voter’s computer via the Internet. EC Section 19295 (b): The RAVBM shall not Review of the requirement validated have the capability, including an optional that the requirement was satisfactorily capability, to store any voter identifiable covered. selections on any remote server. EC Section 19295(c): The RAVBM shall not Review of the requirement validated have the capability, including the optional that the requirement was satisfactorily capability, to tabulate votes. covered. CVSS (3.2.4.1.a): Visual privacy - The ballot, Review of the requirement validated any other visible record containing ballot that the requirement was satisfactorily information, and any input controls shall be covered, as the voter will utilize their visible only to the voter during the voting own hardware in the environment of session and ballot submission. their choosing. CVSS (3.2.4.1.b): Auditory privacy - During Review of the requirement validated the voting session, the audio interface of the that the requirement was satisfactorily voting system shall be audible only to the covered, as the voter will utilize their voter. own hardware in the environment of their choosing. CVSS (3.2.4.1.c): Privacy of warnings - The Review of the requirement validated voting system shall issue all warnings in a that the requirement was satisfactorily way that preserves the privacy of the voter covered, as the voter will utilize their and the confidentiality of the ballot. own hardware in the environment of their choosing. CVSS (3.2.4.1.d): No receipts - The voting Review of the requirements validated system shall not issue a receipt to the voter that this requirement was not that would provide proof to another of how the applicable, as voter will implement their voter voted. own hardware, and print their own marked ballot, as a RAVBMS system. CVSS (3.2.4.2.a): No information shall be Review of the requirement validated kept within an electronic CVR that identifies that the requirement was satisfactorily any alternative language feature(s) used by a covered, as CVR’s are not created. 13 | Pa ge voter. CVSS (3.2.4.2.b): No information shall be Review of the requirement validated kept within an electronic CVR that identifies that the requirement was satisfactorily any accessibility feature(s) used by a voter. covered, as CVR’s are not created. The ImageCast Remote 5.2 system allows for voter privacy. This is achieved by removing client side storage of marked selections. A voter is able to verify and print the paper cast record for submission to a local elections official. 4. Security and Telecommunications Testing Summary SLI Compliance performed Security and Telecommunications testing on the ImageCast Remote 5.2 RAVBMS. During the testing, SLI conducted a documentation review, functional security testing, and telecommunications and data transmission testing. A summary of each phase and the results are listed below. Documentation Review SLI conducted a documentation review of the TDP provided by Dominion for the ImageCast Remote 5.2 RAVBMS. SLI evaluated the documentation for the following applicable portions of the CVSS: Table 4A: Documentation Review CVSS Standard Result 5.5 Vote Secrecy on Review of the TDP validated that the requirement was DRE and EBM satisfactorily covered. Systems 6.1.2 Data Review of the TDP validated that the requirement was Transmissions satisfactorily covered. 6.2 Design, Review of the TDP validated that the requirement was Construction, and satisfactorily covered. Maintenance Requirements 6.2.1 Confirmation Review of the TDP validated that the requirement was satisfactorily covered. 7.1.1 Elements of Review of the TDP validated that the requirement was Security outside satisfactorily covered. Manufacturers Control 7.2 Access control Review of the TDP validated that the requirement was satisfactorily covered. 7.2.1 General Access Review of the TDP validated that the requirement was Control satisfactorily covered. 7.2.2 General Access Review of the TDP validated that the requirement was Control satisfactorily covered. 7.4.5 Software The documentation doesn’t describe a process used to verify 14 | Pa ge Reference the software distributed is the software provided by the NSRL or Information designated repository. The documentation doesn’t provide a procedure or functionality to verify that the software is the certified software by comparison. 7.4.6 Software Setup Review of the TDP determined the documentation doesn’t Validation reference any type of digital verification on software prior to installation. No documentation on the method provided by external interface or equipment used to verify software on the system. No documentation about a mechanism for detecting unauthorized software. Functional Security SLI next conducted functional security testing as part of the Security and Telecommunications testing on the ImageCast Remote 5.2 RAVBMS. The testing consisted of: 1) testing the relevant software and operating security system configuration, for pertinent vulnerabilities; and 2) testing of hardware, including examination of unused hardware ports, and security measures applied to those ports. Table 4B: Functional Security, lists the applicable CVSS Standards, the testing performed, any vulnerabilities, along with the vendor response/mitigation when applicable. Table 4B: Functional Security CVSS Testing Vulnerability Recommended Response Standard Performed Mitigation 5.5 Vote Testing was None N/A N/A Secrecy on performed to DRE and EBM verify how the Systems system handled a ballot being printed and the browser closed, as well as when the ballot is closed prior to being printed. Attempts were made to resume a ballot, as well as to determine if any ballot information resided in history or cache. 15 | Pa ge Table 4B: Functional Security 7.2.1 General The system None N/A N/A Access Control uses an N-tier architecture that consists of separate client applications, application server components, database components, and a central document repository. Authentication included methods for both the voter facing application as well as the administrative application. Security was tested on the architecture pieces, client application, and administrative application, which were accessible remotely. Physical security was not about to be observed as this was hosted at a data center. 7.2.2 Access The system None N/A N/A Control uses a client Identification server system to authenticate registered voters and issue the 16 | Pa ge Table 4B: Functional Security correct ballot for a particular voter using predefined ballot rules and voters can be imported by the jurisdiction. Role based access controls are in place for administrative login purposes. 7.2.4 Access All access to None N/A N/A Control the system is Authorization controlled by Voter ID number and an associated pin number which is created during registration. All administrative access is controlled by username/pass word combinations and there is a role-based administrative access in place. The ability to assign voters to different electoral groups/electoral districts gives the ability to assign ballots to voters in accordance with specific rules. 17 | Pa ge Table 4B: Functional Security 7.4.5 Software No methods or For the Addressed in All code running Reference procedures for vulnerability in Democracy as part of the Information verification if question, the Suite ImageCast the system is level of access ImageCast Remote running certified required to take Remote application unmodified advantage of Identification utilizes a code were this potential Guide Version: trusted code presented. vulnerability 5.2-CA::313. signing key that Testing was would be ensures only unable to inclusive to the Dominion- successfully following actors: signed code is modify the Voter, who allowed to run server code to can attempt within the verify if a to modify application protection the server context. For method was in code anyone to place and remotely. modify the viable. code, they Election would also official need to compile insider, who and run it using could this signing key attempt to which is not modify the available server code outside of remotely. authorized Vendor Dominion Insider, who engineering could personnel. attempt to locally modify the server code. 7.4.6 Software The system Software Setup Addressed in Verification of Setup does not have a Validation, were Democracy ImageCast Validation built in hash not able to be Suite Remote hashes verification verified to ImageCast is obtained by method for the ensure that the Remote following the system which system is Identification procedures provides a running the Guide Version: detailed in the method to verify correct 5.2-CA::313. supplied that the source software. Democracy code is not Suite running ImageCast modified code. Remote Testing was Identification 18 | Pa ge Table 4B: Functional Security unable to Guide Version: successfully 5.2-CA::313. As modify the noted in the server code to response to the verify if a 7.4.5 item, protection modified code method was in cannot be run place and within the same viable. application context as the ImageCast Remote application. 7.6 Tests were None N/A N/A Telecommunica performed to tions and Data verify that if the Transmission system utilizes electrical or optical transmission, that the ballot was sent via SSL, no receipt is utilized to verify. The client generated a blank ballot which does not contain voting selections. Once the blank ballot is delivered, and until the ballot package is saved, there are no external communication s between the voter and the ballot delivery system; all interactions remain local to the voter’s environment. 19 | Pa ge Table 4B: Functional Security 7.8 Testing – Confirmed that None N/A N/A Security ImageCast Remote 5.2 does not have, nor require, internet access once the ballot has been downloaded. There are no external connections from the ballot to any outside server or service. With the exception of printing or saving the ballot package there are no external calls to or from the ballot. 20 | Pa ge Table 4B: Functional Security 7.8.1 Access Testing was None N/A N/A Control performed to confirm that the system access control was maintained. Attempted XXS attacks, SQL Injection attacks, directory listing/scans, attempted to pull directory file lists, scanned for default http login pages, scanned for robots_txt files, and pulled SSL certificate information. Review of this requirement validated that the requirement was satisfactorily covered. 7.8.2 Data Verified that this None N/A N/A Interception and system does Disruption not utilize telecommunicat ions for the transmission of official voting data – only the delivery of a blank ballot that does not contain voter data or choice selections. 21 | Pa ge Telecommunications and Data Transmission Testing Finally in conducting the Security and Telecommunications testing on the ImageCast Remote 5.2 RAVBMS, SLI performed telecommunications and data transmission testing. SLI tested the communication of the ImageCast Remote 5.2 system, including encryption of data, as well as protocols and procedures for access authorization. The following table, Table 4C: Telecommunications and Data Transmission Testing, outlines the testing performed, vulnerabilities, and the vendor response/mitigation when applicable. Table 4C: Telecommunications and Data Transmission Testing CVSS Testing Performed Vulnerability Response/Mitigation Standard 6.1.2 Data After the interactive ballot None N/A Transmission application is launched, connectivity to and from the ballot were confirmed to be nonexistent, with the exception of calls to the local system for print functionality. 6.2. Design, Verified that ImageCast None N/A Construction, Remote 5.2 system consists and of a generated ballot which is Maintenance typically used for absentee Requirements and mail-in ballot marking. ImageCast Remote 5.2 does not utilize specific telecommunications channels once the ballot has been downloaded and opened on the end voter’s machine. 6.2.1 Testing included accessing of None N/A Confirmation a ballot from the host and checking for confirmation. This requirement was determined to be not applicable, as the system only allows the voter to mark and confirm marked ballots prior to printing out a ballot summary card. There are no live 22 | Pa ge connections from the application to a remote server. All selections are cleared after browser has been closed. Security and Telecommunications Results SLI did identified two (2) potential high severity vulnerabilities, as listed below. Table 4D: Vulnerabilities and Mitigation/Response Vulnerability Vendor Mitigation/Response Potential Cross Site Cross site scripting (high severity, firm confidence): The Scripting opportunity in RAVBM application sanitizes all URL parameterized select components of the inputs to protect against the execution of malicious code. system. While the input string added to the URL “lng” parameter was reflected back in the website’s response, if it contained actual code it would have been sanitized such that it could not be executed on the server. This is done by converting certain symbols such as “>” to web-safe character codes, in this case “>” to prevent execution. Potential SQL injection SQL Injection (high severity, tentative confidence): The vulnerabilities arise when two pages referenced in the report are only available to user-controllable data is a user with elevated permissions (“super administrator”) incorporated into database and are not visible to or available to lower level users SQL queries in an unsafe (regular administrator or voter). Further, the application manner. is sanitizing all input fields so that executable code is not passed through, unmodified, to the application and/or database in a way that could exploit a SQL Injection vulnerability. 5. Software Review Testing Summary SLI Compliance performed a review of the ImageCast Remote 5.2 software. The purpose of the review was to identify discrepancies within the software code, and compliance with the California Voting System Standards (CVSS). The results and vendor mitigation/response are listed in the following table: Table 5A: Source Code Review Discrepancy and Mitigation/Response Discrepancy Vendor Mitigation/Response There were 180 source code The Dominion Voting Systems RAVBM product, 23 | Pa ge Table 5A: Source Code Review Discrepancy and Mitigation/Response requirements found to be at ImageCast Remote, utilizes credible published issue within the RAVBMS standards to ensure code quality in conformance with source code base reviewed; as California Voting System Standards section 5.2.3. a result, 180 discrepancies For the C# code, the Microsoft StyleCop standard were written and tool are used. For the JavaScript portion of the against the code base. code, the ESLint standard and tool are used. Reviewing the listed discrepancies, they appear to be 80 instances were noted purely stylistic in nature and do not adversely affect where lines of source the usability, security, or readability of the submitted code exceeded more source code. than 120 characters in length 71 instances were noted where numbers were not set to constant 23 instances were noted where variable declarations were without comment 10 instances were noted where no default case existed Two instances were noted where variable names not differing by more than one character were utilized One instance was noted where inconsistent indentation was implemented IV. COMPLIANCE WITH STATE AND FEDERAL LAWS AND REGULATIONS 1. Elections Code Review 1) §19293 (b) Remote accessible vote by mail system standards adopted by the Secretary of State pursuant to subdivision (a) shall include, but not be limited to, all of the following requirements: 24 | Pa ge (1) The machine or device and its software shall be suitable for the purpose for which it is intended. (2) The remote accessible vote by mail system shall preserve the secrecy of the ballot. (3) The remote accessible vote by mail system shall be safe from fraud or manipulation. (4) The remote accessible vote by mail system shall be accessible to voters with disabilities and to voters who require assistance in a language other than English if the language is one in which a ballot or ballot materials are required to be made available to voters. The system meets these requirements. The system demonstrated English, Spanish and Chinese languages, however with external translation services; the system can support other languages as required. 2) §19295 A remote accessible vote by mail system or part of a remote accessible vote by mail system shall do not any of the following: (a) Have the capability, including an optional capability, to use a remote server to mark a voter’s selections transmitted to the server from the voter’s computer via the Internet. (b) Have the capability, including an optional capability, to store any voter identifiable selections on any remote server. (c) Have the capability, including the optional capability, to tabulate votes. The system meets these requirements. 2. Section 508 and WCAG Compliance Review ImageCast Remote 5.2 was written to WCAG 2.0 guidelines to implement accessibility features. The system also complies with the applicable portions of Section 508. V. Conclusion The Dominion ImageCast Remote 5.2 remote accessible vote by mail system, in the configuration tested and documented by the California Dominion ImageCast Remote 5.2 Use Procedures, meets applicable California and federal laws. 25 | Pa ge Attachment A Dominion ImageCast Remote 5.2 Post-Test Survey Wave I Q1 For each question, please indicate how strongly you agree or disagree with the statement. Answered: 4 Skipped: 0 The marking of my ballot... I feel I can use this sys... I am confident that my ball... The instructions... The system was easy to use. I could read the display... I could understand t... The time frame it took to m... I can use this system witho... Printing was easy. My ballot printed with... 0 1 2 3 4 5 6 7 8 9 10 AGREE AGREE NEUTRAL DISAGREE DISAGREE TOTAL WEIGHTED STRONGLY SOMEWHAT SOMEWHAT STRONGLY AVERAGE The marking of my ballot method 75.00% 25.00% 0.00% 0.00% 0.00% was private. 3 1 0 0 0 4 1.25 I feel I can use this system to 50.00% 25.00% 0.00% 0.00% 25.00% mark my ballot independently. 2 1 0 0 1 4 2.25 I am confident that my ballot was 0.00% 75.00% 0.00% 0.00% 25.00% recorded accurately. 0 3 0 0 1 4 2.75 The instructions were clear and 0.00% 50.00% 0.00% 50.00% 0.00% complete. 0 2 0 2 0 4 3.00 The system was easy to use. 0.00% 25.00% 50.00% 25.00% 0.00% 0 1 2 1 0 4 3.00 I could read the display easily.(If 50.00% 50.00% 0.00% 0.00% 0.00% applicable) 1 1 0 0 0 2 1.50 1 / 6 Dominion ImageCast Remote 5.2 Post-Test Survey Wave I I could understand the speech 50.00% 50.00% 0.00% 0.00% 0.00% output. (If applicable) 1 1 0 0 0 2 1.50 The time frame it took to mark my 50.00% 50.00% 0.00% 0.00% 0.00% ballot was what I expected. 2 2 0 0 0 4 1.50 I can use this system without 25.00% 25.00% 0.00% 25.00% 25.00% technical help. 1 1 0 1 1 4 3.00 Printing was easy. 25.00% 50.00% 0.00% 0.00% 25.00% 1 2 0 0 1 4 2.50 My ballot printed with the correct 66.67% 0.00% 0.00% 0.00% 33.33% selections. 2 0 0 0 1 3 2.33 2 / 6 Dominion ImageCast Remote 5.2 Post-Test Survey Wave I Q2 Would you be satisfied using this system to mark your ballot in an election? Answered: 4 Skipped: 0 Yes No 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% ANSWER CHOICES RESPONSES Yes 75.00% 3 No 25.00% 1 TOTAL 4 3 / 6 Dominion ImageCast Remote 5.2 Post-Test Survey Wave I Q3 Would you rather mark your ballot using another method? Answered: 4 Skipped: 0 Yes No 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% ANSWER CHOICES RESPONSES Yes 25.00% 1 No 75.00% 3 TOTAL 4 4 / 6 Dominion ImageCast Remote 5.2 Post-Test Survey Wave I Q4 If you prefer another method, what method would you prefer? Answered: 1 Skipped: 3 # RESPONSES DATE 1 Electronic machine at poll or "Democracy live" mail system 12/16/2017 12:18 PM 5 / 6 Dominion ImageCast Remote 5.2 Post-Test Survey Wave I Q5 Do you have any suggestions for changes on this system and/or any other comments you would like to provide? Answered: 3 Skipped: 1 # RESPONSES DATE 1 1. "to make a selection, click in the box" not applicable or appropriate wording for screen-reader 12/16/2017 12:18 PM users. 2. Why say "party preference" next to choice rather than the name of the preferred party? 3. For write-in choices, I got warning that no choice had been made but could never find the edit box in which to write my choice. 4. No way to know where, or ability to write email address, date, etc. on the affidavit. 2 The verification failed and my ballot was not recorded. 12/16/2017 12:01 PM 3 The actual ballot was pretty easy to complete. Otherwise, I have concerns with the process of 12/15/2017 9:19 PM obtaining the ballot and how to return it. The registration process, waiting for the email, choosing a PIN and signing back into the site to get the ballot will confuse and frustrate most voters with disabilities. Once a PIN has been selected, will a voter be able use that PIN for every subsequent election? If getting a PIN is necessary, it may be better to have a voter who requests a RAVBM ballot register and choose a PIN at the time they make the request. I understand that, for counties that use the Dominion Imagecast voting system, the RAVBM ballot will not to be remade because the voting system will be able to read the RAVBM ballots directly. Those counties should not have to the second bullet in the affidavit because it is not necessary and will cause unnecessary confusion among voters. I found the envelope part confusing. I think it has to do with the fact that I did not have the actual envelope. I signed the printed envelope and put it in an envelope, but now think that I should have put it on the outside of the envelope. 6 / 6 Dominion ImageCast Remote 5.2 Remote Accessible Vote by Mail Pre-Test Survey Q1 Name Answered: 5 Skipped: 0 1 / 19 Dominion ImageCast Remote 5.2 Remote Accessible Vote by Mail Pre-Test Survey Q2 Age Answered: 5 Skipped: 0 # RESPONSES DATE 1 35 12/22/2017 5:09 PM 2 66 12/22/2017 9:11 AM 3 29 12/21/2017 10:41 AM 4 47 yo 12/18/2017 11:16 AM 5 44 12/18/2017 11:03 AM 2 / 19 Dominion ImageCast Remote 5.2 Remote Accessible Vote by Mail Pre-Test Survey Q3 Phone Answered: 5 Skipped: 0 19 Dominion ImageCast Remote 5.2 Remote Accessible Vote by Mail Pre-Test Survey Q4 Email Answered: 5 Skipped: 0 4 / 19 Dominion ImageCast Remote 5.2 Remote Accessible Vote by Mail Pre-Test Survey Q5 Please describe the disabilities or difficulties you have that may interfere with you being able to mark a ballot privately and independently. Answered: 5 Skipped: 0 # RESPONSES DATE 1 Autism 12/22/2017 5:10 PM 2 When I moved from one state to another; and when I moved to a city in another county 12/22/2017 9:16 AM 3 n/a - work related testing 12/21/2017 10:42 AM 4 Lack of fine motor control due to Cerebral Palsy 12/18/2017 11:26 AM 5 Ballot boxes being at my wheelchair level.. Random marks I make on ballot due to my hands not 12/18/2017 11:06 AM cooperating with my brain. Small print and my eyes tracking to find my choice. 5 / 19 Dominion ImageCast Remote 5.2 Remote Accessible Vote by Mail Pre-Test Survey Q6 Which language would you prefer voting in ? Answered: 5 Skipped: 0 English Spanish Chinese Japanese Tagalog Thai Vietnamese Hindi Khmer Korean 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% ANSWER CHOICES RESPONSES English 100.00% 5 Spanish 0.00% 0 Chinese 0.00% 0 Japanese 0.00% 0 Tagalog 0.00% 0 Thai 0.00% 0 Vietnamese 0.00% 0 Hindi 0.00% 0 Khmer 0.00% 0 Korean 0.00% 0 TOTAL 5 6 / 19 Dominion ImageCast Remote 5.2 Remote Accessible Vote by Mail Pre-Test Survey # OTHER (PLEASE SPECIFY) DATE There are no responses. 7 / 19 Dominion ImageCast Remote 5.2 Remote Accessible Vote by Mail Pre-Test Survey Q7 Have you ever voted before? Answered: 5 Skipped: 0 Yes No 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% ANSWER CHOICES RESPONSES Yes 100.00% 5 No 0.00% 0 TOTAL 5 8 / 19 Dominion ImageCast Remote 5.2 Remote Accessible Vote by Mail Pre-Test Survey Q8 If "No", what has prevented you from voting in the past? Answered: 1 Skipped: 4 # RESPONSES DATE 1 n/a 12/21/2017 10:42 AM 9 / 19 Dominion ImageCast Remote 5.2 Remote Accessible Vote by Mail Pre-Test Survey Q9 What method of voting do you use most often? Answered: 5 Skipped: 0 Polls By Mail Early Voting 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% ANSWER CHOICES RESPONSES Polls 40.00% 2 By Mail 40.00% 2 Early Voting 20.00% 1 TOTAL 5 # OTHER (PLEASE SPECIFY) DATE There are no responses. 10 / 19 Dominion ImageCast Remote 5.2 Remote Accessible Vote by Mail Pre-Test Survey Q10 Have you ever used a voting system with any special accessibility accommodations? Answered: 5 Skipped: 0 Yes No 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% ANSWER CHOICES RESPONSES Yes 60.00% 3 No 40.00% 2 TOTAL 5 11 / 19 Dominion ImageCast Remote 5.2 Remote Accessible Vote by Mail Pre-Test Survey Q11 If "Yes", which type of voting system? Answered: 3 Skipped: 2 Electronic Ballot Marking Device 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% ANSWER CHOICES RESPONSES Electronic 66.67% 2 Ballot Marking Device 33.33% 1 TOTAL 3 # OTHER (PLEASE SPECIFY) DATE There are no responses. 12 / 19 Dominion ImageCast Remote 5.2 Remote Accessible Vote by Mail Pre-Test Survey Q12 What operating system are you using? Answered: 3 Skipped: 2 Windows 10 Windows 8.1 Windows 7 macOS 10.12 Android Linux 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% ANSWER CHOICES RESPONSES Windows 10 33.33% 1 Windows 8.1 0.00% 0 Windows 7 33.33% 1 macOS 10.12 33.33% 1 Android 0.00% 0 Linux 0.00% 0 TOTAL 3 # OTHER (PLEASE SPECIFY) DATE 1 iOS 11.2 12/22/2017 5:12 PM 13 / 19 Dominion ImageCast Remote 5.2 Remote Accessible Vote by Mail Pre-Test Survey Q13 What internet browser are you using? Answered: 4 Skipped: 1 Microsoft Edge 14 Internet Explorer 11 Safari 10.1 Google Chrome 59.0 Mozilla Firefox 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% ANSWER CHOICES RESPONSES Microsoft Edge 14 0.00% 0 Internet Explorer 11 0.00% 0 Safari 10.1 25.00% 1 Google Chrome 59.0 75.00% 3 Mozilla Firefox 0.00% 0 TOTAL 4 # OTHER (PLEASE SPECIFY) DATE There are no responses. 14 / 19 Dominion ImageCast Remote 5.2 Remote Accessible Vote by Mail Pre-Test Survey Q14 List the auxiliary devices and/or software you are using for electronic accessibility. Answered: 0 Skipped: 5 ! No matching responses. ANSWER CHOICES RESPONSES JAWS 0.00% 0 NVDA 0.00% 0 Apple Voiceover 0.00% 0 Windows Narrator 0.00% 0 TOTAL 0 # OTHER (PLEASE SPECIFY) DATE 1 Windows mouse keys, filter keys and slow keys 12/18/2017 12:49 PM 15 / 19 Dominion ImageCast Remote 5.2 Remote Accessible Vote by Mail Pre-Test Survey Q15 What is the make and model of the hardware you are using (PC, Laptop, Tablet, etc.)? Answered: 4 Skipped: 1 # RESPONSES DATE 1 Apple iPad Mini 12/22/2017 5:12 PM 2 Mac Air 12/22/2017 9:22 AM 3 Dell PC 12/21/2017 10:43 AM 4 PC 12/18/2017 12:49 PM 16 / 19 Dominion ImageCast Remote 5.2 Remote Accessible Vote by Mail Pre-Test Survey Q16 What is the printer make and model you are using? Answered: 3 Skipped: 2 # RESPONSES DATE 1 Brother 12/22/2017 5:12 PM 2 Brother DW2700 12/22/2017 9:22 AM 3 HP Laserjet 3011 12/18/2017 12:49 PM 17 / 19 Dominion ImageCast Remote 5.2 Remote Accessible Vote by Mail Pre-Test Survey Q17 What antivirus software are you using? Answered: 4 Skipped: 1 McAfee Norton Windows Defender Kaspersky 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% ANSWER CHOICES RESPONSES McAfee 25.00% 1 Norton 75.00% 3 Windows Defender 0.00% 0 Kaspersky 0.00% 0 TOTAL 4 # OTHER (PLEASE SPECIFY) DATE There are no responses. 18 / 19 Dominion ImageCast Remote 5.2 Remote Accessible Vote by Mail Pre-Test Survey Q18 Is your antivirus software up to date? Answered: 4 Skipped: 1 Yes No 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% ANSWER CHOICES RESPONSES Yes 100.00% 4 No 0.00% 0 TOTAL 4 19 / 19 Dominion ImageCast Remote 5.2 Post-Test Survey Wave II Q1 For each question, please indicate how strongly you agree or disagree with the statement. Answered: 1 Skipped: 0 The marking of my ballot... I feel I can use this sys... I am confident that my ball... The instructions... The system was easy to use. I could read the display... I could understand t... The time frame it took to m... I can use this system witho... Printing was easy. My ballot printed with... 0 1 2 3 4 5 6 7 8 9 10 AGREE AGREE NEUTRAL DISAGREE DISAGREE TOTAL WEIGHTED STRONGLY SOMEWHAT SOMEWHAT STRONGLY AVERAGE The marking of my ballot method 100.00% 0.00% 0.00% 0.00% 0.00% was private. 1 0 0 0 0 1 1.00 I feel I can use this system to 100.00% 0.00% 0.00% 0.00% 0.00% mark my ballot independently. 1 0 0 0 0 1 1.00 I am confident that my ballot was 0.00% 100.00% 0.00% 0.00% 0.00% recorded accurately. 0 1 0 0 0 1 2.00 The instructions were clear and 0.00% 0.00% 100.00% 0.00% 0.00% complete. 0 0 1 0 0 1 3.00 The system was easy to use. 0.00% 0.00% 100.00% 0.00% 0.00% 0 0 1 0 0 1 3.00 I could read the display easily.(If 100.00% 0.00% 0.00% 0.00% 0.00% applicable) 1 0 0 0 0 1 1.00 1 / 6 Dominion ImageCast Remote 5.2 Post-Test Survey Wave II I could understand the speech 0.00% 0.00% 0.00% 0.00% 0.00% output. (If applicable) 0 0 0 0 0 0 0.00 The time frame it took to mark my 100.00% 0.00% 0.00% 0.00% 0.00% ballot was what I expected. 1 0 0 0 0 1 1.00 I can use this system without 0.00% 0.00% 0.00% 100.00% 0.00% technical help. 0 0 0 1 0 1 4.00 Printing was easy. 0.00% 0.00% 0.00% 0.00% 100.00% 0 0 0 0 1 1 5.00 My ballot printed with the correct 100.00% 0.00% 0.00% 0.00% 0.00% selections. 1 0 0 0 0 1 1.00 2 / 6 Dominion ImageCast Remote 5.2 Post-Test Survey Wave II Q2 Would you be satisfied using this system to mark your ballot in an election? Answered: 1 Skipped: 0 Yes No 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% ANSWER CHOICES RESPONSES Yes 100.00% 1 No 0.00% 0 TOTAL 1 3 / 6 Dominion ImageCast Remote 5.2 Post-Test Survey Wave II Q3 Would you rather mark your ballot using another method? Answered: 1 Skipped: 0 Yes No 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% ANSWER CHOICES RESPONSES Yes 100.00% 1 No 0.00% 0 TOTAL 1 4 / 6 Dominion ImageCast Remote 5.2 Post-Test Survey Wave II Q4 If you prefer another method, what method would you prefer? Answered: 1 Skipped: 0 # RESPONSES DATE 1 Having to print & mail a pain. 1/2/2018 12:02 PM 5 / 6 Dominion ImageCast Remote 5.2 Post-Test Survey Wave II Q5 Do you have any suggestions for changes on this system and/or any other comments you would like to provide? Answered: 1 Skipped: 0 # RESPONSES DATE 1 1. Get feedback immediately after test. I took notes, but have forgotten some of what I want to tell 1/2/2018 12:02 PM 2. There was a drop down (I think date) that was not the format on the screen, would have given up instead of trying to enter when couldn't get 01 3 Write in name. Warning looks kill you can't type in 4.Of 2 people doing the test in the same room (dif devices & emails), one did not allow to print. 5. Contest:2of7 form District 1. Ballot said vote for 1, review said vote for 3; 6. Why does my computer say ballot package modified date after I am completing? 7. There is a lot to print. Printing wasted a sheet of paper. Wastes a lot if happens to everyone 8. Printing envelop needs to be redesigned so paper print fits. 9. Positive comment: I get mixed up between my various email addresses. The test let me start process on one and then move to another. 6 / 6 Dominion ImageCast Remote 5.2 Remote Accessible Vote by Mail Pre-Test Survey Wave III Q1 Name Answered: 8 Skipped: 0 19 Dominion ImageCast Remote 5.2 Remote Accessible Vote by Mail Pre-Test Survey Wave III Q2 Age Answered: 8 Skipped: 0 # RESPONSES DATE 1 62 2/2/2018 6:35 PM 2 64 1/25/2018 9:47 AM 3 56 1/25/2018 2:12 AM 4 46 1/23/2018 12:05 PM 5 70 1/22/2018 4:21 PM 6 69 1/22/2018 3:14 PM 7 44 1/22/2018 10:25 AM 8 64 1/22/2018 10:15 AM 2 / 19 Dominion ImageCast Remote 5.2 Remote Accessible Vote by Mail Pre-Test Survey Wave III Q3 Phone Answered: 8 Skipped: 0 3 / 19 Dominion ImageCast Remote 5.2 Remote Accessible Vote by Mail Pre-Test Survey Wave III Q4 Email Answered: 8 Skipped: 0 19 Dominion ImageCast Remote 5.2 Remote Accessible Vote by Mail Pre-Test Survey Wave III Q5 Please describe the disabilities or difficulties you have that may interfere with you being able to mark a ballot privately and independently. Answered: 8 Skipped: 0 # RESPONSES DATE 1 Let's say I am blind 2/2/2018 6:37 PM 2 Leg and foot injuries impair standing. Carpal Tunnel. 1/25/2018 9:48 AM 3 Blind 1/25/2018 2:14 AM 4 Totally blind. Way finding to polls can be an issue, and my arthritic left knee does not allow for 1/23/2018 12:19 PM climbing flights of stairs. I insist on stair-free places as a matter of principle. Have mild CP on right side, so all fine motor chores are left-handed. One-handed typist. This should not effect voting independently, but I will downgrade a polling place machine or system if an assumption of two functioning hands is present. 5 I cannot write with my hand. 1/22/2018 4:23 PM 6 Hands are shaky. Eyesight sometimes sees text as blurry. 1/22/2018 3:15 PM 7 vision, mobility, problem using hands, 1/22/2018 10:26 AM 8 totally blind 1/22/2018 10:16 AM 5 / 19 Dominion ImageCast Remote 5.2 Remote Accessible Vote by Mail Pre-Test Survey Wave III Q6 Which language would you prefer voting in ? Answered: 8 Skipped: 0 English Spanish Chinese Japanese Tagalog Thai Vietnamese Hindi Khmer Korean 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% ANSWER CHOICES RESPONSES English 87.50% 7 Spanish 0.00% 0 Chinese 12.50% 1 Japanese 0.00% 0 Tagalog 0.00% 0 Thai 0.00% 0 Vietnamese 0.00% 0 Hindi 0.00% 0 Khmer 0.00% 0 Korean 0.00% 0 TOTAL 8 6 / 19 Dominion ImageCast Remote 5.2 Remote Accessible Vote by Mail Pre-Test Survey Wave III # OTHER (PLEASE SPECIFY) DATE There are no responses. 7 / 19 Dominion ImageCast Remote 5.2 Remote Accessible Vote by Mail Pre-Test Survey Wave III Q7 Have you ever voted before? Answered: 8 Skipped: 0 Yes No 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% ANSWER CHOICES RESPONSES Yes 100.00% 8 No 0.00% 0 TOTAL 8 8 / 19 Dominion ImageCast Remote 5.2 Remote Accessible Vote by Mail Pre-Test Survey Wave III Q8 If "No", what has prevented you from voting in the past? Answered: 0 Skipped: 8 # RESPONSES DATE There are no responses. 9 / 19 Dominion ImageCast Remote 5.2 Remote Accessible Vote by Mail Pre-Test Survey Wave III Q9 What method of voting do you use most often? Answered: 8 Skipped: 0 Polls By Mail Early Voting 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% ANSWER CHOICES RESPONSES Polls 50.00% 4 By Mail 37.50% 3 Early Voting 12.50% 1 TOTAL 8 # OTHER (PLEASE SPECIFY) DATE 1 I vote early because I do not know whether my polling place is accessible and am not sure exactly 1/23/2018 12:19 PM where it is. Have only had one primary and general election since coming here. Alameda County's polling info was unclear on accessibility of the place, but I knew there was an accessible one in Oakland courthouse. 10 / 19 Dominion ImageCast Remote 5.2 Remote Accessible Vote by Mail Pre-Test Survey Wave III Q10 Have you ever used a voting system with any special accessibility accommodations? Answered: 8 Skipped: 0 Yes No 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% ANSWER CHOICES RESPONSES Yes 62.50% 5 No 37.50% 3 TOTAL 8 11 / 19 Dominion ImageCast Remote 5.2 Remote Accessible Vote by Mail Pre-Test Survey Wave III Q11 If "Yes", which type of voting system? Answered: 5 Skipped: 3 Electronic Ballot Marking Device 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% ANSWER CHOICES RESPONSES Electronic 80.00% 4 Ballot Marking Device 20.00% 1 TOTAL 5 # OTHER (PLEASE SPECIFY) DATE 1 Voice activating machine when being trained as a poll supervisor and a poll instructor 2/2/2018 6:37 PM 2 At least two models. 1/23/2018 12:19 PM 12 / 19 Dominion ImageCast Remote 5.2 Remote Accessible Vote by Mail Pre-Test Survey Wave III Q12 What operating system are you using? Answered: 5 Skipped: 3 Windows 10 Windows 8.1 Windows 7 macOS 10.12 Android Linux 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% ANSWER CHOICES RESPONSES Windows 10 40.00% 2 Windows 8.1 20.00% 1 Windows 7 40.00% 2 macOS 10.12 0.00% 0 Android 0.00% 0 Linux 0.00% 0 TOTAL 5 # OTHER (PLEASE SPECIFY) DATE 1 home prem 1/23/2018 12:46 PM 2 Mac OS X Yosemite Version 10.10 1/22/2018 10:32 AM 13 / 19 Dominion ImageCast Remote 5.2 Remote Accessible Vote by Mail Pre-Test Survey Wave III Q13 What internet browser are you using? Answered: 6 Skipped: 2 Microsoft Edge 14 Internet Explorer 11 Safari 10.1 Google Chrome 59.0 Mozilla Firefox 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% ANSWER CHOICES RESPONSES Microsoft Edge 14 0.00% 0 Internet Explorer 11 33.33% 2 Safari 10.1 0.00% 0 Google Chrome 59.0 50.00% 3 Mozilla Firefox 16.67% 1 TOTAL 6 # OTHER (PLEASE SPECIFY) DATE 1 with latest updates and serv packs 1/23/2018 12:46 PM 14 / 19 Dominion ImageCast Remote 5.2 Remote Accessible Vote by Mail Pre-Test Survey Wave III Q14 List the auxiliary devices and/or software you are using for electronic accessibility. Answered: 2 Skipped: 6 JAWS NVDA Apple Voiceover Windows Narrator 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% ANSWER CHOICES RESPONSES JAWS 50.00% 1 NVDA 50.00% 1 Apple Voiceover 0.00% 0 Windows Narrator 0.00% 0 TOTAL 2 # OTHER (PLEASE SPECIFY) DATE 1 I have both JAWS and NVDA. NVDA is primary. 1/23/2018 12:46 PM 2 Dragon Naturally Speaking 1/22/2018 4:27 PM 3 None 1/22/2018 10:32 AM 15 / 19 Dominion ImageCast Remote 5.2 Remote Accessible Vote by Mail Pre-Test Survey Wave III Q15 What is the make and model of the hardware you are using (PC, Laptop, Tablet, etc.)? Answered: 6 Skipped: 2 # RESPONSES DATE 1 All 2/2/2018 6:38 PM 2 Dell Desktop PC 1/25/2018 9:50 AM 3 Dell Opti-plex 755 PC desktop, win7 Home Premium Ed. 1/23/2018 12:46 PM 4 Dell PC, Intel (R) core (TM) I 7 – 4770 CPU at 3.40 GHz , installed memory (RAM): 12.0 GB, 64 1/22/2018 4:27 PM bit operating system, X 64 based processor 5 MacBook Air 1/22/2018 10:32 AM 6 PC, custom 1/22/2018 10:23 AM 16 / 19 Dominion ImageCast Remote 5.2 Remote Accessible Vote by Mail Pre-Test Survey Wave III Q16 What is the printer make and model you are using? Answered: 6 Skipped: 2 # RESPONSES DATE 1 Various 2/2/2018 6:38 PM 2 Xerox WorkCentre 5845/5855 1/25/2018 9:50 AM 3 none. Will send .tif of ballot via email. 1/23/2018 12:46 PM 4 Canon iP 1700 1/22/2018 4:27 PM 5 Canon MG8200 Series 2 1/22/2018 10:32 AM 6 HP Laserjet 2050 1/22/2018 10:23 AM 17 / 19 Dominion ImageCast Remote 5.2 Remote Accessible Vote by Mail Pre-Test Survey Wave III Q17 What antivirus software are you using? Answered: 4 Skipped: 4 McAfee Norton Windows Defender Kaspersky 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% ANSWER CHOICES RESPONSES McAfee 75.00% 3 Norton 0.00% 0 Windows Defender 25.00% 1 Kaspersky 0.00% 0 TOTAL 4 # OTHER (PLEASE SPECIFY) DATE 1 ESET Endpoint Antivirus 1/25/2018 9:50 AM 2 eSET NOD32 1/22/2018 10:23 AM 18 / 19 Dominion ImageCast Remote 5.2 Remote Accessible Vote by Mail Pre-Test Survey Wave III Q18 Is your antivirus software up to date? Answered: 6 Skipped: 2 Yes No 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% ANSWER CHOICES RESPONSES Yes 83.33% 5 No 16.67% 1 TOTAL 6 19 / 19 Dominion ImageCast Remote 5.2 Post-Test Survey Wave III Q1 For each question, please indicate how strongly you agree or disagree with the statement. Answered: 6 Skipped: 0 The marking of my ballot... I feel I can use this sys... I am confident that my ball... The instructions... The system was easy to use. I could read the display... I could understand t... The time frame it took to m... I can use this system witho... Printing was easy. My ballot printed with... 0 1 2 3 4 5 6 7 8 9 10 AGREE AGREE NEUTRAL DISAGREE DISAGREE TOTAL WEIGHTED STRONGLY SOMEWHAT SOMEWHAT STRONGLY AVERAGE The marking of my ballot method 0.00% 33.33% 50.00% 16.67% 0.00% was private. 0 2 3 1 0 6 2.83 I feel I can use this system to 33.33% 0.00% 33.33% 16.67% 16.67% mark my ballot independently. 2 0 2 1 1 6 2.83 I am confident that my ballot was 50.00% 0.00% 33.33% 0.00% 16.67% recorded accurately. 3 0 2 0 1 6 2.33 The instructions were clear and 50.00% 33.33% 0.00% 16.67% 0.00% complete. 3 2 0 1 0 6 1.83 The system was easy to use. 60.00% 0.00% 0.00% 20.00% 20.00% 3 0 0 1 1 5 2.40 I could read the display easily.(If 40.00% 0.00% 60.00% 0.00% 0.00% applicable) 2 0 3 0 0 5 2.20 1 / 7 Dominion ImageCast Remote 5.2 Post-Test Survey Wave III I could understand the speech 60.00% 20.00% 20.00% 0.00% 0.00% output. (If applicable) 3 1 1 0 0 5 1.60 The time frame it took to mark my 50.00% 16.67% 16.67% 0.00% 16.67% ballot was what I expected. 3 1 1 0 1 6 2.17 I can use this system without 33.33% 0.00% 50.00% 0.00% 16.67% technical help. 2 0 3 0 1 6 2.67 Printing was easy. 50.00% 0.00% 16.67% 16.67% 16.67% 3 0 1 1 1 6 2.50 My ballot printed with the correct 50.00% 0.00% 33.33% 0.00% 16.67% selections. 3 0 2 0 1 6 2.33 2 / 7 Dominion ImageCast Remote 5.2 Post-Test Survey Wave III Q2 Would you be satisfied using this system to mark your ballot in an election? Answered: 6 Skipped: 0 Yes No 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% ANSWER CHOICES RESPONSES Yes 66.67% 4 No 33.33% 2 TOTAL 6 3 / 7 Dominion ImageCast Remote 5.2 Post-Test Survey Wave III Q3 Would you rather mark your ballot using another method? Answered: 4 Skipped: 2 Yes No 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% ANSWER CHOICES RESPONSES Yes 50.00% 2 No 50.00% 2 TOTAL 4 4 / 7 Dominion ImageCast Remote 5.2 Post-Test Survey Wave III Q4 If you prefer another method, what method would you prefer? Answered: 3 Skipped: 3 # RESPONSES DATE 1 As I have said every time I have tested a vbm system, I do not think that anything that requires me 1/23/2018 5:35 PM to interface with print at all is independent or fully accessible. But in this case, I could not even complete the task of completely testing the system. I usually can get to the point of the printing screen, then make a .tif of the ballot. This is a picture I cannot verify, but it will make the sighted paranoid types happy. I prefer electronic voting at polls. I hope that someday we may have the ability to do it from home WITHOUT PRINT since sigs on printed envelopes are also problematic. 2 I found the system easy to use. Please see below my reservations about the possibility of hacking 1/22/2018 8:56 PM my completed ballot and associating it with my name. 3 Polling place, sighted help, anything else! 1/22/2018 1:14 PM 5 / 7 Dominion ImageCast Remote 5.2 Post-Test Survey Wave III Q5 Do you have any suggestions for changes on this system and/or any other comments you would like to provide? Answered: 4 Skipped: 2 # RESPONSES DATE 1 I was unable to test the Dominion voting system because the testing window was too short. On the 1/25/2018 2:28 AM day of the testing I had a busy day at work followed by an evening meeting, so I didn't have enough time to do the testing. I hope future testing sessions can run for at least 48 hours. 2 Here is a copy of the notes I took during testing. There are typos in the secret question choices 1/23/2018 5:35 PM and on the finish page. On the "get pin" page, the heading (lvl 2) that the important part of the page is under only says, "Something here". Spello on page after hitting submit from get pin. Also, relevant heading says "something here". 189114 1st contest: Checked 12, 4, 8 and want a write-in for 4th choice. Checked write-in as 4th choice, but cannot see a resulting edit box if present. Using NVDA. If I can't see one. Will try JAWS. I think I now have managed to make a 5th choice as a write-in that I do not want, but received a msg from system that I could only remove from top down. Damned annoying. Trying to figure out whether I have to start over. If not, how do I fix this? Copying msg here: Warning Only the highest rank can be removed. Please close this message and remove rank from the highest ranked choice. "Warning! There is one or more unused ranks for this contest!" This is to be expected and is correct. Saw this w both JAWS and NVDA. I think I saw at least one warning w NVDA that said a write-in choice had not been filled out. Can't remember. JAWS is showing me 4 warnings that write-in box is empty. It is NOT showing me ANY checkboxes unless I first tab to them. It is also not telling me what the checkbox is associated with. JAWS's accelerator keys show me no edit boxes (possible) and no checkboxes which is incorrect. Having to start over w fresh ballot. 2nd Attempt at 1st contest: Have check the 3 I checked before. I use spacebar to check. Going to try enter to see if edit box on write-in appears. Think I'll try this with JAWS. NVDA doesn't always deal with edit boxes well if not coded exactly. I could not find an edit boxe w either screen reader after hitting enter. Don't know how to force JAWS or NVDA into edit mode to see if I could type, but did not take the time to try to find out since I did not know for sure that true edit box was present. Suche extra work should NOT be required! If write-ins are not possible, they should not be available. Last Attempt, skipping write-in: Checked 3 as before. Got warning that ballot was not valid. Had to figure out that I must click review to proceed. Steps to complete voting: 1 (completed) 2 (completed) 3Review (current step) 4 (next step) As shown here (copied from review page), no step except the current one is actually labeled, so if the review was listed as a step, it is not readable ahead of being on that page, so one does not know to go there intuitively. It is also not listed in instructions for voting page. Curious text from review page pasted here. Rankings correct. Please review your selections carefully. If you are satisfied with your selections, click on the "Mark your ballot" button. If you want to change any of your selections, click on the "Go Back" button to navigate back through the contests. Once you have selected the "Cast Ballot" button below, your vote will be cast and you will be unable to make any further changes. Contest: 1 of 1. Rank no more than TEN Choices to elect 4 positions. Contest is not valid. Preferential Choices Warning!There is one or more unused ranks for this contest! You have ranked these candidates: Choice Candidate 12 Party 12 rank 1. Choice Candidate 4 Party 4 rank 2. Choice Candidate 8 Party 8 rank 3. TODO: Instructions for review visible to screen reader only. (Which instructions are visible to screen readers only? There is nothing listed as an instruction.) the next thing after this msg are the two buttons "go back" and "mark your ballot". *puzzled frown* Tried to mark ballot. No confirmation page or other change appears to have taken place. Just noticed the bit about contest not being valid. Saw that on prev page before clicking review, but don't see anything wrong here and didn't see anything wron there, either. In addition to the foregoing notes, it is clear to me that web accessibility standards were not followed in the coding of the pages of this system. 27.5 years post ADA, that is infuriating. Instructions need to be clearer (the only complete ones were those for signing up) and the step lists need to be fully labeled. One should be able to successfully write in a candidate and be able to change any one of the choices that has been made. The rigors of completing authentication concern me for folks who are not very computer literate and/or have cognative issues. There were a LOT of little details and steps to correctly complete just to be able to vote. 6 / 7 Dominion ImageCast Remote 5.2 Post-Test Survey Wave III 3 It seems possible that my completed ballot could be observed by an outside party, and that the 1/22/2018 8:56 PM ballot could be associated with me. I need significantly more assurance than I have before I would feel comfortable using this method. 4 22 steps before voting is cumbersum. 2. Still says "to make selection, click in the box containing 1/22/2018 1:14 PM your choice". Not useful. 3. Still can't check write-in candidate. No findable place to write name. 4. Could not tell if I had checked a write-in in the candidate list. Warning message tossed me back to the top of the page. When I realized I'd checked an extra write-in, could not remove it because I was told I could only remove top ranking. Very very bad design! So I just left it alone and submitted my ballot with things I didn't want. 6. Affidavit: Still can't write in my email address and date. Picked a random spot for signing. I would not vote by mail if my county used this system. 7 / 7