County of Los Angeles Vsap 2.1 Executive Summary Report

Executive Summary County of Los Angeles’ Voting Solutions for All People (VSAP) 2.1 The County of Los Angeles applied for certification of the VSAP 2.1 voting system on February 27, 2020. Through the formal California contracting process and pursuant to California Elections Code section 19285, the Secretary of State awarded a contract to SLI Compliance as the state-approved testing contractor in May of 2020. Review included all phases of testing, and the scope included reviewing findings from the 2.0 testing. The testing included: (1) Accessibility, Usability and Privacy Testing, (2) Software Testing, (3) Hardware Testing, (4) Security and Telecommunications Testing, (5) Functional Testing, and (6) Volume Testing. 1. Accessibility, Usability and Privacy Testing (July 30 to 31, 2020) VSAP 2.0 Findings and Status: If a voter tries to vote for more than the allowable number of candidates, the ballot marking device cancels the first choice and records the second without informing the voter of the change (Resolved) The ballot marking device has a second headphone jack that allows a poll worker to listen to the audio ballot for the purpose of assisting a voter using the audio ballot. (Resolved) Long candidate lists require the voter to scroll on ballot marking devices. (Resolved) Delays in and/or no audio responses observed. (Resolved) Foreign language ballots only print the yes and no selections for issues in the chosen language. (Resolved) Instructions missing on audio ballot (Resolved) Sanitized headset or handset (Resolved) Tactile discernibility of controls (Resolved) Initial volume (Resolved) Visibility of controls, keys, audio jacks (Resolved) Noted that the symbols are raised but small and are in the same color as the molding of the case. VSAP 2.1 Findings and Status: Test voters consistently reported that they liked the ballot marking device. Most test voters felt that they could independently vote, without assistance, and that their votes were recorded accurately. However, a few voters did note that there was some confusion with the system instructions. One test voter pointed out that the selection button on the tactile remote was the only button that did not have Braille signage near it and suggested that would be nice to have. Another voter felt the “More” button should be brighter to draw your attention to it. Multiple test voters felt the tactile buttons were too sensitive. One of the conditions of VSAP 2.0 was that the “More” button be addressed. The County enhanced the feature of the “More” button by adding an overlay on the screen, for visually impaired voters a verbal queue, that there are additional selections the voter can see by pressing the “More” button. Accessibility, Usability and Privacy Report - https://votingsystems.cdn.sos.ca.gov/vendors/LAC/vsap2-1/vsap21-aup.pdf Page 1 of 4 2. Software Testing (Source Code Review) June 29 to July 31, 2020 VSAP 2.0 Findings and Status: The low-level findings were reviewed, tested, and confirmed to be resolved except for the FIPS findings and the Calico finding, which are both low severity. The FIPS findings resolutions are due by June 30, 2021. The Calico finding cannot be resolved as it is a third- party tool, and the vendor of that tool has not developed a fix. The County is actively monitoring Calico for a fix. VSAP 2.1 Findings and Status Each of the following findings are low severity and have no impact on the functionality of the system. The tool used to review the code identified these as findings, but these are coding format preferences. There were four categorized findings of the VSAP 2.1 code base: catch()” issue, five instances (CVSS 5.2.5.a) Known Language Vulnerability, one instance (CVSS 5.2.8.b.v) Unused function issue, one instance (CVSS 5.2.7.e) compArray issue, one instance (CVSS 5.2.5.a) Software Report - https://votingsystems.cdn.sos.ca.gov/vendors/LAC/vsap2-1/vsap21-source.pdf 3. Hardware Testing June 29, 2020 – July 23, 2020 VSAP 2.0 Findings and Status: There were no outstanding hardware findings. VSAP 2.1 Findings and Status: Several thousand of the ballot marking devices received replacement gears in response to the jamming the machines had during the 2.0 testing. Machines with the replacement gears and machines without replacement gears were tested. All successfully passed the testing without issue. Hardware Report - https://votingsystems.cdn.sos.ca.gov/vendors/LAC/vsap2-1/vsap21-hw.pdf 4. Security and Telecommunications Testing (Red Team) (June 29 to July 23, 2020) VSAP 2.0 Findings and Status: 1. FIPS 140-2 (Pending) – The County has until June 30, 2021 to address the FIPS related findings. 2. Full Disk Encryption (Partially Addressed) – The County submitted a plan to the Secretary of State for full disk encryption on July 24, 2020. It is recommended that the County align the implementation of full disk encryption with the FIPS due date, so that both can be reviewed at that time. 3. Dependency on Root Access (Partially Addressed) – The county removed USB access to the system, however full disk encryption will provide a full remedy. 4. Shared/Static Secrets (Partially Addressed) – This will be resolved with full disk encryption. VSAP 2.1 Findings and Status: Access control and Nessus scan findings are new findings. Access control will be remedied with full disk encryption and an onsite audit. The Nessus scan identified two Page 2 of 4 critical, two high, 14 medium, and five low findings. Each of the findings involve vulnerabilities due to patching or configuration settings in third party or COTS products. The county will be required to, within 10 days of the certification, either apply the patches/updates and make configuration changes to address the vulnerability or provide documentation regarding why the changes cannot be made and how the changes would impact the system’s accuracy and efficiency. Security and Telecommunications Report - https://votingsystems.cdn.sos.ca.gov/vendors/LAC/vsap2- 1/vsap21-sec.pdf 5. Functional Testing (July 6 to July 10, 2020) VSAP 2.0 Findings and Status: No description of log entries or analysis of the logs were found in the System Operating Manual - CVSS 2.1.5: Resolved and Verified Ballot jam messages were not recorded in the logs using the same text string as displayed. - CVSS 2.1.5.1.g: Resolved and Verified If a voter tries to vote for more than the allowable number of candidates, the ballot marking device cancels the first choice and records the second without informing the voter of the change: Resolved Long candidate lists require the voter to scroll on ballot marking device. - CVSS 3.2.7.a: Resolved Delays in and/or no audio responses observed. - CVSS 3.2.8.b: Resolved The buttons on the keypad of the ballot marking devices do not have the variable resistance that allows touch discernible use. The keys depress smoothly and nothing is felt until they bottom out and activate. - CVSS 3.3.3.f: Resolved Although there are no counters for extraneous items in the system extraneous perforations, smudges and folds will be captured in the graphic cast vote records. - CVSS 4.1.5.2.b: Resolved No data plate containing required information was found on any ballot marking device. - CVSS 4.3.4.a: Resolved VSAP 2.1 Findings and Status: Four test elections were conducted, and ballots marked for each election in a pre-determined marking pattern. The ballots were tabulated successfully, and the results audited by a 1% manual tally. There were no new findings. Functional Report - https://votingsystems.cdn.sos.ca.gov/vendors/LAC/vsap2-1/vsap21-funct- reprt.pdf 6. Volume Testing (July 21 to August 2020) VSAP 2.0 Findings and Status: Los Angeles County was required as a condition of certification of the VSAP 2.0 voting system to submit to the Secretary of State a plan to improve the ballot marking device paper handling and remediate the jamming and misfeed rate of the ballot marking device. The former rate was 0.0096%, which was not acceptable as prescribed in the California Voting Systems Standards (CVSS), which has an allowable rate of no more than 0.002%. Page 3 of 4 VSAP 2.1 Findings and Status: The Volume Test was conducted in two parts. During Volume Test Part I, one device out of the 51 overall devices tested experienced a jam, thus bringing the paper handling rate to 0.0002%, which is within the allowable rate as prescribed by the CVSS. Overall for VSAP 2.1 Volume Test Part I there were three findings: Timid Feeds, QR Code Errors, and Blank Screen/Paper Jam. There were 149 timid feeds, which are by design, a countermeasure to prevent jamming. Five ballot marking devices encountered a “QR Code not Read” error upon initial insertion into the ballot box. In all instances, the ballots were tried on another device, and successfully read. One ballot marking device would intermittently reject seemingly valid QR codes. When inserting the ballot, the feeder would start to take the ballot in, and then reject it. The ballot marking device was restarted, but the device continued to reject the ballots. Preventative maintenance was performed on the unit, but the unit continued to reject ballots. The device was removed from the test and taken for additional service and troubleshooting. During Part II of Volume testing, the County updated the firmware of the ballot marking devices in response to the QR code errors that were encountered during the Part I testing. The Secretary of State conducted Volume Test Part II, using 50 machines, at 50 ballots each, and a mixture of devices with the replacement gears and those without. Volume Test Part II resulted in five timid feeds, on five different devices. No QR Code Errors or Blank Screen/Paper Jams were experienced. Volume Report - https://votingsystems.cdn.sos.ca.gov/vendors/LAC/vsap2-1/vsap21-vol.pdf Conclusion Accessibility, Usability and Privacy testing, Hardware testing, and Functional testing verified 2.0 testing and had no new findings. Volume testing verified the fixes applied to the ballot marking devices (firmware update and gear replacement) have brought the devices well within an acceptable rate of paper handling as required by the CVSS. The Source Code testing verified that the findings required as a condition of certification of the VSAP 2.0 voting system were resolved. The VSAP 2.1 testing did uncover four new Source Code findings. The Source Code findings were determined to be low severity as they do not have any functional impact on the system but are coding formatting preferences. Security and Telecommunications testing verified that previous findings were resolved, except for those that are tied to FIPS certification and the implementation of full disk encryption, which will be submitted to the Secretary of State by June 30, 2021. The testing did uncover new Security findings. The new findings are related to access control and Nessus scan findings. The access control will be remedied with full disk encryption. The Nessus scan findings require updates or configuration of third party or COTS products. It is OVSTA’s recommendation that the VSAP 2.1 system be conditionally certified. Page 4 of 4