County of Los Angeles Vsap 2.1 Consultant's Functional Testing Report

County of Los Angeles VSAP Tally 2.1 Functional Test Report for California Secretary of State CAF-20001-FTR-01 Vendor Name County of Los Angeles Vendor System VSAP Tally 2.1 Prepared by: 4720 Independence St. Wheat Ridge, CO 80033 303-422-1566 www.SLICompliance.com Accredited by the U.S. Election Assistance Commission (EAC) for Selected Voting System Test Methods or Services Los Angeles County VSAP Tally 2.1 California Certification Functional Test Report v3.0 Revision History Date Release Author Revision Summary July 24, 2020 1.0 M. Santos Initial Release August 5, 2020 2.0 M. Santos Updated for CASOS comments August 5, 2020 3.0 M. Santos Updated for CASOS comments Disclaimer The information reported herein must not be used by the client to claim product certification, approval, or endorsement by NVLAP, NIST, or any agency of the Federal Government. Trademarks • SLI Compliance is a registered trademark of Gaming Laboratories International, LLC. • All products and company names are used for identification purposes only and may be trademarks of their respective owners. Copyright 2020 by SLI ComplianceSM, a division of Gaming Laboratories International, LLC. California Certification Functional Test Report Page 2 of 28 Report No. CAF-20001-FTR-01 Los Angeles County VSAP Tally 2.1 California Certification Functional Test Report v3.0 TABLE OF CONTENTS INTRODUCTION ........................................................................................................................................... 4 TESTING RESPONSIBILITIES .......................................................................................................................... 4 SCOPE OF THE VSAP TALLY 2.1 VOTING SYSTEM ............................................................................... 4 SYSTEM COMPONENT DESCRIPTION ............................................................................................................. 4 SYSTEM DESCRIPTION .................................................................................................................................. 6 BLOCK DIAGRAM .......................................................................................................................................... 9 SOFTWARE ................................................................................................................................................ 10 FUNCTIONAL TESTING ............................................................................................................................ 10 PHASE ONE – PHYSICAL CONFIGURATION AUDIT ......................................................................................... 11 PHASE TWO – INSTALLATION ...................................................................................................................... 12 PHASE THREE – FUNCTIONAL CONFIGURATION AUDIT (CVSS 9.11.2).......................................................... 14 PHASE FOUR – FUNCTIONAL TESTING ......................................................................................................... 14 EVALUATION OF TESTING ...................................................................................................................... 22 APPENDIX A: FINDINGS FROM VSAP TALLY 2.0 .................................................................................. 23 California Certification Functional Test Report Page 3 of 28 Report No. CAF-20001-FTR-01 Los Angeles County VSAP Tally 2.1 California Certification Functional Test Report v3.0 INTRODUCTION This Functional Test Report details the testing performed during functional testing of the County of Los Angeles’ Voting Solutions for All People (VSAP) Tally 2.1 (VSAP Tally 2.1) voting system against the California Voting System Standards (CVSS). Testing Responsibilities All testing was conducted under the guidance of personnel verified by the California Secretary of State (CASOS) to be qualified to perform the testing. Scope of the VSAP Tally 2.1 Voting System This section provides a description of the scope of the VSAP Tally 2.1 voting system components. System Component Description The VSAP Tally 2.1 voting system is composed of six core components: • Ballot Marking Device (BMD) • BMD Manager (BMG) • Enterprise Signing Authority (ESA) • Interactive Sample Ballot (ISB) • Tally • VSAP Ballot Layout (VBL). Ballot Marking Device (BMD) The BMD is the primary touchpoint for the voter and hub of the voting system, guiding users with screen prompts and symbols. The BMD features a touchscreen, an audio- tactile interface (controller and headphones), paper handler (scanner and printer), QR code scanner, and dual-switch input which voters use to generate, verify, and cast paper ballots. Completed ballots are transferred to the integrated ballot box, which can be detached for unloading. California Certification Functional Test Report Page 4 of 28 Report No. CAF-20001-FTR-01 Los Angeles County VSAP Tally 2.1 California Certification Functional Test Report v3.0 Ballot Marking Device Manager (BMG) The BMG manages and maintains BMDs. Its user interface enables operators to manage software, ballot configurations, and post-election data. The BMG provides files necessary for BMDs to present election data such as candidate information, multi- lingual audio, and supporting text. Enterprise Signing Authority (ESA) The ESA, also referred to as Digital Signing Authority (DSA), establishes the security root and chain of trust for the VSAP voting solution. This subsystem comprises the key management, distribution, and authentication functions. The ESA uses a cryptographic module to generate a public/private key pair to authenticate devices and transactions. The ESA is the basis of data integrity for the voting system. Interactive Sample Ballot (ISB) The ISB is a web-based application that allows voters to mark selections on a sample ballot, either on their computer or mobile device, prior to voting at a Vote Center. The ISB generates a Quick Response (QR) code, called a Poll Pass, containing voter selections to pre-populate selections in the BMD. The ISB also supports Remote Accessible Vote By Mail (RAVBM) and the Uniformed and Overseas Citizens Absentee Voting Act (UOCAVA). Tally Tally captures and processes ballot images to digitally count voter selections from paper ballots, including BMD and Vote by Mail (VBM). Tally scans and creates images of ballots, recording the images as Cast Vote Records (CVRs), tabulates them, and exports the election results. Tally is responsible for counting votes at the end of an election. VSAP Ballot Layout (VBL) The VBL enables election managers to configure and generate ballot layouts. The VBL subsystem ingests election information files and generates ballot layout files for use by other components of the system. The VBL provides a framework for election information. California Certification Functional Test Report Page 5 of 28 Report No. CAF-20001-FTR-01 Los Angeles County VSAP Tally 2.1 California Certification Functional Test Report v3.0 System Description Pre-Election During pre-election, the VBL application enables election managers to configure and generate ballot layouts and election files. The election data is exported to USB drives for use on VSAP Tally 2.1 components, providing them with a definition of the election and ballot layout information. The ESA application uses a cryptographic module to ensure each component of the VSAP Tally 2.1 system conforms to security standards and the data being transmitted to components is secure and authenticated. Initially, the ESA creates a secure environment, known as a Security World, in the Hardware Security Module (HSM). The ESA then uses sets of smart cards for administrators and operators to manage security keys. The ESA provisions Certificate Authorities to establish the security root and chain of trust. Once completed, the ESA generates public/private export key pairs for each target component (BMD, BMG, ISB, Tally, and VBL), and exports the keys, via USB drives, for use in the target servers. At the jurisdiction’s warehouse, the BMDs are connected to the BMG network using network cables. The BMG is equipped with a USB flash drive interface to receive security keys from ESA and election data from VBL. The BMG loads the operating system and software applications onto the BMDs and performs system verification through automated diagnostic tests. Election files are transferred from the BMG through the network to the BMDs. Files are exported from the BMG using USB flash drives. The BMG logs internal processes and user interactions to a database and provides mechanisms for querying, reporting, and exporting the log information. The BMG manages and maintains the BMDs and allows operators to manage software, configurations, and data. The BMG network is a secured, physically isolated and cabled local network, with no external connections, either wired or wireless. A secure, independent network such as this is known as an airgap. The BMG maintains the location information of BMDs connected to the BMG network. Processes and interactions are logged. BMDs that do not communicate on the BMG network or are diagnosed through BMG to have a fault can be located via the BMG for human inspection. California Certification Functional Test Report Page 6 of 28 Report No. CAF-20001-FTR-01 Los Angeles County VSAP Tally 2.1 California Certification Functional Test Report v3.0 The BMG extracts the BMDs’ public key files and sends them to Tally to be used for ballot validation. The ISB is a software application that allows voters to review election information and mark their sample ballots using a computer or mobile device, prior to formally voting at a Vote Center. There are two operating modes for the ISB: Preprocessor and Client application. In Preprocessor mode, ISB takes the election files from VBL and generates data packages optimized for use in the ISB Client application mode. Data mapping functions assemble the data regarding precincts, ballot styles, and parties to associate the voter with the appropriate precinct and identify the ballot style. Depending upon the type of voter, the ISB creates either a Poll Pass, a RAVBM ballot, or a UOCAVA ballot. To set up the ISB session, the voter inputs their address and zip code, allowing the Client application to identify the correct precinct and display the appropriate ballot. During Client application mode, the voter marks their selections on the sample ballot using a mobile device or computer, then reviews their selections. The ISB then generates a Poll Pass, which is a QR code representing the voter’s selection. The Poll Pass is saved on a mobile device or printed, then scanned at a BMD in the Vote Center to populate voting selections. The ISB also facilitates RAVBM and UOCAVA voting by generating a paper ballot that is printed by the voter and returned by mail or fax. Election Before voting starts, election workers scan a pass containing an authentication QR code and then use the touchscreen to enter a personal identification number (PIN) to activate the BMD and perform the poll opening procedures. During the voting day, voters are credentialled through the County’s ePollbook and receive their blank paper ballot from the election workers. The BMD scans the ballot’s Ballot Page Meta-data (BPM) QR code printed by the ePollbook onto the ballot, containing information used by the BMD to determine the appropriate ballot style to display. Voters interact with the BMD to mark their ballot selections using various interfaces including the touchscreen, controller, and dual-switch input device; to ensure privacy, headphones provide the audio interface. California Certification Functional Test Report Page 7 of 28 Report No. CAF-20001-FTR-01 Los Angeles County VSAP Tally 2.1 California Certification Functional Test Report v3.0 Alternatively, the voter can utilize their Poll Pass (obtained from pre-voting via the ISB) and scan its QR code at a BMD in the Vote Center to populate voting selections, after receiving their blank ballot from the poll worker After making selections, the voter’s ballot is printed. It displays election information, voting selections, and a Selection Barcode Encoding (SBE) QR code containing their selections and BMD information. The voter has an opportunity to review and verify the printed ballot before selecting the option to cast the ballot via the BMD’s paper handler, which deposits the validated paper ballot into the integrated ballot box. An election worker empties the ballot box, when it is full or at the close of polls, each night as a security measure, the BMD logs each time the ballot box is opened/emptied. At the end of the voting day, the election worker performs the poll closing procedures. The BMD creates Open Poll and Close Poll reports, election logs, and BMD interaction logs. Post-Election Following the election, the BMDs are returned to carts and moved back to the warehouse. Once the BMDs are reconnected to the BMG network, the BMD log files as well as election and interaction data are uploaded to the BMG. The Tally system is responsible for capturing and processing ballot images so voter selections from paper ballots (including BMD, VBM, RAVBM, and UOCAVA ballots) can be digitally counted. From the perspective of the software system architecture, Tally executes five main functions: a. The ballots are scanned and images are created. b. Images are processed. c. The ballot images are converted into Cast Vote Records (CVRs). d. The CVRs are tabulated. e. The election results from the tabulation are exported to support auditing and reporting. The Tally system uses an image scanner for capturing and processing ballot images to digitally count selections. Public keys from the BMDs used in the election are loaded into the Verifier and used to validate ballots; the BMD public key is contained within the SBE QR code printed on the ballot. Tally converts the scanned images into CVRs, the CVRs are tabulated and the election results from the tabulation are exported for use in other systems. California Certification Functional Test Report Page 8 of 28 Report No. CAF-20001-FTR-01 Los Angeles County VSAP Tally 2.1 California Certification Functional Test Report v3.0 Block Diagram The system overview of the submitted voting system is depicted in Figure 1. Figure 1. LA County VSAP Tally 2.1 Voting System Overview California Certification Functional Test Report Page 9 of 28 Report No. CAF-20001-FTR-01 Los Angeles County VSAP Tally 2.1 California Certification Functional Test Report v3.0 Software The Los Angeles County VSAP Tally 2.1 system is comprised of the following major components: 1. Tally, version 2.2.2.31 2. Ballot Marking Device (BMD), version 1.6 3. FormatOS, version 1.6.1 4. BMD BASI, version 1.6 5. BMD BESI, version 1.6 6. BMD Manager (BMG), version 1.5 7. VSAP Ballot Layout (VBL), version 1.1.3 8. Enterprise Signing Authority (ESA) (commercial off-the-shelf equipment [COTS]), version 1.0 9. IBML - ImageTrac 6400 (COTS) Functional Testing Prior to all testing, the Trusted Build of the software and firmware was created. Functional testing was divided into four phases, with some phases overlapping each other. • In phase one, the Physical Configuration Audit compared components submitted to the actual documentation. • In phase two, the Installation Phase included the steps necessary to install the system. • In phase three, the Functional Configuration Audit verified the system’s hardware and software perform all the functions listed in the documentation. • In phase four, Functional Testing exercised the system using operations necessary to conduct elections following the California Use Procedures for the system, documented the test results, and prepared benchmark data that can be used for system validation by the California Secretary of State (CASOS). During installation and functional testing, it was necessary to make minor edits to the California Use Procedures to provide clarity for end users. California Certification Functional Test Report Page 10 of 28 Report No. CAF-20001-FTR-01 Los Angeles County VSAP Tally 2.1 California Certification Functional Test Report v3.0 During examination and review performance, the system was configured as it would be for normal field use. This included connecting all supporting equipment and peripherals. Phase One – Physical Configuration Audit The Physical Configuration Audit (PCA) compared the voting system components submitted for certification to the manufacturer's technical documentation. This is an audit of all hardware and software in the system to compare the Technical Documentation Package (TDP) to the actual system. For the PCA, Los Angeles County provided: • Identification of all items that are to be a part of the software release. • Specification of compiler (or choice of compilers) to be used to generate executable programs. • Identification of all hardware that interfaces with the software. • Configuration baseline data for all hardware that is unique to the system. • Copies of all software documentation intended for distribution to users, including program listings, specifications, operations manual, voter manual, and maintenance manual. • User acceptance test procedures and acceptance criteria. • Identification of any changes between the physical configuration of the system submitted for the PCA and that submitted for the FCA, with a certification that any differences do not degrade the functional characteristics. • Complete descriptions of its procedures and related conventions used to support this audit by: Establishing a configuration baseline of the software and hardware to be o tested. Confirming whether the system documentation matches the corresponding o system components. All anomalies and omissions in the documentation identified by SOS during the PCA were corrected. California Certification Functional Test Report Page 11 of 28 Report No. CAF-20001-FTR-01 Los Angeles County VSAP Tally 2.1 California Certification Functional Test Report v3.0 Phase Two – Installation System Installation, Configuration, and Validation During Installation testing, the following was verified: • All boxes, system components, etc. have been labeled correctly and accurately. • The voting system has been labeled correctly. (CVSS 8.2.a). • A Configuration Log has been established. (CVSS 8.2.b). • All hardware that was used in the testing, including servers, workstations, monitors, printers, voting devices, and peripherals was documented. Build Software, Servers, and Workstations • All computers were wiped with Darik’s Boot and Nuke (DBAN). • All software and firmware components to be compiled in the trusted build were included and validated with HASHes from the manufacturer for COTS software/firmware components. • All hardware, compilers, and components needed to compile the trusted build were included and available. • The hardware provided was verified to meet or exceed the minimum requirements in the installation procedure manuals. • The required COTS software was verified to be available. • The component list of software and firmware exactly matched what was prescribed to be installed. • All software and firmware components were built per the California Use Procedures. • Los Angeles County Installation Procedures for the server and clients were successfully followed. • The sequence of steps in the installation procedures manual were followed. After the installation, COTS applications, proprietary applications, hardening, and configuration, images of each component of the system were taken. • Post installation, HASHes were taken of every piece of software and firmware. (Note: These HASHes will be provided to California counties along with the Trusted Build so that they may validate the system at any time, such as a post- election reinstall to meet California airgap requirements). • System security policies, data sources, and registry were verified to be properly documented. California Certification Functional Test Report Page 12 of 28 Report No. CAF-20001-FTR-01 Los Angeles County VSAP Tally 2.1 California Certification Functional Test Report v3.0 • System configuration and setup were audited against specifications in the manuals to verify that scripts used in installation and configuration achieved the specifications. • The Los Angeles County System Verification Procedures were verified to be applicable for each machine. • All components were HASH’d and verified to be correct. • The voting system was verified to deploy COTS protection against viruses, worms, Trojan horses, and logic bombs. (CVSS 7.4.2). Install Firmware on Hardware Devices • Hardware devices were examined and determined to have the correct version of firmware installed. • Instructions for firmware upgrades in use procedures and other system documentation were verified to be correct. • After firmware was installed, hardware devices were verified to be operational. • The Los Angeles County System Verification Procedures for each hardware device were verified to be correct to install the firmware on that device. • Verified that no compilers, assemblers, or source code were resident on the system. • Election specific firmware was verified to not be installed on the same component that the operating system is installed on. (CVSS 7.4.1.b.iv). • Verified all software setup validation requirements of CVSS 7.4.6. Post Installation • “Installed Programs” were verified to be as expected on all computer-based machines. • “Drivers” were verified to be as expected on all computer-based machines. • Images of all equipment were taken. • All system software and firmware hashes were taken. • A master copy of the “Trusted Build” directory structure, files, and “County Release” for distribution to counties was created. • “Trusted Build” software and Golden (County Release) images were created. California Certification Functional Test Report Page 13 of 28 Report No. CAF-20001-FTR-01 Los Angeles County VSAP Tally 2.1 California Certification Functional Test Report v3.0 Phase Three – Functional Configuration Audit (CVSS 9.11.2) The Functional Configuration Audit is conducted by SLI to verify that the system performs all the functions described in the system documentation. The documentation was verified for correctness. Phase Four – Functional Testing During the Functional Test Phase, the system was examined to determine that every functional piece of the system is accurate and complete. During the Functional Test Phase, an issue log was maintained of any errors and omissions found in the documentation or anomalies encountered that were not identified during the PCA. Both supported ballot sizes were tested as single sheet and multi-card ballots. The system was maintained in an air-gapped fashion: The architecture shall allow transfer of the election definition and tally database from the permanent server(s) to the sacrificial server (CVSS 7.4.1.a.i). Note that Findings from VSAP Tally 2.0 functional testing were reviewed during this phase. Please see “Appendix A – VSAP Tally 2.0 Functional Testing Findings and Review ” for additional details. Functional Testing Preparation Functional aspects of this phase included: • Kickoff meeting. Reviewed test plan and discussed how the equipment was to be allocated to each test in order to use time and personnel as efficiently as possible. • This project utilized elections from the VSAP Tally 2.0 campaign, except for the Primary Election: Presidential General (2016) (Los Angeles County) (3496) o Recall Election (2003 Election) (797) o Special Election (4084) – A special election with two congressional districts o and one municipality. Presidential Primary (2020 Election) (4085) o • These elections were used for the following functions: Create ballots on BMDs o Create VBM ballots o Create ballots on ISB (Poll Pass [mobile device, printed ballots], RAVBM o printed ballot, UOCAVA printed ballot) California Certification Functional Test Report Page 14 of 28 Report No. CAF-20001-FTR-01 Los Angeles County VSAP Tally 2.1 California Certification Functional Test Report v3.0 Remake RAVBM and UOCAVA ballots into VSAP Tally 2.1 ballots o Scan ballots on Tally. o Tabulate and report. o • Conducted logic and accuracy (L&A) testing in accordance with California Use Procedures. Los Angeles County L&A test deck generation software was used to generate o the test deck for L&A testing, as if being used by the County. Scanned the predetermined test deck through scanners. o Reviewed VSAP Tally 2.1 Use Procedures for L&A procedures per Los o Angeles County documentation. Printed and verified L&A results from scanners. o Tally refined and counted the CVRs to determine election results. o Verified that all components were ready to go after L&A or if they needed to o be re-provisioned/reimaged prior to the actual election. Functional Testing Summary The tests run on the LA County VSAP Tally 2.1 voting system included: • Presidential Primary Election • Presidential General Election • Recall Election • Special Election Test Presidential Primary Election A Primary election was run utilizing: • VSAP Tally 2.1 BMD’s • VSAP Tally 2.1 Central scanners The Presidential Primary had 13 languages, Armenian, Chinese, English, Farsi, Hindi, Japanese, Khmer, Korean, Russian, Spanish, Tagalog, Thai, and Vietnamese , and audio files. The election included seven different parties: Republican (Rep) which had three ballot styles and included 27 contests, Democratic (Dem) which had three ballot styles and included 35 contests, American Independent (AI) which had three ballot styles and included 28 contests, Peace and Freedom (PF) which had three ballot styles and included 27 contests, Green (Grn) which had three ballot styles and included 23 contests, Libertarian (LB) which had three ballot styles and included 25 contests, and a California Certification Functional Test Report Page 15 of 28 Report No. CAF-20001-FTR-01 Los Angeles County VSAP Tally 2.1 California Certification Functional Test Report v3.0 Non-Partisan (NP) which had three ballot styles and included 27 contests. This election was printed on the 8.5”x13” ballot size. The following steps were completed with results as noted: • Finalized Tally and set up for reporting. • Prepared BMDs for election. • Evaluated system for air-gap requirements. • Opened polls in accordance with California Use Procedures. • Printed and verified zero reports for all devices. • Marked ballots on BMD, per marking pattern. • Verified that the voter can review, confirm, and change their selections on the BMD. • Marked VBM Ballots • Used the ISB to generate BMD Poll Passes, RAVBM ballots, and UOCAVA ballots. • Created BMD ballots utilizing the Poll Passes. • Remade RAVBM and UOCAVA ballots into VSAP Tally 2.1 ballots. • Printed zero reports in Tally. • During Tally scanning, tested the following: Fed ballots in all 90-degree orientations on all devices. o Closed polls in accordance with California Use Procedures. o Printed results. o Saved results files as artifacts. o Transferred logs and ballot accounting back to BMG. o Shut down devices. o • Consolidated and reported: Uploaded results to BMG. o Canvass reconciliation. o Processed provisional ballots. ⮚ Using the adjudication component, adjudicated hand marked ballots with ⮚ write-ins. Generated all final reports available on the system. All reports saved to a o flash drive as artifacts of testing. Verified: o California Certification Functional Test Report Page 16 of 28 Report No. CAF-20001-FTR-01 Los Angeles County VSAP Tally 2.1 California Certification Functional Test Report v3.0 Canvass – Statement of Votes (SOV). ⮚ Supplement to the Statement of Votes (SSOV). ⮚ Precinct results. ⮚ Cast Vote Record Report. ⮚ Audit reports (Including tabulation devices). ⮚ • Observed and documented how over-votes and under-votes are tabulated. • Created California Election Night Auto-reporting files per the Calvoter template. • Backed up system to provide “Vote Count Program” to SOS. Evaluated default file name for submission to SOS. Checked backup size. • Verified system logging for all events. Saved system logs to archive. Test Presidential General Election A General election was run utilizing: • VSAP Tally 2.1 BMD’s • VSAP Tally 2.1 Central scanners The Presidential General was conducted in English, Korean, Chinese, and Vietnamese. The election included seven precincts and included 52 contests. This election was printed on the 8.5”x11” ballot size. Prepared all precinct components for election. • Configured one BMD for use in early voting and additional machines for Voting Day precinct voting with three different precincts. • Initialized and loaded election definition on the BMDs. • Opened polls in accordance with California Use Procedures. • Printed and verified zero reports for all devices. • Voted ballots for each precinct on a BMD. • Evaluated machine performance for a “fleeing voter.” • Attempted to vote on the BMD more than once. (CVSS 7.5.4.ix). • During Vote Center voting, tested the following: Marked ballots on BMD, per marking pattern. o Marked VBM ballots. o Verified language support for alternative languages on the following o components: California Certification Functional Test Report Page 17 of 28 Report No. CAF-20001-FTR-01 Los Angeles County VSAP Tally 2.1 California Certification Functional Test Report v3.0 BMD display (contrast, font size etc.). ⮚ BMD audio ballot. ⮚ BMD printing. ⮚ Used the ISB to generate BMD Poll Passes, RAVBM ballots, and UOCAVA o ballots. Created BMD ballots utilizing the Poll Passes. o Remade RAVBM and UOCAVA ballots into VSAP Tally 2.1 ballots. o Transferred logs and ballot accounting back to BMG. o • During Tally scanning, tested the following: Fed ballots in all directions/sides. o Verified language support for alternative languages. o • Closed polls in accordance with California Use Procedures. Printed results from all devices. o Transferred logs and ballot accounting back to BMG. o Shut down devices. o • Consolidated and reported: Canvass reconciliation: o Provisional ballots. ⮚ Using adjudication component, adjudicated all ballots with write-ins. ⮚ Generated all reports available on the system. Saved all reports as artifacts of o testing. • Verified: Canvass – SOV. o SSOV. o Precincts. o Audit reports (Including tabulation devices). o Recall Election A Recall election was run utilizing: • VSAP Tally 2.1 BMD’s • VSAP Tally 2.1 Central scanners The Recall General was conducted in English, Khmer, Japanese, and Hindi. California Certification Functional Test Report Page 18 of 28 Report No. CAF-20001-FTR-01 Los Angeles County VSAP Tally 2.1 California Certification Functional Test Report v3.0 The election included one precinct. It included two contests and 22 choices. This election was printed on the 8.5”x11” ballot size. Prepared all components for election: • Installed election definitions on devices, printed zero reports, and opened polls. • Marked one ballot with non-standard marks and increasingly marginal marks for each type of marker. Included a variety of pens, pencils, and highlighters in various colors. Ran a blank ballot and a fully marked ballot in a separate batch. • During voting, tested the following: Marked ballots on BMD, per marking pattern. o Language support for alternative languages: o BMD display (contrast, font size etc.). ⮚ BMD audio ballot. ⮚ BMD printing. ⮚ • Tested the ability to select candidates across multiple screens on the BMD. • Marked VBM ballots. • Used the ISB to generate BMD Poll Passes, RAVBM ballots, and UOCAVA ballots. • Created BMD ballots utilizing the Poll Passes. • Remade RAVBM and UOCAVA ballots into VSAP Tally 2.1 ballots. • Scanned ballots through Tally. • Closed each machine and printed out results. • Closed polls in accordance with California Use Procedures. • Printed results from all devices. • Transferred logs and ballot accounting back to BMG. • Shut down devices. • Consolidated and reported: Canvass reconciliation: o Provisional ballots. ⮚ Write-ins. ⮚ • Generated final reports and verified: Canvass – SOV. o SSOV. o California Certification Functional Test Report Page 19 of 28 Report No. CAF-20001-FTR-01 Los Angeles County VSAP Tally 2.1 California Certification Functional Test Report v3.0 Precincts. o Other sample user reports. o Audit reports (Including tabulation devices). o Special Election A Special Election was run utilizing: • VSAP Tally 2.1 BMD’s • VSAP Tally 2.1 Central scanners The election was conducted in English. The election included three precincts. It included three contests and eight choices. This election was printed on the 8.5”x11” ballot size. • Prepared all precinct components for election. • Configured BMD and Tally for use in early voting (all precincts). • Initialized and loaded election definition on BMD and BMG. • Opened polls in accordance with California Use Procedures. • Printed and verified zero reports for all devices. • Simulated early voting. Voted ballots on BMD and then suspended voting, re- enabled voting, and voted more ballots. • Marked ballots from marking pattern on a BMD. • Marked VBM ballots. • Used the ISB to generate BMD Poll Passes, RAVBM ballots, and UOCAVA ballots. • Created BMD ballots utilizing the Poll Passes. • Remade RAVBM and UOCAVA ballots into VSAP Tally 2.1 ballots. • Scanned ballots through Tally. • Closed each machine and printed out results. • Closed polls in accordance with California Use Procedures. • Set up adjudication to outstack all options for not counted ballots. Setup all scanners to notify for all error conditions. • Created a ballot manifest to be used for the post-election ballot level comparison risk-limiting audit (RLA). Ballot manifest was created in an Excel spreadsheet with batch and ballot position in that batch. • Printed Cast Vote Record file and exported for RLA. (CVSS7.7.3.b). California Certification Functional Test Report Page 20 of 28 Report No. CAF-20001-FTR-01 Los Angeles County VSAP Tally 2.1 California Certification Functional Test Report v3.0 • Verified ballot images are stored in a random manner. (CVSS 7.7.3). • Printed results from all scanners. • Printed zero reports from BMG. • Transferred results back to BMG. • Shut down devices. • Exercised adjudication workstation. • Consolidated and reported: Uploaded results to BMG from all units. o Canvass reconciliation. o Provisional ballots. o Write-ins. o • Generated final reports and verified: Canvass – SOV. o SSOV. o Precincts. o Other sample user reports. o Audit reports (Including tabulation devices). o • Perform mock RLA audit per EC19204.5. A ballot level comparison audit was performed, with a five percent risk limit. o Twenty-five rolls of a six-sided dice was used to seed the online software from Phillip Stark to determine which ballots were audited Additional Functional Review • Review of this item determined that bar/QR codes used to tally the ballots can be scanned by the voter, and comparing their ballot selection to the codes in the bar/QR code, verify that their votes on a ballot are correct. • Review of this item determined that the system does have the ability to meet EC19103 – backup files in compliance with County requirement to provide vote tally software to SOS. And the size of files are sufficient for VoteCal upload. Final Data Capture and Analysis • Evaluated system for any changes which occurred during testing. Generated new Trusted Build and images utilizing Los Angeles County Use Procedures. HASH’d all software and firmware components utilizing Los Angeles County Use Procedures. California Certification Functional Test Report Page 21 of 28 Report No. CAF-20001-FTR-01 Los Angeles County VSAP Tally 2.1 California Certification Functional Test Report v3.0 • Validated software and firmware on all devices using the procedure provided by Los Angeles County. Evaluation of Testing The above tests were conducted using the executables created in the Trusted Build, in association with the appropriate hardware versions as declared during the current certification project for the County of Los Angeles’ Voting Solutions for All People (VSAP) Tally 2.1 voting system, for the State of California. No functional issues were encountered during testing. As directed by the California Secretary of State, this report does not include any recommendation as to whether or not the system should be approved. End of Functional Test Report California Certification Functional Test Report Page 22 of 28 Report No. CAF-20001-FTR-01 Los Angeles County VSAP Tally 2.1 California Certification Functional Test Report v3.0 Appendix A: Findings from VSAP Tally 2.0 Version 2.0 of the VSAP Functional Testing resulted in 42 findings, of which 17 were reviewed during Functional Testing. The remainder of the findings were reviewed in other phases of testing such as Security and Telecommunications (Red Team Penetration), Software Testing (Source Code Review), Volume, and Accessibility, Usability and Privacy Testing. The 17 findings listed below were reviewed during the functional testing of VSAP Tally 2.1. 1. CVSS 2.1.5: “Because the actual implementation of specific characteristics may vary from system to system, it is the responsibility of the manufacturer to describe each system's characteristics in sufficient detail so that S-ATAs and system users can evaluate the adequacy of the system's audit trail. This description shall be incorporated in the System Operating Manual, which is part of the Technical Data Package.” VSAP Tally 2.0 Finding: No description of log entries or analysis of the logs were found in the System Operating Manual. Review of this item determined that Use Procedure documentation sufficiently covers this issue. 2. CVSS 2.1.5.1.g: “Voting systems shall provide a capability for the status messages to become part of the real-time audit record.” VSAP Tally 2.0 Finding: Ballot jam messages were not recorded in the logs using the same text string as displayed. Review of this item determined that the message recorded in the logs used equivalent text as to the cause of the ballot jam scenarios. 3. CVSS 2.3.3.3.f: “DRE and EBM systems shall notify the voter if he or she has attempted to make more than the allowable number of selections for any contest (e.g., over votes).” VSAP Tally 2.0 Finding: If a voter tries to vote for more than the allowable number of candidates, the BMD cancels the first choice and records the second without informing the voter of the change. Review of this item determined that: • In a vote for one contest, the device will switch to the next picked selection without any message being displayed. The selection is appropriately highlighted to show which option is picked. California Certification Functional Test Report Page 23 of 28 Report No. CAF-20001-FTR-01 Los Angeles County VSAP Tally 2.1 California Certification Functional Test Report v3.0 The expectation here is that the first selection should be manually deselected by the voter before being able to select a different candidate. • In a vote N of M, where N is greater than 1, once the maximum amount of selections have been picked, the device will not allow any additional selections without the voter unselecting a previously selected option. When the max amount of selections has been selected, the unselected options will be greyed out and are no longer selectable, but no messages appear to notify of the attempt at overvoting This is sufficient without the overvote attempt message being displayed, because the system is forcing the voter to deselect a choice, thereby taking them below the max limit threshold of selections. Additionally, there is a counter on every contest that notifies the voter as to how many selections they have left to make. 4. CVSS 3.2.2.1: “Notification of Effect of Over Voting – If the voter attempts to select more than the allowable number of choices within a contest on a VEBD or PCOS, the voting system shall notify the voter of the effect of this action before the ballot is cast and counted. In the case of manual systems, over votes may be mitigated through appropriately placed instructions.” VSAP Tally 2.0 Finding: When a voter attempts to over vote a race, the BMD automatically cancels the first choice and accepts the second without notifying the voter. Review of this item determined that the BMD continues to function as described above. For consistency of experience, the expectation here is that the first selection should be manually deselected by the voter before being able to select a different candidate. 5. CVSS 3.2.4.1. b: “During the voting session, the audio interface of the voting system shall be audible only to the voter.” VSAP Tally 2.0 Finding: The BMD has a second headphone jack that allows a poll worker to listen to the audio ballot for the purpose of assisting a voter using the audio ballot. Review of this item determined that second audio port has been designed and works as intended to allow the poll worker to hear what the voter is hearing while assisting with setting up the ballot. There is an expectation that the poll worker will disconnect from the audio port at the start of the voting session. California Certification Functional Test Report Page 24 of 28 Report No. CAF-20001-FTR-01 Los Angeles County VSAP Tally 2.1 California Certification Functional Test Report v3.0 6. CVSS 3.2.4.2.a: “No information shall be kept within an electronic CVR that identifies any alternative language feature(s) used by a voter.” VSAP Tally 2.0 Finding: Tally keeps a graphic image of each ballot scanned as the CVR. Images of mail in alternative language ballots would show the language used on the ballot. Review of this item determined that the ballot image is not the CVR, nor does the CVR contain the ballot image, only data in a CSV format. 7. CVSS 3.2.7.a: “No page scrolling - Voting systems shall not require page scrolling by the voter.” VSAP Tally 2.0 Finding: Long candidate lists require the voter to scroll on BMDs. Review of this item determined that no scroll bar exists, only a very prominent “More Button” to display more options that can be selected. When the contest is initially viewed, the device will highlight the “More Button” and inform the user of how it’s used. This message will appear on the initial viewing of every contest that has more options. 8. CVSS 3.2.8.b: “When the voter performs an action to record a single vote, the completed system response time of the VEBD shall be no greater than one second in the case of a visual response, and no greater than five seconds in the case of an audio response.” VSAP Tally 2.0 Finding: Delays in and/or no audio responses observed. Review of this item determined that no delayed or missing audios were observed. As soon as the touch screen was utilized on the BMD, an action takes place, such as moving to the next page or selecting a choice. Audio was tested over the course of a vote session, any physical button press will have a delay of about two to three seconds to respond audibly (example, pressing the next button. The next page will display immediately, then about two to three seconds the audio will start reading the page). It did take up to four seconds, but never met or exceeded five seconds. 9. VSS 3.2.9.a.ii: “Any records, including paper ballots and paper verification records, shall have sufficient information to support auditing by poll workers and others who can read only English.” VSAP Tally 2.0 Finding: Foreign language ballots only print the yes and no selections for issues in the chosen language. Review of this item determined that with appropriate documentation regarding the coding of the contest choices, that a poll worker and others would be able to read the ballot. California Certification Functional Test Report Page 25 of 28 Report No. CAF-20001-FTR-01 Los Angeles County VSAP Tally 2.1 California Certification Functional Test Report v3.0 10. CVSS 3.3.3.c.ii: “Sanitized headphone or handset - A sanitized headphone or handset shall be made available to each voter. This requirement can be achieved in various ways, including the use of ‘throwaway’ headphones, or of sanitary coverings.” VSAP Tally 2.0 Finding: The headphones provided are not disposable and no coverings were evident. Review of this item determined disposable headphone covers are made available. 11. CVSS 3.3.3.f: “Mechanically operated controls or keys on an accessible voting station shall be tactilely discernible without activating those controls or keys.” VSAP Tally 2.0 Finding: The buttons on the keypad of the BMD do not have the variable resistance that allows touch discernible use. The keys depress smoothly, and nothing is felt until they bottom out and activate. Review of this item determined that Braille is present next to each button and the buttons are discernible from each other. It was noted that some buttons (The round one in particular) are very sensitive, that by moving the hand over the buttons but not actively pressing it, the button was on occasion still activated as though it was pressed. 12. CVSS 3.3.5.c: “Labels, displays, controls, keys, audio jacks, and any other part of the accessible voting station necessary for the voter to operate the voting system shall be easily legible and visible to a voter in a wheelchair with normal eyesight (no worse than 20/40, corrected) who is in an appropriate position and orientation with respect to the accessible voting station.” VSAP Tally 2.0 Finding: The label for the headphone jack is slightly raised plastic in the same color as the body of the machine, it is almost invisible in diffused lighting with 20/20 vision. Another connection is a symbol which is indecipherable. Review of this item determined that labels and symbols were legible and visible. Symbols are raised but small and are in the same color as the molding of the case. 13. CVSS 4.1.4.2.d.iii: “Ballot boxes and ballot transfer boxes, which serve as secure containers for the storage and transportation of voted ballots, shall provide specific points where ballots are inserted, with all other points on the box constructed in a manner that prevents ballot insertion.” VSAP Tally 2.0 Finding: It is possible to insert or remove ballots from both the BMD and ballot transfer boxes without detection. California Certification Functional Test Report Page 26 of 28 Report No. CAF-20001-FTR-01 Los Angeles County VSAP Tally 2.1 California Certification Functional Test Report v3.0 Review of this item determined that security seals have been added to both the BMD as well as the ballot transfer boxes, to preclude insertion or removal of ballots, without detection. 14. CVSS 4.1.5.2.b: “Ignore, and not record, extraneous perforations, smudges, and folds.” VSAP Tally 2.0 Finding: Although there are no counters for extraneous items in the system, extraneous perforations, smudges, and folds will be captured in the graphic cast vote records. Review of this item determined that extraneous perforations, smudges, and folds do not affect the tabulation (recording) of results. 15. CVSS 4.3.4.a: “All voting systems shall display on each device a separate data plate containing a schedule for and list of operations required to service or to perform preventive maintenance.” VSAP Tally 2.0 Finding: No data plate containing required information was found on any BMD. Review of this item determined that a data plate with a schedule was found under the printer cover. 16. CVSS 5.4.2.c: “The ballot interpretation logic shall test and record the correct installation of ballot formats on voting devices.” VSAP Tally 2.0 Finding: The interpretation logic residing inside Tally does not test the installation of ballot formats on BMDs. Review of this item determined that Tally checks the QR code on the BMD ballot. If it’s missing or does not match the ballot manifest that has been loaded to the Tally, it will be rejected. This same applies to the paper ballots where a QR code is printed on the paper ballot that must be present and match the ballot manifest. If it is missing or not matching, the ballot will be rejected. QR codes can be verified through a third party QR reader app where the code that the Tally reads can be seen by human eyes and matched to the text on the ballot. 17. CVSS 5.4.3.b.iv: “System generated log of all normal process activity and system events that require operator intervention, so that each operator access can be monitored and access sequence can be constructed.” VSAP Tally 2.0 Finding: No log entries were found that appeared to be data quality monitor messages or by hardware condition monitors. California Certification Functional Test Report Page 27 of 28 Report No. CAF-20001-FTR-01 Los Angeles County VSAP Tally 2.1 California Certification Functional Test Report v3.0 Review of this item determined that the logs are uploaded to the network at the end of day from the BMD to the BMG Network. This can be found in the User Guide. End of Appendices California Certification Functional Test Report Page 28 of 28 Report No. CAF-20001-FTR-01