Los Angeles County Vsap 3.0 Consultant’s Functional Test Report

Los Angeles County Voting Solutions for All People (VSAP) 3.0 Functional Test Report for California Secretary of State CAF-21005-FTR-01 Vendor Name Los Angeles County Vendor System VSAP 3.0 Prepared by: 4720 Independence St. Wheat Ridge, CO 80033 303-422-1566 www.SLICompliance.com Accredited by the U.S. Election Assistance Commission (EAC) for Selected Voting System Test Methods or Services Los Angeles County VSAP 3.0 California Certification Functional Test Report v2.0 Revision History Date Release Author Revision Summary January 26th, 2022 1.0 D. Forrester, Initial Release M. Santos February 8th, 2022 2.0 D. Forester, Updated for CASOS comments M. Santos Disclaimer The information reported herein must not be used by the client to claim product certification, approval, or endorsement by NVLAP, NIST, or any agency of the Federal Government. Trademarks • SLI Compliance is a registered trademark of Gaming Laboratories International, LLC. • All products and company names are used for identification purposes only and may be trademarks of their respective owners. Copyright 2022 by SLI ComplianceSM, a division of Gaming Laboratories International, LLC. California Certification Functional Test Report Page 2 of 26 Report No. CAF-21005-FTR-01 Los Angeles County VSAP 3.0 California Certification Functional Test Report v2.0 TABLE OF CONTENTS INTRODUCTION ........................................................................................................................................... 4 TESTING RESPONSIBILITIES .......................................................................................................................... 4 SCOPE OF THE VSAP 3.0 VOTING SYSTEM ............................................................................................ 4 SYSTEM COMPONENT DESCRIPTION ............................................................................................................. 4 SYSTEM DESCRIPTION .................................................................................................................................. 6 BLOCK DIAGRAM .......................................................................................................................................... 9 COMPONENTS ............................................................................................................................................ 10 FUNCTIONAL TESTING ............................................................................................................................ 10 PHASE ONE – PHYSICAL CONFIGURATION AUDIT ......................................................................................... 11 PHASE TWO – INSTALLATION ...................................................................................................................... 12 PHASE THREE – FUNCTIONAL CONFIGURATION AUDIT (CVSS 9.11.2).......................................................... 14 PHASE FOUR – FUNCTIONAL TESTING ......................................................................................................... 14 EVALUATION OF TESTING ...................................................................................................................... 26 California Certification Functional Test Report Page 3 of 26 Report No. CAF-21005-FTR-01 Los Angeles County VSAP 3.0 California Certification Functional Test Report v2.0 INTRODUCTION This Functional Test Report details the testing performed during functional testing of the Los Angeles’ County Voting Solutions for All People 3.0 (VSAP 3.0) voting system against the California Voting System Standards (CVSS). Testing Responsibilities All testing was conducted under the guidance of personnel verified by the California Secretary of State (CASOS) to be qualified to perform the testing. Scope of the VSAP 3.0 Voting System This section provides a description of the scope of the VSAP 3.0 voting system components. System Component Description The VSAP 3.0 voting system is composed of six core components: • Ballot Marking Device (BMD) • BMD Manager (BMG) • Enterprise Signing Authority (ESA) • Interactive Sample Ballot (ISB) • Tally • VSAP Ballot Layout (VBL) Ballot Marking Device (BMD) The BMD is the primary touchpoint for the voter and hub of the voting system, guiding users with screen prompts and symbols. The BMD features a touchscreen, an audio- tactile interface (controller and headphones), paper handler (scanner and printer), QR code scanner, and dual-switch input which voters use to generate, verify, and cast paper ballots. Completed ballots are transferred to the integrated ballot box, which can be detached for unloading. California Certification Functional Test Report Page 4 of 26 Report No. CAF-21005-FTR-01 Los Angeles County VSAP 3.0 California Certification Functional Test Report v2.0 Ballot Marking Device Manager (BMG) The BMG manages and maintains BMDs. Its user interface enables operators to manage software, ballot configurations, and post-election data. The BMG provides files necessary for BMDs to present election data such as candidate information, multi- lingual audio, and supporting text. Enterprise Signing Authority (ESA) The ESA, also referred to as Digital Signing Authority (DSA), establishes the security root and chain of trust for the VSAP voting solution. This subsystem comprises the key management, distribution, and authentication functions. The ESA uses a cryptographic module to generate a public/private key pair to authenticate devices and transactions. The ESA is the basis of data integrity for the voting system. Interactive Sample Ballot (ISB) The ISB is a web-based application that allows voters to mark selections on a sample ballot, either on their computer or mobile device, prior to voting at a Vote Center. The ISB generates a Quick Response (QR) code, called a Poll Pass, containing voter selections to pre-populate selections in the BMD. The ISB also supports Remote Accessible Vote By Mail (RAVBM) and the Uniformed and Overseas Citizens Absentee Voting Act (UOCAVA). Tally Tally captures and processes ballot images to digitally count voter selections from paper ballots, including BMD and Vote by Mail (VBM). Tally scans and creates images of ballots, recording the images as Cast Vote Records (CVRs), tabulates them, and exports the election results. Tally is responsible for counting votes at the end of an election. VSAP Ballot Layout (VBL) The VBL enables election managers to configure and generate ballot layouts. The VBL subsystem ingests election information files and generates ballot layout files for use by other components of the system. The VBL provides a framework for election information. California Certification Functional Test Report Page 5 of 26 Report No. CAF-21005-FTR-01 Los Angeles County VSAP 3.0 California Certification Functional Test Report v2.0 System Description Pre-Election During pre-election, the VBL application enables election managers to configure and generate ballot layouts and election files. The election data is exported to USB drives for use on VSAP 3.0 components, providing them with a definition of the election and ballot layout information. The ESA application uses a cryptographic module to ensure each component of the VSAP 3.0 system conforms to security standards and the data being transmitted to components is secure and authenticated. Initially, the ESA creates a secure environment, known as a Security World, in the Hardware Security Module (HSM). The ESA then uses sets of smart cards for administrators and operators to manage security keys. The ESA provisions Certificate Authorities to establish the security root and chain of trust. Once completed, the ESA generates public/private export key pairs for each target component (BMD, BMG, ISB, Tally, and VBL), and exports the keys, via USB drives, for use in the target servers. At the jurisdiction’s warehouse, the BMDs are connected to the BMG network using network cables. The BMG is equipped with a USB flash drive interface to receive security keys from ESA and election data from VBL. The BMG loads the operating system and software applications onto the BMDs and performs system verification through automated diagnostic tests. Election files are transferred from the BMG through the network to the BMDs. Files are exported from the BMG using USB flash drives. The BMG logs internal processes and user interactions to a database and provides mechanisms for querying, reporting, and exporting the log information. The BMG manages and maintains the BMDs and allows operators to manage software, configurations, and data. The BMG network is a secured, physically isolated, and cabled local network, with no external connections, either wired or wireless. A secure, independent network such as this is known as an airgap. The BMG maintains the location information of BMDs connected to the BMG network. Processes and interactions are logged. BMDs that do not communicate on the BMG network or are diagnosed through BMG to have a fault can be located via the BMG for human inspection. California Certification Functional Test Report Page 6 of 26 Report No. CAF-21005-FTR-01 Los Angeles County VSAP 3.0 California Certification Functional Test Report v2.0 The BMG extracts the BMDs’ public key files and sends them to Tally to be used for ballot validation. The ISB is a software application that allows voters to review election information and mark their sample ballots using a computer or mobile device, prior to formally voting at a Vote Center. There are two operating modes for the ISB: Preprocessor and Client application. In Preprocessor mode, ISB takes the election files from VBL and generates data packages optimized for use in the ISB Client application mode. Data mapping functions assemble the data regarding precincts, ballot styles, and parties to associate the voter with the appropriate precinct and identify the ballot style. Depending on the type of voter, the ISB creates either a Poll Pass, a RAVBM ballot, or a UOCAVA ballot. To set up the ISB session, the voter inputs their address and zip code, allowing the Client application to identify the correct precinct and display the appropriate ballot. During Client application mode, the voter marks their selections on the sample ballot using a mobile device or computer, then reviews their selections. The ISB then generates a Poll Pass, which is a QR code representing the voter’s selection. The Poll Pass is saved on a mobile device or printed, then scanned at a BMD in the Vote Center to populate voting selections. The ISB also facilitates RAVBM and UOCAVA voting by generating a paper ballot that is printed by the voter and returned by mail or fax. Election Before voting starts, election workers scan a pass containing an authentication QR code and then use the touchscreen to enter a personal identification number (PIN) to activate the BMD and perform the poll opening procedures. During the voting day, voters are credentialled through the County’s ePollbook and receive their blank paper ballot from the election workers. The BMD scans the ballot’s Ballot Page Meta-data (BPM) QR code printed by the ePollbook onto the ballot, containing information used by the BMD to determine the appropriate ballot style to display. Voters interact with the BMD to mark their ballot selections using various interfaces including the touchscreen, controller, and dual-switch input device; to ensure privacy, headphones provide the audio interface. California Certification Functional Test Report Page 7 of 26 Report No. CAF-21005-FTR-01 Los Angeles County VSAP 3.0 California Certification Functional Test Report v2.0 Alternatively, the voter can utilize their Poll Pass (obtained from pre-voting via the ISB) and scan its QR code at a BMD in the Vote Center to populate voting selections, after receiving their blank ballot from the poll worker After making selections, the voter’s ballot is printed. It displays election information, voting selections, and a Selection Barcode Encoding (SBE) QR code containing their selections and BMD information. The voter has an opportunity to review and verify the printed ballot before selecting the option to cast the ballot via the BMD’s paper handler, which deposits the validated paper ballot into the integrated ballot box. An election worker empties the ballot box, when it is full or at the close of polls, each night as a security measure, the BMD logs each time the ballot box is opened/emptied. At the end of the voting day, the election worker performs the poll closing procedures. The BMD creates Open Poll and Close Poll reports, election logs, and BMD interaction logs. Post-Election Following the election, the BMDs are returned to carts and moved back to the warehouse. Once the BMDs are reconnected to the BMG network, the BMD log files as well as election and interaction data are uploaded to the BMG. The Tally system is responsible for capturing and processing ballot images so voter selections from paper ballots (including BMD, VBM, RAVBM, and UOCAVA ballots) can be digitally counted. From the perspective of the software system architecture, Tally executes five main functions: a. The ballots are scanned and images are created. b. Images are processed. c. The ballot images are converted into Cast Vote Records (CVRs). d. The CVRs are tabulated. e. The election results from the tabulation are exported to support auditing and reporting. The Tally system uses an image scanner for capturing and processing ballot images to digitally count selections. Public keys from the BMDs used in the election are loaded into the Verifier and used to validate ballots; the BMD public key is contained within the SBE QR code printed on the ballot. Tally converts the scanned images into CVRs, the CVRs are tabulated and the election results from the tabulation are exported for use in other systems. California Certification Functional Test Report Page 8 of 26 Report No. CAF-21005-FTR-01 Los Angeles County VSAP 3.0 California Certification Functional Test Report v2.0 Block Diagram The system overview of the submitted voting system is depicted in Figure 1. Figure 1. VSAP 3.0 Voting System Overview California Certification Functional Test Report Page 9 of 26 Report No. CAF-21005-FTR-01 Los Angeles County VSAP 3.0 California Certification Functional Test Report v2.0 Components The VSAP 3.0 system is comprised of the following major components: 1. Tally, version 3.0.20 2. Ballot Marking Device (BMD), version 3.0.0 3. FormatOS, version 3.0.2 4. BMD BASI, version 3.0.2 5. BMD BESI, version 3.0.2 6. BMD Manager (BMG), version 3.0.0 7. VSAP Ballot Layout (VBL), version 2.0.21 8. Enterprise Signing Authority (ESA) (commercial off-the-shelf equipment [COTS]), version 1.0 9. IBML - ImageTrac 6400 (COTS) 10. Ballot Scanner Fujitsu Scanner fi-7180 11. Ballot Scanner Fujitsu Scanner fi-7180PR Functional Testing Prior to all testing, the Trusted Builds were created for all updated software: FormatOS, VBL, BMG, BMD, Tally. Functional testing was divided into four phases, with some phases overlapping each other. • In phase one, the Physical Configuration Audit compared components submitted to the actual documentation. • In phase two, the Installation Phase included the steps necessary to install the system. • In phase three, the Functional Configuration Audit verified that the system’s hardware and software perform all the functions listed in the documentation. • In phase four, Functional Testing exercised the system using operations necessary to conduct elections following the California Use Procedures for the system, documented the test results, and prepared benchmark data that can be used for system validation by the California Secretary of State. During installation and functional testing, it was necessary to make minor edits to the California Use Procedures to provide clarity for end users. California Certification Functional Test Report Page 10 of 26 Report No. CAF-21005-FTR-01 Los Angeles County VSAP 3.0 California Certification Functional Test Report v2.0 During examination and review performance, the system was configured as it would be for normal field use. This included connecting all supporting equipment and peripherals. Phase One – Physical Configuration Audit The Physical Configuration Audit (PCA) compared the voting system components submitted for certification to the manufacturer's technical documentation. This is an audit of all hardware and software in the system to compare the Technical Documentation Package (TDP) to the actual system. For the PCA, Los Angeles County provided: • Identification of all items that are to be a part of the software release. • Specification of compiler (or choice of compilers) to be used to generate executable programs. • Identification of all hardware that interfaces with the software. • Configuration baseline data for all hardware that is unique to the system. • Copies of all software documentation intended for distribution to users, including program listings, specifications, operations manual, voter manual, and maintenance manual. • User acceptance test procedures and acceptance criteria. • Identification of any changes between the physical configuration of the system submitted for the PCA and that submitted for the Functional Configuration Audit (FCA), with a certification that any differences do not degrade the functional characteristics. • Complete descriptions of its procedures and related conventions used to support this audit by: Establishing a configuration baseline of the software and hardware to be o tested. Confirming whether the system documentation matches the corresponding o system components. All anomalies and omissions in the documentation identified by CASOS during the PCA were corrected. California Certification Functional Test Report Page 11 of 26 Report No. CAF-21005-FTR-01 Los Angeles County VSAP 3.0 California Certification Functional Test Report v2.0 Phase Two – Installation System Installation, Configuration, and Validation During Installation testing, the following was verified: • All boxes, system components, etc. have been labeled correctly and accurately. • The voting system has been labeled correctly. (CVSS 8.2.a). • A Configuration Log has been established. (CVSS 8.2.b). • All hardware that was used in the testing, including servers, workstations, monitors, printers, voting devices, and peripherals was documented. Build Software, Servers, and Workstations • All computers were wiped with Darik’s Boot and Nuke (DBAN). • All software and firmware components to be compiled in the trusted build were included and validated with HASHes from the manufacturer for COTS software/firmware components. • All hardware, compilers, and components needed to compile the trusted build were included and available. • The hardware provided was verified to meet or exceed the minimum requirements in the installation procedure manuals. • The required COTS software was verified to be available. • The component list of software and firmware exactly matched what was prescribed to be installed. • All software and firmware components were built per the California Use Procedures. • Los Angeles County Installation Procedures for the server and clients were successfully followed. • The sequence of steps in the installation procedures manual were followed. After the installation, COTS applications, proprietary applications, hardening, and configuration, images of each component of the system were taken. • Post installation, HASHes were taken of every piece of software and firmware. (Note: These HASHes will be provided to California counties along with the Trusted Build so that they may validate the system at any time, such as a post- election reinstall to meet California airgap requirements). • System security policies, data sources, and registry were verified to be properly documented. California Certification Functional Test Report Page 12 of 26 Report No. CAF-21005-FTR-01 Los Angeles County VSAP 3.0 California Certification Functional Test Report v2.0 • System configuration and setup were audited against specifications in the manuals to verify that scripts used in installation and configuration achieved the specifications. • The Los Angeles County System Verification Procedures were verified to be applicable for each machine. • All components were HASH’d and verified to be correct. • The voting system was verified to deploy COTS protection against viruses, worms, Trojan horses, and logic bombs. (CVSS 7.4.2). Install Firmware on Hardware Devices • Hardware devices were examined and determined to have the correct version of firmware installed. • Instructions for firmware upgrades in use procedures and other system documentation were verified to be correct. • After firmware was installed, hardware devices were verified to be operational. • The Los Angeles County System Verification Procedures for each hardware device were verified to be correct to install the firmware on that device. • Verified that no compilers, assemblers, or source code were resident on the system. • Election specific firmware was verified to not be installed on the same component that the operating system is installed on. (CVSS 7.4.1.b.iv). • Verified all software setup validation requirements of CVSS 7.4.6. Post Installation • “Installed Programs” were verified to be as expected on all computer-based machines. • “Drivers” were verified to be as expected on all computer-based machines. • Images of all equipment were taken. • All system software and firmware hashes were taken. • A master copy of the “Trusted Build” directory structure, files, and “County Release” for distribution to counties was created. • “Trusted Build” software and Golden (County Release) images were created. California Certification Functional Test Report Page 13 of 26 Report No. CAF-21005-FTR-01 Los Angeles County VSAP 3.0 California Certification Functional Test Report v2.0 Phase Three – Functional Configuration Audit (CVSS 9.11.2) The Functional Configuration Audit is conducted by SLI to verify that the system performs all the functions described in the system documentation. The documentation was verified for correctness. Phase Four – Functional Testing During the Functional Test Phase, the system was examined to determine that every functional piece of the system is accurate and complete. During the Functional Test Phase, an issue log was maintained of any errors and omissions found in the documentation or anomalies encountered that were not identified during the PCA. Both supported ballot sizes were tested as single sheet and multi-card ballots. The system was maintained in an air-gapped fashion: The architecture shall allow transfer of the election definition and tally database from the permanent server(s) to the sacrificial server (CVSS 7.4.1.a.i). Functional Testing Preparation Functional aspects of this phase included: • Kickoff meeting. Reviewed test plan and discussed how the equipment was to be allocated to each test in order to use time and personnel as efficiently as possible. • This project utilized the following elections: Presidential General (2020 Election) (4193) o Presidential Primary (2020 Election) (4085) o Gubernatorial Primary (2018 Election) (3793) o Gubernatorial General (2018 Election) (3861) o Recall Election (Special Recall) (972) o Special Election (4084) o • These elections were used for the following functions: Create ballots on BMDs o Create VBM ballots o Create ballots on ISB (Poll Pass [mobile device, printed ballots], RAVBM o printed ballot, UOCAVA printed ballot) Remake RAVBM and UOCAVA ballots into VSAP 3.0 ballots o Scan ballots on Tally o California Certification Functional Test Report Page 14 of 26 Report No. CAF-21005-FTR-01 Los Angeles County VSAP 3.0 California Certification Functional Test Report v2.0 Tabulate and report o • Conducted logic and accuracy (L&A) testing in accordance with California Use Procedures: Los Angeles County L&A test deck generation software was used to generate o the test deck for L&A testing, as if being used by the County. Scanned the predetermined test deck through scanners. o Reviewed VSAP 3.0 Use Procedures for L&A procedures per Los Angeles o County documentation. Printed and verified L&A results from scanners. o Tally refined and counted the CVRs to determine election results. o Verified that all components were ready to go after L&A or if they needed to o be re-provisioned/reimaged prior to the actual election. Functional Testing Summary The tests run on the LA County VSAP 3.0 voting system included: • Presidential Primary Election • Presidential General Election • Gubernatorial Primary • Gubernatorial General • Recall Election • Special Election Test Presidential Primary Election A Primary election was run utilizing: • VSAP 3.0 BMD’s • VSAP 3.0 Central scanners The Presidential Primary had 19 languages tested: Armenian, Chinese (Audio - Cantonese and Mandarin), English, Farsi, Hindi, Japanese, Khmer, Korean, Russian, Spanish, Tagalog, Thai, Vietnamese, Bengali, Burmese, Gujarati, Indonesian, Mongolian, and Telugu. The election included seven different parties: Republican (Rep) which had three ballot styles and included 27 contests, Democratic (Dem) which had three ballot styles and included 35 contests, American Independent (AI) which had three ballot styles and included 28 contests, Peace and Freedom (PF) which had three ballot styles and California Certification Functional Test Report Page 15 of 26 Report No. CAF-21005-FTR-01 Los Angeles County VSAP 3.0 California Certification Functional Test Report v2.0 included 27 contests, Green (Grn) which had three ballot styles and included 23 contests, Libertarian (LB) which had three ballot styles and included 25 contests, and a Non-Partisan (NP) which had three ballot styles and included 27 contests. For this election the VBM ballots were printed on 8.5”x14” ballot paper and the BMD ballots on 8”x13.25” thermal ballot paper. The following steps were completed with results as noted: • Finalized Tally and set up for reporting. • Prepared BMDs for election. • Evaluated system for air-gap requirements. • Opened polls in accordance with California Use Procedures. • Printed and verified zero reports for all devices. • Marked ballots on BMD, per marking pattern. • Verified that the voter can review, confirm, and change their selections on the BMD. • Marked VBM Ballots • Used the ISB to generate BMD Poll Passes, RAVBM ballots, and UOCAVA ballots. • Created BMD ballots utilizing the Poll Passes. • Remade RAVBM and UOCAVA ballots into VSAP 3.0 ballots. • Printed zero reports in Tally. • During Tally scanning, tested the following: Fed ballots in all 90-degree orientations on all devices. o Closed polls in accordance with California Use Procedures. o Printed results. o Saved results files as artifacts. o Transferred logs and ballot accounting back to BMG. o Shut down devices. o • Consolidated and reported: Uploaded results to BMG. o Canvass reconciliation. o Processed provisional ballots. ⮚ Using the adjudication component, adjudicated hand marked ballots with ⮚ write-ins. California Certification Functional Test Report Page 16 of 26 Report No. CAF-21005-FTR-01 Los Angeles County VSAP 3.0 California Certification Functional Test Report v2.0 Generated all final reports available on the system. All reports saved to a o flash drive as artifacts of testing. Verified: o Canvass – Statement of Votes (SOV). ⮚ Supplement to the Statement of Votes (SSOV). ⮚ Precinct results. ⮚ Cast Vote Record Report. ⮚ Audit reports (Including tabulation devices). ⮚ • Observed and documented how over-votes and under-votes are tabulated. • Created California Election Night Auto-reporting files per the Calvoter template. • Backed up system to provide “Vote Count Program” to SOS. Evaluated default file name for submission to SOS. Checked backup size. • Verified system logging for all events. Saved system logs to archive. Test Presidential General Election A General election was run utilizing: • VSAP 3.0 BMD’s • VSAP 3.0 Central scanners The Presidential General was conducted in all 20 languages: Armenian, Chinese (Audio - Cantonese and Mandarin), English, Farsi, Hindi, Japanese, Khmer, Korean, Russian, Spanish, Tagalog, Thai, Vietnamese, Bengali, Burmese, Gujarati, Indonesian, Mongolian, Telugu. The election included seven precincts and included 52 contests. For this election the VBM ballots were printed on 8.5”X14” ballot paper and the BMD ballots on 8”X13.25” thermal ballot paper. The following steps were completed with results as noted: • Prepared all precinct components for election. • Configured one BMD for use in early voting and additional machines for Voting Day precinct voting with three different precincts. • Initialized and loaded election definition on the BMDs. • Opened polls in accordance with California Use Procedures. California Certification Functional Test Report Page 17 of 26 Report No. CAF-21005-FTR-01 Los Angeles County VSAP 3.0 California Certification Functional Test Report v2.0 • Printed and verified zero reports for all devices. • Voted ballots for each precinct on a BMD. • Evaluated machine performance for a “fleeing voter.” • Attempted to vote on the BMD more than once. (CVSS 7.5.4.ix). • During Vote Center voting, tested the following: Marked ballots on BMD, per marking pattern. o Marked VBM ballots. o Verified language support for alternative languages on the following o components: BMD display (contrast, font size etc.). ⮚ BMD audio ballot. ⮚ BMD printing. ⮚ Used the ISB to generate BMD Poll Passes, RAVBM ballots, and UOCAVA o ballots. Created BMD ballots utilizing the Poll Passes. o Remade RAVBM and UOCAVA ballots into VSAP 3.0 ballots. o Transferred logs and ballot accounting back to BMG. o • During Tally scanning, tested the following: Fed ballots in all 90-degree orientations on all devices. o Verified language support for alternative languages. o • Closed polls in accordance with California Use Procedures. Printed results from all devices. o Transferred logs and ballot accounting back to BMG. o Shut down devices. o • Consolidated and reported: Canvass reconciliation: o Provisional ballots. ⮚ Using adjudication component, adjudicated all ballots with write-ins. ⮚ Generated all reports available on the system. Saved all reports as artifacts of o testing. • Verified: Canvass – SOV. o SSOV. o California Certification Functional Test Report Page 18 of 26 Report No. CAF-21005-FTR-01 Los Angeles County VSAP 3.0 California Certification Functional Test Report v2.0 Precincts. o Audit reports (Including tabulation devices). o Test Gubernatorial Primary Election A Gubernatorial Primary election was run utilizing: • VSAP 3.0 BMD’s • VSAP 3.0 Central scanners The Gubernatorial Primary was conducted in 14 languages: Armenian, Chinese (Audio - Cantonese and Mandarin), English, Farsi, Hindi, Japanese, Khmer, Korean, Russian, Spanish, Tagalog, Thai, Vietnamese. The election included seven precincts and included 52 contests. For this election the VBM ballots were printed on 8.5”x14” ballot paper and the BMD ballots on 8”x13.25” thermal ballot paper. The following steps were completed with results as noted: • Prepared all precinct components for election. • Configured one BMD for use in early voting and additional machines for Voting Day precinct voting with three different precincts. • Initialized and loaded election definition on the BMDs. • Opened polls in accordance with California Use Procedures. • Printed and verified zero reports for all devices. • Voted ballots for each precinct on a BMD. • Evaluated machine performance for a “fleeing voter.” • Attempted to vote on the BMD more than once. (CVSS 7.5.4.ix). • During Vote Center voting, tested the following: Marked ballots on BMD, per marking pattern. o Marked VBM ballots. o Verified language support for alternative languages on the following o components: BMD display (contrast, font size etc.). ⮚ BMD audio ballot. ⮚ BMD printing. ⮚ California Certification Functional Test Report Page 19 of 26 Report No. CAF-21005-FTR-01 Los Angeles County VSAP 3.0 California Certification Functional Test Report v2.0 Used the ISB to generate BMD Poll Passes, RAVBM ballots, and UOCAVA o ballots. Created BMD ballots utilizing the Poll Passes. o Remade RAVBM and UOCAVA ballots into VSAP 3.0 ballots. o Transferred logs and ballot accounting back to BMG. o • During Tally scanning, tested the following: Fed ballots in all 90-degree orientations on all devices. o Verified language support for alternative languages. o • Closed polls in accordance with California Use Procedures. Printed results from all devices. o Transferred logs and ballot accounting back to BMG. o Shut down devices. o • Consolidated and reported: Canvass reconciliation: o Provisional ballots. ⮚ Using adjudication component, adjudicated all ballots with write-ins. ⮚ Generated all reports available on the system. Saved all reports as artifacts of o testing. • Verified: Canvass – SOV. o SSOV. o Precincts. o Audit reports (Including tabulation devices). o Test Gubernatorial General Election A Gubernatorial General election was run utilizing: • VSAP 3.0 BMD’s • VSAP 3.0 Central scanners The Gubernatorial General was conducted in 14 languages: Armenian, Chinese (Audio - Cantonese and Mandarin), English, Farsi, Hindi, Japanese, Khmer, Korean, Russian, Spanish, Tagalog, Thai, Vietnamese. California Certification Functional Test Report Page 20 of 26 Report No. CAF-21005-FTR-01 Los Angeles County VSAP 3.0 California Certification Functional Test Report v2.0 The election included seven precincts and included 52 contests. For this election the VBM ballots were printed on 8.5”x14” ballot paper and the BMD ballots on 8”x13.25” thermal ballot paper. The following steps were completed with results as noted: • Prepared all precinct components for election. • Configured one BMD for use in early voting and additional machines for Voting Day precinct voting with three different precincts. • Initialized and loaded election definition on the BMDs. • Opened polls in accordance with California Use Procedures. • Printed and verified zero reports for all devices. • Voted ballots for each precinct on a BMD. • Evaluated machine performance for a “fleeing voter.” • Attempted to vote on the BMD more than once. (CVSS 7.5.4.ix). • During Vote Center voting, tested the following: Marked ballots on BMD, per marking pattern. o Marked VBM ballots. o Verified language support for alternative languages on the following o components: BMD display (contrast, font size etc.). ⮚ BMD audio ballot. ⮚ BMD printing. ⮚ Used the ISB to generate BMD Poll Passes, RAVBM ballots, and UOCAVA o ballots. Created BMD ballots utilizing the Poll Passes. o Remade RAVBM and UOCAVA ballots into VSAP 3.0 ballots. o Transferred logs and ballot accounting back to BMG. o • During Tally scanning, tested the following: Fed ballots in all 90-degree orientations on all devices. o Verified language support for alternative languages. o • Closed polls in accordance with California Use Procedures. Printed results from all devices. o Transferred logs and ballot accounting back to BMG. o Shut down devices. o California Certification Functional Test Report Page 21 of 26 Report No. CAF-21005-FTR-01 Los Angeles County VSAP 3.0 California Certification Functional Test Report v2.0 • Consolidated and reported: Canvass reconciliation: o Provisional ballots. ⮚ Using adjudication component, adjudicated all ballots with write-ins. ⮚ Generated all reports available on the system. Saved all reports as artifacts of o testing. • Verified: Canvass – SOV. o SSOV. o Precincts. o Audit reports (Including tabulation devices). o Recall Election A Recall election was run utilizing: • VSAP 3.0 BMD’s • VSAP 3.0 Central scanners The Recall General was conducted in Armenian, Chinese (Audio - Cantonese and Mandarin), English, Farsi, Hindi, Japanese, Khmer, Korean, Russian, Spanish, Tagalog, Thai, Vietnamese, Bengali, Burmese, Gujarati, Indonesian, Mongolian, Telugu. The election included one precinct. It included a recall question and a contest with 95 candidates. For this election the VBM ballots were printed on 11”x17” ballot paper and the BMD ballots on 8”x13.25” thermal ballot paper The following steps were completed with results as noted: Prepared all components for election. • Installed election definitions on devices, printed zero reports, and opened polls. • Marked one ballot with non-standard marks and increasingly marginal marks for each type of marker. Included a variety of pens, pencils, and highlighters in various colors. Ran a blank ballot and a fully marked ballot in a separate batch. • During voting, tested the following: Marked ballots on BMD, per marking pattern. o Language support for alternative languages: o BMD display (contrast, font size etc.). ⮚ BMD audio ballot. ⮚ California Certification Functional Test Report Page 22 of 26 Report No. CAF-21005-FTR-01 Los Angeles County VSAP 3.0 California Certification Functional Test Report v2.0 BMD printing. ⮚ • Tested the ability to select candidates across multiple screens on the BMD. • Marked VBM ballots. • Used the ISB to generate BMD Poll Passes, RAVBM ballots, and UOCAVA ballots. • Created BMD ballots utilizing the Poll Passes. • Remade RAVBM and UOCAVA ballots into VSAP 3.0 ballots. • Scanned ballots through Tally. • Closed each machine and printed out results. • Closed polls in accordance with California Use Procedures. • Printed results from all devices. • Transferred logs and ballot accounting back to BMG. • Shut down devices. • Consolidated and reported: Canvass reconciliation: o Provisional ballots. ⮚ Write-ins. ⮚ • Generated final reports and verified: Canvass – SOV. o SSOV. o Precincts. o Other sample user reports. o Audit reports (Including tabulation devices). o California Certification Functional Test Report Page 23 of 26 Report No. CAF-21005-FTR-01 Los Angeles County VSAP 3.0 California Certification Functional Test Report v2.0 Special Election A Special Election was run utilizing: • VSAP 3.0 BMD’s • VSAP 3.0 Central scanners The Special Election was conducted in English. The election included three precincts. It included three contests and eight choices. For this election the VBM ballots were printed on 8.5”x11” ballot paper and the BMD ballots 8”x11’ thermal ballot paper. The following steps were completed with results as noted: • Prepared all precinct components for election. • Configured BMD and Tally for use in early voting (all precincts). • Initialized and loaded election definition on BMD and BMG. • Opened polls in accordance with California Use Procedures. • Printed and verified zero reports for all devices. • Simulated early voting. Voted ballots on BMD and then suspended voting, re- enabled voting, and voted more ballots. • Marked ballots from marking pattern on a BMD. • Hand-marked VBM ballots. • Used the ISB to generate BMD Poll Passes, RAVBM ballots, and UOCAVA ballots. • Created BMD ballots utilizing the Poll Passes. • Remade RAVBM and UOCAVA ballots into VSAP 3.0 ballots. • Scanned ballots through Tally. • Closed each machine and printed out results. • Closed polls in accordance with California Use Procedures. • Set up adjudication to out stack all options for not counted ballots. Setup all scanners to notify for all error conditions. • Verified ballot images are stored in a random manner. (CVSS 7.7.3). • Printed results from all scanners. • Printed zero reports from BMD. • Shut down devices. • Exercised adjudication process. California Certification Functional Test Report Page 24 of 26 Report No. CAF-21005-FTR-01 Los Angeles County VSAP 3.0 California Certification Functional Test Report v2.0 • Consolidated and reported: Uploaded results to BMG from all units. o Canvass reconciliation. o Provisional ballots. o Write-ins. o • Generated final reports and verified: Canvass – SOV. o SSOV. o Precincts. o Other sample user reports. o Audit reports (Including tabulation devices). o A 1% Manual Tally was conducted on each election to verify results. o Additional Functional Review • Review of this item determined that bar/QR codes used to tally the ballots can be scanned by the voter and comparing their ballot selection to the codes in the bar/QR code, verify that their votes on a ballot are correct. • Los Angeles County has added two Fujitsu Scanners to the VSAP environment Fujitsu Scanner 1 (fi-7180PR) and Fujitsu Scanner 2 (fi-7800). The scanners are solely intended to use for disaster recovery purposes. The scanners capture the ballot images and are imported into the Tally System for tabulation. Each scanner was tested with BMD ballots, vote by mail ballots, and poll pass (BMD) ballots. The results were verified and accurate. Tested ballots on the disaster recovery Fujitsu scanners for the following elections: 972, 4084 and 4085 BMD Ballots o 972 and 3793 Vote by Mail Ballots. o Final Data Capture and Analysis • Evaluated system for any changes which occurred during testing. Generated new Trusted Build and images utilizing Los Angeles County Use Procedures. HASH’d all software and firmware components utilizing Los Angeles County Use Procedures. • Validated software and firmware on all devices using the procedure provided by Los Angeles County. California Certification Functional Test Report Page 25 of 26 Report No. CAF-21005-FTR-01 Los Angeles County VSAP 3.0 California Certification Functional Test Report v2.0 Evaluation of Testing The above tests were conducted using the executables created in the Trusted Build, in association with the appropriate hardware versions as declared during the current certification project for the Los Angeles County Voting Solutions for All People (VSAP) 3.0 voting system, for the State of California. As directed by the California Secretary of State, this report does not include any recommendation as to whether or not the system should be approved. End of Functional Test Report California Certification Functional Test Report Page 26 of 26 Report No. CAF-21005-FTR-01