Hart Verity Voting 3.1 Voting System Functional Testing Report

Hart InterCivic Verity Voting 3.1 Voting System Functional Test Report for California Secretary of State CAF-19042-FTR-01 Vendor Name Hart InterCivic Vendor System Verity Voting 3.1 Prepared by: 4720 Independence St. Wheat Ridge, CO 80033 303-422-1566 www.SLICompliance.com Accredited by the U.S. Election Assistance Commission (EAC) for Selected Voting System Test Methods or Services California Secretary of State Hart InterCivic Verity Voting 3.1 Functional Test Report Revision History Date Release Author Revision Summary October 29th, 2019 1.0 M. Santos Initial Release Disclaimer The information reported herein must not be used by the client to claim product certification, approval, or endorsement by NVLAP, NIST, or any agency of the Federal Government. Trademarks SLI Compliance is a registered trademark of Gaming Laboratories International, LLC. All products and company names are used for identification purposes only and may be trademarks of their respective owners. Copyright 2019 by SLI ComplianceSM, a division of Gaming Laboratories International, LLC. California Certification Functional Test Report Page 2 of 24 Document No. CAF-19042-FTR-01, v1.0 California Secretary of State Hart InterCivic Verity Voting 3.1 Functional Test Report TABLE OF CONTENTS INTRODUCTION ........................................................................................................................................... 4 TESTING RESPONSIBILITIES .......................................................................................................................... 4 SCOPE OF THE HART VERITY VOTING 3.1 VOTING SYSTEM ............................................................... 4 SYSTEM DESCRIPTION .................................................................................................................................. 4 BLOCK DIAGRAM .......................................................................................................................................... 8 FUNCTIONAL TESTING .............................................................................................................................. 8 PHASE ONE – PHYSICAL CONFIGURATION AUDIT PHASE ................................................................................ 9 PHASE TWO – INSTALLATION PHASE ........................................................................................................... 10 PHASE THREE – FUNCTIONAL CONFIGURATION AUDIT PHASE ...................................................................... 12 PHASE FOUR – FUNCTIONAL TEST PHASE ................................................................................................... 12 EVALUATION OF TESTING ...................................................................................................................... 24 California Certification Functional Test Report Page 3 of 24 Document No. CAF-19042-FTR-01, v1.0 California Secretary of State Hart InterCivic Verity Voting 3.1 Functional Test Report INTRODUCTION This Functional Test Report details the testing performed during functional testing of the Hart InterCivic Verity Voting 3.1 (Hart Verity Voting 3.1) voting system against the California Voting System Standards (CVSS). Testing Responsibilities All testing was conducted under the guidance of personnel verified by the California Secretary of State (SOS) to be qualified to perform the testing. Scope of the Hart Verity Voting 3.1 Voting System This section provides a description of the scope of Hart Verity Voting 3.1 voting system components. System Description The Hart Verity Voting 3.1 voting system is a paper-based, digital scan voting system that consists of the following major components: Verity Data application. Verity Build application. Verity Count application. Verity Central application. Verity Election Management. Verity Desktop. Verity User Manager. Verity Scan firmware/hardware. Verity Touch Writer with Access firmware/hardware. Verity Reader firmware/hardware. Verity Print firmware/hardware. Verity Data Data is used by election officials to enter election data for contests, candidates, proposition text, translations, and audio. Data also provides the user with controls for proofing of data, layout, and performs validation prior to locking the data to ensure its readiness for use in Verity Build, the election definition software. California Certification Functional Test Report Page 4 of 24 Document No. CAF-19042-FTR-01, v1.0 California Secretary of State Hart InterCivic Verity Voting 3.1 Functional Test Report Verity Build An election definition and device settings component. Build is a required component of the Verity Voting system, used by officials to complete pre-voting tasks for creating and generating an election definition and ballots. Build provides a ballot layout proofing process. The process establishes relationships between election data, jurisdiction, and polling place data, for the shared election definition. Build will create the portable media, vDrives, to provide a method of transferring the shared election definition to Verity Voting devices and other Verity components. vDrive uses an “air-gap,” or non-networked transfer method, to provide more secure exchange of election data. Verity Central A Central Ballot Scanning and Adjudication component used by officials for paper ballot scanning, contest resolution, and conversion of voter selection marks to electronic Cast Vote Records (CVRs) capabilities. Once the CVRs are written to vDrive(s) they can be transferred into Verity Count for vote tabulation and reporting of election results. Verity Central records cast vote records only; it does not tabulate. Verity Count Used by officials to complete post-voting functionality to tabulate election results and generate reports. Verity Count receives the CVRs from portable media devices (vDrives) used to record CVRs from Hart voting devices or Verity Central workstations. Verity Count can be used by officials to resolve Verity Scan or Verity Central write-in votes for paper ballots that were manually marked. Verity Count can also be used to collect and store all election logs from every Verity component/device used in the election, allowing for complete election audit log reviews. Verity Election Management The Election Management application is available only on Verity server workstations. This software enables authorized users to add, import, export, archive, restore, and manage elections. Once an election is added or imported in the Election Management application, the election can be opened and handled per the features available within the Verity software installed on that workstation. User Management This software enables authorized users to create and manage user accounts within the Verity system. California Certification Functional Test Report Page 5 of 24 Document No. CAF-19042-FTR-01, v1.0 California Secretary of State Hart InterCivic Verity Voting 3.1 Functional Test Report Verity Desktop Allows authorized users to manage a very limited set of operating system functions. Verity Desktop is workstation management software used for: Setting the system date and time. Exporting Verity application file hashes to removable USB media. Accessing the operating system for a limited time (less than 24-hours per access code). User access to the operating system’s functionality is restricted to software updates and database management. Importing printer configuration files. Verity Print Verity Print is a pre-voting ballot production device for use by election officials and/or poll workers. Verity Print produces unmarked paper ballots. Print is paired with a commercial off-the-shelf (COTS) printer to allow the user to select and print the desired ballot style based on the precinct and voter registration information. The Verity Print device is activated so the election official can print one or more blank ballots from one selected precinct at a time. Ballots can be printed on- demand for immediate use, or they can be printed in advance for additional inventory. Verity Touch Writer Verity Touch Writer is a touch-screen Ballot Marking Device (BMD) that prints voter-marked ballots to a COTS printer. Voters use the electronic touch display interface to privately and independently make their selections on the ballot. Voters can also make selections with Verity Access, an Audio-Tactile interface (ATI) component with three tactile buttons, one audio port (for headphones), and one port for external two-switch devices. When voters finish making their selections, they print the marked ballot. Verity Reader Verity Reader is an optional paper ballot review device suitable for use by all voters, including non-disabled voters and voters with disabilities. Voters can insert their marked paper ballot to visually verify how their ballot will be counted when the ballot is cast in the Verity system, and/or hear audio read-back of their ballot choices. (For voters with disabilities, Verity Reader offers the same accessibility features as the Verity Touch Writer ballot marking device.) California Certification Functional Test Report Page 6 of 24 Document No. CAF-19042-FTR-01, v1.0 California Secretary of State Hart InterCivic Verity Voting 3.1 Functional Test Report Verity Scan Verity Scan is Verity’s digital scanning solution for paper ballots. Verity Scan is paired with a purpose-built ballot box to ensure accurate, secure, and private ballot scanning and vote casting. When opening the polls, authorized users activate the Verity Scan device to prepare it to receive marked paper ballots. Verity Scan indicates when it is appropriate to insert ballots, and when ballots have been successfully cast. Verity Scan records Cast Vote Records (CVRs) and audit log data in redundant, secure storage locations, including the Verity vDrive. vDrive storage is portable flash memory and allows the CVRs to be transferred to the Verity Count tabulation and reporting system. Verity Access Verity Access is an interface module that is connected to Verity Touch Writer and Verity Reader. The module has three tactile buttons, one audio port, and one port for external tactile buttons or sip-n-puff devices. Jacks for headphones and adaptive devices are located on the top edge of the device, and the device has grip surfaces on either side. Verity AutoBallot Verity AutoBallot is an optional barcode scanner kit for Verity Print and Verity Touch Writer that allows air-gapped integration between an e-pollbook check-in process and the task of selecting the proper ballot style for the voting system. Particularly when Verity Print or Touch Writer is configured with dozens or hundreds of ballot styles in Vote Centers, Verity AutoBallot simplifies and automates the ballot style selection process by allowing poll workers to scan a barcode output from an electronic poll book, and activate the correct ballot style with the click of a button, thereby reducing human error. Once the ballot style has been input with the barcode scanner, the poll worker confirms the ballot style on the Verity device display and prints an unmarked ballot (Verity Print) or activates an accessible electronic voting session (Verity Touch Writer). Verity vDrive Verity vDrive is a required Verity Voting component, used as a portable media device generated by Verity Build. Verity vDrive allows election definitions to be moved from Verity Build to Verity Scan, Verity Touch Writer, Verity Reader, and Verity Print. Verity vDrive supports the transfer of Cast Vote Records (CVRs) in Verity Scan and Verity Central. California Certification Functional Test Report Page 7 of 24 Document No. CAF-19042-FTR-01, v1.0 California Secretary of State Hart InterCivic Verity Voting 3.1 Functional Test Report Verity Key Verity Key is electronic media that is created by Verity Build for a specific election. Key is a required Verity component. Key is the electronic media that provides user authentication and configures election security throughout the Verity Voting system. Block Diagram The system overview of the submitted voting system is depicted in Figure 1. Figure 1. Hart Verity Voting 3.1 Voting System Overview Functional Testing Prior to all testing, the Trusted Build of the software and firmware was created. Functional testing was divided into four phases. In phase one, the Physical Configuration Audit compared components submitted to the actual documentation. In phase two, the Installation Phase included the steps necessary to install the system. California Certification Functional Test Report Page 8 of 24 Document No. CAF-19042-FTR-01, v1.0 California Secretary of State Hart InterCivic Verity Voting 3.1 Functional Test Report In phase three, the Functional Configuration Audit verified the system’s hardware and software perform all the functions listed in the documentation. In phase four, Functional Testing exercised the system using operations necessary to conduct elections following the California Use Procedures for the system, documented the test results, and prepared benchmark data that can be used for system validation by the California Secretary of State (SOS). During installation and functional testing, it was necessary to make minor edits to the California Use Procedures to provide clarity for end users. During examination and review performance, the system was configured as it would be for normal field use. This includes connecting all supporting equipment and peripherals. Phase One – Physical Configuration Audit Phase The Physical Configuration Audit (PCA) compared the voting system components submitted for certification to the manufacturer's technical documentation. This was an audit of all hardware and software in the system to compare the Technical Data Package (TDP) to the actual system. For the PCA, Hart provided: Identification of all items that are to be a part of the software release. Specification of compiler (or choice of compilers) to be used to generate executable programs. Identification of all hardware that interfaces with the software. Configuration baseline data for all hardware that is unique to the system. Copies of all software documentation intended for distribution to users, including program listings, specifications, operations manual, voter manual, and maintenance manual. User acceptance test procedures and acceptance criteria. Identification of any changes between the physical configuration of the system submitted for the PCA and that submitted for the Functional Configuration Audit (FCA), with a certification that any differences do not degrade the functional characteristics. Complete descriptions of its procedures and related conventions used to support this audit by: o Establishing a configuration baseline of the software and hardware to be tested. o Confirming whether the system documentation matches the corresponding system components. California Certification Functional Test Report Page 9 of 24 Document No. CAF-19042-FTR-01, v1.0 California Secretary of State Hart InterCivic Verity Voting 3.1 Functional Test Report Phase Two – Installation Phase System Installation, Configuration, and Validation During Installation testing, the following was verified: All boxes, system components, etc. have been labeled correctly and accurately. The voting system has been labeled correctly. (CVSS 8.2.a). That a Configuration Log has been established. (CVSS 8.2.b). All hardware that was used in the testing, including servers, workstations, monitors, printers, voting devices and peripherals, was documented. Build Software, Servers, and Workstations The following activities occurred: All computers were wiped with DBAN. All software and firmware components to be compiled in the trusted build were included and validated with HASHes from the manufacturer for COTS software/firmware components. All hardware, compilers, and components needed to compile the trusted build were included and available. The hardware provided was verified to meet or exceed the minimum requirements in the installation procedure manuals. The required COTS software was verified to be available. The component list of software and firmware exactly matched what was prescribed to be installed. All software and firmware components were built per the California Use Procedures. Hart Installation Procedures for the server and clients were successfully followed. The sequence of steps in the installation procedures manual were followed. After the installation, COTS applications, proprietary applications, hardening, and configuration, images of each component of the system were taken. Post installation, HASHes were taken of every piece of software and firmware. (Note: These HASHes will be provided to California Counties along with the Trusted Build so that they may validate the system at any time, such as a post-election reinstall to meet California airgap requirements). California Certification Functional Test Report Page 10 of 24 Document No. CAF-19042-FTR-01, v1.0 California Secretary of State Hart InterCivic Verity Voting 3.1 Functional Test Report System security policies, data sources, and registry were verified to be properly documented. System configuration and setup were audited against specifications in the manuals to verify that scripts used in installation and configuration achieved the specifications. The Hart System Verification Procedures were verified to be applicable for each machine. All components were HASHed and verified to be correct. The voting system was verified to deploy COTS protection against viruses, worms, trojan horses, and logic bombs. (CVSS 7.4.2). Install Firmware on Hardware Devices Hardware devices were examined and determined to have the correct version of firmware installed. Instructions for firmware upgrades in use procedures and other system documentation were verified to be correct. After firmware was installed, hardware devices were verified to be operational. The Hart System Verification Procedures for each hardware device were verified to be correct to install the firmware on that device. Verified that no compilers, assemblers, or source code were resident on the system. Election specific firmware was verified to not be installed on the same component that the operating system is installed on. (CVSS 7.4.1.b.iv). Verified all software setup validation requirements of CVSS 7.4.6. Post Installation Screenshots of all computer-based machines listed on the “Installed Programs” screen were taken. Screenshots of all computer-based machines listed on the “drivers” page from Windows were taken. Photos of all proprietary machines for the version numbers were taken. Images of all equipment were taken. All system software and firmware were taken. A master copy of the “Trusted Build” directory structure, files, and “County Release” for distribution to counties was created. California Certification Functional Test Report Page 11 of 24 Document No. CAF-19042-FTR-01, v1.0 California Secretary of State Hart InterCivic Verity Voting 3.1 Functional Test Report “Trusted Build” software and Golden (County Release) images were created. Phase Three – Functional Configuration Audit Phase The Functional Configuration Audit was conducted by SLI to verify that the system performs all the functions described in the system documentation. In accordance with CVSS 9.11.2, the Verity Voting 3.1 system does: a. Completely describe its procedures and related conventions used to support this audit for all system components. b. Provide the following information to support this audit: i. Copies of all procedures used for module or unit testing, integration testing, and system testing. ii. Copies of all test cases generated for each module and integration test, and sample ballot formats or other test cases used for system tests. iii. Records of all tests performed by the procedures listed above, including error corrections and retests. The criteria of this phase were adequately met. Phase Four – Functional Test Phase During the Functional Test Phase the system was examined to determine that every functional piece of the system is accurate and complete. During the Functional Test Phase, an issue log was maintained of any errors and omissions found in the documentation or anomalies encountered that were not identified during the PCA. Throughout the Functional Test Phase, Verity Print (BOD) was used to replace ballots that were fatigued or needed replacement, as well as new ballots used for casting votes. The system was maintained in an air-gapped fashion: The architecture shall allow transfer of the election definition and tally database from the permanent server(s) to the sacrificial server (CVSS 7.4.1.a.i). The voting system architecture does allow each installation to use its own Ethernet network, infrastructure devices like switches, and central-scan vote-recording units, including optical scan units, permitting the two installations to be segregated and air-gapped to ensure that there are no cross connections. An air gap is established by keeping two installations/networks physically separate and seeing that no devices attached to the sacrificial installation/network is connected (directly or California Certification Functional Test Report Page 12 of 24 Document No. CAF-19042-FTR-01, v1.0 California Secretary of State Hart InterCivic Verity Voting 3.1 Functional Test Report indirectly) to the first network, ensuring that data cannot flow from one installation/network to the other. Functional Testing Preparation Functional aspects of this phase included: Kickoff meeting. Reviewed test plan; discussed how the equipment was to be allocated to each test in order to use time and personnel as efficiently as possible. Inspected the test decks prepared by Hart and audited them against the Ballot Marking and Expected Results Excel files. Using Verity Data/Build, built an election from scratch and verified all options in the Election Management System (EMS). This election contained a multiple candidate contest for a “choose two of three” contest. o Printed ballots on Verity Print and Verity Build devices. o Scanned ballots. o Tabulated and generated reports. Completed extensive review of Verity Data/Build and Verity Count to verify end user documentation. Evaluated methods to import elections into Verity. Evaluated Access Controls. Verified all CVSS requirements. Proofed the test deck for the Presidential Primary, Presidential General, Special Recall, Ranked Choice Voting, and Vote Center elections. Scanned and tabulated the test deck for the Presidential Primary election on Verity Count. Compared the tabulation results of this scan to expected results. Conducted logic and accuracy testing in accordance with California Use Procedures. o Hart logic and accuracy (L&A) test deck generation software was used to generate the test deck for L&A testing, as if being used by a county. Scanned predetermined test deck through scanners. o Reviewed Verity Scan, Verity Touch Writer, Verity Central, and Verity Count for L&A procedures per Hart documentation. o Printed and verified L&A results from scanners. o Uploaded and consolidated results to Verity Count. o Printed and verified L&A results. o Verified that all components were ready to go after L&A, or they needed to be re-provisioned/re-imaged prior to the actual election. All listed items completed successfully. California Certification Functional Test Report Page 13 of 24 Document No. CAF-19042-FTR-01, v1.0 California Secretary of State Hart InterCivic Verity Voting 3.1 Functional Test Report Functional Testing Summary The tests run on the Hart Verity Voting 3.1 voting system included: Presidential Primary Election Presidential General Election Special Recall Election Ranked Choice Vote (RCV) Election Vote Center Election Pick Two of Three Election Test Presidential Primary Election A primary election was run utilizing: One Verity Touch Writer polling place device. Two Verity Scan polling place devices (one configured for Election Day, one for early voting). One Verity Print polling place device. One Verity Reader polling place device. Three Verity Central environments. One EMS (Verity Data/Build, Verity Count). The Presidential Primary was conducted in English and Spanish. The election included ten precincts and eight party splits per precinct. It included 20 contests and 65 choices. Paper ballots included at least one bi-lingual ballot style consisting of English and Spanish. This election was printed on the 8.5”x14” ballot size and included multiple cards. The following steps were completed with results as noted: Prepared system for test election. Finalized EMS and set up for reporting. Prepared all precinct components (Verity Touch Writer, Verity Scan, Verity Print, and Verity Reader) for election. Evaluated system for air-gap requirements. Opened Polls in accordance with California Use Procedures. Printed and verified zero reports for all devices. Printed 25 ballots on Verity Print for every language available in this election. Added these to the expected results. California Certification Functional Test Report Page 14 of 24 Document No. CAF-19042-FTR-01, v1.0 California Secretary of State Hart InterCivic Verity Voting 3.1 Functional Test Report Voted two ballots from each party on a Verity Touch Writer. Scanned two ballots from each party on a Verity Scan. Verified that the voter can review, confirm, and change their selections on the Verity Touch Writer and Verity Scan. Verified that the voter can review their selections on the Verity Reader. Printed final reports. During voting, tested the following: Fed ballots in all directions/sides on all devices. Closed polls in accordance with California Use Procedures. Printed results from all scanners. Removed results media to transfer results back to Verity Count. Shut down devices. Consolidated and reported: o Uploaded results to Verity Count from all units. o Canvass reconciliation: Processed provisional ballots. Using adjudication component, adjudicated 40 hand marked ballots with write-ins. o Generated final reports and verified: Generated all reports available on the system. Saved all reports to a flash drive as artifacts of testing. Canvass – SOV. Verified accuracy of report. SSOV. Precinct results. Cast Vote Record Report. Audit reports (Including tabulation devices). Created California Election Night auto-reporting files per the Calvoter template. Backed up system to provide “Vote Count Program” to SOS. Evaluated default file name for submission to SOS. Checked backup size. Verified system logging for all events. Saved system logs to archive. Printed ten ballots on Verity Printer for each language. Marked ballots generated on Verity Touch Writer and verified in Verity Reader, as well as scanned through Verity Scan. California Certification Functional Test Report Page 15 of 24 Document No. CAF-19042-FTR-01, v1.0 California Secretary of State Hart InterCivic Verity Voting 3.1 Functional Test Report Verified results. All listed items of election completed successfully. No issues were encountered. Test Presidential General Election A Presidential General election was run utilizing: One Verity Touch Writer polling place device. Two Verity Scan polling place devices (one configured for Election Day, one for early voting). One Verity Print polling place device. One Verity Reader polling place device. Three Verity Central environments. One EMS (Verity Data/Build, Verity Count). Presidential General was conducted in English, Korean, Chinese, and Vietnamese. The election included 10 precincts. It included 18 contests and 45 options with two write-ins. Paper ballots included one bi-lingual ballot style. Hart Verity Voting 3.1 will do bi-lingual ballots, but not tri-lingual ballots. Startup, and maintenance procedures for the polling place machines were evaluated to make sure the procedures were correct. Electronic devices and interfaces were programmed to support the entire election in all four languages. Ballot layouts met the requirements (EC 13111, EC 13109 and EC 13105). This election was printed on the 8.5”x11” ballot size and included multiple cards. The following steps were completed with results as noted: Prepared system for test election. Finalized EMS and set up for reporting. Prepared all precinct components (Verity Print, Verity Touch Writer, Verity Reader, and Verity Scan) for election. Evaluated system for air-gap requirements. Opened Polls in accordance with California Use Procedures. Printed and verified zero reports for all devices. Printed 10 ballots on Verity Print for every language available in this election. Added these to the expected results. Voted two ballots from each party on a Verity Touch Writer. California Certification Functional Test Report Page 16 of 24 Document No. CAF-19042-FTR-01, v1.0 California Secretary of State Hart InterCivic Verity Voting 3.1 Functional Test Report Scanned two ballots from each party on a Verity Scan. Verified that the voter can review, confirm, and change their selections on the Verity Touch Writer and Verity Scan. Verified that the voter can review their selections on the Verity Reader. Printed final reports. During voting, tested the following: Fed ballots in all directions/sides on all devices. Closed polls in accordance with California Use Procedures. Printed results from all scanners. Removed results media to transfer results back to Verity Count. Shut down devices. Consolidated and reported: o Uploaded results to Verity Count from all units. o Canvass reconciliation: Processed provisional ballots. Using adjudication component, adjudicated 40 hand marked ballots with write-ins. o Generated final reports and verified: Generated all reports available on the system. Saved all reports to a flash drive as artifacts of testing. Canvass – SOV. Verified accuracy of report. SSOV. Precinct results. Cast Vote Record Report. Audit reports (Including tabulation devices). Created California Election Night auto-reporting files per the Calvoter template. Backed up system to provide “Vote Count Program” to SOS. Evaluated default file name for submission to SOS. Checked backup size. Verified system logging for all events. Saved system logs to archive. Printed ten ballots on Verity Printer for each language. Marked ballots generated on Verity TouchWriter and verified in Verity Reader, as well as scanned through Verity Scan. Verified results. California Certification Functional Test Report Page 17 of 24 Document No. CAF-19042-FTR-01, v1.0 California Secretary of State Hart InterCivic Verity Voting 3.1 Functional Test Report All listed items of election completed successfully. The Scan machine was unplugged and ran on battery power for two and a half hours while scanning 57 ballots without problem. No issues were encountered. Test Special Recall Election A Special Recall election was run utilizing: One Verity Touch Writer polling place device. Two Verity Scan polling place devices (one configured for Election Day, one for early voting). One Verity Print polling place device. One Verity Reader polling place device. Three Verity Central environments. One EMS (Verity Data/Build, Verity Count). The Special Recall was conducted in English, Khmer, Japanese, and Hindi. The election consisted of one precinct and one contest. The contest included 135 choices with one write-in in a gubernatorial contest. The election was printed on 8.5”x20” ballots. The following steps were completed with results as noted: Prepared all components for election. Installed election definitions on devices, printed zero reports and opened polls. During voting, tested the following: Fed ballots in all directions/sides on all devices. Printed and marked ten ballots on Verity Print for each supported language in the election. Voted two ballots on Verity Touch Writer for each supported language. Added these ballots to the expected results. Verified logging on Verity Print meets batch printing audit requirements. Language support for alternative languages: Khmer, Japanese, and Hindi o Verity Touch Writer display. o Verity Touch Writer audio. o Verity Reader display. o Verity Reader audio ballot. o Verity Scan. California Certification Functional Test Report Page 18 of 24 Document No. CAF-19042-FTR-01, v1.0 California Secretary of State Hart InterCivic Verity Voting 3.1 Functional Test Report Scanned the marked ballots on Verity Scan and Verity Central. Closed each machine and printed out results. Closed polls, in accordance with California Use Procedures. Printed results from all scanners. Transferred results back to Verity Count. Shut down devices. Consolidated and reported: o Uploaded results to Verity Count from all units. o Canvass reconciliation: Provisional ballots. Write-ins. o Generated final reports & verified: Canvass – SOV. SSOV. Precincts. Audit reports (Including tabulation devices). Printed ten ballots on Verity Printer for each language. Marked ballots generated on Verity TouchWriter and verified in Verity Reader, as well as scanned through Verity Scan. Verified results. All listed items of election completed successfully. No issues were encountered. Test Ranked Choice Election A Ranked Choice election was run utilizing: One Verity Touch Writer polling place device. Two Verity Scan polling place devices (one configured for Election Day, one for early voting). One Verity Print polling place device. One Verity Reader polling place device. Three Verity Central environments. One EMS (Verity Data/Build, Verity Count). California Certification Functional Test Report Page 19 of 24 Document No. CAF-19042-FTR-01, v1.0 California Secretary of State Hart InterCivic Verity Voting 3.1 Functional Test Report Ranked Choice Voting in English, Ilocano, Tagalog, and Thai. A fictitious single seat RCV Election with one ballot style with two contests containing twelve candidates and one write-in. The following steps were completed with results as noted: Prepared all components for election. Initialized and loaded election definition on all devices. Evaluated system for air-gap requirements. Opened Polls in accordance with California Use Procedures. Printed and verified zero reports for all devices. Marked two ballots on Verity Touch Writer for each language, removing ballots from the pre-marked test deck and replacing with the Verity Touch Writer ballots. One Scan machine was configured for Election Day voting, and one Scan machine was configured for early voting as in a vote center. Printed zero reports from Verity Central. Cast ballots on each scanner. During voting, tested the following: Fed ballots in all directions/sides on all devices. Language supported included alternative languages: Ilocano, Tagalog, and Thai o Verity Touch Writer display. o Verity Touch Writer audio. o Verity Reader display. o Verity Reader audio ballot. o Verity Scan. Closed polls, in accordance with California Use Procedures. Printed results from all scanners. Removed results media as necessary to transfer results back to EMS. Shut down devices. Ranked results and determine winner. Verified and documented all one seat RCV on the system. Consolidated and reported. Uploaded results to Verity Count from all units. Generated final reports and verified totals. California Certification Functional Test Report Page 20 of 24 Document No. CAF-19042-FTR-01, v1.0 California Secretary of State Hart InterCivic Verity Voting 3.1 Functional Test Report Generated Cast Vote Record Spreadsheet. Audited Cast Vote Record Spreadsheet against ballots. Printed ten ballots on Verity Printer for each language. Marked ballots generated on Verity TouchWriter and verified in Verity Reader, as well as scanned through Verity Scan. Verified results Scanned 12 bar codes on Print, and 12 on Verity Touch Writer. All listed items of election completed successfully. No issues were encountered. Test Vote Center Election A Vote Center election was run utilizing: One Verity Touch Writer polling place device. Two Verity Scan polling place devices (one configured for Election Day, one for early voting). One Verity Print polling place device. One Verity Reader polling place device. Three Verity Central environments. One EMS (Verity Data/Build, Verity Count). The Vote Center Election was conducted in English. A fictitious Vote Center Election that included five contests, 10 choices, and 3000 precincts. Twenty-five percent of Vote Center election ballots were machine pre-folded in a tri-fold format. As a system test of label capacity: One candidate was created as candidate with Last Name starting with the character 'L' and extending with 29 digits as indicated below. The First Name starting with the character 'F' and extending with 19 digits as indicated below. For printed ballots, the following candidate name was used: "F12345678901234567890 L123456789012345678901234567890." The following steps were completed with results as noted: Prepare all precinct components for election. Configure Verity Scan, Verity Reader, Verity Touch Writer, Verity Print, and Verity Central and Verity Count for use in early voting (all precincts) California Certification Functional Test Report Page 21 of 24 Document No. CAF-19042-FTR-01, v1.0 California Secretary of State Hart InterCivic Verity Voting 3.1 Functional Test Report Initialize and load election definition on Verity Scan, Verity Reader, Verity Touch Writer, Verity Print, and Verity Central and Verity Count. Open polls in accordance with California Use Procedures. Printed one offs of every 30th ballot to make sure Verity Print can deal with 3000 precincts. Scanned test decks through each tabulating device. Print and verify zero reports for all devices. Simulate early voting. Vote ballots on vote center voter-facing components and then suspend voting, enable voting, and vote more ballots. Scan pattern of precincts ballots on a Verity Scan. Mark ballots from pattern of precincts on a Verity Touch Writer. Verify ballots from pattern of precinct on a Verity Reader. Unfold pre-folded ballots and process as mail in ballots. Print zero reports from EMS. Scan all preprinted ballots in equal numbers on all scanners. The Scan machine setup for early voting was repeatedly suspended, powered off and back on again, and re-enabled without problem. Setup adjudication to out stack all options. Setup all scanners to notify for all error conditions. Create eight marginal marks ballots including all highlighter and pen/pencil colors. Number the ballots and scan in order on all scanners to verify all scanners scanned consistently and adjudication was consistent across all scanners. o Adjudicate eight hand marked ballots with marginal marks. Evaluate all options in adjudication. Exercise adjudication workstation on marginal marks ballots. Close polls in accordance with California Use Procedures. Print Cast Vote Record file and export for RLA. (CVSS7.7.3.b). Verify ballot images are stored in a random manner. (CVSS 7.7.3). Print results from all scanners. Transfer results back to Verity Count. Shut down devices. Consolidate and report: o Upload results to Verity Count from all units. o Canvass reconciliation. o Provisional ballots. California Certification Functional Test Report Page 22 of 24 Document No. CAF-19042-FTR-01, v1.0 California Secretary of State Hart InterCivic Verity Voting 3.1 Functional Test Report o Write-ins. Generate final reports and verify: o Generate all reports the system is capable of. o Canvass – SOV. o SSOV. o Precincts. o Other sample user reports. o Audit reports (Including tabulation devices). Printed ten ballots on Verity Printer for each language. Marked ballots generated on Verity TouchWriter and verified in Verity Reader, as well as scanned through Verity Scan. Verified results. Verified printer/Verity Touch Writer connection issues from 3.0.1 have been corrected. Tested AutoBallot, scanned 12 bar codes on Print, and 12 on Verity Touch Writer. All listed items of election completed successfully. No issues were encountered. Test Pick Two of Three Election A Pick Two of Three election was run utilizing: One Verity Central environment. One EMS (Verity Data/Build, Verity Count). The Pick Two of Three Election in English was built from scratch utilizing the client server configuration of Verity Data/Build. This election was printed from Verity Build to the C831 printer on 8.5”x14” inch ballots. The ballots were pre-printed with tint and watermark, and 100 ballots were printed on this stock. The ballots were hand marked with either a blue ballpoint pen or a black Sharpie. An expected results spreadsheet was built, polls were opened, and the ballots scanned on Central. Polls were closed. Results were aggregated to Count, and all reports and logs were saved. California Certification Functional Test Report Page 23 of 24 Document No. CAF-19042-FTR-01, v1.0 California Secretary of State Hart InterCivic Verity Voting 3.1 Functional Test Report The counts matched the expected results, and all ballots scanned as expected. All listed items of election completed successfully. No issues were encountered. Final Data Capture and Analysis Took hashes and images from all computers. Validated the system on each server, desktop, and polling place device using the procedure provided by Hart. Generated “Trusted Build” software and Golden (County Release) images to be distributed by Secretary of State to Vendor/County. Verified with vendor trusted build software was complete and ready for distribution. Hart attested to the correctness of the files. Verified that all drivers including monitor video drivers are included in the County Release. Evaluation of Testing The above tests were conducted using the executables created in the Trusted Build, in association with the appropriate hardware versions as declared during the current certification project for the Hart InterCivic Verity Voting 3.1 voting system, for the State of California. No functional issues were encountered during testing. As directed by the California Secretary of State, this report does not include any recommendation as to whether or not the system should be approved. End of Functional Test Report California Certification Functional Test Report Page 24 of 24 Document No. CAF-19042-FTR-01, v1.0