Consultant's Functional Test Report

Dominion Democracy Suite 5.10 Functional Test Report CDV-19011-FTR-01 Vendor Name Dominion Voting Systems Vendor System Democracy Suite 5.10 Prepared by: SLI ComplianceSM 4720 Independence St. Wheat Ridge, CO 80033 303-422-1566 www.SLICompliance.com Accredited by the Election Assistance Commission (EAC) for Selected Voting System Test Methods or Services. Dominion Democracy Suite 5.10 Voting System California Certification Functional Test Report Copyright 2019 SLI ComplianceSM, a Division of Gaming Laboratories International, LLC Revision History DATE RELEASE AUTHOR REVISIONS August 15, 2019 v1.0 M. Santos Initial Release August 20, 2019 v2.0 M. Santos Updates for CASOS comments August 21, 2019 v3.0 J. Panek Updates for CASOS comments August 25, 2019 v4.0 M. Santos Update to FCA section August 26, 2019 v5.0 M. Santos Updates for CASOS comments Disclaimer The Certification Test results reported herein must not be used by the client to claim product certification, approval, or endorsement by NVLAP, NIST, or any agency of the Federal Government. Results herein relate only to the items tested. Trademarks • SLI is a registered trademark of SLI Compliance, a Division of Gaming Laboratories International, LLC. • All other products and company names are used for identification purposes only and may be trademarks of their respective owners. The tests referenced in this document were performed in a controlled environment using specific systems and data sets and results are related to the specific items tested. Actual results in other environments may vary. Opinions and Interpretations There are no SLI opinions or interpretations included in this report. California Certification Page 2 of 20 Functional Test Report v5.0 Report Number CDV-19011-FTR-01 Dominion Democracy Suite 5.10 Voting System California Certification Functional Test Report TABLE OF CONTENTS INTRODUCTION ................................................................................................................................... 4 REFERENCES....................................................................................................................................... 4 SYSTEM OVERVIEW ........................................................................................................................... 4 SCOPE OF THE DOMINION DEMOCRACY SUITE 5.10 VOTING SYSTEM ..................................................... 4 CERTIFICATION TEST RESULTS SUMMARY ................................................................................... 5 PHYSICAL CONFIGURATION AUDIT PHASE ............................................................................................. 6 INSTALLATION PHASE ........................................................................................................................... 6 System Installation, Configuration and Validation ......................................................................... 6 Build Software, Servers, and Workstations ................................................................................... 7 Install Firmware on Hardware Devices .......................................................................................... 8 Post Installation Phase .................................................................................................................. 8 FUNCTIONAL CONFIGURATION AUDIT PHASE ......................................................................................... 8 FUNCTIONAL TEST PHASE .................................................................................................................... 9 Functional Testing Summary ....................................................................................................... 10 Test Presidential Primary Election ............................................................................................... 10 Test Presidential General Election .............................................................................................. 13 Test Special Recall Election ........................................................................................................ 15 Test Ranked Choice Election ...................................................................................................... 17 Test Vote Center Election ............................................................................................................ 18 Final Data Capture and Analysis ................................................................................................. 20 EVALUATION OF TESTING ................................................................................................................... 20 California Certification Page 3 of 20 Functional Test Report v5.0 Report Number CDV-19011-FTR-01 Dominion Democracy Suite 5.10 Voting System California Certification Functional Test Report Introduction SLI Compliance is submitting this test report as a summary of the certification testing efforts for the Dominion Democracy Suite 5.10 voting system. This test report provides results for the functional testing of the Dominion Democracy Suite 5.10 Voting System (Dominion DS 5.10). References • California Voting System Standards (CVSS) System Overview Scope of the Dominion Democracy Suite 5.10 Voting System This section provides a description of the scope of Dominion DS 5.10 voting system components: • Election Management System (EMS) – Election Event Designer (EED) application • EMS – Results Tally Reporting (RTR) application • EMS – Adjudication Services (Adjudication) application • ImageCast Central (ICC) application • ImageCast Evolution (ICE) firmware/hardware • ImageCast X (ICX) firmware/hardware • ImageCast Precinct 2 (ICP2) firmware/hardware The Dominion Democracy Suite 5.10 Election Management System (EMS) represents a set of N-Tier software applications (EMS, RTR, Adjudication) for pre- voting and post-voting election project activities that are applicable to jurisdictions of various sizes and geo-political complexities. The Dominion Democracy Suite ICC system consists of a central, high-speed, optical scan ballot counter (tabulator) called the ICC Ballot Counter and is used for processing absentee ballots (such as vote by mail). This ballot counter unit is based on commercial off the shelf (COTS) hardware coupled with custom-made ballot processing application software. It is used for high-speed, accurate, and reliable centralized scanning and counting of paper ballots. The Dominion Democracy Suite ICE system employs a precinct-level optical scan ballot counter (tabulator) in conjunction with an external ballot box. This tabulator is designed to mark and/or scan paper ballots, interpret voting marks, communicate California Certification Page 4 of 20 Functional Test Report v5.0 Report Number CDV-19011-FTR-01 Dominion Democracy Suite 5.10 Voting System California Certification Functional Test Report these interpretations back to the voter (either visually through the integrated LCD display or audibly via integrated headphones) and, upon the voter’s acceptance, deposit the ballots into the secure ballot box. The Democracy Suite ICX ballot marking platform is used to create paper Electronic Mobile Ballots. These ballots are later scanned and tabulated by the ICC optical ballot counter and/or scanned, verified, and cast by the ICE. The ImageCast Precinct 2 ballot counter device is a precinct optical scan ballot counter designed to provide three major functionalities, including: ballot scanning and tabulation, accessible voting, and ballot review and second chance voting Certification Test Results Summary Prior to all testing, the trusted build of the software and firmware was created. This process created a contractor compiled “Trusted Build” from components provided by the vendor and witnessed by the CA Secretary of State. Functional testing was divided into five phases. 1. The Physical Configuration Audit (PCA) compared components submitted to the actual documentation. 2. The Installation Phase included the steps necessary to install the system, develop any additional test elections from scratch, and provided Dominion with the data needed to print test ballots for any additional elections. 3. The Functional Configuration Audit (FCA) verified that the systems hardware and software perform all the functions listed in the documentation. Note that numerous documentation issues were found which resulted in a fifth phase for documentation verification. 4. The Functional Test Phase exercised the system through the remaining operations necessary to conduct elections following the California Use Procedures for the system, documented the test results, and prepared benchmark data that can be used for system validation by the California Secretary of State (SOS). 5. A fifth phase was not required to perform any regression testing, though numerous documentation issues were found which resulted in a fifth phase for documentation verification. During installation and functional testing, it was necessary to edit the California Use Procedures to provide clarity for end users. A list was maintained of all changes and updated California Use Procedures were generated. California Certification Page 5 of 20 Functional Test Report v5.0 Report Number CDV-19011-FTR-01 Dominion Democracy Suite 5.10 Voting System California Certification Functional Test Report Physical Configuration Audit Phase The PCA compared the voting system components submitted for certification to the manufacturer's technical documentation. This is an audit of all hardware and software in the system to compare the Technical Data Package (TDP) to the actual system. For the PCA, Dominion: • Identified all items that were to be a part of the software release. • Specified the compiler (or choice of compilers) to be used to generate executable programs. • Identified all hardware that interfaces with the software. • Configured baseline data for all hardware that is unique to the system. • Made copies of all software documentation intended for distribution to users, including program listings, specifications, operations manual, voter manual, and maintenance manual. • Complied with user acceptance test procedures and acceptance criteria. • Identified any changes between the physical configuration of the system submitted for the PCA and that submitted for the FCA, with a certification that any differences did not degrade the functional characteristics. • Completed descriptions of its procedures and related conventions used to support this audit by: o Establishing a configuration baseline of the software and hardware to be tested. o Confirming whether the system documentation matches the corresponding system components. Note that the PCA was conducted in conjunction with testing activities; as such, all anomalies and omissions in the documentation identified by SLI during the PCA were not corrected prior to starting any testing. A re-review of the documentation was conducted in order to verify the documentation was accurate and free of anomalies or omissions. Installation Phase System Installation, Configuration and Validation During Installation Phase testing, • An issue log was created of any errors and omissions found in the documentation or anomalies encountered that were not identified in the PCA. • Verified that all boxes, system components, etc. had been labeled correctly and accurately. California Certification Page 6 of 20 Functional Test Report v5.0 Report Number CDV-19011-FTR-01 Dominion Democracy Suite 5.10 Voting System California Certification Functional Test Report • Verified that the voting system had been labeled correctly. • Verified Configuration Log. Installation Phase completed successfully. Build Software, Servers, and Workstations • All hardware was documented and photographed that was used in the tests, including servers, workstations, monitors, printers, voting devices, and peripherals. • Verified that the servers and workstations have been wiped. • Verified that the hardware provided meets or exceeds the minimum requirements in the four installation procedure manuals. • Verified that the required commercial-off-the-shelf (COTS) software licenses are available. • Verified that the component list of software and firmware exactly matches what should be installed. • Followed the Dominion Installation Procedures for the server and clients. • Followed the sequence of steps in the installation procedures manual. After the installation, COTS applications, proprietary applications, hardening, and configuration, an image of the system was taken. • Installed/setup Mobile Ballot Printing. • Installed imprinters on all scanners that will provide this option. • Verified the functionality of ICE. • When the installation was completed, in addition to the images, HASH values were taken for each component software. • Verified system security policies, services, data sources, and registry are documented. • Audited the system configuration and setup against specifications in the manuals to verify that scripts used in installation and configuration achieved the specifications. • Followed the Dominion System Verification Procedures applicable for each machine and logged any errors and omissions or anomalies in the issue log. • Verified the voting system has deployed COTS protection against viruses, worms, Trojan horses, and logic bombs. All listed items completed successfully California Certification Page 7 of 20 Functional Test Report v5.0 Report Number CDV-19011-FTR-01 Dominion Democracy Suite 5.10 Voting System California Certification Functional Test Report Install Firmware on Hardware Devices • Examined the hardware devices and determined which version of firmware is currently installed. • Took steps necessary to allow installation of the trusted build of firmware on each hardware device. • Took steps as necessary to validate instructions for firmware upgrades in use procedures and other system documentation. • After firmware was installed, verified that hardware devices are operational. • Followed the Dominion System Verification Procedures for each hardware device to validate the firmware installed on that device. • Verified no compilers, assemblers, or source code are resident on the system. • Verified that election specific firmware is not installed on the same component that the operating system is installed on. • Verified all validation requirements. • Logged any errors and omissions or anomalies in the issue log. • Provided a copy of the issue log to Dominion personnel for their resolution. All listed items completed successfully. Post Installation Phase • Took Acronis images of all equipment. • All images sent to SLI for restore on equipment used for Security testing. • Created Master copy of “Trusted Build” directory structure, files, and “Golden Images” for distribution to counties. Post Installation Phase completed successfully. Functional Configuration Audit Phase The Functional Configuration Audit is conducted to verify that the system performs all the functions described in the system documentation. The manufacturer shall: • Completely describe its procedures and related conventions used to support this audit for all system components. • Provide the following information to support this audit: o Copies of all procedures used for module or unit testing, integration testing, and system testing. o Copies of all test cases generated for each module and integration test, and sample ballot formats or other test cases used for system tests. California Certification Page 8 of 20 Functional Test Report v5.0 Report Number CDV-19011-FTR-01 Dominion Democracy Suite 5.10 Voting System California Certification Functional Test Report o Records of all tests performed by the procedures listed above, including error corrections and retests. Dominion provided test cases, results records, procedures used for module or unit testing, integration testing, and system testing. All required information was verified to satisfactorily meet the requirements of this section. Functional Test Phase Functional Test Phase testing examined the system to determine that every functional piece of the system is accurate and complete. During Functional Phase testing, an issue log was maintained of any errors and omissions found in the documentation or anomalies encountered that were not identified during the PCA. Throughout the Functional Testing Phase, Mobile Ballot Production (BOD) was used to generate ballots that have been fatigued or needed replacement. The system was maintained in an air gapped fashion: The architecture did allow transfer of the election definition and tally database from the permanent server(s) to the sacrificial server using USB devices. The voting system architecture does allow each installation to use its own Ethernet network, port server, and central-office vote-recording units, including any DRE and optical scan units, permitting the two installations to be segregated and air-gapped to ensure that there are no cross connections. An air gap was established by keeping two installations/networks physically separate and seeing that no device attached to the sacrificial installation/network was connected (directly or indirectly) to the first network, ensuring that data cannot flow from one installation/network to the other. • Inspected the test decks prepared by Dominion and audited them against the Ballot Marking and Expected Results Excel files. • Using EED, built an election from scratch and verified all options in EED. This election contained a multiple candidate contest for a “choose two of three” contest. o Printed ballots on MBP device. o Scanned ballots. o Tabulated and generated reports. • Extensive review of EED and RTR to verify end user documentation. • Evaluated methods to import elections into EED. • Evaluated Access Controls. Verified all CVSS requirements. • Proofed the test deck for the Presidential Primary, Presidential General, Special Recall, Ranked Choice Voting, and Vote Center elections. Scanned and tabulated the test deck for the Presidential Primary election on ICC. California Certification Page 9 of 20 Functional Test Report v5.0 Report Number CDV-19011-FTR-01 Dominion Democracy Suite 5.10 Voting System California Certification Functional Test Report Compared the tabulation results of this scan to expected results. Investigated any anomalies encountered in the results and resolved them by modifying either the deck or the expected results. • Conducted logic and accuracy testing in accordance with California Use Procedures. o Dominion logic and accuracy (L&A) test deck generation software was used to generate the test deck for L&A testing, as if being used by a County. Scanned predetermined test deck through scanners. o Reviewed ICE, ICP2, ICX Classic 21 and ICX Prime 21, ICC, and EMS for L&A procedures per DVS documentation. o Printed and verified L&A results from scanners. o Uploaded and consolidated results to EMS. o Printed and verified L&A results. o Verified that all components are ready to go after L&A or needed to be reprovisioned/reimaged prior to the actual election. All listed items completed successfully. Functional Testing Summary The tests run on the Democracy Suite 5.10 voting system included: • Presidential Primary Election • Presidential General Election • Special Recall Election • Ranked Choice Vote (RCV) Election • Vote Center Election Test Presidential Primary Election A primary election was run utilizing: • 2 ICX polling place devices • 1 ICE polling place device • 1 ICP2 polling place device • 1 ICC environment • 1 EMS (EED, Adjudication, RTR) The following steps were completed with results as noted: Presidential Primary was conducted in English and Spanish. The election included ten precincts and eight party splits per precinct. It included 20 contests and 65 California Certification Page 10 of 20 Functional Test Report v5.0 Report Number CDV-19011-FTR-01 Dominion Democracy Suite 5.10 Voting System California Certification Functional Test Report choices. Paper ballots included one bi-lingual ballot style consisting of English and Spanish. Electronic devices and interfaces were programmed to support the entire election in English and Spanish. Ballots were of medium size (8.5” x 17”) for the system, and double sided. • Prepared system for test election • Finalized EMS and set up for reporting • Prepared all precinct components (ICE, ICP2, ICX Classic 21, and ICX Prime 21) for election • Evaluated system for air-gap requirements • Opened Polls in accordance with California Use Procedures • Printed and verified zero reports for all devices • Printed ten ballots on Mobile Ballot Production for every language available in this election. Added these to the expected results • Scanned two ballots from each party on an ICE • Scanned two ballots from each party on an ICP2 • Marked one ballot from each party on an ICX Classic • Marked one ballot from each party on an ICX Prime • Verified that the voter can review, confirm, and change their selections on the ICX Classic 21, ICX Prime 21, ICE and ICP2 • Used preprinted ballots for the ICX’s above so the results did not need to be added to the expected results • Printed zero reports • Casted votes according to the following table: Definition Equipment Voting activity Absentee ICC All Precincts Early ICE #1 Precinct 11400 Voting ICP2 #1 Precinct 13102 ICX Classic 21 #1 All Precincts ICX Classic 21 #2 All Precincts ICX Prime 21 #1 All Precincts ICX Prime 21 #2 All Precincts Precinct ICE #2 Precinct 13102 Voting ICE #3 Precinct 21728 ICP2 #2 Precinct 13102 ICP2 #3 Precinct 21728 California Certification Page 11 of 20 Functional Test Report v5.0 Report Number CDV-19011-FTR-01 Dominion Democracy Suite 5.10 Voting System California Certification Functional Test Report ICX Classic 21 #3 Precincts 22740 and ICX Prime 21 #3 890240 Precincts 22740 and 890240 During voting, tested the following: • Fed ballots in all directions/sides on all devices • Closed polls in accordance with California Use Procedures • Printed results from all scanners • Removed results media to transfer results back to EMS/RTR • Shut down devices • Consolidated and reported: o Uploaded results to EMS from all units o Canvass reconciliation Processed provisional ballots Using adjudication component, adjudicated 40 hand marked ballots with write-ins o Generated final reports and verified: Generated all reports available on the system. Saved all reports to a flash drive as artifacts of testing Canvass – SOV. Verified accuracy of report SSOV Precinct results Cast Vote Record Report Audit reports (Including tabulation devices) • Backed up system to provide “Vote Count Program” to SOS. Evaluated default file name for submission to SOS. Checked backup size. • Verified system logging for all events. Saved system logs to archive. All listed items of election completed successfully. No issues were encountered. California Certification Page 12 of 20 Functional Test Report v5.0 Report Number CDV-19011-FTR-01 Dominion Democracy Suite 5.10 Voting System California Certification Functional Test Report Test Presidential General Election A general election was run utilizing: • 2 ICX polling place devices • 1 ICE polling place device • 1 ICP2 polling place device • 1 ICC environment • 1 EMS (EED, Adjudication, RTR) The following steps were completed with results as noted: Presidential Primary was conducted in English and Spanish. The election included ten precincts and eight party splits per precinct. It included twenty contests and sixty-five choices. Paper ballots included one bi-lingual ballot style consisting of English and Spanish. Where supported, electronic devices and interfaces should be programmed to support the entire election in English and Spanish. Ballots were of medium size (8.5” x 11”) for the system, and double sided. • Prepared system for test election • Finalized EMS and set up for reporting • Prepared all precinct components (ICE, ICP2, ICX Classic 21, and ICX Prime 21) for election • Evaluated system for air-gap requirements • Opened Polls in accordance with California Use Procedures • Printed and verified zero reports for all devices • Printed ten ballots on Mobile Ballot Production for every language available in this election. Added these to the expected results • Scanned two ballots from each party on an ICE • Scanned two ballots from each party on an ICP2 • Marked one ballot from each party on an ICX Classic • Marked one ballot from each party on an ICX Prime • Verified that the voter can review, confirm and change their selections on the ICX Classic 21, ICX Prime 21, ICE and ICP2 • Used preprinted ballots for the ICX’s above so the results did not need to be added to the expected results • Printed zero reports California Certification Page 13 of 20 Functional Test Report v5.0 Report Number CDV-19011-FTR-01 Dominion Democracy Suite 5.10 Voting System California Certification Functional Test Report • Casted votes according to the following table: Definition Equipment Voting activity Absentee ICC All Precincts Early ICE #1 Precinct 11400 Voting ICP2 #1 Precinct 13102 ICX Classic 21 #1 All Precincts ICX Classic 21 #2 All Precincts ICX Prime 21 #1 All Precincts ICX Prime 21 #2 All Precincts Precinct ICE #2 Precinct 13102 Voting ICE #3 Precinct 21728 ICP2 #2 Precinct 13102 ICP2 #3 Precinct 21728 ICX Classic 21 #3 Precincts 22740 and 890240 ICX Prime 21 #3 Precincts 22740 and 890240 During voting, tested the following: • Fed ballots in all directions/sides on all devices • Closed polls in accordance with California Use Procedures • Printed results from all scanners • Removed results media to transfer results back to EMS/RTR • Shut down devices • Consolidated and reported: o Uploaded results to EMS from all units o Canvass reconciliation Processed provisional ballots Using adjudication component, adjudicated 40 hand marked ballots with write-ins o Generated final reports & verified: Generated all reports available on the system. Saved all reports to a flash drive as artifacts of testing Canvass – SOV. Verified accuracy of report SSOV Precinct results Cast Vote Record Report California Certification Page 14 of 20 Functional Test Report v5.0 Report Number CDV-19011-FTR-01 Dominion Democracy Suite 5.10 Voting System California Certification Functional Test Report Audit reports (Including tabulation devices) • Backed up system to provide “Vote Count Program” to SOS. Evaluated default file name for submission to SOS. Checked backup size • Verified system logging for all events. Saved system logs to archive All listed items of election completed successfully. No issues were encountered. Test Special Recall Election A Recall election was run utilizing: • 2 ICX polling place devices • 1 ICE polling place device • 1 ICP2 polling place device • 1 ICC environment • 1 EMS (EED, Adjudication, RTR) Special Recall in English, Khmer, Japanese, and Hindi. The election consisted of one precinct and one contest. The contest included 135 choices with one write-in in a gubernatorial contest. The election was printed on the longest ballot size possible for the system. • Prepared all components for election • Installed election definitions on devices, printed zero reports and opened polls • Scanned test decks through each tabulating device • Marked one ballot with non-standard marks and increasingly marginal marks for each type of marker. Included a variety of pens, pencils and highlighters in various colors. A blank ballot and a fully marked ballot were run in a separate batch • During voting, tested the following: o Fed ballots in all directions/sides on all devices o Printed ten ballots on Mobile Ballot Production for each supported language in the election. Voted five on ICE, and hand marked five. Added these ballots to the expected results. Verified logging on MBP meets batch printing audit requirements o Language support for alternative languages: Khmer, Japanese, and Hindi ICE display ICE audio ballot ICX Classic 21-inch display California Certification Page 15 of 20 Functional Test Report v5.0 Report Number CDV-19011-FTR-01 Dominion Democracy Suite 5.10 Voting System California Certification Functional Test Report ICX Classic 21-inch audio ballot ICX Classic 21-inch printing ICX Prime 21-inch display ICX Prime 21-inch audio ballot ICX Prime 21-inch printing ICP2 display o Scanned the marked ballot ten times on each scanner device o Closed each machine and printed out results o Evaluated consistency of the scanners in the way they handle the marginal marks o Tested the ability to select candidates across multiple screens on the ICEs, ICX Classic 21’s and ICX Prime 21’s o Closed polls, in accordance with California Use Procedures • Printed results from all scanners • Transferred results back to EMS • Shut down devices • Consolidated and reported o Uploaded results to EMS from all units o Canvass reconciliation Provisional ballots Write-ins o Generated final reports & verified Canvass – SOV SSOV Precincts Audit reports (Including tabulation devices) All listed items of election completed successfully. No issues were encountered. California Certification Page 16 of 20 Functional Test Report v5.0 Report Number CDV-19011-FTR-01 Dominion Democracy Suite 5.10 Voting System California Certification Functional Test Report Test Ranked Choice Election A Ranked Choice Vote election was run utilizing: • 2 ICX polling place devices • 1 ICE polling place device • 1 ICP2 polling place device • 1 ICC DR-g1130 environment • 1 EMS (EED, Adjudication, RTR) Ranked Choice Voting in English, Ilocano, Tagalog, and Thai. A fictitious single and multiple seat RCV Election with one ballot style with one contest containing six to ten candidates was utilized. • Prepared all components for election • Initialized and loaded election definition on one ICE, ICP2, ICX Classic 21, ICX Prime 21, and all ICCs • Evaluated system for air-gap requirements • Opened Polls in accordance with California Use Procedures • Printed and verified zero reports for all devices • Scanned one ballot on ICE • Scanned one ballot on ICP2 • Marked one ballot on an ICX Classic 21-inch • Marked one ballot on an ICX Prime 21-inch • Removed ballots from the pre-marked test deck, and remade these on the ICX Classic 21’s and ICX Prime 21’s • Printed zero reports from EMS • Cast all ballots on each scanner • During voting, tested the following o Fed ballots in all directions/sides on all devices o Language supported included alternative languages: Ilocano, Tagalog, and Thai ICE display ICE audio ICX Classic 21-inch display ICX Classic 21-inch audio ICX Classic 21-inch printing ICX Prime 21-inch display ICX Prime 21-inch audio California Certification Page 17 of 20 Functional Test Report v5.0 Report Number CDV-19011-FTR-01 Dominion Democracy Suite 5.10 Voting System California Certification Functional Test Report ICX Prime 21-inch printing ICP2 display • Closed polls, in accordance with California Use Procedures • Printed results from all scanners • Removed results media as necessary to transfer results back to EMS • Shut down devices • Ranked results and determine winner • Verified and documented all one seat and multi seat RCV on the system • Consolidated and reported • Uploaded results to EMS from all units • Generated final reports & verified totals • Generated Cast Vote Record Spreadsheet • Audited Cast Vote Record Spreadsheet against ballots All listed items of election completed successfully. No issues were encountered. Test Vote Center Election An election was run utilizing: • 2 ICX polling place devices • 1 ICE polling place device • 1 ICP2 polling place device • 1 ICC DR-g1130 environment • 1 EMS (EED, Adjudication, RTR) Vote Center Election in English. A fictitious election including approximately five contests, ten choices, and three thousand precincts, or the maximum number of precincts supported by the system was utilized. As a system test of label capacity, one candidate was created as candidate Last Name starting with the character 'L' and extending with 29 digits as indicated below. Entered the First Name starting with the character 'F' and extending with 19 digits as indicated below. For printed ballots, used the following for the candidate name: “F12345678901234567890 L123456789012345678901234567890”. Twenty-five percent of vote center election ballots were machine pre-folded in a tri-fold format. Pattern of precincts were 3000 divided by 125: 1, 125, 250, 375, 500, 625, …. • Prepared all precinct components for election • Configured ICX Classic 21-inch /ICX Prime 21-inch /ICE/ICP2 for use in early voting (all precincts) California Certification Page 18 of 20 Functional Test Report v5.0 Report Number CDV-19011-FTR-01 Dominion Democracy Suite 5.10 Voting System California Certification Functional Test Report • Initialized and loaded election definition on ICEs, ICP2, ICC, ICX Classic 21’s, ICX Prime 21’s, and Mobile Ballot Production • Opened Polls in accordance with California Use Procedures • Printed and verified zero reports for all devices • Printed ballots from the pattern of precincts, on Mobile Ballot Production. Added ballots to the expected results • Created activation cards per the pattern of precincts • Simulated early voting. Voted ballots on vote center voter facing components and then suspended voting, enabled voting and voted more ballots • Scanned pattern of precincts ballots on an ICE • Scanned pattern of precincts ballots on an ICP2 • Marked ballots from pattern of precincts on an ICX Classic • Marked ballots from pattern of precinct on an ICX Prime • Unfolded pre-folded ballots and processed as mail in ballots • Printed zero reports from EMS • Scanned all preprinted ballots in equal numbers on all ICC scanners • Created eight marginal marks ballots including all highlighter and pen/pencil colors. Numbered the ballots and scanned in order on all scanners to verify all scanners scanned consistently and adjudication was consistent across all scanners • Exercised adjudication workstation on marginal marks ballots • Closed polls, in accordance with California Use Procedures • Verified ballot images are stored in a random manner. • Printed results from all scanners • Transferred results back to EMS • Shut down devices • Consolidated and reported o Uploaded results to EMS from all units o Canvass reconciliation o Provisional ballots o Write-ins • Generated final reports & verified o Generated all reports the system is capable of o Canvass – SOV o SSOV California Certification Page 19 of 20 Functional Test Report v5.0 Report Number CDV-19011-FTR-01 Dominion Democracy Suite 5.10 Voting System California Certification Functional Test Report o Precincts o Audit reports (Including tabulation devices) All listed items of election completed successfully. No issues were encountered. Final Data Capture and Analysis • Took hashes and images from all computers • Validated the system on each server, desktop, and one each of the ICE, ICP2, ICC, ICX Classic 21, ICX Prime 21 using the procedure provided by Dominion • Generated “Trusted Build” software and Golden (County Release) images to be distributed by Secretary of State to Vendor/County • Verified with vendor trusted build software was complete and ready for distribution. Dominion attested to the correctness of the files • Verified that all drivers including monitor video drivers are included in the County Release Evaluation of Testing The above tests were conducted using the executables created in the Trusted Build, in association with the appropriate hardware versions as declared during the current certification project for the Dominion Democracy Suite 5.10 voting system, for the State of California. Summary of Testing includes: • No functional issues were encountered during testing • Documentation needed significant re-write which resulted in an additional round of review and verification. Documentation is deemed sufficient at this time. End of Functional Test Report California Certification Page 20 of 20 Functional Test Report v5.0 Report Number CDV-19011-FTR-01