Consultant's Functional Test Report

Dominion Democracy Suite 5.2 Functional Test Report CDV-306-FTR-01 Prepared for: Vendor Name Dominion Voting Systems Vendor System DVS 5.2 Prepared by: SLI ComplianceSM 4720 Independence St. Wheat Ridge, CO 80033 303-422-1566 www.SLICompliance.com Accredited by the Election Assistance Commission (EAC) for Selected Voting System Test Methods or Services Test Report v1.0 Page 1 of 13 Report Number HRT-17002-CTR-01 Dominion Voting Democracy Suite 5.2 California Certification Test Report Copyright 2017 SLI ComplianceSM Revision History Release Author Revision Summary 1.0 M. Santos Initial Release 2.0 M. Santos Added results detail and formatting Disclaimer The Certification Test results reported herein must not be used by the client to claim product certification, approval, or endorsement by NVLAP, NIST, or any agency of the Federal Government. Results herein relate only to the items tested. Trademarks • SLI is a registered trademark of SLI Compliance, a Division of Gaming Laboratories International, LLC. • Intel and Pentium are registered trademarks of Intel Corporation. • Microsoft, MS are registered trademarks and Internet Explorer, Windows, Visual C++, Visual Basic, VBX, ODBC, and MFC are trademarks of Microsoft Corporation. • All other products and company names are used for identification purposes only and may be trademarks of their respective owners. The tests referenced in this document were performed in a controlled environment using specific systems and data sets and results are related to the specific items tested. Actual results in other environments may vary. Opinions and Interpretations There are no SLI opinions or interpretations included in this report. Test Report v2.0 Page 2 of 13 Report Number CDV-306-FTR-01 Dominion Voting Democracy Suite 5.2 California Certification Test Report TABLE OF CONTENTS INTRODUCTION ........................................................................................................................ 4 References ....................................................................................................................................................... 4 SYSTEM OVERVIEW ................................................................................................................. 4 Scope of the Dominion Democracy Suite 5.2 Voting System .......................................................................... 4 System Topology Diagram ............................................................................................................................... 5 CERTIFICATION TEST RESULTS SUMMARY ................................................................................. 6 Functional Testing Summary ............................................................................................................................ 6 Primary Election ........................................................................................................................................... 6 General Election .......................................................................................................................................... 7 Recall Election ............................................................................................................................................. 8 RCV Election ................................................................................................................................................ 9 Accessibility ............................................................................................................................................... 10 Maximum Ballot Styles ............................................................................................................................... 11 Two Hour Battery Test, ICE ....................................................................................................................... 12 Two Hour Battery Test, ICX ....................................................................................................................... 12 Evaluation of Testing ...................................................................................................................................... 13 Test Report v2.0 Page 3 of 13 Report Number CDV-306-FTR-01 Dominion Voting Democracy Suite 5.2 California Certification Test Report Introduction SLI Compliance is submitting this test report as a summary of the certification testing efforts for the Dominion Democracy Suite 5.2 voting system. The purpose of this document is to provide an overview of the certification testing effort and the findings of the testing effort for the Dominion Democracy Suite 5.2 voting system. This effort included functional testing of the Dominion Democracy Suite 5.2 voting system. References • California Voting System Standards (CVSS) System Overview Scope of the Dominion Democracy Suite 5.2 Voting System This section provides a description of the scope of Dominion Democracy Suite 5.2 voting system components: • Election Management System (EMS) – Election Event Designer (EED) application • EMS – Results Tally and Reporting (RTR) application • EMS – Adjudication Services (Adjudication) application • ImageCast Central (ICC) application • ImageCast Evolution (ICE) firmware/hardware • ImageCast X (ICX) firmware/hardware The Dominion Democracy Suite 5.2 Election Management System (EMS) represents a set of N-Tier software applications (EMS, RTR, Adjudication) for pre- voting and post-voting election project activities that are applicable to jurisdictions of various sizes and geo-political complexities. The Dominion Democracy Suite 5.2 ICC system consists of a central high-speed optical scan ballot counter (tabulator) called the ICC Ballot Counter and is used for processing absentee ballots (such as vote by mail). This ballot counter unit is based on commercial off the shelf (COTS) hardware coupled with custom-made ballot processing application software. It is used for high-speed centralized scanning and counting of paper ballots. Test Report v2.0 Page 4 of 13 Report Number CDV-306-FTR-01 Dominion Voting Democracy Suite 5.2 California Certification Test Report The Dominion Democracy Suite 5.2 ICE system employs a precinct-level optical scan ballot counter (tabulator) in conjunction with an external ballot box. This tabulator is designed to mark and/or scan paper ballots, interpret voting marks, communicate these interpretations back to the voter (either visually through the integrated LCD display or audibly via integrated headphones) and, upon the voter’s acceptance, deposit the ballots into the secure ballot box. The Dominion Democracy Suite 5.2 ICX ballot marking platform is used for creation of paper Electronic Mobile Ballots. These ballots are later scanned and tabulated by the ICC optical ballot counter and/or scanned, verified, and cast by the ICE. System Topology Diagram Test Report v2.0 Page 5 of 13 Report Number CDV-306-FTR-01 Dominion Voting Democracy Suite 5.2 California Certification Test Report Certification Test Results Summary Functional Testing Summary The tests run on the Dominion Democracy Suite 5.2 voting system included: • Primary Election • General Election • Recall Election • Ranked Choice Vote (RCV) Election • Accessibility • Maximum Ballot Styles • ICE Two Hour Battery Test • ICX Two Hour Battery Test Primary Election A primary election was run utilizing: • 1 ICX polling place device • 1 ICE polling place device • 1 ICC DR-g1130 environment • 1 ICC DR-x10c environment • 1 EMS (EED, Adjudication, RTR) The following steps were completed with results as noted: 1. Polls were opened in accordance with California Use Procedures. Zero reports for all devices were printed and verified. No issues were encountered. 2. The ICX polling place ballot marking device was utilized to create voted ballots in English, Spanish, and Chinese, including changing from English to Spanish midway through the ballot being voted. Text size and contrast were verified to meet expected criteria. No issues were encountered. 3. The ICE polling place scanning device was utilized to scan voted ballots, including those from ICX, and including English, Spanish and Chinese ballots. No issues were encountered. Test Report v2.0 Page 6 of 13 Report Number CDV-306-FTR-01 Dominion Voting Democracy Suite 5.2 California Certification Test Report 4. The ICC central count location scanning devices were utilized to scan voted ballots, including those from ICX, and including English, Spanish and Chinese ballots. The DR-x10c implementation of the ICC did have a roller issue, apparently because it was idle for over a year. Replacement of the roller resolved the issue. No other issues were encountered. 5. Polls were closed in accordance with California Use Procedures, including printing results from ICE and ICC, removing results media to transfer results back to EMS, and then shutting down devices. No issues were encountered. 6. Post Election results were consolidated and reported based on the upload of results to EMS from all tabulating (ICE and ICC) units. Results included reconciliation of write-ins as well as generation of final reports and verifying Canvass – SOV, SSOV, Precinct results, overvotes, and undervotes. No issues were encountered. General Election A general election was run utilizing: • 1 ICX polling place device • 1 ICE polling place device • 1 ICC DR-g1130 environment • 1 EMS (EED, Adjudication, RTR) The following steps were completed with results as noted: 1. Polls were opened in accordance with California Use Procedures. Zero reports for all devices were printed and verified. No issues were encountered. 2. The ICX polling place ballot marking device was utilized to create voted ballots in English. No issues were encountered. 3. The ICE polling place scanning device was utilized to scan voted ballots, including those from ICX. Included were overvoted ballots, ballots with marginal marks, and ballots with write-ins. No issues were encountered. Test Report v2.0 Page 7 of 13 Report Number CDV-306-FTR-01 Dominion Voting Democracy Suite 5.2 California Certification Test Report 4. The ICC central count location scanning devices were utilized to scan voted ballots, including those from ICX. Included were overvoted ballots, ballots with marginal marks, and ballots with write-ins. Ballots were adjudicated within ICC as needed. No issues were encountered. 5. Polls were closed in accordance with California Use Procedures, including printing results from ICE and ICC, removing results media to transfer results back to EMS, and then shutting down devices. No issues were encountered. 6. Post-Election results were consolidated and reported based on upload of results to EMS from all tabulating (ICE and ICC) units. Results included reconciliation of write-ins as well as generation of final reports and verifying Canvass – SOV, SSOV, Precinct results, overvotes, and undervotes. One issue observed was that third party generated reports did not report ambiguous marks correctly as undervotes. No other issues were encountered. Recall Election A Recall election was run utilizing: • 1 ICX polling place device • 1 ICE polling place device • 1 ICC DR-g1130 environment • 1 EMS (EED, Adjudication, RTR) One ballot style with one contest containing 155 candidates and one write-in was utilized. The156 ballots were marked such that each candidate and the write-in received one vote. The following steps were completed with results as noted: 1. Polls were opened in accordance with California Use Procedures. Zero reports for all devices were printed and verified. No issues were encountered. 2. The ICE polling place scanning device was utilized to scan all 156 voted ballots. No issues were encountered. Test Report v2.0 Page 8 of 13 Report Number CDV-306-FTR-01 Dominion Voting Democracy Suite 5.2 California Certification Test Report 3. The ICX polling place ballot marking device was utilized to create voted ballots which represented the last five ballots in the test deck, including the write-in candidate. No issues were encountered. 4. The ICC central count location scanning device was utilized to scan voted ballots 1 through 151 from the test deck, as well as the 5 from ICX. No issues were encountered. 5. Polls were closed in accordance with California Use Procedures, including printing results from ICE and ICC, removing results media to transfer results back to EMS, and then shutting down devices. No issues were encountered. 6. Post-Election results were consolidated and reported based on upload of results to EMS from all tabulating (ICE and ICC) units. Results included reconciliation of write-ins as well as generation of final reports and verifying Canvass – SOV, SSOV, Precinct results, overvotes, and undervotes. No issues were encountered. RCV Election A Ranked Choice Vote election was run utilizing: • 1 ICX polling place device • 1 ICE polling place device • 1 ICC DR-g1130 environment • 1 EMS (EED, Adjudication, RTR) One ballot style with one contest containing Ranked Choice candidates and one write-in was utilized. The 24 ballots were marked such that each candidate and the write-in received one vote. The following steps were completed with results as noted: 1. Polls were opened in accordance with California Use Procedures. Zero reports for all devices were printed and verified. No issues were encountered. 2. The ICE polling place scanning device was utilized to scan the 24 voted ballots. No issues were encountered. Test Report v2.0 Page 9 of 13 Report Number CDV-306-FTR-01 Dominion Voting Democracy Suite 5.2 California Certification Test Report 3. The ICX polling place ballot marking device was utilized to create five voted ballots—recreations of five of the original ballots—which were then placed in the deck of 24, replacing the originals. No issues were encountered. 4. The ICC central count location scanning device was utilized to scan the updated deck of 24 voted ballots, including those from ICX. The deck was run twice through the ICC. No issues were encountered 5. Polls were closed in accordance with California Use Procedures, including printing results from ICE and ICC, removing results media to transfer results back to EMS, and then shutting down devices. No issues were encountered. 6. At this point a total of 72 ballots had been counted, with results matching what was expected. Seven ballots were added, that were marked for write-in candidate John Doe. These results were run through ICC once for a total of 79 ballots. Results were added to totals and verified to match expected outcomes. No issues were encountered. 7. Post-Election results were consolidated and reported based on upload of results to EMS from all tabulating (ICE and ICC) units. Results included reconciliation of write-ins as well as generation of final reports and verifying Canvass – SOV, SSOV, Precinct results, overvotes, and undervotes. No issues were encountered. Accessibility An election was run utilizing: • 1 ICX polling place device The following steps were completed with results as noted: 1. Polls were opened in accordance with California Use Procedures. Zero report was printed and verified. No issues were encountered. 2. The ICX polling place ballot marking device was utilized, employing audio and ATI controller to create voted ballots. Ballot was successfully navigated and printed No issues were encountered. Test Report v2.0 Page 10 of 13 Report Number CDV-306-FTR-01 Dominion Voting Democracy Suite 5.2 California Certification Test Report Maximum Ballot Styles An election was run utilizing: • 1 ICX polling place device • 1 ICE polling place device • 1 ICC DR-g1130 environment • 1 EMS (EED, Adjudication, RTR) ICX was set up with 1500 precincts, 12,000 ballot styles. Additionally, three languages were added per ballot style for a total of 36,000 unique ballot styles. ICE was set up with 1500 precincts, 12,000 ballot styles. Additionally, two languages were added per ballot style for a total of 24,000 unique ballot styles. The following steps were completed with results as noted: 1. Polls were opened in accordance with California Use Procedures. Zero reports for all devices were printed and verified. No issues were encountered. 2. The ICX polling place ballot marking device was utilized to create voted ballots in English, Spanish, and Chinese. A total of 30 ballots were voted. No issues were encountered. 3. Utilizing both 1-sided and 2-sided ballots, 169 voted ballots were created. The ICE polling place scanning device was utilized to scan the 169 voted ballots. One additional ballot was created and voted through the ICE AVS. A total of 170 ballots were scanned. No issues were encountered. 4. The ICC central count location scanning device was utilized to scan the 30 voted ballots from ICX. No issues were encountered 5. Polls were closed in accordance with California Use Procedures, including printing results from ICE and ICC, removing results media to transfer results back to EMS, and then shutting down devices. No issues were encountered. 6. Post-Election results were consolidated and reported based on upload of results to EMS from all tabulating (ICE and ICC) units. Results included reconciliation of write-ins as well as generation of final reports and verifying Canvass – SOV, SSOV, Precinct results, overvotes and undervotes. No issues were encountered. Test Report v2.0 Page 11 of 13 Report Number CDV-306-FTR-01 Dominion Voting Democracy Suite 5.2 California Certification Test Report Two Hour Battery Test, ICE An election was run utilizing: • 1 ICE polling place device The following steps were completed with results as noted: 1. Polls were opened in accordance with California Use Procedures. Zero report was printed and verified. No issues were encountered. 2. The ICE polling place scanning device was utilized to scan voted ballots for two hours on battery power; 22-inch ballots were employed, scanning one ballot per minute. At the two hour mark, the ICE device was at 70% battery strength. 3. Polls were closed in accordance with California Use Procedures, including printing results from ICE and ICC, removing results media to transfer results back to EMS, and then shutting down devices. No issues were encountered. Two Hour Battery Test, ICX An election was run utilizing: • 1 ICX polling place device The following steps were completed with results as noted: 1. Polls were opened in accordance with California Use Procedures. Zero report was printed and verified. No issues were encountered. 2. The ICE polling place scanning device was utilized to vote ballots for two hours on battery power voting one ballot every three minutes. The ICX device ran out of power at the 45 minute mark, after 15 ballots. This result is considered a failure of the 2 hour battery test for the ICX. Dominion believed that the Uninterruptable Power Supply (UPS) had not been fully charged. CASOS determined that the test will be re-run, after the UPS has been charged for a minimum of 48 hours. Test Report v2.0 Page 12 of 13 Report Number CDV-306-FTR-01 Dominion Voting Democracy Suite 5.2 California Certification Test Report Evaluation of Testing The above tests were conducted using the executables created in the EAC Trusted Build, in association with the appropriate hardware versions as declared during the national certification project for the Dominion Democracy Suite 5.2 voting system. Summary of Testing includes: In the Primary election, the DR-x10c implementation of the ICC did have a roller issue, apparently because it was idle for over a year. Replacement of the roller resolved the issue. In the General election, one issue observed was that third party generated reports did not report ambiguous marks correctly as undervotes. In the ICX Two Hour Battery Test, the ICX device ran out of power at the 45 minute mark. This result is considered a failure of the 2 hour battery test for the ICX. End of Test Report Test Report v2.0 Page 13 of 13 Report Number CDV-306-FTR-01