Consultant's Functional Report

Election Systems & Software EVS 6.3.0.2 Voting System Functional Test Report for California Secretary of State CAF-ESS22001-FTR-01 Vendor Name Election Systems and Software (ES&S) Vendor System EVS 6.3.0.2 Prepared by: SLI ComplianceSM 4720 Independence St. Wheat Ridge, CO 80033 303-422-1566 www.SLICompliance.com Accredited by the Election Assistance Commission (EAC) for VSTL status. Election Systems and Software EVS 6.3.0.2 California Certification Functional Test Report Copyright © 2022 SLI ComplianceSM, a Division of Gaming Laboratories International, LLC Revision History Date Release Author Revisions 11/22/2022 1.0 M. Santos Initial Release Disclaimer The Certification Test results reported herein must not be used by the client to claim product certification, approval, or endorsement by NVLAP, NIST, or any agency of the Federal Government. Results herein relate only to the items tested. Trademarks • SLI is a registered trademark of SLI Compliance, a Division of Gaming Laboratories International, LLC. • All other products and company names are used for identification purposes only and may be trademarks of their respective owners. The tests referenced in this document were performed in a controlled environment using specific systems and data sets and results are related to the specific items tested. Actual results in other environments may vary. Opinions and Interpretations There are no SLI opinions or interpretations included in this report. Functional Test Report v1.0 Page 2 of 14 Report Number CAF-ESS22001-FTR-01 Election Systems and Software EVS 6.3.0.2 California Certification Functional Test Report TABLE OF CONTENTS INTRODUCTION ................................................................................................................................... 4 REFERENCES....................................................................................................................................... 4 SYSTEM OVERVIEW ........................................................................................................................... 4 VOTING SYSTEM SCOPE ....................................................................................................................... 4 CERTIFICATION TEST RESULTS SUMMARY ................................................................................... 5 FUNCTIONAL TESTING SUMMARY .......................................................................................................... 5 Equipment Tested .......................................................................................................................... 5 Physical Configuration Audit .......................................................................................................... 5 Installation Process/Functional Configuration Audit ...................................................................... 5 Functional Test Phase ................................................................................................................... 6 Gubernatorial Primary Election ...................................................................................................... 8 Language Special Election ............................................................................................................ 8 Marked Review Election ................................................................................................................ 9 Presidential General Election ...................................................................................................... 10 Presidential Primary Election ....................................................................................................... 11 Special Recall Election ................................................................................................................ 12 Vote Center Election .................................................................................................................... 13 Regression Testing ...................................................................................................................... 14 EVALUATION OF TESTING ................................................................................................................... 14 Functional Test Report v1.0 Page 3 of 14 Report Number CAF-ESS22001-FTR-01 Election Systems and Software EVS 6.3.0.2 California Certification Functional Test Report Introduction SLI Compliance is submitting this test report as a summary of the certification testing efforts for the Election Systems & Software EVS 6.3.0.2 (ES&S EVS 6.3.0.2) voting system. The purpose of this document is to provide an overview of the certification testing effort and the resulting findings for the ES&S EVS 6.3.0.2 voting system. References California Voting System Standards (CVSS) System Overview Voting System Scope This section provides a description of the scope of the following ES&S EVS 6.3.0.2 voting system components: • Electionware Election Management System application • DS450 application/hardware • DS850 application/hardware • DS950 application/hardware • DS200 firmware/hardware • DS300 firmware/hardware • ExpressVote HW2.1 firmware/hardware The ES&S EVS 6.3.0.2 Election Management System (Electionware, Paper Ballot) represents a set of N-Tier software applications (Election Management System (EMS), Results Tally & Reporting (RTR), and Adjudication) for pre-voting and post- voting election project activities that are applicable to jurisdictions of various sizes and geo-political complexities. The ES&S EVS 6.3.0.2 DS450, DS850, and DS950 central, high-speed, optical scan ballot counters (tabulators) are used for processing absentee ballots (such as vote by mail). This ballot counter unit is based on commercial off-the-shelf (COTS) hardware coupled with custom-made ballot processing application software. It is used for high-speed, accurate, and reliable centralized scanning and counting of paper ballots. The ES&S EVS 6.3.0.2 DS200 and DS300 systems are precinct-level optical scan ballot counters (tabulators) in conjunction with an external ballot box. These tabulators are designed to scan paper ballots, interpret voting marks, and deposit the ballots into the secure ballot box. Functional Test Report v1.0 Page 4 of 14 Report Number CAF-ESS22001-FTR-01 Election Systems and Software EVS 6.3.0.2 California Certification Functional Test Report The ES&S EVS 6.3.0.2 ExpressVote HW2.1 ballot marking device is a solution used to assist voters with marking ballots. These ballots are later scanned and tabulated by the DS200, DS300, DS450, DS850 or DS950 optical ballot counters. Certification Test Results Summary Functional Testing Summary Functional testing is divided into five phases: • Physical Configuration Audit • Installation Process • Functional Configuration Audit • Functional Test Phase • Regression Testing (If needed) Equipment Tested The equipment that was tested included: • Two DS200 devices • Two DS300 devices • Two DS450 devices • Two DS850 devices • Two DS950 devices • Two ExpressVote HW2.1 ballot marking devices • Two Electionware Benches each containing: o EMS Client o EMS Server o EMS Standalone Laptop Physical Configuration Audit The physical configuration audit took place from October 11–13, 2022. During this phase, an SLI engineer took photos, documented part numbers and serial numbers of each device under test and updated the Physical Configuration Audit traveler (equipment inventory) accordingly. Installation Process/Functional Configuration Audit The installation process and functional configuration audit phase occurred from October 11–13, 2022. During the initial process of this phase, all the servers, clients, and standalone stations were wiped clean using the KillDisk’s NIST 800-88 Functional Test Report v1.0 Page 5 of 14 Report Number CAF-ESS22001-FTR-01 Election Systems and Software EVS 6.3.0.2 California Certification Functional Test Report method. Once each device was cleared, and following the vendor provided installation documentation, SLI performed the following actions: • Windows 10 Enterprise LTSC was installed on the client and standalone with the server using Windows Server2022. • The SLI employee then installed all missing device drivers onto their respective devices. • Windows 10 was activated using the call number listed (Only for the Client and Standalone) • Configure network between Server and Client • The Engineer then completed the hardening of the bench pre-install by running the SecureSetup_6.3.0.2, provided by ES&S. • At this stage, all relevant programs were installed, such as Removable Media Service (RMS), Event Log Service (ELS), SumatraPDF, Amyuni, Electionware, Electionware Additional Reporting Module, Report Printer w/ Printer Control Language (PCL) driver, Arial Fonts. • Cerberus FTP Server program installed and configured on the Server. • The Post Install script was run using the SecureSetup_6.3.0.2 program • Using the SecureSetup_6.3.0.2 program, created User Accounts and then the Secure Administrator Account in Windows • Configure Accounts for Multi-factor Authentication • At this point, the trusted hash files were taken. • The System BIOS for the Server, Clients, and Standalone were hardened. Test Outcome It was found that the accessibility jack on one of the ExpressVote HW2.1 units was not functioning properly. ES&S sent a technician out to replace the jack, clean rollers, calibrate and other minor routine maintenances and the issue was resolved. Functional Test Phase Functional Testing was performed on October 13–17, 2022. This phase simulated an election process from start to finish. The following eight elections were tested: • Gubernatorial General Election • Gubernatorial Primary Election • Language Special Election • Marked Review Election • Presidential General Election • Presidential Primary Election • Special Recall Election Functional Test Report v1.0 Page 6 of 14 Report Number CAF-ESS22001-FTR-01 Election Systems and Software EVS 6.3.0.2 California Certification Functional Test Report • Vote Center Election Gubernatorial General Election The Gubernatorial General Election used 17-inch ballots and was coded by SLI. This election was created by the tester engineer on the standalone Electionware configuration. The SLI representative first created all the media USB devices for the following Election Devices: • One DS200 • One DS300 • One ExpressVote HW2.1 • One DS450 • One DS850 • One DS950 Once the polls had been opened, the zero reports were printed and verified on all devices. Five hand marked ballots were processed on the DS200, DS300, DS450, DS850, and DS950. An additional five ExpressVote Cards, voted and printed on ExpressVote HW2.1, were run through all tabulator devices once. A total of 25 hand marked ballots were processed. A total of 25 ExpressVote Card ballots were processed. A Grand total of 50 ballots were processed. Once all ballots had been processed and created on their respective devices, the polls were closed on the DS200 and DS300 in accordance with vendor procedures. Results and Audit Logs were then obtained from each scanner and all results media removed. DS200 and DS300 media was then brought into Electionware along with the results from the DS450, DS850, and DS950. Results reports were verified against the expected results and the devices in test were powered off. Test Outcome This election was run as described above, results verified successfully, and testing concluded with no issues. Functional Test Report v1.0 Page 7 of 14 Report Number CAF-ESS22001-FTR-01 Election Systems and Software EVS 6.3.0.2 California Certification Functional Test Report Gubernatorial Primary Election The Gubernatorial Primary Election used 17-inch ballots for an Open Primary election with Top Two Election. This election was imported into the Client/Server Electionware configuration using the import functionality. The SLI representative first created all the media USB devices for the following Election Devices: • One DS200 • One DS300 • One ExpressVote HW2.1 • One DS450 • One DS850 • One DS950 Once the polls had been opened, the zero reports were printed and verified on all devices. A deck of 20 hand marked ballots was utilized. Ten of the hand marked ballots were processed in the DS200, DS300, and all 20 hand marked ballots were processed through the DS450, DS850 and DS950 devices. An additional five ExpressVote Cards, voted and printed on ExpressVote HW2.1, were run through all tabulator devices once. A total of 80 hand marked ballots were processed. A total of 25 ExpressVote Card ballots were processed. A Grand total of 105 ballots were processed. Once all ballots had been processed and created on their respective devices, the polls were closed on the DS200 and DS300 in accordance with vendor procedures. Results and Audit Logs were then obtained from each scanner and all results media removed. DS200 and DS300 media was then brought into Electionware along with the results from the DS450, DS850, and DS950.Results reports were verified against the expected results and the devices in test were powered off. Test Outcome This election was run as described above, results verified successfully, and testing concluded with no issues. Language Special Election The Marked Review Election used 17-inch ballots, contained the languages English, Tagalog, Hmong, Ilocano, Punjabi, and Bengali. This election was imported into the Standalone laptop configuration using the import functionality. Functional Test Report v1.0 Page 8 of 14 Report Number CAF-ESS22001-FTR-01 Election Systems and Software EVS 6.3.0.2 California Certification Functional Test Report The SLI representative first created all the media USB devices for the following Election Devices: • One DS200 • One DS300 • One DS450 • One DS850 • One DS950 Logic and Accuracy testing was performed in accordance with California Use Procedure. Polls were then opened, the zero reports were printed and verified on all devices. Once the polls were opened, a stack of six hand marked ballots was then fed through all devices. A Grand total of 30 ballots were processed for this election. Once all ballots had been processed and created on their respective devices, the polls were closed on the DS200 and DS300 in accordance with vendor procedures. Results and Audit Logs were then obtained from each scanner and all results media removed. DS200 and DS300 media was then brought into Electionware along with the results from the DS450, DS850, and DS950. Results reports were verified against the expected results and the devices in test were powered off. Test Outcome This election was run as described above, results verified successfully, and testing concluded with no issues. Marked Review Election The Marked Review Election used 14-inch ballots and contained the languages English and Spanish along with Marked Review box functionality. This election was imported into the Standalone laptop configuration using the import functionality. The SLI representative first created all the media USB devices for the following Election Devices: • One DS200 • One DS300 • One ExpressVote HW2.1 • One DS450 • One DS850 • One DS950 Functional Test Report v1.0 Page 9 of 14 Report Number CAF-ESS22001-FTR-01 Election Systems and Software EVS 6.3.0.2 California Certification Functional Test Report Once the polls had been opened, the zero reports were printed and verified on all devices. A hand marked paper ballot and a ExpressVote card, voted and printed on ExpressVote HW2.1, were voted and had their review box filled in. Marked review ballots were scanned in the tabulators where they were rejected and flagged for review. The tester then turned the “Review Box” and verified the ballots were accepted by the tabulator. Test Outcome This election was run as described above, results verified successfully, and testing concluded with no issues. Presidential General Election The Presidential General Election used 11-inch ballots, contained the languages English, Korean, Chinse, and Vietnamese along with California ballot rotation, write- ins and testing of Accessible Equipment. Five Activation Cards were spoiled and then replaced with five Activation Cards printed from the ExpressVote Activation Card Printer and recreated at the ExpressVote HW2.1. This election was imported into the Client/Server Electionware configuration using the import functionality. The SLI representative first created all the media USB devices for the following Election Devices: • One DS200 • One DS300 • One ExpressVote HW2.1 • One DS450 • One DS850 • One DS950 Once the polls had been opened, the zero reports were printed and verified on all devices. A deck of 938 ballots were processed, with a partial deck of 469 ballots processed in the DS200, DS300, and a full deck of 938 ballots processed through the DS450, DS850, and DS950. Additionally, 80 ExpressVote Cards were voted and printed on the ExpressVote HW2.1. These 80 ExpressVote Cards were then tabulated once each, through the DS200 and DS300. The same 80 ExpressVote Cards were then tabulated twice each in the DS450, DS850, DS950. A total of 3,806 hand marked ballots were processed. A total of 640 ExpressVote Card ballots were processed. A Grand total of 4,446 ballots were processed. Functional Test Report v1.0 Page 10 of 14 Report Number CAF-ESS22001-FTR-01 Election Systems and Software EVS 6.3.0.2 California Certification Functional Test Report While creating the ballots on the ExpressVote HW2.1, accessibility and language support were tested by adjusting the display, audio, font size, and contrast and by testing the Jelly switches, Sip/Puff device, and printing device. Once all ballots had been processed and created on their respective devices, the polls were closed on the DS200 and DS300 in accordance with vendor procedures. Results and Audit Logs were then obtained from each scanner and all results media removed. DS200 and DS300 media was then brought into Electionware along with the results from the DS450, DS850, and DS950. Results reports were verified against the expected results and the devices in test were powered off. SOV, and SOVV reports were also created from the brought in results. Test Outcome This election was run as described above, results verified successfully, and testing concluded with no issues. Presidential Primary Election The Presidential Primary Election used 14-inch ballots, California ballot rotation setup, and testing of Accessible Equipment. Five Activation Cards were spoiled and then replaced with five Activation Cards printed from the ExpressVote Activation Card Printer and utilized at the ExpressVote HW2.1. This election did not contain any provisional ballots or write-ins. This election was imported into the Client/Server Electionware configuration using the import functionality. The SLI representative first created all the media USB devices for the following Election Devices: • One DS200 • One DS300 • One ExpressVote HW2.1 • One DS450 • One DS850 • One DS950 Once the polls had been opened, the zero reports were printed and verified on all devices. A deck of 938 ballots were processed, with a half partial deck of 496 hand marked ballots processed in the DS200, DS300, and a full deck of 938 ballots processed in through each of the DS450, DS850, and DS950 devices. While creating the ballots on the ExpressVote HW2.1, the accessibility and language support were tested by adjusting the display, audio, font size, and contrast and by testing the Jelly switches, Sip/Puff device, and printing device. Functional Test Report v1.0 Page 11 of 14 Report Number CAF-ESS22001-FTR-01 Election Systems and Software EVS 6.3.0.2 California Certification Functional Test Report 80 ExpressVote Card ballots were run through once on the DS200 and DS300 then two runs for a total of 160 ExpressVote Cards through the DS450, DS850, DS950. A total of 3,806 hand marked ballots were processed. A total of 640 ExpressVote Card ballots were processed. A Grand total of 4,446 ballots were processed. Once all ballots had been processed and created on their respective devices, the polls were closed on the DS200 and DS300 in accordance with vendor procedures. Results and Audit Logs were then obtained from each scanner and all results media removed. DS200 and DS300 media was then brought into Electionware along with the results from the DS450, DS850, and DS950. Finally, the results reports were verified against the expected results and the devices in test were powered off. SOV, and SOVV reports were also created from the brought in results. Test Outcome This election was run as described above, results verified successfully, and testing concluded with no issues. Special Recall Election The Special Recall Election used 19-inch ballots and activation cards containing the maximum number of candidates and were marked with stray marks of different marking devices, colors, and placements. The languages used were English and Khmer. This election was imported into the Client/Server Electionware configuration using the import functionality. The SLI representative first created all the media USB devices for the following Election Devices: • One DS200 • One DS300 • One DS450 • One DS850 • One DS950 Ten ballots were marked with stray marks, additionally one Khmer ballot was marked yes to the recall question with a vote for the candidate, and another Khmer ballot was marked no to the recall question and a vote for the candidate. A total of 12 ballots were marked. Once the polls had been opened, the zero reports were printed and verified on all devices. All marked ballots were scanned through the devices. A Grand total of 60 ballots were processed. Functional Test Report v1.0 Page 12 of 14 Report Number CAF-ESS22001-FTR-01 Election Systems and Software EVS 6.3.0.2 California Certification Functional Test Report then the polls were closed on the DS200 and DS300 in accordance with vendor procedures. Results and Audit Logs were then obtained from each scanner and all results media removed. All media from the devices used were then brought into Electionware where the results reports were verified against the expected results and the devices in test were powered off. Test Outcome This election was run as described above, results verified successfully, and testing concluded with no issues. Vote Center Election The Vote Center Election used 17-inch ballots and activation cards containing over 5000 precincts, long character name strings, pre-folded ballots, network configuration, California ballot rotation setup, and testing of Accessible Equipment. This election did not contain any provisional ballots or write-ins. This election was imported into the Client/Server Electionware configuration using the import functionality. The SLI representative first created all the media USB devices for the following Election Devices: • Two DS200 • Two DS300 • Two ExpressVote HW2.1 • One DS450 • One DS850 • One DS950 A ping test was then conducted on the DS450, DS850 and DS950 to verify that the network functionality operated as expected. A User.xml file was created for Cerberus FTP for the networking of results and then polls were opened. Once the polls had been opened, the zero reports were printed and verified on all devices. Three-hundred and twenty-four ballots from the folded test deck were tabulated through each available device: DS200, DS300, DS450, DS850 and DS950 yielding a total of 1620 ballots. Additionally, 50 ExpressVote Cards were tabulated once each on both the DS200 and DS300, those 50 ExpressVote Cards were then tabulated twice each on the DS450, DS850 and DS950. In the end this yielded a total of 400 ExpressVote Cards that were tabulated. A total of 1,620 hand marked ballots were processed. A total of 400 ExpressVote Card ballots were processed. Functional Test Report v1.0 Page 13 of 14 Report Number CAF-ESS22001-FTR-01 Election Systems and Software EVS 6.3.0.2 California Certification Functional Test Report A Grand total of 2,020 ballots were processed. While creating the ballots on the ExpressVote HW2.1, the accessibility and language support were tested by adjusting the display, audio, font size, and contrast and by testing the Jelly switches, Sip/Puff device, and printing device. Once all ballots had been processed and created on their respective devices, the polls were closed on the DS200 and DS300 in accordance with vendor procedures. Results and Audit Logs were then obtained from each scanner and all results media removed. This media was then brought into Electionware along with the networked results from the DS450 and DS850. Finally, the results reports were verified against the expected results and the devices in test were powered off. Test Outcome This election was run as described above, results verified successfully, and testing concluded with no issues. Regression Testing While regression testing was planned, it was not performed as all testing was completed without issue in the previous four test phases. Evaluation of Testing The above testing was conducted using the executables created in the Trusted Build, in association with the appropriate hardware versions as declared during the certification project for the ES&S EVS 6.3.0.2 voting system. As directed by the California Secretary of State, this Functional Test Report does not include any recommendation as to whether or not the system should be approved. End of Functional Test Report Functional Test Report v1.0 Page 14 of 14 Report Number CAF-ESS22001-FTR-01