Consultant's Functional Report

ES&S EVS 6.0.4.2 Functional Test Report ESS-18S52059-FTR-01 Vendor Name Election Systems and Software (ES&S) Vendor System EVS 6.0.4.2 Prepared by: SLI ComplianceSM 4720 Independence St. Wheat Ridge, CO 80033 303-422-1566 www.SLICompliance.com Accredited by the Election Assistance Commission (EAC) for VSTL status. Election Systems and Software EVS 6.0.4.2 California Certification Functional Test Report Copyright 2019 SLI ComplianceSM, a Division of Gaming Laboratories International, LLC Revision History Date Release Author Revisions D. George Initial Release 7/29/2019 1.0 Updates based on D. George 8/12/2019 1.1 comments D. George Additional edits 8/16/2019 1.2 D. George Additional edits 8/20/2019 1.3 D. George Final edits 8/21/2019 1.4 Disclaimer The Certification Test results reported herein must not be used by the client to claim product certification, approval, or endorsement by NVLAP, NIST, or any agency of the Federal Government. Results herein relate only to the items tested. Trademarks SLI is a registered trademark of SLI Compliance, a Division of Gaming Laboratories International, LLC. All other products and company names are used for identification purposes only and may be trademarks of their respective owners. The tests referenced in this document were performed in a controlled environment using specific systems and data sets and results are related to the specific items tested. Actual results in other environments may vary. Opinions and Interpretations There are no SLI opinions or interpretations included in this report. Functional Test Report v1.4 Page 2 of 12 Report Number ESS-18S52059-FTR-01 Election Systems and Software EVS 6.0.4.2 California Certification Functional Test Report TABLE OF CONTENTS INTRODUCTION ........................................................................................................................................... 4 REFERENCES............................................................................................................................................... 4 SYSTEM OVERVIEW ................................................................................................................................... 4 VOTING SYSTEM SCOPE ............................................................................................................................... 4 CERTIFICATION TEST RESULTS SUMMARY ........................................................................................... 5 FUNCTIONAL TESTING SUMMARY .................................................................................................................. 5 Equipment Tested .................................................................................................................................. 5 Physical Configuration Audit .................................................................................................................. 5 Installation Process/Functional Configuration Audit .............................................................................. 6 Functional Test Phase ........................................................................................................................... 7 Regression Testing .............................................................................................................................. 12 EVALUATION OF TESTING ........................................................................................................................... 12 Functional Test Report v1.4 Page 3 of 12 Report Number ESS-18S52059-FTR-01 Election Systems and Software EVS 6.0.4.2 California Certification Functional Test Report Introduction SLI Compliance is submitting this test report as a summary of the certification testing efforts for the ES&S EVS 6.0.4.2 voting system. The purpose of this document is to provide an overview of the certification testing effort and the resulting findings for the ES&S EVS 6.0.4.2 voting system. This effort included functional testing of the ES&S EVS 6.0.4.2 voting system’s DS200, DS450, DS850, ExpressVote, ExpressVote XL, and Election Management System (EMS) components. References California Voting System Standards (CVSS) System Overview Voting System Scope This section provides a description of the scope of the ES&S EVS 6.0.4.2 voting system components: DS200 HW 1.3/ 2.20.0.0 DS450 HW 1.0/ 3.3.0.0 DS850 HW 1.0/ 3.3.0.0 ExpressVote HW 2.1/ 2.5.0.0 ExpressVote XL 1.2.0.0 Electionware (EMS) 5.3.0.0 The EVS 6.0.4.2 Election Management System (Electionware, PaperBallot) represents a set of N-Tier software applications (EMS, RTR, Adjudication) for pre- voting and post-voting election project activities that are applicable to jurisdictions of various sizes and geo-political complexities. The EVS 6.0.4.2 DS450 and DS850 central, high-speed, optical scan ballot counters (tabulators) and are used for processing absentee ballots (such as vote by mail). This ballot counter unit is based on commercial off the shelf (COTS) hardware coupled with custom-made ballot processing application software. It is used for high-speed, accurate, and reliable centralized scanning and counting of paper ballots. The EVS 6.0.4.2 DS200 system employs a precinct-level optical scan ballot counter (tabulator) in conjunction with an external ballot box. This tabulator is designed to Functional Test Report v1.4 Page 4 of 12 Report Number ESS-18S52059-FTR-01 Election Systems and Software EVS 6.0.4.2 California Certification Functional Test Report scan paper ballots, interpret voting marks, and deposit the ballots into the secure ballot box. The EVS 6.0.4.2 ExpressVote and ExpressVote XL ballot marking platforms are solutions used for creation of paper ballots. These ballots are later scanned and tabulated by the DS200, DS850 and DS450 optical ballot counters. Certification Test Results Summary Functional Testing Summary Functional testing is divided into five phases: Physical Configuration Audit Installation Process Functional Configuration Audit Functional Test Phase Regression Testing (If needed) Equipment Tested The equipment that was tested included: Six DS200 Devices Two DS450 devices Two DS850 devices Two ExpressVote devices Two ExpressVote XL devices Additional Equipment ES&S provided two (2) servers (Dell PowerEdge T430), two (2) test workstations (Dell Optiplex 5050) for use as clients to the servers and two (2) standalone workstations (Dell Latitude 5580) to serve as Election Reporting Manager (ERM) systems. ES&S also provided two (2) DS850s, two (2) DS450s, two (2) ExpressVotes, two (2) ExpressVote XLs, and six (6) DS200 machines. Physical Configuration Audit The physical configuration audit took place from May 6 to May 30. During this phase, an SLI engineer took photos, documented part numbers and serial numbers of each device under test, and updated the Physical Configuration Audit traveler (equipment inventory) accordingly. Functional Test Report v1.4 Page 5 of 12 Report Number ESS-18S52059-FTR-01 Election Systems and Software EVS 6.0.4.2 California Certification Functional Test Report Installation Process/Functional Configuration Audit The installation process and functional configuration audit phase occurred from May 24 to May 30. During the initial process of this phase, all the servers, clients, and standalone stations were wiped clean using the KillDisk’s US DoD 5220 method. Once each device was cleared, and following the vendor provided installation documentation, SLI performed the following actions, depending on the applicable setup: Windows 7 Enterprise was installed on the client and standalone with the server using Windows Server 2008 R2. The Microsoft Windows Using WSUS Offline Update was then run multiple times until a message stating “Any missing update was either black listed or not found. Nothing to do!” was displayed. The SLI employee then installed any missing device drivers onto their respective devices. Windows 7 was activated using the call number listed (Only for the Client and Standalone). Visual C++ was installed. The Engineer then completed the hardening of the bench pre-install by running the SecureSetup_2.1.0.3 program, provided by ES&S. At this stage, all relevant programs were installed, such as Adobe Acrobat, Removable Media Service, ExpressVote Previewer, Event Log Service, Electionware, the Report printer driver, Symantec Endpoint Protection, and the updates for Symantec Endpoint Protection. Only Electionware, Symantec Endpoint Protection, and the updates for Symantec Endpoint Protection were installed on the server. Additionally, the Cerberus FTP Server program was installed on the server. The Post Install script was run using the SecureSetup_2.1.0.3 program. At this point, the trusted hash files were taken using the document and scripts provided by ES&S. The server, clients, and standalone were then hardened and bit locked. Test Outcome During the initial phase of testing, attempting to secure the system using bit locker, the system on the server did not operate as expected. Whenever the server began the process for bit locking, the device would need to be restarted. Once the server restarted, an error would occur. This issue was later remedied by attempting the bit lock process a second time after the initial failure. The second attempt at bit locking the system completed without any errors. During the process of setting up the server, client, and standalone, images were taken at key points, such as after the Windows Server Update Services (WSUS) Functional Test Report v1.4 Page 6 of 12 Report Number ESS-18S52059-FTR-01 Election Systems and Software EVS 6.0.4.2 California Certification Functional Test Report offline update had completed and before the pre-install hardening; the golden images were taken at the end of the hardening process but before the hash files were taken. These golden images were then used to initialize the second server, client, and standalone. The DS200, DS450, DS850, ExpressVote, and ExpressVote XL were updated with their respective trusted builds. The DS200, DS450, DS850, and ExpressVote XL had their flash cards removed and the trusted build was installed using these flash cards. The ExpessVote was updated using a USB flash drive. After the trusted build was burned onto their flash drives, the hash files were taken to be used as the golden hash files for each device. Functional Test Phase Functional Testing began on May 31 and was completed on June 4. This phase simulated an election process from start to finish. The following seven different functions were tested: Vote Center Election Ranked Choice Vote (RCV) Election Primary Election General Election Recall Election (Restore) Recall Election (Build) A Mock Risk-Limiting Audit (RLA) was also performed Vote Center Election The Vote Center Election used 17-inch ballots and activation cards containing over 5000 precincts, long character name strings, pre-folded ballots, network configuration, and the California ballot rotation setup. This election did not contain any provisional ballots or write-ins. This election was created by ES&S and was imported into the Client/Server Electionware configuration using the import functionality. The SLI representative first created all the media USB devices to insert the election into two DS200s, one DS450, one DS850, one ExpressVote, and one ExpressVote XL devices. A ping test was then conducted on the DS850 to verify that the network functionality operated as expected. The calibration of each device was then verified, and the polls opened. Once the polls had been opened, the zero reports were printed and verified on all devices. A total of 324 ballots from the folded test deck were processed through the DS200, DS450, and DS850. With the test engineer creating 50 ballots each on the ExpressVote and ExpressVote XL. These ballots were then processed through the DS200, DS450, and DS850 for a total of 100 activation cards on each device. Functional Test Report v1.4 Page 7 of 12 Report Number ESS-18S52059-FTR-01 Election Systems and Software EVS 6.0.4.2 California Certification Functional Test Report While creating the ballots on the ExpressVote and ExpressVote XL, the accessibility and language support were tested by adjusting the display, audio, font size, and contrast and by testing the Jelly switches, Sip/Puff device, and printing device. Once all ballots had been processed and created on their respective devices, the polls were closed in accordance with vendor procedures. Results and Audit Logs were then obtained from each scanner and all results media removed. This media was then brought into Electionware along with the networked results from the DS450 and DS850. Finally, the results reports were verified against the expected results and the devices in test were powered off. Test Outcome This election was run as described above, results verified successfully, and concluded with no issues. RCV Election The RCV Election used 17-inch ballots and activation cards containing four different languages: English, Tagalog, Thai, and Ilocano. This election was created by ES&S and was imported into the standalone Electionware configuration using the import function. The SLI representative first created all the media USB devices to insert the election into two DS200, one DS450, one DS850, one ExpressVote, and one ExpressVote XL devices. The calibration of each device was then verified, and the polls opened. Once the polls had been opened, the zero reports were printed and verified on all devices. The ExpressVote and ExpressVote XL were each used to create a single activation card. While creating the ballots on the ExpressVote and ExpressVote XL, the accessibility and language support were tested by adjusting the display, audio, font size, and contrast, and by testing the Jelly switches, Sip/Puff device, and printing device. Fifty ballots and two activation cards were processed through each DS200 device. The DS450 and DS850 processed 100 Ballots and two activation cards. Once all ballots had been processed and created on their respective devices, the polls were closed in accordance with vendor procedures. Results and Audit Logs were then obtained from each scanner and all results media removed. This media was then brought into Electionware. Finally, the results reports were verified against the expected results and the devices in test were powered off. Additionally, a CVR spreadsheet was generated, uploaded into ExpressRunOff, and CVR reports generated to verify the results. Functional Test Report v1.4 Page 8 of 12 Report Number ESS-18S52059-FTR-01 Election Systems and Software EVS 6.0.4.2 California Certification Functional Test Report Test Outcome: This election was run as described above, results verified successfully, and concluded with no issues. Primary Election The Primary Election used 14-inch ballots and activation cards containing three different languages: English, Spanish, and Chinese. This election also utilized the two ballots in a single election, the California ballot rotation, and included write-ins. This election was created by ES&S and was imported into the Client/Server Electionware configuration using the import function. The SLI representative first created all the media USB devices to insert the election into two DS200, one DS450, one DS850, one ExpressVote, and one ExpressVote XL devices. The calibration of each device was then verified, and the polls opened. Once the polls had been opened, the zero reports were printed and verified on all devices. The ExpressVote and ExpressVote XL were each used to create 70 activation cards. While creating the ballots on the ExpressVote and ExpressVote XL, the accessibility and language support were tested by adjusting the display, audio, font size, and contrast and by testing the Jelly switches, Sip/Puff device, and printing device. Each DS200 device processed 405 ballots and 70 activation cards. The DS450 and DS850 processed a total of 810 Ballots and 140 activation cards. Once all ballots had been processed and created on their respective devices, the polls were closed in accordance with vendor procedures. Results and Audit Logs were then obtained from each scanner and all results media removed. This media was then brought into Electionware. Finally, canvass reconciliation was performed for write-ins and the results reports were verified against the expected results and the devices in test were powered off. Test Outcome: This election was run as described above, results verified successfully, and concluded with no issues. General Election The General Election used 11-inch ballots and activation cards containing four different languages: English, Korean, Chinese, and Vietnamese. This election also utilized the ENR files, California ballot rotation, and included write-ins. This election was created by ES&S and was imported into the Client/Server Electionware configuration using the import function. The SLI representative first created all the media USB devices to insert the election into two DS200, one DS450, one DS850, one ExpressVote, and one Functional Test Report v1.4 Page 9 of 12 Report Number ESS-18S52059-FTR-01 Election Systems and Software EVS 6.0.4.2 California Certification Functional Test Report ExpressVote XL devices. The calibration of each device was then verified, and the polls opened. Once the polls had been opened, the zero reports were printed and verified on all devices. The ExpressVote and ExpressVote XL were each used to create ten activation cards. While creating these ballots, the accessibility and language support were tested by adjusting the display, audio, font size, and contrast and by testing the Jelly switches, Sip/Puff device, and printing device. Each DS200 device processed 75 ballots and 20 activation cards. The DS450 and DS850 processed a total of 150 ballots and 20 activation cards. Once all ballots had been processed and created on their respective devices, the polls were closed in accordance with vendor procedures. Results and Audit Logs were then obtained from each scanner and all results media removed. This media was then brought into Electionware. Finally, the results reports were verified against the expected results and the devices in test were powered off. Additionally, the California Election Night Auto-reporting per vendor provided procedures was tested and verified. Test Outcome: This election was run as described above, results verified successfully, and concluded with no issues. Recall Election (Restore) The Recall Election (Restore) used 19-inch ballots and activation cards containing four different languages: English, Khmer, Japanese, and Hindi. This election also utilized the California ballot rotation and Electronic Adjudication. This election was created by ES&S and was imported into the Standalone Electionware configuration using the import function. The SLI representative first created all the media USB devices to insert the election into two DS200, one DS450, one DS850, one ExpressVote, and one ExpressVote XL devices. The calibration of each device was then verified, and the polls opened. Once the polls had been opened, the zero reports were printed and verified on all devices. The ExpressVote and ExpressVote XL were each used to create a single activation card. While creating the ballots on the ExpressVote and ExpressVote XL, the accessibility and language support were tested by adjusting the display, audio, font size, and contrast and by testing the Jelly switches, Sip/Puff device, and printing device. These two activation cards were then scanned through the DS200, DS450, and DS850. Once all ballots had been processed and created on their respective devices, the polls were closed in accordance with vendor procedures. Results and Audit Logs were then obtained from each scanner and all results media removed. This media was then brought into Electionware. Once the results were brought into Functional Test Report v1.4 Page 10 of 12 Report Number ESS-18S52059-FTR-01 Election Systems and Software EVS 6.0.4.2 California Certification Functional Test Report Electionware, the ballot adjudication process was verified to be operational. Finally, the results reports were verified against the expected results, the devices in test were powered off. Test Outcome: This election was run as described above, results verified successfully, and concluded with no issues. Recall Election (Build) The Recall Election (Build) used 19-inch ballots and activation cards containing four different languages: English, Khmer, Japanese, and Hindi. This election also utilized the California ballot rotation, Marginal Marks Test, and hand marked ballots. This election was created from scratch by an SLI employee on the Standalone Electionware configuration. The SLI representative first created all the media USB devices to insert the election into two DS200, one DS450, one DS850, one ExpressVote, and one ExpressVote XL devices. The calibration of each device was then verified, and the polls opened. Once the polls had been opened, the zero reports were printed and verified on all devices. The ExpressVote and ExpressVote XL were each used to create a single activation card. While creating the ballots on the ExpressVote and ExpressVote XL, the accessibility and language support were tested by adjusting the display, audio, font size, and contrast and by testing the Jelly switches, Sip/Puff device, and printing device. The SLI representative then created a blank ballot and hand marked each selection using increasingly marginal marks, using a variety of pens, pencils, and highlighters of various colors. The two activation cards and the hand marked ballot were then scanned through the DS200, DS450, and DS850. This test also included a test of the tabulator’s ability to identify marginal marks. All marks produced in accordance with the instructions provided on the ballot were read correctly across all scanners. The polls were closed in accordance with vendor procedures. Results and Audit Logs were then obtained from each scanner and all results media removed. This media was then brought into Electionware. From here reconciliation was performed for the hand marked ballots as well as write-in votes. Finally, the results reports were verified against the expected results, the devices in test were powered off. Test Outcome: This election was run as described above, results verified successfully, and concluded with no issues. Mock RLA Using the Recall Election (Build) the SLI and California representative audited the data using professor Stark’s “Tools for Ballot-Polling Risk-Limiting Election Audit.” Functional Test Report v1.4 Page 11 of 12 Report Number ESS-18S52059-FTR-01 Election Systems and Software EVS 6.0.4.2 California Certification Functional Test Report This process had two representatives auditing a small sample size of the total batch of ballots to verify that everything processed accurately. Test Outcome: This test was executed as described above, results verified successfully, and concluded with no issues. Regression Testing While regression testing was planned, it was not performed as all testing was completed in the previous four test phases. Evaluation of Testing The above testing was conducted using the executables created in the Trusted Build, in association with the appropriate hardware versions as declared during the certification project for the ES&S EVS 6.0.4.2 voting system. As directed by the California Secretary of State, the Functional Test Report does not include any recommendation as to whether or not the system should be approved. End of Volume Test Report Functional Test Report v1.4 Page 12 of 12 Report Number ESS-18S52059-FTR-01