Consultant's Volume Report

ES&S EVS 6.0.4.2 Volume Test Report ESS-18S52059-VTR-01 Vendor Name Election Systems and Software (ES&S) Vendor System EVS 6.0.4.2 Prepared by: SLI ComplianceSM 4720 Independence St. Wheat Ridge, CO 80033 303-422-1566 www.SLICompliance.com Accredited by the Election Assistance Commission (EAC) for VSTL status. Election Systems and Software EVS 6.0.4.2 California Certification Volume Test Report Copyright 2019 SLI Compliance SM, a Division of Gaming Laboratories International, LLC Revision History Date Release Author Revisions 8/5/2019 1.0 D. George Initial Release Updates based on 8/12/2019 1.1 D. George comments 8/16/2019 1.2 D. George Additional edits 8/20/2019 1.3 D. George Additional edits 8/21/2019 1.4 D. George Final edits Disclaimer The Certification Test results reported herein must not be used by the client to claim product certification, approval, or endorsement by NVLAP, NIST, or any agency of the Federal Government. Results herein relate only to the items tested. Trademarks SLI is a registered trademark of SLI Compliance, a Division of Gaming Laboratories International, LLC. All other products and company names are used for identification purposes only and may be trademarks of their respective owners. The tests referenced in this document were performed in a controlled environment using specific systems and data sets; results are related to the specific items tested. Actual results in other environments may vary. Opinions and Interpretations There are no SLI opinions or interpretations included in this report. Volume Test Report v1.4 Page 2 of 7 Report Number ESS-18S52059-VTR-01 Election Systems and Software EVS 6.0.4.2 California Certification Volume Test Report TABLE OF CONTENTS INTRODUCTION ........................................................................................................................................... 4 REFERENCES............................................................................................................................................... 4 SYSTEM OVERVIEW ................................................................................................................................... 4 VOTING SYSTEM SCOPE ............................................................................................................................... 4 CERTIFICATION TEST RESULTS SUMMARY ........................................................................................... 5 VOLUME TESTING SUMMARY ........................................................................................................................ 5 Volume Test ........................................................................................................................................... 5 EVALUATION OF TESTING ............................................................................................................................. 7 Volume Test Report v1.4 Page 3 of 7 Report Number ESS-18S52059-VTR-01 Election Systems and Software EVS 6.0.4.2 California Certification Volume Test Report Introduction SLI Compliance is submitting this test report as a summary of the certification testing efforts for the ES&S EVS 6.0.4.2 voting system. The purpose of this document is to provide an overview of the certification testing effort and the findings of the testing effort for the ES&S EVS 6.0.4.2 voting system. This effort included volume testing of the ES&S EVS 6.0.4.2 voting system’s DS200, ExpressVote, and ExpressVote XL components. References California Voting System Standards (CVSS) System Overview Voting System Scope This section provides a description of the ES&S EVS 6.0.4.2 voting system components: DS200 HW1.3 / 2.20.0.0 ExpressVote HW 2.1/ 2.5.0.0 ExpressVote XL / 1.2.0.0 The EVS 6.0.4.2 DS200 system employs a precinct-level optical scan ballot counter (tabulator) in conjunction with an external ballot box. This tabulator is designed to scan paper ballots, interpret voting marks, and deposit the ballots into the secure ballot box. The EVS 6.0.4.2 ExpressVote and ExpressVote XL ballot marking platforms are solutions used for creation of paper ballots. These ballots are later scanned and tabulated by the DS200, DS850 and DS450 optical ballot counters. Volume Test Report v1.4 Page 4 of 7 Report Number ESS-18S52059-VTR-01 Election Systems and Software EVS 6.0.4.2 California Certification Volume Test Report Certification Test Results Summary Volume Testing Summary The tests run on the ES&S EVS 6.0.4.2 voting system included a Volume test. Volume Test An election was run utilizing the following devices: 25 DS200 devices 25 ExpressVote devices 30 ExpressVote XL devices 25 plastic, stackable ballot boxes 4,250 ExpressVote ballots 12,500 paper ballots The election definition was the same general election used in functional testing. 500 single sheet paper ballots were scanned on each DS200 device twice, for a total of one thousand ballots tabulated. The ballots were printed in all supported languages. 50 blank ExpressVote Cards were marked and printed on each ExpressVote device. 100 ExpressVote Cards were marked and printed on each ExpressVote XL device. ES&S supplied sufficient identical test decks of ballots, pre-marked in a pattern approved by the Secretary of State. Additional blank ballots were provided to replace any ballots that were damaged during testing. Sufficient quantities of blank ballots or ballot paper were supplied to mark on the ExpressVote and ExpressVote XL devices. DS200 Testing The following steps for testing the DS200 devices were completed with results as noted: 1. Polls were opened for the DS200 devices in accordance with California Use Procedures. Devices were brought to a state where ballots could be tabulated. o Zero Reports were printed 2. One DS200 devices were removed from AC power for two hours, to test the backup batteries. Volume Test Report v1.4 Page 5 of 7 Report Number ESS-18S52059-VTR-01 Election Systems and Software EVS 6.0.4.2 California Certification Volume Test Report o No devices failed. 3. Once the time expired, the DS200 devices were reconnected to AC power. 4. For each DS200 device, 500 paper ballots were scanned and tabulated. o One issue was encountered and was resolved. The DS200 experienced multiple errors related to ballot jams encountered by a single DS200 device. The ballot jams were caused by rails that connect the device and ballot box having been incorrectly installed. The rails were replaced, installed properly per the vendor instructions and the ballot jam issues were resolved. The vendor documented instructions to resolve such issues in the vendor provided use procedures. 5. Once the first 500 paper ballots were scanned and tabulated, the polls were closed and a Results Report was printed. The totals were then compared to the expected results. o The printed results matched the expected results. 6. The paper ballots were removed from the ballot bin. The DS200 devices were zeroed out and re-opened. The 500 paper ballots were then scanned and tabulated a second time. o There were no issues. 7. The polls were closed once again and a Results Report was printed and compared with the expected results. o The printed results matched the expected results. 8. Once the results had been verified with no issues, the DS200 devices were shut down to prepare for transportation by ES&S personnel. ExpressVote Testing The following steps for testing the ExpressVote devices were completed with results as noted: 1. Polls were opened for the ExpressVote devices in accordance with California Use Procedures. o Devices were brought to a state where ballots could marked and printed 2. One ExpressVote devices were removed from AC power for two hours, to test the backup batteries. o No devices failed. 3. Once the time expired, the ExpressVote devices were reconnected to AC power. 4. For each ExpressVote device, 50 ExpressVote cards were marked and printed. 5. All devices printed their cards in the expected manner, no errors occurred. Volume Test Report v1.4 Page 6 of 7 Report Number ESS-18S52059-VTR-01 Election Systems and Software EVS 6.0.4.2 California Certification Volume Test Report ExpressVote XL Testing The following steps for testing the ExpressVote XL devices were completed with results as noted: 1. Polls were opened for the ExpressVote XL devices in accordance with California Use Procedures. o Devices were brought to a state where ballots could marked and printed 2. One ExpressVote XL devices were removed from AC power for 2 hours, to test the backup batteries. o No devices failed. 3. Once the time expired, the ExpressVote XL devices were reconnected to AC power. 4. For each ExpressVote XL device, 100 ExpressVote cards were marked and printed. 5. All devices printed their cards in the expected manner, no errors occurred. Issues Identified One issue was encountered and was resolved. The DS200 experienced multiple errors related to ballot jams encountered by a single DS200 device. The ballot jams were caused by rails that connect the device and ballot box having been incorrectly installed. The rails were replaced, installed properly per the vendor instructions and the ballot jam issues were resolved. The vendor documented instructions to resolve such issues in the vendor user documentation. The rails which hold the DS200 tabulation device to the carry case were not only incorrectly installed but were also of an older model. The resolution was to swap out the installed rails for the correct rails harvested from a machine not utilized in testing, place the tabulation machine back into place and close the carry case allowing for continuation of feeding ballots. Evaluation of Testing The above testing was conducted using the executables created in the Trusted Build, in association with the appropriate hardware versions as declared during the certification project for the ES&S EVS 6.0.4.2 voting system. As directed by the California Secretary of State, this volume testing report does not include any recommendation as to whether or not the system should be approved. End of Volume Test Report Volume Test Report v1.4 Page 7 of 7 Report Number ESS-18S52059-VTR-01