Consultant's Volume Report

Election Systems & Software EVS 6.3.0.2 Voting System Volume Test Report for California Secretary of State CAF-ESS22007-VTR-01 Vendor Name Election Systems and Software (ES&S) Vendor System EVS 6.3.0.2 Prepared by: SLI ComplianceSM 4720 Independence St. Wheat Ridge, CO 80033 303-422-1566 www.SLICompliance.com Accredited by the Election Assistance Commission (EAC) for VSTL status. Election Systems and Software EVS 6.3.0.2 California Certification Volume Test Report v1.0 Copyright ©2022 SLI Compliance SM, a Division of Gaming Laboratories International, LLC Revision History Date Release Author Revisions Dec. 5, 2022 1.0 M. Santos Initial Release Disclaimer The Certification Test results reported herein must not be used by the client to claim product certification, approval, or endorsement by NVLAP, NIST, or any agency of the Federal Government. Results herein relate only to the items tested. Trademarks • SLI is a registered trademark of SLI Compliance, a Division of Gaming Laboratories International, LLC. • All other products and company names are used for identification purposes only and may be trademarks of their respective owners. The tests referenced in this document were performed in a controlled environment using specific systems and data sets; results are related to the specific items tested. Actual results in other environments may vary. Opinions and Interpretations There are no SLI opinions or interpretations included in this report. Volume Test Report v1.0 Page 2 of 10 Report Number CAF-ESS22007-VTR-01 Election Systems and Software EVS 6.3.0.2 California Certification Volume Test Report v1.0 TABLE OF CONTENTS INTRODUCTION ................................................................................................................................... 4 REFERENCES....................................................................................................................................... 4 SYSTEM OVERVIEW ........................................................................................................................... 4 VOTING SYSTEM SCOPE ....................................................................................................................... 4 CERTIFICATION TEST RESULTS SUMMARY ................................................................................... 4 VOLUME TESTING SUMMARY ................................................................................................................ 4 Volume Test ................................................................................................................................... 4 EVALUATION OF TESTING ............................................................................................................... 9910 Volume Test Report v1.0 Page 3 of 10 Report Number CAF-ESS22007-VTR-01 Election Systems and Software EVS 6.3.0.2 California Certification Volume Test Report v1.0 Introduction SLI Compliance is submitting this test report as a summary of the certification testing efforts for the Election Systems & Software EVS 6.3.0.2 (ES&S EVS 6.3.0.2) voting system. The purpose of this document is to provide an overview of the certification testing effort and the findings of the testing effort for the ES&S EVS 6.3.0.2 voting system. This effort included volume testing of the ES&S EVS 6.3.0.2 voting system’s DS300 optical ballot counter, ExpressVote HW2.1 ballot marking device in a kiosk, and ExpressVote HW2.1 ballot marking device on a table, components. References California Voting System Standards (CVSS) System Overview Voting System Scope This section provides a description of the ES&S EVS 6.3.0.2 voting system components: • DS300 optical ballot counter • ExpressVote HW2.1 ballot marking device (Kiosk and Table) The ES&S EVS 6.3.0.2 DS300 system employs a precinct-level optical scan ballot counter (tabulator) in conjunction with an external ballot box. This tabulator is designed to scan paper ballots, interpret voting marks, and deposit the ballots into the secure ballot box. The ES&S EVS 6.3.0.2 ExpressVote HW2.1 ballot marking devices are solutions to assist voters with marking ballots. These ballots are later scanned and tabulated by the DS300, DS850, and DS450 optical ballot counters. Certification Test Results Summary Volume Testing Summary The tests run on the ES&S EVS 6.3.0.2 voting system included a Volume Test. Volume Test An election was run utilizing the following devices: Volume Test Report v1.0 Page 4 of 10 Report Number CAF-ESS22007-VTR-01 Election Systems and Software EVS 6.3.0.2 California Certification Volume Test Report v1.0 • 15 DS300 optical ballot counters • 10 ExpressVote HW2.1 ballot marking devices, in kiosk • 10 ExpressVote HW2.1 ballot marking devices, on table • 15 plastic, stackable ballot boxes • 2,000 ExpressVote ballots • 17,700 paper ballots The election definition was the same general election used in functional testing. • Two decks of 590 single sheet paper ballots were scanned on each DS300 device twice, for a total of 2,360 ballots tabulated. The ballots were printed in English and Spanish. • 50 to 130 blank ExpressVote Cards were marked and printed using each ExpressVote HW2.1 ballot marking device. • ES&S supplied sufficient identical test decks of ballots, pre-marked in a pattern approved by the Secretary of State. • Additional blank ballots were provided to replace any ballots that were damaged during testing. • Sufficient quantities of blank ballots or ballot paper were supplied to mark using the ExpressVote HW2.1 ballot marking devices. DS300 Optical Ballot Counter Testing The following steps for testing the DS300 optical ballot counters were completed with results as noted: 1. Polls were opened for the DS300 optical ballot counter in accordance with California Use Procedures. Devices were brought to a state where ballots could be tabulated. o Zero Reports were printed 2. One DS300 optical ballot counter was removed from AC power for two hours to test the backup battery. o No device issues were encountered. 3. Once the time expired, the DS300 optical ballot counter was reconnected to AC power. 4. For each DS300 optical ballot counter, two decks of 590 paper ballots were scanned and tabulated. o One DS300 optical ballot counter experienced multiple errors related to ballot jams. The ballot jams were caused by the end of the ballot being slightly folded up, which didn’t allow the ballot to be fed through the paper path. The vendor documented instructions to resolve such issues in the vendor provided use procedures. Volume Test Report v1.0 Page 5 of 10 Report Number CAF-ESS22007-VTR-01 Election Systems and Software EVS 6.3.0.2 California Certification Volume Test Report v1.0 5. After the first deck of 590 paper ballots were scanned and tabulated, the ballot box was opened, and the paper ballots were removed from the ballot bin to be scanned again. The totals displayed on screen were then compared to the expected results. o The total displayed results matched the expected results. 6. The first deck of 590 paper ballots were then scanned and tabulated a second time. o One DS300 optical ballot counter experienced multiple ballots not falling into the internal bin. This was due to the lids of the internal blue ballot bin being tilted inward slightly and blocking the path. This was caused by human error during the removal of ballots process. The lids were properly positioned per the vendor instructions and the ballot issues were resolved. 7. After the deck of 590 paper ballots were cast a second time, the ballot box was opened, and the paper ballots were removed from the ballot bin. The displayed totals were compared with the expected results. o The total displayed results matched the expected results. 8. For each DS300 optical ballot counter, a second test deck of 590 paper ballots were scanned and tabulated. The displayed totals were compared with the expected results. o The total displayed results matched the expected results. 9. The second test deck was removed then scanned and tabulated a second time. The displayed totals were compared with the expected results. o The total displayed results matched the expected results. 10. The polls were closed, and a Results Report was printed and compared with the expected results. o The printed results matched the expected results. o One DS300 optical ballot counter encountered issues with its thermal printer device. It was expected that the thermal printer would begin printing the results report after two minutes; however, the device would not print the results report even after fifteen minutes. The thermal printer door was opened to readjust the paper and rolling wheel mechanism. However, even after readjusting the thermal paper and rolling wheel mechanism, the results report would still not print. After consulting with the vendor, it was deemed to be a printer error. The results were available on the device and were able to be verified against the expected totals. 11. Once the results had been verified with no issues, the DS300 optical ballot counters were shut down to prepare for transportation by ES&S personnel. Volume Test Report v1.0 Page 6 of 10 Report Number CAF-ESS22007-VTR-01 Election Systems and Software EVS 6.3.0.2 California Certification Volume Test Report v1.0 ExpressVote HW2.1 Ballot Marking Devices in Kiosk Testing The following steps for testing the ExpressVote HW2.1 ballot marking devices in kiosk were completed with results as noted: 1. Polls were opened for the ExpressVote HW2.1 ballot marking devices in kiosks in accordance with California Use Procedures. o Devices were brought to a state where ballots could be marked and printed. 2. One ExpressVote HW2.1 ballot marking device in a kiosk was removed from AC power for two hours, to test the backup battery. o No device issues were encountered. 3. Once the time expired, the ExpressVote HW2.1 ballot marking device in a kiosk was reconnected to AC power. 4. For each ExpressVote HW2.1 ballot marking devices in a kiosk, 50 to 130 ExpressVote cards were marked and printed. 5. All devices printed their cards in the expected manner with two exceptions. o One issue was encountered and was resolved. A single ExpressVote HW2.1 ballot marking device in a kiosk experienced a paper jam issue. The paper jam was caused by pulling on the voter card before being prompted by the machine to do so. This caused the voter card to get caught on the rubber wheels in the paper path. The issue was resolved by pressing the red error sign on the screen, then slowly pulling out the voter card from where the paper path ejects the voter card. The vendor documented instructions to resolve such issues are available in the vendor user documentation. o Another issue occurred with a different ExpressVote HW2.1 ballot marking device in a kiosk when it displayed a “Printed Card Not Read” message after printing a card. Nothing was visibly wrong with the card. On screen instructions were followed. The card was fed back into the same device. This time the card could be read and displayed details were accurate. The issue could not be duplicated on the same device. ExpressVote HW2.1 Ballot Marking Device on Table Testing The following steps for testing the ExpressVote HW2.1 ballot marking devices on tables were completed with results as noted: 1. Polls were opened for the ExpressVote HW2.1 ballot marking devices on tables in accordance with California Use Procedures. Volume Test Report v1.0 Page 7 of 10 Report Number CAF-ESS22007-VTR-01 Election Systems and Software EVS 6.3.0.2 California Certification Volume Test Report v1.0 o Devices were brought to a state where ballots could be marked and printed 2. One ExpressVote HW2.1 ballot marking device on a table was removed from AC power for two hours, to test the backup battery. o No device issues were encountered. 3. Once the time expired, the ExpressVote HW2.1 ballot marking device on a table was reconnected to AC power. 4. For each ExpressVote HW2.1 ballot marking device on a table, 50 to 130 ExpressVote cards were marked and printed. 5. All devices printed their cards in the expected manner with two exceptions. o One issue was encountered and was resolved. A single ExpressVote HW2.1 ballot marking device on a table experienced a paper jam issue. The paper jam was caused by inserting the voter card slightly skewed into the paper path. The issue was resolved by pressing the red error sign on the screen then opening the paper path covering using a special key which the vendor provided. After the paper path covering was opened, the voter card was clearly visible and by turning the rubber wheels located on the inside, the voter card was able to be ejected. The vendor documented instructions to resolve such issues are available in the vendor user documentation. o Another issue occurred when the ExpressVote HW2.1 ballot marking device on a table displayed a “Printed Card Not Read” message after printing a card. Nothing was visibly wrong with the card. On screen instructions were followed. The card was fed back into the same device. This time the card could be read and displayed details were accurate. The issue was not able to be duplicated on the same device. Issues Identified The following issues were noted during testing: • One DS300 optical ballot counter experienced multiple errors related to ballot jams. The ballot jams were caused by ballots where the ends were slightly folded due to slight mishandling while removing the previously scanned deck. Some ballots were mishandled and the ends of the ballots curled up at the ends as a result. The vendor documented instructions to resolve such issues are available in the vendor user documentation. • One DS300 optical ballot counter experienced multiple ballots not falling into the internal bin. This was due to the lids of the internal blue ballot bin being tilted inward slightly and blocking the path. This was caused by human error Volume Test Report v1.0 Page 8 of 10 Report Number CAF-ESS22007-VTR-01 Election Systems and Software EVS 6.3.0.2 California Certification Volume Test Report v1.0 during the removal of ballots process. The lids were properly positioned per the vendor instructions and the ballot issues were resolved. One DS300 optical ballot counter encountered issues with its thermal printer. It was expected that the thermal printer would begin printing the results report after two minutes; however, the device would not print the results report even after fifteen minutes. The thermal printer door was opened to readjust the paper and rolling wheel mechanism. Even after readjusting the thermal paper and rolling wheel mechanism, the results report would still not print. After consulting with the vendor, it was deemed to be a printer error. The results were available on the device and were able to be verified against the expected totals. • One ExpressVote HW2.1 ballot marking device in a kiosk experienced a paper jam issue. The paper jam was caused by pulling on the voter card before prompted by the machine to do so. This caused the voter card to get caught on the rubber wheels in the paper path. The issue was resolved by pressing the red error sign on the screen then slowly pulling out the voter card from where the paper path ejects the voter card. The vendor documented instructions to resolve such issues are available in the vendor user documentation. • One ExpressVote HW2.1 ballot marking device on a table experienced a paper jam issue. The paper jam was caused by inserting the voter card slightly skewed into the paper path. The issue was resolved by pressing the red error sign on the screen then opening the paper path covering using a special key which the vendor provided. After the paper path covering was opened, the voter card was clearly visible and by turning the rubber wheels located on the inside, the voter card was able to be ejected. The vendor documented instructions to resolve such issues are available in the vendor user documentation. • One issue occurred twice, once on the kiosk and once on the table version of the ExpressVote HW2.1 ballot marking device. A ‘Printed Card Not Read.’ message displayed after printing a card. Nothing was visibly wrong with the card. On screen instructions were followed. The card was fed back into the same device and the card could be read and displayed details were accurate. The issue was not able to be duplicated on either of the two devices. Evaluation of Testing The above testing was conducted using the executables created in the Trusted Build, in association with the appropriate hardware versions as declared during the certification project for the ES&S EVS 6.3.0.2 voting system. Volume Test Report v1.0 Page 9 of 10 Report Number CAF-ESS22007-VTR-01 Election Systems and Software EVS 6.3.0.2 California Certification Volume Test Report v1.0 As directed by the California Secretary of State, this volume testing report does not include any recommendation as to whether or not the system should be approved. End of Volume Test Report Volume Test Report v1.0 Page 10 of 10 Report Number CAF-ESS22007-VTR-01