Consultants' Report

California Secretary of State Consultant’s Report on: HART INTERCIVIC SYSTEM 6.2.1: Ballot Now3.3, Release 3.3.11 BOSS 4.3, Release 4.3.13 Rally 2.3, Release 2.3.7 Tally 4.3, Release 4.3.10 SERVO 4.2, Release 4.2.10 eScan, version 1.3.14 JBC, version 4.3.0 eSlate/DAU, version 4.2.13 VBO, version 1.8.3 eCM Manager 1.1, Release 1.1.7 Prepared September 5, 2006 By Paul W. Craft Scope of Work and Reporting This report is prepared as a supplement and attachment to the “Staff Review and Analysis” (SOS Report) as prepared by the California Secretary of State’s Office of Voting Systems Technology Assessment (OVSTA). As a part of our consulting work product, we were asked to provide assistance to OVSTA in planning and conducting voting system tests. The majority of the findings may be reported in the SOS Report. This report will be limited to a description of the system tested, the testing tasks which we conducted, events we observed and findings. Our expertise is in methodologies for examining computerized voting systems, analysis of systems operation, developing measurements of system compliance with established criteria, identification and analysis of system anomalies, and collecting evidence of system characteristics and compliance. We are not attorneys and do not offer legal advice. We have assisted the California Secretary of State in the collection of facts and evidence that he will use in reaching certification decisions. However, to advise him on the determination of whether the system complies with California’s certification requirements would require an interpretation of law. Accordingly, we do not provide recommendations or any opinion Consultant’s report on Hart Voting System Version 6.2.1 September 5, 2006 Page 2 as to whether the system can be certified. Recommendations to the Secretary for or against certification are within the duties of the OVSTA and are included in their report. The work that we have performed and our findings are strictly limited to the specific hardware elements and specific software elements tested during the examination. The results described in this report should be reliable and repeatable for those specific items. The decision to apply those results to decisions about other items is solely at the discretion of and risk of the Secretary of State and the purchasers of systems. Although the content of this report can be used as part of a baseline for reaching conclusions about compliance of other items, users of this report who wish to arrive at such conclusions about compliance of purchased systems or the compliance of a system in use should conduct appropriate acceptance testing or system validation analysis to support those conclusions. If they do not have a high level of well-founded confidence in their ability to conduct acceptance testing or validation analysis, we strongly recommend that they contract for the assistance of someone with the required knowledge and experience. Description of the System. The election management system components, consisting of BOSS, Ballot Now, Rally, and Tally may be placed in one or more server/workstations consisting of PC-compatible units supported with appropriate printers and peripheral devices. The MBB (Mobile Ballot Box) is a proprietary PCMCIA memory card that is used in the JBC devices and eScan units for storage and which is also used to move data between other components of the system. BOSS supports the election definition management and provides support for programming the other components of the system. BOSS requires an MBB reader/writer as a peripheral as well as access to a printer for various review and audit reports Ballot Now provides a ballot-on-demand ballot printing service and supports the scanning of the paper ballots. It can use a variety of compatible high-speed scanners and laser printers. It can be configured in either a stand-alone or networked configuration. In the stand-alone mode, all of the Ballot Now processing is done on a single processor. In the networked configuration, one or more Ballot Now workstations can be attached to the Ballot Now server for multiple scanning and resolution workstations. In addition to the high-speed scanner, Ballot Now requires access to a PCMCIA reader/writer (to read and write to the MBB) and a hard drive of sufficient size to store the ballot images captured by the scanner. Rally supports reading the MBBs produced by the JBC and transferring the ballot images, called Cast Voter Records (CVRs) to the Tally subsystem. Rally requires access to a PCMCIA card reader/writer and a connection to the Tally subsystem. Consultant’s report on Hart Voting System Version 6.2.1 September 5, 2006 Page 3 Tally receives all the CSV results from either MBBs or from Rally and consolidates the ballot counts for final counting and voting result reporting. Tally requires access to a MBB reader/writer and a printer. ECM manager is an application that can create, save or copy crypto module tokens and can validate a token. Servo is an application that is used to prepare eSlates, JBCs and eScan units for an election and to harvest backup copies of cast vote records and logs from the equipment. Servo is usually run on a laptop so it can be easily moved from one device to the next in the elections warehouse. It uses a hard-wired connection to the devices. In the test configuration, these applications were split between two workstations: the BOSS/Ballot Now/Tally/ECM workstation and the Rally/SERVO workstation. The eSlate Precinct Voting System (PVS) consists of a Judge’s Booth Controller (JBC) (pictured to the left) connected to at least one and up to twelve eSlate units. The JBC is used by the poll worker to select the ballot style for the voter, to print authorization codes for each voter, and to print out results from the eSlates when the polls close. The eSlate is the DRE voter interface unit. It allows voters to make selections and cast ballots using an array of push buttons and a revolving wheel. The DAU (Disability Access Unit) on the back of the eSlate provides an audio ballot and connectors which allow the eSlate to be operated using jelly switches or a sip and puff device. As configured for use in California, each eSlate is equipped with a VBO (Verifiable Ballot Option) device which displays a thermal paper printout of each voter’s selections, which the voter can examine before the final step of casting their ballot. Consultant’s report on Hart Voting System Version 6.2.1 September 5, 2006 Page 4 The eScan unit is a precinct based mark-sense ballot tabulator. Voters insert their ballots into the eScan unit. It scans the ballots and feeds the ballots into the ballot box, which serves as the base for the unit. It is configured to reject ballots with overvoted contests and ballots which are blank. Tasks We Performed During the Test We participated in testing of the system from: • June 26 through June 30, 2006 at the Hart Intercivic office in Lafayette, Colorado • July 11 through July 13, 2006 at the Orange County Registrar of Voters’ warehouse in Santa Ana, California • July 24 through July 26, 2006 at the Secretary of State’s office in Sacramento, California. • August 22, 2006 at the Hart Intercivic office in Lafayette, Colorado. The July 11th testing was the Volume Test. The July 24th test was a retest required when portions of the June 26 testing work product was negated by discovery that the vendor had used a prior version of the BOSS Software to prepare the election definition for the test. The August 22, test was required when Hart upgraded the JBC Firmware from version 4.2.13 to version 4.3.0 and incremented the overall version of the system to 6.2.1. Our tasks during the tests included: • Verification that the test platforms were clean machines • Installing all of the software and firmware used in the test from trusted builds of the system components as prepared by the NASED ITA laboratories • Verification that the pc based machines had not been tampered with between rounds of testing • Documenting the system configuration • Executing the steps required in the Secretary of State’s test plans Consultant’s report on Hart Voting System Version 6.2.1 September 5, 2006 Page 5 • Recording the results of testing • Taking many of the photographs used to document the test Findings With the exception of some elements of the specific findings below, the system generally performed to the level expected during testing. With thorough acceptance testing by purchasers of the system, and appropriate operating procedures, the system appears to be capable of being used to conduct elections producing accurate results meeting the functional requirements of state law, with appropriate security and user friendly interfaces. This final revision of the system including JBC Firmware 4.3.0 successfully eliminated a software error, which was discovered in the prior version of the system. In JBC firmware version 4.2.13, when running a primary with a split precinct and a different ballot style on each side of the split, if the JBC was programmed to print a precinct results report which consolidated the two sides of the split into one report, the report would only contain the totals for the first side of the split. In version 4.3.0 of the JBC firmware the vendor has corrected the error. The tests conducted on August 22, 2006 verified the correction of this error and included regression testing to verify that no other functions of the JBC were adversely affected by the change. All other findings in this report are unchanged from our August 4, 2006 report on system version 6.2.0. • Although a substantial improvement over the version of the eScan in the prior version of the system, the eScans units still experienced anomalies. One of the units used in the June 26 round of testing rebooted itself multiple times and had to be taken out of the test. It was discovered that when a ballot is rejected and the override button to accept the ballot is pressed, a second ballot can be fed into the machine immediately behind the first and the second ballot will not be counted. During the Volume Test, one machine failed to boot and was taken out of the test. Ballot jams were significantly lowered due to a combination of changes to the feed routine in the new firmware and test voters who exercised greater care than in the volume test of the prior version. One machine experienced an error that required cycling power. We had no machines that required rebooting after ballot jams. All ballot jams could have been handled without opening the ballot box. • The eSlate units were substantially improved over the prior version. Printer errors were substantially reduced by the new design. There were eight printer error anomalies. Four of the errors were on one machine, which was taken out of the test. Except for the machine that was taken out of service, all of the others were remedied with a hot swap of the VBO unit. The cryptic error messages noted in the testing of version 6.1 of the system still exist. Consultant’s report on Hart Voting System Version 6.2.1 September 5, 2006 Page 6 • During the volume test, there were a noticeable number of instances of voter codes that were invalid. In several of these instances, we were unable to reconcile the statements of the voters, time stamps on the codes, and the status the system would show for the code. This issue is probably an artifact of the test methodology and operator error. Nonetheless, it deserves attention in the test plans for future tests of the system. • As in the previous version of the system, when in the Backup and Reset Window of the Servo application, the “Reset” button has no second chance warning the user. If the eSlate or JBC connected to the Servo has not been backed up and the backup button in the window is not checked, when the reset button is clicked the data on the attached eSlate of JBC will be erased without warning. A user who does this by mistake will lose data. That said, the data exists in multiple places and there is no function that would erase all copies at one time. The use procedures need to address this, and we suggest providing a second chance warning or a default setting to always back up data in future releases. • While voting the audio ballot on the eSlate, write-in input could not be reviewed or edited. Each letter was sounded out as it was entered. However, at the point, when the voter has completed his or her entry there is no way to review it. At all the opportunities for review, the machine merely says “Write-in Selected.” • When a voter selects a letter in the write-in screen, the audio ballot states “you have entered…” and states the letter entered. If, however, the voter presses the enter key and begins turning the select wheel before the letter is read, the machine will read the letter which is highlighted by the wheel instead of the letter actually entered. So, if you turn the wheel until you hear B, push enter and then turn the wheel one click to the right the machine will say “You have entered C.” The actual entry will have been B. Because there is no way to review the entries made, the voter cannot really figure out what letter was actually entered. • The Clear last function on the audio ballot write-in is also not intuitive. If you successfully enter “ABCD” go to the Clear Last Digit or Letter Entered, and press enter, the machine reads “You have entered C.” The next press yields “You have entered B” and then “You have entered A” and, on the last press, there is no audio response. • As with the prior version of the system, the heading on the voter verified ballot does not print out in the alternative languages. Reading the header is not necessary for the voter to review and verify their choices. If users discover that voters who require alternative languages are disturbed by this, it might be appropriate for change in future releases of the system. Consultant’s report on Hart Voting System Version 6.2.1 September 5, 2006 Page 7 • As with the prior version, the curbside voting application will require the entire booth to be taken off line and carried out into the parking lot. Although it is not heavy, it will be cumbersome and will likely require two people to move it. It has a wide footprint and is somewhat top heavy. The booth must be activated while connected to the JBC, disconnected and carried to the parking lot. The booth used for this must be at the end of the line of booths. After the voter votes, the booth must be brought back into the precinct and attached to the JBC for the vote to be recorded and to enable the booth for subsequent voters. It will require one trip to and back from the parking lot for each voter. Jurisdictions should plan for this and arrange the booths in each polling place so as to make this as easy as possible. In this new version of the system, if a machine is disconnected prematurely, it can simply be reconnected for a graceful recovery. With the previous version this error required rebooting the JBC and all connected eSlates. • It was discovered that a test “long candidate name” with 25 characters in the first name and 35 characters in the second name would not be displayed on the eSlate using a single column ballot. On the printed ballot, the name would appear but run off the page to the right. Hart’s representatives had no explanation for this anomaly. If this version of the system is certified this will need to be addressed in the use procedures. • A feature in Tally that allows administrative users to adjust vote totals for any candidate or issue does not have any data validation or controls to enforce the basic mathematical relationships of the election. In each race, candidate votes plus overvotes and undervotes will always equal total ballots cast. All county wide contests in a general election and each county wide race in each party’s portion of a primary election will always have the same number of ballots cast. So, if the presidential race has 1000 ballots cast the U.S. Senate race must have 1000 ballots cast. The Tally vote total adjustment feature does not enforce this relationship. An administrative user who does not understand these relationships within the election structure, or who is simply not very careful, can go into the utility and change the candidate votes, overvotes or undervotes in a single race and not be required to make matching adjustments to the other races in the election. If this incident occurred, the jurisdiction would suffer the embarrassment of turnout figures and ballot accounting figures that could not be reconciled. The system use procedures must require strict limitations on the use of administrator user ids in Tally and also provide instructions on how to reconcile any adjustments to a vote total throughout the election. Summary of System Benchmarking During the testing in the week of June 26, 2006 the software and firmware applications were loaded from trusted builds. MD5 Check files were created before, at various stages during, and at the end of testing. Consultant’s report on Hart Voting System Version 6.2.1 September 5, 2006 Page 8 On July 24, 2006, because there was no advance in versioning for the Ballot Now, BOSS, Rally, Tally, Servo and eCM Manager applications we ran FileCheck MD5 on the check files created at the end of the last round of testing and verified that the installations of the applications and database on the test machines had not been changed since the end of the last round of testing. Additional MD5 Check files were created at the end of testing. On August 22, 2006, because the only advance in versioning was for the JBC firmware, we again ran FileCheck MD5 on the check files from the prior round of testing verifying that the installations of the applications and database on the test machines had not been changed since the July 24th Round of Testing. The JBC and eSlate firmware were installed from trusted builds. System Configuration for the June 26 through June 30, 2006 Test 1. Server Dell Optiplex GX520 Service tag 8573T71 Express Service Code 17728524685 2. Monitor Dell MY0X37824760345RB9RU Running Microsoft Windows 2000 v. 5.0 build 2195 Service Pack 4 3. Laptop Dell Latitude D610 Service Tag 292DV71 Express Service Code 4901766445 Running Microsoft Windows 2000 v. 5.0 build 2195 Service Pack 4 4. Card Readers High Speed USB brad card reader/writer Model UISDMA2W Serial Number W02105802918F (new) Flashreader brand Model UISA2SE Serial Number U29031000004181 (old) 5. Scanner Kodak I260 Service Code 3178 Serial Number 12011222 Replaced during testing Replaced with Kodak I260 Service Code 3178 Serial Number 12811223 Replaced with Kodak I660 Service Code 4168 Serial Number 12742275 Replaced with Kodak I660 Service Code 4168 Serial Number 12737513 6. Printer HP 2420DN Product Number Q5959A Serial Number CNGKK28521 7. eScan Serial Number G778FE Replaced by Serial Number G77C74 Replaced by Serial Number G779FE 8. Judge’s Booth Controller (JBC) Model Number JBC1000B Serial Number C010E3 Consultant’s report on Hart Voting System Version 6.2.1 September 5, 2006 Page 9 9. VVPAT Module Model Number VBO Serial Number V00019 Running software version 1.8.3 10. DAU device Serial Number B0CCE5 Model Number DAUVBO 11. Hart eSlate 3000 Model Number eSlate 3000 Serial Number A0504B 12. VVPAT Module Model Number VPO Serial Number V000C1 Running software version 1.8.3 13. DAU device Model Number DAUVBO Serial Number B0CC67 14. Hart eSlate Model 3000 Serial Number A000C6 System Configuration for the July 24 through July 26, 2006 Test 1. Server Dell Optiplex GX520 Service tag 8573T71 Express Service Code 17728524685 2. Monitor Dell MY0X37824760345RB9RU Running Microsoft Windows 2000 v. 5.0 build 2195 Service Pack 4 3. Laptop Dell Latitude D610 Service Tag 292DV71 Express Service Code 4901766445 Running Microsoft Windows 2000 v. 5.0 build 2195 Service Pack 4 4. Card Readers High Speed USB brad card reader/writer Model UISDMA2W Serial Number W02105802918F (new) Flashreader brand Model UISA2SE Serial Number U29031000004181 (old) 5. Scanner Kodak I260 Service Code 3178 Serial Number 12011222 6. Printer HP 2420DN Product Number Q5959A Serial Number CNGKB95996 7. Replaced by Serial Number G779FE 8. Judge’s Booth Controller (JBC) Model Number JBC1000B Serial Number C010E3 9. VVPAT Module Model Number VBO Serial Number V000C1 Running software version 1.8.3 10. DAU device Serial Number B0CCE5 Model Number DAUVBO 11. Hart eSlate 3000 Model Number eSlate 3000 Serial Number A0504B Consultant’s report on Hart Voting System Version 6.2.1 September 5, 2006 Page 10 System Configuration for the August 22, 2006 Test 1. Server Dell Optiplex GX520 Service tag 8573T71 Express Service Code 17728524685 2. Monitor - Dell MY0X37824760345RB9RU Running Microsoft Windows 2000 v. 5.0 build 2195 Service Pack 4 3. Laptop Dell Latitude D610 Service Tag 292DV71 Express Service Code 4901766445 Running Microsoft Windows 2000 v. 5.0 build 2195 Service Pack 4 4. Card Readers High Speed USB brad card reader/writer Model UISDMA2W Serial Number W02105802918F (new) 5. Printer HP 2420DN Product Number Q5959A Serial Number CNGKB95996 6. Judge’s Booth Controller (JBC) Model Number JBC1000B Serial Number C01161 7. Judge’s Booth Controller (JBC) Model Number JBC1000B Serial Number C0011B 8. VVPAT Module Model Number VBO Serial Number V000C1 Running software version 1.8.3 9. VVPAT Module Model Number VBO Serial Number V001171 Running software version 1.8.3 10. DAU device Serial Number B0CCE5 Model Number DAUVBO 11. Hart eSlate 3000 Model Number eSlate 3000 Serial Number A0504B 12. Hart eSlate 3000 Model Number eSlate 3000 Serial Number A00040