Staff Report

A P LEX ADILLA | SECRETARY OF STATE | STATE OF CALIFORNIA OFFICE OF VOTING SYSTEMS TECHNOLOGY ASSESSMENT 1500 11th Street | Sacramento, CA 95814 | Tel 916.653.7244 | Fax 916.653.4620 | www.sos.ca.gov Five Cedars Group Alternate Format Ballot Remote Accessible Vote by Mail System Staff Report Prepared by: Secretary of State’s Office of Voting Systems Technology Assessment August 28, 2017 Table of Contents I. INTRODUCTION .................................................... 3 II. SUMMARY OF THE SYSTEM .............................. 3 III. TESTING INFORMATION AND RESULTS .......... 4 IV. COMPLIANCE WITH STATE AND FEDERAL LAWS…………………………….….…..35 V. CONCLUSION…………………………………….….37 2 | Page SECRETARY OF STATE’S STAFF REPORT – FIVE CEDARS GROUP AFB RAVBM I. INTRODUCTION 1. Scope This report presents the test results for all phases of the certification test of the Five Cedars Group, Alternate Format Ballot Remote Accessible Vote by Mail system. The purpose of the testing is to test the compliance of the voting system with the relevant California Voting System Standards, State and Federal laws. Testing also uncovers other findings, which do not constitute non-compliance, and those findings are reported to the voting system vendor to address the issues procedurally. 2. Summary of the Application Five Cedars Group submitted an application for the Alternate Format Ballot (AFB) Remote Accessible Vote by Mail system, which is comprised of the following major components: HTML Alternate Format Ballot AFB Ballot Generator In addition to the ballot, which includes the HTML source code, Five Cedars Group was required to submit the following: 1) the technical documentation package (TDP); 2) all the software components to field a complete working version of the ballot, including all peripheral devices, for the Functional Test Phase. 3. Contracting and Consultants Upon receipt of a complete application, the Secretary of State released a Request for Proposal (RFP) for assistance with the Security Review, both Source Code and Security testing. The statement of work (SOW) also had an option for the Secretary of State to use the awarded contractor for Functional testing, if it deemed necessary. Through the formal California contracting process, the Secretary of State awarded a contract to SLI Compliance (SLI), 4720 Independence Street, Wheat Ridge Colorado. II. SUMMARY OF THE SYSTEM 1. HTML Alternate Format Ballot The AFB is an HTML ballot that is generated from text files supplied to Five Cedars by a County that implements the AFB. The implementing County will be 3 | Page SECRETARY OF STATE’S STAFF REPORT – FIVE CEDARS GROUP AFB RAVBM responsible for identifying the correct ballot style for a given voter, and then downloading the AFB ballot to that voter. Currently, AFB ballots can only be generated from text files exported from the Hart system. Once downloaded, the AFB ballot is marked on the voter’s home equipment, and then the voter prints a cast vote record (CVR) on their home printer. A cast vote record is a record of the ballot that has been cast, but is not an actual ballot. The AFB CVR is then mailed back to the jurisdiction using a vote by mail ballot envelope. The QR code on the cast vote record contains the following information: a random number which is also printed on the cast vote record in human readable format, the ballot style, a version number , and the codes for the contest choices. The QR code does not contain any voter information. The contest choices are printed in the format 1:3, where 1 represents the first contest on the ballot, and the three represents the third choice in the contest. The CVR is duplicated on ballot at the implementing County. The computer downloading the AFB ballot can be disconnected from the internet after the ballot is delivered with no adverse results. 2. AFB Ballot Generator The AFB Ballot Generator is a Windows application that reads County supplied Hart BOSS ballot definition files, creates logical internal data tables, which it uses to build the accessible HTML ballots. The program was written in Microsoft VB.NET using Framework 4.5. The ballots are built by populating a set of pre- built ballot HTML templates which are assembled into a single HTML file for each ballot style required. If the county has supplied XLF ballot translation files, the AFB Ballot Generator will use the translation pairs, and a set of the HTML templates already translated into the desired language, to build ballots in the desired language. III. TESTING INFORMATION AND RESULTS 1. Background Five Cedars Group submitted an application to the Secretary of State for certification of the Alternate Format Ballot on April 24, 2017. California assigned AFB the project number CA-AFB1. California certification testing of the AFB system began in June 2017. The testing began with the Functional Testing, followed by Accessibility Testing, Source Code Review, and finally Security Review. 4 | Page SECRETARY OF STATE’S STAFF REPORT – FIVE CEDARS GROUP AFB RAVBM 2. Functional Test Data and Results The Functional Test of the Five Cedars AFB system was conducted by Secretary of State staff and Five Cedars staff at the Secretary of State’s Office located at 1500 11th Street, Sacramento, California from June 5 through June 6, 2017. The Secretary of State ran the Functional Test as if it were a voter using the system for the first time. Testing was conducted with four (4) laptop computers and one (1) printer provided by Five Cedars. OVSTA tested the Alternate Format Ballot using the following end user equipment: Table 2A: Functional Test Equipment Hardware Software Windows 10, Microsoft Narrator, JAWS version 18 Hewlett Packard (HP) screen reader, and a free reader from Australia called Spectre laptop NVDA Apple MacBook Air Apple accessibility software laptop ChromeVox version 53.0.2785.154 accessibility Chromebook software Apple iPad Standard Apple accessibility software The cast vote records were printed on a Canon P1100 ink jet printer. The Five Cedars representative generated ballots with the Five Cedars Ballot Generator software. The ballots were generated from the San Mateo 2012 General Election using comma separated text files exported from the Hart BOSS 4.3 system. The seven text files exported from the San Mateo 2012 General Election were: Candidate.txt Candidate_detail.txt Contest.txt Contest_and_Precinct.txt Election.txt Party.txt Precinct.txt 5 | Page SECRETARY OF STATE’S STAFF REPORT – FIVE CEDARS GROUP AFB RAVBM The Five Cedars Ballot Generator software worked as expected and generated AFB ballots correctly. At this time, the Five Cedars system will only generate ballots from text files that are exported from a Hart BOSS 4.3 system. The AFB ballot will allow over-votes, but warns the voter of the over-vote condition. The ballot also warns the voter of under-votes if you click on the “Test This Ballot” button. The AFB performed as expected against all California Secretary of State test cases, as well as the vendor supplied AFB test cases supplied by Five Cedars. Table 2B: Test Environment and Results Test Environment Result HP Spectre laptop using Windows 10 Performed as expected. and Narrator with Internet Explorer browser. HP Spectre laptop using Windows 10 Narrator encountered many problems and Narrator with Microsoft Edge with the Microsoft EDGE browser. browser. HP Spectre laptop using Windows 10 Performed as expected. and JAWS version 18 screen reader with Internet Explorer browser. HP Spectre laptop using Windows 10 JAWS encountered many errors when and JAWS version 18 screen reader using the Microsoft Edge browser. with Microsoft Edge browser. HP Spectre laptop using Windows 10 Performed as expected. and NVDA free reader with Internet Explorer browser. HP Spectre laptop using Windows 10 NVDA encountered many errors when and NVDA free reader with Microsoft using the Microsoft Edge browser. Edge browser. Apple MacAir laptop running Sierra Performed as expected. version 10.12.4, and the standard Apple accessible software 6 | Page SECRETARY OF STATE’S STAFF REPORT – FIVE CEDARS GROUP AFB RAVBM Apple iPad running IOS 11, and the Performed as expected. standard Apple accessibility software Chromebook laptop using ChromeVox Experienced one failure. version 53.0.2785.154 accessibility software. Dell laptop running Windows 7 with Worked approximately 50% of the time. Narrator The QR code was scanned from the cast vote record using an iPhone 4 smart phone running the following apps: Free QR Code Reader & BarCode Scanner from MixerBox Inc. QR Reader for iPhone by TapMedia Ltd. I-nigma QR Code and Data Matrix and 1D BarCode Reader from 3GVision. The free QR code reader from MixerBox, Inc. would not read the QR code. The QR Reader for iPhone by TapMedia Ltd. would not read the QR code. The I- nigma QR code Reader from 3GVision read the QR code as expected with no problems or errors. Findings The computer downloading the AFB ballot can be disconnected from the internet after the ballot is downloaded with no adverse results. The AFB performed as expected against all California Secretary of State test cases, as well as the vendor supplied AFB test cases supplied by Five Cedars. 3. Source Code Review The Source Code Review for the Five Cedars AFB system was conducted by SLI. The Source Code Review includes proprietary source code. The AFB system code was tested to the applicable California Voting System Standards (CVSS) requirements, and any applicable industry standards, as detailed below. SLI conducted a source code review of the source code for compliance to the CVSS. The source code was reviewed for adherence to the applicable standards in sections 5 and 7 of the CVSS. The expected outcome was that no issue would be found. The actual outcome was a determination that for the “Dead Code” (CVSS 5.2.7.e) requirement found in the source code base reviewed, two discrepancies were written against the code base, and for the 7 | Page SECRETARY OF STATE’S STAFF REPORT – FIVE CEDARS GROUP AFB RAVBM “Sufficient Header Comments” (CVSS 5.2.6.a-h) requirement found in the source code base reviewed, three discrepancies were written against the code base. The source code was reviewed for adherence to other applicable coding format conventions and standards including best practices for the coding language used. The expected outcome for this review was that no issue would be found. The actual outcome for this review was a determination that the source code was clean and met all CVSS and applicable standards requirements in this category. An analysis of the program logic and branching structure was conducted. The expected outcome was that no issue would be found. The actual outcome was a determination that the program logic and branching structure was reasonable and sufficient for the functionality implemented. An evaluation of whether the system is designed in a way that allows meaningful analysis, was conducted, including: Whether the architecture and code is amenable to an external review. Whether code analysis tools can be usefully applied. Whether the code complexity is at a level that obfuscates its logic. The expected outcome was that no issue would be found. The actual outcome was a determination that the architecture and code is amenable to external review and that the code complexity does not obfuscate the logic. Code analysis tools could be applied to this code base, but it is of a small quantity that manual review was as useful, if not more so. The AFB source code was searched for exposures to commonly exploited vulnerabilities including buffer overflows and SQL issues. The expected outcome for this review was that no exposures to commonly exploited vulnerabilities would be found in the AFB source code. The actual outcome for this review was a determination that no exposures to commonly exploited vulnerabilities were found in the AFB source code. The AFB source code was evaluated for the use and correct implementation of cryptography and key management. The expected outcome for this review was that cryptography and key management would be found to be correctly implemented in the AFB source code, as per the CVSS. The actual outcome for this review was a determination that cryptography and key management is correctly implemented in the AFB source code. 8 | Page SECRETARY OF STATE’S STAFF REPORT – FIVE CEDARS GROUP AFB RAVBM The AFB source code was analyzed for its ability to appropriately accommodate error and exception handling. The expected outcome for this review was that no issues with error and exception handling would be found in the AFB source code. The actual outcome for this review was a determination that no error and exception handling issues were found in the AFB source code. The AFB source code was evaluated in two areas for the likelihood of security failures being detected. a. Evaluate whether audit mechanisms are reliable and tamper resistant. The expected outcome for this review was that audit mechanisms in the AFB source code would be found to be reliable and tamper resistant. The actual outcome for this review that no issues were found – audit mechanisms in the AFB source code were found to be reliable and tamper resistant. b. Evaluate whether data that might be subject to tampering is properly validated and authenticated The expected outcome for this review was that any data in the AFB source code that might be subject to tampering would be properly validated and authenticated. The actual outcome for this review was that no issues were found – any data in the AFB source code that might be subject to tampering is properly validated and authenticated. The AFB source code was evaluated for the risk that a user can escalate his or her capabilities beyond those authorized. The expected outcome for this review was that in the AFB source code, a user cannot escalate his or her capabilities beyond those authorized. The actual outcome for this review was a determination that in the AFB source code, a user cannot escalate his or her capabilities beyond those authorized. The AFB source code was evaluated for embedded, exploitable code (such as “Easter eggs”) that can be triggered to affect the system. The expected outcome for this review was that no embedded, exploitable code (such as “Easter eggs”) that can be triggered to affect the system would be found to be resident in the AFB source code. The actual outcome for this review was a determination that no embedded, exploitable code (such as “Easter eggs”) that can be triggered to affect the system was found to be resident in the AFB source code. The AFB source code was evaluated that no code for dynamic memory access features which would permit the replacement of certificated executable code or control data or insertion of exploitable code or data would be found. The expected outcome for this review was that code for dynamic memory access features which would permit the replacement of certificated executable code or control data or insertion of exploitable code or data would not be found in the AFB source code. The actual outcome for this review was a determination that 9 | Page SECRETARY OF STATE’S STAFF REPORT – FIVE CEDARS GROUP AFB RAVBM no code for dynamic memory access features which would permit the replacement of certificated executable code or control data or insertion of exploitable code or data was found in the AFB source code. The AFB source code was evaluated for use of runtime scripts, instructions, or other control data that can affect the operation of security relevant functions or the integrity of the data. The expected outcome for this review was that no use of runtime scripts, instructions, or other control data would be found in the AFB source code. The actual outcome for this review was a determination that no use of runtime scripts, instructions, or other control data that can affect the operation of security relevant functions or the integrity of the data was found in the AFB source code. The AFB source code was evaluated that no code for dynamic memory access features which would permit the replacement of certificated executable code or control data or insertion of exploitable code or data would be found. The expected outcome for this review was that code for dynamic memory access features which would permit the replacement of certificated executable code or control data or insertion of exploitable code or data would not be found in the AFB source code. The actual outcome for this review was an determination that no code for dynamic memory access features which would permit the replacement of certificated executable code or control data or insertion of exploitable code or data was found in the AFB source code. The AFB source code was evaluated for design and implementation to ensure that sound, generally accepted engineering practices are followed, checking to verify that code is defensively written against bad data, errors in other modules, changes in environment, user errors, and other adverse conditions. The expected outcome for this review was that generally accepted engineering practices are followed and the code is defensively written in the AFB source code. The expected outcome for this review was a determination that in the AFB source code, generally accepted engineering practices are followed and the code is defensively written against bad data, errors in other modules, changes in environment, user errors, and any other potential adverse conditions. Discrepancies Ten discrepancies for the “Sufficient Header comments” requirement were found in the AFB source code base reviewed, as a result, ten discrepancies were written against the code base. Vulnerabilities For any vulnerabilities discovered, SLI was tasked with identifying the particular 10 | Pag e SECRETARY OF STATE’S STAFF REPORT – FIVE CEDARS GROUP AFB RAVBM standards applicable to each vulnerability. To the extent possible, reported vulnerabilities included an indication of whether the exploitation of the vulnerability would require access by: A Voter. Voters usually have low knowledge of the Remote Accessible Vote by Mail Machine System (RAVBMS) design and configuration. Some may have more advanced knowledge. A voter may carry out attacks designed by others. An Elections official insider. Elections official have a wide range of knowledge of the RAVBMS design and configuration. An official may have unrestricted access to the RAVBMS for long periods of time. Their designated activities include: o Set up and pre-election procedures; o Election operation; o Post-election processing of results; and o Archiving and storage operations. A Vendor insider: A vendor insider has great knowledge of the RAVBMS design and configuration. They have unlimited access to the RAVBMS before it is delivered to the purchaser and, thereafter, may have unrestricted access when performing warranty and maintenance service and when providing election administration services. No vulnerabilities were found within the source code reviewed, as a result, no findings were written against the code base. Findings Ten discrepancy findings were located within the AFB system. No potential vulnerabilities were identified within the AFB code base. Within the AFB code base, all findings were low risk vulnerabilities that would require an in-depth knowledge of the code base and how it operates to be able to successfully subvert the system. To exploit them successfully, it would require modifying the code. 4. Security Security testing was done at SLI. Testing was implemented without any prior knowledge of the source code. The testing was divided into three phases. 11 | Pag e SECRETARY OF STATE’S STAFF REPORT – FIVE CEDARS GROUP AFB RAVBM Phase I included a review of all pertinent documents for appropriate processes and procedures for implementing a secure system. This included review of the system design and architecture. Phase II included testing of relevant software, operating systems and hardware configurations. Phase III included testing of all telecommunications aspects of the system. Phase I Table 4A: Documentation Review Testing Performed Results 5.5 Vote Secrecy on Electronic Ballot Marking (EBM) Review of the Systems Technical Data Package (TDP) a. Immediately after the ballot is recorded to validated that the persistent electronic storage or printed, erasing requirement was the selections from the device’s display, working satisfactorily covered. memory, and all other storage, including all forms of temporary storage. b. Immediately after the voter chooses to cancel his or her ballot, erasing the selections from the display and all other storage, including buffers and other temporary storage. 6.1.2 Data Transmissions Review of the Technical Data These requirements apply to the use of Package (TDP) telecommunications to transmit data for the preparation validated that the of the system for an election, the execution of an requirement was election, and the preservation of the system data and satisfactorily covered. audit trails during and following an election. While this section does not assume a specific model of voting system operations and does not assume a specific model for the use of telecommunications to support such operations, it does address the following types of data, where applicable: Voter Authentication: Coded information that confirms the identity of a voter for security purposes for a system that transmits votes individually. Ballot Definition: Information that describes to a 12 | Pag e SECRETARY OF STATE’S STAFF REPORT – FIVE CEDARS GROUP AFB RAVBM voting machine the content and appearance of the ballots to be used in an election. Vote Count: Information representing the tabulation of votes at any level within the control of the jurisdiction, such as the polling place, precinct or central count. List of Voters: A listing of the individual voters who have cast ballots in a specific election. Additional data transmissions used to operate a voting system in the conduct of an election, but not explicitly listed above, are also subject to the requirements of this section. 6.2 Design, Construction, and Maintenance Review of the Requirements Technical Data Package (TDP) Design, construction, and maintenance requirements for validated that the telecommunications represent the operational capability requirement was of both system hardware and software. These satisfactorily covered. capabilities shall be considered basic to all data transmissions. 6.2.1 Confirmation Review of the Technical Data Confirmation occurs when the system notifies the user of Package (TDP) the successful or unsuccessful completion of the data validated that the transmission, where successful completion is defined as requirement was accurate receipt of the transmitted data. To provide satisfactorily covered. confirmation, the telecommunications components of a voting system shall notify the user of the successful or unsuccessful completion of the data transmission. In the event of unsuccessful transmission the user shall be notified of the action to be taken. 7.1.1 Elements of Security Outside Manufacturers Review of the Control Technical Data Package (TDP) The requirements of this section apply to the capabilities validated that the of a voting system that must be provided by the requirement was manufacturer. However, an effective security program satisfactorily covered. requires well defined security practices by the purchasing jurisdiction and the personnel managing and operating the system. These practices include: Administrative and management --including access controls. 13 | Pag e SECRETARY OF STATE’S STAFF REPORT – FIVE CEDARS GROUP AFB RAVBM Internal security procedures. Adherence to, and enforcement of, operational procedures (e.g., effective password management). Security of physical facilities. Organizational responsibilities and personnel screening. 7.2 Access control Review of the Technical Data Access controls are procedures and system capabilities Package (TDP) that detect or limit access to system components in validated that the order to guard against loss of system integrity, requirement was availability, confidentiality, and accountability. Access satisfactorily covered. controls provide reasonable assurance that system resources such as data files, application programs, and computer-related facilities and equipment are protected against unauthorized operation, modification, disclosure, loss or impairment. Unauthorized operations include modification of compiled or interpreted code, run-time alteration of flow control logic or of data, and abstraction of raw or processed voting data in any form other than a standard output report by an authorized operator. Access controls may include physical controls, such as keeping computers in locked rooms to limit physical access, and technical controls, such as security software programs designed to prevent or detect unauthorized access to sensitive files. The access controls described in this section are limited to those controls required to be provided by system manufacturers. 7.2.1 General Access Control Review of the Technical Data Voting system equipment shall provide access Package (TDP) control mechanisms designed to permit validated that the authorized access to the voting system and to requirement was prevent unauthorized access to the voting satisfactorily covered. system. Access control mechanisms on the EMS shall be capable of identifying and authenticating individuals permitted to perform operations on the EMS. Voting system equipment shall provide controls that permit or deny access to the device’s software and files. The default access control permissions shall 14 | Pag e SECRETARY OF STATE’S STAFF REPORT – FIVE CEDARS GROUP AFB RAVBM implement the minimum permissions needed for each role or group identified by a device. The voting device shall prevent a lower-privileged process from modifying a higher-privileged process. An administrator of voting system equipment shall authorize privileged operations. Voting system equipment shall prevent modification to or tampering with software or firmware through any means other than the documented procedure for software upgrades. 7.2.2 General Access Control Review of the Technical Data Identification requirements provide controls for Package (TDP) accountability when operating and administering a voting validated that the system. requirement was satisfactorily covered. The voting system shall identify users and processes to which access is granted and the specific functions and data to which each entity holds authorized access. 7.4.5 Software Reference Information Review of the Technical Data The manufacturer shall provide the National Software Package (TDP) Reference Library (NSRL), any California certified validated that the escrow facility, pursuant to Title 2, Division 7, Chapter 6 requirement was of the California Code of Regulation, and the Office of satisfactorily covered. the Secretary of State with a copy of the software installation disk, including the executable binary images of all third party software. Further, the manufacturer shall deposit the source code, tools, and documentation, to allow the complete and successful compilation of a system in its production/operation environment. The manufacturer shall document that the process used to verify the software distributed on unalterable storage media is the certified software by using the reference information provided by the NSRL or other designated repository before installing the software. The manufacturers shall document to whom they provide voting system software. 7.4.6 Software Setup Validation Review of the Technical Data Setup validation methods shall verify the contents of all Package (TDP) 15 | Pag e SECRETARY OF STATE’S STAFF REPORT – FIVE CEDARS GROUP AFB RAVBM system storage locations (e.g., system registers, validated that the variables, files, etc.) containing election specific requirement was information (e.g., ballot style, candidate registers, satisfactorily covered. measure registers, etc.). 7.8 Testing – Security Review of the Technical Data The S-ATA shall design and perform test procedures Package (TDP) that test the security capabilities of the voting system validated that the against the requirements. These procedures shall focus requirement was on the ability of the system to detect, prevent, log, and satisfactorily covered. recover from the broad range of security risks identified. These procedures shall also examine system capabilities and safeguards claimed by the manufacturer in the TDP to go beyond these risks. The range of risks tested is determined by the design of the system and potential exposure to risk. Regardless of system design and risk profile, all systems shall be tested for effective access control and physical data security. The S-ATA may meet these testing requirements by confirming proper implementation of proven commercial security software. In this case, the manufacturer must provide the published standards and methods used by the U.S. Government to test and accept this software, or it may provide references to free, publicly available publications of these standards and methods, such as government web sites. At its discretion, the S-ATA may conduct or simulate attacks on the system to confirm the effectiveness of the system's security capabilities. Phase II In this phase, functional tests were exercised in order to verify and validate security requirements, following are the requirements with their accompanying results: 5.5 Vote Secrecy on DRE and EBM Systems 7.2.1 General Access Control 7.2.2 Access Control Identification 7.2.4 Access Control Authorization 7.4.5 Software Reference Information 16 | Pag e SECRETARY OF STATE’S STAFF REPORT – FIVE CEDARS GROUP AFB RAVBM 7.4.6 Software Setup Validation 7.6 Telecommunications and Data Transmission 7.8 Testing – Security 7.8.1 Access Control 7.8.2 Data Interception and Disruption Table 4B: Phase II Functional Security Test CVSS Requirement Testing Performed Result 5.5 Vote Secrecy on Electronic Testing was performed AFB Ballot Marking (EBM) Systems to verify how the system performed as handled a ballot being expected Immediately after the ballot is printed and the browser and the recorded to persistent electronic closed, as well as when requirement storage or printed, erasing the the ballot is closed prior is met. selections from the device’s to being printed. display, working memory, and Attempts were made to all other storage, including all resume a ballot, as well forms of temporary storage. as to determine if any Immediately after the voter ballot information chooses to cancel his or her resided in history or ballot, erasing the selections cache. from the display and all other storage, including buffers and other temporary storage. 7.2.1 General Access Control Review of the For this requirement and particular General requirements address the attempted validation product and high-level functionality of a voting concludes that the Five suggested system. These are the fundamental Cedars AFB product delivery access control requirements upon does not have any built system this which other requirements in this in access control requirement section are based. mechanisms. Paradigm is not used is for the applicable. Voting system equipment shall jurisdiction to host ballot provide access control files on their voter mechanisms designed to permit registration system. authorized access to the voting system and to prevent unauthorized access to the voting system. 7.2.2 Access Control Identification Review of the For this requirement and particular Identification requirements provide attempted validation product and controls for accountability when concludes that the Five suggested 17 | Pag e SECRETARY OF STATE’S STAFF REPORT – FIVE CEDARS GROUP AFB RAVBM operating and administering a voting Cedars AFB product delivery system. does not have any built system this in access control requirement The voting system shall identify identification is not users and processes to which mechanisms. Paradigm applicable. access is granted and the used is for the specific functions and data to jurisdiction to host ballot which each entity holds files on their voter authorized access. registration system. 7.2.4 Access Control Authorization Review of the For this requirement and particular Voting systems shall explicitly deny attempted validation product and subject’s access based on access concludes that the Five suggested control lists or policies. Cedars AFB product delivery does not have any built system this in access control requirement authorization is not mechanisms. Paradigm applicable. used is for the jurisdiction to host ballot files on their voter registration system. 7.4.5 Software Reference Review of the For this Information requirement and particular attempted validation product and The voting system equipment shall be concludes that the suggested designed to allow the voting system system does not have delivery administrator to verify that the software checks in place to system this is the certified software by comparing it validate that any requirement to reference information produced by software is the certified is not the NSRL or other designated software, as there is no applicable. repository. software involved, simply custom made HTML ballots. 7.4.6 Software Setup Validation Review of the N/A requirement and Setup validation methods shall attempted validation include a software verification concludes that the method that ensures that the system doesn’t have voting system software has not checks in place to been modified illegitimately. validate that the AFB ballot system is the o The voting systems shall correct system, as there 18 | Pag e SECRETARY OF STATE’S STAFF REPORT – FIVE CEDARS GROUP AFB RAVBM include any supporting is no software involved, software and hardware simply custom made necessary to conduct the HTML ballots. software verification method. o The manufacturer shall document the process used to conduct the software verification method. o The software verification method shall not modify the voting system software on the voting system. 7.6 Telecommunications and Data Review of the For this Transmission requirement confirmed particular that the system utilizes product and There are four areas that must be electrical or optical suggested addressed by telecommunications and transmission, and that delivery data transmission security capabilities: the ballot may be sent system this access control, data integrity, detection via SSL or in other requirement and prevention of data interception, unspecified mediums. It is not and protection against external threats. was determined that no applicable. technology is utilized to verify unaltered receipt by the voter. What is sent/served is a blank ballot that does not contain any voter data or voting selections. Main security protocol is that once the blank ballot is delivered, there are no more communications between the voter and the ballot delivery system, all interactions remain local to the voter’s environment. 7.8 Testing Security Confirmed that the AFB AFB HTML ballot doesn’t performed as The S-ATA shall design and perform require internet access expected test procedures that test the security once the ballot has been and the capabilities of the voting system downloaded. Confirmed requirement against the requirements. These there are no external was met. 19 | Pag e SECRETARY OF STATE’S STAFF REPORT – FIVE CEDARS GROUP AFB RAVBM procedures shall focus on the ability of connections from the the system to detect, prevent, log, and ballot to any outside recover from the broad range of server or service. With security risks identified. These the exception of printing procedures shall also examine system the Summary there are capabilities and safeguards claimed by no external connections the manufacturer in the TDP to go from the ballot. beyond these risks. The range of risks tested is determined by the design of the system and potential exposure to risk. Regardless of system design and risk profile, all systems shall be tested for effective access control and physical data security. The S-ATA may meet these testing requirements by confirming proper implementation of proven commercial security software. In this case, the manufacturer must provide the published standards and methods used by the U.S. Government to test and accept this software, or it may provide references to free, publicly available publications of these standards and methods, such as government web sites. At its discretion, the S-ATA may conduct or simulate attacks on the system to confirm the effectiveness of the system's security capabilities. 7.8.1 Access Control Review of the N/A requirement and For those access control features built attempted validation in as components of the voting system, determined that the Five the S-ATA shall design tests to Cedars AFB product confirm that these security elements contains no access work as specified. control capabilities beyond those of which Specific activities to be conducted by the jurisdiction plans to the S-ATA shall include: implement. The requirement for security Specific tests designed by the S-ATA of the interactive ballots to verify the correct operation of all are based upon the documented access control already in place 20 | Pag e SECRETARY OF STATE’S STAFF REPORT – FIVE CEDARS GROUP AFB RAVBM procedures and capabilities, including Absentee/Mail-in ballot tests designed to circumvent controls system and the security provided by the manufacturer. These of the delivery method tests shall include: (Email, HTTPS, File sharing). o Performing the activities that the jurisdiction will perform in specific accordance with the manufacturer’s access control policy and procedures to create a secure system, including procedures for software and firmware installation. o Performing tests intended to bypass or otherwise defeat the resulting security environment. These tests shall include simulation of attempts to physically destroy components of the voting system in order to validate the correct operation of system redundancy and backup capabilities. This review applies to the full scope of system functionality. It includes functionality for defining the ballot and other pre-voting functions, as well as functions for casting and storing votes, vote canvassing, vote reporting, and maintenance of the system’s audit trail. 7.8.2 Data Interception and Review of the AFB Disruption requirement verified that performed as this system does not expected For systems that use utilize and the telecommunications, as provided for in telecommunications for requirement section 6 of the Standards and the transmission of was met. consistent with California law, to official voting data. Only transmit official voting data, the SATA delivery of blank ballot shall review, and conduct tests of, the that does not contain 21 | Pag e SECRETARY OF STATE’S STAFF REPORT – FIVE CEDARS GROUP AFB RAVBM data interception and prevention voter data or choice safeguards specified by the selections. manufacturer in its TDP. The S-ATA shall evaluate safeguards provided by the manufacturer to ensure their proper operation, including the proper response to the detection of efforts to monitor data or otherwise compromise the system. Phase III Table 4C: Phase III Telecommunications Testing CVSS Requirement Testing Performed Result 6.1.2 Data This requirement is here AFB performed as Transmission because of the expected and the transmission of requirement was met. These requirements electronic AFB HTML apply to the use of ballots from the telecommunications to jurisdiction to the Voter. transmit data As such all for the preparation of the of the security associated system for an election, with these AFB HTML the execution of an ballots falls to the election, jurisdictions and their and the preservation of absentee / mail-in voting the system data and programs. The AFB audit trails during and HTML ballots does not following an election. retain voter vote While this section does selections. The AFB not assume a specific HTML ballots does not model of voting system transmit results or operations and does not selections to a remote assume a specific model server. The AFB HTML for the use of ballots requires no telecommunications to telecommunications to support such operations, use with the exception of it does address the the initial download of the following types of data, ballot. where applicable: Voter Authentication: Coded information that confirms the identity of a 22 | Pag e SECRETARY OF STATE’S STAFF REPORT – FIVE CEDARS GROUP AFB RAVBM voter for security purposes for a system that transmits votes individually Ballot Definition: Information that describes to a voting machine the content and appearance of the ballots to be used in an election Vote Count: Information representing the tabulation of votes at any level within the control of the jurisdiction, such as the polling place, precinct or central count List of Voters: A listing of the individual voters who have cast ballots in a specific election. Additional data transmissions used to operate a voting system in the conduct of an election, but not explicitly listed above, are also subject to the requirements of this section. 6.2 Design, Review of the AFB performed as Construction, and requirement confirmed expected and the Maintenance that this system consists requirement was met. Requirements of an HTML Alternate format ballot which is Design, construction, and typically used for maintenance absentee and mail in requirements for ballot marking. All telecommunications telecommunications represent the operational aspects of the system are capability of both system controlled by state and 23 | Pag e SECRETARY OF STATE’S STAFF REPORT – FIVE CEDARS GROUP AFB RAVBM hardware and software. local jurisdiction These capabilities shall requirements. AFB HTML be considered basic to all ballots do not utilize data transmissions. specific telecommunications channels once the AFB HTML ballots has been downloaded and opened on the voter’s machine. 6.2.1 Confirmation This requirement was determined to be not Confirmation occurs applicable. As the AFB when the system notifies HTML ballots don’t the user of the successful transmit votes to a or unsuccessful remote system. The completion of the data Voter is able to fill out the transmission, where ballot, confirm selections successful completion is on the ballot and then defined as accurate print the ballot for mail in receipt of the transmitted ballot processing at a data. To provide remote location For this confirmation, the particular product and telecommunications suggested delivery components of a voting system this requirement system shall notify the is not applicable. user of the successful or unsuccessful completion of the data transmission. In the event of unsuccessful transmission the user shall be notified of the action to be taken. Potential Vulnerabilities To the extent possible, reported vulnerabilities include an indication of whether the exploitation of the vulnerability would require access by: o Voter: Usually has low knowledge of the voting machine design and configuration. Some may have more advanced knowledge. May carry out attacks designed by others. They have access to the machine(s) for less than an hour. 24 | Pag e SECRETARY OF STATE’S STAFF REPORT – FIVE CEDARS GROUP AFB RAVBM o Poll worker: Usually has low knowledge of the voting machine design and configuration. Some may have more advanced knowledge. May carry out attacks designed by others. They have access to the machine(s) for up to one week, but all physical security has been put into place before the machines are received. o Elections official insider: Wide range of knowledge of the voting machine design and configuration. May have unrestricted access to the machine for long periods of time. Their designated activities include: Set up and pre-election procedures; Election operation; Post-election processing of results; and Archiving and storage operations. o Vendor insider: With great knowledge of the voting machine design and configuration. They have unlimited access to the machine before it is delivered to the purchaser and, thereafter, may have unrestricted access when performing warranty and maintenance service, and when providing election administration services. 7.2.1 General Access Control The AFB system is a replacement ballot delivery method, designed for voters with disabilities to mark and print a CVR. Since AFB doesn’t require or utilize internet connectivity during the process of marking the CVR, the risk of the being compromised is negligible unless the main source of the AFB system at the jurisdiction is compromised, which can be addressed through manual processes and procedures. For the vulnerability in question, the level of access required to take advantage of this potential vulnerability would be inclusive to the following actors: Voter, who can attempt to modify the delivered HTML ballot, or attempt to remotely exploit the web server serving the HTML ballot files. Election official insider, who could attempt to remotely exploit the web server serving the HTML ballot files. Vendor Insider, who could attempt to locally exploit the web server serving the HTML ballot files. 7.4.5 Software Reference Information Due to the nature of the application it is possible to locally modify the AFB HTML ballot with advanced knowledge of HTML and JavaScript, however the impact is 25 | Pag e SECRETARY OF STATE’S STAFF REPORT – FIVE CEDARS GROUP AFB RAVBM mitigated by the voter being able to proof the CVR prior to and after printing, and then again before utilizing the jurisdictions already defined vote-by-mail procedures. For the vulnerability in question, the level of access required to take advantage of this potential vulnerability would be inclusive to the following actors: Voter, who can attempt to modify the server code remotely. Election official insider, who could attempt to modify the server code remotely. Vendor Insider, who could attempt to locally modify the server code. Findings The AFB system is designed for voters with disabilities to mark and print a CVR, which means that once the HTML file is delivered, the entire file/application runs in the current browser session. Since the AFB doesn’t utilize incoming or outgoing connections once the ballot is loaded this reduces the possibility of interception or manipulation through network attack vectors. This however poses a risk of server side contamination, to help mitigate this the vendor provides high level documentation about the processes/procedures and security to mitigate these risks. Including but not limited to: Secure hosting Physical security of hosting sites Network security Inventory and configuration management Access control Monitoring and logging For the vulnerability in question, the level of access required to take advantage of this potential vulnerability would be inclusive to the following actors: Voter, who could attempt to modify the delivered HTML, or attempt to remotely exploit the web server serving the HTML files. Election Official Insider, who could attempt to remotely exploit the web server serving the HTML files. Vendor Insider, who could attempt to locally exploit the web server serving the HTML files. Voter Privacy is ensured by removing client side storage of marked selections, which allows the voter to verify and print a CVR. 26 | Pag e SECRETARY OF STATE’S STAFF REPORT – FIVE CEDARS GROUP AFB RAVBM The ability to tamper with the client side HTML is always present due to the fact there are no server side verifications or validations in place. In this context however the ability to affect large numbers of ballots is reliant upon server side compromise (initial AFB ballot delivery), and the failure of the Absentee/Mail-in ballot system. The voter is given the ability to proof and confirm selections within the interactive ballot system as well as the paper ballot summary. For the vulnerability in question, the level of access required to take advantage of this potential vulnerability would be inclusive to the following actors: Voter, who could attempt to modify the delivered HTML. No discrepancy findings were located within the AFB system. Two potential vulnerabilities were identified within the AFB system, as listed in the Potential Vulnerabilities section above. 5. Accessibility For Accessibility, Usability and Privacy Testing the SOS conducted two phases of End User Usability testing. Additionally, SLI conducted three phases of testing for Accessibility, Usability and Privacy: documentation of usability and accessibility testing performed during system development, functional usability and accessibility testing and privacy testing. SOS End User Usability Testing The SOS conducted end user Accessibility testing from July 5 to July 7, 2017. Approximately twenty-four (24) testers participated in the testing. The testers were recruited through several channels including several groups within the accessibility community. Each tester used his/her own technology, including any auxiliary peripheral devices and screen reading software to access, mark, and print their paper cast vote record. The testers were asked to complete pre and post surveys, documenting such information as demographics, the technology used for testing, and the tester’s post-test experience using the system. The survey results of each are included in Attachment A of this report. Please note that personally identifying information such as names, email addresses, and telephone have been redacted. One (1) technical issue was identified, requiring mitigation. A tester using an iMAC computer system, with the Safari 10 browser, experienced the following issue when pressing the “check your ballot for mistakes” and “show my ballot summary page” buttons. The system would not advance to the ballot summary page. Five Cedars implemented a code change to fix the issue. The tester 27 | Pag e SECRETARY OF STATE’S STAFF REPORT – FIVE CEDARS GROUP AFB RAVBM attempted to use the system again, with success. The issue was marked as resolved. SLI Accessibility, Usability and PrivacyTesting The California Voting Systems Standards (CVSS) were written in such a way to be applicable to a wide variety of voting technology. Therefore, the relevant portions of the CVSS are reviewed as they relate to the Remote Accessible Vote by Mail (RAVBM) for purposes of this report. The use of “voting system” shall apply to RAVBM. Testing was divided into three phases. Phase I includes review of all pertinent documents as an inspection of the manufacturer’s documentation of usability and accessibility testing performed during system development. Phase II includes all Usability and Accessibility testing. Phase III includes Privacy testing. This testing will verify California Election Code specific requirements, o The RAVBM shall not have the capability, including an optional capability, to use a remote server to mark a voter’s selections transmitted to the server from the voter’s computer via the Internet. [EC19295(a)] o The RAVBM shall not have the capability, including an optional capability, to store any voter identifiable selections on any remote server. [EC19295(b)] o The RAVBM shall not have the capability, including the optional capability, to tabulate votes. [EC19295(c)] Phase I In this phase, documentation was reviewed to verify and validate the following: Review of the manufacturer’s documentation of usability and accessibility testing performed during system development. Review of the Technical Data Package (TDP) validated that the requirement was satisfactorily covered. Phase II In this phase, functional tests were exercised in order to verify and validate the requirements noted in the following sections. The tools listed below were utilized during testing in Phase II. 28 | Pag e SECRETARY OF STATE’S STAFF REPORT – FIVE CEDARS GROUP AFB RAVBM Table 5A : Consultant Tools Tool Name Tool Purpose Tool Use Wave Web Tool to evaluate accessibility of The vendor provided Accessibility web sites, to WCAG 2.0 and url’s were run through evaluation tool Section 508. MAUVE to help determine accessibility. A11y.css Tool to evaluate web application The vendor provided regarding conformance to ISO url’s were run with 9241-171: Ergonomics of human- A11y to help determine system interaction Guidance on compliance. software accessibility. Evaluera website Tool to evaluate web applications The vendor provided regarding conformance to WCAG url’s were run through 2.0 standards. 508 checker to help determine compliance. Chromevox – Tool is a screen reader that is an Screen reader used for Screen reader extension of Google Chrome, to Accessibility and extension of assist visually impaired users. Usability testing. Google Chrome. Usability Usability: The test lab shall report all the effectiveness metrics for usability as defined and measured by the Voting Performance Protocol (VPP). The test lab shall report the average voting session time, as measured by the VPP. Note that this requirement does not apply to the audio interface of a system or to the use of special input devices for voters with dexterity disabilities. The test lab shall report the average voter confidence, as measured by the VPP. SLI evaluated AFB against Section 3.2 et al of the CVSS. This section covers the requirements for Usability of a system. SLI found that the vast majority of the requirements were satisfactorily met. AFB failed on three (3) of the requirements. Each is listed in the table along with a response/mitigation: Table 5B: Usability Testing CVSS Requirement Result Mitigation/Response CVSS (3.2.7.c.ii) No AFB does not None repeating keys - No key or meet this control on a voting system requirement. shall have a repetitive effect as a result of being 29 | Pag e SECRETARY OF STATE’S STAFF REPORT – FIVE CEDARS GROUP AFB RAVBM held in its active position. CVSS (3.2.8.e): Voter AFB does not AFB cannot regulate a voter’s Inactivity Time meet this inactivity, voting session time, or requirement. place any other time restraints on a CVSS (3.2.8.f): Alert Time AFB does not voter using the system. The – Upon expiration of the meet this system concludes the session voter inactivity time, the requirement. once the voter retrieves the ballot. voting system shall issue Thus, there is no longer any an alert and provide a communication with the system to means by which the voter place a time restriction of any sort may receive additional on the voter. time. The alert time shall be between 20 and 45 seconds. If the voter does not respond to the alert within the alert time, the system shall go into an inactive state. Accessibility SLI evaluated AFB against the applicable portions of the CVSS for Accessibility Testing for compliance. The table below shows each of the applicable CVSS sections, and the result of compliance. Table 5C: Accessibility Testing CVSS Requirement Result CVSS (3.3.1.a.i.1): Accessibility Review of the requirement validated throughout the voting session - A VEBD that the requirement was satisfactorily shall be integrated into the covered. manufacturer’s complete voting system so as to support accessibility for disabled voters throughout the voting session. Documentation of Accessibility Procedures - The manufacturer shall supply documentation describing: recommended procedures that fully implement accessibility for voters with disabilities; and CVSS (3.3.1.a.i.2): How a VEBD supports those procedures. CVSS (3.3.1.b) : Complete information Review of the requirement validated 30 | Pag e SECRETARY OF STATE’S STAFF REPORT – FIVE CEDARS GROUP AFB RAVBM in alternative formats - When the that the requirement was satisfactorily provision of accessibility involves an covered. alternative format for ballot presentation, then all information presented to non- disabled voters, including instructions, warnings, error and other messages, and contest choices, shall be presented in that alternative format CVSS (3.3.1.d): Secondary means of Review of the requirement showed voter identification - If a voting system that no biometric measures are provides for voter identification or required, so this requirement is not authentication by using biometric applicable. measures that require a voter to possess particular biological characteristics, then the system shall provide a secondary means that does not depend on those characteristics. CVSS (3.3.1.e.i): Accessibility of paper- Review of the requirement validated based vote verification - If a VEBD that the requirement was not generates a paper record (or some applicable, as a voter will implement other durable, human-readable record) their own hardware. If a voter has for the purpose of allowing voters to equipment that utilizes either OCR verify their votes, then the system shall technology, or a QR reader with audio provide a means to ensure that the playback, then they would be able to verification record is accessible to all obtain an audio read back of their voters with disabilities. printed ballot. i. Audio readback for paper-based vote verification - If a VEBD generates a paper record (or some other durable, human-readable record) for the purpose of allowing voters to verify their votes, then the system shall provide a mechanism that can read that record and generate an audio representation of its contents. CVSS (3.3.2.c): Distinctive buttons and Review of the requirement validated controls - Buttons and controls on that the requirement was not accessible voting stations shall be applicable, as a voter will implement distinguishable by both shape and color. their own hardware. This applies to buttons and controls implemented either "on-screen" or in hardware. This requirement does not apply to sizeable groups of keys, such as a conventional 4x3 telephone keypad or a full alphabetic keyboard. CVSS (3.3.2.d): Synchronized audio Review of the requirement validated 31 | Pag e SECRETARY OF STATE’S STAFF REPORT – FIVE CEDARS GROUP AFB RAVBM and video - The voting station shall that the requirement was not provide synchronized audio output to applicable, as a voter will implement convey the same information as that their own hardware. Synchronized which is displayed on the screen. There audio/video, video-only, or audio only shall be a means by which the voter can are all obtainable. If voter environment disable either the audio or the video contains a screen, audio output and a output, resulting in a video-only or screen reader, synchronized output is audio-only presentation, respectively. available. Voter can turn off audio to The system shall allow the voter to have video only. Likewise voter can switch among the three modes turn off video display to have Audio (synchronized audio/video, video-only, only. or audio-only) throughout the voting session while preserving the current votes. CVSS (3.3.3.b.1): Audio-tactile interface Review of the requirements validated - The accessible voting station shall that these requirements were not provide an audio-tactile interface (ATI) applicable, as voter will implement that supports the full functionality of the their own hardware. visual ballot interface. Full functionality includes at a minimum: o Instructions and feedback on initial activation of the ballot (such as insertion of a smart card), if applicable; CVSS (3.3.3.b.2): Instructions and feedback to the voter on how to operate the accessible voting station, including settings and options (e.g., volume control, repetition); CVSS (3.3.3.b.3): Instructions and feedback for navigation of the ballot; CVSS (3.3.3.b.4): Instructions and feedback for contest choices, including write-in candidates; CVSS (3.3.3.b.5): Instructions and feedback on confirming and changing votes; CVSS (3.3.3.b.6): Instructions and feedback on final submission of ballot. CVSS (3.3.3.c.vii): Audio features and Review of the requirements validated 32 | Pag e SECRETARY OF STATE’S STAFF REPORT – FIVE CEDARS GROUP AFB RAVBM characteristics - Voting stations that that these requirements were not provide audio presentation of the ballot applicable, as voter will implement shall do so in a usable way, as detailed their own hardware. in the following sub-requirements. iv. Intelligible audio - The audio presentation of verbal information by both recorded and synthetic speech shall be readily comprehensible by voters who have normal hearing and are proficient in the language. This includes such characteristics as proper enunciation, normal intonation, appropriate rate of speech, and low background noise. Candidate names shall be pronounced as the candidate intends. This requirement applies to those aspects of the audio content that are inherent to the voting system or that are generated by default. CVSS (3.3.3.d): Ballot activation - If the Review of the requirement validated voting station supports ballot activation that the requirement was satisfactorily for nonblind voters, then it shall also covered. provide features that enable voters who are blind to perform this activation. CVSS (3.3.4.b): Support for non-manual Review of the requirements validated input - The accessible voting station that these requirements were not shall applicable, as a voter will implement provide a mechanism to enable non- their own hardware. manual input that is functionally equivalent to tactile input. All the functionality of the accessible voting station (e.g., straight party voting, write- in candidates) that is available through the conventional forms of input, such as tactile, shall also be available through non-manual input mechanisms such as mouth sticks and "sip and puff" switches. CVSS (3.3.6.a): Reference to audio Review of the requirements validated requirements - The accessible voting that these requirements were not station shall incorporate the features applicable, as voter will implement listed under the requirements for voting their own hardware. equipment that provides audio presentation of the ballot. CVSS (3.3.6.b): Visual redundancy for Review of the requirement validated 33 | Pag e SECRETARY OF STATE’S STAFF REPORT – FIVE CEDARS GROUP AFB RAVBM sound cues - If the voting system that the requirement was satisfactorily provides sound cues as a method to covered. alert the voter, the tone shall be accompanied by a visual cue, unless the station is in audio-only mode. CVSS (3.3.7): Use of ATI- For voters Review of the requirements validated who lack proficiency in reading English, that this requirement was not the voting equipment shall provide an applicable, as a voter will implement audio interface for instructions and their own hardware. ballots. CVSS (3.3.8): Speech not to be required Review of the requirement validated by equipment - Voting equipment shall that the requirement was satisfactorily not require voter speech for its covered. operation. Phase III SLI evaluated AFB, for compliance with California Elections Code (EC) requirements for privacy within an RAVBM system, in addition to the applicable portions of the CVSS. The table below shows each of the applicable code sections and CVSS sections, and the result of compliance. Table 5D: Privacy Testing Applicable California Elections Code Result Section & CVSS Requirement EC Section 19295(a): The RAVBM shall Review of the requirement validated not have the capability, including an that the requirement was optional capability, to use a remote server satisfactorily covered. to mark a voter’s selections transmitted to the server from the voter’s computer via the Internet. EC Section 19295 (b): The RAVBM shall Review of the requirement validated not have the capability, including an that the requirement was optional capability, to store any voter satisfactorily covered. identifiable selections on any remote server. EC Section 19295(c): The RAVBM shall Review of the requirement validated not have the capability, including the that the requirement was optional capability, to tabulate votes. satisfactorily covered. CVSS (3.2.4.1.a): Visual privacy - The Review of the requirement validated ballot, any other visible record containing that the requirement was ballot information, and any input controls satisfactorily covered, as the voter shall be visible only to the voter during the will utilize their own hardware in the voting session and ballot submission. environment of their choosing. CVSS (3.2.4.1.b): Auditory privacy - Review of the requirement validated 34 | Pag e SECRETARY OF STATE’S STAFF REPORT – FIVE CEDARS GROUP AFB RAVBM During the voting session, the audio that the requirement was interface of the voting system shall be satisfactorily covered, as the voter audible only to the voter. will utilize their own hardware in the environment of their choosing. CVSS (3.2.4.1.c): Privacy of warnings - Review of the requirement validated The voting system shall issue all warnings that the requirement was in a way that preserves the privacy of the satisfactorily covered, as the voter voter and the confidentiality of the ballot. will utilize their own hardware in the environment of their choosing. CVSS (3.2.4.1.d): No receipts - The voting Review of the requirements system shall not issue a receipt to the validated that this requirement was voter that would provide proof to another not applicable, as voter will of how the voter voted. implement their own hardware, and print their own marked ballot, as a RAVBMS system. CVSS (3.2.4.2.a): No information shall be Review of the requirement validated kept within an electronic CVR that that the requirement was identifies any alternative language satisfactorily covered, as CVR’s are feature(s) used by a voter. not created. CVSS (3.2.4.2.b): No information shall be Review of the requirement validated kept within an electronic CVR that that the requirement was identifies any accessibility feature(s) used satisfactorily covered, as CVR’s are by a voter. not created. Findings The AFB application is an HTML 5 SPA (Single Page Application), which means that once the initial server call for the application is processed the entire application runs in the current browser session. Voter privacy is achieved by removing client side storage of marked selections, the voter is allowed to verify and print a ballot summary card for use in currently setup jurisdiction absentee /mail in voting programs. The system was written to WCAG 2.0 guidelines to implement accessibility features. Usability was generally met, with the exception of repetitive keys. The voter is given the ability to proof and confirm ballot selections within the AFB interactive ballot system as well as the paper ballot summary. One discrepancy finding was determined within the Five Cedars AFB RAVBMS. That of the repetitive key, that when a key is pressed, if held, the action is repeated multiple times. The other two discrepancies found are not applicable, as a voter’s session time cannot be regulated. 35 | Pag e SECRETARY OF STATE’S STAFF REPORT – FIVE CEDARS GROUP AFB RAVBM IV. COMPLIANCE WITH STATE AND FEDERAL LAWS 1. Elections Code Review §19293 (b) Remote accessible vote by mail system standards adopted by the Secretary of State pursuant to subdivision (a) shall include, but not be limited to, all of the following requirements: (1) The machine or device and its software shall be suitable for the purpose for which it is intended. (2) The remote accessible vote by mail system shall preserve the secrecy of the ballot. (3) The remote accessible vote by mail system shall be safe from fraud or manipulation. (4) The remote accessible vote by mail system shall be accessible to voters with disabilities and to voters who require assistance in a language other than English if the language is one in which a ballot or ballot materials are required to be made available to voters. The system meets these requirements. Review of the requirement showed that English, Spanish, Hindi, Khmer, Korean, Tagalog, Thai, Vietnamese and Chinese are supported, however with external translation services, the system can support other languages as required. §19295 A remote accessible vote by mail system or part of a remote accessible vote by mail system shall do not any of the following: (a) Have the capability, including an optional capability, to use a remote server to mark a voter’s selections transmitted to the server from the voter’s computer via the Internet. (b) Have the capability, including an optional capability, to store any voter identifiable selections on any remote server. (c) Have the capability, including the optional capability, to tabulate votes. The system meets these requirements. 2. Section 508 and WCAG Compliance Review AFB was written to WCAG 2.0 guidelines to implement accessibility features. The system also complies with the applicable portions of Section 508. 36 | Pag e SECRETARY OF STATE’S STAFF REPORT – FIVE CEDARS GROUP AFB RAVBM V. CONCLUSION The Five Cedars Group AFB Remote Accessible Vote by Mail system, in the configuration tested and documented by the California Installation meets all applicable California and federal laws. 37 | Pag e SECRETARY OF STATE’S STAFF REPORT – FIVE CEDARS GROUP AFB RAVBM Attachment A 38 | Pag e SECRETARY OF STATE’S STAFF REPORT – FIVE CEDARS GROUP AFB RAVBM Five Cedars Alternate Ballot Format 1.0 Pre-Test Survey Q2 Age Answered: 14 Skpped: 0 # RESPONSES DATE 1 55 7/9/2017 8:48 PM 2 66 7/7/2017 9:37 AM 3 65 7/6/2017 10:54 AM 4 53 7/5/2017 5:07 PM 5 64 7/5/2017 4:46 PM 6 69 7/5/2017 2:58 PM 7 65 7/5/2017 2:20 PM 8 51 7/5/2017 1:42 PM 9 45 7/5/2017 1:36 PM 10 72 7/5/2017 11:54 AM 11 63 7/5/2017 10:59 AM 12 53 7/5/2017 10:57 AM 13 57 7/5/2017 10:49 AM 14 46 7/5/2017 9:40 AM 2 / 19 Five Cedars Alternate Ballot Format 1.0 Pre-Test Survey Q4 Email Answered: 14 Skpped: 0 4 / 19 Five Cedars Alternate Ballot Format 1.0 Pre-Test Survey Q5 Please describe the disabilities or difficulties you have that may interfere with you being able to mark a ballot privately and independently. Answered: 13 Skpped: 1 # RESPONSES DATE 1 B nd 7/9/2017 8:50 PM 2 I am tota y b nd. 7/7/2017 9:39 AM 3 I am tota y b nd and requre a systemwhch a ows the seectons to be read and the ab ty to 7/6/2017 10:57 AM check a box eectronca y. 4 I have a cogntve dsab ty that requres someone to support me n the votng process. I am 7/5/2017 5:13 PM perfecty capabe of makng my own decsons on what to vote on the ba ot. But, the votng process (as t s rght now) s too compex for me to do t a by mysef. 5 I have Cerebra Pasy and cannot wrte or hande paper 7/5/2017 4:49 PM 6 Tota y b nd 7/5/2017 3:12 PM 7 Tota y b nd. 7/5/2017 2:23 PM 8 none 7/5/2017 1:44 PM 9 I am b nd. 7/5/2017 11:56 AM 10 I sustaned a severe cosed TBI whch eft me wth severe Ataxa and other ssues. My Ataxa 7/5/2017 11:02 AM makes t neear mpossbe to mark a ba ot as I shake so bady. 11 Tota y b nd 7/5/2017 11:01 AM 12 I am amost tota y b nd and cannot read prnt. 7/5/2017 10:51 AM 13 I have Cerebra Pasy, whch causes a sgnfcant oss of fne motor sk s and mpared speech. I 7/5/2017 9:48 AM cannot physca y wrte we enough to mark a paper ba ot. 5 / 19 Five Cedars Alternate Ballot Format 1.0 Pre-Test Survey Q6 Which language would you prefer voting in ? Answered: 12 Skpped: 2 English Spanish Chinese Japanese Tagalog Thai Vietnamese Hindi Khmer Korean 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% ANSWER CHOICES RESPONSES Eng sh 100.00% 12 Spansh 0.00% 0 Chnese 0.00% 0 Japanese 0.00% 0 Tagaog 0.00% 0 Tha 0.00% 0 Vetnamese 0.00% 0 Hnd 0.00% 0 Khmer 0.00% 0 Korean 0.00% 0 TOTAL 12 6 / 19 Five Cedars Alternate Ballot Format 1.0 Pre-Test Survey # OTHER (PLEASE SPECIFY) DATE There are no responses. 7 / 19 Five Cedars Alternate Ballot Format 1.0 Pre-Test Survey Q7 Have you ever voted before? Answered: 13 Skpped: 1 Yes No 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% ANSWER CHOICES RESPONSES Yes 100.00% 13 No 0.00% 0 TOTAL 13 8 / 19 Five Cedars Alternate Ballot Format 1.0 Pre-Test Survey Q8 If "No", what has prevented you from voting in the past? Answered: 0 Skpped: 14 # RESPONSES DATE There are no responses. 9 / 19 Five Cedars Alternate Ballot Format 1.0 Pre-Test Survey Q9 What method of voting do you use most often? Answered: 12 Skpped: 2 Polls By Mail Early Voting 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% ANSWER CHOICES RESPONSES Po s 58.33% 7 By Ma 41.67% 5 Eary Votng 0.00% 0 TOTAL 12 # OTHER (PLEASE SPECIFY) DATE 1 I request and receve ba ot by ma then return competed to the po 7/5/2017 1:44 PM 2 Automark 7/5/2017 11:02 AM 10 / 19 Five Cedars Alternate Ballot Format 1.0 Pre-Test Survey Q10 Have you ever used a voting system with any special accessibility accommodations? Answered: 14 Skpped: 0 Yes No 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% ANSWER CHOICES RESPONSES Yes 71.43% 10 No 28.57% 4 TOTAL 14 11 / 19 Five Cedars Alternate Ballot Format 1.0 Pre-Test Survey Q11 If "Yes", which type of voting system? Answered: 9 Skpped: 5 Electronic Ballot Marking Device 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% ANSWER CHOICES RESPONSES Eectronc 88.89% 8 Ba ot Markng Devce 11.11% 1 TOTAL 9 # OTHER (PLEASE SPECIFY) DATE 1 DRE, ABMD, VBM, VBP, OBM, and VotePad. 7/5/2017 3:12 PM 12 / 19 Five Cedars Alternate Ballot Format 1.0 Pre-Test Survey Q12 What operating system are you using? Answered: 14 Skpped: 0 Windows 10 Windows 8.1 Windows 7 macOS 10.12 Android Linux 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% ANSWER CHOICES RESPONSES Wndows 10 35.71% 5 Wndows 8.1 7.14% 1 Wndows 7 42.86% 6 macOS 10.12 14.29% 2 Androd 0.00% 0 Lnux 0.00% 0 TOTAL 14 # OTHER (PLEASE SPECIFY) DATE 1 Home premum Ed. 7/5/2017 1:45 PM 13 / 19 Five Cedars Alternate Ballot Format 1.0 Pre-Test Survey Q13 What internet browser are you using? Answered: 13 Skpped: 1 Microsoft Edge 14 Internet Explorer 11 Safari 10.1 Google Chrome 59.0 Mozilla Firefox 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% ANSWER CHOICES RESPONSES Mcrosoft Edge 14 0.00% 0 Internet Exporer 11 53.85% 7 Safar 10.1 15.38% 2 Googe Chrome 59.0 23.08% 3 Moz a Frefox 7.69% 1 TOTAL 13 # OTHER (PLEASE SPECIFY) DATE 1 IE, Frefox, Chrome. 7/5/2017 2:26 PM 14 / 19 Five Cedars Alternate Ballot Format 1.0 Pre-Test Survey Q14 List the auxiliary devices and/or software you are using for electronic accessibility. Answered: 9 Skpped: 5 JAWS NVDA Apple Voiceover Windows Narrator 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% ANSWER CHOICES RESPONSES JAWS 66.67% 6 NVDA 11.11% 1 Appe Voceover 22.22% 2 Wndows Narrator 0.00% 0 TOTAL 9 # OTHER (PLEASE SPECIFY) DATE 1 Actvewords and MS mouse-keys 7/5/2017 4:53 PM 2 Wndow-Eyes 7/5/2017 3:38 PM 3 Openbook OCR scannng software. 7/5/2017 2:26 PM 4 Can use Wndow-Eyes as 2ndary test, but t s now defunked. 7/5/2017 1:45 PM 5 None 7/5/2017 11:05 AM 6 The "mouse keys," "f ter keys" and "stcky keys" optons ava abe under "Ease of Access" n the 7/5/2017 9:57 AM Wndows 7 (and 10) Contro Pane. 15 / 19 Five Cedars Alternate Ballot Format 1.0 Pre-Test Survey Q15 What is the make and model of the hardware you are using (PC, Laptop, Tablet, etc.)? Answered: 14 Skpped: 0 # RESPONSES DATE 1 Appe MacBook Pro 7/9/2017 8:56 PM 2 Wndows Pc sony vo 7/7/2017 9:43 AM 3 Ths s a De desktop PC. 7/6/2017 11:00 AM 4 Mac 21.5 nch md 2014 7/5/2017 5:17 PM 5 PC 7/5/2017 4:53 PM 6 Generc PC desktop 7/5/2017 3:38 PM 7 Asus Mode: X555YA-DB84Q 7/5/2017 2:26 PM 8 de A-6 INSPIRON 15 3000 seres aptop AMD processor 7/5/2017 1:46 PM 9 De Opt-pex 755 (pc) 7/5/2017 1:45 PM 10 MSI GT60 7/5/2017 12:04 PM 11 Unknown (at the brary) 7/5/2017 11:05 AM 12 Custom PC 7/5/2017 11:03 AM 13 Laptop 7/5/2017 10:53 AM 14 PC 7/5/2017 9:57 AM 16 / 19 Five Cedars Alternate Ballot Format 1.0 Pre-Test Survey Q16 What is the printer make and model you are using? Answered: 14 Skpped: 0 # RESPONSES DATE 1 Brother MFD 7/9/2017 8:56 PM 2 hp azar jet prnter 7/7/2017 9:43 AM 3 and aso a Canoscan Cannon scanner. 7/6/2017 11:00 AM 4 HP Envy 4500 7/5/2017 5:17 PM 5 N/A 7/5/2017 4:53 PM 6 Canon MX920 7/5/2017 3:38 PM 7 None. 7/5/2017 2:26 PM 8 HP desk jet 2600 seres 7/5/2017 1:46 PM 9 I own none. W provde tfs of competed ba ots 7/5/2017 1:45 PM 10 my company's combo prnter 7/5/2017 12:04 PM 11 Unknown (at the brary) 7/5/2017 11:05 AM 12 HP 2050 7/5/2017 11:03 AM 13 do not know 7/5/2017 10:53 AM 14 HP Laserjet 3011 7/5/2017 9:57 AM 17 / 19 Five Cedars Alternate Ballot Format 1.0 Pre-Test Survey Q17 What antivirus software are you using? Answered: 7 Skpped: 7 McAfee Norton Windows Defender Kaspersky 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% ANSWER CHOICES RESPONSES McAfee 42.86% 3 Norton 0.00% 0 Wndows Defender 42.86% 3 Kaspersky 14.29% 1 TOTAL 7 # OTHER (PLEASE SPECIFY) DATE 1 Don't know 7/9/2017 8:56 PM 2 Mcrosoft securty essentas 7/7/2017 9:43 AM 3 Mcrosoft Essentas 7/6/2017 11:00 AM 4 McroSoft 7/5/2017 3:38 PM 5 Maware Bytes. 7/5/2017 2:26 PM 6 Mcrosoft Sec. Essentas (/WnDefender?) 7/5/2017 1:45 PM 7 System Center Endpont Protecton, Mawarebytes and 7/5/2017 12:04 PM 8 Unknown (at the brary) 7/5/2017 11:05 AM 9 eSet NOD32 7/5/2017 11:03 AM 18 / 19 Five Cedars Alternate Ballot Format 1.0 Pre-Test Survey Q18 Is your antivirus software up to date? Answered: 14 Skpped: 0 Yes No 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% ANSWER CHOICES RESPONSES Yes 92.86% 13 No 7.14% 1 TOTAL 14 19 / 19 Five Cedars Alternate Ballot Format 1.0 Post-Test Survey Q1 For each question, please indicate how strongly you agree or disagree with the statement. Answered: 12 Skipped: 0 The marking of my ballot... I feel I can use this sys... I am confident that my ball... The instructions... The system was easy to use. I could read the display... I could understand t... The time frame it took to m... I can use this system witho... Printing was easy. My ballot printed with... 0 1 2 3 4 5 6 7 8 9 10 AGREE AGREE NEUTRAL DISAGREE DISAGREE TOTAL WEIGHTED STRONGLY SOMEWHAT SOMEWHAT STRONGLY AVERAGE The marking of my ballot method 75.00% 25.00% 0.00% 0.00% 0.00% was private. 9 3 0 0 0 12 1.25 I feel I can use this system to 72.73% 9.09% 9.09% 0.00% 9.09% mark my ballot independently. 8 1 1 0 1 11 1.64 I am confident that my ballot was 63.64% 0.00% 0.00% 27.27% 9.09% recorded accurately. 7 0 0 3 1 11 2.18 The instructions were clear and 54.55% 27.27% 9.09% 0.00% 9.09% complete. 6 3 1 0 1 11 1.82 The system was easy to use. 54.55% 27.27% 0.00% 9.09% 9.09% 6 3 0 1 1 11 1.91 I could read the display easily.(If 55.56% 0.00% 33.33% 11.11% 0.00% applicable) 5 0 3 1 0 9 2.00 1 / 7 Five Cedars Alternate Ballot Format 1.0 Post-Test Survey I could understand the speech 80.00% 20.00% 0.00% 0.00% 0.00% output. (If applicable) 8 2 0 0 0 10 1.20 The time frame it took to mark my 63.64% 27.27% 0.00% 0.00% 9.09% ballot was what I expected. 7 3 0 0 1 11 1.64 I can use this system without 63.64% 9.09% 0.00% 18.18% 9.09% technical help. 7 1 0 2 1 11 2.00 Printing was easy. 27.27% 18.18% 18.18% 18.18% 18.18% 3 2 2 2 2 11 2.82 My ballot printed with the correct 45.45% 18.18% 18.18% 9.09% 9.09% selections. 5 2 2 1 1 11 2.18 2 / 7 Five Cedars Alternate Ballot Format 1.0 Post-Test Survey Q2 Would you be satisfied using this system to mark your ballot in an election? Answered: 11 Skipped: 1 Yes No 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% ANSWER CHOICES RESPONSES Yes 81.82% 9 No 18.18% 2 TOTAL 11 3 / 7 Five Cedars Alternate Ballot Format 1.0 Post-Test Survey Q3 Would you rather mark your ballot using another method? Answered: 11 Skipped: 1 Yes No 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% ANSWER CHOICES RESPONSES Yes 27.27% 3 No 72.73% 8 TOTAL 11 4 / 7 Five Cedars Alternate Ballot Format 1.0 Post-Test Survey Q4 If you prefer another method, what method would you prefer? Answered: 5 Skipped: 7 # RESPONSES DATE 1 LA County BMD 7/18/2017 9:40 PM 2 I would prefer a ballot which printed all on one page. Marking the ballot was easy, but scanning it 7/7/2017 7:35 PM after printing it was not so simple. Keeping the pages in order required other software such as Open Book 9.0. The ideal situation would be to actually transmit the ballot and print a receipt as proof, if necessary. 3 I sometimes use absentee ballots and someone to help me mark the items. 7/7/2017 1:18 PM 4 Electronic voting at polling place. Marked neutral for visibility of display question, since I am totally 7/7/2017 12:40 PM blind and do not have a screen physiccally connected to my computer. Recommend an "NA" choice be added to such a question, since that would be a more accurate statement. Marked "disagree somewhat" for the speech component because there were occasional typos in ballot/glitches in web coding that made me have to look at some things more closely to figure out what was going on, but my speech synth has very clear audio. Also recommend "NA" option here for those who do not use/need speech access. I imagine that if this were a real ballot, such glitches and typos would be fixed, but must not assume this for purposes of testing. I could not verify accuracy of .tif ballot, so marked neutral wich also should have been "NA". I am in VERY STRONG disagreement with ANY kind of vbm system that REQUIRES printed ballots, signed envelopes or other documentationthat MUST be relegated to the ancient methods of quill and parchment (ink on paper). Though marking would be private and independent, sighted assistance would be REQUIRED to complete the task of actuating the mailing, which renders such systems unusable. Since I can neither read nor write print without either technological or human intervention, I will not participate in programs or systems that propagate the enforcement of such arcane measures, which means *heavy sigh* I'll probably never be able to vote by mail. Though I know that electronic voting machines at polling places spit out print scrolls (receipts or something), I have the poll workers take them directly. Those pages shut me out (can neither read nor verify their physical content independently), so I don't work with them. This means that people who can't physically read, write or manipulate print will never actually be able to be a part of this important voting alternative until such barbaric rules as paper proof are removed from it. *heavy sigh* Very disappointing. That makes the act of voting more difficult than it has to be, wich will prevent some from participating and make others have to fight even harder to make it possible for pwd to get out and vote. This means that some important voices from our community are silenced. We should make vbm truly accessible for ALL CALIFORNIANS, not just those with access to print. Voting can be rigged whether paper is present or not. We have seen this, have we not? 5 So far, Democracy live is better. 7/7/2017 9:37 AM 5 / 7 Five Cedars Alternate Ballot Format 1.0 Post-Test Survey Q5 Do you have any suggestions for changes on this system and/or any other comments you would like to provide? Answered: 11 Skipped: 1 # RESPONSES DATE 1 Yes, and I will send them to the Sec. of State in an Email report. 7/18/2017 9:40 PM 2 I had no problems with this system. 7/9/2017 10:22 PM 3 In general, your system is superb, very easy to navigate, and highly accessible. My only 7/8/2017 10:17 AM suggestion is to expand a bit your instructions on "printing" the ballot. What the user wants to do is to save as PDF the ballot, and attach that to an email. This is what the system ends up doing, but the general user would be confused, thinking that printing and mailing a hard copy ballot is the only option. I can help write or review any such amended instructions; contact me at ttford@gmail.com 4 The write-in option should be available for all candidates. So far, Democracy live seemed to have 7/7/2017 7:35 PM the easiest system to print and scan into email. 5 This basic system was easy enough to use. I'd only suggest that the final printed ballot come out 7/7/2017 7:20 PM as one page very like what happens at the pole. 6 I really like the option of seeing either the whole ballot or one race at a time. I tried using both 7/7/2017 6:32 PM methods, and I think both were good. I can imagine that going one race at a time would help voters with certain disabilities a great deal. 7 No, I was quite happy. 7/7/2017 1:18 PM 8 See above. *rueful smile* should have looked at all questions befor deciding where to put the long 7/7/2017 12:40 PM rant. 9 When I was trying this voting system, I had gotten to the bottom of my ballot. There were three 7/7/2017 12:03 PM buttons there. One button wanted to check the ballot for mistakes (which I pushed with my mouse) and it refused to work for me! The other button at the end of ballot that refused to work (even though I pressed it up to 20 times) was the button to get to the ballot summary page (that I had to get to for the SoS to make a PDF of the ballot summary page to then email to the SoS). i am not entirely sure that my votes were accurately recorded by the SoS because of the fact I couldn't get to my ballot summary page. I do like this voting system very much. But, the SoS has to overcome the technical difficulties of such a voting system. The only button on bottom of the ballot that was indeed working was the button to take you back up to the beginning of the ballot. I pushed that button several times (to test it) and each time it immediately took me to the beginning of the ballot. I personally feel that this particular voting system is worth pursuing because it can be a great help to people with disabilities exercise their right as US citizens to vote in local, state, and national elections. Just overcome the technical difficulties in administrating the voting system. FYI: I was using an iMac with Safari 10 when I ran into my technical difficulties with the new voting system. 10 Again, voter should be required to enter address manually. Less confusing chatter and 7/7/2017 9:37 AM explanations: for instance, having a link for ballot and one for mark your ballot, puzzling. For the Presidential race, checkboxes were not announced. If I make an error like over-voting, I shouldn't have to hunt and up-arrow to the error message. Jaws routinely did not work properly after I entered a write-in candidate. When I'd try to return to the page, Jaws said, "end of ballot" even though it wasn't. I had to restart Jaws in order to read with arrow keys again. I also found it confusing being presented with "show ballot" and "check for mistakes". Simpler if I simply can review ballot and then hit "back if I find any mistakes. Clunky. The Laguna Beach city council race choices were incorrectly recorded. 6 / 7 Five Cedars Alternate Ballot Format 1.0 Post-Test Survey 11 Overall I am quite impressed with the quality of the online voting interface. The markup and error 7/7/2017 9:26 AM handling messages are excellent. The two areas in which enhancement could be considered are the registration and supplemental information. Since this was a demonstration I do not expect the interaction leading to the ballot is necessarily reflective of the experience for a live ballot because I would like to see more verification and display of voter registration information. For the ballot itself, there are frequently cases where more information is desired about candidates and measures so it would be helpful to provide access to this information either through links to the supplemental resources or through buttons that toggle the display of such information on the page under each item. Also, as I recall, there was a question in my mind when starting the ballot what shortcut keys would apply for quick navigation and selection. So perhaps there could be a reference to this sort of information for JAWS users and other assistive software or devices? 7 / 7