OVSTA
Staff Report
Democracy Live Secure Select 1.0
Read the report at Democracy Live ↗
A P
LEX ADILLA | SECRETARY OF STATE | STATE OF CALIFORNIA
OFFICE OF VOTING SYSTEMS TECHNOLOGY ASSESSMENT
1500 11th Street | Sacramento, CA 95814 | Tel 916.695.1680 | Fax 916.653.4620 | www.sos.ca.gov
Democracy Live
Secure Select 1.0
Remote Accessible Vote by Mail System
Staff Report
Prepared by:
Secretary of State’s Office of
Voting Systems Technology Assessment
August 29, 2017
Table of Contents
I. Introduction ......................................................................................... 1
1. Scope ................................................................................... 1
2. Summary of the Application ................................................. 1
3. Contracting and Outsourcing ............................................... 1
II. Summary of the System .................................................................... 2
III. Testing Information and Results ........................................................ 2
1. Background .......................................................................... 3
2. Functional Testing ................................................................ 3
3. Accessibility, Usability and Privacy Testing .......................... 4
4. Security and Telecommunications Testing ........................... 13
5. Software Review Testing ..................................................... 20
IV. Compliance with State and Federal Laws and Regulations ............... 20
1. Elections Code Review ......................................................... 20
3. Section 508 and WCAG Compliance Review ....................... 21
V. Conclusion .......................................................................................... 21
SECRETARY OF STATE’S STAFF REPORT –Democracy Live Secure Select 1.0 2 | Page
A P
LEX ADILLA | SECRETARY OF STATE | STATE OF CALIFORNIA
OFFICE OF VOTING SYSTEMS TECHNOLOGY ASSESSMENT
1500 11th Street | Sacramento, CA 95814 | Tel 916.695.1680 | Fax 916.653.4620 | www.sos.ca.gov
I. INTRODUCTION
1. Scope
This report presents the test results for the all phases of the certification test of
the Democracy Live Secure Select 1.0 Remote Accessible Vote by Mail
(RAVBM) system. The purpose of the testing is to test the compliance of the
RAVBM system with California and Federal laws. Testing also uncovers other
findings, which do not constitute non-compliance, and those findings are reported
to the RAVBM system vendor to address the issues procedurally. The
procedures for mitigating any additional findings are made to the documentation,
specifically the Secure Select 1.0 Use Procedures.
2. Summary of the Application
Democracy Live submitted an application for the Secure Select 1.0 system on
December 19, 2016. Secure Select 1.0 is a cloud based application, solely for
the purposes of ballot marking pursuant to Elections Code sections 303.3 and
19283.
Democracy Live was required to submit the following: 1) the technical
documentation package (TDP); 2) hardware for functional testing (Windows and
Apple OS systems, including all supported browsers, screen readers; 3) source
code; and 4) the Democracy Live Secure Select 1.0 Use Procedures.
3. Contracting and Consulting
Upon receipt of a complete application, the Secretary of State (SOS) released a
Request for Quote (RFQ) for assistance with the Source Code Review,
Telecommunications and Security Review, and Usability, Accessibility and
Privacy testing. The statement of work (SOW) also had the option for assistance
with Functional Testing. That testing was conducted by SOS staff.
Through the formal California contracting process, the Secretary of State
awarded a contract to SLI Compliance, a division of Gaming Laboratories
International, LLC.
II. SUMMARY OF THE SYSTEM
Secure Select 1.0 application is an HTML 5 SPA (Single Page Application),
which means that once the initial server call for the application is processed the
entire application runs in the current browser session. Secure Select has three
main components. A Ballot Definition File is created and passed into Secure
Select. Secure Select parses the Ballot Definition File and presents a ballot style
to a voter. The voter can optionally use a Screen Reader to navigate through the
ballot. After marking their ballots and reviewing their selections, the voter must
print their selections. The printed paper cast record is returned to the local
elections official, where it will be remade into a ballot. The printed paper cast
record and the ballot is kept together for auditing purposes.
III. TESTING INFORMATION AND RESULTS
1. Background
Democracy Live submitted an application to the Secretary of State for
certification of the Secure Select 1.0 RAVBM system on December 19, 2016.
The system has two hosting mechanisms, vendor hosted or county hosted. For
the purposes of this examination and certification, the SOS staff and the
consultant tested the vendor hosted solution.
State examination and functional testing of this system was conducted by SOS
staff. The functional test was conducted at the Secretary of State’s office in
Sacramento, California from May 15 to May 17, 2017. Phase I Accessibility
testing was performed by thirty (30) volunteers throughout the state of California,
from June 14 to June 16, 2017. The Software Review was performed by SLI from
July 31 to August 7, 2017. Security and Telecommunications testing was
performed by SLI from August 4 to August 21, 2017. A final Accessibility test
SECRETARY OF STATE’S STAFF REPORT –Democracy Live Secure Select 1.0 2 | P a g e
was administered by SOS staff from August 11 to August 17, 2017, with
approximately nine (9) volunteers.
Secure Select 1.0 was evaluated against the applicable portions of the California
Voting Systems Standards (CVSS). The CVSS were written in such a way to be
applicable to a wide variety of voting technology, therefore the relevant portions
of the CVSS were reviewed as it relates to the Secure Select 1.0 RAVBM system
for purposes of this report. The use of “voting system” shall apply to RAVBM.
2. Functional Testing Summary
System Configuration:
The Secure Select 1.0 Remote Accessible Vote by Mail System is a software
only solution. The vendor only requires that a customer provide ballot definition
data.
For the functional test, the SOS provided Democracy Live with three test ballot
data sets of California Elections as described below:
1. State of California Primary Election – Fictional Jurisdiction
2. State of California General Election - Fictional Jurisdiction
3. State of California Recall Election – October 2003 Statewide Recall Election
The vendor provided two test machines for the Functional Test. The base
configurations of the systems were as follows:
Machine Type OS Screen Reader Browser
Apple Laptop OS X 10.12 VoiceOver Safari 10.1
Lenovo Laptop Windows 10 Narrator Internet Explorer 11,Edge 10
SOS staff used each machine provided to access, mark and print the paper cast vote
record as described in the Secure Select 1.0 Use Procedures. Each test environment
exercised undervotes, overvotes, all contest marking, and write-ins. Staff also used
keyboard functions, standard USB mice, and the touchpad on the laptops to mark
selections.
Undervotes
The system warned of undervotes, and allowed the tester to proceed to the end of the
ballot to print the paper cast record.
SECRETARY OF STATE’S STAFF REPORT –Democracy Live Secure Select 1.0 3 | Page
Overvotes
The system warned of overvotes, and allowed the tester to proceed to the end of the
ballot and print the paper cast record.
Write-Ins
The system allowed the tester to make write in selections with a fifty (50) character limit.
Printed Paper Cast Record
Each paper cast record printed displays the Election Type, the precinct number, and the
ballot style/type. The paper cast record also produces a QR Code near the top right of
the paper cast record. State of California staff used multiple QR Code readers to
decipher the information contained within the QR Code. An Example of the QR Code
data is below:
Table 2A: QR Code Data
v:1.2 bs:1 pid:CP05 id:1497039770174.532 1:1 2:2 3:1 4:1 5:1 6:1 7:1 14:1 15:1
Key Data Value
v QR Code Format Version
bs Ballot Style Code
pid Precinct Identifier
id Unique ballot identifier generated by
Secure Select
1:1 Contest Number: Selection Number (First
Contest, First Candidate Marked)
SOS staff identified some minor issues and clarification with the Use Procedures. The
vendor was provided with the issues and adequately modified the Use Procedures.
3. Accessibility, Usability, and Privacy Testing
For Accessibility, Usability and Privacy Testing the SOS conducted two phases of End
User Usability testing. Additionally, SLI conducted three phases of testing for
Accessibility, Usability and Privacy: documentation of usability and accessibility testing
performed during system development, functional usability and accessibility testing and
privacy testing.
SOS End User Usability Testing - Phase I
Phase I of End User Usability testing was conducted from June 14 to June 16, 2017.
The testing was conducted by SOS staff. Approximately thirty (30) testers participated in
the testing. The testers were recruited through several channels including several
SECRETARY OF STATE’S STAFF REPORT –Democracy Live Secure Select 1.0 4 | Page
groups within the accessibility community. Each tester used his/her own technology,
including any auxiliary peripheral devices and screen reading software to access, mark,
and print their paper cast vote record.
The testers were asked to complete pre and post surveys, documenting such
information as demographics, the technology used for testing, and the tester’s post-test
experience using the system. The survey results of each are included in Attachment A
of this report. Please note that personally identifying information such as names, email
addresses, and telephone have been redacted.
The technical issues identified as requiring mitigation or responses are listed below in
Table 3A.
Table 3A: Phase I Tester Technical Issues
Issue Response/Mitigation
1. Non-accessible return The SOS will develop a model template for the return
package. instructions of the paper cast vote record. The SOS
along with jurisdictions will educate voters on the
restrictions of Remote Accessible Vote by Mail voting,
including email return of the paper cast vote record is
not an option at this time.
2. Time limitations. Test configuration had a five minute time limit enabled,
which impacted testers who needed longer to complete
their selections. Because this was a test configuration
and not part of Secure Select 1.0, this is not an issue.
3. Unlabeled graphic - Regarding the language links, this is because it is
language choice. written using Chinese characters. This request is the
same as writing "Chinese" in English which would not be
accessible for Chinese users.
4. Tester logged out of The system cannot restrict printing, as there is no data
system and returned transmitting back to the server once the ballot marking
later to complete session starts. Jurisdictions have a system of “checks
ballot. and balances” in place to mitigate voters from voting
multiple times in a single election.
5. When scrolling down JAWS did tend to repeat the words "x checkboxes." The
the list of contests, 35 grouping was removed and a label for "x checkboxes"
is repeated for each was added before the set to remove that repetition for
selection. JAWS users. This was retested with IE, Edge, Firefox,
Safari, Chrome, Narrator+IE, NVDA + Firefox, Jaws
18+IE, Jaws 17+IE, Jaws 16+IE.
6. Tester used the page Animated scroll added to smooth the transition between
down button, which focus items. Tested with IE, Edge, Firefox, Safari,
appeared to display a Chrome, Narrator+IE, NVDA + Firefox, Jaws 18+IE,
“different screen”, Jaws 17+IE, Jaws 16+IE
causing some
disorientation.
SECRETARY OF STATE’S STAFF REPORT –Democracy Live Secure Select 1.0 5 | Page
7. The screen only If the keyboard is used to change focus, then scroll with
moved down when the mouse, select and option using the mouse, then go
used the arrows (or back to keyboard control it jumps back to the last
mouse) were used to keyboard position. Logic added to sync keyboard focus
advance it. This meant with mouse interaction. Tested with IE, Edge, Firefox,
that each time the Safari, Chrome, Narrator+IE, NVDA + Firefox, Jaws
tester jumped from 18+IE, Jaws 17+IE, Jaws 16+IE.
contest to contest, the
screen jumped
significantly, causing
them to lose their
place. The tester had
to scroll up several
times during the
propositions to
reorient.
8. When using NVDA, This is a known issue/functionality for NVDA
tester was unable to (https://github.com/nvaccess/nvda/issues/5742). NVDA
read the labels support mentions pressing tab to change focus does
associated with read the label correctly. Users can also press
checkboxes for the NVDA+Tab to read the label. This issue is also not
various contests when present in Firefox, which is NVDA's preferred browser.
using the arrow keys
to move the screen
reader’s virtual cursor
through the ballot.
9. Tester text description This is a known issue (marked as low priority) for NVDA
of a contest was + IE. https://github.com/nvaccess/nvda/issues/6639.
improperly labeled as This does not happen on any version of JAWS, or
“edit.” NVDA + Firefox.
10. The secure ballot QR Code library has been updated (which supports UTF
allows a person to put 8 characters).
in non-english
characters in the
write-in box, but if they
do, the barcode won’t
print correctly at the
end screen (and the
application becomes
mostly unusable after
the attempt to print).
11. “No Selections” could Changed to “Zero Selections Made.”
probably be
something that doesn’t
contain the word “No”
in it, as “No” is a valid
selection on many
choices. Maybe
SECRETARY OF STATE’S STAFF REPORT –Democracy Live Secure Select 1.0 6 | Page
“Selection Left Blank”
would be better.
SOS End User Usability - Phase II
Phase II of End User Usability testing was conducted by the SOS from August 18 to
August 24, 2017. Approximately nine (9) volunteers tested the Secure Select 1.0
system for usability and accessibility. During this phase of testing only two (2) issues
were identified amongst the users. Both were issues identified in Phase I of testing,
Items 1 and 8 from Table 3A. Both issues identified are not reflective of the Secure
Select 1.0 system.
Consultant Phase Accessibility, Usability, and Privacy Testing
SLI conducted three phases of testing for Accessibility, Usability and Privacy. Phase I
was a review of the Secure Select 1.0 documentation of usability and accessibility
testing performed during system development. Phase II included all Accessibility and
Usability testing. Phase III included Privacy testing.
SLI used the following tools in Table 3B to during testing to evaluate Secure Select 1.0.
Table 3B: Consultant Tools
Tool Name Tool Purpose Tool Use
Wave Web Tool to evaluate accessibility of web The vendor provided url’s
Accessibility sites, to WCAG 2.0 and Section 508. were run through MAUVE
evaluation tool to help determine
accessibility.
A11y.css Tool to evaluate web application The vendor provided url’s
regarding conformance to ISO 9241- were run with A11y to
171: Ergonomics of human-system help determine
interaction Guidance on software compliance.
accessibility.
Evaluera website Tool to evaluate web applications The vendor provided url’s
regarding conformance to WCAG 2.0 were run through 508
standards. checker to help
determine compliance.
Chromevox – Tool is a screen reader that is an Screen reader used for
Screen reader extension of Google Chrome, to Accessibility and Usability
extension of Google assist visually impaired users. testing.
Chrome.
SECRETARY OF STATE’S STAFF REPORT –Democracy Live Secure Select 1.0 7 | Page
Phase I Documentation Review
The documentation was reviewed by SLI to verify and validate the manufacturer’s
documentation of usability and accessibility performed during system development.
Review of the TDP validated that the requirement was satisfactorily covered.
Phase II-A Usability Testing
SLI evaluated Secure Select 1.0 against Section 3.2 et al of the CVSS. This section
covers the requirements for Usability of a system. SLI found that the vast majority of the
requirements were satisfactorily met. Secure Select 1.0 failed on two (2) of the
requirements. Each is listed in the table along with a response/mitigation:
Table 3C: Phase II-A – Usability Testing
CVSS Requirement Result Mitigation/Response
CVSS (3.2.8.e): Voter Secure Select Secure Select 1.0 cannot regulate a
Inactivity Time 1.0 does not voter’s inactivity, voting session time,
meet this or place any other time restraints on a
requirement. voter using the system. The system
CVSS (3.2.8.f): Alert Time – Secure Select concludes the session once the voter
Upon expiration of the voter 1.0 does not retrieves the ballot. Thus, there is no
inactivity time, the voting meet this longer any communication with the
system shall issue an alert requirement. system to place a time restriction of
and provide a means by which any sort on the voter.
the voter may receive
additional time. The alert time
shall be between 20 and 45
seconds. If the voter does not
respond to the alert within the
alert time, the system shall go
into an inactive state.
Phase II-B Accessibility Testing
SLI evaluated Secure Select 1.0 against the applicable portions of the CVSS for
Accessibility Testing for compliance. The table below shows each of the applicable
CVSS sections, and the result of compliance.
Table 3D: Phase II-B – Accessibility Testing
CVSS Requirement Result
CVSS (3.3.1.a.i.1): Accessibility throughout Review of the requirement validated that
the voting session - A VEBD shall be the requirement was satisfactorily
integrated into the manufacturer’s complete covered.
voting system so as to support accessibility
for disabled voters throughout the voting
SECRETARY OF STATE’S STAFF REPORT –Democracy Live Secure Select 1.0 8 | Page
session.
Documentation of Accessibility Procedures
- The manufacturer shall supply
documentation describing: recommended
procedures that fully implement
accessibility for voters with disabilities; and
CVSS (3.3.1.a.i.2): How a VEBD supports
those procedures.
CVSS (3.3.1.b) : Complete information in Review of the requirement validated that
alternative formats - When the provision of the requirement was satisfactorily
accessibility involves an alternative format covered.
for ballot presentation, then all information
presented to non-disabled voters, including
instructions, warnings, error and other
messages, and contest choices, shall be
presented in that alternative format
CVSS (3.3.1.d): Secondary means of voter Review of the requirement showed that
identification - If a voting system provides no biometric measures are required, so
for voter identification or authentication by this requirement is not applicable.
using biometric measures that require a
voter to possess particular biological
characteristics, then the system shall
provide a secondary means that does not
depend on those characteristics.
CVSS (3.3.1.e.i): Accessibility of paper- Review of the requirement validated that
based vote verification - If a VEBD the requirement was not applicable, as a
generates a paper record (or some other voter will implement their own hardware.
durable, human-readable record) for the If a voter has equipment that utilizes
purpose of allowing voters to verify their either OCR technology, or a QR reader
votes, then the system shall provide a with audio playback, then they would be
means to ensure that the verification record able to obtain an audio read back of their
is accessible to all voters with disabilities. printed ballot.
i. Audio readback for paper-based vote
verification - If a VEBD generates a paper
record (or some other durable, human-
readable record) for the purpose of allowing
voters to verify their votes, then the system
shall provide a mechanism that can read
that record and generate an audio
representation of its contents.
CVSS (3.3.2.c): Distinctive buttons and Review of the requirement validated that
controls - Buttons and controls on the requirement was not applicable, as a
accessible voting stations shall be voter will implement their own hardware.
distinguishable by both shape and color.
This applies to buttons and controls
implemented either "on-screen" or in
hardware. This requirement does not apply
SECRETARY OF STATE’S STAFF REPORT –Democracy Live Secure Select 1.0 9 | Page
to sizeable groups of keys, such as a
conventional 4x3 telephone keypad or a full
alphabetic keyboard.
CVSS (3.3.2.d): Synchronized audio and Review of the requirement validated that
video - The voting station shall provide the requirement was not applicable, as a
synchronized audio output to convey the voter will implement their own hardware.
same information as that which is displayed Synchronized audio/video, video-only, or
on the screen. There shall be a means by audio only are all obtainable. If voter
which the voter can disable either the audio environment contains a screen, audio
or the video output, resulting in a video-only output and a screen reader, synchronized
or audio-only presentation, respectively. output is available. Voter can turn off
The system shall allow the voter to audio to have video only. Likewise voter
switch among the three modes can turn off video display to have Audio
(synchronized audio/video, video-only, or only.
audio-only) throughout the voting session
while preserving the current votes.
CVSS (3.3.3.b.1): Audio-tactile interface - Review of the requirements validated that
The accessible voting station shall provide these requirements were not applicable,
an audio-tactile interface (ATI) that supports as voter will implement their own
the full functionality of the visual ballot hardware.
interface. Full functionality includes at a
minimum:
o Instructions and feedback on initial
activation of the ballot
(such as insertion of a smart card), if
applicable;
CVSS (3.3.3.b.2): Instructions and
feedback to the voter on how to operate the
accessible voting station, including settings
and options (e.g.,
volume control, repetition);
CVSS (3.3.3.b.3): Instructions and
feedback for navigation of the ballot;
CVSS (3.3.3.b.4): Instructions and
feedback for contest choices, including
write-in candidates;
CVSS (3.3.3.b.5): Instructions and
feedback on confirming and changing
votes;
CVSS (3.3.3.b.6): Instructions and
feedback on final submission of ballot.
CVSS (3.3.3.c.vii): Audio features and Review of the requirements validated that
characteristics - Voting stations that provide these requirements were not applicable,
SECRETARY OF STATE’S STAFF REPORT –Democracy Live Secure Select 1.0 10 | Page
audio presentation of the ballot shall do so as voter will implement their own
in a usable way, as detailed in the following hardware.
sub-requirements.
iv. Intelligible audio - The audio
presentation of verbal information by both
recorded and synthetic speech shall be
readily comprehensible by voters who have
normal hearing and are proficient in the
language. This includes such
characteristics as proper enunciation,
normal intonation, appropriate rate of
speech, and low background noise.
Candidate names shall be pronounced as
the candidate intends. This requirement
applies to those aspects of the audio
content that are inherent to the voting
system or that are generated by default.
CVSS (3.3.3.d): Ballot activation - If the Review of the requirement validated that
voting station supports ballot activation for the requirement was satisfactorily
nonblind voters, then it shall also provide covered.
features that enable voters who are blind to
perform this activation.
CVSS (3.3.4.b): Support for non-manual Review of the requirements validated that
input - The accessible voting station shall these requirements were not applicable,
provide a mechanism to enable non-manual as a voter will implement their own
input that is functionally equivalent to tactile hardware.
input. All the functionality of the accessible
voting station (e.g., straight party voting,
write-in candidates) that is available
through the conventional forms of input,
such as tactile, shall also be available
through non-manual input mechanisms
such as mouth sticks and "sip and puff"
switches.
CVSS (3.3.6.a): Reference to audio Review of the requirements validated that
requirements - The accessible voting these requirements were not applicable,
station shall incorporate the features listed as voter will implement their own
under the requirements for voting hardware.
equipment that provides audio presentation
of the ballot.
CVSS (3.3.6.b): Visual redundancy for Review of the requirement validated that
sound cues - If the voting system provides the requirement was satisfactorily
sound cues as a method to alert the voter, covered.
the tone shall be accompanied by a visual
cue, unless the station is in audio-only
mode.
CVSS (3.3.7): Use of ATI- For voters who Review of the requirements validated that
SECRETARY OF STATE’S STAFF REPORT –Democracy Live Secure Select 1.0 11 | Page
lack proficiency in reading English, the this requirement was not applicable, as a
voting equipment shall provide an audio voter will implement their own hardware.
interface for instructions and ballots.
CVSS (3.3.8): Speech not to be required by Review of the requirement validated that
equipment - Voting equipment shall not the requirement was satisfactorily
require voter speech for its operation. covered.
Phase III Privacy Testing
SLI evaluated Secure Select 1.0, for compliance with California Elections Code
requirements for privacy within an RAVBM system, in addition to the applicable portions
of the CVSS. The table below shows each of the applicable code sections and CVSS
sections, and the result of compliance.
Table 3D: Phase III – Privacy Testing
Applicable California Elections Code Result
Section & CVSS Requirement
EC Section 19295(a): The RAVBM shall not Review of the requirement validated
have the capability, including an optional that the requirement was satisfactorily
capability, to use a remote server to mark a covered.
voter’s selections transmitted to the server
from the voter’s computer via the Internet.
EC Section 19295 (b): The RAVBM shall not Review of the requirement validated
have the capability, including an optional that the requirement was satisfactorily
capability, to store any voter identifiable covered.
selections on any remote server.
EC Section 19295(c): The RAVBM shall not Review of the requirement validated
have the capability, including the optional that the requirement was satisfactorily
capability, to tabulate votes. covered.
CVSS (3.2.4.1.a): Visual privacy - The ballot, Review of the requirement validated
any other visible record containing ballot that the requirement was satisfactorily
information, and any input controls shall be covered, as the voter will utilize their
visible only to the voter during the voting own hardware in the environment of
session and ballot submission. their choosing.
CVSS (3.2.4.1.b): Auditory privacy - During Review of the requirement validated
the voting session, the audio interface of the that the requirement was satisfactorily
voting system shall be audible only to the covered, as the voter will utilize their
voter. own hardware in the environment of
their choosing.
CVSS (3.2.4.1.c): Privacy of warnings - The Review of the requirement validated
voting system shall issue all warnings in a that the requirement was satisfactorily
way that preserves the privacy of the voter covered, as the voter will utilize their
and the confidentiality of the ballot. own hardware in the environment of
their choosing.
CVSS (3.2.4.1.d): No receipts - The voting Review of the requirements validated
system shall not issue a receipt to the voter that this requirement was not
SECRETARY OF STATE’S STAFF REPORT –Democracy Live Secure Select 1.0 12 | Page
that would provide proof to another of how the applicable, as voter will implement their
voter voted. own hardware, and print their own
marked ballot, as a RAVBMS system.
CVSS (3.2.4.2.a): No information shall be Review of the requirement validated
kept within an electronic CVR that identifies that the requirement was satisfactorily
any alternative language feature(s) used by a covered, as CVR’s are not created.
voter.
CVSS (3.2.4.2.b): No information shall be Review of the requirement validated
kept within an electronic CVR that identifies that the requirement was satisfactorily
any accessibility feature(s) used by a voter. covered, as CVR’s are not created.
The Secure Select 1.0 systems allows for voter privacy. This is achieved by removing
client side storage of marked selections. A voter is able to verify and print the paper
cast record for submission to a local elections official.
4. Security and Telecommunications Testing Summary
SLI Compliance performed Security and Telecommunications testing on the Secure
Select 1.0 RAVBMS. During the testing, SLI conducted a documentation review,
functional security testing, and telecommunications and data transmission testing. A
summary of each phase and the results are listed below.
Documentation Review
SLI conducted a documentation review of the TDP provided by Democracy Live for the
Secure Select 1.0 RAVBMS. SLI evaluated the documentation for the following
applicable portions of the CVSS:
Table 4A: Documentation Review
CVSS Standard Result
5.5 Vote Secrecy on DRE and EBM Review of the TDP validated that the
Systems requirement was satisfactorily covered.
6.1.2 Data Transmissions Review of the TDP validated that the
requirement was satisfactorily covered.
6.2 Design, Construction, and Review of the TDP validated that the
Maintenance Requirements requirement was satisfactorily covered.
6.2.1 Confirmation Review of the TDP validated that the
requirement was satisfactorily covered.
7.2 Access control Review of the TDP validated that the
requirement was satisfactorily covered.
7.2.1 General Access Control Review of the TDP validated that the
requirement was satisfactorily covered.
7.2.2 General Access Control Review of the TDP validated that the
requirement was satisfactorily covered.
7.4.5 Software Reference Information Review of the TDP validated that the
requirement was satisfactorily covered.
SECRETARY OF STATE’S STAFF REPORT –Democracy Live Secure Select 1.0 13 | Page
7.4.6 Software Setup Validation Review of the TDP validated that the
requirement was satisfactorily covered.
Functional Security
SLI next conducted functional security testing as part of the Security and
Telecommunications testing on the Secure Select 1.0 RAVBMS. The testing consisted
of 1) testing the relevant software and operating security system configuration, for
pertinent vulnerabilities and 2) testing of hardware, including examination of unused
hardware ports, and security measures applied to those ports. Table 4B: Functional
Security, lists the applicable CVSS Standards, the testing performed, any
vulnerabilities, along with the vendor response/mitigation when applicable.
Table 4B: Functional Security
CVSS Standard Testing Vulnerability Recommended Response
Performed Mitigation
5.5 Vote Secrecy Testing was N/A N/A N/A
on DRE and EBM performed to verify
Systems how the system
handled a ballot
being printed and
the browser
closed, as well as
when the ballot is
closed prior to
being printed.
Attempts were
made to resume a
ballot, as well as
to determine if any
ballot information
resided in history
or cache.
7.2.1 General Testing was Possible Recommended Democracy
Access Control performed of attack vector mitigation is to Live server
multiple server would be minimize users administrat
setups which whitelisting a and rights to ors are in
included an commonly web server, as control of
unrestricted test used JSON well as to whitelisting
setup, and a data repository monitor JSON domains
domain whitelist domain. So files and server allowed to
protected system such domain audit logs as host ballot
to verify that the whitelisting continuously as definition
whitelisting of should remain possible, while files. By
JavaScript Object jurisdiction or the web server default,
SECRETARY OF STATE’S STAFF REPORT –Democracy Live Secure Select 1.0 14 | Page
Notation (JSON) vendor is running. only
domains controlled Democracy
successfully specific Live owned
blocks load of domains
JSON files from will be
an invalid domain. allowed to
host ballot
definition
files. If a
county
requires
county
hosted
ballot
definition
files, only
domains
owned by
counties
will be
added to
the
whitelist.
No public
or
commonly
used
JSON
repositorie
s will be
added to
the
whitelist at
any time.
7.2.2 Access Testing was N/A N/A N/A
Control performed to verify
Identification the system’s
ability to
correctly process
invalid data
domain white
listings. Two
different test
systems
were tested in
order to verify that
an open system
accepts
SECRETARY OF STATE’S STAFF REPORT –Democracy Live Secure Select 1.0 15 | Page
JavaScript Object
Notation (JSON)
ballot definition
files from
anywhere, and
that the locked
down
SS-CA application
successfully
blocks non-
whitelisted
domains.
7.2.4 Access Testing was N/A N/A N/A
Control performed to verify
Authorization the systems
Invalid data
domain
whitelisting. Two
different test
systems to verify
that an open
system accepts
JSON ballot
definition files from
anywhere, and
that the locked
down SSCA
application
successfully
blocks non-
whitelisted
domains.
7.4.5 Software Testing was Testing was N/A Democracy
Reference performed to unable to find Live
Information confirm that the a method of confirms
Secure Select 1.0 successfully that the
system contains a modifying the hash code
Verification URL server code to verification
that contains a verify if the page does,
Hash code Value protection in fact,
that can be method was change
checked to verify if viable. In with any
the source code theory the change to
has been functionality is the
modified. there however codebase.
at this point
testing is
unable to
SECRETARY OF STATE’S STAFF REPORT –Democracy Live Secure Select 1.0 16 | Page
verify validity.
7.4.6 Software Testing was N/A N/A N/A
Setup Validation performed to
confirm that the
Secure Select 1.0
system contains a
Verification URL
that contains a
Hash code Value
that can be
checked to verify if
the source code
has been
modified.
7.6 Tests were N/A N/A N/A
Telecommunicatio performed to verify
ns and Data that if the system
Transmission utilizes electrical
or optical
transmission, that
the ballot was sent
via SSL, no
receipt is utilized
to verify. What is
sent is a blank
ballot, that does
not contain any
voter data or
voting selections.
7.8 Testing – Confirmed that N/A N/A N/A
Security Secure Select 1.0
does not have, nor
require, internet
access once the
ballot has been
downloaded.
There are no
external
connections from
the ballot to any
outside server or
service. With the
exception of
sending the ballot
to a connected
printer to be
printed, there are
SECRETARY OF STATE’S STAFF REPORT –Democracy Live Secure Select 1.0 17 | Page
no external
connections to or
from the ballot.
7.8.1 Access Testing was N/A N/A N/A
Control performed to
confirm that
access control
was maintained.
Attempts to attack
the system
included XSS,
SQL Injection,
Directory listings,
and http login
pages, as well as
SSL certificate
information.
7.8.2 Data Verified that this N/A N/A N/A
Interception and system does not
Disruption utilize
telecommunication
s for the
transmission of
official voting data
– only the delivery
of a blank ballot
that does not
contain voter data
or choice
selections.
Telecommunications and Data Transmission Testing
Finally in conducting the Security and Telecommunications testing on the Secure Select
1.0 RAVBMS, SLI performed telecommunications and data transmission testing. SLI
tested the communication of the Secure Select 1.0 system, including encryption of data,
as well as protocols and procedures for access authorization. The following table,
Table 4C: Telecommunications and Data Transmission Testing, outlines the testing
performed, vulnerabilities, and the vendor response/mitigation when applicable.
Table 4C: Telecommunications and Data Transmission Testing
CVSS Testing Performed Vulnerability Response/Mitigation
Standard
6.1.2 Data Testing included N/A N/A
Transmission transmissions of the Secure
Select 1.0 SPA (Single Page
SECRETARY OF STATE’S STAFF REPORT –Democracy Live Secure Select 1.0 18 | Page
Application) voting ballot that
is served from a Hosted
webserver, to the voter. Scans
were performed to determine
if there were any basic web
server vulnerabilities in the
initial serving of the browser
application that houses the
Secure Select 1.0 ballot. None
were found.
6.2. Design, Verified that Secure Select 1.0 N/A N/A
Construction, system consists of a SPA
and format ballot which is typically
Maintenance used for absentee and mail-in
Requirements ballot marking. Secure Select
1.0 does not utilize specific
telecommunications channels
once the ballot has been
downloaded and opened on
the end voter’s machine.
6.2.1 Testing included accessing of N/A N/A
Confirmation a ballot from the host and
checking for confirmation.
This requirement was
determined to be not
applicable, as the system only
allows the voter to mark and
confirm marked ballots prior to
printing out a ballot summary
card. There are no live
connections from the
application to a remote server.
All selections are cleared after
browser has been closed.
Security and Telecommunications Results
SLI found no discrepancies within Democracy Lives’ Secure Select 1.0 RAVBMS.
SLI did identify two potential vulnerabilities, as listed in Table 4B.
SECRETARY OF STATE’S STAFF REPORT –Democracy Live Secure Select 1.0 19 | Page
5. Software Review Testing Summary
SLI Compliance performed a review of the Secure Select 1.0 software. The purpose of
the review was to identify discrepancies within the software code, and compliance with
the California Voting System Standards (CVSS). The results and vendor
mitigation/response are listed in the following table:
Table 5A: Source Code Review Discrepancy and Mitigation/Response
Discrepancy Vendor Mitigation/Response
5.2.7.e: Dead Code – Two Secure Select has been developed with
discrepancies written for the “Dead multiple return statements in some functions
Code” (CVSS 5.2.7.e) requirement as a method of defensive programming to
found in the source code base keep application logic clear and concise.
reviewed, as a result, two Democracy Live believes these discrepancies
discrepancies were written against the are acceptable for the following reasons.
code base. 1. Using multiple return statements to avoid
complicated nesting logic is considered a best
practice.
2. The files identified to not actually have any
“dead code.” All code is executable depending
on logic conditions.
3. CVSS specifies it is ok to use multiple return
statements in a function when using “defensive
programming.”
5.2.6.a-h: Sufficient Header Header comments are used to document and
Comments – Three discrepancies explain functionality of code. Democracy Live
written for the “Sufficient Header agrees additional comments can be added to
Comments” (CVSS 5.2.6.a-h) the codebase to document all functions and is
requirement found in the source code willing to include these comments in the next
base reviewed, as a result, three release of Secure Select.
discrepancies were written against the
code base.
IV. COMPLIANCE WITH STATE AND FEDERAL LAWS AND
REGULATIONS
1. Elections Code Review
1) §19293 (b) Remote accessible vote by mail system standards adopted by the
Secretary of State pursuant to subdivision (a) shall include, but not be limited to,
all of the following requirements:
(1) The machine or device and its software shall be suitable for the purpose for
which it is intended.
(2) The remote accessible vote by mail system shall preserve the secrecy of the
ballot.
SECRETARY OF STATE’S STAFF REPORT –Democracy Live Secure Select 1.0 20 | Page
(3) The remote accessible vote by mail system shall be safe from fraud or
manipulation.
(4) The remote accessible vote by mail system shall be accessible to voters with
disabilities and to voters who require assistance in a language other than English
if the language is one in which a ballot or ballot materials are required to be
made available to voters.
The system meets these requirements. The system demonstrated English,
Spanish and Chinese languages, however with external translation services; the
system can support other languages as required.
2) §19295 A remote accessible vote by mail system or part of a remote
accessible vote by mail system shall do not any of the following:
(a) Have the capability, including an optional capability, to use a remote
server to mark a voter’s selections transmitted to the server from the voter’s
computer via the Internet.
(b) Have the capability, including an optional capability, to store any voter
identifiable selections on any remote server.
(c) Have the capability, including the optional capability, to tabulate votes.
The system meets these requirements.
2. Section 508 and WCAG Compliance Review
Secure Select 1.0 was written to WCAG 2.0 guidelines to implement
accessibility features. The system also complies with the applicable portions
of Section 508.
V. Conclusion
The Democracy Live Secure Select 1.0 remote accessible vote by mail
system, in the configuration tested and documented by the California
Democracy Live Secure Select 1.0 Use Procedures, meets all applicable
California and federal laws. The Democracy Live Secure Select 1.0 remote
accessible vote by mail system is compliant with all California and federal
laws.
SECRETARY OF STATE’S STAFF REPORT –Democracy Live Secure Select 1.0 21 | Page
ATTACHMENT A
SECRETARY OF STATE’S STAFF REPORT –Democracy Live Secure Select 1.0 22 | Page
Democracy Live Secure Select 1.0 Pre-Test Survey
Q1 Name
Answered: 18 Skipped: 1
# RESPONSES DATE
Q2 Age
Answered: 19 Skipped: 0
# RESPONSES DATE
1 / 13
Democracy Live Secure Select 1.0 Pre-Test Survey
Q3 Phone
Answered: 19 Skipped: 0
# RESPONSES DATE
Q4 Email
Answered: 19 Skipped: 0
2 / 13
Democracy Live Secure Select 1.0 Pre-Test Survey
Q5 Please describe the disabilities or difficulties you have that may
interfere with you being able to mark a ballot privately and independently.
Answered: 18 Skipped: 1
# RESPONSES DATE
1 I am totally blind. 6/18/2017 2:44 PM
2 I work with the SOS on accessibility matters at the polling place and have been asked to assist. 6/15/2017 7:42 PM
3 Totally blind 6/15/2017 5:04 PM
4 I am almost totally blind, and cannot see ballots. 6/14/2017 4:49 PM
5 I am blind and do not trust accessible voting machines so would prefer to continue voting by 6/14/2017 12:19 PM
absentee ballot. I am unable to fill out an absentee ballot independently because it requires sight.
6 Cerebral Palsy, which impacts my fine motor skills and speech. 6/14/2017 10:35 AM
7 legal blindness, can not read printed material 6/14/2017 10:02 AM
8 I have a physical disability and use a motorized scooter. 6/14/2017 9:58 AM
9 blindness 6/14/2017 6:41 AM
10 blind 6/14/2017 4:18 AM
11 Optic nerve hypoplasia 6/14/2017 12:12 AM
12 Totally blind. Also, can't do stairs. Would like the ability to vote at home which most others seem to 6/13/2017 11:54 PM
take blithely for granted. The only means of "absentee voting" politicians and bureaucrats have
seemed to come up with, so far, to make that happen is sending print ballots and calling the case
closed. This is WAY overdue. Thanks for finally getting there!
13 I am totally blind in both eyes without any visual acuity at all, not even light perception. 6/13/2017 9:01 PM
14 I am totally blind and cannot read the print ballot or determine which holes pertain to which answer 6/13/2017 8:20 PM
I wish to mark.
3 / 13
Democracy Live Secure Select 1.0 Pre-Test Survey
15 I am totally blind and cannot read the ballot or mark the ballot independently. It is difficult to get to 6/13/2017 7:49 PM
my poling place so I have my husband mark my absentee ballot.
16 Legally blind for the past 16 years 6/13/2017 7:09 PM
17 totally blind 6/13/2017 7:07 PM
18 I am blind / visually impaired 6/13/2017 7:02 PM
Q6 Which language would you prefer voting in ?
Answered: 18 Skipped: 1
English
Spanish
Chinese
Japanese
Tagalog
Thai
Vietnamese
Hindi
Khmer
Korean
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
ANSWER CHOICES RESPONSES
English 100.00% 18
Spanish 0.00% 0
Chinese 0.00% 0
Japanese 0.00% 0
Tagalog 0.00% 0
Thai 0.00% 0
Vietnamese 0.00% 0
Hindi 0.00% 0
4 / 13
Democracy Live Secure Select 1.0 Pre-Test Survey
Khmer 0.00% 0
Korean 0.00% 0
TOTAL 18
# OTHER (PLEASE SPECIFY) DATE
1 English 6/13/2017 7:09 PM
Q7 Have you ever voted before?
Answered: 18 Skipped: 1
Yes
No
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
ANSWER CHOICES RESPONSES
Yes 100.00% 18
No 0.00% 0
TOTAL 18
Q8 If "No", what has prevented you from voting in the past?
Answered: 0 Skipped: 19
# RESPONSES DATE
There are no responses.
Q9 What method of voting do you use most often?
Answered: 18 Skipped: 1
5 / 13
Democracy Live Secure Select 1.0 Pre-Test Survey
Polls
By Mail
Early Voting
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
ANSWER CHOICES RESPONSES
Polls 66.67% 12
By Mail 33.33% 6
Early Voting 0.00% 0
TOTAL 18
# OTHER (PLEASE SPECIFY) DATE
1 I sometimes vote early, since I'm not sure my polling place is accessible. 6/13/2017 11:54 PM
2 When voting by mail, someone must mark the ballot and hope that my signature matches the 6/13/2017 9:01 PM
signature on file.
3 accessable voting machine 6/13/2017 7:02 PM
Q10 Have you ever used a voting system with any special accessibility
accommodations?
Answered: 19 Skipped: 0
Yes
No
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
ANSWER CHOICES RESPONSES
Yes 89.47% 17
6 / 13
Democracy Live Secure Select 1.0 Pre-Test Survey
No 10.53% 2
TOTAL 19
Q11 If "Yes", which type of voting system?
Answered: 17 Skipped: 2
Electronic
Ballot Marking
Device
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
ANSWER CHOICES RESPONSES
Electronic 82.35% 14
Ballot Marking Device 17.65% 3
TOTAL 17
# OTHER (PLEASE SPECIFY) DATE
1 both DRE and BMD 6/15/2017 5:04 PM
2 electronic voting machine with talking menus 6/14/2017 12:19 PM
3 Being assisted by poll worker or a friend on a mail in vote. 6/14/2017 11:21 AM
4 Verbal machine with headphones 6/13/2017 7:09 PM
5 accessible voting machine for the blind and visually impaired 6/13/2017 7:02 PM
Q12 What operating system are you using?
Answered: 19 Skipped: 0
7 / 13
Democracy Live Secure Select 1.0 Pre-Test Survey
Windows 10
Windows 8.1
Windows 7
macOS 10.12
Android
Linux
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
ANSWER CHOICES RESPONSES
Windows 10 36.84% 7
Windows 8.1 5.26% 1
Windows 7 47.37% 9
macOS 10.12 10.53% 2
Android 0.00% 0
Linux 0.00% 0
TOTAL 19
# OTHER (PLEASE SPECIFY) DATE
1 iOS 6/14/2017 11:27 AM
2 IOS 9.2 on my iphone 6/14/2017 12:25 AM
3 Home premium edition 6/14/2017 12:01 AM
Q13 What internet browser are you using?
Answered: 18 Skipped: 1
8 / 13
Democracy Live Secure Select 1.0 Pre-Test Survey
Microsoft Edge
14
Internet
Explorer 11
Safari 10.1
Google Chrome
59.0
Mozilla Firefox
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
ANSWER CHOICES RESPONSES
Microsoft Edge 14 0.00% 0
Internet Explorer 11 72.22% 13
Safari 10.1 11.11% 2
Google Chrome 59.0 5.56% 1
Mozilla Firefox 11.11% 2
TOTAL 18
# OTHER (PLEASE SPECIFY) DATE
1 firefox 6/14/2017 11:27 AM
2 Internet Explorer 9? 6/14/2017 10:10 AM
3 with most recent updates 6/14/2017 12:01 AM
Q14 List the auxiliary devices and/or software you are using for electronic
accessibility.
Answered: 14 Skipped: 5
9 / 13
Democracy Live Secure Select 1.0 Pre-Test Survey
JAWS
NVDA
Apple Voiceover
Windows
Narrator
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
ANSWER CHOICES RESPONSES
JAWS 64.29% 9
NVDA 7.14% 1
Apple Voiceover 21.43% 3
Windows Narrator 7.14% 1
TOTAL 14
# OTHER (PLEASE SPECIFY) DATE
1 System Access 6/18/2017 2:48 PM
2 None. 6/15/2017 7:45 PM
3 Window-Eyes 6/15/2017 5:07 PM
4 Focus 80 braille display 6/14/2017 12:26 PM
5 I also use an iPhone with VoiceOver enabled. 6/14/2017 11:27 AM
6 Mouse keys, filter keys and sticky keys (accessibility features in Windows 7 (and 10). 6/14/2017 10:50 AM
7 Also have access to Window-Eyes and System Access. Can download JAWS at great need. Can 6/14/2017 12:01 AM
also test on Samsung Galaxy S6 w Google TalkBack w optional Bluetooth keyboard if desired.
8 Jws for Windows for speech and a Power Braile forty on a Dell Desktop computer. 6/13/2017 9:06 PM
9 iPad 6/13/2017 7:20 PM
10 Window-Eyes 6/13/2017 7:14 PM
11 JAWS 18 6/13/2017 7:10 PM
Q15 What is the make and model of the hardware you are using (PC,
Laptop, Tablet, etc.)?
Answered: 19 Skipped: 0
10 / 13
Democracy Live Secure Select 1.0 Pre-Test Survey
# RESPONSES DATE
1 Nenovo IdeaPad 6/18/2017 2:48 PM
2 HP Pro Desk 6/15/2017 7:45 PM
3 desktop PC 6/15/2017 5:07 PM
4 labtop 6/14/2017 4:51 PM
5 Dell desktop PC, Optiplex 7010 6/14/2017 12:26 PM
6 Sony Vio computer. Its a PC using windows 7. 6/14/2017 11:27 AM
7 Dell Desktop PC 6/14/2017 10:50 AM
8 Lenovo desktop 6/14/2017 10:10 AM
9 Dell PC 6/14/2017 10:03 AM
10 pc 6/14/2017 6:41 AM
11 Apple MacBook Pro 6/14/2017 4:23 AM
12 iPhone 5s 6/14/2017 12:25 AM
13 Dell OptiPlex 755 (PC) 6/14/2017 12:01 AM
14 This is a Dell PC, desktop purchased in 2011. 6/13/2017 9:06 PM
15 Dell Laptop PC 6/13/2017 8:24 PM
16 Dell Optiplex 3010 6/13/2017 7:53 PM
17 PC And tablet 6/13/2017 7:20 PM
18 PC, custom, from Standard computers, APX 64-based system, 64-bit 6/13/2017 7:14 PM
19 Dell PC 6/13/2017 7:10 PM
Q16 What is the printer make and model you are using?
Answered: 19 Skipped: 0
# RESPONSES DATE
1 HP DeskJet 6/18/2017 2:48 PM
2 Xerox Copier - model unknown 6/15/2017 7:45 PM
3 Canon MX922 6/15/2017 5:07 PM
4 I do not know 6/14/2017 4:51 PM
5 HP Laserjet P3005DN 6/14/2017 12:26 PM
6 HP Lazar jet 1102 6/14/2017 11:27 AM
7 HP Laserjet P3011 6/14/2017 10:50 AM
8 HP photosmart C4400 series 6/14/2017 10:10 AM
9 HP 6/14/2017 10:03 AM
10 HP 6/14/2017 6:41 AM
11 Brother multifunction printer 6/14/2017 4:23 AM
12 Epsom 6/14/2017 12:25 AM
13 none 6/14/2017 12:01 AM
14 This printer is a Laser Jet Hp Printer. 6/13/2017 9:06 PM
15 HP Monochrome Laser HP1102 Professional 6/13/2017 8:24 PM
11 / 13
Democracy Live Secure Select 1.0 Pre-Test Survey
16 HP Laser Jet P1505n 6/13/2017 7:53 PM
17 Panasonic 550 6/13/2017 7:20 PM
18 HP Laserjet P2015 6/13/2017 7:14 PM
19 HP laser professional 1102? 6/13/2017 7:10 PM
Q17 What antivirus software are you using?
Answered: 12 Skipped: 7
McAfee
Norton
Windows
Defender
Kaspersky
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
ANSWER CHOICES RESPONSES
McAfee 16.67% 2
Norton 16.67% 2
Windows Defender 58.33% 7
Kaspersky 8.33% 1
TOTAL 12
# OTHER (PLEASE SPECIFY) DATE
1 Other MS 6/15/2017 5:07 PM
2 Microsoft security essentials 6/14/2017 11:27 AM
3 Malware Bytes 6/14/2017 10:10 AM
4 nod32 6/14/2017 6:41 AM
5 don't know 6/14/2017 4:23 AM
6 Not sure of my antivirus software. Nor do I know whether or not I have antivirus software 6/14/2017 12:25 AM
7 Microsoft Security essentials? 6/14/2017 12:01 AM
8 Microsoft Security Essentials. 6/13/2017 9:06 PM
9 Microsoft Security 6/13/2017 7:53 PM
12 / 13
Democracy Live Secure Select 1.0 Pre-Test Survey
10 AT&T 6/13/2017 7:20 PM
11 eSet Nod32 6/13/2017 7:14 PM
12 ESET NOD32 Antivirus. 6/13/2017 7:10 PM
Q18 Is your antivirus software up to date?
Answered: 19 Skipped: 0
Yes
No
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
ANSWER CHOICES RESPONSES
Yes 89.47% 17
No 10.53% 2
TOTAL 19
13 / 13
Democracy Live Secure Select 1.0 Post-Test Survey
Q1 For each question, please indicate how strongly you agree or
disagree with the statement.
Answered: 13 Skipped: 0
The marking of
my ballot...
I feel I can
use this sys...
I am confident
that my ball...
The
instructions...
The system was
easy to use.
I could read
the display...
I could
understand t...
The time frame
it took to m...
I can use this
system witho...
Printing was
easy.
My ballot
printed with...
0 1 2 3 4 5 6 7 8 9 10
AGREE AGREE NEUTRAL DISAGREE DISAGREE TOTAL WEIGHTED
STRONGLY SOMEWHAT SOMEWHAT STRONGLY AVERAGE
The marking of my ballot method 61.54% 23.08% 15.38% 0.00% 0.00%
was private. 8 3 2 0 0 13 1.54
I feel I can use this system to 75.00% 8.33% 8.33% 8.33% 0.00%
mark my ballot independently. 9 1 1 1 0 12 1.50
I am confident that my ballot was 58.33% 25.00% 16.67% 0.00% 0.00%
recorded accurately. 7 3 2 0 0 12 1.58
The instructions were clear and 54.55% 18.18% 0.00% 0.00% 27.27%
complete. 6 2 0 0 3 11 2.27
The system was easy to use. 50.00% 33.33% 8.33% 0.00% 8.33%
6 4 1 0 1 12 1.83
I could read the display easily.(If 66.67% 0.00% 16.67% 0.00% 16.67%
applicable) 4 0 1 0 1 6 2.00
1 / 6
Democracy Live Secure Select 1.0 Post-Test Survey
I could understand the speech 80.00% 10.00% 10.00% 0.00% 0.00%
output. (If applicable) 8 1 1 0 0 10 1.30
The time frame it took to mark my 54.55% 18.18% 0.00% 9.09% 18.18%
ballot was what I expected. 6 2 0 1 2 11 2.18
I can use this system without 54.55% 27.27% 0.00% 0.00% 18.18%
technical help. 6 3 0 0 2 11 2.00
Printing was easy. 63.64% 9.09% 18.18% 0.00% 9.09%
7 1 2 0 1 11 1.82
My ballot printed with the correct 58.33% 16.67% 16.67% 0.00% 8.33%
selections. 7 2 2 0 1 12 1.83
2 / 6
Democracy Live Secure Select 1.0 Post-Test Survey
Q2 Would you be satisfied using this system to mark your ballot in an
election?
Answered: 13 Skipped: 0
Yes
No
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
ANSWER CHOICES RESPONSES
Yes 46.15% 6
No 53.85% 7
TOTAL 13
3 / 6
Democracy Live Secure Select 1.0 Post-Test Survey
Q3 Would you rather mark your ballot using another method?
Answered: 11 Skipped: 2
Yes
No
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
ANSWER CHOICES RESPONSES
Yes 45.45% 5
No 54.55% 6
TOTAL 11
4 / 6
Democracy Live Secure Select 1.0 Post-Test Survey
Q4 If you prefer another method, what method would you prefer?
Answered: 8 Skipped: 5
# RESPONSES DATE
1 LA County BMD 7/18/2017 9:02 PM
2 I don't mind this system, but want to be able to e-mail my ballot and I couldn't successfully 6/19/2017 4:41 PM
download the instructions for that part of the process.
3 Electronic voting at polls. Does not require my use of, or interaction with print. If I must mail 6/18/2017 4:51 AM
something in print, it requires a sighted person. Independence lost! It's like being thrown back
before women's suffrage or the advent of black folks being able to vote.
4 N/A 6/17/2017 9:25 PM
5 The current vote-by-mail process is more accessible and easier to complete. 6/16/2017 3:00 PM
6 I'd prefer a touch tone phone as an alternative to be available for those that are not blessed to 6/16/2017 1:36 PM
have or be able to use a computer.
7 Voting by touch tone phone would also be great! 6/16/2017 1:14 PM
8 I usually vote absentee, but need a sighted person's help 6/16/2017 9:43 AM
5 / 6
Democracy Live Secure Select 1.0 Post-Test Survey
Q5 Do you have any suggestions for changes on this system and/or any
other comments you would like to provide?
Answered: 12 Skipped: 1
# RESPONSES DATE
1 I will send my suggestions to the Sec of State's office separately. 7/18/2017 9:02 PM
2 Voter should have to enter his/her address manually; too easy to fake. 7/5/2017 11:12 AM
3 The download appeared to be a PDF document and that is apparently what precluded me from a 6/19/2017 4:41 PM
successful download. It would be better to have a document integrated into the system to show
how to e-mail back the document.
4 If I had to mail it I would have an issue in getting my envelope address. other than that the system 6/18/2017 8:34 PM
was great
5 I'm disappointed that the paranoya and fear mongering of those who are terrified that their votes 6/18/2017 4:51 AM
will be misappropriated is allowing mine to be negated. Further, the system in question must be
adjusted to be truly accessible, which IT IS NOT currently. It is clunky at best. The first question of
this survey was whether I believed the MARKING of my ballot was private. The marking of the
ballot itself is private and allows for independence, but if you force print down our throats, you
negate those two very important things which are JUST AS important as a vote's lack of
corruptability.
6 The time limit for filling out the ballot needs to be increased. It took me four tries to fill in my 6/17/2017 9:25 PM
general election ballot because the system kept timing me out and kicking me back to the start of
the process. Also, there is a problem with the language selection buttons on the top of the pages.
The first language choice is an unlabeled graphic. It needs alt text to allow a screen reader user to
know what language is being indicated.
7 Simplify the process for printing out the ballot and mailing it. The sequence of instructions seems 6/16/2017 3:00 PM
out of order. It would be better to have you complete your selections first, then go through the
process of printing out the materials to mail your ballot back in.
8 In the initial instructions, needing to have a scanner was not made clear. Actually, the initial survey 6/16/2017 1:36 PM
asked about operating system, computing device, and printer, but there was no mention of a
scanner or ocr program.
9 Instructions were not clear on how to submit the ballot after printing. We do have both a scanner 6/16/2017 1:14 PM
and printer, but are unable to verify if the method we chose to send it by email was the correct
method, a pdf image. This system would be great if our ballot could just be transmitted without
having to scan a printed copy.
10 I wasn't able to log out of one of the ballots. When I did log out, I went back in, and it did not tell me 6/16/2017 10:14 AM
that I had already voted. I'm concerned about that. I'm thinking it was because it was a test, and
that if I really had voted it would not let me vote again.
11 The printing process which I chose to implement as a saved PDF had some issues but they may 6/16/2017 9:44 AM
have been on my end, i.e. Office13 and its interactions with Adobe probably need soe technical
troubleshooting. I was unable to test actual printing to paper. Definitely have screen reader users
test for verbosity on the ballots as some folks may be confused by the repetition of the number of
choices, i.e. when scrolling down the list you hear 35 repeated for each selection. That may have
to do with the verbosity level of the screen reader. Again, just needs blind folks with screen
readers to test the final ballot formats. Overall I am very satisfied. .
12 I do not know this system that well 6/16/2017 9:43 AM
6 / 6