Consultant's Source Code Test Report

6705 Odyssey Drive Suite C Huntsville, AL 35806 Phone (256)713-1111 Fax (256)713-1112 Dominion Democracy Suite 5.10-A Software Test Report for the State of California Version: 00 Date: 06/25/2020 TR v. 01-02-DVS-40-01.00 SIGNATURES Approved by: 06/25/2 020 Michael Walker, VSTL Project Manager Date Approved by: 06/25/2 020 Wendy Owens, VSTL Program Manager Date TR v. 01-02-DVS-40-01.00 REVISIONS Revision Description Date 00 Final Report-Initial Release 06/25/2020 TR v. 01-02-DVS-40-01.00 TABLE OF CONTENTS 1.0 INTRODUCTION.......................................................................................................................... 1 1.1 References ........................................................................................................................................ 1 1.2 Terms and Abbreviations ................................................................................................................. 1 1.3 Background ...................................................................................................................................... 2 1.4 System Overview ............................................................................................................................. 2 1.5 Description of Component Code ..................................................................................................... 2 1.6 Scope of Testing .............................................................................................................................. 2 1.7 Testing Overview ............................................................................................................................. 3 2.0 TEST CANDIDATE ...................................................................................................................... 3 3.0 TEST PROCESS AND RESULTS ............................................................................................... 4 3.1 Summary Findings and Recommendation ...................................................................................... 4 3.1.1 Source Code Review, Trusted Build, and Documentation Review .................................... 5 3.1.2 Installation of the Trusted Builds of the Windows-based Components ............................. 6 3.1.3 System Level Image Creation ............................................................................................ 6 4.0 SUMMARY .................................................................................................................................... 7 TR v. 01-02-DVS-40-01.00 1.0 INTRODUCTION The purpose of this Test Report is to document the results of D-Suite 5.10-A System Level Image Creation. The scope of the evaluation included performing a source code review of the submitted modified source code, generating trusted builds, installing the trusted builds for the Windows- based components and creating trusted images of all Windows-based components. 1.1 References The documents listed below were utilized in the development of this Test Report: California Voting System Standards (CVSS) Election Assistance Commission Testing and Certification Program Manual, Version 2.0 Election Assistance Commission Voting System Test Laboratory Program Manual, Version 2.0 National Voluntary Laboratory Accreditation Program NIST Handbook 150-2016, “NVLAP Procedures and General Requirements (NIST Handbook 150)”, dated July 2016 National Voluntary Laboratory Accreditation Program NIST Handbook 150-22, 2008 Edition, “Voting System Testing (NIST Handbook 150-22)”, dated May 2008 United States 107th Congress Help America Vote Act (HAVA) of 2002 (Public Law 107- 252), dated October 2002 Pro V&V, Inc. Quality Assurance Manual, Version 7.0 D-Suite 5.10-A Technical Documents (A listing of the TDP documents submitted for this test campaign is listed in Table 3.1 of this Test Report) 1.2 Terms and Abbreviations The terms and abbreviations applicable to the development of this Test Report are listed below: “COTS” – Commercial-Off-The-Shelf “CVSS” – California Voting System Standards “D-Suite” – Democracy Suite “EAC” – United States Election Assistance Commission “EMS” – Election Management System “ICC” – ImageCast Central “ICE” – ImageCast Evolution TR v. 01-02-DVS-40-01.00 Page 1 of 7 “ICX” – ImageCast X “ICP2” – ImageCast Precinct 2 “ICVA” – ImageCast Voter Activation “MBP” – Mobile Ballot Printing “VSTL” – Voting System Test Laboratory 1.3 Background The D-Suite 5.10-A system is a modification baselined from the current California-approved D- Suite 5.10 system configuration. The baseline system for the source code review report is D-Suite 5.10, which has been previously approved by the state. 1.4 System Overview The Democracy Suite 5.10-A Voting System is a paper-based optical scan voting system consisting of the following major components: The Election Management System (EMS), the ImageCast Central (ICC) ballot scanner, the ImageCast Precinct 2 (ICP2) precinct count tabulator, ImageCast Evolution (ICE) precinct count tabulator, ImageCast Voter Activation (ICVA), Mobile Ballot Printing (MBP), and ImageCast X (ICX) BMD ballot marking device. 1.5 Description of Component Code The table below provides the component lines of code for D-Suite 5.10-A. Table 1-1 Democracy Suite 5.10-A Component Lines of Code Lines of Component Language/s Standard Code EMS* C# 1,721,537 SD_CSharp_AutomatedCodeReview_5.10_A.pdf ICP2 C/C++ 486,907 SD_CplusPlus_CodingStandard_5.10_A.pdf ICX Java 224,307 SD_DVSJavaCodingStandards_5.10_A.pdf ICE C/C++ 852,980 SD_CplusPlus_CodingStandard_5.10_A.pdf ICC C/C++ 231,903 SD_CplusPlus_CodingStandard_5.10_A.pdf ADJ C# 200,913 SD_Csharp_AutomatedCodeReview_5.10_A.pdf *Note: EMS Includes ICVA and MBP. 1.6 Scope of Testing Pro V&V performed an evaluation of the results from the previous test campaign along with the changes made to the system to determine the scope of testing required for the submitted modification. It was determined the tasks listed below would be required to verify successful system implementation of all modifications: Source Code Review, Trusted Build, and Build Document Review TR v. 01-02-DVS-40-01.00 Page 2 of 7 Installation of the Trusted Builds of the Windows-based Components System Level Image Creation 1.7 Testing Overview The evaluation of D-Suite 5.10-A addressed each of the following test goals in the following manner: Table 1-2: Testing Overview Test Goal Description Trusted Builds were generated during the test campaign. The source code submitted by Dominion was reviewed Perform Source Code Review, by Pro V&V to the California Voting System Standards Compliance/Trusted Builds, and (CVSS). The reviewed code was successfully built using Documentation Review the submitted COTS and third-party software products. Additionally, build documentation was reviewed. Installation of the Trusted Builds Windows-based components were installed onto the of the Windows-based respective workstations to ensure that all workstations Components were set up correctly. Trusted images were created during the campaign for Perform System Level Image California to load to ensure that the integrity of the Creation system. 2.0 TEST CANDIDATE The Democracy Suite 5.10-A Voting System is a paper-based optical scan voting system that includes proprietary software and firmware. D-Suite 5.10-A consists of the following major components: Election Management System (EMS) 5.10.50.83 o Election Event Designer 5.10.50.83 o Results Tally Reporting 5.10.50.83 o Audio Studio 5.10.50.83 o File System Service 5.10.50.83 o Data Center Manager 5.10.50.83 TR v. 01-02-DVS-40-01.00 Page 3 of 7 o Application Server 5.10.50.85 o Election Data Translator 5.10.50.83 o EMS Service 5.10.50.83 o Adjudication Services 5.10.50.10 Note: EMS APPS component version is 5.10.50.85 Adjudication Client (ADJ) 5.10.50.12 ImageCast Central (ICC) ballot scanner 5.10.2.0001 ImageCast Precinct 2 (ICP2) precinct count tabulator 5.10.5.1 ImageCast X (ICX) BMD ballot marking device 5.10.12.4 ImageCast Voter Activation (ICVA) 5.10.50.83 ImageCast Evolution (ICE) precinct count tabulator 5.10.10.3 Mobile Ballot Printing (MBP) 5.10.50.83 Reformatting Workstation 3.0 TEST PROCESS AND RESULTS The following sections outline the test process that was followed to evaluate the D-Suite 5.10-A System under the scope defined in Section 1.6. All testing was conducted under the guidance of Pro V&V by personnel verified by Pro V&V to be qualified to perform the testing. 3.1 Summary Findings and Recommendation Summary findings for this evaluation are detailed in the following sections. 3.1.1 Source Code Review, Trusted Build, and Build Document Review A source code review was performed to review the submitted source code to the California Voting Systems Standard (CVSS) and the manufacturer-submitted coding standards. Prior to initiating the review, Pro V&V verified that the submitted documentation was sufficient to enable: (1) a review of the source code and (2) Pro V&V to design and conduct tests at every level of the software structure to verify that design specifications and performance guidelines were met. TR v. 01-02-DVS-40-01.00 Page 4 of 7 The source code review was based on the source code changes made since the previous system was certified. Both manual and automated review techniques were used per EAC approved procedures. A combination of Automated Source Code Review (utilizing the StyleCop statistical code analysis tool) and Manual Source Code Review methods were used to review the changes in the source code. In addition, 10% of the source code comments were manually reviewed. The Source Code Review included a Trusted Build of the submitted source code. To perform the Trusted Build, Dominion-submitted source code, COTS, and Third-Party software products were inspected and combined to create the executable code. Additionally, during the performance of the Trusted Build, the build documentation was reviewed. Summary Findings Pro V&V performed a differential analysis of the EMS 5.10.50.85 source code against the baseline EMS 5.10.50.83 and noted only two source code files were changed (DocumentToHtmlConverter.cs and HtmlTagAttributes.cs), which are described below. DocumentToHtmlConverter.cs. This file contains logic for converting the text documents from the EMS database that contain the ballot information (headers, contest, candidates, etc.) into the Standard HTML format required by the ICX to display the information. A few methods were changed to update the logic of which HTML elements are used and also how they are put together to create the HTML file. HtmlTagAttributes.cs. This file contains the constant strings (static definitions) of the HTML elements that are being used. This file was changed due to different notation for the Standard HTML file required by the ICX. The changes to these two files only affect the EMS Application Server (APPS) component. The affected functionality is the election file generation for the ICX. This is the process where the EMS creates the election definition files for the ICX machines. More specifically, it’s in the process of the creation of the HTML documents used by the ICX to display content. The review of the source code included: A review for adherence to the applicable standards in Sections 5 and 7 of the CVSS A review for adherence to other applicable coding format conventions and standards including best practices for the coding language used An evaluation as to whether the system is designed in a way that allows meaningful analysis, including: o Whether the architecture and code is amenable to an external review TR v. 01-02-DVS-40-01.00 Page 5 of 7 o Whether code analysis tools can be usefully applied Security considerations reviewed against the code included an analysis of error exception handling. Pro V&V considered the modifications to be minor in nature; therefore, an in-depth security review was not required as part of the scope of the evaluation. 3.1.2 Installation of the Trusted Builds of the Windows-based Components Installation of the Trusted Builds for the Windows-based components included the installation of all Windows-based components onto their respective workstations. Installation documentation review was performed to ensure that all required equipment and software were current and installed correctly. Summary Findings Build documents and installation guides utilized during the evaluation are listed below. Table 3-1: D-Suite 5.10-A TDP Documents Description Version Build Documents Democracy Suite ImageCast Evolution Firmware Build Prerequisites Setup and 5.10-A::1 Installation Democracy Suite ImageCast Precinct 2 Build Environment and Prerequisite 5.10-A::4 Setup, Firmware Build and Installation Democracy Suite ImageCast X Build 5.10-A::5 Democracy Suite Windows Build Document 5.10-A::16 Installation Guides Democracy Suite EMS Client Workstation Installation and Configuration 5.10-A::1 Procedure Democracy Suite EMS Standard System Installation and Configuration 5.10-A::1 Procedure Democracy Suite ImageCast Central Installation and Configuration Procedures 5.10-A::3 Supplemental Documents Appendix to 5.10-A Integrated Build Document --- 3.1.3 System Level Image Creation System Level Image Creation included the production of creating Windows-based images that could be reloaded onto new workstations to verify the integrity of the systems. Summary Findings Once all setup was complete, the procedures detailed in the submitted documentation were followed to make trusted images of all Windows-based systems. TR v. 01-02-DVS-40-01.00 Page 6 of 7 4.0 SUMMARY Based on the evaluation performed and the results obtained, the D-Suite 5.10-A successfully completed the evaluation with no issues noted. For each source code base submitted (EMS/ICVA/MBP, ICP2, ICX, ICE, ICC, and ADJ) there were no source code requirements found to be at issue with the source code reviewed; as a result, no discrepancies were noted. Additionally, there were no vulnerabilities found within the reviewed source code; therefore, no findings were written against any of the source code bases. Trusted builds, source code, and images were provided to the State of California per Dominion’s request. Along with the trusted build and images, hash values contained in the Hash_Cert_Folder.txt file were archived in the media that Pro V&V provided to the State of California. As directed by the California Secretary of State Office, this software evaluation report does not include any recommendation as to whether or not the system should be approved. TR v. 01-02-DVS-40-01.00 Page 7 of 7