California Secretary of State Debra Bowen’s Report to the Election Assistance Commission Concerning Errors and Deficiencies in Diebold/Premier GEMS Version 1.18.19
GEMS is the central software component of the voting systems developed and marketed by Diebold Election Systems, Inc. (renamed Premier Election Solutions, Inc. in August 2007).1 GEMS stands for Global Election Management System. This report describes how a serious software programming error in GEMS version 1.18.19 caused the loss of 197 tallied ballots in the November 4, 2008, General Election. The report also describes several deficiencies in the audit trail logs in GEMS version 1.18.19.
I. The “Deck 0” Software Error Deleted 197 Tallied Ballots From The Official Results For The November 4, 2008, General Election In Humboldt County, California
GEMS version 1.18.19 contains a serious software error in its Central Count Server. The GEMS Central Count Server tallies votes from central count optical scanners. Humboldt County, California, uses the Central Count Server to tally optical scan vote-by-mail (or “absentee”) ballots, mail-only precinct ballots and provisional ballots. The software error silently deletes all tallied votes from the first batch or “deck” of optical scan paper ballots after they have been scanned into GEMS. The deletion results whenever the following commonplace sequence of events occurs:
1. At any point after the first deck of voted ballots (automatically named “Deck 0” in GEMS 1.18.19) is scanned into the GEMS database, the Central Count Server window is closed and re-opened; and
2. The GEMS operator deletes any subsequent deck of ballots because a problem is encountered.
In the November 4, 2008, General Election in Humboldt County, this software error deleted all of the votes cast on the 197 vote-by-mail ballots tallied in Deck 0 from the GEMS database, resulting in certification (which was subsequently corrected) of incomplete and inaccurate official election results. The loss of votes could have been greater; its magnitude was limited only by the number of ballots the county elections official had chosen to scan as part of Deck 0.
1 Diebold Election Systems, Inc./Premier Election Solutions, Inc. is a wholly owned subsidiary of Diebold, Inc. In this report, “Diebold” and “Premier” are used interchangeably to refer to the voting system subsidiary.
March 2, 2009
PAGE 2
The miscount of votes caused by this software flaw greatly exceeds the maximum error rate permitted by the Help America Vote Act of 2002 (HAVA).2
Diebold knew of this serious software error no later than October 2004. The company, however, did not notify the Election Assistance Commission (EAC), the National Association of State Election Directors (NASED) or the California Secretary of State. Instead, the company sent a vague email to elections officials in the 11 California counties using GEMS version 1.18.19 with the Central Count Server at the time. (Six other counties used GEMS version 1.18.19, but did not use it with the Central Count Server.) The email, reproduced below, advised the county officials to create and immediately delete an empty Deck 0 before scanning any real ballots, but did not explain why this new procedure was necessary.
2 Section 301(a)(5) of HAVA requires that every voting system used in an election for federal office meet the maximum error rate standard in the 2002 VVSG. Volume II, Section 4.7.1.1 of the 2002 VVSG provides:
[D]ata accuracy is defined in terms of ballot position error rate. This rate applies to the voting functions and supporting equipment that capture, record, store, consolidate and report the specific selections, and absence of selections, made by the voter for each ballot position. Volume I, Section 3.2.1 identifies the specific functions to be tested.
For each processing function, the system shall achieve a target error rate of no more than one in 10,000,000 ballot positions, with a maximum acceptable error rate in the test process of one in 500,000 ballot positions. This error rate includes errors from any source while testing a specific processing function and its related equipment. (Italics added.)
Volume I, Section 3.2.1(e) identifies as subject to testing “[c]onsolidation of vote selection data from multiple counting devices to generate jurisdiction-wide vote counts, including storage and reporting of the consolidated vote data.”
March 2, 2009
PAGE 3
From: Runyan, Therese (Tari) [mailto:Tarir@dieboldes.com] Sent: Tuesday, October 19, 2004 7:40 PM To: Billie Alvarez (E-mail); Debbie Hench (E-mail); DeJusto, Madelyn (E-mail); Elaine Ginnold (E-mail); Julie Rodewald (E-mail); McWilliams, Lindsey; Mark Gonzales (E-mail); Ryan Ronco (E-mail); Sally McPherson (E-mail); Sandy Brockman (E-mail); Tulare - Hiley Wallis (E-mail) Cc: Robert Chen (E-mail) Subject: Central Count for the upcoming election
I have attached a document that details using Central Count for November - specifically beginning Central Count and the Deck 0. It is very important that you follow these instructions - please contact Rob or myself if you have any questions . Thank you <<Gems1-18-19CentralCount.doc>>
Tari Runyan Western Regional Support Manager Diebold Election Systems 10675 N. Abilene St. Commerce City, Co 80022
voice 303-600-8280 cell 303-520-0710 email tarir@dieboldes.com
The file attached to the email consisted of the following:
This Document is to Provide a Working Solution for the Following
ISSUE:
When running Gems 1.18.19.0 and processing ballots with the Central Count Server an issue exists with correctly sorting committed decks, in some reports, and also deleting other decks under certain conditions, when “deck 0” has not been deleted.
RESOLUTION:
When the election is invoked and there has been no Central Count ballot processing ever done in the database then start the Central Count server and process a “Start” card and then immediately afterwards an “Ender” card. This will commit deck 0 without any ballots and allow the deletion of the committed deck 0 from the database. You should delete Deck 0.
This must be done as the first action after starting Central Count
March 2, 2009
PAGE 4
The email and attachment did not inform the elections officials that failure to follow these instructions would likely result in deletion of tallied votes by GEMS without any warning or notice to the system operator. The email and attachment also failed to inform counties that it was a programming flaw in the GEMS software that made the special instructions necessary.3 The chief elections official for Humboldt County, Registrar of Voters Carolyn Crnich, states that she never saw the email or the attached instructions. The former county elections officer apparently had received the email in 2004, but left Humboldt County in 2007 to work in another county's elections office. That county elections officer did not, according to Registrar of Voters Carolyn Crnich, pass the information along to her or anyone else in her office.4 This helps to explain why the Deck 0 software error manifested itself in the November 2008 election.5
Secretary of State staff reviewing the matter confirmed that Diebold corrected the Deck 0 software error in GEMS version 1.18.24. Many jurisdictions across the United States continue to use GEMS version 1.18.19 and later versions, such as 1.18.22 and 1.18.23 that Diebold released before fixing the “Deck 0” error in version 1.18.24. Company representatives have stated, however, that no jurisdiction outside California uses GEMS version 1.18.19 or these later versions with the Central Count Server. Research by the Secretary of State’s staff found nothing that would call into question the validity of the claim.
How the Deletion of Votes Was Detected
The deletion of 197 ballots from the official election results for Humboldt County was discovered through a unique collaboration between Registrar of Voters Carolyn Crnich and the Humboldt County Election Transparency Project, a group of community volunteers. The
3 Diebold has never added the special instructions to the official user documentation for GEMS version 1.18.19 or the subsequent GEMS versions (1.18.20, 1.18.21, 1.18.22 and 1.18.23) that contain the same software error.
4 The account of this communications failure is included because it helps to explain why the software error was triggered in an election held in 2008. It does not absolve the company of its responsibility to include in its official voting system documentation all information and instructions necessary to ensure accurate results.
5 The two other California counties that used the GEMS version 1.18.19 Central Count Server in the November 2008 election used the company’s “work-around” and did not experience a loss of tallied votes. The company's remaining California county customers had previously upgraded to GEMS version 1.18.24. Diebold released version 1.18.24 on May 13, 2005. Version 1.18.24 was federally qualified on October 3, 2005, with NASED No. N-1-06-22-22-002, and approved for use in California by the previous Secretary of State on February 17, 2006.
March 2, 2009
PAGE 5
Project's purpose was to provide the public with unofficial, independently scanned digital images of all ballots so anyone could perform their own count as a check on the accuracy of the official count. Ms. Crnich agreed to have all ballots scanned, using a high-speed scanner running on open source firmware and counted by freely-available open source software written by Mitch Trachtenberg, a Humboldt County Election Transparency Project volunteer.
The November 2008 election was the second trial of this system, which Humboldt County had first used following the June 6, 2008, Statewide Direct Primary Election.6 The Project used a Fujitsu high-speed scanner that Humboldt County's Board of Supervisors had authorized the Registrar of Voters to purchase for this purpose, although it is used year-round for other tasks in the Elections department. The County paid approximately $35,000 for the scanner.
The Humboldt County Election Transparency Project donated the open source firmware and software needed to tally the ballots and supplied the volunteers to conduct the scanning. Under strict security and chain-of-custody requirements, the two-sided ballots were scanned one side at a time (simplex mode).7 With shifts drawn from 14 deputized volunteers supervised by Humboldt County Elections Department staff, scanning the 64,000+ ballots (128,000+ ballot sides) cast in the November 4, 2008, election took 65 person hours to complete. As a result, Project volunteers were not able to inform the Registrar of Voters that their unofficial ballot total exceeded the official total by 216 ballots until Sunday, November 30, 2008. At that point, they had not identified the precinct involved or the reason for the discrepancy. A special meeting of the Humboldt County Board of Supervisors was scheduled for the next day, Monday, December 1, 2008, at which Registrar of Voters Crnich was required to certify the final results. Without enough time to confirm that the official tally in fact omitted ballots, she certified the results that Monday.
Meanwhile, Project volunteers studied the GEMS precinct turnout report, looking for precincts in which the reported turnout was substantially lower than in adjacent precincts. This method was available only because Humboldt County had pre-sorted voted vote-by-mail ballots by precinct;
6 A full description of the Humboldt County Election Transparency Project and the open source ballot scanning and tallying system it uses is available at www.mitchtrachtenberg.com. Ballot Browser, the open source application developed by Mitch Trachtenberg, can also be downloaded from this location, and links are provided to scanned images of the ballots from the June and November 2008 elections in Humboldt County. Additional information about the Project can be found at http://humtp.com/.
7 In future elections, the Project intends to switch to duplex scanning, halving the time required to scan the ballots.
March 2, 2009
PAGE 6
many California counties do not follow this practice. Project volunteers, working with the Registrar of Voters, quickly zeroed in on precinct 1E-45, for which GEMS reported a turnout rate significantly lower than in neighboring precincts. The Registrar of Voters retrieved and counted the paper vote-by-mail ballots for that precinct. Her count showed 197 more vote-by-mail ballots for precinct 1E-45 than were listed in the final GEMS report. The revised count was confirmed by reviewing the printout tapes from the central count scanners for November 1, 2008, when hand-written logs showed Deck 0 had been scanned. This accounted for 197 of the 216-ballot differential between the overall totals reported by the Project and GEMS.8
According to Registrar of Voters Carolyn Crnich, it was only after two telephone conferences with Premier representatives on December 3, 2008, that she learned what the company had known for over four years: The company's voting system was indeed capable of losing tallied ballots, and the cause was the “Deck 0” error in the GEMS software.
Whether the fact that ballots had been omitted from the tally could or should have been discovered through ballot reconciliation processes that are part of the standard canvass process misses what was at issue in Humboldt County. No software error affecting the accuracy of election results should ever be excused based on claims that the effects of the error could or should be detected and corrected through adherence to sound election administration procedures. In this particular case, a reconciliation of the Registrar’s count of vote-by-mail ballots returned by voters with the count reported by GEMS was performed on November 1, the day Deck 0 was tallied, and no discrepancy was found. GEMS reports generated on Election Day and on November 23, 2008, two and a half weeks after the election, continued to accurately reflect the 197 ballots in Deck 0.
It was only later, after the GEMS Central Count Server was re-opened and new decks of vote-by-mail ballots that had been received on Election Day were tallied for the first time, that Deck 0 was deleted, without any warning or notification to the elections official, as a result of the software programming flaw. Because the deletion of the votes from the 197 ballots in Deck 0 occurred long after they were counted and after repeated reports showed them properly accounted for, nothing indicated any need to recheck the earlier reconciliation for a third time.
8 The Registrar of Voters subsequently resolved the remaining 19 discrepancies, which were not voting system related.
March 2, 2009
PAGE 7
II. Deficiencies In The Gems Version 1.18.19 Audit Logs
A second set of serious problems related to electronic audit logs was discovered during the Secretary of State Office’s investigation of the Deck 0 software programming flaw. First, GEMS version 1.18.19 fails to record in any log important system events such as the deletion of decks of optical scan ballots after they have been scanned and entered into the GEMS election results database. Second, it records the wrong entry date and time for certain decks of ballots. Third, it permits deletion of certain audit logs that contain – or should contain – records that would be essential to reconstruct operator actions during the vote tallying process.
Failure to Log Important System Events
GEMS version 1.18.19 creates no audit trail record—in the GEMS Central Count Server log, the POSTER log, or the primary Audit Log—when a GEMS operator intentionally deletes a deck of ballots after it has been tallied with the Central Count Server and committed to the GEMS election results database. Intentional deletion of committed decks is a common occurrence, performed when the operator detects that ballots were not scanned properly. During the course of tallying results for the November 4, 2008, General Election, Humboldt County elections officials intentionally deleted 26 committed decks. Those decks were re-scanned and assigned different numbers by GEMS. The re-scanning events are reflected in the Central Count Server log, but the deletion events are not reflected anywhere in the GEMS system.
Inaccurate Date and Time Stamps
The GEMS “Status Report by Deck” logs the date and time each batch of ballots was scanned. For the November 2008 election in Humboldt County, at least three of these date and time “stamps” were wrong. For example, a Status Report by Deck listed a batch of ballots scanned and committed to GEMS on November 3 with the date November 25.
Prohibited “Clear” Buttons Allow Deletion of Log Records
GEMS version 1.18.19 is designed to permit the operator to delete the audit trail records in two important audit logs, intentionally or inadvertently. The records can be deleted by selecting “Clear” buttons that appear on the audit log screens between the “Save As…” and “Close” buttons.
Diebold/Premier revised its code to remove these “Clear” buttons from all subsequent versions of GEMS including version 1.18.20, which the company released only two weeks after issuing version 1.18.19. However, the company never removed the “Clear” buttons, from version
March 2, 2009
PAGE 8
1.18.19 as that version made its way through the certification process at the federal level and in numerous states, including California.
As noted above, the “Deck 0” software error affects only jurisdictions that use the Central Count Server. The “Clear” buttons, by contrast, allow inadvertent or malicious destruction of critical audit trail records in all GEMS version 1.18.19 jurisdictions, risking the accuracy and integrity of elections conducted using this voting system. Five years after the company recognized the need to remove the “Clear” buttons from the GEMS audit log screens, not only Humboldt, San Luis Obispo and Santa Barbara Counties in California but jurisdictions in other parts of the country, including several counties in Texas and Florida, continue to use GEMS version 1.18.19.
III. The Deck 0 and Audit Log Deficiencies in GEMS Version 1.18.19 Violate the 1990 Federal Standards Under Which It Was Qualified.
NASED qualified GEMS version 1.18.19 on February 3, 2004, pursuant to the Federal Election Commission’s 1990 Performance And Test Standards For Punchcard, Marksense, And Direct Recording Electronic Voting Systems (“1990 VSS”). NASED assigned qualification No. N-1-06-12-12-001 to GEMS version 1.18.19.
The 1990 VSS, like its 2002 and 2005 successors, requires that the software in a voting system automatically create and permanently retain electronic audit logs of important system events during tallying of the votes cast in an election. As detailed below, GEMS version 1.18.19 fails to meet these requirements.
The 1990 Standards
The 1990 VSS assigned great importance to election audit trails.9 The audit trails are intended to provide “a concrete, indestructible archival record of all system activity related to the vote tally”
9 The Glossary to the 1990 VSS defines “Audit Trail” in relevant part as follows:
Audit Trail-The continuous trail of evidence linking individual transactions related to the vote count with the summary record of vote totals. It permits verification of the accuracy of the count and detection and correction of problems. A combination of manual and computer-generated documentation provides a record of each step taken in: defining and producing ballots and generating related software for specific elections: installing ballots and software; testing system readiness; casting and tabulating ballots; and producing reports of vote totals. The record incorporates system status and error messages generated during election processing, including a log of machine activities and routine and unusual intervention by authorized and unauthorized individuals. (1990 VSS App. L – Glossary, page L-2, italics added.)
March 2, 2009
PAGE 9
and are “essential for public confidence in the accuracy of the tally, for recounts, and in the event of litigation. (1990 VSS, Section 4.8, italics added.)
One class of audit records required by the 1990 VSS is “in-process audit records” which “consist of data documenting precinct and central count system operation during diagnostic routines and in the casting and tallying of ballots.” (1990 VSS, Section 4.8.2.3.) The 1990 VSS specifically requires at a minimum a “[s]ystem generated log of all normal process activity and system events that require operator intervention, so that each operator access can be monitored and access sequence can be constructed.” (Ibid.) For central count, optical scan (“Marksense”) systems, the “in-process audit records” must also include the “quantity and identification number of aborted precincts.” (Section 4.8.2.3(e)(iii).)
In apparent violation of the 1990 VSS, GEMS version 1.18.19 includes “Clear” buttons that allow the operator to permanently delete records in at least two audit trail logs required by the standards to be permanent: the Central Count Server Log and the Poster Log. Screenshots of Central Count Server Log and Poster Log displays in GEMS version 1.18.19 are reproduced below:
March 2, 2009
PAGE 10
Each of these logs records, or should record, system activity related to the vote tally, including all system events that require operator intervention. GEMS version 1.18.19 not only includes “Clear” buttons that permit deletion of these records, it provides no warning to the operator that exercising the “Clear” command will result in permanent deletion of the records in the log, nor does it require the operator to confirm the command before GEMS executes it. Deletion of the records in either log would make it impossible to monitor operator access to GEMS or to reconstruct the sequence of operator access, defeating the purposes of the 1990 VSS, that GEMS version 1.18.19 was required to adhere to. Regardless of whether Diebold knew that permitting deletion of audit records would violate the federal standards, it clearly knew as early as 2001 that “[a]dding a 'clear' button is easy, but there are too many reasons why doing that is a bad idea.”10
10 In 2001, Ken Clark, the principal developer of GEMS, discussed the propriety of including a “clear” button in the following internal email exchange:
March 2, 2009
PAGE 11
Deletion of records of an election tally in these critical GEMS logs is not merely a hypothetical possibility. In the course of this investigation, Secretary of State office staff contacted the elections official in another California county that had also used GEMS version 1.18.19 in the November 4, 2008, General Election. Following the election, the county official received a public records request for documents related to the investigation, including the GEMS Poster Log. The county official informed the Secretary of State’s office that, while attempting to print a copy of the Poster Log records from the November 4, 2008, General Election, she inadvertently deleted them instead. As can be seen in the Poster Log screenshot above, the “Print” and “Clear” command buttons appear near each other in the same horizontal row.
To: <support@gesn.com>
Subject: RE: GEMS Audit Log - Ability to clear entries
From: "Ken Clark" <ken@gesn.com>
Date: Wed, 3 Oct 2001 12:31:58 -0700
Importance: Normal
In-reply-to: <002801c14c42$59cf60c0$0903a8c0@Jeff>
You are correct, you can't clear the log. Adding a "clear" button is easy, but there are too many reasons why doing that is a bad idea.
What Bill should do is create an "election template" database, and build his new elections from that, not from his "last" election.
Ken
-----Original Message----- From: owner-support@gesn.com [mailto:owner-support@gesn.com]On Behalf Of Jeff Hintz Sent: Wednesday, October 03, 2001 12:34 PM To: Support Team (E-mail) Subject: GEMS Audit Log - Ability to clear entries
Bill Bowers of GBS has asked the question whether we can clear or delete the Audit Log in GEMS??? He does backups of old elections, then creates new elections from the old elections. He wants the log to be clean when he starts over. I am assuming that this in internal to GEMS and cannot be cleared or deleted, but I just want to get confirmation.
Thanks,
JEFF HINTZ
Global Election Systems
March 2, 2009
PAGE 12
The relevant provisions in the 1990 VSS testing requirements and rejection standards are set forth below. Under those provisions, each of the errors and deficiencies in the GEMS version 1.18.19 software described in this report standing alone would warrant a finding by an Independent Testing Authority (ITA) of “Total Failure” (indicated by a score of 1.0) had the flaw been detected. Under the 1990 VSS, a finding of “Total Failure” required failure of the voting system.
G.4 Functional Failures and Scores
. . . .
G.4.3.2 Obtaining Reports
This function includes all operations and capabilities necessary to consolidate voting data from all voting devices and polling places, to process absent voter ballots and any other ballots which require exceptional handling, to produce voting data reports, and other reports associated with the results of the election.
Defect Score
Total Loss of Function: 1.0
Any failure to correctly process voting data, audit data and administrative data at any level of reporting, or to support testing required to validate these operations. (Italics added.) 11
11 To place Appendix G.4.3.2 in context, related excerpts from the 1990 VSS are set forth below.
7. Qualification Test and Measurement Procedures
An independent test authority (ITA) shall conduct qualification tests to evaluate system compliance with the requirements of Sections 2 through 6. The examination shall encompass tests of hardware under conditions simulating the intended storage, operation, transportation, and maintenance environments; the selectively in-depth examination of software; the inspection and evaluation of system documentation; and operational tests verifying system performance and function under normal and abnormal conditions.
. . . .
7.1.1 Scope of Tests
. . . .
The procedure for disposition of failures or deficiencies discovered during qualification testing is described in Appendix G. This procedure recognizes that some but not necessarily all operational malfunctions (apart from software logic defects) may result in rejection. Basically, any
March 2, 2009
PAGE 13
IV. Conclusion
GEMS version 1.18.19 contains a serious software error that caused the omission of 197 ballots from the official results (which was subsequently corrected) in the November 4, 2008, General Election in Humboldt County. The potential for this error to corrupt election results is confined to jurisdictions that tally ballots using the GEMS Central Count Server. Key audit trail logs in GEMS version 1.18.19 do not record important operator interventions such as deletion of decks of ballots, assign inaccurate date and time stamps to events that are recorded, and can be deleted by the operator. The number of votes erroneously deleted from the election results reported by GEMS in this case greatly exceeds the maximum allowable error rate established by HAVA. In addition, each of the foregoing defects appears to violate the 1990 Voting System Standards to an extent that would have warranted failure of the GEMS version 1.18.19 system had they been detected and reported by the Independent Testing Authority that tested the system.
defect that results in or may result in the loss or corruption of voting data, whether through failure of system hardware and software, through procedural deficiency, or through deficiencies in security and audit provisions, shall be cause for rejection. (1990 VSS, Section 7.1.1—Scope of Tests, italics added.)
Appendix G - Voting System Failure Definition and Scoring Criteria
. . . .
The emphasis of this Appendix [G] is upon identifying failure modes which may result in the loss of a critical performance attribute, or in the loss or corruption of voting data. These failures are defined below as "total" failures. They are so important as to require that testing procedures be interrupted if they occur, so that they can be corrected. The effectiveness of the corrective action must be verified by ancillary tests before the qualification or acceptance tests may be resumed. (1990 VSS, App. G, italics added.)
. . . .