In This Issue
Fall Issue of The Bridge on Nuclear Energy Revisited
September 15, 2020 Volume 50 Issue 3
The desire to reduce the carbon intensity of human activities and strengthen the resilience of infrastructure key to economic prosperity and geopolitical stability shines a new spotlight on the value and challenges of nuclear energy.

Risk-Limiting Audits in Colorado Elections: A Brief Overview

Tuesday, September 15, 2020

Author: Matthew Fitzgerald

Elections are the bedrock of America’s democracy. Citizens hold their govern­ment accountable by voting to elect or remove representatives and other officials and make important decisions about which policies are ­enacted. Elections must therefore be accurate and trustworthy. However, electoral integrity becomes more challenging to ensure as electronic voting is more widely used. Electronic voting introduces a myriad of potential ways for those who seek to sway the outcome of an election to tamper with the results.

To ensure accurate election results and protect against the risks of a cyberattack, Colorado uses an election auditing technique called risk-limiting audits (RLAs). On November 7, 2017, the state conducted the first such audit in US history.

RLAs use statistically rigorous processes to confirm that reported election outcomes are correct. This article provides an overview of RLA ­measures, with a focus on techniques and procedures used in Colorado. It also expounds on ways this process may improve integrity safeguards, and suggests potential future research and development of RLA processes.

What Are Risk-Limiting Audits?

Since the passage of the Help America Vote Act in 2002, states have increasingly relied on electronic voting equipment in the administration of elections. However, electronic machines, which may be used either to collect electronic ballots or to scan paper ballots, introduce a host of potential vulnerabilities that threaten the integrity of election results. Faulty machines may record, store, or transmit votes improperly because of flaws in product design or lack of proper maintenance. They can also be a target of cyberattacks, as both foreign and domestic parties may seek to manipulate the vote tallies recorded by electronic voting equipment.

To combat the threat of invalid election results, whether due to errors in ballot counting, malicious cyberattacks, or other means, those interested in election security have proposed measures that may enhance the security of elections. Many of these measures involve improvements to electronic voting equipment, changes in voting procedures, or more stringent auditing of the equipment used to record or tally ballots.

RLAs focus on the actual votes cast and provide a means for election officials to audit the result of an election, rather than focusing on the voting process or the equipment involved. In this way RLAs help to provide assurance to voters, candidates, and other interested parties that the correct winner was chosen in a given election.

Components of Risk-Limiting Audits

RLAs can be used in a variety of election formats, but existing RLA procedures rely on paper ballots to ensure a verifiable audit and therefore cannot be used in elections in which votes are cast exclusively on digital recording equipment. Because paper ballots are the foundation of the process, it is essential to accurately maintain the paper trail. This necessitates procedures to certify its integrity.

A Ballot Manifest

Election officials must craft a ballot manifest description, detailing the way ballots are collected, the sequence in which they are stored, and the location where they are held (Branscomb 2017). A well-defined ballot manifest can help election officials ensure that errors in counting or sorting do not cause a discrepancy that impairs their ability to conduct an effective audit.

In elections in which votes are tabulated by direct-recording electronic (DRE) voting machines, files designed using the ballot manifest determine how votes are collected and tabulated. Optical scanners that tabulate paper ballots similarly use the ballot manifest to dictate how the machine reads and stores the votes on each ballot. In Colorado, statewide voting by mail necessitates that paper ballots be collected and sent to optical scanners for tabulation.

In preparation for an RLA, election officials must establish a risk limit. This is described by the Colorado secretary of state as “the largest statistical probability that an incorrect reported tabulation outcome is not detected and corrected” (Colorado Secretary of State 2020). As an example, if a risk limit is set at 9 percent, 91 of every 100 possible invalid election outcomes should be caught through the use of the audit.

Random Ballot Selection

To determine which ballots will be audited, a seed number is generated. A random number generator uses the seed number to calculate pseudorandom values that each correspond with a cast ballot that will be audited. There are many ways the seed number can be identified; the goal is to generate a number that is as randomized as possible.

The randomness of the selected ballots is a vital component of the RLA. To maintain public confidence in the audit’s integrity, it must be ensured that (i) any voter’s ballot may be reviewed in the audit process and (ii) it is not possible to predict which ballots will be reviewed (so anyone seeking to undermine the election’s integrity cannot manipulate ballot order to avoid review of compromised ballots). With the ability to predict the seed number, an individual or group seeking to undermine the audit process could circumvent the audit, rendering it ineffective.

A random selection of ballots can be achieved through measures that use a physical source of randomness and the cohesive effort of multiple individuals. In Colorado, the secretary of state creates a random seed number by inviting members of the public to a meeting where they take turns rolling a 10-sided die to generate a number of at least 20 digits. This number is used to initialize a pseudorandom number generator to determine which ballots will be randomly selected for the audit (Colorado Secretary of State 2020).

Only a very small percentage of ballots may be ­audited. The exact number depends on the reported margin of victory in an election as well as the degree to which audited ballots have affirmed the accuracy of the election outcome.

Two Postelection Ballot Audit Methods

After polls have closed and ballots have been ­tabulated, the randomly selected ballots are reviewed by ballot officials. There are several ways they might be audited, two of which are ballot-polling audits and comparison audits (Lindeman and Stark 2012).

In a ballot-polling audit, ballots are examined after identifying the reported winner after the initial vote tabulation and comparing this initial result with the results found in the limited number of randomly ­selected ballots during the RLA process. This method typically requires a far greater number of ballots to be reviewed than a comparison audit. In a comparison audit, the reviewed ballots are compared to a cast vote record to confirm the accuracy of the original ballot tabulated.

With both types of audit, reviews continue until the statistical risk limit has been fulfilled or, if necessary, a full recount is completed (Lindeman and Stark 2012).

Advantages of the RLA

Software Independence

The use of electronic equipment in US elections has become commonplace. But DRE machines may report invalid results, because of either faulty programming or malicious activity. The same may be true of optical scanning equipment used in elections administered with paper ballots. In both cases, the tabulation of voters’ ballots must, at some point, be entrusted to computer programmers who, despite their best efforts, cannot deliver software that is perfect in terms of either functionality or security.

The fallibility of electronic voting equipment has resulted in calls from election cybersecurity experts for the implementation of software-independent voting infrastructure (Rivest 2008). Software independence in this context does not entail the elimination of all electronic voting equipment. Instead, it embraces procedures that allow elections to proceed effectively in the absence of reliable electronic voting equipment. RLAs support software independence by creating a post­election environment in which there is an organized paper trail that can be sampled for accuracy.

Diverse Applicability

Rules governing election procedures are primarily made at the state and local levels. The result is an American electoral landscape with diverse procedures for voter registration, ballot collection, and voting equipment selection and monitoring. An advantage of the RLA is that it can apply to a variety of elections (Lutz 2019).

As long as a locality’s electoral procedures meet the criteria mentioned above, it has a high probability of being able to conduct an RLA. With existing RLA software and procedures, “complicated calculations or in-house statistical expertise” (Lindeman and Stark 2012) are not required to implement a risk-limiting audit.


In addition to versatility of implementation, RLAs provide a level of efficiency that does not exist in more traditional election recount procedures. Because of the statistical methods used in the development of the RLA, a relatively small number of ballots may satisfy the risk limit, as indicated in the following description of a ballot-polling audit procedure (Lindeman and Stark 2012, p. 44):

[I]n the 2008 presidential election, 13.7 million ballots were cast in California; Barack Obama was reported to have received 61.1% of the vote. A ballot-polling audit could confirm that Obama won California at 10% risk (with t = 1%) by auditing roughly 97 ballots—seven ten-thousandths of one percent of the ballots cast—if Obama really received over 61% of the votes.

A useful analogy provided by Lindeman and Stark (2012) is that of taste-testing soup. If a chef desires to know whether the soup needs more salt, she needs only to stir it thoroughly and taste a spoonful. The spoon will contain only a small volume of soup relative to the pot, but the characteristics of its contents should be indicative of the whole. Similarly, only a small number of ­ballots need be used for an audit, so long as the selection process is characterized by thorough randomization.

One of the reasons for this efficiency is that an RLA is not implemented to confirm the exact total of votes received by each candidate. Instead, it confirms, within a selected unit of risk, whether the outcome of an election, in terms of winners and losers, is likely to be ­correct. As Lindeman and Stark (2012, p. 42) comment, “Risk-limiting audits do not guarantee that the electoral outcome is right, but they have a large chance of correct­ing the outcome if it is wrong.”


RLAs may also prove to be much more expedient in comparison with auditing processes that confirm the functionality of electronic voting equipment (­Lindeman and Stark 2012). Generally, the more complex a voting software system, the higher the chance that bugs or security gaps may be identified through conventional auditing measures and the longer such procedures may take. Simply confirming the accuracy of voting results seems to achieve the goals of such measures—making sure that the winner indeed won, and the loser truly lost—at a comparatively lower resource cost.

The use of RLA procedures provides election officials with a statistically reliable, transparent method by which they might increase voter confidence in election outcomes. Incorrect election outcomes can be identified, mitigating the potential of faulty tabulation, regardless of whether this is due to machine or human error. Confirming the accuracy of electoral outcomes may enhance and justify public trust (Hall et al. 2009).

Possible Improvements to the RLA

RLA systems remain relatively novel in practice. Only four states currently require them by law: Colorado, Nevada, Rhode Island, and Virginia (Lynch 2019). As more states consider the use of RLAs in their election processes, it may be prudent to consider how existing RLA systems may be improved.

While RLAs can be used across a variety of races on a ballot, no state has used them for all races across the entirety of the ballot. In Colorado the secretary of state is required to choose at least two statewide contests in a general election and at least one countywide contest in a primary election for an RLA. Colorado’s election rules indicate criteria that the secretary of state uses when choosing which contests to audit, such as the reported margin of victory, the number of ballots cast, or any causes for concern about the accuracy of the outcome (Colorado Secretary of State 2020).

These criteria are commendable, but the audit may be improved with additional criteria. If election officials want audits to enhance the perceived legitimacy of electoral contests, they may be well advised to also consider, as recommended by Citizens’ Oversight founder Ray Lutz (2019), the amount of publicly disclosed campaign funds in a political race when deciding which contests to audit. The amount of money spent on a political contest may indicate perceived competitiveness or significance, making high-dollar contests prime targets for fraud. Such contests may also be more visible and of greater interest to voters.

The RLA may increase public confidence in election outcomes (Hall et al. 2009), but because of its relative novelty there is little evidence thus far. While state laws may be written to emphasize transparency in the audit process, the statistical processes involved are unlikely to be considered easily comprehensible by the public.

Whether public trust in the results of an audit is ­higher than in the software programs and algorithms used in electronic voting equipment may be worthy of future research. At a minimum, education of the ­electorate about the advantages of RLAs would be beneficial.


There is no simple approach for securing elections in the United States. With the current patchwork of state and local election regulations, the scope of threats to the election infrastructure is vast. The use of electronic equipment for collecting, storing, and tabulating votes brings with it the potential for invalid vote records due to either intentional or unintentional equipment malfunction. Paper ballot systems, although typically less susceptible to fraud, require that voters mark their ­ballots properly and that these marks be clearly distinguishable for those counting votes. Any new election process vetted for implementation must take into account the specific electoral environment in which it will be introduced.

RLAs may nonetheless prove to be sufficiently useful for more states to consider incorporating them in their election procedures. They provide supplemental statistical proof that election outcomes are valid, and can be implemented in a variety of ways to accommodate different types of elections.

While the extent to which RLAs increase public trust in elections is thus far unconfirmed, their potential makes them an important option to consider in democratic election best practices.


Branscomb H. 2017. Ballot manifest discussion: Colorado risk limiting vote tabulation audit. Informal Conf of Risk-­Limiting Audit Representative Group, Mar 17, ­Denver.

Colorado Secretary of State. 2020. Rule 25: Post-election Audit. Election Rules [8 CCR 1505-1]. Denver.

Hall J, Miratrix L, Stark P, Briones M, Ginnold E, Oakley F, Peaden M, Pellerin G, Stanionis T, Webber T. 2009. Implementing risk-limiting post-election audits in California. Electronic Voting Technology Workshop/Workshop on Trustworthy Elections, Aug 10–11, Montreal.

Lindeman M, Stark P. 2012. A gentle introduction to risk-limiting audits. IEEE Security and Privacy 10(5):42–49.

Lutz R. 2019. Auditing. 3rd National Election Integrity Conf, Oct 5–6, Berkeley.

Lynch D. 2019. Checking the election: Risk-limiting audits. LegisBrief 27(26):1–2.

Rivest R. 2008. On the notion of ‘software independence’ in voting systems. Philosophical Transactions, Royal Society A 366(1881):3759–67.

About the Author:Matt Fitzgerald is a graduate student in the Technology, Cybersecurity, and Policy Program at the University of Colorado Boulder.