How Is a Random Number Generator Tested for Statistical Fairness?

A random number generator’s output is tested with statistical hypothesis tests, such as those defined in NIST Special Publication 800-22, which check a long sequence for detectable patterns — unequal bit frequency, run lengths, repeated blocks — at a significance level of 0.01. A passing result is evidence about the generator’s long-run behaviour, not proof that any single output was fair.

A worked example with numbers

One test in the suite, the frequency (monobit) test, checks whether the proportion of 1-bits in a long sequence is close to one half. Testing 1,000,000 bits from a working generator, the count of 1s should land close to 500,000; the test converts the observed deviation into a p-value and compares it against the α = 0.01 significance threshold. A p-value at or above 0.01 counts as a pass; below 0.01 counts as a fail. Because the threshold is 0.01, roughly 1% of sequences from a genuinely unbiased generator are still expected to fail any single test purely by chance, which is why the full suite is judged as a whole battery, not by one test alone. A second test in the suite, the runs test, instead checks the number and length of consecutive identical-bit sequences — for example, whether unbroken runs of 1s appear far more often than an unbiased source would produce — catching a different kind of pattern than the frequency test alone would miss.

How to set the rule before the result

To evaluate a generator responsibly, run the full statistical test battery on a large output sample — not one short sequence — then check the proportion of sub-sequences that pass each test and whether the resulting p-values are themselves roughly evenly spread, rather than clustered. A generator used for this site’s random number generator or integer range picker ultimately relies on the browser’s own cryptographic implementation, which browser vendors — not individual websites — are responsible for validating against this kind of test battery.

Common mistakes that change the odds or the process

The most common mistake is treating “passed NIST SP 800-22” as proof that a generator’s output is unpredictable or secure. Statistical randomness and cryptographic unpredictability are different properties: a deterministic algorithm with a known, guessable seed can produce a sequence that passes every statistical test in the suite while still being fully predictable to anyone who knows or can guess the seed. A second mistake is applying a suite designed for long sequences to a handful of coin flips or dice rolls; the tests need thousands to millions of bits to produce a meaningful p-value, so a ten-flip session cannot be meaningfully “NIST tested” at all. A third mistake is quoting a generator’s test results without stating which version or configuration was tested; a software update, a changed seeding method, or a different underlying hardware entropy source can all change a generator’s statistical behaviour, so a certification from years ago does not automatically describe the code running today.

Where this method stops being appropriate

Statistical testing characterises a generator’s long-run output distribution; it cannot certify that one specific past draw — the coin flip that picked tonight’s restaurant, or the raffle winner announced an hour ago — was generated fairly, because a single output carries no statistical signal on its own. It also does not evaluate the surrounding system: a correctly tested random source can still be misused by flawed range-conversion code layered on top of it. Nor does statistical testing evaluate whether a page correctly discloses which generator it uses; a claim of “cryptographically secure randomness” next to code that quietly falls back to a weaker source on an older browser is a disclosure problem the test battery has no way to detect from the output alone.

How the random source fits into the rule

NIST SP 800-22 defines the statistical test battery, its α = 0.01 significance level, and the pass/fail rule used to evaluate long random and pseudorandom sequences. The publication is maintained by the US National Institute of Standards and Technology, a government metrology body, and is the reference implementation most cryptographic libraries and hardware random-number generator vendors test against before publishing their own compliance claims.

What passing a statistical test suite does not prove

A statistical test battery answers one narrow question: does this long output sequence look the way an unbiased, independent source would be expected to look? It does not verify who controls the generator, whether its seed is secret, whether the implementation has since changed, or whether a single displayed result actually came from the tested code path rather than a different, unvalidated fallback. Those are separate engineering and security questions a purely statistical pass cannot answer.

It also does not evaluate any single website’s specific use of that generator. A browser vendor testing its own crypto.getRandomValues() implementation is answering “does our output stream look statistically unbiased,” not “does this particular page on this particular site convert that output into a fair choice among five names.” The second question depends on the page’s own range-conversion code, which a passing NIST result says nothing about.

Statistical randomness vs. cryptographic security

A generator can satisfy the first without the second, which is exactly why browsers expose a separate, dedicated API for any use where unpredictability, not just statistical evenness, is the actual requirement — the distinction covered in why Math.random() can be the wrong choice for a random picker.

A simple mental test for the difference: a spreadsheet formula that outputs a fixed, well-shuffled 1,000-number sequence built once by a human and reused every time would fail badly on statistical tests only if the sequence were poorly built — it could in principle be constructed to pass most of them — yet it would still be completely predictable to anyone who obtained the sequence once. Passing statistical tests is necessary for a good generator; it is not sufficient for a secure one.

This is also why security-sensitive systems, such as those generating session tokens or cryptographic keys, are evaluated with additional entropy-source standards on top of statistical output testing — a topic covered by NIST’s SP 800-90 series rather than SP 800-22 alone. A casual browser tool for coins, dice, or list picks does not need that additional layer, because nothing of security value depends on an attacker being unable to guess the next result; the layer becomes necessary only once prediction would let someone gain an unfair advantage or bypass a protection.

The full NIST SP 800-22 test battery

Each test targets a different failure mode — the Runs test catches too few or too many alternations, the Spectral test catches periodic patterns a simple frequency count would miss entirely, and so on. A generator is judged on the whole battery, since a flaw one test misses is often exactly what a different test in the list is designed to catch.

A worked frequency-test example: how sensitive the test becomes at scale

The frequency test’s statistic is s_obs = |n₁ − n₀| ÷ √n, converted to a p-value; a p-value at or above 0.01 passes. At a sequence length of 1,000,000 bits, that formula is far more sensitive than intuition suggests:

At this sample size, a count of 1-bits that misses an even split by just 0.13 percentage points is already enough to fail. That sensitivity is the point: a generator meant to protect something valuable needs to be accurate far beyond what a casual glance at a printed figure like “50.1% ones” would ever flag as suspicious.

Why failing one test out of fifteen is not automatically alarming

Because the significance level is set at α = 0.01, a genuinely unbiased generator is still expected to fail any single test about 1% of the time purely by chance. Run all fifteen tests in the battery on independent sequences from a good generator, and the chance that at least one of the fifteen comes back a “fail” by chance alone is 1 − 0.99¹⁵ ≈ 13.99% — not a rare event at all. This is exactly why a responsible evaluation looks at the pattern across many repeated sequences and across the whole battery, rather than treating one isolated failing result, on its own, as proof that a generator is broken.

How much data the battery actually needs from you

NIST’s documentation for the battery is built around evaluating long sequences — commonly cited guidance for the suite recommends sequences on the order of at least one million bits each, with the fuller evaluation run across many such sequences rather than just one, so both the individual test outcomes and the spread of resulting p-values can be checked. A home experiment recording a few hundred or even a few thousand coin flips or dice rolls, however carefully logged, is nowhere near the scale this methodology was designed to evaluate, which is one reason this kind of certification realistically belongs to hardware and software vendors rather than to an individual page owner testing a single website feature.

What a single-sequence test does not check

Every test in the battery evaluates one sequence’s own internal structure — its bit balance, its run lengths, its periodicity — not its relationship to any other sequence generated separately. It has nothing to say about whether two independently produced outputs, from two different processes drawing on a shared but limited entropy source at nearly the same moment, might end up correlated with each other in a way that never shows up when each sequence is examined on its own. That kind of cross-sequence independence is a different property from the one this statistical battery is built to measure, and it needs its own separate evaluation.

Related DiceDecide tools

• Generate a random number
• Pick a bounded random integer
• Read whether browser randomness is truly random
• Run a streak simulation