Flaky test
A flaky test is one that sometimes passes and sometimes fails without any code change — typically caused by timing dependencies, shared state, network/external-system reliance, or race conditions. Flaky tests destroy trust in the test suite: developers learn to ignore failures and miss real regressions.
Flakiness is the silent killer of test discipline. The progression is predictable: one flaky test gets ignored, then re-run; the team learns to re-run failures; eventually any test failure is presumed flaky and re-run. By the time a real regression hits, nobody catches it. The remediation hierarchy: fix the root cause (eliminate the timing dependency, isolate the shared state, mock the external service), quarantine the flaky test if the fix is non-trivial, and delete the test if it's both flaky and low-value. The most expensive option — accepting flakiness — is also the most common; the cost surfaces later as missed regressions.
Discussed in our use-cases
ICP-targeted pages where flaky test is part of the framing.
Related terms
- Test quarantine
Test quarantine moves a flaky test out of the blocking suite into a tagged 'quarantine' suite that runs but doesn't fail the build.
- Test isolation
Test isolation is the property that each test runs independently — no shared state, no execution-order dependency, no side effects bleeding to the next test.
- Test impact analysis
Test impact analysis identifies the subset of tests relevant to a code change — by analysing the dependency graph between source files and test files — so CI runs only the affected tests instead of the entire suite.