Flake Tracker
Identify, monitor, and prioritise unreliable tests by retry rate and run-time trend.
What is a Flaky Test?
A flaky test is one that does not produce a consistent result — it sometimes passes and sometimes fails on the same code, environment, and inputs. The most common cause is timing issues (async waits, network latency), but shared state, random data, and environment differences are also common culprits.
Flaky tests are expensive: they erode trust in the CI pipeline, cause false red builds, and require engineers to re-run jobs manually to get a green status. Flake Tracker surfaces these tests before they become a team-wide problem.
Measured by retry rate
Flake Tracker uses retry rate as the primary flakiness signal — the percentage of runs where a test only passed after at least one retry. A test with a 0% retry rate is stable. Anything above 10% warrants investigation; above 20% is critical.
Quick Start
- Navigate to QA → Flake Tracker in the sidebar.
- Review the four summary cards at the top: Total Tests, Flaky Count, Avg Retry Rate, Avg Run Time.
- Use the filter bar to narrow by severity: All, Critical (>20%), or Moderate (10–20%).
- Click any row in the table to expand detail. On desktop, the top-3 offenders panel on the right updates automatically.
- Sort the Retry Rate column descending to see the worst offenders first.
UI Walkthrough
Summary Cards
Four stat tiles at the top of the screen give you an instant health snapshot:
Total Tests
Total number of unique tests in the tracked suite. This count updates as new tests are added to the CI pipeline.
Flaky Count
Tests with a retry rate above 0%. Includes both critical and moderate severity tests.
Avg Retry Rate
Mean retry rate across all flaky tests. Rising trend indicates degrading suite reliability.
Avg Run Time
Average test execution time across all tracked tests. Used alongside retry rate to identify slow-and-flaky combinations.
Severity Filters
Three filter tabs above the test table let you focus on the tests that matter most:
- All — Show every tracked test, stable and flaky.
- Critical >20% — Tests failing more than one in five runs. These block CI confidence and should be fixed immediately.
- Moderate 10–20% — Tests that are unreliable but not yet blocking. Investigate before they cross the critical threshold.
Test Table
The main table lists every tracked test with the following columns:
| Column | Description |
|---|---|
| File | Source file path of the test (e.g. tests/auth/login.spec.ts) |
| Suite | Test suite or describe block containing the test |
| Retry Rate | Percentage of runs that required at least one retry to pass. Colour-coded: red for critical, amber for moderate, green for stable. |
| Run Time | Average execution time in milliseconds for the most recent runs |
| Trend | Mini sparkline chart showing retry rate over the last 10 runs. Upward trend is a warning signal. |
| Last Seen | Timestamp of the most recent run (passed or failed) |
Top Offenders Panel (Desktop)
On wider screens, a right-side panel automatically highlights the three tests with the highest retry rate. Each entry shows the file, suite, retry percentage, and a larger trend chart. Use this panel for a quick daily health check without scrolling the full table.
Run Time Trend
Below the test table, a bar chart shows average run time per test over recent CI runs. Spikes in run time often correlate with flakiness — a test that suddenly takes 3× longer usually indicates a timing-dependent assertion that is now racing against a slower network or database response.
Severity Thresholds Explained
The thresholds are based on common industry benchmarks for test suite reliability:
- 0% — Stable. The test always passes on the first attempt.
- 1–9% — Low flakiness. Worth watching but not urgent.
- 10–20% — Moderate. The test fails roughly one in ten runs. Investigate root cause within the current sprint.
- >20% — Critical. The test fails more than one in five runs. Engineers are likely re-running CI jobs manually because of this test. Fix immediately or quarantine.
Quarantine flaky tests
If a critical test cannot be fixed immediately, quarantine it (skip or tag it as flaky in your test runner config) to stop it polluting the CI signal. Track it in your backlog and use Flake Tracker to confirm the fix when it lands.
Common Root Causes
When you identify a flaky test, the following checklist covers the most common causes:
- Missing await / race condition — An async operation completes after the assertion fires. Add explicit waits or use polling assertions.
- Shared global state — One test mutates state that another test depends on. Isolate test data or reset state in beforeEach/afterEach.
- Network / API timing — The test calls a real API that occasionally times out. Mock the API for unit tests; add a higher timeout for integration tests.
- Date/time dependency — The test uses
new Date()and behaviour changes near midnight or in a different timezone. Freeze time in tests. - Random data without seed — Test uses random data that occasionally produces an edge case. Seed the random number generator or use deterministic fixtures.
- Resource leak — A previous test leaves a database record, open socket, or file handle that interferes with the next test. Add teardown cleanup.