A test fails. Someone re-runs it. It passes. Nobody investigates further, because the build is green now and there’s a release to ship. A week later the same test fails again, and this time a different engineer re-runs it, again without digging in, because that’s what happened last time too.
This is how flaky tests quietly destroy trust in a test suite. Not through one dramatic incident, but through hundreds of small ones, each individually easy to dismiss.
Why teams stop trusting red builds
Once a handful of tests earn a reputation for failing “for no reason,” the behaviour around them changes. Developers start re-running failures before reading them. Pull requests get merged on a second or third attempt without anyone confirming what actually broke. Tickets get filed for “intermittent failures” and then quietly closed months later, unresolved, because nobody could reproduce them on demand.
The real cost isn’t the re-run itself: it’s the erosion of judgement. When enough failures turn out to be noise, engineers stop treating any failure as reliable signal. That’s the moment a genuine regression can slip through: the one time a red build really did mean something, it got treated the same as the ninety-nine times it didn’t.
Worse, this problem compounds. A team that can’t tell which tests are flaky has no principled way to decide where to spend fixing effort. Should you rewrite the test, add a wait condition, fix a race condition in the code under test, or just delete it? Without data, that decision is a guess dressed up as an opinion.
How Obvyr detects and ranks flaky tests
Obvyr’s core job is turning that guesswork into evidence. Every test execution captured by the CLI or Gradle plugin (whether it ran on a developer’s laptop or in a CI pipeline) is stored individually, tied to the specific test it belongs to. Over time, that builds a full pass/fail history for every test in your suite, not just a snapshot of the last run.
From that history, Obvyr automatically detects and ranks flaky tests: ones whose pass/fail behaviour doesn’t correlate cleanly with code changes, and instead looks like noise. Every test gets a health profile built from its own execution record (pass/fail counts, pass rate, flakiness, and full history), so “is this test flaky” stops being a matter of institutional memory (“I think that one’s always been a bit off”) and becomes something you can actually look up.
Those individual health profiles roll up into a project-level flakiness rate: a single number that tells you, at a glance, whether your suite’s overall reliability is improving or getting worse.
How the flakiness score is tuned
A flakiness score is only useful if it reflects the runs you actually care about, so Obvyr lets you configure that per project rather than forcing a one-size-fits-all default. Two settings control it: which executions count toward the score at all, filtered by required tags, and a minimum-executions threshold a test has to clear before it’s judged flaky or not.
That tag filter matters in practice more than it sounds. A test that looks flaky across “every execution anywhere” might actually just be sensitive to a developer’s local machine: different timing, different data, different network conditions. Tag your local runs distinctly (most teams already do this for CI detection) and exclude them from the flakiness calculation, and the score reflects what happens in the environment that actually ships, not the noisiest one. The minimum-executions threshold does similar work from the other direction: a test that’s only run twice hasn’t earned a verdict yet, so it doesn’t get flagged as flaky (or trusted as stable) on too little evidence.
The result is a concrete answer to the question that usually gets settled by gut feeling: which of today’s failures should you trust, and which ones are just the suite being the suite?
Reliability is only half the picture, though. Once you know which tests are flaky, the natural next question is which tests are worth keeping at all.