{"id":184,"date":"2026-10-21T15:00:00","date_gmt":"2026-10-21T15:00:00","guid":{"rendered":"https:\/\/devopsschool.cn\/blog\/?p=184"},"modified":"2026-10-21T15:00:00","modified_gmt":"2026-10-21T15:00:00","slug":"testops-flaky-team-story","status":"publish","type":"post","link":"https:\/\/devopsschool.cn\/blog\/testops-flaky-team-story\/","title":{"rendered":"A Quarter of TestOps: One Team’s Flaky-Test Story"},"content":{"rendered":"

The team lead printed a screenshot of the CI dashboard and pinned it to the wall of the standup room. Ninety days of build history, and the red-green pattern looked like static on a broken television. “This,” she said, “is our quality signal.” Nobody argued, which was itself the problem. This is the story of what that twelve-person platform team did over the following quarter, told here because their situation is ordinary and their recovery is repeatable. The discipline they stumbled into has a name — TestOps, the practice of running your automated checks like a production service rather than a pile of scripts — and every step they took can be copied without asking anyone’s permission.<\/p>\n

Week One: Counting the Damage<\/h2>\n

The first move was not a tool. It was a tally. For two weeks, whenever a build failed, someone recorded the cause: real defect, environment trouble, or a check that failed and later passed with no change to the code. The results shocked nobody and convinced everybody. Roughly four failures in five fell into the last bucket, which meant the team was spending its evenings arbitrating arguments between machines. Their verdict stream, they finally admitted, was not a signal at all. It was weather.<\/p>\n

What TestOps Would Call This Moment<\/h2>\n

Every operations discipline begins with the admission that a service is failing its users. Here the users were the developers, the promised product was a trustworthy pass-or-fail verdict per change, and the service was defaulting on the promise daily. The Ops Atlas write-up of TestOps<\/a> frames the whole discipline around that promise, which is what makes it a useful label for the quarter that followed: the team stopped asking “how do we fix this build?” and started asking “how do we run a verdict service people can believe?”<\/p>\n

Week Three: Naming the Flakes Out Loud<\/h2>\n

Morale improved the moment the problem got vocabulary. Google’s engineers, publishing their own fleet-wide numbers in 2016, reported that about 1.5 percent of test runs were affected by flakiness, that roughly 16 percent of their tests showed some flaky behaviour, and that when a run flipped from green to red, 84 percent of the time the culprit was a flake, not a true defect. The team’s wall chart matched the shape of those numbers almost exactly. That was the turning point in the argument with management: the pattern was not a local failure of discipline, it was the documented default state of large suites everywhere, and it had standard remedies.<\/p>\n

Week Six: Quarantine With Dates Attached<\/h2>\n

The first remedy was containment. Each check confirmed as unreliable was moved out of the blocking path — but every quarantine entry now carried three things it had never had before: the name of the person responsible, a ticket tracking the fix, and a date after which the check would be deleted rather than forgiven. The rule was simple and posted on the wall next to the dashboard: a check sits in quarantine to be repaired, not to retire. Within a fortnight the green builds started meaning something again, because a green now said “no defects and no known liars,” instead of “nobody re-ran it enough times yet.”<\/p>\n

Week Ten: Burn-In for Newcomers<\/h2>\n

Then the team attacked the supply side. Research on flaky tests — Luo’s analysis is the usual citation — has found that about 78 percent of flaky checks are flaky from the moment they are first written, which reframes the whole hiring pipeline for new tests: a check that has just been merged is a suspect, not a colleague. So new checks entered a probation period, running daily without the power to block anyone, and earned gating rights only after a clean stretch. The false-red rate on fresh code, which had quietly been the worst in the suite, fell to near zero. Training teams take note: this single habit — prove a check before you trust it — is worth more than any tool purchase the team considered all quarter.<\/p>\n

Week Twelve: Running the Numbers Like a TestOps Team<\/h2>\n

The last habit arrived late and stayed longest. Every Monday, fifteen minutes: the false-red share for the week, the typical time a change waited for its verdict, and the slowest tenth of those waits. The team kept the review short and the chart small, and resisted the temptation to add metrics nobody acted on. What the ritual bought them was early warning. When the false-red share ticked up two weeks in a row, the cause — a batch of new checks someone had rushed past probation — was found while it was still one conversation long. In the TestOps way of working, this weekly look at a handful of honest numbers is the entire management overhead; everything else is just engineering responding to what the numbers say.<\/p>\n

What the Team Would Tell You Now<\/h2>\n

Asked what mattered, the lead gave four answers, in order:<\/p>\n