Back to Cookbook
KiloClaw

Flaky Test Quarantine

Stop flaky tests from blocking PRs while you fix them

Quarantine known flaky tests into a separate lane so they don't block merges, while preserving visibility and enforcing a fix-by deadline.

CommunitySubmitted by CommunityWork15 min

INGREDIENTS

🐙GitHub

PROMPT

Create a skill called "Flaky Test Quarantine". The skill must: - Define how to identify flaky tests (reruns + fingerprints) - Propose the quarantine mechanism for our test framework - Specify CI changes: separate lane, reporting, and non-blocking behavior - Provide a tracking template (owner, SLA, exit criteria) Output in Markdown plus example snippets (pseudo-code acceptable).

How It Works

Flaky tests destroy trust and waste developer time. This recipe sets up a quarantine

pattern: detect → tag → isolate → fix → unquarantine.

Triggers

  • Tests fail intermittently with no code changes
  • Developers rerun CI until it passes
  • PR merges are blocked by "random red" failures

Steps

  1. Identify flaky tests by rerun history and failure fingerprints.
  2. Tag tests as flaky (framework tag/annotation) and route them to a separate CI stage.
  3. Make PR gating depend on non-flaky suites; still run flaky lane for signal.
  4. Track flaky tests in a single dashboard/issue with owners and deadlines.
  5. Require a root-cause fix (not just retries) to remove from quarantine.

Expected Outcome

  • PR pipeline regains trustworthiness.
  • Flakiness becomes bounded work with ownership, not background chaos.

Example Inputs

  • "These 8 tests fail randomly on main."
  • "E2E tests flake due to timing/race conditions."
  • "Network-dependent tests fail in CI only."

Tips

  • Quarantine is a temporary containment strategy, not acceptance of flakiness.
Tags:#flaky-tests#testing#ci-cd#developer-productivity