Separate a test failure from an execution failure
Open the failing test's log and result, including individual shards and attempts when present. An assertion failure points into the test or the behavior it checks. A timeout, missing input, failed input fetch, or worker loss needs a different investigation. A CI job failing before the test starts is not evidence that the test is flaky.
Keep the commit, Bazel version, target, configuration, test arguments, platform, and original failure output. Compare the failing and passing invocations before changing retries or timeouts.
Repeat the test without hiding failed attempts
Replace the label below with one test target. Keep the configuration and test arguments used by the failure. This diagnostic run disables reuse of cached test results and sets one attempt per requested run. It does not disable build-action caching.
bazel test //your/package:suspect_test \
--nocache_test_results \
--runs_per_test=20 \
--flaky_test_attempts=1 \
--test_output=errors \
--build_event_json_file=/tmp/suspect-test.bep.json--runs_per_test requests repeated runs. --flaky_test_attempts controls retries after a failure. They answer different questions. Preserve the BEP file and the test logs, including attempt and shard results, before another invocation overwrites outputs.
Twenty passes mean no failure was observed in that sample. They do not prove the absence of a timing race or a problem that occurs only on a particular worker. If no failure appears, reproduce the original environment and workload rather than immediately increasing the sample size.
Change one condition at a time
| What differs | What to investigate |
|---|---|
| Only parallel runs fail | Shared files, fixed ports, global state, resource pressure, or missing synchronization. |
| Only CI or one worker pool fails | Toolchain and runtime versions, platform, locale, time zone, environment, and machine resources. |
| Only remote execution fails | Undeclared inputs, unavailable host tools, runfiles, network dependencies, and worker compatibility. |
| The same assertion fails intermittently | Random seeds, ordering assumptions, asynchronous completion, clocks, and external services. |
| A retry passes after a failure | Keep both attempts. Check what changed; a passing retry alone is not a root cause. |
Use the test framework's seed or filtering option when available and record its value. Use Bazel-provided temporary directories rather than shared fixed paths. Fix the dependency, isolation, or synchronization problem identified by the evidence; adding a sleep can leave the race intact.
Check the flags with a small reproducible example
The accompanying fixture deliberately fails run 2 when repeated. It is a controlled demonstration of test-run reporting, not a naturally flaky test or a measurement of Hermetiq's detection rate. It also includes a passing target so you can compare cached results with fresh execution.
| Recorded case | Observed result |
|---|---|
| Fresh passing target | Exit 0; one passing result, not cached locally. |
| Unchanged passing rerun | Exit 0; the result is cached locally. |
| Passing target with --nocache_test_results | Exit 0; a new passing execution. |
| Three runs of the controlled failure | Exit 3; runs 1 and 3 pass, run 2 fails. |
Verify a test repair on shared workers
The earlier local example isolates test-result reuse. This additional recording connects the test to Hermetiq Cloud and runs it on Buildbarn workers. Keep the before-and-after run results, not just the final green invocation.
Recorded November 19, 2026 · Bazel 9.2.0 · macOS arm64 client · Hermetiq Cloud shared Buildbarn workers
This test deliberately fails when TEST_RUN_NUMBER=2. We ran it three times on shared workers with --cache_test_results=no, removed that condition, then ran the same three-test check again. It is a controlled failure, not a naturally flaky test.
| Step | Recorded outcome | Evidence |
|---|---|---|
| Before the repair | Run 1: passed; Run 2: failed; Run 3: passed Bazel exit code 3 | Action records Bazel event excerpt |
| After the repair | Run 1: passed; Run 2: passed; Run 3: passed Bazel exit code 0 | Action records Bazel event excerpt |
The exact repair changes fail_run_two = True to False in the sample BUILD rule. The published event excerpts retain each run number and status. Six fresh remote test runs verify this condition and its removal; three passes do not establish that an arbitrary real test is free of intermittency.
Read the one-line repair · Read the failed run’s log · Read the verification log
This small fixture verifies a setup and investigation workflow. It does not measure production performance, migration effort, flake-detection accuracy, or every toolchain’s compatibility. The downloadable JSON includes selected evidence fields, with local paths and project identifiers removed where noted.
Download source and recorded evidence · Reproduction steps and scope · Offline verifier
Verify the fix where the failure occurred
Rerun the target with cached test results disabled and the same repetition settings. Then exercise the affected suite, concurrency, and execution platform. Keep the original failure and the verification output together. If you temporarily quarantine a test, assign an owner and a condition for restoring it.
Hermetiq uses connected Bazel build events for test history and outcomes. Open the failing invocation and compare it with relevant passes; use failed-action and remote-execution data when those sources are connected. A coding client connected through MCP can investigate that context and make separately authorized code changes. Those permissions and the verification step still matter.
Sources and review
Reviewed against the primary references below and Hermetiq's setup and deployment documentation on November 19, 2026. Check flags and requirements against the versions you run. The adoption and operating checklists are engineering guidance, not measured performance claims.