Bazel & Buildbarn guides

Troubleshooting · Reviewed November 19, 2026

Find out why a Bazel test sometimes fails.

Start with the exact test target and a failing invocation. A later pass does not explain the failure. First establish whether the test executed, what failed, and which conditions changed between attempts.

Separate a test failure from an execution failure

Open the failing test's log and result, including individual shards and attempts when present. An assertion failure points into the test or the behavior it checks. A timeout, missing input, failed input fetch, or worker loss needs a different investigation. A CI job failing before the test starts is not evidence that the test is flaky.

Keep the commit, Bazel version, target, configuration, test arguments, platform, and original failure output. Compare the failing and passing invocations before changing retries or timeouts.

Repeat the test without hiding failed attempts

Replace the label below with one test target. Keep the configuration and test arguments used by the failure. This diagnostic run disables reuse of cached test results and sets one attempt per requested run. It does not disable build-action caching.

bazel test //your/package:suspect_test \
  --nocache_test_results \
  --runs_per_test=20 \
  --flaky_test_attempts=1 \
  --test_output=errors \
  --build_event_json_file=/tmp/suspect-test.bep.json

--runs_per_test requests repeated runs. --flaky_test_attempts controls retries after a failure. They answer different questions. Preserve the BEP file and the test logs, including attempt and shard results, before another invocation overwrites outputs.

Twenty passes mean no failure was observed in that sample. They do not prove the absence of a timing race or a problem that occurs only on a particular worker. If no failure appears, reproduce the original environment and workload rather than immediately increasing the sample size.

Change one condition at a time

What differsWhat to investigate
Only parallel runs failShared files, fixed ports, global state, resource pressure, or missing synchronization.
Only CI or one worker pool failsToolchain and runtime versions, platform, locale, time zone, environment, and machine resources.
Only remote execution failsUndeclared inputs, unavailable host tools, runfiles, network dependencies, and worker compatibility.
The same assertion fails intermittentlyRandom seeds, ordering assumptions, asynchronous completion, clocks, and external services.
A retry passes after a failureKeep both attempts. Check what changed; a passing retry alone is not a root cause.

Use the test framework's seed or filtering option when available and record its value. Use Bazel-provided temporary directories rather than shared fixed paths. Fix the dependency, isolation, or synchronization problem identified by the evidence; adding a sleep can leave the race intact.

Check the flags with a small reproducible example

The accompanying fixture deliberately fails run 2 when repeated. It is a controlled demonstration of test-run reporting, not a naturally flaky test or a measurement of Hermetiq's detection rate. It also includes a passing target so you can compare cached results with fresh execution.

Recorded caseObserved result
Fresh passing targetExit 0; one passing result, not cached locally.
Unchanged passing rerunExit 0; the result is cached locally.
Passing target with --nocache_test_resultsExit 0; a new passing execution.
Three runs of the controlled failureExit 3; runs 1 and 3 pass, run 2 fails.

Verify a test repair on shared workers

The earlier local example isolates test-result reuse. This additional recording connects the test to Hermetiq Cloud and runs it on Buildbarn workers. Keep the before-and-after run results, not just the final green invocation.

Recorded November 19, 2026 · Bazel 9.2.0 · macOS arm64 client · Hermetiq Cloud shared Buildbarn workers

This test deliberately fails when TEST_RUN_NUMBER=2. We ran it three times on shared workers with --cache_test_results=no, removed that condition, then ran the same three-test check again. It is a controlled failure, not a naturally flaky test.

StepRecorded outcomeEvidence
Before the repairRun 1: passed; Run 2: failed; Run 3: passed
Bazel exit code 3
Action records
Bazel event excerpt
After the repairRun 1: passed; Run 2: passed; Run 3: passed
Bazel exit code 0
Action records
Bazel event excerpt

The exact repair changes fail_run_two = True to False in the sample BUILD rule. The published event excerpts retain each run number and status. Six fresh remote test runs verify this condition and its removal; three passes do not establish that an arbitrary real test is free of intermittency.

Read the one-line repair · Read the failed run’s log · Read the verification log

This small fixture verifies a setup and investigation workflow. It does not measure production performance, migration effort, flake-detection accuracy, or every toolchain’s compatibility. The downloadable JSON includes selected evidence fields, with local paths and project identifiers removed where noted.

Download source and recorded evidence · Reproduction steps and scope · Offline verifier

Try the shared-worker setup in your own project

Verify the fix where the failure occurred

Rerun the target with cached test results disabled and the same repetition settings. Then exercise the affected suite, concurrency, and execution platform. Keep the original failure and the verification output together. If you temporarily quarantine a test, assign an owner and a condition for restoring it.

Hermetiq uses connected Bazel build events for test history and outcomes. Open the failing invocation and compare it with relevant passes; use failed-action and remote-execution data when those sources are connected. A coding client connected through MCP can investigate that context and make separately authorized code changes. Those permissions and the verification step still matter.

Sources and review

Reviewed against the primary references below and Hermetiq's setup and deployment documentation on November 19, 2026. Check flags and requirements against the versions you run. The adoption and operating checklists are engineering guidance, not measured performance claims.