Hermetiq Cloud pilot: execution, cache invalidation, and a controlled test repair
Recorded November 19, 2026. Bazel 9.2.0, macOS arm64 client, shared Hermetiq
Cloud Buildbarn workers. No customer source, credentials, or customer workload.

What ran
The manifest rule declares one text input and one environment variable. A shell
action writes the mode and copies the input. The sample test deliberately fails
for TEST_RUN_NUMBER=2 until the one-line repair removes that condition. It is
not a naturally flaky test or a demonstration of automatic root-cause repair.
All six scenarios used new output bases. The first disabled remote cache reads.
The test scenarios used --runs_per_test=3 and --cache_test_results=no.

Observed
cold: 1 remote manifest execution, exit 0.
warm: 1 remote cache hit from a fresh output base, exit 0.
environment-change: PILOT_MODE changes from release to debug; 1 remote
execution, a different action key and output digest, exit 0.
restored: release restores the original key and remote cache hit, exit 0.
test-failure: remote runs 1 and 3 pass; run 2 fails; Bazel exits 3.
test-fixed: after the supplied patch, all three remote runs pass; Bazel exits 0.

Hermetiq verification
The invocations appeared in the authenticated Hermetiq dashboard. On the changed
manifest invocation, Cache Metrics reported one remote Action Cache lookup, one
miss and Env Changed. A project-scoped Barn Owl investigation called
find_cache_events and compare_build_configurations. The retained tool-result
excerpts identify ENV_CHANGED and the added --define=PILOT_MODE=debug. The
independent Bazel action records show the exact release-to-debug value change,
unchanged shell arguments and input, and the restored hit. The Command digest
includes environment variables; its change does not imply changed argv.

Reproduce in your own project
1. Extract the ZIP into a new directory. Install Bazelisk. The .bazelversion file
   selects 9.2.0. Keep the original BUILD.bazel for the first five scenarios.
2. Connect this workspace to your own Hermetiq Cloud project using Quickstart.
   Install its credential helper and store your own credentials outside source.
   Enable the shared-worker remote-execution settings in the hermetiq config.
   No helper, credentials, or usable project configuration is included here.
   See https://www.hermetiq.com/docs/remote-execution#managed-rbe-start
3. Use the commands below. Choose a new directory for PILOT_OUTPUT. Do not reuse
   another experiment's output bases. Omit disk caching from this config.

PILOT_OUTPUT=$(mktemp -d)
bazelisk --batch --output_base="$PILOT_OUTPUT/cold" build --config=hermetiq //:manifest --noremote_accept_cached --remote_download_outputs=all
bazelisk --batch --output_base="$PILOT_OUTPUT/warm" build --config=hermetiq //:manifest --remote_download_outputs=all
bazelisk --batch --output_base="$PILOT_OUTPUT/changed" build --config=hermetiq //:manifest --define=PILOT_MODE=debug --remote_download_outputs=all
bazelisk --batch --output_base="$PILOT_OUTPUT/restored" build --config=hermetiq //:manifest --remote_download_outputs=all
bazelisk --batch --output_base="$PILOT_OUTPUT/test-before" test --config=hermetiq //:test_sample --runs_per_test=3 --cache_test_results=no --test_output=errors
patch -p1 < fix.diff
bazelisk --batch --output_base="$PILOT_OUTPUT/test-after" test --config=hermetiq //:test_sample --runs_per_test=3 --cache_test_results=no --test_output=errors

The penultimate test command is expected to fail. Check each execution log and
testResult event; a green cached result does not establish a fresh test run.
To capture your own records, add --execution_log_json_file=<NEW_FILE> and
--build_event_json_file=<NEW_FILE> to each command. Do not overwrite this bundle's
recorded files. Your invocation IDs, platform paths and action keys may differ.

Verify this recording offline
python3 verify.py
The verifier checks action keys and outputs across the four build cases,
unchanged inputs/argv/platform, all six test outcomes and remote test-setup
actions, the exact repair, and every SHA256SUMS entry. Bazel also labels its
test.xml generation actions TestRunner; some of those auxiliary actions hit
cache in the fixed run. The actual test-setup actions all executed remotely.
No service or credentials
are needed to verify the published recording.

Evidence format and limits
recorded/*.actions.json contains selected original execution-log fields, not
complete logs. recorded/*.bep-excerpt.json retains selected started, finished,
testResult and testSummary events. Project IDs in artifact URIs are replaced by
DEMO_PROJECT. Text logs replace local recording/workspace paths with placeholders.
The two tool-result excerpts select fields from the actual Barn Owl tool details;
they omit project identifiers, user identifiers, local paths and unrelated fields.
recorded/runs.json records invocation IDs, outcomes and output hashes.
Shared service versions were not pinned; this is a dated observation, not a
cross-version guarantee. It does not measure production speedups, detection
accuracy, migration effort or universal workload compatibility. Three passes
after removing a deliberately failing condition do not establish that an
arbitrary real test is free from intermittent failures.

References
https://bazel.build/remote/cache-remote
https://bazel.build/reference/test-encyclopedia
https://github.com/bazelbuild/remote-apis/blob/main/build/bazel/remote/execution/v2/remote_execution.proto
