Your Bazel cache hit rate is lying to you: a field guide

The process summary at the end of a Bazel build is a poor diagnosis. It lists remote cache hits, remote runs, local runs, and internal spawns. Teams treat the hit share as cache health. CI still misses its window, laptops and runners rebuild the same source, and a local test fails on a worker that never had an input.

That line counts actions. It is not wall time, and it does not name the action key that changed. This guide is for people running a remote cache or remote execution. The records are an execution log, the Build Event Protocol, and a JSON profile.

The percentage mixes different events

remote cache hit means the remote cache returned the result. remote means a worker ran the action. linux-sandbox, darwin-sandbox, and worker are local execution. disk cache hit is the local disk cache. internal is mostly bookkeeping, such as symlink actions. Bazel's remote-cache debugging guide says the internal bucket can be ignored here, and that a warm local cache is absent from the summary. If the process count looks too small, bazel clean forces the next run to consult the remote cache. Keep that in a diagnosis. Do not put it in CI.

One ratio hides which mnemonic missed, whether it sat upstream of a fan-out, and whether the cause was a new key, an eviction, or another --remote_instance_name. It also hides analysis, cache RPCs, uploads, queue time, execution, and downloads.

buildMetrics.actionSummary in a JSON BEP file is that count in a form you can query. It is still a count.

Why an aggregate hit rate misleads is the dashboard version of that number. BEP does not carry the full key-forming input set, so an execution log is what shows why a key changed.

Environment variables and absolute paths

A remote hit requires the same action digest: the command, declared inputs and their contents, the environment variables Bazel included, the execution platform, and the listed outputs. The execution log stores that set as a SpawnExec record from src/main/protobuf/spawn.proto.

--incompatible_strict_action_env defaults to true: a fixed PATH, and no inherited LD_LIBRARY_PATH. --action_env=NAME copies the client value. --action_env=NAME=value pins one. Inheriting USER, HOME, JAVA_HOME, or a CI-only CI=true splits compiles whose sources match. The command-line reference says so.

Absolute paths leak through outputs later actions read. In a genrule, $ starts a Bazel expansion, so a shell variable is $$. A cmd that writes $$PWD or the output base into a header differs on every machine. C and C++ do the same when __FILE__ is a host path and the compile line lacks -ffile-prefix-map or -fdebug-prefix-map. A repository rule that reads a host path changes the repo, and the miss appears downstream.

bazel-out/k8-fastbuild/bin and bazel-out/darwin_arm64-fastbuild/bin are different command lines, so Linux CI and an Apple Silicon laptop do not share those actions. --experimental_output_paths=strip can rewrite eligible paths. It is experimental, needs a cache or remote execution, and Starlark actions opt in with supports-path-mapping.

Non-hermetic genrules and local-only files

A genrule stays cacheable when cmd reads declared srcs and tools and writes declared outs. Point at those files with $(location), not a laptop path. Reading an undeclared file, running date, or calling an internal host changes the digest whenever that outside input changes. local = True forces the local strategy: no sandbox, and no remote execution.

The tags local, no-remote, and no-remote-exec keep an action off workers. no-remote-cache skips the remote cache. no-cache skips caching. Use them for a credential that exists on one machine. They are also how a remote build still depends on the CI image.

Remote execution uploads the declared input tree. An undeclared path is not in it. The laptop succeeds because the file is on disk. The worker fails, or it reads a different file from its image. cacheable, remotable, inputs, and targetLabel in the log name the spawn. Print the hashed command before comparing machines:

bazel aquery 'mnemonic("Genrule", //pkg:target)'

Stamping

--stamp is off by default. A genrule overrides it: stamp = 1 always stamps, stamp = 0 never does, and stamp = -1 follows the flag. Stamped actions can read stable-status.txt and volatile-status.txt. --workspace_status_command prints the pairs. STABLE_ keys belong in the stable file and should change the digest with the commit. Other keys are volatile. BUILD_TIMESTAMP changes every invocation, so an action that inputs the volatile file misses every build and invalidates dependents. CI with --stamp and laptops with --nostamp never share those actions. Each side can still look fine alone.

Toolchains, platforms, and instance names

Linux x86-64 CI and macOS arm64 laptops are different execution platforms. Different compilers should miss. The bug is both sides resolving different platforms when they were supposed to match. A host-autodetected C++ toolchain puts that machine's compiler into the action digest. Pin the toolchain and select it with --platforms and an explicit execution platform. Diff --host_platform, --extra_execution_platforms, and legacy --cpu and --host_cpu as well. They change resolution and the prefix under bazel-out.

--remote_default_exec_properties and per-target exec_properties are the platform workers see. A different property is a different cache key and can select a different queue. Workers run actions they advertise. The rest wait until --remote_timeout and often fail as DEADLINE_EXCEEDED, while other hits keep the percentage up.

--remote_instance_name is the remote API namespace. One host and two instance names do not share results. Read the expanded flags from BEP, not the rc file you expect: write --build_event_text_file=/tmp/bep.txt and find the structuredCommandLine labeled canonical.

Remote execution after the hit rate

A warm branch can report a high hit rate while a clean laptop or a new VM runs actions that were never uploaded. That is a cold cache. Identical SpawnExec records with remoteCacheHit false means the key matched and the result was not reused. cacheable: false means the rule opted out. --noremote_accept_cached disables lookups even when the digest matches, including after --config expands.

--remote_local_fallback runs locally after a remote failure. If that action can see host files, uploading the result shares bytes a worker would not have produced. --remote_download_outputs=all, toplevel, or minimal changes what comes back to the client, not the digest.

Why CI stays slow

Analysis runs on the CI machine. A new VM pays it every job, and a long-lived laptop server hides it. BEP's BuildMetrics carry aggregate timing. The JSON profile separates analysis from execution and shows the critical path. When analysis dominates, hit rate is the wrong knob.

Cache checks are RPCs, including hits. A warm server skips unchanged actions. Ephemeral CI does not. --disk_cache is local, and a fresh VM does not keep it unless you persist that directory.

Misses also wait. A busy pool, or a platform and size class with no eligible workers, turns a few misses into queue time. The profile is the client's view of that wait. Queue, input fetch, execution, and upload inside the cluster come from completed-action records, not from the percentage.

--generate_json_trace_profile
--experimental_profile_include_primary_output
--experimental_profile_include_target_label
--noslim_profile

--generate_json_trace_profile writes command.profile.gz under the output base from bazel info output_base. The other flags attach the target and primary output to the trace. bazel analyze-profile summarizes a --profile file. Open the JSON trace for the timeline.

Flaky tests

--cache_test_results reuses passes and reruns failures. A flake that eventually passes stays cached until an input changes. --nocache_test_results forces execution. It does not add a missing data file.

--flaky_test_attempts retries failures. Bazel reports FLAKY when a later attempt passes, and the command can still exit 0. default tries ordinary tests once and retries rules with flaky = True. TestResult is one attempt. TestSummary is the rollup. Read both in --build_event_json_file.

--runs_per_test repeats on purpose. Keep each attempt. Retries hide a test that reads the clock, the network, or an undeclared file. A remote sandbox has no laptop network unless the execution requirements and the worker allow it. --test_env changes the test action the same way --action_env changes a build action.

A checklist for one pair of builds

Run one command on two machines that should share a cache, with the same small source change on both.

1. Save the process summary and set it aside.
2. Record a log, BEP, and a profile. This bazel clean is for the diagnosis only:

   bazel clean
   bazel build //your:target \
     --execution_log_compact_file=/tmp/exec1.log \
     --build_event_json_file=/tmp/bep1.json \
     --build_event_publish_all_actions \
     --generate_json_trace_profile \
     --experimental_profile_include_primary_output \
     --experimental_profile_include_target_label \
     --noslim_profile

   Name the second machine's files exec2.log and bep2.json. Compact logs are the format Bazel recommends. --execution_log_json_file and --execution_log_binary_file are the other formats, and only one execution-log flag can be set. Delete bazel clean before anyone copies this into CI.

3. Diff with the parser from a Bazel checkout, not from the application repo:

   bazel build //src/tools/execlog:parser
   bazel-bin/src/tools/execlog/parser \
     --log_path=/tmp/exec1.log \
     --log_path=/tmp/exec2.log \
     --output_path=/tmp/exec1.log.txt \
     --output_path=/tmp/exec2.log.txt
   diff -u /tmp/exec1.log.txt /tmp/exec2.log.txt

   Sort changes into environment variables, command arguments, input digests, platform properties, and listed outputs. Keep mnemonic and targetLabel on each. Action JSON you already have can be compared in the action-diff tool.

4. Query buildMetrics.actionSummary and group actions by mnemonic. --build_event_publish_all_actions is what publishes successes. Fix the upstream mnemonic, then debug that cache miss.
5. Open command.profile.gz. Analysis time is a loading and configuration problem. A few long remote actions go back to the log diff. Queue time is worker eligibility and completed-action timing.
6. Diff canonical flags: --action_env, --test_env, --stamp, --platforms, --host_platform, --extra_execution_platforms, --remote_instance_name, --remote_cache, --remote_executor, --remote_download_outputs, and --define. In BUILD files, look for stamp = 1, local = True, and no-cache or no-remote.
7. Read testSummary.status. Use --nocache_test_results to execute again and --runs_per_test to repeat. Do not raise --flaky_test_attempts just to hide the failure under study.

Matching logs plus execution of actions this branch had never uploaded means the percentage was right. It is still the wrong diagnosis for every slow build.

Read it from build data you already emit

The checklist explains one pair of builds. Hermetiq can ask the same questions from data a project already emits: BEP for the invocation, targets, tests, and counts; a JSON profile for phases and the critical path when those flags are on; and remote-cache events for hits and misses. Where the cache reports reasons, they separate changed inputs, commands, environments, platforms, a first-seen action, and an instance mismatch. BEP alone does not include that breakdown.

A coding agent can query the project through Hermetiq's MCP server. Connect a build from the getting-started guide, then attach the agent with the MCP guide. Which questions come back answered depends on the data sources that project has connected.