Why Your Bazel Cache Hit Rate Is Terrible (And How to Fix It)

TL;DR: A remote cache can underperform for systemic reasons that aggregate hit-rate metrics do not reveal: environment variable leakage, non-deterministic toolchains, volatile status stamps, CI-vs-local divergence, overly broad dependencies, and undersized caches. This post explains each one, shows the concrete fixes, and argues that sustainable cache performance requires action-level observability.

The Uncomfortable Truth About Your Cache Hit Rate

You adopted Bazel for fast, reproducible builds. You set up a remote cache — maybe a cloud bucket behind --remote_cache, maybe a dedicated cache server, maybe a full remote execution cluster. Early results were promising. CI times dropped. Local rebuilds felt snappy. Someone gave a presentation about it.

Then reality set in. Build times started creeping back up. Engineers reported that "clean builds" were happening when they shouldn't be. Someone dug into the numbers and found that only 50% of actions were hitting cache. Half of your remote cache investment was going to waste.

Here is the uncomfortable truth: an aggregate cache hit rate can look acceptable while expensive actions still miss repeatedly. The right baseline depends on workload shape, branch strategy, cache retention, and whether the build is clean or incremental. Compare like-for-like invocations, then investigate unexpected misses at the action level.

This post covers the six most common architectural reasons your cache hit rate is bad — not as a debugging guide for individual misses (we wrote that guide separately), but as a systemic diagnosis of the patterns that keep entire teams stuck at mediocre cache performance.

Reason 1: Environment Variable Leakage

This is the single most common cause of poor cache sharing across machines, and it's one of the easiest to fix — once you know it's happening.

Bazel computes action keys from the full set of inputs to each action, and that includes environment variables. If an action inherits variables from the host environment — PATH, HOME, USER, LANG, LC_ALL, or anything else that differs between machines — the action key will be different on every machine. Two engineers building the exact same target with the exact same source code will get different cache keys because one has PATH=/usr/local/bin:/usr/bin and the other has PATH=/opt/homebrew/bin:/usr/local/bin:/usr/bin.

The result: zero cache sharing between their machines, and zero cache sharing between either machine and CI. Every action executes from scratch everywhere.

The fix is to enable strict action environment, which restricts the environment variables available to actions to a minimal, deterministic set:

# .bazelrc
# Use a fixed, minimal environment for all actions
build --incompatible_strict_action_env

# If specific actions need PATH, set it explicitly to a fixed value
build --action_env=PATH=/usr/local/bin:/usr/bin:/bin

The --incompatible_strict_action_env flag has been the default since Bazel 0.26, but many teams override it — sometimes intentionally, sometimes because a rule or toolchain they depend on requires specific environment variables. If you're not sure whether your builds are using strict action env, check your full effective .bazelrc chain. A single build --action_env=HOME anywhere in the chain will bleed HOME into every action key.

Audit your .bazelrc for any --action_env or --host_action_env flags that reference variables whose values differ across machines. Each one is a potential cache key divergence point:

# BAD: These will differ across machines and kill cache sharing
build --action_env=HOME
build --action_env=TMPDIR
build --action_env=USER

# GOOD: Fixed values that are identical everywhere
build --action_env=CC=/usr/bin/clang
build --action_env=GOCACHE=/tmp/go-cache

Reason 2: Non-Deterministic Toolchains

Even with strict action environment, your cache can still diverge if the tools themselves produce different output on different machines. This is the toolchain hermiticity problem, and it's more pervasive than most teams realize.

Consider a simple C++ compilation. The action key includes the compiler binary's digest. If Machine A has GCC 12.2.0 and Machine B has GCC 12.3.1, every compilation action will have a different key — even if both compilers would produce byte-identical output. The same applies to linkers, Java compilers, Go toolchains, protoc, and every other tool in your build graph.

The fix is to use hermetic toolchains — toolchains that are downloaded and managed by Bazel itself, rather than picked up from the host system. This ensures every machine uses the exact same compiler binary, byte-for-byte:

# WORKSPACE or MODULE.bazel — register a hermetic CC toolchain
# Instead of relying on the system compiler:
#   cc_configure()  <-- picks up whatever is installed locally

# Use a pinned, downloaded toolchain:
http_archive(
    name = "toolchain_gcc_12",
    urls = ["https://your-mirror/gcc-12.2.0-linux-x86_64.tar.gz"],
    sha256 = "abc123...",
)

register_toolchains("@toolchain_gcc_12//:cc_toolchain")

For languages with first-party Bazel support, hermetic toolchains are usually available out of the box. rules_go downloads the Go SDK. rules_java manages JDK versions. rules_rust pins the Rust compiler. The problem is C/C++, where many teams still rely on the system-installed compiler via cc_configure().

Even with hermetic toolchains, watch out for system headers. If your compilation includes system headers from /usr/include, different OS versions will produce different sets of input files to the compile action. The hermetic build model requires that all inputs — tools, headers, libraries — come from Bazel-managed dependencies.

Reason 3: Volatile Status Stamps

Bazel's workspace status mechanism lets you embed build metadata — git commit hashes, build timestamps, CI job IDs — into your build outputs. It's incredibly useful for binary provenance. It's also a cache killer.

When a rule sets stamp = True, the workspace status values become inputs to that action. If your workspace status script includes BUILD_TIMESTAMP or the current git commit SHA, every stamped action gets a new cache key on every single build. A team that stamps their main binaries might not notice the impact on a single target — but if stamping propagates through libraries or test binaries, it can invalidate hundreds of actions per build.

# workspace_status.sh — common but cache-hostile setup
echo "BUILD_TIMESTAMP $(date +%s)"
echo "STABLE_GIT_COMMIT $(git rev-parse HEAD)"
echo "BUILD_USER ${USER}"
echo "BUILD_HOST $(hostname)"

The key distinction in Bazel is between stable and volatile status values. Variables prefixed with STABLE_ are considered stable — if they change, stamped actions are rebuilt. Variables without the prefix are volatile — Bazel tracks them but does not use them to invalidate the action cache. The problem is that many teams put everything (including the git commit) in stable status, not realizing the cache implications.

# BETTER: Only put truly stable values in STABLE_ prefix
# Volatile values won't bust the action cache
echo "BUILD_TIMESTAMP $(date +%s)"           # volatile — safe
echo "BUILD_HOST $(hostname)"                 # volatile — safe
echo "STABLE_GIT_COMMIT $(git rev-parse HEAD)" # stable — will bust cache on every commit!

# BEST: Move git commit to volatile if you don't need
# strict cache invalidation on every commit
echo "GIT_COMMIT $(git rev-parse HEAD)"      # volatile — won't bust cache

If you need the git commit in your binaries but don't want it to invalidate the cache on every commit, move it to a volatile status variable. Stamped targets will still embed the value, but the action cache won't treat each commit as a reason to rebuild everything. Only use STABLE_ prefix for values that genuinely require a full rebuild when they change — like a release version number that changes once per release cycle.

Also audit which targets actually need stamp = True. Many teams stamp all their binaries by default. In most cases, only the final deployment artifacts need stamps — not test binaries, not libraries, not dev builds. Set --stamp=false for development and CI test builds:

# .bazelrc
build --nostamp
# Only stamp release builds
build:release --stamp

Reason 4: CI vs. Local Divergence

This is where the pain gets real. You have a remote cache shared between CI and developer laptops. In theory, CI populates the cache and developers get free cache hits. In practice, developers get almost zero hits because their build configuration doesn't match CI's.

The divergence happens in layers. Maybe CI uses --config=ci which sets optimization flags that developers don't use. Maybe CI runs on Linux and developers are on macOS. Maybe CI uses a Docker image with a specific toolchain version while developers use whatever Homebrew installed. Each layer of difference multiplies the cache miss rate.

The most insidious variant is the --config mismatch. Bazel allows configs to be defined in .bazelrc and composed together. A common pattern is:

# .bazelrc
build:ci --remote_cache=grpcs://cache.example.com
build:ci --google_default_credentials
build:ci --config=linux_x86_64
build:ci --compilation_mode=opt

# Developer config
build:dev --remote_cache=grpcs://cache.example.com
build:dev --config=linux_x86_64
build:dev --compilation_mode=dbg   # <-- different from CI!

That single flag difference — -c opt vs -c dbg — means that every action compiled in CI has a completely different action key from the same action compiled locally. The developer's laptop will never hit CI's cache entries. The remote cache is effectively two separate caches that happen to share the same storage.

The fix requires discipline: define a single, canonical build configuration that both CI and developers use for cacheable builds. Separate concerns that must differ (like authentication credentials) from concerns that should be identical (like compilation mode, platform, and toolchain selection):

# .bazelrc — shared configuration for cache-compatible builds
build --remote_cache=grpcs://cache.example.com
build --config=hermetic_toolchain
build --compilation_mode=fastbuild

# CI-specific settings that DON'T affect action keys
build:ci --google_default_credentials
build:ci --remote_upload_local_results

# Developer settings that DON'T affect action keys
build:dev --credential_helper=%workspace%/tools/credential-helper

Cross-platform cache sharing (macOS developers, Linux CI) is a harder problem. Bazel's platform system intentionally separates action keys by target platform, which means macOS and Linux builds will not share cache entries for platform-specific actions. The architectural solutions are either remote execution (developers submit actions to a Linux build farm) or cross-compilation from macOS targeting Linux. Both require meaningful infrastructure investment.

Reason 5: Unnecessary Target Dependencies

Cache invalidation is transitive. When a file changes, every action that depends on it — directly or transitively — gets a new cache key. This is correct behavior, but it means that overly broad dependency graphs amplify the impact of every change.

The most common version of this problem is the monolithic header file. One team creates a common/config.h that contains every configuration constant for the entire project. Every C++ target depends on it. When anyone changes a single constant, every compilation action in the project misses cache.

Overly broad glob() patterns have a similar effect:

# BAD: Every .h file in the directory is an input to every target
cc_library(
    name = "mylib",
    srcs = ["mylib.cc"],
    hdrs = glob(["*.h"]),  # Includes unrelated headers
)

# GOOD: Explicit header list — only actual dependencies
cc_library(
    name = "mylib",
    srcs = ["mylib.cc"],
    hdrs = ["mylib.h", "mylib_internal.h"],
)

The same principle applies to deps. If a BUILD file lists a library in deps that isn't actually used, any change to that library will unnecessarily invalidate the depending target's cache. At scale, these phantom dependencies accumulate. A target might have 50 entries in its transitive deps closure that it doesn't actually need — and each one is a source of unnecessary cache invalidation.

Tools like unused_deps can help identify unnecessary dependency edges. For header files, restructuring monolithic headers into smaller, focused headers reduces the blast radius of changes. This is build hygiene, not a Bazel-specific concern — but Bazel's content-addressed caching makes the cost of poor dependency hygiene directly measurable.

Reason 6: Cache Eviction Under Pressure

Sometimes your build configuration is perfect and the cache still misses. The action was cached — but the entry was evicted before anyone reused it.

Remote caches have finite storage. Most implementations use LRU (least recently used) eviction: when the cache is full, the oldest-accessed entries are removed to make room for new ones. If your cache is undersized for your workload, useful entries get evicted before they can be reused. The symptom is a cache hit rate that looks reasonable for sequential builds on a single branch but degrades when multiple teams or branches are competing for cache space.

Common scenarios where eviction becomes a problem:

• A large refactoring lands that produces thousands of new cache entries, evicting entries that other teams' builds depend on
• A monorepo with many independent projects sharing a single cache — each project's builds push out entries that other projects need
• Feature branches that diverge significantly from main, filling the cache with branch-specific entries that will never be reused
• Test actions that produce large output artifacts, consuming disproportionate cache space

The difficulty is that most remote cache implementations provide no visibility into eviction rates. You know your cache hit rate dropped, but you don't know whether the cause is configuration divergence or capacity pressure. Without eviction metrics, you're debugging in the dark.

Practical fixes for cache eviction pressure include: increasing cache storage (the cheapest fix if your cache is backed by object storage), setting TTLs that match your reuse patterns (shorter TTLs for test outputs, longer TTLs for compilation outputs), partitioning the cache by project or team to prevent noisy neighbors, and monitoring cache size and eviction rates as infrastructure metrics.

How to Measure Your Actual Cache Hit Rate

Here's a question that reveals more about a team's build maturity than any technical interview: "What is your Bazel cache hit rate?"

Most teams cannot answer this question. They might have a rough sense ("it feels fast" or "CI is slow lately"), but they don't have a number. And without a number, they can't tell whether it's getting better or worse, can't set improvement targets, and can't measure the impact of fixes.

The most reliable way to measure cache hit rates is through the Build Event Protocol (BEP). Bazel can stream detailed build events — including per-action cache hit/miss status — to a backend via --build_event_binary_file or --bes_backend. Every action executed during a build reports whether it was a cache hit, a remote cache hit, or a cache miss, along with the mnemonic (CppCompile, Javac, GoCompile, etc.) and the target label.

# Stream BEP to a file for offline analysis
bazel build //... --build_event_binary_file=/tmp/bep.pb

# Or stream to a BES backend in real time
bazel build //... --bes_backend=grpcs://your-bes-endpoint.example.com

From the BEP data, you can compute cache hit rate at any granularity: per-build, per-target, per-mnemonic, per-developer, per-CI-pipeline. You can track it over time and correlate drops with specific commits or configuration changes. This is the foundation of meaningful cache performance management.

The Fix: Systematic Observability

Each of the six reasons above has a specific technical fix. Environment leakage? Enable strict action env. Toolchain drift? Pin hermetic toolchains. Stamp poisoning? Audit your workspace status script. CI divergence? Unify your build configs. Dependency bloat? Clean up your BUILD files. Cache eviction? Increase capacity and add monitoring.

But here's the pattern: every fix requires first knowing that the problem exists, then identifying which specific variant you have, and then verifying that the fix actually worked. Without systematic observability into your build's cache behavior, you're flying blind. You fix one issue and another quietly takes its place. You improve the hit rate for CppCompile actions but don't notice that GoCompile actions degraded. You fix CI cache sharing but a new team onboards with a different .bazelrc and silently fragments the cache again.

This is the problem that Hermetiq solves. By ingesting BEP data from every build across your organization, Hermetiq provides continuous visibility into cache hit rates — broken down by target, mnemonic, team, branch, and time period. When the hit rate drops, Hermetiq identifies the cause: which actions are missing, what category of miss it is (input changed, environment changed, platform changed, never cached, cache evicted), and what changed between the last good build and the current one.

Instead of the reactive cycle of "notice builds are slow, spend a day debugging execution logs, fix one problem, repeat in a month," teams using Hermetiq get proactive alerts when cache performance degrades and can resolve the root cause in minutes. That's the difference between treating cache hit rate as an afterthought and treating it as the production metric it actually is.

Key Takeaways

• There is no universal healthy hit rate. Compare equivalent workloads over time, then investigate unexpected misses by action, target, platform, and environment.
• The six root causes are systemic, not random: environment variable leakage, non-deterministic toolchains, volatile status stamps, CI-vs-local configuration divergence, overly broad dependencies, and cache eviction pressure.
• Each cause has a concrete fix. Strict action env, hermetic toolchains, volatile status variables, unified build configs, precise dependency declarations, and appropriately sized caches.
• Measurement is the prerequisite. If you don't know your cache hit rate, you can't manage it. Use BEP-based measurement to establish a baseline and track improvements.
• Fixes don't stick without observability. Build configuration drifts. New teams onboard. Dependencies change. Without continuous monitoring, cache performance will degrade again.
• Treat cache hit rate as a production metric. Monitor it, alert on it, and hold yourself to a target — just like latency or uptime for a production service.

Related Articles

• How to Debug Bazel Cache Misses — The tactical companion to this post. Once you know which systemic issue you have, this guide walks through the step-by-step debugging workflow to isolate individual cache misses.
• What Is Hermeticity and Why It Matters — The foundational property that makes Bazel caching reliable. Non-hermetic builds are the root cause behind several of the issues described above.
• How to Speed Up Bazel Builds — Cache hit rate is one lever for build speed. This guide covers the full set of techniques including parallelism, remote execution, and build graph optimization.