← Back to Blog

February 17, 2026

How to Debug Bazel Cache Misses: A Practical Guide

By Hermetiq Engineering

TL;DR: Cache misses are the single biggest source of wasted time and compute in Bazel builds. This guide walks through the most common causes — input changes, command changes, environment drift, platform mismatches, non-deterministic actions, and cache evictions — and shows you how to systematically identify and fix them using execution logs, action key diffs, and purpose-built tooling.

You set up Bazel with remote caching. The first few builds felt magical — targets that used to take minutes resolved in seconds. Your team celebrated. Then, slowly, things started to slip. Build times crept up. Someone noticed the cache hit rate on the dashboard (if you even have one) dropped from 85% to 60%. Engineers started complaining that "Bazel is slow again."

The problem isn't Bazel. The problem is cache misses — and unless you know how to debug them, they'll silently erode every performance gain your build system was supposed to deliver.

This guide is for engineers who already use Bazel and need to understand why their cache isn't working the way it should. We'll cover what cache misses actually are at the action level, walk through the six most common causes, show you the concrete commands and tools for diagnosing them, and describe a systematic workflow you can follow every time cache performance degrades.

What Is a Cache Miss in Bazel?

Bazel's caching model operates at the action level. Every discrete unit of work — compiling a source file, linking a binary, running a test — is an action. Each action has a unique key derived from its inputs: the command line, the input file digests, the environment variables declared in the action, and the execution platform properties. When Bazel needs to execute an action, it computes this key and checks whether a result already exists in the cache (local disk cache or a remote cache). If it finds a matching entry, that's a cache hit — Bazel skips execution and reuses the cached output. If it doesn't find one, that's a cache miss, and the action must execute from scratch.

The remote caching documentation describes the mechanism in detail, but the key insight is this: any change to any component of the action key produces a completely different hash. There is no partial matching. A single bit flip in any input means a full re-execution of that action and every action that transitively depends on it.

This is by design. Bazel's correctness model depends on it. But it also means that small, unintentional changes can cascade into massive cache invalidation events — and figuring out which change caused the invalidation is where the debugging begins.

The Six Common Causes of Cache Misses

1. Input file changes

The most obvious cause: a source file changed, so every action that depends on it (directly or transitively) gets a new cache key. This is correct behavior — you want cache invalidation when inputs change. The problem arises when inputs change that shouldn't be changing. Common culprits include generated files with embedded timestamps, .bazelrc files that differ between machines, or workspace status files that inject volatile data into the build graph.

A particularly insidious variant: files that are semantically identical but differ at the byte level. Two machines generate the same protobuf output, but one has a different protoc version that produces a slightly different binary encoding. The content is equivalent, but the digest is not, and the cache misses.

2. Command line changes

The full command line used to execute an action — including compiler flags, linker options, and tool paths — is part of the action key. If your .bazelrc adds --copt=-O2 on your laptop but CI uses --copt=-O3, every C++ compilation action will miss the cache. This is one of the most common causes of poor cache sharing between local development and CI, and it's often invisible because the flags are buried in layered .bazelrc imports or Starlark transitions.

Watch out for flags that seem innocuous but change the action graph. --compilation_mode (-c opt vs -c dbg) is an obvious one. Less obvious: --host_copt flags that affect tools built for the exec platform, or --features flags that enable different sets of compiler options through the toolchain.

3. Environment variable drift

Bazel allows actions to declare which environment variables they depend on via use_default_shell_env or explicit env attributes. If an action inherits environment variables, any change to those variables changes the action key. The classic scenario: PATH differs between two machines. One developer has /usr/local/bin before /usr/bin; another has it reversed. If the action's environment includes PATH, the cache keys diverge.

Bazel's --incompatible_strict_action_env flag was designed to address this by restricting the environment variables available to actions, and it's been the default since Bazel 0.26. But custom rules that explicitly pass through environment variables can still introduce drift. If you're writing Starlark rules, audit every ctx.actions.run() call for unnecessary env entries.

4. Platform mismatches

Bazel's platform system encodes the execution and target platform as part of the build configuration. If two machines declare different platform properties — different OS version, different CPU architecture, different custom constraints — their actions will have different keys even if the actual compilation would produce identical output. This is especially common in heterogeneous CI environments where some runners are on x86 and others on ARM, or where Linux distributions differ between machines.

Remote execution makes this worse. The platform properties you declare in your BUILD file's platform() definition must exactly match what the remote execution cluster expects. A mismatch doesn't just cause cache misses — it can cause actions to be scheduled on the wrong worker type entirely.

5. Non-deterministic actions

If an action produces different output for the same inputs, it poisons the cache for every downstream consumer. Non-determinism is the most dangerous category because it doesn't just cause one cache miss — it causes an unpredictable cascade of misses across the action graph. Common sources include: timestamps embedded in build outputs (Java .jar files with timestamps in the ZIP metadata, C++ __DATE__ and __TIME__ macros), random UUIDs or seeds in code generators, dictionary iteration order in Python-based build tools, and parallel file system operations that produce non-deterministic directory listings.

Bazel's dynamic execution and --experimental_remote_cache_check flag can help detect non-determinism by re-executing actions and comparing outputs, but the root fix must happen at the toolchain level.

6. Cache evictions

Sometimes the action was cached and the cache entry has simply expired or been evicted. Remote caches have finite storage, and most implementations use LRU (least recently used) or TTL (time-to-live) eviction policies. If your cache is undersized for your workload, or if a large build flooded the cache with entries that pushed out frequently-needed artifacts, you'll see cache misses that have nothing to do with build configuration.

This is a capacity planning problem, not a build configuration problem. If your cache hit rate drops after a large refactor or a new project onboards to the shared cache, check your cache size and eviction metrics before debugging action keys.

How to Identify Cache Misses

Using the execution log

The most powerful built-in tool for diagnosing cache misses is Bazel's execution log. This log records every action that Bazel executed or looked up in the cache during a build, including the full set of inputs, the command line, environment variables, and the platform. To capture it, run your build with the --execution_log_json_file flag:

bazel build //my/target:app \
  --execution_log_json_file=/tmp/exec_log.json

This produces a JSON file where each entry represents a single action. The key fields to examine are:

  • commandArgs — the full command line for the action
  • inputFiles — every file input, with its digest
  • environmentVariables — environment variables passed to the action
  • platform — the execution platform properties
  • remoteCacheHit — whether this action was served from cache

Starting with Bazel 7.x, you can also use the more efficient compact execution log format:

bazel build //my/target:app \
  --experimental_execution_log_compact_file=/tmp/exec_log.compact

The compact format is significantly smaller and faster to produce, but requires the bazel dump --execution_log command to convert it to a human-readable format.

Comparing action keys across builds

The real diagnostic power comes from comparing execution logs between two builds: one that you expect to be a cache hit and one that isn't. Capture logs from both builds and diff the action entries for the same target. Here's a workflow:

# Build 1: Capture baseline (this should be fully cached)
bazel build //my/target:app \
  --execution_log_json_file=/tmp/exec_log_baseline.json

# Build 2: Capture the problematic build
bazel build //my/target:app \
  --execution_log_json_file=/tmp/exec_log_current.json

# Compare the two logs for a specific action
diff <(jq 'select(.mnemonic == "CppCompile") | select(.targetLabel == "//my/target:lib")' \
  /tmp/exec_log_baseline.json) \
  <(jq 'select(.mnemonic == "CppCompile") | select(.targetLabel == "//my/target:lib")' \
  /tmp/exec_log_current.json)

The diff will show you exactly what changed between the two builds for that action. Maybe an input file digest changed. Maybe a compiler flag was added. Maybe an environment variable differs. Whatever it is, it'll be right there in the diff.

Using --explain and --verbose_explanations

For a quicker (but less detailed) diagnostic, Bazel's --explain flag tells you why Bazel decided to re-execute an action rather than use a cached result:

bazel build //my/target:app \
  --explain=/tmp/explain.log \
  --verbose_explanations

This produces output like:

Compiling my/target/lib.cc: one of the inputs changed
  changed input: my/target/config.h (content changed)

The --explain flag is useful for quick triage. It tells you what changed but not always why — for that, you need the full execution log comparison.

Checking aquery for action graph analysis

Bazel's aquery (action query) command lets you inspect the action graph without actually running the build. This is useful for understanding what actions would be executed and what their inputs look like:

# Show all actions for a target
bazel aquery //my/target:app

# Filter for a specific mnemonic
bazel aquery 'mnemonic("CppCompile", //my/target:app)'

# Compare action graphs between two configurations
bazel aquery //my/target:app --output=proto > /tmp/aquery_opt.pb
bazel aquery //my/target:app -c dbg --output=proto > /tmp/aquery_dbg.pb

If you suspect a configuration change is causing cache divergence, aquery lets you compare the action graphs side by side without waiting for a full build.

A Systematic Debugging Workflow

When cache hit rates degrade, resist the temptation to jump straight to execution logs. Follow a top-down approach:

Step 1: Quantify the problem. Before debugging individual actions, understand the scope. What is your overall cache hit rate? When did it start declining? Is the drop across all targets or localized to specific packages or mnemonics? If you have historical build telemetry (via build observability tooling), check the trend. A sudden cliff suggests a single breaking change. A gradual decline suggests environmental drift.

Step 2: Isolate the variable. Determine whether the misses are happening between:

  • Local builds on the same machine — if consecutive builds of the same target miss cache, you likely have non-determinism or volatile inputs.
  • Different developer machines — environment drift, different .bazelrc settings, or toolchain version differences.
  • Local and CI — the most common scenario. Usually caused by flag differences, platform mismatches, or different toolchain installations.
  • Different CI jobs — different runner configurations, stale Docker images, or job-specific .bazelrc imports.

Step 3: Check the obvious things first. Before diving into execution logs, verify:

  • Are both builds pointing at the same remote cache endpoint? (Check --remote_cache flag.)
  • Are both builds using the same --remote_instance_name? Different instance names mean completely separate cache namespaces.
  • Are both builds using the same Bazel version? Different Bazel versions can produce different action keys for the same build graph.
  • Is the cache healthy? Check that the cache service is running, has sufficient storage, and isn't rejecting writes.

Step 4: Capture and compare execution logs. Run the same build in both environments with --execution_log_json_file and diff the results for the actions that are missing cache. The diff will tell you exactly which component of the action key differs.

Step 5: Fix and verify. Once you've identified the divergence, fix it — align the flags, pin the environment variable, update the platform definition — and run a validation build. Confirm the cache hit rate recovers before calling it done.

Step 6: Prevent regressions. The most important step is the one most teams skip. Add a CI check that monitors cache hit rates. Set up alerts when they drop below a threshold. The next time someone adds a flag to .bazelrc that breaks cache sharing, you'll catch it in hours instead of weeks.

Why This Is Hard at Scale

The workflow above works well when you're debugging a single target or a small set of actions. It breaks down when you're managing a build system that serves hundreds of engineers building thousands of targets across multiple platforms.

At scale, the challenges multiply. Execution logs for a full build can be gigabytes in size. Comparing them manually is impractical. The root cause of a cache miss might be a transitive dependency seven levels deep in the graph. Environment drift happens continuously as developers update their local toolchains, and you can't diff execution logs from every machine against a golden baseline.

More fundamentally, the built-in tools tell you what changed but not why it matters. A diff showing that a single header file changed doesn't tell you whether that change was intentional (a developer modified the file) or accidental (a code generator produced non-deterministic output). A flag difference doesn't tell you which .bazelrc layer introduced it or who changed it. You need context that lives outside the execution log.

How Hermetiq Automates Cache Miss Debugging

This is the problem we built Hermetiq to solve. Instead of manually capturing execution logs, diffing JSON files, and correlating changes across builds, Hermetiq ingests every build event via the Build Event Protocol and automatically performs cache miss analysis across your entire build history.

When your cache hit rate drops, Hermetiq shows you exactly which targets are affected, categorizes the miss reason — input changed, command changed, environment changed, platform changed, cache evicted, or never cached — and correlates it with the specific change that caused it. No manual log capture. No JSON diffing. No guesswork.

For teams running hermetic builds at scale, this changes cache miss debugging from a multi-hour investigation into a 30-second lookup. And because Hermetiq tracks cache behavior over time, you get trend data that lets you catch regressions early — before they compound into the kind of slow, frustrating build experience that makes engineers dread hitting "build."

If you've connected Hermetiq's MCP server, you can even ask your AI assistant directly: "Why is my cache hit rate down this week?" The assistant queries the build telemetry, identifies the miss pattern, and gives you the answer and the fix — without opening a single log file.

Key Takeaways

Cache misses are not a mystery. They're the deterministic result of a change to some component of the action key. The debugging process is systematic: quantify the problem, isolate the variable, compare execution logs, identify the divergence, fix it, and prevent regressions.

The hard part isn't the debugging technique — it's doing it consistently at scale, across hundreds of engineers and thousands of daily builds, without burning out your platform team. That's where tooling matters. Whether you build your own monitoring around execution logs or use a platform like Hermetiq, the important thing is to have a system for catching and diagnosing cache misses before they erode your build performance.

Your remote cache is only as good as your ability to keep it working. Treat cache hit rate as a production metric. Monitor it, alert on it, and debug it with the same rigor you'd bring to a latency regression in your API. Your builds — and your engineers — will thank you.

Stop debugging cache misses manually. Hermetiq automatically categorizes every cache miss, identifies the root cause, and tracks trends over time — so your team can fix cache regressions in minutes, not hours.

Related Articles