← Back to Blog

TL;DR: Slow Bazel builds are rarely caused by one thing. They compound: cold caches, deep dependency graphs, non-hermetic actions busting cache hits, sequential bottlenecks on the critical path. This guide covers ten practical techniques — from enabling remote caching to profiling the critical path to continuous build observability — that experienced teams use to cut build times from minutes to seconds.

Bazel is fast by design. Incremental builds, content-addressable caching, and fine-grained parallelism give it a structural advantage over most build systems. But "fast by design" doesn't mean fast by default. Real-world Bazel builds accumulate performance problems the same way real-world codebases accumulate tech debt: gradually, then suddenly.

One day your CI takes 3 minutes. Six months later it takes 18 minutes and nobody can explain why. The dependency graph got deeper, someone introduced a non-hermetic genrule, the remote cache started evicting before builds could read, and the critical path grew a 90-second bottleneck that serializes everything downstream.

These ten techniques are the ones that actually move the needle. They're ordered roughly by impact — start at the top and work down.

1. Enable Remote Caching

If you're running Bazel without remote caching, every CI run rebuilds from scratch. That's the single biggest source of wasted compute in most Bazel setups. Remote caching lets one machine's build results become every machine's build results. A developer builds a target locally, the artifacts get written to a shared cache, and the next CI run or teammate who builds the same target with the same inputs gets an instant cache hit instead of re-executing the action.

The setup is straightforward:

# .bazelrc
build --remote_cache=grpcs://your-cache.example.com
build --remote_upload_local_results=true

The typical impact is dramatic. Teams going from no remote cache to a properly configured one see 60-90% reductions in CI build time on incremental builds. The key word is "properly configured" — if your cache hit rate is low, skip ahead to technique #4.

2. Use Remote Execution

Remote caching avoids re-executing actions that have already been computed. Remote execution goes further: it distributes the actions that do need to run across a fleet of workers. Instead of building on a single machine — even a beefy one — you build on dozens or hundreds simultaneously.

# .bazelrc
build --remote_executor=grpcs://your-cluster.example.com
build --jobs=200

Remote execution is especially powerful for large monorepos where the action graph has high parallelism. If your build has 500 actions that can run concurrently but your CI machine has 8 cores, you're leaving massive parallelism on the table. A remote execution cluster with 200 workers can execute those 500 actions in a fraction of the wall time.

The catch: remote execution requires hermetic builds. Every action must produce the same output given the same inputs, regardless of which machine executes it. If your builds depend on host-specific state — local toolchains, environment variables, absolute paths — remote execution will surface those problems as flaky failures. Fix hermeticity first.

3. Optimize the Critical Path

Bazel parallelizes your build automatically based on the dependency graph. But parallelism only helps for the parts of the graph that can run in parallel. The critical path — the longest chain of sequentially dependent actions — determines your minimum possible build time, no matter how many cores or workers you throw at it.

If one target takes 90 seconds to compile and everything else depends on it, your build cannot be faster than 90 seconds plus the time for everything downstream. Adding more workers won't help. The only fix is to shorten the critical path.

How to find the critical path

Use --profile to generate a JSON trace (see technique #5), then look for the longest sequential chain. Alternatively, Bazel's --experimental_profile_additional_actions flag adds action-level detail to the profile.

How to shorten it

Split large targets. If a single cc_library or java_library has hundreds of source files, it compiles as one action — and that action sits on the critical path blocking everything downstream. Breaking it into smaller libraries with explicit dependencies lets Bazel parallelize more of the compilation.

# Before: one giant library, single compilation action
cc_library(
    name = "core",
    srcs = glob(["core/**/*.cc"]),
    hdrs = glob(["core/**/*.h"]),
)

# After: split into focused libraries
cc_library(
    name = "core_parsing",
    srcs = glob(["core/parsing/**/*.cc"]),
    hdrs = glob(["core/parsing/**/*.h"]),
)

cc_library(
    name = "core_execution",
    srcs = glob(["core/execution/**/*.cc"]),
    hdrs = glob(["core/execution/**/*.h"]),
    deps = [":core_parsing"],
)

This doesn't reduce total compile time. It distributes it across more parallelizable actions — which reduces wall time when you have available workers.

4. Fix Cache Misses (Hermeticity)

A remote cache is only useful if builds actually hit it. The most common reason for unexpectedly low cache hit rates is non-hermetic actions: something about the build depends on state that isn't captured in the action's declared inputs, so the cache key changes even though the code hasn't.

Common sources of cache misses:

  • Timestamps in build outputs — embedding __DATE__ or __TIME__ in C/C++ builds guarantees a unique output every time.
  • Absolute paths — if a toolchain bakes the workspace path into the output, different machines produce different cache keys.
  • Undeclared environment variables — a genrule that reads $HOME or $USER without declaring the dependency will produce different results per user.
  • Non-deterministic tool output — some code generators produce output with random ordering (e.g., hash map iteration order).

Use --execution_log_json_file to dump the full execution log, then compare action keys between two builds that should have been identical. The diff will show exactly which input changed.

bazel build //... --execution_log_json_file=/tmp/exec_log.json

For a detailed walkthrough, see our guide on how to debug Bazel cache misses.

5. Use --profile and Analyze the JSON Trace

Bazel has a built-in profiler that generates a JSON trace file compatible with Chrome's chrome://tracing viewer. This is the most underused performance tool in the Bazel ecosystem.

bazel build //your:target --profile=/tmp/bazel_profile.json

Open the resulting file in chrome://tracing (or Perfetto) and you'll see a timeline of every phase of your build: loading, analysis, action execution. You can see exactly which actions ran sequentially, which were waiting on dependencies, and where time was spent on network I/O vs. actual computation.

Things to look for:

  • Wide gaps between actions — usually means the dependency graph is forcing serialization.
  • Long single actions — candidates for splitting (technique #3).
  • High "remote cache check" time — your cache server may be slow or overloaded.
  • Loading and analysis phases taking more than a few seconds — may indicate expensive repository rules or macros.

Make profiling a regular practice, not a one-off investigation. The profile data tells you exactly where your build time goes — without it, you're optimizing blind.

6. Reduce Dependency Graph Depth

Deep dependency graphs — where target A depends on B depends on C depends on D, ten levels deep — create long critical paths even if each individual target is fast. Bazel can't start building a target until all of its transitive dependencies are complete.

Flattening the graph where possible gives Bazel more opportunity to parallelize. This doesn't mean removing real dependencies — it means restructuring so that targets depend on what they actually use, not on monolithic "util" libraries that transitively pull in the world.

Use bazel query to visualize dependency depth:

# Find the longest dependency chain to your target
bazel query 'deps(//your:target)' --output graph | dot -Tpng > deps.png

# Count the depth of the dependency graph
bazel query 'deps(//your:target)' --output maxrank

If you find a target with a max rank of 30+, there's almost certainly an opportunity to restructure. Look for intermediate targets that could depend directly on leaf libraries instead of going through layers of abstraction.

7. Use Persistent Workers

Some compilers and tools have expensive startup costs. The Java compiler, for instance, needs to bootstrap the JVM and load class data before it can compile a single file. Persistent workers keep these processes alive between actions, amortizing the startup cost across the entire build.

# .bazelrc — enable persistent workers for Java and Kotlin
build --strategy=Javac=worker
build --strategy=KotlinCompile=worker

Persistent workers are particularly effective for Java, Kotlin, Scala, and TypeScript builds — any toolchain where the compiler has significant startup overhead. The improvement varies, but 20-40% reductions in action execution time are common for JVM-based builds.

Not all rules support workers out of the box. Check the documentation for the specific rule set you're using. The --worker_verbose flag helps debug worker lifecycle issues if you run into problems.

8. Tune Garbage Collection (--host_jvm_args)

Bazel itself runs on the JVM, and for large builds the JVM's garbage collector can become a real bottleneck. If you see Bazel pausing during builds — especially during the analysis phase — GC pressure is likely the cause.

# .bazelrc — give Bazel more heap and use G1GC
startup --host_jvm_args=-Xmx12g
startup --host_jvm_args=-XX:+UseG1GC
startup --host_jvm_args=-XX:MaxGCPauseMillis=200

The default heap size is often too small for monorepos with tens of thousands of targets. If bazel info used-heap-size-after-gc is close to the max, you're spending time in GC that should be spent building. Increase -Xmx until you have comfortable headroom.

On CI machines with plenty of RAM, allocating 8-16 GB to the Bazel JVM is reasonable for large builds. Monitor GC behavior using --host_jvm_args=-verbose:gc during profiling runs to confirm whether GC is actually a bottleneck before tuning further.

9. Use build --keep_going for Parallel Failure Collection

By default, Bazel stops at the first build or test failure. In CI, this means you fix one error, push, wait for the build, and discover the next error. Repeat ten times for ten errors.

# .bazelrc — especially useful for CI
build --keep_going
test --keep_going

With --keep_going, Bazel continues building everything that doesn't depend on the failed target. You get all the failures in a single run instead of discovering them one at a time. This doesn't make any individual build faster, but it dramatically reduces the total time to get a green build by eliminating round trips.

Pair this with --build_event_json_file or the Build Event Protocol to collect structured failure data for analysis. Knowing all the failures up front lets developers — and automated systems — batch fixes instead of playing whack-a-mole.

10. Monitor and Measure with Build Observability

Every technique above requires measurement to validate. Did enabling remote caching actually improve build times? By how much? Did splitting that library shorten the critical path or just shuffle the bottleneck somewhere else? Without continuous measurement, you're guessing.

Build observability means collecting structured telemetry from every build — not just the ones you manually profile. Cache hit rates, critical path durations, action execution times, remote execution queue depths, flaky test frequencies — all of it, tracked over time, across branches and developers.

Key metrics to track:

  • P50/P90/P99 build duration — the headline number, but meaningless without the others.
  • Cache hit rate — compare equivalent incremental CI workloads and investigate unexpected action-level misses rather than relying on a universal threshold.
  • Critical path duration — the theoretical floor for your build time.
  • Remote execution queue wait time — high queue times mean your worker fleet is undersized.
  • Action count per build — if this is growing faster than your codebase, your build graph has a structural problem.

The Build Event Protocol (BEP) is the best source for this data. It emits structured events for every phase of the build — from loading to execution to test results. Feed it to a build observability platform and you get dashboards, trend lines, and alerting that make build performance visible to the entire team.

Build performance is not a one-time fix. Codebases evolve, dependency graphs grow, and new sources of non-hermeticity creep in. The teams that keep their builds fast are the ones that measure continuously and catch regressions before they compound. Start with the techniques that match your current bottleneck — cache hit rates if you're not caching effectively, critical path analysis if you are — and build from there.

Stop guessing where your build time goes. Hermetiq gives you cache hit rates, critical path breakdowns, remote execution analytics, and cost tracking for every Bazel build — out of the box. Connect your builds and start measuring in minutes.


Related Articles