TL;DR: Remote Build Execution (RBE) lets Bazel distribute build and test actions across a cluster of worker machines instead of running everything locally. It's the single biggest lever for reducing build times at scale — but it requires hermetic builds, correct toolchain configuration, and real observability to operate well. This guide covers the full picture: how RBE works, the major providers, the configuration you need, the pitfalls you'll hit, and how to monitor the system once it's running.
If you've been running Bazel for a while, you've probably hit the ceiling on local builds. You've optimized your dependency graph, enabled remote caching, and maybe thrown more cores at the CI machines. But at a certain codebase size, local execution can't keep up. A full build touches tens of thousands of actions. Even with caching, a clean build or a large change set means saturating every core on a single machine for minutes — or hours.
That's the problem Remote Build Execution solves. Instead of running actions on the machine that invoked Bazel, RBE dispatches them to a fleet of remote workers. The parallelism isn't limited by your local hardware anymore. It's limited by the size of your cluster.
What RBE Is and Why It Matters
Remote execution is one of Bazel's most powerful features and one of the least understood. At its core, it's a distributed computing system that leverages Bazel's action graph — the directed acyclic graph of build and test steps — to execute work across many machines simultaneously.
When you build locally, Bazel walks the action graph and executes each action on your machine, respecting dependencies. With RBE enabled, Bazel still constructs the action graph locally, but instead of executing actions itself, it sends them to a remote service that schedules, executes, and caches the results.
The value proposition is straightforward: if your build has 10,000 independent actions and you have 500 workers, you can complete them roughly 500x faster than a single machine — minus the overhead of network transfer and scheduling. In practice, teams moving from local to remote execution typically see 3x-10x wall-clock improvements on large builds, with cache-cold builds benefiting the most.
How RBE Works Under the Hood
The Remote Execution API (REAPI) defines the contract between Bazel and the remote execution service. Understanding the data flow is essential for debugging problems when they arise.
The action graph
Bazel analyzes your BUILD files and constructs an action graph — a DAG where each node is a build action (compile, link, test, etc.) with declared inputs and outputs. This graph is computed locally. RBE doesn't change how Bazel plans the build; it changes where the work runs.
The Content Addressable Store (CAS)
Before a remote worker can execute an action, it needs the inputs — source files, toolchains, intermediate artifacts. These are stored in the CAS, a content-addressed blob store where each file is identified by the hash of its contents. When Bazel sends an action to the remote service, it uploads any inputs that aren't already in the CAS. The worker pulls them down, executes the action, and uploads the outputs back to the CAS.
The scheduler
The remote execution service includes a scheduler that receives action requests from Bazel clients, matches them to available workers based on platform requirements (OS, architecture, toolchain), and queues them for execution. Good schedulers handle priority, fairness across clients, and load balancing across the worker fleet.
The workers
Workers are the machines that actually run the build actions. Each worker pulls an action from the scheduler, downloads its inputs from the CAS, executes the command in a sandboxed environment, and uploads the outputs back. Workers are typically stateless and interchangeable — any worker that matches the platform requirements can execute any action.
The Action Cache (AC)
The Action Cache maps an action's inputs (the hash of its command, environment, input files, and platform) to its outputs. Before executing an action, the system checks the AC. If there's a hit, the outputs are returned directly from the CAS without re-executing. This is the remote cache — and it works identically whether you're using remote execution or just remote caching.
The full flow
Putting it together: Bazel constructs the action graph locally. For each action, it computes a cache key and checks the AC. On a cache hit, it downloads the output from the CAS. On a cache miss, it uploads the inputs to the CAS, sends the action to the scheduler, waits for a worker to execute it, and downloads the result. All of this happens in parallel across the entire action graph, limited only by dependency ordering and cluster capacity.
Prerequisites: Getting Your Builds RBE-Ready
You can't just flip a flag and get remote execution. RBE requires your builds to meet certain properties — and the gap between "works locally" and "works remotely" is where most teams get stuck.
Hermetic builds are non-negotiable
Remote execution amplifies every hermeticity violation in your build. When an action runs locally, it can accidentally depend on host-installed tools, environment variables, or filesystem state — and it works fine because that state is always there. When the same action runs on a remote worker with a different OS, different packages, and a clean filesystem, it fails. Or worse, it produces silently different output.
Before enabling RBE, you need to audit your builds for undeclared dependencies on the host environment. Actions should depend only on their declared inputs — nothing else.
Toolchain configuration
Locally, Bazel often uses whatever compiler is installed on the machine. With RBE, you need to tell Bazel exactly which toolchain to use on the remote workers — and that toolchain needs to be available there. This means configuring Bazel platform and toolchain rules to specify the execution platform, register the correct C++/Java/Python toolchains, and ensure they match what's installed on the workers.
This is the single biggest configuration hurdle for teams adopting RBE. Getting the platform and toolchain definitions right often takes more time than setting up the remote execution cluster itself.
Network considerations
RBE moves data over the network — source files up, artifacts down. If your build produces large intermediate artifacts or depends on large input trees, network bandwidth between the Bazel client and the RBE cluster matters. Latency matters too: every action that misses the cache requires a round trip to upload inputs, wait for execution, and download outputs. Teams running RBE across regions or with limited bandwidth may find that network transfer time dominates execution time for small, fast actions.
The Major RBE Providers
The Remote Execution API is an open standard, and several implementations exist. Each makes different trade-offs around self-hosting, managed infrastructure, and feature scope.
Buildbarn
Buildbarn is a fully open-source REAPI implementation written in Go. It's the most popular self-hosted option, used by companies that want full control over their execution cluster. Buildbarn includes a scheduler, worker infrastructure, CAS, and AC — everything you need to run RBE on your own Kubernetes cluster. The trade-off is operational complexity: you're responsible for provisioning, scaling, monitoring, and maintaining the infrastructure. Setting up Buildbarn is a meaningful infrastructure project, but it gives you complete ownership and no per-action costs.
BuildBuddy
BuildBuddy offers both a managed cloud service and a self-hosted open-source version. The managed service handles the infrastructure for you — you point Bazel at their endpoint, and they handle scheduling, execution, and caching. BuildBuddy also includes a build results UI, timing profiles, and invocation storage. It's a good option for teams that want RBE without the operational overhead of running their own cluster.
EngFlow
EngFlow is a commercial remote execution platform focused on performance and enterprise features. It offers a managed service and private deployments. EngFlow was founded by engineers from Google's Bazel team and emphasizes low-latency execution and deep Bazel integration.
Google Cloud RBE
Google offers Remote Build Execution as a managed service on Google Cloud. It's tightly integrated with the Google Cloud ecosystem and supports the REAPI natively. It's a natural choice for teams already running on GCP, though pricing is based on execution time and can add up at scale.
Key .bazelrc Flags for Remote Execution
Configuring RBE in Bazel comes down to a set of flags in your .bazelrc file. Here's a practical starting configuration:
# .bazelrc — remote execution configuration
# Enable remote execution
build:remote --remote_executor=grpcs://your-rbe-endpoint.example.com
# Enable remote caching (used alongside execution)
build:remote --remote_cache=grpcs://your-rbe-endpoint.example.com
# Set the remote instance name (namespace for your project)
build:remote --remote_instance_name=my-project/default
# Define the execution platform
build:remote --extra_execution_platforms=//platforms:linux_x86_64
build:remote --host_platform=//platforms:linux_x86_64
# Upload local results to the remote cache
build:remote --remote_upload_local_results=true
# Number of concurrent remote actions
build:remote --jobs=200
# Timeout for remote actions (seconds)
build:remote --remote_timeout=3600For authentication, you'll typically add TLS certificates or API tokens depending on your provider:
# mTLS authentication (common with Buildbarn)
build:remote --tls_certificate=/path/to/client.crt
build:remote --tls_client_certificate=/path/to/client.crt
build:remote --tls_client_key=/path/to/client.key
# Or header-based authentication (common with managed services)
build:remote --remote_header=x-api-key=YOUR_API_KEYPlatform configuration is equally critical. You'll need to define a platform that matches your remote workers:
# platforms/BUILD
platform(
name = "linux_x86_64",
constraint_values = [
"@platforms//os:linux",
"@platforms//cpu:x86_64",
],
exec_properties = {
"OSFamily": "Linux",
"container-image": "docker://your-registry/rbe-ubuntu22:latest",
},
)Then invoke Bazel with the remote config:
bazel build --config=remote //...Common Pitfalls
RBE is powerful, but it introduces failure modes that don't exist with local builds. These are the problems teams hit most often.
Non-hermetic actions
This is the number one issue. An action that works locally because it reads /usr/bin/python3 will fail on a remote worker where Python is installed at a different path — or not at all. Actions that embed timestamps, read environment variables, or depend on filesystem ordering will produce different results on different workers, leading to flaky builds and poisoned caches.
The fix is straightforward but labor-intensive: audit every action for undeclared inputs, use Bazel-managed toolchains, and enable strict sandboxing. Cache miss analysis is invaluable here — if an action's cache miss rate is higher than expected, it's likely non-hermetic.
Platform mismatches
The platform defined in your Bazel configuration must match what's actually running on the workers. If your platform specifies Ubuntu 22.04 but the workers are running Ubuntu 20.04, you'll get subtle failures from library version mismatches. If the CPU architecture doesn't match, actions will fail outright. Container image tags that resolve to different images over time (like :latest) break reproducibility.
Pin everything. Use digest-based container image references. Verify that the toolchain versions in your platform definition match the worker environment exactly.
Network latency and transfer overhead
Every cache-miss action requires uploading inputs and downloading outputs over the network. For actions that are fast to execute but have large input trees (like linking steps that need many object files), the network transfer time can exceed the execution time. This is especially painful when the Bazel client is geographically far from the RBE cluster.
Mitigations include colocating CI machines with the RBE cluster, using BES (Build Event Service) to observe where time is spent, and restructuring the build to reduce the size of action input trees where possible.
Scheduler saturation and queue depth
When multiple developers or CI jobs submit builds simultaneously, the scheduler queue can back up. If the worker fleet is undersized for peak load, actions wait in queue longer than they take to execute. This shows up as high queue wait times in your build telemetry — and it silently turns a 5-minute build into a 20-minute build.
Monitor queue depth and wait times. Autoscale the worker fleet based on demand. Set up alerts for when queue wait times exceed acceptable thresholds.
Cache poisoning
If a non-hermetic action produces incorrect output and that output gets cached, every subsequent build that hits that cache entry gets the wrong result. This is cache poisoning, and it's one of the most dangerous failure modes in remote execution. The symptoms are bizarre: builds that fail for no apparent reason, tests that pass locally but fail in CI, artifacts that work on one machine but not another.
Prevention is better than cure: enforce hermeticity strictly, use sandboxing, and monitor cache hit rates for anomalies. If you suspect cache poisoning, invalidate the action cache and rebuild from scratch.
Monitoring and Observability for RBE
Running an RBE cluster without observability is like running a production service without monitoring — you're flying blind. The system has many moving parts, and problems manifest as slow builds long before they surface as errors.
What to monitor
The essential metrics for an RBE deployment fall into a few categories:
Build-level metrics: Wall-clock build time, action count, cache hit rate, remote vs. local execution ratio. These tell you whether RBE is actually helping.
Action-level metrics: Per-action execution time, queue wait time, input fetch time, output upload time. These tell you where time is going within a build. If queue wait times are high, you need more workers. If input fetch times are high, you have a CAS performance or network problem.
Infrastructure metrics: Scheduler queue depth, worker utilization, CAS storage capacity, CAS operation latency, worker error rates. These tell you the health of the RBE cluster itself.
Cost metrics: Total compute hours consumed, cost per build, cost per team or project. RBE clusters aren't free to run — whether you're paying a provider per-action or running your own infrastructure on Kubernetes, understanding the cost is essential for capacity planning and optimization.
The observability gap
Most teams have some of these metrics but not all of them — and critically, they don't have them correlated. Knowing that a build took 20 minutes isn't useful unless you can drill into which actions were slow, whether those actions were waiting in queue or actually executing, and whether the slowness is a one-time event or a trend. Knowing that your cache hit rate is 70% isn't useful unless you can identify which targets are missing and why.
This is where dedicated build observability tooling matters. The Build Event Protocol (BEP) streams rich telemetry for every build — action timings, cache events, test results, remote execution metadata. But BEP produces raw event streams. Turning those into actionable dashboards, alerts, and root-cause analysis requires infrastructure that most teams don't build in-house.
Making RBE Work Long-Term
Getting RBE running is a milestone. Keeping it running well is the real challenge. Build graphs evolve. New languages and toolchains get added. The worker fleet needs scaling and maintenance. Cache hit rates drift. New engineers write non-hermetic rules without realizing it.
The teams that succeed with RBE long-term share a few traits: they treat the RBE cluster as a production system with SLOs, they invest in observability to catch regressions early, they enforce hermeticity through tooling rather than documentation, and they track the cost of their build infrastructure alongside its performance.
Remote execution is the most impactful optimization you can make to a Bazel build system at scale. It's also the most operationally demanding. The difference between a team that loves RBE and a team that fights it comes down to visibility — into what's happening in the cluster, where time and money are going, and what's broken before it breaks the build.
Get full visibility into your RBE cluster. Hermetiq integrates directly with Buildbarn to give you build-level and action-level telemetry, cache miss analysis, queue wait tracking, worker fleet health, and per-build cost breakdowns — so you can run remote execution with confidence.
Related Articles
- What Is Hermeticity and Why It Matters — The foundational build property that makes remote execution reliable.
- How to Debug Bazel Cache Misses — Trace cache misses to their root cause: input changes, command drift, or platform mismatches.
- How to Set Up Buildbarn — A step-by-step guide to deploying the most popular open-source RBE backend.