TL;DR: Buildbarn is the leading open-source implementation of the Remote Execution API for Bazel. It gives you self-hosted remote execution with full control over cost, performance, and infrastructure. This guide covers architecture, deployment on Kubernetes, storage configuration, worker setup, and the observability layer you need to actually operate it in production.
What Is Buildbarn?
Buildbarn is an open-source implementation of the Remote Execution API (REAPI) — the protocol that Bazel uses to offload build and test actions to remote workers. Instead of compiling everything on a developer's laptop or a single CI machine, Bazel sends action requests to a remote cluster that executes them in parallel, caches the results, and returns outputs.
Google runs its own proprietary remote execution backend internally. In the open-source world, Buildbarn is the most mature and widely deployed alternative. It implements the full REAPI surface: content-addressable storage (CAS), action cache (AC), remote execution, and the bytestream protocol for large blob transfers.
Why Teams Choose Buildbarn
There are managed remote execution services available — Google's Remote Build Execution (now deprecated), and various commercial offerings. But many engineering organizations choose to self-host with Buildbarn for specific reasons:
Cost control. Remote execution at scale is compute-intensive. When you're running thousands of actions per build across hundreds of daily builds, the bill from a managed service adds up fast. Self-hosting lets you right-size your cluster, use spot instances, and avoid per-action pricing.
Customization. Buildbarn's components are designed to be configured independently. You can tune storage backends, worker pool sizes, scheduling policies, and platform definitions to match your workload. Need GPU workers for ML compilation? Custom toolchain containers? Buildbarn supports it.
Data sovereignty. Your source code, build artifacts, and action metadata stay on infrastructure you control. For organizations with strict compliance requirements, self-hosting is often the only option.
Transparency. When something goes wrong — and it will in a distributed system — you have full access to logs, metrics, and configuration. No filing support tickets and waiting.
Architecture Overview
Buildbarn is not a single binary. It's a set of cooperating services, each handling a specific part of the remote execution workflow. Understanding the architecture is essential before you start deploying.
bb-storage: CAS and Action Cache
bb-storage is the data layer. It implements two core abstractions from the REAPI:
The Content Addressable Store (CAS) holds all blobs — source files, compiled outputs, intermediate artifacts — indexed by their SHA-256 hash. When Bazel uploads an input tree or downloads an output, it talks to the CAS. The Action Cache (AC) maps action digests (the hash of an action's inputs, command, and environment) to their execution results. This is what makes remote caching work: if the AC already has a result for an action, the scheduler can skip execution entirely.
bb-storage supports multiple storage backends. You can use local disk with Buildbarn's built-in block storage, or configure it to use cloud object stores like S3 or GCS for the CAS while keeping the AC in memory or on fast local SSDs.
bb-scheduler: Task Routing
The scheduler is the brain of the system. When Bazel submits an execution request, bb-scheduler receives it, matches the action's platform requirements against available workers, and routes the task. It manages queue depth, handles retries for failed actions, and implements scheduling policies like fair queuing across multiple clients.
The scheduler is also where you define platform queues and size classes. A platform queue groups workers that share the same execution environment (e.g., Linux x86_64 with a specific toolchain). Size classes let you route small actions to lightweight workers and reserve larger workers for memory-intensive compilations.
bb-worker: Action Execution
Workers are where actions actually run. Each bb-worker instance pulls tasks from the scheduler, fetches input files from the CAS, executes the command in a sandboxed environment, and uploads outputs back to the CAS. Workers are stateless — they don't cache anything locally between actions (though some configurations use a local CAS cache for performance).
Workers run actions inside containers or chroot environments, depending on your configuration. In Kubernetes deployments, each worker pod typically runs one or more concurrent actions, with resource limits controlling CPU and memory per action.
bb-browser: Debugging UI
bb-browser is a web UI for inspecting the contents of the CAS and AC. You can look up an action by its digest, view the command that was executed, inspect input and output trees, and read stdout/stderr. It's invaluable for debugging cache misses and failed actions during development.
Deploying Buildbarn on Kubernetes
Kubernetes is the standard deployment target for Buildbarn. The components map naturally to Kubernetes primitives: StatefulSets for storage (to preserve data across restarts), Deployments for stateless schedulers and browsers, and either Deployments or DaemonSets for workers depending on your cluster topology.
Here's a minimal Kubernetes manifest structure for the storage layer:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: storage
namespace: buildbarn
spec:
replicas: 2
selector:
matchLabels:
app: bb-storage
template:
metadata:
labels:
app: bb-storage
spec:
containers:
- name: storage
image: ghcr.io/buildbarn/bb-storage:latest
args:
- /config/bb-storage.jsonnet
ports:
- containerPort: 8980 # gRPC
- containerPort: 8981 # HTTP metrics
resources:
requests:
memory: "4Gi"
cpu: "2"
limits:
memory: "8Gi"
cpu: "4"
volumeMounts:
- name: config
mountPath: /config
- name: cas-data
mountPath: /storage/cas
- name: ac-data
mountPath: /storage/ac
volumes:
- name: config
configMap:
name: bb-storage-config
volumeClaimTemplates:
- metadata:
name: cas-data
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 500Gi
- metadata:
name: ac-data
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 50GiThe scheduler deployment is simpler since it's stateless:
apiVersion: apps/v1
kind: Deployment
metadata:
name: scheduler
namespace: buildbarn
spec:
replicas: 1
selector:
matchLabels:
app: bb-scheduler
template:
metadata:
labels:
app: bb-scheduler
spec:
containers:
- name: scheduler
image: ghcr.io/buildbarn/bb-scheduler:latest
args:
- /config/bb-scheduler.jsonnet
ports:
- containerPort: 8982 # Client gRPC
- containerPort: 8983 # Worker gRPC
- containerPort: 8984 # HTTP metrics
resources:
requests:
memory: "1Gi"
cpu: "1"
limits:
memory: "2Gi"
cpu: "2"Configuration Basics
Buildbarn uses Jsonnet for configuration, which gives you variables, functions, and imports on top of JSON. This is powerful but adds a learning curve. Here's a simplified storage configuration:
local common = import 'common.libsonnet';
{
blobstore: {
contentAddressableStorage: {
'local': {
keyLocationMapOnBlockDevice: {
file: {
path: '/storage/cas/key_location_map',
sizeBytes: 16 * 1024 * 1024,
},
},
keyLocationMapMaximumGetAttempts: 8,
keyLocationMapMaximumPutAttempts: 32,
oldBlocks: 8,
currentBlocks: 24,
newBlocks: 3,
blocksOnBlockDevice: {
source: {
file: {
path: '/storage/cas/blocks',
sizeBytes: 500 * 1024 * 1024 * 1024, // 500 GiB
},
},
spareBlocks: 3,
},
},
},
actionCache: {
completenessChecking: {
backend: {
'local': {
keyLocationMapOnBlockDevice: {
file: {
path: '/storage/ac/key_location_map',
sizeBytes: 1024 * 1024,
},
},
keyLocationMapMaximumGetAttempts: 8,
keyLocationMapMaximumPutAttempts: 32,
oldBlocks: 8,
currentBlocks: 24,
newBlocks: 1,
blocksOnBlockDevice: {
source: {
file: {
path: '/storage/ac/blocks',
sizeBytes: 50 * 1024 * 1024 * 1024, // 50 GiB
},
},
spareBlocks: 3,
},
},
},
},
},
},
grpcServers: [{
listenAddresses: [':8980'],
authenticationPolicy: { allow: {} },
}],
httpServers: [{
listenAddress: ':8981',
}],
}Storage Backend Options
Local block devices are the default and best-performing option. Buildbarn implements its own block-level storage engine that avoids filesystem overhead. This gives you predictable latency and efficient use of SSDs. For most production deployments, local NVMe storage on dedicated nodes is the recommended approach.
Cloud object stores (S3, GCS) work for the CAS if you need virtually unlimited capacity or want to share a cache across multiple clusters. The tradeoff is higher latency per operation. You can mitigate this with a local read cache in front of the cloud backend.
Sharding distributes data across multiple storage instances. Buildbarn supports hash-based sharding where each blob's digest determines which storage shard owns it. This lets you scale horizontally — more shards means more aggregate throughput and capacity.
Worker Platform Configuration
Workers advertise their capabilities to the scheduler via platform properties. Bazel actions specify required platform properties, and the scheduler matches them to compatible workers. A typical worker configuration defines the platform, concurrency level, and execution environment:
{
buildDirectories: [{
native: {
buildDirectoryPath: '/worker/build',
cacheDirectoryPath: '/worker/cache',
maximumMessageSizeBytes: 16 * 1024 * 1024,
},
runners: [{
endpoint: { address: 'unix:///worker/runner' },
concurrency: 8,
platformKeyCount: 0,
}],
mount: { mountPath: '/worker/build' },
}],
schedulers: {
'main': {
endpoint: { address: 'scheduler:8983' },
platform: {
properties: [
{ name: 'OSFamily', value: 'Linux' },
{ name: 'container-image', value: 'docker://your-registry/worker:latest' },
],
},
sizeClasses: [{ value: 0 }],
},
},
}Observability: The Part Everyone Skips
Here's the uncomfortable truth about Buildbarn: deploying it is the easy part. Operating it is where teams struggle. Buildbarn is a distributed system with multiple interacting components, and distributed systems fail in distributed ways. Without proper observability, you're flying blind.
What You Need to Monitor
Scheduler queue depth and wait times. When the queue grows, actions are waiting for workers. If wait times spike, your cluster is undersized for the workload — or a platform queue is misconfigured and workers aren't picking up tasks.
Worker health and resource utilization. CPU saturation, memory pressure, and I/O throughput per worker. Workers that are consistently at their memory limit will eventually get OOM-killed, which means in-flight actions fail and need to be retried.
Storage operation rates and latencies. CAS read/write latencies directly impact action execution time. A storage node under pressure — high eviction rates, growing operation latencies — is a sign that you've undersized your storage or your workload has outgrown the current configuration.
gRPC error rates between components. The scheduler talks to storage. Workers talk to the scheduler and storage. Bazel talks to the scheduler and storage. If any of these gRPC connections are showing elevated error rates or timeouts, something is degraded.
Completed Action Logging
One of Buildbarn's most powerful but underused features is completed action logging. When enabled, the scheduler emits a structured log entry for every action that completes — including the action digest, worker that executed it, queue wait time, execution time, input fetch duration, output upload duration, and exit code.
This data is essential for understanding where time is spent in your remote execution pipeline. Is an action slow because of execution, or because it waited in the queue for 30 seconds? Is input fetching the bottleneck, or output upload? Without completed action logs, you're guessing.
OpenTelemetry Integration
Buildbarn supports exporting traces and metrics via OpenTelemetry. You can send trace data to Jaeger, Tempo, or any OTLP-compatible backend to get distributed traces across the entire execution path — from Bazel's initial request through scheduling, input fetch, execution, and output upload.
Metrics should be scraped by Prometheus (Buildbarn exposes a /metrics endpoint on each component's HTTP port) and visualized in Grafana. At minimum, you want dashboards for queue depth, worker count and utilization, storage latency percentiles, and gRPC error rates.
Common Pitfalls
After helping teams deploy and operate Buildbarn, these are the problems that come up repeatedly:
1. Undersized Storage
The CAS grows fast. A mid-sized monorepo doing full builds can push tens of gigabytes of blobs per day. If your CAS is too small, eviction rates increase, which means more cache misses, which means more actions need to re-execute, which means more blobs get written — a vicious cycle. Size your CAS for at least 2-3 days of build output retention, and monitor eviction rates aggressively.
2. Worker OOM Kills
This is the single most common operational issue. Actions that exceed their worker's memory limit get killed by the kernel OOM killer. The action fails, the scheduler retries it, it fails again — and your build hangs or eventually times out. The fix is twofold: set appropriate memory limits on worker pods, and configure Buildbarn's action size classes so memory-intensive actions get routed to workers with more headroom.
3. Queue Depth Tuning
A healthy scheduler queue is not empty — it has a small buffer of pending actions so workers always have something to pick up. But a queue that grows unboundedly means you don't have enough workers. The tricky part is that queue behavior changes throughout the day: CI peak hours look very different from overnight. Use autoscaling (Kubernetes HPA or KEDA) to scale workers based on queue depth, and set alerts for sustained queue growth.
4. Network Bottlenecks
Remote execution is network-intensive. Every action requires downloading inputs from the CAS and uploading outputs back. If your Kubernetes nodes have limited network bandwidth, you'll hit a throughput ceiling even if CPU and memory are underutilized. Use nodes with high network performance (e.g., AWS instances with enhanced networking), and consider co-locating storage and workers on the same nodes or in the same availability zone.
5. Misconfigured Platform Properties
If the platform properties in your Bazel configuration don't exactly match what workers advertise, actions will sit in the queue indefinitely. There's no automatic partial matching — the strings must match exactly. This is one of the most frustrating debugging experiences because everything looks healthy except actions never get scheduled.
Connecting Bazel to Buildbarn
Once your cluster is running, the Bazel configuration is straightforward. Add these flags to your .bazelrc:
# Remote execution
build:remote --remote_executor=grpc://scheduler.buildbarn:8982
build:remote --remote_cache=grpc://storage.buildbarn:8980
build:remote --remote_instance_name=main
# Platform
build:remote --host_platform=//platforms:remote_linux
build:remote --platforms=//platforms:remote_linux
build:remote --extra_execution_platforms=//platforms:remote_linux
# Performance tuning
build:remote --jobs=200
build:remote --remote_timeout=3600
build:remote --remote_retries=5The --jobs flag controls Bazel's parallelism — how many actions it submits concurrently. Set this high enough to keep your worker pool busy, but not so high that you overwhelm the scheduler. A good starting point is 2-4x your total worker concurrency.
Where to Go from Here
Buildbarn gives you the raw infrastructure for remote execution. But operating it at scale requires more than just deploying pods. You need visibility into how builds interact with the cluster, where time is being spent, and whether your investment in infrastructure is paying off.
That's the gap between having a Buildbarn cluster and having a build infrastructure you can actually reason about. Completed action logs tell you what happened. Scheduler metrics tell you how healthy the cluster is. But correlating that data across builds, tracking trends over time, and understanding cost-per-build requires a layer on top.
Hermetiq has native Buildbarn integration. We ingest completed action logs, correlate them with Bazel Build Event Protocol data, and give you infrastructure health dashboards, cost tracking per build, and cache analytics — out of the box. See how it works.
Related Articles
- Bazel Remote Execution Guide — A comprehensive overview of how Bazel remote execution works and why it matters at scale.
- Understanding the Bazel Build Event Protocol — The telemetry layer that lets you see inside your builds, from action timing to test results.
- What Is Build Observability? — Metrics, logs, and traces for understanding what happens inside your Bazel builds.