← Back to Blog

February 16, 2026

Bazel Build Cost Visibility: Track What Your Builds Actually Cost

By Hermetiq Engineering

TL;DR: Engineering teams spend $50K to $500K+ per year on build infrastructure — remote execution clusters, CI machines, cloud compute — but almost none of them can tell you what a single build costs. Without per-build and per-target cost attribution, you are optimizing blind. This post explains what build costs actually consist of, why they are hard to measure, and how to set up real cost tracking using OpenCost, Buildbarn's completed action log, and the Bazel Build Event Protocol.

The Hidden Cost of Builds

Every engineering organization that uses Bazel at scale has a build infrastructure bill. It shows up as a line item on the monthly cloud invoice — a Kubernetes cluster running remote execution workers, persistent volumes for the content-addressable store, load balancers for gRPC traffic, network egress between availability zones. The number might be $5,000 per month or $50,000 per month, depending on team size and build volume.

But here is the problem: that number is a black box. You know the total. You do not know what is inside it. You cannot answer basic questions like: Which target is the most expensive to build? How much does a full CI build cost? What is the per-developer build cost per day? If a team adds 100 new targets to the monorepo, what will that do to the infrastructure bill next quarter?

Most teams do not even try to answer these questions. The build cluster is treated as a shared utility — like electricity — and the bill is absorbed into "infrastructure costs." This works until the bill doubles in six months and nobody can explain why. Or until leadership asks for a cost breakdown by team, and engineering has nothing to show.

The gap is not a lack of caring. It is a lack of tooling. Cloud cost management tools like AWS Cost Explorer or GCP Billing work at the resource level — they can tell you what an EC2 instance costs per hour, but not what a CppCompile action costs per invocation. Build systems like Bazel have no concept of cost at all. The cost information and the build information live in entirely separate systems, and nobody has connected them.

Why Build Costs Are Hard to Track

Build cost attribution is a genuinely difficult problem, not just a missing feature. Several properties of modern build infrastructure make it resistant to simple cost allocation.

Builds are ephemeral. A Bazel remote execution worker picks up an action, runs it for anywhere from 100 milliseconds to 10 minutes, and then picks up the next one. There is no persistent mapping between a build and a machine. A single build might execute actions on 50 different workers, and each worker might serve 20 different builds in the same hour. Traditional server-based cost attribution does not apply.

Infrastructure is shared. Multiple teams, multiple CI pipelines, and multiple developers share the same remote execution cluster simultaneously. The scheduler interleaves actions from different builds on the same workers. Allocating the cost of a worker pod to a specific build requires knowing exactly which actions ran on it and for how long — second-by-second.

Cost is multi-dimensional. An action's cost is not just CPU time. It includes memory allocation, disk I/O for reading inputs and writing outputs, network bandwidth for transferring blobs to and from the content-addressable store, and queue wait time. Some actions are CPU-bound (compilation), some are memory-bound (linking), and some are I/O-bound (large test suites with big data dependencies). A single cost metric does not capture the full picture.

Kubernetes adds complexity. In a Kubernetes-based remote execution cluster, the actual cost depends on node types, spot vs. on-demand instances, resource requests vs. limits, pod scheduling efficiency, and autoscaler behavior. A worker pod that requests 4 CPU cores but only uses 1.5 on average is wasting 62% of its allocated compute. But from the cloud provider's billing perspective, you are paying for all 4 cores.

What You Should Be Measuring

Effective build cost tracking requires attribution at multiple levels of granularity. Each level answers different questions.

Per-Action Cost

The most granular unit. Every action Bazel executes remotely consumes CPU-seconds, memory-seconds, I/O bandwidth, and network transfer. If you know the cost-per-CPU-second for your worker fleet (derived from node instance cost divided by available CPU cores divided by seconds per hour), you can compute a cost for each action. Here is a simplified calculation:

# Per-action cost estimation
#
# Worker node: n2-standard-8 (8 vCPUs, 32GB RAM)
# On-demand hourly cost: $0.3880/hr
# Cost per CPU-second: $0.3880 / 8 / 3600 = $0.0000135
#
# Example: CppCompile action
#   Wall time: 12.4 seconds
#   CPU usage: 1.0 core (single-threaded compilation)
#   CPU-seconds consumed: 12.4
#   Estimated cost: 12.4 * $0.0000135 = $0.000167
#
# Example: CppLink action
#   Wall time: 45.2 seconds
#   CPU usage: 2.0 cores (parallel linking)
#   CPU-seconds consumed: 90.4
#   Estimated cost: 90.4 * $0.0000135 = $0.00122
#
# Example: TestRunner action
#   Wall time: 180.0 seconds
#   CPU usage: 4.0 cores
#   CPU-seconds consumed: 720.0
#   Estimated cost: 720.0 * $0.0000135 = $0.00972

Those numbers look small individually. They are not small at scale. A monorepo build that executes 15,000 actions can cost $5-15 per build. Run 200 builds per day across CI and developer builds, and you are looking at $1,000-3,000 per day in compute alone.

Per-Target Cost

A Bazel target (like //services/payment:server) consists of multiple actions — compilation, linking, possibly code generation and resource processing. The per-target cost is the sum of all action costs attributed to that target. This is the level where you find optimization opportunities: a single expensive target that is rebuilt frequently can dominate your total build cost.

Per-Build Cost

The total cost of a single Bazel invocation — all actions, all targets. This is what you need for capacity planning and for answering the question "how much does a CI build cost?" Tracking per-build cost over time reveals trends: are builds getting more expensive? Did a recent change increase cost? Is one branch significantly more expensive to build than another?

Queue Wait Cost

Time that actions spend sitting in the scheduler queue is not free. It is not consuming worker CPU, but it is consuming developer time. If a build takes 10 minutes but 3 minutes of that is queue wait, the developer is blocked for 3 minutes of unproductive waiting. At an engineering cost of $100/hour, that is $5 per build in developer time wasted on queue delays. Multiply by build frequency and headcount.

Cache Miss Cost

Every cache miss is an action that ran on a remote worker instead of returning a cached result in milliseconds. The cost of a cache miss is the full execution cost of that action — CPU, memory, I/O, and wall time. If your cache hit rate drops from 85% to 70%, you are not just losing some performance. You are paying for 15% more actions to run on workers. That is real money.

The OpenCost Approach

OpenCost is a CNCF project that provides Kubernetes-native cost allocation. It integrates with cloud provider billing APIs and Prometheus to map real infrastructure costs to Kubernetes workloads — namespaces, deployments, pods, and containers.

For build infrastructure, OpenCost gives you a starting point. It can tell you: this namespace running Buildbarn costs $X per day. The worker deployment costs $Y. The storage StatefulSet costs $Z. It breaks down costs into CPU, RAM, persistent volume, and network categories, and it accounts for the difference between requested resources and actual utilization — the efficiency gap.

The deployment is straightforward. OpenCost runs as a pod in your cluster, scrapes Kubernetes resource metrics, and exposes a cost allocation API. You can query it for historical cost data segmented by any Kubernetes label or annotation. If your Buildbarn worker pods carry labels that identify the platform queue or size class, OpenCost can break costs down by those dimensions.

The limitation: OpenCost's granularity stops at the pod level. It can tell you what a worker pod cost over the last hour, but it cannot tell you what each individual action running on that pod cost. For that, you need to go deeper.

Going Deeper: Action-Level Cost Attribution

True per-action cost attribution requires combining two data sources: pod-level cost data from OpenCost (or direct cloud billing), and action-level resource consumption from the build system itself.

Buildbarn's completed action log is the key. When enabled, the scheduler emits a structured record for every completed action that includes the worker that executed it, the execution start and end time, the queue wait duration, the input fetch duration, the output upload duration, and the total wall time. Combined with the action's resource requirements from the Remote Execution API platform properties, you have enough data to compute per-action cost.

The approach works like this: take the hourly cost of a worker pod (from OpenCost or cloud billing). Divide by the number of CPU-seconds that pod can deliver per hour (its CPU allocation times 3600). That gives you a cost-per-CPU-second. For each completed action, multiply the action's CPU-seconds consumed (wall time times CPU cores used) by the cost-per-CPU-second. Sum those costs per target to get per-target cost. Sum per target to get per-build cost.

This model is an approximation — it does not account for memory-bound actions that use more RAM than CPU, or I/O-bound actions that saturate disk bandwidth. But for most build workloads where the majority of actions are CPU-bound compilations, it is accurate enough to identify the top cost drivers and track trends over time.

The Cost of Cache Misses

Cache misses are not just a performance problem. They are a cost problem with a dollar amount attached.

Consider a concrete scenario. Your monorepo build executes 20,000 actions on a typical CI run. With an 80% cache hit rate, 4,000 actions execute remotely on workers. With a 90% cache hit rate, only 2,000 execute remotely. The difference — 2,000 fewer remote executions — translates directly to saved compute.

If the average action costs $0.001 in compute (a reasonable estimate for a mid-size compilation), 2,000 fewer executions saves $2 per build. Run 300 builds per day and that is $600 per day, or roughly $18,000 per month. A 10 percentage point improvement in cache hit rate just saved $216,000 per year. And that is before accounting for the developer time saved from faster builds.

This is why cache hit rate is the single highest-leverage metric for build cost optimization. Every percentage point improvement compounds: fewer remote executions means less worker compute, less network transfer, less storage I/O, and shorter build times. If you are not tracking cache miss cost, you are missing the biggest optimization lever you have.

The first step is understanding why cache misses happen — input changes, command changes, environment drift, platform mismatches, or cache eviction. Each cause has a different fix. Hermetiq's cache miss analysis breaks down miss reasons so you can prioritize the highest-value fixes.

Practical Steps to Reduce Build Costs

Visibility is the first step. Once you can see where money goes, the optimization path becomes clear. Here are the highest-impact levers, roughly in order of effort.

Improve cache hit rates. This is the single best return on effort. Audit your most-missed targets, fix nondeterministic inputs (timestamps, absolute paths, volatile environment variables), and ensure your .bazelrc does not inadvertently vary platform properties across builds. A few hours of investigation can save thousands per month.

Right-size your workers. If workers request 4 CPU cores but actions typically use 1-2, you are overprovisioning. Use completed action data to understand actual resource consumption per action, then adjust worker pod resource requests accordingly. Smaller workers mean more workers per node, which means better bin-packing and less waste.

Use size classes. Not all actions need the same resources. A simple Genrule that copies a file needs far less than a CppLink action that links a large binary. Buildbarn's size class feature lets you route lightweight actions to small workers and reserve large workers for actions that actually need them. This improves utilization and reduces cost.

Tune your .bazelrc for cost efficiency. Several Bazel flags directly impact remote execution cost:

# Limit concurrent remote actions to avoid over-scheduling
build:remote --jobs=150

# Use remote caching without remote execution for targets
# that build faster locally
build:remote --strategy=Genrule=local
build:remote --strategy=CppCompile=remote
build:remote --strategy=CppLink=remote

# Set timeouts to avoid runaway actions consuming resources
build:remote --remote_timeout=600

# Enable BES (Build Event Service) for cost tracking
build:remote --bes_backend=grpc://hermetiq.example.com:443
build:remote --bes_results_url=https://app.hermetiq.com/invocation/

Identify and optimize expensive targets. In most monorepos, cost follows a power law: 10% of targets account for 50%+ of total build cost. Identify those targets through per-target cost attribution, and then optimize them — break up large compilation units, reduce dependency fan-out, improve action cacheability, or restructure the build graph to enable more parallelism.

Avoid over-provisioning your cluster. Many teams set up autoscaling but configure the minimum pool size too high "just in case." Monitor your actual queue depth and worker utilization over a full week. If utilization is consistently below 50% outside of peak hours, your minimum pool is too large. Scale down the floor and let the autoscaler handle bursts.

How Hermetiq Tracks Build Costs

Hermetiq integrates cost tracking directly into the build observability workflow. Instead of building a custom pipeline to connect cloud billing data, Kubernetes metrics, and build events, you get cost attribution out of the box.

OpenCost integration. Hermetiq connects to your cluster's OpenCost deployment to pull namespace-level and pod-level cost data for the Buildbarn namespace. This gives you the total infrastructure cost broken down by component — storage, scheduler, workers — with CPU, RAM, persistent volume, and network cost categories.

Per-build cost breakdown. By correlating Buildbarn completed action logs with OpenCost data and Bazel Build Event Protocol events, Hermetiq computes the cost of every build invocation. You can see which builds are expensive, compare cost across branches, and track whether cost is increasing over time.

Cost trends. The trends dashboard shows cost metrics over 7, 15, and 30 day windows — total compute spend, average cost per build, cost per developer, and the most expensive targets. This is the data you need for capacity planning and for justifying infrastructure investment to leadership.

Expensive target identification. Hermetiq's remote execution analytics surface the targets and mnemonics (action types) that consume the most resources. You can see which targets have the highest total CPU-seconds, the most expensive individual actions, and the biggest I/O footprint — all correlated to dollar amounts.

Cache miss cost attribution. Beyond just counting cache misses, Hermetiq computes the cost of each miss — what would have been saved if that action had been a cache hit instead. Aggregated across an invocation or a time window, this shows you the dollar value of improving your cache hit rate. It turns an abstract metric into a concrete business case.

Stop guessing what your builds cost. Hermetiq gives you per-build cost attribution, cache miss cost analysis, and infrastructure spend trends — connected to every Bazel invocation. See how it works.

Related Articles