Buildbarn production monitoring checklist.
Follow one Bazel action through queueing, input fetch, execution, and output upload, then connect each infrastructure signal to the affected build.
Monitor the build, not only the cluster
Buildbarn tells you where infrastructure time and failures occurred. Bazel evidence tells you which builds, targets, tests, and engineers felt the impact. Keep both sides of that relationship available during every investigation.
- Track build completion, elapsed time, action counts, and failed actions from Bazel Build Event Protocol data.
- Preserve the invocation identifier, action digest, target or mnemonic, platform, and instance-name prefix needed to join build and execution evidence.
- Compare an affected build with a healthy build that represents the same workload before calling a change a regression.
Verify coverage before writing alerts
An alert is useful only when the underlying signals cover the complete request path and contain enough dimensions to isolate the failing pool or service.
- Scrape every scheduler, worker or executor, storage frontend, and gRPC endpoint that serves production traffic.
- Record component version, configuration revision, deployment time, instance prefix, platform, and size class alongside the measurements.
- Treat Buildbarn’s maintained monitoring assets as a starting point: review the dashboards and recording rules against your deployed versions and topology.
- Derive thresholds from workload objectives and historical baselines instead of copying universal queue, latency, or retention numbers.
Scheduler and queue checklist
Segment scheduler behavior before adding capacity. A global average can hide one incompatible platform queue while every other pool remains healthy.
- Watch queued, executing, completed, and removed task flow—not queue depth by itself.
- Measure queue and stage duration by instance-name prefix, platform properties, and size class.
- Track execution retries, completion result, and gRPC status for the same cohorts.
- When queue count and duration rise while queued-to-executing throughput stays flat, confirm worker eligibility before scaling the whole fleet.
Worker and execution checklist
Break remote execution into input fetch, command execution, and output upload. Each phase points to a different first investigation.
- Verify worker registration, synchronization, advertised platform, size class, and degraded or idle state.
- Compare FetchingInputs, Running, and UploadingOutputs duration distributions for the affected workload.
- Review CPU time, maximum resident set size, page faults, swaps, block I/O, and file-pool pressure when the command phase or retries regress.
- Interpret fetch regressions as storage, network, file-population, or input-tree candidates; run regressions as workload or worker contention; upload regressions as output-size, network, or storage-write candidates.
CAS and Action Cache checklist
Monitor storage health and retention without assuming every Bazel cache miss is an eviction. Legitimate action-key changes create misses even when storage is healthy.
- Track Get, Put, and FindMissing operation rate, duration, blob size, batch size, backend, storage type, and gRPC status.
- Monitor worst-case retention by shard and replica, and account for counters that reset when a storage process restarts.
- Watch key-location-map exhaustion signals such as TooManyAttempts and TooManyIterations.
- When cache hits regress, compare retention with equivalent Bazel actions and diff the key-forming execution-log fields before assigning a storage cause.
gRPC and network checklist
Identify the specific failing edge and RPC method before concluding that the Buildbarn cluster is unavailable.
- Track in-flight requests, operation rate, status codes, messages sent and received, and duration by service and method.
- Separate scheduler, execution, ByteStream, CAS, and Action Cache failures instead of grouping every non-OK response together.
- Use Bazel network metrics or its JSON trace profile to distinguish a client-side bandwidth constraint from service latency.
Symptom → likely layer → first check
Use this table to narrow the first investigation. Confirm the cause with action- and build-level evidence before changing capacity or configuration.
| Symptom | Likely layer | First check |
|---|---|---|
| Actions remain queued | Scheduler or platform routing | Queue count and duration by platform and size class; eligible worker registration |
| Queue grows while workers appear idle | Routing | Instance prefix and exact requested versus advertised platform properties |
| Time rises before command execution | Worker, storage, or network | FetchingInputs duration and CAS Get or ByteStream read latency |
| Command execution becomes slower | Worker or workload | Running duration, resource pressure, action mnemonic, target, and comparable build |
| Time rises after command completion | Worker, storage, or network | UploadingOutputs duration, output blob size, and storage write latency |
| Remote-cache hits regress | Storage or action keys | Retention plus a comparable Bazel execution-log diff |
| NOT_FOUND increases | Storage or routing | Exact RPC method, digest, shard or replica visibility, upload completion, and retention |
| DEADLINE_EXCEEDED or UNAVAILABLE increases | gRPC or service | Exact method, service, duration, status, and current service availability |
| Retries increase | Worker or scheduler | Completion result, worker health, resource pressure, platform, and size class |
Move from a slow build to the responsible layer.
- Select one affected build and one comparable healthy build.
- Find the remote-execution phase whose duration or result changed.
- Segment that phase by platform, size class, service, shard, and replica.
- Inspect the matching action metadata, completed-action evidence, and Bazel events.
- Change one variable, then compare the same workload cohort after the change.
Alert on customer impact and an actionable owner.
Every alert should name the component, instance or platform, time window, and a link to affected builds—not only a global infrastructure number.
- A production component or scrape target disappears.
- A sustained non-OK RPC rate affects a specific service and method.
- Queue duration or depth moves outside the workload’s objective.
- Worst-case retention falls below the team’s required reuse window.
- Key-location-map exhaustion or iteration-limit signals appear.
- An execution-phase percentile regresses against its comparable baseline.
Verify details against the deployed versions.
Investigate your own Buildbarn telemetry.
Hermetiq connects Bazel and Buildbarn telemetry to the answers your engineers and AI agents need.
Technical review: August 29, 2026 · Verify flags, metric availability, and behavior against the versions deployed in your environment.