# Controlled Buildbarn queue-capacity experiment

Recorded September 8, 2026 UTC. This is a real local experiment with a synthetic waiting workload. It is not a customer or production benchmark. Codex used actual Hermetiq MCP infrastructure responses, changed one local worker setting, and verified the unchanged workload with real remote builds.

## Finding and all six runs

One worker execution slot queued most of 16 independent waiting actions. Changing that worker's concurrency to four reduced mean queue wait in each of the three after runs. The median of the three per-run mean waits fell from 7.893954s to 1.809266s. Each mean covers exactly the 16 waiting actions; the dependent verification action is excluded. No universal total-build speedup is claimed.

| Recorded run | Slots | Mean queue wait, s | Mean action execution, s | Bazel elapsed, s | BEP invocation interval, s | Enclosing process, s |
|---|---:|---:|---:|---:|---:|---:|
| baseline-1 | 1 | 7.893954 | 1.014649 | 18.400 | 17.806 | 19.902347 |
| baseline-2 | 1 | 7.885843 | 1.014484 | 18.358 | 17.810 | 20.056586 |
| baseline-3 | 1 | 7.905235 | 1.015866 | 19.089 | 18.253 | 20.647383 |
| fixed-1 | 4 | 1.655201 | 1.038829 | 6.165 | 5.495 | 7.641868 |
| fixed-2 | 4 | 1.809266 | 1.057227 | 6.627 | 6.125 | 7.955241 |
| fixed-3 | 4 | 2.446663 | 1.208559 | 19.435 | 15.182 | 40.506617 |

The final run resumed after a pause of about 2 hours 37 minutes. Host conditions across that pause were not monitored or held constant; the cause of the slower total time is not established. The final run's queue wait improved, but its total Bazel elapsed time was slower than every baseline. It is retained in the summary, medians, complete evidence and validation script.

Bazel elapsed, BEP invocation duration and enclosing process wall time have different start/end boundaries. The process measurement includes starting the batch Bazel process and waiting for it to exit. Do not interchange those clocks. `record.json` stores each one, start/end timestamps, critical paths and invocation UUIDs.

## Workload and controls

- Bazel 9.1.0 on macOS ARM64, sending actions to a Linux ARM64 Buildbarn worker.
- `sample/BUILD.bazel` defines 16 independent genrules: `sleep 1` followed by writing a numbered line. A seventeenth genrule concatenates, sorts and checks all 16 lines.
- One worker process and the same runner/image/platform before and after. Its runner concurrency changes from 1 to 4; no additional worker machines are added.
- Docker Desktop was configured for two CPUs and 3072 MiB VM memory. The temporary worker container was capped at 192 MiB and 0.5 CPU; the separate existing runner executes the action processes. These are configured limits, not continuous measurements of available host resources. `environment-snapshot.json` records a post-experiment check.
- One successful warmup, followed by all three baseline runs and then all three fixed runs. Order was not randomized or counterbalanced. The warmup is not in the measured six-run set.
- Every build uses `--jobs=16 --nouse_action_cache --disk_cache= --remote_accept_cached=false`, a dedicated output base and `--batch`. Remote cache writes and worker input-file reuse are allowed; action-result cache reads and local action reuse are disabled. This is not a cold-storage benchmark.
- All six runs exit successfully, execute 17 remote actions, and report zero remote cache hits. Each also has one internal Bazel action.
- All 17 target/action digests, commands, input digests, platforms, environment variables and output digests match across all six runs. The verification output is 48 bytes with SHA-256 `eb3becdaa9f75ea4c63169dba168dd18e273eed64e0f4c4ad694cf63ac9418f6`.

Waiting commands illustrate scheduling concurrency. They do not measure compiler throughput, CPU saturation, production autoscaling, costs, or the right concurrency for CPU/memory-heavy builds. Four slots on two CPUs is not a general recommendation.

## Diagnosis and actual MCP evidence

`decision.json` was recorded before the configuration change. Codex called the local Hermetiq MCP server, inspected the responses and the one-slot worker configuration, then applied the change using local terminal access.

`baseline-1-scheduler-mcp.json` reports 15 queued actions and one executing action at peak. `baseline-1-worker-mcp.json` reports low estimated per-action user/system CPU time. Those figures are CPU seconds per action, not CPU utilization percentages. This combination supports the specific hypothesis of a slot-constrained waiting workload.

The metric path is real Buildbarn Prometheus exposition -> one-second local VictoriaMetrics collection -> the existing Hermetiq MCP server. `config/prometheus.yml` and `config/promql.json` are the actual collector and native-metric query configurations. The lab labels these scheduler/worker metrics with the single local project `local-dev`. No other projects or customer metrics are included.

The scheduler backlog is derived from scheduled task count minus queued-duration count. Executing actions are derived from queued-duration count minus executing-duration count. The exported PromQL applies those differences to cumulative native Buildbarn counters and takes the maximum over the query window. There was no scheduler restart during the measured runs; do not apply those differences blindly across independent counter resets.

Important query limits:

1. Infrastructure queries are project-wide, not scoped to one invocation. Hermetiq uses at least a one-minute lookback for these summary queries. The measured runs were shorter than one minute. The first two after snapshots therefore include earlier baseline activity and still report `congested` with peak backlog 15. Those responses are retained without relabeling them.
2. The isolated `fixed-3-scheduler-mcp.json` reports four executing actions and 12 queued at peak, with assessment `healthy`.
3. Histogram quantiles are estimates. The baseline queue p90 estimate exceeds the exact maximum wait of this small run because bucket interpolation and the wider window differ from per-action measurements. It is not used in the result table.
4. Worker stage labels matter: the response contains separate `FetchingInputs`, `Running`, and other stage rows. Its generic assessment string may mention the first stage, not whole-action execution. Exact action execution times in the table are calculated from worker timestamps.
5. `find_remote_actions` returned no rows in this local completed-action-log configuration; that empty response is preserved. Detailed action verification was obtained from actual worker `ExecuteResponse` logs and Bazel execution logs, not from Hermetiq CAL query results.

This is a client-assisted infrastructure change. It demonstrates Codex using Hermetiq's actual connected infrastructure data plus local configuration access. It does not assert that an unattended Hermetiq service modified a production environment.

## Recalculate the existing evidence without a stack

Use Python 3.9 or later:

```sh
python3 verify-evidence.py
```

The script verifies all 102 remote action records, cache flags, matching identities/outputs, the single configuration change, logged total times, six per-run queue/execution means, summary medians and every exported file checksum. It reads only these public files; it neither accesses a server nor changes them.

Queue time is `workerStartTimestamp - queuedTimestamp`; execution time is `executionCompletedTimestamp - executionStartTimestamp`. Both use metadata in authentic Buildbarn `ExecuteResponse` records, joined by `toolInvocationId` and `targetId`. Means cover targets `//:unit_00` through `//:unit_15`. The verification action is excluded because it depends on the others. Python datetime parsing truncates sub-microsecond timestamp precision; values are reported to six decimal places, and the webpage rounds to three. Nanosecond-precision recalculation can differ by one microsecond in the final rounded digit.

The worker response's informational browser message says `Action details (cached result)` even for these executed actions. Cache status is established by Bazel `runner=remote`, `cacheHit=false`, successful actual worker execution timestamps and the response's false/default `cachedResult` field, not that browser-link message.

## Run fresh measurements

This bundle does not provision infrastructure or include the Hermetiq backend. Start a compatible local Buildbarn environment and a configured local Hermetiq BEP/MCP service first. The recorded Buildbarn configuration derives from its [Docker Compose reference](https://github.com/buildbarn/bb-deployments/tree/main/docker-compose). Use the exact exported configuration files to understand the worker/platform/query choices; their Docker service addresses are local lab names, not public services.

Recorded endpoints and settings:

| Connection | Recorded local value |
|---|---|
| Bazel remote cache/executor | `grpc://127.0.0.1:8980` |
| Remote instance | `local-dev` |
| BEP receiver | `grpc://127.0.0.1:50091` |
| BEP project/instance | `local-dev` |
| Local development identity | `X-Forwarded-User=public-example-agent` |
| Hermetiq MCP | `http://127.0.0.1:5150/mcp` |
| Metrics store | `http://127.0.0.1:8428` |
| Worker image | `ghcr.io/buildbarn/bb-worker:20260527T162223Z-c48dcda` |
| Runner platform image | `docker://ghcr.io/catthehacker/ubuntu:act-22.04@sha256:dd7654ffb01d5b7b54b23b9ce928a1f7f2d08c7b3d7e320b6574b55d7ccde78b` |

The public example identity and unauthenticated localhost endpoints are development settings, not production authentication instructions. In another environment use its provisioned endpoints, project, identity and transport. The recorded hardlinking worker depends on a compatible runner sharing `/worker`; `common.libsonnet` refers to two local storage shards. The configurations are evidence, not a complete deployment recipe.

1. Set one compatible worker's runner concurrency to 1 in an isolated local environment. Keep the source and other settings constant.
2. From the extracted bundle directory, run one warmup and three labeled baseline runs, serially:

   ```sh
   python3 run-build.py --output-dir ./fresh-runs warmup
   python3 run-build.py --output-dir ./fresh-runs baseline-1
   python3 run-build.py --output-dir ./fresh-runs baseline-2
   python3 run-build.py --output-dir ./fresh-runs baseline-3
   ```

   Use `--help` for configurable endpoints. Output filenames are never reused: choose a new label or directory to retry. The runner script preserves the recorded build flags, creates new invocation UUIDs and stores separate run metadata, logs, BEP and execution JSON. It does not change worker configuration.
3. Query Hermetiq `get_scheduler_health` and `get_worker_fleet_health` with the UTC `startTime`/`endTime` from each new `.run.json`; query `get_invocation` with its `invocationId`. Use the actual tool schemas in your installed MCP server. If isolated summaries are needed, leave at least one minute idle before each measured run, then query with that run's timestamps. Delaying the query afterward does not change a historical lookback window. Retain raw one-second metrics for per-run analysis.
4. Change only worker concurrency to 4 using `worker-concurrency.diff` as a reference, and restart that worker through your local lab's normal process. Wait for four slots to register. Run the three fixed labels with the same `run-build.py` flags.
5. Save worker `ExecuteResponse` logs for all six invocation IDs. Join action metadata by invocation and target, then compare queue waits, action execution, total build time, cache flags and verified output. Preserve slower or interrupted runs. Longer and counterbalanced experiments are appropriate before applying capacity recommendations to production work.

## File map and export transformations

- `sample/`: the exact BUILD/MODULE source and the recorded Bazel version.
- `config/`: actual before/after worker config, common config, scraper and PromQL overrides.
- `worker-concurrency.diff`: the complete one-field before/after difference.
- `decision.json`: the recorded pre-change diagnosis and verification plan.
- `record.json`: all six computed results and invocation provenance.
- `*.run.json`, `*.log`: build arguments, process timestamps and original Bazel console output with private paths replaced.
- `*.execution.json`: original consecutive Bazel execution JSON objects normalized into arrays.
- `*.worker-responses.json`: actual worker JSON responses selected only for the six recorded invocation IDs. No synthetic response generation.
- `*-mcp.json`: actual MCP `result.structuredContent`, preserving all response data while removing duplicated text/transport wrappers. Local dashboard URLs still refer to the recording environment.
- `*-metrics.raw.jsonl`: unchanged raw VictoriaMetrics exports for the selected local Buildbarn metrics and run windows.
- `environment-snapshot.json`, `provenance.json`: version, configured resource and export details.
- `run-build.py`: parameterized reproduction driver for fresh runs.
- `verify-evidence.py`, `SHA256SUMS`: offline recomputation and integrity checks. Checksums exclude `SHA256SUMS` itself and the enclosing ZIP.

Private recording-root paths are replaced with `/recording`; the local Bazel binary path becomes `bazel`; remaining local user paths become `/home/example`. These changes affect path/identity presentation, not timestamps, invocation UUIDs, action hashes, outputs, metric samples or measured durations. Raw BEP profiles, the backend binary/source, backend diagnostic logs, database, credentials and unrelated worker logs are not included.

## Configuration sources and attribution

The local worker/common configuration was adapted from the Buildbarn project's [bb-deployments reference configuration](https://github.com/buildbarn/bb-deployments). The exact upstream reference revision of that earlier local adaptation was not recorded; the precise files used for this experiment are included and checksummed. Modifications include a single local instance, a completed-action-logger endpoint, and the two recorded concurrency values. The reference is Apache-2.0 licensed; the license is included in `LICENSE.buildbarn`.

- [Buildbarn runner configuration schema](https://github.com/buildbarn/bb-remote-execution/blob/main/pkg/proto/configuration/bb_worker/bb_worker.proto)
- [Remote Execution API metadata and result schema](https://github.com/bazelbuild/remote-apis/blob/main/build/bazel/remote/execution/v2/remote_execution.proto)
- [Buildbarn deployment license](https://github.com/buildbarn/bb-deployments/blob/main/LICENSE)

The newly written sample, reproduction script and verifier are provided by Hermetiq for reproducing this example. The bundle grants no rights to unbundled Hermetiq software or third-party container images; obtain those separately under their own terms.