A Buildbarn queue, measured before and after.

1. The work is waiting for a slot.

The sample has 16 independent Bazel genrules. Each runs sleep 1 and writes a numbered line. A seventeenth action checks the combined output. Bazel allows 16 concurrent jobs, while Buildbarn’s single worker initially advertises one execution slot.

After the first baseline build, Codex queries the actual Hermetiq MCP server with get_scheduler_health and get_worker_fleet_health. These tools use Buildbarn’s scheduler and worker metrics, collected into the local metrics store every second.

assessment: congested
scheduler_queue_depth: 15 actions (peak)
scheduler_executing_actions: 1 action (peak)

The worker response shows low per-action CPU use: estimated p90 user CPU time of 0.0043 seconds and system CPU time of 0.0054 seconds. These are histogram estimates, not CPU utilization percentages. Together with the waiting commands and one-slot configuration, the evidence points to insufficient concurrency for this workload.

The recorded decision precedes the change. Its hypothesis is specific: allow more of these independent waiting actions to execute together, then check queue time and build correctness.

2. Change one worker setting.

Codex changes concurrency from 1 to 4 in the same worker configuration and restarts that worker. This provides four execution slots in one worker process, following Buildbarn’s runner concurrency configuration. The Docker VM remains limited to two CPUs and 3 GiB of configured memory.

-      concurrency: 1,
+      concurrency: 4,

The source, runner image, platform properties, Bazel job limit and cache flags stay the same. Local action reuse, the disk cache and remote cache reads are disabled in every measured run, so a cached result cannot stand in for execution.

This concurrency setting is appropriate to test here because the actions mostly wait. CPU-heavy compilation or memory-heavy linking needs its own resource measurements; four slots on two CPUs is not a general fleet-sizing recommendation.

3. Less queueing in every run after the change.

The median of the three per-run mean queue waits falls from 7.89 seconds to 1.81 seconds. Each mean covers the same 16 waiting actions. It is calculated from Buildbarn’s actual worker-start and queued timestamps in the Remote Execution API metadata; the dependent verification action is excluded.

* The third after run resumed following a pause of about 2 hours 37 minutes. Queue wait remained lower, but its total Bazel elapsed time was slower than all three baseline runs. Its Hermetiq BEP invocation interval was 15.182 seconds, Bazel reported 19.435 seconds, and the enclosing process took 40.507 seconds. These clocks measure different boundaries. Host conditions across the pause were not controlled; the evidence does not establish the cause of the slower total time.

All six builds succeed with 17 remote executions and zero remote cache hits. Their target/action digests, commands, inputs and outputs match. The verification action checks all 16 numbered lines and produces the same 48-byte output in every run.

The result is a demonstrated reduction in queue wait for this workload. It does not show that every build gets faster. The complete measurement record, resumed invocation and individual worker records preserve every run, including the slower one.

assessment: healthy
scheduler_queue_depth: 12 actions (peak)
scheduler_executing_actions: 4 actions (peak)

Infrastructure queries cover the project and use at least a one-minute lookback. The first two after responses therefore include earlier baseline activity and still report congestion. Those responses are retained. The table uses invocation-matched worker timestamps instead of these overlapping summaries or interpolated histogram percentiles.

Reproduce the workload. Recalculate the evidence.

The download contains the source, before and after worker configurations, metric query templates, raw metric exports, MCP responses and all six sets of execution records. It also contains two Python scripts: one verifies the recorded results without a running stack; the other runs the workload against an existing local environment.

1. Extract the example ZIP into a new directory. Recompute the means, verify matching actions and outputs, and check all file hashes:

   python3 verify-evidence.py

2. For fresh measurements, provide Bazel 9.1.0, a compatible local Buildbarn Linux worker and the Hermetiq BEP and MCP endpoints. The example does not provision these services. Use the README to map the recorded configuration and queries to your own environment.

3. Start with one worker slot, run one warmup and three baseline builds, then query Hermetiq’s scheduler and worker tools for those windows. Apply the single concurrency change and run three more builds. Keep cache reads disabled and store each run separately.

   python3 run-build.py --output-dir ./fresh-runs baseline-1

4. Capture worker ExecuteResponse records and join them to builds using toolInvocationId and targetId. Compare queue wait, action execution, total build time and identical verified output. Let infrastructure query windows clear between runs if you need isolated summaries.

The captured records identify the Linux remote platform and runner. Bazel’s darwin_arm64 output path reflects the macOS client configuration.

What the evidence supports.

This is a client-assisted infrastructure optimization. Codex used the real Hermetiq MCP server to inspect scheduler and worker telemetry, changed the local worker configuration with terminal access, and reran Bazel. Hermetiq supplied the infrastructure evidence; this recording does not claim an unattended service changed a production fleet.

The Buildbarn metric responses add evidence beyond BEP. Exact per-action timing verification comes from authentic Buildbarn worker logs and Bazel execution logs. The local completed-action-log query returned no rows, so these detailed worker records are not presented as results from Hermetiq’s completed-action-log tools.

There was one warmup, then three before runs and three after runs in sequence. The order was not randomized. This small waiting workload demonstrates a queue-capacity mechanism, not production throughput, compilation performance or a customer benchmark.

Recorded

September 8, 2026 UTC

Bazel

9.1.0

Worker image

bb-worker:20260527T162223Z-c48dcda

Hermetiq backend

Existing binary, compiled unchanged from efbd9de19d5cfccf46039d3431ccbda7b33df2b0

The lab connects native Buildbarn metrics to Hermetiq using the exported query configuration. Public responses retain their values, timestamps and action IDs; local user paths are replaced with generic paths. The README documents transformations, limitations and configuration attribution.

• Complete source and evidence ZIP
• Methodology and reproduction
• Environment and export provenance
• Actual infrastructure query configuration
• File checksums

Related reading

• Buildbarn and remote execution resources
• Buildbarn intelligence with Hermetiq
• Bazel remote execution guide
• Set up Buildbarn
• Watch a build failure become a verified fix →