Concurrent Bazel builds: find the queue before adding workers

A coding agent reports that a build is taking too long. Another agent is already running tests. Adding remote workers sounds plausible, but first answer a narrower question: has the slow invocation reached remote execution at all?

There are three places to investigate: admission to a Bazel server, resources on the client machine, and a remote execution worker. They need different evidence and different fixes. Treating all three as “the build queue” can turn a coordination problem into a larger infrastructure bill.

This guide is for engineers running concurrent Bazel commands from coding agents, terminals or CI jobs. A small downloadable example demonstrates the server boundary. The remaining checks explain how to investigate contention once a command gets past it.

Check whether the commands share one server

Bazel's client connects to a server associated with an output base. That server handles one invocation at a time. Two agents using the same checkout and output base can therefore serialize before the second command does useful build work. Multiple actions running concurrently inside one build are a separate kind of parallelism. See Bazel's client/server implementation.

Capture the command, terminal output, launch and completion times, workspace path and output base for each invocation. Keep any message identifying another running command. That message tells you a particular Bazel server is busy; it does not tell you whether remote workers are full.

For a diagnostic in a disposable checkout, the startup option --noblock_for_lock makes a command exit rather than wait. Startup options go before the command. Check your installed version's help and preserve the exit status. An unattended agent should back off or join a queue instead of retrying in a tight loop. The command reference documents the option.

Reproduce the boundary without a remote service

Download the example, runner and recorded results. It uses Bazel 8.4.2, Python 3 and a POSIX shell. We ran it on macOS arm64 on November 3, 2026. Bazelisk honors the pinned version; a standalone Bazel executable must already be version 8.4.2. The first run may download Bazel and its dependencies.

unzip bazel-concurrency.zip
cd bazel-concurrency
python3 run.py

The runner copies the sample into a new temporary directory and prints its evidence path. It ignores user rc files, uses two explicit output bases and disables workspace convenience symlinks. It leaves logs, profiles, commands, timestamps and a results.json file there, then shuts down only the two servers it created. It does not configure a remote cache, executor or Hermetiq service.

There are two targets. //:quick copies a declared source file. //:hold deliberately sleeps for 12 seconds before copying it. That sleep keeps the first server occupied; it is not representative compiler work. The runner warms both servers before the concurrency check.

These are synthetic behavior checks, not a throughput benchmark, a memory comparison or a remote-execution test. The runner fails if the hold ends before independence can be demonstrated. Its recorded logs replace only the temporary run-directory prefix with <RUN_ROOT>; the commands and outcomes remain inspectable.

The fail-fast command in the runner has this shape, with its temporary paths supplied explicitly:

bazel --ignore_all_rc_files \
  --output_user_root="$run_root/user" \
  --output_base="$run_root/a" \
  --noblock_for_lock build //:quick \
  --incompatible_autoload_externally= --symlink_prefix=/ \
  --color=no --curses=no --subcommands \
  --show_progress_rate_limit=0 \
  --profile="$run_root/fail_fast.profile.gz"

run_root represents the directory printed by the runner; the command runs inside its workspace subdirectory. The rc isolation and autoload setting belong to this controlled example. Retain your repository's required wrapper, toolchains, platforms and execution settings when investigating a real build.

Choose server and source isolation separately

An explicit --output_base selects another output base. Bazel normally derives the default from the workspace location; an ordinary symlink to the same checkout does not necessarily give you a new one. The output directory layout explains the paths and overrides.

Separate output bases do not make a mutable source directory safe for independent edits. If an agent changes files while another agent's tests run, a green result may not describe the intended revision. Use a stable source snapshot for comparisons. Decide source ownership and revision isolation separately from build-state storage.

There is also a warm-state cost. A server retains loaded packages and its analyzed dependency graph. A second server starts without the first server's analysis state. Action-result caching is another layer and may be shared remotely; a remote hit does not eliminate local loading and analysis. Bazel's iteration-speed guide describes those layers.

Before creating a server per agent, decide how much memory the host can afford, whether warm analysis is valuable, and who owns each source snapshot and output base. If admission is the limiting step, a coordinated validation queue may be sufficient. Have it reject obsolete work, retain the newest relevant revision and attach each result to the revision actually tested. Those are orchestration policies to implement and verify.

Measure the client after admission

Save a uniquely named profile for each invocation with --profile. Bazel's JSON trace profile helps inspect loading, analysis, critical-path actions and concurrency. Available rows vary by version and flags. Keep launcher timestamps too: a profile alone should not be assumed to account for every second the agent experienced.

Observe CPU pressure, memory pressure, disk activity and the number of Bazel servers over the same interval. Compare a stable workload running alone with the same workload under controlled concurrency. A slowdown under load establishes contention; it takes resource measurements to identify the bottleneck.

Local action scheduling controls such as --local_resources and --jobs apply within an invocation. Independently configured servers do not thereby negotiate one host-wide resource budget. Start conservatively and measure their combined effect. Use the resource flag reference for your version.

Look at remote phases when actions reach the backend

For remotely executed actions, distinguish queue, input-fetch, execution and output-upload time. Choose representative slow actions on the path that determines build completion. Repeated queueing suggests investigating scheduler capacity and worker eligibility; long input fetch calls for storage and network evidence. Neither observation proves the entire build would improve by the same amount.

Do not add every action duration and call the sum build elapsed time. Actions overlap. A useful change shortens the path that determines completion without moving the delay somewhere else.

In Hermetiq, profile analysis requires uploaded profile data; remote queue, fetch, execution, upload and worker detail require completed-action data. The documented build views describe those dependencies. An empty or unavailable view leaves that part of the investigation unknown.

Run one comparison you can interpret

Use one revision, target set, toolchain and execution platform. Record warm or cold state. Run a baseline, then change one thing: admission policy, the concurrency budget, or a backend setting supported by the evidence. Repeat enough to see whether the result is stable rather than relying on one favorable run.

• Keep the time from the agent's request to a usable result, including observable admission delay.
• Retain build phases, critical path, host pressure and available remote action phases.
• Record tests actually executed, cached results, failures and the exact revision validated.

Start with the downloadable example, then capture one slow invocation in your own workflow. Locate the delay before deciding whether coordination, local resources or remote infrastructure should change.

Related reading:

• Bazel performance guide
• Validate Bazel changes with the right tests and evidence
• What a cache hit rate hides
• Measured Buildbarn queue-capacity walkthrough