Adopt an AI build engineer without handing it production control

An AI build engineer should take investigation work off your platform team. That does not require giving it permission to change production on its first day. Start by asking it to explain one build problem, connect the evidence, and produce a next step that an engineer can review.

For platform leads evaluating AI assistance for Bazel and Buildbarn, the useful question is concrete: what can the agent establish with the data and access available today? A plausible explanation, a proposed patch, a successful test, and a deployed improvement are different results. Your evaluation should preserve those distinctions.

Separate inspection, proposals, and execution

Treat the first investigation as an explicit working agreement. Specify the affected build or time window, the evidence it may inspect, and the outcome you want. Then separate three kinds of work:

This agreement describes the task; your tooling still needs to enforce the access you intend. A sentence in a prompt does not remove credentials or prevent a connected terminal from running a command. Check both the build-data connection and the coding agent's repository, terminal, CI, and infrastructure access.

MCP makes the distinction easy to miss. It connects a host application to servers exposing capabilities. The MCP architecture assigns permission and consent responsibilities to the host and describes isolated client connections. Seeing “MCP connected” does not tell you which operations your particular client and server permit. Inspect their actual configuration and available tools.

Give the agent enough evidence to answer the question

The goal is fewer investigations for your engineers to assemble. An agent that gives you a list of dashboards to inspect has left much of that work with you. Ask it to retrieve and connect the evidence it can access, explain what each observation supports, and identify the specific gap when it cannot finish.

Different sources answer different questions. Bazel's Build Event Protocol describes a build through structured events, including configuration, progress, and results. It is a starting point for identifying the invocation and affected work. It is not a substitute for every other signal.

Profiles help locate time within the Bazel invocation. Completed-action records can separate remote phases. Cache records and action inputs help investigate reuse. Worker and scheduler telemetry show the infrastructure during the relevant interval. Configuration describes how work is routed and which workers can run it. The data-source documentation explains the integrations behind these views.

The agent should say which of those sources were available for this investigation. A configuration file can explain intended routing; it does not establish that the running deployment loaded that version. A missing telemetry series is a coverage gap, not a measured zero.

A queue is a question, not yet a capacity recommendation

Consider a hypothetical investigation: some actions wait while a worker pool appears idle. Increasing replicas sounds reasonable, but the observations alone do not establish that the waiting work can run on those workers.

A useful answer connects the waiting actions to their requested execution platform, the corresponding queue, eligible worker capacity, and the same time interval. It distinguishes a saturated compatible pool from workers serving a different platform or route. It also states when the connection cannot be made from the available evidence.

That is investigation work the agent should do. The engineer should receive an explanation of which relationship is supported and a bounded next step. If the agent lacks the relevant routing or worker evidence, “capacity shortage is not established” is a better answer than an unsupported scaling patch. No measurements or product result are claimed in this hypothetical example.

Require a proposal you can actually review

Ask for a compact decision record, rather than a confident paragraph followed by a patch:

Affected work: invocation, target/action, platform, and time window
Evidence: observed result, source, and link or identifier
Explanation: what the evidence supports
Uncertainty: missing evidence and plausible alternatives
Proposed change: exact file/field and intended effect, or no change yet
Verification: affected work to rerun and outcome to inspect
Action taken: what actually ran or changed, with its result

This is a suggested evaluation format, not a claim that every tool returns these fields. It is useful because a reviewer can trace the recommendation without repeating the entire investigation. “No change yet” is a valid proposal when the evidence only supports narrowing the question.

The verification must match the claim. Rendering a configuration demonstrates something different from deploying it. Passing an affected target establishes something different from reducing fleet-wide build time. Record the strongest outcome you observed and leave the next stage open until it happens.

Try it: an investigation record with missing evidence

Suppose an agent proposes increasing worker capacity after seeing a slow build. It has the invocation duration and a saved configuration draft. It has no queue or eligible-worker observations. Neither a long build nor a readable draft establishes that the deployed worker pool is saturated.

We wrote a small offline Python example to make that gap inspectable. It uses five synthetic records and Python's standard library. It did not connect to Hermetiq, run Bazel, or change a configuration. Its fields are an educational format, not Hermetiq MCP response fields.

For this example, a capacity proposal must reference queue evidence, eligible-worker evidence and deployed-configuration evidence for the same project, execution platform and time window. This is our example policy, not a universal diagnostic rule; a narrower question may need fewer sources.

Download the editable example, inputs, tests and recorded output. Extract it into a new directory and run:

cd worked-example
python3 review_record.py
python3 -m unittest -v test_review_record.py

The first synthetic record contains only an invocation duration and a saved draft. Its actual local output is:

{
  "case": "bep_and_draft_only",
  "review_status": "coverage_gaps",
  "gaps": [
    "queue: no observation",
    "eligible_workers: no observation",
    "active_config: saved draft is not deployment evidence"
  ],
  "configuration_changed": false
}

The checker follows each evidence reference and compares its availability, completeness and scope. For the configuration reference, it additionally checks whether the record identifies deployed configuration. The complete source is included in the download; this excerpt shows two of those checks:

if item['scope'] != scope:
    gaps.append(f'{role}: different project, platform, or window')
if role == 'active_config' and item['provenance'] != 'deployed':
    gaps.append('active_config: saved draft is not deployment evidence')

Adding a tool name does not repair missing telemetry. A second input lists a worker-health tool but marks its data unavailable. A third supplies idle-worker observations for another execution platform. A fourth marks worker evidence incomplete. All three return a specific coverage gap.

The fifth supplies matching, complete metadata. Its result is ready_for_engineering_review, with no configuration action performed. Metadata coverage does not prove saturation, validate a replica count, establish trustworthy observations or authorize execution. An engineer still needs to assess the measurements and verification plan.

All eight tests passed in the local run recorded with the example. They also check a different time window, an unlisted source tool, an irrelevant permission claim and preservation of the input record. This tests our example's policy, not the behavior or reliability of a live AI agent. No network or mutation code exists in it; it is not an access-control boundary.

The useful requirement is that the agent return the observations it has, name the exact gap, and explain which proposed change remains unsupported. Your engineer should not have to discover that a capacity recommendation rests on missing data. A narrower supported answer can still remove investigation work without producing a patch.

Use a public fix to calibrate the distinction

Hermetiq's recorded AI-assisted Bazel fix provides a small, inspectable example. A script reads a file absent from its rule's declared inputs. Hermetiq supplies action evidence; Codex checks the source and execution inputs, changes the rule, and reruns the affected target. A subsequent unchanged run records two remote cache hits.

The evaluation lesson is the chain of support: failure evidence explains the proposed edit, and actual reruns establish the reported outcome. A patch alone would stop before verification. The example used repository and terminal access in a local lab. It is neither a customer benchmark nor evidence that every failure can be fixed unattended.

Read the linked patch, logs, and methodology before using the example as a reference. For your own trial, use the same separation between evidence, edit, and observed result, even if your problem needs no patch at all.

Run the first evaluation on one bounded problem

Choose an invocation your team understands well enough to judge. Keep the target set and relevant conditions explicit. Give the agent a concrete first request:

Investigate this build using the available evidence. Explain the likely cause and uncertainty. Propose the smallest supported next step. Do not change configuration or infrastructure.

Supply the build identifier and permitted scope alongside that request. Apply the corresponding access controls in the client and connected systems. If you later authorize a patch or a test run, make the scope explicit and record its result.

Evaluate the output on five practical questions:

1. Does the explanation cite evidence for the affected work, rather than a generic Bazel tip?
2. Can an engineer review the proposed change without rebuilding the investigation from scratch?
3. How long did it take to reach a useful, supported answer, including follow-ups?
4. When challenged with conflicting evidence, did the agent correct its explanation and preserve uncertainty?
5. What outcome was actually verified: diagnosis, patch, test, deployment, or a measured change under comparable conditions?

Keep the questions and results together. One successful sample shows that a workflow can work. Repeat with your own workloads before expanding the operating scope. Count a supported decision to avoid an unnecessary change as useful work, too.

Choose the operating scope separately

You can evaluate build intelligence while keeping the infrastructure you already operate. Hermetiq Build Analytics & AI brings Barn Owl into investigations with connected build, action, infrastructure, and configuration evidence. Available analysis depends on the sources and access connected to the deployment.

Barn Owl is available directly in Hermetiq and through MCP for your own compatible agent. The public plugin requires a Hermetiq project receiving BEP events. Its documented tools depend on the connected deployment's capabilities; optional infrastructure tools are not universally available. Its skill requires explicit authorization for ConfigSet mutations. Check the live tool catalog and your client's permissions before evaluating a particular operation.

Managed Cache & Execution is a separate operating decision. Agree the supported workload and responsibilities with Hermetiq. Adding analytics to your existing build infrastructure does not itself transfer those responsibilities.

Start with one useful answer and an inspectable next step. After that, decide which further actions the evidence and your operating agreement justify.

Related reading:

• MCP for Bazel
• Why Everyone Is Talking About MCP
• What Is Build Observability?
• Bazel Build Platforms