Buildbarn Helm Values Explained: Scheduler, Storage, Workers, Frontend

Buildbarn Helm values decide the hosts, disks, auth, and queues you get on the first upgrade. The packaged defaults and the starter override are different files, and a README example is a third thing. This page is the field guide for chart 0.9.7: what to set before helm upgrade, which knobs are scheduler, storage, frontend, and browser, and which worker settings this chart still owns after you accept that it does not create workers.

The install companion is Buildbarn Helm Chart: Install Production Buildbarn on Kubernetes. That page and this one use chart 0.9.7: the version in Chart.yaml on tag buildbarn-v0.9.7, the version the chart README passes to helm show and helm upgrade, and the version Artifact Hub shows for hermetiq-buildbarn/buildbarn. That listing is under the Hermetiq organization and is a verified publisher. The short companion page is Buildbarn Helm chart. Facts below are from values.yaml, custom-values/buildbarn-values.yaml, values.schema.json, and charts/buildbarn/README.md at that tag.

TL;DR

Read the packaged defaults, then edit a copy of the starter file. Do not install either one unchanged on a cluster that is not the repository’s GKE-oriented example.

helm show values oci://ghcr.io/hermetiq/buildbarn --version 0.9.7
helm show readme oci://ghcr.io/hermetiq/buildbarn --version 0.9.7

Apply the edited override with the chart README’s upgrade command, from the directory that contains buildbarn-values.yaml:

helm upgrade --install --namespace hermetiq buildbarn \
  oci://ghcr.io/hermetiq/buildbarn \
  --version 0.9.7 \
  --values buildbarn-values.yaml

Before that upgrade, the keys that usually need a human are hosts, StorageClass names, routing, frontend JWKS if you keep the starter file’s auth, and scheduler.predeclaredPlatformQueues if any pool will scale to zero. Workers are still a second chart. keda.frontend scales the frontend only.

Those resource and size figures are configuration from the two files. They are not a measured recommendation for your workload.

Starter values versus helm show values

helm show values is the full default set inside the OCI chart. The starter file is only an override. Helm merges them. A key absent from your --values file stays at the packaged default, including the StorageClass name premium-rwo and routing.provider: contour if you drop the starter’s routing block.

The chart README’s install section tells you to start from the starter file, then inspect the packaged README and values at --version 0.9.7. Its storage section later shows a different stanza, with replicas: 5, a 1Ti CAS volume, blocksSizeGi: 900, and a 20Gi Action Cache. That block is an example. The packaged defaults are the ones in the table above: three replicas, CAS 1000Gi with blocksSizeGi: 950, Action Cache 12Gi with blocksSizeGi: 10.

values.schema.json rejects some mistakes at render time. routing.provider has to be one of gateway, gateway-httproute-only, contour, ingress, none, or an empty string. Scheduler actionRouter.mode has to be simple, demultiplexing, or custom. The schema types many nested blocks as objects, so it does not restate every default. helm show values is still the list of defaults. configOverrides replaces a Jsonnet file verbatim when the values surface is not enough. Unknown override filenames fail helm template.

Scheduler values

The chart renders Deployment scheduler-ubuntu22-04 and Service scheduler. scheduler.replicas defaults to 1. That replica count is the scheduler process, not a worker pool. Platform matching compares {instanceNamePrefix, the marshalled platform} by exact string equality. An action whose platform matches no pool fails with No matching worker pool, or waits. There is no silent fallback in the default scheduler.actionRouter.mode of simple.

Scale-to-zero pools need a queue that exists before any worker is registered. Declare it under scheduler.predeclaredPlatformQueues. The packaged list is empty. Without a declaration, an unregistered platform is rejected with No workers exist, so KEDA never sees the queue signal it needs to start a worker. Queues learned from workers disappear after scheduler.platformQueueWithNoWorkersTimeout, which defaults to 900s. defaultExecutionTimeout is 1800s and maximumExecutionTimeout is 7200s. Those are scheduler configuration, not a service objective.

The chart README’s example queue, which you must match to the pool you actually run:

scheduler:
  predeclaredPlatformQueues:
    - instanceNamePrefix: ""
      platform:
        - name: container-image
          value: docker://ghcr.io/catthehacker/ubuntu:act-22.04
      sizeClasses: [0]

Match the full platform to the pool’s spec.config.generated.platformProperties, sorted by name then value. The scheduler rejects an unsorted platform. Match sizeClasses to the pool’s sizeClass. A single class can use [0]. The empty instanceNamePrefix accepts every client instance name. Changing only the worker’s Prometheus instanceNamePrefix does not change worker registration.

actionRouter.mode: demultiplexing is how you add a catch-all. Every platform you still want to route explicitly has to be listed under backends. Everything else is rewritten onto catchAll.platform. The chart README states the two costs: a typo that used to fail fast now runs on the catch-all pool, and the Action Cache keeps the client’s platform, so the catch-all shares no cached results with the pool it imitates. Leave the mode at simple until you mean to accept that.

Storage values

Storage is a sharded StatefulSet named storage, with a headless Service of the same name. The shard count is storage.replicas. common.libsonnet generates one CAS shard and one Action Cache shard per replica. The packaged default is 3. storage.podAntiAffinity.enabled defaults to true, and the values comment says three replicas then need three eligible nodes.

Persistence defaults to mode: pvc. Both CAS and AC use StorageClass premium-rwo. The chart README’s install section says to run kubectl get storageclass and set storage.persistence.cas.storageClassName and storage.persistence.ac.storageClassName to a class the cluster has. On EKS, use gp3 unless the cluster provides a premium-rwo alias backed by EBS gp3. The starter file does not set those names, so an install that only copies the starter still asks for premium-rwo.

storage:
  replicas: 3
  persistence:
    mode: pvc
    cas:
      storageClassName: premium-rwo
      size: 1000Gi
      blocksSizeGi: 950
    ac:
      storageClassName: premium-rwo
      size: 12Gi
      blocksSizeGi: 10

blocksSizeGi has to leave headroom for metadata. The CAS comment says the blocks file must be smaller than size. Changing a block count, or the blocks or device size, changes the derived block size, and a persistent store discards existing data on the next start. ISCC and FSAC are separate stores and default to enabled: false. Enable ISCC together with scheduler.sizeClassAnalysis.enabled. Enable FSAC together with worker prefetching. The key-location-map sizing comments say their average object sizes come from one dev deployment and tell you to measure your own cluster. This page does not turn those comments into a sizing formula.

Packaged storage requests are cpu: 500m and memory: 4Gi, with limits cpu: 7100m and memory: 16Gi. The starter file raises requests to 7100m and 16Gi and says to tune them for shard count and cache load. emptyDir and hostPath are cache storage: pod recreation, drain, or node loss can erase a shard. The chart README says there is no automatic migration between pvc, emptyDir, and hostPath, and changing an existing release requires recreating the StatefulSet because volumeClaimTemplates are effectively immutable. The storage PodDisruptionBudget defaults to enabled with maxUnavailable: 1. The storage gRPC port accepts unauthenticated traffic; storage.networkPolicy.enabled defaults to false.

Frontend values

The frontend Deployment is the client API for CAS, ByteStream, Action Cache, and Execute. frontend.replicas defaults to 2. That field is rendered only while frontend autoscaling is off. The template omits spec.replicas when both keda.enabled and keda.frontend.enabled are true, so Helm does not fight the HPA. Packaged keda.enabled is true and keda.frontend.enabled is false, so a default install uses two replicas and no ScaledObject. The starter file turns the frontend scaler on:

keda:
  frontend:
    enabled: true

With the rest of the packaged keda.frontend block left in place, that means minReplicaCount: 2, maxReplicaCount: 20, an in-flight gRPC threshold of "100", CPU target utilization "70", and memory scaling left off. The Prometheus address defaults to http://vmselect-vmks.hermetiq.svc.cluster.local:8481/select/0/prometheus. The chart README says to keep minReplicaCount at least 1, because the signal comes from the frontend pods, and that the chart rejects zero. It also says scale-down keeps recent capacity for five minutes and then removes at most 25% or one pod per minute, whichever is smaller. Memory scaling stays off by default because a disk-backed read cache can inflate the working set. frontend.readCache.enabled itself defaults to false. This scaler does not scale workers. KEDA and a scraper that keeps the kubernetes_service="frontend" label have to exist if you enable it. The chart does not install either one.

JWKS is off in the packaged defaults and on in the starter file. The starter’s issuer, audience, and JWKS URL are placeholders. Replace them so they match the token before you install that file. The chart comment says the frontend reloads the mounted JWKS file every 300 seconds, and kubelet propagates ConfigMap updates within about 60 seconds. The sync CronJob defaults to 0 */6 * * *. You can manage ConfigMap frontend-jwks yourself and leave frontend.jwks.sync.enabled false.

frontend:
  jwks:
    enabled: true
    issuer: https://<tenant>.auth0.com/
    audience: https://bep.helm.hermetiq.dev
    sync:
      enabled: true
      url: https://<tenant>.auth0.com/.well-known/jwks.json
  actionCache:
    putAuthorizer:
      mode: requireCanWriteToCache
  contentAddressableStorage:
    putAuthorizer:
      mode: requireCanWriteToCache
  executeAuthorizer:
    mode: requireCanWriteToCache

With JWKS enabled, the authentication policy is any of a JWT check and allow. Unauthenticated callers are still admitted. CAS get, FindMissing, and Action Cache reads stay allow. The starter file requires canWriteToCache for CAS puts, Action Cache puts, and Execute. The IdP does not need a separate canWriteToCache claim. The JWT arm sets that metadata. Packaged authorizers are allow, so a default install without the starter file leaves writes and Execute open. Closing reads is a configOverrides["frontend.jsonnet"] change, and the chart README says the browser, bb-portal, and tokenless CI lose access at the same time.

config.maximumMessageSizeBytes is shared through common.libsonnet by frontend, storage, scheduler, browser, workers, and remote asset. The packaged value is 10485760, which the comment labels 10 MiB. The starter file sets 838860800 and labels it 800 MiB, for large unary BatchUpdateBlobs and Tree requests. Streaming ByteStream transfers are unaffected. The starter comment says to size pod memory for large requests. The optional grpc-cache-proxy sidecar has its own maxMessageSizeBytes of 67108864, and the values comment says that limit is floored at config.maximumMessageSizeBytes so a smaller proxy limit cannot reject a Buildbarn response. Packaged frontend requests are cpu: 300m and memory: 1Gi, with limits cpu: "2" and memory: 3Gi. The starter file sets requests to cpu: "2" and memory: 3Gi.

Browser, routing, and hosts

These are the values operators usually get wrong on the first install, because the README example, the packaged default, and the starter file do not all say the same thing.

Packaged hosts.domainBase is helm.hermetiq.dev. Empty host fields derive browser.<domainBase>, bb.<domainBase> for frontend gRPC, rbe-web.<domainBase>, and, when those features are enabled, asset., portal., and bes.. With the packaged string, frontend gRPC is bb.helm.hermetiq.dev and the browser is browser.helm.hermetiq.dev. The chart README’s service-topology snippet sets domainBase: bb.helm.hermetiq.dev. Pasting that snippet derives bb.bb.helm.hermetiq.dev. Set one suffix you control, or set hosts.frontendGrpc, hosts.browser, and the other host fields individually. For Contour and Ingress, an empty certificate.dnsNames derives a *.domainBase wildcard plus the concrete hosts, and that derivation needs hosts.domainBase. Gateway mode does not render that Certificate. TLS has to terminate on the Gateway listener.

hosts.otel defaults to otel.cloud-grpc.hermetiq.io. The grpc-cache-proxy comment says that hostname is a Hermetiq-operated host and that on-prem installs should point it at their own collector. The starter file sets otel-collector.hermetiq.svc.cluster.local, which exists only if that collector is installed. The proxy’s OTLP endpoint falls through to http://<hosts.otel>:4317 when tracing does not supply one.

routing.provider defaults to contour. The starter file sets gateway and gateway.name: hermetiq-gateway. The packaged gateway.name is empty, so turning on Gateway routing without a name does not attach to a Gateway you forgot to mention. Use gateway-httproute-only for GKE Gateway, which has HTTPRoute and not GRPCRoute. Use none or routing.enabled: false when another system owns the routes. routing.provider: gateway also renders Envoy Gateway BackendTrafficPolicy and Browser CORS resources. On a GRPCRoute controller that is not Envoy Gateway, the chart README says those kinds fail the install until you disable them. gateway.clientTrafficPolicy.enabled defaults to false. The README describes it as a Gateway-scoped policy for high-RTT clients, and says to enable it in only one chart when Hermetiq and Buildbarn share a Gateway.

The scheduler web UI is the other early mistake. ingress.rbeWeb.enabled defaults to true. Contour’s HTTPProxy and the nginx Ingress both follow that flag. gateway.httpRoutes.rbeWeb.enabled also defaults to true. The starter file sets the Gateway flag to false and says not to expose the scheduler web UI until you have access controls for it. That flag does nothing while routing.provider is still contour. Scheduler admin HTTP authentication defaults to allow. The values comment says that HTTP policy cannot do mutual TLS. Browser SSO is a separate switch: browser.oauth2Proxy.enabled defaults to false, and the starter file turns it on, reusing ConfigMap oauth2-proxy-config-web-ui and Secret oauth2-proxy-client from the Hermetiq chart. When that chart is absent, turn the proxy off or set existingConfigMap empty and supply oidcIssuerUrl. With the proxy enabled, the Browser Service exposes oauth2-proxy on port 80. Without it, the browser container listens on 7984. browser.service.targetPortOverride is mutually exclusive with the oauth2-proxy.

Workers are not in this chart

The Buildbarn chart does not render worker Deployments or worker KEDA ScaledObjects. The supported bundle table in the repository README at the same tag pairs Buildbarn 0.9.7 with bb-worker-operator 0.3.8 (appVersion v0.3.9). Install that operator, then apply RbeWorker resources in the Buildbarn namespace. The install steps for the operator and the sample pools are in the install article and in the operator README. A pool with minReplicas: 0 stays at zero ready replicas until its predeclared queue has work.

This chart still owns the worker-facing configuration that has to match the pools:

• ConfigMap buildbarn-worker-config, which holds shared common.libsonnet. The operator mounts it and generates each pool’s worker and runner Jsonnet from the RbeWorker. Shard addresses and config.maximumMessageSizeBytes reach workers through that file.
• scheduler.predeclaredPlatformQueues, platformQueueWithNoWorkersTimeout, and scheduler.actionRouter. Worker autoscaling under spec.autoscaling on the RbeWorker does not create those queues.
• NetworkPolicy peers. Storage and scheduler policies, when enabled, already allow pods labeled app=worker in the release namespace. Extra clients go in additionalWorkerPeers or additionalClientPeers.
• worker-vmpodscrape when vmPodScrapes.enabled is true, which it is by default. The chart README says to keep the app=worker label on operator-managed pods or the scrape misses them.

keda.frontend does not scale those pools. The chart does not create worker node pools. Sample pools select amd64 Linux nodes and add no custom tolerations. FUSE and Docker-in-Docker pools need privileged containers, while the chart-managed frontend, scheduler, storage, and browser pods stay on the restricted security context.

Limitations

• Installing oci://ghcr.io/hermetiq/buildbarn does not start workers.
• The starter file targets the repository’s GKE-oriented examples. Other clusters need their own StorageClass, identity, Gateway or Ingress, DNS, and TLS. The chart does not install those controllers.
• Frontend reads stay open when JWKS is enabled. Writes and Execute follow the authorizer mode, which the starter file tightens and the packaged defaults leave at allow.
• No comparative performance numbers are published for this chart. Do not treat replica counts, disk sizes, the starter resource requests, or a chart comment about one observed overflow as a benchmark.
• Changing storage.replicas or block geometry after data is written can strand or discard the cache. Plan the shard count before the first production upgrade.
• The README host example and the packaged hosts.domainBase are different strings. Derive names from the string you set.
• Platform matching has no silent fallback. A catch-all router is optional, and turning it on stops mismatches from failing loudly.
• Frontend autoscaling needs KEDA and the configured Prometheus address. It will not scale workers, and the chart rejects a zero frontend minimum.

Related reading:

• Buildbarn Helm chart — landing page for the same OCI chart.
• Buildbarn Helm chart install guide — prerequisites, what the chart deploys, and the worker operator.
• How to set up Buildbarn — architecture and operating choices around a cluster.
• Deployments — Hermetiq Cloud and customer-managed Kubernetes.
• Buildbarn monitoring checklist — queues, workers, storage, and gRPC after the cluster is up.
• Build Analytics & AI — investigations on Buildbarn you operate.
• Managed Cache & Execution — Hermetiq operates the remote cache and execution.