Skip to main content
The Helm chart at deploy/helm/charts/vexa deploys the same control plane as the Compose stack, with the runtime pointed at the cluster: RUNTIME_BACKEND=k8s makes every bot and every agent dispatch its own bare Pod (restart: Never), so capacity is your cluster’s scheduler — not a bigger box.

​
What the chart deploys

Hardened defaults throughout: containers run non-root with all capabilities dropped, PodDisruptionBudgets on the control plane, and rolling updates that never take the last replica down.

​
Install

values-staging.yaml and values-test.yaml show environment overlays. Chart tests (helm lint + template tests) live under deploy/helm/tests/.

​
Is this install actually working? — make probe SURFACE=helm

The standing full-journey smoke probe drives spawn → schedule → boot → join → transcribe → live-view → stop through the gateway front door, then sweeps every deployment’s logs once. Each stage prints Expected / Actual / Verdict; a red stage names where the journey broke and fails the command, so you get a truthful install verdict in minutes with no meeting and no humans.
It mints its API key from the release secret’s ADMIN_API_TOKEN (or pass VEXA_API_KEY). The same journey runs on the other surfaces: make probe (compose, the fast default) and make probe SURFACE=lite.

​
The runtime on Kubernetes

The runtime kernel owns one runtime.v1 lifecycle (starting → running → stopping → stopped → destroyed) and delegates only the substrate question — how do I start, observe, and stop a workload? — to the backend. On k8s, a workload is a bare Pod; the same dispatch, contracts, and worker run identically to the Docker backend. See Execution.

​
Scheduling spawned Pods on a tainted / dedicated pool

A spawned bot or agent Pod is created with kubectl run — it is not a child of the runtime Deployment, so it does not inherit global.nodeSelector / global.tolerations. On a cluster whose nodes are all tainted (the standard pool-segregation pattern — dedicated, spot/temp, or GPU node pools), a spawned Pod with no tolerations sits Pending forever (untolerated taint(s)) and the meeting silently fails while the runtime itself — a Deployment that does apply those constraints — looks healthy. The chart closes this by handing the runtime its own scheduling constraints as env, which the spawn backend stamps onto every Pod it creates: By default a spawned Pod schedules wherever the runtime itself can — set global.tolerations / global.nodeSelector for your pool and both the runtime and everything it spawns land there, with no extra action:
Set runtime.tolerations / runtime.nodeSelector only to send spawned workloads to a different pool than the kernel (e.g. GPU bots). Both values are serialized to JSON; malformed JSON fails the spawn loudly rather than dropping the constraint (a dropped constraint is the stranded-Pod bug).
Spawned-workload commands must exist in the target image under k8s (entrypoint-replace) semantics. A runtime profile may carry a command; the k8s backend passes it as kubectl run --command -- …, which replaces the image ENTRYPOINT (it becomes the Pod’s argv[0]). So a command path that is not present in the image StartErrors every spawn on k8s — even though the Docker backend, which appends the command to the entrypoint as arguments, silently ignores the same bad path. The meeting-bot profile therefore carries no command: the bot image’s own ENTRYPOINT (/app/entrypoint.sh) boots it on every backend. At release time an image↔profile conformance gate (image-identity) asserts each profile command is either empty or an in-image executable against the published images, so a profile that diverges from its image fails the release rather than a customer’s install.

​
Autoscaling and bot bursts — the fresh-node reachability gate

Because every bot is its own bare Pod, a burst of meeting requests can push the cluster autoscaler to add nodes, and bots get scheduled onto them the moment they report Ready. But a brand-new node’s network can take up to a few minutes to fully converge after Ready — the CNI/NetworkPolicy programming, kube-proxy, and DNS warmup all lag node readiness. A bot scheduled into that window can find its control plane (its meeting-api callback URL and redis) unreachable on its first outbound hop. The bot handles this with a pre-join reachability gate: it makes its first joining lifecycle emit load-bearing. If the meeting-api callback is reachable (any HTTP response), it joins immediately — zero added latency. If not, it probes redis; if either channel is up it proceeds (it can still report), and only if both are down does it refuse to join, terminating fast (< a few seconds) with:
  • Pod exit code 3 — the attributable terminal signal (kubectl describe pod <bot> → Last State: Terminated, Exit Code: 3). Distinct from a real join failure (exit 1).
  • a failed lifecycle event carrying failure_stage: requested and infra_fault: control_plane_unreachable on whichever channel recovers (often none — the exit code is the signal).
This converts an opaque CrashLoop / stuck-requested meeting into a fast, attributed failure an operator can read in one kubectl describe. See Troubleshooting → exit code 3. What stays cluster-side (not solved by the gate): the node-readiness window itself. Consider pre-pulling the bot image onto new nodes (a cold pull can itself take minutes and mask the network window), and treat node Ready as insufficient for scheduling network-dependent work — the gate fails fast and, on k8s, the exit lets the operator (or an autoscaler policy) reschedule onto a warmed node.

​
Scaling meeting-api

meetingApi.replicaCount defaults to 2, and that is safe for the segment consumer. Each replica now joins the transcription_segments consumer group under a per-pod identity (collector-<hostname>, overridable with COLLECTOR_CONSUMER_NAME), and a surviving replica periodically XAUTOCLAIMs a crashed replica’s un-acked segment batch and drains it through the normal persist path — so no transcript segments are orphaned when a pod dies mid-batch. The reclaim only fires once a batch has idled past COLLECTOR_RECLAIM_MIN_IDLE_MS (default 60000), so a live peer’s in-flight batch is never stolen. /health exposes pipeline.pending_depth (the group’s delivered-but-un-acked count) alongside pipeline.consumer_lag, and degrades to 503 once it exceeds PIPELINE_PENDING_ALARM (default 100) — a stuck batch is a reportable state, not a silent stall. A pod’s hostname changes on every recreate (a rolling deploy, a pod restart, a compose --force-recreate), so each new container joins collector_group under a new collector-<hostname> and the old name is left behind. Those abandoned consumers hold no pending entries (they read their last message and acked it), so they are harmless to correctness — but left unchecked the group fills with dead names that inflate operator XINFO CONSUMERS reads and muddy /health. The same reclaim sweep now prunes them: any consumer with pending == 0 that has idled past COLLECTOR_CONSUMER_TTL_MS (default 1800000, 30 min — well above the reclaim idle gate, so a briefly-quiet live replica is never touched) is removed with XGROUP DELCONSUMER. A consumer holding an in-flight batch (pending > 0) is never pruned, and the running replica never prunes itself; a live consumer re-registers on its next XREADGROUP, so the prune is idempotent and self-healing. Reading XINFO CONSUMERS collector_group should therefore list only the replicas that actually exist — dead collector-<oldhash> names are expected to disappear within a sweep of the TTL. For k8s you can eliminate the churn at the source by pinning a stable per-replica identity via COLLECTOR_CONSUMER_NAME (e.g. a StatefulSet ordinal) so a restart re-uses the same consumer.
Every surface now ships Valkey 8.x, which has XAUTOCLAIM — so orphan reclaim is active everywhere Vexa ships, including Vexa Lite (compose, helm, and Lite all run Valkey, the Linux Foundation BSD-3 fork of Redis 7.2.4; see the changelog and #653). Reclaim, per-pod identity, and ghost-consumer pruning all work on the shipped stack. Check yours with valkey-server --version.The degradation path below still exists as a safety net for a bring-your-own backing store older than the floor: on a Redis/Valkey without XAUTOCLAIM the reclaim disables itself and logs once at startup; everything else (normal XREADGROUP consumption, pending_depth) is unaffected. The cost is narrow but real — if a replica dies mid-batch on such a store, that batch stays pending instead of being drained by a peer, so run a single replica or move to a store with XAUTOCLAIM (Redis ≥ 6.2 / Valkey ≥ 7.2). Ghost-consumer pruning degrades the same way: if the backing store rejects XINFO CONSUMERS, the prune logs once and no-ops, leaving the consume path untouched.

​
In-cluster self-addressing

Under Helm the meeting-api Service is release-qualified (<release>-vexa-meeting-api), not the bare meeting-api that the Compose stack uses. So the chart sets MEETING_API_URL explicitly on the meeting-api deployment to http://<release>-vexa-meeting-api:8080 — the address a spawned bot calls back on for its lifecycle callback and recording upload. Left unset it would fall back to the compose-only default http://meeting-api:8080, which does not resolve in-cluster. The chart sets the same address as VEXA_MEETING_API_URL on the agent-api deployment. Before opening a live-transcript SSE stream (GET /api/meeting/stream), agent-api verifies the caller owns the meeting by calling GET /meetings/{id} on meeting-api, and fails closed — an unreachable meeting-api returns 403 "not authorized for this meeting" for the meeting’s own owner. Left unset it would fall back to the compose-only http://meeting-api:8080, so the owner sees a permanent “Reconnecting to live stream…” instead of streaming words. The chart likewise sets ADMIN_API_URL on the meeting-api deployment to http://<release>-vexa-admin-api:8001 (mirroring Compose’s ADMIN_API_URL=http://admin-api:8001). This is required for two background loops: calendar sync (discovering each user’s connected ICS feed through admin-api’s internal edge) and capped auto-join (fetching the per-user max-bots cap for every scheduled-meeting spawn). With ADMIN_API_URL unset, calendar sync no-ops and the auto-join sweep cannot resolve the per-user cap. The meeting-api then fails closed and refuses to spawn rather than spawn uncapped; set AUTO_JOIN_ALLOW_UNCAPPED=1 only if you deliberately want uncapped auto-join spawns on a self-host.

​
The deprecated dashboard (optional)

The 0.10 dashboard — the multi-user web UI that predates the Terminal — ships as an off-by-default component while it is still load-bearing: hosted production runs it against the 0.12 core, and the authenticated-session flows are only walkable through it today. Enable it explicitly:
It runs the pinned external image vexaai/dashboard on its own tag — global.imageTag does not apply to it — and talks to the gateway’s hosted-compat surface. It is deprecated: when the Terminal covers its remaining flows, the component is deleted, not ported. On Compose the same option is docker compose --profile dashboard up -d (port 13001).

​
Honest status

The k8s backend lifecycle and the workspace mount are implemented (the workspace store binds into each worker Pod as a PVC, scoped per-mount). The Compose stack is the path with the most production mileage today; the chart is the right starting point for a cluster evaluation — track the status page.

​
Air-gapped clusters

Everything the chart deploys pulls from images you build and host in your own registry; pair it with the self-hosted transcription unit and your own LLM endpoint (Configuration) for a zero-egress posture. See Security & compliance.