> ## Documentation Index > Fetch the complete documentation index at: https://docs.vexa.ai/llms.txt > Use this file to discover all available pages before exploring further. # Kubernetes (Helm) > Deploy Vexa on Kubernetes or OpenShift with the Helm chart — a Pod per bot and per agent, scalable to thousands of users. The Helm chart at [`deploy/helm/charts/vexa`](https://github.com/Vexa-ai/vexa/tree/main/deploy/helm) deploys the same control plane as the Compose stack, with the runtime pointed at the cluster: `RUNTIME_BACKEND=k8s` makes **every bot and every agent dispatch its own bare Pod** (`restart: Never`), so capacity is your cluster's scheduler — not a bigger box. ## What the chart deploys | Component | Form | | ------------------------------------------------------- | -------------------------------------------------- | | gateway · admin-api · meeting-api · agent-api · runtime | Deployments (RollingUpdate, `maxUnavailable: 0`) | | terminal (web workbench) | Deployment + ingress | | postgres | StatefulSet (or point values at your managed DB) | | redis (Valkey) · minio | Deployments (+ minio init Job) | | DB migrations | Job | | pgbouncer | optional connection pooling | | dashboard (deprecated 0.10 UI) | optional Deployment + Service, **off by default** | | RBAC for the k8s spawn backend | Role/RoleBinding scoped to Pod create/watch/delete | Hardened defaults throughout: containers run non-root with all capabilities dropped, PodDisruptionBudgets on the control plane, and rolling updates that never take the last replica down. ## Install ```bash theme={null} git clone https://github.com/Vexa-ai/vexa.git && cd vexa helm install vexa deploy/helm/charts/vexa -f deploy/helm/charts/vexa/values.yaml ``` `values-staging.yaml` and `values-test.yaml` show environment overlays. Chart tests (`helm lint` + template tests) live under `deploy/helm/tests/`. ### Is this install actually working? — `make probe SURFACE=helm` The standing full-journey smoke probe drives spawn → schedule → boot → join → transcribe → live-view → stop through the gateway front door, then sweeps every deployment's logs once. Each stage prints Expected / Actual / Verdict; a red stage names where the journey broke and fails the command, so you get a truthful install verdict in minutes with no meeting and no humans. ```bash theme={null} make probe SURFACE=helm # port-forwards gateway + admin-api GATEWAY_URL=http://: make probe SURFACE=helm # or drive a NodePort directly ``` It mints its API key from the release secret's `ADMIN_API_TOKEN` (or pass `VEXA_API_KEY`). The same journey runs on the other surfaces: `make probe` (compose, the fast default) and `make probe SURFACE=lite`. ## The runtime on Kubernetes The runtime kernel owns one `runtime.v1` lifecycle (`starting → running → stopping → stopped → destroyed`) and delegates only the substrate question — *how do I start, observe, and stop a workload?* — to the backend. On `k8s`, a workload is a bare Pod; the same dispatch, contracts, and worker run identically to the Docker backend. See [Execution](/architecture/execution). ### Scheduling spawned Pods on a tainted / dedicated pool A spawned bot or agent Pod is created with `kubectl run` — it is **not** a child of the runtime Deployment, so it does **not** inherit `global.nodeSelector` / `global.tolerations`. On a cluster whose nodes are all tainted (the standard pool-segregation pattern — dedicated, spot/temp, or GPU node pools), a spawned Pod with no tolerations sits `Pending` forever (`untolerated taint(s)`) and the meeting silently fails while the runtime itself — a Deployment that *does* apply those constraints — looks healthy. The chart closes this by handing the runtime its **own** scheduling constraints as env, which the spawn backend stamps onto every Pod it creates: | Env on the runtime | Chart source | Applied to | | --------------------------- | ---------------------------------------------------------- | --------------------------------------- | | `RUNTIME_K8S_TOLERATIONS` | `runtime.tolerations` ⟶ defaults to `global.tolerations` | every spawned Pod's `spec.tolerations` | | `RUNTIME_K8S_NODE_SELECTOR` | `runtime.nodeSelector` ⟶ defaults to `global.nodeSelector` | every spawned Pod's `spec.nodeSelector` | By default a spawned Pod schedules **wherever the runtime itself can** — set `global.tolerations` / `global.nodeSelector` for your pool and both the runtime and everything it spawns land there, with no extra action: ```yaml theme={null} global: tolerations: - key: vexa.ai/pool operator: Equal value: main effect: NoSchedule nodeSelector: vexa.ai/pool: main ``` Set `runtime.tolerations` / `runtime.nodeSelector` only to send spawned workloads to a **different** pool than the kernel (e.g. GPU bots). Both values are serialized to JSON; malformed JSON fails the spawn loudly rather than dropping the constraint (a dropped constraint is the stranded-Pod bug). **Spawned-workload commands must exist in the target image under k8s (entrypoint-replace) semantics.** A runtime profile may carry a `command`; the k8s backend passes it as `kubectl run --command -- …`, which **replaces** the image `ENTRYPOINT` (it becomes the Pod's `argv[0]`). So a command path that is not present in the image `StartError`s **every** spawn on k8s — even though the Docker backend, which *appends* the command to the entrypoint as arguments, silently ignores the same bad path. The meeting-bot profile therefore carries **no** command: the bot image's own `ENTRYPOINT` (`/app/entrypoint.sh`) boots it on every backend. At release time an image↔profile conformance gate (`image-identity`) asserts each profile command is either empty or an in-image executable against the published images, so a profile that diverges from its image fails the release rather than a customer's install. ### Autoscaling and bot bursts — the fresh-node reachability gate Because every bot is its own bare Pod, a burst of meeting requests can push the cluster autoscaler to add nodes, and bots get scheduled onto them the moment they report `Ready`. But a brand-new node's network can take up to a few minutes to fully converge *after* `Ready` — the CNI/NetworkPolicy programming, kube-proxy, and DNS warmup all lag node readiness. A bot scheduled into that window can find its control plane (its meeting-api callback URL and redis) unreachable on its first outbound hop. The bot handles this with a **pre-join reachability gate**: it makes its first `joining` lifecycle emit load-bearing. If the meeting-api callback is reachable (any HTTP response), it joins immediately — zero added latency. If not, it probes redis; **if either channel is up it proceeds** (it can still report), and **only if both are down does it refuse to join**, terminating fast (\< a few seconds) with: * **Pod exit code `3`** — the attributable terminal signal (`kubectl describe pod ` → `Last State: Terminated, Exit Code: 3`). Distinct from a real join failure (exit `1`). * a `failed` lifecycle event carrying `failure_stage: requested` and `infra_fault: control_plane_unreachable` on whichever channel recovers (often none — the exit code is the signal). This converts an opaque CrashLoop / stuck-`requested` meeting into a fast, attributed failure an operator can read in one `kubectl describe`. See [Troubleshooting → exit code 3](/troubleshooting). **What stays cluster-side (not solved by the gate):** the node-readiness window itself. Consider pre-pulling the bot image onto new nodes (a cold pull can itself take minutes and mask the network window), and treat node `Ready` as insufficient for scheduling network-dependent work — the gate fails fast and, on k8s, the exit lets the operator (or an autoscaler policy) reschedule onto a warmed node. ## Scaling meeting-api `meetingApi.replicaCount` defaults to `2`, and that is safe for the segment consumer. Each replica now joins the `transcription_segments` consumer group under a **per-pod identity** (`collector-`, overridable with `COLLECTOR_CONSUMER_NAME`), and a surviving replica periodically `XAUTOCLAIM`s a crashed replica's un-acked segment batch and drains it through the normal persist path — so no transcript segments are orphaned when a pod dies mid-batch. The reclaim only fires once a batch has idled past `COLLECTOR_RECLAIM_MIN_IDLE_MS` (default `60000`), so a live peer's in-flight batch is never stolen. `/health` exposes `pipeline.pending_depth` (the group's delivered-but-un-acked count) alongside `pipeline.consumer_lag`, and degrades to `503` once it exceeds `PIPELINE_PENDING_ALARM` (default `100`) — a stuck batch is a reportable state, not a silent stall. A pod's hostname changes on **every recreate** (a rolling deploy, a pod restart, a compose `--force-recreate`), so each new container joins `collector_group` under a **new** `collector-` and the old name is left behind. Those abandoned consumers hold no pending entries (they read their last message and acked it), so they are harmless to correctness — but left unchecked the group fills with dead names that inflate operator `XINFO CONSUMERS` reads and muddy `/health`. The same reclaim sweep now **prunes** them: any consumer with `pending == 0` that has idled past `COLLECTOR_CONSUMER_TTL_MS` (default `1800000`, 30 min — well above the reclaim idle gate, so a briefly-quiet live replica is never touched) is removed with `XGROUP DELCONSUMER`. A consumer holding an in-flight batch (`pending > 0`) is **never** pruned, and the running replica never prunes itself; a live consumer re-registers on its next `XREADGROUP`, so the prune is idempotent and self-healing. Reading `XINFO CONSUMERS collector_group` should therefore list only the replicas that actually exist — dead `collector-` names are expected to disappear within a sweep of the TTL. For k8s you can eliminate the churn at the source by pinning a stable per-replica identity via `COLLECTOR_CONSUMER_NAME` (e.g. a StatefulSet ordinal) so a restart re-uses the same consumer. **Every surface now ships Valkey 8.x**, which has `XAUTOCLAIM` — so orphan reclaim is active everywhere Vexa ships, including Vexa Lite (compose, helm, and Lite all run Valkey, the Linux Foundation BSD-3 fork of Redis 7.2.4; see [the changelog](/changelog) and #653). Reclaim, per-pod identity, and ghost-consumer pruning all work on the shipped stack. Check yours with `valkey-server --version`. The degradation path below still exists as a safety net for a *bring-your-own* backing store older than the floor: on a Redis/Valkey without `XAUTOCLAIM` the reclaim disables itself and logs once at startup; everything else (normal `XREADGROUP` consumption, `pending_depth`) is unaffected. The cost is narrow but real — if a replica dies mid-batch on such a store, that batch stays pending instead of being drained by a peer, so run a single replica or move to a store with `XAUTOCLAIM` (Redis ≥ 6.2 / Valkey ≥ 7.2). Ghost-consumer pruning degrades the same way: if the backing store rejects `XINFO CONSUMERS`, the prune logs once and no-ops, leaving the consume path untouched. ## In-cluster self-addressing Under Helm the meeting-api Service is release-qualified (`-vexa-meeting-api`), not the bare `meeting-api` that the Compose stack uses. So the chart sets `MEETING_API_URL` explicitly on the meeting-api deployment to `http://-vexa-meeting-api:8080` — the address a spawned bot calls back on for its lifecycle callback and recording upload. Left unset it would fall back to the compose-only default `http://meeting-api:8080`, which does not resolve in-cluster. The chart sets the same address as `VEXA_MEETING_API_URL` on the **agent-api** deployment. Before opening a live-transcript SSE stream (`GET /api/meeting/stream`), agent-api verifies the caller owns the meeting by calling `GET /meetings/{id}` on meeting-api, and **fails closed** — an unreachable meeting-api returns `403 "not authorized for this meeting"` for the meeting's own owner. Left unset it would fall back to the compose-only `http://meeting-api:8080`, so the owner sees a permanent "Reconnecting to live stream…" instead of streaming words. The chart likewise sets `ADMIN_API_URL` on the meeting-api deployment to `http://-vexa-admin-api:8001` (mirroring Compose's `ADMIN_API_URL=http://admin-api:8001`). This is **required** for two background loops: calendar sync (discovering each user's connected ICS feed through admin-api's internal edge) and capped auto-join (fetching the per-user max-bots cap for every scheduled-meeting spawn). With `ADMIN_API_URL` unset, calendar sync no-ops and the auto-join sweep cannot resolve the per-user cap. The meeting-api then **fails closed and refuses to spawn** rather than spawn uncapped; set `AUTO_JOIN_ALLOW_UNCAPPED=1` only if you deliberately want uncapped auto-join spawns on a self-host. ## The deprecated dashboard (optional) The 0.10 dashboard — the multi-user web UI that predates the Terminal — ships as an **off-by-default** component while it is still load-bearing: hosted production runs it against the 0.12 core, and the authenticated-session flows are only walkable through it today. Enable it explicitly: ```bash theme={null} helm upgrade --install vexa deploy/helm/charts/vexa -n vexa \ --set dashboard.enabled=true \ --set dashboard.publicUrl=https://dashboard.example.com \ --set dashboard.publicApiUrl=https://api.example.com ``` It runs the pinned external image `vexaai/dashboard` on its own tag — `global.imageTag` does not apply to it — and talks to the gateway's hosted-compat surface. It is **deprecated**: when the Terminal covers its remaining flows, the component is deleted, not ported. On Compose the same option is `docker compose --profile dashboard up -d` (port 13001). ## Honest status The k8s backend **lifecycle and the workspace mount are implemented** (the workspace store binds into each worker Pod as a PVC, scoped per-mount). The Compose stack is the path with the most production mileage today; the chart is the right starting point for a cluster evaluation — track the [status page](/roadmap/status). ## Air-gapped clusters Everything the chart deploys pulls from images you build and host in your own registry; pair it with the [self-hosted transcription unit](/deployment#transcription-the-separate-gpu-unit) and your own LLM endpoint ([Configuration](/configuration)) for a zero-egress posture. See [Security & compliance](/security-compliance).