NATS JetStream Integration

:::warning NATS is no longer an internal transport Command notifications moved to the EHDB feed (noetl/ai-meta#212). NATS no longer carries commands or events between the server and workers, and there is no NATS deployment in production.

What NATS still is: a dependency the local kind bootstrap deploys, and the backing service for playbook-facing use — the nats tool kind and NATS-sourced subscription specs. The provisioning and verification sections below remain valid for that. See Architecture for the command bus. :::

This page documents the concrete NATS setup used by NoETL across local kind and GKE deployments, including:

where configuration lives (Helm/manifests/playbooks/code),
which parameters are required,
what environment variables are expected by Gateway, NoETL Server, and NoETL Worker,
how to provision and verify in kind and GKE.

Runtime Contract (historical)

note

This describes the retired internal command path, kept because deployments and fixtures created before the cutover still carry these stream/subject names. The live command bus is the EHDB feed.

NoETL used JetStream for command notifications, while command state stayed in the NoETL API/event model.

The worker still reads these variables (for the user-facing nats paths), with these actual code defaults:

Variable	Code default
`NATS_URL`	empty — deliberately, so a missing URL fails visibly rather than silently against a phantom one
`NATS_STREAM`	`noetl_commands` (lowercase)
`NATS_CONSUMER`	`worker-pool`

The server publishes lightweight notifications:

{
  "execution_id": 123,
  "event_id": 124,
  "command_id": "123:step:124",
  "step": "step",
  "server_url": "http://noetl.noetl.svc.cluster.local:8082"
}

Workers pull notifications, claim command execution through NoETL API, execute tools, and emit resulting events.

Source of Truth: Files and Profiles

Required Parameters

Infrastructure-level (playbook inputs)

Common GKE knobs (from ops playbooks):

nats_user
nats_password
nats_jetstream_file_size
nats_jetstream_mem_size (in some profiles, e.g. Autopilot)

These are used when Helm installs NATS (nats/nats) with JetStream enabled and auth users provisioned.

Expected Environment Variables by Component

NoETL Server

Primary NATS variables (from NATS_* aliases in noetl/core/config.py):

Variable	Required	Default
`NATS_URL`	Yes	`nats://nats.nats.svc.cluster.local:4222`
`NATS_STREAM`	No	`NOETL_COMMANDS`
`NATS_SUBJECT`	No	`noetl.commands`
`NATS_CONSUMER`	No	`noetl_worker_pool`
`NATS_USER`	No	`noetl`
`NATS_PASSWORD`	No	`noetl`

In Helm, these are set in config.server.* in automation/helm/noetl/values.yaml.

NoETL Worker

Worker uses the same NATS_* variables plus worker-specific pull tuning:

Variable	Required	Default
`NATS_URL`	Yes	empty (code default — fails visibly if unset)
`NATS_STREAM`	No	`noetl_commands`
`NATS_SUBJECT`	No	`noetl.commands`
`NATS_CONSUMER`	No	`noetl_worker_pool`
`NOETL_WORKER_NATS_FETCH_TIMEOUT_SECONDS`	No	`30`
`NOETL_WORKER_NATS_FETCH_HEARTBEAT_SECONDS`	No	`5`
`NOETL_WORKER_NATS_MAX_ACK_PENDING`	No	`64`
`NOETL_WORKER_NATS_MAX_DELIVER`	No	`1000`

In Helm, these are set in config.worker.* in automation/helm/noetl/values.yaml.

Gateway

Gateway variables are read from gateway_config.rs.

Variable	Required	Default
`NATS_URL`	Yes	`nats://127.0.0.1:4222`
`NATS_UPDATES_SUBJECT_PREFIX`	No	`playbooks.executions.`
`NATS_CALLBACK_SUBJECT_PREFIX`	No	`noetl.callbacks`
`NATS_SESSION_BUCKET`	No	`sessions`
`NATS_SESSION_CACHE_TTL_SECS`	No	`300`
`NATS_REQUEST_BUCKET`	No	`requests`
`NATS_REQUEST_TTL_SECS`	No	`1800`
`CORS_ALLOWED_ORIGINS`	Required for browser clients	localhost defaults only

In Helm, these map from:

env.natsUrl -> NATS_URL
env.natsUpdatesSubjectPrefix -> NATS_UPDATES_SUBJECT_PREFIX

and CORS from env.corsAllowedOrigins -> CORS_ALLOWED_ORIGINS.

Stream/Consumer Compatibility Rules

NoETL worker consumer uses:

deliver_policy=new
ack_policy=explicit
pull subscription durable consumer

Stream created by NoETL runtime uses retention=limits (not workqueue) with in-memory storage and max_age=3600.

Important:

If stream retention is changed to workqueue, consumer creation can fail with: consumer must be deliver all on workqueue stream.
Keep stream subject aligned with NATS_SUBJECT (default noetl.commands).

Provisioning with Playbooks

All commands below are run from the ops repository root.

kind: minimal path (NATS + NoETL)

# 1) Deploy NATS
noetl run automation/infrastructure/nats.yaml \
  --runtime local \
  --set action=deploy

# 2) Build+load+deploy NoETL into kind
noetl run automation/development/noetl.yaml \
  --runtime local \
  --set action=redeploy \
  --set noetl_repo_dir=../noetl

kind: full environment bootstrap

noetl run automation/setup/bootstrap.yaml --runtime local

Bootstrap includes PostgreSQL, NATS, NoETL, and (optionally) Gateway.

GKE: standard IAP stack

noetl run automation/iap/gcp/deploy_gke_stack.yaml \
  --set project_id=<gcp-project-id> \
  --set deploy_nats=true \
  --set deploy_noetl=true \
  --set deploy_gateway=true

Useful NATS knobs:

--set nats_jetstream_size=5Gi

GKE: fresh stack profile (more control)

noetl run automation/gcp_gke/noetl_gke_fresh_stack.yaml \
  --set action=provision-deploy \
  --set project_id=<gcp-project-id> \
  --set nats_user=noetl \
  --set nats_password=noetl \
  --set nats_jetstream_file_size=5Gi

This profile wires NATS credentials into both NoETL (config.server.NATS_URL, config.worker.NATS_URL) and Gateway (env.natsUrl).

Verification Checklist

# NATS pods
kubectl -n nats get pods

# Stream and consumers
kubectl -n nats exec deploy/nats-box -- \
  nats --server nats://noetl:[email protected]:4222 stream info NOETL_COMMANDS
kubectl -n nats exec deploy/nats-box -- \
  nats --server nats://noetl:[email protected]:4222 consumer ls NOETL_COMMANDS

# NoETL components
kubectl -n noetl get deploy noetl-server noetl-worker

# Gateway env spot-check
kubectl -n gateway get deploy gateway -o yaml | rg \"NATS_URL|NATS_UPDATES_SUBJECT_PREFIX|CORS_ALLOWED_ORIGINS\"

Troubleshooting

Symptom: browser shows CORS Failed to fetch on gateway auth. Check if backend is returning 5xx via proxy/CDN and missing CORS headers on error response.
Symptom: nats: no response from stream in NoETL server. Verify stream exists and subject matches NATS_SUBJECT.
Symptom: worker crash-loop with deliver all on workqueue stream. Recreate NOETL_COMMANDS with retention=limits and restart NoETL server/workers.

Runtime Contract (historical)​

Source of Truth: Files and Profiles​

Helm profiles (Kubernetes)​

kind deployment profiles (manifest based)​

Playbooks used for provisioning​

Runtime behavior in code​

Required Parameters​

Infrastructure-level (playbook inputs)​

Expected Environment Variables by Component​

NoETL Server​

NoETL Worker​

Gateway​

Stream/Consumer Compatibility Rules​

Provisioning with Playbooks​

kind: minimal path (NATS + NoETL)​

kind: full environment bootstrap​

GKE: standard IAP stack​

GKE: fresh stack profile (more control)​

Verification Checklist​

Troubleshooting​