Provisioning NoETL and authoring playbooks with AI agents

A working guide for engineers who want Copilot — or any AI agent — to generate NoETL playbooks that actually run.

:::danger The one thing to know before you start Schema-valid is not the same as runnable. NoETL's JSON Schema is deliberately permissive: it documents the shape rather than constraining it. It will happily validate a playbook that issues zero commands and stalls.

The schema's own text says so — ToolSpec requires only kind (a stricter version "rejected 268 of 279 real playbooks"), and NextDoc states: "It therefore DOCUMENTS the router; it does not CONSTRAIN it."

So an agent that validates against the schema and stops has done half the job. The Before-you-run checklist is the other half. :::

Part 1 — Provision NoETL

Install the CLI

The CLI is the Rust binary from noetl/cli.

brew tap noetl/tap && brew install noetl     # Homebrew
cargo install --bins noetl                   # Cargo (needs a Rust toolchain)
noetl --version

Local mode needs nothing else

Local execution runs in-process. No server, no worker, no PostgreSQL:

noetl run ./playbooks/hello_world.yaml
noetl run ./playbooks/hello_world.yaml -r local -v    # explicit + verbose

Which runtime you get. A three-rung ladder, and the reference type is never consulted:

--runtime / -r on the command line
the active context's runtime, if it pins one
default: local

auto is the sentinel for "not pinned" and falls through to rung 3. Every run prints the resolved runtime and which rung chose it to stderr.

note

noetl exec still works but is a deprecated hidden alias for run and prints a nudge. Use run in anything you generate.

Registering to the catalog (distributed mode)

Local runs read a file directly and need no registration. To run through the server, register first:

noetl catalog register ./playbooks/hello_world.yaml

The REST equivalent is POST /api/catalog/register.

The catalog assigns the version, you do not. Each registration "takes the next version for its path", and the response reports it:

{ "message": "Resource 'path/to/playbook' version '1' registered.", "version": 1 }

Then run it by catalog path — which requires an explicit distributed runtime:

noetl run examples/hello_world -r distributed

Cluster deployment

For a full local cluster (kind + PostgreSQL + workers + observability), see Quick Start. The bootstrap is automation/setup/bootstrap.yaml in noetl/ops — that file is the source of truth for what gets deployed; this guide does not duplicate it.

Part 2 — The playbook schema

Where it lives

A machine-readable JSON Schema is generated from the Rust model and guarded against drift by a test:


File	`schema/playbook.schema.json` in noetl/cli
Dialect	JSON Schema draft 2020-12
Title	`NoETL Playbook`
Generated by	`executor/src/playbook.rs`
Drift guard	`tests/schema_guard.rs`

How to get it — hand this to your agent

noetl schema                                  # print to stdout
noetl schema --output playbook.schema.json    # write to a file

Because it is emitted by the binary you have installed, noetl schema always matches your CLI version. Prefer it over a copy pasted into a prompt.

Top-level shape

apiVersion: noetl.io/v2     # required
kind: Playbook              # required
metadata:                   # required
  name: my_playbook         #   required
  path: examples/my_playbook
workflow:                   # required
  - step: start

Required: apiVersion, kind, metadata, workflow. Also accepted: executor, keychain, workbook, workload.

A Step requires only step (its name). Other keys: tool, next, when, case, loop, set, vars, input, spec, desc.

What the schema will NOT catch

This is the part that matters for generated playbooks:

Not constrained	Consequence
Tool `kind` values — `tool:` requires only that `kind` is present	a typo'd or non-existent kind validates fine
The `next:` router — documented, not constrained	a malformed router validates fine
`metadata.version` — not even defined in the schema	see checklist item 1

Tool kinds — there are three different lists

Docs and agents routinely conflate these. They are genuinely different:

1. What actually executes (the worker/local dispatch registry, noetl-tools/src/tools/mod.rs) — 20 kinds:

shell rhai http duckdb ducklake postgres python snowflake transfer script playbook provider noop task_sequence result_fetch artifact container nats mcp subscription

2. What the server accepts at validation (orchestrate-core/src/playbook.rs::ToolKind) — a larger set; also includes Workbook, Playbooks, Secrets, Iterator, Gcs, Gateway, SnowflakeTransfer, and a WASM plugin variant.

3. Capability tokens for executor.requires.tools (RuntimeCapabilities::local()) — a third, smaller list:

shell http duckdb rhai playbook auth sink

warning

auth and sink are capability tokens only — they are not dispatchable tool kinds. Writing tool: { kind: auth } does not do what it looks like.

Declared features for executor.requires.features: case_v1, case_v2 (Rhai conditions), loop_v1, vars_v1, jinja2. Anything outside these two lists is a hard error that aborts the run, not a warning.

Local execution dispatches through the same registry as the distributed worker, so it is not limited to list 3 — that list only gates executor.requires.

Part 3 — Instructing an AI agent

Prompt snippet

Paste this into a Copilot instructions file, a system prompt, or a CLAUDE.md:

You generate NoETL playbooks (YAML, apiVersion `noetl.io/v2`).

AUTHORITATIVE SCHEMA — read it before writing:
  Run `noetl schema` and follow the JSON Schema it prints.
  It is generated from the Rust model and drift-guarded, so it matches the
  installed CLI exactly.

THE SCHEMA IS PERMISSIVE. It documents shape, it does not constrain
correctness. It will validate a playbook that issues zero commands. After
generating, you MUST apply the checklist below — schema validation alone is
not sufficient.

REQUIRED STRUCTURE
  apiVersion: noetl.io/v2
  kind: Playbook
  metadata: { name, path, version }      # ALWAYS set version — see #1
  workflow: [ ... steps ... ]

CHECKLIST — apply to every playbook before returning it
  1. metadata.version is set.
  2. Every value a `code:` block reads is bound through `tool.input:`.
     `workload` is NOT in scope inside a code block.
  3. No two exclusive arcs target the same join step.
  4. Every terminal branch ends in a step whose tool kind is `noop`.
  5. Each arc `when:` is a single comparison, not a compound expression.
  6. Tool `kind` is one of the dispatchable kinds (run `noetl schema` and check
     the docs list); `auth` and `sink` are NOT tool kinds.

VALIDATE before claiming success:
  noetl run <file> -r local --dry-run
  noetl run <file> -r local -v
Report the output. If `commands_generated` is 0, the playbook is broken —
start at checklist item 1.

Provisioning prompt

Install the NoETL CLI (`brew install noetl` or `cargo install --bins noetl`),
confirm with `noetl --version`, then `noetl schema --output playbook.schema.json`
so the schema is on disk for reference. Do not start a server: `noetl run`
defaults to local, in-process execution.

Part 4 — Worked example

Prompt

Write a NoETL playbook that takes a city from the workload, calls an HTTP endpoint for its weather, and prints a summary with a Python step.

Generated playbook

apiVersion: noetl.io/v2
kind: Playbook
metadata:
  name: weather_summary
  path: examples/weather_summary
  version: "1.0"            # checklist 1

workload:
  city: "Berlin"

workflow:
  - step: start
    next:
      - step: fetch_weather

  - step: fetch_weather
    tool:
      kind: http
      method: GET
      endpoint: "https://api.open-meteo.com/v1/forecast"
      params:
        latitude: 52.52
        longitude: 13.41
        current_weather: true
    next:
      - step: summarize

  - step: summarize
    tool:
      kind: python
      input:                 # checklist 2 — bind everything the code reads
        city: "{{ workload.city }}"
        weather: "{{ fetch_weather.data.current_weather }}"
      code: |
        city = input_data["city"]
        weather = input_data["weather"]
        result = {
            "status": "success",
            "data": {"summary": f"{city}: {weather.get('temperature')}C"},
        }
    next:
      - step: end

  - step: end                # checklist 4 — terminal noop
    tool:
      kind: noop

Note input: on the Python step. Inside a code: block the runtime exposes exactly two names — args and input_data — because the body is wrapped as def __noetl_step__(args, input_data, **kw). workload is not one of them.

Validate, then run

noetl run ./weather_summary.yaml -r local --dry-run   # validate + show the plan
noetl run ./weather_summary.yaml -r local -v          # run it
noetl run ./weather_summary.yaml -r local --set city=Lisbon

For distributed:

noetl catalog register ./weather_summary.yaml
noetl run examples/weather_summary -r distributed

Before-you-run checklist

The failure modes below are silent. Each produces a green-looking run that does nothing useful, and none is caught by schema validation.

1. `metadata.version` — set it, always

Symptom: the execution is accepted, the response reports "commands_generated": 0, and nothing ever runs. No error anywhere.

Why it traps agents: metadata.version is not defined in the JSON Schema, and it is not a typed field on either Metadata model — it is absorbed by #[serde(flatten)] extra. So nothing rejects a playbook without it. An agent that validates against the schema gets a clean pass and ships a playbook that stalls.

metadata:
  name: my_playbook
  path: examples/my_playbook
  version: "1.0"      # <- cheapest line in the file

:::note Status This symptom is recorded from a live debugging session. Setting the field is free and fixes it. The internal mechanism has not been traced to a specific code path — the catalog assigns its own version sequence independently, so the two are not the same thing. Flagged for maintainer confirmation. :::

2. `workload` is not in scope inside `code:`

Symptom: NameError: name 'workload' is not defined.

The body of a code: block is wrapped into def __noetl_step__(args, input_data, **kw) and called as result = __noetl_step__(args, input_data). Only args and input_data exist. Anything the code needs must be bound through tool.input: — which is the standard DSL key (args: is the legacy alias).

# WRONG — NameError at runtime
tool:
  kind: python
  code: |
    result = {"data": workload["city"]}

# RIGHT
tool:
  kind: python
  input:
    city: "{{ workload.city }}"
  code: |
    result = {"data": input_data["city"]}

Symptom: the join step never runs, and is reported SKIPPED rather than failed.

Under mode: exclusive the first matching arc wins and the orchestrator emits step.skipped for every unmatched arc target. A skipped step counts as done for join purposes, so a join reachable from two exclusive alternatives is resolved by whichever arc lost — and never executes.

# WRONG — `merge` is the target of two exclusive alternatives
next:
  spec:
    mode: exclusive
  arcs:
    - step: merge
      when: "{{ x > 10 }}"
    - step: merge
      when: "{{ x <= 10 }}"

# RIGHT — distinct targets that converge afterwards
next:
  spec:
    mode: exclusive
  arcs:
    - step: high_path
      when: "{{ x > 10 }}"
    - step: low_path
      when: "{{ x <= 10 }}"

4. Terminal branches end in `noop`

Every branch that ends should end on a step whose tool kind is noop. A dangling branch with no terminal step leaves the execution without a clean finish.

5. Keep arc conditions to a single comparison

when: on an arc is evaluated per arc. Compound boolean expressions are a frequent source of arcs that silently never match — split them into separate arcs, or use a case: block.

Verification status of this page

Every command, path and field above was checked against origin/main at the time of writing, except where explicitly marked.

Claim	Status
`schema/playbook.schema.json`, generated + guarded by `tests/schema_guard.rs`	VERIFIED
`noetl schema` / `--output`	VERIFIED
Runtime ladder, default `local`; `exec` deprecated	VERIFIED
`noetl catalog register`, `POST /api/catalog/register`, server-assigned versions	VERIFIED
Required top-level keys and `Step` keys	VERIFIED (from the schema)
The three tool-kind lists	VERIFIED
Schema does not constrain tool kinds or the router	VERIFIED (schema's own descriptions)
Trap 2 (`args` / `input_data` only)	VERIFIED
Trap 3 (`step.skipped` for unmatched arcs)	VERIFIED
Trap 1 (`metadata.version` → `commands_generated: 0`)	OBSERVED, mechanism not traced

Part 1 — Provision NoETL​

Install the CLI​

Local mode needs nothing else​

Registering to the catalog (distributed mode)​

Cluster deployment​

Part 2 — The playbook schema​

Where it lives​

How to get it — hand this to your agent​

Top-level shape​

What the schema will NOT catch​

Tool kinds — there are three different lists​

Part 3 — Instructing an AI agent​

Prompt snippet​

Provisioning prompt​

Part 4 — Worked example​

Prompt​

Generated playbook​

Validate, then run​

Before-you-run checklist​

1. metadata.version — set it, always​

2. workload is not in scope inside code:​

3. Exclusive arcs that share a join target​

4. Terminal branches end in noop​

5. Keep arc conditions to a single comparison​

Verification status of this page​

See also​