Skip to main content

Quick Start

Get NoETL running quickly with the maintained Rust CLI or a full local development environment.

Option 1: Install The NoETL CLI​

The NoETL CLI is the Rust binary maintained in https://github.com/noetl/cli.

Homebrew​

brew tap noetl/tap
brew install noetl
noetl --version

APT​

echo 'deb [trusted=yes] https://noetl.github.io/apt jammy main' | sudo tee /etc/apt/sources.list.d/noetl.list
sudo apt-get update
sudo apt-get install noetl
noetl --version

Cargo​

cargo install --bins noetl
noetl --version

Prerequisites:

CLI Capabilities:

  • noetl server start/stop - Manage NoETL server
  • noetl worker start/stop - Manage workers
  • noetl db init/validate - Database management
  • noetl build - Build container images
  • noetl k8s deploy/redeploy/reset - Kubernetes deployment
  • noetl register playbook - Register playbooks
  • noetl run <reference> - Execute playbooks (local or distributed)
  • noetl context - Manage execution contexts

Option 2: Local Development Environment​

For a complete environment with server, workers, PostgreSQL, and observability stack:

# Clone repository
git clone https://github.com/noetl/noetl.git
cd noetl

# Install NoETL CLI from the separate CLI repository when developing locally
git clone https://github.com/noetl/cli.git ../cli
cargo install --path ../cli

# (Optional) Install dev tools - auto-detects macOS vs Linux/WSL2
noetl run automation/development/setup_tooling.yaml --set action=install-devtools

# Bootstrap entire environment (one command)
noetl run boot

Prerequisites Setup​

NoETL provides OS-aware tooling playbooks that automatically install required development tools:

# Detect your OS and see recommended setup
noetl run automation/development/setup_tooling.yaml --set action=detect

# Install all development tools (auto-detects OS)
noetl run automation/development/setup_tooling.yaml --set action=install-devtools

# Or use platform-specific playbooks:
# macOS (uses Homebrew)
noetl run automation/development/tooling_macos.yaml --set action=install-devtools

# Linux/WSL2 (uses apt-get)
noetl run automation/development/tooling_linux.yaml --set action=install-devtools

Tools installed: git, curl, jq, podman, kind, kubectl, helm, pyenv, uv, tfenv

What noetl run boot does:

  1. Validates prerequisites (podman, kind, kubectl, helm, python3, uv)
  2. Checks for existing Rust CLI binary (target/release/noetl) - builds only if missing
  3. Builds NoETL container images
  4. Creates Kind Kubernetes cluster with pre-configured NodePort mappings
  5. Deploys PostgreSQL database
  6. Deploys NoETL server (control plane) and workers (data plane)
  7. Initializes database schema
  8. Deploys NATS JetStream, Qdrant, ClickHouse, and an S3-compatible object store
  9. Deploys the monitoring stack (VictoriaMetrics, Grafana)
  10. Provisions the Gateway auth schema and a pagination test server for integration tests

Bootstrap options:

# Force rebuild Rust CLI (even if binary exists)
noetl run boot --set build_rust_cli=true

# Use minimal kind config (fewer port mappings)
noetl run boot --set kind_config=ci/kind/config-minimal.yaml

Services available after boot:

:::note Host ports assume the default kind config These are host ports under ci/kind/config.yaml, which is what noetl run boot uses by default. The alternative ci/kind/config-minimal.yaml maps several ports identically to their nodePort (NATS client and monitoring stay on 30422 / 30822), so if you booted with --set kind_config=…-minimal.yaml, use the nodePort number instead. :::

ServiceURLCredentials
NoETL Serverhttp://localhost:8082-
PostgreSQLlocalhost:54321demo/demo
ClickHouse HTTPhttp://localhost:30123-
Qdrant HTTPhttp://localhost:30633-
NATS Monitoringhttp://localhost:32822-

Your First Playbook​

1. Create a playbook file​

Create hello_world.yaml:

apiVersion: noetl.io/v2
kind: Playbook
metadata:
  name: hello_world
  path: examples/hello_world
workload:
  message: "Hello from NoETL!"
workflow:
  - step: start
    next:
      - step: greet

  - step: greet
    tool:
      kind: python
      libs: {}
      args:
        message: "{{ workload.message }}"
      code: |
        result = {"status": "success", "data": {"greeting": message}}
    next:
      - step: end

  - step: end

2. Register the playbook​

noetl register playbook hello_world.yaml --host localhost --port 8082

3. Execute the playbook​

# Local execution (file path)
noetl run hello_world.yaml -v

# Distributed execution (catalog path - after registration)
noetl run examples/hello_world -r distributed

4. Check the result​

# List executions
curl http://localhost:8082/api/executions | jq

# Get execution details
curl http://localhost:8082/api/executions/1 | jq

Development Workflow​

Common development commands:

# Rebuild and redeploy after code changes
noetl k8s redeploy

# Reset database and redeploy (full reset)
noetl k8s reset

# Re-bootstrap entire environment
noetl run destroy && noetl run boot

# Check deployment status
kubectl get pods -n noetl

# View server logs
kubectl logs -n noetl -l app=noetl-server --tail=100

# View worker logs
kubectl logs -n noetl -l app=noetl-worker --tail=100

Register Test Fixtures​

NoETL includes example playbooks in tests/fixtures/playbooks/. First, set up the context for distributed mode:

# Setup context for Kind cluster with distributed runtime
noetl context add kind-cluster --server-url http://localhost:8082
noetl context use kind-cluster
noetl context set-runtime distributed

The CLI supports recursive directory scanning:

# Register all playbooks from a directory (recursive)
noetl register playbook tests/fixtures/playbooks/ --host localhost --port 8082

# Register all credentials from a directory (recursive)
noetl register credential tests/fixtures/credentials/ --host localhost --port 8082

# Register a single playbook
noetl register playbook tests/fixtures/playbooks/hello_world/hello_world.yaml --host localhost --port 8082

# Register a single credential
noetl register credential tests/fixtures/credentials/pg_k8s.yaml --host localhost --port 8082

Cleanup​

# Destroy entire environment (cluster, deployments, resources)
noetl run destroy

Next Steps​