6
1
Fork 0
mirror of https://github.com/systeminit/swamp.git synced 2026-05-20 19:25:15 +00:00
Stinemates, Watson, Adam, Mahir, Paul
  • TypeScript 99.6%
  • Python 0.2%
  • Shell 0.1%
Find a file
keeb 02eee3f8bf
fix(extensions): prune empty per-extension scaffold dirs on rm (swamp-club#383) (#1412)
## Summary

Closes swamp-club#383.

`pull` unconditionally `Deno.mkdir`s the seven per-extension kind dirs
(`models`, `workflows`, `vaults`, `drivers`, `datastores`, `reports`,
`files`) regardless of whether the extension ships content for that
kind. These scaffolds are never recorded in the lockfile's tracked-file
list. `RemoveExtensionService`'s `pruneEmptyDirs` walks UP from
tracked-file parents and stops at the first non-empty dir — at the
extension root it sees the untracked scaffolds as entries and bails,
leaving the entire per-extension subtree on disk after `rm`.

Fix: push the seven scaffold paths into `parentDirs` before pruning so
the existing walk sweeps them. A new `PER_EXTENSION_SCAFFOLD_DIRS`
constant in `layout.ts` keeps the kind list authoritative (sibling to
`PULLED_TYPE_DIRS`).

## Backwards compatibility

- Legacy gen-1 / gen-2 installs (no per-extension subtree) are
unaffected — the pushed paths don't exist on disk and `pruneEmptyDirs`'s
existing `ENOENT` catch absorbs the `Deno.readDir` failure.
- Pre-existing repos with already-leaked scaffold dirs: the next `rm`
cleans them up. No migration required.

## Follow-ups (out of scope)

- swamp-club#392 — sibling leak for `.swamp/<kind>-bundles/<hash>/`
namespace dirs (filed during verification of this fix). Same shape of
bug at the repo-root level.
- Pull-side lazy-`mkdir` cleanup in `pull.ts:866-960` (deferred).
Removes the redundant scaffold mkdirs so new installs don't create empty
dirs in the first place.

## Test plan

- [x] Three pinning tests added to `remove_extension_service_test.ts`:
canonical empty-scaffold cleanup, sibling-extension preservation under a
shared `@scope/`, and flat (non-scoped) extension name handling.
- [x] All three tests FAIL on `main` (confirmed via `git stash`) and
PASS with the fix.
- [x] Full suite: 6016 passed, 0 failed.
- [x] `deno check`, `deno lint`, `deno fmt` clean.
- [x] End-to-end re-verification: `pull @alvagante/docker-image-test`
then `rm --force` leaves no per-extension artifacts on disk (12 dirs
pruned vs 4 pre-fix).

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 17:51:39 +01:00
.agents/skills feat: warn when model definitions use env var expressions (#784) 2026-03-19 23:05:48 +00:00
.claude fix(cli): replace stdin auto-detect with explicit --stdin flag (#1407) 2026-05-19 23:26:04 +01:00
.github fix(release): install pinned canary Deno via direct download (#1309) 2026-05-05 17:36:37 +01:00
agent-constraints chore(agent-constraints): add state-unwind heuristic to Risk dimension (#1217) 2026-04-24 01:09:45 +01:00
design feat(cli): add --extensions-dir for git worktree support (#1410) 2026-05-20 15:59:50 +01:00
evals/promptfoo refactor(skills): consolidate 16 swamp-* skills into 11 (#1364) 2026-05-12 00:28:50 +01:00
extensions/models feat(domain): migrate default swamp-club URL to swamp-club.com (#1228) 2026-04-27 12:37:07 +01:00
integration feat(models): expose cel-js Environment to extensions via ctx.createCelEnvironment() (#1406) 2026-05-19 23:21:04 +01:00
packages feat(models): expose cel-js Environment to extensions via ctx.createCelEnvironment() (#1406) 2026-05-19 23:21:04 +01:00
scripts fix(release): pin host Deno to bundled canary so denort matches (#1308) 2026-05-05 17:13:28 +01:00
src fix(extensions): prune empty per-extension scaffold dirs on rm (swamp-club#383) (#1412) 2026-05-20 17:51:39 +01:00
.gitattributes ci: add .gitattributes to pin line endings to LF (#1234) 2026-04-28 15:14:54 +01:00
.gitignore feat(extensions): namespace pulled extensions by scoped name (per-extension layout) (#1186) 2026-04-17 19:58:26 +01:00
banner.png docs: Refresh README for open alpha (#307) 2026-02-12 23:51:15 +00:00
BUG_BOUNTY.md feat(domain): migrate default swamp-club URL to swamp-club.com (#1228) 2026-04-27 12:37:07 +01:00
CLAUDE.md docs: strengthen skill-creator prerequisite in CLAUDE.md (#1341) 2026-05-08 00:29:56 +01:00
CODE_OF_CONDUCT.md fix: Update Discord invite link to swamp-club (#356) 2026-02-16 16:56:56 +00:00
CONTRIBUTING.md Update CONTRIBUTING.md (#294) 2026-02-12 18:32:25 +00:00
COPYING license: Add AGPLv3 copyright headers to all TypeScript files (#287) 2026-02-11 23:55:22 +00:00
COPYING-EXCEPTION Update COPYING-EXCEPTION (#408) 2026-02-19 22:16:59 +00:00
COPYRIGHT license: Add AGPLv3 copyright headers to all TypeScript files (#287) 2026-02-11 23:55:22 +00:00
deno.json chore(deps): bump cel-js 7.5.1 → 7.6.1 (#1318) 2026-05-05 21:25:10 +01:00
deno.lock feat(models): expose cel-js Environment to extensions via ctx.createCelEnvironment() (#1406) 2026-05-19 23:21:04 +01:00
Dockerfile fix: pin Deno Docker image and pass release version via job outputs (#883) 2026-03-26 18:32:28 +00:00
FILE-LICENSE-TEMPLATE.md license: Add AGPLv3 copyright headers to all TypeScript files (#287) 2026-02-11 23:55:22 +00:00
LICENSE license: Add AGPLv3 copyright headers to all TypeScript files (#287) 2026-02-11 23:55:22 +00:00
logo.png docs: add README with project logo (#1) 2026-01-28 10:40:57 -07:00
main.ts fix(tracing): wire up W3C trace context propagation (swamp-club#317) (#1358) 2026-05-11 14:29:53 +01:00
main_test.ts license: Add AGPLv3 copyright headers to all TypeScript files (#287) 2026-02-11 23:55:22 +00:00
README.md chore: Remove references to github issue filing (#1232) 2026-04-28 14:50:43 +01:00
SECURITY.md chore: Remove references to github issue filing (#1232) 2026-04-28 14:50:43 +01:00
TRADEMARKS.md feat(domain): migrate default swamp-club URL to swamp-club.com (#1228) 2026-04-27 12:37:07 +01:00

Swamp — AI Automation for Hackers

Swamp

AI Native Automation, built for agents - Welcome to Swamp.

Swamp is a CLI that supercharges AI agents to create operational workflows that are reviewable, shareable, and accurate. Built for agents, there to empower humans. All the data lives in the .swamp/ (the swamp).

Come join the swamp party on discord.

Getting Started

curl -fsSL https://swamp-club.com/install.sh | sh

Quick Start

swamp repo init                    # Claude Code (default)
swamp repo init --tool cursor      # Cursor
swamp repo init --tool opencode    # OpenCode
swamp repo init --tool codex       # Codex

Start your AI agent in the repo and tell it what you want to do. Just ask:

  • "Manage my EC2 fleet — inventory every instance across all regions and flag anything without a cost-center tag"
  • "Set up a workflow to check my bare metal Minecraft servers are online and under 80% memory"
  • "Audit our DNS records and compare them against what's actually running"
  • "Build a workflow that rotates database credentials and stores them in the vault"

The agent will create models, wire up workflows, and run them — all reviewable in .swamp/ before anything touches production.

Core Concepts

  • Models — Typed representations of external systems (cloud resources, CLI tools, APIs). Each model type defines metadata, arguments, methods, and inputs.
  • Definitions — YAML files that instantiate a model type with specific configuration. Support CEL expressions for dynamic values and cross-model references.
  • Workflows — Orchestrate model method executions across parallel jobs and steps, with dependency ordering and trigger conditions.
  • Data — Versioned, immutable artifacts (resources, logs, files) produced by method runs. Searchable by tags.
  • Vaults — Secure storage for secrets and credentials, referenced in definitions via CEL expressions.
  • Tags — Key-value labels on definitions, workflows, and data. Flow from definitions to produced data, overridable at runtime with --tag.

Everything lives in a .swamp/ directory inside a Git repository, with human-friendly symlink views under /models/ and /workflows/.

Local Execution

Swamp runs entirely on your machine. It picks up the environment variables from the shell you run it in — your AWS credentials, SSH keys, kubeconfig, whatever the task needs. No credentials leave your laptop unless a model explicitly calls an external API.


Update

Swamp can update itself:

swamp update

Shell Completions

Tab-completion for commands, model names, and workflow names:

# Bash — add to ~/.bashrc
eval "$(swamp completions bash)"

# Zsh with oh-my-zsh
mkdir -p ~/.oh-my-zsh/completions
swamp completions zsh > ~/.oh-my-zsh/completions/_swamp
rm -f ~/.zcompdump* && exec zsh

# Zsh without oh-my-zsh — add to ~/.zshrc
eval "$(swamp completions zsh)"

# Fish
swamp completions fish > ~/.config/fish/completions/swamp.fish

Completions are directory-dependent — they return names from the current directory's swamp repository.

Using Swamp with AI Agents

Swamp ships first-class skills for four AI coding tools:

Tool Init flag Skills dir Instructions file
Claude Code (default) .claude/skills/ CLAUDE.md
Cursor --tool cursor .cursor/skills/ .cursor/rules/swamp.mdc
OpenCode --tool opencode .agents/skills/ AGENTS.md
Codex --tool codex .agents/skills/ AGENTS.md

The skills are bundled into the swamp binary and written into the appropriate directory so your agent discovers them automatically. Each skill teaches the agent how to work with swamp — search for models, create definitions, run workflows, manage vaults, and more.

You can switch tools later or run multiple tools side-by-side — each tool's skills directory is independent and gitignored:

swamp repo upgrade --tool cursor

User-Defined Models

Extend Swamp with custom TypeScript models. Place them in extensions/models/ (or configure via SWAMP_MODELS_DIR or .swamp.yaml).

See the extension model skill in your skills directory (e.g. .claude/skills/swamp-extension-model/SKILL.md for Claude Code) for details.

Developer Guide

Prerequisites

Commands

deno run dev          # Run the CLI from source
deno run test         # Run the test suite
deno check            # Type-check
deno lint             # Lint
deno fmt              # Format
deno run compile      # Compile the binary

Contributing

Swamp uses an issue-driven contribution model. We don't accept pull requests from external contributors — fork PRs are automatically closed. This isn't about gatekeeping; it's about supply chain security in the age of AI-generated code. When AI agents can produce large, plausible-looking changes, the only way to maintain quality and security is to tightly control the inputs to the development process.

Here's how it works:

  1. You file an issuebug reports and feature requests are very welcome. Be as detailed as you like.
  2. We triage it — A maintainer triages the issue locally using Claude, confirms bugs by tracing through the codebase, and generates a detailed implementation plan. Plans are revised interactively until the approach is solid.
  3. We build it — System Initiative engineers (with AI agents under our direct control) implement the plan, with full test coverage and code review.
  4. You get credit — We're happy to include you as a co-author on any PR generated from your request.

This means you get the feature you asked for, maintained over time, without having to worry about keeping a fork in sync. See CONTRIBUTING.md for the full details.

Datastores

By default, swamp stores all runtime data (model data, workflow runs, outputs, audit logs, etc.) in the local .swamp/ directory. You can configure a different datastore backend to share state across machines or centralise data.

Default (local filesystem)

When you run swamp repo init, the datastore is .swamp/ inside the repo. No extra configuration needed.

Setting up an external filesystem datastore

Move runtime data to a directory outside the repo (e.g. a shared NFS mount):

swamp datastore setup filesystem --path /mnt/shared/swamp-data

This migrates existing .swamp/ runtime data to the new path and updates .swamp.yaml. A file-based lock prevents concurrent access from multiple processes.

Setting up an S3 datastore

Store runtime data in S3 for team collaboration using the @swamp/s3-datastore extension:

swamp datastore setup extension @swamp/s3-datastore \
  --config '{"bucket":"my-swamp-bucket","prefix":"my-project","region":"us-east-1"}'

This pushes existing local data to S3 and updates .swamp.yaml. Subsequent commands automatically pull changes before execution and push changes after. A distributed lock (S3 conditional writes) prevents concurrent access.

Use --skip-migration on either setup command to skip the initial data migration.

Migrating between datastores

Run swamp datastore setup again with the new backend. For example, to move from a filesystem datastore to S3:

swamp datastore setup extension @swamp/s3-datastore \
  --config '{"bucket":"my-bucket","region":"us-east-1"}'

Or from S3 back to local filesystem:

swamp datastore setup filesystem --path /path/to/data

Each setup command migrates existing data to the new backend.

Checking datastore status

swamp datastore status        # Shows type, health, and config
swamp datastore sync          # Manual bidirectional sync (S3 only)
swamp datastore sync --pull   # Pull-only from S3
swamp datastore sync --push   # Push-only to S3

Stuck locks

If a process crashes without releasing the datastore lock, subsequent commands will wait up to 60 seconds before timing out (locks auto-expire after 30 seconds). To inspect or force-release a stuck lock:

swamp datastore lock status           # Show who holds the lock
swamp datastore lock release --force  # Force-release the lock

Environment variable override

For CI/CD, override the datastore without modifying .swamp.yaml:

export SWAMP_DATASTORE=s3:my-bucket/my-prefix
export SWAMP_DATASTORE=filesystem:/tmp/swamp-data

Repository Directory

Every command runs against a swamp repository. By default this is the current working directory. Pass --repo-dir to point at a different repo per invocation, or set SWAMP_REPO_DIR to persist the override across a shell session or CI job:

swamp model search --repo-dir /path/to/repo
export SWAMP_REPO_DIR=/path/to/repo
swamp model search

Priority order (highest to lowest): --repo-dir flag → SWAMP_REPO_DIR env var → current working directory.

Log Level

By default, swamp outputs at the info level. You can change this once rather than repeating a flag on every command.

Per-invocation flags (highest priority):

swamp -q workflow run my-workflow               # error level only
swamp --log-level debug workflow run my-workflow

Via environment variable (useful for CI/CD):

export SWAMP_LOG_LEVEL=warning
swamp workflow run my-workflow

Permanently for a repository — add to .swamp.yaml:

logLevel: error

Valid levels: trace, debug, info, warning, error, fatal.

Priority order (highest to lowest): -q / --log-level flag → SWAMP_LOG_LEVEL env var → .swamp.yaml logLevel → default (info).

Tracing

Swamp has native OpenTelemetry tracing for diagnosing slow or failing operations. Tracing is opt-in and has zero overhead when disabled.

Set OTEL_EXPORTER_OTLP_ENDPOINT to enable:

# Send traces to a local Jaeger instance
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 swamp workflow run my-workflow

# Quick debug: print spans to stderr (no collector needed)
OTEL_TRACES_EXPORTER=console swamp workflow run my-workflow

Traces capture the full execution hierarchy — CLI command, workflow, job, step, model method, and driver execution — with automatic context propagation to in-process extensions and Docker containers via TRACEPARENT.

Local development

Run Jaeger for a local trace UI:

docker run -d --name jaeger -p 16686:16686 -p 4318:4318 jaegertracing/all-in-one:latest

Then open http://localhost:16686 and search for the swamp service.

Configuration

Variable Purpose Default
OTEL_EXPORTER_OTLP_ENDPOINT Collector URL (tracing off when unset) (unset = off)
OTEL_TRACES_EXPORTER otlp, console, or none otlp
OTEL_SERVICE_NAME Service name in traces swamp
OTEL_EXPORTER_OTLP_HEADERS Auth headers (key=val,key=val) (none)

Telemetry

Swamp collects anonymous usage telemetry to help us understand which commands are used, how long they take, and what errors occur. All user-identifiable values are redacted before transmission — nothing sensitive is ever sent.

Here is a complete example of a telemetry event:

{
  "event": "cli_invocation",
  "distinct_id": "a3f1b2c4-5678-9abc-def0-1234567890ab",
  "properties": {
    "id": "550e8400-e29b-41d4-a716-446655440000",
    "invocation": {
      "command": "model",
      "subcommand": "create",
      "args": ["prompt", "<REDACTED>"],
      "optionKeys": ["--force"],
      "globalOptions": ["--json"]
    },
    "result": {
      "status": "success",
      "exitCode": 0
    },
    "startedAt": "2026-02-16T10:00:00.000Z",
    "completedAt": "2026-02-16T10:00:01.234Z",
    "durationMs": 1234,
    "swampVersion": "0.13.0",
    "denoVersion": "2.1.0",
    "platform": "linux"
  }
}

Note that positional arguments containing user data (model names, file paths, queries) are replaced with <REDACTED>. Only categorical values defined by swamp itself (like model types) are recorded. Option values are never recorded — only the option keys.

Disabling Telemetry

Per-invocation:

swamp --no-telemetry workflow run my-workflow

Via environment variable (useful for CI/UAT environments):

export SWAMP_NO_TELEMETRY=1
swamp workflow run my-workflow

Permanently for a repository — add to .swamp.yaml:

telemetryDisabled: true

Priority order (highest to lowest): --no-telemetry flag → SWAMP_NO_TELEMETRY env var → .swamp.yaml telemetryDisabled: true.

License

Swamp is licensed under the GNU Affero General Public License v3.0 with the Swamp Extension and Definition Exception. See COPYRIGHT and CONTRIBUTING.md for details.