fix(ci): Review Integrity fits the runner's disk (swamp-club #3174) #500

Open

seth wants to merge 2 commits from cue/3174-swamp-extensions-ci-review into main

pull from: cue/3174-swamp-extensions-ci-review

merge into: swamp-club:main

swamp-club:main

swamp-club:probe/3174-runner-lifetime

swamp-club:automated/regenerate-models-aws

swamp-club:automated/regenerate-models-vercel

swamp-club:cue/3144-stagecraft-recover-gracefully

swamp-club:cue/2947-stagecraft-manifest

swamp-club:fix/916-workflow-stage-recovery

swamp-club:issue-2249

swamp-club:stack72-patch-2

swamp-club:stack72-patch-1

swamp-club:feat/agent-extensions-claude

Conversation 0 Commits 2 Files changed 1 +41 -11

seth commented 2026-10-15 21:27:35 +00:00

Owner

Copy link

Fixes swamp-club #3174.

Cause

Review Integrity jobs went dark a few seconds after the claude download and were reaped by Forgejo about 10 minutes later, with no comment. The runners are KEDA ScaledJob pods on GKE Autopilot using the host executor, so every checkout and download lands on the pod's own disk. That disk had a 1 GiB ephemeral-storage limit. Every frozen task matched a GKE eviction ("Pod ephemeral local storage usage exceeds the total limit of containers 1Gi") 4 to 13 seconds after its last log line: six tasks on 10-08, run 2067 for PR 494 on 10-15, and the ec2 publish job in #3390.

The job needed about 1.07 to 1.1 GB:

• base checkout: 188 MB
• head checkout with fetch-depth: 0: 268 MB
• setup-deno, which keeps the zip and the extracted folder as well as the cached copy: about 350 MB
• the claude binary: 239 MB

The runner limit has since been raised in infra, and PR 494's re-run passed. This PR keeps the job well under any modest limit.

Change (.forgejo/workflows/ci.yml, review-integrity only)

• Shallow head fetch. The head is fetched as git objects with no working tree; later steps only read it through git diff, log and show. The fetch starts one commit deep and deepens by commit until base and head share an ancestor, falling back to --unshallow, so the three-dot diff keeps its merge base. A shallow history can only yield an older common ancestor, never a newer one, so the audit can only ever see more than the PR changed, never less. Fails closed if there is no shared history. The repo is public, so no credential is used. Measured: 37 MB for a branch based on today's main, 53 MB for one 30 commits behind, against 268 MB before.
• Deno download cleanup. After setup-deno, remove the zip and the extracted folder it leaves in RUNNER_TEMP (about 200 MB). The copy Deno runs from, and everything else in RUNNER_TEMP, is left alone.
• The header comment now says how head/ is obtained in each job.

How this PR is audited

ci.yml runs as the pull request has it, so this PR's own Review Integrity run exercises the new fetch and cleanup steps. The audit prompt, the verdict rule and the poster script still come from the base commit.

Testing

• The new steps' scripts, extracted from ci.yml, were run against a local bare remote with a refs/pull/N/head ref. For branches based 0, 30 and 150 commits behind main, the true merge base was found while staying shallow, and the right trust-root files were flagged. This caught a bug: an up-to-date ref is not deepened, so deepening now names commits.
• Deno cleanup was run against a fake RUNNER_TEMP: only the zip and the deno-only folder were removed.
• verify-build and verify-reviews passed; the attestation is posted for 1bfa2f080.

Follow-ups

• #3514: streamed progress, a timeout and a did-not-finish comment in the audit step (deferred).
• #3515: swamp install.sh peak disk use.
• #3516: promptfoo optional dependencies.

🤖 Generated with Claude Code

Fixes swamp-club #3174. ## Cause Review Integrity jobs went dark a few seconds after the claude download and were reaped by Forgejo about 10 minutes later, with no comment. The runners are KEDA ScaledJob pods on GKE Autopilot using the host executor, so every checkout and download lands on the pod's own disk. That disk had a 1 GiB ephemeral-storage limit. Every frozen task matched a GKE eviction ("Pod ephemeral local storage usage exceeds the total limit of containers 1Gi") 4 to 13 seconds after its last log line: six tasks on 10-08, run 2067 for PR 494 on 10-15, and the ec2 publish job in #3390. The job needed about 1.07 to 1.1 GB: - base checkout: 188 MB - head checkout with `fetch-depth: 0`: 268 MB - setup-deno, which keeps the zip and the extracted folder as well as the cached copy: about 350 MB - the claude binary: 239 MB The runner limit has since been raised in infra, and PR 494's re-run passed. This PR keeps the job well under any modest limit. ## Change (`.forgejo/workflows/ci.yml`, review-integrity only) - **Shallow head fetch.** The head is fetched as git objects with no working tree; later steps only read it through `git diff`, `log` and `show`. The fetch starts one commit deep and deepens by commit until base and head share an ancestor, falling back to `--unshallow`, so the three-dot diff keeps its merge base. A shallow history can only yield an older common ancestor, never a newer one, so the audit can only ever see more than the PR changed, never less. Fails closed if there is no shared history. The repo is public, so no credential is used. Measured: 37 MB for a branch based on today's main, 53 MB for one 30 commits behind, against 268 MB before. - **Deno download cleanup.** After setup-deno, remove the zip and the extracted folder it leaves in RUNNER_TEMP (about 200 MB). The copy Deno runs from, and everything else in RUNNER_TEMP, is left alone. - The header comment now says how head/ is obtained in each job. ## How this PR is audited ci.yml runs as the pull request has it, so this PR's own Review Integrity run exercises the new fetch and cleanup steps. The audit prompt, the verdict rule and the poster script still come from the base commit. ## Testing - The new steps' scripts, extracted from ci.yml, were run against a local bare remote with a `refs/pull/N/head` ref. For branches based 0, 30 and 150 commits behind main, the true merge base was found while staying shallow, and the right trust-root files were flagged. This caught a bug: an up-to-date ref is not deepened, so deepening now names commits. - Deno cleanup was run against a fake RUNNER_TEMP: only the zip and the deno-only folder were removed. - verify-build and verify-reviews passed; the attestation is posted for 1bfa2f080. ## Follow-ups - #3514: streamed progress, a timeout and a did-not-finish comment in the audit step (deferred). - #3515: swamp install.sh peak disk use. - #3516: promptfoo optional dependencies. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

seth added 1 commit 2026-10-15 21:27:37 +00:00

fix(ci): Review Integrity fits the runner's disk (swamp-club #3174) ... 1bfa2f080e

Runner pods were evicted for exceeding their 1 GiB ephemeral-storage
limit while the audit downloaded claude, so the job went dark and
Forgejo reaped it with no comment. The head is now fetched shallow with
no working tree, deepening only until base and head share an ancestor,
and setup-deno's leftover zip and extracted folder are removed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

seth added 1 commit 2026-10-16 00:35:06 +00:00

fix(ci): fetch the head's full commit history, without blobs (swamp-club #3174) ... 9f6f09c229

Review Integrity on PR 500 found that the shallow, deepen-until-merge-base
fetch can settle on an older common ancestor. A three-dot diff from there
compares two trees, so a pull request that restores a trust-root file to
that ancestor's state would show no change and skip the audit. The safety
argument in the previous commit was wrong.

Fetch every commit with --filter=blob:none instead: the merge base is
exact again, as it was with fetch-depth 0, and only the contents the diff
reads are fetched. Measured 11 MB against 268 MB for the old full
checkout. A server that ignores the filter sends everything, which costs
disk but changes nothing the audit sees.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

All checks were successful

Hide checks

CI / Validate Attestation (pull_request) Successful in 50s

Details

CI / Review Integrity (pull_request) Successful in 1m13s

Details

This pull request can be merged automatically.

This branch is out-of-date with the base branch

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin cue/3174-swamp-extensions-ci-review:cue/3174-swamp-extensions-ci-review

git switch cue/3174-swamp-extensions-ci-review

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main

git merge --no-ff cue/3174-swamp-extensions-ci-review

git switch cue/3174-swamp-extensions-ci-review

git rebase main

git switch main

git merge --ff-only cue/3174-swamp-extensions-ci-review

git switch cue/3174-swamp-extensions-ci-review

git rebase main

git switch main

git merge --no-ff cue/3174-swamp-extensions-ci-review

git switch main

git merge --squash cue/3174-swamp-extensions-ci-review

git switch main

git merge --ff-only cue/3174-swamp-extensions-ci-review

git switch main

git merge cue/3174-swamp-extensions-ci-review

git push origin main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

swamp-club/swamp-extensions!500

No description provided.