swamp-club/swamp-extensions
Watch
8
0
Fork
You've already forked swamp-extensions
 0
Code Pull requests 3 Activity Actions

fix(codegen): PUT updates keep unset fields, GCP generation errors are loud and non-destructive, stable-only GCP versions (swamp-club #2662, #2660, #2657, #2647, #2656) #349

Merged
stack72 merged 19 commits from 2662 into main 2026-09-29 09:22:46 +00:00
Conversation 0 Commits 19 Files changed 388 +8604 -4314
stack72 commented 2026-09-29 09:10:17 +00:00
Owner
Copy link

Fixes swamp-club #2662, #2660, #2657, #2647 and #2656.

Problem

  • #2662: the GCP same-surface merge copied a whole parameters list onto an existing method when the preferred version had none, so required flags survived. That tightened a method the preferred version defines.
  • #2660: a GCP merge that broke parsing dropped every resource of the API, and generate:gcp still exited 0. Investigating this turned up a second silent path: parseGcpDiscoveryDocument swallowed per-resource exceptions. In both cases orphan pruning then deleted the published models.
  • #2647: fetch-schema:gcp pulled preview versions, so three preview-only resources shipped in @swamp/gcp/compute.
  • #2656: full-replacement PUT updates built their body only from the fields set in global arguments, so every optional field left unset was cleared or reset.
  • #2657: GCP filled create-required PUT fields from stored state, which silently reverted changes made outside swamp.

Fix

GCP pipeline

  • #2662: every entry of a parameters/properties collection that the merge creates on an existing definition is added as optional. A new method keeps its own flags.
  • #2660:
    • If the merged document fails, the pipeline falls back to the unmerged preferred document, and each merged version then contributes only the resources the preferred document lacks.
    • Per-resource parse failures are now reported instead of swallowed.
    • When anything in a service fails (a schema file, a document or a resource), that service's existing models that did not generate are kept, file and manifest entry, at their last good version.
    • generate:gcp exits 2 once it has written everything that did generate. An uncaught crash still exits 1.
  • #2647: isStableGcpVersion excludes alpha, beta and preview versions, both at fetch time and at generate time.

PUT updates keep unset fields (#2656, #2657)

  • A full-replacement PUT update GETs the live resource once and copies every update-body field that global arguments leave unset. The GET is skipped when every field is set. Stored state is never used for body fields.
  • codegen/shared/liveFill.ts chooses which fields to copy:
    • the response must describe the field with the same shape as the request (a DigitalOcean load balancer's region object is never echoed back where the request takes a slug);
    • fields whose own description says output-only or read-only are skipped;
    • string fields named like secrets are never copied. That includes create-required ones, which must now be set explicitly: secondary-dns tsigs.secret and ai-search tokens.cf_api_key.
  • GCP:
    • The live read replaces the stored-state fallback.
    • Fingerprint and etag come from that live read when there was one.
    • A PUT that takes an updateMask is a partial update, so it behaves like PATCH.
    • A resource whose GET cannot be built from the update's params (compute autoscalers) keeps the old stored-state fallback for create-required fields only.
  • Cloudflare: the #2645 fill is widened from create-required fields to all update-body fields.
  • DigitalOcean and Vercel: new fill.
  • Hetzner: unchanged. Its PUT only updates the fields sent, so #2656 does not apply.

Nightly regenerate-models: each provider step now continues on error.

  • A provider that crashed or failed to fetch has its partial model/ output discarded.
  • A generator exiting 2 keeps its complete output.
  • A final step fails the run and names the providers that failed.

Docs:

  • The five design docs are updated.
  • gcp.md now states correctly that method runs validate GlobalArgsSchema with .partial().

Model changes

  • A schema-drift commit generated with main's unchanged codegen: 5 GCP and 2 Vercel models change, and networksecurity/firewallendpoints_wildfireverdictchangerequests is removed because the API no longer publishes it.
  • One regeneration from that commit with the new codegen: 159 GCP, 86 Cloudflare and 19 DigitalOcean models change. Each gets a single version bump and an identity upgrade entry.
  • Removed: @swamp/gcp/compute acceleratorinterconnects, acceleratorinterconnectmemberinstances and hacontrollers. These were preview-only resources.
  • A second generation run produces no changes.

Behaviour change to call out

On a PUT update, leaving out an optional field now keeps its value instead of clearing it. To clear a field, set it explicitly to an empty value where the API accepts one.

Testing

  • Codegen: 365 tests pass.
    • gcp/pipeline_test.ts covers the merge, the fallback, parse errors, kept models (resource-level and document-level failures) and preview skipping.
    • shared/liveFill_test.ts covers the field-selection rules.
    • Generator tests cover all five providers.
    • The GCP and Cloudflare mock-server integration tests run cases where stored state and the live resource differ.
  • Emulator smoke test: BigQuery datasets against floci-gcp. On main, an update setting only description wiped friendlyName and labels. On this branch they are kept, including a value changed outside swamp.
  • Verification: verify-build passed 14/14 on de6c611c3 and verify-reviews passed 6/6. The attestation is posted to swamp-club #2662.

Not tested / known limits

  • Cloudflare and DigitalOcean PUT updates have not been run against the real APIs.
  • Each PUT update now makes an extra GET, so a 404 or a transient read failure fails the update.
  • The secret-name and output-only checks are heuristics. They don't catch a secret with a generic name, or a server-set sub-field inside an object.

Follow-ups (non-blocking review findings)

  • A GCP resource that fails to parse on every run now turns each nightly run red and blocks pruning for that service. This is intended, but it is new noise.
  • The nightly PR still opens when a provider failed, and its description doesn't say so.
  • Only GCP uses the exit-code-2 contract so far. The other providers treat any failure as a crash, which is safe.
  • The six provider steps in the workflow duplicate the same script, and the GCP live-fill emission has a redundant bare block.
  • Separate bug, already on main: the BigQuery datasets update reads the dataset id from existing["name"], a field datasets don't have. I'm filing it as its own issue.

🤖 Generated with Claude Code

https://claude.ai/code/session_01M8fdcEbuKuk5xeY7UNrJP6

Fixes swamp-club #2662, #2660, #2657, #2647 and #2656. ## Problem - **#2662**: the GCP same-surface merge copied a whole `parameters` list onto an existing method when the preferred version had none, so `required` flags survived. That tightened a method the preferred version defines. - **#2660**: a GCP merge that broke parsing dropped every resource of the API, and `generate:gcp` still exited 0. Investigating this turned up a second silent path: `parseGcpDiscoveryDocument` swallowed per-resource exceptions. In both cases orphan pruning then deleted the published models. - **#2647**: `fetch-schema:gcp` pulled `preview` versions, so three preview-only resources shipped in `@swamp/gcp/compute`. - **#2656**: full-replacement PUT updates built their body only from the fields set in global arguments, so every optional field left unset was cleared or reset. - **#2657**: GCP filled create-required PUT fields from stored state, which silently reverted changes made outside swamp. ## Fix **GCP pipeline** - **#2662**: every entry of a `parameters`/`properties` collection that the merge creates on an existing definition is added as optional. A new method keeps its own flags. - **#2660**: - If the merged document fails, the pipeline falls back to the unmerged preferred document, and each merged version then contributes only the resources the preferred document lacks. - Per-resource parse failures are now reported instead of swallowed. - When anything in a service fails (a schema file, a document or a resource), that service's existing models that did not generate are kept, file and manifest entry, at their last good version. - `generate:gcp` exits **2** once it has written everything that did generate. An uncaught crash still exits 1. - **#2647**: `isStableGcpVersion` excludes alpha, beta and preview versions, both at fetch time and at generate time. **PUT updates keep unset fields (#2656, #2657)** - A full-replacement PUT update GETs the live resource once and copies every update-body field that global arguments leave unset. The GET is skipped when every field is set. Stored state is never used for body fields. - `codegen/shared/liveFill.ts` chooses which fields to copy: - the response must describe the field with the same shape as the request (a DigitalOcean load balancer's `region` object is never echoed back where the request takes a slug); - fields whose own description says output-only or read-only are skipped; - string fields named like secrets are never copied. That includes create-required ones, which must now be set explicitly: secondary-dns `tsigs.secret` and ai-search `tokens.cf_api_key`. - **GCP**: - The live read replaces the stored-state fallback. - Fingerprint and etag come from that live read when there was one. - A PUT that takes an `updateMask` is a partial update, so it behaves like PATCH. - A resource whose GET cannot be built from the update's params (compute `autoscalers`) keeps the old stored-state fallback for create-required fields only. - **Cloudflare**: the #2645 fill is widened from create-required fields to all update-body fields. - **DigitalOcean and Vercel**: new fill. - **Hetzner**: unchanged. Its PUT only updates the fields sent, so #2656 does not apply. **Nightly `regenerate-models`**: each provider step now continues on error. - A provider that crashed or failed to fetch has its partial `model/` output discarded. - A generator exiting 2 keeps its complete output. - A final step fails the run and names the providers that failed. **Docs**: - The five design docs are updated. - `gcp.md` now states correctly that method runs validate `GlobalArgsSchema` with `.partial()`. ## Model changes - A schema-drift commit generated with main's unchanged codegen: 5 GCP and 2 Vercel models change, and `networksecurity/firewallendpoints_wildfireverdictchangerequests` is removed because the API no longer publishes it. - One regeneration from that commit with the new codegen: 159 GCP, 86 Cloudflare and 19 DigitalOcean models change. Each gets a single version bump and an identity upgrade entry. - **Removed**: `@swamp/gcp/compute` `acceleratorinterconnects`, `acceleratorinterconnectmemberinstances` and `hacontrollers`. These were preview-only resources. - A second generation run produces no changes. ## Behaviour change to call out On a PUT update, leaving out an optional field now **keeps** its value instead of clearing it. To clear a field, set it explicitly to an empty value where the API accepts one. ## Testing - **Codegen**: 365 tests pass. - `gcp/pipeline_test.ts` covers the merge, the fallback, parse errors, kept models (resource-level and document-level failures) and preview skipping. - `shared/liveFill_test.ts` covers the field-selection rules. - Generator tests cover all five providers. - The GCP and Cloudflare mock-server integration tests run cases where stored state and the live resource differ. - **Emulator smoke test**: BigQuery `datasets` against floci-gcp. On main, an update setting only `description` wiped `friendlyName` and `labels`. On this branch they are kept, including a value changed outside swamp. - **Verification**: `verify-build` passed 14/14 on `de6c611c3` and `verify-reviews` passed 6/6. The attestation is posted to swamp-club #2662. ## Not tested / known limits - Cloudflare and DigitalOcean PUT updates have not been run against the real APIs. - Each PUT update now makes an extra GET, so a 404 or a transient read failure fails the update. - The secret-name and output-only checks are heuristics. They don't catch a secret with a generic name, or a server-set sub-field inside an object. ## Follow-ups (non-blocking review findings) - A GCP resource that fails to parse on every run now turns each nightly run red and blocks pruning for that service. This is intended, but it is new noise. - The nightly PR still opens when a provider failed, and its description doesn't say so. - Only GCP uses the exit-code-2 contract so far. The other providers treat any failure as a crash, which is safe. - The six provider steps in the workflow duplicate the same script, and the GCP live-fill emission has a redundant bare block. - Separate bug, already on main: the BigQuery `datasets` update reads the dataset id from `existing["name"]`, a field datasets don't have. I'm filing it as its own issue. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01M8fdcEbuKuk5xeY7UNrJP6
When a same-surface additional version defines parameters on a method
the preferred version defines without any, the merge created the
collection with structuredClone, so required flags survived and a
method the preferred version defines gained a required parameter.
Entries of a parameters or properties collection created on an existing
definition now go through withoutRequirement, the same as entries added
to an existing collection. A whole new method keeps its own flags.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
If the preferred document with same-surface versions merged in fails to
dereference or parse, the merge failure is recorded and the unmerged
preferred document is parsed instead, with each merged version going
through the per-resource dedup path. Before, every resource of the API
was lost and only an error line was printed.

A model that exists on disk but fails to generate is now kept on disk
and in the manifest at its last good version instead of being pruned
as an orphan. generate:gcp exits non-zero when any error is reported,
after writing everything that did generate.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
fetchGcpSchema excluded only alpha and beta versions, so compute's
preview and 2026-10-01-preview documents were fetched, and resources that
exist only in preview were generated as separate models into
@swamp/gcp/compute. isStableGcpVersion now also excludes preview, at fetch
time and in selectBestVersion, and generation skips any non-stable
additional version still on disk.

Regenerating removes the three preview-only compute models
(acceleratorinterconnects, acceleratorinterconnectmemberinstances,
hacontrollers). They call the compute/v1 endpoint, although these resources are
published only in the preview versions.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A full-replacement PUT update built its body only from the fields set in
global arguments, so every optional field left unset was cleared or
reset to its default by the provider. GCP also filled create-required
fields from stored state, which silently reverted changes made outside
swamp since the last get/sync.

Every generator now emits the same rule for PUT updates: when any
update-body field is unset in global arguments, the update GETs the live
resource once and copies those fields from it, so an unset field keeps
its current value. Stored state is never used for body fields. The GET
is skipped when global arguments set every field. Create-required fields
that are optional in GlobalArgsSchema (GCP, Cloudflare) are still
checked, and update throws before the PUT if one is missing from both.
PATCH updates are unchanged.

shared/liveFill.ts picks the fields to copy: create-required fields
always, and any other field only when the GET response describes it with
the same shape as the request body. A differently shaped value, such as
DigitalOcean's region object where the request takes a slug, is never
echoed back.

- GCP: live read via readResource when the GET shares the update's path
  params (fixes #2657). Fingerprints and etags come from that read when
  there was one, so they are no longer stale either.
- Cloudflare: the #2645 fill widened from create-required fields to all
  update-body fields.
- DigitalOcean, Vercel, Hetzner: new fill using the read helper each
  model already imports. Vercel requires the read and update paths to
  match. Hetzner updates are always PUT.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8fdcEbuKuk5xeY7UNrJP6
generate:gcp now exits non-zero when it reports errors (swamp-club
#2660), and any provider step can already fail on a fetch or generate
crash. Such a failure skipped every later provider and the PR step.
Each provider step now continues on error, so the remaining providers
regenerate and the PR still opens with everything that did generate. A
final step fails the run and names the failed providers.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8fdcEbuKuk5xeY7UNrJP6
Regenerated with main's unchanged codegen from today's upstream schemas,
so the fix commits below show only their own effect.

- gcp: agentidentity, apigee and networksecurity pick up upstream schema
  changes; networksecurity drops
  firewallendpoints_wildfireverdictchangerequests, which the API no
  longer publishes.
- vercel: projects and teams pick up upstream schema changes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8fdcEbuKuk5xeY7UNrJP6
compute autoscalers name their target in the PUT body, not the path, so
the live GET cannot be built from the update's params. Without the
previous stored-state fallback for create-required fields, update threw
unless name was set in global arguments. Such resources now fall back to
stored state for create-required fields only, exactly as before, and
generate unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8fdcEbuKuk5xeY7UNrJP6
Some schemas, notably Google's, keep server-set fields such as creation
times, emails and unique ids in the request body and mark them only in
the description ("Output only.", "Read-only."). The PUT live fill no
longer echoes those back; create-required fields are still always
filled. Immutable fields stay filled, since a full-replacement PUT must
repeat their current value.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8fdcEbuKuk5xeY7UNrJP6
One regeneration after all codegen commits, so each model gets a single
version bump with an identity upgrade entry.

- PUT updates fill unset fields from the live resource (swamp-club
  #2656, #2657): 164 GCP models across 43 services, 86 Cloudflare,
  19 DigitalOcean and 11 Hetzner. Vercel's only PUT model has no
  fillable fields and is unchanged. compute/autoscalers, which cannot
  read live, is unchanged.
- compute drops the three models that existed only in preview versions
  (swamp-club #2647): acceleratorinterconnects,
  acceleratorinterconnectmemberinstances and hacontrollers.

A second generation run produces no further changes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8fdcEbuKuk5xeY7UNrJP6
parseGcpDiscoveryDocument swallowed any exception from building a
resource, so a resource that failed to parse vanished without an error,
and orphan pruning then deleted its published model. Parse failures are
now reported as errors (so generate:gcp exits non-zero), and when any
resource of a service fails to parse or generate, that service's existing
models that did not generate are kept on disk and in the manifest at
their last good version. This replaces the per-model keep from the
earlier #2660 commit, which could not see parse failures. A clean run
still prunes models removed upstream.

Today's schemas produce no such errors and the regenerated models are
unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8fdcEbuKuk5xeY7UNrJP6
- gcp.md: stable-only fetch (#2647), the merge fallback, merge-created
  parameters and the generation-error rules (#2660, #2662), and a new
  "PUT updates keep unset fields" section (#2657). Corrects the claim
  that swamp validates GlobalArgsSchema in full for every method: method
  runs use .partial(); the full check is at swamp model create with any
  --global-arg and in workflow validate for type-based steps.
- cloudflare.md, digitalocean.md, hetzner.md, vercel.md: the PUT live
  fill and which fields it copies (#2656).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8fdcEbuKuk5xeY7UNrJP6
Found by the verify-reviews adversarial review. Keeping existing models
only applied when a single resource failed. When a whole document failed
(a separate-surface version that cannot be dereferenced, a preferred
document that fails with nothing merged, or an unreadable or mismatched
schema file) while other documents of the service still generated, the
service was not marked, and orphan pruning deleted every model the
failed document used to produce. Every document-level error now goes
through reportError, which marks the service, so those models are kept.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8fdcEbuKuk5xeY7UNrJP6
Found by the verify-reviews adversarial review (high). Some GCP PUT
updates also take an updateMask query parameter (logging sinks, dataflow
jobs, bigtable instances and clusters, apigee environments, and others),
so they only change the masked fields. The live fill ran before the mask
was built, so every filled field joined the mask: updating a sink's
filter sent nine fields, including name, which the API can reject, and a
concurrent change between the read and the write was reverted. Such
updates are now treated like PATCH: nothing is filled or guarded, and
the mask lists only the fields set in global arguments.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8fdcEbuKuk5xeY7UNrJP6
Found by the verify-reviews adversarial review (medium). Hetzner's PUT
body fields are all optional and a field left out keeps its value
(labels sent replace the label set; labels left out are unchanged), so
swamp-club #2656 does not apply. The live fill only added a GET that can
fail and a window in which a concurrent change is reverted. Hetzner
updates again send only the fields set in global arguments.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8fdcEbuKuk5xeY7UNrJP6
Found by the verify-reviews adversarial review (low). Shape matching
compares types only, so an API that returns a secret masked (e.g.
"********") would have the mask written over the real value. Fields
named like secrets (secret, password, token, api_key, private_key,
credential, and similar) are never filled, not even when create-required:
they must be set explicitly, and update throws before the PUT if a
required one is not.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8fdcEbuKuk5xeY7UNrJP6
Found by the verify-reviews adversarial review (low). With every
provider step continuing on error, a generator that crashed after
rewriting or pruning some services still had its partial output
committed into the nightly PR. generate:gcp now exits 2 when it finished
but reported errors (an uncaught crash exits 1). Each provider step
keeps its output on exit 2 and otherwise, on a crash or fetch failure,
discards its model/ changes before failing.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8fdcEbuKuk5xeY7UNrJP6
Documents the fixes from the adversarial review, and the output-only
rule for the PUT live fill, which the earlier docs commit missed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8fdcEbuKuk5xeY7UNrJP6
Only a string can come back masked, and a name ending in an id
(token_id, apiKeyId) references a secret rather than holding one. The
rule matched on name alone, so it also dropped fields such as admin
users' boolean changePasswordAtNextLogin and cloudflare ai-search
instances' token_id from the live fill.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8fdcEbuKuk5xeY7UNrJP6
fix(gcp, cloudflare, hetzner): regenerate after the adversarial-review fixes
All checks were successful
CI / Validate Attestation (pull_request) Successful in 1m8s
Details
CI / Review Integrity (pull_request) Successful in 1m43s
Details
de6c611c31
 
Regenerated from the schema-drift commit, so each model still carries a
single version bump. Against the earlier regeneration commit:

- hetzner-cloud: all 11 models return to exactly their main versions
  (no live fill, since Hetzner's PUT is a partial update).
- gcp, updateMask PUTs: androidenterprise devices, chat spaces_messages,
  cloudsearch settings_searchapplications, dataflow jobs and logging
  sinks return to their main versions.
- secret-named string fields leave the live-fill lists: gcp admin users
  (password), games achievement and leaderboard configurations (token),
  sqladmin instances (rootPassword), storage objects (restoreToken), and
  cloudflare ai-search tokens (cf_api_key), alerting webhooks (secret),
  leaked-credential detections (password) and secondary-dns tsigs
  (secret). Create-required ones (tsigs secret, ai-search tokens
  cf_api_key) must now be set explicitly.

A second generation run produces no further changes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8fdcEbuKuk5xeY7UNrJP6
stack72 deleted branch 2662 2026-09-29 09:22:56 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
swamp-club/swamp-extensions!349
No description provided.