fix(kubernetes): node list now syncs — prunes departed nodes (#1975) #257
Loading…
Reference in a new issue
No description provided.
Delete branch "fix/kubernetes-node-list-prune-1975"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
listmethod now performs a full sync: upserts live nodes, then prunes datastore records for nodes no longer present in the clusterdeleteResourceandreadModelDataonK8sContextandNodeContext— these runtime methods were always available but never declared by the kubernetes extensionreadModelData(context.definition.name, "node")to query existing records scoped to the current model instance, thendeleteResourceto remove stale ones2026.09.03.1with upgrade entryContext
@swamp/kubernetes/node'slistmethod upserted every node it found in the live cluster but never deleted records for nodes that departed. Records for scaled-down, drained, or replaced nodes accumulated in the datastore forever, misleading debugging (phantom nodes with reused IPs).The
swamp data deletesilent revert bug that compounded this issue was fixed separately in swamp-club/swamp#1996.nodePod pruning is intentionally skipped —
nodePodresources already havelifetime: "1h"which handles stale pod records automatically.Closes swamp-club#1975
Filed swamp-club#1991 (swamp data delete silent revert — fixed by swamp-club/swamp#1996)
Test plan
deno check extensions/models/*.ts— all 16 models passdeno lint extensions/models/— cleandeno fmt --check extensions/models/— cleandeno install --frozen— lockfile matches🤖 Generated with Claude Code
Code Review
Blocking Issues
None.
Suggestions
Unhandled error in prune loop (
node.ts:333–338): IfdeleteResourcethrows for one record, the loop aborts and remaining departed nodes are left un-pruned with no error logged. Wrapping each call in atry/catchwith acontext.logger.warning(...)would make partial failures visible and allow the loop to continue.Transitive semver range in deno.lock (
deno.lock): The new@systeminit/swamp-testing@0.20260521.16dependency introduces"npm:zod@^4.3.6"as a range entry in the lock file. This is a transitive dep declared by the testing package (not something this PR can control), and the lock file resolves it to4.3.6at runtime, so there's no real drift risk. Flagging for awareness — the lock file itself is committed and frozen, so this is safe.No tests for prune logic: The
listmethod now has meaningful stateful behavior (reading existing records, diffing, deleting stale ones). The CLAUDE.md testing rules specifically targetvault/anddatastore/extensions, so this isn't a rule violation, but the prune path (readModelData→ set diff →deleteResource) is the kind of logic that benefits from a mock-context unit test to catch regressions (e.g., a future sanitization change that causes a name-match mismatch betweenliveNamesandrecord.name).Summary: The change is well-structured. The optional-chaining guard (
if (context.deleteResource && context.readModelData)) correctly degrades gracefully on runtimes that don't yet support the new context methods. The sanitized name symmetry (bothliveNamesandrecord.nameuse the post-sanitizeInstanceNameform) is consistent. Version bump, upgrade entry, and manifest are all aligned. No type errors, noanyusage, no default exports.Adversarial Review
Critical / High
Label-filtered list prunes nodes that exist but don't match the filter —
node.ts:306-335The
listmethod fetches nodes withlabelSelector: labels(line 306–308), so only nodes matching the current label filter enterliveNames. The prune loop (lines 327–344) then deletes every existing record whose name is not inliveNames. This means nodes that are alive in the cluster but excluded by the label filter get pruned as "departed."Breaking example: User configures
labels: "role=worker". Three worker nodes and two control-plane nodes are synced in a previous full-list run. User then runslist— only the three workers come back from the API. The two control-plane node records are pruned, even though those nodes are perfectly healthy. Any downstream consumer that relies on the complete node inventory now sees a truncated view.Even without label changes between runs: if a user always uses a label filter, the first
listcall will prune records written by other methods likegetfor nodes outside the filter. Thegetmethod (line 351–369) writes individual node records without a label check, andlistwill delete them on its next sync.Suggested fix: Either (a) skip pruning entirely when
labelsis set, or (b) scopereadModelDatato only return records that were originally written by a label-filtered list (e.g., by tagging records with the filter used). Option (a) is simpler and safer:Empty API response wipes all node records —
node.ts:308-344If
coreApi.listNode()returns an emptyitemsarray — due to transient API failure returning200 {}, RBAC misconfiguration returning an empty list, or a cluster with no matching nodes —liveNameswill be empty. The prune loop will then delete every existing node record.Breaking example: A network blip causes the K8s API server to return
{ items: [] }. The extension deletes all node records. When the API recovers on the nextlistcall, the records are re-created, but any downstream system that read the datastore between those two calls sees zero nodes.Suggested fix: Guard against pruning when the live set is empty, or when the ratio of pruned-to-existing exceeds a threshold:
This prevents a complete wipe while still allowing incremental pruning when at least some nodes are returned.
Medium
A failed delete aborts the entire method, discarding successful write handles —
node.ts:327-345If
deleteResourcethrows for any single record during pruning, the exception propagates out oflistand thereturn { dataHandles: handles }on line 347 is never reached. The upserts (writes) already succeeded as side effects, but the caller never receives the handles. Depending on the framework's contract, this could mean the caller treats the entire sync as failed and retries, potentially causing duplicate work or confusing error reporting.Suggested fix: Wrap the prune loop in a try/catch so that prune failures are logged but don't prevent the method from returning its write handles:
Concurrent
listcalls can race on read-then-delete —node.ts:327-344Two concurrent
listinvocations both callreadModelData, get the same set of existing records, and both attempt to delete the same "departed" records. Depending ondeleteResource's behavior on already-deleted records, this could throw or silently double-count. IfdeleteResourcethrows on a missing record, one of the concurrent calls will fail (see finding #3).Low
readModelDatais not guaranteed to return sanitized names —node.ts:335The comparison
!liveNames.has(record.name)assumesreadModelDatareturns names in the same format assanitizeInstanceName(). If the framework stores or returns the raw name instead of the sanitized instance name, the comparison will never match and no pruning will occur (a silent no-op, not data loss). This is safe-by-default but worth verifying against the framework contract.Verdict
FAIL — The label-filter pruning interaction (#1) and the empty-response wipe (#2) are both data-loss bugs in realistic production scenarios. The core sync idea is sound but the prune guard needs to account for filtered queries and degenerate API responses before this is safe to merge.
Code Review
Blocking Issues
None.
Suggestions
NodeContextduplicates methods now inK8sContext(node.ts:35-39,helpers.ts:37-41).deleteResourceandreadModelDatawere just added to the sharedK8sContextinterface, butNodeContextre-declares them verbatim. A future pass could haveNodeContextextendK8sContextto avoid drift. Out of scope for this PR per the "only touch what's necessary" rule, but worth noting for follow-up.liveNamesvsreadModelDataname comparison assumption (node.ts:337-339). The prune loop comparesrecord.name(fromreadModelData) againstliveNames, which contains sanitized instance names. This is correct only ifreadModelDatareturns instance names (i.e., the keys passed towriteResource), not raw data-payload names. In practice, Kubernetes node names are DNS subdomain labels wheresanitizeInstanceNameis a no-op, so this works — but a brief comment documenting the expected shape ofreadModelDataresults would help future maintainers reason about the invariant.No test for the pruning path. The prune-on-label-filter and prune-on-empty-response guards are subtle. CLAUDE.md only mandates tests for
vault/anddatastore/extensions, so this isn't blocking, but a unit test that mocksreadModelData/deleteResourceand exercises the!labels && liveNames.size > 0guard condition would significantly improve confidence — especially the "skip when 0 live nodes" safety rail.Adversarial Review
Medium
node.ts:326-348 -- Prune failure discards all upsert handles
If
readModelDataordeleteResourcethrows during the prune phase (network error, permission denied, backend unavailable), the exception propagates out ofexecute, and the return statement at line 350 never executes. The upserts at lines 315-325 have already been committed viawriteResource, but the caller never receives the handles. Depending on how the framework uses those handles (e.g. to confirm what was synced), this could cause the caller to believe the entire operation failed even though the core work (upserting live nodes) succeeded.Breaking scenario:
readModelDatasucceeds and returns 200 existing records, butdeleteResourcefails on the 3rd call (transient network error). The 2 successful deletes are committed, the remaining 197 stale records stay, and the caller gets an unhandled rejection instead of the 50 upserted handles.Suggested fix: Wrap the prune block in try/catch. On prune failure, log a warning and continue to return the upsert handles.
node.ts:329 --
!labelsis falsy check on a string, empty string passes throughlabelsisz.string().optional(), so it can beundefinedor any string. The check!labelsistruefor bothundefinedand"". An empty string label selector is equivalent to no selector in Kubernetes, so this is currently correct behavior. However, it is a fragile idiom -- if the schema were ever changed to.default(""), pruning would silently activate even when the caller intended to pass a label selector that happened to be empty. Not a bug today, but a latent footgun.Suggested fix: Use
labels === undefinedorlabels == nullfor an explicit check.Low
node.ts:23 --
definitionis required in the type but unguarded at runtimedefinitionis required (not optional) inNodeContext, butdeleteResourceandreadModelDataare optional. The prune block (line 327) guards on the optional methods but not ondefinition. If a runtime provides the optional methods but does not providedefinition,context.definition.nameat line 332 would throw TypeError. In practice this is unlikely -- a runtime new enough to supply the new methods would supplydefinitiontoo -- but the asymmetry between the type-level guarantee (required) and the runtime contract (implicitly assumed) is worth noting.node.ts:23 vs pod_summary.ts:26 -- inconsistent
definitionshapeNodeContextdeclaresdefinitionwith anamefield whilePodSummaryContext(out of scope, but the only other model usingdefinition) declares it with anidfield. If the framework provides a singledefinitionobject, one of these models is looking at the wrong field. This is not a bug in this PR (out of scope), but the divergence is suspicious and worth verifying.Verdict
PASS -- The core sync logic is sound. The prune guards (skip when label-filtered, skip when API returns empty) are correct and match the commit messages. The two medium findings are robustness improvements, not correctness bugs in the happy path.
Code Review
Blocking Issues
None.
Suggestions
No tests for the new pruning logic — The
listmethod gained meaningful new behavior (record pruning), but no test file is in the changed files list. CLAUDE.md only mandates tests forvault/anddatastore/extensions, so this doesn't block merge, but a unit test exercising the prune path (mockreadModelDatareturning a stale record, assertdeleteResourceis called) would make future refactors safer.definitionis not guarded alongsidedeleteResource/readModelData— The prune block correctly gates oncontext.deleteResource && context.readModelDatabefore using them, butcontext.definition.nameis accessed inside that branch without a guard.definitionis declared as a required field onNodeContext, so TypeScript ensures the harness provides it; if the harness ever makes it optional in the future this would silently become a runtime crash. Low risk given current conventions, but worth noting.liveNames.size > 0semantics — The guard correctly prevents mass deletion when the API returns an empty list (transient error or cluster tear-down). This is a good safety choice. One minor edge case: if a cluster genuinely has no nodes but previously had records, pruning will never run. This is probably acceptable for operational safety, and can be revisited if it becomes a real problem.Overall: The sync/prune implementation is correct. The two key safety guards (
!labelsskips pruning on partial label-filtered results;liveNames.size > 0prevents catastrophic wipe on empty API response) are both present and correct. The interface additions inhelpers.ts(deleteResource?,readModelData?) are consistent with howNodeContextuses them. Version bump and upgrade entry look correct. No security, type, or CLAUDE.md compliance issues found.Adversarial Review
Critical / High
None.
Medium
None.
Low
kubernetes/extensions/models/node.ts:327-329— Race between K8s list and model-data read could prune a just-added node.If another sync concurrently writes a new node record between the
listNodecall (line 308) and thereadModelDatacall (line 331), that node would appear inexistingbut not inliveNames, and would be incorrectly pruned. Self-corrects on the next sync cycle, and theliveNames.size > 0guard already prevents the catastrophic variant (mass-delete on empty API response). No fix needed — this is inherent to non-transactional sync and the impact is a single missed record that reappears next cycle.kubernetes/extensions/models/node.ts:329—!labelsrelies on JavaScript falsiness forstring | undefined.The check
!labelsistruefor bothundefinedand"". An empty-string label selector is semantically equivalent to no selector (K8s API returns all nodes), so pruning correctly runs in both cases. However, this relies on the caller never passing a whitespace-only string like" "which would be truthy, bypass the guard, and the K8s API would likely return all nodes — meaning pruning runs correctly anyway. No practical risk, but documenting the intent (e.g.labels == nullvs!labels) would make the distinction explicit if this ever matters.Verdict
PASS — The pruning logic is well-guarded: it skips when label-filtered (partial view), when the API returns zero nodes (protects against accidental mass deletion), and when the context lacks
deleteResource/readModelData(graceful degradation). The interface additions toK8sContextare non-breaking (both optional). The upgrade chain and version bump are consistent acrossnode.tsandmanifest.yaml. No correctness, security, or resource management issues found.