S3 run-state retention/lifecycle to bound operator run-index prefix growth

[marcusrbrown]

S3 run-state retention/lifecycle to bound operator run-index prefix growth

Summary

The gateway's operator run index reads run-state objects from S3 per repo. There is no retention policy on those prefixes, so they grow unbounded over a deployment's lifetime. Beyond storage cost, this has a correctness edge: listWithMetadata paginates with a bounded iteration cap and then sorts, so an extremely large per-repo run-state prefix can miss newer objects that fall beyond the iteration cap. An S3 lifecycle/retention policy that bounds prefix growth is the operational mitigation.

Where it comes from

Split out from fro-bot/agent#1036 (operator run-index scale follow-ups). The two code-side items in that issue shipped in fro-bot/agent:

• dedup bindings before the authz fan-out cap (Daily Autohealing Report — 2026-08-04 #1038)
• bound per-repo read amplification (Daily Autohealing Report — 2026-08-08 #1063)

The remaining item is this retention policy, which is infra-side (object-store lifecycle), not a gateway code change.

What's needed

• A lifecycle/retention rule on the run-state object prefixes in the gateway's S3 bucket that bounds how large any single per-repo prefix can grow (by age and/or count), so the gateway's bounded list pagination always sees the newest objects.
• Decide the retention window in light of how the operator run index is used (it returns at most the newest 100 across repos; very old run-state objects are not surfaced).

Acceptance

• Per-repo run-state prefixes are bounded so the gateway's listWithMetadata pagination cannot miss newer objects behind the iteration cap.
• Storage growth is bounded over a long-lived deployment.
• The retention window is documented alongside the gateway deploy config.

Cross-reference

Closes out the last item of fro-bot/agent#1036.

[fro-bot]

Triage

This is a valid infra-side follow-up. The current marcusrbrown/infra gateway deploy path wires S3 credentials and object-store hosts, but I found no bucket lifecycle/retention management for gateway run-state objects in apps/gateway/src/deploy.ts, apps/gateway/AGENTS.md, apps/gateway/README.md, or .github/workflows/deploy-gateway.yaml.

Reproduction / Verification

I did not mutate or populate the production bucket. A full live reproduction would require creating an extremely large per-repo run-state prefix, likely beyond the S3 adapter's 100-page list cap, which is not appropriate for issue triage.

Static verification confirms the risk:

• Upstream fro-bot/agent#1036 closed after splitting this exact remaining item into this infra issue.
• The pinned gateway upstream is fro-bot/agent@v0.79.1.
• Upstream listWithMetadata uses bounded S3 ListObjectsV2 pagination with S3_LIST_MAX_ITERATIONS = 100, then the run index sorts the returned subset by LastModified and reads at most MAX_RUNS_PER_REPO = 200 objects.
• Run-state keys are built as <store-prefix>/<identity>/<owner>/<repo>/runs/<runId>.json.
• Because S3 listing is key-order/page-order, not newest-first, a very large .../runs/ prefix can make the post-list sort operate on a truncated subset and miss newer objects outside the listed pages.

Recommended Next Steps

1. Add an explicit gateway run-state retention policy with a conservative initial window of 30 days. The operator run index returns at most the newest 100 summaries, so older terminal run-state objects are not useful for the UI path this issue targets.
2. Choose the lifecycle mechanism carefully. Native S3 lifecycle filters support prefixes/tags, not a glob like */runs/*; a broad <store-prefix>/<identity>/ rule could delete other object types (artifacts, bindings, locks, metadata, sessions). Safer options are per-repo lifecycle rules for known .../<owner>/<repo>/runs/ prefixes, or tagging run-state objects upstream and applying a tag-filtered lifecycle rule.
3. Document the selected retention window and exact key scope in apps/gateway/AGENTS.md and apps/gateway/README.md alongside the S3 deploy config.
4. Add verification that the lifecycle rule exists without exposing secrets. For AWS S3 this can be a get-bucket-lifecycle-configuration/SDK readback in deploy or provision; for R2/MinIO, confirm equivalent lifecycle support before choosing the implementation path.
5. If keeping this strictly infra-only, prefer a small provision/deploy helper that applies and verifies lifecycle configuration for configured per-repo run prefixes. If dynamic repos make that unmanageable, the clean long-term fix is an upstream object tag or key-shape change so one lifecycle rule can target run-state objects only.

Status

Confirmed as a real operational gap. Not currently reproducible safely in CI without synthetic high-volume object-store state. No code changes were made in this triage run.

---

Run Summary

Field	Value
Event	issues
Repository	marcusrbrown/infra
Run ID	28418856087
Cache	hit
Session	ses_0e95c9ecbffeIOC76cq89oH7fa

[marcusrbrown]

Dashboard-side operator contract 1.6.0 support is now deployed in dashboard release 2026.07.11. The run-index retention/lifecycle concern here remains separate; dashboard now renders sanitized failure reasons but does not change run-index storage retention behavior.

[marcusrbrown]

Design decision: wait for upstream run-state object tagging rather than building deploy-time per-repo lifecycle machinery here.

Native S3 lifecycle filters can't express */runs/*, and run-state objects share the <store-prefix>/<identity>/ tree with artifacts/bindings/locks/metadata/sessions, so the safe infra-only alternative was per-repo prefix rules regenerated at every deploy — permanent repo-discovery/drift machinery for what should be one declarative rule. Not worth it.

Filed fro-bot/agent#1192 asking for an object-type=run-state tag on run-state writes. Once a tagged release ships and the gateway pin picks it up, this issue closes with a single tag-filtered 30-day lifecycle rule on the bucket (window per the triage above: the operator index only surfaces the newest 100 runs, so terminal run-state >30d is dead weight).

Blocked on fro-bot/agent#1192.

[marcusrbrown]

Implemented and verified the AWS-native 30-day run-state lifecycle rule. The dedicated lifecycle principal applied the rule with the repository helper; helper readback and a separate S3 readback both confirmed exactly one enabled rule for object-type=run-state, with no unrelated lifecycle rules changed.

[marcusrbrown]

Reopening. The acceptance criteria here were never met, and the closure rested on a premise that turned out to be false.

The lifecycle rule shipped on 2026-07-19 and this issue closed on 2026-07-21. The rule filters on the object-type=run-state tag. No run-state object has ever carried that tag, because the fro-bot-gateway IAM identity was never granted s3:PutObjectTagging — so the rule has matched nothing since the day it was installed.

Evidence

The object inventory is unambiguous:

Run-state objects in the bucket	43
Oldest	2026-06-03
Newest	2026-07-17
Written since 2026-07-19	0

The gateway pin moved to v0.93.1 on 2026-07-19. Run-state tagging first shipped upstream in v0.92.0. Every write since that pin has been rejected:

AccessDenied: User: .../fro-bot-gateway is not authorized to perform:
s3:PutObjectTagging on resource: .../fro-bot-state/discord-gateway/<owner>/<repo>/runs/<id>.json

A tagged PutObject requires both s3:PutObject and s3:PutObjectTagging. Only the first was granted.

Two separate problems, not one

Run execution was broken, not just retention. createRun tags, and a failed create rejects run admission. Both entry paths share that admission code, so Discord mentions were as broken as dashboard launches — the dashboard was simply the first path to surface it visibly. That is tracked in #1400 and the grant is now applied and verified with a tagged write.

Retention was never bounded, which is this issue. Fixing the grant does not close it:

• The 43 existing objects were written before tagging existed. They are untagged and permanently outside the tag filter. The rule will never expire them.
• The lifecycle rule may not even be installed. apps/gateway/server/provision-droplet.ts applies it using the gateway's own AWS_* credentials, and that identity is denied s3:GetLifecycleConfiguration — verified live. It cannot read the bucket's lifecycle configuration, let alone write it. So the provisioning step that installs the rule could not have succeeded with those credentials.

What closing this actually requires

1. Confirm whether the lifecycle rule is present on the bucket. This needs a credential that can read lifecycle configuration; the runtime identity cannot.
2. Decide what to do with the 43 legacy untagged objects. They are small — about 15 KB total — so this is a correctness question about bounded prefix growth, not a storage-cost one. Either tag them retroactively, add a prefix-scoped rule that catches untagged objects, or delete them deliberately.
3. Confirm newly written run-state objects actually carry the tag, once a run has exercised the repaired grant.
4. Re-check the original concern behind this issue: listWithMetadata pagination missing newer objects behind its iteration cap. With 43 objects the cap was never approached, so that risk has been theoretical the whole time — but it becomes real again once writes resume and the rule stays inert.

Worth noting for the future: upstream's deploy/README.md already stated that an operator must grant s3:PutObjectTagging before deploying a release that emits these tags, and it cites this issue by number. The prerequisite was documented; nothing in this repo enforced or restated it. apps/gateway/AGENTS.md listed no IAM actions at all for the gateway identity, which is the gap that let a documented requirement go unmet for two months.

[marcusrbrown]

@fro-bot — findings on what remains here, after the s3:PutObjectTagging grant landed and run-state writes resumed.

One of the four items is now confirmed. One cannot be verified with any credential this repository holds. The other two are decisions rather than checks.

1. Is the lifecycle rule installed? Unknown.

s3:GetBucketLifecycleConfiguration is denied to all three identities available locally — the gateway runtime identity, the agent provisioner, and the VPN provisioner. Confirming the rule exists needs an admin credential.

This matters more than it sounds. apps/gateway/server/provision-droplet.ts applies the rule using the gateway's own object-only credentials, and those credentials cannot read lifecycle configuration, let alone write it. So the provisioning step that installs the rule could not have succeeded. The rule may never have existed at all.

2. Do newly written objects carry the tag? Yes — confirmed.

Verified by reading tags back on every run-state object written since the repair:

Written	Tags
05:52:44	object-type=run-state
06:25:12	object-type=run-state
06:31:53	object-type=run-state

A legacy sample returns no tags at all, as expected:

Written	Tags
2026-06-03	none
2026-06-03	none
2026-06-04	none

This required granting s3:GetObjectTagging temporarily. apps/gateway/AGENTS.md records that the runtime identity should not carry it, because the daemon has zero call sites for it — that reasoning is about the daemon, not about an operator confirming the tag lands. The grant was added for this verification and is being removed again; the documented rule stands.

So the write path is fully proven end to end: the permission is present, the tag is applied, and the object carries it.

3. The legacy objects

43 run-state objects, 15,123 bytes total, written 2026-06-03 through 2026-07-17 — all before tagging existed, all untagged, all permanently outside a tag-filtered rule.

Repo	Objects
marcusrbrown/infra	21
marcusrbrown/systematic	14
fro-bot/space-bus	4
fro-bot/dashboard	2
fro-bot/agent	1
marcusrbrown/dev-like	1

Three objects have been written since the repair.

4. A gap in the rule's shape

The rule filters on object-type=run-state. There are 14 non-run objects under the same prefix — bindings, locks, and similar. Those are not tagged as run-state and never will be, so even a working rule does not bound them. If the goal is bounding prefix growth rather than specifically run-state growth, the tag filter is narrower than the stated objective.

Where that leaves the issue

The original concern was listWithMetadata pagination missing newer objects behind its iteration cap. At 46 run-state objects and 15 KB, that risk is not live and has never been close to live. This was always about preventing unbounded growth over a long horizon, not about a present problem.

So the remaining work is small and specific:

• Confirm with an admin credential whether the lifecycle rule exists on the bucket. If it does not, decide who installs it, given that the deploy identity provably cannot.
• Decide the legacy objects. At 15 KB, deleting them is cosmetic; the argument for it is removing a permanently un-expirable set rather than reclaiming space. Tagging them retroactively is possible now — the grant covers it — but tagging objects that are all more than 30 days old means a working rule deletes them on its next pass, so that path and deletion have the same outcome.
• Decide whether the rule should cover the whole prefix rather than one tag, given the 14 objects it structurally cannot reach.

None of these need to happen quickly. They do need to be decided rather than assumed, which is what closing this issue prematurely the first time actually cost.

[marcusrbrown]

@fro-bot — correcting item 4 of my previous comment. I had it backwards, and the inverted version would have been destructive if acted on.

I wrote that the rule's object-type=run-state tag filter is "narrower than the stated objective" because 14 objects under the prefix are structurally unreachable by it, and suggested considering a prefix-wide rule instead.

All 14 of those objects are bindings:

fro-bot-state/discord-gateway/_/_/bindings/by-channel/<channel-id>.json   × 7
fro-bot-state/discord-gateway/<owner>/<repo>/bindings/repo.json           × 7

Bindings are durable configuration, not accumulating state. They are what associates a Discord channel with a repository, and they are meant to outlive everything around them — docs/solutions/integration-issues/gateway-mention-loop-permission-and-empty-workspace-2026-06-05.md records a case where /fro-bot add-project refused with already bound precisely because the binding survived a full workspace wipe.

A prefix-wide expiration rule would delete every binding older than 30 days. Channels would silently unbind, and the failure would surface later as repos that used to work no longer responding — with nothing pointing back at a lifecycle rule as the cause.

So the tag filter is not a shortcoming of the design. It is the mechanism that makes a retention rule safe to have at all: it expires exactly the class of object that accumulates, and cannot touch the class that must persist. The narrowness is the feature.

That removes one of the four decisions. The remaining three stand:

• Confirm whether the rule is installed, which needs an admin credential.
• Decide the 43 legacy untagged run-state objects, which a tag-filtered rule can never reach.
• Decide who owns installing it, given the deploy identity provably cannot.

Item 2 from that comment is unchanged and confirmed: newly written run-state objects do carry the tag.

[fro-bot]

Confirming what I can from this run and adding two data points that landed on main since this comment was posted.

What I could independently verify

This workflow (issue_comment) has zero AWS credentials — no AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY in the environment, and aws sts get-caller-identity fails with NoCredentials. So I can't add a fourth "yes/no" on whether the lifecycle rule exists on the bucket; I can only confirm that this execution context is not a candidate for checking it either. Item 1 stays genuinely blocked on an admin credential this repo does not hold anywhere in CI.

Two relevant commits landed on main after this comment (both by @marcusrbrown, 2026-09-20)

85f25d8 — "docs: state the gateway identity's required S3 IAM actions" adds a formal "S3 State-Bucket IAM Actions" section to apps/gateway/AGENTS.md. It now states in-repo, as a documented known gap, exactly what this comment found live:

apps/gateway/server/provision-droplet.ts reads and writes the bucket lifecycle configuration using these same AWS_* credentials, which needs bucket-level s3:GetLifecycleConfiguration and s3:PutLifecycleConfiguration. The fro-bot-gateway identity does not currently have these — the lifecycle provisioning step fails closed with AccessDenied. ... Prefer a separate provisioning-scoped identity or manual lifecycle configuration over widening fro-bot-gateway.

It also adds an explicit anti-pattern: never grant the runtime identity IAM self-management to route around this. So the "may never have existed" finding from item 1 is now the documented, accepted state of the repo rather than a live discovery — it doesn't resolve the unknown, but it stops the gap from being silent.

cb0daab — "feat(gateway): fail the deploy when the S3 identity cannot tag" hardens the write-path verification this comment already confirmed by hand: the deploy now performs its own tagged-object probe against the real bucket/prefix shape and fails closed if s3:PutObjectTagging is denied, specifically because an untagged write (the old self-test) reports healthy through exactly the failure mode that caused the two-month outage. This is #1400's territory, not #729's, but it means the "3 objects since repair" evidence in this comment now has a standing regression guard instead of resting on a one-time manual check.

Neither commit touches items 1 (rule existence), 3 (legacy objects), or 4 (rule shape) from this comment — those remain open decisions.

Recommendation on the three open items

Rule existence (item 1): consistent with the 85f25d8 guidance, don't widen fro-bot-gateway to check this. A one-time, narrowly-scoped credential (s3:GetLifecycleConfiguration + s3:PutLifecycleConfiguration on this bucket only, held outside the gateway GitHub Environment) is the smallest way to close the unknown — confirm the rule, and only create it if the readback comes back empty.

Legacy objects (item 3): delete rather than retag. This comment's own analysis is that a working rule expires them on its next pass either way since all 43 predate tagging by well over 30 days — retagging buys nothing and requires re-granting s3:PutObjectTagging-adjacent scope for a one-off. Deletion is the same outcome with fewer steps and no lingering grant to remember to revoke.

Rule shape (item 4): keep the rule scoped to object-type=run-state rather than broadening it to the whole prefix. This issue's acceptance criteria are specifically about bounding the objects listWithMetadata's pagination reads (run-state), not about bindings/locks/sessions, which likely have different retention semantics (e.g., a binding may need to outlive 30 days; a lock is transient by construction and probably shouldn't be governed by a day-based expiration at all). Blanket-expiring the other 14 objects under this rule risks silently deleting state nothing has audited for that lifetime. If unbounded growth of those other object types is a real concern, it's a distinct problem with its own retention semantics per object type and warrants its own issue rather than folding into #729's run-state-specific rule.

Given item 1 is unresolved and is the acceptance criterion this issue is actually about ("prefixes are bounded"), this should stay open until an admin-credential readback confirms the rule is live — a documented known gap is progress, not closure.

---

Run Summary

Field	Value
Event	issue_comment
Repository	marcusrbrown/infra
Run ID	35648467813
Cache	hit
Session	ses_f3df09a8effeMu85OJVfZiizO1

[marcusrbrown]

@fro-bot — this is resolved, and one of my earlier claims was wrong.

The lifecycle rule was installed all along

I said its existence was unknown and reasoned that it "may never have existed," because the identity provisioning uses cannot read or write lifecycle configuration. An admin read settles it — the rule is present and byte-exact:

{
  "ID": "run-state-30d-expiration",
  "Filter": {"Tag": {"Key": "object-type", "Value": "run-state"}},
  "Status": "Enabled",
  "Expiration": {"Days": 30}
}

That matches RUN_STATE_LIFECYCLE_RULE in apps/gateway/server/provision-droplet.ts field for field. It was applied out-of-band at some point, which the deploy-identity argument could not have told me either way. I inferred absence from an inability to install, and that inference was unsound.

Retention was configured correctly the entire time. It simply had nothing to match, because nothing was ever tagged.

Bucket versioning is off

get-bucket-versioning returns an empty response, which is what AWS returns when versioning was never configured. So no NoncurrentVersionExpiration companion rule is needed, and expiration deletes cleanly rather than leaving noncurrent versions behind.

The legacy objects are now tagged

All 43 pre-tagging run-state objects have been tagged object-type=run-state, each verified by reading the tag back. Since every one is well past 30 days, the existing rule will expire them on its next pass.

Full audit of the prefix afterwards:

	Count	Tagged
run-state objects	46	46
bindings	14	0

The second row is the one worth checking, and it is correct: bindings carry no tag and the rule cannot reach them.

Provisioning no longer attempts lifecycle configuration

bun run provision:gateway currently fails. main() called ensureRunStateLifecycleRule with the runtime credentials, which throws AccessDenied on s3:GetLifecycleConfiguration before reading anything — confirmed by invoking the function directly with the real credentials.

Lifecycle configuration is now explicitly operator-owned. Automating it would require some identity to permanently hold s3:PutLifecycleConfiguration, and the only candidate is a credential that lives on the droplet and is used by the running daemon. That is bucket-administration power held forever to automate a change that happens roughly once — a lifecycle rule is bucket-level state, untouched by rebuilding the droplet and never read or written by a deploy.

ensureRunStateLifecycleRule remains exported and unchanged for an operator to run with a privileged credential. Missing inputs now throw rather than logging skipped and returning success.

Where this issue stands

The acceptance criteria are now actually met rather than assumed:

• Per-repo run-state prefixes are bounded — the rule is installed, enabled, and every run-state object carries the tag it filters on.
• Storage growth is bounded over a long-lived deployment, for the same reason.
• The retention window is documented alongside the gateway deploy config, in apps/gateway/AGENTS.md.

The original pagination concern behind this issue — listWithMetadata missing newer objects behind its iteration cap — was never close to live at 46 objects and 15 KB. It is now structurally bounded rather than merely small.

One thing that will confirm the rule end to end: the 43 tagged legacy objects should disappear on the next lifecycle pass. That is the first time this rule will have expired anything, and it is worth checking rather than assuming.