feat(seer-billing): Check Seer credits before starting autofix steps by srest2021 · Pull Request #130283 · getsentry/sentry

srest2021 · 2026-11-06T18:45:02Z

Relates to CW-3061

Adds check_autofix_billing, called from all autofix entrypoints: trigger_autofix_agent, trigger_autofix_feature, trigger_push_changes and continue_single_run.

Token-billed orgs are checked for credits before every step, before Open PR and before a reply to a single run, and get SeerOutOfCreditsException if they can't pay. All credit checks are also separately feature-flagged.
Agentic triage runs aren't billed for token-billed orgs.
Other orgs keep the same quota check.

Autofix single run replies get their own billable operation with a placeholder estimate.

WIP followup to handle the out of credits exceptions: #130284

Return a 402 before sending a chat message when a token-billed org does not have enough Seer credits. Dashboard generation runs are not gated. can_run_seer_operation now passes orgs that are not token-billed. Refs CW-3061

Token-billed orgs are checked for Seer credits before each Autofix step and before opening a PR, and get SeerOutOfCreditsException when they can't pay. Other orgs keep the seat-based quota check. Agentic triage runs are not gated. Refs CW-3061

linear-code · 2026-11-06T18:45:10Z

CW-3061

…tofix-gating

Autofix billing checks the token-billed flag itself, so it only needs the credit check.

…tofix-gating # Conflicts: # src/sentry/seer/credits.py

sentry · 2026-11-06T18:58:32Z

Sentry Snapshot Testing

Name	Added	Removed	Changed	Renamed	Unchanged	Skipped	Status
sentry-frontend `sentry-frontend`	0	0	0	0	798	0	✅ Unchanged

⚙️ sentry-frontend Snapshot Settings

…gent-chat # Conflicts: # src/sentry/seer/endpoints/organization_seer_agent_chat.py

…tofix-gating # Conflicts: # src/sentry/seer/autofix/feature/dispatch.py # tests/sentry/seer/autofix/feature/test_dispatch.py

Chat replies to single runs continue the run through continue_single_run, which skipped billing. The run's own referrer decides, so agentic triage runs stay free.

…gent-chat # Conflicts: # src/sentry/seer/endpoints/organization_seer_agent_chat.py

…tofix-gating

…tofix-gating Check the credit check flag before the token-billed flag. has_enough_seer_credits no longer checks the flag; its callers do.

Cover orgs that aren't token-billed, group the chat credit tests after the access tests, and match the neighboring tests' names and setup.

…tofix-gating

…utofix-gating # Conflicts: # src/sentry/seer/credits.py

Single runs are behind organizations:autofix-single-run and need their own estimate, so they're out of scope here.

Agentic triage runs skip the Seer credit check, and the Autofix endpoint accepted any referrer from the request body. Treat a client-supplied agentic triage referrer as unknown.

Restores the single run reply check, using the placeholder estimate for now.

srest2021 · 2026-11-06T21:50:34Z

+    if referrer == AutofixReferrer.AGENTIC_TRIAGE:
+        logger.warning("group_ai_autofix.internal_referrer", extra={"referrer": raw})
+        return AutofixReferrer.UNKNOWN


Make sure people can't spoof AT referrer and skip getting billed. AT is not exposed via API.

…utofix-gating

billyvg

🤖 AI-assisted review, posted on behalf of @billyvg.

billyvg · 2026-11-09T19:26:38Z

+    check_autofix_billing(
+        group,
+        SeerOperation.AUTOFIX_SINGLE_RUN_REPLY,
+        agent_run.run.referrer,
+        is_new_run=False,
+    )


🤖 AI-assisted review, posted on behalf of @billyvg.

Should a user's reply to an agentic triage run skip the credit check? continue_single_run only runs when a user replies in Seer chat. agent_run.run.referrer is the run's original referrer, though, and SeerRun.referrer is set when the run starts and never updated. So every reply on a run that agentic triage started skips the check. A reply can write code and open a PR, which means a user on a token-billed org with no credits left can keep driving an agentic triage run indefinitely.

Continuing a step-based agentic triage run from the issue page does get checked, because trigger_autofix_agent passes the caller's referrer. If only agentic triage's own automated work is meant to be free, passing the reply's referrer here would match that path.

Resolved in #130552

billyvg · 2026-11-09T19:26:38Z

+    check_autofix_billing(
+        group,
+        SeerOperation.AUTOFIX_OPEN_PR,
+        seer_run.referrer if seer_run else None,
+        is_new_run=False,
+    )


🤖 AI-assisted review, posted on behalf of @billyvg.

Same question for Open PR. When a user clicks Open PR (issue page or Slack) on a run that agentic triage started, this skips the credit check, because the run's referrer is night_shift. A user continuing a step on that same run is checked under their own referrer, though. The completion hook's automatic push for agentic triage runs does need the run's referrer to stay free. If user-triggered pushes should be checked, one option is to use the run's referrer only when referrer == AutofixReferrer.ON_COMPLETION_HOOK, and the caller's referrer otherwise.

Resolved in #130552

Co-Authored-By: Sofia Rest <sofia.rest@sentry.io>

sehr-m

okay wait i think i get the operations for estimate gating, but a few qs for changes :))

sehr-m · 2026-11-09T19:07:00Z

+def check_autofix_billing(
+    group: Group,
+    operation: SeerOperation,
+    referrer: str | None,


don't love us gating on referrer- these are not standardized and currently there is no plan to standardize them

Resolved in #130552

trigger_autofix_agent checked billing, then called trigger_autofix_feature, which checked it again with the same arguments. Keep only the dispatch check, and test a token-billed org with credit checks off.

…tion (#130463) relates to CW-2332 depends on #129563, #130283, #130284 The chat's credit check only runs when the panel opens or the window regains focus. If credits run out partway through a conversation, the message box stays enabled and the next send fails with a generic error. This rechecks credits when: - Seer stops a reply for credits (`failure_reason: out_of_credits`) - a send comes back 402 so the out-of-credits notice and disabled message box show right away. A 402 no longer shows "There was an error sending your message"; the recheck brings up the notice instead. The recheck invalidates the credit check query, like `useExplorerAutofix` does. That needed `useSeerCreditCheck` to turn off with `enabled` instead of `skipToken`, so the key is the same either way. Example (it would show any intermediate thinking as well): <img width="402" height="615" alt="× Seer Agent I" src="https://github.com/user-attachments/assets/c7e056e8-73a7-4a5a-be32-62bcb80e1609" /> --------- Co-authored-by: Billy Vong <billy@sentry.io> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

Credit checks no longer skip runs by referrer. Agentic triage runs are checked like any other until a follow-up marks them by feature instead. Credit checks are behind seer-credit-checks, so nothing changes in production. Refs CW-3061

srest2021 · 2026-11-09T23:27:32Z

I'm going to drop agentic triage gating from this PR, wiring this feature through and gating only manual iterations on AT is a bit more complicated than I thought and deserves its own PR

srest2021 added 2 commits November 6, 2026 10:42

feat(seer): Gate agent chat messages on Seer credits

cbdde3a

Return a 402 before sending a chat message when a token-billed org does not have enough Seer credits. Dashboard generation runs are not gated. can_run_seer_operation now passes orgs that are not token-billed. Refs CW-3061

srest2021 mentioned this pull request Nov 6, 2026

feat(seer-billing): Handle SeerOutOfCreditsException at Autofix callers #130284

Open

github-actions Bot added the Scope: Backend Automatically applied to PRs that change backend components label Nov 6, 2026

srest2021 added 5 commits November 6, 2026 10:48

ref(seer): Keep the token-billed check inside can_run_seer_operation

e2855ba

Merge branch 'srest2021/CW-3061-agent-chat' into srest2021/CW-3061-au…

33f5287

…tofix-gating

ref(seer): Split has_enough_seer_credits out of can_run_seer_operation

d70d6cb

Autofix billing checks the token-billed flag itself, so it only needs the credit check.

ref(seer): Combine the credit check flag conditions

58a1b63

Merge branch 'srest2021/CW-3061-agent-chat' into srest2021/CW-3061-au…

8856cd4

…tofix-gating # Conflicts: # src/sentry/seer/credits.py

srest2021 added 9 commits November 6, 2026 11:06

Merge remote-tracking branch 'origin/master' into srest2021/CW-3061-a…

c49f7ed

…gent-chat # Conflicts: # src/sentry/seer/endpoints/organization_seer_agent_chat.py

Merge branch 'srest2021/CW-3061-agent-chat' into srest2021/CW-3061-au…

8e5ca8f

…tofix-gating # Conflicts: # src/sentry/seer/autofix/feature/dispatch.py # tests/sentry/seer/autofix/feature/test_dispatch.py

feat(seer): Check Seer credits before replying to an Autofix single run

0ab3b10

Chat replies to single runs continue the run through continue_single_run, which skipped billing. The run's own referrer decides, so agentic triage runs stay free.

Merge remote-tracking branch 'origin/master' into srest2021/CW-3061-a…

69f5488

…gent-chat # Conflicts: # src/sentry/seer/endpoints/organization_seer_agent_chat.py

Merge branch 'srest2021/CW-3061-agent-chat' into srest2021/CW-3061-au…

0f648e8

…tofix-gating

ref(seer): Check the credit check flag first

a3dae0d

Merge branch 'srest2021/CW-3061-agent-chat' into srest2021/CW-3061-au…

f738ddf

…tofix-gating Check the credit check flag before the token-billed flag. has_enough_seer_credits no longer checks the flag; its callers do.

test(seer): Align credit check tests with their files

94b979c

Cover orgs that aren't token-billed, group the chat credit tests after the access tests, and match the neighboring tests' names and setup.

Merge branch 'srest2021/CW-3061-agent-chat' into srest2021/CW-3061-au…

302de4f

…tofix-gating

Base automatically changed from srest2021/CW-3061-agent-chat to master November 6, 2026 20:03

Merge remote-tracking branch 'origin/master' into srest2021/CW-3061-a…

0ec42da

…utofix-gating # Conflicts: # src/sentry/seer/credits.py

vercel Bot deployed to Preview November 6, 2026 20:07

View deployment

ref(seer): Leave single run replies out of credit gating

6f7cb6f

Single runs are behind organizations:autofix-single-run and need their own estimate, so they're out of scope here.

sentry-warden Bot reviewed Nov 6, 2026

View reviewed changes

Comment thread src/sentry/seer/autofix/autofix_agent.py Resolved

srest2021 added 5 commits November 6, 2026 12:43

fix(seer): Don't let API callers claim the agentic triage referrer

99eb2ac

Agentic triage runs skip the Seer credit check, and the Autofix endpoint accepted any referrer from the request body. Treat a client-supplied agentic triage referrer as unknown.

ref(seer): Clarify why the endpoint ignores the agentic triage referrer

e8f415e

feat(seer): Check Seer credits before replying to an Autofix single run

396779d

Restores the single run reply check, using the placeholder estimate for now.

cleaning up diff

90c5db5

ref(seer): Restore the billing comment in trigger_autofix_agent

ff4faae

srest2021 added 8 commits November 6, 2026 13:18

cleaning up

b86335d

fix ordering

245aca9

Restore comment

80bba23

ref(seer): Keep all billing skip rules in check_autofix_billing

914a60c

refactor billing helper

28e5990

move logs out of helper

766bc10

small test fix:

6c0043d

cleaning up tests

1c7c190

srest2021 commented Nov 6, 2026

View reviewed changes

srest2021 changed the title ~~feat(seer): Check Seer credits before Autofix steps and Open PR~~ feat(seer): Check Seer credits before starting autofix steps Nov 6, 2026

srest2021 changed the title ~~feat(seer): Check Seer credits before starting autofix steps~~ feat(seer-billing): Check Seer credits before starting autofix steps Nov 6, 2026

srest2021 marked this pull request as ready for review November 6, 2026 21:54

srest2021 requested a review from a team as a code owner November 6, 2026 21:54

srest2021 requested a review from a team November 6, 2026 21:54

Merge remote-tracking branch 'origin/master' into srest2021/CW-3061-a…

4e289d3

…utofix-gating

vercel Bot deployed to Preview November 7, 2026 00:15

View deployment

This was referenced Nov 9, 2026

feat(seer-billing): Show out-of-credits Seer failures getsentry/toolkit#2031

Open

feat(seer): Show the credits notice when a chat runs out mid-conversation #130463

Merged

billyvg reviewed Nov 9, 2026

View reviewed changes

Comment thread src/sentry/seer/autofix/feature/dispatch.py Resolved

billyvg reviewed Nov 9, 2026

View reviewed changes

ref(seer-billing): Pass organization to check_autofix_billing

451f3bd

Co-Authored-By: Sofia Rest <sofia.rest@sentry.io>

sehr-m reviewed Nov 9, 2026

View reviewed changes

ref(seer-billing): Check Autofix billing only in dispatch

ab0d0aa

trigger_autofix_agent checked billing, then called trigger_autofix_feature, which checked it again with the same arguments. Keep only the dispatch check, and test a token-billed org with credit checks off.

srest2021 mentioned this pull request Nov 9, 2026

feat(slack): Hide Fix with Seer on issue alerts when Seer credits run out #130271

Open

srest2021 mentioned this pull request Nov 10, 2026

feat(seer-billing): Skip credit checks on runs agentic triage starts #130552

Open

srest2021 added this pull request to stack #130567 November 10, 2026 01:16

Conversation

srest2021 commented Nov 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

linear-code Bot commented Nov 6, 2026

Uh oh!

sentry Bot commented Nov 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Sentry Snapshot Testing

Uh oh!

Uh oh!

srest2021 Nov 6, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

billyvg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

billyvg Nov 9, 2026

Choose a reason for hiding this comment

Uh oh!

srest2021 Nov 10, 2026

Choose a reason for hiding this comment

Uh oh!

billyvg Nov 9, 2026

Choose a reason for hiding this comment

Uh oh!

srest2021 Nov 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sehr-m left a comment

Choose a reason for hiding this comment

Uh oh!

sehr-m Nov 9, 2026

Choose a reason for hiding this comment

Uh oh!

srest2021 Nov 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

srest2021 commented Nov 9, 2026

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

srest2021 commented Nov 6, 2026 •

edited

Loading

sentry Bot commented Nov 6, 2026 •

edited

Loading