Problem
Insight-alert Slack destinations created through the API can still miss the insight chart block, even after the one-off backfill from #90273 ran.
#88990 added the chart block to the insight-alert-firing Slack sub-template so investigated anomaly alerts post the insight chart. That block list only lives in client-side copies: frontend/src/scenes/hog-functions/sub-templates/sub-templates.ts and the example in products/cdp/mcp/cdp_functions.yaml. Nothing on the server owns the layout.
A destination's inputs.blocks is a snapshot the client sends at creation. create_alert_destination_hog_functions in products/alerts/backend/destinations.py passes the client payload straight into HogFunctionSerializer. So any caller that carries its own copy of the block list (a script, an agent skill, an MCP client with a cached example) creates a destination with whatever layout that copy has. Today that is the pre-#88990 layout, with a plain divider where the chart goes.
Observed: an insight-alert Slack destination created via the API on 2026-08-28, after the #90273 backfill had already run on that team, holds the old divider layout. The next investigated alert on it will post without a chart. The backfill is a one-off, so it does not catch these, and every new destination from such a client re-opens the gap.
slack_blocks() in products/alerts/backend/destination_configs.py is not the culprit. It serves the logs, replay-vision, and billing alert kinds, which have EventKindSpec entries; insight alerts do not have one, so their destinations never go through that builder.
Proposed fix
Make the server the source of truth for the insight-alert Slack layout, and repair incoming block lists on write.
-
Canonical layout in Python. Move the insight-alert Slack block list into products/alerts/backend next to INSIGHT_CHART_BLOCK in destination_backfill.py, so the chart expression and the surrounding blocks live in one place. Add a test that pins sub-templates.ts and the cdp_functions.yaml example to that constant, so the copies cannot drift again.
-
Upgrade on write. When an internal_destination on template-slack filtered on $insight_alert_firing is created or updated with hog-templated blocks that match the pre-chart shape, swap the divider above the buttons for the chart block before saving. blocks_with_chart() in destination_backfill.py already recognizes that shape and leaves hand-edited layouts alone; reuse it. The natural hook is create_alert_destination_hog_functions, or HogFunctionSerializer under allow_managed_alert_destination. This turns the backfill's recognizer into a steady-state guard instead of a one-off.
-
Optional: an EventKindSpec for insight alerts. With one, build_alert_destination_config could build insight-alert destinations from just slack_workspace + channel (+ context suffix), the way logs and replay-vision alerts already do, and API/MCP callers would no longer need to send blocks at all. This is the cleaner end state but a larger change; (1) and (2) close the gap on their own.
Acceptance criteria
- Creating an insight-alert Slack destination through the API with the pre-chart block list stores the chart block in
inputs.blocks.value, with matching recompiled bytecode.
- Creating one with a hand-edited layout (divider not directly above the buttons, or liquid templating) stores it unchanged.
sub-templates.ts, the MCP example, and the Python constant carry the same chart block string, enforced by a test.- Re-running
backfill_insight_alert_slack_chart_block on a team after new destinations are created reports them as already current.
Out of scope
- Alert state after a
false_positive investigation verdict: should_trigger_investigation in posthog/temporal/alerts/investigation.py skips the investigation when previous_state == FIRING, so a fire that follows a suppressed false positive goes out un-investigated (and so without a chart). Separate issue.
- Client-side copies of the layout that live outside this repo (agent skills, scripts). Once the server upgrades on write they stop mattering.
Related
Problem
Insight-alert Slack destinations created through the API can still miss the insight chart block, even after the one-off backfill from #90273 ran.
#88990 added the chart block to the
insight-alert-firingSlack sub-template so investigated anomaly alerts post the insight chart. That block list only lives in client-side copies:frontend/src/scenes/hog-functions/sub-templates/sub-templates.tsand the example inproducts/cdp/mcp/cdp_functions.yaml. Nothing on the server owns the layout.A destination's
inputs.blocksis a snapshot the client sends at creation.create_alert_destination_hog_functionsinproducts/alerts/backend/destinations.pypasses the client payload straight intoHogFunctionSerializer. So any caller that carries its own copy of the block list (a script, an agent skill, an MCP client with a cached example) creates a destination with whatever layout that copy has. Today that is the pre-#88990 layout, with a plain divider where the chart goes.Observed: an insight-alert Slack destination created via the API on 2026-08-28, after the #90273 backfill had already run on that team, holds the old divider layout. The next investigated alert on it will post without a chart. The backfill is a one-off, so it does not catch these, and every new destination from such a client re-opens the gap.
slack_blocks()inproducts/alerts/backend/destination_configs.pyis not the culprit. It serves the logs, replay-vision, and billing alert kinds, which haveEventKindSpecentries; insight alerts do not have one, so their destinations never go through that builder.Proposed fix
Make the server the source of truth for the insight-alert Slack layout, and repair incoming block lists on write.
Canonical layout in Python. Move the insight-alert Slack block list into
products/alerts/backendnext toINSIGHT_CHART_BLOCKindestination_backfill.py, so the chart expression and the surrounding blocks live in one place. Add a test that pinssub-templates.tsand thecdp_functions.yamlexample to that constant, so the copies cannot drift again.Upgrade on write. When an
internal_destinationontemplate-slackfiltered on$insight_alert_firingis created or updated with hog-templated blocks that match the pre-chart shape, swap the divider above the buttons for the chart block before saving.blocks_with_chart()indestination_backfill.pyalready recognizes that shape and leaves hand-edited layouts alone; reuse it. The natural hook iscreate_alert_destination_hog_functions, orHogFunctionSerializerunderallow_managed_alert_destination. This turns the backfill's recognizer into a steady-state guard instead of a one-off.Optional: an
EventKindSpecfor insight alerts. With one,build_alert_destination_configcould build insight-alert destinations from justslack_workspace+channel(+ context suffix), the way logs and replay-vision alerts already do, and API/MCP callers would no longer need to send blocks at all. This is the cleaner end state but a larger change; (1) and (2) close the gap on their own.Acceptance criteria
inputs.blocks.value, with matching recompiledbytecode.sub-templates.ts, the MCP example, and the Python constant carry the same chart block string, enforced by a test.backfill_insight_alert_slack_chart_blockon a team after new destinations are created reports them as already current.Out of scope
false_positiveinvestigation verdict:should_trigger_investigationinposthog/temporal/alerts/investigation.pyskips the investigation whenprevious_state == FIRING, so a fire that follows a suppressed false positive goes out un-investigated (and so without a chart). Separate issue.Related