Skip to content

fix(health): recover stuck loads and stop premature healthy state #91351

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
posthog wants to merge 3 commits into
base: master
Choose a base branch
from
Draft

fix(health): recover stuck loads and stop premature healthy state #91351

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
3ad626d
fix(health): recover stuck loads and stop premature healthy state
posthog[bot] Aug 30, 2026
e69351b
fix(health): defer empty state until the team resolves
posthog[bot] Aug 30, 2026
741f30f
fix(health): hide healthy summary cards for a project with no events
posthog[bot] Aug 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions frontend/src/scenes/health/components/HealthEmptyState.test.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
import '@testing-library/jest-dom'

import { cleanup, render, screen } from '@testing-library/react'

import { HealthEmptyState } from './HealthEmptyState'

describe('HealthEmptyState', () => {
afterEach(cleanup)

it('points a project with no events at install instead of declaring it healthy', () => {
render(<HealthEmptyState hasIngestedEvents={false} />)

expect(screen.getByText('Health checks have not run yet')).toBeInTheDocument()
// LemonBanner renders the action twice for its responsive layout, so there is at least one.
expect(screen.getAllByText('Install PostHog').length).toBeGreaterThan(0)
expect(screen.queryByText('All systems healthy')).not.toBeInTheDocument()
})

it('declares a project with events healthy when no issues are found', () => {
render(<HealthEmptyState hasIngestedEvents={true} />)

expect(screen.getByText('All systems healthy')).toBeInTheDocument()
expect(screen.queryByText('Install PostHog')).not.toBeInTheDocument()
})
})
38 changes: 38 additions & 0 deletions frontend/src/scenes/health/components/HealthEmptyState.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
import { LemonBanner } from '@posthog/lemon-ui'

import { urls } from 'scenes/urls'

import { ProductKey } from '~/queries/schema/schema-general'
import { OnboardingStepKey } from '~/types'

export function HealthEmptyState({ hasIngestedEvents }: { hasIngestedEvents: boolean }): JSX.Element {
// Without any ingested events the checks have nothing to run against, so treat an empty result
// set as "not set up yet" and point the user at install rather than claiming everything is fine.
if (!hasIngestedEvents) {
return (
<LemonBanner
type="info"
action={{
to: urls.onboarding({
productKey: ProductKey.PRODUCT_ANALYTICS,
stepKey: OnboardingStepKey.INSTALL,
}),
children: 'Install PostHog',
'data-attr': 'health-empty-install',
}}
>
<p className="font-semibold mb-0">Health checks have not run yet</p>
<p className="text-sm mt-1 mb-0">
Health checks start once your project receives data. Install PostHog to send your first events.
</p>
</LemonBanner>
)
}

return (
<LemonBanner type="success">
<p className="font-semibold mb-0">All systems healthy</p>
<p className="text-sm mt-1 mb-0">No active health issues found for your project.</p>
</LemonBanner>
)
}
10 changes: 3 additions & 7 deletions frontend/src/scenes/health/components/HealthIssueList.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,11 @@ import type { HealthIssueCategory } from '../healthCategories'
import { healthSceneLogic } from '../healthSceneLogic'
import { severityToTagType, worstSeverity } from '../healthUtils'
import type { HealthIssue } from '../types'
import { HealthEmptyState } from './HealthEmptyState'
import { HealthIssueCard } from './HealthIssueCard'

export const HealthIssueList = (): JSX.Element => {
const { issues, healthIssuesLoading, healthIssues } = useValues(healthSceneLogic)
const { issues, healthIssuesLoading, healthIssues, hasIngestedEvents } = useValues(healthSceneLogic)
const { snoozeIssue, dismissIssue, undismissIssue, loadHealthIssues } = useActions(healthSceneLogic)

if (healthIssuesLoading && !healthIssues) {
Expand All @@ -35,12 +36,7 @@ export const HealthIssueList = (): JSX.Element => {
}

if (issues.length === 0) {
return (
<LemonBanner type="success">
<p className="font-semibold mb-0">All systems healthy</p>
<p className="text-sm mt-1 mb-0">No active health issues found for your project.</p>
</LemonBanner>
)
return <HealthEmptyState hasIngestedEvents={hasIngestedEvents} />
Comment thread
posthog[bot] marked this conversation as resolved.
}

const groupedByCategory: Partial<Record<HealthIssueCategory, HealthIssue[]>> = {}
Expand Down
5 changes: 5 additions & 0 deletions frontend/src/scenes/health/components/HealthTables.stories.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import DataModelingDetailContent from '../categoryDetail/categories/DataModeling
import { SdkOutdatedRenderer } from '../renderers/SdkOutdatedRenderer'
import type { HealthIssue, HealthIssueSeverity } from '../types'
import { DataModelingHealthTable } from './DataModelingHealthTable'
import { HealthEmptyState } from './HealthEmptyState'
import { IngestionWarningTable } from './IngestionWarningTable'
import { PipelineHealthTable } from './PipelineHealthTable'
import { WebAnalyticsHealthTable } from './WebAnalyticsHealthTable'
Expand Down Expand Up @@ -298,6 +299,10 @@ export const WebAnalyticsEmpty: StoryFn = () => (
<WebAnalyticsHealthTable issues={[]} onSnooze={noop} onDismiss={noop} onUndismiss={noop} />
)

export const EmptyAllHealthy: StoryFn = () => <HealthEmptyState hasIngestedEvents={true} />

export const EmptyNoEventsYet: StoryFn = () => <HealthEmptyState hasIngestedEvents={false} />

export const SdkOutdatedDefault: StoryFn = () => <SdkOutdatedRenderer issue={SDK_OUTDATED_ISSUE} />

export const SdkOutdatedEmpty: StoryFn = () => <SdkOutdatedRenderer issue={SDK_OUTDATED_EMPTY_ISSUE} />
Expand Down
23 changes: 19 additions & 4 deletions frontend/src/scenes/health/healthSceneLogic.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,17 @@ import { sceneConfigurations } from 'scenes/scenes'
import { Scene } from 'scenes/sceneTypes'
import { teamLogic } from 'scenes/teamLogic'

import { Breadcrumb } from '~/types'
import { Breadcrumb, TeamPublicType, TeamType } from '~/types'

import { CATEGORY_ORDER, HEALTH_CATEGORY_CONFIG, categoryForKind } from './healthCategories'
import type { CategoryHealthSummary, HealthIssue, HealthIssueSeverity } from './types'
import { REFRESH_COOLDOWN_MS, REFRESH_POLL_COUNT, REFRESH_POLL_INTERVAL_MS, SEVERITY_ORDER } from './types'
import {
HEALTH_ISSUES_LOAD_TIMEOUT_MS,
REFRESH_COOLDOWN_MS,
REFRESH_POLL_COUNT,
REFRESH_POLL_INTERVAL_MS,
SEVERITY_ORDER,
} from './types'

export interface HealthIssuesResponse {
results: HealthIssue[]
Expand All @@ -23,9 +29,11 @@ export interface HealthIssuesResponse {

// Generated by kea-typegen. Update if you're an agent, ignore if you're human.
export interface healthSceneLogicValues {
currentTeam: TeamPublicType | TeamType | null // teamLogic
currentTeamIdStrict: number | string // teamLogic
breadcrumbs: Breadcrumb[]
categorySummaries: CategoryHealthSummary[]
hasIngestedEvents: boolean
healthIssues: HealthIssuesResponse | null
healthIssuesLoading: boolean
isManualRefresh: boolean
Expand Down Expand Up @@ -88,6 +96,7 @@ export interface healthSceneLogicMeta {
__keaTypeGenInternalSelectorTypes: {
issues: (healthIssues: HealthIssuesResponse | null) => HealthIssue[]
totalCount: (healthIssues: HealthIssuesResponse | null) => number
hasIngestedEvents: (currentTeam: TeamPublicType | TeamType | null) => boolean
categorySummaries: (issues: HealthIssue[]) => CategoryHealthSummary[]
}
}
Expand All @@ -102,7 +111,7 @@ export type healthSceneLogicType = MakeLogicType<
export const healthSceneLogic = kea<healthSceneLogicType>([
path(['scenes', 'health', 'healthSceneLogic']),
connect({
values: [teamLogic, ['currentTeamIdStrict']],
values: [teamLogic, ['currentTeamIdStrict', 'currentTeam']],
Comment thread
posthog[bot] marked this conversation as resolved.
}),
actions({
setShowDismissed: (show: boolean) => ({ show }),
Expand Down Expand Up @@ -156,7 +165,7 @@ export const healthSceneLogic = kea<healthSceneLogicType>([
const queryString = new URLSearchParams(params).toString()
const url = `api/environments/${values.currentTeamIdStrict}/health_issues/?${queryString}`

return await api.get(url)
return await api.get(url, { signal: AbortSignal.timeout(HEALTH_ISSUES_LOAD_TIMEOUT_MS) })

@posthog posthog Bot Aug 30, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The category detail loader can still hang forever

should_fix

Why we think it's a valid issue
  • Checked: The category detail loader, its scene's render branches, which categories actually reach that scene, and how the overview links to it.
  • Found: The loader takes no abort signal. frontend/src/scenes/health/categoryDetail/healthCategoryDetailLogic.ts:154 calls return await api.get(url) with no signal, against the same health_issues/ endpoint the overview loader uses. A request that never settles leaves healthIssues at its initial null.
  • Found: The scene has no error or retry branch at all. HealthCategoryDetailScene.tsx:53 renders {!healthIssues ? <LoadingSkeleton /> : ...}, and the refresh button sits inside GenericCategoryContent, which only renders once healthIssues is non-null. So there is no affordance to recover from the skeleton state.
  • Found: The dead end is worse than a hang-only case. The loader's catch at healthCategoryDetailLogic.ts:155-158 returns values.healthIssues, which is null on the first load. A rejecting request — network error, 5xx, or any transient failure — therefore resolves the loader with null, healthIssuesLoading goes false, and the scene sits on three skeleton bars permanently. The only signal is one transient lemonToast.error('Failed to load health issues'). So the scene dead-ends on any failed initial load, not just an unsettled one.
  • Found: The scene is reachable by normal navigation. afterMount redirects away only when the category has a redirectUrl (healthCategoryDetailLogic.ts afterMount, reading CATEGORY_DETAIL_CONFIG). In categoryDetailConfig.ts:23-62 only ingestion, sdk, web_analytics, pipelines, and error_tracking set one. data_modeling has an entry without a redirectUrl, and other has no entry, so both render this scene. HealthIssueList.tsx sends users there with CATEGORY_DETAIL_CONFIG[category]?.redirectUrl ?? urls.healthCategory(category) on the "View details" button, on the same page this change touches.
  • Impact: Confirmed. A user who clicks "View details" for data_modeling or other and whose first request hangs or fails gets a permanently loading screen with no retry and no error text. This is the same failure the change removes from the overview, left in place one click away, and in a harsher form because the overview at least reaches its retry banner.
  • Impact: Scope is the one thing that argues against acting now. Neither healthCategoryDetailLogic.ts nor HealthCategoryDetailScene.tsx is in this diff, so this is a pre-existing defect in a sibling file rather than a regression the change introduces. It is worth surfacing because the author has the fix pattern loaded and the two loaders duplicate the same request, but it is fair to land it as a follow-up rather than a blocker.
Issue description

The timeout protects only the overview loader. healthCategoryDetailLogic calls the same endpoint without a timeout. An unresolved initial request leaves the category detail scene on skeletons forever.

Suggested fix

Move the bounded request into a shared function and use it from both loaders. Add a failure state and retry action to the category detail scene.

Prompt to fix with AI (copy-paste) 
## Context
@frontend/src/scenes/health/healthSceneLogic.tsx#L168

<issue_description>
The timeout protects only the overview loader. `healthCategoryDetailLogic` calls the same endpoint without a timeout. An unresolved initial request leaves the category detail scene on skeletons forever.
</issue_description>

<issue_validation>
- **Checked:** The category detail loader, its scene's render branches, which categories actually reach that scene, and how the overview links to it.
- **Found:** The loader takes no abort signal. `frontend/src/scenes/health/categoryDetail/healthCategoryDetailLogic.ts:154` calls `return await api.get(url)` with no `signal`, against the same `health_issues/` endpoint the overview loader uses. A request that never settles leaves `healthIssues` at its initial `null`.
- **Found:** The scene has no error or retry branch at all. `HealthCategoryDetailScene.tsx:53` renders `{!healthIssues ? <LoadingSkeleton /> : ...}`, and the refresh button sits inside `GenericCategoryContent`, which only renders once `healthIssues` is non-null. So there is no affordance to recover from the skeleton state.
- **Found:** The dead end is worse than a hang-only case. The loader's `catch` at `healthCategoryDetailLogic.ts:155-158` returns `values.healthIssues`, which is `null` on the first load. A rejecting request — network error, 5xx, or any transient failure — therefore resolves the loader with `null`, `healthIssuesLoading` goes false, and the scene sits on three skeleton bars permanently. The only signal is one transient `lemonToast.error('Failed to load health issues')`. So the scene dead-ends on any failed initial load, not just an unsettled one.
- **Found:** The scene is reachable by normal navigation. `afterMount` redirects away only when the category has a `redirectUrl` (`healthCategoryDetailLogic.ts` afterMount, reading `CATEGORY_DETAIL_CONFIG`). In `categoryDetailConfig.ts:23-62` only `ingestion`, `sdk`, `web_analytics`, `pipelines`, and `error_tracking` set one. `data_modeling` has an entry without a `redirectUrl`, and `other` has no entry, so both render this scene. `HealthIssueList.tsx` sends users there with `CATEGORY_DETAIL_CONFIG[category]?.redirectUrl ?? urls.healthCategory(category)` on the "View details" button, on the same page this change touches.
- **Impact:** Confirmed. A user who clicks "View details" for `data_modeling` or `other` and whose first request hangs or fails gets a permanently loading screen with no retry and no error text. This is the same failure the change removes from the overview, left in place one click away, and in a harsher form because the overview at least reaches its retry banner.
- **Impact:** Scope is the one thing that argues against acting now. Neither `healthCategoryDetailLogic.ts` nor `HealthCategoryDetailScene.tsx` is in this diff, so this is a pre-existing defect in a sibling file rather than a regression the change introduces. It is worth surfacing because the author has the fix pattern loaded and the two loaders duplicate the same request, but it is fair to land it as a follow-up rather than a blocker.
</issue_validation>

## Task
Investigate the issue and solve it

<potential_solution>
Move the bounded request into a shared function and use it from both loaders. Add a failure state and retry action to the category detail scene.
</potential_solution>

@posthog posthog Bot Aug 30, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed this is a real, still-present defect — but it belongs in a separate follow-up, not this PR. The category detail loader (healthCategoryDetailLogic.ts) still fires the request with no timeout, its catch returns the previous value (null on the first load), and the detail scene (HealthCategoryDetailScene.tsx) has only a loading-skeleton branch with no error or retry. So a hung or failing first load on the data_modeling or other detail pages does dead-end on skeletons, one click from the overview.

I did not fix it here because both of those files are outside this PR's diff — this change only touches the overview scene (the empty-state, issue list, and scene logic). The fix the reviewer suggests (extract the bounded request into a shared helper used by both loaders, and add a failure state + retry action to the detail scene) would edit sibling files and add new UI to a scene this PR never touched, which is beyond its intent. That's a deliberate scope boundary, not a disagreement that the bug is worth fixing.

For a human to decide: track this as a follow-up PR that gives the category-detail loader the same AbortSignal.timeout the overview now uses (ideally via a shared helper so the two loaders don't drift) and adds an error/retry branch to HealthCategoryDetailScene. It's a clean, self-contained follow-up.

},
},
],
Expand All @@ -170,6 +179,12 @@ export const healthSceneLogic = kea<healthSceneLogicType>([
(s) => [s.healthIssues],
(healthIssues: HealthIssuesResponse | null): number => healthIssues?.count ?? 0,
],
// A project that has never ingested an event has no data for any check to run against, so an
// empty result set there means "checks have not run yet", not "everything is healthy".
hasIngestedEvents: [
(s) => [s.currentTeam],
(currentTeam: TeamPublicType | TeamType | null): boolean => !!currentTeam?.ingested_event,
],
Comment on lines +182 to +187

@posthog posthog Bot Aug 30, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An ingested event does not prove that health checks ran

must_fix

Why we think it's a valid issue
  • Checked: The Temporal schedule creation for health checks, every registered check's cron cadence, the per-team batch selection, the HealthIssue model and its API, and the new frontend branch that picks the empty-state copy.
  • Found: posthog/temporal/health_checks/schedule.py:45 calls a_create_schedule(..., trigger_immediately=False), and posthog/temporal/common/schedule.py:51-64 also defaults that flag to False. So a newly created schedule waits for its first cron tick.
  • Found: The schedules are global, not per team. create_health_check_schedules runs once per deploy from init_schedules() (posthog/temporal/schedule.py:913, invoked by bin/migrate:236-247). No hook creates or triggers a check when a team is created or when a team sends its first event. _get_team_id_batches_sync (posthog/temporal/health_checks/activities.py:26-73) recomputes the team list on each scheduled run, so a new team is only evaluated at the next tick.
  • Found: Every check cron is daily between 01:00 and 08:15 UTC, except path_cleaning_suggestions (products/web_analytics/backend/temporal/health_checks/path_cleaning_suggestions.py:45) at "23 6 * * 1" — weekly, Monday. A project that sends its first event after the last daily tick waits up to about 24 hours for the daily checks, and up to 7 days for the weekly one.
  • Found: No run-tracking exists that the page could read. HealthIssue (posthog/models/health_issue.py:42) has only per-row created_at/updated_at/resolved_at, and the sole "last run" signal is a Prometheus gauge pushed to a pushgateway (posthog/temporal/health_checks/observability.py:83-89), not an API field. The reviewer's premise that no such signal exists is correct.
  • Found: The scene does not force a run on mount. healthSceneLogic.tsx:335-346 loads results only, with a comment stating that auto-firing the refresh endpoint caused 429 storms. So the page shows the last cron output and nothing else.
  • Found: ingested_event is a plain team flag (posthog/models/team/team.py:352) that other surfaces use for "this project ever received data" (frontend/src/layout/panel-layout/installationStatusNavLogic.ts:108, frontend/src/scenes/insights/EmptyStates/sampleDataStateLogic.ts:47). It carries no information about check execution.
  • Impact: Confirmed. Once the first event arrives, hasIngestedEvents (healthSceneLogic.tsx:184-187) turns true, the issue list is still empty, and HealthEmptyState.tsx:32-37 renders "All systems healthy". No check has evaluated the project at that point. The window is up to a day for the daily checks and up to a week for the weekly one, and every new project passes through it. This is the same premature healthy state the change sets out to remove, so the fix closes only the zero-event part of it.
  • Priority: Lowered to should_fix. The claim is wrong and users see it, but the change does not regress anything — master already showed "All systems healthy" for any empty set, including projects with no events — and the state corrects itself at the next tick or when the user presses the manual refresh button. The primary remedy the reviewer asks for is a new backend signal, which is outside this frontend-only change; the cheap in-scope remedy is neutral copy. That makes it worth fixing but not a blocker.
Issue description

Health checks run from daily or weekly Temporal schedules, and schedule creation sets trigger_immediately=False. ingested_event becomes true when the first event arrives, before those checks necessarily run. An empty response during this interval renders “All systems healthy,” although no check result exists. The false green state can last until the next scheduled run.

Suggested fix

Expose a backend signal that records whether the relevant checks completed, and require it before showing the success banner. Until that signal exists, use a neutral empty state such as “No active health issues” instead of claiming that all checks passed.

Prompt to fix with AI (copy-paste) 
## Context
@frontend/src/scenes/health/healthSceneLogic.tsx#L182-187

<issue_description>
Health checks run from daily or weekly Temporal schedules, and schedule creation sets `trigger_immediately=False`. `ingested_event` becomes true when the first event arrives, before those checks necessarily run. An empty response during this interval renders “All systems healthy,” although no check result exists. The false green state can last until the next scheduled run.
</issue_description>

<issue_validation>
- **Checked:** The Temporal schedule creation for health checks, every registered check's cron cadence, the per-team batch selection, the `HealthIssue` model and its API, and the new frontend branch that picks the empty-state copy.
- **Found:** `posthog/temporal/health_checks/schedule.py:45` calls `a_create_schedule(..., trigger_immediately=False)`, and `posthog/temporal/common/schedule.py:51-64` also defaults that flag to `False`. So a newly created schedule waits for its first cron tick.
- **Found:** The schedules are global, not per team. `create_health_check_schedules` runs once per deploy from `init_schedules()` (`posthog/temporal/schedule.py:913`, invoked by `bin/migrate:236-247`). No hook creates or triggers a check when a team is created or when a team sends its first event. `_get_team_id_batches_sync` (`posthog/temporal/health_checks/activities.py:26-73`) recomputes the team list on each scheduled run, so a new team is only evaluated at the next tick.
- **Found:** Every check cron is daily between 01:00 and 08:15 UTC, except `path_cleaning_suggestions` (`products/web_analytics/backend/temporal/health_checks/path_cleaning_suggestions.py:45`) at `"23 6 * * 1"` — weekly, Monday. A project that sends its first event after the last daily tick waits up to about 24 hours for the daily checks, and up to 7 days for the weekly one.
- **Found:** No run-tracking exists that the page could read. `HealthIssue` (`posthog/models/health_issue.py:42`) has only per-row `created_at`/`updated_at`/`resolved_at`, and the sole "last run" signal is a Prometheus gauge pushed to a pushgateway (`posthog/temporal/health_checks/observability.py:83-89`), not an API field. The reviewer's premise that no such signal exists is correct.
- **Found:** The scene does not force a run on mount. `healthSceneLogic.tsx:335-346` loads results only, with a comment stating that auto-firing the refresh endpoint caused 429 storms. So the page shows the last cron output and nothing else.
- **Found:** `ingested_event` is a plain team flag (`posthog/models/team/team.py:352`) that other surfaces use for "this project ever received data" (`frontend/src/layout/panel-layout/installationStatusNavLogic.ts:108`, `frontend/src/scenes/insights/EmptyStates/sampleDataStateLogic.ts:47`). It carries no information about check execution.
- **Impact:** Confirmed. Once the first event arrives, `hasIngestedEvents` (`healthSceneLogic.tsx:184-187`) turns true, the issue list is still empty, and `HealthEmptyState.tsx:32-37` renders "All systems healthy". No check has evaluated the project at that point. The window is up to a day for the daily checks and up to a week for the weekly one, and every new project passes through it. This is the same premature healthy state the change sets out to remove, so the fix closes only the zero-event part of it.
- **Priority:** Lowered to `should_fix`. The claim is wrong and users see it, but the change does not regress anything — master already showed "All systems healthy" for any empty set, including projects with no events — and the state corrects itself at the next tick or when the user presses the manual refresh button. The primary remedy the reviewer asks for is a new backend signal, which is outside this frontend-only change; the cheap in-scope remedy is neutral copy. That makes it worth fixing but not a blocker.
</issue_validation>

## Task
Investigate the issue and solve it

<potential_solution>
Expose a backend signal that records whether the relevant checks completed, and require it before showing the success banner. Until that signal exists, use a neutral empty state such as “No active health issues” instead of claiming that all checks passed.
</potential_solution>

@posthog posthog Bot Aug 30, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed and worth addressing, but it needs a human product call rather than an unattended change. The issue is real: hasIngestedEvents is derived only from the team's ingested_event flag, which turns true the moment the first event arrives and tells us nothing about whether a health check has actually run. Because checks run on daily/weekly schedules that don't fire immediately, a project that has just started sending data sits in a window (up to ~24h for the daily checks, up to a week for the weekly path-cleaning check) where the issue list is empty simply because nothing has evaluated it yet — and the page shows the green "All systems healthy" banner anyway. That's the same premature-healthy state this PR set out to remove, just for projects that have events rather than none.

The clean fix is the one you suggested first: a backend signal that records whether the relevant checks completed, gating the success banner on it. That's a new API field plus Temporal changes, which is out of scope for this frontend-only PR, so I'm leaving it for the owning team to decide.

The only in-scope alternative is neutral copy (e.g. "No active health issues" instead of "All systems healthy"). I didn't make that change unattended because it removes the green healthy affordance for every project, including genuinely-healthy mature ones where the banner is accurate and reassuring — that's a product tradeoff someone owning the Health page should weigh, not a mechanical fix. For what it's worth this isn't a regression (master showed the same banner for any empty set) and it self-corrects at the next check run.

Decision needed: build the backend "checks completed" signal to gate the success state, or accept the frontend-only compromise of neutral copy that drops the "All systems healthy" state for all projects.

categorySummaries: [
(s) => [s.issues],
(issues: HealthIssue[]): CategoryHealthSummary[] => {
Expand Down
4 changes: 4 additions & 0 deletions frontend/src/scenes/health/types.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,10 @@ export const REFRESH_COOLDOWN_MS = 5 * 60 * 1000
export const REFRESH_POLL_INTERVAL_MS = 5000
export const REFRESH_POLL_COUNT = 12

// Bound the issues load so a request that never settles fails instead of leaving the page on
// skeletons forever. On timeout the loader rejects and the scene falls through to its retry banner.
export const HEALTH_ISSUES_LOAD_TIMEOUT_MS = 30 * 1000

export interface HealthIssue {
id: string
kind: string
Expand Down
Loading