feat: stream-driven progress for explain --generate by peyton-alt · Pull Request #964 · entireio/cli

peyton-alt · 2026-04-16T04:26:22Z

Summary

Switches entire explain --generate from silent wait + wall-clock timeout to real-time stream-driven progress and a TTY-gated deadline strategy.

Previously stacked on #963 (typed error classification). Rebased off main — now fully independent. See Scope note below.

What users see now

Interactive (TTY):

Generating checkpoint summary... (transcript: 47.0 KB)
→ Sending request to Anthropic…
→ Anthropic responded (TTFT 1.9s, 35.9k cached input tokens) — generating…
→ Writing summary… (~1.2k tokens)
✓ Summary generated (3.1s, 1.9k output tokens)
✓ Summary generated and saved

Non-TTY (CI): same lines, one per event (no in-place \r updates). 5-minute idle-based safety deadline + 30-minute hard wall-clock cap.

ACCESSIBLE=1: one line per event, no ANSI escape sequences.

Architecture

Stream-json mode: claude --print --output-format stream-json --verbose consumed incrementally by an NDJSON parser (streamClaudeResponse). Falls back internally to --output-format json if the CLI rejects streaming flags.
agent.StreamingTextGenerator optional capability interface with AsStreamingTextGenerator helper. Claude Code implements it via GenerateTextStreaming in a new generate_streaming.go file — existing GenerateText path untouched.
summaryProgressWriter renders GenerationProgress phases using existing statusStyles (lipgloss) for consistent styling with entire status. In-place \r updates on TTY; one line per event in CI/accessible mode.
TTY-gated deadline: no deadline interactive (Ctrl+C is the timeout); idle-based 5min watchdog non-TTY with a hard 30min wall-clock cap on top.
ProgressFn threading: summarize.GenerateFromTranscript gains a ProgressFn parameter, forwarded through ClaudeGenerator to GenerateTextStreaming. Condensation path passes nil (no progress).

Scope note (re: #963)

This PR was originally stacked on #963 (typed ClaudeError classification) so that stream failures mapped to actionable messages (auth → claude login, rate-limit, etc.). Rebasing off main means the streaming path currently returns plain error values — users hitting stream failures see a generic claude CLI reported error: <text> (with HTTP status when present) instead of kind-specific advice.

When #963 lands on main, a small follow-up will wire typed errors into the streaming path (~1 file, map *ClaudeError kinds → user-facing messages in generate_streaming.go and explain.go). Happy-path UX is unaffected; this only changes how rare failures are phrased.

Test plan

mise run check (fmt + lint + unit + integration + E2E canary) passes
Manual: entire explain -c <id> --generate on TTY — verify progress phases render
Manual: entire explain -c <id> --generate | cat — verify non-TTY mode (one line per event)
Manual: ACCESSIBLE=1 entire explain -c <id> --generate — verify no ANSI escapes

🤖 Generated with Claude Code

Copilot

Pull request overview

This PR upgrades entire explain --generate to use Claude CLI streaming output for incremental progress reporting, and introduces a TTY-gated deadline strategy (no deadline on interactive TTY by default; idle-based safety deadline in non-TTY).

Changes:

Add a streaming text-generation capability (StreamingTextGenerator) and implement it for the Claude Code agent using --output-format stream-json with an NDJSON stream parser.
Thread streaming progress events through summarization and render them in explain via summaryProgressWriter (TTY in-place updates vs. line-per-event for non-TTY/accessible mode).
Replace the prior fixed wall-clock timeout behavior with a resolver-driven deadline policy plus an idle watchdog for non-TTY default behavior.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
cmd/entire/cli/explain.go	Adds progress rendering, deadline resolution, idle watchdog, and transcript-size preflight message for `explain --generate`.
cmd/entire/cli/explain_test.go	Updates and adds tests for progress writer, idle watchdog, and deadline policy behavior.
cmd/entire/cli/summarize/summarize.go	Extends `GenerateFromTranscript` with an optional progress callback and threads it into Claude generation.
cmd/entire/cli/summarize/summarize_test.go	Updates call sites for new signature and adds tests for progress callback threading.
cmd/entire/cli/summarize/claude.go	Prefers streaming generation when available and forwards progress callback into the agent.
cmd/entire/cli/summarize/claude_test.go	Adds test coverage for streaming preference/fallback in `ClaudeGenerator`.
cmd/entire/cli/strategy/manual_commit_condensation.go	Updates summarization call to match new `GenerateFromTranscript` signature.
cmd/entire/cli/agent/agent.go	Introduces progress phase/types and the `StreamingTextGenerator` interface.
cmd/entire/cli/agent/capabilities.go	Adds capability detection helper `AsStreamingTextGenerator` and declared-cap field.
cmd/entire/cli/agent/capabilities_test.go	Adds tests validating `AsStreamingTextGenerator` behavior.
cmd/entire/cli/agent/claudecode/generate.go	Switches Claude Code generation to streaming NDJSON parsing and emits progress callbacks.
cmd/entire/cli/agent/claudecode/response.go	Adds NDJSON stream event types and `streamClaudeResponse` parser.
cmd/entire/cli/agent/claudecode/response_test.go	Replaces legacy response parsing tests with stream parser tests using fixtures.
cmd/entire/cli/agent/claudecode/claude_test.go	Adds streaming-generation tests and delegation test for `GenerateText`.
cmd/entire/cli/agent/claudecode/testdata/stream_success.jsonl	Adds real-world success fixture for stream-json NDJSON.
cmd/entire/cli/agent/claudecode/testdata/stream_error_404.jsonl	Adds error fixture (`is_error:true`, 404) for stream-json NDJSON.

Copilot · 2026-04-16T04:31:23Z

+			case <-ticker.C:
+				ts := lastActivity.Load()
+				if ts > 0 && time.Since(time.Unix(0, ts)) >= idle {
+					fired.Store(true)
+					cancel()
+					return
+				}
+				// ts == 0 means no activity yet; keep waiting.


startIdleWatchdog treats lastActivity == 0 as “no activity yet; keep waiting”, which means the watchdog will never fire if the subprocess produces no stream events at all (the default non-TTY safety deadline becomes ineffective). Consider initializing lastActivity to time.Now() before starting the watchdog, or treating ts==0 as “idle since start” and canceling after idle elapses.

cursor

Cursor Bugbot has reviewed your changes and found 3 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 33a095c. Configure here.}

cursor · 2026-04-16T04:33:30Z

+	Type    string         `json:"type"` // "message_start" | "content_block_delta" | "message_delta" | ...
+	Delta   *StreamDelta   `json:"delta,omitempty"`
+	Message *StreamMessage `json:"message,omitempty"`
+	TTFTms  int            `json:"ttft_ms,omitempty"`


TTFTms field mapped to wrong JSON nesting level

Medium Severity

The TTFTms field is defined on StreamInnerEvent (the nested event JSON object), but the real Claude CLI output places ttft_ms at the outer NDJSON line level alongside session_id and uuid. The captured fixture testdata/stream_success.jsonl confirms this: ttft_ms sits at the top level, not inside event. Since StreamEvent has no TTFTms field, the value is silently dropped during deserialization, so ev.Event.TTFTms is always 0 in production. The synthetic test fixture in claude_test.go incorrectly puts ttft_ms inside event, masking the bug.

Additional Locations (2)

cmd/entire/cli/agent/claudecode/generate.go#L137-L138

cmd/entire/cli/agent/claudecode/claude_test.go#L35-L36

^{Reviewed by Cursor Bugbot for commit 33a095c. Configure here.}

cursor · 2026-04-16T04:33:31Z

+					cancel()
+					return
+				}
+				// ts == 0 means no activity yet; keep waiting.


Idle watchdog never fires without initial stream activity

Medium Severity

The idle watchdog only fires when ts > 0, but lastActivity starts at its zero value (0). If the Claude CLI subprocess hangs before producing any stream events, lastActivity is never updated, so the watchdog never cancels the context. In the non-interactive (CI) code path — where the idle watchdog is the sole timeout mechanism — this causes the process to hang indefinitely. The old code had a hard 30-second wall-clock default that would have caught this.

^{Reviewed by Cursor Bugbot for commit 33a095c. Configure here.}

cursor · 2026-04-16T04:33:31Z

-		timeoutDuration = time.Until(deadline)
+func generateCheckpointAISummary(ctx context.Context, errW io.Writer, scopedTranscript []byte, filesTouched []string, agentType types.AgentType) (*checkpoint.Summary, error) {
+	// Create progress writer so the user sees phase updates on stderr.
+	progressWriter := newSummaryProgressWriter(errW)


Nil errW causes panic in progress writer

Medium Severity

newSummaryProgressWriter(errW) is called unconditionally, but the caller generateCheckpointSummary has an existing if errW != nil guard (line 488), indicating errW can be nil. When errW is nil and progress events arrive, handle calls fmt.Fprintln(s.w, line) on a nil io.Writer, which panics. The old code always guarded writes to errW with a nil check; the new progress writer path does not.

Additional Locations (1)

cmd/entire/cli/explain.go#L2099-L2106

^{Reviewed by Cursor Bugbot for commit 33a095c. Configure here.}

Switches `entire explain --generate` from silent wait to real-time stream-driven progress. Independent of typed Claude error classification (#963). ### User-visible changes **Interactive (TTY):** live progress phases with in-place updates Generating checkpoint summary... (transcript: 47.0 KB) → Sending request to Anthropic... → Anthropic responded (TTFT 1.9s, 35.9k cached input tokens) -- generating... → Writing summary... (~1.2k tokens) ✓ Summary generated (3.1s, 1.9k output tokens) ✓ Summary generated and saved **Non-TTY (CI):** same lines, one per event (no in-place updates) **ACCESSIBLE=1:** one line per event, no ANSI escape sequences ### Architecture - `agent.StreamingTextGenerator` optional capability interface with `AsStreamingTextGenerator` helper. Plain-error contract — typed error classification will be wired separately (see #963). - Claude Code implements it via `GenerateTextStreaming` in a new file (`generate_streaming.go`), leaving the existing `GenerateText` path untouched. Falls back internally to `--output-format json` when the CLI rejects streaming flags. - `summaryProgressWriter` renders `GenerationProgress` phases using existing `statusStyles` for consistent styling with `entire status`. - TTY-gated deadline: no deadline interactive (Ctrl+C is the timeout); idle-based 5min watchdog non-TTY with a 30min wall-clock cap. - `ProgressFn` threads through `summarize.GenerateFromTranscript` and `ClaudeGenerator`; preserved as nil for the condensation path. ### Scope note This PR intentionally does not map Claude auth/rate-limit/config errors to actionable messages — that's #963's typed-error work. When #963 lands on main, a small follow-up will wire typed errors into the streaming path. Users hitting stream errors in the meantime see a generic "claude CLI reported error: <text>" (HTTP status included when present). ### Tests - Stream parser: success + error fixtures, malformed-line resilience, reader errors - GenerateTextStreaming: phase emission, envelope error surfacing, legacy fallback, context cancellation - Progress writer: all phases, Generating dedup, lastActivity tracking - Idle watchdog: stale-fire, bump-prevents-fire, zero-idle noop - Deadline resolver: interactive no-deadline vs non-interactive idle Entire-Checkpoint: 56ee3789dcea

Copilot AI review requested due to automatic review settings April 16, 2026 04:26

Copilot started reviewing on behalf of peyton-alt April 16, 2026 04:26 View session

Copilot AI reviewed Apr 16, 2026

View reviewed changes

cursor bot reviewed Apr 16, 2026

View reviewed changes

peyton-alt changed the title ~~feat: stream-driven progress and TTY-gated deadline for explain --generate~~ feat: stream-driven progress for explain --generate Apr 16, 2026

peyton-alt force-pushed the feat/explain-stream-progress branch from 33a095c to 5e80233 Compare April 16, 2026 17:05

peyton-alt force-pushed the feat/explain-summary-observability branch from cd538a4 to 92449c4 Compare April 16, 2026 17:19

peyton-alt force-pushed the feat/explain-stream-progress branch 3 times, most recently from 3a3b257 to c6c0c5a Compare April 17, 2026 01:19

peyton-alt force-pushed the feat/explain-stream-progress branch from c6c0c5a to 9c500a7 Compare April 17, 2026 02:22

peyton-alt changed the base branch from feat/explain-summary-observability to main April 17, 2026 02:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: stream-driven progress for explain --generate#964

feat: stream-driven progress for explain --generate#964
peyton-alt wants to merge 1 commit intomainfrom
feat/explain-stream-progress

peyton-alt commented Apr 16, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Apr 16, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor bot left a comment

Uh oh!

cursor bot Apr 16, 2026

Uh oh!

cursor bot Apr 16, 2026

Uh oh!

cursor bot Apr 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

2 participants

Conversation

peyton-alt commented Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What users see now

Architecture

Scope note (re: #963)

Test plan

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Apr 16, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor bot Apr 16, 2026

Choose a reason for hiding this comment

TTFTms field mapped to wrong JSON nesting level

Uh oh!

cursor bot Apr 16, 2026

Choose a reason for hiding this comment

Idle watchdog never fires without initial stream activity

Uh oh!

cursor bot Apr 16, 2026

Choose a reason for hiding this comment

Nil errW causes panic in progress writer

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

2 participants

peyton-alt commented Apr 16, 2026 •

edited

Loading