Skip to content

feat(cli): Add GitHub Actions annotations for CLI errors and warnings#15683

Open
Swimburger wants to merge 3 commits intomainfrom
niels/cli/gha-annotations
Open

feat(cli): Add GitHub Actions annotations for CLI errors and warnings#15683
Swimburger wants to merge 3 commits intomainfrom
niels/cli/gha-annotations

Conversation

@Swimburger
Copy link
Copy Markdown
Member

@Swimburger Swimburger commented May 4, 2026

Summary

Surface CLI errors and warnings as GitHub Actions annotations when running under GHA, so failures appear inline in the PR/run UI instead of being buried in the raw step log.

The screenshot motivating this: fern automations generate produced an "SDK generation failed (2/8 succeeded, 4 skipped)" summary table, but the GHA Annotations panel showed only the generic Process completed with exit code 1 — the user had to scroll the raw log to find which generator failed and why.

How it works

Two layers, both gated on GITHUB_ACTIONS=true:

  1. Generic logger hook (in @fern-api/cli-logger) — every logger.error / logger.warn across the CLI also emits a ::error:: / ::warning:: workflow command on stdout. Covers fern check, fern generate, validation errors, etc. Status-only logs (omitOnTTY: true, e.g. the per-task "Failed." line) are filtered out so they don't burn through GitHub's per-step cap.

  2. Structured per-generator annotations for fern automations generate — emitted from the GeneratorRunCollector after the run, anchored on the exact line in generators.yml (file=fern/apis/foo/generators.yml,line=42) so the annotation appears inline on the file in the PR's "Files changed" tab. Title format is <generator> failed (group=<group>, api=<api>) — explicit key=value qualifiers because generator names like fernapi/fern-python-sdk already contain /.

The structured emitter wraps its run in withSuppressedLoggerAnnotations(...) (a scoped runner with try/finally restoration) so per-generator failures don't get annotated twice — once with raw text from logger.error, once with the rich file= / line= metadata.

When the failure count exceeds GitHub's per-step cap (10), the structured emitter still emits one ::error:: per failure (GHA drops the overflow itself) and appends a trailing ::warning:: like "3 additional generator failures were hidden by GitHub's per-step annotation cap. See the step summary table for the full list." so reviewers know what's missing.

Related fix bundled in: auto-retry on 429 in automations mode

While verifying this PR with a real GHA run, every generator call hit 429 Too Many Requests from Fiddle. The CLI already has a --retry-rate-limited flag (bounded: 3 attempts, 2s→120s exponential backoff with jitter), but fern automations generate was passing false. Automation runs are unattended — failing the run and asking a human to re-trigger with a flag isn't useful — so this PR flips that default to true for automations only. fern generate (interactive) still requires the explicit flag. The retry warnings emitted during backoff are absorbed by withSuppressedLoggerAnnotations, so the panel stays clean and only the final post-retry failure (if any) becomes a ::error:: annotation.

Sanitization (per the GHA workflow command spec)

  • ANSI escape codes (chalk colors) stripped from bodies and property values — they'd render as literal [31m...[39m noise in the panel.
  • Embedded newlines in bodies encoded as %0A so multi-line errors stay one workflow command (GHA renders them as a multi-line annotation).
  • Property values escape , : \r \n % per spec so they can't break out of the property list.

Out of scope

  • fern check validation errors annotate via the generic logger hook with no file= / line= metadata. Anchoring those properly would need structured emission from a check-command emitter, not from logger.error — separate design pass.
  • The 429s observed during empirical verification are independent of this PR's changes; the request/concurrency path is unchanged. Real fixes (client-side throttling on the createJob fan-out, or revisiting Fiddle's rate limit) are follow-ups.

Test plan

  • pnpm turbo run test --filter @fern-api/cli-logger --filter @fern-api/cli — 33 + 637 tests pass
  • pnpm check:fix clean
  • End-to-end smoke test: simulated logger and structured annotation paths with GITHUB_ACTIONS=true and verified output matches expectations across env-var gate, suppression scope (incl. nested), level filter, omitOnTTY filter, prefix→title, and the cap-overflow warning
  • Empirical verification on a real GHA run — published 5.9.0-rc.0, ran automations generate against a multi-workspace fern config, and confirmed: rich annotations rendered inline on generators.yml lines, %2C / %3A decoded correctly in titles, and no duplicate generic+structured annotations. Initial run also exposed the 429 issue, which motivated the retry default flip above.

- Introduced `githubAnnotations.ts` to handle rendering of GitHub Actions annotations.
- Implemented structured error and warning annotations for CLI commands, particularly for `fern automations generate`.
- Enhanced `GeneratorRunResult` to include workspace-relative paths and line numbers for better annotation accuracy.
- Added tests for rendering annotations and handling workspace-relative paths.
- Suppressed generic logger annotations during structured annotation emission to avoid duplication.
- Emitted a trailing warning when the number of failures exceeds GitHub's annotation cap.
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 4, 2026

🌱 Seed Test Selector

Select languages to run seed tests for:

  • Python
  • TypeScript
  • Java
  • Go
  • Ruby
  • C#
  • PHP
  • Swift
  • Rust
  • OpenAPI

How to use: Click the ⋯ menu above → "Edit" → check the boxes you want → click "Update comment". Tests will run automatically and snapshots will be committed to this PR.

@Swimburger Swimburger changed the title feat: Add GitHub Actions annotations for CLI errors and warnings feat(cli): Add GitHub Actions annotations for CLI errors and warnings May 4, 2026
@Swimburger Swimburger marked this pull request as ready for review May 4, 2026 21:35
@Swimburger Swimburger requested a review from amckinney as a code owner May 4, 2026 21:35
Copy link
Copy Markdown

@claude claude Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review to trigger a review and subscribe this PR to future pushes, or @claude review once for a one-time review.

Tip: disable this comment in your organization's Code Review settings.

@Swimburger Swimburger enabled auto-merge (squash) May 4, 2026 21:35
devin-ai-integration[bot]

This comment was marked as resolved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants