Skip to content

docs: document posthog-js release flow #4591

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
dustinbyrne wants to merge 3 commits into
base: main
Choose a base branch
from
+172 −25
Open

docs: document posthog-js release flow #4591

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
52687bd
docs: document S3 release recovery
dustinbyrne Aug 20, 2026
232d7ee
fix(ci): add recovery approval button
dustinbyrne Aug 20, 2026
afe9dc1
docs: clarify S3 release recovery
dustinbyrne Aug 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 43 additions & 7 deletions .github/workflows/recover-s3-release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ jobs:
runs-on: ubuntu-latest
outputs:
source-sha: ${{ steps.source.outputs.source-sha }}
source-version: ${{ steps.source.outputs.source-version }}
toolbar-sha: ${{ steps.toolbar.outputs.toolbar-sha }}
toolbar-matrix: ${{ steps.regions.outputs.toolbar-matrix }}
upload-matrix: ${{ steps.regions.outputs.upload-matrix }}
Expand Down Expand Up @@ -172,7 +173,10 @@ jobs:
fi
fi

echo "source-sha=$source_sha" >> "$GITHUB_OUTPUT"
{
echo "source-sha=$source_sha"
echo "source-version=$source_version"
} >> "$GITHUB_OUTPUT"
echo "✓ source: posthog-js@$source_version from $source_sha"
echo "✓ S3 target: /static/$TARGET_VERSION/"
echo "✓ publish to npm: $PUBLISH_TO_NPM"
Expand Down Expand Up @@ -242,14 +246,15 @@ jobs:
PUBLISH_TO_NPM: ${{ inputs.publish_to_npm }}
REGION: ${{ inputs.region }}
SOURCE_SHA: ${{ steps.source.outputs.source-sha }}
SOURCE_VERSION: ${{ steps.source.outputs.source-version }}
TARGET_VERSION: ${{ inputs.target_version }}
TOOLBAR_SHA: ${{ steps.toolbar.outputs.toolbar-sha }}
UPDATE_LATEST_ALIASES: ${{ inputs.update_latest_aliases }}
run: |
{
echo "### S3 recovery approval requested for posthog-js@$TARGET_VERSION"
echo
echo "- SDK source: [\`$SOURCE_SHA\`](https://github.com/PostHog/posthog-js/commit/$SOURCE_SHA)"
echo "- SDK source: [\`$SOURCE_SHA\`](https://github.com/PostHog/posthog-js/commit/$SOURCE_SHA) (\`posthog-js@$SOURCE_VERSION\`)"
echo "- Toolbar source: [\`$TOOLBAR_SHA\`](https://github.com/PostHog/posthog/commit/$TOOLBAR_SHA)"
echo "- Regions: \`$REGION\`"
echo "- Update latest CDN aliases: \`$UPDATE_LATEST_ALIASES\`"
Expand All @@ -274,24 +279,55 @@ jobs:
SLACK_CHANNEL_ID: ${{ vars.SLACK_APPROVALS_CLIENT_LIBRARIES_CHANNEL_ID }}
SLACK_USER_GROUP_ID: ${{ vars.GROUP_CLIENT_LIBRARIES_SLACK_GROUP_ID }}
SOURCE_SHA: ${{ needs.validate.outputs.source-sha }}
SOURCE_VERSION: ${{ needs.validate.outputs.source-version }}
TARGET_VERSION: ${{ inputs.target_version }}
TOOLBAR_SHA: ${{ needs.validate.outputs.toolbar-sha }}
UPDATE_LATEST_ALIASES: ${{ inputs.update_latest_aliases }}
run: |
set -euo pipefail
text=$(printf '%s\n' \
details=$(printf '%s\n' \
":bufo-hotdog-rocket: <!subteam^$SLACK_USER_GROUP_ID> S3 recovery approval needed for posthog-js@$TARGET_VERSION" \
"• SDK source: <https://github.com/PostHog/posthog-js/commit/$SOURCE_SHA|$SOURCE_SHA>" \
"• SDK source: <https://github.com/PostHog/posthog-js/commit/$SOURCE_SHA|$SOURCE_SHA> (posthog-js@$SOURCE_VERSION)" \
"• Toolbar source: <https://github.com/PostHog/posthog/commit/$TOOLBAR_SHA|$TOOLBAR_SHA>" \
"• Regions: $REGION" \
"• Update latest aliases: $UPDATE_LATEST_ALIASES" \
"• Force overwrite immutable assets: $FORCE_OVERWRITE" \
"• Publish and finalize on npm: $PUBLISH_TO_NPM" \
"<$ACTION_URL|Review and approve on GitHub>")
"• Publish and finalize on npm: $PUBLISH_TO_NPM")
fallback_text=$(printf '%s\n%s' "$details" "<$ACTION_URL|Review and approve on GitHub>")
payload=$(jq -n \
--arg action_url "$ACTION_URL" \
--arg channel "$SLACK_CHANNEL_ID" \
--arg details "$details" \
--arg text "$fallback_text" \
'{
channel: $channel,
text: $text,
blocks: [
{
type: "section",
text: { type: "mrkdwn", text: $details }
},
{
type: "actions",
elements: [
{
type: "button",
text: {
type: "plain_text",
text: "Approve/Reject on GitHub",
emoji: true
},
url: $action_url,
style: "primary"
}
]
}
]
}')
response=$(curl -sS --connect-timeout 10 --max-time 30 -X POST 'https://slack.com/api/chat.postMessage' \
-H "Authorization: Bearer $SLACK_BOT_TOKEN" \
-H 'Content-Type: application/json; charset=utf-8' \
--data "$(jq -n --arg channel "$SLACK_CHANNEL_ID" --arg text "$text" '{channel: $channel, text: $text}')")
--data "$payload")
if [ "$(printf '%s' "$response" | jq -r '.ok')" != 'true' ]; then
printf '%s\n' "$response" >&2
exit 1
Expand Down
147 changes: 129 additions & 18 deletions RELEASING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,34 +1,145 @@
## Releases

Releases are managed with changeset, you can find more information on the [changeset repository](https://github.com/changesets/changesets).
Releases are managed with [Changesets](https://github.com/changesets/changesets).

Before submitting a PR, create a changeset by running:
Before submitting a PR with a publishable change, create a changeset by running:

```
```bash
pnpm changeset
```

CLI will prompt questions about the changes you've made and will generate a changeset file for you.
The CLI will ask which packages changed, what version bump they need, and how the change should appear in the changelog.

When a PR containing a changeset is merged to `main`, the [`Release` workflow](https://github.com/PostHog/posthog-js/actions/workflows/release.yml) automatically:

1. Waits for approval and creates the version-bump commit on `main`.
2. If the version of the `posthog-js` browser package changed, builds and uploads its browser and toolbar assets to the US and EU S3 buckets.
3. Publishes the changed packages to npm.
4. Creates package tags and GitHub releases.
5. Dispatches downstream dependency upgrades and reports the result in Slack.

### Browser S3/CDN gate

The browser S3 release happens before npm publishing. When `posthog-js` changes, every npm publish job is gated on:

- successfully identifying the new browser version;
- building the browser distribution;
- building the toolbar for both regional asset hosts; and
- successfully uploading to both the US and EU S3 buckets.

If any of that work fails, no package from the coordinated npm release is published. If `posthog-js` did not change, the S3 jobs are skipped and npm publishing proceeds normally.

A stable browser release uploads:

- immutable versioned assets under `/static/<version>/`;
- mutable major-version aliases under `/static/<major>/`; and
- top-level compatibility aliases under `/static/`.

Prerelease versions receive only immutable versioned assets. The workflow does not purge CDN caches, so mutable aliases can continue serving cached bytes until their normal TTL expires.
Comment thread
dustinbyrne marked this conversation as resolved.
Outdated

PostHoggers can join [`#alerts-posthog-js`](https://posthog.slack.com/archives/C07HTMN9X47) for release workflow failure notifications.

## Manual S3 recovery

S3 recovery is an exceptional path for repairing a failed or incomplete browser release. If the failure was transient and the original run is still safe to resume, first retry its failed jobs. Use recovery when the original run cannot be completed safely, for example when one regional upload succeeded before the other failed or an existing release is missing immutable assets.

Recovery always rebuilds and uploads the browser SDK and toolbar. It is not an npm-only, alias-only, artifact-promotion, rollback, or S3 deletion mechanism.
Comment thread
dustinbyrne marked this conversation as resolved.
Outdated

### Starting recovery

1. Open the [`Release` workflow](https://github.com/PostHog/posthog-js/actions/workflows/release.yml).
2. Select **Run workflow** and run it from `main`.
3. Set `target_version` and review the recovery inputs below.
4. Confirm that **Validate recovery inputs** resolved the intended SDK version, SDK commit, and toolbar commit.
5. Have an eligible reviewer approve the protected `S3 Recovery` environment.
6. Monitor every requested regional build and upload, then verify the resulting assets.

After validation, the workflow notifies `#approvals-client-libraries` with the target, validated source commits, regions, and selected recovery options. Slack delivery is best-effort and does not bypass the protected approval.

### Recovery inputs

When a PR containing a changeset is merged to `main`, the release workflow will automatically:
| Input | Default | Behavior |
| ----------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `target_version` | empty | Required S3 destination version, for example `1.418.2`. |
| `source_sha` | empty | Full `PostHog/posthog-js` commit to build. When empty, resolves the `posthog-js@<target_version>` tag. A version that never received a tag requires its full version-bump commit SHA. The commit must be current `main` or an ancestor. |
| `toolbar_sha` | empty | Full `PostHog/posthog` commit used to build the toolbar. When empty, uses current `master`. The commit must be current `master` or an ancestor. Prefer the toolbar commit recorded in the original failed run when reproducing that release. |
Comment thread
dustinbyrne marked this conversation as resolved.
Outdated
| `region` | `all` | `all`, `us`, or `eu`. Updating latest aliases or publishing to npm requires `all`. |
| `update_latest_aliases` | `false` | Also updates `/static/` and `/static/<major>/`. Leave disabled for an older release or immutable-only repair. |
| `publish_to_npm` | `false` | After both S3 regions succeed, publish and finalize an unpublished current `posthog-js` version. This requires latest aliases, both regions, `NPM Release` approval, and OIDC. |
| `force_overwrite` | `false` | Allows existing immutable `/static/<version>/` objects to be replaced. By default the workflow refuses existing immutable assets and protects writes against races. Overwriting does not purge CDN caches. |

1. Bump versions based on changesets
2. Commit version updates directly to main
3. Publish packages to npm
4. Create GitHub releases
S3-only recovery permits the selected SDK package version to differ from `target_version` so an explicitly reviewed source can repair a destination path. The resolved source version appears in the approval summary and Slack message. Approve a mismatch only when it is intentional; otherwise the destination path would identify bytes from a different package version.

# for posthog-js browser sdk
Use `force_overwrite` only after checking which objects the earlier attempt created and confirming the selected SDK and toolbar commits. If immutable assets already exist, `force_overwrite=true` is required before the workflow writes anything, including aliases. Recovery still rebuilds and rewrites the immutable assets even when only the aliases are wrong.

When we run post-merge actions for the browser SDK, the release workflow publishes the package to npm and uploads the browser bundles to S3 for the CDN.
### Approval and publication safeguards

The CDN upload happens in `.github/workflows/release.yml` via the `upload-s3` job. For a new stable `posthog-js` version it uploads:
Before selected source code is built, recovery validates that:

- immutable versioned assets under `/static/<version>/`
- mutable major-version aliases under `/static/<major>/`
- top-level compatibility aliases under `/static/`
- the recovery workflow is running from `posthog-js@main`;
- the SDK source is current `PostHog/posthog-js@main` or an ancestor; and
- the toolbar source is current `PostHog/posthog@master` or an ancestor.

Prerelease versions only get immutable versioned assets.
An independent `S3 Recovery` approval is then required before checking out and executing the selected sources. Actual bucket access uses AWS OIDC under `S3 Upload`.

A mismatch can still happen if npm publish succeeds but the S3/CDN upload fails: npm users may get version N+1 while CDN/snippet users remain on version N or see incomplete CDN assets until the failed upload is retried. The release workflow sends a partial-release Slack warning for this case.
Optional npm finalization is allowed only when the source and target versions match, the source is the current version-bump commit on `main`, and neither the npm version nor release tag already exists. It requires both regions and latest aliases, waits for S3 success, and runs under the protected `NPM Release` environment with npm OIDC.

PostHoggers can join the [#alerts-posthog-js channel in Slack](https://posthog.slack.com/archives/C07HTMN9X47), which gets notified about release workflow failures.
### Regional failures

US and EU builds and uploads run independently. Requiring both regions before npm publishing prevents an npm-ahead-of-S3 release, but it does not make regional S3 writes atomic. One region can succeed before the other fails.

For an immutable-only retry, select only the failed region and leave latest aliases disabled. If a stable-alias update partially succeeds, inspect both regions and ask `#team-client-libraries` before retrying; alias updates require `region=all`.

### Verifying recovery

Verify representative immutable assets from both direct asset hosts:

```bash
set -euo pipefail
VERSION=1.418.2

for host in us-assets.i.posthog.com eu-assets.i.posthog.com; do
for file in array.js recorder.js toolbar.js; do
curl --fail --silent --show-error --output /dev/null \
"https://${host}/static/${VERSION}/${file}"
done
done
```

These requests verify availability, not the bytes held by every cache. After `force_overwrite`, inspect the upload jobs and compare a cache-busted response with the workflow's `s3-recovery-posthog-js-dist` artifact:

```bash
set -euo pipefail
RUN_ID=123456789
artifact_dir=$(mktemp -d)

gh run download "$RUN_ID" \
--repo PostHog/posthog-js \
--name s3-recovery-posthog-js-dist \
--dir "$artifact_dir"

for host in us-assets.i.posthog.com eu-assets.i.posthog.com; do
downloaded_file="$artifact_dir/${host}-array.js"
curl --fail --silent --show-error \
"https://${host}/static/${VERSION}/array.js?recovery-check=$(date +%s)" \
--output "$downloaded_file"
cmp "$artifact_dir/array.js" "$downloaded_file"
done
```

Immutable responses have long-lived cache headers, and recovery does not purge copies already held by browsers or CDN edges. A successful cache-busted comparison does not invalidate those older cached responses.

If latest aliases were updated, also verify `/static/array.js` and `/static/<major>/array.js` in both regions after allowing for their cache TTL.

If npm finalization was requested, verify npm, the tag, and the GitHub release:

```bash
set -euo pipefail

test "$(npm view "posthog-js@${VERSION}" version)" = "$VERSION"
test -n "$(git ls-remote --tags https://github.com/PostHog/posthog-js.git \
"refs/tags/posthog-js@${VERSION}")"
gh release view "posthog-js@${VERSION}" \
--repo PostHog/posthog-js \
--json tagName,url
```