chore(signals): sync prod snapshots to run the inbox ranking training job locally - #91322
Conversation
andrewm4894
added
the
skip-agent-review
|
😎 Merged successfully - details. |
🤖 CI report
|
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
/trunk merge |
Prompt To Fix All With AI### Issue 1
products/signals/dags/inbox_ranking/README.md:83
**Internal access details exposed**
The new public workflow names the internal secrets role, profile, access system, production bucket, and reader-secret identifier. Move these operational details to internal documentation so they are not exposed publicly or presented to external readers as usable setup instructions.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.Reviews (1): Last reviewed commit: "chore(signals): sync prod snapshots to r..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
|
A Codex review is actively running on the latest commit (started 2026-08-29T22:11:43 per the discussion comment, no completed verdict yet) — the earlier Codex/Greptile feedback only covers an older commit, so the current head hasn't been cleared. Auto-approving over an in-flight review risks conflicting with findings that land moments later.
Gate mechanics and policy version
|
… and mirror deletions in the snapshot sync Claude-Session: https://claude.ai/code/session_01DciwXDou2MVrmrGupjjJCh
andrewm4894
added
stamphog
|
/trunk merge |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2d9d5465f9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
/trunk merge |
- prefer OBJECT_STORAGE_ENDPOINT over the localhost default for the local hop - turn xtrace off before the reader credential enters the shell - list partitions from parquet files, not directories a scrub leaves empty - README: check .env.local for INBOX_RANKING_DATASET_S3_BUCKET, pick the newest common partition, note the dev bucket is unauthenticated Claude-Session: https://claude.ai/code/session_01Y9VyXUkutac4vDhgEsDjDT
|
/trunk merge |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ee66b05def
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| SECRETS_PROFILE="${SECRETS_PROFILE:-prod-us-secrets}" | ||
| PROD_BUCKET="${INBOX_RANKING_PROD_BUCKET:-posthog-inbox-ranking-dataset-prod-us}" | ||
| PREFIX="${INBOX_RANKING_DATASET_S3_PREFIX:-inbox_ranking}" |
There was a problem hiding this comment.
When local Dagster uses a supported custom INBOX_RANKING_DATASET_S3_PREFIX, passing that setting to this script also changes the production source path, so it looks for snapshots under the local-only prefix and a fresh sync finds nothing. If the value exists only in .env.local, the inverse mismatch occurs: the script uploads under inbox_ranking while Dagster reads the custom prefix. Use separate source and destination prefix settings so both configurations can stay aligned.
Useful? React with 👍 / 👎.
| AWS_ACCESS_KEY_ID=object_storage_root_user AWS_SECRET_ACCESS_KEY=object_storage_root_password \ | ||
| aws --endpoint-url http://localhost:19000 s3 cp s3://posthog/inbox_ranking/inbox_ranking_models/v1/dt=2026-08-25/metadata.json - |
There was a problem hiding this comment.
When a developer uses supported custom object-storage credentials, endpoint, bucket, or dataset prefix, the sync and training run can succeed but this command still queries the default SeaweedFS instance with its default credentials and path. It then fails authentication or reports a missing result from the wrong store; construct this command from the same OBJECT_STORAGE_* and prefix values used by the workflow.
AGENTS.md reference: AGENTS.md:L199-L199
Useful? React with 👍 / 👎.
2 participants
Problem
Changes
products/signals/dags/inbox_ranking/bin/sync_snapshots_local.shcopies theinbox_report_stateandinbox_report_labelspartitions from the prod bucket into local SeaweedFS, through a disk cache, and prints the days present in both tables.posthogbucket. Nothing touches the prod prefix.metadata.json, and the stuck-queue gotcha (runs leftSTARTEDby a killeddagster devcount against the local cap of 10).How did you test this code?
inbox_ranking_training_jobfordt=2026-08-25from local Dagster: 20 snapshots present, the run completed in about 40 seconds and wrote a candidate with four readable heads.Automatic notifications
Docs update
The README section is the doc.
🤖 Agent context
Autonomy: Human-driven (agent-assisted)
Claude Code (Fable 5), session. Skills invoked:
/writing-pr-descriptionsand theinbox-rankingteam skill from the skills store. The prod-side sync (SSO login, secret read,aws s3 sync) ran from the user's terminal; the agent ran the local half and the Dagster run.