fix(analyze-hook): AI must canonicalize hook name and description#566
Merged
Conversation
Submitter-provided name and description fields flowed verbatim into the registry because assemble_hook.py's priority chain was submitter > claude > contractName, and classify-hook.md explicitly told the AI to defer to the submitter. This let an issue creator put promotional, false-audit, or affiliation claims in name/description and have them land in the registry unchanged. Changes: - classify-hook.md §6-7: rewrite as acceptance gates. AI always derives a baseline from source; submitter suggestion is adopted only if factually consistent with the source and free of promotional / audit / safety / affiliation language. Otherwise AI's text wins. - classify-hook.md §8 (new): structured `warnings` array. Records rejected submitter suggestions and flag mismatches. Replaces the prior hack of burying flag discrepancies inside the description. - analyze-hook.yml: structured-output schema gains required `warnings: string[]` (max 20 entries, 300 chars each). - assemble_hook.py: submitter removed from canonical name/description chain. Claude is canonical; contractName is defense-in-depth fallback. Warnings render as bullets in the PR body's Warnings section. - review-hook.md §2d: second-line PR review now applies the same acceptance criteria so it would reject any promotional name/description that slipped past analyze-hook. - test_assemble_hook.py: tests flipped to assert claude > contractName, with submitter text never adopted directly; warnings rendering covered. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Zizmor's github-app audit flagged the create-github-app-token step as high-severity because the token inherited blanket installation permissions. The token is only used by the "Create PR" step (push branch, gh pr create, gh pr merge --auto), so scope it explicitly to contents:write and pull-requests:write. This finding existed before but only surfaced now because the zizmor workflow only audits files changed in the PR diff, and this PR is the first one to modify analyze-hook.yml. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes a trust gap in the analyze-hook workflow: submitter-provided
nameanddescriptionfields from the GitHub issue form were flowing verbatim into the registry. A submitter could put promotional, false-audit, or affiliation claims (e.g. "Uniswap Official — Audited by Trail of Bits") in those fields and have them land unchanged inhooks/<chain>/<address>.json.Root cause was two-sided:
scripts/assemble_hook.pypriority chain wassubmitter > claude > contractName— submitter won unconditionally if non-empty..claude/prompts/classify-hook.md§6-7 explicitly told the AI to defer to the submitter. So even if you flipped the priority, Claude would obediently echo the submitter's text back in its own structured output. The prompt was load-bearing.This PR fixes both, plus tightens the second-line PR review and adds a structured warnings channel.
What changed
.claude/prompts/classify-hook.mdname) rewritten as an acceptance gate. AI always derives a baseline name from the verified source (contractName+ project-qualified labels). Submitter's suggestion is adopted only if all of: factually consistent with the source, free of promotional / audit / safety / affiliation language ("Official", "Audited", "Safe", "Trusted", brand names not in source), and ≤100 chars. Otherwise AI's text wins.description) same model. AI drafts from source; submitter version adopted only if every claim is substantiated by the Solidity logic.warnings— structured array surfacing rejected submitter suggestions andgetHookPermissions()vs. address-bitmask mismatches. Replaces the prior practice of burying flag discrepancies inside the description (which made them invisible).submission.jsonname/descriptionas untrusted by default..github/workflows/analyze-hook.ymlwarnings: string[]field (max 20 entries, 300 chars each).scripts/assemble_hook.pyname:claude_output["name"]→source_meta["contractName"]→"UnnamedHook"(last two are defense-in-depth)description:claude_output["description"]→""Nonewhen empty)..claude/prompts/review-hook.md§2dscripts/test_assemble_hook.pytest_assemble_claude_name_always_wins_over_submitter,test_assemble_falls_back_to_contract_name_when_claude_empty,test_assemble_description_empty_when_claude_empty,test_generate_pr_body_renders_warnings_as_bullets.What this does NOT defend against
Subtle / plausibly-correct misframings (e.g. "Liquidity Manager" for a one-pair AMM). The acceptance criteria target unverifiable claims, not nuanced framing. The
review-hook.ymlsecond-line check is meant to catch this, and rejected suggestions now appear in the PR body's Warnings section as an audit trail for the human approver. The merge gate is still load-bearing.Test plan
python -m pytest scripts/— all 48 tests pass locallyanalyze-hook.ymlparses and includeswarningsin required"Uniswap Official Hook") and confirm the AI rejects it, usesContractName, and adds a warning bullet to the PR bodyNone🤖 Generated with Claude Code