fix(validator): 0.0 veto labels are silently overridden by co-applied reward labels - #1662
Open
RealDiligent wants to merge 1 commit into
Open
fix(validator): 0.0 veto labels are silently overridden by co-applied reward labels#1662RealDiligent wants to merge 1 commit into
RealDiligent wants to merge 1 commit into
Conversation
…el override them Repositories use a `0.0` entry in `label_multipliers` as a hard veto — a rejection / void marker. The live registry has these on seven repos: `eval:REJECT`, `dataset:REJECT`, `invalid-pr`, `duplicate-pr`, `slop`, `kata:invalid`, `kata:defeat:*`, `eval:none`, ... `resolve_highest_label_multiplier` took `max()` across a PR's matching labels, so a veto label applied alongside a reward label was silently ignored. A maintainer could never revoke a PR that already carried a reward label: gt-imagent round-winner + invalid-pr -> 10.00 (should be 0) sparkinfer eval:L + eval:REJECT -> 2.50 (should be 0) sparkdistill dataset:xl + dataset:REJECT -> 4.00 (should be 0) loopover gittensor:priority + slop -> 1.50 (should be 0) kata kata:winner:s1 + kata:invalid -> 0.70 (should be 0) A `0.0`-configured label now vetoes: if any matched label resolves to `0.0` the label multiplier is `0.0`, regardless of a co-applied higher label. Among purely positive labels the highest still wins, so normal scoring is unchanged. The veto is also detected within a single label that matches both a `0.0` pattern and a positive pattern (e.g. an `eval:*` catch-all plus `eval:REJECT`). Both the OSS PR path and the issue-discovery solving-PR path resolve labels through this helper, so both are fixed. A low positive multiplier (e.g. 0.05) is a reward, not a veto, and is unaffected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Repositories use a
0.0entry inlabel_multipliersas a hard veto — a rejection / void marker that should zero a PR's score.resolve_highest_label_multipliertookmax()across a PR's matching labels, so a veto label applied alongside a reward label was silently ignored. A maintainer could never revoke a PR that had already earned a reward label.This is live in the committed
master_repositories.jsonon seven repos (eval:REJECT,dataset:REJECT,invalid-pr,duplicate-pr,slop,kata:invalid,kata:defeat:*,eval:none,type:*, …).Root cause
gittensor/validator/oss_contributions/label_resolution.py:max()treats a0.0veto as just "the lowest value", so any positive label out-ranks it. A0.0label is categorically different from a low reward — it means disqualified, not low-value.Fix
A
0.0-configured label now vetoes: if any matched label resolves to0.0, the label multiplier is0.0regardless of a co-applied higher label. Among purely positive labels the highest still wins, so normal scoring is unchanged. The veto is also detected within a single label that matches both a0.0pattern and a positive pattern (e.g. aneval:*catch-all plus an expliciteval:REJECT). A low positive multiplier (e.g.mult:contribution: 0.05) is a reward, not a veto, and is unaffected.Both the OSS PR path (
resolve_trusted_label_multiplier→ this helper) and the issue-discovery solving-PR path (_mirror_issue_for_scoring→ same helper) resolve labels here, so both are fixed by the one change.Evidence
Resolved multiplier for a reward + veto label pair, against the committed
master_repositories.json(each row is a maintainer applying a documented rejection marker to a PR that already carries a reward label):round-winner+invalid-prround-winner+duplicate-preval:L+eval:REJECTdataset:xl+dataset:REJECTgittensor:priority+slopkata:winner:s1+kata:invalidRelated Issues
None open.
Type of Change
Testing
New
tests/validator/test_label_veto.py(9 cases): veto-beats-reward for the real registry shapes, order-independence, the single-label-matches-both case, and three guards that non-veto behaviour is unchanged (highest positive wins, a low positive is not a veto, default multiplier still applies). The six veto assertions were verified to fail against the current helper and pass with the fix.Full CI on this branch: 969 tests passed;
ruff lint,ruff format,pyright,vultureall pass with no auto-format changes.No CLI output changes, so the CONTRIBUTING before/after-screenshot requirement does not apply.
Risk / tradeoffs
Behaviour change, and that is the intent: PRs that carry both a reward and a
0.0veto label now score0.0instead of the reward. This only affects PRs where a veto label is actually present — purely positive label sets are unchanged (verified by the "highest positive still wins" test). A repo that genuinely wanted an overridable low weight would configure a small positive value, not0.0; that path is preserved.Checklist
🤖 Generated with Claude Code