Skip to content

fix: treat unverifiable submitter access as warning instead of error #452

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jwm4 merged 2 commits into from
May 20, 2026
Merged

fix: treat unverifiable submitter access as warning instead of error #452

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
246fde3
fix: treat unverifiable submitter access as warning instead of error
jwm4 May 20, 2026
10309b2
fix: simplify submitter access check per review feedback
jwm4 May 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions .github/workflows/leaderboard.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -155,13 +155,14 @@ jobs:
if [ "$HOST" = "github" ]; then
# GitHub: verify via API
ORG_REPO=$(echo "$REPO_URL" | sed 's|git@github.com:||' | sed 's|https://github.com/||' | sed 's|\.git$||')
if gh api "/repos/$ORG_REPO/collaborators/$SUBMITTER" 2>/dev/null; then
HTTP_CODE=$(gh api "/repos/$ORG_REPO/collaborators/$SUBMITTER" --silent 2>/dev/null; echo $?)
if [ "$HTTP_CODE" = "0" ]; then
echo "✅ $SUBMITTER is a collaborator on $ORG_REPO"
elif [ "$(gh api "/repos/$ORG_REPO" -q '.owner.login')" == "$SUBMITTER" ]; then
echo "✅ $SUBMITTER is the owner of $ORG_REPO"
else
echo "::error::$SUBMITTER does not have commit access to $ORG_REPO"
exit 1
echo "::warning::Cannot verify submitter access for $ORG_REPO (API returned 403). Manual review required."
echo "⚠️ Submitter access must be verified manually by maintainers."
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Outdated
Comment on lines +158 to +164

@coderabbitai coderabbitai Bot May 20, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Scope warning fallback to unverifiable cases only

This branch now warns on all non-success outcomes, so a submitter without verified access is no longer blocked. That broadens behavior beyond “unverifiable only” and weakens the gate.

Use a three-way outcome: collaborator/owner ✅, definitive no-access ❌ (fail), API-unverifiable ⚠️ (warn).

Suggested patch 
-            if gh api "/repos/$ORG_REPO/collaborators/$SUBMITTER" --silent 2>/dev/null; then
+            COLLAB_STATUS="$(gh api -i "/repos/$ORG_REPO/collaborators/$SUBMITTER" 2>/dev/null || true)"
+            COLLAB_HTTP="$(printf '%s\n' "$COLLAB_STATUS" | awk 'NR==1 {print $2}')"
+            if [ "$COLLAB_HTTP" = "204" ]; then
               echo "✅ $SUBMITTER is a collaborator on $ORG_REPO"
             elif [ "$(gh api "/repos/$ORG_REPO" -q '.owner.login')" == "$SUBMITTER" ]; then
               echo "✅ $SUBMITTER is the owner of $ORG_REPO"
+            elif [ "$COLLAB_HTTP" = "404" ]; then
+              echo "::error::$SUBMITTER does not appear to have access to $ORG_REPO."
+              exit 1
             else
               echo "::warning::Cannot verify submitter access for $ORG_REPO (API check failed). Manual review required."
               echo "⚠️ Submitter access must be verified manually by maintainers."
             fi
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/leaderboard.yml around lines 158 - 164, The workflow
currently treats any non-success outcome as "unverifiable" and warns; change the
logic to three outcomes: if gh api "/repos/$ORG_REPO/collaborators/$SUBMITTER"
succeeds -> collaborator ✅; else if gh api "/repos/$ORG_REPO" succeeds (repo
exists) and the owner check (gh api "/repos/$ORG_REPO" -q '.owner.login' ==
"$SUBMITTER") is false -> definitive no-access ❌ (echo a failure message and
exit 1); otherwise (any gh api call for the repo fails) -> API-unverifiable ⚠️
(emit the warning but do not fail). Use the existing gh api calls
("/repos/$ORG_REPO/collaborators/$SUBMITTER" and "/repos/$ORG_REPO") and the
owner check to distinguish repo-exists vs API-error cases and ensure only the
definitive no-access path fails the job.

fi
else
# Non-GitHub: cannot verify cross-platform access automatically
Expand Down
Loading