Skip to content

Sync .github/skills directory with azure-sdk-tools for PR 16932 - #39897

Merged
Helen Gao (helen229) merged 1 commit into
mainfrom
sync-.github-skills-fix/skill-eval-environment-skills-6787054-16932
Sep 8, 2026
Merged

Helen Gao (helen229) merged 1 commit into
mainfrom
sync-.github-skills-fix/skill-eval-environment-skills-6787054-16932

Conversation

@azure-sdk-automation

Copy link
Copy Markdown
Contributor

Sync .github/skills directory with azure-sdk-tools for PR Azure/azure-sdk-tools#16932 See eng/common workflow

Copilot AI balanced review requested due to automatic review settings September 8, 2026 18:37
@azure-sdk-automation
azure-sdk-automation Bot requested review from a team as code owners September 8, 2026 18:37
@azure-sdk-automation azure-sdk-automation Bot added EngSys This issue is impacting the engineering system. Central-EngSys This issue is owned by the Engineering System team. labels Sep 8, 2026
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
11 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Several eval stimuli reference skill directories that do not exist in this repo (e.g., azure-typespec-author, markdown-token-optimizer, skill-authoring), which will break eval environment loading.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR syncs the .github/skills eval trigger definitions with updates from azure-sdk-tools (per azure-sdk-tools PR 16932), primarily by making trigger/anti-trigger prompts more specific and by explicitly scoping eval environments to particular skill sets.

Changes:

  • Updated trigger prompts to include concrete identifiers (e.g., release plan 12345, build 6447834) and clearer intent (“check readiness”, “trigger pipeline”, etc.).
  • Added reusable YAML anchors for environment blocks and expanded anti-trigger cases to assert correct skill routing via required/disallowed grader config.
  • Introduced cross-skill anti-trigger scenarios (e.g., pipeline fixer vs. pipeline analysis) by explicitly listing multiple skills in environment.skills.
File summaries
File Description
.github/skills/azsdk-common-sdk-release/evals/trigger.eval.yaml Refines release prompts and adds explicit env scoping + cross-skill anti-triggers.
.github/skills/azsdk-common-prepare-release-plan/evals/trigger.eval.yaml Adds env anchor reuse and strengthens anti-trigger routing checks.
.github/skills/azsdk-common-pipeline-fixer/evals/trigger.eval.yaml Separates “fix” vs “analyze” routing via env boundaries and required/disallowed assertions.
.github/skills/azsdk-common-pipeline-analysis/evals/trigger.eval.yaml Adds env scoping and cross-skill anti-triggers to keep analysis-only behavior.
.github/skills/azsdk-common-apiview-feedback-resolution/evals/trigger.eval.yaml Adds env anchor reuse and anti-trigger routing checks for non-APIView tasks.
Review details

Suppressed comments (2)

.github/skills/azsdk-common-pipeline-fixer/evals/trigger.eval.yaml:114

  • The eval references ../../skill-authoring, but this repo does not contain a .github/skills/skill-authoring skill directory, so the eval environment will fail to load.
    environment:
      skills:
        - ..
        - ../../skill-authoring
    graders:

.github/skills/azsdk-common-pipeline-analysis/evals/trigger.eval.yaml:98

  • The eval references ../../markdown-token-optimizer, but this repo does not contain a .github/skills/markdown-token-optimizer skill directory, so the eval environment will fail to load.
    environment:
      skills:
        - ..
        - ../../markdown-token-optimizer
    graders:
  • Files reviewed: 5/5 changed files
  • Comments generated: 6
  • Review effort level: Lite

Comment thread .github/skills/azsdk-common-pipeline-fixer/evals/trigger.eval.yaml
Comment thread .github/skills/azsdk-common-sdk-release/evals/trigger.eval.yaml
Comment thread .github/skills/azsdk-common-sdk-release/evals/trigger.eval.yaml
@helen229
Helen Gao (helen229) merged commit d25849a into main Sep 8, 2026
126 of 127 checks passed
@helen229
Helen Gao (helen229) deleted the sync-.github-skills-fix/skill-eval-environment-skills-6787054-16932 branch September 8, 2026 19:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Central-EngSys This issue is owned by the Engineering System team. EngSys This issue is impacting the engineering system.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants