Sync agenteye-evaluator skill (agenteye@3ddd9de)

[NiveditJain]

Automated mirror — do not hand-edit this folder.

Source of truth	FailproofAI/agenteye → evaluator-sdk/skill/
Target	skills/agenteye-evaluator/
Triggered from	agenteye main @ 3ddd9de

The skills repo's validate-skills.py passed. To change the skill, edit evaluator-sdk/skill/ in agenteye and re-run the Sync agenteye-evaluator skill workflow — this same PR refreshes.

Summary by CodeRabbit

• Documentation
  • Clarified evaluator workflows by distinguishing plan-only and build modes.
  • Added a comprehensive brainstorming guide for creating evaluation plans from session data.
  • Documented dataset exploration, candidate metric validation, and evaluation-plan artifact structure.
  • Improved SDK installation guidance by clarifying when to stop the installation process.

[coderabbitai]

📝 Walkthrough

Walkthrough

The evaluator skill now documents separate plan and build modes, adds a detailed brainstorm reference for producing evaluation plans from logs, and clarifies SDK installation guidance.

Changes

Evaluator skill workflows

Layer / File(s)	Summary
Plan and build mode contract skills/agenteye-evaluator/SKILL.md	The skill distinguishes written evaluation planning from scoring-service construction, clarifies the build design loop, and adds a CI trigger comment.
Brainstorm evaluation-planning workflow skills/agenteye-evaluator/references/brainstorm.md	A new reference documents CLI and SQL exploration, corpus selection, session comparison, candidate dimension filtering, convergence, and eval-plan output.
SDK installation ladder guidance skills/agenteye-evaluator/references/scaffold.md	The installation guidance clarifies the public PyPI warning and stopping at the first applicable rung.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

• FailproofAI/skills#4: Overlapping synchronization and updates to the agenteye-evaluator skill documentation.

Suggested reviewers: siddarthaa

Poem

I hopped through plans where evals may grow,
Or build their scoring code below.
I read the logs, both good and bad,
And shaped the measures agents had.
The ladder’s clear, the paths align—
A tidy burrow, by design!

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately describes an automated sync of the agenteye-evaluator skill and matches the main change.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@skills/agenteye-evaluator/references/brainstorm.md`:
- Around line 130-138: Update the SQL example in the population-confirmation
section to explicitly compare good and bad session cohorts, using reviewed
session-ID lists or equivalent verdict-based cohort membership. Ensure the
output reports the metric by cohort so it can demonstrate separation, or clearly
relabel the query as feature extraction rather than discrimination evidence.
- Around line 163-171: Update the Filter A guidance and example around the
agenteye events query to measure payload-field coverage across the full
candidate population rather than a single session. Add a population-level query
that reports both sessions with the field populated and the total
eligible-session denominator, and require that coverage result before retaining
the dimension.
- Around line 90-94: Update the session inspection guidance around the `agenteye
--json events` command to explicitly page through results beyond the `--limit
1000` cap using the documented cursor mechanism. Require verifying that the
complete event stream has been retrieved before analyzing or cataloging a
session, while preserving the payload-free-first and selective `--full`
workflow.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: bc74dde9-6a53-437d-9cc9-81d9da3d47d5

📥 Commits

Reviewing files that changed from the base of the PR and between 0ae1292 and 89dc7a9.

📒 Files selected for processing (3)

• skills/agenteye-evaluator/SKILL.md
• skills/agenteye-evaluator/references/brainstorm.md
• skills/agenteye-evaluator/references/scaffold.md