Skip to content

feat(annotate-mlflow-trace): add trace tagging and feedback skill#20

Open
V-3604 wants to merge 2 commits into
mlflow:mainfrom
V-3604:feat/annotate-mlflow-trace
Open

feat(annotate-mlflow-trace): add trace tagging and feedback skill#20
V-3604 wants to merge 2 commits into
mlflow:mainfrom
V-3604:feat/annotate-mlflow-trace

Conversation

@V-3604

@V-3604 V-3604 commented Jun 4, 2026

Copy link
Copy Markdown
Contributor

annotate-mlflow-trace lets a coding agent tag and leave feedback on the current session's MLflow trace from the same conversation, so sessions can be labeled for evaluation without opening the UI. A single script (trace_annotate.py) exposes tag, feedback, and list.

Fixes #13

Trace selection (updated for MLflow 3.14)

tag and feedback default to the most recent trace from the current project, resolved by the mlflow.trace.working_directory metadata that mlflow.claude_code writes in 3.14, so a shared experiment does not surface another project's trace. Selection falls back to the most recent trace on older versions, and --trace-id targets a specific trace. This is the agent and CLI path for writing assessments; it complements Review Queues, which is the UI path for the same data.

Decisions on the open questions in #13

  1. One skill, not two. tag-trace and feedback-trace share a workflow and a script, so they are a single annotate-mlflow-trace skill with subcommands rather than two directories.
  2. skip_skill_traces.py is included here as an optional Stop-hook pre-filter so annotation turns do not log their own trace. A native skip list in stop_hook_handler is probably the better long-term home, and I can move it there if that is preferred. The skill works without the hook; the hook only suppresses the administrative trace.

Notes

  • The hook detects the skill turn by the Base directory for this skill: marker Claude Code injects, so it is specific to Claude Code. Other agents can still annotate by --trace-id.
  • Unit tests cover tag parsing, value coercion, and trace selection, including project scoping and fallback. Verified on MLflow 3.12 and 3.14.

Signed-off-by: Varshith <kvarshithgowda@gmail.com>
@adamgurary

Copy link
Copy Markdown
Contributor

@B-Step62 can we route this to someone to look at please

@V-3604

V-3604 commented Jun 17, 2026

Copy link
Copy Markdown
Contributor Author

This PR and #21 both touch the skills array in .claude-plugin/plugin.json; #21 adds a ./ prefix to those entries. Once it merges I'll rebase and re-add annotate-mlflow-trace as ./annotate-mlflow-trace so it lands in the new format.

…ject

Signed-off-by: Varshith <kvarshithgowda@gmail.com>
@V-3604

V-3604 commented Jun 27, 2026

Copy link
Copy Markdown
Contributor Author

@B-Step62 Hi, I updated this to fit MLflow 3.14. Trace selection now scopes to the current project using the mlflow.trace.working_directory metadata the Claude Code integration writes, with a fallback for older versions, so it complements Review Queues instead of overlapping. The description now answers the two design questions from #13: I went with a single skill, and I left skip_skill_traces.py as an optional hook with a note that a native skip list in stop_hook_handler may be the better home. I can take either direction on the hook.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add trace annotation skills (tag-trace, feedback-trace)

2 participants