RFC-0008 and RFC-0009: Skill Registry and Harness Integration#10
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds RFC-0005 proposing a new governed, metadata-first Skill Registry for MLflow, including versioning, lifecycle/publish-state governance, typed source pointers (Git/OCI/ZIP), and first-class skill groups.
Changes:
- Introduces the Skill Registry conceptual model (skills, skill versions, tags, aliases) and publish-state lifecycle.
- Defines skill groups with versioned membership snapshots, plus associated tags/aliases.
- Specifies proposed DB schema (12 tables) and API surfaces (REST, Python SDK, CLI, UI).
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| published skills, not the process of writing them. | ||
| - **Skill format specification.** The registry is format-agnostic. It | ||
| does not define or enforce what a skill looks like (SKILL.md, plugin | ||
| manifests, etc.). |
There was a problem hiding this comment.
Could we consider MLflow artifact storage as a first-class source type here? That would give us a natural UI upload flow and keeps the door open for more MLflow-native packaging/optimization later, even if Git and OCI stay the main paths.
There was a problem hiding this comment.
Good idea. We've added mlflow as an explicit future source type in the "Source type extensibility" section — it would allow storing skill content directly in MLflow artifact storage, providing a natural UI upload flow. It's deferred from the initial implementation to keep the registry metadata-first, but can be added as a follow-up without breaking changes. The adoption strategy's "Follow-up" section also calls this out.
— This comment was posted by Claude Code under the supervision of Bill Murdock.
| RETIRED = "retired" | ||
|
|
||
|
|
||
| class SkillSourceType(StrEnum): |
There was a problem hiding this comment.
Can we line this lifecycle up more closely with the MCP registry RFC in PR 12 (#12)? Right now this introduces draft, published, deprecated, and retired plus a separate skill-level status, which makes the cross-registry pattern feel less consistent unless there’s a strong skill-specific reason to diverge.
There was a problem hiding this comment.
Fully aligned — replaced the 4-state publish lifecycle (draft/published/deprecated/retired) with RFC-0004's 3-state model using status (active/deprecated/deleted). SkillStatus is now the only lifecycle enum. Parent entity status (Skill.status, SkillGroup.status) is derived from the latest version rather than set independently.
— This comment was posted by Claude Code under the supervision of Bill Murdock.
| does not perform scans. Scanning tools are separate. | ||
| - **Agent harness integration.** How a specific agent harness (Claude | ||
| Code, Codex, Cursor, etc.) installs or loads skills from the registry | ||
| is outside this RFC. The registry provides the metadata; harness |
There was a problem hiding this comment.
Do we want one short paragraph on external skill conventions we expect to interoperate with? Even if the registry stays format-agnostic, it would help to say whether we’re aligning with existing Claude/Cursor/GitHub-style skill packaging or deliberately avoiding standardization for now.
There was a problem hiding this comment.
Done — fully aligned with RFC-0004. Version lifecycle now uses status with three states (active, deprecated, deleted), matching MCPStatus. New versions default to active. Skill.status and SkillGroup.status are now read-only, derived from the latest version's status. Also added latest_version_alias on both Skill and SkillGroup, following RFC-0004's pattern for deterministic "latest" resolution. Store interface switched from AbstractSkillRegistryStore with @abstractmethod to SkillRegistryMixin with raise NotImplementedError, matching MCPServerRegistryMixin.
(Comment from Claude Code under the supervision of Bill Murdock.)
| #### SkillVersion | ||
|
|
||
| A versioned record containing a typed source pointer, publish state, | ||
| and tags. |
There was a problem hiding this comment.
I initially expected aliases to live on SkillVersion, but after checking MLflow model registry, the existing pattern is that aliases are owned by the top-level entity and point to a version. If the goal is consistency with MLflow and PR 12 (#12), I think this design is fine. It may just be worth making that pattern more explicit in the text.
There was a problem hiding this comment.
Aligned — set_skill_alias takes (name, alias, version) and sets the alias on the Skill entity, pointing to a specific version. Same pattern for set_skill_group_alias. This matches the MCP RFC's approach where aliases are owned by the parent entity rather than the version.
— This comment was posted by Claude Code under the supervision of Bill Murdock.
| ) | ||
| ``` | ||
|
|
||
| ## Create a skill group with a versioned membership snapshot |
There was a problem hiding this comment.
What's the justification for skill groups? Why not just using tags on skills
There was a problem hiding this comment.
Added a "Why groups instead of tags?" section to the RFC. The key reasons: (1) groups pin specific member versions for reproducibility, (2) groups have their own lifecycle and aliases independent of member status, (3) groups can reference capabilities across registries (skill registry + MCP registry), (4) groups map naturally to the "plugin" concept in agent harnesses, and (5) tags are flat key-value pairs that can't express ordered, versioned membership.
— This comment was posted by Claude Code under the supervision of Bill Murdock.
There was a problem hiding this comment.
Quick terminology update: "skill groups" have been renamed to "skill bundles" in the latest revision, and the section is now titled "Why bundles instead of tags?" The reasoning in my earlier reply still applies, just with the updated name.
This comment was posted by Bill Murdock with assistance from Claude Code.
| trace data back to a governed skill record to understand adoption | ||
| across an organization. | ||
|
|
||
| ### Use cases |
There was a problem hiding this comment.
Could we add explicit JTBDs per persona (developer, admin etc.)? It would make it easier to evaluate which parts of the data model are required
There was a problem hiding this comment.
Done — replaced the abstract use case bullets with three end-to-end persona flows: a platform administrator (registers, scans, groups, aliases, deprecates), a developer (searches, resolves alias, pulls), and a security engineer (queries scan tags, deprecates failed versions, tracks compliance). These ground the data model in concrete user workflows.
— This comment was posted by Claude Code under the supervision of Bill Murdock.
There was a problem hiding this comment.
The persona flows have been updated in the latest revision to reflect the new entity model. "Skill groups" are now "skill bundles," and the platform administrator scenario now references subagents explicitly (e.g., bundling a code-review skill with a security-auditor subagent and a GitHub MCP server). The three personas and their workflows are otherwise unchanged.
This comment was posted by Bill Murdock with assistance from Claude Code.
Update: Review Feedback Round 1Pushed changes addressing the straightforward review feedback: YAML frontmatter, mermaid ER diagram, field renames ( Still openStructural:
Scope:
Content:
(Comment from Claude Code under the supervision of Bill Murdock.) |
Status update (2026-04-27)Changes since last updateRFC-0005 (Skill Registry):
RFC-0006 (Skill Registry Harness Integration) — new:
Resolved review comments (newly replied)
Still open
(Comment from Claude Code under the supervision of Bill Murdock.) |
jwm4
changed the title
|
Also, I've converted this to draft given all the open issues listed above. |
Status update (2026-04-29)Changes since last updateRFC-0005 (Skill Registry):
RFC-0006 (Harness Integration):
Still open (deferred to future pass)
(Comment from Claude Code under the supervision of Bill Murdock.) |
Status update (2026-04-29, part 2)The four items listed as "still open" in the previous update are now all addressed:
All review comments from mprahl, serena-ruan, and Copilot now have replies. I don't have further edits planned at this time. The RFC is ready for another round of review. (Comment from Claude Code under the supervision of Bill Murdock.) |
| rfc_pr: https://github.com/mlflow/rfcs/pull/10 | ||
| --- | ||
|
|
||
| # RFC: Skill Registry |
There was a problem hiding this comment.
@Bill Murdock — Reading through the RFC and the proposed API, one thing struck me: given that the registry already supports kind=[skill, agent, mcp-server, hook], should we consider a more generic name than "skills" / "skill groups"?
The data model is already kind-agnostic — a Skill entity with kind=agent is conceptually an asset that happens to be an agent, not a skill. Calling the registry an Asset Registry (or Capability Registry) and using create_asset() / create_asset_group() instead of create_skill() / create_skill_group() would better reflect what the API actually does and avoid the naming tension where "skill" is both a specific kind and the umbrella term for all kinds.
The REST paths and CLI would follow naturally: mlflow assets create, mlflow asset-groups create-version, etc.
There was a problem hiding this comment.
+1
I'd like to also propose bundles. It satisfies the generic 'kind-agnostic' semantics of assets, while also implying plurality even in the singular term (skill-group and bundle both imply grouping of some asset).
I also agree that having an independent name (assets/bundles/capabilities) greatly improves function discoverability in the SDK
There was a problem hiding this comment.
Alternatively, if none of the names appeal, I'd suggest at least in the CLI dropping skill-groups in favor of skills with a --group flag. Something like:
mlflow skills pull --group pr-workflow --alias production
There was a problem hiding this comment.
+1 to @etirelli here, the umbrella term should not be a skill. Currently in the RFC the skill that is a kind/skill is more consistent with the industry's usage of the term, but the top level usage of the word skill would be very confusing.
My preference is asset registry or capability registry, or something equally broader.
There was a problem hiding this comment.
after further discussion last week, +1 to using the term bundle or skill bundle as per @khaledsulayman 's suggestion above.
There was a problem hiding this comment.
Resolved. Rather than renaming the umbrella term, we split the single Skill(kind=...) entity into three separate types: Skill, Subagent, and Hook. Each has full lifecycle (versions, aliases, tags, status). SkillGroup is renamed to SkillBundle with typed member lists, following the "skill bundle" suggestion from this thread. The registry is still "Skill Registry" and the SDK namespace stays mlflow.genai.skills.*. See commit d16abe0.
Bill Murdock, with assistance from Claude Code
There was a problem hiding this comment.
Adopted. SkillGroup is now SkillBundle throughout both RFCs. See commit d16abe0 for the full restructuring.
This comment was posted by Bill Murdock with assistance from Claude Code.
There was a problem hiding this comment.
Adopted a variant of this: the CLI now uses --bundle instead of --group, following the SkillGroup-to-SkillBundle rename. So mlflow skills pull --bundle pr-workflow --alias production.
This comment was posted by Bill Murdock with assistance from Claude Code.
| Add a Skill Registry to MLflow: a governed, metadata-first registry for | ||
| AI agent capabilities. The registry stores metadata and typed source | ||
| pointers (to Git repos, OCI registries, ZIP archives, etc.) rather | ||
| than artifacts directly. It provides enterprise governance on top of |
There was a problem hiding this comment.
Could we support both approaches? The user would either set pointers to external storage (Git, OCI, ZIP) or store assets directly in MLflow's internal artifact store.
The reason: source pointers are the right default for customers who already manage assets in Git or other systems. But for customers who don't have that infrastructure in place, or who already use MLflow to store models and want their agent capabilities alongside them, or who operate in airgapped or restricted environments where reaching external sources isn't practical — the ability to store assets directly in the registry would be a better fit.
There was a problem hiding this comment.
Agreed. The latest revision adds source_type="mlflow" as a first-class source type in the initial implementation. Content is stored as a directory tree of individual files in MLflow's artifact store, consistent with how model artifacts are stored. Users can choose external source pointers (Git, OCI, ZIP) or direct artifact storage depending on their environment. This covers the airgapped and "capabilities alongside models" use cases you described.
-- This comment was posted by Claude Code under the supervision of Bill Murdock.
|
|
||
| ### Out of scope | ||
|
|
||
| - **Artifact storage.** The registry stores metadata and source |
There was a problem hiding this comment.
given that mlflow already has an "artifact store" to store models and traces, I think the proposal could just embrace it and support it as an option, as per my previous comment.
There was a problem hiding this comment.
Addressed in the same change. See reply to your earlier comment.
-- This comment was posted by Claude Code under the supervision of Bill Murdock.
| permissions. | ||
| - **Alias management requires MANAGE.** Aliases like `production` | ||
| control which version downstream consumers resolve to. Changing an | ||
| alias has the same blast radius as a status transition. |
There was a problem hiding this comment.
Given the impact and blast radius of alias management, should alias changes be auditable?
Currently, last_updated_by / last_updated_timestamp on the parent Skill captures that something changed, but doesn't record alias-specific history (which alias, what it pointed to before, who changed it, when). For enterprise governance scenarios — "who promoted this to production and when?" or "what was production pointing to yesterday before the incident?" — an alias change log would be valuable.
This could be as lightweight as an append-only skill_alias_history table (alias, old_version, new_version, changed_by, timestamp), or even just emitting an event that external audit systems can consume. Worth considering for the initial design, since retrofitting audit onto alias mutations is harder than including it upfront.
There was a problem hiding this comment.
Added. Every alias mutation (set or delete) now appends a record to an append-only history table with the alias name, old version, new version, who changed it, and when. This covers "who promoted this to production?" and "what was production pointing to before the incident?" queries.
The design includes:
SkillAliasHistoryandSkillGroupAliasHistoryentitiesskill_alias_historyandskill_group_alias_historydatabase tables (append-only, no updates or deletes through the API)get_skill_alias_history()andget_skill_group_alias_history()store methods- REST endpoints at
/{name}/aliases/historyand/{name}/aliases/{alias}/history
History is recorded automatically by the store on every alias mutation. No separate API call needed.
-- This comment was posted by Claude Code under the supervision of Bill Murdock.
| cross-registry concern that applies equally to skills, MCP servers, | ||
| and other AI asset registries. It is expected to be solved at the | ||
| platform level across all MLflow registries rather than piecemeal in | ||
| each one. |
There was a problem hiding this comment.
Cross-workspace visibility can be managed at the infrastructure level (e.g., Kubernetes RBAC in OpenShift AI), so deferring that is reasonable. However, the RFC should consider supporting export/import of assets between workspaces — or at minimum, a move/duplicate operation. Use case: a platform team develops and tests assets in a staging workspace or even on a staging instance, then promotes them to a production workspace or instance. Without export/import, they'd need to re-register everything manually, including versions, tags, aliases, and group memberships. Even a simple mlflow skills export --name code-review --output bundle.json / mlflow skills import --input bundle.json --workspace production would cover this.
There was a problem hiding this comment.
Good use case. We'd like to follow the other MLflow registries on this. Neither the model registry nor the MCP registry (RFC-0004) has export/import yet, and we want to stay consistent rather than designing a serialization format in isolation. If export/import lands as a cross-registry capability in MLflow, the skill registry should adopt the same approach.
Added a mention in the adoption strategy's follow-up section. The data model is fully serializable, so this can be built on top of the existing API without schema changes.
-- This comment was posted by Claude Code under the supervision of Bill Murdock.
Status update (2026-04-30)Changes since last updateRFC-0005 (Skill Registry):
RFC-0006 (Harness Integration):
Open discussion
All review comments from mprahl, serena-ruan, Copilot, and etirelli now have replies (except the naming question, which is under active discussion). (Comment from Claude Code under the supervision of Bill Murdock.) |
| class SkillStatus(StrEnum): | ||
| ACTIVE = "active" | ||
| DEPRECATED = "deprecated" | ||
| DELETED = "deleted" |
There was a problem hiding this comment.
We added back a fourth DRAFT status in the mcp registry proposal fyi. If feedback on it is positive, might be useful here too?
bef24e7
There was a problem hiding this comment.
Added. The RFC now uses a 4-state lifecycle (draft, active, deprecated, deleted) aligned with the MCP Registry RFC. New versions default to draft, and activation requires MANAGE permission. See the updated status table and transitions in the latest push.
(This comment is from Claude Code under the supervision of Bill Murdock.)
| schema changes since the column stores a string value. `kind` is | ||
| immutable after creation. | ||
|
|
||
| **MCP servers: two registration paths.** The MCP server registry |
There was a problem hiding this comment.
Do we need both paths? I can see the benefit when embedded as part of an artifact, but wanted to double check if we can provide a cleaner path like a way for an embedded mcp to be automatically registered. Sorry if this was already discussed!
There was a problem hiding this comment.
Good question. The two paths serve different ownership models:
MCP Registry (RFC-0004) is for standalone MCP servers that have their own lifecycle, versioning, and deployment bindings. Skill groups reference these via registry="mcp" in their membership. This is the recommended path for most MCP servers.
kind=mcp-server in this registry is for MCP configs that are embedded in a group-level artifact (e.g., an .mcp.json inside an OCI image containing a complete plugin bundle). These configs exist only as part of their containing artifact and do not have an independent lifecycle.
Auto-registering embedded configs into the MCP registry would create governance confusion: the embedded config's lifecycle is tied to its containing group version, but an MCP registry entry has its own independent lifecycle. If someone deprecates the group version, the auto-registered MCP entry would still appear active in the MCP registry. Keeping them separate avoids that mismatch.
In practice, we expect most MCP servers to use the MCP registry path. The kind=mcp-server path is a narrow escape hatch for bundled artifacts where separating the config from its plugin would be artificial.
(This comment is from Claude Code under the supervision of Bill Murdock.)
There was a problem hiding this comment.
Update: the kind=mcp-server path described in the previous reply has been removed. MCP servers are now handled in two ways, both cleaner than before:
- Standalone MCP servers are registered in the MCP registry (RFC-0004) and referenced via the
mcp_serverslist onSkillBundleVersion. - Embedded MCP configs (e.g.,
.mcp.jsoninside a bundle-level OCI artifact) are just part of the artifact. They do not need separate registry entries or membership references.
See commit d16abe0.
Bill Murdock, with assistance from Claude Code
|
|
||
| ```python | ||
| @dataclass | ||
| class SkillGroup: |
There was a problem hiding this comment.
Could it make sense for skill group to simply be a Skill with kind=group? A skill can already be pointing to any number of things. Could potentially simplify/unify the data model a bit. Just putting it out there as an idea!
There was a problem hiding this comment.
Thanks for the suggestion! We discussed this on Slack and agreed to keep them separate. The main concern is naming clarity: collapsing groups into Skill with kind=group makes the "Skill" name even more confusing, since a skill group already includes not just skills but also sub-agents, hooks, and MCP configurations. People generally don't refer to a group of skills as a single "skill." With a name like "module" the composition would feel natural, but "skill" carries a more atomic connotation.
Keeping them separate also stays consistent with other MLflow registries (the MCP registry has separate server and server group entities) and avoids conditional complexity in the data model (groups have a membership table that individual skills don't need).
(This comment is from Claude Code under the supervision of Bill Murdock.)
There was a problem hiding this comment.
The naming clarity and data model complexity arguments make sense to me, but I'm a little confused about the "server group entities" analogy. What exactly does this refer to in the MCP registry rfc?
There was a problem hiding this comment.
Oh, sorry, that was an error in the comment that I should have caught! I've crossed it out. My main point was the other one. Thank you for catching this!
|
Status update (2026-05-07) Addressed jonburdo's three review comments:
The naming question (etirelli's comment on "skills" vs. a more generic name) remains open pending alignment on the broader question of unified vs. separate registries in MLflow. No further edits planned at this time. The RFC is ready for additional review. (This comment is from Claude Code under the supervision of Bill Murdock.) |
|
Status update (2026-05-12) The MCP Registry RFC (PR #12) has merged. Updated RFC-0005 to align with its final design: Changes made:
Reviewed and kept as-is (ahead of MCP RFC, not incompatible):
Differs by design (not a gap):
(This comment is from Claude Code under the supervision of Bill Murdock.) |
khaledsulayman
left a comment
khaledsulayman
left a comment
There was a problem hiding this comment.
A few questions and some naming suggestions, including:
registerinstead ofcreatebundlesinstead ofskill-groups- verb-based skill status lifecycling (
publish,unpublish,deprecate,delete)
| skill = mlflow.genai.skills.create_skill( | ||
| name="code-review", | ||
| description="Reviews pull requests for correctness, style, and security", | ||
| ) | ||
|
|
||
| # Register a version pointing to a Git source | ||
| version = mlflow.genai.skills.create_skill_version( | ||
| name="code-review", | ||
| version="1.0.0", | ||
| source_type="git", | ||
| source="https://github.com/acme/agent-skills/tree/v1.0.0/code-review", | ||
| content_digest="sha256:a3f2b8c...", | ||
| ) | ||
| # version.status == "draft" | ||
|
|
There was a problem hiding this comment.
*assuming the register naming from my above comment, but also works for create:
It's not clear to me why these can't be consolidated into one entrypoint. For an unregistered skill, calling mlflow.genai.skills.register() (or mlflow skills register in the CLI) without specifying a version should suffice as the current create_skill() operation, whereas specifying a version flag fulfills create_skill_version(). For already registered skills, it seems reasonable for mlflow to internally handle idempotency in the case of an existing skill/skill-version.
If I'm missing something as to why the python API should split this across two methods, I'd still push for at least having one register/create verb in the CLI, if possible.
There was a problem hiding this comment.
Addressed in the same change as the register naming comment above. register_skill() is now a single entrypoint that handles both skill creation and version registration with idempotency. The CLI equivalent is mlflow skills register --name X --version Y ....
(This comment is from Claude Code under the supervision of Bill Murdock.)
| mlflow skills pull-group --name pr-workflow --alias production \ | ||
| --destination ./plugins/pr-workflow | ||
| ``` |
There was a problem hiding this comment.
Could we unify pull and pull-group under a single pull command?
The harness RFC already does this for install, so it makes sense for pull to follow the same convention:
mlflow skills pull --name code-review --alias production --destination ./skills/
mlflow skills pull --group pr-workflow --alias production --destination ./plugins/
There was a problem hiding this comment.
Alternatively, if the skill-group naming resolves to assets or bundles:
mlflow bundles pull --name pr-workflow --alias production --destination ./plugins/
There was a problem hiding this comment.
Done. pull now uses --name vs --group flags, matching how RFC-0006's install already works.
(This comment is from Claude Code under the supervision of Bill Murdock.)
There was a problem hiding this comment.
I adopted your --group flag suggestion from the naming thread. The CLI now uses mlflow skills pull --group pr-workflow --alias production rather than a separate skill-groups command. This addresses the CLI ergonomics regardless of how the top-level naming question resolves: if the namespace stays skills, the command works as-is; if it changes to bundles or assets, the --group flag pattern carries over directly.
(This comment is from Claude Code under the supervision of Bill Murdock.)
There was a problem hiding this comment.
Terminology update: --group is now --bundle after the entity model restructuring. The CLI is mlflow skills pull --bundle pr-workflow --alias production. Same pattern you suggested, just with the updated name.
This comment was posted by Bill Murdock with assistance from Claude Code.
There was a problem hiding this comment.
Resolved: the entity is now called SkillBundle, and the CLI uses mlflow skills pull --bundle pr-workflow --alias production. Thanks for pushing on this naming question.
This comment was posted by Bill Murdock with assistance from Claude Code.
| **MLflow artifact storage (`source_type="mlflow"`).** In addition to | ||
| external source pointers, the registry supports storing skill content | ||
| directly in MLflow's artifact storage. This serves users who do not | ||
| have external Git/OCI infrastructure, who want agent capabilities | ||
| stored alongside their models, or who operate in airgapped | ||
| environments where external sources are not reachable. | ||
|
|
||
| Content is stored as a directory tree of individual files under an | ||
| artifact path, consistent with how MLflow stores model artifacts. For | ||
| example, a skill with a SKILL.md, scripts, and reference material is | ||
| stored as separate artifacts under a version-specific prefix: | ||
|
|
||
| ``` | ||
| skills/code-review/1.0.0/ | ||
| SKILL.md | ||
| scripts/analyze.sh | ||
| scripts/lint-config.json | ||
| reference/style-guide.md | ||
| ``` | ||
|
|
||
| The `source` field contains the MLflow artifact URI (e.g., | ||
| `mlflow-artifacts:/skills/code-review/1.0.0/`). Pull downloads the | ||
| directory tree from the artifact store. The MLflow UI can browse | ||
| individual files within a stored skill version. | ||
|
|
||
| The upload API accepts a local directory path and stores each file as | ||
| a separate artifact. The `content_digest` is computed over the full | ||
| directory contents at upload time. |
There was a problem hiding this comment.
I'm a bit unclear on the actual upload path here. Initially, I assumed this would be done via create_skill()/create_skill_version() by passing (..., source_type=mlflow, source=filepath), but in keeping with the semantics of git and oci, source should probably be reserved for the URI in MLflow's artifact storage.
Do we need to add an explicit skills registry API around uploads, or do we delegate it to users manually calling mlflow.log_artifacts() and maybe document that path?
There was a problem hiding this comment.
The register_skill() convenience function now has a content_path parameter. When provided, it uploads the local directory to MLflow artifact storage and sets source_type and source automatically. No manual log_artifacts() call needed.
At the store layer, create_skill_version() also accepts content_path. The server stores each file as a separate artifact under a version-specific prefix (e.g., skills/code-review/1.0.0/SKILL.md) and computes the content_digest at upload time. See the "MLflow artifact storage" subsection in the data model for details.
This comment was posted by Bill Murdock with assistance from Claude Code.
| **MCP servers: two registration paths.** The MCP server registry | ||
| (RFC-0004) is the default and recommended path for registering MCP | ||
| servers. It provides deployment tracking via hosted bindings, | ||
| deduplication across skill groups, and the full MCP governance model. | ||
| Skill groups reference MCP registry entries via `registry="mcp"` in | ||
| their membership. | ||
|
|
||
| `kind=mcp-server` in this registry is reserved for MCP configs that | ||
| are embedded in a group-level artifact (e.g., an OCI image containing | ||
| a complete plugin with an `.mcp.json` file). These are not | ||
| independently managed and exist only as part of their containing | ||
| artifact. Standalone MCP servers should always be registered in the | ||
| MCP registry, not as skills. |
There was a problem hiding this comment.
Apologies if it's already implied but should we explicitly add a guard against registering a standalone mcp server via mlflow skills <create/register> --name mcp-name --kind mcp-server --source-type git --source https://...? Such attempts should probably fail and redirect the user to mlflow mcp-servers register
There was a problem hiding this comment.
Now moot. The kind field has been removed entirely. Skill, Subagent, and Hook are separate entity types with separate registration functions (register_skill(), register_subagent(), register_hook()). MCP servers are only referenced via cross-registry pointers in SkillBundleVersion.mcp_servers, not registered through the skill registry. See commit d16abe0.
Bill Murdock, with assistance from Claude Code
| skill = mlflow.genai.skills.create_skill( | ||
| name="code-review", | ||
| description="Reviews pull requests for correctness, style, and security", | ||
| ) |
There was a problem hiding this comment.
nit: to me, create reads as authoring. Could we go with register instead? This also mimics RFC-0004's convention.
There was a problem hiding this comment.
Added. The RFC now provides register_skill() as a convenience function that auto-creates the parent Skill entity if needed and registers a version in one call, matching the MCP RFC's register_mcp_server() pattern. The store layer retains create_skill() / create_skill_version() for fine-grained operations, consistent with how the MCP RFC structures its store interface vs. SDK convenience layer.
(This comment is from Claude Code under the supervision of Bill Murdock.)
| rfc_pr: https://github.com/mlflow/rfcs/pull/10 | ||
| --- | ||
|
|
||
| # RFC: Skill Registry |
There was a problem hiding this comment.
Alternatively, if none of the names appeal, I'd suggest at least in the CLI dropping skill-groups in favor of skills with a --group flag. Something like:
mlflow skills pull --group pr-workflow --alias production
| status lifecycle, security scan tracking, and federated discovery. | ||
| The two approaches are complementary. | ||
|
|
||
| # Adoption strategy |
There was a problem hiding this comment.
Is it worth adding a note about shared base extraction as per the MCP RFC? If even just to say we will coordinate with MCP registry implementation.
There was a problem hiding this comment.
Added. The adoption strategy now includes a note about coordinating with the MCP Registry's shared base extraction effort where practical to reduce duplication. See the "Shared base extraction" bullet in the adoption strategy section.
(This comment is from Claude Code under the supervision of Bill Murdock.)
| def get_skill_group_version( | ||
| self, name: str, version: str, | ||
| ) -> SkillGroupVersion: | ||
| raise NotImplementedError | ||
|
|
||
| def get_skill_group_version_by_alias( | ||
| self, name: str, alias: str, | ||
| ) -> SkillGroupVersion: | ||
| raise NotImplementedError | ||
|
|
||
| def get_latest_skill_group_version( | ||
| self, name: str, | ||
| ) -> SkillGroupVersion: | ||
| raise NotImplementedError |
There was a problem hiding this comment.
Same rationale as my reply to your get_skill_version consolidation suggestion: keeping separate methods to match the MCP Registry pattern. The group get methods parallel the skill get methods, which parallel the MCP server get methods.
(This comment is from Claude Code under the supervision of Bill Murdock.)
There was a problem hiding this comment.
Same terminology update: "group get methods" are now "bundle get methods" (get_skill_bundle_version, get_skill_bundle_version_by_alias, etc.).
This comment was posted by Bill Murdock with assistance from Claude Code.
| # Search active skill versions | ||
| mlflow skills search-versions --name code-review \ | ||
| --filter "status = 'active'" | ||
|
|
||
| # Search active groups | ||
| mlflow skill-groups search --filter "status = 'active'" |
There was a problem hiding this comment.
I'd consider a unified list command which will list all skills or skill-groups/assets/bundles unless provided --name or --group, in which case it will list the registered versions of that skill/skill-group/asset/bundle
There was a problem hiding this comment.
I like the ergonomics of a unified list, but I'm keeping separate search_skills() and search_skill_versions() to match the MCP Registry's pattern (search_mcp_servers() / search_mcp_server_versions()). Consistency across MLflow's registries is a priority so users who learn one API can transfer that knowledge.
In the CLI, this maps to mlflow skills search and mlflow skills search-versions. Skill groups have their own parallel commands.
(This comment is from Claude Code under the supervision of Bill Murdock.)
There was a problem hiding this comment.
Terminology update: "skill groups" are now "skill bundles" with their own parallel commands (mlflow skill-bundles search). The rationale for keeping separate search commands is unchanged.
This comment was posted by Bill Murdock with assistance from Claude Code.
| | `POST` | `/ajax-api/3.0/mlflow/skills/{name}/install` | Install a single capability for a harness | | ||
| | `POST` | `/ajax-api/3.0/mlflow/skill-groups/{name}/install` | Install a skill group for a harness | |
There was a problem hiding this comment.
Do we need REST endpoints for install? Would this not be a client-side operation?
There was a problem hiding this comment.
Agreed. Install is now a client-side operation: the SDK resolves from the registry, pulls content, and writes harness-specific manifests locally. The only server-side endpoint is the marketplace.json GET, which harnesses query to discover available plugins.
(This comment is from Claude Code under the supervision of Bill Murdock.)
|
|
||
| - **Skills** (SKILL.md) — reusable agent instructions | ||
| - **Agents** (agent .md) — sub-agent definitions | ||
| - **MCP servers** (JSON config) — tool server integrations |
There was a problem hiding this comment.
Now that the MCP registry RFC has been merged, is this still in scope?
There was a problem hiding this comment.
No. I removed kind=mcp-server from the skill registry entirely in the previous commit, in response to a similar point from Khaled. MCP servers belong exclusively in the MCP Registry (RFC-0004). Skill groups reference MCP registry entries via registry="mcp" in their membership, and harness-specific .mcp.json configs are generated at install time by RFC-0006.
-- Bill Murdock (with assistance from Claude Code)
| Add a Skill Registry to MLflow: a governed, metadata-first registry for | ||
| AI agent capabilities. The registry stores metadata and typed source | ||
| pointers (to Git repos, OCI registries, ZIP archives, etc.) rather | ||
| than artifacts directly. It provides enterprise governance on top of |
There was a problem hiding this comment.
Could you clarify this? When I first read it, it seemed like artifact repository was out of scope but later on I saw that it was.
There was a problem hiding this comment.
Clarified. The summary now says the registry stores metadata and typed source pointers, but can also store content directly via MLflow artifact storage. The primary design is metadata-first. The source_type="mlflow" section in the detailed design explains the artifact storage path.
-- Bill Murdock (with assistance from Claude Code)
|
|
||
| The list view shows skills and skill groups in a card-based or table | ||
| layout, with name, description, latest version, status, and tags. Users | ||
| can filter by status, source type, and search by name or description. A |
There was a problem hiding this comment.
It would be useful to track install counts and surface them in the MLflow UI. That would help enterprises understand which capabilities are actually being adopted, and it would also let end users sort or rank skills / skill groups by popularity.
🤖 Generated with Claude
There was a problem hiding this comment.
Good idea. We added install count tracking and surfacing in the UI to the adoption strategy section, enabling users to sort or rank capabilities by adoption.
-- Bill Murdock (with assistance from Claude Code)
|
|
||
| Version strings must follow [semantic versioning](https://semver.org/) | ||
| (e.g., `1.0.0`, `2.1.0-beta.1`). `get_latest_skill_version(name)` | ||
| returns the highest semantic version among `active` versions. |
There was a problem hiding this comment.
Just make sure this is aligned with the MCP Registry rules. Latest first resolves to the highest active but if there are no active versions, then it fallsback to the latest non deleted. We may want to do the same here.
There was a problem hiding this comment.
Done. get_latest_skill_version(name) now returns the highest active semantic version if one exists, otherwise the highest non-deleted non-active version. Same rule applied to entity-level status derivation. Both the main RFC and implementation details are aligned with the MCP Registry pattern.
(This comment is from Bill Murdock with assistance from Claude Code.)
| | `version` | `String(256)` | PK, valid semantic version | | ||
| | `version_major` | `Integer` | extracted from validated semantic version | | ||
| | `version_minor` | `Integer` | extracted from validated semantic version | | ||
| | `version_patch` | `Integer` | extracted from validated semantic version | |
There was a problem hiding this comment.
You may want to align with the SQL prerelease ordering approach in Jon's PR: mlflow/mlflow#23952
There was a problem hiding this comment.
Done. Added version_prerelease_sort_key (String(512)) to both skill_versions and skill_bundle_versions tables (and noted that subagent/hook tables follow the same pattern). The sort key is a lexicographically sortable encoding of prerelease identifiers, following Jon's approach in mlflow/mlflow#23952. Release versions encode to a sentinel that sorts above all prerelease encodings, so full semver precedence resolves in SQL without application-level tie-breaking.
(This comment is from Bill Murdock with assistance from Claude Code.)
| Git transport format. Implementers must decide which Git URL forms are | ||
| accepted, how repository, ref, and path are extracted, and how | ||
| reproducibility is handled when a ref is mutable. Resolving this | ||
| challenge is left to the implementer, but feasible options include the |
There was a problem hiding this comment.
One option is the URL is to an actual Git remote with an @<ref> suffix that can be cloned and then depend on subpath for the path within the repo.
There was a problem hiding this comment.
Adopted this approach. For source_type="git", source is now a Git clone URL with an @<ref> suffix (e.g., https://github.com/acme/agent-skills.git@v1.0.0), and subpath identifies the path within the repository. This separates the clone target, ref, and content path into distinct fields rather than relying on hosting-provider-specific tree URL conventions. Updated all examples across both RFCs to use this format.
(This comment is from Bill Murdock with assistance from Claude Code.)
| member versions may omit `source` because their content is embedded in | ||
| the bundle artifact. The optional membership `member_subpath` identifies | ||
| where the member lives inside the bundle artifact. For assembled bundles, | ||
| `member_subpath` is usually empty because the member's own `source` and `subpath` |
There was a problem hiding this comment.
I think member_subpath being empty should be a hard requirement when the bundle has members with sources.
There was a problem hiding this comment.
Done. For assembled bundles, member_subpath must now be null. The API rejects attempts to set member_subpath on a membership whose member version has its own source.
(This comment is from Bill Murdock with assistance from Claude Code.)
| name="pr-workflow", | ||
| version="1.0.0", | ||
| skills=[ | ||
| ("code-review", "1.0.0"), |
There was a problem hiding this comment.
I'd prefer not having a tuple and use specific data type to give us flexibility in the future for more fields.
There was a problem hiding this comment.
Done. Replaced all tuples with typed member refs: SkillMemberRef, SubagentMemberRef, HookMemberRef, and McpServerMemberRef. The first three carry an optional member_subpath field for monolithic bundles. Updated the dataclass, store interface, and all examples to use the new types.
(This comment is from Bill Murdock with assistance from Claude Code.)
|
|
||
| ```python | ||
| # A bundle version can include skills, subagents, hooks, and MCP servers | ||
| bundle_version = mlflow.genai.skills.create_skill_bundle_version( |
There was a problem hiding this comment.
Could you show an example where the skill bundle is the source?
There was a problem hiding this comment.
Added two monolithic bundle creation examples. One uses a Git source (with a comment showing how the browsable GitHub URL maps to clone URL + subpath), and one uses an OCI source. Both show member_subpath on embedded member refs. These are in the "Create a skill bundle with cross-registry references" section alongside the assembled example.
(This comment is from Bill Murdock with assistance from Claude Code.)
| bundle version and any registered embedded skill versions discovered | ||
| during import. Automatic per-skill SKILL spans require the local skill | ||
| name to resolve to a registered skill version in the manifest. Users of | ||
| harnesses without hook support can still use |
There was a problem hiding this comment.
This might require some clarity if we are leaning in on enhancing existing MLflow autolog.
There was a problem hiding this comment.
Clarified. The RFC now explicitly recommends enhancing the MLflow autologger to recognize skill invocations using the install-time manifest and emit SKILL spans with registry coordinates. The autologger runs in the same process that owns the active trace, so it has natural access to thread-local trace context and produces correctly nested spans. Updated both the main RFC-0009 text and the implementation details to lead with this approach, with hook-based correlation and other strategies listed as fallbacks for harnesses without in-process integration.
(This comment is from Bill Murdock with assistance from Claude Code.)
| metadata. Import fails if no valid semantic version is available.""" | ||
| ``` | ||
|
|
||
| Import is append-only with respect to governed versions. It must not |
There was a problem hiding this comment.
I don't think we should allow reimports or edits of a skill bundle version. It should be immutable after creation.
There was a problem hiding this comment.
Agreed. RFC-0008 already has an explicit immutability contract: "The member lists and source fields of a bundle version are immutable after creation." The import section in RFC-0009 now references that contract directly and states that import must not overwrite any existing version. If the target (name, version) already exists, import fails with a conflict.
(This comment is from Bill Murdock with assistance from Claude Code.)
| "name": "imported-plugin", | ||
| "version": "1.0.0", | ||
| "scope": "project", | ||
| "mode": "monolithic", |
There was a problem hiding this comment.
Nit: This is probably not needed as the MLflow client should likely be querying the server to validate the information in the lock file.
There was a problem hiding this comment.
Agreed. Removed registry_uri and workspace from the lock file format. The registry URI and workspace now come from the MLflow client configuration, so the same lock file works across environments. Lock replay still contacts the registry to verify version status.
(This comment is from Bill Murdock with assistance from Claude Code.)
There was a problem hiding this comment.
Good call on the pip lock pattern. Updated: registry_uri is now optional in the lock file. If specified at lock time, replay uses it; if omitted, replay falls back to the MLflow client configuration. This lets users choose between full determinism and cross-environment portability.
(This comment is from Bill Murdock with assistance from Claude Code.)
| For harnesses where in-process integration is not available, other | ||
| approaches may be feasible. These are not exhaustive; the implementer | ||
| may choose another approach if it satisfies the same trace-parenting | ||
| requirements. | ||
|
|
||
| - **Explicit trace correlation for hook commands.** Harness hooks may | ||
| pass trace context to external commands through stdin, environment | ||
| variables, or a temporary context file. That context could include a | ||
| trace ID, parent span ID, or opaque MLflow correlation token. Hook | ||
| commands would use this context to create SKILL spans under the | ||
| correct parent and would need a way to preserve span lifecycle state | ||
| between start and end events. | ||
|
|
||
| - **Invocation event annotation.** Instead of opening and closing live | ||
| spans from hook commands, hooks can emit timestamped skill invocation | ||
| events containing registry coordinates and any available trace | ||
| correlation data. MLflow can then attach those events to traces or | ||
| materialize derived SKILL spans during ingestion or display. | ||
|
|
||
| - **Harness-native extension.** If a harness exposes an in-process | ||
| extension or plugin API for skill invocation events, the installer | ||
| can configure that extension to call the MLflow tracing SDK directly. | ||
| This has similar trace-context advantages to autologger | ||
| instrumentation while using the harness's native extension surface. |
There was a problem hiding this comment.
This can likely be removed since we probably don't want to speculate on implementation details here.
There was a problem hiding this comment.
Removed. The "Other harnesses" section was speculative and didn't add anything beyond what the "Automatic skill-span instrumentation challenge" section already covers.
(This comment is from Bill Murdock with assistance from Claude Code.)
mprahl
left a comment
mprahl
left a comment
There was a problem hiding this comment.
Just a few comments but this looks good!
| ```json | ||
| { | ||
| "harness": "claude-code", | ||
| "locked_at": "2026-05-17T21:00:00Z", |
There was a problem hiding this comment.
I'm personally on the fence on whether or not to rely on the client configuration or to put the MLflow server URL and workspace here for easy mlflow install (just need to provide auth).
@HumairAK what do you think?
There was a problem hiding this comment.
For Pip lock, it seems like you can specify it optionally. Per @mprahl for NPM it seems it's more opinionated and always includes it.
I personally prefer the Pip lock approach and making this optional and let the user decide what level of determinism they want. The way pip does it is:
You specify the index at lock time the same way you do with normal pip operations:
pip lock -i https://my-registry/simple ...
pip lock --extra-index-url https://another-registry/simple ...
or via PIP_INDEX_URL / PIP_EXTRA_INDEX_URL
I think we can take a similar approach here.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
5088cc7 to
2dfb845
Compare
mprahl
left a comment
mprahl
left a comment
There was a problem hiding this comment.
Great job! A lot of hard work went into this RFC and I think this will be a critical feature for MLflow!
| - **Subagents**: sub-agent definitions that can be invoked by a | ||
| parent agent | ||
| - **Hooks**: event-triggered actions (harness-specific) |
There was a problem hiding this comment.
These are not skills, we should not mix them under the name of skill registry and skills modules.
There was a problem hiding this comment.
We went back and forth a lot on the terminology and we want something generic enough to convey an agent capability that includes these subtypes.
Do you have any alternative naming suggestions @B-Step62 ? My personal preference is "agent capabilities" or similar.
There was a problem hiding this comment.
There is some precedent in the industry for "skills" as an umbrella term - e.g. Claude Code plugins created via claude plugin init are saved to ~/.claude/skills. I agree with your point in another comment about having one CLI with a --type or --kind flag. Personally, I like the discoverability of "Skills." They are the first-class entity that people search for, while subagents, hooks, and mcp-server references are mainly expected to be used in conjunction with them.
If we feel strongly about wanting something more generic, I lean towards "assets." Anecdotally, I've heard it used to describe the collective, similar to "agent capabilities," but I slightly prefer one-word naming, especially for CLI verbage.
| - **Subagents**: sub-agent definitions that can be invoked by a | ||
| parent agent | ||
| - **Hooks**: event-triggered actions (harness-specific) | ||
| - **Skill bundles**: versioned, governed units that group related |
There was a problem hiding this comment.
- If this maps to plugin, why don't we name it "plugin"? We don't prefer inventing an extra concept that increases cognitive load to users.
- Why do we treat MCP server particularly as a part of skill bundle? This feels like just a logical group that bundles any sort of assets the proposal is adding. Also calling MCP server as "skill bundle" is confusing.
There was a problem hiding this comment.
I'll let @jwm4 chime in more here, but my recollection is that "plugin" is vendor specific and this was trying to use more generic terminology.
On the MCP server part, I'm kind of leaning towards not including it here and tackling that cross-registry integration with a future agents registry.
There was a problem hiding this comment.
The vendor-specific issue is a factor, but really my main concern about having a "plugin registry" in MLflow is that I would expect users to see this as a registry for MLflow plugins, which are unrelated. One of the things I like about the names "skill registry" and "skill bundle" is that most of the time the main content in these things are skills, so if a user comes in with the question "where do I find the skills?", I'd like the name of this capability to make it extremely clear and intuitive to understand that this is the place to look for the skills. However, I definitely recognize that that desire has come at the cost of now being confusing when you look more closely at it and see that it is skills plus other stuff so I think "Agent Capabilities" as Matt proposes in the other thread would be a fine conclusion here. Maybe the registry is called "Agent Capabilities Registry" and the bundles are called "Agent Capability Bundles"? I think the words "context" or "prompting" would also be fine to include in some naming scheme somehow (e.g, "Context Registries" and "Context Bundles" or "Context Plugins" or whatever). Personally, I still would prefer to have skill in the name somehow, because I still worry about the user trying to find the skills and not knowing where to look. But I also want to try to keep these names as concise as possible so I wouldn't want something like "Skills and Related Capabilities Plugin Registry" because it's just too wordy.
There was a problem hiding this comment.
Related to this comment, I'd research a bit more about existing standard or spec for bundling these concepts rather than defining a new one. I see the vendor neutral benefit, but defining our own spec means that we lock users down to MLflow as a vendor, unless we commit to evangelize and push this to be an industry standard like OpenTelemetry. Considering the number of players in the industry and how influential some of vendors are, I'm not very optimistic for that direction.
| mlflow skills register --name code-review --version 1.0.0 \ | ||
| --source https://github.com/acme/agent-skills.git@v1.0.0 \ | ||
| --subpath code-review | ||
| mlflow subagents register --name security-auditor --version 1.0.0 \ | ||
| --source https://github.com/acme/agent-skills.git@v1.0.0 \ | ||
| --subpath security-auditor | ||
| mlflow hooks register --name pre-commit-scan --version 1.0.0 \ | ||
| --source https://github.com/acme/agent-skills.git@v1.0.0 \ | ||
| --subpath pre-commit-scan |
There was a problem hiding this comment.
Learning different CLI for different types of entities is hectic. We also don't like to maintain many CLI modules with lots of overlapping functionality and configurations.
Can we instead have a single CLI with type argument?
There was a problem hiding this comment.
That sounds reasonable to me. Once we settle on a name / scope for the overall registry, it should be relatively simple to have a single command like mlflow agentic-capability register --type skill ... (or whatever we settle on in place of agentic-capability).
| 3. Transition the bundle version from draft to active: | ||
| ```bash | ||
| mlflow skill-bundles update-version --name pr-workflow \ | ||
| --version 1.0.0 --status active |
There was a problem hiding this comment.
What does "active" mean for skill? It sounds like just a special kind of tags or alias.
There was a problem hiding this comment.
I think the goal here is to align with the MCP registry statuses. The benefit of having a status on a skill is to still have lineage tracking to traces and version history but you can mark a skill version as deprecated without having to delete it.
| 1. Install the bundle for a harness | ||
| ([RFC-0009](../0009-skill-harness-integration/0009-skill-harness-integration.md)): | ||
| ```bash | ||
| mlflow skills install-bundle --name pr-workflow --alias production \ |
There was a problem hiding this comment.
People often install a set of skills from github directory using a single command gh skills install repo/name. In order to do this through MLflow, do users need to define bundle themselves? It can be a major overhead for adopting MLflow.
There was a problem hiding this comment.
My main goal here is to have a single command that's compatible with all the different source types that we support (GitHub, OCI, .zip file URLs, stored in MLflow). That way users can pull skill content from the registry without having to pay much attention to whether it's coming from GitHub, OCI, etc. We're seeing a lot of arguments out there about OCI in particular as a better data layer for highly managed content than GitHub, so we really want to at least be compatible with OCI and GitHub. The other source types are more debatable, but if we're going to have multiple source types anyway, including those two seems reasonable to me. Storing the skills in MLflow itself was not part of the original proposal, but it was requested in an earlier review of this RFC and it didn't seem like an unreasonable request to me.
| ``` | ||
| 2. Install v1.0.0 and run it on a set of test inputs. Traces are | ||
| recorded in MLflow under experiment A. | ||
| 3. Install v2.0.0 and run it on the same test inputs. Traces are |
There was a problem hiding this comment.
I imagine it is a common scenario to test the performance between with and without a skill, or a different set of skills. Do we have a plan for supporting that use case?
There was a problem hiding this comment.
That as a first class experience would likely be better though an agents registry (e.g. agent v1 used the skill but agent v2 did not).
I think this is still possible with separate evaluation runs and so we should include that as a user journey.
There was a problem hiding this comment.
I agree this is an important scenario. As Matt requested, I've added a new user journey ("Compare agent performance with and without a skill") that walks through this: run the agent without the skill as a baseline, install the skill, run again, evaluate both with the same scorers, and compare side-by-side. The same pattern works for comparing different skill sets.
As Matt notes, a first-class experience for managing which skills are active per agent configuration would benefit from a future agents registry.
| 4. Use `mlflow.genai.evaluate()` with a `make_judge` scorer that | ||
| uses the `{{ trace }}` template variable to score both sets of | ||
| traces against quality criteria (correctness, helpfulness, safety). | ||
| 5. Compare the evaluation results side by side in the MLflow UI to |
There was a problem hiding this comment.
How do users manage lineage between the MLflow metadata of the loaded skill and the evaluation result?
There was a problem hiding this comment.
The intended lineage path is through the trace: evaluation operates on traces, and skill lineage is recorded on SKILL spans via the {workspace, name, version} coordinates. So an evaluation result can be related back to the loaded skill by following the associated trace and inspecting its SKILL span(s).
To me, the RFC does not currently make that workflow explicit enough. If the expectation is that users traverse trace -> spans -> skill coordinates, we should document that clearly. Otherwise, a more first-class lookup/linkage mechanism may be worth adding. cc @jwm4
There was a problem hiding this comment.
I've added a new user journey ("Trace skill lineage to evaluation results") to the RFC that makes this explicit. The lineage path flows through traces:
- Skill invocations produce
SKILLspans with{workspace, name, version}registry coordinates. mlflow.genai.evaluate()produces results linked to traces viatrace_id.Trace.search_spans(span_type="SKILL")retrieves the skill version from any traced evaluation result.mlflow.search_traces()with span-level filters finds all traces involving a specific skill version.
The current approach is two-step (query traces by skill attributes, then retrieve evaluation results). I also added a note as a phase 3 follow up to do richer integration such as filtering evaluation results directly by skill version.
There was a problem hiding this comment.
How does trace records lineage to skills in MLflow? Once the skill is pulled from MLflow, it doesn't contain metadata about MLflow.
In my old prototype I tried to resolve this by saving a special metadata file during pull and make MLflow tracing look at that. However, this does not always work because users may not use MLflow SDK for tracing. For example, Claude Code support Otel tracing as beta, and once it is fully supported we may migrate to that. At that point, the tracing logic is not owned by us hence the file based approach doesn't work.
| - **MCP server hosting.** MCP server deployment and runtime management | ||
| are covered by the MCP Server Registry (RFC-0004) and the MCP | ||
| Gateway. | ||
| - **Prompts.** MLflow already has a Prompt Registry for versioned |
There was a problem hiding this comment.
There is not a strong distinction between two in the industry. Custom agent frameworks like pydantic AI, Google ADK, Claude Agent SDK support skills and MCPs. Also managed harness still inputs prompts. The boundary is merely whether a harness is owned by framework or users, not about what primitive they use.
There was a problem hiding this comment.
To me the biggest distinction between a skill and a prompt is that a skill includes some mechanism to invoke it. I think the common practice for skills is:
- Skills can be invoked directly by the user (e.g.,
/create-cookie-recipe). - Skills can be invoked via the model using a skill tool.
The skill format includes meta-data controlling both of these (whether one of these or both of these is allowed, the description of the skill that the model uses to decide whether to invoke it).
Another key distinction between a skill and a prompt is that a skill is typically a directory containing both the SKILL.md and some additional content (typically scripts, templates, and resources), where the SKILL.md file is the prompt that gets loaded up when the skill invoked and the other content is only accessed progressively when directed by that prompt. For example, a skill could have a resources directory with many resource files, each handling a specific situation that sometimes arises and collectively, they might be all too big to put into a single prompt because they would exceed the context window, but the skill never needs to know all of them at once. It only needs to know the one that's relevant to the particular situation that it finds itself in.
Now, of course, a regular prompt in the prompt registry could also provide instructions to load some resource or invoke some script under certain circumstances. But those resources and scripts need to be delivered, managed, and governed separately because they're not part of the prompt itself. Skills address this limitation.
Anyway, I'm not sure what the consequences for this document are. Is there something in this bullet point that you would like revised?
There was a problem hiding this comment.
Sorry my wording was vague - the question is not distinction between prompt and skills, but harness-based agents vs. custom agentic code. IIUC the claim here is that the skills are for the former (e.g. claude code) and prompts are for the latter (e.g. langgraph). However, the mapping is not that distinct so audience overlaps.
The indication to the design is that I don't think we should treat prompt as an exception from the "general assets registry" concept.
|
|
||
| # Summary | ||
|
|
||
| Add harness-specific installation to the MLflow Skill Registry |
There was a problem hiding this comment.
I might be missing something, but isn't installation of skill just copying files into the target machine with specific location prefix? It is what gh skills install command does. What problem are we trying to solve with this proposal above that?
There was a problem hiding this comment.
I addressed this in an earlier response, but I want to put the answer here too for anyone looking at this thread in isolation:
My main goal here is to have a single command that's compatible with all the different source types that we support (GitHub, OCI, .zip file URLs, stored in MLflow). That way users can pull skill content from the registry without having to pay much attention to whether it's coming from GitHub, OCI, etc. We're seeing a lot of arguments out there about OCI in particular as a better data layer for highly managed content than GitHub, so we really want to at least be compatible with OCI and GitHub. The other source types are more debatable, but if we're going to have multiple source types anyway, including those two seems reasonable to me. Storing the skills in MLflow itself was not part of the original proposal, but it was requested in an earlier review of this RFC and it didn't seem like an unreasonable request to me.
There was a problem hiding this comment.
It seems to me like the skills in gh skills install are primarily about installing Agent Skills, not full “plugin bundles.” Where as harness-integration RFC defines a plugin as an installable package that can bundle skills, custom agents, hooks, and MCP server configurations.
There was a problem hiding this comment.
Oh, I didn't realize that. Thank you for clarifying, @HumairAK ! It does seem like another important differentiator in addition to the multi-source support.
There was a problem hiding this comment.
I understand there is some extension, but I'm not sure if we should step into the "plugin bundles" concept in this RFC. It is non trivial complexity to define such standard "bundle" for many different downstream agents and respective mapping logic between them. I feel it's too premature for MLflow to take that responsibility.
Also curious if you have looked at https://github.com/microsoft/apm, which sounds like solving the similar problem here. We could adopt an existing spec like this and just store them as a pure artifact rather than inventing a new one.
@jwm4 Btw this PR is getting super long, even Github UI takes me minutes to load comments😅 Can we at least split two RFCs into separate PRs? The general rule of thumb is one PR for one RFC.
|
Thanks for the proposal, @jwm4 @mprahl. Honestly, it feels broader in scope than the original skill management problem. If the intent is to cover a wider range of assets than skills, I'm supportive for the direction itself, but in that case the design should work backward from that broader goal rather than extend the mechanics built for skills. I'd also stress extensibility, since a year from now the industry could easily have five more concepts we'll want to fit into this. |
| shared artifact (used with OCI and ZIP). The optional `content_digest` | ||
| field enables integrity verification. | ||
|
|
||
| #### Subagent and Hook |
There was a problem hiding this comment.
Afaik, there's no standard schema for subagents and hooks. What kind of format will we use for these, and do we think we can have a unified schema to support multiple harnesses?
There was a problem hiding this comment.
I think the proposed schema is generic/vendor neutral, but the actual contents may not be, so it may be worthwhile to distinguish what vendor format the hook is in and let each harness determine if they support that format.
Thoughts @jwm4?
| lifecycle patterns. The store interface, REST API, and SDK expose | ||
| parallel operations for each type. | ||
|
|
||
| #### SkillBundle |
There was a problem hiding this comment.
IMO, it's a bit odd to call it as "skill" bundle while it contains subagents and hooks
There was a problem hiding this comment.
I think we can consolidate the naming discussions to https://github.com/mlflow/rfcs/pull/10/changes#r3533636917.
| cross-registry reference handling for all entity types are in | ||
| [implementation-details.md](implementation-details.md#skill-entity). | ||
|
|
||
| #### Aliases and tags |
There was a problem hiding this comment.
Now MLflow has too many tag and alias tables, can we define a canonical table with generic fields like Tag(entity_type="skill_version", entity_id="1", key="team", value="db")?
There was a problem hiding this comment.
I actually prefer separate tag and alias tables to allow database indexes to be lean. Using generic EAV tables causes scale issues with large tables on joins.
I think we can consider consolidating it for this registry but I'd like to not make this a generic tags table of all future entities.
There was a problem hiding this comment.
I also prefer the lean table approach Matt advocates for, but I can accept whatever the conclusion of this discussion is.
| repositories. This works well for individual developers and small | ||
| teams. GitHub provides versioning, collaboration, and access control. | ||
|
|
||
| However, enterprises face governance challenges that Git alone does not |
There was a problem hiding this comment.
I remember previously we discussed that this limitation of git is not strong enough to add a skill registry to MLflow. What are the new value adds and/or evidence for us to try this again? cc: @B-Step62
There was a problem hiding this comment.
@TomeHirata the following section is intended to address that by outlining why Git alone is insufficient here. It would be helpful to understand which of those arguments, or any previously discussed concerns, you still find unconvincing so we can either strengthen the case or clarify the RFC.
| "which skill versions are most used?" or "show me all traces where | ||
| the deprecated code-review v1.0 was loaded." | ||
|
|
||
| 5. **No pull mechanism.** Once a user discovers a capability in the |
There was a problem hiding this comment.
I think we can install skills from git (e.g., https://cli.github.com/manual/gh_skill_install)
There was a problem hiding this comment.
Very cool! I didn't know about that.
There was a problem hiding this comment.
Yes, I think this would be a good solution for users who have their skills in GitHub. I discussed the issues with the other source types in this other thread.
| servers, and hooks together. But there is no agent-neutral way to | ||
| represent these bundles for governance and discovery. | ||
|
|
||
| 4. **No trace-to-skill linkage.** MLflow already traces agent |
There was a problem hiding this comment.
I believe this can be done even without having "registry". We can create a span for each skill invocation and store skill name as span attributes.
There was a problem hiding this comment.
Yes, but I think the user experience is significantly better if MLflow is aware of what the meaning of those attributes are. Otherwise, it's up to each user to define what meaning the span attributes have. The meaning would not be entirely clear to a person or agent viewing the trace such as where did this skill come from, what were the contents of the skill, did the skill change recently, etc?
There was a problem hiding this comment.
I agree with Matt, and also I'm thinking head towards what we want in a future agent registry and I feel like mostly what we want is to be able to aggregate the contents of the other registries (i.e., the registry entry for an agent links to the registry entries for the models, prompts, MCP servers, skills, etc. that it uses).
There was a problem hiding this comment.
I believe this can be done even without having "registry". We can create a span for each skill invocation and store skill name as span attributes.
The value add here is that the registry turns those attributes from individual user managed metadata into a governed, versioned API. That makes the data compatible across clients, tools, teams, and organizations, and enables queries like "which registered skill version was used here?" or "show traces involving deprecated skills."
Add a new user journey showing how evaluation results trace back to specific skill versions through SKILL spans, with code examples using search_traces and search_spans. Note richer evaluation-to-skill query integration as Phase 3 follow-up work. Addresses reviewer feedback from B-Step62 and HumairAK on PR mlflow#10. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Show how to measure the impact of adding or removing a skill using the same evaluation infrastructure as version comparison. Addresses reviewer feedback from B-Step62 on PR mlflow#10. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
After more discussion, we've decided to close this PR and open new ones that include a much simpler phase one to get to value sooner followed by a more complete phase two later. I will be opening those new PRs ASAP (probably some time next week). |
Summary
Two companion RFCs for adding a governed Skill Registry to MLflow:
RFC-0008 (Skill Registry): A metadata-first registry for AI agent capabilities (skills, subagents, hooks). Stores metadata and typed source pointers (Git, OCI, ZIP, MLflow artifacts). Provides lifecycle management, usage analytics via MLflow traces (
mlflow.skill_context()), and federated discovery. Skill bundles group related capabilities into versioned units that map to agent harness plugins, with cross-registry references to MCP servers (RFC-0004).RFC-0009 (Harness Integration): Adds
mlflow skills installto generate harness-specific manifests and place files in the correct directories. Adapters for Claude Code, Codex CLI, and Cursor. Bundle import for onboarding existing plugins into the registry. Trace instrumentation via install-time manifests and harness hooks.Related Issues
This PR is from Bill Murdock with assistance from Claude Code.