[FEATURE] Feature request: consumer-side arbitrary source mappings

[yogurtearl]

Feature request: consumer-side arbitrary source mappings

At a glance

• Problem: APM can only install content when the repo or subpath already looks like a recognized package shape.
• Why it matters: many useful internal repos already contain agent guidance, but were never authored as APM packages.
• Example syntax:

dependencies:
  apm:
    - git: git@github.example.com:platform/engineering-guidance.git
      ref: main
      mappings:
        - from: docs/agents/**/*.instructions.md
          to: .github/instructions/
        - from: skills/review/**
          to: .agents/skills/review/

Is your feature request related to a problem? Please describe.
Yes. Teams often know exactly which files they want from an existing repo, but APM rejects the source unless the selected repo or subpath already looks like .apm/, SKILL.md, or plugin.json.

That blocks a very common adoption path:

• the source repo has useful agent docs or skill folders
• the consumer knows what to pull
• but the source repo has no APM metadata and no package boundary at the repo root

Describe the solution you'd like
Allow the consumer to declare file mappings for arbitrary repo layouts.

The key pieces would be:

• repo URL
• ref rules
• optional sparse checkout hints
• one or more from -> to mappings into APM-managed outputs

Example syntax:

dependencies:
  apm:
    - git: git@github.example.com:platform/engineering-guidance.git
      ref: v1.2.3
      mappings:
        - from: docs/agents/**/*.instructions.md
          to: .github/instructions/
        - from: skills/android-upgrade/**
          to: .agents/skills/android-upgrade/

This should work for both remote git dependencies and local paths, including repo-root-plus-subdirectory installs.

Describe alternatives you've considered

• Reshape the source repo into an APM package
• Point directly at a pre-existing SKILL.md folder
• Create a separate APM-only wrapper repo

Those options work sometimes, but they add migration cost, duplicate content, or depend on the source repo already being laid out the "right" way.

Additional context
This would make APM much easier to adopt in large codebases where shared guidance already exists, but nobody is going to stop and repackage every source repo before consumers can benefit from it.

[github-actions]

Triage decision

needs-design

Proposed labels

theme/security
area/content-security
area/distribution
type/feature
status/needs-design

(Note: type/feature label already present and preserved.)

Milestone

null

Suggested next action

Draft a design note specifying: glob validation rules (relative-only, sandbox enforcement), output path restrictions (APM-managed outputs only), git transport policy (https-only or also SSH?), lockfile schema for mapping dependencies, and conflict resolution when two mappings target the same output path.

Suggested issue comment

Thanks for filing this -- the adoption blocker is real: many useful repos contain agent guidance but were never packaged as APM packages, and the current "only recognized package shapes" restriction is a hard barrier.

Consumer-side arbitrary source mappings are accepted in principle. Before implementation we need a design note that answers:

1. **Glob validation**: the `from:` glob must be validated to stay within the checked-out repo (no `../`, no absolute paths, no symlink traversal). What glob library, and what validation rules?
2. **Output path restrictions**: the `to:` path must map to an APM-managed output directory, not an arbitrary filesystem location. What is the allowed set?
3. **Git transport policy**: the example syntax uses `git@...` (SSH SCP form). #1288 explicitly rejected SSH for confused-deputy reasons. Does this feature use the same policy, or does it need a separate justification?
4. **Lockfile schema**: mapping dependencies need a lock entry. What fields? (URL, ref, resolved SHA, mapping fingerprint for drift detection?)
5. **Conflict resolution**: if two mappings write to the same output path, which wins? What is the error behavior?
6. **Ref pinning**: branch refs (e.g., `ref: main`) are mutable. Should the schema recommend or require SHA/tag pinning?

Once the design note is in, this is a substantial but well-scoped feature. Would you like to draft it?

Per-lens notes (collapsed)

DevX UX Expert -- User-Need Reviewer

The adoption blocker is real and common: teams have existing repos with useful agent guidance that pre-date APM. Requiring source repos to be re-packaged as APM packages before consumers can use them is a significant adoption tax. The consumer-side mapping approach is a well-known pattern (similar to Cargo path dependencies or npm file: dependencies). The proposed syntax is clean and intuitive. High value if implemented safely.

Supply Chain Security Expert -- Risk-Surface Reviewer

This feature introduces arbitrary file copy from git remotes with user-defined glob patterns -- a significant new attack surface. Key risks: (1) the from: glob could escape the checkout sandbox via path traversal or symlinks; (2) the to: path could write to arbitrary filesystem locations; (3) the git@... SSH form in the example may bypass the https-only policy from #1288; (4) mutable branch refs (ref: main) mean the same lockfile entry can deliver different content on each install; (5) conflict resolution between mappings can silently overwrite managed files. theme/security and area/content-security apply strongly. The design must address all five before code lands.

OSS Growth Hacker -- Contributor-Tone Reviewer

yogurtearl is NONE association, with a clear adoption-blocker report and well-considered alternatives. Reply must be warm, validate the real pain, indicate direction (accepted in principle), and make the design questions concrete and actionable.

Python Architect -- Architecture Reviewer

This is a significant architecture addition: new dependency kind, new resolver path, new lockfile schema entries, glob matching with sandbox validation, output path restriction enforcement, and conflict resolution logic. Cross-cutting impact is high (dependency schema, lockfile schema, resolver, compile pipeline, apm install). The design surface is broad enough to warrant a formal design note before implementation. Status: needs-design.

Doc Writer -- Documentation Reviewer

New dependency syntax needs a guide page (example apm.yml, security guidance on ref pinning, output path rules). area/docs-site as secondary but pruned from label set to stay within the 6-label limit.

APM CEO -- Triage Arbiter

Accepted in principle -- this genuinely lowers the adoption barrier. But the security surface (arbitrary file copy with user-defined globs from arbitrary git remotes) is the highest-risk new feature in this sweep. Verdict: needs-design. The design must address glob sandbox, output path restrictions, transport policy, lockfile schema, and conflict resolution before any code lands. theme/security is mandatory.

{
  "decision": "needs-design",
  "decision_detail": "Design must specify: glob sandbox validation rules (no path traversal, no absolute paths), output path restrictions (APM-managed only), git transport policy (SSH SCP form policy vs #1288), lockfile schema for mapping deps, conflict resolution for overlapping to: paths, and ref pinning guidance.",
  "theme": "theme/security",
  "areas": ["area/content-security", "area/distribution"],
  "type": "type/feature",
  "status": "status/needs-design",
  "priority": null,
  "preserved_labels": [],
  "milestone": null,
  "next_action": "Draft a design note specifying glob validation rules, output path restrictions, git transport policy, lockfile schema for mapping dependencies, and conflict resolution rules.",
  "comment_markdown": "Consumer-side arbitrary source mappings accepted in principle. Before implementation, a design note is needed covering: glob sandbox validation (no path traversal), output path restrictions (APM-managed only), transport policy (does git@ SSH work here?), lockfile schema for mapping deps, conflict resolution, and ref pinning guidance. Would you like to draft it?"
}

---

Triage status: agentic proposal pending human ratification.
Silence is approval. Maintainers can:

• Override any label or milestone above by editing it directly --
  human edits are authoritative and will not be reverted on
  subsequent runs.
• Re-trigger triage by applying the status/needs-triage label, or
  by removing status/triaged to enroll the issue in the next
  daily sweep.

Posted by the Triage Panel workflow. See .apm/skills/apm-triage-panel for the panel skill.

Generated by Triage Panel · sonnet46 7.4M · ◷

[danielmeppiel]

North-star design proposal: consumer-side source mappings as a validated adapter, not a gate bypass

Posted by the apm-issue-autopilot design panel. Design-only: no code, no PR. Grounded in HEAD (deps/, models/dependency/reference.py, models/validation.py, PRINCIPLES.md).

(a) Problem framing

Today an APM source is admitted only if detect_package_type()
(src/apm_cli/models/validation.py) classifies the repo or selected
subpath as a recognized shape (APM_PACKAGE, CLAUDE_SKILL,
SKILL_BUNDLE, MARKETPLACE_PLUGIN, HOOK_PACKAGE); anything else is
INVALID and rejected. APM already supports narrowing (virtual
subdirectory deps like owner/repo/skills/review, sparse-checkout via
virtual_path), but the narrowed target must itself still look like a
package. The request is for repos that contain useful guidance but carry
no package boundary anywhere.

The crux: a raw consumer-supplied from -> to copy would let any file
from any repo land in APM-managed output dirs (.github/instructions/,
.agents/skills/, ...) with no shape validation, no integrity
record, and an unconstrained to: destination. That is a direct
supply-chain regression -- it bypasses the gate rather than extending it.

(b) North-star approach -- the consumer-authored adapter

Reframe the feature from "let the consumer copy arbitrary files" to
"let the consumer declare the package boundary that the producer never
wrote, and route that declaration through the SAME validation +
lockfile-integrity pipeline."

The gate's real job is that someone names and validates the boundary.
Today only the producer can. We let the consumer do it on the producer's
behalf -- but the artifacts still flow through the existing integrators
and the existing lockfile contract. The gate moves from
repo-level ("is this repo a package?") to artifact-level ("is
each mapped file a valid canonical primitive landing in a managed
namespace?"). It is preserved, not removed.

Concretely, a mapping entry:

1. Resolves the source exactly like any git dep (same host/URL parser in
   models/dependency/reference.py), pinned by ref: to an exact commit
   SHA. Dependency-confusion / typosquat posture is unchanged -- mappings
   do not widen which repo you trust, only which files inside an
   already-pinned, already-trusted repo are admitted.
2. For each from glob, materializes only the matched subset (reuse the
   existing sparse-checkout path; from globs become the sparse cone).
3. Routes each matched file into a declared primitive type so it
   passes through the corresponding integrator/validator
   (instruction_integrator, skill_transformer, etc.). A file that is
   not a valid primitive of its declared type is rejected -- exactly the
   guarantee the shape gate gave at repo level.
4. Constrains to: to the APM-managed canonical output namespaces
   only (the integrator target set), validated with
   ensure_path_within + validate_path_segments
   (utils/path_security.py). Arbitrary project paths (src/, .git/,
   ~) are rejected.

(c) How it preserves the supply-chain gate

• Which-repo trust: unchanged. Same URL/host validation, same pinned
  ref: -> resolved SHA in the lockfile.
• Which-bytes trust: the lockfile records resolved_commit plus a
  content_hash (already present, deps/lockfile.py) computed over the
  mapped subset, so apm install --frozen and apm audit re-verify
  the exact admitted bytes. Source tampering is tamper-evident.
• Shape trust: relocated to the artifact. Each mapped file must
  validate as the canonical primitive type it claims (P1: we do NOT
  inject apm-* frontmatter into the source; the file must already be a
  valid instruction / skill on its own -- consistent with "readable and
  useful with zero APM-specific tooling").
• Blast-radius trust: to: clamped to managed namespaces; no
  traversal, no overwrite of unmanaged files.
• Governance: add an org policy knob (extends the existing policy
  schema referenced in packages/apm-guide/.apm/skills/apm-usage/ governance.md) to disable consumer mappings or allowlist source hosts,
  so regulated orgs keep the strict repo-shape gate.
• Auditability (P6): every mapping is named and recorded; apm why /
  apm audit can explain "file X came from repo Y@SHA via mapping N." No
  silent normalization.

(d) Schema / compat / migration sketch

Purely additive. Existing string forms and detect_package_type are
untouched; the adapter branch is taken only when mappings: is present.

dependencies:
  apm:
    # existing forms keep working unchanged
    - owner/repo#v1.0.0
    # new: consumer adapter form
    - git: git@github.example.com:platform/engineering-guidance.git
      ref: v1.2.3              # required for adapters -> pinned to a SHA
      mappings:
        - from: docs/agents/**/*.instructions.md
          to:   instructions   # managed namespace, not a raw path
        - from: skills/review/**
          to:   skills/review  # under the skills namespace

• models/dependency/reference.py: new optional mappings (+
  is_consumer_adapter); to: is a namespace selector resolved by the
  integrator registry, not a free path, to keep the destination clamp
  total.
• Resolver: new branch synthesizes an in-memory package from the mapped
  subset, then runs artifact-level validation + integrators.
• deps/lockfile.py: persist mappings and per-mapping content_hash.
• Compat: no change to lockfile format for non-adapter deps; old
  lockfiles still load. No breaking change (P4 -- apm install ergonomics
  unchanged for everyone not using mappings).

(e) Open questions for the maintainer

1. to: surface: constrain to a fixed namespace selector set
   (recommended -- maximally safe, fully managed), or allow managed-root
   relative paths? The former is stricter; the latter is more flexible
   but widens the clamp logic.
2. Required pin: should ref: be mandatory for adapters (no floating
   main)? Recommended yes -- floating refs make content_hash
   re-verification meaningless and invite drift.
3. Primitive-type inference: infer the declared type from the to:
   namespace, or require an explicit as:/type: per mapping? Affects
   how strict artifact validation can be.
4. Local-path parity: the issue asks for local paths too. Do mapped
   local sources inherit the REMOTE-parent dual-reject boundary
   (deps/path_anchoring.py, issue fix: resolve transitive local paths against declaring package (#857) #940), or get their own rule?
5. Governance default: ship consumer mappings on with an opt-out
   policy, or off behind an allow flag for the first release?
6. Glob over-capture: hard-fail on zero matches (typo guard); should
   we also warn/deny when a from glob would pull obviously sensitive
   files (.env, keys), or leave that to the artifact validator?

(f) Panel perspectives

• devx-ux (lead): mental model is "authoring an adapter," not
  "hacking a copy." Least-surprise comes from to: being a managed
  namespace (can't scribble into src/) and from lock-pinned
  reproducibility. Keep the virtual-subdir shorthand for the common case;
  reserve mappings: for genuinely unstructured repos.
• apm-ceo (lead): north-star fit is strong -- this is the long-tail
  adoption unlock for the package-manager vision without violating P1 (no
  producer-side invented frontmatter) and without a breaking change. The
  boundary still exists; it just becomes consumer-declarable. Accept the
  written trade-off: shape validation moves repo -> artifact.
• oss-growth (lead): the wedge. Every org has internal guidance repos
  nobody will repackage; "add 4 lines to apm.yml" widens the funnel and
  makes consumer adapter manifests shareable (network effect, gateway to
  authoring real packages later).
• supply-chain-security: flexibility without bypass = unchanged
  repo-trust + pinned SHA + content_hash over the mapped subset +
  to: clamp + per-artifact integrator validation + an org governance
  knob. Threats explicitly handled: malicious source content (artifact
  validation), to: traversal (ensure_path_within), glob
  over-capture (zero-match fail + optional deny-list), ref drift (require
  pin).
• python-architect: a new DependencyReference variant parsed in
  models/dependency/reference.py; resolution reuses bare_cache +
  sparse-checkout with from globs as the cone; lockfile gains
  mappings + per-mapping content_hash. Additive, no migration for
  existing deps.

---

Recommendation: accept for design, implement behind the artifact-level
gate above. The one strategic decision the maintainer must ratify is
the written trade-off in PRINCIPLES terms: shape validation relocates
from repo-level to artifact-level, which keeps the supply-chain gate
intact while unlocking the long tail.

-- apm-issue-autopilot design panel

[yogurtearl]

Design response for microsoft/apm#1535 (status/needs-design)

TL;DR

Accept the feature in principle, but implement it as a validated consumer adapter (not a raw file-copy bypass):

• keep existing repo trust + ref resolution model
• allow consumer-declared mappings for unstructured repos
• enforce strict sandbox + destination restrictions
• keep lockfile integrity and deterministic replay

This unlocks adoption for legacy guidance repos while preserving APM's supply-chain posture.

---

Problem we need to solve

Many teams have useful guidance in repos that were never authored as APM packages.
Current install rules require a recognized package shape (.apm/, SKILL.md, plugin.json, etc.), so these sources are rejected even when the consumer knows exactly what content they need.

The outcome is predictable: teams either repackage everything (high friction) or duplicate content elsewhere (drift and governance pain).

---

Proposed design

1. New dependency form: consumer adapter mappings

Add an additive schema branch under dependencies.apm[]:

dependencies:
  apm:
    - git: https://github.com/org/repo.git
      ref: v1.2.3
      mappings:
        - from: docs/agents/**/*.instructions.md
          to: instructions
        - from: skills/review/**
          to: skills/review

Key point: to: is a managed namespace selector, not an arbitrary filesystem path.

2. Keep the gate, move it to artifact level

This should not bypass APM validation. Instead:

1. resolve source repo and pin commit SHA exactly as today
2. materialize only mapping matches
3. validate matched files as canonical primitives for the declared to: namespace
4. integrate only through existing managed outputs

Invalid primitives fail install.

3. Hard security constraints

from: validation

• must be repo-relative
• no absolute paths
• no ../
• no symlink escapes outside checkout root
• fail on zero matches (typo guard)

to: restrictions

• allow only APM-managed output namespaces
• reject unmanaged destinations (src/, .git/, home paths, etc.)
• reject path traversal / non-canonical targets

transport + trust

• use existing transport policy (no special SSH carveout unless separately approved)
• enforce pinned refs for adapter mappings (ref required, resolved SHA locked)

4. Lockfile changes (additive)

For mapping dependencies, persist:

• source URL + normalized identity
• requested ref + resolved commit SHA
• mapping spec fingerprint
• content hash over admitted mapped subset

This preserves deterministic replay and drift detection.

5. Conflict model

If two mappings target the same final managed path, default behavior should be hard error with clear provenance output.

Optional later extension: explicit conflict policy per mapping (e.g., strategy: fail|replace) but v1 should be strict fail.

---

Why this should pass design review

Security

It does not widen trusted source identity rules and does not allow arbitrary writes.
It keeps pinning, hashes, and managed-output constraints.

Architecture

It is additive: existing dependency forms and package-shape detection remain unchanged.
Only mappings enter the new adapter path.

Product adoption

It removes a major migration blocker for real-world repos without forcing immediate source repackaging.

---

Suggested phased rollout

Phase 1 (safe baseline)

• feature behind explicit opt-in
• strict validation (from, to, zero-match fail)
• ref required
• conflict strategy = fail
• lockfile + audit support

Phase 2 (ergonomics)

• better diagnostics (apm why provenance for mapped files)
• optional policy controls (org-level allow/deny for mappings)
• scoped enhancements after usage data

---

Open decisions for maintainers

1. to: as fixed namespace enum vs managed-root relative path
2. whether adapter mappings are enabled by default or policy-gated at first release
3. explicit primitive type declaration vs inference from to:
4. local-path adapter parity with remote dependencies

---

Recommendation

Approve a design based on consumer adapters with strict validation, not unconstrained copy semantics.

This approach captures the adoption upside while staying aligned with APM's security and reproducibility principles.