marketplace.packages[].source: generalize beyond `host/owner/repo` (deeply-nested GitLab subgroups, arbitrary git hosts)

[leocamello]

Summary

PR #1288 (merged) extended marketplace.packages[].source to accept <host.tld>/<owner>/<repo> and https://<host.tld>/<owner>/<repo>[.git]. That unblocks GitHub Enterprise and shallow GitLab layouts, but the regex pins exactly three path segments (host + owner + repo), and the test suite explicitly rejects 4+ segments (test_four_segment_path_rejected).

Self-hosted GitLab deployments commonly nest projects under multiple subgroups, e.g.:

gitlab.example.com/group/sub-group/team/projects/my-package

That's host + 4 path segments + repo. With the post-#1288 schema there is still no source form that names this repo — the schema rejects every URL form, the git: { url: ... } object form is rejected as "must be a non-empty string", and the bare <owner>/<repo> shorthand silently routes to github.com regardless of marketplace.owner.url. The apm view resolver, by contrast, already handles arbitrary-depth host-prefixed paths correctly — so this is a marketplace-schema gap, not a resolver-capability gap.

Repro

apm.yml:

name: test-mp
version: 0.1.0
description: repro
marketplace:
  owner:
    name: my-group
    url: https://gitlab.example.com/group/sub-group/team/projects
  outputs:
    claude: {}
  build:
    tagPattern: "v{version}"
  packages:
    - name: my-package
      description: deeply nested repo
      source: gitlab.example.com/group/sub-group/team/projects/my-package
      version: "^0.1.0"

$ apm marketplace check
[x] marketplace config error: 'packages[0].source' must be one of
    '<owner>/<repo>', '<host.tld>/<owner>/<repo>',
    'https://<host.tld>/<owner>/<repo>[.git]', or './<path>',
    got 'gitlab.example.com/group/sub-group/team/projects/my-package'

The object form sometimes recommended in authoring guides also fails:

      source:
        git: https://gitlab.example.com/group/sub-group/team/projects/my-package.git

[x] marketplace config error: 'packages[0].source' must be a non-empty string

Evidence the resolver itself isn't the blocker

$ apm view gitlab.example.com/group/sub-group/team/projects/my-package versions
Available versions:
  v0.1.0    tag      5d908cad
  main      branch   5d908cad

Evidence marketplace.owner.url is unused at resolution time

With:

marketplace:
  owner:
    url: https://gitlab.example.com/group/sub-group/team/projects
  packages:
    - source: my-group/my-package          # the only schema-accepted shorthand

I wrapped git on PATH to log every invocation. apm marketplace check ran:

git ls-remote --tags --heads https://github.com/my-group/my-package.git

The shorthand routed to github.com, ignoring owner.url. That's consistent with the schema docs (§7.3): owner.url is documented as "Owner homepage" — descriptive metadata, not a resolution input.

Why this matters

Self-hosted GitLab with subgroup nesting is the canonical enterprise pattern in many orgs. Until the schema accepts a form that names these repos, APM marketplaces can't list them — and the only schema-accepted shorthand silently exfiltrates the lookup to github.com under whatever auth happens to be in scope, which is also a small footgun for anyone publishing a marketplace from an enterprise tenant.

Suggested directions

Two options, not mutually exclusive:

Option A (preferred): a marketplace-level git base, packages name themselves relative to it

This is the shape every other package manager has converged on — npm's registry, cargo's [registries], Maven's <repositories>. Declare the host/path once, then entries don't repeat it. It sidesteps the "how many slashes" parser problem entirely.

A new field is cleaner than repurposing owner.url (which is documented as "Owner homepage" and the official example, https://github.com/contoso, is a profile page with no git semantics):

marketplace:
  sourceBase: https://gitlab.example.com/group/sub-group/team/projects
  owner:
    name: my-group
    url: https://gitlab.example.com/groups/group/sub-group/team/projects   # still a homepage
  packages:
    - name: my-package
      source: my-package                             # → sourceBase + my-package

    - name: external
      source: github.com/some-org/external           # #1288's host-prefixed form
                                                      # overrides sourceBase

Properties:

• Mixed-host marketplaces still work — fix: add support for custom github endpoints (host.tld/owner/repo) as apm.yml entrypoint for command apm pack (marketplace) #1288's host.tld/owner/repo form is the per-package escape hatch.
• The marketplace stays portable: forking/relocating means changing one line, not every entry.
• sourceBase is a strict prefix concatenation (no segment counting), so it composes with arbitrary GitLab subgroup depth, Bitbucket workspaces, Gitea orgs, etc.
• Validation: require https:// scheme, reject userinfo / port / query / fragment (same security posture as fix: add support for custom github endpoints (host.tld/owner/repo) as apm.yml entrypoint for command apm pack (marketplace) #1288).

(Field name sourceBase is illustrative — host, defaultHost, registry, etc. are all reasonable.)

Option B (mechanical fallback): widen the source regex to accept >3 path segments

Treat the last two slash-separated segments as owner/repo, everything before as host + group path. Same security guards from #1288 (no userinfo, no port, no query, https-only for URL form). Less elegant than Option A but a smaller diff if sourceBase is too big a change to land soon.

These compose well — Option A is the principled fix, Option B is available for one-off packages that don't fit the marketplace's base.

Decision needed from maintainers

Before anyone invests in implementation, it would help to know which direction lands cleanly with your design intent for the marketplace block:

• Option A only — introduce a marketplace-level base field; keep the source regex as-is.
• Option B only — widen the source regex to arbitrary depth; don't introduce a new field.
• Both — Option A as the primary path, Option B as the per-package escape hatch (my read of the trade-offs, but happy to be redirected).
• Neither / different shape — there's a direction we haven't considered.

Once a direction is chosen I'm happy to take a stab at the implementation PR (schema + tests + docs), tested against an actual deeply-nested self-hosted GitLab on our side. Just need the maintainer steer first to avoid building the wrong thing.

Out of scope (suggested for separate issues)

• SSH SCP form (git@host:path) — explicitly rejected by fix: add support for custom github endpoints (host.tld/owner/repo) as apm.yml entrypoint for command apm pack (marketplace) #1288 for confused-deputy reasons; not asking to revisit here.
• Non-https URL schemes — same.
• Per-host auth wiring beyond what fix: add support for custom github endpoints (host.tld/owner/repo) as apm.yml entrypoint for command apm pack (marketplace) #1288 already added.

References

• fix: add support for custom github endpoints (host.tld/owner/repo) as apm.yml entrypoint for command apm pack (marketplace) #1288 (merged 2026-05-26) — adds 3-segment host.tld/owner/repo form; pins the limit via test_four_segment_path_rejected.
• fix(artifactory): deterministic boundary probe for nested GitLab paths #1472 — handles arbitrary GitLab subgroup depth on the install side via Artifactory probe. Different surface, but informs how the marketplace schema could think about depth.
• [Feature] GitLab host support for marketplace and install #1195 (closed) — earlier GitLab host support pass.
• Marketplace add source parity with the Anthropic spec #676, [FEATURE] URL-based marketplace work Extension #692 — adjacent source-shape parity work.

[github-actions]

Triage decision

needs-design

Proposed labels

theme/governance
area/marketplace
area/enterprise
type/feature
status/needs-design

Milestone

null

Suggested next action

Draft a design note specifying the sourceBase field name, validation rules (https-only, no userinfo/port/query/fragment), interaction with the existing host-prefixed form from #1288, default value (none vs github.com), and migration behavior for existing bare owner/repo sources.

Suggested issue comment

Thanks for a thorough report -- the evidence trail (resolver handles arbitrary depth, schema pins at 3 segments, bare shorthand silently routes to github.com) is exactly the level of detail that makes a triage decision easy.

The direction is Option A: a marketplace-level `sourceBase` field that declares the git base for package entries, with the per-package host-prefixed form from #1288 as the per-entry override. That design matches how npm, cargo, and Maven handle this and avoids the "how many slashes" parser problem.

Before implementation we need a design note that answers:
- Exact field name (`sourceBase`, `registry`, `host`, `defaultHost` -- pick one with rationale)
- Validation rules for `sourceBase` (https-only, no userinfo, no port, same security posture as #1288)
- Default behavior when `sourceBase` is absent (current behavior preserved: no base, bare owner/repo routes to github.com)
- Interaction with #1288's host-prefixed per-entry form (override rule)
- Whether Option B (widen the source regex) is included as a complementary escape hatch or deferred

If you would like to own the implementation, a design note in a GitHub comment or linked doc is a good first step. The #1288 PR and its security rationale are the starting point for the validation rules.

See also: https://github.com/microsoft/apm/discussions/116 for the public roadmap.

Per-lens notes (collapsed)

DevX UX Expert -- User-Need Reviewer

Self-hosted GitLab with subgroup nesting is the canonical enterprise pattern. The current 3-segment limit makes APM marketplaces unusable for a large class of enterprise deployments. Option A (sourceBase field) matches the mental model of every major package manager and is the right shape. The author's evidence that marketplace.owner.url is unused at resolution time (it routes to github.com regardless) is a concrete footgun that Option A also fixes.

Supply Chain Security Expert -- Risk-Surface Reviewer

The sourceBase field is a new source-routing input in the marketplace schema -- a governance surface. The security guards from #1288 (https-only, no userinfo, no port, no query, no fragment) must apply to sourceBase as strictly as they do to per-entry source values. Option A actually reduces the footgun of bare owner/repo silently routing to github.com. Design must specify that sourceBase must be https-scheme and that per-entry absolute URLs override it without inheriting its restrictions. theme/governance, area/marketplace, area/enterprise.

OSS Growth Hacker -- Contributor-Tone Reviewer

leocamello is NONE association, with an enterprise deployment and a polished, actionable report. The reply must give a concrete direction steer (Option A is preferred) immediately -- the author explicitly asked for this before implementing. Warmly invite them to own the implementation once the design note is in.

Python Architect -- Architecture Reviewer

Option A requires: new sourceBase field in the marketplace schema validator, updated resolver to apply the base as a prefix, interaction with the existing per-entry source forms (#1288), and migration semantics (existing configs without sourceBase must resolve as before). The cross-cutting impact touches schema validation, resolver, and docs. A design note is warranted before code lands so that the field name, validation rules, and override semantics are agreed. Status: needs-design.

Doc Writer -- Documentation Reviewer

The marketplace schema docs need a new sourceBase entry with examples for deeply-nested GitLab and other hosts. A migration note (existing configs continue to work) is needed. area/docs-site is a secondary area pruned from the label set to stay within the 6-label limit.

APM CEO -- Triage Arbiter

Direction is clear (Option A), but the design surface (field name, validation, override semantics) must be settled before code lands to avoid a second schema revision. Verdict: needs-design. theme/governance because the sourceBase field is a trust-routing input for marketplace source resolution.

{
  "decision": "needs-design",
  "decision_detail": "Design must specify: sourceBase field name, validation rules (https-only, no userinfo/port/query/fragment), interaction with #1288 host-prefixed per-entry form, default behavior when absent, and whether Option B (widen source regex) is included.",
  "theme": "theme/governance",
  "areas": ["area/marketplace", "area/enterprise"],
  "type": "type/feature",
  "status": "status/needs-design",
  "priority": null,
  "preserved_labels": [],
  "milestone": null,
  "next_action": "Draft a design note specifying the sourceBase field name, validation rules, interaction with the existing host-prefixed form from #1288, default value, and migration behavior for existing bare owner/repo sources.",
  "comment_markdown": "Direction is Option A (marketplace-level sourceBase field). Before implementation, a design note is needed specifying the field name, validation rules (https-only, matching #1288), default behavior, and override semantics. If you would like to own the implementation, a design note in a GitHub comment is a good first step."
}

---

Triage status: agentic proposal pending human ratification.
Silence is approval. Maintainers can:

• Override any label or milestone above by editing it directly --
  human edits are authoritative and will not be reverted on
  subsequent runs.
• Re-trigger triage by applying the status/needs-triage label, or
  by removing status/triaged to enroll the issue in the next
  daily sweep.

Posted by the Triage Panel workflow. See .apm/skills/apm-triage-panel for the panel skill.

Generated by Triage Panel · sonnet46 7.4M · ◷

[leocamello]

Design note for the agreed direction (Option A — a marketplace-level sourceBase), answering the five questions raised in triage. Happy to own the implementation PR (schema + resolver/build + tests + docs) once you confirm the shape below.

Decision summary

Following the triage steer (Option A), this note specifies a marketplace-level
sourceBase field that declares the git base for package entries, with the
per-entry host-prefixed / full-URL forms from #1288 acting as per-entry
overrides. Option B (widening the per-entry source regex to arbitrary depth)
is deferred — sourceBase covers the enterprise-nesting case, and a deeply
nested one-off entry can always be written as a full https://… URL override.

The five questions the triage panel asked are answered below.

---

1. Field name: sourceBase

sourceBase over the alternatives:

• registry implies an HTTP API contract (npm registry, cargo [registries]
  resolve through a service). APM has no such service — the value is purely a git
  base that gets string-composed into a clone URL. Calling it registry would
  promise resolution semantics we do not have.
• host / defaultHost are misleading: the value is host + group path
  (gitlab.example.com/group/sub-group/team/projects), not a bare host. The
  field's whole reason to exist is that the path is deeper than a host.
  defaultHost additionally connotes "fallback when none given" rather than
  "base everything composes onto".

sourceBase reads as "the base that source values are relative to", and the
camelCase matches the existing build.tagPattern sub-key convention in the
marketplace block.

2. Validation rules

sourceBase is a source-routing input, so it inherits #1288's security posture
exactly, with one deliberate relaxation (depth):

Accepted:

• Scheme must be https://. (No http, git, ssh, file, or SCP
  git@host:path — same confused-deputy rationale as fix: add support for custom github endpoints (host.tld/owner/repo) as apm.yml entrypoint for command apm pack (marketplace) #1288.)
• Host must be a FQDN (the existing _HOST_PAT: dotted, RFC-1123 charset).
• Arbitrary path depth after the host — this is the one rule that differs
  from per-entry source. Each path segment is restricted to the same charset
  as owner/repo ([A-Za-z0-9._-]) and run through
  validate_path_segments() (no ./.., no traversal, no empty segments).

Rejected (mirrors #1288):

• userinfo (user@host)
• port (host:443)
• query string (?…)
• fragment (#…)
• trailing .git (a base is not a repo)

Normalization: a trailing / is stripped at parse time so composition is a clean
base + "/" + source.

A new regex sits beside the existing ones in yml_schema.py, e.g.:

# https://host.tld/<segment>(/<segment>)*  — arbitrary depth, no .git, no query/fragment/port/userinfo
SOURCE_BASE_RE = re.compile(rf"^https://{_HOST_PAT}/{_SEGMENT}(?:/{_SEGMENT})*$")

with _SEGMENT = r"[A-Za-z0-9._-]+".

3. Default behavior when sourceBase is absent

Unchanged from today. No base is applied; a bare owner/repo resolves to the
default host (GITHUB_HOST env or github.com). Every existing marketplace
manifest keeps resolving byte-for-byte as it does now — the behavioral fork in §4
only exists when sourceBase is explicitly declared. There is no migration step
for existing configs.

4. Interaction with #1288 per-entry forms (override rule) — "relative to base, always"

When sourceBase is set, a source value is classified by the existing
split_host_from_source() detector (which already keys off "first segment is a
dotted FQDN"):

source shape	Has host?	Resolves to
./local-pkg	n/a (local)	local path, base ignored
https://host/a/b[.git] (full URL)	yes (URL)	the URL, base ignored (override)
host.tld/owner/repo (#1288 host-prefixed)	yes	that host, base ignored (override)
my-package (relative, 1 seg)	no	sourceBase + /my-package
owner/repo (relative, 2 seg)	no	sourceBase + /owner/repo
a/b/c (relative, N seg)	no	sourceBase + /a/b/c

So the rule is simply: split_host_from_source() returns a host → override;
returns no host → compose onto sourceBase. No segment counting (this is what
the issue meant by "sidesteps the 'how many slashes' parser problem").

Consequences, explicitly:

• To target github.com from inside a sourceBase marketplace, write the explicit
  github.com/owner/repo form. Bare owner/repo no longer silently routes to
  github.com when a base is declared — this is the footgun the supply-chain
  lens flagged, and removing it is intentional.
• Mixed-host marketplaces still work: every non-base host is reachable via the
  fix: add support for custom github endpoints (host.tld/owner/repo) as apm.yml entrypoint for command apm pack (marketplace) #1288 host-prefixed form or a full URL.
• Per-entry overrides do not inherit the base's path — they are absolute.

Validation interaction

When sourceBase is present, the per-entry source validator must additionally
accept a single-segment relative form (my-package) and arbitrary-depth
relative forms (a/b/c), which today's SOURCE_RE rejects. When sourceBase is
absent, the single-segment form stays rejected (there is no base to attach it to —
ambiguous). Validation thus becomes context-aware of whether the base is set.

5. Option B (widen per-entry source regex): deferred

Not included in this PR. Rationale: sourceBase + relative entries already names
deeply nested repos, and a one-off deeply nested entry on a different host can
be written as a full https://host/a/b/c/repo URL override. If a concrete need
for bare host-prefixed depth (host.tld/a/b/c/repo without a base) appears later,
it is a small complementary regex widening tracked as a follow-up. Flagging this
explicitly so the override surface stays minimal for the security review.

---

Resolution / architecture impact (for the python-architect & primitives lenses)

The clean integration point already exists. _marketplace_https_git_url()
(resolver.py) prefers source.url — the canonical URL — and its docstring
already calls out "self-managed GitLab nested groups that owner/repo
round-tripping cannot reconstruct correctly". So:

1. Parse (yml_schema.py): add source_base: str | None to
   MarketplaceConfig; register sourceBase in _APM_MARKETPLACE_KEYS; validate
   with SOURCE_BASE_RE + validate_path_segments. _parse_package_entry keeps
   using split_host_from_source; a relative entry simply keeps host=None and
   its raw source string.
2. Compose (builder / output_mappers.py): at the boundary that emits each
   package's effective git URL, if the entry is relative (host is None, not
   local) and source_base is set, emit the canonical
   source_base.rstrip("/") + "/" + source as a full https://… URL. Host-prefixed
   and URL entries already carry their own host/url and flow through unchanged.
3. The resolver then consumes the canonical URL via the existing source.url
   path — no change to _marketplace_https_git_url itself, and no need to teach
   the host field to carry a path.

This keeps sourceBase a manifest-and-build-time concern; the resolver stays
URL-driven exactly as it is for the GitLab-nested case today.

Files in the implementation PR (preview, for scope agreement)

• src/apm_cli/marketplace/yml_schema.py — field, regex, validation, override.
• src/apm_cli/marketplace/yml_editor.py — mirror the validation + error string.
• src/apm_cli/marketplace/builder.py / output_mappers.py — base composition.
• tests/unit/marketplace/ + tests/integration/test_yml_schema*.py — relative
  resolution, override precedence, https-only / no-userinfo / no-port / no-query /
  no-fragment rejection, absent-base unchanged behavior, the 4+ segment GitLab
  repro from this issue now passing.
• docs/.../reference/manifest-schema.md + package-authoring guide — schema entry,
  GitLab example, migration note ("existing configs unchanged").
• OpenAPM spec three-edit ritual iff the maintainers consider the marketplace
  source-resolution rule normative (anchor + requirements.yml row + @req test) —
  flagging for a maintainer call.

Security posture recap (for the supply-chain lens)

• sourceBase validated as strictly as fix: add support for custom github endpoints (host.tld/owner/repo) as apm.yml entrypoint for command apm pack (marketplace) #1288's per-entry source: https-only,
  no userinfo/port/query/fragment, FQDN host, validate_path_segments on the path.
• Net reduction in footgun: bare owner/repo no longer silently exfiltrates a
  lookup to github.com under ambient auth once a base is declared.
• Per-entry absolute overrides are validated by the existing fix: add support for custom github endpoints (host.tld/owner/repo) as apm.yml entrypoint for command apm pack (marketplace) #1288 rules; they do
  not inherit the base, so a base cannot be used to smuggle a path onto an override.

Out of scope (unchanged from the issue)

SSH SCP form, non-https schemes, per-host auth wiring beyond #1288.