feat(bundle): add non-activating package resources

[Daniel Meppiel (danielmeppiel)]

add(bundle): preserve non-activating package resources

Description

TL;DR

Add opt-in resources directory roots, apm pack --format apm --source, and apm unpack --source for package-owned contracts/checks that must travel as inert bytes. Preserve original manifest, lockfile, resource paths and contents in the existing APM directory/ZIP/tar envelope. Restore only into a new directory after mandatory verification; unsupported deployment/plugin routes refuse instead of silently dropping resources.

Important

This is working-draft, source-only functionality, not a normative OpenAPM change or an official native release. No APMX backend repin, model invocation, version bump, release or tag is included.

Problem (WHY)

• [!] Independent contracts/checks need packaging without becoming discoverable skills or deployed instructions. Keeping them out of fake carrier skills follows the Agent Skills guidance to "encapsulate a coherent unit of work that composes well with other skills.".
• Review reproduced an envelope-hash replacement race, a legacy-lock marker bypass, and replacement of a concurrently created empty destination. Regression traps now exercise each real boundary, following "do the work, run a validator (a script, a reference checklist, or a self-check), fix any issues, and repeat until validation passes.".

The missing capability is an explicit non-activating resource route, not a claim that plugin formats must represent arbitrary package files.

Approach (WHAT)

• Select nonempty, nonoverlapping package-relative directories with resources: [contracts, checks]; omit $schema to use the working draft.
• Reuse the existing APM envelope: outer apm.lock.yaml holds pack.source: true and pack.bundle_files; package/ holds untouched author metadata and resources.
• Restore exact payload bytes without hooks, checks, scripts, dependency installation, primitive activation, compilation, marketplace output or network acquisition.
• Reject unsafe/ambiguous content, unverifiable inventories, verification bypasses and occupied outputs. Probe source markers in both apm.lock.yaml and apm.lock.
• Preserve ordinary unmarked legacy YAML semantics, including large locks with merge overrides; strict source metadata rules stay source-only.

Implementation (HOW)

Code paths below are relative to src/apm_cli/; other paths are repository-relative.

Files	Intent
models/apm_package.py, models/package_resources.py	Canonical working-draft parsing and bounded directory selection; no per-file resource inventory or activation semantics.
bundle/source_package.py	Exact source envelope, original-hash authority through staged verification, marker admission and fresh-output publication.
bundle/packer.py, bundle/unpacker.py, bundle/lockfile_enrichment.py	Route explicit source operations through existing pack metadata, archive and integrity machinery.
bundle/plugin_exporter.py, bundle/agent_plugin_exporter.py, bundle/local_bundle.py, models/format_detection.py	Refuse unsupported resource export and source-envelope deployment across admission paths.
commands/pack.py, cli.py, core/build_orchestrator.py	Source flags, matching lazy/resolved help, compatible-option checks, source-only production, JSON output and truthful restore guidance.
agent_plugins/assets.py, utils/archive.py, utils/path_security.py, utils/yaml_io.py	Bounded verified reads; source-only strict archive/path/YAML admission; ordinary-mode preservation and special-bit rejection.
utils/atomic_io.py	Shared native atomic no-replace directory publication; no replacing POSIX fallback. ZIP/tar publication retains exclusive hard-link creation.
.apm/architecture/owners/{contracts-tooling,marketplace-plugins}.json, scripts/architecture_linter/checks/marketplace_package_and_registration.py	Extend canonical ownership and register mutation guards for resource, integrity and publication boundaries.
tests/unit/bundle/test_source_package.py, tests/unit/utils/test_atomic_io.py, tests/unit/scripts/test_architecture_runner.py, tests/integration/test_source_package_cli.py, tests/integration/test_architecture_source_resources.py, tests/utils/source_package.py	Real CLI roundtrips, adversarial input, review regression traps, exact rule inventory, native-capability controls and reusable inert fixtures.
docs/src/content/docs/producer/pack-a-bundle.md, docs/src/content/docs/reference/{manifest-schema,cli/pack,cli/unpack}.md, packages/apm-guide/.apm/skills/apm-usage/{commands,package-authoring}.md	Describe opt-in, supported formats, nonactivation, exact metadata, safety limits and trust boundaries.

Diagrams

The new source path keeps original author files separate from envelope metadata; deployment admission is a refusal branch, not restoration.

flowchart LR
    subgraph Pack["Opt-in source packing"]
        A["apm.yml resources and original lock"]
        B["collect_package_resources"]
        C["pack_source_package"]
    end
    subgraph Envelope["Existing APM envelope"]
        D["pack.source and bundle_files hashes"]
        E["package: exact author files"]
    end
    subgraph Restore["Verified inert restoration"]
        F["_verify_source_package"]
        G["_copy_assets: original envelope hashes"]
        H["publish_directory_noreplace: new output only"]
    end
    A --> B --> C
    C --> D
    C --> E
    D --> F
    E --> F
    F --> G --> H
    D --> J["reject_source_deployment"]
    J --> K["install and legacy unpack refuse"]
    classDef new stroke-dasharray: 5 5;
    class B,C,D,F,G,H,J new;

Loading

Trade-offs

• Explicit local route, not universal export: no plugin representation, remote non-activating acquisition, transitive resource flattening or fake carrier skills. Existing git install is not relabeled non-activating.
• Envelope rather than metadata rewriting: the extra package/ prefix preserves exact original apm.yml and apm.lock.yaml, including comments and line endings; restoration removes only that prefix.
• Unsigned hashes, not publisher authentication: SHA-256 detects byte corruption against the envelope, but an attacker can replace both. Ordinary modes are preserved where supported, not authenticated by the byte inventory; trust an external archive checksum when modes/ownership/timestamps matter. No arbitrary-content secret scanning is claimed.
• Fail closed rather than overwrite: regular bounded content only; no escaping paths, symlinks, aliases or special bits. Source metadata is capped at 4 MiB each, inventory at 10,000 entries/512 MiB, paths at 4,096 UTF-8 bytes/128 segments. Unsupported native no-replace capability or filesystem support causes failure.
• Platform evidence is bounded: native execution was macOS. Linux binding/flag/failure controls are mocked; native Linux and Windows execution is not claimed by these local results. Root README, CHANGELOG, normative spec and release metadata are unchanged.

Benefits

1. Directory, ZIP and tar.gz roundtrips preserve the in-repository 15-resource fixture plus both original metadata files.
2. A separate current factory proof preserves 19 resources plus manifest/lock: 21 payload files with identical paths, SHA-256, sizes and ordinary modes after author removal.
3. Mutation, legacy-marker bypass and concurrent-destination replacement now refuse before publishing output; consumer-owned output remains unchanged.

Issue and approved scope

Issue: Related to #3174 (bounded upstream source-resource tracking).

Human scope-approval comment: Pending / not publicly recorded. The bounded implementation and draft publication were explicitly approved in private maintainer conversation, but no public human scope-approval comment or review-contact designation exists yet. This draft does not substitute conversational approval, an automated recommendation, or a label for that required record.

This PR implements the bounded upstream source-resource route. Public human scope approval, review-contact designation, and PR/CI review remain pending; no issue-closing keyword is used. Official release/native assets and downstream APMX repinning remain separate, excluded gates.

Warning

Keep this PR draft and ineligible for ready/merge until a responsible maintainer posts the real public scope record. No scope-approval decision, acceptance label, private-security exemption, or trivial-documentation exemption is asserted.

Type of change

• Bug fix
• New feature
• Documentation
• Maintenance / refactor

Testing

• Tested locally
• All existing tests pass
• Added tests for new functionality (if applicable)

Validation

Scoped results at reviewed 726f492, not a full-suite or remote-CI claim: 868 passed, 1 skipped, 64 quality tests passed, 14 docs-tool tests passed. The expected duplicate-ZIP warning comes from an adversarial fixture. Independent follow-up reported 156 focused tests passed plus disposable mutation, legacy-admission and real macOS collision probes; all three findings were cleared.

CI follow-up: the first Linux shard-2 run exposed a stale lazy unpack help string and the missing new rule ID in the exact architecture inventory. Both failures were reproduced locally, then corrected without weakening either assertion or changing resource logic. The expanded regression command passed:

uv run --frozen --extra dev pytest -q tests/unit/test_cli_lazy_dispatch.py tests/unit/scripts/test_architecture_runner.py tests/unit/bundle/test_source_package.py tests/unit/utils/test_atomic_io.py tests/integration/test_source_package_cli.py tests/integration/test_architecture_source_resources.py --tb=short

Result: 236 passed, 1 expected duplicate-ZIP warning in 39.73s. Updated remote CI remains a separate gate.

Exact impacted test command and recorded summary

uv run --extra dev pytest -q \
  tests/unit/bundle \
  tests/unit/test_packer.py tests/unit/test_unpacker.py tests/unit/test_unpack_security.py \
  tests/unit/test_bundle_formats.py tests/unit/commands/test_pack.py \
  tests/unit/commands/test_pack_cli_flags.py tests/unit/commands/test_pack_cli_surface.py \
  tests/unit/commands/test_pack_phase3.py tests/unit/commands/test_unpack_deprecation.py \
  tests/unit/core/test_build_orchestrator.py tests/unit/utils/test_archive.py \
  tests/unit/utils/test_atomic_io.py tests/unit/test_yaml_io.py tests/unit/test_apm_package.py \
  tests/unit/agent_plugins tests/unit/install/test_install_local_bundle.py \
  tests/unit/install/test_install_local_bundle_issue1207.py tests/unit/install/test_frozen.py \
  tests/unit/install/test_frozen_roots.py tests/integration/test_source_package_cli.py \
  tests/integration/test_architecture_source_resources.py \
  tests/integration/test_architecture_package_format_precedence.py \
  tests/integration/test_architecture_bundle_format.py \
  tests/integration/test_architecture_pack_lockfile_read.py --tb=short

868 passed, 1 skipped, 1 warning in 38.73s

Exact quality, docs-tool and canonical pre-push commands

uv run --extra dev pytest -p no:cacheprovider -q tests/quality --tb=short

64 passed in 43.40s

npm --prefix docs run test:links

ℹ tests 14
ℹ pass 14
ℹ fail 0

The docs tool's intentional broken-link fixture diagnostic is expected; no full Astro build is claimed. Each command below passed on the reviewed HEAD after refreshing/merging origin/main (already up to date):

git merge --no-edit origin/main
uv run --extra dev ruff check src/ tests/ scripts/lint_architecture_boundaries.py scripts/architecture_linter/ scripts/windows_native_symlink_probe_entry.py
uv run --extra dev ruff format --check src/ tests/ scripts/lint_architecture_boundaries.py scripts/architecture_linter/ scripts/windows_native_symlink_probe_entry.py
uv run --extra dev python -m pylint --disable=all --enable=R0801 --min-similarity-lines=10 --fail-on=R0801 src/apm_cli/ scripts/lint_architecture_boundaries.py scripts/architecture_linter/ scripts/windows_native_symlink_probe_entry.py
bash scripts/lint-auth-signals.sh
bash scripts/lint-architecture-boundaries.sh
uv run --frozen python scripts/check_test_assertions.py
uv run --frozen python scripts/check_exact_test_duplicates.py
git diff --check

Ruff: 1930 files formatted; pylint: 10.00/10; auth/architecture and both test ratchets clean. The YAML-write, 2100-line and raw-relative-path guards also passed using equivalent Python regex/file scans on macOS.

Scenario Evidence

#	Scenario (user promise)	Principle(s)	Test(s) proving it	Type
1	Restore exact author files after the author checkout is removed; preserve the consumer seed.	Portability by manifest, Secure by default	tests/integration/test_source_package_cli.py::test_source_archive_survives_author_removal	e2e
2	Packing/restoring data never runs scripts or invokes other build producers.	Secure by default	tests/unit/bundle/test_source_package.py::test_no_execution_and_no_other_build_producers	integration (in-process component)
3	Reject corrupted, missing, unsafe or ambiguous source content.	Secure by default	tests/integration/test_source_package_cli.py::test_real_cli_rejects_damaged_source_archives; tests/unit/bundle/test_source_package.py::test_strict_archives_reject_ambiguous_members	e2e / integration
4	A source mutation cannot replace the original envelope's expectations.	Secure by default	tests/unit/bundle/test_source_package.py::test_reinventory_cannot_replace_envelope_hash_authority (review regression trap)	integration (in-process component)
5	Renaming a source marker to the legacy lock name cannot enable deployment; ordinary legacy locks still work.	Secure by default, DevX (pragmatic as npm)	tests/unit/bundle/test_source_package.py::test_all_source_marker_names_block_deployment; test_source_probe_preserves_legacy_metadata_semantics (review regression traps)	integration (in-process component)
6	A competing output created during publication is never replaced.	Secure by default	tests/unit/bundle/test_source_package.py::test_directory_publication_preserves_racing_empty_destination; test_archive_publication_preserves_racing_destination (review regression traps)	integration (in-process component)
7	Unsupported export formats and explicit normative schema opt-ins refuse the new resource contract.	Portability by manifest, DevX (pragmatic as npm)	tests/unit/bundle/test_source_package.py::test_deployment_formats_refuse_resource_loss; test_resources_are_not_a_normative_contract	integration (in-process component)

Separate real-factory evidence, without private paths or logs

An isolated copy of the current factory contained 19 resources plus original author-fixture manifest and a genuine CLI-generated lock. Frozen dry-run validated the retained lock; ZIP/tar source packing and restoration preserved all 21 path/hash/size/mode identities, including lock mode 0600. Only the owned author copy was deleted before restoration; the original checkout stayed unchanged and staging was cleaned.

Recorded argument vectors below replace the machine-specific interpreter and scratch paths with variables; they are not native-release commands:

"$PYTHON" -m apm_cli.cli install --frozen --dry-run --target copilot --no-trust-bin
"$PYTHON" -m apm_cli.cli pack --format apm --source --archive --archive-format zip -o "$PROOF/archives"
"$PYTHON" -m apm_cli.cli pack --format apm --source --archive --archive-format tar.gz -o "$PROOF/archives"
"$PYTHON" -m apm_cli.cli unpack --source "$PROOF/archives/checkout-factory-0.1.0-dev.zip" -o "$PROOF/restored-zip"
"$PYTHON" -m apm_cli.cli unpack --source "$PROOF/archives/checkout-factory-0.1.0-dev.tar.gz" -o "$PROOF/restored-tar-gz"

All five commands exited 0. The ordinary frozen dry-run is author-fixture lock validation, not a claim that install is non-activating. No raw proof logs, private paths or tokens are published.

How to test

• Run uv run --extra dev pytest -q tests/unit/bundle/test_source_package.py tests/unit/utils/test_atomic_io.py tests/integration/test_source_package_cli.py tests/integration/test_architecture_source_resources.py from this checkout; roundtrips and review regression traps must pass.
• Use the public tests/utils/source_package.py::make_source_package fixture in a disposable directory, then run this checkout's CLI with pack --format apm --source --archive; restore with unpack --source ARCHIVE -o NEW_DIRECTORY and compare every payload byte.
• Repeat with --archive-format tar.gz, and without --archive for directory form; original manifest/lock bytes must remain unchanged.
• Run restore against an existing directory or with --skip-verify; expect refusal and untouched consumer content. Try ordinary install/unpack of a source envelope; expect refusal, not activation.

Spec conformance (OpenAPM v0.1)

If this PR changes behaviour that an OpenAPM v0.1 req-XXX covers,
confirm the three-step ritual in the
development guide:

• Spec edit: docs/src/content/docs/specs/openapm-v0.1.md updated
  (new/changed <a id="req-XXX"></a> anchor + prose + Appendix C
  row).
• Manifest edit: docs/src/content/docs/specs/manifests/openapm-v0.1.requirements.yml
  updated.
• Test edit: a @pytest.mark.req("req-XXX") test under
  tests/spec_conformance/ added or extended.
• CONFORMANCE.{md,json} regenerated via
  uv run --extra dev python -m tests.spec_conformance.gen_statement
  and committed.
• N/A -- this PR does not change OpenAPM-observable behaviour.

resources is rejected with an explicit normative $schema; no normative specification or schema is revised.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com