Build registry-driven wrangle documentation

registry/CONTRACT.md

@@ -1,6 +1,6 @@
 # Wrangles Registry Contract
 
-Status: pilot
+Status: pre-production
 
 Contract version: 0.1
 
@@ -45,7 +45,7 @@ manifest and any normalized Registry entry. Aggregate category pages, the
 index, and the template are inventoried separately as supporting Markdown;
 they are not treated as one-to-one wrangle records.
 
-For the first migration pass, conflicting or incomplete source content is
+During the initial migration, conflicting or incomplete source content is
 resolved in this order:
 
 1. WranglesPY callable signature, implementation, and tests
@@ -83,10 +83,10 @@ required for Registry inclusion. Its signature and tests may be used for parity
 checks, but the Registry pipeline must not depend on finding one.
 
 Registry Markdown keeps the full parameter records, including descriptions,
-roles, accepted-value constraints that cannot be inferred reliably from Python
-annotations, and examples. The compiler merges those records with the runtime
-manifest, rejects name/required/default drift, and generates the recipe JSON
-Schema from the merged result.
+presentation groups, accepted-value constraints that cannot be inferred
+reliably from Python annotations, and examples. The compiler merges those
+records with the runtime manifest, rejects name/required/default drift, and
+generates the recipe JSON Schema from the merged result.
 
 The current embedded JSON Schema docstrings in `recipe_wrangles` are migration
 input, not a second permanent authoring source. Their complete overview and
@@ -157,7 +157,7 @@ Each entry separates identity, executable naming, compatibility, and routing:
 - `slug` is the public documentation path below `/wrangles/`. Changing it
   requires a redirect.
 
-`aliases` contains still-supported legacy recipe keys. The first-pass Registry
+`aliases` contains still-supported legacy recipe keys. The initial Registry
 keeps `classify`, `lookup`, and other ungrouped keys exactly as the current code
 exposes them. When the dotted migration is implemented, the new key can become
 canonical without changing `id`, and the former key can move to `aliases` for
@@ -197,6 +197,22 @@ Parameter records contain human guidance plus a JSON Schema fragment. Every
 runtime-owned parameter fact must reconcile with the pinned manifest and be
 traceable to the named runtime symbol.
 
+Every parameter also declares one `param_group` from a small shared vocabulary:
+
+- `I/O` identifies input and output columns or structures.
+- `Options` contains wrangle-specific settings users need to achieve the
+  desired behavior.
+- `Formatting` controls the shape or presentation of results.
+- `Conditions` contains the shared `if`, `where`, and `where_params` controls.
+- `Execution` controls batching, concurrency, time limits, and nested execution.
+- `Errors` controls fallbacks, retries, and failure handling.
+- `Details` contains model, provider, connection, credential, cache, and other
+  supporting technical settings.
+
+The groups are intentionally fixed rather than specialized by wrangle. They
+organize the human documentation and remain available to agent and product
+consumers in the compiled JSON contracts.
+
 `runtime_default` means the value used when the argument is omitted. It is not
 a playground placeholder. Example or UI starter values belong in examples,
 not in `runtime_default`.
@@ -222,7 +238,7 @@ verification level:
 - `live`: a separate controlled job executes an external service.
 - `manual`: the example requires a documented human check.
 
-The pilot compiler implements `static` verification. The entries remain
+The compiler currently implements `static` verification. The entries remain
 `static` until an automated WranglesPY execution check is added; output
 fixtures do not by themselves constitute execution evidence.
 
@@ -241,8 +257,8 @@ The compiler produces:
 - one comprehensive JSON contract per wrangle under
   `wrangles-docs/static/registry/contracts/`, including all parameters
 - raw source Markdown and sanitized example fixtures
-- a pilot recipe JSON Schema under
-  `wrangles-docs/static/schemas/recipes/pilot/schema.json`
+- a pre-production recipe JSON Schema under
+  `wrangles-docs/static/schemas/recipes/registry/schema.json`
 - deterministic JSON and Markdown reconciliation reports under
   `registry/reports/`
 
@@ -256,7 +272,8 @@ and environment-specific values are excluded. A clean compile followed by
 
 ## Versioning and lifecycle
 
-The pilot Registry version is `0.1.0-pilot`. A production release will contain:
+The pre-production Registry version is `0.1.0`. A production release will
+contain:
 
 - an immutable Registry version
 - the compatible WranglesPY version or version range
@@ -265,8 +282,12 @@ The pilot Registry version is `0.1.0-pilot`. A production release will contain:
 - a convenience `latest` alias that is not used for reproducible execution
 
 Wrangles progress through `draft`, `active`, `deprecated`, and `removed`.
-Deprecated records stay discoverable and identify their replacement. Removed
-records remain in versioned artifacts needed to understand historical recipes.
+Deprecated records stay discoverable and must identify their canonical
+replacement with `replaced_by`. The compiler rejects a missing, self-referential,
+or unknown replacement key and generates a visible migration notice in the
+human documentation. Deprecated entries sort after active entries within their
+documentation group. Removed records remain in versioned artifacts needed to
+understand historical recipes.
 
 ## Change workflow
 
@@ -277,17 +298,31 @@ the database projection are invalid.
 The current wiki and database extraction scripts are migration inputs. Once
 the cutover is complete, they must not remain ongoing authoring paths.
 
-## Pilot exit criteria
-
-The pilot is complete when:
-
-1. All callable recipe wrangles compile into deterministic Markdown, manifest,
-   and recipe-schema artifacts; curated entries are preserved during bootstrap.
-2. Invalid frontmatter, duplicate keys, invalid recipes, missing fixtures, and
-   stale generated outputs fail CI.
-3. The pages build successfully in Docusaurus at stable per-wrangle routes.
-4. A pinned WranglesPY contract manifest is reconciled with all existing
-   per-wrangle quasi-registry Markdown and the normalized pilot entries. The
-   report distinguishes records awaiting normalization from runtime wrangles
-   that genuinely lack Markdown, and preserves embedded-schema differences.
-5. An offline example runner is designed for the next implementation slice.
+## Production readiness
+
+The contract remains `pre-production`. The detailed migration sequence is in
+[README.md](README.md#migration-plan), and production requires all of the
+following outcomes:
+
+1. Every callable recipe wrangle and public parameter reconciles with a pinned
+   WranglesPY runtime manifest, with no unexplained key, required-state,
+   default, common-control, or accepted-value conflicts.
+2. Registry records contain reviewed descriptions, constraints, examples,
+   access and lifecycle metadata, provenance, and canonical database UUIDs.
+3. The entry schema, compiled contract, recipe-schema URLs, compatibility
+   policy, and change-management rules are stable and versioned.
+4. Recipe JSON Schema and any temporary WranglesPY `_schema` compatibility view
+   are generated from the Registry and protected by deterministic parity checks.
+5. WranglesXL, VS Code, APIs, the docs site, and Recipe Writer clients consume
+   the Registry-generated contracts instead of hand-maintained schema copies.
+6. Hand-maintained `_schema` docstrings have been removed, or only a generated
+   compatibility view remains for explicitly supported legacy readers.
+7. Local examples have offline execution verification, and service-backed
+   examples have controlled live verification.
+8. Invalid source, duplicate keys, invalid recipes, missing fixtures, runtime
+   drift, and stale generated outputs fail CI.
+9. Registry pages build at stable routes, and immutable public artifacts have
+   been deployed and verified at canonical URLs.
+10. Ownership, release, rollback, and deprecation procedures are documented;
+    the first production Registry version is tagged before this status changes
+    to `production`.

registry/README.md

@@ -4,16 +4,35 @@ This directory is the authoring source for the Wrangles knowledge registry.
 It contains curated, reviewable Markdown records and the small supporting
 schemas and fixtures needed to compile those records into product artifacts.
 
-Start with [CONTRACT.md](CONTRACT.md). The first-pass pilot covers all callable
-recipe wrangles reported by the pinned WranglesPY runtime manifest. Three
-entries (`convert.case`, `convert.data_type`, and `convert.from_json`) remain
-hand-curated; the migration bootstrap creates the rest without overwriting
-those records.
+Start with [CONTRACT.md](CONTRACT.md). The Registry currently covers all
+callable recipe wrangles reported by the pinned WranglesPY runtime manifest.
+Every existing Registry record is authoritative editorial content;
+`bootstrap:registry` creates missing records but never overwrites an existing
+one.
 
 The generated outputs are intentionally committed so changes to public docs,
 the agent-facing bundle, and recipe hinting can be reviewed in the same pull
 request as their source records.
 
+## Source and generated directories
+
+The similarly named Registry directories have different responsibilities:
+
+| Directory | Responsibility | Edit directly? |
+| --- | --- | --- |
+| `registry/wrangles/` | Authoritative Markdown records for wrangle metadata, parameters, guidance, examples, and lifecycle | Yes |
+| `wrangles-docs/registry-docs/` | Docusaurus pages generated from the authoritative records | No |
+| `wrangles-docs/static/registry/` | Public machine-readable Markdown, JSON contracts, schemas, fixtures, and discovery metadata | No |
+
+The normal flow is therefore:
+
+```text
+registry/wrangles/
+        -> npm run compile:registry
+wrangles-docs/registry-docs/     (human-facing pages)
+wrangles-docs/static/registry/   (machine-facing artifacts)
+```
+
 `runtime/wranglespy.json` is the pinned mechanical contract exported from the
 WranglesPY recipe namespace. Its producer-owned schema is copied alongside the
 Registry schema under `schema/`. The compiler reconciles the manifest with the
@@ -35,17 +54,103 @@ npm run check:registry
 npm run build
 ```
 
-`bootstrap:registry` creates only missing Registry records. Use
-`refresh:registry-bootstrap` to regenerate the non-curated first-pass records
-after updating the pinned runtime manifest or migration logic.
+`bootstrap:registry` creates only missing Registry records. It deliberately
+does not regenerate existing records because `registry/wrangles/` is the
+authoritative editorial source.
 
-`compile:registry` produces the pilot Docusaurus pages and public raw
+`compile:registry` produces the Docusaurus pages and public raw
 artifacts. `check:registry` fails when the source is invalid or the committed
 generated files are stale.
 
+## Migration Plan
+
+The documentation-source migration and the recipe-schema migration are
+separate activities. Neither is the same as deprecating or removing a callable
+wrangle from the product.
+
+Progress is tracked in
+[#27: Complete Docs Registry migration and production cutover](https://github.com/wrangleworks/Wrangles-Docs/issues/27).
+
+### Retiring Baver's local source files
+
+The committed files under `wrangles-docs/wrangle-docs/**/_sources/` came from
+the previous documentation workflow. They are currently retained as migration
+input and reconciliation evidence; they are not the ongoing authoring source.
+
+Complete this migration in the following order:
+
+1. Inventory every `_sources` file and confirm that useful descriptions,
+   examples, access requirements, and provenance are represented in
+   `registry/wrangles/` or intentionally omitted.
+2. Review the Registry-generated pages and establish the final routes and any
+   required redirects from the previous documentation pages.
+3. Remove `_sources` from Registry reconciliation and retire the old
+   `_sources`-based generation and synchronization workflow, including
+   `.github/agents/sync.md`.
+4. Preserve a tagged or archived rollback snapshot, deploy the Registry pages,
+   and then remove the obsolete `_sources` files.
+
+This activity is complete when no supported build, compiler, or authoring
+workflow reads from or writes to `_sources`.
+
+### Migrating the schema from WranglesPY to the Docs Registry
+
+The embedded `_schema` JSON Schema docstrings in WranglesPY remain migration
+input today. Runtime behavior and callable signatures continue to be owned by
+WranglesPY; descriptions, parameter guidance, accepted-value constraints, and
+examples move to `registry/wrangles/`.
+
+The Registry contract remains **pre-production** until these steps are
+complete:
+
+1. Reconcile every callable wrangle and public parameter against a pinned
+   WranglesPY runtime manifest, resolving every unexplained key, required-state,
+   default, common-control, and accepted-value conflict.
+2. Complete the Registry descriptions, constraints, examples, access metadata,
+   lifecycle metadata, and provenance; assign canonical database UUIDs where
+   they are still missing.
+3. Stabilize and version the Registry entry schema, compiled contract, and
+   recipe-schema URLs, including compatibility and change-management rules.
+4. Generate the recipe JSON Schema and, where still needed, a compatible
+   WranglesPY `_schema` view from the Registry; add deterministic parity checks
+   so manually maintained copies cannot drift.
+5. Cut over supported consumers—including WranglesXL, VS Code, APIs, the docs
+   site, and Recipe Writer clients—to the Registry-generated schema and
+   contracts.
+6. Remove the hand-maintained WranglesPY `_schema` content, or retain only a
+   generated compatibility view until every supported direct reader has
+   migrated.
+7. Add offline execution verification for local examples and controlled live
+   verification for service-backed examples.
+8. Publish and verify immutable Registry artifacts at their canonical public
+   URLs, document ownership and release procedures, and tag the first
+   production Registry release.
+
+After all eight steps are complete, change the contract status from
+`pre-production` to `production`. Until then, generated artifacts are suitable
+for integration and review but are not the final production authority for all
+consumers.
+
+## Deprecating or removing a callable wrangle
+
+This is the lifecycle process for an executable recipe key. It is unrelated to
+retiring `_sources` or migrating the WranglesPY `_schema` content.
+
+1. Add or verify the canonical replacement in WranglesPY, retain the old key
+   for the supported compatibility period, and test that both paths behave as
+   intended.
+2. Export the refreshed WranglesPY runtime manifest, then set the old Registry
+   record to `status: deprecated` and add `replaced_by: <canonical wrangle_key>`.
+3. Run `compile:registry`, `check:registry`, and `build`. The compiler validates
+   the replacement and automatically labels, links, and sorts the deprecated
+   entry in both human and machine artifacts.
+
+Do not model a deprecated callable as an alias of its replacement: keeping its
+own Registry record preserves compatibility and historical recipe discovery.
+
 Do not edit these generated directories by hand:
 
 - `wrangles-docs/registry-docs/`
 - `wrangles-docs/static/registry/`
-- `wrangles-docs/static/schemas/recipes/pilot/`
+- `wrangles-docs/static/schemas/recipes/registry/`
 - `registry/reports/`

registry/common/wrangle-controls.yaml

@@ -3,17 +3,20 @@ controls:
   where:
     description: Filter rows before applying the wrangle using SQL-like criteria, such as `column1 = 123 OR column2 = 'abc'`.
     required: false
+    param_group: Conditions
     schema:
       type: string
   where_params:
     description: Values used with `where` for parameterized criteria. Uses SQLite placeholder syntax such as `?` or `:name`.
     required: false
+    param_group: Conditions
     schema:
       type:
         - array
         - object
   if:
     description: Condition that determines whether the wrangle runs as a whole. Recipe variables may be referenced with `${variable}`.
     required: false
+    param_group: Conditions
     schema:
       type: string

registry/index.md

@@ -3,7 +3,7 @@ okf_version: "0.2"
 type: collection
 title: Wrangles Registry
 description: Versioned knowledge and machine contracts for Wrangles recipe primitives.
-status: pilot
+status: pre-production
 tags:
   - wrangles
   - recipes
@@ -15,6 +15,7 @@ tags:
 The Registry describes the recipe vocabulary shared by WranglesPY,
 WranglesXL, the documentation site, and future Recipe Writer clients.
 
-The pilot contains three Convert wrangles. Generated indexes and manifests
-should be used for discovery; individual files provide the detailed contract,
-guidance, examples, provenance, and lifecycle state.
+The Registry contains the callable recipe wrangles reported by the pinned
+WranglesPY runtime manifest. Generated indexes and manifests should be used for
+discovery; individual files provide the detailed contract, guidance, examples,
+provenance, and lifecycle state.

registry/reports/runtime-reconciliation.json

@@ -1,7 +1,7 @@
 {
   "format": "wrangles-registry-reconciliation",
   "format_version": "0.2",
-  "registry_version": "0.1.0-pilot",
+  "registry_version": "0.1.0",
   "runtime_source": {
     "repository": "https://github.com/wrangleworks/WranglesPY",
     "revision": "7916bf158e8b7e561270a1bea7b808f88956edc4"

registry/reports/runtime-reconciliation.md

@@ -3,7 +3,7 @@
 Generated file. Do not edit directly.
 
 - Runtime source: [`7916bf158e8b7e561270a1bea7b808f88956edc4`](https://github.com/wrangleworks/WranglesPY/commit/7916bf158e8b7e561270a1bea7b808f88956edc4)
-- Registry version: `0.1.0-pilot`
+- Registry version: `0.1.0`
 - Runtime entries: 98
 - Embedded Python schema docstrings: 96
 - Existing quasi-registry records: 93