|
| 1 | +--- |
| 2 | +tags: [backend, otf, toolchain, workflows, naming, observability] |
| 3 | +--- |
| 4 | + |
| 5 | +# Toolchain Naming and Pipeline Simplification |
| 6 | + |
| 7 | +- **Status**: valid |
| 8 | +- **Authors**: Enrique González Paredes (@egparedes) |
| 9 | +- **Created**: 2026-07-30 |
| 10 | +- **Updated**: 2026-07-30 |
| 11 | + |
| 12 | +In the context of the on-the-fly compilation toolchain, facing a root object |
| 13 | +whose names (`Backend`, `transforms`, `executor`) no longer match the |
| 14 | +established *toolchain* vocabulary and a workflow-combinator framework whose |
| 15 | +reading cost exceeds its value, we decided to rename the root object to |
| 16 | +`Toolchain` (with `frontend` / `backend` halves), rename the envelope and |
| 17 | +stage types accordingly, replace the combinator tower with explicit, fully |
| 18 | +typed pipelines, and add a sanctioned stage-observability seam. We considered |
| 19 | +keeping the combinators and trimming only the dead ones, and accept that the |
| 20 | +renames break the old names outright, a one-time rotation of persistent |
| 21 | +translation-cache keys, and a migration burden on downstream code that |
| 22 | +subclassed the combinators. |
| 23 | + |
| 24 | +This partially supersedes [0011 - On The Fly Compilation](0011-On_The_Fly_Compilation.md) |
| 25 | +(the workflow-combinator framework and the `otf.step_types` naming — its stage |
| 26 | +vocabulary and build-system decisions remain valid) and refines |
| 27 | +[0017 - Toolchain Configuration](0017-Toolchain-Configuration.md) (whose term |
| 28 | +*toolchain* becomes the name of the root object; its configuration decisions |
| 29 | +remain valid). |
| 30 | + |
| 31 | +## Context |
| 32 | + |
| 33 | +ADR 0017 already defines **toolchain** as "all the code components that work |
| 34 | +together to go from DSL code to an optimized, runnable python callable" — |
| 35 | +JIT/OTF pipelines, transformation passes, lowerings, parsers. The code grew |
| 36 | +into exactly that shape while keeping older names: the root object is called |
| 37 | +`Backend`, its frontend half `transforms`, its backend half `executor`, and |
| 38 | +the pipeline pieces are spread over five vaguely-named modules (`workflow`, |
| 39 | +`toolchain`, `definitions`, `stages`, `recipes`). |
| 40 | + |
| 41 | +Separately, the workflow-combinator framework introduced by ADR 0011 had grown |
| 42 | +to thirteen abstractions (`Workflow`, two mixins, `NamedStepSequence`, |
| 43 | +`MultiWorkflow`, `StepSequence`, `CachedStep`, `SkippableStep`, `make_step`, |
| 44 | +the `ConcreteArtifact` envelope and three adapters) to express function |
| 45 | +composition (some of them already deleted as dead code in the preparatory |
| 46 | +cleanup PRs). Measured against actual use, only `CachedStep` is a deep module; |
| 47 | +the others are shallow wrappers around `Callable[[S], T]`, and the |
| 48 | +`NamedStepSequence.__call__` reflection loop is `Any`-typed, defeating the |
| 49 | +static typing ADR 0011 prized. |
| 50 | + |
| 51 | +Finally, the seam between the toolchain and its consumers has no sanctioned |
| 52 | +interface for observing or running intermediate stages: the one consumer that |
| 53 | +needs an intermediate stage (the dace `__sdfg__` path) duck-types into the |
| 54 | +default pipeline and mutates its stages. |
| 55 | + |
| 56 | +## Decision |
| 57 | + |
| 58 | +### Naming |
| 59 | + |
| 60 | +| Old | New | |
| 61 | +| ---------------------------- | --------------------------------- | |
| 62 | +| `Backend` | `Toolchain` | |
| 63 | +| `Backend.transforms` | `Toolchain.frontend` | |
| 64 | +| `Backend.executor` | `Toolchain.backend` | |
| 65 | +| `OTFCompileWorkflow` | `CompilePipeline` | |
| 66 | +| `ConcreteArtifact` / `.data` | `ProgramWithArgs` / `.definition` | |
| 67 | +| `CompilableProgramDef` | `CompilableProgram` | |
| 68 | + |
| 69 | +These are hard renames: no deprecation aliases are kept, so the old names stop |
| 70 | +resolving in the same release. The public `gtx.*` names (`gtx.gtfn_cpu`, |
| 71 | +`gtx.wait_for_compilation`, ...) are unchanged, with one exception — |
| 72 | +`gtx.typing.Backend` becomes `gtx.typing.Toolchain`. |
| 73 | + |
| 74 | +We considered shipping one-release read aliases for the renamed attributes and |
| 75 | +re-export shims for the moved modules. We rejected that: an alias that silently |
| 76 | +keeps working is exactly what lets a downstream stay on the old vocabulary past |
| 77 | +the release where it was supposed to migrate, and the aliases would have been |
| 78 | +partial anyway (constructing with the old keyword names could not be aliased |
| 79 | +without hand-writing `__init__`, so `Backend(executor=...)` would have raised |
| 80 | +`TypeError` regardless). A single loud break at a known release is easier to |
| 81 | +act on than a half-working compatibility layer. `CompilableProgram` stays a type |
| 82 | +alias (of the parameterized envelope) for now. |
| 83 | + |
| 84 | +### Pipeline, not combinators |
| 85 | + |
| 86 | +Named pipelines become frozen dataclasses with an explicit, fully typed |
| 87 | +`__call__`; steps are plain callables (`Step[S, T] = Callable[[S], T]`); |
| 88 | +customization stays composition-time via `dataclasses.replace`. The |
| 89 | +combinator framework (both mixins, `NamedStepSequence`, `MultiWorkflow`, |
| 90 | +`StepSequence`, `make_step`, `.chain`, the three adapters) is deleted; |
| 91 | +`CachedStep` is kept unchanged. |
| 92 | + |
| 93 | +What ADR 0011's decisions become: |
| 94 | + |
| 95 | +| ADR 0011 requirement | Kept by | |
| 96 | +| ------------------------------------------- | ----------------------------------------------------------------------- | |
| 97 | +| Named steps, order visible | dataclass fields + explicit `__call__` (order literally visible) | |
| 98 | +| Statically typed composition | explicit `__call__` — checked end-to-end, unlike the reflection loop | |
| 99 | +| Customization at composition, not via flags | `dataclasses.replace` on frozen pipelines | |
| 100 | +| Steps compose across backends | `Step[S, T]` is a `Callable` — every existing step already satisfies it | |
| 101 | +| Linear workflows | unchanged (`Transforms` keeps its input-dependent step *selection*) | |
| 102 | + |
| 103 | +### Stage observability |
| 104 | + |
| 105 | +A `stage_hook(name, artifact)` event hook (on the existing |
| 106 | +`instrumentation.hook_machinery`) is emitted by `CompilePipeline` and |
| 107 | +`Transforms` after each step; artifacts are treated opaquely. |
| 108 | +`GT4PY_DUMP_STAGES=<dir>` registers a subscriber that writes each stage |
| 109 | +artifact per program. `Toolchain.translate(definition, compile_time_args)` is |
| 110 | +the sanctioned partial run (frontend + translation only); it narrows to the |
| 111 | +standard `CompilePipeline` shape and raises a clear error on monolithic |
| 112 | +backends. Per-call step options are deliberately not offered — a caller |
| 113 | +needing a variant step builds a variant pipeline with `dataclasses.replace`. |
| 114 | + |
| 115 | +### Phasing |
| 116 | + |
| 117 | +The decisions in this ADR land as a stack of PRs. The naming decisions above |
| 118 | +land first, except the `OTFCompileWorkflow` → `CompilePipeline` rename, which |
| 119 | +lands with the pipeline PR; the "Pipeline, not combinators" and "Stage |
| 120 | +observability" decisions are implemented in follow-up PRs of the same stack. |
| 121 | + |
| 122 | +## Consequences |
| 123 | + |
| 124 | +- The dace `__sdfg__` reach-in (duck-typing through `executor.translation` |
| 125 | + and mutating frozen stages) will be replaced by a dace-owned translate-only |
| 126 | + step built with `dataclasses.replace`. |
| 127 | +- Downstream code subclassing the deleted combinators or overriding |
| 128 | + `Transforms.step_order` must migrate to explicit `__call__` composition. |
| 129 | +- Downstream code must migrate to the new names in the same release: the old |
| 130 | + ones are removed, not aliased. That includes the modules dissolved along the |
| 131 | + way — `otf.definitions`, `otf.code_specs`, `otf.recipes` and `otf.toolchain` |
| 132 | + cease to exist, and `gtx.typing.Backend` becomes `gtx.typing.Toolchain`. |
| 133 | +- ADR 0011's `otf.step_types` naming section no longer describes the code. |
| 134 | +- Cache keys rotate once: fingerprints embed the qualified class and field |
| 135 | + names, so the `ProgramWithArgs` / `definition` rename changes the keys of |
| 136 | + the persistent translation caches (gtfn and dace). This is subsumed by |
| 137 | + `BUILD_CACHE_VERSION_ID`, which already rotates them at every release, so |
| 138 | + there is no behavior change beyond a one-time cache miss. |
| 139 | + |
| 140 | +## Alternatives considered |
| 141 | + |
| 142 | +- **Keep the combinators but trim dead ones**: rejected — the tower is a |
| 143 | + shallow-module archetype and every recorded ADR 0011 requirement survives |
| 144 | + its removal; trimming keeps the reading cost and the `Any`-typed |
| 145 | + composition. |
| 146 | +- **A per-call options API on `translate()`**: rejected — reconfiguration |
| 147 | + stays composition-time, ADR 0011's own rule. |
0 commit comments