Skip to content

Commit 5dfb78f

Browse files
committed
refactor[next]: rename Backend to Toolchain and align field names
Adopt the vocabulary ADR 0017 already defines: the root object that carries a program from DSL source to a runnable callable is the *toolchain*, not the *backend* (which is one of its two halves). Renamed: - `next.backend.Backend` -> `next.backend.Toolchain` - `Toolchain.transforms` -> `Toolchain.frontend` - `Toolchain.executor` -> `Toolchain.backend` - `otf.workflow.ConcreteArtifact` -> `otf.workflow.ProgramWithArgs` - `ProgramWithArgs.data` -> `ProgramWithArgs.definition` - `otf.stages.CompilableProgramDef` -> `otf.stages.CompilableProgram` - `gtx.typing.Backend` -> `gtx.typing.Toolchain` These are hard renames: no deprecation aliases are kept, so downstream has to move in this release rather than drift on a compatibility layer. An alias would have been partial anyway -- constructing with the old keyword names cannot be aliased on a dataclass without hand-writing `__init__` -- and a half-working compatibility layer is harder to act on than one loud break. Also resolves the `TODO(tehrengruber)` in `backend.py` asking for exactly this rename, and records the decision in a new ADR 0027, which partially supersedes ADR 0011 and refines ADR 0017. No behavior change. Fingerprints embed qualified class and field names, so the `ProgramWithArgs` rename rotates the two persistent translation-cache keys (gtfn, dace) once; `BUILD_CACHE_VERSION_ID` already rotates them on every release, so this only means one extra cache miss on upgrade. Claude-Session: https://claude.ai/code/session_01R8zRtFMhdJ8c96XJYCXkRk
1 parent fa8a324 commit 5dfb78f

46 files changed

Lines changed: 404 additions & 241 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎docs/development/ADRs/next/0011-On_The_Fly_Compilation.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,8 @@ tags: [backend, bindings, build, compile, otf]
1111

1212
This supersedes [0009 - Compiled Backend Integration](0009-Compiled_Backend_Integration.md) and concentrates on the API design for on-the-fly compilation of GT4Py programs and all the steps in between IR and compiled Python extension.
1313

14+
Partially superseded by [0027 - Toolchain Naming and Pipeline Simplification](0027-Toolchain-Naming-and-Pipeline-Simplification.md) (workflow combinators and step-type naming).
15+
1416
## Context
1517

1618
The on-the-fly compilation (OTFC) in gt4py encompasses everything necessary to go from an IR representation of a GT4Py program to an executable Python function. Depending on the chosen route, this may include:

‎docs/development/ADRs/next/0017-Toolchain-Configuration.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,8 @@ tags: [backend, otf, workflows, toolchain]
1111

1212
In order to provide a streamlined user experience, we attempt to standardize how users of GT4Py stencils can configure how those stencils are optimized without editing GT4Py code. This describes the design of the first minimal implementation.
1313

14+
Refined by [0027 - Toolchain Naming and Pipeline Simplification](0027-Toolchain-Naming-and-Pipeline-Simplification.md) (the term *toolchain* becomes the name of the root object).
15+
1416
## Context
1517

1618
In this document the word toolchain is used to mean all the code components that work together to go from DSL code to an optimized, runnable python callable.
Lines changed: 147 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,147 @@
1+
---
2+
tags: [backend, otf, toolchain, workflows, naming, observability]
3+
---
4+
5+
# Toolchain Naming and Pipeline Simplification
6+
7+
- **Status**: valid
8+
- **Authors**: Enrique González Paredes (@egparedes)
9+
- **Created**: 2026-07-30
10+
- **Updated**: 2026-07-30
11+
12+
In the context of the on-the-fly compilation toolchain, facing a root object
13+
whose names (`Backend`, `transforms`, `executor`) no longer match the
14+
established *toolchain* vocabulary and a workflow-combinator framework whose
15+
reading cost exceeds its value, we decided to rename the root object to
16+
`Toolchain` (with `frontend` / `backend` halves), rename the envelope and
17+
stage types accordingly, replace the combinator tower with explicit, fully
18+
typed pipelines, and add a sanctioned stage-observability seam. We considered
19+
keeping the combinators and trimming only the dead ones, and accept that the
20+
renames break the old names outright, a one-time rotation of persistent
21+
translation-cache keys, and a migration burden on downstream code that
22+
subclassed the combinators.
23+
24+
This partially supersedes [0011 - On The Fly Compilation](0011-On_The_Fly_Compilation.md)
25+
(the workflow-combinator framework and the `otf.step_types` naming — its stage
26+
vocabulary and build-system decisions remain valid) and refines
27+
[0017 - Toolchain Configuration](0017-Toolchain-Configuration.md) (whose term
28+
*toolchain* becomes the name of the root object; its configuration decisions
29+
remain valid).
30+
31+
## Context
32+
33+
ADR 0017 already defines **toolchain** as "all the code components that work
34+
together to go from DSL code to an optimized, runnable python callable" —
35+
JIT/OTF pipelines, transformation passes, lowerings, parsers. The code grew
36+
into exactly that shape while keeping older names: the root object is called
37+
`Backend`, its frontend half `transforms`, its backend half `executor`, and
38+
the pipeline pieces are spread over five vaguely-named modules (`workflow`,
39+
`toolchain`, `definitions`, `stages`, `recipes`).
40+
41+
Separately, the workflow-combinator framework introduced by ADR 0011 had grown
42+
to thirteen abstractions (`Workflow`, two mixins, `NamedStepSequence`,
43+
`MultiWorkflow`, `StepSequence`, `CachedStep`, `SkippableStep`, `make_step`,
44+
the `ConcreteArtifact` envelope and three adapters) to express function
45+
composition (some of them already deleted as dead code in the preparatory
46+
cleanup PRs). Measured against actual use, only `CachedStep` is a deep module;
47+
the others are shallow wrappers around `Callable[[S], T]`, and the
48+
`NamedStepSequence.__call__` reflection loop is `Any`-typed, defeating the
49+
static typing ADR 0011 prized.
50+
51+
Finally, the seam between the toolchain and its consumers has no sanctioned
52+
interface for observing or running intermediate stages: the one consumer that
53+
needs an intermediate stage (the dace `__sdfg__` path) duck-types into the
54+
default pipeline and mutates its stages.
55+
56+
## Decision
57+
58+
### Naming
59+
60+
| Old | New |
61+
| ---------------------------- | --------------------------------- |
62+
| `Backend` | `Toolchain` |
63+
| `Backend.transforms` | `Toolchain.frontend` |
64+
| `Backend.executor` | `Toolchain.backend` |
65+
| `OTFCompileWorkflow` | `CompilePipeline` |
66+
| `ConcreteArtifact` / `.data` | `ProgramWithArgs` / `.definition` |
67+
| `CompilableProgramDef` | `CompilableProgram` |
68+
69+
These are hard renames: no deprecation aliases are kept, so the old names stop
70+
resolving in the same release. The public `gtx.*` names (`gtx.gtfn_cpu`,
71+
`gtx.wait_for_compilation`, ...) are unchanged, with one exception —
72+
`gtx.typing.Backend` becomes `gtx.typing.Toolchain`.
73+
74+
We considered shipping one-release read aliases for the renamed attributes and
75+
re-export shims for the moved modules. We rejected that: an alias that silently
76+
keeps working is exactly what lets a downstream stay on the old vocabulary past
77+
the release where it was supposed to migrate, and the aliases would have been
78+
partial anyway (constructing with the old keyword names could not be aliased
79+
without hand-writing `__init__`, so `Backend(executor=...)` would have raised
80+
`TypeError` regardless). A single loud break at a known release is easier to
81+
act on than a half-working compatibility layer. `CompilableProgram` stays a type
82+
alias (of the parameterized envelope) for now.
83+
84+
### Pipeline, not combinators
85+
86+
Named pipelines become frozen dataclasses with an explicit, fully typed
87+
`__call__`; steps are plain callables (`Step[S, T] = Callable[[S], T]`);
88+
customization stays composition-time via `dataclasses.replace`. The
89+
combinator framework (both mixins, `NamedStepSequence`, `MultiWorkflow`,
90+
`StepSequence`, `make_step`, `.chain`, the three adapters) is deleted;
91+
`CachedStep` is kept unchanged.
92+
93+
What ADR 0011's decisions become:
94+
95+
| ADR 0011 requirement | Kept by |
96+
| ------------------------------------------- | ----------------------------------------------------------------------- |
97+
| Named steps, order visible | dataclass fields + explicit `__call__` (order literally visible) |
98+
| Statically typed composition | explicit `__call__` — checked end-to-end, unlike the reflection loop |
99+
| Customization at composition, not via flags | `dataclasses.replace` on frozen pipelines |
100+
| Steps compose across backends | `Step[S, T]` is a `Callable` — every existing step already satisfies it |
101+
| Linear workflows | unchanged (`Transforms` keeps its input-dependent step *selection*) |
102+
103+
### Stage observability
104+
105+
A `stage_hook(name, artifact)` event hook (on the existing
106+
`instrumentation.hook_machinery`) is emitted by `CompilePipeline` and
107+
`Transforms` after each step; artifacts are treated opaquely.
108+
`GT4PY_DUMP_STAGES=<dir>` registers a subscriber that writes each stage
109+
artifact per program. `Toolchain.translate(definition, compile_time_args)` is
110+
the sanctioned partial run (frontend + translation only); it narrows to the
111+
standard `CompilePipeline` shape and raises a clear error on monolithic
112+
backends. Per-call step options are deliberately not offered — a caller
113+
needing a variant step builds a variant pipeline with `dataclasses.replace`.
114+
115+
### Phasing
116+
117+
The decisions in this ADR land as a stack of PRs. The naming decisions above
118+
land first, except the `OTFCompileWorkflow` → `CompilePipeline` rename, which
119+
lands with the pipeline PR; the "Pipeline, not combinators" and "Stage
120+
observability" decisions are implemented in follow-up PRs of the same stack.
121+
122+
## Consequences
123+
124+
- The dace `__sdfg__` reach-in (duck-typing through `executor.translation`
125+
and mutating frozen stages) will be replaced by a dace-owned translate-only
126+
step built with `dataclasses.replace`.
127+
- Downstream code subclassing the deleted combinators or overriding
128+
`Transforms.step_order` must migrate to explicit `__call__` composition.
129+
- Downstream code must migrate to the new names in the same release: the old
130+
ones are removed, not aliased. That includes the modules dissolved along the
131+
way — `otf.definitions`, `otf.code_specs`, `otf.recipes` and `otf.toolchain`
132+
cease to exist, and `gtx.typing.Backend` becomes `gtx.typing.Toolchain`.
133+
- ADR 0011's `otf.step_types` naming section no longer describes the code.
134+
- Cache keys rotate once: fingerprints embed the qualified class and field
135+
names, so the `ProgramWithArgs` / `definition` rename changes the keys of
136+
the persistent translation caches (gtfn and dace). This is subsumed by
137+
`BUILD_CACHE_VERSION_ID`, which already rotates them at every release, so
138+
there is no behavior change beyond a one-time cache miss.
139+
140+
## Alternatives considered
141+
142+
- **Keep the combinators but trim dead ones**: rejected — the tower is a
143+
shallow-module archetype and every recorded ADR 0011 requirement survives
144+
its removal; trimming keeps the reading cost and the `Any`-typed
145+
composition.
146+
- **A per-call options API on `translate()`**: rejected — reconfiguration
147+
stays composition-time, ADR 0011's own rule.

‎docs/development/ADRs/next/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,7 @@ Writing a new ADR is simple:
4848
- [0016 - Multiple Backends and Build Systems](0016-Multiple-Backends-and-Build-Systems.md)
4949
- [0017 - Toolchain Configuration](0017-Toolchain-Configuration.md)
5050
- [0018 - Canonical Form of an SDFG in GT4Py (Especially for Optimizations)](0018-Canonical_SDFG_in_GT4Py_Transformations.md)
51+
- [0027 - Toolchain Naming and Pipeline Simplification](0027-Toolchain-Naming-and-Pipeline-Simplification.md)
5152

5253
### Python Integration
5354

‎docs/user/next/advanced/HackTheToolchain.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -22,8 +22,8 @@ cached_lowering_toolchain = gtx.backend.DEFAULT_TRANSFORMS.replace(
2222
## Skip Steps / Change Order
2323

2424
```python
25-
DUMMY_FOP = workflow.ConcreteArtifact(
26-
data=ff_stages.DSLFieldOperatorDef(definition=None), args=None
25+
DUMMY_FOP = workflow.ProgramWithArgs(
26+
definition=ff_stages.DSLFieldOperatorDef(definition=None), args=None
2727
)
2828
```
2929

@@ -57,7 +57,7 @@ class Cpp2BindingsGen: ...
5757

5858
class PureCpp2WorkflowFactory(gtx.program_processors.runners.gtfn.GTFNCompileWorkflowFactory):
5959
translation: workflow.Workflow[
60-
gtx.otf.stages.CompilableProgramDef, gtx.otf.artifacts.ProgramSource
60+
gtx.otf.stages.CompilableProgram, gtx.otf.artifacts.ProgramSource
6161
] = MyCodeGen()
6262
bindings: workflow.Workflow[
6363
gtx.otf.artifacts.ProgramSource, gtx.otf.artifacts.ExtensionSource

‎docs/user/next/advanced/WorkflowPatterns.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -409,7 +409,7 @@ gtx.backend.DEFAULT_PROG_TRANSFORMS??
409409
```
410410

411411
```python
412-
gtx.program_processors.runners.gtfn.run_gtfn_gpu.executor.otf_workflow??
412+
gtx.program_processors.runners.gtfn.run_gtfn_gpu.backend??
413413
```
414414

415415
```python

‎src/gt4py/next/AGENTS.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,9 @@ field operators / programs (ffront)
3131
Distinct from `iterator/embedded.py`, which runs one level lower.
3232
- `otf/` — on-the-fly compilation toolchain (workflow steps, caching,
3333
argument descriptors).
34+
- `backend.py` — the toolchain root object `Toolchain` (formerly `Backend`):
35+
`frontend` (definition transforms) + `backend` (compile pipeline) +
36+
allocator.
3437
- `program_processors/runners/` — the backends: `gtfn` (GridTools C++),
3538
`dace` (DaCe SDFG), `roundtrip` / `double_roundtrip` (pure Python).
3639
- `type_system/` — `next` type specifications and the type inference the

‎src/gt4py/next/backend.py‎

Lines changed: 17 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@ def adapted_jit_to_aot_args_factory() -> workflow.Workflow[
4545
class Transforms(
4646
workflow.MultiWorkflow[
4747
stages.ConcreteProgramDef[stages.IRDefinitionT, stages.ArgsDefinitionT],
48-
stages.CompilableProgramDef,
48+
stages.CompilableProgram,
4949
]
5050
):
5151
"""
@@ -92,14 +92,14 @@ class Transforms(
9292
] = dataclasses.field(default_factory=past_process_args.transform_program_args_factory)
9393

9494
past_to_itir: workflow.Workflow[
95-
ffront_stages.ConcretePASTProgramDef, stages.CompilableProgramDef
95+
ffront_stages.ConcretePASTProgramDef, stages.CompilableProgram
9696
] = dataclasses.field(default_factory=past_to_itir.past_to_gtir_factory)
9797

9898
def step_order(self, inp: stages.ConcreteProgramDef) -> list[str]:
9999
steps: list[str] = []
100100
if isinstance(inp.args, arguments.JITArgs):
101101
steps.append("aotify_args")
102-
match inp.data:
102+
match inp.definition:
103103
case ffront_stages.DSLFieldOperatorDef():
104104
steps.extend(
105105
[
@@ -140,22 +140,27 @@ def step_order(self, inp: stages.ConcreteProgramDef) -> list[str]:
140140
DEFAULT_TRANSFORMS: Transforms = Transforms()
141141

142142

143-
# TODO(tehrengruber): Rename class and `executor` & `transforms` attribute. Maybe:
144-
# `Backend` -> `Toolchain`
145-
# `transforms` -> `frontend_transforms`
146-
# `executor` -> `backend_transforms`
147143
@dataclasses.dataclass(frozen=True)
148-
class Backend(Generic[core_defs.DeviceTypeT]):
144+
class Toolchain(Generic[core_defs.DeviceTypeT]):
145+
"""
146+
Complete pipeline from a program definition to an executable program.
147+
148+
The `frontend` workflow transforms any supported program definition into a
149+
`CompilableProgram`, which the `backend` workflow then compiles into a
150+
loadable compilation artifact. The `allocator` describes the device the
151+
compiled program expects its buffers on.
152+
"""
153+
149154
name: str
150-
executor: workflow.Workflow[stages.CompilableProgramDef, artifacts.CompilationArtifact]
155+
backend: workflow.Workflow[stages.CompilableProgram, artifacts.CompilationArtifact]
151156
allocator: next_allocators.FieldBufferAllocatorProtocol[core_defs.DeviceTypeT]
152-
transforms: workflow.Workflow[stages.ConcreteProgramDef, stages.CompilableProgramDef]
157+
frontend: workflow.Workflow[stages.ConcreteProgramDef, stages.CompilableProgram]
153158

154159
def compile(
155160
self, program: stages.IRDefinitionT, compile_time_args: arguments.CompileTimeArgs
156161
) -> artifacts.ExecutableProgram:
157-
artifact = self.executor(
158-
self.transforms(stages.ConcreteProgramDef(data=program, args=compile_time_args))
162+
artifact = self.backend(
163+
self.frontend(stages.ConcreteProgramDef(definition=program, args=compile_time_args))
159164
)
160165
return artifact.load()
161166

‎src/gt4py/next/config.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -48,10 +48,10 @@ class BuildJobsMode(enum.Enum):
4848
#: Run compilation in a ``ThreadPoolExecutor``.
4949
THREAD = "thread"
5050
#: Run compilation in a ``ProcessPoolExecutor`` with the ``spawn`` start
51-
#: method. Requires the backend's ``executor`` to be stdlib-picklable
51+
#: method. Requires the toolchain's ``backend`` pipeline to be stdlib-picklable
5252
#: (standard runners and factory-constructed variants are) and to return a
5353
#: picklable ``CompilationArtifact``; backends that don't qualify (or that
54-
#: customize ``Backend.compile``) are compiled in the calling thread
54+
#: customize ``Toolchain.compile``) are compiled in the calling thread
5555
#: instead, with a warning.
5656
PROCESS = "process"
5757

0 commit comments

Comments
 (0)