refactor[next]: add stage observability and Toolchain.translate, fix dace __sdfg__ reach-in - #2742
Open
egparedes wants to merge 1 commit into
Open
refactor[next]: add stage observability and Toolchain.translate, fix dace __sdfg__ reach-in#2742egparedes wants to merge 1 commit into
egparedes wants to merge 1 commit into
Conversation
egparedes
marked this pull request as ready for review
July 30, 2026 17:58
egparedes
force-pushed
the
otf-split-3-observability
branch
from
July 30, 2026 17:58
e3d68d2 to
1c59da0
Compare
egparedes
force-pushed
the
otf-split-3-observability
branch
from
July 31, 2026 16:15
1c59da0 to
292deb1
Compare
egparedes
force-pushed
the
otf-split-3-observability
branch
from
August 17, 2026 15:07
292deb1 to
b79be9b
Compare
egparedes
force-pushed
the
otf-split-3-observability
branch
from
August 20, 2026 16:17
b79be9b to
573e320
Compare
egparedes
force-pushed
the
otf-split-3-observability
branch
from
August 20, 2026 16:52
573e320 to
5e0b9c1
Compare
egparedes
force-pushed
the
otf-split-3-observability
branch
from
August 20, 2026 20:28
5e0b9c1 to
05ddb45
Compare
egparedes
force-pushed
the
otf-split-3-observability
branch
from
August 26, 2026 11:05
05ddb45 to
658a51e
Compare
egparedes
force-pushed
the
otf-split-3-observability
branch
from
August 28, 2026 13:06
658a51e to
4b16df7
Compare
egparedes
force-pushed
the
otf-split-3-observability
branch
from
August 28, 2026 13:09
4b16df7 to
aa3bb68
Compare
egparedes
force-pushed
the
otf-split-3-observability
branch
from
September 1, 2026 18:30
aa3bb68 to
e53ca53
Compare
egparedes
force-pushed
the
otf-split-3-observability
branch
from
September 9, 2026 16:34
e53ca53 to
3799397
Compare
egparedes
force-pushed
the
otf-split-3-observability
branch
from
September 9, 2026 16:51
3799397 to
c95d6e7
Compare
…dace __sdfg__ reach-in Give the toolchain a sanctioned way to observe and partially run its pipelines, and remove the one consumer that had to reach into them: - `otf.workflow.stage_hook(name, artifact)`: an event hook emitted after each step by the two named pipelines (`backend.Transforms` and `recipes.OTFCompileWorkflow`, via the generic `MultiWorkflow` / `NamedStepSequence` loops) and by `Toolchain.translate`. Artifacts are passed opaquely and never formatted at the emit site; with nothing subscribed the hook body is empty, so emission is an empty callback loop (~160 ns). - `GT4PY_DUMP_STAGES=<dir>` (`config.DUMP_STAGES`) plus the `instrumentation.stage_dump` subscriber, which writes every stage artifact to `<dir>/<program>/<NNN>_<step>.<ext>`. Registered at import time when the variable is set, so a plain program run dumps its stages without any user code; `enable()` / `disable()` cover programmatic use. Exclusive-create with an index bump keeps concurrent compilation workers and same-named programs from overwriting each other. - `Toolchain.translate(definition, compile_time_args)`: the sanctioned partial run (frontend + translation only). It narrows to the standard `OTFCompileWorkflow` shape and raises a clear error on a monolithic backend. Per-call step options are deliberately not offered; a caller needing a variant step builds a variant pipeline with `dataclasses.replace`. - The dace `__sdfg__` reach-in is gone. It duck-typed through `backend.backend.translation`, unwrapped the `CachedStep` by hand, and used `object.__setattr__` to write into the *in-memory-cached* `past_to_itir` stage and its frozen `CompileTimeArgs`. It is replaced by a dace-owned translate-only toolchain variant built with `dataclasses.replace`, plus `Toolchain.translate`. That last change also fixes a real bug: because the mutated stage was the cached one, a second `__sdfg__` call re-transformed an already-transformed program, and any other consumer of that cache entry saw args whose `offset_provider` had been replaced by offset-provider *types*, losing the runtime connectivity tables. A regression test covers it. Removing the reach-in drops the last non-doc consumer of the workflow mixins' `.replace`, which unblocks the pipeline simplification. `GT4PY_DUMP_STAGES` is new opt-in functionality; nothing else changes behavior, and no fingerprinted dataclass gains a field, so no cache keys rotate. Claude-Session: https://claude.ai/code/session_01R8zRtFMhdJ8c96XJYCXkRk
egparedes
force-pushed
the
otf-split-3-observability
branch
from
September 10, 2026 11:54
c95d6e7 to
43b76e2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Give the toolchain a sanctioned way to observe and partially run its
pipelines, and remove the one consumer that had to reach into them:
otf.workflow.stage_hook(name, artifact): an event hook emitted aftereach step by the two named pipelines (
backend.Transformsandrecipes.OTFCompileWorkflow, via the genericMultiWorkflow/NamedStepSequenceloops) and byToolchain.translate. Artifacts arepassed opaquely and never formatted at the emit site; with nothing
subscribed the hook body is empty, so emission is an empty callback loop
(~160 ns).
GT4PY_DUMP_STAGES=<dir>(config.DUMP_STAGES) plus theinstrumentation.stage_dumpsubscriber, which writes every stageartifact to
<dir>/<program>/<NNN>_<step>.<ext>. Registered at importtime when the variable is set, so a plain program run dumps its stages
without any user code;
enable()/disable()cover programmatic use.Exclusive-create with an index bump keeps concurrent compilation workers
and same-named programs from overwriting each other.
Toolchain.translate(definition, compile_time_args): the sanctionedpartial run (frontend + translation only). It narrows to the standard
OTFCompileWorkflowshape and raises a clear error on a monolithicbackend. Per-call step options are deliberately not offered; a caller
needing a variant step builds a variant pipeline with
dataclasses.replace.__sdfg__reach-in is gone. It duck-typed throughbackend.backend.translation, unwrapped theCachedStepby hand, andused
object.__setattr__to write into the in-memory-cachedpast_to_itirstage and its frozenCompileTimeArgs. It is replaced bya dace-owned translate-only toolchain variant built with
dataclasses.replace, plusToolchain.translate.That last change also fixes a real bug: because the mutated stage was the
cached one, a second
__sdfg__call re-transformed an already-transformedprogram, and any other consumer of that cache entry saw args whose
offset_providerhad been replaced by offset-provider types, losing theruntime connectivity tables. A regression test covers it.
Removing the reach-in drops the last non-doc consumer of the workflow
mixins'
.replace, which unblocks the pipeline simplification.GT4PY_DUMP_STAGESis new opt-in functionality; nothing else changesbehavior, and no fingerprinted dataclass gains a field, so no cache keys
rotate.