Skip to content

refactor[next]: add stage observability and Toolchain.translate, fix dace __sdfg__ reach-in - #2742

Open
egparedes wants to merge 1 commit into
otf-split-2-toolchain-namingfrom
otf-split-3-observability
Open

refactor[next]: add stage observability and Toolchain.translate, fix dace __sdfg__ reach-in#2742
egparedes wants to merge 1 commit into
otf-split-2-toolchain-namingfrom
otf-split-3-observability

Conversation

@egparedes

@egparedes egparedes commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Give the toolchain a sanctioned way to observe and partially run its
pipelines, and remove the one consumer that had to reach into them:

  • otf.workflow.stage_hook(name, artifact): an event hook emitted after
    each step by the two named pipelines (backend.Transforms and
    recipes.OTFCompileWorkflow, via the generic MultiWorkflow /
    NamedStepSequence loops) and by Toolchain.translate. Artifacts are
    passed opaquely and never formatted at the emit site; with nothing
    subscribed the hook body is empty, so emission is an empty callback loop
    (~160 ns).
  • GT4PY_DUMP_STAGES=<dir> (config.DUMP_STAGES) plus the
    instrumentation.stage_dump subscriber, which writes every stage
    artifact to <dir>/<program>/<NNN>_<step>.<ext>. Registered at import
    time when the variable is set, so a plain program run dumps its stages
    without any user code; enable() / disable() cover programmatic use.
    Exclusive-create with an index bump keeps concurrent compilation workers
    and same-named programs from overwriting each other.
  • Toolchain.translate(definition, compile_time_args): the sanctioned
    partial run (frontend + translation only). It narrows to the standard
    OTFCompileWorkflow shape and raises a clear error on a monolithic
    backend. Per-call step options are deliberately not offered; a caller
    needing a variant step builds a variant pipeline with
    dataclasses.replace.
  • The dace __sdfg__ reach-in is gone. It duck-typed through
    backend.backend.translation, unwrapped the CachedStep by hand, and
    used object.__setattr__ to write into the in-memory-cached
    past_to_itir stage and its frozen CompileTimeArgs. It is replaced by
    a dace-owned translate-only toolchain variant built with
    dataclasses.replace, plus Toolchain.translate.

That last change also fixes a real bug: because the mutated stage was the
cached one, a second __sdfg__ call re-transformed an already-transformed
program, and any other consumer of that cache entry saw args whose
offset_provider had been replaced by offset-provider types, losing the
runtime connectivity tables. A regression test covers it.

Removing the reach-in drops the last non-doc consumer of the workflow
mixins' .replace, which unblocks the pipeline simplification.

GT4PY_DUMP_STAGES is new opt-in functionality; nothing else changes
behavior, and no fingerprinted dataclass gains a field, so no cache keys
rotate.

@egparedes
egparedes marked this pull request as ready for review July 30, 2026 17:58
@egparedes
egparedes force-pushed the otf-split-3-observability branch from e3d68d2 to 1c59da0 Compare July 30, 2026 17:58
@egparedes
egparedes force-pushed the otf-split-3-observability branch from 1c59da0 to 292deb1 Compare July 31, 2026 16:15
@egparedes
egparedes force-pushed the otf-split-3-observability branch from 292deb1 to b79be9b Compare August 17, 2026 15:07
@egparedes
egparedes force-pushed the otf-split-3-observability branch from b79be9b to 573e320 Compare August 20, 2026 16:17
@egparedes
egparedes force-pushed the otf-split-3-observability branch from 573e320 to 5e0b9c1 Compare August 20, 2026 16:52
@egparedes
egparedes force-pushed the otf-split-3-observability branch from 5e0b9c1 to 05ddb45 Compare August 20, 2026 20:28
@egparedes
egparedes force-pushed the otf-split-3-observability branch from 05ddb45 to 658a51e Compare August 26, 2026 11:05
@egparedes
egparedes force-pushed the otf-split-3-observability branch from 658a51e to 4b16df7 Compare August 28, 2026 13:06
@egparedes
egparedes force-pushed the otf-split-3-observability branch from 4b16df7 to aa3bb68 Compare August 28, 2026 13:09
@egparedes egparedes changed the title refactor[next]: add stage observability and Toolchain.translate refactor[next]: add stage observability and Toolchain.translate, fix dace __sdfg__ reach-in Aug 28, 2026
@egparedes
egparedes force-pushed the otf-split-3-observability branch from aa3bb68 to e53ca53 Compare September 1, 2026 18:30
@egparedes
egparedes force-pushed the otf-split-3-observability branch from e53ca53 to 3799397 Compare September 9, 2026 16:34
@egparedes
egparedes force-pushed the otf-split-3-observability branch from 3799397 to c95d6e7 Compare September 9, 2026 16:51
…dace __sdfg__ reach-in

Give the toolchain a sanctioned way to observe and partially run its
pipelines, and remove the one consumer that had to reach into them:

- `otf.workflow.stage_hook(name, artifact)`: an event hook emitted after
  each step by the two named pipelines (`backend.Transforms` and
  `recipes.OTFCompileWorkflow`, via the generic `MultiWorkflow` /
  `NamedStepSequence` loops) and by `Toolchain.translate`. Artifacts are
  passed opaquely and never formatted at the emit site; with nothing
  subscribed the hook body is empty, so emission is an empty callback loop
  (~160 ns).
- `GT4PY_DUMP_STAGES=<dir>` (`config.DUMP_STAGES`) plus the
  `instrumentation.stage_dump` subscriber, which writes every stage
  artifact to `<dir>/<program>/<NNN>_<step>.<ext>`. Registered at import
  time when the variable is set, so a plain program run dumps its stages
  without any user code; `enable()` / `disable()` cover programmatic use.
  Exclusive-create with an index bump keeps concurrent compilation workers
  and same-named programs from overwriting each other.
- `Toolchain.translate(definition, compile_time_args)`: the sanctioned
  partial run (frontend + translation only). It narrows to the standard
  `OTFCompileWorkflow` shape and raises a clear error on a monolithic
  backend. Per-call step options are deliberately not offered; a caller
  needing a variant step builds a variant pipeline with
  `dataclasses.replace`.
- The dace `__sdfg__` reach-in is gone. It duck-typed through
  `backend.backend.translation`, unwrapped the `CachedStep` by hand, and
  used `object.__setattr__` to write into the *in-memory-cached*
  `past_to_itir` stage and its frozen `CompileTimeArgs`. It is replaced by
  a dace-owned translate-only toolchain variant built with
  `dataclasses.replace`, plus `Toolchain.translate`.

That last change also fixes a real bug: because the mutated stage was the
cached one, a second `__sdfg__` call re-transformed an already-transformed
program, and any other consumer of that cache entry saw args whose
`offset_provider` had been replaced by offset-provider *types*, losing the
runtime connectivity tables. A regression test covers it.

Removing the reach-in drops the last non-doc consumer of the workflow
mixins' `.replace`, which unblocks the pipeline simplification.

`GT4PY_DUMP_STAGES` is new opt-in functionality; nothing else changes
behavior, and no fingerprinted dataclass gains a field, so no cache keys
rotate.

Claude-Session: https://claude.ai/code/session_01R8zRtFMhdJ8c96XJYCXkRk
@egparedes
egparedes force-pushed the otf-split-3-observability branch from c95d6e7 to 43b76e2 Compare September 10, 2026 11:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant