Add mutable runtime recipe variables with step-time resolution and explicit nested export

[Copilot]

Linked issue

Implements general-purpose mutable runtime variables in a separate PR from #1179; this branch does not include or depend on attachment/feature-branch changes from that draft.

What changes

Adds an opt-in runtime variable system so recipes can set/mutate values during execution, bind action return values, and consume current values in later steps (including on_start -> read/wrangles/write) without changing existing static ${...} behavior.

• Runtime variable model

  • New execution-time references: ${runtime.name} and ${runtime.name.path}
  • Runtime resolution happens per-step (not eagerly at load)
  • Escaped literal support for runtime templates (\${runtime...})
  • Per-run mutable state cloning to avoid mutating caller-owned inputs

• Mutation + result binding

  • New variables run action:
    • set (atomic assignment)
    • update (deep merge for dict values)
    • mark_secret
    • inspect / max_items
  • New framework run parameter: result_variable to capture connector/action return values into runtime state (excluded from connector args)

• Isolation and export semantics

  • Runtime state is isolated per recipe.run() invocation
  • Nested recipes inherit a snapshot by default
  • Explicit parent export via export_runtime_variables on recipe connector
  • Deterministic conflict handling: export_conflict_policy: error|overwrite
  • Protected names blocked from assignment (row_count, column_count, columns, df, recipe_variables, applied_permission_group, internal runtime keys)

• Inspection/logging safety

  • log wrangle supports runtime_variables and runtime_max_items
  • Redaction for marked secrets + sensitive-name heuristics + signed URL query secrets
  • Bounded/truncated rendering for large containers and non-text payloads

• Schema/docs/tests

  • Updated run common schema for result_variable
  • Added connector schema for variables
  • README section documents syntax, compatibility boundaries, and nested/parallel semantics
  • Added focused tests for sequencing, reassignment, typed values, missing/escaped refs, no-input startup, nested export conflicts, branch isolation, redaction, and caller-input isolation

run:
  on_start:
    - custom.build_reference:
        result_variable: upload_ref
    - variables:
        update:
          upload_ref:
            status: ready
        inspect:
          - upload_ref.file_id

wrangles:
  - create.column:
      output: reference_id
      value: ${runtime.upload_ref.file_id}

How it was verified

Focused suites covering runtime behavior and regressions:

• tests/recipes/test_run.py
• tests/recipes/test_variables.py
• tests/connectors/test_recipe.py
• tests/connectors/test_concurrent.py
• tests/connectors/test_matrix.py
• targeted tests/recipes/wrangles/test_main.py log/python/matrix cases
• parallel_validation run (CodeQL clean; code-review subtool unavailable in this environment)

Compatibility and risk

• Backward compatible by default: existing ${variable} remains load-time/static; mutable behavior is opt-in via ${runtime...} and variables action.
• System/computed values remain read-only and separate from assignable runtime state.
• No global/process-wide runtime store introduced; isolation is preserved across parallel branches and concurrent recipe.run() calls unless explicitly exported.
• Rollback is low-risk: remove runtime-resolution/mutation paths and keep legacy static templating behavior.

Ready-for-review checklist

• One human delivery owner is assigned
• The linked issue and intended milestone are correct
• The branch is current with main and has no merge conflicts
• Focused tests pass
• New or changed behavior has direct test coverage
• Documentation/schema/configuration is updated where applicable
• The PR contains no unrelated changes
• The PR description reflects the branch's current scope and latest validation
• One primary reviewer is requested only when this PR is ready

See the pull request workflow.

[mborodii-prog]

@ebhills export_runtime_variables silently fails to propagate when the same recipe:/read:/write: action also passes an explicit variables: override — no error is raised; the parent recipe later fails trying to read a variable that was never actually exported.
For example: import wrangles

recipe = """
run:
on_start:
- variables: {set: {parent_flag: parent}}
- recipe:
variables: {SOME_OVERRIDE: hello} # <- explicit override present
run:
on_start: [{variables: {set: {child_export: exported_value}}}]
export_runtime_variables: [child_export]
wrangles:

• create.column: {output: child_export, value: ${runtime.child_export}}
  """
  wrangles.recipe.run(recipe)

Expected: child_export column populated with exported_value. Actual: ValueError: create.column (line 8) - Runtime variable '${runtime.child_export}' was not found. Removing the variables: {SOME_OVERRIDE: hello} line makes the export work correctly — confirming the override is what breaks propagation.

[mborodii-prog]

@ebhills set + update on the same key in one variables: action either raises a spurious "not found for update" error or silently drops the set value — the merge logic validates/reads against pre-action state instead of the in-flight set values.
import wrangles

recipe = """
run:
on_start:
- variables:
set: {counter: {a: 1}}
update: {counter: {b: 2}}
wrangles:

• create.column: {output: out, value: ${runtime.counter}}
  """
  wrangles.recipe.run(recipe)

Expected: counter == {a: 1, b: 2}. Actual: ValueError: variables (line 4) - Runtime variable 'counter' was not found for update. (If counter already existed before this action, the error doesn't fire, but the set value is silently discarded — update merges onto the stale pre-action value instead.)

[mborodii-prog]

@ebhills if: gates are evaluated after eager resolution of all other parameters, across all four execution paths (_run_actions, _read_data, _execute_wrangles, _write_data). A step that should be skipped can still raise ValueError from an unrelated, not-yet-set runtime variable referenced elsewhere in the same action. This is the exact pattern used in the README's own worked example.

import wrangles

recipe = """
run:
on_start:
- variables: {set: {enabled: false}}
wrangles:

• create.column:
  if: ${runtime.enabled} == True
  output: out
  value: ${runtime.not_set_variable}
  """
  wrangles.recipe.run(recipe)

Expected: step skipped (condition is false), no error. Actual: ValueError: create.column (line 8) - Runtime variable '${runtime.not_set_variable}' was not found. This is exactly the "declare a variable only inside the branch that needs it" pattern used in the feature's own README example — so the flagship example is expected to fail the same way. Same root cause reproduces in _read_data, _execute_wrangles, and _write_data (just substitute a read:/run:/write: step with the same if:/unrelated-${runtime.X} shape).