Skip to content

fix: guard Fusion's None target.threads and empty failures in all adapter overrides - #569

Open
mbiyashev3 wants to merge 2 commits into
brooklyn-data:mainfrom
mbiyashev3:fix/fusion-none-guards-all-adapters
Open

fix: guard Fusion's None target.threads and empty failures in all adapter overrides#569
mbiyashev3 wants to merge 2 commits into
brooklyn-data:mainfrom
mbiyashev3:fix/fusion-none-guards-all-adapters

Conversation

@mbiyashev3

@mbiyashev3 mbiyashev3 commented Aug 18, 2026

Copy link
Copy Markdown

Overview

#549 guarded default__ only. Because adapter.dispatch prefers an adapter-specific override, every warehouse that ships its own macro still renders the bare Python literal None into the invocations INSERT — so BigQuery, Postgres, Trino and SQL Server remain broken under dbt Fusion. This applies the same guard to those four overrides, and (second commit) aligns the five test_executions overrides with the failures guard #549 applied to default__.

Update type - breaking / non-breaking

  • Minor bug fix
  • Documentation improvements
  • Quality of Life improvements
  • New features (non-breaking change)
  • New features (breaking change)
  • Other (non-breaking change)
  • Other (breaking change)
  • Release preparation

What does this solve?

Under dbt Fusion (dbt_version 2.0.0, e.g. the dbt Cloud Fusion runtime) target.threads is None. In on-run-end, dbt_artifacts.upload_results() renders that None straight into the invocations INSERT as a bare Python literal, and the warehouse rejects the statement.

Observed on BigQuery with dbt_artifacts 2.11.0 — which already contains #549:

BigQuery error: Unrecognized name: None at [42:9]

The emitted SQL:

insert into `<proj>`.`dbt_artifacts`.`invocations`
  (command_invocation_id, ..., target_schema, target_threads, ...)
values (
  '01a0152c-...', '2.0.0', 'myproject', '2026-08-18 14:00:31.639301+00:00',
  'build', False, 'user', 'default', 'analytics',
  None,                      -- <-- positional column 10, target_threads
  ...
)

All 19 values in that tuple are valid warehouse literals except position 10. None is the sole defect.

Impact is worse than a failed statement. invocations is 8th in the upload order, so the failure kills the tail of the sequence — sources, tests and models never upload (0 rows) — and every already-inserted model_executions / test_executions row is left orphaned on command_invocation_id. stg_dbt__invocations and fct_dbt__invocations come out empty, so every invocation-joined mart is empty too. That is silent bad data, not just staleness.

#549 was an incomplete fix

Merge commit c8c0b59 is a 3-line diff touching two files, and in both cases it patched only the default__ macro. The four adapter overrides were left with a bare {{ target.threads }}, and this is still unfixed on main as of the 2.11.0 release merge (3d27645) — so no released version of the package fixes BigQuery:

$ git show 3d27645:macros/upload_individual_datasets/upload_invocations.sql | grep -n "target.threads"
57:        {# dbt-fusion sets target.threads to None — guard against it #}
58:        {% if target.threads is not none %}{{ target.threads }}{% else %}cast(null as int){% endif %}, {# target_threads #}
112:        {{ target.threads }}, {# target_threads #}
175:            {{ target.threads }}, {# target_threads #}
237:            {{ target.threads }}, {# target_threads #}
320:        {{ target.threads }}, {# target_threads #}

Snowflake is unaffected for invocations (it falls through to default__), but not for test_executions, which does have its own override.

Commit 1 — target.threads

Guards bigquery__, postgres__, trino__, sqlserver__. The null literal is chosen per branch based on the surrounding construct rather than copied verbatim from default__:

Branch Construct Literal Why
bigquery__, postgres__, trino__ bare tuple, consumed by insert into <rel> (<cols>) values (<tuple>) null The insert target's column type is known, so an untyped NULL is unambiguous; a cast would be noise.
sqlserver__ select "1"…"19" from (values ( ... )) v (...) cast(null as {{ dbt.type_int() }}) The derived column's type comes from the VALUES row, not the insert target — the same reason #549 needed a cast in default__.

For the typed case I used dbt.type_int() rather than hardcoding, to match how the column is declared in models/sources/invocations.sql:19 (cast(null as {{ type_int() }}) as target_threads) and how sibling macros in this package reference dbt's type helpers. dbt.type_int() is dispatched and resolves to int everywhere except BigQuery, where it resolves to int64 — so it stays correct if a future branch needs a typed null.

Note: this package has no type_int helper of its own in macros/database_specific_helpers/; type_int is dbt-core's global macro. I deliberately did not add a dispatched helper for this, to keep a values-only fix from growing into database_specific_helpers/ churn.

Commit 2 — failures (independent; drop it if you'd rather ship the target.threads fix alone)

#549 also replaced the failures expression in default__get_test_executions_dml_sql with a guard covering both None and '', but left the five overrides on the older form:

{{ 'null' if test.failures is none else test.failures }}

That handles None but not ''. When test.failures is an empty string the expression renders to nothing at all, and the tuple collapses to an empty positional value — a syntax error on every warehouse:

..., null, , 'message', ...
             ^ position 12, failures

This aligns bigquery__, postgres__, snowflake__, sqlserver__ and trino__ with the default__ form. A literal 0 still renders as 0 — the guard tests is not none and != '' rather than truthiness, so a legitimate zero-failure count is not collapsed to null.

Commit 2 is independent of commit 1 and can be dropped without touching the target.threads fix.

Not changed

Values only — no columns added, removed or reordered, so the column-order invariant across the upload macros, models/sources/* and get_column_name_lists.sql (CONTRIBUTING.md) is untouched. No version bump beyond a patch is implied.

Outstanding questions

  • I could not run any integration test. This machine has no container runtime (no docker / colima / podman), so the Postgres, Trino and SQL Server legs could not run, and ./scripts/ci/setup.sh fails before completing because pyodbc cannot build without the MS ODBC Driver 18. Please treat Tier 1 CI on this PR as the first real execution of these paths. What I did verify is described under "What databases have you tested with?" below.
  • No committed integration coverage for the None path. target.threads is never None under dbt-core — profiles.yml sets it and dbt supplies a default — so a green integration run exercises only the non-None branch and would pass identically with or without this patch. I could not find a way to force target.threads = None through the existing harness without a Fusion runtime in CI. Reproducing this in CI likely needs a Fusion leg, which feels like its own piece of work; happy to take direction if you'd like it attempted here.
  • Same pattern, out of scope here: macros/upload_individual_datasets/upload_model_executions.sql renders {{ model.execution_time }} bare with no guard at all (6 branches), and renders config_full_refresh unquoted in the bigquery__ and trino__ branches. These are plausibly the next thing Fusion breaks. config_full_refresh is the less exposed of the two — it already falls back to flags.FULL_REFRESH when None, so it only breaks if that flag is itself None — whereas model.execution_time has no guard whatsoever. I left both alone to keep this PR to the reported bug; happy to open a follow-up.
  • Lint: ./scripts/ci/lint.sh runs SQLFluff against models/ only, and this PR touches macros/ exclusively — so these changes fall outside sqlfluff's scope entirely, rather than being unlinted for want of Snowflake credentials.

What databases have you tested with?

  • Snowflake
  • Google BigQuery
  • Databricks
  • Spark
  • N/A

No warehouse was executed against — I want to be explicit about that. No container runtime was available for the Postgres / Trino / SQL Server legs (see Outstanding questions), and fork PRs cannot reach the Snowflake or BigQuery secrets in any case.

What I verified instead, offline:

  1. Both files parse. Rendered macros/upload_individual_datasets/upload_invocations.sql and upload_test_executions.sql through a Jinja2 environment with jinja2.ext.do enabled (matching dbt's) — both parse cleanly, confirming the nested {{ dbt.type_int() }} inside an {% if %}/{% else %} output branch is valid.
  2. Every guard renders the intended literal. For all five target_threads sites: threads=88; threads=Nonenull (bigquery/postgres/trino) and cast(null as int) (default__, sqlserver). For all six failures sites: 33, 00, Nonenull, ''null.
  3. The failures bug reproduces. Running the same render against the pre-patch overrides with failures='' produces an empty value, which is what commit 2 fixes.

This is template-level verification, not warehouse verification — it proves the Jinja is correct and emits the literals I claim, and does not prove those literals are accepted by each engine. The per-branch reasoning for null vs cast(null as int) is set out in the table above and is the part most worth a reviewer's eye. Postgres / Trino / SQL Server should be covered by Tier 1 CI on this PR — though the Tier 1 run is currently sitting at action_required, so it needs a maintainer to approve workflows for this fork before it will execute. Snowflake, BigQuery and Databricks will need a maintainer post-merge.

Related

I searched open and closed issues and PRs for target.threads, Unrecognized name: None and fusion. There is no existing issue or PR for this bug and no duplicate of this change. The Fusion-tagged issues that do exist are unrelated: #523 (+column_store config), #515 (+as_columnstore warning), #555 (; in docs.md breaking dollar-quoted comments on Postgres), #552 (state-aware orchestration). #549 is the PR this one completes.

mbiyashev3 and others added 2 commits August 18, 2026 12:45
dbt Fusion (dbt_version 2.0.0) leaves target.threads as None. brooklyn-data#549 added a
guard, but only to default__get_invocations_dml_sql. Because
adapter.dispatch prefers an adapter-specific override, every warehouse with
its own macro still rendered the bare Python literal `None` into the
invocations INSERT, which the warehouse rejects:

  BigQuery: Unrecognized name: None at [42:9]

invocations is 8th in the upload order, so the failure also aborts the
sources/tests/models uploads and orphans the already-inserted
model_executions / test_executions rows on command_invocation_id.

Guards bigquery__, postgres__, trino__ and sqlserver__.

Null literal chosen per branch by surrounding construct:
  - bigquery/postgres/trino emit a bare tuple consumed as
    `insert into <rel> (<cols>) values (<tuple>)`, so the target column type
    is known and an untyped `null` is unambiguous.
  - sqlserver wraps its tuple in `select ... from (values (...)) v (...)`,
    where the derived column's type comes from the row rather than the
    insert target, so a typed null is required. Uses dbt.type_int() to match
    how target_threads is declared in models/sources/invocations.sql.

Values-only change: no columns added, removed or reordered.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ides

brooklyn-data#549 replaced the `failures` expression in
default__get_test_executions_dml_sql with a guard that handles both None and
the empty string, but left the five adapter overrides on the older form:

  {{ 'null' if test.failures is none else test.failures }}

That form handles None but not ''. When test.failures is an empty string the
expression renders to nothing at all, so the VALUES tuple collapses to
`..., null, , 'message', ...` — an empty positional value, which is a syntax
error on every warehouse.

Aligns bigquery__, postgres__, snowflake__, sqlserver__ and trino__ with the
default__ form from brooklyn-data#549. Note snowflake__ was previously fixed for
invocations only by falling through to default__; test_executions has its
own override and so was still on the old form.

A literal 0 continues to render as 0 -- the guard tests `is not none` and
`!= ''` rather than truthiness, so a legitimate zero-failure count is not
collapsed to null.

Values-only change: no columns added, removed or reordered.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@mbiyashev3

Copy link
Copy Markdown
Author

@mtcarlone — tagging you since you reviewed and merged #549; this PR finishes that fix for the adapter overrides adapter.dispatch actually selects, so BigQuery/Postgres/Trino/SQL Server stop rendering a bare None under Fusion. (I don't have permission to add you via the reviewers field from a fork, hence the mention.)

Two things that would help when you get a chance:

  1. Tier 1 CI is stuck at action_required and needs a maintainer to approve workflows for this fork before Postgres/Trino/SQL Server will run. I had no container runtime available locally, so that CI run is the first real execution of these code paths — worth having green before this is taken seriously.
  2. The per-branch null literal is the part worth reviewing. I used a bare null for bigquery/postgres/trino (their tuple is consumed by insert into <rel> (<cols>) values (...), so the column type is known) and cast(null as {{ dbt.type_int() }}) for sqlserver (its tuple sits inside from (values (...)) v (...), where the type comes from the row — the same reason Update upload_invocations.sql #549 needed a cast in default__). Details in the PR description.

The second commit (failures empty-string guard) is independent and can be dropped if you'd rather ship the target.threads fix alone.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant