Skip to content

BE-457: HashQL: MIR execution pipeline extensions for postgres compilation#8525

Open
indietyp wants to merge 4 commits intobm/be-456-hashql-entity-type-model-enrichment-and-graph-store-queryfrom
bm/be-457-hashql-mir-execution-pipeline-extensions-for-postgres
Open

BE-457: HashQL: MIR execution pipeline extensions for postgres compilation#8525
indietyp wants to merge 4 commits intobm/be-456-hashql-entity-type-model-enrichment-and-graph-store-queryfrom
bm/be-457-hashql-mir-execution-pipeline-extensions-for-postgres

Conversation

@indietyp
Copy link
Copy Markdown
Member

@indietyp indietyp commented Mar 8, 2026

🌟 What is the purpose of this PR?

Prepares the MIR execution pipeline for consumption by the postgres compiler. The execution analysis now produces a complete IslandGraph (not just a flat island list), the placement solver and island placement accept external allocators, and the traversal system gains the TraversalPathBitMap and as_symbol() APIs that the SQL generator needs. Also adds a backend switch cost to the terminator placement so cross-backend transitions are no longer free.

🔍 What does this change?

Execution analysis (pass/execution/mod.rs):

  • run() becomes run_in() with an explicit allocator parameter, returning ExecutionAnalysisResidual (assignment + island graph). The island graph is now constructed as part of the analysis rather than left to the caller.
  • Adds run_all_in() which runs the execution analysis over all graph-read bodies in a DefIdSlice.

Traversal system (traversal/mod.rs, traversal/entity.rs):

  • TraversalPathBitMap: per-vertex-type collection of TraversalPathBitSets with pointwise lattice operations. The postgres compiler uses this to track which paths each island accesses across all vertex types.
  • TraversalPath::as_symbol(): returns a static symbol for each path variant, used as SQL column aliases so the interpreter can locate result columns by name.
  • TraversalPathBitSet::vertex(): returns the vertex type for a bitset.
  • EntityPath::as_symbol() and EntityPath::column_name(): per-path SQL identifiers.

Terminator placement (terminator_placement/mod.rs):

  • TransMatrix gains AddAssign for element-wise saturating addition.
  • Adds backend_switch_cost() which encodes a fixed overhead for cross-backend transitions (Postgres to Interpreter: 8, Interpreter to Embedding: 4, etc.). Previously cross-backend transitions had zero inherent cost, so empty blocks were arbitrarily assigned to the interpreter even when staying on postgres was free.

Placement solver (placement/solve/):

  • PlacementSolver::run() becomes run_in() with allocator parameter.
  • CSP solver internal allocations use the provided allocator.

Island graph (island/graph/mod.rs):

  • IslandGraph::new_in() now takes an allocator for its output storage.

Pretty printer (pretty/text.rs):

  • TextFormatAnnotations gains annotate_basic_block() and BasicBlockAnnotation associated type, plus a blanket impl for &mut T.

Builder (builder/rvalue.rs):

  • Adds RValueBuilder::opaque_entity_uuid() convenience constructor for the common pattern of extracting an entity's UUID.

Pre-Merge Checklist 🚀

🚢 Has this modified a publishable library?

This PR:

  • does not modify any publishable blocks or libraries, or modifications do not need publishing

📜 Does this require a change to the docs?

The changes in this PR:

  • are internal and do not require a docs change

🕸️ Does this require a change to the Turbo Graph?

The changes in this PR:

  • do not affect the execution graph

⚠️ Known issues

The backend_switch_cost() values (8, 4, etc.) are hand-tuned heuristics. They correctly prevent the solver from arbitrarily switching backends for empty blocks, but a proper cost model would derive these from measured overhead.

🛡 What tests cover this?

  • Updated execution pass end-to-end tests (execution/tests.rs) including entity_uuid_equality, mixed_postgres_embedding_interpreter, projection_and_apply_splits
  • Updated statement placement tests for interpret and postgres backends (eq_opaque_entity_uuid)
  • Updated terminator placement and island tests
  • Placement solver tests updated for allocator API

❓ How to test this?

cargo nextest run -p hashql-mir

feat: checkpoint (II)

feat: checkpoint (III)

feat: snapshot vec

feat: add dedicated filter

feat: checkpoint

feat: filter implementation

feat: filter implementation (mostly) done

chore: environment capture note

chore: always postgres bigint

feat: target clone

feat: simplify lookup

feat: move storage up

feat: eval entity path

chore: checkpoint

chore: checkpoint

chore: find entrypoint

feat: eval context

feat: eval cleanup

chore: cleanup

feat: track index

feat: wire up filter

feat: add error reporting

chore: checkpoint

feat: add traverse, and first postgres compiler outline

feat: traverse bitmap

feat: move traversal out

feat: projections

feat: projections

fix: clippy

feat: subquery projection for lateral

feat: checkpoint

feat: test plan

feat: checkpoint

feat: checkpoint – failing tests ;-;

feat: checkpoint – failing tests ;-;

feat: checkpoint — passing tests

fix: import

fix: entity type

feat: checkpoint

feat: attribute a cost to terminator placement switches

fix: import

feat: checkpoint

feat: checkpoint

chore: lint
@cursor
Copy link
Copy Markdown

cursor Bot commented Mar 8, 2026

PR Summary

Medium Risk
Medium risk because it changes core execution placement/graph construction and introduces new cross-backend transition costs, which can shift scheduling/target assignments across backends.

Overview
Extends the MIR execution pipeline for downstream Postgres compilation. ExecutionAnalysis now returns an ExecutionAnalysisResidual containing both the per-block target assignment and a fully built IslandGraph, with new run_in(...)/run_all_in(...) APIs that take an explicit output allocator.

Refactors placement/island plumbing to support external allocators and richer graph queries. The placement solver and island placement APIs are renamed to *_in variants and updated to allocate outputs via a caller-provided allocator; IslandGraph gains helpers like members(), contains(), and find().

Adds SQL-oriented traversal identifiers and multi-vertex traversal tracking. Traversal paths now expose as_symbol() (and EntityPath::as_symbol()), TraversalPathBitSet::vertex(), and a new TraversalPathBitMap with pointwise lattice ops for tracking accessed paths across vertex types.

Adjusts backend switching economics and MIR test/pretty output. Terminator placement now adds fixed backend-switch overheads via backend_switch_cost() (using TransMatrix +=), which updates solver behavior and snapshots; the MIR builder adds opaque aggregate support and the pretty printer can annotate basic blocks, with updated/added execution and statement-placement tests (notably EntityUuid equality).

Reviewed by Cursor Bugbot for commit fd9a3a8. Bugbot is set up for automated code reviews on this repo. Configure here.

@vercel
Copy link
Copy Markdown

vercel Bot commented Mar 8, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
hash Ready Ready Preview, Comment May 8, 2026 8:41am
petrinaut Ready Ready Preview May 8, 2026 8:41am
2 Skipped Deployments
Project Deployment Actions Updated (UTC)
hashdotdesign Ignored Ignored Preview May 8, 2026 8:41am
hashdotdesign-tokens Ignored Ignored Preview May 8, 2026 8:41am

Copy link
Copy Markdown
Member Author

indietyp commented Mar 8, 2026

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

This stack of pull requests is managed by Graphite. Learn more about stacking.

@augmentcode
Copy link
Copy Markdown

augmentcode Bot commented Mar 8, 2026

🤖 Augment PR Summary

Summary: Prepares the HashQL MIR execution pipeline for Postgres compilation by returning a full island dependency graph and exposing traversal/placement APIs needed by the SQL generator.
Key Changes:

  • Execution analysis now takes an explicit allocator (run_in) and returns ExecutionAnalysisResidual (assignment + IslandGraph); adds run_all_in for batch analysis.
  • Traversal adds TraversalPathBitMap, TraversalPath::as_symbol(), and EntityPath::as_symbol()/column_name() for stable SQL column aliases.
  • Placement solver and island placement APIs accept external allocators and route CSP internal allocations through the provided bump allocator.
  • Terminator placement introduces a fixed backend_switch_cost() so cross-backend transitions have non-zero overhead (avoids “free” backend flips).
  • Island graph gains helper queries (members, contains, find) and is now built/resolved within execution analysis.
  • Pretty printer supports per-basic-block annotations; MIR builder gains an opaque aggregate constructor and new tests cover opaque UUID equality.
Tests: Updates/extends execution, statement placement, terminator placement, and allocator-related solver snapshots (incl. entity_uuid_equality). Notes: Backend switch costs are heuristics intended to be refined by a measured cost model.

🤖 Was this summary useful? React with 👍 or 👎

Copy link
Copy Markdown

@augmentcode augmentcode Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review completed. No suggestions at this time.

Comment augment review to trigger a new review at any time.

@codecov
Copy link
Copy Markdown

codecov Bot commented Mar 8, 2026

Codecov Report

❌ Patch coverage is 64.24581% with 128 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (bm/be-456-hashql-entity-type-model-enrichment-and-graph-store-query@1a97257). Learn more about missing BASE report.

Files with missing lines Patch % Lines
...cal/hashql/mir/src/pass/execution/traversal/mod.rs 0.00% 44 Missing ⚠️
libs/@local/hashql/mir/src/pretty/text.rs 21.62% 28 Missing and 1 partial ⚠️
.../hashql/mir/src/pass/execution/traversal/entity.rs 0.00% 28 Missing ⚠️
libs/@local/hashql/mir/src/pass/execution/mod.rs 44.00% 14 Missing ⚠️
.../hashql/mir/src/pass/execution/island/graph/mod.rs 35.29% 11 Missing ⚠️
.../mir/src/pass/execution/placement/solve/csp/mod.rs 90.00% 0 Missing and 1 partial ⚠️
libs/@local/hashql/mir/src/pass/execution/tests.rs 99.07% 1 Missing ⚠️
Additional details and impacted files
@@                                          Coverage Diff                                           @@
##             bm/be-456-hashql-entity-type-model-enrichment-and-graph-store-query    #8525   +/-   ##
======================================================================================================
  Coverage                                                                       ?   85.26%           
======================================================================================================
  Files                                                                          ?      349           
  Lines                                                                          ?    55110           
  Branches                                                                       ?     1339           
======================================================================================================
  Hits                                                                           ?    46991           
  Misses                                                                         ?     7672           
  Partials                                                                       ?      447           
Flag Coverage Δ
rust.hashql-compiletest 29.63% <ø> (?)
rust.hashql-mir 92.08% <64.24%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@codspeed-hq
Copy link
Copy Markdown

codspeed-hq Bot commented Mar 8, 2026

Merging this PR will not alter performance

✅ 24 untouched benchmarks
⏩ 56 skipped benchmarks1


Comparing bm/be-457-hashql-mir-execution-pipeline-extensions-for-postgres (60da845) with bm/be-456-hashql-entity-type-model-enrichment-and-graph-store-query (1a97257)

Open in CodSpeed

Footnotes

  1. 56 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@indietyp indietyp force-pushed the bm/be-456-hashql-entity-type-model-enrichment-and-graph-store-query branch from 6ef6d13 to ef9b858 Compare March 8, 2026 16:15
@indietyp indietyp force-pushed the bm/be-457-hashql-mir-execution-pipeline-extensions-for-postgres branch from 5d95ba7 to 1aa0f1c Compare March 8, 2026 16:15
@indietyp indietyp force-pushed the bm/be-456-hashql-entity-type-model-enrichment-and-graph-store-query branch from ef9b858 to a91293c Compare March 8, 2026 16:56
@indietyp indietyp force-pushed the bm/be-457-hashql-mir-execution-pipeline-extensions-for-postgres branch from 1aa0f1c to 87024df Compare March 8, 2026 16:56
@indietyp indietyp force-pushed the bm/be-456-hashql-entity-type-model-enrichment-and-graph-store-query branch from 61d002f to bc50eb0 Compare April 21, 2026 17:06
@indietyp indietyp force-pushed the bm/be-457-hashql-mir-execution-pipeline-extensions-for-postgres branch from c67124e to 0bd3b25 Compare April 21, 2026 17:06
@indietyp indietyp force-pushed the bm/be-457-hashql-mir-execution-pipeline-extensions-for-postgres branch from 0bd3b25 to a675701 Compare April 29, 2026 13:08
@indietyp indietyp force-pushed the bm/be-456-hashql-entity-type-model-enrichment-and-graph-store-query branch from bc50eb0 to 8b6ee87 Compare April 29, 2026 13:08
Comment thread libs/@local/hashql/mir/src/pass/execution/tests.rs
Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 5552cbb. Configure here.

Comment thread libs/@local/hashql/mir/src/pass/execution/mod.rs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/libs Relates to first-party libraries/crates/packages (area) area/tests New or updated tests type/eng > backend Owned by the @backend team

Development

Successfully merging this pull request may close these issues.

2 participants