BE-456: HashQL: Entity type model enrichment and graph store query extensions#8524
Conversation
feat: checkpoint (II) feat: checkpoint (III) feat: snapshot vec feat: add dedicated filter feat: checkpoint feat: filter implementation feat: filter implementation (mostly) done chore: environment capture note chore: always postgres bigint feat: target clone feat: simplify lookup feat: move storage up feat: eval entity path chore: checkpoint chore: checkpoint chore: find entrypoint feat: eval context feat: eval cleanup chore: cleanup feat: track index feat: wire up filter feat: add error reporting chore: checkpoint feat: add traverse, and first postgres compiler outline feat: traverse bitmap feat: move traversal out feat: projections feat: projections fix: clippy feat: subquery projection for lateral feat: checkpoint feat: test plan feat: checkpoint feat: checkpoint – failing tests ;-; feat: checkpoint – failing tests ;-; feat: checkpoint — passing tests fix: import fix: entity type feat: checkpoint feat: attribute a cost to terminator placement switches fix: import feat: checkpoint feat: checkpoint chore: lint
|
The latest updates on your projects. Learn more about Vercel for GitHub.
2 Skipped Deployments
|
PR SummaryMedium Risk Overview Extends the Postgres query builder with new SQL constructs (e.g., Enriches HashQL’s entity type model to include metadata/temporal/provenance/confidence/encodings and updates graph read path resolution plus a large set of compiletest fixtures to reflect the new Reviewed by Cursor Bugbot for commit 959fba1. Bugbot is set up for automated code reviews on this repo. Configure here. |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
Autofix Details
Bugbot Autofix prepared fixes for both issues found in the latest run.
- ✅ Fixed: Misleading enum variant name for
jsonb_aggfunction- Renamed
Function::JsonAggtoFunction::JsonbAggand updated all internal usages/tests to match the emittedjsonb_agg(...)SQL.
- Renamed
- ✅ Fixed: Multiple new symbols defined but never referenced
- Removed the eight unreferenced symbol definitions from
sym.rsto eliminate dead entries and duplicate provenance naming.
- Removed the eight unreferenced symbol definitions from
Or push these changes by commenting:
@cursor push 928f4a2c0b
Preview (928f4a2c0b)
diff --git a/libs/@local/graph/postgres-store/src/store/postgres/query/expression/conditional.rs b/libs/@local/graph/postgres-store/src/store/postgres/query/expression/conditional.rs
--- a/libs/@local/graph/postgres-store/src/store/postgres/query/expression/conditional.rs
+++ b/libs/@local/graph/postgres-store/src/store/postgres/query/expression/conditional.rs
@@ -17,7 +17,7 @@
pub enum Function {
Min(Box<Expression>),
Max(Box<Expression>),
- JsonAgg(Box<Expression>),
+ JsonbAgg(Box<Expression>),
JsonExtractText(Box<Expression>),
JsonExtractAsText(Box<Expression>, PathToken<'static>),
JsonExtractPath(Vec<Expression>),
@@ -55,7 +55,7 @@
expression.transpile(fmt)?;
fmt.write_char(')')
}
- Self::JsonAgg(expression) => {
+ Self::JsonbAgg(expression) => {
fmt.write_str("jsonb_agg(")?;
expression.transpile(fmt)?;
fmt.write_char(')')
@@ -821,9 +821,9 @@
}
#[test]
- fn transpile_json_agg() {
+ fn transpile_jsonb_agg() {
assert_eq!(
- Expression::Function(Function::JsonAgg(Box::new(Expression::Parameter(1))))
+ Expression::Function(Function::JsonbAgg(Box::new(Expression::Parameter(1))))
.transpile_to_string(),
"jsonb_agg($1)"
);
diff --git a/libs/@local/hashql/core/src/symbol/sym.rs b/libs/@local/hashql/core/src/symbol/sym.rs
--- a/libs/@local/hashql/core/src/symbol/sym.rs
+++ b/libs/@local/hashql/core/src/symbol/sym.rs
@@ -36,7 +36,6 @@
entity,
entity_edition_id,
entity_id,
- entity_type_ids,
entity_uuid,
eq,
Err,
@@ -55,8 +54,6 @@
left_entity_confidence,
left_entity_id,
left_entity_provenance,
- left_entity_uuid,
- left_entity_web_id,
link_data,
List,
lt,
@@ -76,10 +73,7 @@
or,
pow,
properties,
- property_metadata,
provenance,
- provenance_edition,
- provenance_inferred,
provided,
r#as: "as",
r#as_force: "as!",
@@ -101,8 +95,6 @@
right_entity_confidence,
right_entity_id,
right_entity_provenance,
- right_entity_uuid,
- right_entity_web_id,
Some,
special_form,
String,
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #8524 +/- ##
==========================================
+ Coverage 62.29% 62.31% +0.01%
==========================================
Files 1353 1354 +1
Lines 136575 137003 +428
Branches 5788 5792 +4
==========================================
+ Hits 85085 85372 +287
- Misses 50582 50725 +143
+ Partials 908 906 -2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
🤖 Augment PR SummarySummary: This PR enriches the HashQL entity type model to better reflect the full graph database schema, and extends the Postgres query-builder/AST to support additional SQL constructs needed by the HashQL→Postgres compiler. Changes:
Technical Notes: Query builder aliasing is standardized on an 🤖 Was this summary useful? React with 👍 or 👎 |
6ef6d13 to
ef9b858
Compare
a4534ce to
4503abc
Compare
b9e2eb9 to
1a97257
Compare
4503abc to
8452cc0
Compare
1a97257 to
4f33a35
Compare
4f33a35 to
959fba1
Compare
Benchmark results
|
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 2002 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 1001 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: high, policies: 3314 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: medium, policies: 1526 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 2078 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 1033 | Flame Graph |
policy_resolution_medium
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 102 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 51 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: high, policies: 269 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: medium, policies: 107 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 133 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 63 | Flame Graph |
policy_resolution_none
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 2 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 8 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 3 | Flame Graph |
policy_resolution_small
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 52 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 25 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: high, policies: 94 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: medium, policies: 26 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 66 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 29 | Flame Graph |
read_scaling_complete
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_id;one_depth | 1 entities | Flame Graph | |
| entity_by_id;one_depth | 10 entities | Flame Graph | |
| entity_by_id;one_depth | 25 entities | Flame Graph | |
| entity_by_id;one_depth | 5 entities | Flame Graph | |
| entity_by_id;one_depth | 50 entities | Flame Graph | |
| entity_by_id;two_depth | 1 entities | Flame Graph | |
| entity_by_id;two_depth | 10 entities | Flame Graph | |
| entity_by_id;two_depth | 25 entities | Flame Graph | |
| entity_by_id;two_depth | 5 entities | Flame Graph | |
| entity_by_id;two_depth | 50 entities | Flame Graph | |
| entity_by_id;zero_depth | 1 entities | Flame Graph | |
| entity_by_id;zero_depth | 10 entities | Flame Graph | |
| entity_by_id;zero_depth | 25 entities | Flame Graph | |
| entity_by_id;zero_depth | 5 entities | Flame Graph | |
| entity_by_id;zero_depth | 50 entities | Flame Graph |
read_scaling_linkless
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_id | 1 entities | Flame Graph | |
| entity_by_id | 10 entities | Flame Graph | |
| entity_by_id | 100 entities | Flame Graph | |
| entity_by_id | 1000 entities | Flame Graph | |
| entity_by_id | 10000 entities | Flame Graph |
representative_read_entity
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/block/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/book/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/building/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/organization/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/page/v/2
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/person/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/playlist/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/song/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/uk-address/v/1
|
Flame Graph |
representative_read_entity_type
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| get_entity_type_by_id | Account ID: bf5a9ef5-dc3b-43cf-a291-6210c0321eba
|
Flame Graph |
representative_read_multiple_entities
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_property | traversal_paths=0 | 0 | |
| entity_by_property | traversal_paths=255 | 1,resolve_depths=inherit:1;values:255;properties:255;links:127;link_dests:126;type:true | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:0;link_dests:0;type:false | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:1;link_dests:0;type:true | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:2;links:1;link_dests:0;type:true | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:2;properties:2;links:1;link_dests:0;type:true | |
| link_by_source_by_property | traversal_paths=0 | 0 | |
| link_by_source_by_property | traversal_paths=255 | 1,resolve_depths=inherit:1;values:255;properties:255;links:127;link_dests:126;type:true | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:0;link_dests:0;type:false | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:1;link_dests:0;type:true | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:2;links:1;link_dests:0;type:true | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:2;properties:2;links:1;link_dests:0;type:true |
scenarios
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| full_test | query-limited | Flame Graph | |
| full_test | query-unlimited | Flame Graph | |
| linked_queries | query-limited | Flame Graph | |
| linked_queries | query-unlimited | Flame Graph |


🌟 What is the purpose of this PR?
Enriches the HashQL entity type model to match the full database schema, extends the graph-store SQL query builder with new expression types needed by the postgres compiler, and adds a database migration for the
continuationcomposite type. This is the foundation layer for the postgres translation stack (BE-457, BE-306).🔍 What does this change?
hashql-core:
IdSnapshotVec, a typed-ID-indexed vector with transactional snapshot/rollback support (used by the postgres compiler for scoped state)TemporalInterval,EntityTemporalMetadata,Confidence, provenance types,EntityMetadata, andLinkDatafields to match the full database schema. Previously onlyEntityId,EntityRecordId, and partialLinkDatawere modeled.edition_idandontology_type_idssymbolsgraph-postgres-store query builder:
FieldAccess(composite field extraction),JsonAgg,Constant::Null,Constant::U128,UnaryOperator::IsNotFalseJsonB,Continuation,Numeric,Int,BigIntUnnestnow accepts multiple expressions (UNNEST(a, b, c))SelectStatementgainsOFFSETsupport.cast(),.not()and.grouped()now takeself&'static strtoIdentifierfor consistencymod postgresandmod tablemadepubfor use by the HashQL postgres compilerDatabase migration (v010__query):
continuationcomposite type:(filter boolean, block int, locals int[], values jsonb[]). This is the row type returned by compiled filter subqueries; the interpreter reads.filterto decide row inclusion and.block/.locals/.valuesto resume multi-block CFGs.hashql-eval:
graph/read/path.rsto handle the expanded entity field structurelib.rsanderror.rsfor the postgres module (filled in by BE-306)Test output updates:
.jsonc/.stdout/.stderrfixtures across eval (graph/read), HIR (lower/checking, specialization, graph-hoisting), and MIR (reify, inline) test suites, all cascading from the entity type changesPre-Merge Checklist 🚀
🚢 Has this modified a publishable library?
This PR:
📜 Does this require a change to the docs?
The changes in this PR:
🕸️ Does this require a change to the Turbo Graph?
The changes in this PR:
The entity type model still does not cover every field in the database schema (e.g.,
property_metadatainner structure is opaque). Fields are added as needed by the postgres compiler; remaining gaps will be filled incrementally.🛡 What tests cover this?
IdSnapshotVechas doc tests and inline unit tests covering push/set/rollback/nested snapshotsFieldAccess,JsonAgg,Null,U128,IsNotFalse,UNNESTwith multiple args, cast types, andOFFSET❓ How to test this?