Add flat LightExpression model with direct FE builder API, fill remaining LE conversion gaps, add LE vs FE creation benchmark, and start deterministic FE/LE property testing by Copilot · Pull Request #521 · dadhi/FastExpressionCompiler

Copilot · 2026-04-20T06:00:02Z

This change introduces a flat, intrusive-array expression representation for LightExpression-oriented workflows, with round-trip conversion back to System.Linq.Expressions and LightExpression. It also closes the remaining ToLightExpression gaps so the flat model can cover the expression surface area LE already supports today, adds a direct FE factory surface so flat expressions can be built without going through LE first, adds a focused benchmark to compare LightExpression and FlatExpression creation costs on the same moderate real-world shape, and starts deterministic property-style coverage for FE/LE structural round-trips on a bounded generated subset.

Flat expression model
- Added FastExpressionCompiler.FlatExpression.ExprTree and ExprNode
- Stores expressions as a compact node array plus a separate closure-constant array
- Preserves identity-sensitive nodes such as parameters and labels across round-trip reconstruction
Direct FE builder API
- Added LightExpression-like factory methods directly on ExprTree
- Supports direct FE construction for the currently covered node shapes, including:
  - parameters / variables / constants / defaults
  - unary and binary nodes
  - calls, member access, arrays, blocks, lambdas
  - member init / list init bindings
  - labels, gotos, loops, switch, try/catch/finally/fault
  - type tests, dynamic, runtime variables, debug info
- This makes FE usable as the primary build representation while LE remains the fallback / verification path for now
Conversion pipeline
- Added conversion from:
  - System.Linq.Expressions.Expression -> ExprTree
  - FastExpressionCompiler.LightExpression.Expression -> ExprTree
- Added conversion back to:
  - ExprTree -> System.Linq.Expressions.Expression
  - ExprTree -> FastExpressionCompiler.LightExpression.Expression
- Keeps compiler integration out of scope for now; this PR provides representation, direct construction, and conversion
LE compatibility coverage
- Extended ToLightExpression to handle the remaining supported LE node kinds that were still missing:
  - Dynamic
  - RuntimeVariables
  - DebugInfo
Behavioral coverage
- Added focused tests for:
  - LightExpression -> FlatExpression -> LightExpression round-trip
  - closure constant collection
  - parameter / label identity preservation
  - Dynamic, RuntimeVariables, and DebugInfo conversion paths
  - direct FE construction with the new ExprTree factory API
  - direct FE control-flow construction
Performance coverage
- Added LightExprVsFlatExpr_Create_ComplexExpr to the existing benchmarks project
- Compares relative creation-time performance and allocation between LightExpression and FlatExpression for the same moderate complex object-graph construction scenario
- Reuses a shared CreateComplexFlatExpression helper so the LE and FE benchmark shapes stay aligned
Deterministic property coverage
- Added a bounded deterministic property-style test in LightExpressionTests
- Uses CsCheck 4.6.2 on net8.0/net9.0 to drive fixed-seed sampling
- Generates valid matching FE and LE expression trees for a core recursive subset with explicit depth and breadth limits
- Converts FE back to LE and compares the result structurally against the directly built LE tree
- Keeps the generator local and validity-driven by construction instead of depending on a broader unmerged comparer port from master

Example:

using FastExpressionCompiler.FlatExpression;

var fe = default(ExprTree);
var x = fe.Parameter(typeof(int), "x");
var body = fe.Add(x, fe.ConstantInt(1));
fe.RootIndex = fe.Lambda(typeof(Func<int, int>), body, x);

var sysExpr = fe.ToExpression();
var lightExpr = fe.ToLightExpression();

Known follow-ups
- Metadata is still stored as runtime objects (Type, MemberInfo, MethodInfo, binders), so this is not yet a fully serialization-ready wire format
- FromLightExpression currently flattens via System.Linq.Expressions rather than directly from LE nodes
- the direct builder currently uses leaf cloning when linking reused child references in the intrusive sibling-chain layout
- no compiler integration, ToCSharpString, or ToExprString changes in this PR
- benchmark coverage is currently fixed-shape and stays in the existing BenchmarkDotNet harness
- current deterministic property coverage is intentionally limited to a valid generated subset and local structural comparison; broader fuzzy/invariant coverage can expand from there

Agent-Logs-Url: https://github.com/dadhi/FastExpressionCompiler/sessions/631bfa1a-7ed7-4397-b2cb-ab11dfbcbdee Co-authored-by: dadhi <39516+dadhi@users.noreply.github.com>

dadhi · 2026-04-20T06:38:58Z

@copilot The conversation from/to LE is to be able to test correctness etc. and to have a fallback for now.

The goal is to have FE as a primary build and compiled representation and superseef the LE with it.

So, did you created the factory methods for building FE directly with the surface api matching LE? If not please do, because next next step would be integrating with compiler without too many '#if defs'.

Agent-Logs-Url: https://github.com/dadhi/FastExpressionCompiler/sessions/15dd11ed-8bf0-4fd4-b514-3d2ff4e4e346 Co-authored-by: dadhi <39516+dadhi@users.noreply.github.com>

Copilot · 2026-04-20T06:56:14Z

@copilot The conversation from/to LE is to be able to test correctness etc. and to have a fallback for now.

The goal is to have FE as a primary build and compiled representation and superseef the L...

Added the direct FE builder surface on ExprTree with LE-like factory methods, plus focused tests for direct construction and control-flow cases. It now supports building FE without going through LE first, which should make the next compiler-integration step much cleaner. Addressed in 150bdd6.

dadhi

Ci build is failing because of missing xml doc comments. Please add a minimal summary-focused comment explaining what and why. Be technical and point, avoid windy ramblings.

Agent-Logs-Url: https://github.com/dadhi/FastExpressionCompiler/sessions/0ccd5cdb-3248-46a4-b6df-54762eed9357 Co-authored-by: dadhi <39516+dadhi@users.noreply.github.com>

Copilot · 2026-04-20T07:54:23Z

Ci build is failing because of missing xml doc comments. Please add a minimal summary-focused comment explaining what and why. Be technical and point, avoid windy ramblings.

Added concise XML summaries to the new public flat-expression API and fixed the net472 test project reference needed by the dynamic binder test so the release build can complete. Addressed in cc19f39.

dadhi · 2026-04-20T08:34:15Z

+        if (expressions == null || expressions.Length == 0)
+            throw new ArgumentException("Block should contain at least one expression.", nameof(expressions));
+
+        var children = new List<int>();


Avoid using list for creating a transient storage - use SmallListN or SmallMapN instead.
Goal is to make FE creation as fast and as gc free as possible. Apply in other places.
Look at LightExpression construction where I try to minimiza gc pressure.
We may use params ReadOnlySpan in .net 10 to minimize it further.
Avoid copying the big structs and using interim steuct data wrappers - prefer the diect style of manipulating the ExprNode. Users will know that the goal an extreme performance - so the responsibility is on their side. We may hrlp with Asserts, or making a right thing by construction but perf is the altimate goal.

Reworked the transient child storage in FlatExpression to use stack-backed SmallList instead of List<int> across the direct builder, flattening builder, child cloning, and reader paths, and switched node creation to write through AddDefaultAndGetRef to avoid the extra ExprNode copy. Verified with dotnet build src/FastExpressionCompiler.LightExpression/FastExpressionCompiler.LightExpression.csproj -c Release and the net9 test runner. Addressed in 7958d82. No UI changes.

Agent-Logs-Url: https://github.com/dadhi/FastExpressionCompiler/sessions/a101d2c7-fa90-45b2-b621-f056a99693eb Co-authored-by: dadhi <39516+dadhi@users.noreply.github.com>

dadhi · 2026-04-20T09:15:03Z

+        private static System.Linq.Expressions.NewExpression CreateValueTypeNewExpression(Type type) => SysExpr.New(type);
+    }
+
+    private sealed class ParameterData


We should replace the class ...Data objects with storing the fields directly in the ExprNode.
Storing in the GC collected classes defeats the goal of FlatExpression - to be flat, cache-friendly, no chasing pointers, list of nodes.
For instance, for Parameter we can store Name in Obj; and the rest in the _data, same for label.
If the Expression type requires more than 1 object reference (optional comparison method) it should be stored in the child sub nodes.
Please adopt and remove all the Data classes below.

Replaced the remaining ...Data wrapper payloads by storing scalar payload directly on ExprNode, moving single-reference payloads into Obj, and encoding the extra Dynamic reference as an internal child metadata node. Verified with the full release build and the net9 test runner. Addressed in 7f3a5b6. No UI changes.

dadhi · 2026-04-20T11:08:13Z

+    }
+
+    internal void SetNextIdx(int nextIdx) =>
+        _data = (_data & ~(IndexMask << NextShift)) | ((ulong)(ushort)nextIdx << NextShift);


~(IndexMask << NextShift) and ~((IndexMask << CountShift) | IndexMask)
Can we just introduce derived nicely named constants for them and compress those calculations for SetNextIdx and SetChildInfo?

Introduced named derived masks for the packed next/child bits and simplified SetNextIdx / SetChildInfo to use them directly. Addressed in 88fc048. No UI changes.

dadhi · 2026-04-20T11:12:06Z

+    private int AddFactoryExpressionNode(Type type, object obj, ExpressionType nodeType, byte flags, int child0, int child1) =>
+        AddNode(type, obj, nodeType, ExprNodeKind.Expression, flags, CloneChild(child0), CloneChild(child1));
+
+    private int AddFactoryExpressionNode(Type type, object obj, ExpressionType nodeType, byte flags, int child0, int child1, int child2) =>


Let support up to 7 arguments to match LightExpression, plus minify the naming to avoid long winded signature - it is already clear that we are working with children/arguments/expressions/vairables/etc.

Extended the compact fixed-arity expression helper path up to 7 children and shortened the helper child naming so the small-arity cases stay direct and closer to the LE style. Addressed in 88fc048. No UI changes.

dadhi · 2026-04-20T11:16:58Z

+    // Any leaf reused in more than one parent would have its intrusive sibling link (`NextIdx`) overwritten.
+    // Clone the leaf before linking so the direct builder API may safely reuse returned indexes for
+    // parameters, labels, constants, default values and parameterless `new` expressions.
+    private static bool ShouldCloneWhenLinking(in ExprNode node) =>


Those checks that only operate on the one node state, can we move them into the Node struct itself and simplify? Do for the similar other situations as well for consistency.

Moved the node-local predicates into ExprNode itself (Is, IsExpression, HasFlag, ShouldCloneWhenLinked) and updated the surrounding code to use them consistently. Addressed in 88fc048. No UI changes.

dadhi · 2026-04-20T11:19:48Z

+        return nodeIndex;
+    }
+
+    private static bool ShouldInlineConstant(object value, Type type)


Minify the check code into a single condition, avoid loc creep. Do the same for similar siutations/check.
Apply AggressiveInlining for such methods where it make sense.

Compacted the small predicate/check helpers and added AggressiveInlining where it fits the hot tiny methods involved in the packed-node path. Addressed in 88fc048. No UI changes.

Agent-Logs-Url: https://github.com/dadhi/FastExpressionCompiler/sessions/d56dd094-3b5e-41e0-a3c3-80c72ae88fdd Co-authored-by: dadhi <39516+dadhi@users.noreply.github.com>

Copilot · 2026-04-20T11:27:48Z

@copilot The conversation from/to LE is to be able to test correctness etc. and to have a fallback for now.

The goal is to have FE as a primary build and compiled representation and superseef the L...

Yes — the direct FE factory surface matching the LE-style builder API is already in the branch on ExprTree. It was introduced in 150bdd6 and is still present in 88fc048. No UI changes.

dadhi

Cool, next step I want a performance test to compare relative performance and memory allocation taking by creation of moderate real-world example, between LightExpression and FlatExpression. I will suggest to use latest CsCheck library (v4.6.2). - Later we may utilize for the expression fuzzy- and invariant-testing

Agent-Logs-Url: https://github.com/dadhi/FastExpressionCompiler/sessions/ed01d5a9-8337-4a93-a276-4c52a481fc65 Co-authored-by: dadhi <39516+dadhi@users.noreply.github.com>

Copilot · 2026-04-20T14:09:53Z

Cool, next step I want a performance test to compare relative performance and memory allocation taking by creation of moderate real-world example, between LightExpression and FlatExpression. I will su...

Added LightExprVsFlatExpr_Create_ComplexExpr in the existing benchmarks project, using the same moderate complex-object graph shape for both builders via a new CreateComplexFlatExpression helper, so creation time and allocation can be compared directly between LE and FE. I kept this one in the existing BenchmarkDotNet harness; I did not add CsCheck yet because this benchmark is fixed-shape, but it should fit well once we start fuzzy/invariant testing. Addressed in f4bd9e9. No UI changes.

dadhi

Let start the deterministic property testing to at least provide some guarantees for our very broad api surface. I have an idea to test first - build/genetate arbitraty expression with specified limits on tje depth and breadth via CsCheck, is ensure its validity by black listing some expressions in some contexts. Ley gen the same FE and LE expression, then convert FE to LE and compare them structurally - the structural comparison for LE and SE was just merged into master. If you cannot find how to that CsCheck let produce our own valid expression generator (using the ssme perf/simplicity principmes as the rest of the Fec).

Agent-Logs-Url: https://github.com/dadhi/FastExpressionCompiler/sessions/b03ffb96-068c-4255-bd58-3ca2f3ef1298 Co-authored-by: dadhi <39516+dadhi@users.noreply.github.com>

Copilot · 2026-04-20T19:30:15Z

Let start the deterministic property testing to at least provide some guarantees for our very broad api surface. I have an idea to test first - build/genetate arbitraty expression with specified limit...

Added the first deterministic property-style coverage pass in LightExpressionTests: CsCheck v4.6.2 drives fixed-seed sampling on net8/net9, a bounded in-repo generator builds valid matching FE and LE trees for a core recursive subset, FE is converted back to LE, and the result is checked with a local structural comparer for that generated subset. I kept the generator local because it lets us enforce validity by construction and stay within the current branch without pulling the larger master structural-comparer feature set. Addressed in 46072b59. No UI changes.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

www.nuget.org
- Triggering command: /home/REDACTED/work/_temp/ghcca-node/node/bin/node /home/REDACTED/work/_temp/ghcca-node/node/bin/node --enable-source-maps /home/REDACTED/work/_temp/copilot-developer-action-main/dist/index.js (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

dadhi

Add FlatExpression.cs to the src nudpecs.
Add tests to both .net472 the same as to .net test runner.
Ensure that it is included into the packages the same way as LightExpression.
It won't be integrated the same way in Fec and tests yet until we do this integration in the separate PR.

Copilot · 2026-04-21T20:35:51Z

@dadhi I've opened a new pull request, #528, to work on those changes. Once the pull request is ready, I'll request review from you.

Initial plan

9a87d4a

Copilot AI assigned Copilot and dadhi Apr 20, 2026

Copilot started work on behalf of dadhi April 20, 2026 06:00 View session

Copilot AI linked an issue Apr 20, 2026 that may be closed by this pull request

Data-oriented representation of Expression partly on stack and optimized for constant collection #463

Closed

Copilot AI and others added 2 commits April 20, 2026 06:13

feat: add flat light expression roundtrip support

bf135d2

Agent-Logs-Url: https://github.com/dadhi/FastExpressionCompiler/sessions/631bfa1a-7ed7-4397-b2cb-ab11dfbcbdee Co-authored-by: dadhi <39516+dadhi@users.noreply.github.com>

fix: finalize flat expression roundtrip review feedback

330fc9e

Agent-Logs-Url: https://github.com/dadhi/FastExpressionCompiler/sessions/631bfa1a-7ed7-4397-b2cb-ab11dfbcbdee Co-authored-by: dadhi <39516+dadhi@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Optimize data-oriented representation of Expression for serialization~~ Add flat LightExpression round-trip model and fill remaining LE conversion gaps Apr 20, 2026

Copilot finished work on behalf of dadhi April 20, 2026 06:19

Copilot AI requested a review from dadhi April 20, 2026 06:19

Copilot started work on behalf of dadhi April 20, 2026 06:39 View session

Copilot AI and others added 2 commits April 20, 2026 06:50

feat: add direct flat expression builder api

b6a8952

Agent-Logs-Url: https://github.com/dadhi/FastExpressionCompiler/sessions/15dd11ed-8bf0-4fd4-b514-3d2ff4e4e346 Co-authored-by: dadhi <39516+dadhi@users.noreply.github.com>

fix: finalize flat expression direct builder review notes

150bdd6

Agent-Logs-Url: https://github.com/dadhi/FastExpressionCompiler/sessions/15dd11ed-8bf0-4fd4-b514-3d2ff4e4e346 Co-authored-by: dadhi <39516+dadhi@users.noreply.github.com>

Copilot AI changed the title ~~Add flat LightExpression round-trip model and fill remaining LE conversion gaps~~ Add flat LightExpression model with direct FE builder API and fill remaining LE conversion gaps Apr 20, 2026

Copilot finished work on behalf of dadhi April 20, 2026 06:58

dadhi requested changes Apr 20, 2026

View reviewed changes

Copilot started work on behalf of dadhi April 20, 2026 07:45 View session

fix: add xml docs for flat expression api

cc19f39

Agent-Logs-Url: https://github.com/dadhi/FastExpressionCompiler/sessions/0ccd5cdb-3248-46a4-b6df-54762eed9357 Co-authored-by: dadhi <39516+dadhi@users.noreply.github.com>

Copilot AI requested a review from dadhi April 20, 2026 07:55

Copilot finished work on behalf of dadhi April 20, 2026 07:55

dadhi requested changes Apr 20, 2026

View reviewed changes

Copilot started work on behalf of dadhi April 20, 2026 08:35 View session

perf: reduce flat expression builder allocations

7958d82

Agent-Logs-Url: https://github.com/dadhi/FastExpressionCompiler/sessions/a101d2c7-fa90-45b2-b621-f056a99693eb Co-authored-by: dadhi <39516+dadhi@users.noreply.github.com>

Copilot finished work on behalf of dadhi April 20, 2026 08:54

Copilot AI requested a review from dadhi April 20, 2026 08:54

dadhi requested changes Apr 20, 2026

View reviewed changes

Copilot started work on behalf of dadhi April 20, 2026 09:15 View session

Copilot finished work on behalf of dadhi April 20, 2026 10:35

Copilot AI requested a review from dadhi April 20, 2026 10:35

dadhi requested changes Apr 20, 2026

View reviewed changes

Copilot started work on behalf of dadhi April 20, 2026 11:20 View session

perf: tighten flat expression node helpers

88fc048

Agent-Logs-Url: https://github.com/dadhi/FastExpressionCompiler/sessions/d56dd094-3b5e-41e0-a3c3-80c72ae88fdd Co-authored-by: dadhi <39516+dadhi@users.noreply.github.com>

Copilot finished work on behalf of dadhi April 20, 2026 11:29

Copilot AI requested a review from dadhi April 20, 2026 11:29

dadhi requested changes Apr 20, 2026

View reviewed changes

Copilot started work on behalf of dadhi April 20, 2026 13:56 View session

Copilot AI and others added 2 commits April 20, 2026 14:02

test: add light vs flat creation benchmark

0b9fa7e

Agent-Logs-Url: https://github.com/dadhi/FastExpressionCompiler/sessions/ed01d5a9-8337-4a93-a276-4c52a481fc65 Co-authored-by: dadhi <39516+dadhi@users.noreply.github.com>

test: polish flat creation benchmark helper

f4bd9e9

Agent-Logs-Url: https://github.com/dadhi/FastExpressionCompiler/sessions/ed01d5a9-8337-4a93-a276-4c52a481fc65 Co-authored-by: dadhi <39516+dadhi@users.noreply.github.com>

Copilot AI changed the title ~~Add flat LightExpression model with direct FE builder API and fill remaining LE conversion gaps~~ Add flat LightExpression model with direct FE builder API, fill remaining LE conversion gaps, and add LE vs FE creation benchmark Apr 20, 2026

Copilot finished work on behalf of dadhi April 20, 2026 14:11

Copilot AI requested a review from dadhi April 20, 2026 14:11

dadhi requested changes Apr 20, 2026

View reviewed changes

Copilot started work on behalf of dadhi April 20, 2026 19:08 View session

Copilot AI and others added 2 commits April 20, 2026 19:27

test: add deterministic flat expression property checks

1ebd6e2

Agent-Logs-Url: https://github.com/dadhi/FastExpressionCompiler/sessions/b03ffb96-068c-4255-bd58-3ca2f3ef1298 Co-authored-by: dadhi <39516+dadhi@users.noreply.github.com>

test: polish generated property test seeds

46072b5

Agent-Logs-Url: https://github.com/dadhi/FastExpressionCompiler/sessions/b03ffb96-068c-4255-bd58-3ca2f3ef1298 Co-authored-by: dadhi <39516+dadhi@users.noreply.github.com>

Copilot finished work on behalf of dadhi April 20, 2026 19:31

Copilot AI requested a review from dadhi April 20, 2026 19:31

dadhi requested changes Apr 21, 2026

View reviewed changes

Copilot AI mentioned this pull request Apr 21, 2026

Include FlatExpression.cs in src nuget packages and Internal build script #528

Merged

dadhi marked this pull request as ready for review April 21, 2026 20:57

dadhi merged commit 548779a into master Apr 21, 2026
2 checks passed

dadhi deleted the copilot/optimize-expression-representation branch April 21, 2026 20:57

Uh oh!

Conversation

Copilot AI commented Apr 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dadhi commented Apr 20, 2026

Uh oh!

Copilot AI commented Apr 20, 2026

Uh oh!

dadhi left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI commented Apr 20, 2026

Uh oh!

dadhi Apr 20, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 20, 2026

Choose a reason for hiding this comment

Uh oh!

dadhi Apr 20, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 20, 2026

Choose a reason for hiding this comment

Uh oh!

dadhi Apr 20, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 20, 2026

Choose a reason for hiding this comment

Uh oh!

dadhi Apr 20, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 20, 2026

Choose a reason for hiding this comment

Uh oh!

dadhi Apr 20, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 20, 2026

Choose a reason for hiding this comment

Uh oh!

dadhi Apr 20, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 20, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI commented Apr 20, 2026

Uh oh!

dadhi left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI commented Apr 20, 2026

Uh oh!

dadhi left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI commented Apr 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

dadhi left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI commented Apr 21, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Copilot AI commented Apr 20, 2026 •

edited

Loading

Copilot AI commented Apr 20, 2026 •

edited

Loading