Builder deposits optimisation by pawanjay176 · Pull Request #9311 · sigp/lighthouse

pawanjay176 · 2026-05-17T04:26:30Z

Issue Addressed

N/A

Proposed Changes

Adds an OnboardBuildersCache to the beacon chain to pre-verify and cache builder deposits. Caching is important in 2 places:

onboard_builders_from_pending_deposits is a fork transition function that scales with the number of pending deposits. Under worst case, the pending deposits queue can be dos'd with a number of 1eth deposits to make nodes do more work verifying it at the fork boundary. Even though the pending_deposits queue is effectively capped by the gas limit, this cache makes even a theoretic attack ineffective by doing the full verification in miliseconds instead of seconds.
Some numbers claude cooked up

Deposits	Capital	Cached	Batch verify (without cache)
10K	$26M	~35ms	~520ms
50K	$130M	~175ms	~2.6s
96K	$250M	271ms (measured)	~5s
100K	$260M	~285ms	~5.2s

Post fork, process_operations may need to verify all builder deposits in the hot path. the engine api currently allows max 8192 deposit requests to be sent for block production, so in worst case, we may need to verify 8192 signatures during block processing. The deposits we need to process are received in the payload envelope ~6 seconds into the slot. we process these deposits when a new beacon block that builds on the payload arrives ~3 seconds into the next slot. So we have a lot of time to verify these signatures before we actually need to process them

The cache is threaded to both the per_slot_processing for the first case and per_block_processing for the second case.

Additional Info

tested with the following kurtosis config:

participants_matrix:
     el:
       - el_type: geth
         el_image: ethpandaops/geth:bal-devnet-6
         el_extra_params: ["--rpc.txfeecap=0", "--rpc.gascap=0"]
     cl:
       - cl_type: lighthouse
         cl_image: lighthouse-local:latest
         cl_log_level: debug
         count: 2
       - cl_type: prysm
         cl_image: ethpandaops/prysm-beacon-chain:glamsterdam-devnet-3-deposits
         count: 2
network_params:
  gloas_fork_epoch: 1
  withdrawal_type: "0x01"
  validator_balance: 40000
  gas_limit: 5000000000
  genesis_gaslimit: 5000000000

additional_services:
  - dora
  - assertoor

assertoor_params:
  image: ethpandaops/assertoor:master
  run_stability_check: false
  run_block_proposal_check: false
  tests:
    - file: "https://raw.githubusercontent.com/ethpandaops/assertoor/refs/heads/master/playbooks/gloas-dev/builder-deposit-spam.yaml"
      config:
        batchSize: 256
        pendingBatches: 16 # 8192 deposits per block
        # totalDeposits: 96214
        totalDeposits: 262140
        skipForkActivationCheck: true

dora_params:
  image: ethpandaops/dora:master⏎

pawanjay176 · 2026-05-19T00:36:18Z

This is ready for review now.

mergify · 2026-05-19T00:53:51Z

Some required checks have failed. Could you please take a look @pawanjay176? 🙏

eserilev

looks good on the whole, just some small suggestion, a question and a few nits

eserilev · 2026-05-19T09:20:47Z

+    let decompressed = deposit_data
+        .par_iter()
+        .enumerate()
+        .map(|(index, deposit)| {
+            deposit_pubkey_signature_message(deposit, spec)
+                .map(|(public_key, signature, message)| (index, public_key, signature, message))
+        })
+        .collect::<Vec<_>>();


I think we should use one of the scoped rayon pools instead of the global pool

See below comment

eserilev · 2026-05-19T09:22:14Z

+    let mut results = vec![false; decompressed.len()];
+
+    let batch_results = decompressed
+        .par_chunks(DEPOSIT_SIGNATURE_BATCH_SIZE)


I think we should use a scoped rayon pool here as well

I think instead of using the scoped rayon pool here, which would involve threading the task executor all the way to state_processing, we can instead spawn the tasks that trigger signature verification with the rayon pool. Implemented in 50cd378

The only place where we might call rayon without a scoped pool is in process_deposit_requests when we have cache misses for the signature verification. I think that is okay.

eserilev · 2026-05-19T15:05:28Z

+    /// This can be significantly slower if there are many builder deposits
+    /// that need to be onboarded at the fork boundary. This variant should be used
+    /// for tests and other non-production paths.
+    FullVerification,


it looks like we are only running tests for GloasVerificationContext::FullVerification, would be nice to write tests for the other two variants if possible

Added more tests in de89987

…ache everywhere.

pawanjay176 · 2026-05-19T23:56:58Z

@eserilev Removed the GloasVerificationContext::SkipBuilderOnboarding variant because I think it wasn't safe.
partial_state_advance promises to return a valid state just without the roots calculated so not doing the builder onboarding there feels like violating that contract and the assumption that the advanced state doesn't need builders might be misguided with a later refactor. Ended up threading the cache everywhere which is a little ugly but I think its necessary.

mergify · 2026-05-20T00:10:03Z

Some required checks have failed. Could you please take a look @pawanjay176? 🙏

jimmygchen · 2026-05-20T08:05:50Z

 // it is `O(n * m)` where `n` is max 8192 and `m` is max 128M.
-fn is_pending_validator<E: EthSpec>(
-    state: &BeaconState<E>,
+#[instrument(skip_all, level = "debug")]


this may create a large number of spans, we probably don't need the span per validator?

…nsertion

pawanjay176 · 2026-05-22T00:01:11Z

Note to reviewer: Changed process_deposit_requests_post_gloas significantly with 70b8594 which also diverges quite a bit from the spec function to optimise it.

The observation was that with a bigger sized state.builders, inserting a new builder to the builders list was taking a full iteration of the builder list. This is because builder indices are reusable and add_builder_to_registry uses get_index_for_new_builder which iterates through the entire list to check if any index is available for reuse. With higher builder counts, this becomes significant.

We now cache all reusable indices in a first sweep before reusing anything and that dropped the time to insert with big builder count much more manageable. Again, this is highly unlikely on mainnet.

mergify · 2026-05-22T00:15:53Z

Some required checks have failed. Could you please take a look @pawanjay176? 🙏

jimmygchen · 2026-05-22T05:36:23Z

+                }
+
+                builder_deposit_keys.push(key);
+                builder_deposits.push(deposit_data);


i thought about whether its worth de-dup here, but it seems like the risk and potential impact is low?

Yeah i was considering getting rid of cache_pending_deposits post fork so this could make it easier.
I'm happy to consolidate logic in one place though.

jimmygchen · 2026-05-22T05:42:20Z

+}
+
 /// Transform a `Fulu` state into a `Gloas` state.
+#[instrument(skip_all)]


Do you see this span when you test it locally?
I think we might have to rename the advance_head span to lh_advance_head so it gets exported to tempo.

I'm happy to just drop it too. I have already tested and benchmarked it with way worse cases than we'll ever see on mainnet and this happens just once at the fork transtion. I think its fine to remove it.

jimmygchen · 2026-05-22T07:12:36Z

+            // perform the signature verification in batches.
+            // We have until the fork transition for the cache to be used, so we use the low priority pool.
+            executor.spawn_blocking_with_rayon(
+                move || cache.add_new_pending_deposits::<T::EthSpec>(&state, &spec),


Is this pretty much a no-op after the fork?

yeah it is. Hadn't considered it. can potentially only do this for pre-gloas states and then delete it post gloas

jimmygchen · 2026-05-22T07:23:47Z

+
+    for (index, builder) in state_builders.iter().enumerate() {
+        builder_index_map.insert(builder.pubkey, index as BuilderIndex);
+        if builder.withdrawable_epoch <= current_epoch && builder.balance == 0 {


if the builder tops up in the same block and its balance increases, then we could accidentally make this index reusable right? is this possible

lighthouse/consensus/state_processing/src/per_block_processing/process_operations.rs

Lines 1052 to 1058 in 70b8594

if let Some(builder_index) = builder_index {

state

.builders_mut()?

.get_mut(builder_index as usize)

.ok_or(BeaconStateError::UnknownBuilder(builder_index))?

.balance

.safe_add_assign(deposit_request.amount)?;

awesome catch!

Fixed in 656e70a and added a test as well. Really great catch. I'm going to try and upstream this to the EF tests

jimmygchen · 2026-05-22T07:40:09Z

+            // perform the signature verification in batches.
+            executor.spawn_blocking_with_rayon(
+                move || cache.cache_deposit_requests(&deposits, &spec),
+                task_executor::RayonPoolType::HighPriority,


This isn't work in the hot path, however i think its fine leaving it high prio, as we want to be ready asap in case if the payload arrive late in the slot? is this what you were thinking?

Yeah pretty much. Better to have everything verified to reduce cache misses in case of late envelopes

jimmygchen · 2026-05-22T07:53:47Z

+}
+
+/// Helper to create a harness with Fulu genesis and gloas at a later epoch.
+async fn get_fulu_harness_with_gloas_scheduled<E: EthSpec>(


nit: move this to the top near the other similar functions

jimmygchen · 2026-05-22T07:59:46Z

Looks like the failing test here revealed a bug, and the invalid deposit got added to pending deposits.

Might want to skip (continue) if the siganture is invalid here:

lighthouse/consensus/state_processing/src/per_block_processing/process_operations.rs

Line 1080 in 70b8594

if is_valid {

https://github.com/sigp/lighthouse/actions/runs/26260083391/job/77291404904?pr=9311

pawanjay176 added 11 commits May 14, 2026 18:15

Batch verify builder deposit signatures

eda918d

Refactor onboard_builders_from_pending_deposits based on new spec

ed15ac6

Add a builder onboarding cache

369eecd

Working wiring with pre-gloas cache

112b093

More threading

0aa7d7a

Clean up threading with a GloasVerificationContext

5e46d38

Add a hashmap for keeping track of added builders

18bcc24

Update onboarding cache size

826a521

Add better docs

49410fa

fmt

e2e8406

lint

9738905

pawanjay176 requested a review from michaelsproul as a code owner May 17, 2026 04:26

pawanjay176 added work-in-progress PR is a work-in-progress optimization Something to make Lighthouse run more efficiently. gloas labels May 17, 2026

pawanjay176 added 3 commits May 18, 2026 16:44

Cleanup

fd43fdc

Fix lint

44dcf5d

Call initialize_ptc in all cases and rename variant

bc1a156

pawanjay176 added ready-for-review The code is ready for review and removed work-in-progress PR is a work-in-progress labels May 19, 2026

pawanjay176 requested a review from eserilev May 19, 2026 00:34

mergify Bot added waiting-on-author The reviewer has suggested changes and awaits thier implementation. and removed ready-for-review The code is ready for review labels May 19, 2026

eserilev reviewed May 19, 2026

View reviewed changes

pawanjay176 added 4 commits May 19, 2026 14:04

Use spawn_blocking_with_rayon for batch signature verification tasks

50cd378

Review comments

16f615e

Add unit tests

33bccd0

Remove SkipBuilderOnboarding variant and thread the onboard builder c…

f89c099

…ache everywhere.

Add more tests

de89987

pawanjay176 added ready-for-review The code is ready for review and removed waiting-on-author The reviewer has suggested changes and awaits thier implementation. labels May 19, 2026

mergify Bot added waiting-on-author The reviewer has suggested changes and awaits thier implementation. and removed ready-for-review The code is ready for review labels May 20, 2026

jimmygchen reviewed May 20, 2026

View reviewed changes

Comment thread beacon_node/beacon_chain/src/beacon_chain.rs Outdated

Optimise by not iterating through the state.builders list for every i…

70b8594

…nsertion

pawanjay176 added ready-for-review The code is ready for review and removed waiting-on-author The reviewer has suggested changes and awaits thier implementation. labels May 22, 2026

mergify Bot added waiting-on-author The reviewer has suggested changes and awaits thier implementation. and removed ready-for-review The code is ready for review labels May 22, 2026

jimmygchen reviewed May 22, 2026

View reviewed changes

pawanjay176 added 3 commits May 22, 2026 16:54

Only onboard state deposits before gloas

ecbb7e7

Fix consensus bugs and add test

656e70a

Merge branch 'unstable' into builder-deposits-optimisation

31b6f3d

	if let Some(builder_index) = builder_index {
	state
	.builders_mut()?
	.get_mut(builder_index as usize)
	.ok_or(BeaconStateError::UnknownBuilder(builder_index))?
	.balance
	.safe_add_assign(deposit_request.amount)?;

Conversation

pawanjay176 commented May 17, 2026

Issue Addressed

Proposed Changes

Additional Info

Uh oh!

pawanjay176 commented May 19, 2026

Uh oh!

mergify Bot commented May 19, 2026

Uh oh!

eserilev left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

pawanjay176 commented May 19, 2026

Uh oh!

mergify Bot commented May 20, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pawanjay176 commented May 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mergify Bot commented May 22, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jimmygchen commented May 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

pawanjay176 commented May 22, 2026 •

edited

Loading