feat(discovery-core): add iobudget batching, cursor types, and new cost constants

[m-kus]

TL;DR

Added cursor types and improved I/O budget functionality to support batch discovery operations.

What changed?

• Added new cursor types (DiscoveryCursor, ChannelCursor, SubchannelCursor) to track progress across paginated discovery calls
• Enhanced IoBudget with batch budget functionality and a new consume_up_to method for more efficient resource allocation
• Added cost constants for various discovery operations
• Added the futures crate dependency to support async operations

---

This change is 

[m-kus]

• feat(discovery-service): add docker image and ci workflow #487
• feat(e2e): starknet integration demo explorer #467
• feat(discovery-service): add fallback for chain state, e2e test in the starknet integration environment #466
• feat(e2e): add indexer e2e test suite with sdk devnet helpers #486
• feat(sdk): add indexer discovery provider #485
• feat(discovery-service): add preflight check endpoint #484
• feat(discovery-service): add outgoing state sync endpoint #483
• feat(discovery-service): add incoming sync endpoint with validation and error handling #482
• feat(discovery-service): centralize configuration into config module #481
• feat(discovery-core): add transfer preflight check #474
• feat(discovery-core): add outgoing state sync orchestrator #473
• feat(discovery-core): add incoming state sync orchestrator #472
• feat(discovery-core): add paginated last-note-index search with bisection #471
• feat(discovery-core): add paginated outgoing channel discovery #470
• feat(discovery-core): paginated notes discovery with nullifier filtering #462
• feat(discovery-core): paginated subchannel discovery with cursor support #461
• feat(discovery-core): paginated incoming channel discovery with cursor support #460
• feat(discovery-core): add iobudget batching, cursor types, and new cost constants #459 👈 (View in Graphite)
• fix: use patched devnet and mocked proof facts #479
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[Yoni-Starkware]

@Yoni-Starkware reviewed 4 files and all commit messages, and made 2 comments.
Reviewable status: 4 of 6 files reviewed, 2 unresolved discussions (waiting on @m-kus).

---

crates/discovery-core/src/io_budget.rs line 73 at r1 (raw file):

        if cost_per_item == 0 || max_items == 0 {
            return 0;
        }

cost_per_item=0 means you can consume as many items as you want.
If you don't want to support that, please assert that cost_per_item != 0

Code quote:

    pub fn consume_up_to(&self, max_items: usize, cost_per_item: usize) -> usize {
        if cost_per_item == 0 || max_items == 0 {
            return 0;
        }

---

crates/discovery-core/src/io_budget.rs line 86 at r1 (raw file):

                    Some(current - num_items * cost_per_item)
                }
            })

Suggestion, a bit clearer

Suggestion:

        let max_items_cap = max_items.min(self.batch_budget / cost_per_item);
        if max_items_cap == 0 {
            return 0;
        }
        self.remaining
            .fetch_update(Ordering::SeqCst, Ordering::SeqCst, |current| {
                let num_items = (current / cost_per_item).min(max_items_cap);
                let count = num_items * cost_per_item;
                current.checked_sub(count)
            })

[Yoni-Starkware]

@Yoni-Starkware resolved 1 discussion.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on @m-kus).

[Yoni-Starkware]

@Yoni-Starkware made 1 comment.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @m-kus).

---

crates/discovery-core/src/io_budget.rs line 38 at r2 (raw file):

    /// Sets the batch budget (max budget units per batch). Returns `self` for chaining.
    pub fn with_batch_budget(mut self, batch_budget: usize) -> Self {
        self.batch_budget = batch_budget;

Thinking about this again, why do we need the batch budget here?

Why not as a config of the state reader? batch_size merely limits the size of a single read request. I don't see a problem in handling bigger requests, just in chunks of that size, iteratively.

[m-kus]

@m-kus made 1 comment.
Reviewable status: 5 of 7 files reviewed, 1 unresolved discussion (waiting on @Yoni-Starkware).

---

crates/discovery-core/src/io_budget.rs line 38 at r2 (raw file):

Previously, Yoni-Starkware (Yoni) wrote…

Thinking about this again, why do we need the batch budget here?

Why not as a config of the state reader? batch_size merely limits the size of a single read request. I don't see a problem in handling bigger requests, just in chunks of that size, iteratively.

Done.

[Yoni-Starkware]

@Yoni-Starkware reviewed 2 files and all commit messages, and made 1 comment.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on m-kus).

[Yoni-Starkware]

@Yoni-Starkware made 1 comment and resolved 1 discussion.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on m-kus).

---

crates/discovery-core/src/io_budget.rs line 38 at r2 (raw file):

Previously, m-kus (Michael Zaikin) wrote…

Done.

Thanks!