Skip to content

initializing ErrorEventsThreshold - #11955

Draft
Brett Samblanet (brettsam) wants to merge 3 commits into
devfrom
brettsam/retry_count
Draft

initializing ErrorEventsThreshold#11955
Brett Samblanet (brettsam) wants to merge 3 commits into
devfrom
brettsam/retry_count

Conversation

@brettsam

@brettsam Brett Samblanet (brettsam) commented Sep 1, 2026

Copy link
Copy Markdown
Member

Resolves #11871

Pull request checklist

IMPORTANT: Currently, changes must be backported to the in-proc branch to be included in Core Tools and non-Flex deployments.

  • Backporting to the in-proc branch is not required
    • Otherwise: Backport PR: TBD
  • My changes do not require documentation changes
  • My changes should not be added to the release notes for the next release
    • Otherwise: I've added my notes to release_notes.md
  • My changes do not need to be backported to a previous version
  • My changes do not require diagnostic events changes
  • I have added all required tests (Unit tests, E2E tests)

Additional information

During worker indexing, a webhost language worker can publish a startup error before RpcFunctionInvocationDispatcher.InitializeAsync receives function metadata. The dispatcher subscribed to worker errors in its constructor, but its restart threshold was not initialized until after the empty-function early return. This left the threshold at 0, causing the first startup error to immediately exhaust the retry budget and stop the host.

This change:

  • Defaults the restart threshold to 3 and initializes it from the configured worker process count before subscribing to worker events.
  • Retains the existing post-metadata recalculation for runtimes resolved from function metadata.
  • Avoids setting up invocation buffers or sending function-load requests when a replacement worker starts before metadata is available.

The regression test covers worker errors both before dispatcher initialization and after empty metadata initialization. Before the fix, it failed with Expected: RestartWorker / Actual: StopApplication; after the fix, the configured retry budget is consumed and worker recovery is attempted. Additional coverage verifies the default threshold remains at least 3.

@brettsam
Brett Samblanet (brettsam) requested a review from a team as a code owner September 1, 2026 14:49
Copilot AI lite review requested due to automatic review settings September 1, 2026 14:49

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adjusts RpcFunctionInvocationDispatcher to ensure ErrorEventsThreshold is initialized early (with a safe default and pre-event-subscription initialization) so worker startup/indexing errors don’t immediately exhaust the retry budget and stop the host, addressing #11871.

Changes:

  • Default ErrorEventsThreshold to 3 and initialize it earlier based on configured process count when possible.
  • Recalculate the threshold after function metadata initialization using the resolved max process count.
  • Avoid setting up invocation buffers / sending load requests when starting replacement workers before metadata is available, and add regression tests for the indexing-first-error scenario.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
test/WebJobs.Script.Tests/Workers/Rpc/RpcFunctionInvocationDispatcherTests.cs Adds coverage for default threshold behavior and worker-indexing error recovery before/after empty-metadata initialization.
src/Functions.Rpc.Server/Rpc/FunctionRegistration/RpcFunctionInvocationDispatcher.cs Initializes ErrorEventsThreshold earlier and skips buffer/load setup when _functions is not yet available.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@brettsam
Brett Samblanet (brettsam) marked this pull request as draft September 1, 2026 16:24
@brettsam

Copy link
Copy Markdown
Member Author

Ok this is more complicated than I first assumed. We've never tried to restart the process in this state and it's caused all kind of issues. Converting to draft to do more investigation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Worker startup timeout during .NET isolated metadata indexing immediately exceeds retry count instead of retrying

3 participants