Skip to content

Cleanly stop inbound streams during shutdown.#9983

Open
robholland wants to merge 6 commits intotemporalio:mainfrom
robholland:rh-inbound-stream-close
Open

Cleanly stop inbound streams during shutdown.#9983
robholland wants to merge 6 commits intotemporalio:mainfrom
robholland:rh-inbound-stream-close

Conversation

@robholland
Copy link
Copy Markdown
Contributor

@robholland robholland commented Apr 17, 2026

What changed?

When gracefully shutting down, close inbound streams, not just outbound.

Why?

This avoids unexpected connection reset on the sender side.

Flag enabled (default — new behavior):

  1. Evict from membership + drain wait
  2. handler.Stop() ← stops StreamReceiverMonitor (closes inbound streams + outbound),
    replicationTaskFetcherFactory, shardController, eventNotifier
  3. server.GracefulStop() ← stream handler goroutines already unblocked → clean drain
  4. visibilityManager.Close()

Flag disabled (current behavior):

  1. Evict from membership + drain wait
  2. handler.controller.Stop() ← only stops the shard controller (inbound streams left open)
  3. server.GracefulStop() ← stream handler goroutines still blocked → falls back to Stop()
  4. handler.Stop() ← remainder of handler teardown
  5. visibilityManager.Close()

How did you test it?

  • built
  • run locally and tested manually
  • covered by existing tests
  • added new unit test(s)
  • added new functional test(s)

Note

Medium Risk
Touches history service shutdown sequencing and replication stream lifecycle, which can affect rolling restarts and cross-cluster replication behavior if the new ordering has unintended side effects.

Overview
Improves history service graceful shutdown by optionally closing inbound replication streams (in addition to outbound) so remote senders are signaled to stop and stream handler goroutines can drain cleanly.

Introduces new dynamic config history.enableCloseInboundReplicationStreamOnShutdown (default true), threads it through history Config, and uses it to (a) stop inbound streams in StreamReceiverMonitor.Stop() and (b) reorder service.go shutdown to call handler.Stop() before server.GracefulStop() when enabled; adds unit tests covering both enabled/disabled behaviors.

Reviewed by Cursor Bugbot for commit 8aab9ee. Bugbot is set up for automated code reviews on this repo. Configure here.

@robholland robholland requested review from a team as code owners April 17, 2026 18:33
Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

Reviewed by Cursor Bugbot for commit b887cf5. Configure here.

Comment thread service/history/service.go
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants