Skip to content

fix(envoy-client): dont abort drain after 20s #5043

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
MasterPtato wants to merge 1 commit into
base: 05-11-fix_wasm_point_wasm-pack_build_to_new_wasm-bindgen_repo
Choose a base branch
from
Draft

fix(envoy-client): dont abort drain after 20s #5043

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions engine/packages/config/src/config/pegboard.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -159,7 +159,7 @@ pub struct Pegboard {

impl Pegboard {
pub fn base_retry_timeout(&self) -> usize {
self.base_retry_timeout.unwrap_or(2000)
self.base_retry_timeout.unwrap_or(2_000)
}

pub fn actor_allocation_threshold(&self) -> i64 {
Expand All @@ -177,7 +177,7 @@ impl Pegboard {
}

pub fn actor_retry_duration_threshold(&self) -> i64 {
self.actor_retry_duration_threshold.unwrap_or(300_000)
self.actor_retry_duration_threshold.unwrap_or(5 * 60 * 1000)
}

pub fn retry_reset_duration(&self) -> i64 {
Expand All @@ -202,7 +202,7 @@ impl Pegboard {
}

pub fn serverless_base_retry_timeout(&self) -> usize {
self.serverless_base_retry_timeout.unwrap_or(2000)
self.serverless_base_retry_timeout.unwrap_or(2_000)
}

pub fn serverless_retry_reset_duration(&self) -> i64 {
Expand Down Expand Up @@ -237,7 +237,7 @@ impl Pegboard {

pub fn gateway_response_start_timeout_ms(&self) -> u64 {
self.gateway_response_start_timeout_ms
.unwrap_or(5 * 60 * 1000) // 5 minutes
.unwrap_or(5 * 60 * 1000)
}

pub fn gateway_update_ping_interval_ms(&self) -> u64 {
Expand Down
2 changes: 1 addition & 1 deletion examples/kitchen-sink/Dockerfile.local
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion examples/kitchen-sink/package.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

13 changes: 8 additions & 5 deletions examples/kitchen-sink/src/index.ts
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

25 changes: 25 additions & 0 deletions examples/kitchen-sink/src/mode.ts
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

22 changes: 12 additions & 10 deletions examples/kitchen-sink/src/server.ts
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

26 changes: 15 additions & 11 deletions rivetkit-rust/packages/rivetkit-core/src/registry/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,11 +74,6 @@
use inspector::build_actor_inspector;
use websocket::is_actor_connect_path;

/// Bound on `handle.shutdown_and_wait` inside `serve_with_config` teardown.
/// Protects against indefinite hangs if the envoy reconnect loop is stuck;
/// the TS/outer-host grace period is the ultimate backstop.
const SHUTDOWN_DRAIN_TIMEOUT: Duration = Duration::from_secs(20);

#[derive(Debug, Default)]
pub struct CoreRegistry {
factories: HashMap<String, Arc<ActorFactory>>,
Expand Down Expand Up @@ -530,10 +525,22 @@
// trip the `shutdown` token instead.
shutdown.cancelled().await;

// TODO: Move into envoy-client since timing out has to do with protocol compliance
// Read threshold from protocol metadata, fall back to 30 min
let stop_threshold = handle
.get_protocol_metadata

Check failure on line 531 in rivetkit-rust/packages/rivetkit-core/src/registry/mod.rs

View workflow job for this annotation

GitHub Actions / Check 

attempted to take value of method `get_protocol_metadata` on type `rivet_envoy_client::handle::EnvoyHandle`
.await
.map(|x| x.actor_stop_threshold)
.unwrap_or(30 * 60 * 1000);
// Bounded drain. If envoy cannot reach the engine (reconnect loop stuck),
// we fall back to immediate `Stop` rather than hanging indefinitely.
// The outer host (TS signal handler / Rust binary) is the backstop.
match timeout(SHUTDOWN_DRAIN_TIMEOUT, handle.shutdown_and_wait(false)).await {
match timeout(
Duration::from_millis(SHUTDOWN_DRAIN_TIMEOUT as u64),

Check failure on line 539 in rivetkit-rust/packages/rivetkit-core/src/registry/mod.rs

View workflow job for this annotation

GitHub Actions / Check 

cannot find value `SHUTDOWN_DRAIN_TIMEOUT` in this scope
handle.shutdown_and_wait(false),
)
.await
{
Ok(()) => {}
Err(_) => {
tracing::warn!("envoy shutdown drain exceeded timeout; forcing immediate stop");
Expand Down Expand Up @@ -718,11 +725,8 @@

let (start_tx, start_rx) = oneshot::channel();
let result: Result<Arc<ActorTaskHandle>> = async {
try_send_lifecycle_command(
&lifecycle_tx,
LifecycleCommand::Start { reply: start_tx },
)
.context("send actor task start command")?;
try_send_lifecycle_command(&lifecycle_tx, LifecycleCommand::Start { reply: start_tx })
.context("send actor task start command")?;
start_rx
.await
.context("receive actor task start reply")?
Expand Down
2 changes: 1 addition & 1 deletion rivetkit-typescript/packages/rivetkit-napi/index.d.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
/* tslint:disable */

Check failure on line 1 in rivetkit-typescript/packages/rivetkit-napi/index.d.ts

View workflow job for this annotation

GitHub Actions / RivetKit / Quality Check

format 

Formatter would have printed the following content:
/* eslint-disable */

/* auto-generated by NAPI-RS */
Expand Down Expand Up @@ -314,7 +314,7 @@
* Idempotent. Safe to call when neither mode has been activated.
* Does not block on the `serve()` future; TS awaits that promise
* separately to avoid re-entrancy.
*/
*/
shutdown(): Promise<void>
health(): Promise<JsRegistryRouteResponse>
metadata(): JsRegistryRouteResponse
Expand Down
Loading