Skip to content

bug(CL): SIGTERM after the store opens and before the handler is installed still forces a full redb repair #473

Description

@crazywriter1

Summary

#467 moved open_store to after the execution engine is reachable, which closes the EL-retry window in #466. On current main (6e76402) the store is still opened before install_sigterm_handler runs.

A SIGTERM in that gap kills the process with the redb store open and without the quick-repair commit that savepoint writes. The next start then does a full repair. #466 measured that as 1h04 on a 103 GB testnet store. This report is the same failure mode on the window #467 left open. I have not re-run that store-size measurement.

This is not #360. That issue is the race after the handler is already installed (the handler task is aborted when the runtime drops). Here no handler is installed yet.

What the code does

In crates/malachite-app/src/node.rs, App::start opens the store and then awaits consensus-engine startup:

let (store, store_monitor) = self
    .open_store(db_metrics, env_config.db_cache_size)
    .await?;
// ...
let (channels, engine_handle) = self.start_consensus_engine(ctx, identity).await?;

start_consensus_engine awaits validator-proof creation and malachitebft_app_channel::start_engine.

Node::run installs the handler only after start() returns:

let mut handles = match self.start().await { /* ... */ };
install_sigterm_handler(&handles);

install_sigterm_handler is what calls store.savepoint() before exit. An unhandled SIGTERM does not run Drop, so that commit never happens. An orderly Err return from start() still drops the store and is fine. Only the signal in this window is the problem.

#467's commit message already calls this out: the patch is minimal, and a proper fix installs the signal handler before the store is opened.

Expected

SIGTERM at any point after open_store should take the same path as the installed handler: stop, savepoint, then exit. The next start should log Database opened, not Database repair in progress.

Proposed fix

Keep the change in node.rs. Install a SIGTERM handler that can savepoint before open_store, and keep it in place through start_consensus_engine. No consensus or Engine API behavior change.

Activity

  1. crazywriter1 commented on Sep 28, 2026

    @crazywriter1
    ContributorAuthor

    Requesting assignment to @crazywriter1 for this issue, per CONTRIBUTING.md.

    The change stays in crates/malachite-app/src/node.rs: install a SIGTERM handler that can savepoint before open_store, and keep it through start_consensus_engine. No consensus or Engine API behavior change.

    Could a maintainer please assign this issue to me? I will open the PR referencing Closes: #473 once assigned.

  2. romac commented on Oct 5, 2026

    @romac
    Contributor

    Thank you very much for the report! We are now tracking this internally. You can expect a fix in the next few releases.

  3. added
    tracked internallyThis issue is already tracked internally by the Arc team.
    and removed on Oct 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    tracked internallyThis issue is already tracked internally by the Arc team.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions