[MySQL] Fix replication stalls caused by silently dropped idle connections - #760
[MySQL] Fix replication stalls caused by silently dropped idle connections#760bean1352 wants to merge 7 commits into
Conversation
🦋 Changeset detectedLatest commit: 2a2e209 The changes in this PR will be included in the next version bump. This PR includes changesets to release 12 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
rkistner
left a comment
rkistner
left a comment
There was a problem hiding this comment.
To clarify, this specifically enables TCP keepalive, not a mysql-specific keepalive?
Is there an option on MySQL / Zongji to additionally do a mysql-level keepalive? If there is no built-in functionality for that, maybe something along the lines of a SELECT 1 query every minute?
Your work here is definitely an improvement, but the firewall silently dropping the connection due to being idle is not the only case this can happen, and TCP keepalives typically take very long to detect the issue on their own. A protocol-level keepalive with a socketTimeout or equivalent can be a much stronger check.
Yes the change was TCP The service now creates the control connection itself and passes it into I'll also open a small types PR to |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3edb4060bb
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| const responded = await new Promise<boolean>((resolve) => { | ||
| controlConnection.query('SELECT 1', (error) => resolve(!error)); | ||
| timers.setTimeout(timeout, undefined, { ref: false }).then(() => resolve(false)); |
There was a problem hiding this comment.
When Zongji is already running a table-metadata query on this control connection, the MySQL client queues this SELECT 1 behind that query. If the metadata query legitimately takes longer than five seconds—for example on an overloaded server—the timer wins even though the connection is healthy, and the code stops replication with an error. Since this repeats every minute, a slow source can enter a restart loop; the probe should use a dedicated connection or avoid applying its response timeout while another control query is in progress.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Using a separate dedicated connection would defeat the purpose of the probe. But apart from that this makes sense - need to take into account the case where the connection is already waiting on another query.
| private async keepAliveControlConnectionUntilStopped(): Promise<void> { | ||
| const interval = this.options.ctrlConnectionKeepAliveIntervalMs ?? CTRL_CONNECTION_KEEPALIVE_INTERVAL; | ||
| let idleTime = 0; | ||
| while (!this.isStopped) { | ||
| await timers.setTimeout(1_000); | ||
| idleTime += 1_000; | ||
| if (idleTime >= interval && !(this.isStopped || this.isStopping)) { | ||
| idleTime = 0; | ||
| await this.probeControlConnection(); | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
Instead of having a separate mechanism here, can this use the async keepAlive() trigger in BinLogReplicationJob?
2 participants
Summary
The PowerSync Service keeps two connections open to MySQL: one that streams changes, and a helper connection used for occasional lookups. The helper can sit idle for hours. Some firewalls quietly close connections that have been idle for an hour. The next time the PowerSync Service tries to use that closed connection, it waits ~15 minutes for the operating system to give up. During that wait replication is completely frozen, nothing is logged, and the PowerSync Service still reports itself as healthy. Shutdown has the same problem, because it sends one last query over that same connection and waits for the answer with no time limit. A self-hosted customer hit this issue five times in two days and traced it to their firewall's one hour idle timeout.
Fix
The MySQL connections now send a
keepalivesignal every 40 seconds. That is enough for a firewall to see the connection as active, so it never gets closed in the first place. Shutdown also gets a 5 second limit: if the final query gets no answer in that time, the service closes the connection itself instead of waiting.AI disclaimer
I developed this change using Claude Opus 4.8 for the investigation and Claude Fable 5 for the implementation. I reviewed and tested the changes myself.