Skip to content

fix(ckpt): open loop device in direct-io mode - #2523

Open
qiwu575757 wants to merge 1 commit into
alibaba:mainfrom
qiwu575757:fix/ckpt/loop-direct-io
Open

fix(ckpt): open loop device in direct-io mode#2523
qiwu575757 wants to merge 1 commit into
alibaba:mainfrom
qiwu575757:fix/ckpt/loop-direct-io

Conversation

@qiwu575757

Copy link
Copy Markdown
Collaborator

The btrfs-loop backend attached its image with losetup --find --show, which leaves the loop device in buffered mode (DIO=0). Any O_DIRECT issued by btrfs on top of the loop is then silently downgraded to buffered IO against the host image file: data crosses two page caches (the loop filesystem's and the host ext4's) and every request is serialized through the single loop kernel thread. The loop backend therefore had no end-to-end O_DIRECT at all, and its data path became the bottleneck under concurrent IO.

Measured on 5.10.134-19.6.3, ws-ckpt 0.4.2, 40 GiB image on a 100 GiB ESSD PL1 host, comparing the same workloads before and after:

checkpoint under fio pressure 305.10 ms -> 62.55 ms (5.3x -> 1.08x
of btrfs-base; p50 42 vs 41 ms and
p99 237 vs 237 ms now match base)
checkpoint under dual pressure 327.98 ms -> 139.75 ms
10h soak throughput 705 -> 1355 cycles/h
soak checkpoint / rollback 291.07 -> 120.58 ms / 332.13 -> 107.84 ms

Add --direct-io=on to all four losetup --find --show call sites, so bootstrap, post-rename remount, legacy rollback reattach and image-grow reattach all open the device in direct mode. No plain attach remains in this file, so a fallback path cannot silently reintroduce buffered mode.

Validated on the same host with the full ws-ckpt suite after the change: 421075 operations, zero non-zero exits and zero SHA256 integrity mismatches, covering daemon kill -9 recovery, host filesystem exhaustion, image size cap, 4-process and 4-workspace concurrency, a 10h soak and 32k snapshot accumulation.

@qiwu575757
qiwu575757 requested a review from Ziqi002 as a code owner August 14, 2026 02:28
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.


Qi Wu seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

@github-actions github-actions Bot added the component:ckpt src/ws-ckpt label Aug 14, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1dafa3672d

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/ws-ckpt/src/crates/daemon/src/backends/btrfs_loop.rs Outdated
Comment thread src/ws-ckpt/src/crates/daemon/src/backends/btrfs_loop.rs Outdated
@qiwu575757
qiwu575757 force-pushed the fix/ckpt/loop-direct-io branch from 1dafa36 to 3025db0 Compare August 14, 2026 08:30
Comment thread src/ws-ckpt/src/crates/daemon/src/backends/btrfs_loop.rs Outdated
The btrfs-loop backend attached its image with `losetup --find --show`,
which leaves the loop device in buffered mode (DIO=0). Any O_DIRECT
issued by btrfs on top of the loop is then silently downgraded to
buffered IO against the host image file: data crosses two page caches
(the loop filesystem's and the host ext4's) and every request is
serialized through the single loop kernel thread. The loop backend
therefore had no end-to-end O_DIRECT at all, and its data path became
the bottleneck under concurrent IO.

Measured on 5.10.134-19.6.3, ws-ckpt 0.4.2, 40 GiB image on a 100 GiB
ESSD PL1 host, comparing the same workloads before and after:

  checkpoint under fio pressure   305.10 ms -> 62.55 ms  (5.3x -> 1.08x
                                  of btrfs-base; p50 42 vs 41 ms and
                                  p99 237 vs 237 ms now match base)
  checkpoint under dual pressure  327.98 ms -> 139.75 ms
  10h soak throughput             705 -> 1355 cycles/h
  soak checkpoint / rollback      291.07 -> 120.58 ms / 332.13 -> 107.84 ms

Add `--direct-io=on` to all four `losetup --find --show` call sites, so
bootstrap, post-rename remount, legacy rollback reattach and image-grow
reattach all open the device in direct mode. All direct-IO handling is
best-effort: when the kernel or tooling rejects it, the daemon logs a
warning and keeps running in buffered mode exactly as before, never
blocking bootstrap. Loops already attached in buffered mode are not
switched to direct-IO on daemon restart, to keep overall system
performance stable; they converge on the next reattach or host reboot.

Validated on the same host with the full ws-ckpt suite after the change:
421075 operations, zero non-zero exits and zero SHA256 integrity
mismatches, covering daemon kill -9 recovery, host filesystem
exhaustion, image size cap, 4-process and 4-workspace concurrency, a
10h soak and 32k snapshot accumulation.

Assisted-by: Qoder
Signed-off-by: Qi Wu <shijing.wq@alibaba-inc.com>
@qiwu575757
qiwu575757 force-pushed the fix/ckpt/loop-direct-io branch from 3025db0 to 83569e0 Compare August 17, 2026 03:06

@yummypeng yummypeng left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component:ckpt src/ws-ckpt

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants