Skip to content

Add opt-in FilesystemStorageFactory::with_lazy_open (open files on first access) - #629

Open
ikaradimas wants to merge 1 commit into
ikatson:mainfrom
ikaradimas:lazy-open
Open

Add opt-in FilesystemStorageFactory::with_lazy_open (open files on first access)#629
ikaradimas wants to merge 1 commit into
ikatson:mainfrom
ikaradimas:lazy-open

Conversation

@ikaradimas

Copy link
Copy Markdown

Problem

FilesystemStorage::init opens every file of a torrent up front, and init runs on the add path (via create_and_init in Session::add_torrent) — so restoring a session opens every file of every torrent before the session is usable. For a large, many-file library that is thousands of open() calls: on one user's Windows session, ~87s to restore 41 torrents even with all data already complete and no hashing (see also the trust_fastresume PR). Files are only actually read/written on demand during transfer, so opening them all eagerly is wasted work at the worst possible time.

Change

Add an opt-in FilesystemStorageFactory::with_lazy_open(bool) (default false, so existing behavior is unchanged). When enabled, init records each file's path instead of opening it; the backing file is opened on first pread/pwrite/ensure_file_length (guarded so it happens once). In addition, ensure_file_length becomes stat-first: a file already at the target length is neither opened nor resized. A restored complete torrent therefore opens none of its files at startup — they open on demand.

MmapFilesystemStorage builds its inner FilesystemStorage via FilesystemStorageFactory::default() and needs the files open after init to mmap them, so it keeps eager opening (flag off) and is unaffected. The read and write paths are unchanged apart from the lazy-open guard.

Testing

Unit tests in storage/filesystem/fs.rs cover lazy read of an existing file (opens only on first access, returns correct bytes), stat-first ensure_file_length (no open when already the right length), and lazy write (creates the file and parent dir). With the flag false (default), behavior is unchanged.

…pen files on first access)

FilesystemStorage::init opens every file of a torrent up front, and init runs on
the add path (via create_and_init in Session::add_torrent) - so restoring a
session opens every file of every torrent before the session is usable. For a
large, many-file library that is thousands of open() calls (~87s to restore 41
complete torrents on Windows in one report), even when all data is already
present and no hashing is needed.

Add an opt-in FilesystemStorageFactory::with_lazy_open (default false). When
enabled, init records each file's path instead of opening it; the backing file
is opened on first pread/pwrite/ensure_file_length (guarded so it happens once).
ensure_file_length is also stat-first: a file already at the target length is
neither opened nor resized. A restored complete torrent therefore opens none of
its files at startup - they open on demand during transfer.

MmapFilesystemStorage builds its inner FilesystemStorage via
FilesystemStorageFactory::default() and needs the files open after init to mmap
them, so it keeps eager opening (flag off) and is unaffected. The read/write
paths are unchanged apart from the lazy-open guard.

Unit tests cover lazy read of an existing file, stat-first ensure_file_length,
and lazy write (creates file + parent dir).
@ikatson

ikatson commented Aug 15, 2026

Copy link
Copy Markdown
Owner

Will this do anything at all without your other "trust_fastresume" change?

As each file either needs to be checked (at least to a degree, with fastresume), or written to, this would just move the opening to a later stage (during download).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants