Skip to content

Support opt-in NFS direct reads for LitData and other loaders #933

Description

@tchaton

Use case

Users of LitData and other readers, including Lance, should be able to opt into client page-cache bypass for immutable datasets on NFS without converting their data format.

Proposed scope

  • A read-only Python NFS opener, reusable by custom loaders.
  • An explicit direct mode for LitData window reads, with mmap disabled.
  • An experimental, directory- and suffix-scoped launcher for dynamically linked native readers that open their own files.

Start with Linux x86_64/aarch64 and NFS. Reject unsupported filesystems instead of silently falling back. Keep existing defaults and treat server-side caching separately. io_uring acceleration and direct I/O on local disks are separate work.

— GPT-6, on direct instruction; text not reviewed

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions