Edit files in-place with open modes "e" and "eb" (and enable "a" and "ab" too)

[lhoestq]

Implements HfFileSystemEditFile which is available via hffs.open("hf://buckets/..", "eb") and its methods:

• edit((start, end), data)
• insert(loc, data)
• append(data)
• delete(loc, length)
• truncate(size=None)

This also enables append mode "a" and "ab".

This only uploads the new data (and neighboring chunks), which enables efficient edits even on big files.

Close #4153
Related to huggingface/huggingface.js#2407
Follows huggingface/xet-core#717

Still in draft mode until huggingface/xet-core#951 is merged, but feel free to review already !

PS: Currently editing a big files still requires downloading ~64MB of data from neighboring chunks to generate verification and range hashes. Hopefully this limitation can be removed soon

Feedback on the mode name

I was thinking of naming it "mutate" for "m" and "mb" but ended up with "edit" for "e" and "eb"

Implementation details

It uses a buffer to keep the edits in memory and send them once the buffer is big enough and enough time has passed. Once ready, the buffer is sent to the Hub in a background thread.

The buffer uses bytearray to be able to efficiently support fast small appends.

I had to implement HfApi.edit_bucket_file() which calls hf_xet.

Try it yourself

Requires hf_xet from huggingface/xet-core#951 to get the python bindings for XetSession.new_range_upload()

Some examples:

• Write logs progressively using the append mode "a":

from huggingface_hub import hffs

with hffs.open("buckets/username/my-bucket/logs.txt", "a") as f:
    for log in logs:
        f.write(log)

• Edit a file header using the edit mode "e":

from huggingface_hub import hffs

header_length = 16
new_header = b"MY_NEW_HEADER_00"
with hffs.open("buckets/username/my-bucket/data.bin", "eb") as f:
    f.edit((0, header_length), new_header)

• Remove a certain line using the edit mode "e":

from huggingface_hub import hffs

with hffs.open("buckets/username/my-bucket/doc.txt", "e") as f:
    for line in f:
        if line == "this is a bad line\n":
            break
    f.delete(loc=f.loc - len(line), length=len(line))

TODO

• tests

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[Wauplin]

Will have a look at it this week! Sorry for the delay