Document native hf://buckets/ support in DuckDB (2.0+) - #2768
Draft
davanstrien wants to merge 2 commits into
Draft
Document native hf://buckets/ support in DuckDB (2.0+)#2768davanstrien wants to merge 2 commits into
davanstrien wants to merge 2 commits into
Conversation
duckdb-httpfs now parses hf://buckets/ paths directly (duckdb/duckdb-httpfs#293, merged 2026-09-07, ships with DuckDB 2.0). - storage-buckets-access: new "Query with DuckDB" section (native route first, DuckDB 1.x HfFileSystem / S3 fallback in a note); split the overview table row; DuckDB snippet leaves "Python Data Tools" - storage-buckets-integrations: DuckDB entry; drop it from "Coming soon" - datasets-duckdb: bucket tip leads with the native query - storage-buckets-s3: pointer to the native route Verified against a CI build of DuckDB c38d7e0739 (v2.0.0-dev) with the matching httpfs binary: public read, private read via the huggingface credential_chain secret, globs, @revision rejection on buckets. COPY TO a bucket is not supported yet, so the section says read-only for now. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
julien-c
reviewed
Sep 7, 2026
| ### Query a bucket with DuckDB | ||
|
|
||
| With the `httpfs` extension, [DuckDB](https://duckdb.org/) can read Parquet (and other formats) straight from a bucket: | ||
| With the `httpfs` extension, [DuckDB](https://duckdb.org/) can read Parquet (and other formats) straight from a bucket over the S3 gateway. If you are on DuckDB 2.0 or later, `httpfs` also reads `hf://buckets/` paths directly with your Hugging Face token and no S3 secret, see [Query with DuckDB](./storage-buckets-access#query-with-duckdb). |
Member
There was a problem hiding this comment.
maybe reverse the two sentences
Member
Author
There was a problem hiding this comment.
updated to put the native path first
Per review: reverse the two sentences and state only what each route needs. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
lhoestq
reviewed
Sep 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DuckDB's
httpfsextension now parseshf://buckets/paths directly (duckdb/duckdb-httpfs#293, merged 2026-09-07). It ships with DuckDB 2.0, which is not released yet, so this PR stays a draft until it is.Changes
HfFileSystemfrom Python, S3 gateway elsewhere) in a note. The overview table gets a DuckDB row, and the DuckDB snippet leaves "Python Data Tools", which is now pandas-only.Verified
Ran on an HF Job against a CI build of DuckDB
c38d7e0739(v2.0.0-dev) with the matching httpfs binary from the #293 pipeline:**/*.parquetglobCREATE SECRET (TYPE huggingface, PROVIDER credential_chain)withHF_TOKEN@revisionon a bucket path is rejected ("buckets do not support revisions")COPY ... TO 'hf://buckets/...'is not supported yet (DirectoryExists is not implemented), so the docs say read-only for nowBefore marking ready
datasets-duckdb.md) has mergedcc @cfahlgren1 @lhoestq
🤖 Generated with Claude Code