From e6e8a1e417964e6b6705b9610c3bf1d5fc4ed21f Mon Sep 17 00:00:00 2001 From: Daniel van Strien Date: Mon, 7 Sep 2026 13:55:58 +0100 Subject: [PATCH 1/2] Document native hf://buckets/ support in DuckDB (2.0+) duckdb-httpfs now parses hf://buckets/ paths directly (duckdb/duckdb-httpfs#293, merged 2026-09-07, ships with DuckDB 2.0). - storage-buckets-access: new "Query with DuckDB" section (native route first, DuckDB 1.x HfFileSystem / S3 fallback in a note); split the overview table row; DuckDB snippet leaves "Python Data Tools" - storage-buckets-integrations: DuckDB entry; drop it from "Coming soon" - datasets-duckdb: bucket tip leads with the native query - storage-buckets-s3: pointer to the native route Verified against a CI build of DuckDB c38d7e0739 (v2.0.0-dev) with the matching httpfs binary: public read, private read via the huggingface credential_chain secret, globs, @revision rejection on buckets. COPY TO a bucket is not supported yet, so the section says read-only for now. Co-Authored-By: Claude Fable 5.1 --- docs/hub/datasets-duckdb.md | 11 +++----- docs/hub/storage-buckets-access.md | 34 ++++++++++++++++-------- docs/hub/storage-buckets-integrations.md | 12 ++++++++- docs/hub/storage-buckets-s3.md | 2 +- 4 files changed, 39 insertions(+), 20 deletions(-) diff --git a/docs/hub/datasets-duckdb.md b/docs/hub/datasets-duckdb.md index 50ae1b70c3..f5f683b6e4 100644 --- a/docs/hub/datasets-duckdb.md +++ b/docs/hub/datasets-duckdb.md @@ -74,11 +74,8 @@ SELECT * FROM 'hf://datasets/ibm/duorc/ParaphraseRC/*.parquet' LIMIT 3; In the following sections, we will cover more complex operations you can perform with DuckDB on Hugging Face datasets. > [!TIP] -> **Querying Storage Buckets**: When using the DuckDB Python client, you can query data stored in [Storage Buckets](./storage-buckets) by registering the Hugging Face filesystem: -> ```python -> import duckdb -> from huggingface_hub import HfFileSystem -> duckdb.register_filesystem(HfFileSystem()) -> duckdb.sql("SELECT * FROM 'hf://buckets/username/my-bucket/data.parquet' LIMIT 10") +> **Querying Storage Buckets**: DuckDB 2.0 and later also read [Storage Buckets](./storage-buckets) natively, from the CLI and every client: +> ```sql +> SELECT * FROM 'hf://buckets/username/my-bucket/data.parquet' LIMIT 10; > ``` -Native `hf://buckets/` support in DuckDB is expected in a future release. +> Buckets are not versioned, so the `@revision` syntax does not apply to bucket paths. On DuckDB 1.x, register the Hugging Face filesystem from Python instead: `duckdb.register_filesystem(HfFileSystem())`. See [Query with DuckDB](./storage-buckets-access#query-with-duckdb) for details. diff --git a/docs/hub/storage-buckets-access.md b/docs/hub/storage-buckets-access.md index 2d93f21631..430c3e8e8a 100644 --- a/docs/hub/storage-buckets-access.md +++ b/docs/hub/storage-buckets-access.md @@ -8,7 +8,8 @@ Beyond the [CLI and Python SDK](./storage-buckets#managing-files), there are sev |--------|----------|---------| | **hf-mount** | Mount as local filesystem — any tool works | [See below](#mount-as-a-local-filesystem) | | **Volume mounts** | HF Jobs & Spaces (same idea, managed for you) | [See below](#volume-mounts-in-jobs-and-spaces) | -| **hf:// paths** (fsspec) | Python data tools (pandas, DuckDB) | [See below](#python-data-tools) | +| **DuckDB** | SQL queries over bucket files, from the CLI or any client | [See below](#query-with-duckdb) | +| **hf:// paths** (fsspec) | Python data tools (pandas and other fsspec-aware libraries) | [See below](#python-data-tools) | | **CLI sync** | Batch transfers, backups | [Sync docs](./storage-buckets#syncing-directories) | | **S3 API** | Existing S3 tooling (AWS CLI, boto3, s5cmd) | [S3-Compatible API](./storage-buckets-s3) | @@ -47,6 +48,27 @@ Jobs can also take a **local directory** as the volume source (`-v ./training-da For the full volume mount syntax and Python API, see the [Jobs configuration docs](./jobs-configuration#volumes) and the [Spaces volume mount guide](/docs/huggingface_hub/guides/manage-spaces#mount-volumes-in-your-space). +## Query with DuckDB + +[DuckDB](https://duckdb.org/) 2.0 and later read `hf://buckets/` paths natively through the `httpfs` extension. From the DuckDB CLI or any client, load `httpfs` and query a bucket: + +```sql +LOAD httpfs; + +SELECT * FROM 'hf://buckets/username/my-bucket/data.parquet' LIMIT 10; +``` + +Glob patterns work as they do for datasets, e.g. `'hf://buckets/username/my-bucket/data/**/*.parquet'`. Buckets are not versioned, so the `@revision` syntax does not apply to bucket paths. Native `hf://buckets/` access is read-only for now; to write query results to a bucket, use the [S3-compatible API](./storage-buckets-s3#query-a-bucket-with-duckdb) or the Python client with `HfFileSystem`. + +**Private buckets:** create a Hugging Face secret first. It picks up the token from `hf auth login` or the `HF_TOKEN` environment variable: + +```sql +CREATE SECRET hf (TYPE huggingface, PROVIDER credential_chain); +``` + +> [!NOTE] +> On DuckDB 1.x, `httpfs` does not recognize `hf://buckets/` paths. Register [`HfFileSystem`](/docs/huggingface_hub/guides/hf_file_system) from Python instead, or use the [S3-compatible API](./storage-buckets-s3#query-a-bucket-with-duckdb) from the CLI and other clients. + ## Python Data Tools The [`HfFileSystem`](/docs/huggingface_hub/guides/hf_file_system) provides [fsspec](https://filesystem-spec.readthedocs.io)-compatible access to buckets using `hf://buckets/` paths. Any Python library that supports fsspec can read and write bucket data directly. @@ -60,14 +82,4 @@ df = pd.read_parquet("hf://buckets/username/my-bucket/data.parquet") df.to_parquet("hf://buckets/username/my-bucket/output.parquet") ``` -**DuckDB** (Python client): - -```python -import duckdb -from huggingface_hub import HfFileSystem - -duckdb.register_filesystem(HfFileSystem()) -duckdb.sql("SELECT * FROM 'hf://buckets/username/my-bucket/data.parquet' LIMIT 10") -``` - For more on `hf://` paths and supported operations, see the [`HfFileSystem` guide](/docs/huggingface_hub/guides/hf_file_system) and the [Buckets Python guide](/docs/huggingface_hub/guides/buckets). diff --git a/docs/hub/storage-buckets-integrations.md b/docs/hub/storage-buckets-integrations.md index 24fa653c75..a1118a8804 100644 --- a/docs/hub/storage-buckets-integrations.md +++ b/docs/hub/storage-buckets-integrations.md @@ -137,10 +137,20 @@ files = hffs.ls("buckets/username/my-bucket") text_files = hffs.glob("buckets/username/my-bucket/*.txt") ``` +## DuckDB + +DuckDB 2.0 and later read `hf://buckets/` paths natively through the `httpfs` extension, from the CLI and every client: + +```sql +SELECT * FROM 'hf://buckets/username/my-bucket/data/**/*.parquet' LIMIT 10; +``` + +See [Query with DuckDB](./storage-buckets-access#query-with-duckdb) for private buckets and older DuckDB versions. + ## Other languages [OpenDAL](https://opendal.apache.org/) provides a similar filesystem interface for Rust, Java, Go, JavaScript, and more. ## Coming soon -Native `hf://` URL support is on the way for more libraries — including Polars, DuckDB, and webdataset. In the meantime, all of these already work today through the [S3-compatible API](./storage-buckets-s3). +Native `hf://` URL support is on the way for more libraries — including Polars and webdataset. In the meantime, all of these already work today through the [S3-compatible API](./storage-buckets-s3). diff --git a/docs/hub/storage-buckets-s3.md b/docs/hub/storage-buckets-s3.md index 3c500d8e13..cd93e98c5e 100644 --- a/docs/hub/storage-buckets-s3.md +++ b/docs/hub/storage-buckets-s3.md @@ -163,7 +163,7 @@ s3.download_file("my-bucket", "models/model.safetensors", "model.safetensors") ### Query a bucket with DuckDB -With the `httpfs` extension, [DuckDB](https://duckdb.org/) can read Parquet (and other formats) straight from a bucket: +With the `httpfs` extension, [DuckDB](https://duckdb.org/) can read Parquet (and other formats) straight from a bucket over the S3 gateway. If you are on DuckDB 2.0 or later, `httpfs` also reads `hf://buckets/` paths directly with your Hugging Face token and no S3 secret, see [Query with DuckDB](./storage-buckets-access#query-with-duckdb). ```sql INSTALL httpfs; From f58d78d4baacf408eb8f2e7e6f51c9016a70c786 Mon Sep 17 00:00:00 2001 From: Daniel van Strien Date: Mon, 7 Sep 2026 14:41:22 +0100 Subject: [PATCH 2/2] S3 page: lead with the native DuckDB route, gateway for earlier versions Per review: reverse the two sentences and state only what each route needs. Co-Authored-By: Claude Fable 5.1 --- docs/hub/storage-buckets-s3.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/hub/storage-buckets-s3.md b/docs/hub/storage-buckets-s3.md index cd93e98c5e..e0b6587229 100644 --- a/docs/hub/storage-buckets-s3.md +++ b/docs/hub/storage-buckets-s3.md @@ -163,7 +163,9 @@ s3.download_file("my-bucket", "models/model.safetensors", "model.safetensors") ### Query a bucket with DuckDB -With the `httpfs` extension, [DuckDB](https://duckdb.org/) can read Parquet (and other formats) straight from a bucket over the S3 gateway. If you are on DuckDB 2.0 or later, `httpfs` also reads `hf://buckets/` paths directly with your Hugging Face token and no S3 secret, see [Query with DuckDB](./storage-buckets-access#query-with-duckdb). +On DuckDB 2.0 or later, the `httpfs` extension reads `hf://buckets/` paths directly. See [Query with DuckDB](./storage-buckets-access#query-with-duckdb). + +For earlier versions of DuckDB, or to write query results back to a bucket with `COPY ... TO`, go through the S3 gateway instead: ```sql INSTALL httpfs;