Skip to content
Draft
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 4 additions & 7 deletions docs/hub/datasets-duckdb.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,11 +74,8 @@ SELECT * FROM 'hf://datasets/ibm/duorc/ParaphraseRC/*.parquet' LIMIT 3;
In the following sections, we will cover more complex operations you can perform with DuckDB on Hugging Face datasets.

> [!TIP]
> **Querying Storage Buckets**: When using the DuckDB Python client, you can query data stored in [Storage Buckets](./storage-buckets) by registering the Hugging Face filesystem:
> ```python
> import duckdb
> from huggingface_hub import HfFileSystem
> duckdb.register_filesystem(HfFileSystem())
> duckdb.sql("SELECT * FROM 'hf://buckets/username/my-bucket/data.parquet' LIMIT 10")
> **Querying Storage Buckets**: DuckDB 2.0 and later also read [Storage Buckets](./storage-buckets) natively, from the CLI and every client:
> ```sql
> SELECT * FROM 'hf://buckets/username/my-bucket/data.parquet' LIMIT 10;
> ```
Native `hf://buckets/` support in DuckDB is expected in a future release.
> Buckets are not versioned, so the `@revision` syntax does not apply to bucket paths. On DuckDB 1.x, register the Hugging Face filesystem from Python instead: `duckdb.register_filesystem(HfFileSystem())`. See [Query with DuckDB](./storage-buckets-access#query-with-duckdb) for details.
34 changes: 23 additions & 11 deletions docs/hub/storage-buckets-access.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,8 @@ Beyond the [CLI and Python SDK](./storage-buckets#managing-files), there are sev
|--------|----------|---------|
| **hf-mount** | Mount as local filesystem — any tool works | [See below](#mount-as-a-local-filesystem) |
| **Volume mounts** | HF Jobs & Spaces (same idea, managed for you) | [See below](#volume-mounts-in-jobs-and-spaces) |
| **hf:// paths** (fsspec) | Python data tools (pandas, DuckDB) | [See below](#python-data-tools) |
| **DuckDB** | SQL queries over bucket files, from the CLI or any client | [See below](#query-with-duckdb) |
| **hf:// paths** (fsspec) | Python data tools (pandas and other fsspec-aware libraries) | [See below](#python-data-tools) |
| **CLI sync** | Batch transfers, backups | [Sync docs](./storage-buckets#syncing-directories) |
| **S3 API** | Existing S3 tooling (AWS CLI, boto3, s5cmd) | [S3-Compatible API](./storage-buckets-s3) |

Expand Down Expand Up @@ -47,6 +48,27 @@ Jobs can also take a **local directory** as the volume source (`-v ./training-da

For the full volume mount syntax and Python API, see the [Jobs configuration docs](./jobs-configuration#volumes) and the [Spaces volume mount guide](/docs/huggingface_hub/guides/manage-spaces#mount-volumes-in-your-space).

## Query with DuckDB

[DuckDB](https://duckdb.org/) 2.0 and later read `hf://buckets/` paths natively through the `httpfs` extension. From the DuckDB CLI or any client, load `httpfs` and query a bucket:

```sql
LOAD httpfs;

SELECT * FROM 'hf://buckets/username/my-bucket/data.parquet' LIMIT 10;
```

Glob patterns work as they do for datasets, e.g. `'hf://buckets/username/my-bucket/data/**/*.parquet'`. Buckets are not versioned, so the `@revision` syntax does not apply to bucket paths. Native `hf://buckets/` access is read-only for now; to write query results to a bucket, use the [S3-compatible API](./storage-buckets-s3#query-a-bucket-with-duckdb) or the Python client with `HfFileSystem`.

**Private buckets:** create a Hugging Face secret first. It picks up the token from `hf auth login` or the `HF_TOKEN` environment variable:

```sql
CREATE SECRET hf (TYPE huggingface, PROVIDER credential_chain);
```

> [!NOTE]
> On DuckDB 1.x, `httpfs` does not recognize `hf://buckets/` paths. Register [`HfFileSystem`](/docs/huggingface_hub/guides/hf_file_system) from Python instead, or use the [S3-compatible API](./storage-buckets-s3#query-a-bucket-with-duckdb) from the CLI and other clients.

## Python Data Tools

The [`HfFileSystem`](/docs/huggingface_hub/guides/hf_file_system) provides [fsspec](https://filesystem-spec.readthedocs.io)-compatible access to buckets using `hf://buckets/` paths. Any Python library that supports fsspec can read and write bucket data directly.
Expand All @@ -60,14 +82,4 @@ df = pd.read_parquet("hf://buckets/username/my-bucket/data.parquet")
df.to_parquet("hf://buckets/username/my-bucket/output.parquet")
```

**DuckDB** (Python client):

```python
import duckdb
from huggingface_hub import HfFileSystem

duckdb.register_filesystem(HfFileSystem())
duckdb.sql("SELECT * FROM 'hf://buckets/username/my-bucket/data.parquet' LIMIT 10")
```

For more on `hf://` paths and supported operations, see the [`HfFileSystem` guide](/docs/huggingface_hub/guides/hf_file_system) and the [Buckets Python guide](/docs/huggingface_hub/guides/buckets).
12 changes: 11 additions & 1 deletion docs/hub/storage-buckets-integrations.md
Original file line number Diff line number Diff line change
Expand Up @@ -137,10 +137,20 @@ files = hffs.ls("buckets/username/my-bucket")
text_files = hffs.glob("buckets/username/my-bucket/*.txt")
```

## DuckDB

DuckDB 2.0 and later read `hf://buckets/` paths natively through the `httpfs` extension, from the CLI and every client:

```sql
SELECT * FROM 'hf://buckets/username/my-bucket/data/**/*.parquet' LIMIT 10;
```

See [Query with DuckDB](./storage-buckets-access#query-with-duckdb) for private buckets and older DuckDB versions.

## Other languages

[OpenDAL](https://opendal.apache.org/) provides a similar filesystem interface for Rust, Java, Go, JavaScript, and more.

## Coming soon

Native `hf://` URL support is on the way for more libraries — including Polars, DuckDB, and webdataset. In the meantime, all of these already work today through the [S3-compatible API](./storage-buckets-s3).
Native `hf://` URL support is on the way for more libraries — including Polars and webdataset. In the meantime, all of these already work today through the [S3-compatible API](./storage-buckets-s3).
2 changes: 1 addition & 1 deletion docs/hub/storage-buckets-s3.md
Original file line number Diff line number Diff line change
Expand Up @@ -163,7 +163,7 @@ s3.download_file("my-bucket", "models/model.safetensors", "model.safetensors")

### Query a bucket with DuckDB

With the `httpfs` extension, [DuckDB](https://duckdb.org/) can read Parquet (and other formats) straight from a bucket:
With the `httpfs` extension, [DuckDB](https://duckdb.org/) can read Parquet (and other formats) straight from a bucket over the S3 gateway. If you are on DuckDB 2.0 or later, `httpfs` also reads `hf://buckets/` paths directly with your Hugging Face token and no S3 secret, see [Query with DuckDB](./storage-buckets-access#query-with-duckdb).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe reverse the two sentences

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated to put the native path first


```sql
INSTALL httpfs;
Expand Down
Loading