Add blog: Ingesting into BigLake with OLake Go - #474
Conversation
| Google renamed BigLake to Lakehouse for Apache Iceberg in April 2026, and BigLake Metastore to the Lakehouse runtime catalog. The API endpoints, `gcloud` commands, and IAM role names still use `biglake`, so every configuration value in this post works as written. This post says "BigLake" throughout to match what the config fields say. | ||
| ::: | ||
|
|
||
| For most of Apache Iceberg's history, BigQuery could read Iceberg tables, but it could not own them. The tables themselves were still managed elsewhere — by a Hive Metastore, AWS Glue, or an outside REST catalog. BigQuery was a consumer, not the source of truth. |
There was a problem hiding this comment.
please remove em dashes wherever used in the blog
| For most of Apache Iceberg's history, BigQuery could read Iceberg tables, but it could not own them. The tables themselves were still managed elsewhere — by a Hive Metastore, AWS Glue, or an outside REST catalog. BigQuery was a consumer, not the source of truth. | ||
|
|
||
|
|
||
| Apache Iceberg 1.10.0 changed that. Google contributed two things to the release. The first is a native BigQuery Metastore Catalog (`BigQueryMetastoreCatalog` and `BigQueryMetastoreClient`), merged through PR #12808, which lets Spark, Flink, Trino, and any other Iceberg-aware engine use BigQuery's own metadata service as the catalog of record. The same catalog was added to the Iceberg Kafka Connect runtime bundle, so streaming pipelines get it too. |
There was a problem hiding this comment.
can we link the PR here cause just mentioning the PR number does not look good
There was a problem hiding this comment.
added the link to the PR
|
|
||
| The second is `GoogleAuthManager`, which plugs Google credentials into the standard Iceberg REST catalog path. That second piece is what this post relies on. Google now recommends the BigLake Iceberg REST catalog endpoint for new setups and describes the BigQuery Metastore Catalog as the earlier integration, so the destination below points at the REST endpoint and authenticates through `GoogleAuthManager`. | ||
|
|
||
| ## What BigLake actually does |
There was a problem hiding this comment.
can we change this heading to something better
There was a problem hiding this comment.
changed it to "BigLake is a governance layer, not a storage layer"
|
|
||
| ## What BigLake actually does | ||
|
|
||
| BigLake is easiest to understand as a governance layer rather than a storage layer. A BigLake Iceberg table is a governed pointer to Iceberg data and metadata sitting in Cloud Storage, not a copy of it inside BigQuery. BigLake applies BigQuery's IAM controls to that external data, and the BigLake Iceberg REST catalog lets engines talk to it directly as a catalog endpoint. |
There was a problem hiding this comment.
BigLake applies BigQuery's IAM controls to that external data, and the BigLake Iceberg REST catalog lets engines talk to it directly as a catalog endpoint.
The line can be simplified.
|
|
||
| One caveat worth knowing up front: row-level security and column masking are available on BigLake external tables generally, but Google documents fine-grained access control as unsupported for tables managed through the Iceberg REST catalog endpoint. Access control on those tables runs through IAM at the catalog, namespace, and table level instead. | ||
|
|
||
| Google has also been building BigLake managed tables, a separate table type that brings BigQuery's usual table upkeep (compaction, clustering, garbage collection) to Iceberg tables, along with full DML: inserts, updates, deletes, and schema evolution. Google reports that streaming writes into managed tables avoid the small-file buildup that normally comes with frequent inserts. |
There was a problem hiding this comment.
If we are not talking about the managed tables further is this paragraph even necessary? Like the blog is on ingesting into biglake using olake so if this is not possible from olake side why are we mentioning this?
There was a problem hiding this comment.
Trimmed it instead of cutting it, one line to head off readers confusing this with BigLake managed tables since Google's docs use both terms.
|
|
||
| Before native catalog support, connecting BigQuery to an outside engine meant bridging them indirectly — typically by writing Iceberg through BigLake and using change notifications (Pub/Sub events on the GCS bucket, for example) to refresh a table definition somewhere else. It worked, but it meant an extra moving part and a small window where the two sides could disagree about table state. | ||
|
|
||
| With the BigQuery Metastore Catalog and the BigLake Iceberg REST catalog, that bridge is no longer necessary. An engine writing Iceberg tables straight into BigLake's catalog means BigQuery reads the same metadata everyone else does. Ingesting directly into BigLake is now a first-class path. |
There was a problem hiding this comment.
we can simplify this section more. can we first inform the problem like why did bigquery have to do things indirectly? it was because bigquery and the external engines never shared the same iceberg catalog.
Then talk about what the old method was, like we used to write data to GCS using lets say spark using some catalog and then bigquery used to read the metadata file and for every change we need to manually point bigquery to latest metadata files. For instance, spark writing iceberg data using glue as catalog to GCS and then bigquery creates external iceberg table pointing to metadata file in gcs.
But now bigquery and the other query engines share the same catalog. so no separate work needs to be done as bigquery can now talk to the same catalog.
| - how to confirm the tables landed, by querying them from BigQuery | ||
| - the failure modes worth knowing about, and how to tell them apart | ||
|
|
||
|  |
There was a problem hiding this comment.
the before image needs to be changed. the query engines wrote through their own catalog to GCS and then the bigquery created the external tables pointing to the metadata file. also in the image we have not shown bigquery anywhere. The external table is bigquery iceberg external table its important to mention bigquery in the diagram.
Also the after image can we have like:
Engine <- (read/write) -> BigLake REST Catalog <- BigQuery
And also have one arrow poiting to GCS (data files) from BigLake REST Catalog.
|
|
||
| ## Getting BigLake ready | ||
|
|
||
| Before touching OLake, the catalog needs to exist on the Google Cloud side. |
There was a problem hiding this comment.
Please make sure everywhere we are changing OLake to OLake Go
|
|
||
| **Project and API.** A Google Cloud project with billing enabled and the BigLake API turned on. | ||
|
|
||
| **Cloud Storage bucket.** A bucket to hold Iceberg table data and metadata. Every storage location used by one catalog must sit in the same geographic region group. A US bucket and a European bucket cannot belong to the same catalog. |
There was a problem hiding this comment.
can we mention this is only in the case when using multi bucket catalog. It is not mentioned anywhere and suddenly talking about this can confuse the user.
|
|
||
| **Cloud Storage bucket.** A bucket to hold Iceberg table data and metadata. Every storage location used by one catalog must sit in the same geographic region group. A US bucket and a European bucket cannot belong to the same catalog. | ||
|
|
||
| **Create the catalog in end-user credentials mode.** Multi-bucket is the type to use. Google calls the single-bucket type highly discouraged for new projects, since it locks the catalog name to the bucket name permanently. |
There was a problem hiding this comment.
this is not the sole region actually. and this is just an added advantage not the reason to choose multi bucket catalog.
one of the main reason is:
if the bucket is in us-east-1 and bigquery is in ap-south-1 or any other region other than the bucket region then bigquery cannot query the data.
there are also other reasons why biglake suggests to use multi bucket catalog as well but definitely not because user is able to name their catalog based on their preference.
|
|
||
| **Create the catalog in end-user credentials mode.** Multi-bucket is the type to use. Google calls the single-bucket type highly discouraged for new projects, since it locks the catalog name to the bucket name permanently. | ||
|
|
||
| Use end-user credentials mode. OLake Go authenticates as the service account in `gcp_service_account_json` and does not request vended credentials, so the catalog has to pass that identity through to Cloud Storage. This is also why the service account needs Storage Object User directly on the buckets. |
There was a problem hiding this comment.
the other points above are in bold then why is this point not bold? also can we have a colon or dash instead of full stop after each point?
| [--primary-location LOCATION] | ||
| ``` | ||
|
|
||
| Set `--primary-location` to `US` or `EU` if the tables need to be queryable from the matching BigQuery multi-region. Getting this wrong is expensive: changing the primary location on an existing multi-region catalog means deleting and recreating the catalog. |
There was a problem hiding this comment.
this entire process only talks about the multi bucket catalog and that is not the only option that olake go supports so we need to also inform the single bucket catalog method as well.
|
|
||
| Set `--primary-location` to `US` or `EU` if the tables need to be queryable from the matching BigQuery multi-region. Getting this wrong is expensive: changing the primary location on an existing multi-region catalog means deleting and recreating the catalog. | ||
|
|
||
| **Grant roles.** The service account OLake uses needs BigLake Editor (`roles/biglake.editor`) on the project and Storage Object User (`roles/storage.objectUser`) on every bucket the catalog touches. Registering new tables also needs BigLake Admin (`roles/biglake.admin`) on the project. Missing Storage Object User on a bucket is the single most common reason a connection test passes and the first sync fails. |
There was a problem hiding this comment.
can you please confirm from which docs was the last line in this statement referenced from? To be specific this line:
Missing Storage Object User on a bucket is the single most common reason a connection test passes and the first sync fails.
Are we sure this is true?
|
|
||
| **Grant roles.** The service account OLake uses needs BigLake Editor (`roles/biglake.editor`) on the project and Storage Object User (`roles/storage.objectUser`) on every bucket the catalog touches. Registering new tables also needs BigLake Admin (`roles/biglake.admin`) on the project. Missing Storage Object User on a bucket is the single most common reason a connection test passes and the first sync fails. | ||
|
|
||
| **Know the limits before syncing.** The REST catalog endpoint supports Parquet data files only. Iceberg V1 tables are not supported and have to be upgraded to V2 first. The table properties `write.data.path` and `write.metadata.path` cannot be moved off their defaults. Table locations get a random suffix appended automatically, so a table lands at `gs://bucket/namespace/table/RANDOM` rather than the path you would expect when looking in Cloud Storage. |
There was a problem hiding this comment.
in this can we segregate things a little like parquet file format limitation comes under data limitation and the write.data.path comes under the table limitation.
Also is it necessary to mention these limitations in the blog wont it be better to add a link to their docs where these limitations are mentioned and just inform that these are the limitations that needs to be taken care of? Cause when I read the docs there are some more table limitations that can be useful for the user.
|
|
||
| **Know the limits before syncing.** The REST catalog endpoint supports Parquet data files only. Iceberg V1 tables are not supported and have to be upgraded to V2 first. The table properties `write.data.path` and `write.metadata.path` cannot be moved off their defaults. Table locations get a random suffix appended automatically, so a table lands at `gs://bucket/namespace/table/RANDOM` rather than the path you would expect when looking in Cloud Storage. | ||
|
|
||
| Google's Lakehouse documentation covers catalog creation and IAM setup in full. Once the catalog exists and the roles are granted, the rest happens on the OLake side. |
There was a problem hiding this comment.
can we add a link to this as well
|
|
||
| ## Configuring OLake Go to write into BigLake | ||
|
|
||
| BigLake catalog support ships in OLake Go 0.9.0 and above, so confirm the version before configuring the destination. |
There was a problem hiding this comment.
Lets check on this comment in the end as the version will be changed based on the release that is happening.
|
|
||
| BigLake catalog support ships in OLake Go 0.9.0 and above, so confirm the version before configuring the destination. | ||
|
|
||
| Set up the source as usual first (any of OLake Go's supported databases, Kafka, or S3 work here). Then, when configuring the destination, choose the Iceberg writer with the REST catalog type, and select BigLake as the catalog. |
There was a problem hiding this comment.
this is not how we set up big lake currently as well. currently we only select rest catalog and then fill in the details related to biglake.
also the destination config is changing now and we will be having BigLake as the catalog in the dropdown. so lets update the doc accordingly.
below are some images that will be useful for you to understand how the new destination setup will look like:
| | Field | Value | | ||
| |-------|-------| | ||
| | `catalog_type` | `rest` | | ||
| | `rest_catalog_url` | `https://biglake.googleapis.com/iceberg/v1/restcatalog` (fixed, same for everyone) | | ||
| | `iceberg_s3_path` | The BigLake catalog path. Two formats, described below. | | ||
| | `rest_auth_type` | `org.apache.iceberg.gcp.auth.GoogleAuthManager` (the only supported value) | | ||
| | `gcp_service_account_json` | The full JSON content of the service account key | | ||
| | `catalog_name` | Defaults to `olake_iceberg` if left blank | | ||
| | `gcp_auth_scopes` | Defaults to `https://www.googleapis.com/auth/cloud-platform` | |
There was a problem hiding this comment.
this table will also need some changes as mentioned above please check the new destination config for biglake
2 participants
Adds a new blog post covering how to ingest Apache Iceberg tables into Google BigLake using OLake Go, now that Iceberg 1.10.0 ships native BigQuery catalog support