feat: support V0 iceberg_tables schema for SqlCatalog by rchowell · Pull Request #2380 · apache/iceberg-rust

rchowell · 2026-04-28T16:54:28Z

Which issue does this PR close?

Closes column "iceberg_type" does not exist #2068

What changes are included in this PR?

This PR adds support for using a V0 SqlCatalog from other implementations like iceberg-python or iceberg-java, and it follows the iceberg-java behavior of checking an explicit schema-version property and migrating to V1 only-if the user requested this. I also added logging at the same level as iceberg-java.

Probe to see if we have a V0 or V1 table, then add the iceberg_type column if it does not exist. More details in apache/iceberg-python#3263

Are these changes tested?

Unit tests with migration path

rchowell · 2026-05-01T16:02:07Z

Ok @blackmwk thanks again for reviewing, I was mistaken and have fixed this PR to match iceberg-java much more closely. I also updated the PR title and description to be more accurate since this is more of a feature to support reading V0 tables in iceberg-rust. Thanks!

rchowell · 2026-05-01T16:04:09Z

 pub const SQL_CATALOG_PROP_BIND_STYLE: &str = "sql_bind_style";
+/// catalog schema version, setting to "V1" will migrate from V0 to V1 schema
+pub const SQL_CATALOG_PROP_SCHEMA_VERSION: &str = "sql.schema-version";



Seeing these together makes me think sql_schema_version yet iceberg-java has jdbc.schema-version. I do prefer consistency for something like sql.<kebab>.

CTTY · 2026-05-08T18:56:42Z

+        ))
+        .execute(&pool)
+        .await
+        .is_ok();


What if the SELECT failed here but the schema is actually V1?

I assume in that case it would pass silently here and fail later when trying to update it. This would be confusing to users

With above said, I don't have a better idea to probe the schema type... at least we should check the error type? any thoughts?

JDBC has a method to check meta table to inspect the column if exists. Is there anything similar in sqlx?

CTTY · 2026-05-08T20:58:47Z

        let tbl_metadata_location_str = tbl_metadata_location.to_string();
        self.execute(&format!(
            "INSERT INTO {CATALOG_TABLE_NAME}
             ({CATALOG_FIELD_CATALOG_NAME}, {CATALOG_FIELD_TABLE_NAMESPACE}, {CATALOG_FIELD_TABLE_NAME}, {CATALOG_FIELD_METADATA_LOCATION_PROP}, {CATALOG_FIELD_RECORD_TYPE})


Should we consider V0 schema here as well?

In iceberg-java, it looks like we only support creating V0 table. So I think it's fine for us to leave it here.

But still quite curious about the case when TableCreation has V1 schema.

https://github.com/apache/iceberg/blob/bb37293484468f0502de9986955860505aef9776/core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java#L128

CTTY · 2026-05-08T21:08:29Z

+            let requested: SchemaVersion = config
+                .props
+                .get(SQL_CATALOG_PROP_SCHEMA_VERSION)
+                .and_then(|v| v.parse().ok())


We should fail on parse error here to avoid silent fallback for misinputs

Similar to sql_bind_style parsing: https://github.com/apache/iceberg-rust/blob/main/crates/catalog/sql/src/catalog.rs#L183

CTTY · 2026-05-08T21:11:26Z

 //! };
 //!
-//! #[tokio::main]
+//! #[tokio::main(flavor = "current_thread")]


why do we need to change this?

blackmwk

Thanks @rchowell for this pr!

blackmwk · 2026-05-11T08:30:00Z

 pub const SQL_CATALOG_PROP_WAREHOUSE: &str = "warehouse";
 /// catalog sql bind style
 pub const SQL_CATALOG_PROP_BIND_STYLE: &str = "sql_bind_style";
+/// catalog schema version, setting to "V1" will migrate from V0 to V1 schema


Please make the comment more clear: this is expected catalog schema version. If it's set to v1 and it's actually v0, the catalog will do a migration.

blackmwk · 2026-05-11T08:34:43Z

 pub const SQL_CATALOG_PROP_BIND_STYLE: &str = "sql_bind_style";
+/// catalog schema version, setting to "V1" will migrate from V0 to V1 schema
+pub const SQL_CATALOG_PROP_SCHEMA_VERSION: &str = "sql.schema-version";



blackmwk · 2026-05-11T08:36:07Z

+        ))
+        .execute(&pool)
+        .await
+        .is_ok();


JDBC has a method to check meta table to inspect the column if exists. Is there anything similar in sqlx?

blackmwk · 2026-05-11T08:36:44Z

+
+        // Migrate the schema to V1 if the catalog table does not support views and the caller opted in.
+        let schema_version = if is_v1 {
+            tracing::debug!("{CATALOG_TABLE_NAME} already supports views");


This is confusing, why not just print messages like detect v1 schema

blackmwk · 2026-05-11T09:08:28Z

+            let requested: SchemaVersion = config
+                .props
+                .get(SQL_CATALOG_PROP_SCHEMA_VERSION)
+                .and_then(|v| v.parse().ok())


blackmwk · 2026-05-11T09:11:17Z

+                .and_then(|v| v.parse().ok())
+                .unwrap_or(SchemaVersion::V0);
+            if requested == SchemaVersion::V1 {
+                tracing::debug!("{CATALOG_TABLE_NAME} is being updated to support views");


This should be a warning, and the message should be sth like It's going to be updated to v1

blackmwk · 2026-05-11T09:14:02Z


+    fn record_type_filter(&self) -> &'static str {
+        match self.schema_version {
+            SchemaVersion::V1 => "AND (iceberg_type = 'TABLE' OR iceberg_type IS NULL)",


I prefer to put these sqls into the SchemaVersion enum, as following:

impl SchemaVersion { fn alter_table_sql(&self) -> Result<String> { match self { V0 => ... V1 => ... } } }

fix: automatically migrate V0 SqlCatalog catalog tables

ec717a9

rchowell mentioned this pull request Apr 28, 2026

column "iceberg_type" does not exist #2068

Open

blackmwk reviewed Apr 29, 2026

View reviewed changes

Comment thread crates/catalog/sql/src/catalog.rs Outdated

Migration based upon sql.schema-version prop

22d1206

rchowell changed the title ~~fix: automatically migrate V0 SqlCatalog catalog tables~~ feat: support V0 iceberg_tables schema for SqlCatalog May 1, 2026

rchowell commented May 1, 2026

View reviewed changes

rchowell requested a review from blackmwk May 5, 2026 17:49

rchowell mentioned this pull request May 8, 2026

fix: add iceberg_type column for SqlCatalog apache/iceberg-python#3263

Open

CTTY reviewed May 8, 2026

View reviewed changes

blackmwk reviewed May 11, 2026

View reviewed changes

Conversation

rchowell commented Apr 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

What changes are included in this PR?

Are these changes tested?

Uh oh!

Uh oh!

rchowell commented May 1, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

blackmwk left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

rchowell commented Apr 28, 2026 •

edited

Loading