[BUG] allow_mismatched_pq_schemas rejects sources that disagree on Parquet field ID presence

[vuule]

Describe the bug

When reading multiple Parquet sources with allow_mismatched_pq_schemas(true) and a column projection, libcudf rejects the read if the sources disagree on whether a column carries a Parquet field ID, even when the column names, types, and nesting all match and selection is by name.

The cause is in equal_to_except_num_children in cpp/src/io/parquet/reader_impl_helpers.cpp, used by map_column to validate a projected column across sources:

auto const names_match =
  (match_schema_by_field_id and lhs.field_id.has_value() and rhs.field_id.has_value())
    ? lhs.field_id == rhs.field_id
    : are_column_paths_equal(lhs.name, rhs.name, case_sensitive_names);
return lhs.type == rhs.type and lhs.converted_type == rhs.converted_type and
       lhs.type_length == rhs.type_length and names_match and
       lhs.decimal_scale == rhs.decimal_scale and
       lhs.decimal_precision == rhs.decimal_precision and lhs.field_id == rhs.field_id;

The predicate correctly falls back to name matching when field IDs are unavailable, but then unconditionally requires lhs.field_id == rhs.field_id regardless of the selection mode. std::optional<int32_t>{1} == std::nullopt is false, so a column written with a field ID by one producer and without one by another can never match, and the read fails with std::invalid_argument.

This is a realistic input: whether field IDs are emitted varies by writer and by configuration, so a dataset assembled from files produced by different tools can easily be inconsistent. Reading such a dataset is the case allow_mismatched_pq_schemas exists to support.

Steps/Code to reproduce bug

Two sources with identical column names, types, and order. The only difference is that one writer assigns field IDs.

TEST_F(ParquetReaderTest, MismatchedFieldIdPresence)
{
  auto const col0  = cudf::test::fixed_width_column_wrapper<int32_t>{1, 2, 3};
  auto const col1  = cudf::test::fixed_width_column_wrapper<int32_t>{4, 5, 6};
  auto const table = cudf::table_view{{col0, col1}};

  auto const write_buffer = [&](bool set_field_ids) {
    auto buffer   = std::vector<char>{};
    auto metadata = cudf::io::table_input_metadata{table};
    metadata.column_metadata[0].set_name("col0");
    metadata.column_metadata[1].set_name("col1");
    if (set_field_ids) {
      metadata.column_metadata[0].set_parquet_field_id(1);
      metadata.column_metadata[1].set_parquet_field_id(2);
    }
    cudf::io::write_parquet(
      cudf::io::parquet_writer_options::builder(cudf::io::sink_info{&buffer}, table)
        .metadata(std::move(metadata))
        .build());
    return buffer;
  };

  auto const read_projected = [&](std::vector<std::vector<char>> const& buffers) {
    return cudf::io::read_parquet(
      cudf::io::parquet_reader_options::builder(build_source_info(buffers))
        .allow_mismatched_pq_schemas(true)
        .column_names({"col0"})
        .build());
  };

  // Passes: both sources agree on field IDs
  {
    auto const buffers = std::vector<std::vector<char>>{write_buffer(true), write_buffer(true)};
    EXPECT_NO_THROW(read_projected(buffers));
  }
  // Fails: only the first source carries field IDs
  {
    auto const buffers = std::vector<std::vector<char>>{write_buffer(true), write_buffer(false)};
    EXPECT_NO_THROW(read_projected(buffers));
  }
  // Fails: only the second source carries field IDs
  {
    auto const buffers = std::vector<std::vector<char>>{write_buffer(false), write_buffer(true)};
    EXPECT_NO_THROW(read_projected(buffers));
  }
}

Both asymmetric cases fail with:

CUDF failure at: cpp/src/io/parquet/reader_impl_helpers.cpp:2135: Encountered mismatching data type or schema across the Parquet sources for column 'col0'

The symmetric baseline passes, which isolates the field_id term as the cause.

Expected behavior

When selection is by name, the read should succeed: the projected column matches across sources by name, type, and nesting, and differing field-ID presence should not be treated as a schema mismatch. Dropping the field_id term from the comparison when selection_mode != column_selection_mode::BY_FIELD_ID would do it. When selecting by field ID, requiring the IDs to match is of course correct.

A related note: SchemaElement::operator== in cpp/include/cudf/io/parquet_schema.hpp also compares field_id, so without allow_mismatched_pq_schemas these sources are rejected by the "All sources must have the same schema" check instead. That path is arguably working as intended, since strict equality is the point, but it is worth confirming that field-ID presence should participate in schema equality at all.

No existing test covers asymmetric field-ID presence: tests either set no field IDs, or set matching IDs on every source, so the term is trivially satisfied.

Environment overview

• Environment location: Docker (devcontainer)
• Method of cuDF install: from source

Additional context

Affects the C++ reader directly; reachable from Python wherever allow_mismatched_pq_schemas is exposed.