[DISCUSS] Proposal for a Canonical BigDecimal Extension Type

[Mandukhai-Alimaa]

I would like to propose a new canonical extension type for BigDecimal (arbitrary-precision decimals). While Arrow supports fixed-precision Decimal128 and Decimal256, these types require a uniform scale for an entire column. Many modern data engines (PostgreSQL, Trino, Oracle) utilize numeric types that allow for variable scale per row. This flexibility is the primary driver for this proposal, as it enables the representation of data that cannot be shoehorned into a fixed-scale schema without loss of integrity or precision.

This proposal aims to provide a high-fidelity transport format for these values to ensure data integrity when moving data between these systems and Arrow-native engines.

The proposal document

I look forward to your feedback!

[kou]

Thanks for the proposal but could you use dev@arrow.apache.org instead of here?
See also: https://arrow.apache.org/community/#mailing-lists

This GitHub Discussions is synchronized with user@arrow.apache.org not dev@arrow.apache.org. user@ is for users but this proposal is for developers. dev@ is for developers.

[Mandukhai-Alimaa]

I just posted the proposal to the dev@ mailing list. Thanks!

[Mandukhai-Alimaa]

The mailing list thread: https://lists.apache.org/thread/34opw343xrzkb0vhhbqto9vx061c7j7f