Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions dialects/tests/types_test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ dependencies:
supported_types:
- I8
- I16
- type: UNBOUND
- type: I32
metadata:
storage_bytes: 4
Expand Down
67 changes: 0 additions & 67 deletions extensions/unknown.yaml

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm confused. did this file exist before?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, that has been there for a while #79

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 to removing this. IMO unbound is a better approach for this, and I don't think anyone is using it. If they are, we'll find out when they come yelling 😅

This file was deleted.

9 changes: 9 additions & 0 deletions proto/substrait/type.proto
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,13 @@ message Type {
Map map = 28;
Func func = 38;

// A placeholder type whose concrete type has not yet been bound.
// Unbound may be used where a concrete type would normally appear in
// partially bound plans and expressions, and must be bound to a
// concrete type before execution. Substrait defines no runtime semantics
// for values whose type is unbound.
Unbound unbound = 39;

UserDefined user_defined = 30;

// Deprecated in favor of user_defined, which allows nullability and
Expand Down Expand Up @@ -209,6 +216,8 @@ message Type {
Nullability nullability = 4;
}

message Unbound {}

// A function type for higher-order functions.
// Represents a function that takes parameters of specified types and
// returns a value of a specified type.
Expand Down
14 changes: 14 additions & 0 deletions site/docs/types/type_classes.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,20 @@ Simple type classes are those that don't support any form of configuration. For
| date | A date within [1000-01-01..9999-12-31]. | `int32` days since `1970-01-01`
| interval_year | Interval year to month. Supports a range of [-10,000..10,000] years with month precision (= [-120,000..120,000] months). Usually stored as separate integers for years and months, but only the total number of months is significant, i.e. `1y 0m` is considered equal to `0y 12m` or `1001y -12000m`. | `int32` years and `int32` months, with the added constraint that each component can never independently specify more than 10,000 years, even if the components have opposite signs (e.g. `-10000y 200000m` is **not** allowed)
| uuid | A universally-unique identifier composed of 128 bits. Typically presented to users in the following hexadecimal format: `c48ffa9e-64f4-44cb-ae47-152b4e60e77b`. Any 128-bit value is allowed, without specific adherence to RFC4122. | 16-byte `binary`
| unbound | A placeholder for a type that has not yet been bound to a concrete type. May only appear in partially bound plans, which must be bound before execution. | n/a

### Unbound Type

The `unbound` type class is a placeholder for a type that has not yet been bound to a concrete type. It may appear wherever a concrete type would normally appear, for example as a field type within `ReadRel.base_schema`. This allows a producer to serialize the shape of a plan — its relations and the names and ordering of its fields — before the concrete types of those fields are known. A downstream binder later replaces each occurrence of `unbound` with a concrete type once the necessary schema or catalog information is available.

A plan that contains `unbound` anywhere within it is *partially bound*. Partially bound plans may be serialized and exchanged, but they are not executable, and Substrait defines no runtime semantics for values of unbound type. Specifically:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
A plan that contains `unbound` anywhere within it is *partially bound*. Partially bound plans may be serialized and exchanged, but they are not executable, and Substrait defines no runtime semantics for values of unbound type. Specifically:
A plan that contains `unbound` type is *partially bound*. Partially bound plans may be serialized and exchanged, but they are not executable, and Substrait defines no runtime semantics for values of unbound type. Specifically:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks like unbound is consistently used instead of unbound type. I don't feel strongly but if it were type, spelling out type is not ambiguous.


- `unbound` is not a wildcard: it does not unify with, or implicitly coerce to, any other type. In particular, `unbound` is distinct from `any`: `any` is a function-signature wildcard that participates in overload matching, while `unbound` is a plan-level placeholder for a type that is not yet known.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm wondering whether we should specify the reverse direction: any matches unbound just like any other type while it is unbound?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm... interesting.... any2 f(any1, any2) would match unbound f(unbound, unbound)? 😆 I don't know these matching actually making sense or we want to try to bind this.

- `unbound` carries no nullability and accepts no parameters; those properties are determined when the type is bound to a concrete type.
- There is no literal of type `unbound`.
- Substrait prescribes no resolution policy for `unbound`. A function invocation whose argument types are unbound is part of a partially bound plan; how a binder assigns concrete types, and how it subsequently resolves function bindings, is left entirely to the system performing the binding.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A function invocation whose argument types are unbound is part of a partially bound plan;

One thing I'm still not sure about with what we're discussing here is how we would declare functions in plans that are consuming unbound arguments. From the example you gave

Project[add($0, $1)]
  Filter[gt($0, 1)]
    NamedTable("t", columns=["a"::unbound, "b"::unbound])

if we treat unbound as a type we would have the following signatures in the plan

  • gt:unbound_unbound
  • add:unbound_unbound

Even if we

took the direction from the sync that Substrait shouldn't prescribe a resolution policy.

As of today, neither of those are valid function signatures based on the current specification. substrait-java and substrait-go currently enforce function validity based on extension catalogs, so trying to convert a protobuf plan with these signatures would would fail because they wouldn't map to an existing function.

This is behaviour that we can change, but I think we need an have explicit mechanism to declare / identify unbound functions in the same way that we declare unbound columns. It sound like the plan is to look into this as a follow-up, which sounds reasonable to me, but until we do so I think that any plan that uses an unbound column in a function argument will be malformed.

Putting this another way: I'm not sure that we can re-use the existing mechanism to references functions with unbound types, because those mechanism assume concrete type arguments.

Caveat

unbound columns may work today with functions that consume only any arguments like eq:any_any.


Consumers that do not support partially bound plans should reject any plan containing `unbound`.

### Compound Types

Expand Down
2 changes: 2 additions & 0 deletions site/docs/types/type_system.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,3 +15,5 @@ Refer to [Type Parsing](type_parsing.md) for a description of the syntax used to

!!! note "Note"
Substrait employs a strict type system without any coercion rules. All changes in types must be made explicit via [cast expressions](../expressions/specialized_record_expressions.md).

Partially bound plans and expressions may use the [`unbound`](type_classes.md#unbound-type) type as a placeholder until a downstream binder assigns a concrete type.
14 changes: 14 additions & 0 deletions text/dialect_schema.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ properties:
- INTERVAL_DAY
- INTERVAL_YEAR
- UUID
- UNBOUND
- DECIMAL
- STRUCT
- LIST
Expand All @@ -78,6 +79,7 @@ properties:
- $ref: "#/definitions/type_interval_day"
- $ref: "#/definitions/type_interval_year"
- $ref: "#/definitions/type_uuid"
- $ref: "#/definitions/type_unbound"
- $ref: "#/definitions/type_decimal"
- $ref: "#/definitions/type_struct"
- $ref: "#/definitions/type_list"
Expand Down Expand Up @@ -640,6 +642,18 @@ definitions:
description: Arbitrary data created by the dialect author.
system_metadata:
$ref: "#/definitions/system_type_metadata"
type_unbound:
type: object
additionalProperties: false
required: [type]
properties:
type:
const: UNBOUND
metadata:
type: object
description: Arbitrary data created by the dialect author.
system_metadata:
$ref: "#/definitions/system_type_metadata"
type_decimal:
type: object
additionalProperties: false
Expand Down