diff --git a/site/docs/expressions/dynamic_parameters.md b/site/docs/expressions/dynamic_parameters.md index ec7689585..2f3d9751c 100644 --- a/site/docs/expressions/dynamic_parameters.md +++ b/site/docs/expressions/dynamic_parameters.md @@ -10,3 +10,133 @@ A dynamic parameter expression includes the following properties: |-----------------------|-------------------------------------------------------------------------------|----------| | `type` | Specifies the expected data type of the dynamic parameter. | Yes | | `parameter_reference` | A surrogate key used within a plan to reference a specific parameter binding. | Yes | + +## Parameter Bindings in Plans + +Dynamic parameters are referenced in expressions using a `parameter_reference` anchor. The actual values for these parameters are provided at the plan level using the `parameter_bindings` field in the `Plan` message. + +### DynamicParameterBinding + +Each binding maps a parameter anchor to a concrete literal value: + +```protobuf +Plan { + parameter_bindings: [ + DynamicParameterBinding { + parameter_anchor: 1 + value: Literal { i32: 100 } + }, + DynamicParameterBinding { + parameter_anchor: 2 + value: Literal { string: "example" } + } + ] + relations: [ + // Relations containing DynamicParameter expressions with parameter_reference 1 and 2 + ] +} +``` + +### Properties + +| Property | Description | Required | +|----------|-------------|----------| +| `parameter_anchor` | The anchor that identifies the dynamic parameter reference (must match a `parameter_reference` in a `DynamicParameter` expression) | Yes | +| `value` | The literal value assigned to the parameter at runtime. The type of the literal must match the type of the corresponding `DynamicParameter` expression. | Yes | + +## Use Cases + +### Parameterized Queries + +Dynamic parameters enable the same plan to be used with different input values: + +```sql +-- SQL query +SELECT * FROM users WHERE age > ? AND city = ? + +-- Substrait plan uses DynamicParameter expressions with parameter_reference 1 and 2 +-- Different executions provide different parameter_bindings +``` + +**Execution 1:** +```protobuf +parameter_bindings: [ + { parameter_anchor: 1, value: { i32: 25 } }, + { parameter_anchor: 2, value: { string: "New York" } } +] +``` + +**Execution 2:** +```protobuf +parameter_bindings: [ + { parameter_anchor: 1, value: { i32: 30 } }, + { parameter_anchor: 2, value: { string: "San Francisco" } } +] +``` + +### Plan Sharing Without Sensitive Data + +Plans can be shared without embedding sensitive values: + +```sql +-- SQL query with sensitive value +SELECT * FROM accounts WHERE ssn = '123-45-6789' + +-- Substrait plan uses DynamicParameter instead of embedding SSN +-- The actual SSN value is provided separately via parameter_bindings +``` + +This allows the plan to be: +- Cached and reused +- Logged without exposing sensitive data +- Shared across system boundaries without security concerns + +## Validation + +When using dynamic parameters, consumers must validate: + +1. **Type Matching**: The type of the literal in `parameter_bindings` must match the type specified in the `DynamicParameter` expression +2. **Completeness**: All `parameter_reference` values used in expressions must have corresponding bindings in `parameter_bindings` +3. **Uniqueness**: Each `parameter_anchor` should appear at most once in `parameter_bindings` + +## Example + +Complete example showing dynamic parameters in a filter expression: + +```protobuf +Plan { + parameter_bindings: [ + DynamicParameterBinding { + parameter_anchor: 100 + value: Literal { i32: 42 } + } + ] + relations: [ + PlanRel { + root: RelRoot { + input: Rel { + filter: FilterRel { + input: Rel { read: ReadRel { ... } } + condition: Expression { + scalar_function: ScalarFunction { + function_reference: 0 // greater_than function + arguments: [ + Expression { selection: FieldReference { ... } }, + Expression { + dynamic_parameter: DynamicParameter { + type: { i32: { nullability: REQUIRED } } + parameter_reference: 100 + } + } + ] + } + } + } + } + } + } + ] +} +``` + +In this example, the filter condition compares a field to a dynamic parameter with anchor 100, which is bound to the value 42. diff --git a/site/docs/relations/logical_relations.md b/site/docs/relations/logical_relations.md index c2317a0ca..17ea8b20d 100644 --- a/site/docs/relations/logical_relations.md +++ b/site/docs/relations/logical_relations.md @@ -457,6 +457,17 @@ The write operator is an operator that consumes one input and writes it to stora | Create Mode | This determines what should happen if the table already exists (ERROR/REPLACE/IGNORE) | Required only for CTAS | | Output Mode | For views that modify a DB it is important to control which records to "return". Common default is NO_OUTPUT where we return nothing. Alternatively, we can return MODIFIED_RECORDS, that can be further manipulated by layering more rels ontop of this WriteRel (e.g., to "count how many records were updated"). This also allows to return the after-image of the change. To return before-image (or both) one can use the reference mechanisms and have multiple return values. | Required for VIEW CREATE/CREATE_OR_REPLACE/ALTER | +### CreateMode Values + +When using CTAS (CREATE TABLE AS), the CreateMode determines what happens if the table already exists: + +| Mode | Value | Description | +|---------------------------|-------|------------------------------------------------------------------------------| +| CREATE_MODE_UNSPECIFIED | 0 | Behavior is unspecified and should not be used | +| CREATE_MODE_APPEND_IF_EXISTS | 1 | If the table exists, append the new data to it | +| CREATE_MODE_REPLACE_IF_EXISTS | 2 | If the table exists, replace it entirely (equivalent to DROP + CREATE) | +| CREATE_MODE_IGNORE_IF_EXISTS | 3 | If the table exists, do nothing (equivalent to "IF NOT EXISTS") | +| CREATE_MODE_ERROR_IF_EXISTS | 4 | If the table exists, throw an error (default behavior) | ### Write Definition Types diff --git a/site/docs/relations/physical_relations.md b/site/docs/relations/physical_relations.md index 51ecc27ea..407c79a56 100644 --- a/site/docs/relations/physical_relations.md +++ b/site/docs/relations/physical_relations.md @@ -23,12 +23,48 @@ The hash equijoin join operator will build a hash table out of one input (defaul |---------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------| | Left Input | A relational input. | Required | | Right Input | A relational input. | Required | -| Build Input | Specifies which input is the `Build`. | Optional, defaults to build `Right`, probe `Left`. | +| Build Input | Specifies which input side to build the hash table from. See [Build Input Details](#build-input-details) below. | Optional, defaults to `BUILD_INPUT_RIGHT` | | Left Keys | References to the fields to join on in the left input. | Required | | Right Keys | References to the fields to join on in the right input. | Required | | Post Join Predicate | An additional expression that can be used to reduce the output of the join operation post the equality condition. Minimizes the overhead of secondary join conditions that cannot be evaluated using the equijoin keys. | Optional, defaults true. | | Join Type | One of the join types defined in the Join operator. | Required | +### Build Input Details + +The `build_input` field specifies which side of the join to use for building the hash table. The hash join algorithm consists of two phases: + +1. **Build Phase**: Read one input and build a hash table on the join keys +2. **Probe Phase**: Read the other input and probe the hash table for matching records + +The choice of build side can significantly impact performance: + +| Value | Description | When to Use | +|-------|-------------|-------------| +| BUILD_INPUT_UNSPECIFIED | Behavior is unspecified; consumers may choose either side | Default behavior, typically builds on right | +| BUILD_INPUT_LEFT | Build hash table from left input, probe with right | When left input is smaller than right | +| BUILD_INPUT_RIGHT | Build hash table from right input, probe with left | When right input is smaller than left (common case) | + +**Performance Considerations:** + +* The build side should typically be the smaller input to minimize memory usage +* The build side must fit entirely in memory (or be spilled to disk) +* For very large builds, memory pressure may cause degraded performance +* Some join types have natural build side preferences: + - **Left Semi/Anti Join**: Build on right is more efficient + - **Right Semi/Anti Join**: Build on left is more efficient + - **Inner/Outer Joins**: Build on smaller side + +**Example:** +``` +HashJoinRel { + left: scan_large_table // 1M rows + right: scan_small_table // 10K rows + build_input: BUILD_INPUT_RIGHT // Build hash table from smaller right side + keys: [...] + type: INNER +} +``` + ## NLJ (Nested Loop Join) Operator diff --git a/site/docs/spec/_config b/site/docs/spec/_config index 77dde9a2b..607f5adcb 100644 --- a/site/docs/spec/_config +++ b/site/docs/spec/_config @@ -3,3 +3,4 @@ arrange: - specification.md - technology_principles.md - extending.md + - dialects.md diff --git a/site/docs/spec/dialects.md b/site/docs/spec/dialects.md new file mode 100644 index 000000000..66c826c2c --- /dev/null +++ b/site/docs/spec/dialects.md @@ -0,0 +1,311 @@ +# Substrait Dialects + +## Overview + +Substrait dialects provide a standardized way to describe what subset of the Substrait specification a particular system supports. A dialect file codifies system-specific behaviors, supported types, functions, and options, enabling better interoperability and validation between systems. + +Dialects serve multiple purposes: + +* **Capability Declaration**: Systems can declare which Substrait features they support +* **Compatibility Validation**: Producers can validate plans against a consumer's capabilities before sending +* **Documentation**: Dialects serve as machine-readable documentation of system capabilities +* **Testing**: Enable automated testing of system conformance to declared capabilities + +## Dialect File Format + +Dialect files are YAML documents that describe a system's Substrait support. The basic structure includes: + +```yaml +name: system_name +type: sql # or other system type +dependencies: + extension_name: extension_uri_or_path +supported_types: + - type definitions +scalar_functions: + - function definitions +aggregate_functions: + - function definitions +``` + +### Core Fields + +| Field | Description | Required | +|-------|-------------|----------| +| `name` | The name of the dialect (e.g., `duckdb`, `postgres`) | Yes | +| `type` | The type of system (e.g., `sql`) | No | +| `dependencies` | Extension URIs that this dialect depends on | No | +| `supported_types` | List of Substrait types supported by the system | No | +| `scalar_functions` | Scalar functions supported | No | +| `aggregate_functions` | Aggregate functions supported | No | + +## Supported Types + +The `supported_types` section declares which Substrait types are supported by the system. Types can be specified in simple or detailed format. + +### Simple Format + +```yaml +supported_types: + - I8 + - I16 + - I32 + - I64 + - FP32 + - FP64 + - STRING +``` + +### Detailed Format + +```yaml +supported_types: + i8: + sql_type_name: tinyint + i32: + sql_type_name: integer + supported_as_column: true + user_defined: + source: geo # reference to dependency + name: geometry +``` + +The detailed format allows specifying: + +* `sql_type_name`: The SQL type name used by the system (for SQL dialects) +* `supported_as_column`: Whether the type can be used as a column type +* `source`: For user-defined types, references a dependency extension +* `name`: The specific type name from the extension + +## Function Support + +Functions are specified with their Substrait name and can include system-specific customization. + +### Basic Function Declaration + +```yaml +scalar_functions: + - name: arithmetic.add + local_name: + + infix: true + supported_kernels: + - i8_i8 + - i16_i16 + - i32_i32 + - i64_i64 + - fp32_fp32 + - fp64_fp64 +``` + +### Function Fields + +| Field | Description | Required | +|-------|-------------|----------| +| `name` | The Substrait function name (e.g., `arithmetic.add`) | Yes | +| `local_name` | The system's native name for the function | No | +| `infix` | Whether the function uses infix notation (e.g., `+`) | No | +| `required_options` | Options that must be set to specific values | No | +| `supported_kernels` | List of type combinations (kernels) supported | No | +| `unsupported_kernels` | List of kernels explicitly not supported | No | + +### Required Options + +Some systems may require specific option values for functions: + +```yaml +scalar_functions: + - name: arithmetic.add + required_options: + overflow: ERROR + rounding: TIE_TO_EVEN +``` + +This indicates that for `arithmetic.add`, the system only supports the `ERROR` overflow behavior and `TIE_TO_EVEN` rounding mode. + +### Supported Kernels + +Kernels specify which type combinations are supported for a function: + +```yaml +scalar_functions: + - name: arithmetic.multiply + supported_kernels: + - i32_i32 # multiply(i32, i32) + - i64_i64 # multiply(i64, i64) + - fp32_fp32 # multiply(fp32, fp32) + - fp64_fp64 # multiply(fp64, fp64) + - dec_dec # multiply(decimal, decimal) +``` + +## Dependencies + +Dependencies declare which extension URIs the dialect relies on: + +```yaml +dependencies: + aggregate_generic: + https://github.com/substrait-io/substrait/blob/main/extensions/substrait/extensions/functions_aggregate_generic.yaml + arithmetic: + https://github.com/substrait-io/substrait/blob/main/extensions/substrait/extensions/functions_arithmetic.yaml + string: + https://github.com/substrait-io/substrait/blob/main/extensions/substrait/extensions/functions_string.yaml +``` + +Dependencies can be referenced using short names (like `arithmetic`) throughout the dialect file. + +## Complete Example + +Here's a simplified dialect file for a hypothetical system: + +```yaml +name: example_system +type: sql + +dependencies: + arithmetic: + https://github.com/substrait-io/substrait/blob/main/extensions/functions_arithmetic.yaml + comparison: + https://github.com/substrait-io/substrait/blob/main/extensions/functions_comparison.yaml + +supported_types: + i32: + sql_type_name: integer + i64: + sql_type_name: bigint + fp64: + sql_type_name: double + str: + sql_type_name: varchar + +scalar_functions: + - name: arithmetic.add + local_name: + + infix: true + required_options: + overflow: ERROR + supported_kernels: + - i32_i32 + - i64_i64 + - fp64_fp64 + + - name: arithmetic.subtract + local_name: '-' + infix: true + required_options: + overflow: ERROR + supported_kernels: + - i32_i32 + - i64_i64 + - fp64_fp64 + + - name: comparison.equal + local_name: = + infix: true + supported_kernels: + - any_any + +aggregate_functions: + - name: sum + supported_kernels: + - i32 + - i64 + - fp64 +``` + +## Use Cases + +### Plan Validation + +Before sending a plan to a consumer, a producer can validate it against the consumer's dialect: + +```python +# Pseudocode +plan = create_substrait_plan() +consumer_dialect = load_dialect("duckdb.yaml") + +if not validate_plan_against_dialect(plan, consumer_dialect): + print("Plan uses features not supported by DuckDB") +``` + +### Feature Discovery + +A system can query a dialect to determine capabilities: + +```python +# Pseudocode +dialect = load_dialect("postgres.yaml") + +if dialect.supports_function("arithmetic.add", ["i32", "i32"]): + # Use add function +else: + # Use alternative approach +``` + +### Testing + +Dialects enable automated testing: + +```python +# Pseudocode +for dialect in discover_dialects(): + for function in dialect.scalar_functions: + test_function_conformance(dialect, function) +``` + +## Available Dialects + +Several system dialects are maintained as part of the Substrait ecosystem: + +* **DuckDB** - An in-process SQL OLAP database +* **DataFusion** - Apache Arrow-native query engine +* **PostgreSQL** - Popular relational database +* **SQLite** - Lightweight embedded database +* **Snowflake** - Cloud data warehouse +* **Velox/Presto** - High-performance vectorized execution engine +* **cuDF** - GPU-accelerated DataFrame library + +Dialect files can be found in the [BFT (Basic Function Tests) dialect directory](https://github.com/substrait-io/substrait/tree/main/bft/dialects). + +## Creating a Custom Dialect + +To create a dialect for your system: + +1. **Start with a template**: Copy an existing dialect file that's similar to your system +2. **Update basic information**: Change the `name` field to your system name +3. **Declare dependencies**: List all extension URIs your system uses +4. **Specify supported types**: List all Substrait types you support +5. **Document functions**: For each function, specify: + - The Substrait function name + - Your system's local name (if different) + - Required option values + - Supported type kernels +6. **Test the dialect**: Use the dialect testing framework to validate +7. **Contribute back**: Consider contributing your dialect to the Substrait repository + +## Dialect Testing + +Substrait provides testing infrastructure for dialects to ensure: + +* Syntax validity of the dialect YAML +* Consistency between declared capabilities and actual support +* Proper reference to extension URIs +* Valid type and function declarations + +Test files can be found in the `substrait/dialects/tests/` directory. + +## Best Practices + +1. **Be specific about kernels**: Rather than declaring all possible kernels, list only those you actually support +2. **Document required options**: If your system only supports specific option values, declare them +3. **Keep dialects updated**: Update your dialect file when you add or remove support for features +4. **Version your dialects**: Consider versioning dialect files alongside system versions +5. **Test thoroughly**: Use the dialect testing framework to validate your declarations + +## Related Concepts + +* [Extensions](../extensions/index.md) - Dialect files reference extension URIs +* [Simple Extensions](../extensions/index.md#simple-extensions) - Function and type extensions +* [Type System](../types/type_system.md) - Understanding Substrait types +* [Scalar Functions](../expressions/scalar_functions.md) - Function semantics + + diff --git a/site/docs/types/type_aliases.md b/site/docs/types/type_aliases.md index 04ec0d52d..1178533dd 100644 --- a/site/docs/types/type_aliases.md +++ b/site/docs/types/type_aliases.md @@ -4,7 +4,7 @@ In a Substrait plan, types are spelled out whenever and wherever they are needed To alleviate the problem, Substrait offers a type alias mechanism. -Type aliases allow a plan to declare a type once and reference it multiple times within a plan. A type alias can be used wherever a type is expected. +Type aliases allow a plan to declare a type once and reference it multiple times within a plan. A type alias can be used wherever a type is expected. Type aliases are scoped to a single plan and are defined in the `Plan.type_aliases` field. ## Type Alias @@ -32,4 +32,72 @@ type alias 5 --> STRUCT // NOT OK to reference type alias 6 --> STRUCT STRUCT type alias 8 --> STRUCT // NOT OK because type alias 7 and 8 have a circular dependency. -``` \ No newline at end of file +``` + +## Using Type Aliases in Plans + +Type aliases are defined at the plan level using the `type_aliases` field in the `Plan` message. Each type alias has: + +* **Anchor**: A surrogate key (uint32) used to reference the alias within the plan +* **Type**: The concrete Substrait type being aliased + +### Example Plan with Type Aliases + +```protobuf +Plan { + type_aliases: [ + TypeAlias { + type_alias_anchor: 1 + type: VarChar { length: 100, nullability: REQUIRED } + }, + TypeAlias { + type_alias_anchor: 2 + type: Struct { + types: [ + Type { i32: { nullability: REQUIRED } }, + Type { alias: { type_alias_reference: 1, nullability: NULLABLE } } + ] + nullability: REQUIRED + } + } + ] + relations: [ + // Relations can now reference type aliases using TypeAliasReference + ] +} +``` + +In this example: +- Type alias 1 defines `VARCHAR<100>` that can be reused throughout the plan +- Type alias 2 defines a struct that itself references type alias 1 +- Both aliases can be referenced using `TypeAliasReference` wherever a type is expected + +### Referencing Type Aliases + +To reference a type alias, use a `TypeAliasReference` type: + +```protobuf +Type { + alias: TypeAliasReference { + type_alias_reference: 1 // References the alias with anchor 1 + nullability: NULLABLE // Nullability must be specified + } +} +``` + +The nullability in the `TypeAliasReference` determines the nullability of the referenced type at the point of use, overriding any nullability specified in the type alias definition itself. + +## Benefits + +Type aliases provide several benefits: + +1. **Reduced Plan Size**: Complex types are defined once and referenced multiple times +2. **Improved Readability**: Named types make plans easier to understand +3. **Consistency**: Ensures the same type definition is used throughout the plan +4. **Maintainability**: Changes to a type only need to be made in one place + +Type aliases are particularly useful for: +- User-defined types with many parameters +- Deeply nested struct types +- Types with string parameters (e.g., long VARCHAR lengths) +- Frequently reused complex types \ No newline at end of file