Skip to content
1 change: 1 addition & 0 deletions proto/substrait/extensions/extensions.proto
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ message SimpleExtensionURN {

// The extension URN that uniquely identifies this extension. This must follow the
// format extension:<OWNER>:<ID> and serves as the "namespace" of this extension.
// This must conform to the following regex: ^extension:[^:]+:[^:]+$
Comment thread
benbellick marked this conversation as resolved.
Outdated
string urn = 2;
}

Expand Down
4 changes: 3 additions & 1 deletion site/docs/extensions/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,15 @@ Some kinds of primitives are so frequently extended that Substrait defines a sta
* Window Functions
* Table Functions

To extend these items, developers can create one or more YAML files that describe the properties of each of these extensions. Each YAML file must include a required `urn` field that uniquely identifies the extension. While these identifiers are URN-like but not technically URNs (they lack the `urn:` prefix), they will be referred to as `extension URNs` for clarity.
To extend these items, developers can create one or more YAML files that describe the properties of each of these extensions. Each YAML file must include a required `urn` field that uniquely identifies the extension. While these identifiers are URN-like but not technically URNs (they lack the `urn:` prefix), and they will be referred to as `extension URNs` for clarity.

This extension URN uses the format `extension:<OWNER>:<ID>`, where:

- `OWNER` represents the organization or entity providing the extension and should follow [reverse domain name convention](https://en.wikipedia.org/wiki/Reverse_domain_name_notation) (e.g., `io.substrait`, `com.example`, `org.apache.arrow`) to prevent name collisions
- `ID` is the specific identifier for the extension (e.g., `functions_arithmetic`, `custom_types`)

These URNs must match the regex `^extension:[a-zA-Z0-9_.-]+:[a-zA-Z0-9_.-]+$`.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that I see it again, why do we allow upper case if we were to allow only narrow set of characters?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you recommend an even more restrictive urn? How about:
^extension:[a-z0-9_.-]+:[a-z0-9_.-]+$

i.e same thing without capital letters.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I updated it to be without capital letters, let me know what you think!


The YAML file is constructed according to the [YAML Schema](https://github.com/substrait-io/substrait/blob/main/text/simple_extensions_schema.yaml). Each definition in the file corresponds to the YAML-based serialization of the relevant data structure. If a user only wants to extend one of these types of objects (e.g. types), a developer does not have to provide definitions for the other extension points.

A Substrait plan can reference one or more YAML files via their extension URN. In the places where these entities are referenced, they will be referenced using an extension URN + name reference. The name scheme per type works as follows:
Expand Down
2 changes: 1 addition & 1 deletion site/docs/serialization/binary_serialization.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ For simple extensions, a plan references the extension URNs associated with the

Simple extensions within a plan are split into three components: an extension URN, an extension declaration and a number of references.

* **Extension URN**: A unique identifier for the extension following the format `extension:<OWNER>:<ID>` that identifies a YAML document specifying one or more specific extensions. Declares an anchor that can be used in extension declarations.
* **Extension URN**: A unique identifier for the extension following the format `extension:<OWNER>:<ID>` that identifies a YAML document specifying one or more specific extensions. Declares an anchor that can be used in extension declarations. The extension URN must conform to the regex `^extension:[^:]+:[^:]+$`.
* **Extension Declaration**: A specific extension within a single YAML document. The declaration combines a reference to the associated extension URN along with a unique key identifying the specific item within that YAML document (see [Function Signature](../extensions/index.md#function-signature)). It also defines a declaration anchor. The anchor is a plan-specific unique value that the producer creates as a key to be referenced elsewhere.
* **Extension Reference**: A specific instance or use of an extension declaration within the plan body.

Expand Down
1 change: 1 addition & 0 deletions text/simple_extensions_schema.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ required: [urn]
properties:
urn:
type: string
pattern: "^extension:[a-zA-Z0-9_.-]+:[a-zA-Z0-9_.-]+$"
dependencies:
# For reusing type classes and type variations from other extension files.
# The keys are namespace identifiers that you can then use as dot-separated
Expand Down
Loading