From 3b9cbc561067cb693fcb57e3c0f49ec8b867ae7f Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Mon, 26 Jun 2023 10:22:18 +0200 Subject: [PATCH 01/70] Intermediate progress writing property ranges parameter. --- optimade.rst | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-) diff --git a/optimade.rst b/optimade.rst index b91b89d43..7870af3a2 100644 --- a/optimade.rst +++ b/optimade.rst @@ -459,6 +459,9 @@ The default partial data format is named "jsonlines" and is described in the App An implementation SHOULD always include this format as one of the partial data formats provided for a property that has been omitted from the response to the initial query. Implementations MAY provide links to their own non-standard formats, but non-standard format names MUST be prefixed by a database-provider-specific prefix. +To enable requesting only a part of the data for such properties via the :param:`property\_ranges` query parameter, there is additional meta data stored under the field :field:`meta..range`. +The metadata field of the ranged property, :field:`meta..range`, MUST include these fields: + Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format that communicates that the property value has been omitted from the response, with three different links for different partial data formats provided. .. code:: jsonc @@ -493,6 +496,7 @@ Below follows an example of the :field:`data` and :field:`meta` parts of a respo // ... } + Metadata properties ------------------- @@ -1077,6 +1081,30 @@ Standard OPTIONAL URL query parameters not in the JSON:API specification: If provided, these fields MUST be returned along with the REQUIRED fields. Other OPTIONAL fields MUST NOT be returned when this parameter is present. Example: :query-url:`http://example.com/optimade/v1/structures?response_fields=last_modified,nsites` +- **property\_ranges**: A set of slices for partial data properties. + This parameter is used to select a subrange of partial data properties so only a part of the data needs to be returned. + In general support is OPTIONAL, property definitions may however deviate from this and place stricter requirements on servers. + It consists of the name of a dimension directly followed by the requested slice in this dimension. + A slice consists of a pair of brackets ("(", ASCII 40(0x28)) and (")", ASCII 41(0x29)) enclosing three integers, which are separated by commas (",", ASCII 91(0x5B)) + It is defined in the same manner as the `slice object`_. + The first integer specifies the first index in that dimension for which values should be returned. + The second integer specifies the last index for which values should be returned. + The third integer specifies the step size in that dimension. + Slices can be specified for multiple dimensions by separating them with a comma. + The slices are 0 based, i.e. the first value has index 0, and inclusive i.e. for the range :val:`(10,20,1)` the last value returned belongs to index 20. + Databases MUST use these ranges for properties where the dimension is listed under indexable_dimensions, if this is not the case the database MAY return more data than was specified in the range. + + If a dimension is not specified, it is assumed that the whole range in that dimension is requested. + If a value is not present at a set of the indexes, no value SHOULD be returned. + However, when a value is explicitly set to :val:`null` and :val:`null` has a meaning beyond indicating that no value has been defined :val:`null` MUST be returned. + Incase the requested property_range covers more data than the server wants to return the server may choose to return only a part of the data. + For each combination of indexes for which data is returned all the values for all requested properties however need to be returned. + If the server does not return all the requested data, a link MUST be provided in the :field:`next` field, that applies to an entry as a whole, from which the remainder of the data can be retrieved. + + + + + Additional OPTIONAL URL query parameters not described above are not considered to be part of this standard, and are instead considered to be "custom URL query parameters". These custom URL query parameters MUST be of the format "". @@ -3621,7 +3649,7 @@ The strings below contain Extended Regular Expressions (EREs) to recognize ident #BEGIN ERE identifiers [a-z_][a-z_0-9]* #END ERE identifiers - +== #BEGIN ERE numbers [-+]?([0-9]+(\.[0-9]*)?|\.[0-9]+)([eE][-+]?[0-9]+)? #END ERE numbers From eae167938a1c10bf3722ea754960123192340b0d Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Tue, 27 Jun 2023 15:12:12 +0200 Subject: [PATCH 02/70] Added metadata fields to be able to use property ranges query parameter. --- optimade.rst | 50 +++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 47 insertions(+), 3 deletions(-) diff --git a/optimade.rst b/optimade.rst index 7870af3a2..30f8fb062 100644 --- a/optimade.rst +++ b/optimade.rst @@ -459,10 +459,34 @@ The default partial data format is named "jsonlines" and is described in the App An implementation SHOULD always include this format as one of the partial data formats provided for a property that has been omitted from the response to the initial query. Implementations MAY provide links to their own non-standard formats, but non-standard format names MUST be prefixed by a database-provider-specific prefix. -To enable requesting only a part of the data for such properties via the :param:`property\_ranges` query parameter, there is additional meta data stored under the field :field:`meta..range`. -The metadata field of the ranged property, :field:`meta..range`, MUST include these fields: +If the server supports the :query-param:`property_ranges` query parameter, as described in section `Entry Listing URL Query Parameters`_ additional metadata needs to be present under :field:`property_metadata` field, for each property that is returned in the partial data format. +This meta data allows clients to request specific subsections of properties. -Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format that communicates that the property value has been omitted from the response, with three different links for different partial data formats provided. +If the field can be sliced with the :query-param:`property_ranges`, the following fields MUST be defined under :field:`property_metadata` Otherwise implementation is OPTIONAL: + +- :field:`range_ids`: List of Strings. + A list with an identifier for each dimension of the property. + The outermost dimension of a nested array should come first. + If, within one entry, dimensions for two or more properties share the same :field:`range_id` those dimensions should be thought of as the same dimension. + For example, both the :property:`energy` and :property:`cartesian_site_positions` of a molecular dynamics trajectory share a range_id of :val:`frames`. + This means that the energy at index x(in the dimension labelled by this range_id) belongs to the cartesian_site_positions at the same index x. + +- :field:`indexable_dim`: List of Strings. + The list of range_ids of the dimensions for which slicing is supported, i.e. the client can request a subrange via the :query-param:`property_ranges` query parameter. + +- :field:`data_range`: List of Slice Objects. + This field describes how the values are distributed in the different dimensions. + It consists of a slice object for each dimension of the property. + The order of the slice objects must be the same as in the :field:`range_ids` field. + +- :field:`nvalues`: Integer. + The total number of values in the property. + SHOULD be a queryable property with support for all mandatory filter features. + +- :field:`"layout"`: String. + A string either equal to :val:`"dense"` or :val:`"sparse"` to indicate whether the property is returned in a dense or sparse layout. + +Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format. It communicates that the property value has been omitted from the response and includes three different links for different partial data formats provided and metadata that makes it possible to only request only a part of the data for this property. .. code:: jsonc @@ -475,6 +499,25 @@ Below follows an example of the :field:`data` and :field:`meta` parts of a respo "a": null } "meta": { + "property_metadata":{ + "a":{ + "range":{ + "range_ids":["frames","particles"], + "indexable_dim": ["frames"], + "data_range": [{ + "start": 1, + "step": 1, + "stop": 200, + },{ + "start": 1, + "step": 1, + "stop": 3, + }], + "format":"1.2", + "layout":"dense", + } + } + }, "partial_data_links": { "a": [ { @@ -3680,6 +3723,7 @@ The dictionary has the following OPTIONAL fields: The default is 1, i.e., every value in the range indicated by :field:`start` and :field:`stop` is included in the slice. Hence, a value of 2 denotes a slice of every second value in the array. + For example, for the array :val:`["a", "b", "c", "d", "e", "f", "g", "h", "i", "j"]` the slice object :val:`{"start": 1, "end": 7, "step": 3}` refers to the items :val:`["b", "e", "h"]`. Furthermore, we also define the following special markers: From f00b52daf4693aba3348d3db8db6c98183cc0597 Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Wed, 28 Jun 2023 15:36:55 +0200 Subject: [PATCH 03/70] Small corrections. --- optimade.rst | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/optimade.rst b/optimade.rst index 30f8fb062..810aa18cf 100644 --- a/optimade.rst +++ b/optimade.rst @@ -462,29 +462,34 @@ Implementations MAY provide links to their own non-standard formats, but non-sta If the server supports the :query-param:`property_ranges` query parameter, as described in section `Entry Listing URL Query Parameters`_ additional metadata needs to be present under :field:`property_metadata` field, for each property that is returned in the partial data format. This meta data allows clients to request specific subsections of properties. -If the field can be sliced with the :query-param:`property_ranges`, the following fields MUST be defined under :field:`property_metadata` Otherwise implementation is OPTIONAL: +If the field can be sliced with the :query-param:`property_ranges`, the following fields MUST be defined under :field:`property_metadata`, otherwise implementation is OPTIONAL. +Except for the `nvalues` field, the queryability of all fields is OPTIONAL. - :field:`range_ids`: List of Strings. A list with an identifier for each dimension of the property. The outermost dimension of a nested array should come first. + If, within one entry, dimensions for two or more properties share the same :field:`range_id` those dimensions should be thought of as the same dimension. - For example, both the :property:`energy` and :property:`cartesian_site_positions` of a molecular dynamics trajectory share a range_id of :val:`frames`. + For example, if both the :property:`energy` and :property:`cartesian_site_positions` of a molecular dynamics trajectory share a range_id of :val:`frames`. This means that the energy at index x(in the dimension labelled by this range_id) belongs to the cartesian_site_positions at the same index x. - :field:`indexable_dim`: List of Strings. - The list of range_ids of the dimensions for which slicing is supported, i.e. the client can request a subrange via the :query-param:`property_ranges` query parameter. + The list of range_ids of the dimensions for which slicing is supported, i.e. the client can request a subrange in this dimension via the :query-param:`property_ranges` query parameter. + +- :field:`"layout"`: String. + A string either equal to :val:`"dense"` or :val:`"sparse"` to indicate whether the property is returned in a dense or sparse layout. - :field:`data_range`: List of Slice Objects. This field describes how the values are distributed in the different dimensions. It consists of a slice object for each dimension of the property. The order of the slice objects must be the same as in the :field:`range_ids` field. + If the :field:`layout` field is set to :val:`"sparse"` the value of the :field:`step` sub-field has no meaning. - :field:`nvalues`: Integer. The total number of values in the property. SHOULD be a queryable property with support for all mandatory filter features. -- :field:`"layout"`: String. - A string either equal to :val:`"dense"` or :val:`"sparse"` to indicate whether the property is returned in a dense or sparse layout. + Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format. It communicates that the property value has been omitted from the response and includes three different links for different partial data formats provided and metadata that makes it possible to only request only a part of the data for this property. @@ -513,7 +518,6 @@ Below follows an example of the :field:`data` and :field:`meta` parts of a respo "step": 1, "stop": 3, }], - "format":"1.2", "layout":"dense", } } From 2a0bef8770fe17863ae134defe6f0d950a8160c2 Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Thu, 29 Jun 2023 14:07:16 +0200 Subject: [PATCH 04/70] moved property ranges to single_entryendpoint query param + further refinement text --- optimade.rst | 51 +++++++++++++++++++++++++++------------------------ 1 file changed, 27 insertions(+), 24 deletions(-) diff --git a/optimade.rst b/optimade.rst index 810aa18cf..4dc9d4698 100644 --- a/optimade.rst +++ b/optimade.rst @@ -1128,30 +1128,6 @@ Standard OPTIONAL URL query parameters not in the JSON:API specification: If provided, these fields MUST be returned along with the REQUIRED fields. Other OPTIONAL fields MUST NOT be returned when this parameter is present. Example: :query-url:`http://example.com/optimade/v1/structures?response_fields=last_modified,nsites` -- **property\_ranges**: A set of slices for partial data properties. - This parameter is used to select a subrange of partial data properties so only a part of the data needs to be returned. - In general support is OPTIONAL, property definitions may however deviate from this and place stricter requirements on servers. - It consists of the name of a dimension directly followed by the requested slice in this dimension. - A slice consists of a pair of brackets ("(", ASCII 40(0x28)) and (")", ASCII 41(0x29)) enclosing three integers, which are separated by commas (",", ASCII 91(0x5B)) - It is defined in the same manner as the `slice object`_. - The first integer specifies the first index in that dimension for which values should be returned. - The second integer specifies the last index for which values should be returned. - The third integer specifies the step size in that dimension. - Slices can be specified for multiple dimensions by separating them with a comma. - The slices are 0 based, i.e. the first value has index 0, and inclusive i.e. for the range :val:`(10,20,1)` the last value returned belongs to index 20. - Databases MUST use these ranges for properties where the dimension is listed under indexable_dimensions, if this is not the case the database MAY return more data than was specified in the range. - - If a dimension is not specified, it is assumed that the whole range in that dimension is requested. - If a value is not present at a set of the indexes, no value SHOULD be returned. - However, when a value is explicitly set to :val:`null` and :val:`null` has a meaning beyond indicating that no value has been defined :val:`null` MUST be returned. - Incase the requested property_range covers more data than the server wants to return the server may choose to return only a part of the data. - For each combination of indexes for which data is returned all the values for all requested properties however need to be returned. - If the server does not return all the requested data, a link MUST be provided in the :field:`next` field, that applies to an entry as a whole, from which the remainder of the data can be retrieved. - - - - - Additional OPTIONAL URL query parameters not described above are not considered to be part of this standard, and are instead considered to be "custom URL query parameters". These custom URL query parameters MUST be of the format "". @@ -1242,6 +1218,33 @@ The rules for which properties are to be present for an entry in the response ar Single Entry URL Query Parameters ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +- **property\_ranges**: A set of slices which specify which range of a property should be returned for partial data properties. + The property to which this range should be applied can be specified via the :query-param:`response_fields` query parameter. + Each slice consists of the name of a dimension directly followed by the requested slice in this dimension. + The dimension name and the start, stop and step values of the slice are separated by colons (":", ASCII 58(0x3A)) + The slice is defined in the same manner as the `slice object`_. + If no value is placed between the colons for a component of the slice the default value is used. + The first integer specifies the start of the slice, i.e. the first index in that dimension for which values should be returned. The default is the index of the first value. + The second integer specifies the end/stop of the slice, i.e. the last index for which values should be returned. The default is the index of the last value of the property. + And the third integer specifies the step size in that dimension. The default is :val:`1` + Multiple dimensional slices can be defined by specifying a range for each dimension. These ranges are separated by a comma (",", ASCII 44(0x2C)). + The slices are 0 based, i.e. the first value has index 0, and inclusive i.e. for the range :val:`:10:20:1` the last value returned belongs to index 20. + + In general support is OPTIONAL, property definitions may however deviate from this and place stricter requirements on servers. + Databases must use the methods described in the section `Transmission of large property values`_ to return the requested sub-range of a property. +Databases MUST use these ranges for properties where the dimension is listed under indexable_dimensions, if this is not the case the database MAY return more data than was specified in the range. + + If a dimension is not specified, it is assumed that the whole range in that dimension is requested. + If one or more values are not present at one of the requested combination of indexes, the server MAY either decide to return null or if possible adjust the returned range so the indexes for which no value is defined are no longer part of the range. + The latter is only allowed when no defined values would be lost. + For dense arrays that may mean that the field :field:`returned_range` differs from the requested range. + When a value is, however, explicitly set to :val:`null` and :val:`null` has a meaning beyond indicating that no value has been defined :val:`null` MUST be returned + + Example: In this example the Cartesian site positions are requested for particles 30 through 70 for 1 out of every 10 frames of the first 1000 frames of a trajectory. + +:query-url:`http://example.com/optimade/v1/structures/id_12345?response_fields=cartesian_site_positions&property_ranges=frames::1000:10,particles:30:70::` + + The client MAY provide a set of additional URL query parameters for this endpoint type. URL query parameters not recognized MUST be ignored. While the following URL query parameters are OPTIONAL for clients, API implementations MUST accept and handle them: From 4a00af4d4f1005576bb334142768e3ee6ba2ac4b Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Thu, 29 Jun 2023 18:21:24 +0200 Subject: [PATCH 05/70] Some small changes after proof reading. --- optimade.rst | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/optimade.rst b/optimade.rst index 4dc9d4698..b93221ac4 100644 --- a/optimade.rst +++ b/optimade.rst @@ -459,8 +459,8 @@ The default partial data format is named "jsonlines" and is described in the App An implementation SHOULD always include this format as one of the partial data formats provided for a property that has been omitted from the response to the initial query. Implementations MAY provide links to their own non-standard formats, but non-standard format names MUST be prefixed by a database-provider-specific prefix. -If the server supports the :query-param:`property_ranges` query parameter, as described in section `Entry Listing URL Query Parameters`_ additional metadata needs to be present under :field:`property_metadata` field, for each property that is returned in the partial data format. -This meta data allows clients to request specific subsections of properties. +If the server supports the :query-param:`property_ranges` query parameter, as described in section `Single Entry URL Query Parameters`_, additional metadata SHOULD be present in the property specific field under the :field:`property_metadata` field, for each property that is returned in the partial data format. +This metadata allows clients to request specific subsections of properties. If the field can be sliced with the :query-param:`property_ranges`, the following fields MUST be defined under :field:`property_metadata`, otherwise implementation is OPTIONAL. Except for the `nvalues` field, the queryability of all fields is OPTIONAL. @@ -468,7 +468,7 @@ Except for the `nvalues` field, the queryability of all fields is OPTIONAL. - :field:`range_ids`: List of Strings. A list with an identifier for each dimension of the property. The outermost dimension of a nested array should come first. - + If, within one entry, dimensions for two or more properties share the same :field:`range_id` those dimensions should be thought of as the same dimension. For example, if both the :property:`energy` and :property:`cartesian_site_positions` of a molecular dynamics trajectory share a range_id of :val:`frames`. This means that the energy at index x(in the dimension labelled by this range_id) belongs to the cartesian_site_positions at the same index x. @@ -519,6 +519,7 @@ Below follows an example of the :field:`data` and :field:`meta` parts of a respo "stop": 3, }], "layout":"dense", + "nvalues": 600 } } }, @@ -1231,8 +1232,8 @@ Single Entry URL Query Parameters The slices are 0 based, i.e. the first value has index 0, and inclusive i.e. for the range :val:`:10:20:1` the last value returned belongs to index 20. In general support is OPTIONAL, property definitions may however deviate from this and place stricter requirements on servers. - Databases must use the methods described in the section `Transmission of large property values`_ to return the requested sub-range of a property. -Databases MUST use these ranges for properties where the dimension is listed under indexable_dimensions, if this is not the case the database MAY return more data than was specified in the range. + Databases MUST use the methods described in the section `Transmission of large property values`_ to return the requested sub-range of a property. + Databases MUST use these ranges for properties where the dimension is listed under indexable_dimensions, if this is not the case the database MAY return more data than was specified in the range. If a dimension is not specified, it is assumed that the whole range in that dimension is requested. If one or more values are not present at one of the requested combination of indexes, the server MAY either decide to return null or if possible adjust the returned range so the indexes for which no value is defined are no longer part of the range. @@ -1242,7 +1243,7 @@ Databases MUST use these ranges for properties where the dimension is listed und Example: In this example the Cartesian site positions are requested for particles 30 through 70 for 1 out of every 10 frames of the first 1000 frames of a trajectory. -:query-url:`http://example.com/optimade/v1/structures/id_12345?response_fields=cartesian_site_positions&property_ranges=frames::1000:10,particles:30:70::` + :query-url:`http://example.com/optimade/v1/structures/id_12345?response_fields=cartesian_site_positions&property_ranges=frames::1000:10,particles:30:70::` The client MAY provide a set of additional URL query parameters for this endpoint type. From 6905d7f208cccd1bc0bc7214ab590772e1a42bd3 Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Thu, 29 Jun 2023 19:55:33 +0200 Subject: [PATCH 06/70] added that fields should be placed in the range dictionary. --- optimade.rst | 63 ++++++++++++++++++++++++++-------------------------- 1 file changed, 32 insertions(+), 31 deletions(-) diff --git a/optimade.rst b/optimade.rst index b93221ac4..6c1d0f18b 100644 --- a/optimade.rst +++ b/optimade.rst @@ -459,37 +459,38 @@ The default partial data format is named "jsonlines" and is described in the App An implementation SHOULD always include this format as one of the partial data formats provided for a property that has been omitted from the response to the initial query. Implementations MAY provide links to their own non-standard formats, but non-standard format names MUST be prefixed by a database-provider-specific prefix. -If the server supports the :query-param:`property_ranges` query parameter, as described in section `Single Entry URL Query Parameters`_, additional metadata SHOULD be present in the property specific field under the :field:`property_metadata` field, for each property that is returned in the partial data format. -This metadata allows clients to request specific subsections of properties. - -If the field can be sliced with the :query-param:`property_ranges`, the following fields MUST be defined under :field:`property_metadata`, otherwise implementation is OPTIONAL. -Except for the `nvalues` field, the queryability of all fields is OPTIONAL. - -- :field:`range_ids`: List of Strings. - A list with an identifier for each dimension of the property. - The outermost dimension of a nested array should come first. - - If, within one entry, dimensions for two or more properties share the same :field:`range_id` those dimensions should be thought of as the same dimension. - For example, if both the :property:`energy` and :property:`cartesian_site_positions` of a molecular dynamics trajectory share a range_id of :val:`frames`. - This means that the energy at index x(in the dimension labelled by this range_id) belongs to the cartesian_site_positions at the same index x. - -- :field:`indexable_dim`: List of Strings. - The list of range_ids of the dimensions for which slicing is supported, i.e. the client can request a subrange in this dimension via the :query-param:`property_ranges` query parameter. - -- :field:`"layout"`: String. - A string either equal to :val:`"dense"` or :val:`"sparse"` to indicate whether the property is returned in a dense or sparse layout. - -- :field:`data_range`: List of Slice Objects. - This field describes how the values are distributed in the different dimensions. - It consists of a slice object for each dimension of the property. - The order of the slice objects must be the same as in the :field:`range_ids` field. - If the :field:`layout` field is set to :val:`"sparse"` the value of the :field:`step` sub-field has no meaning. - -- :field:`nvalues`: Integer. - The total number of values in the property. - SHOULD be a queryable property with support for all mandatory filter features. - - +If the server supports the :query-param:`property_ranges` query parameter, as described in section `Single Entry URL Query Parameters`_, additional metadata SHOULD be present for each field that is returned via the partial data mechanism. +The client needs this metadata to be able to select only a part of the data of a property. +This metadata is stored in a dictionary in the field :field:`range` which is stored as per property metadata as described in section `Metadata properties`_ + +- :field:`range`: Dictionary + A dictionary that contains the data necessary for the client to select only a sub_range of a property. + If the property_ranges query parameter is supported for this property the following keys MUST be present. + Otherwise these keys are optional. + + - :field:`range_ids`: List of Strings. + A list with an identifier for each dimension of the property. + The outermost dimension of a nested array should come first. + + If, within one entry, dimensions for two or more properties share the same :field:`range_id` those dimensions should be thought of as the same dimension. + For example, if both the :property:`energy` and :property:`cartesian_site_positions` of a molecular dynamics trajectory share a range_id of :val:`frames`. + This means that the energy at index x(in the dimension labelled by this range_id) belongs to the cartesian_site_positions at the same index x. + + - :field:`indexable_dim`: List of Strings. + The list of range_ids of the dimensions for which slicing is supported, i.e. the client can request a subrange in this dimension via the :query-param:`property_ranges` query parameter. + + - :field:`"layout"`: String. + A string either equal to :val:`"dense"` or :val:`"sparse"` to indicate whether the property is returned in a dense or sparse layout. + + - :field:`data_range`: List of Slice Objects. + This field describes how the values are distributed in the different dimensions. + It consists of a slice object for each dimension of the property. + The order of the slice objects must be the same as in the :field:`range_ids` field. + If the :field:`layout` field is set to :val:`"sparse"` the value of the :field:`step` sub-field has no meaning. + + - :field:`nvalues`: Integer. + The total number of values in the property. + SHOULD be a queryable property with support for all mandatory filter features. Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format. It communicates that the property value has been omitted from the response and includes three different links for different partial data formats provided and metadata that makes it possible to only request only a part of the data for this property. From d579c6ece94a74402d4c43896451f2cccac97373 Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Fri, 30 Jun 2023 08:56:20 +0200 Subject: [PATCH 07/70] Some more sentences that had to be moved to a new line. --- optimade.rst | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/optimade.rst b/optimade.rst index 6c1d0f18b..963caeb19 100644 --- a/optimade.rst +++ b/optimade.rst @@ -1226,10 +1226,14 @@ Single Entry URL Query Parameters The dimension name and the start, stop and step values of the slice are separated by colons (":", ASCII 58(0x3A)) The slice is defined in the same manner as the `slice object`_. If no value is placed between the colons for a component of the slice the default value is used. - The first integer specifies the start of the slice, i.e. the first index in that dimension for which values should be returned. The default is the index of the first value. - The second integer specifies the end/stop of the slice, i.e. the last index for which values should be returned. The default is the index of the last value of the property. - And the third integer specifies the step size in that dimension. The default is :val:`1` - Multiple dimensional slices can be defined by specifying a range for each dimension. These ranges are separated by a comma (",", ASCII 44(0x2C)). + The first integer specifies the start of the slice, i.e. the first index in that dimension for which values should be returned. + The default is the index of the first value. + The second integer specifies the end/stop of the slice, i.e. the last index for which values should be returned. + The default is the index of the last value of the property. + And the third integer specifies the step size in that dimension. + The default is :val:`1` + Multiple dimensional slices can be defined by specifying a range for each dimension. + These ranges are separated by a comma (",", ASCII 44(0x2C)). The slices are 0 based, i.e. the first value has index 0, and inclusive i.e. for the range :val:`:10:20:1` the last value returned belongs to index 20. In general support is OPTIONAL, property definitions may however deviate from this and place stricter requirements on servers. From 679296214e098d3a5c1ac6de80a3e6684551a381 Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Fri, 30 Jun 2023 08:53:59 +0200 Subject: [PATCH 08/70] Update optimade.rst Make sure each sentence starts on a new line. Co-authored-by: Antanas Vaitkus --- optimade.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/optimade.rst b/optimade.rst index 963caeb19..d334d5b1d 100644 --- a/optimade.rst +++ b/optimade.rst @@ -492,7 +492,8 @@ This metadata is stored in a dictionary in the field :field:`range` which is sto The total number of values in the property. SHOULD be a queryable property with support for all mandatory filter features. -Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format. It communicates that the property value has been omitted from the response and includes three different links for different partial data formats provided and metadata that makes it possible to only request only a part of the data for this property. +Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format. +It communicates that the property value has been omitted from the response and includes three different links for different partial data formats provided and metadata that makes it possible to request only a part of the data for this property. .. code:: jsonc From 93ecd0b80b9dc8c71aa713d35756d96f97f465bc Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Fri, 30 Jun 2023 08:58:23 +0200 Subject: [PATCH 09/70] removed unintentional changes in appendix. --- optimade.rst | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/optimade.rst b/optimade.rst index d334d5b1d..5629ed741 100644 --- a/optimade.rst +++ b/optimade.rst @@ -3706,7 +3706,7 @@ The strings below contain Extended Regular Expressions (EREs) to recognize ident #BEGIN ERE identifiers [a-z_][a-z_0-9]* #END ERE identifiers -== + #BEGIN ERE numbers [-+]?([0-9]+(\.[0-9]*)?|\.[0-9]+)([eE][-+]?[0-9]+)? #END ERE numbers @@ -3737,7 +3737,6 @@ The dictionary has the following OPTIONAL fields: The default is 1, i.e., every value in the range indicated by :field:`start` and :field:`stop` is included in the slice. Hence, a value of 2 denotes a slice of every second value in the array. - For example, for the array :val:`["a", "b", "c", "d", "e", "f", "g", "h", "i", "j"]` the slice object :val:`{"start": 1, "end": 7, "step": 3}` refers to the items :val:`["b", "e", "h"]`. Furthermore, we also define the following special markers: From a84c72e423f797cfd355e7ba61e3ac0d82e70c0f Mon Sep 17 00:00:00 2001 From: Johan Bergsma <29785380+JPBergsma@users.noreply.github.com> Date: Tue, 4 Jul 2023 16:24:23 +0200 Subject: [PATCH 10/70] Seperated property_ranges query parameter from the partial data. --- optimade.rst | 99 +++++++++++++++++++++++++++++++++++----------------- 1 file changed, 67 insertions(+), 32 deletions(-) diff --git a/optimade.rst b/optimade.rst index 5629ed741..7b8c7d1d4 100644 --- a/optimade.rst +++ b/optimade.rst @@ -459,33 +459,65 @@ The default partial data format is named "jsonlines" and is described in the App An implementation SHOULD always include this format as one of the partial data formats provided for a property that has been omitted from the response to the initial query. Implementations MAY provide links to their own non-standard formats, but non-standard format names MUST be prefixed by a database-provider-specific prefix. -If the server supports the :query-param:`property_ranges` query parameter, as described in section `Single Entry URL Query Parameters`_, additional metadata SHOULD be present for each field that is returned via the partial data mechanism. +Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format. +It communicates that the property value has been omitted from the response and includes three different links for different partial data formats provided. + +.. code:: jsonc + + { + // ... + "data": { + "type": "structures", + "id": "2345678", + "attributes": { + "a": null + } + "meta": { + "partial_data_links": { + "a": [ + { + "format": "jsonlines", + "link": "https://example.org/optimade/v1.2/extensions/partial_data/structures/2345678/a/default_format" + }, + { + "format": "_exmpl_bzip2_jsonlines", + "link": "https://db.example.org/assets/partial_values/structures/2345678/a/bzip2_format" + }, + { + "format": "_exmpl_hdf5", + "link": "https://cloud.example.org/ACCHSORJGIHWOSJZG" + } + ] + } + } + } + // ... + } + + +Metadata for Property_ranges query parameter +-------------------------------------------- + +If the server supports the :query-param:`property_ranges` query parameter, as described in section `Single Entry URL Query Parameters`_, additional metadata SHOULD be present for each field for which the :query-param:`property_ranges` query parameter can be used. The client needs this metadata to be able to select only a part of the data of a property. This metadata is stored in a dictionary in the field :field:`range` which is stored as per property metadata as described in section `Metadata properties`_ - :field:`range`: Dictionary - A dictionary that contains the data necessary for the client to select only a sub_range of a property. - If the property_ranges query parameter is supported for this property the following keys MUST be present. - Otherwise these keys are optional. - - :field:`range_ids`: List of Strings. - A list with an identifier for each dimension of the property. - The outermost dimension of a nested array should come first. - - If, within one entry, dimensions for two or more properties share the same :field:`range_id` those dimensions should be thought of as the same dimension. - For example, if both the :property:`energy` and :property:`cartesian_site_positions` of a molecular dynamics trajectory share a range_id of :val:`frames`. - This means that the energy at index x(in the dimension labelled by this range_id) belongs to the cartesian_site_positions at the same index x. + A dictionary that contains the data necessary for the client to select only a sub_range of a property. + If the property_ranges query parameter is supported for this property the following keys MUST be present. + Otherwise this dictionary and these keys are optional. - :field:`indexable_dim`: List of Strings. The list of range_ids of the dimensions for which slicing is supported, i.e. the client can request a subrange in this dimension via the :query-param:`property_ranges` query parameter. - - :field:`"layout"`: String. + - :field:`layout`: String. A string either equal to :val:`"dense"` or :val:`"sparse"` to indicate whether the property is returned in a dense or sparse layout. - :field:`data_range`: List of Slice Objects. This field describes how the values are distributed in the different dimensions. It consists of a slice object for each dimension of the property. - The order of the slice objects must be the same as in the :field:`range_ids` field. + The order of the slice objects must be the same as in the :field:`range_ids` field in the property definition. If the :field:`layout` field is set to :val:`"sparse"` the value of the :field:`step` sub-field has no meaning. - :field:`nvalues`: Integer. @@ -493,7 +525,7 @@ This metadata is stored in a dictionary in the field :field:`range` which is sto SHOULD be a queryable property with support for all mandatory filter features. Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format. -It communicates that the property value has been omitted from the response and includes three different links for different partial data formats provided and metadata that makes it possible to request only a part of the data for this property. +It communicates that the property value has been omitted from the response and metadata that makes it possible to request only a part of the data for this property. .. code:: jsonc @@ -526,20 +558,7 @@ It communicates that the property value has been omitted from the response and i } }, "partial_data_links": { - "a": [ - { - "format": "jsonlines", - "link": "https://example.org/optimade/v1.2/extensions/partial_data/structures/2345678/a/default_format" - }, - { - "format": "_exmpl_bzip2_jsonlines", - "link": "https://db.example.org/assets/partial_values/structures/2345678/a/bzip2_format" - }, - { - "format": "_exmpl_hdf5", - "link": "https://cloud.example.org/ACCHSORJGIHWOSJZG" - } - ] + //... } } } @@ -1221,24 +1240,26 @@ The rules for which properties are to be present for an entry in the response ar Single Entry URL Query Parameters ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -- **property\_ranges**: A set of slices which specify which range of a property should be returned for partial data properties. - The property to which this range should be applied can be specified via the :query-param:`response_fields` query parameter. +- **property\_ranges**: A set of slices which specify which sub-range of a property should be returned. + + The property to which this sub-range should be applied can be specified via the :query-param:`response_fields` query parameter. Each slice consists of the name of a dimension directly followed by the requested slice in this dimension. The dimension name and the start, stop and step values of the slice are separated by colons (":", ASCII 58(0x3A)) - The slice is defined in the same manner as the `slice object`_. + The slice is defined in a similar manner as the `slice object`_. If no value is placed between the colons for a component of the slice the default value is used. + The first integer specifies the start of the slice, i.e. the first index in that dimension for which values should be returned. The default is the index of the first value. The second integer specifies the end/stop of the slice, i.e. the last index for which values should be returned. The default is the index of the last value of the property. And the third integer specifies the step size in that dimension. The default is :val:`1` + Multiple dimensional slices can be defined by specifying a range for each dimension. These ranges are separated by a comma (",", ASCII 44(0x2C)). The slices are 0 based, i.e. the first value has index 0, and inclusive i.e. for the range :val:`:10:20:1` the last value returned belongs to index 20. In general support is OPTIONAL, property definitions may however deviate from this and place stricter requirements on servers. - Databases MUST use the methods described in the section `Transmission of large property values`_ to return the requested sub-range of a property. Databases MUST use these ranges for properties where the dimension is listed under indexable_dimensions, if this is not the case the database MAY return more data than was specified in the range. If a dimension is not specified, it is assumed that the whole range in that dimension is requested. @@ -1251,6 +1272,7 @@ Single Entry URL Query Parameters :query-url:`http://example.com/optimade/v1/structures/id_12345?response_fields=cartesian_site_positions&property_ranges=frames::1000:10,particles:30:70::` +| The client MAY provide a set of additional URL query parameters for this endpoint type. URL query parameters not recognized MUST be ignored. @@ -2195,6 +2217,19 @@ A Property Definition MUST be composed according to the combination of the requi - :val:`partial`: the defined property MUST be queryable with support for a subset of the filter language operators as specified by the field :field:`query-support-operators`. - :val:`none`: the defined property does not need to be queryable with any features of the filter language. + - :field:`property_range`: Dictionary. + A dictionary that contains information needed to use the :query-param:`property_range` query parameter + + - :field:`supported`: Boolean. + This field indicates whether the it is possible to use the :query-param:`property_range` query parameter in combination with this field. If true the use of the :query-param:`property_range` is supported + + - :field:`range_ids`: List of Strings. + A list with an identifier for each dimension of the property. + The outermost dimension of a nested array should come first. + If, within one entry, dimensions for two or more properties share the same :field:`range_id` those dimensions should be thought of as the same dimension. + For example, if both the :property:`energy` and :property:`cartesian_site_positions` of a molecular dynamics trajectory share a range_id of :val:`frames`. + This means that the energy at index x(in the dimension labelled by this range_id) belongs to the cartesian_site_positions at the same index x. + - :field:`query-support-operators`: List of Strings. Defines the filter language features supported on this property. Each string in the list MUST be one of :val:`<`, :val:`<=`, :val:`>`, :val:`>=`, :val:`=`, :val:`!=`, :val:`CONTAINS`, :val:`STARTS WITH`, :val:`ENDS WITH`:, :val:`HAS`, :val:`HAS ALL`, :val:`HAS ANY`, :val:`HAS ONLY`, :val:`IS KNOWN`, :val:`IS UNKNOWN` with the following meanings: From 17beeef72512666fe3776583be51a3511b9b218e Mon Sep 17 00:00:00 2001 From: Matthew Evans <7916000+ml-evs@users.noreply.github.com> Date: Thu, 18 Jan 2024 11:52:58 +0000 Subject: [PATCH 11/70] Apply formatting suggestions from code review --- optimade.rst | 66 ++++++++++++++++++++++++++-------------------------- 1 file changed, 33 insertions(+), 33 deletions(-) diff --git a/optimade.rst b/optimade.rst index 49fdc2875..852f02170 100644 --- a/optimade.rst +++ b/optimade.rst @@ -495,28 +495,26 @@ It communicates that the property value has been omitted from the response and i } -Metadata for Property_ranges query parameter +Metadata for :query-param:`property_ranges` query parameter -------------------------------------------- If the server supports the :query-param:`property_ranges` query parameter, as described in section `Single Entry URL Query Parameters`_, additional metadata SHOULD be present for each field for which the :query-param:`property_ranges` query parameter can be used. The client needs this metadata to be able to select only a part of the data of a property. -This metadata is stored in a dictionary in the field :field:`range` which is stored as per property metadata as described in section `Metadata properties`_ +This metadata is given in a dictionary field :field:`range` which stores per property metadata as described in section `Metadata properties`_ -- :field:`range`: Dictionary - - A dictionary that contains the data necessary for the client to select only a sub_range of a property. - If the property_ranges query parameter is supported for this property the following keys MUST be present. - Otherwise this dictionary and these keys are optional. +- :field:`range`: Dictionary. + A dictionary that contains the data necessary for the client to select only a subset or slice of a property. + If the :query-param:`property_ranges` query parameter is supported for this property the following keys MUST be present: - :field:`indexable_dim`: List of Strings. - The list of range_ids of the dimensions for which slicing is supported, i.e. the client can request a subrange in this dimension via the :query-param:`property_ranges` query parameter. + The list of :field:`range_ids` of the dimensions for which slicing is supported, i.e. the client can request a slice in this dimension via the :query-param:`property_ranges` query parameter. - :field:`layout`: String. A string either equal to :val:`"dense"` or :val:`"sparse"` to indicate whether the property is returned in a dense or sparse layout. - - :field:`data_range`: List of Slice Objects. + - :field:`data_range`: List of slice objects. This field describes how the values are distributed in the different dimensions. - It consists of a slice object for each dimension of the property. + It consists of a `slice object`_ for each dimension of the property. The order of the slice objects must be the same as in the :field:`range_ids` field in the property definition. If the :field:`layout` field is set to :val:`"sparse"` the value of the :field:`step` sub-field has no meaning. @@ -535,24 +533,27 @@ It communicates that the property value has been omitted from the response and m "type": "structures", "id": "2345678", "attributes": { - "a": null + "a": null } "meta": { - "property_metadata":{ - "a":{ - "range":{ - "range_ids":["frames","particles"], + "property_metadata": { + "a": { + "range": { + "range_ids": ["frames", "particles"], "indexable_dim": ["frames"], - "data_range": [{ + "data_range": [ + { "start": 1, "step": 1, "stop": 200, - },{ + }, + { "start": 1, "step": 1, "stop": 3, - }], - "layout":"dense", + } + ], + "layout": "dense", "nvalues": 600 } } @@ -1240,39 +1241,38 @@ The rules for which properties are to be present for an entry in the response ar Single Entry URL Query Parameters ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -- **property\_ranges**: A set of slices which specify which sub-range of a property should be returned. +- **property\_ranges**: A set of slice objects which specify which sub-range of a property should be returned. The property to which this sub-range should be applied can be specified via the :query-param:`response_fields` query parameter. Each slice consists of the name of a dimension directly followed by the requested slice in this dimension. The dimension name and the start, stop and step values of the slice are separated by colons (":", ASCII 58(0x3A)) The slice is defined in a similar manner as the `slice object`_. - If no value is placed between the colons for a component of the slice the default value is used. + If no value is placed between the colons for a component of the slice then the default value is used. The first integer specifies the start of the slice, i.e. the first index in that dimension for which values should be returned. The default is the index of the first value. The second integer specifies the end/stop of the slice, i.e. the last index for which values should be returned. The default is the index of the last value of the property. - And the third integer specifies the step size in that dimension. + The third integer specifies the step size in that dimension. The default is :val:`1` - Multiple dimensional slices can be defined by specifying a range for each dimension. + Multidimensional slices can be defined by specifying a range for each dimension. These ranges are separated by a comma (",", ASCII 44(0x2C)). - The slices are 0 based, i.e. the first value has index 0, and inclusive i.e. for the range :val:`:10:20:1` the last value returned belongs to index 20. + The slices are 0-based, i.e. the first value has index 0, and inclusive i.e. for the range :val:`:10:20:1` the last value returned belongs to index 20. - In general support is OPTIONAL, property definitions may however deviate from this and place stricter requirements on servers. - Databases MUST use these ranges for properties where the dimension is listed under indexable_dimensions, if this is not the case the database MAY return more data than was specified in the range. + General support for :field:`property_ranges` is OPTIONAL, however particular property definitions may themselves deviate from this and place stricter requirements on implementations. + Databases MUST use these ranges for properties where the dimension is listed under :field:`indexable_dimensions`, if this is not the case, the database MAY return more data than was specified in the range. If a dimension is not specified, it is assumed that the whole range in that dimension is requested. If one or more values are not present at one of the requested combination of indexes, the server MAY either decide to return null or if possible adjust the returned range so the indexes for which no value is defined are no longer part of the range. The latter is only allowed when no defined values would be lost. For dense arrays that may mean that the field :field:`returned_range` differs from the requested range. - When a value is, however, explicitly set to :val:`null` and :val:`null` has a meaning beyond indicating that no value has been defined :val:`null` MUST be returned + However, when a value is explicitly set to :val:`null` to indicate that the underlying property has no defined values, then :val:`null` MUST be returned. - Example: In this example the Cartesian site positions are requested for particles 30 through 70 for 1 out of every 10 frames of the first 1000 frames of a trajectory. + Example: In this example the Cartesian site positions are requested for particles 30 through 70 for 1 out of every 10 frames of the first 1000 frames of a trajectory with ID :val:`id_12345`. - :query-url:`http://example.com/optimade/v1/structures/id_12345?response_fields=cartesian_site_positions&property_ranges=frames::1000:10,particles:30:70::` + :query-url:`http://example.com/optimade/v1/trajectories/id_12345?response_fields=cartesian_site_positions&property_ranges=frames::1000:10,particles:30:70::` -| The client MAY provide a set of additional URL query parameters for this endpoint type. URL query parameters not recognized MUST be ignored. @@ -2218,17 +2218,17 @@ A Property Definition MUST be composed according to the combination of the requi - :val:`none`: the defined property does not need to be queryable with any features of the filter language. - :field:`property_range`: Dictionary. - A dictionary that contains information needed to use the :query-param:`property_range` query parameter + A dictionary that contains information needed to use the :query-param:`property_range` query parameter. - :field:`supported`: Boolean. - This field indicates whether the it is possible to use the :query-param:`property_range` query parameter in combination with this field. If true the use of the :query-param:`property_range` is supported + This field indicates whether it is possible to use the :query-param:`property_range` query parameter in combination with this field. - :field:`range_ids`: List of Strings. A list with an identifier for each dimension of the property. The outermost dimension of a nested array should come first. If, within one entry, dimensions for two or more properties share the same :field:`range_id` those dimensions should be thought of as the same dimension. For example, if both the :property:`energy` and :property:`cartesian_site_positions` of a molecular dynamics trajectory share a range_id of :val:`frames`. - This means that the energy at index x(in the dimension labelled by this range_id) belongs to the cartesian_site_positions at the same index x. + This means that the energy at index *x* (in the dimension labelled by this :field:`range_id`) belongs to the :field:`cartesian_site_positions` at the same index *x*. - :field:`query-support-operators`: List of Strings. Defines the filter language features supported on this property. From 6d8e24ff58c124039238931c61599f420eaad555 Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Fri, 22 Mar 2024 02:07:48 +0100 Subject: [PATCH 12/70] Apply suggestions from review Co-authored-by: Antanas Vaitkus --- optimade.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/optimade.rst b/optimade.rst index bb04a1cf9..2cf31c798 100644 --- a/optimade.rst +++ b/optimade.rst @@ -1265,12 +1265,12 @@ Single Entry URL Query Parameters The slices are 0-based, i.e. the first value has index 0, and inclusive i.e. for the range :val:`:10:20:1` the last value returned belongs to index 20. General support for :field:`property_ranges` is OPTIONAL, however particular property definitions may themselves deviate from this and place stricter requirements on implementations. - Databases MUST use these ranges for properties where the dimension is listed under :field:`indexable_dimensions`, if this is not the case, the database MAY return more data than was specified in the range. + Databases MUST use these ranges for properties where the dimension is listed under :field:`indexable_dim`, if this is not the case, the database MAY return more data than was specified in the range. If a dimension is not specified, it is assumed that the whole range in that dimension is requested. If one or more values are not present at one of the requested combination of indexes, the server MAY either decide to return null or if possible adjust the returned range so the indexes for which no value is defined are no longer part of the range. The latter is only allowed when no defined values would be lost. - For dense arrays that may mean that the field :field:`returned_range` differs from the requested range. + For dense arrays that may mean that the field :field:`returned_ranges` differs from the requested range. However, when a value is explicitly set to :val:`null` to indicate that the underlying property has no defined values, then :val:`null` MUST be returned. Example: In this example the Cartesian site positions are requested for particles 30 through 70 for 1 out of every 10 frames of the first 1000 frames of a trajectory with ID :val:`id_12345`. From 63e382143874c7b6b07110f7302cae78cf8d46b8 Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Fri, 22 Mar 2024 12:48:03 +0100 Subject: [PATCH 13/70] Minor corrections from review Co-authored-by: Andrius Merkys --- optimade.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/optimade.rst b/optimade.rst index a88cb8225..7a94c5851 100644 --- a/optimade.rst +++ b/optimade.rst @@ -1295,14 +1295,14 @@ Single Entry URL Query Parameters Multidimensional slices can be defined by specifying a range for each dimension. These ranges are separated by a comma (",", ASCII 44(0x2C)). - The slices are 0-based, i.e. the first value has index 0, and inclusive i.e. for the range :val:`:10:20:1` the last value returned belongs to index 20. + The slices are 0-based, i.e. the first value has index 0, and inclusive i.e. for the dimension :val:`dim` the range :val:`dim:10:20:1` the last value returned belongs to index 20. General support for :field:`property_ranges` is OPTIONAL, however particular property definitions may themselves deviate from this and place stricter requirements on implementations. Databases MUST use these ranges for properties where the dimension is listed under :field:`indexable_dim`, if this is not the case, the database MAY return more data than was specified in the range. If a dimension is not specified, it is assumed that the whole range in that dimension is requested. - If one or more values are not present at one of the requested combination of indexes, the server MAY either decide to return null or if possible adjust the returned range so the indexes for which no value is defined are no longer part of the range. - The latter is only allowed when no defined values would be lost. + If one or more values are not present at one of the requested combination of indexes, the server MAY either decide to return :val:`null` or if possible adjust the returned range so the indexes for which no value is defined are no longer part of the range. + The latter is only allowed when no defined values would be omitted. For dense arrays that may mean that the field :field:`returned_ranges` differs from the requested range. However, when a value is explicitly set to :val:`null` to indicate that the underlying property has no defined values, then :val:`null` MUST be returned. From bf7e5b4acf7cf111e1ebe28c7e9a239ea469ac15 Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Thu, 13 Jun 2024 13:12:40 +0200 Subject: [PATCH 14/70] Edit property slice definition [WIP] --- optimade.rst | 194 +++++++++++++++++++++++++++------------------------ 1 file changed, 103 insertions(+), 91 deletions(-) diff --git a/optimade.rst b/optimade.rst index 5ca81834c..4a3bff0d9 100644 --- a/optimade.rst +++ b/optimade.rst @@ -527,78 +527,6 @@ It communicates that the property value has been omitted from the response and i } -Metadata for :query-param:`property_ranges` query parameter --------------------------------------------- - -If the server supports the :query-param:`property_ranges` query parameter, as described in section `Single Entry URL Query Parameters`_, additional metadata SHOULD be present for each field for which the :query-param:`property_ranges` query parameter can be used. -The client needs this metadata to be able to select only a part of the data of a property. -This metadata is given in a dictionary field :field:`range` which stores per property metadata as described in section `Metadata properties`_ - -- :field:`range`: Dictionary. - A dictionary that contains the data necessary for the client to select only a subset or slice of a property. - If the :query-param:`property_ranges` query parameter is supported for this property the following keys MUST be present: - - - :field:`indexable_dim`: List of Strings. - The list of :field:`range_ids` of the dimensions for which slicing is supported, i.e. the client can request a slice in this dimension via the :query-param:`property_ranges` query parameter. - - - :field:`layout`: String. - A string either equal to :val:`"dense"` or :val:`"sparse"` to indicate whether the property is returned in a dense or sparse layout. - - - :field:`data_range`: List of slice objects. - This field describes how the values are distributed in the different dimensions. - It consists of a `slice object`_ for each dimension of the property. - The order of the slice objects must be the same as in the :field:`range_ids` field in the property definition. - If the :field:`layout` field is set to :val:`"sparse"` the value of the :field:`step` sub-field has no meaning. - - - :field:`nvalues`: Integer. - The total number of values in the property. - SHOULD be a queryable property with support for all mandatory filter features. - -Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format. -It communicates that the property value has been omitted from the response and metadata that makes it possible to request only a part of the data for this property. - -.. code:: jsonc - - { - // ... - "data": { - "type": "structures", - "id": "2345678", - "attributes": { - "a": null - } - "meta": { - "property_metadata": { - "a": { - "range": { - "range_ids": ["frames", "particles"], - "indexable_dim": ["frames"], - "data_range": [ - { - "start": 1, - "step": 1, - "stop": 200, - }, - { - "start": 1, - "step": 1, - "stop": 3, - } - ], - "layout": "dense", - "nvalues": 600 - } - } - }, - "partial_data_links": { - //... - } - } - } - // ... - } - - Metadata properties ------------------- @@ -676,6 +604,85 @@ Example of the corresponding metadata property definition contained in the field } // ... +Slices of array properties +-------------------------- + +# respnse_fields=property_metadata means MUST RETURN ALL METADATA. + +The OPTIMADE standard defines a way for a client to request only a subset of the items of an array, referred to as a slice. +The protocol for this functionality allows the server to allow slicing on a per-entry basis. +It is important to note that this functionality is separate from the protocol described in `Transmission of large property values`. +Slices are used for a client to ask the server to only provide a subset of items, which may result in a small or large set of items. +The protocol for large property values is used by the server implementation to transmit a set of items that it deems too large to provide inside the normal OPTIMADE response. + +If the server supports the :query-param:`property_ranges` query parameter, as described in section `Single Entry URL Query Parameters`_, additional metadata SHOULD be present for each field for which the :query-param:`property_ranges` query parameter can be used. +The client needs this metadata to be able to select only a part of the data of a property. +This metadata is given in a dictionary field :field:`range` which stores per property metadata as described in section `Metadata properties`_ + +- :field:`range`: Dictionary. + A dictionary that contains the data necessary for the client to select only a subset or slice of a property. + If the :query-param:`property_ranges` query parameter is supported for this property the following keys MUST be present: + + - :field:`indexable_dim`: List of Strings. + The list of :field:`range_ids` of the dimensions for which slicing is supported, i.e. the client can request a slice in this dimension via the :query-param:`property_ranges` query parameter. + + - :field:`layout`: String. + A string either equal to :val:`"dense"` or :val:`"sparse"` to indicate whether the property is returned in a dense or sparse layout. + + - :field:`data_range`: List of slice objects. + This field describes how the values are distributed in the different dimensions. + It consists of a `slice object`_ for each dimension of the property. + The order of the slice objects must be the same as in the :field:`range_ids` field in the property definition. + If the :field:`layout` field is set to :val:`"sparse"` the value of the :field:`step` sub-field has no meaning. + + - :field:`nvalues`: Integer. + The total number of values in the property. + SHOULD be a queryable property with support for all mandatory filter features. + +Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format. +It communicates that the property value has been omitted from the response and metadata that makes it possible to request only a part of the data for this property. + +.. code:: jsonc + + { + // ... + "data": { + "type": "structures", + "id": "2345678", + "attributes": { + "a": null + } + "meta": { + "property_metadata": { + "a": { + "range": { + "range_ids": ["frames", "particles"], + "indexable_dim": ["frames"], + "data_range": [ + { + "start": 1, + "step": 1, + "stop": 200, + }, + { + "start": 1, + "step": 1, + "stop": 3, + } + ], + "layout": "dense", + "nvalues": 600 + } + } + }, + "partial_data_links": { + //... + } + } + } + // ... + } + Responses ========= @@ -1189,6 +1196,10 @@ Standard OPTIONAL URL query parameters not in the JSON:API specification: Other OPTIONAL fields MUST NOT be returned when this parameter is present. Example: :query-url:`http://example.com/optimade/v1/structures?response_fields=last_modified,nsites` + +********* + + Additional OPTIONAL URL query parameters not described above are not considered to be part of this standard, and are instead considered to be "custom URL query parameters". These custom URL query parameters MUST be of the format "". These names adhere to the requirements on implementation-specific query parameters of `JSON:API v1.1 `__ since the database-provider-specific prefixes contain at least two underscores (a LOW LINE character '\_'). @@ -1278,7 +1289,16 @@ The rules for which properties are to be present for an entry in the response ar Single Entry URL Query Parameters ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -- **property\_ranges**: A set of slice objects which specify which sub-range of a property should be returned. +The client MAY provide a set of additional URL query parameters for this endpoint type. +URL query parameters not recognized MUST be ignored. +While the following URL query parameters are OPTIONAL for clients, API implementations MUST accept and handle them: +:query-param:`response_format`, :query-param:`email_address`, :query-param:`response_fields`. +The URL query parameter :query-param:`include` is OPTIONAL for both clients and API implementations. +The meaning of these URL query parameters are as defined above in section `Entry Listing URL Query Parameters`_. + +One additional query parameter :query-param:`property_slices` MUST be handled by the API implementation either as defined below or by returning the error :http-error:`501 Not Implemented`: + +- **property\_slices**: A set of slice objects which specify which sub-range of a property should be returned. The property to which this sub-range should be applied can be specified via the :query-param:`response_fields` query parameter. Each slice consists of the name of a dimension directly followed by the requested slice in this dimension. @@ -1297,26 +1317,15 @@ Single Entry URL Query Parameters These ranges are separated by a comma (",", ASCII 44(0x2C)). The slices are 0-based, i.e. the first value has index 0, and inclusive i.e. for the dimension :val:`dim` the range :val:`dim:10:20:1` the last value returned belongs to index 20. - General support for :field:`property_ranges` is OPTIONAL, however particular property definitions may themselves deviate from this and place stricter requirements on implementations. - Databases MUST use these ranges for properties where the dimension is listed under :field:`indexable_dim`, if this is not the case, the database MAY return more data than was specified in the range. - - If a dimension is not specified, it is assumed that the whole range in that dimension is requested. - If one or more values are not present at one of the requested combination of indexes, the server MAY either decide to return :val:`null` or if possible adjust the returned range so the indexes for which no value is defined are no longer part of the range. - The latter is only allowed when no defined values would be omitted. - For dense arrays that may mean that the field :field:`returned_ranges` differs from the requested range. - However, when a value is explicitly set to :val:`null` to indicate that the underlying property has no defined values, then :val:`null` MUST be returned. + If an array dimension is indicated as slicable, the response MUST include the items in the requested slice. - Example: In this example the Cartesian site positions are requested for particles 30 through 70 for 1 out of every 10 frames of the first 1000 frames of a trajectory with ID :val:`id_12345`. + If an array dimension is not indicated as slicable, the response MUST be one of the following: (i) An error is returned; or (ii) the items in the requested slice is returned. - :query-url:`http://example.com/optimade/v1/trajectories/id_12345?response_fields=cartesian_site_positions&property_ranges=frames::1000:10,particles:30:70::` + If a dimension is not specified, it is assumed that all list items in that dimension is requested. + Example: In the following example, items from the Cartesian site positions array is requested only for the 31st to 71st sites (i.e., with indexes 30 through 70 inclusive) for 1 out of every 10 frames of the first 1000 frames (i.e., taking steps of 10 over indexes 0 through 999 inclusive, which requests the frames with index 0, 10, 20, 30, ..., 990) of a trajectory with ID :val:`id_12345`. -The client MAY provide a set of additional URL query parameters for this endpoint type. -URL query parameters not recognized MUST be ignored. -While the following URL query parameters are OPTIONAL for clients, API implementations MUST accept and handle them: -:query-param:`response_format`, :query-param:`email_address`, :query-param:`response_fields`. -The URL query parameter :query-param:`include` is OPTIONAL for both clients and API implementations. -The meaning of these URL query parameters are as defined above in section `Entry Listing URL Query Parameters`_. + :query-url:`http://example.com/optimade/v1/trajectories/id_12345?response_fields=cartesian_site_positions&property_ranges=dim_frames::999:10,dim_sites:30:70::` Single Entry JSON Response Schema ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -2291,8 +2300,11 @@ A Property Definition MUST be composed according to the combination of the requi - :field:`names`: List of Strings. A list of names of the dimensions of the underlying one or multi-dimensionsional data represented as mutiple levels of lists. The order is that the the first name applies to the outermost list, the next name to the lists embedded in that list, etc. - If, within one entry, dimensions for two or more properties share the same :field:`name` those dimensions represents the same dimension. - For example, if both the :property:`energy` and :property:`cartesian_site_positions` of a molecular dynamics trajectory share a range_id of :val:`frames` this means that the energy at index *i* (in the dimension labelled by this :field:`range_id`) belongs to the :field:`cartesian_site_positions` at the same index *i*. + Dimension names defined by the OPTIMADE standard are prefixed by ``dim_``. + Dimension names defined by database or definition providers MUST be prefixed by the corresponding namespace prefix and SHOULD also be prefixed with ``dim_``, e.g., ``_exmpl_dim_particles``. + If, within one entry, two or more array dimensions in one or more properties share the same :field:`name`, those represent the same dimension. + For example, in a trajectory entry there is a property :property:`cartesian_site_positions` where one of the dimension names is :val:`dim_frames`. + If there is another one-dimensional array property :property:`_exmpl_energy` that specifies the same dimension name in its property definition, then this declares that the energy and site positions at index *i* pertain to the same frame. - :field:`sizes`: List of Integers or :val:`null`. A list of fixed length requirements on the underlying one or multi-dimensionsional data represented as mutiple levels of lists. From d02975663cd4abf9bee4e8d0bfb6854cfed15eef Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Thu, 13 Jun 2024 16:16:37 +0200 Subject: [PATCH 15/70] Edit property slice definition [WIP] --- optimade.rst | 101 ++++++++++++++++++++++++++++++++++++--------------- 1 file changed, 72 insertions(+), 29 deletions(-) diff --git a/optimade.rst b/optimade.rst index 4a3bff0d9..49b01bd96 100644 --- a/optimade.rst +++ b/optimade.rst @@ -607,37 +607,80 @@ Example of the corresponding metadata property definition contained in the field Slices of array properties -------------------------- -# respnse_fields=property_metadata means MUST RETURN ALL METADATA. +# response_fields=property_metadata means MUST RETURN ALL METADATA. The OPTIMADE standard defines a way for a client to request only a subset of the items of an array, referred to as a slice. -The protocol for this functionality allows the server to allow slicing on a per-entry basis. -It is important to note that this functionality is separate from the protocol described in `Transmission of large property values`. -Slices are used for a client to ask the server to only provide a subset of items, which may result in a small or large set of items. -The protocol for large property values is used by the server implementation to transmit a set of items that it deems too large to provide inside the normal OPTIMADE response. - -If the server supports the :query-param:`property_ranges` query parameter, as described in section `Single Entry URL Query Parameters`_, additional metadata SHOULD be present for each field for which the :query-param:`property_ranges` query parameter can be used. -The client needs this metadata to be able to select only a part of the data of a property. -This metadata is given in a dictionary field :field:`range` which stores per property metadata as described in section `Metadata properties`_ - -- :field:`range`: Dictionary. - A dictionary that contains the data necessary for the client to select only a subset or slice of a property. - If the :query-param:`property_ranges` query parameter is supported for this property the following keys MUST be present: - - - :field:`indexable_dim`: List of Strings. - The list of :field:`range_ids` of the dimensions for which slicing is supported, i.e. the client can request a slice in this dimension via the :query-param:`property_ranges` query parameter. - - - :field:`layout`: String. - A string either equal to :val:`"dense"` or :val:`"sparse"` to indicate whether the property is returned in a dense or sparse layout. - - - :field:`data_range`: List of slice objects. - This field describes how the values are distributed in the different dimensions. - It consists of a `slice object`_ for each dimension of the property. - The order of the slice objects must be the same as in the :field:`range_ids` field in the property definition. - If the :field:`layout` field is set to :val:`"sparse"` the value of the :field:`step` sub-field has no meaning. - - - :field:`nvalues`: Integer. - The total number of values in the property. - SHOULD be a queryable property with support for all mandatory filter features. +The protocol for this functionality allows the server to allow slicing independently per entry, per array, and per array dimension. +This functionality is separate from the protocol described in `Transmission of large property values`_. +Slices are used for a client to ask the server to only provide a subset of items of an array, which can result in a small or large set of items. +In contrast, the protocol for large property values is used by the server implementation to transmit a set of items that it deems too large to provide inside the normal OPTIMADE response. + +The main mechanism is provided through the query parameter :query-param:`property_slices` defined in section `Single Entry URL Query Parameters`_. +Information relating to the ability of the server to handle this query parameter and the relevant ranges of indexes is provided using metadata property fields (see `Metadata properties`_) of the array property as defined below: + +- :field:`dimensions`: List of Dictionary. + A list of dictionaries which provide information related to the extents of an array property, which is relevant for slicing. + Each item, in order, represents the array dimension as declared in the corresponding property definition. + + Each item MUST be a dictionary with the following REQUIRED fields: + + - :field:`name`: String. + The dimension name of the corresponding array dimension. + + If the request specifies the :query-param:`property_slices` query parameter for any of the array dimensions of this array property the following key MUST be present: + + - :field:`requested_slice`: Dictionary. + A field that describes the requested slice that was provided via the query parameter :query-param:`property_slices`. + The subfields MUST reflect the values provided via the :query-param:`property_slices`. + The implementation MUST preserve the values as given in the query parameter, including the distinction between specific values and default values even when they are equivalent. + It MAY contain the following subfields that are defined according to the specification of a `slice object`_. + + - :field:`start`: Integer or :val:`null`. + + - :field:`stop`: Integer or :val:`null`. + + - :field:`step`: Integer or :val:`null`. + + A :val:`null` value for any of these fields means it has the default value defined for a `slice object`_. + A missing value is equivalent to a :val:`null` value. + (Consequently, the dictionary MAY be empty if all fields take the default values.) + + Examples: + + - For :?:`property_slices=dim_frames:::` some of the equivalent valid representations are: + + - ``{}`` + - ``{"start": null, "stop": null, "step": null}`` + - ``{"start": null}`` + - ``{"stop": null, "step": null}`` + + Whereas, e.g., the following representation is not valid: + + - ``{"start": 0}`` (despite that :val:`0` is the default value of the :field:`start` field.) + But it is instead a valid representation if the query parameter was :?:`property_slices=dim_frames:0::`. + + - :?:`property_slices=dim_frames:3:5:2` the representation MUST be: + + - ``{"start": 3, "stop": 5, "step": 2}`` + + The dictionary MAY contain the following fields: + + - :field:`size`: Integer. + The length of this dimension for the specific entry that this metadata entry pertains to. + + - :field:`sliceable`: Boolean. + If true, the server MUST handle slices for that dimension. + If false (which is the default), the server MAY handle slices for that dimension, or MAY return the error :http-error:`501 Not Implemented` when a client requests a slice involving this dimension. + + - :field:`available_slice`: Dictionary or :val:`null`. + This field describes a `slice object`_ where there MAY be non-null values in the data. + By including this field, the server certifies that there are only :val:`null` values outside this slice. + If not provided, or equal to :val:`null` or an empty dictionary, the client cannot make any assumptions about what part of the array contains :val:`null` values. + + Examples: + + - ``{"start": 3, "stop": 7, "step": 2}`` means the server certifies that values at indexes 0,1,2,4,6 and any index from 8 to the end of the array are :val:`null`. + Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format. It communicates that the property value has been omitted from the response and metadata that makes it possible to request only a part of the data for this property. From 63f122f635792be59525cb1956499524702f62f3 Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Thu, 13 Jun 2024 20:57:17 +0200 Subject: [PATCH 16/70] Edit property slice definition [WIP] --- optimade.rst | 104 ++++++++++++++++++++++++++++++++++----------------- 1 file changed, 70 insertions(+), 34 deletions(-) diff --git a/optimade.rst b/optimade.rst index 49b01bd96..7e70658bd 100644 --- a/optimade.rst +++ b/optimade.rst @@ -610,7 +610,7 @@ Slices of array properties # response_fields=property_metadata means MUST RETURN ALL METADATA. The OPTIMADE standard defines a way for a client to request only a subset of the items of an array, referred to as a slice. -The protocol for this functionality allows the server to allow slicing independently per entry, per array, and per array dimension. +The protocol for this functionality allows the server to allow slicing independently per entry, per array, and per array axis. This functionality is separate from the protocol described in `Transmission of large property values`_. Slices are used for a client to ask the server to only provide a subset of items of an array, which can result in a small or large set of items. In contrast, the protocol for large property values is used by the server implementation to transmit a set of items that it deems too large to provide inside the normal OPTIMADE response. @@ -618,16 +618,16 @@ In contrast, the protocol for large property values is used by the server implem The main mechanism is provided through the query parameter :query-param:`property_slices` defined in section `Single Entry URL Query Parameters`_. Information relating to the ability of the server to handle this query parameter and the relevant ranges of indexes is provided using metadata property fields (see `Metadata properties`_) of the array property as defined below: -- :field:`dimensions`: List of Dictionary. - A list of dictionaries which provide information related to the extents of an array property, which is relevant for slicing. - Each item, in order, represents the array dimension as declared in the corresponding property definition. +- :field:`array_axes`: List of Dictionary. + A list of dictionaries which provide information related to the axes of an array property, which is relevant for slicing. + Each item, in order, represents the array axis as declared in the corresponding property definition. Each item MUST be a dictionary with the following REQUIRED fields: - - :field:`name`: String. - The dimension name of the corresponding array dimension. + - :field:`dimension_name`: String. + The dimension name of the corresponding array axis. - If the request specifies the :query-param:`property_slices` query parameter for any of the array dimensions of this array property the following key MUST be present: + If the request specifies the :query-param:`property_slices` query parameter for any of the array axes of this array property the following key MUST be present, otherwise it MUST be omitted or equal to :val:`null`: - :field:`requested_slice`: Dictionary. A field that describes the requested slice that was provided via the query parameter :query-param:`property_slices`. @@ -665,12 +665,15 @@ Information relating to the ability of the server to handle this query parameter The dictionary MAY contain the following fields: - - :field:`size`: Integer. - The length of this dimension for the specific entry that this metadata entry pertains to. + - :field:`length`: Integer. + The length of this array axis which MUST be the same as the length of the declared dimension for this axis in the corresponding property definition. + Note that the length of a dimension can be different for different entries if the length is not explicitly declared by the property definition. + For example, the number of frames, i.e., the length of the dimension named ``dim_frames``, is generally different for different trajectories. - :field:`sliceable`: Boolean. - If true, the server MUST handle slices for that dimension. - If false (which is the default), the server MAY handle slices for that dimension, or MAY return the error :http-error:`501 Not Implemented` when a client requests a slice involving this dimension. + If :val:`true`, the server MUST handle slices for that dimension. + If :val:`false`, the server MAY handle slices for that dimension, or MAY return the error :http-error:`501 Not Implemented` when a client requests a slice involving this dimension. + If the field is omitted or :val:`null`, it means the same thing as :val:`false`. - :field:`available_slice`: Dictionary or :val:`null`. This field describes a `slice object`_ where there MAY be non-null values in the data. @@ -682,40 +685,73 @@ Information relating to the ability of the server to handle this query parameter - ``{"start": 3, "stop": 7, "step": 2}`` means the server certifies that values at indexes 0,1,2,4,6 and any index from 8 to the end of the array are :val:`null`. -Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format. -It communicates that the property value has been omitted from the response and metadata that makes it possible to request only a part of the data for this property. +Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format for a request to the trajectory endpoint with the query parameter :query-param:`property_slices=dim_frames:3:37:5` and :query-param:`response_fields=cartesian_site_positions,_exmpl_temperature` where the trajectory consists of 432934 frames (with indexes 0 to 432933) and where the :field:`cartisian_site_positions` contains 7 sites: .. code:: jsonc { // ... "data": { - "type": "structures", + "type": "trajectories", "id": "2345678", "attributes": { - "a": null - } + "cartesian_site_positions": null, + "_exmpl_temperature": null + }, "meta": { "property_metadata": { - "a": { - "range": { - "range_ids": ["frames", "particles"], - "indexable_dim": ["frames"], - "data_range": [ - { - "start": 1, - "step": 1, - "stop": 200, + "cartesian_site_positions": { + "array_axes": [ + { + "dimension_name": "dim_frames", + "requested_slice": { + "start": 3, + "stop": 37, + "step": 5 }, - { - "start": 1, - "step": 1, - "stop": 3, - } - ], - "layout": "dense", - "nvalues": 600 - } + "available_slice": { + "start": 0, + "stop": 432933, + "step": 1 + }, + "sliceable": true, + "length": 432934, + } + { + "dimension_name": "dim_sites", + "requested_slice": { + "start": 3, + "stop": 37, + "step": 5 + }, + "available_slice": { + "start": 0, + "stop": 432933, + "step": 1 + }, + "sliceable": true, + "length": 432934, + } + ], + }, + "_exmpl_temperature": { + "array_axes": [ + { + "dimension_name": "dim_frames", + "requested_slice": { + "start": 3, + "stop": 37, + "step": 5 + }, + "available_slice": { + "start": 1000, + "stop": 4000, + "step": 30 + }, + "sliceable": true, + "length": 432934, + } + ] } }, "partial_data_links": { From f86330818bda0774c205eecad713c95a0830ff62 Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Thu, 13 Jun 2024 22:49:14 +0200 Subject: [PATCH 17/70] Edit property slice definition [WIP] --- optimade.rst | 37 +++++++++++++++++++------------------ 1 file changed, 19 insertions(+), 18 deletions(-) diff --git a/optimade.rst b/optimade.rst index 7e70658bd..be1eb7711 100644 --- a/optimade.rst +++ b/optimade.rst @@ -539,8 +539,10 @@ Database providers are allowed to define their own metadata properties in :field For example, the metadata property definition of the field :field:`_exmpl_example_field` MUST NOT define a metadata field named, e.g., :field:`accuracy`; the field rather needs to be named, e.g., :field:`_exmpl_accuracy`. The reason for this limitation is to avoid name collisions with metadata fields defined by the OPTIMADE standard in the future that apply also to database-specific data fields. -Implementation of the :field:`meta` field is OPTIONAL. -However, when an implementation supports the :field:`property_metadata` field, it SHOULD include metadata fields for all properties which have metadata and are present in the data part of the response. +Implementation of the :field:`meta` field is OPTIONAL, unless the server implements slicing, in which case it is MANDATORY (see `Slices of array properties`_). +When an implementation supports the :field:`property_metadata` field, it SHOULD include metadata fields for all properties which have metadata and are present in the data part of the response. +If the client includes the string ``property_metadata`` in the query parameter :query-param:`response_fields`, the server MUST include metadata fields for all properties which have metadata. +Furthermore, if the server returns a metadata property, it must be included in its entirety, i.e., including all non-null fields. Example of a response in the JSON response format with two structure entries that each include a metadata property for the attribute field :field:`elements_ratios` and the database-specific per entry metadata field :field:`_exmpl_originates_from_project`: @@ -607,8 +609,6 @@ Example of the corresponding metadata property definition contained in the field Slices of array properties -------------------------- -# response_fields=property_metadata means MUST RETURN ALL METADATA. - The OPTIMADE standard defines a way for a client to request only a subset of the items of an array, referred to as a slice. The protocol for this functionality allows the server to allow slicing independently per entry, per array, and per array axis. This functionality is separate from the protocol described in `Transmission of large property values`_. @@ -616,7 +616,9 @@ Slices are used for a client to ask the server to only provide a subset of items In contrast, the protocol for large property values is used by the server implementation to transmit a set of items that it deems too large to provide inside the normal OPTIMADE response. The main mechanism is provided through the query parameter :query-param:`property_slices` defined in section `Single Entry URL Query Parameters`_. -Information relating to the ability of the server to handle this query parameter and the relevant ranges of indexes is provided using metadata property fields (see `Metadata properties`_) of the array property as defined below: +Information relating to the ability of the server to handle this query parameter and the relevant ranges of indexes is provided using metadata property field :field:`array_axes` (see `Metadata properties`_). +When the client request includes the query parameter :query-param:`property_slices`, the server MUST provide metadata for all properties for which including the subfield :field:`requested_slice` of the :field:`array_axes` is MANDATORY (see below). +The field :field:`array_axes` is defined as follows: - :field:`array_axes`: List of Dictionary. A list of dictionaries which provide information related to the axes of an array property, which is relevant for slicing. @@ -684,8 +686,7 @@ Information relating to the ability of the server to handle this query parameter - ``{"start": 3, "stop": 7, "step": 2}`` means the server certifies that values at indexes 0,1,2,4,6 and any index from 8 to the end of the array are :val:`null`. - -Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format for a request to the trajectory endpoint with the query parameter :query-param:`property_slices=dim_frames:3:37:5` and :query-param:`response_fields=cartesian_site_positions,_exmpl_temperature` where the trajectory consists of 432934 frames (with indexes 0 to 432933) and where the :field:`cartisian_site_positions` contains 7 sites: +Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format for a request to the trajectory endpoint with the query parameter :query-param:`property_slices=dim_frames:3:37:5` and :query-param:`response_fields=cartesian_site_positions,_exmpl_temperature` where the trajectory consists of 432934 frames (with indexes 0 to 432933) and where the :field:`cartisian_site_positions` contains 7 sites. Furthermore, the :field:`_exmpl_temperature` contains only :val:`null` values except for items with indexes 1000, 1030, 1060, ..., 4000 (where the values can be either numeric or :val:`null`). .. code:: jsonc @@ -716,21 +717,20 @@ Below follows an example of the :field:`data` and :field:`meta` parts of a respo }, "sliceable": true, "length": 432934, - } + }, { "dimension_name": "dim_sites", - "requested_slice": { - "start": 3, - "stop": 37, - "step": 5 - }, "available_slice": { "start": 0, - "stop": 432933, + "stop": 6, "step": 1 }, - "sliceable": true, - "length": 432934, + "sliceable": false, + "length": 7, + }, + { + "dimension_name": "dim_spatial", + "length": 3, } ], }, @@ -1277,11 +1277,12 @@ Standard OPTIONAL URL query parameters not in the JSON:API specification: ********* - + Additional OPTIONAL URL query parameters not described above are not considered to be part of this standard, and are instead considered to be "custom URL query parameters". These custom URL query parameters MUST be of the format "". These names adhere to the requirements on implementation-specific query parameters of `JSON:API v1.1 `__ since the database-provider-specific prefixes contain at least two underscores (a LOW LINE character '\_'). +If the server receives an unrecognized query parameter that does not adhere to this format it MUST return the error :http-error:`400 Bad Request`. Example uses of custom URL query parameters include providing an access token for the request, to tell the database to increase verbosity in error output, or providing a database-specific extended searching format. @@ -2440,7 +2441,7 @@ A Property Definition MUST be composed according to the combination of the requi The value :val:`TRUE` means the implementation includes the property in responses by default, i.e., when not specifically requested. The value :val:`FALSE` means that the property is only included when requested. Omitting the field or :val:`null` means the implementation does not declare if the property will be included in responses by default or not. - + - :field:`dimensions_supporting_ranges`: List of String or :val:`null`. A list of names of dimensions that appears in :field:`x-optimade-dimensions` fields within the property definition. A name appearing in the list means this dimension can be used in the :query-param:`property_ranges` query paramter to request subsets of the data along that dimension. From a73029a553d19fed344d5637626055e2e8c1be58 Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Thu, 13 Jun 2024 23:04:37 +0200 Subject: [PATCH 18/70] Edit property slice definition [WIP] --- optimade.rst | 5 ----- 1 file changed, 5 deletions(-) diff --git a/optimade.rst b/optimade.rst index be1eb7711..55b73eef3 100644 --- a/optimade.rst +++ b/optimade.rst @@ -526,7 +526,6 @@ It communicates that the property value has been omitted from the response and i // ... } - Metadata properties ------------------- @@ -1275,10 +1274,6 @@ Standard OPTIONAL URL query parameters not in the JSON:API specification: Other OPTIONAL fields MUST NOT be returned when this parameter is present. Example: :query-url:`http://example.com/optimade/v1/structures?response_fields=last_modified,nsites` - -********* - - Additional OPTIONAL URL query parameters not described above are not considered to be part of this standard, and are instead considered to be "custom URL query parameters". These custom URL query parameters MUST be of the format "". These names adhere to the requirements on implementation-specific query parameters of `JSON:API v1.1 `__ since the database-provider-specific prefixes contain at least two underscores (a LOW LINE character '\_'). From c098e4782b93c5e9091931ea897d2fa5df95898b Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Fri, 14 Jun 2024 09:11:47 +0200 Subject: [PATCH 19/70] Improve handing of Python dependencies --- optimade.rst | 37 +++++++++++-------------------------- 1 file changed, 11 insertions(+), 26 deletions(-) diff --git a/optimade.rst b/optimade.rst index 55b73eef3..bfbf3e16d 100644 --- a/optimade.rst +++ b/optimade.rst @@ -541,7 +541,7 @@ The reason for this limitation is to avoid name collisions with metadata fields Implementation of the :field:`meta` field is OPTIONAL, unless the server implements slicing, in which case it is MANDATORY (see `Slices of array properties`_). When an implementation supports the :field:`property_metadata` field, it SHOULD include metadata fields for all properties which have metadata and are present in the data part of the response. If the client includes the string ``property_metadata`` in the query parameter :query-param:`response_fields`, the server MUST include metadata fields for all properties which have metadata. -Furthermore, if the server returns a metadata property, it must be included in its entirety, i.e., including all non-null fields. +Furthermore, if the server returns metadata for a property, it must be included in its entirety, i.e., including all non-null fields. Example of a response in the JSON response format with two structure entries that each include a metadata property for the attribute field :field:`elements_ratios` and the database-specific per entry metadata field :field:`_exmpl_originates_from_project`: @@ -1373,34 +1373,27 @@ The meaning of these URL query parameters are as defined above in section `Entry One additional query parameter :query-param:`property_slices` MUST be handled by the API implementation either as defined below or by returning the error :http-error:`501 Not Implemented`: -- **property\_slices**: A set of slice objects which specify which sub-range of a property should be returned. - - The property to which this sub-range should be applied can be specified via the :query-param:`response_fields` query parameter. - Each slice consists of the name of a dimension directly followed by the requested slice in this dimension. - The dimension name and the start, stop and step values of the slice are separated by colons (":", ASCII 58(0x3A)) - The slice is defined in a similar manner as the `slice object`_. - If no value is placed between the colons for a component of the slice then the default value is used. +- **property\_slices**: A number of slice specifications to specify parts of array properties for the functionality described in `Slices of array properties`_. + The query parameter contains a comma-separated (",", ASCII 44(0x2C)) list of slice specification. + Each slice specification consists of a dimension name and the start, stop and step values of the slice separated by the colon character (":", ASCII 58(0x3A)) + Omitting the value between two colons specifies a default value. + The components of the slice are defined in the same way as for a `slice object`_. The first integer specifies the start of the slice, i.e. the first index in that dimension for which values should be returned. The default is the index of the first value. The second integer specifies the end/stop of the slice, i.e. the last index for which values should be returned. The default is the index of the last value of the property. The third integer specifies the step size in that dimension. The default is :val:`1` + The slices are 0-based, i.e. the first value has index 0, and inclusive i.e. for the dimension :val:`dim_frames` the range :val:`dim_frames:10:20:1` the last value returned belongs to index 20. - Multidimensional slices can be defined by specifying a range for each dimension. - These ranges are separated by a comma (",", ASCII 44(0x2C)). - The slices are 0-based, i.e. the first value has index 0, and inclusive i.e. for the dimension :val:`dim` the range :val:`dim:10:20:1` the last value returned belongs to index 20. - - If an array dimension is indicated as slicable, the response MUST include the items in the requested slice. - - If an array dimension is not indicated as slicable, the response MUST be one of the following: (i) An error is returned; or (ii) the items in the requested slice is returned. + Requirements and conventions for the response when this query parameter is used are described in `Slices of array properties`_. - If a dimension is not specified, it is assumed that all list items in that dimension is requested. + Example: - Example: In the following example, items from the Cartesian site positions array is requested only for the 31st to 71st sites (i.e., with indexes 30 through 70 inclusive) for 1 out of every 10 frames of the first 1000 frames (i.e., taking steps of 10 over indexes 0 through 999 inclusive, which requests the frames with index 0, 10, 20, 30, ..., 990) of a trajectory with ID :val:`id_12345`. + - :query-url:`http://optimade.example.com/v1/trajectories/id_12345?response_fields=cartesian_site_positions&property_ranges=dim_frames::999:10,dim_sites:30:70::` - :query-url:`http://example.com/optimade/v1/trajectories/id_12345?response_fields=cartesian_site_positions&property_ranges=dim_frames::999:10,dim_sites:30:70::` + This query URL requests items from the array :field:`cartesian_site_positions` is requested only for the 31st to 71st sites (i.e., with indexes 30 through 70 inclusive) for 1 out of every 10 frames of the first 1000 frames (i.e., taking steps of 10 over indexes 0 through 999 inclusive, which requests the frames with index 0, 10, 20, 30, ..., 990) of a trajectory with ID :val:`id_12345`. Single Entry JSON Response Schema ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -2437,10 +2430,6 @@ A Property Definition MUST be composed according to the combination of the requi The value :val:`FALSE` means that the property is only included when requested. Omitting the field or :val:`null` means the implementation does not declare if the property will be included in responses by default or not. - - :field:`dimensions_supporting_ranges`: List of String or :val:`null`. - A list of names of dimensions that appears in :field:`x-optimade-dimensions` fields within the property definition. - A name appearing in the list means this dimension can be used in the :query-param:`property_ranges` query paramter to request subsets of the data along that dimension. - - :field:`x-optimade-requirements`: Dictionary. A dictionary describing the level of OPTIMADE API functionality required by all implementations of this property. Omitting this field means the corresponding functionality is OPTIONAL. @@ -2478,10 +2467,6 @@ A Property Definition MUST be composed according to the combination of the requi Omitting the field is equivalent to :val:`may`. - - :field:`dimensions_supporting_ranges`: List of String or :val:`null`. - A list of names of dimensions that appears in :field:`x-optimade-dimensions` fields within the property definition. - Expresses if an implementation of this property is required to support using this dimension in the :query-param:`property_ranges` query paramter to request subsets of the data along that dimension. - Property Definition keys from JSON Schema ----------------------------------------- From 4f704c161dd8d6010d6b9523ea8c28953fcc0e21 Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Fri, 14 Jun 2024 09:12:55 +0200 Subject: [PATCH 20/70] Improve handing of Python dependencies --- optimade.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/optimade.rst b/optimade.rst index bfbf3e16d..35c37ea7d 100644 --- a/optimade.rst +++ b/optimade.rst @@ -648,7 +648,7 @@ The field :field:`array_axes` is defined as follows: Examples: - - For :?:`property_slices=dim_frames:::` some of the equivalent valid representations are: + - For :query-string:`property_slices=dim_frames:::` some of the equivalent valid representations are: - ``{}`` - ``{"start": null, "stop": null, "step": null}`` @@ -658,9 +658,9 @@ The field :field:`array_axes` is defined as follows: Whereas, e.g., the following representation is not valid: - ``{"start": 0}`` (despite that :val:`0` is the default value of the :field:`start` field.) - But it is instead a valid representation if the query parameter was :?:`property_slices=dim_frames:0::`. + But it is instead a valid representation if the query parameter was :query-string:`property_slices=dim_frames:0::`. - - :?:`property_slices=dim_frames:3:5:2` the representation MUST be: + - :query-string:`property_slices=dim_frames:3:5:2` the representation MUST be: - ``{"start": 3, "stop": 5, "step": 2}`` From 266545b3c0279c20eae243c0990120c667f8f8e9 Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Fri, 14 Jun 2024 10:16:41 +0200 Subject: [PATCH 21/70] Improve handing of Python dependencies --- optimade.rst | 801 ++++++++++++++++++++++++++------------------------- 1 file changed, 401 insertions(+), 400 deletions(-) diff --git a/optimade.rst b/optimade.rst index 35c37ea7d..4e688f709 100644 --- a/optimade.rst +++ b/optimade.rst @@ -660,106 +660,106 @@ The field :field:`array_axes` is defined as follows: - ``{"start": 0}`` (despite that :val:`0` is the default value of the :field:`start` field.) But it is instead a valid representation if the query parameter was :query-string:`property_slices=dim_frames:0::`. - - :query-string:`property_slices=dim_frames:3:5:2` the representation MUST be: +- :query-string:`property_slices=dim_frames:3:5:2` the representation MUST be: - - ``{"start": 3, "stop": 5, "step": 2}`` +- ``{"start": 3, "stop": 5, "step": 2}`` - The dictionary MAY contain the following fields: +The dictionary MAY contain the following fields: - - :field:`length`: Integer. - The length of this array axis which MUST be the same as the length of the declared dimension for this axis in the corresponding property definition. - Note that the length of a dimension can be different for different entries if the length is not explicitly declared by the property definition. - For example, the number of frames, i.e., the length of the dimension named ``dim_frames``, is generally different for different trajectories. +- :field:`length`: Integer. +The length of this array axis which MUST be the same as the length of the declared dimension for this axis in the corresponding property definition. +Note that the length of a dimension can be different for different entries if the length is not explicitly declared by the property definition. +For example, the number of frames, i.e., the length of the dimension named ``dim_frames``, is generally different for different trajectories. - - :field:`sliceable`: Boolean. - If :val:`true`, the server MUST handle slices for that dimension. - If :val:`false`, the server MAY handle slices for that dimension, or MAY return the error :http-error:`501 Not Implemented` when a client requests a slice involving this dimension. - If the field is omitted or :val:`null`, it means the same thing as :val:`false`. +- :field:`sliceable`: Boolean. +If :val:`true`, the server MUST handle slices for that dimension. +If :val:`false`, the server MAY handle slices for that dimension, or MAY return the error :http-error:`501 Not Implemented` when a client requests a slice involving this dimension. +If the field is omitted or :val:`null`, it means the same thing as :val:`false`. - - :field:`available_slice`: Dictionary or :val:`null`. - This field describes a `slice object`_ where there MAY be non-null values in the data. - By including this field, the server certifies that there are only :val:`null` values outside this slice. - If not provided, or equal to :val:`null` or an empty dictionary, the client cannot make any assumptions about what part of the array contains :val:`null` values. +- :field:`available_slice`: Dictionary or :val:`null`. +This field describes a `slice object`_ where there MAY be non-null values in the data. +By including this field, the server certifies that there are only :val:`null` values outside this slice. +If not provided, or equal to :val:`null` or an empty dictionary, the client cannot make any assumptions about what part of the array contains :val:`null` values. - Examples: +Examples: - - ``{"start": 3, "stop": 7, "step": 2}`` means the server certifies that values at indexes 0,1,2,4,6 and any index from 8 to the end of the array are :val:`null`. +- ``{"start": 3, "stop": 7, "step": 2}`` means the server certifies that values at indexes 0,1,2,4,6 and any index from 8 to the end of the array are :val:`null`. Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format for a request to the trajectory endpoint with the query parameter :query-param:`property_slices=dim_frames:3:37:5` and :query-param:`response_fields=cartesian_site_positions,_exmpl_temperature` where the trajectory consists of 432934 frames (with indexes 0 to 432933) and where the :field:`cartisian_site_positions` contains 7 sites. Furthermore, the :field:`_exmpl_temperature` contains only :val:`null` values except for items with indexes 1000, 1030, 1060, ..., 4000 (where the values can be either numeric or :val:`null`). .. code:: jsonc - { - // ... - "data": { - "type": "trajectories", - "id": "2345678", - "attributes": { - "cartesian_site_positions": null, - "_exmpl_temperature": null - }, - "meta": { - "property_metadata": { - "cartesian_site_positions": { - "array_axes": [ - { - "dimension_name": "dim_frames", - "requested_slice": { - "start": 3, - "stop": 37, - "step": 5 - }, - "available_slice": { - "start": 0, - "stop": 432933, - "step": 1 - }, - "sliceable": true, - "length": 432934, - }, - { - "dimension_name": "dim_sites", - "available_slice": { - "start": 0, - "stop": 6, - "step": 1 - }, - "sliceable": false, - "length": 7, - }, - { - "dimension_name": "dim_spatial", - "length": 3, - } - ], - }, - "_exmpl_temperature": { - "array_axes": [ - { - "dimension_name": "dim_frames", - "requested_slice": { - "start": 3, - "stop": 37, - "step": 5 - }, - "available_slice": { - "start": 1000, - "stop": 4000, - "step": 30 - }, - "sliceable": true, - "length": 432934, - } - ] - } - }, - "partial_data_links": { - //... - } - } - } - // ... - } +{ +// ... +"data": { +"type": "trajectories", +"id": "2345678", +"attributes": { +"cartesian_site_positions": null, +"_exmpl_temperature": null +}, +"meta": { +"property_metadata": { +"cartesian_site_positions": { +"array_axes": [ + { + "dimension_name": "dim_frames", + "requested_slice": { + "start": 3, + "stop": 37, + "step": 5 + }, + "available_slice": { + "start": 0, + "stop": 432933, + "step": 1 + }, + "sliceable": true, + "length": 432934, + }, + { + "dimension_name": "dim_sites", + "available_slice": { + "start": 0, + "stop": 6, + "step": 1 + }, + "sliceable": false, + "length": 7, + }, + { + "dimension_name": "dim_spatial", + "length": 3, + } +], +}, +"_exmpl_temperature": { +"array_axes": [ + { + "dimension_name": "dim_frames", + "requested_slice": { + "start": 3, + "stop": 37, + "step": 5 + }, + "available_slice": { + "start": 1000, + "stop": 4000, + "step": 30 + }, + "sliceable": true, + "length": 432934, + } +] +} +}, +"partial_data_links": { +//... +} +} +} +// ... +} Responses ========= @@ -790,305 +790,305 @@ In the JSON response format, property types translate as follows: Every response SHOULD contain the following fields, and MUST contain at least :field:`meta`: - **meta**: a `JSON:API meta member `__ that contains JSON:API meta objects of non-standard meta-information. - It MUST be a dictionary with these fields: +It MUST be a dictionary with these fields: - - **api\_version**: a string containing the full version of the API implementation. - The version number string MUST NOT be prefixed by, e.g., "v". - Examples: :field-val:`1.0.0`, :field-val:`1.0.0-rc.2`. +- **api\_version**: a string containing the full version of the API implementation. +The version number string MUST NOT be prefixed by, e.g., "v". +Examples: :field-val:`1.0.0`, :field-val:`1.0.0-rc.2`. - - **query**: information on the query that was requested. - It MUST be a dictionary with this field: +- **query**: information on the query that was requested. +It MUST be a dictionary with this field: - - **representation**: a string with the part of the URL following the versioned or unversioned base URL that serves the API. - Query parameters that have not been used in processing the request MAY be omitted. - In particular, if no query parameters have been involved in processing the request, the query part of the URL MAY be excluded. - Example: :field-val:`/structures?filter=nelements=2`. +- **representation**: a string with the part of the URL following the versioned or unversioned base URL that serves the API. +Query parameters that have not been used in processing the request MAY be omitted. +In particular, if no query parameters have been involved in processing the request, the query part of the URL MAY be excluded. +Example: :field-val:`/structures?filter=nelements=2`. - - **more\_data\_available**: :field-val:`false` if the response contains all data for the request (e.g., a request issued to a single entry endpoint, or a :query-param:`filter` query at the last page of a paginated response) and :field-val:`true` if the response is incomplete in the sense that multiple objects match the request, and not all of them have been included in the response (e.g., a query with multiple pages that is not at the last page). +- **more\_data\_available**: :field-val:`false` if the response contains all data for the request (e.g., a request issued to a single entry endpoint, or a :query-param:`filter` query at the last page of a paginated response) and :field-val:`true` if the response is incomplete in the sense that multiple objects match the request, and not all of them have been included in the response (e.g., a query with multiple pages that is not at the last page). - :field:`meta` SHOULD also include these fields: +:field:`meta` SHOULD also include these fields: - - **time\_stamp**: a timestamp containing the date and time at which the query was executed. - - **data\_returned**: an integer containing the total number of data resource objects returned for the current :query-param:`filter` query, independent of pagination. - - **provider**: information on the database provider of the implementation. - It MUST be a dictionary with these fields: +- **time\_stamp**: a timestamp containing the date and time at which the query was executed. +- **data\_returned**: an integer containing the total number of data resource objects returned for the current :query-param:`filter` query, independent of pagination. +- **provider**: information on the database provider of the implementation. +It MUST be a dictionary with these fields: - - **name**: a short name for the database provider. - - **description**: a longer description of the database provider. - - **prefix**: database-provider-specific prefix (see section `Database-Provider-Specific Namespace Prefixes`_). +- **name**: a short name for the database provider. +- **description**: a longer description of the database provider. +- **prefix**: database-provider-specific prefix (see section `Database-Provider-Specific Namespace Prefixes`_). - :field:`provider` MAY include these fields: +:field:`provider` MAY include these fields: - - **homepage**: a `JSON API link `__, pointing to the homepage of the database provider, either directly as a string, or as an object which can contain the following fields: +- **homepage**: a `JSON API link `__, pointing to the homepage of the database provider, either directly as a string, or as an object which can contain the following fields: - - **href**: a string containing the homepage URL. - - **meta**: a meta object containing non-standard meta-information about the database provider's homepage. +- **href**: a string containing the homepage URL. +- **meta**: a meta object containing non-standard meta-information about the database provider's homepage. - :field:`meta` MAY also include these fields: +:field:`meta` MAY also include these fields: - - **data\_available**: an integer containing the total number of data resource objects available in the database for the endpoint. - - **last\_id**: a string containing the last ID returned. - - **response\_message**: response string from the server. - - **request\_delay**: a non-negative float giving time in seconds that the client is suggested to wait before issuing a subsequent request. +- **data\_available**: an integer containing the total number of data resource objects available in the database for the endpoint. +- **last\_id**: a string containing the last ID returned. +- **response\_message**: response string from the server. +- **request\_delay**: a non-negative float giving time in seconds that the client is suggested to wait before issuing a subsequent request. - Implementation note: the functionality of this field overlaps to some degree with features provided by the HTTP error :http-error:`429 Too Many Requests` and the `Retry-After HTTP header `__. - Implementations are suggested to provide consistent handling of request overload through both mechanisms. +Implementation note: the functionality of this field overlaps to some degree with features provided by the HTTP error :http-error:`429 Too Many Requests` and the `Retry-After HTTP header `__. +Implementations are suggested to provide consistent handling of request overload through both mechanisms. - - **database**: a dictionary describing the specific database accessible at this OPTIMADE API. - If provided, the dictionary fields SHOULD match those provided in the corresponding links entry for the database in the provider's index meta-database, outlined in `Links Endpoint JSON Response Schema`_. - The dictionary can contain the following OPTIONAL fields: +- **database**: a dictionary describing the specific database accessible at this OPTIMADE API. +If provided, the dictionary fields SHOULD match those provided in the corresponding links entry for the database in the provider's index meta-database, outlined in `Links Endpoint JSON Response Schema`_. +The dictionary can contain the following OPTIONAL fields: - - **id**: the identifier of this database within those served by this provider, i.e., the ID under which this database is served in this provider's index meta-database. - - **name**: a human-readable name for the database, e.g., for use in clients. - - **version**: a string describing the version of the database. - - **description**: a human-readable description of the database, e.g., for use in clients. - - **homepage**: a `JSON API link `__, pointing to a homepage for the particular database. - - **maintainer**: a dictionary providing details about the maintainer of the database, which MUST contain the single field: +- **id**: the identifier of this database within those served by this provider, i.e., the ID under which this database is served in this provider's index meta-database. +- **name**: a human-readable name for the database, e.g., for use in clients. +- **version**: a string describing the version of the database. +- **description**: a human-readable description of the database, e.g., for use in clients. +- **homepage**: a `JSON API link `__, pointing to a homepage for the particular database. +- **maintainer**: a dictionary providing details about the maintainer of the database, which MUST contain the single field: - - **email** with the maintainer's email address. +- **email** with the maintainer's email address. - - **implementation**: a dictionary describing the server implementation, containing the OPTIONAL fields: +- **implementation**: a dictionary describing the server implementation, containing the OPTIONAL fields: - - **name**: name of the implementation. - - **version**: version string of the current implementation. - - **homepage**: a `JSON API link `__, pointing to the homepage of the implementation. - - **source\_url**: a `JSON API link `__ pointing to the implementation source, either downloadable archive or version control system. +- **name**: name of the implementation. +- **version**: version string of the current implementation. +- **homepage**: a `JSON API link `__, pointing to the homepage of the implementation. +- **source\_url**: a `JSON API link `__ pointing to the implementation source, either downloadable archive or version control system. - - **maintainer**: a dictionary providing details about the maintainer of the implementation, MUST contain the single field: +- **maintainer**: a dictionary providing details about the maintainer of the implementation, MUST contain the single field: - - **email** with the maintainer's email address. +- **email** with the maintainer's email address. - - **issue\_tracker**: a `JSON API link `__ pointing to the implementation's issue tracker. +- **issue\_tracker**: a `JSON API link `__ pointing to the implementation's issue tracker. - - **warnings**: a list of warning resource objects representing non-critical errors or warnings. - A warning resource object is defined similarly to a `JSON:API error object `__, but MUST also include the field :field:`type`, which MUST have the value :field-val:`"warning"`. - The field :field:`detail` MUST be present and SHOULD contain a non-critical message, e.g., reporting unrecognized search attributes or deprecated features. - The field :field:`status`, representing an HTTP response status code, MUST NOT be present for a warning resource object. - This is an exclusive field for error resource objects. +- **warnings**: a list of warning resource objects representing non-critical errors or warnings. +A warning resource object is defined similarly to a `JSON:API error object `__, but MUST also include the field :field:`type`, which MUST have the value :field-val:`"warning"`. +The field :field:`detail` MUST be present and SHOULD contain a non-critical message, e.g., reporting unrecognized search attributes or deprecated features. +The field :field:`status`, representing an HTTP response status code, MUST NOT be present for a warning resource object. +This is an exclusive field for error resource objects. - Example for a deprecation warning: +Example for a deprecation warning: - .. code:: jsonc +.. code:: jsonc - { - "id": "dep_chemical_formula_01", - "type": "warning", - "code": "_exmpl_dep_chemical_formula", - "title": "Deprecation Warning", - "detail": "chemical_formula is deprecated, use instead chemical_formula_hill" - } +{ +"id": "dep_chemical_formula_01", +"type": "warning", +"code": "_exmpl_dep_chemical_formula", +"title": "Deprecation Warning", +"detail": "chemical_formula is deprecated, use instead chemical_formula_hill" +} - **Note**: warning :field:`id`\ s MUST NOT be trusted to identify the exceptional situations (i.e., they are not error codes), use instead the field :field:`code` for this. - Warning :field:`id`\ s can *only* be trusted to be unique in the list of warning resource objects, i.e., together with the :field:`type`. +**Note**: warning :field:`id`\ s MUST NOT be trusted to identify the exceptional situations (i.e., they are not error codes), use instead the field :field:`code` for this. +Warning :field:`id`\ s can *only* be trusted to be unique in the list of warning resource objects, i.e., together with the :field:`type`. - General OPTIMADE warning codes are specified in section `Warnings`_. +General OPTIMADE warning codes are specified in section `Warnings`_. - - Other OPTIONAL additional information *global to the query* that is not specified in this document, MUST start with a database-provider-specific prefix (see section `Database-Provider-Specific Namespace Prefixes`_). +- Other OPTIONAL additional information *global to the query* that is not specified in this document, MUST start with a database-provider-specific prefix (see section `Database-Provider-Specific Namespace Prefixes`_). - - Example for a request made to :query-url:`http://example.com/optimade/v1/structures/?filter=a=1 AND b=2`: +- Example for a request made to :query-url:`http://example.com/optimade/v1/structures/?filter=a=1 AND b=2`: - .. code:: jsonc - - { - "meta": { - "query": { - "representation": "/structures/?filter=a=1 AND b=2" - }, - "api_version": "1.0.0", - "schema": "http://schemas.optimade.org/openapi/v1/optimade.json", - "time_stamp": "2007-04-05T14:30:20Z", - "data_returned": 10, - "data_available": 10, - "more_data_available": false, - "provider": { - "name": "Example provider", - "description": "Provider used for examples, not to be assigned to a real database", - "prefix": "exmpl", - "homepage": "http://example.com" - }, - "implementation": { - "name": "exmpl-optimade", - "version": "0.1.0", - "source_url": "http://git.example.com/exmpl-optimade", - "maintainer": { - "email": "admin@example.com" - }, - "issue_tracker": "http://tracker.example.com/exmpl-optimade" - }, - "database": { - "id": "example_db", - "name": "Example database 1 (of many)", - "description": "The first example database in a series hosted by the Example Provider.", - "homepage": "http://database_one.example.com", - "maintainer": { - "email": "science_lead@example.com" - } - } - } - // ... - } - - - **schema**: a `JSON:API links object `__ that points to a schema for the response. - If it is a string, or a dictionary containing no :field:`meta` field, the provided URL MUST point at an `OpenAPI `__ schema. - It is possible that future versions of this specification allow for alternative schema types. - Hence, if the :field:`meta` field of the JSON:API links object is provided and contains a field :field:`schema_type` that is not equal to the string :field-val:`OpenAPI` the client MUST NOT handle failures to parse the schema or to validate the response against the schema as errors. +.. code:: jsonc - **Note**: The :field:`schema` field was previously RECOMMENDED in all responses, but is now demoted to being OPTIONAL since there now is a standard way of specifying a response schema in JSON:API through the :field:`describedby` subfield of the top-level :field:`links` field. +{ +"meta": { +"query": { +"representation": "/structures/?filter=a=1 AND b=2" +}, +"api_version": "1.0.0", +"schema": "http://schemas.optimade.org/openapi/v1/optimade.json", +"time_stamp": "2007-04-05T14:30:20Z", +"data_returned": 10, +"data_available": 10, +"more_data_available": false, +"provider": { +"name": "Example provider", +"description": "Provider used for examples, not to be assigned to a real database", +"prefix": "exmpl", +"homepage": "http://example.com" +}, +"implementation": { +"name": "exmpl-optimade", +"version": "0.1.0", +"source_url": "http://git.example.com/exmpl-optimade", +"maintainer": { +"email": "admin@example.com" +}, +"issue_tracker": "http://tracker.example.com/exmpl-optimade" +}, +"database": { +"id": "example_db", +"name": "Example database 1 (of many)", +"description": "The first example database in a series hosted by the Example Provider.", +"homepage": "http://database_one.example.com", +"maintainer": { +"email": "science_lead@example.com" +} +} +} +// ... +} + +- **schema**: a `JSON:API links object `__ that points to a schema for the response. +If it is a string, or a dictionary containing no :field:`meta` field, the provided URL MUST point at an `OpenAPI `__ schema. +It is possible that future versions of this specification allow for alternative schema types. +Hence, if the :field:`meta` field of the JSON:API links object is provided and contains a field :field:`schema_type` that is not equal to the string :field-val:`OpenAPI` the client MUST NOT handle failures to parse the schema or to validate the response against the schema as errors. + +**Note**: The :field:`schema` field was previously RECOMMENDED in all responses, but is now demoted to being OPTIONAL since there now is a standard way of specifying a response schema in JSON:API through the :field:`describedby` subfield of the top-level :field:`links` field. - **data**: The schema of this value varies by endpoint, it can be either a *single* `JSON:API resource object `__ or a *list* of JSON:API resource objects. - Every resource object needs the :field:`type` and :field:`id` fields, and its attributes (described in section `API Endpoints`_) need to be in a dictionary corresponding to the :field:`attributes` field. +Every resource object needs the :field:`type` and :field:`id` fields, and its attributes (described in section `API Endpoints`_) need to be in a dictionary corresponding to the :field:`attributes` field. - Every resource object MAY also contain a :field:`meta` field which MAY contain the following keys: +Every resource object MAY also contain a :field:`meta` field which MAY contain the following keys: - - **property_metadata**: an object containing per-entry and per-property metadata. - The keys are the names of the fields in :field:`attributes` for which metadata is available. - The values belonging to these keys are dictionaries containing the relevant metadata fields. - See also `Metadata properties`_ +- **property_metadata**: an object containing per-entry and per-property metadata. +The keys are the names of the fields in :field:`attributes` for which metadata is available. +The values belonging to these keys are dictionaries containing the relevant metadata fields. +See also `Metadata properties`_ - - **partial_data_links**: an object used to list links which can be used to fetch data that has been omitted from the :field:`data` part of the response. - The keys are the names of the fields in :field:`attributes` for which partial data links are available. - Each value is a list of objects that MUST have the following keys: +- **partial_data_links**: an object used to list links which can be used to fetch data that has been omitted from the :field:`data` part of the response. +The keys are the names of the fields in :field:`attributes` for which partial data links are available. +Each value is a list of objects that MUST have the following keys: - - **format**: String. - The name of the format provided via this link. - For one of the objects this :field:`format` field SHOULD have the value "jsonlines", which refers to the format in `OPTIMADE JSON lines partial data format`_. +- **format**: String. +The name of the format provided via this link. +For one of the objects this :field:`format` field SHOULD have the value "jsonlines", which refers to the format in `OPTIMADE JSON lines partial data format`_. - - **link**: String. - A `JSON API link `__ that points to a location from which the omitted data can be fetched. - There is no requirement on the syntax or format for the link URL. +- **link**: String. +A `JSON API link `__ that points to a location from which the omitted data can be fetched. +There is no requirement on the syntax or format for the link URL. - For more information about the mechanism to transmit large property values, including an example of the format of :field:`partial_data_links`, see `Transmission of large property values`_. +For more information about the mechanism to transmit large property values, including an example of the format of :field:`partial_data_links`, see `Transmission of large property values`_. The response MAY also return resources related to the primary data in the field: - **links**: a `JSON API links object `__ is REQUIRED for implementing pagination. - (see section `Entry Listing URL Query Parameters`_.) - Each field of a links object, i.e., a "link", MUST be one of: +(see section `Entry Listing URL Query Parameters`_.) +Each field of a links object, i.e., a "link", MUST be one of: - - :field-val:`null` - - a string representing a URI, or - - a dictionary ("link object") with fields +- :field-val:`null` +- a string representing a URI, or +- a dictionary ("link object") with fields - - **href**: a string representing a URI - - **meta**: (OPTIONAL) a meta object containing non-standard meta-information about the link +- **href**: a string representing a URI +- **meta**: (OPTIONAL) a meta object containing non-standard meta-information about the link - Example links objects: +Example links objects: - - **base\_url**: a links object representing the base URL of the implementation. Example: +- **base\_url**: a links object representing the base URL of the implementation. Example: - .. code:: jsonc +.. code:: jsonc - { - "links": { - "base_url": { - "href": "http://example.com/optimade", - "meta": { - "_exmpl_db_version": "3.2.1" - } - } - // ... - } - // ... - } +{ +"links": { +"base_url": { +"href": "http://example.com/optimade", +"meta": { +"_exmpl_db_version": "3.2.1" +} +} +// ... +} +// ... +} - The :field:`links` field SHOULD include the following links objects: +The :field:`links` field SHOULD include the following links objects: - - **describedby**: a links object giving the URL for a schema that describes the response. - The URL SHOULD resolve into a JSON formatted response returning a JSON object with top level :field:`$schema` and/or :field:`$id` fields that can be used by the client to identify the schema format. +- **describedby**: a links object giving the URL for a schema that describes the response. +The URL SHOULD resolve into a JSON formatted response returning a JSON object with top level :field:`$schema` and/or :field:`$id` fields that can be used by the client to identify the schema format. - **Note**: This field is the standard facility in JSON:API to communicate a response schema. - It overlaps in function with the field :field:`schema` in the top level :field:`meta` field. +**Note**: This field is the standard facility in JSON:API to communicate a response schema. +It overlaps in function with the field :field:`schema` in the top level :field:`meta` field. - The following fields are REQUIRED for implementing pagination: +The following fields are REQUIRED for implementing pagination: - - **next**: represents a link to fetch the next set of results. - When the current response is the last page of data, this field MUST be either omitted or :field-val:`null`\ -valued. +- **next**: represents a link to fetch the next set of results. +When the current response is the last page of data, this field MUST be either omitted or :field-val:`null`\ -valued. - An implementation MAY also use the following reserved fields for pagination. - They represent links in a similar way as for :field:`next`. +An implementation MAY also use the following reserved fields for pagination. +They represent links in a similar way as for :field:`next`. - - **prev**: the previous page of data. :field-val:`null` or omitted when the current response is the first page of data. - - **last**: the last page of data. - - **first**: the first page of data. +- **prev**: the previous page of data. :field-val:`null` or omitted when the current response is the first page of data. +- **last**: the last page of data. +- **first**: the first page of data. - Finally, the :field:`links` field MAY also include the following links object: +Finally, the :field:`links` field MAY also include the following links object: - - **self**: a links object giving the URL from which the response was obtained. +- **self**: a links object giving the URL from which the response was obtained. - **included**: a list of `JSON:API resource objects `__ related to the primary data contained in :field:`data`. - Responses that contain related resources under :field:`included` are known as `compound documents `__ in the JSON:API. +Responses that contain related resources under :field:`included` are known as `compound documents `__ in the JSON:API. - The definition of this field is found in the `JSON:API specification `__. - Specifically, if the query parameter :query-param:`include` is included in the request, :field:`included` MUST NOT include unrequested resource objects. - For further information on the parameter :query-param:`include`, see section `Entry Listing URL Query Parameters`_. +The definition of this field is found in the `JSON:API specification `__. +Specifically, if the query parameter :query-param:`include` is included in the request, :field:`included` MUST NOT include unrequested resource objects. +For further information on the parameter :query-param:`include`, see section `Entry Listing URL Query Parameters`_. - This value MUST be either an empty array or an array of related resource objects. +This value MUST be either an empty array or an array of related resource objects. If there were errors in producing the response all other fields MAY be present, but the top-level :field:`data` field MUST be skipped, and the following field MUST be present: - **errors**: a list of `JSON:API error objects `__, where the field :field:`detail` MUST be present. - All other fields are OPTIONAL. +All other fields are OPTIONAL. An example of a full response: .. code:: jsonc - { - "links": { - "next": null, - "base_url": { - "href": "http://example.com/optimade", - "meta": { - "_exmpl_db_version": "3.2.1" - } - } - }, - "meta": { - "query": { - "representation": "/structures?filter=a=1 AND b=2" - }, - "api_version": "1.0.0", - "time_stamp": "2007-04-05T14:30:20Z", - "data_returned": 10, - "data_available": 10, - "last_id": "xy10", - "more_data_available": false, - "provider": { - "name": "Example provider", - "description": "Provider used for examples, not to be assigned to a real database", - "prefix": "exmpl", - "homepage": { - "href": "http://example.com", - "meta": { - "_exmpl_title": "This is an example site" - } - } - }, - // - }, - "data": [ - // ... - ], - "included": [ - // ... - ] - } +{ +"links": { +"next": null, +"base_url": { +"href": "http://example.com/optimade", +"meta": { +"_exmpl_db_version": "3.2.1" +} +} +}, +"meta": { +"query": { +"representation": "/structures?filter=a=1 AND b=2" +}, +"api_version": "1.0.0", +"time_stamp": "2007-04-05T14:30:20Z", +"data_returned": 10, +"data_available": 10, +"last_id": "xy10", +"more_data_available": false, +"provider": { +"name": "Example provider", +"description": "Provider used for examples, not to be assigned to a real database", +"prefix": "exmpl", +"homepage": { +"href": "http://example.com", +"meta": { +"_exmpl_title": "This is an example site" +} +} +}, +// +}, +"data": [ +// ... +], +"included": [ +// ... +] +} - **@context**: A JSON-LD context that enables interpretation of data in the response as linked data. - If provided, it SHOULD be one of the following: +If provided, it SHOULD be one of the following: - - An object conforming to a JSON-LD standard, which includes a :field:`@version` field specifying the version of the standard. - - A string containing a URL that resolves to such an object. +- An object conforming to a JSON-LD standard, which includes a :field:`@version` field specifying the version of the standard. +- A string containing a URL that resolves to such an object. - **jsonapi**: A `JSON:API object `__. - The :field:`version` subfield SHOULD be :field-val:`"1.1"`. - The :field:`meta` subfield SHOULD be included and contain the following subfields: +The :field:`version` subfield SHOULD be :field-val:`"1.1"`. +The :field:`meta` subfield SHOULD be included and contain the following subfields: - - **api**: A string with the value "OPTIMADE". - - **api-version**: A string with the full version of the OPTIMADE standard that the processing and response adheres to. - This MAY be the version indicated at the top of this document, but MAY also be another version if the client, e.g., has used the query parameter :query-param:`api_hint` to request processing according to another version. +- **api**: A string with the value "OPTIMADE". +- **api-version**: A string with the full version of the OPTIMADE standard that the processing and response adheres to. +This MAY be the version indicated at the top of this document, but MAY also be another version if the client, e.g., has used the query parameter :query-param:`api_hint` to request processing according to another version. - If the server is able to handle serialization in such a way that it can dictate the order of the top level object members in the response, it is RECOMMENDED to put the :field:`jsonapi` as the first top level member to simplify identification of the response. +If the server is able to handle serialization in such a way that it can dictate the order of the top level object members in the response, it is RECOMMENDED to put the :field:`jsonapi` as the first top level member to simplify identification of the response. HTTP Response Status Codes -------------------------- @@ -1160,9 +1160,9 @@ Example response: .. code:: CSV - version - 1 - 0 +version +1 +0 The above response means that the API versions 1 and 0 are served under the versioned base URLs :query-url:`/v1` and :query-url:`/v0`, respectively. The order of the versions indicates that the API implementation regards version 1 as preferred over version 0. @@ -1178,14 +1178,14 @@ Each entry in the list includes a set of properties and their corresponding valu The section `Entry list`_ specifies properties as belonging to one of three categories: 1. Properties marked as REQUIRED in the response. - These properties MUST always be present for all entries in the response. +These properties MUST always be present for all entries in the response. 2. Properties marked as REQUIRED only if the query parameter :query-param:`response_fields` is not part of the request, or if they are explicitly requested in :query-param:`response_fields`. - Otherwise they MUST NOT be included. - One can think of these properties as constituting a default value for :query-param:`response_fields` when that parameter is omitted. +Otherwise they MUST NOT be included. +One can think of these properties as constituting a default value for :query-param:`response_fields` when that parameter is omitted. 3. Properties not marked as REQUIRED in any case, MUST be included only if explicitly requested in the query parameter :query-param:`response_fields`. - Otherwise they SHOULD NOT be included. +Otherwise they SHOULD NOT be included. Examples of valid entry listing endpoint URLs: @@ -1214,65 +1214,65 @@ Standard OPTIONAL URL query parameters standardized by the JSON:API specificatio - **filter**: a filter string, in the format described below in section `API Filtering Format Specification`_. - **page\_limit**: sets a numerical limit on the number of entries returned. - See `JSON:API 1.1 `__. - The API implementation MUST return no more than the number specified. - It MAY return fewer. - The database MAY have a maximum limit and not accept larger numbers (in which case the :http-error:`403 Forbidden` error code MUST be returned). - The default limit value is up to the API implementation to decide. - Example: :query-url:`http://example.com/optimade/v1/structures?page_limit=100` +See `JSON:API 1.1 `__. +The API implementation MUST return no more than the number specified. +It MAY return fewer. +The database MAY have a maximum limit and not accept larger numbers (in which case the :http-error:`403 Forbidden` error code MUST be returned). +The default limit value is up to the API implementation to decide. +Example: :query-url:`http://example.com/optimade/v1/structures?page_limit=100` - **page\_{offset, number, cursor, above, below}**: A server MUST implement pagination in the case of no user-specified :query-param:`sort` parameter (via the :field:`links` response field, see section `JSON Response Schema: Common Fields`_). - A server MAY implement pagination in concert with :query-param:`sort`. - The following parameters, all prefixed by "page\_", are RECOMMENDED for use with pagination. - If an implementation chooses +A server MAY implement pagination in concert with :query-param:`sort`. +The following parameters, all prefixed by "page\_", are RECOMMENDED for use with pagination. +If an implementation chooses - - *offset-based pagination*: using :field:`page_offset` and :field:`page_limit` is RECOMMENDED. - - *cursor-based pagination*: using :field:`page_cursor` and :field:`page_limit` is RECOMMENDED. - - *page-based pagination*: using :field:`page_number` and :field:`page_limit` is RECOMMENDED. It is RECOMMENDED that the first page has number 1, i.e., that :field:`page_number` is 1-based. - - *value-based pagination*: using :field:`page_above`/:field:`page_below` and :field:`page_limit` is RECOMMENDED. +- *offset-based pagination*: using :field:`page_offset` and :field:`page_limit` is RECOMMENDED. +- *cursor-based pagination*: using :field:`page_cursor` and :field:`page_limit` is RECOMMENDED. +- *page-based pagination*: using :field:`page_number` and :field:`page_limit` is RECOMMENDED. It is RECOMMENDED that the first page has number 1, i.e., that :field:`page_number` is 1-based. +- *value-based pagination*: using :field:`page_above`/:field:`page_below` and :field:`page_limit` is RECOMMENDED. - Examples (all OPTIONAL behavior a server MAY implement): +Examples (all OPTIONAL behavior a server MAY implement): - - skip 50 structures and fetch up to 100: :query-url:`/structures?page_offset=50&page_limit=100`. - - fetch page 2 of up to 50 structures per page: :query-url:`/structures?page_number=2&page_limit=50`. - - fetch up to 100 structures above sort-field value 4000 (in this example, server chooses to fetch results sorted by increasing :field:`id`, so :field:`page_above` value refers to an :field:`id` value): :query-url:`/structures?page_above=4000&page_limit=100`. +- skip 50 structures and fetch up to 100: :query-url:`/structures?page_offset=50&page_limit=100`. +- fetch page 2 of up to 50 structures per page: :query-url:`/structures?page_number=2&page_limit=50`. +- fetch up to 100 structures above sort-field value 4000 (in this example, server chooses to fetch results sorted by increasing :field:`id`, so :field:`page_above` value refers to an :field:`id` value): :query-url:`/structures?page_above=4000&page_limit=100`. - **sort**: If supporting sortable queries, an implementation MUST use the :query-param:`sort` query parameter with format as specified by `JSON:API 1.1 `__. - An implementation MAY support multiple sort fields for a single query. - If it does, it again MUST conform to the JSON:API 1.1 specification. +An implementation MAY support multiple sort fields for a single query. +If it does, it again MUST conform to the JSON:API 1.1 specification. - If an implementation supports sorting for an `entry listing endpoint `_, then the :endpoint:`/info/` endpoint MUST include, for each field name :field:`` in its :field:`data.properties.` response value that can be used for sorting, the key :field:`sortable` with value :field-val:`true`. - If a field name under an entry listing endpoint supporting sorting cannot be used for sorting, the server MUST either leave out the :field:`sortable` key or set it equal to :field-val:`false` for the specific field name. - The set of field names, with :field:`sortable` equal to :field-val:`true` are allowed to be used in the "sort fields" list according to its definition in the JSON:API 1.1 specification. - The field :field:`sortable` is in addition to each property description and other OPTIONAL fields. - An example is shown in section `Entry Listing Info Endpoints`_. +If an implementation supports sorting for an `entry listing endpoint `_, then the :endpoint:`/info/` endpoint MUST include, for each field name :field:`` in its :field:`data.properties.` response value that can be used for sorting, the key :field:`sortable` with value :field-val:`true`. +If a field name under an entry listing endpoint supporting sorting cannot be used for sorting, the server MUST either leave out the :field:`sortable` key or set it equal to :field-val:`false` for the specific field name. +The set of field names, with :field:`sortable` equal to :field-val:`true` are allowed to be used in the "sort fields" list according to its definition in the JSON:API 1.1 specification. +The field :field:`sortable` is in addition to each property description and other OPTIONAL fields. +An example is shown in section `Entry Listing Info Endpoints`_. - **include**: A server MAY implement the JSON:API concept of returning `compound documents `__ by utilizing the :query-param:`include` query parameter as specified by `JSON:API 1.0 `__. - All related resource objects MUST be returned as part of an array value for the top-level :field:`included` field, see section `JSON Response Schema: Common Fields`_. +All related resource objects MUST be returned as part of an array value for the top-level :field:`included` field, see section `JSON Response Schema: Common Fields`_. - The value of :query-param:`include` MUST be a comma-separated list of "relationship paths", as defined in the `JSON:API `__. - If relationship paths are not supported, or a server is unable to identify a relationship path a :http-error:`400 Bad Request` response MUST be made. +The value of :query-param:`include` MUST be a comma-separated list of "relationship paths", as defined in the `JSON:API `__. +If relationship paths are not supported, or a server is unable to identify a relationship path a :http-error:`400 Bad Request` response MUST be made. - The **default value** for :query-param:`include` is :query-val:`references`. - This means :entry:`references` entries MUST always be included under the top-level field :field:`included` as default, since a server assumes if :query-param:`include` is not specified by a client in the request, it is still specified as :query-string:`include=references`. - Note, if a client explicitly specifies :query-param:`include` and leaves out :query-val:`references`, :entry:`references` resource objects MUST NOT be included under the top-level field :field:`included`, as per the definition of :field:`included`, see section `JSON Response Schema: Common Fields`_. +The **default value** for :query-param:`include` is :query-val:`references`. +This means :entry:`references` entries MUST always be included under the top-level field :field:`included` as default, since a server assumes if :query-param:`include` is not specified by a client in the request, it is still specified as :query-string:`include=references`. +Note, if a client explicitly specifies :query-param:`include` and leaves out :query-val:`references`, :entry:`references` resource objects MUST NOT be included under the top-level field :field:`included`, as per the definition of :field:`included`, see section `JSON Response Schema: Common Fields`_. - **Note**: A query with the parameter :query-param:`include` set to the empty string means no related resource objects are to be returned under the top-level field :field:`included`. +**Note**: A query with the parameter :query-param:`include` set to the empty string means no related resource objects are to be returned under the top-level field :field:`included`. Standard OPTIONAL URL query parameters not in the JSON:API specification: - **response\_format**: the output format requested (see section `Response Format`_). - Defaults to the format string 'json', which specifies the standard output format described in this specification. - Example: :query-url:`http://example.com/optimade/v1/structures?response_format=xml` +Defaults to the format string 'json', which specifies the standard output format described in this specification. +Example: :query-url:`http://example.com/optimade/v1/structures?response_format=xml` - **email\_address**: an email address of the user making the request. - The email SHOULD be that of a person and not an automatic system. - Example: :query-url:`http://example.com/optimade/v1/structures?email_address=user@example.com` +The email SHOULD be that of a person and not an automatic system. +Example: :query-url:`http://example.com/optimade/v1/structures?email_address=user@example.com` - **response\_fields**: a comma-delimited set of fields to be provided in the output. - If provided, these fields MUST be returned along with the REQUIRED fields. - Other OPTIONAL fields MUST NOT be returned when this parameter is present. - Example: :query-url:`http://example.com/optimade/v1/structures?response_fields=last_modified,nsites` +If provided, these fields MUST be returned along with the REQUIRED fields. +Other OPTIONAL fields MUST NOT be returned when this parameter is present. +Example: :query-url:`http://example.com/optimade/v1/structures?response_fields=last_modified,nsites` Additional OPTIONAL URL query parameters not described above are not considered to be part of this standard, and are instead considered to be "custom URL query parameters". These custom URL query parameters MUST be of the format "". @@ -1287,8 +1287,8 @@ Examples: - :query-url:`http://example.com/optimade/v1/structures?_exmpl_warning_verbosity=10` - :query-url:`http://example.com/optimade/v1/structures?_exmpl_filter="elements all in [Al, Si, Ga]"` - **Note**: the specification presently makes no attempt to standardize access control mechanisms. - There are security concerns with access control based on URL tokens, and the above example is not to be taken as a recommendation for such a mechanism. +**Note**: the specification presently makes no attempt to standardize access control mechanisms. +There are security concerns with access control based on URL tokens, and the above example is not to be taken as a recommendation for such a mechanism. Entry Listing JSON Response Schema ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -1301,51 +1301,51 @@ In the default JSON response format every dictionary (`resource object `__ can OPTIONALLY contain the field - - **self**: the entry's URL +- **self**: the entry's URL - **meta**: a `JSON API meta object `__ that is used to communicate metadata. - See `JSON Response Schema: Common Fields`_ for more information about this field. +See `JSON Response Schema: Common Fields`_ for more information about this field. - **relationships**: a dictionary containing references to other entries according to the description in section `Relationships`_ encoded as `JSON:API Relationships `__. - The OPTIONAL human-readable description of the relationship MAY be provided in the :field:`description` field inside the :field:`meta` dictionary of the JSON:API resource identifier object. - All relationships to entries of the same entry type MUST be grouped into the same JSON:API relationship object and placed in the relationships dictionary with the entry type name as key (e.g., :entry:`structures`). +The OPTIONAL human-readable description of the relationship MAY be provided in the :field:`description` field inside the :field:`meta` dictionary of the JSON:API resource identifier object. +All relationships to entries of the same entry type MUST be grouped into the same JSON:API relationship object and placed in the relationships dictionary with the entry type name as key (e.g., :entry:`structures`). Example: .. code:: jsonc - { - "data": [ - { - "type": "structures", - "id": "example.db:structs:0001", - "attributes": { - "chemical_formula_descriptive": "Es2 O3", - "url": "http://example.db/structs/0001", - "immutable_id": "http://example.db/structs/0001@123", - "last_modified": "2007-04-05T14:30:20Z" - } - }, - { - "type": "structures", - "id": "example.db:structs:1234", - "attributes": { - "chemical_formula_descriptive": "Es2", - "url": "http://example.db/structs/1234", - "immutable_id": "http://example.db/structs/1234@123", - "last_modified": "2007-04-07T12:02:20Z" - } - } - // ... - ] - // ... - } +{ +"data": [ +{ +"type": "structures", +"id": "example.db:structs:0001", +"attributes": { +"chemical_formula_descriptive": "Es2 O3", +"url": "http://example.db/structs/0001", +"immutable_id": "http://example.db/structs/0001@123", +"last_modified": "2007-04-05T14:30:20Z" +} +}, +{ +"type": "structures", +"id": "example.db:structs:1234", +"attributes": { +"chemical_formula_descriptive": "Es2", +"url": "http://example.db/structs/1234", +"immutable_id": "http://example.db/structs/1234@123", +"last_modified": "2007-04-07T12:02:20Z" +} +} +// ... +] +// ... +} Single Entry Endpoints ---------------------- @@ -1373,25 +1373,26 @@ The meaning of these URL query parameters are as defined above in section `Entry One additional query parameter :query-param:`property_slices` MUST be handled by the API implementation either as defined below or by returning the error :http-error:`501 Not Implemented`: -- **property\_slices**: A number of slice specifications to specify parts of array properties for the functionality described in `Slices of array properties`_. +- **property\_slices**: A number of slice specifications to request only parts of array properties for the functionality described in `Slices of array properties`_. + + The query parameter contains a comma-separated (",", ASCII 44(0x2C)) list of slice specifications. + Each slice specification consists of an ordered sequence of four elements separated by the colon character (":", ASCII 58(0x3A)). + The elements in the sequence are the dimension name and the three components of the slice, i.e., the start, stop, and step values defined in the same way as for a `slice object`_. + Omitting the value for any of the components of the slice specifies a default value (however, all the colon separators MUST be included). + The start value specifies the first index in that dimension for which values should be returned (which is 0-based and inclusive). + The default is :val:`0`. + The stop value specifies the last index for which values should be returned (inclusive). + The default is the last index of the array along the specified dimension. + The step value specifies the step size in that dimension. + The default is :val:`1`. - The query parameter contains a comma-separated (",", ASCII 44(0x2C)) list of slice specification. - Each slice specification consists of a dimension name and the start, stop and step values of the slice separated by the colon character (":", ASCII 58(0x3A)) - Omitting the value between two colons specifies a default value. - The components of the slice are defined in the same way as for a `slice object`_. - The first integer specifies the start of the slice, i.e. the first index in that dimension for which values should be returned. - The default is the index of the first value. - The second integer specifies the end/stop of the slice, i.e. the last index for which values should be returned. - The default is the index of the last value of the property. - The third integer specifies the step size in that dimension. - The default is :val:`1` - The slices are 0-based, i.e. the first value has index 0, and inclusive i.e. for the dimension :val:`dim_frames` the range :val:`dim_frames:10:20:1` the last value returned belongs to index 20. + An empty value of the :query-param:`property_slices` query parameter MUST be interpreted as equivalent to the query parameter not being included in the request. Requirements and conventions for the response when this query parameter is used are described in `Slices of array properties`_. Example: - - :query-url:`http://optimade.example.com/v1/trajectories/id_12345?response_fields=cartesian_site_positions&property_ranges=dim_frames::999:10,dim_sites:30:70::` + - :query-url:`http://optimade.example.com/v1/trajectories/id_12345?response_fields=cartesian_site_positions&property_ranges=dim_frames::999:10,dim_sites:30:70:` This query URL requests items from the array :field:`cartesian_site_positions` is requested only for the 31st to 71st sites (i.e., with indexes 30 through 70 inclusive) for 1 out of every 10 frames of the first 1000 frames (i.e., taking steps of 10 over indexes 0 through 999 inclusive, which requests the frames with index 0, 10, 20, 30, ..., 990) of a trajectory with ID :val:`id_12345`. From e562f4922ffbfb5652815c5fa356359fb182d171 Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Fri, 14 Jun 2024 10:20:32 +0200 Subject: [PATCH 22/70] Revert "Improve handing of Python dependencies" This reverts commit 266545b3c0279c20eae243c0990120c667f8f8e9. --- optimade.rst | 801 +++++++++++++++++++++++++-------------------------- 1 file changed, 400 insertions(+), 401 deletions(-) diff --git a/optimade.rst b/optimade.rst index 4e688f709..35c37ea7d 100644 --- a/optimade.rst +++ b/optimade.rst @@ -660,106 +660,106 @@ The field :field:`array_axes` is defined as follows: - ``{"start": 0}`` (despite that :val:`0` is the default value of the :field:`start` field.) But it is instead a valid representation if the query parameter was :query-string:`property_slices=dim_frames:0::`. -- :query-string:`property_slices=dim_frames:3:5:2` the representation MUST be: + - :query-string:`property_slices=dim_frames:3:5:2` the representation MUST be: -- ``{"start": 3, "stop": 5, "step": 2}`` + - ``{"start": 3, "stop": 5, "step": 2}`` -The dictionary MAY contain the following fields: + The dictionary MAY contain the following fields: -- :field:`length`: Integer. -The length of this array axis which MUST be the same as the length of the declared dimension for this axis in the corresponding property definition. -Note that the length of a dimension can be different for different entries if the length is not explicitly declared by the property definition. -For example, the number of frames, i.e., the length of the dimension named ``dim_frames``, is generally different for different trajectories. + - :field:`length`: Integer. + The length of this array axis which MUST be the same as the length of the declared dimension for this axis in the corresponding property definition. + Note that the length of a dimension can be different for different entries if the length is not explicitly declared by the property definition. + For example, the number of frames, i.e., the length of the dimension named ``dim_frames``, is generally different for different trajectories. -- :field:`sliceable`: Boolean. -If :val:`true`, the server MUST handle slices for that dimension. -If :val:`false`, the server MAY handle slices for that dimension, or MAY return the error :http-error:`501 Not Implemented` when a client requests a slice involving this dimension. -If the field is omitted or :val:`null`, it means the same thing as :val:`false`. + - :field:`sliceable`: Boolean. + If :val:`true`, the server MUST handle slices for that dimension. + If :val:`false`, the server MAY handle slices for that dimension, or MAY return the error :http-error:`501 Not Implemented` when a client requests a slice involving this dimension. + If the field is omitted or :val:`null`, it means the same thing as :val:`false`. -- :field:`available_slice`: Dictionary or :val:`null`. -This field describes a `slice object`_ where there MAY be non-null values in the data. -By including this field, the server certifies that there are only :val:`null` values outside this slice. -If not provided, or equal to :val:`null` or an empty dictionary, the client cannot make any assumptions about what part of the array contains :val:`null` values. + - :field:`available_slice`: Dictionary or :val:`null`. + This field describes a `slice object`_ where there MAY be non-null values in the data. + By including this field, the server certifies that there are only :val:`null` values outside this slice. + If not provided, or equal to :val:`null` or an empty dictionary, the client cannot make any assumptions about what part of the array contains :val:`null` values. -Examples: + Examples: -- ``{"start": 3, "stop": 7, "step": 2}`` means the server certifies that values at indexes 0,1,2,4,6 and any index from 8 to the end of the array are :val:`null`. + - ``{"start": 3, "stop": 7, "step": 2}`` means the server certifies that values at indexes 0,1,2,4,6 and any index from 8 to the end of the array are :val:`null`. Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format for a request to the trajectory endpoint with the query parameter :query-param:`property_slices=dim_frames:3:37:5` and :query-param:`response_fields=cartesian_site_positions,_exmpl_temperature` where the trajectory consists of 432934 frames (with indexes 0 to 432933) and where the :field:`cartisian_site_positions` contains 7 sites. Furthermore, the :field:`_exmpl_temperature` contains only :val:`null` values except for items with indexes 1000, 1030, 1060, ..., 4000 (where the values can be either numeric or :val:`null`). .. code:: jsonc -{ -// ... -"data": { -"type": "trajectories", -"id": "2345678", -"attributes": { -"cartesian_site_positions": null, -"_exmpl_temperature": null -}, -"meta": { -"property_metadata": { -"cartesian_site_positions": { -"array_axes": [ - { - "dimension_name": "dim_frames", - "requested_slice": { - "start": 3, - "stop": 37, - "step": 5 - }, - "available_slice": { - "start": 0, - "stop": 432933, - "step": 1 - }, - "sliceable": true, - "length": 432934, - }, - { - "dimension_name": "dim_sites", - "available_slice": { - "start": 0, - "stop": 6, - "step": 1 - }, - "sliceable": false, - "length": 7, - }, - { - "dimension_name": "dim_spatial", - "length": 3, - } -], -}, -"_exmpl_temperature": { -"array_axes": [ - { - "dimension_name": "dim_frames", - "requested_slice": { - "start": 3, - "stop": 37, - "step": 5 - }, - "available_slice": { - "start": 1000, - "stop": 4000, - "step": 30 - }, - "sliceable": true, - "length": 432934, - } -] -} -}, -"partial_data_links": { -//... -} -} -} -// ... -} + { + // ... + "data": { + "type": "trajectories", + "id": "2345678", + "attributes": { + "cartesian_site_positions": null, + "_exmpl_temperature": null + }, + "meta": { + "property_metadata": { + "cartesian_site_positions": { + "array_axes": [ + { + "dimension_name": "dim_frames", + "requested_slice": { + "start": 3, + "stop": 37, + "step": 5 + }, + "available_slice": { + "start": 0, + "stop": 432933, + "step": 1 + }, + "sliceable": true, + "length": 432934, + }, + { + "dimension_name": "dim_sites", + "available_slice": { + "start": 0, + "stop": 6, + "step": 1 + }, + "sliceable": false, + "length": 7, + }, + { + "dimension_name": "dim_spatial", + "length": 3, + } + ], + }, + "_exmpl_temperature": { + "array_axes": [ + { + "dimension_name": "dim_frames", + "requested_slice": { + "start": 3, + "stop": 37, + "step": 5 + }, + "available_slice": { + "start": 1000, + "stop": 4000, + "step": 30 + }, + "sliceable": true, + "length": 432934, + } + ] + } + }, + "partial_data_links": { + //... + } + } + } + // ... + } Responses ========= @@ -790,305 +790,305 @@ In the JSON response format, property types translate as follows: Every response SHOULD contain the following fields, and MUST contain at least :field:`meta`: - **meta**: a `JSON:API meta member `__ that contains JSON:API meta objects of non-standard meta-information. -It MUST be a dictionary with these fields: + It MUST be a dictionary with these fields: -- **api\_version**: a string containing the full version of the API implementation. -The version number string MUST NOT be prefixed by, e.g., "v". -Examples: :field-val:`1.0.0`, :field-val:`1.0.0-rc.2`. + - **api\_version**: a string containing the full version of the API implementation. + The version number string MUST NOT be prefixed by, e.g., "v". + Examples: :field-val:`1.0.0`, :field-val:`1.0.0-rc.2`. -- **query**: information on the query that was requested. -It MUST be a dictionary with this field: + - **query**: information on the query that was requested. + It MUST be a dictionary with this field: -- **representation**: a string with the part of the URL following the versioned or unversioned base URL that serves the API. -Query parameters that have not been used in processing the request MAY be omitted. -In particular, if no query parameters have been involved in processing the request, the query part of the URL MAY be excluded. -Example: :field-val:`/structures?filter=nelements=2`. + - **representation**: a string with the part of the URL following the versioned or unversioned base URL that serves the API. + Query parameters that have not been used in processing the request MAY be omitted. + In particular, if no query parameters have been involved in processing the request, the query part of the URL MAY be excluded. + Example: :field-val:`/structures?filter=nelements=2`. -- **more\_data\_available**: :field-val:`false` if the response contains all data for the request (e.g., a request issued to a single entry endpoint, or a :query-param:`filter` query at the last page of a paginated response) and :field-val:`true` if the response is incomplete in the sense that multiple objects match the request, and not all of them have been included in the response (e.g., a query with multiple pages that is not at the last page). + - **more\_data\_available**: :field-val:`false` if the response contains all data for the request (e.g., a request issued to a single entry endpoint, or a :query-param:`filter` query at the last page of a paginated response) and :field-val:`true` if the response is incomplete in the sense that multiple objects match the request, and not all of them have been included in the response (e.g., a query with multiple pages that is not at the last page). -:field:`meta` SHOULD also include these fields: + :field:`meta` SHOULD also include these fields: -- **time\_stamp**: a timestamp containing the date and time at which the query was executed. -- **data\_returned**: an integer containing the total number of data resource objects returned for the current :query-param:`filter` query, independent of pagination. -- **provider**: information on the database provider of the implementation. -It MUST be a dictionary with these fields: + - **time\_stamp**: a timestamp containing the date and time at which the query was executed. + - **data\_returned**: an integer containing the total number of data resource objects returned for the current :query-param:`filter` query, independent of pagination. + - **provider**: information on the database provider of the implementation. + It MUST be a dictionary with these fields: -- **name**: a short name for the database provider. -- **description**: a longer description of the database provider. -- **prefix**: database-provider-specific prefix (see section `Database-Provider-Specific Namespace Prefixes`_). + - **name**: a short name for the database provider. + - **description**: a longer description of the database provider. + - **prefix**: database-provider-specific prefix (see section `Database-Provider-Specific Namespace Prefixes`_). -:field:`provider` MAY include these fields: + :field:`provider` MAY include these fields: -- **homepage**: a `JSON API link `__, pointing to the homepage of the database provider, either directly as a string, or as an object which can contain the following fields: + - **homepage**: a `JSON API link `__, pointing to the homepage of the database provider, either directly as a string, or as an object which can contain the following fields: -- **href**: a string containing the homepage URL. -- **meta**: a meta object containing non-standard meta-information about the database provider's homepage. + - **href**: a string containing the homepage URL. + - **meta**: a meta object containing non-standard meta-information about the database provider's homepage. -:field:`meta` MAY also include these fields: + :field:`meta` MAY also include these fields: -- **data\_available**: an integer containing the total number of data resource objects available in the database for the endpoint. -- **last\_id**: a string containing the last ID returned. -- **response\_message**: response string from the server. -- **request\_delay**: a non-negative float giving time in seconds that the client is suggested to wait before issuing a subsequent request. + - **data\_available**: an integer containing the total number of data resource objects available in the database for the endpoint. + - **last\_id**: a string containing the last ID returned. + - **response\_message**: response string from the server. + - **request\_delay**: a non-negative float giving time in seconds that the client is suggested to wait before issuing a subsequent request. -Implementation note: the functionality of this field overlaps to some degree with features provided by the HTTP error :http-error:`429 Too Many Requests` and the `Retry-After HTTP header `__. -Implementations are suggested to provide consistent handling of request overload through both mechanisms. + Implementation note: the functionality of this field overlaps to some degree with features provided by the HTTP error :http-error:`429 Too Many Requests` and the `Retry-After HTTP header `__. + Implementations are suggested to provide consistent handling of request overload through both mechanisms. -- **database**: a dictionary describing the specific database accessible at this OPTIMADE API. -If provided, the dictionary fields SHOULD match those provided in the corresponding links entry for the database in the provider's index meta-database, outlined in `Links Endpoint JSON Response Schema`_. -The dictionary can contain the following OPTIONAL fields: + - **database**: a dictionary describing the specific database accessible at this OPTIMADE API. + If provided, the dictionary fields SHOULD match those provided in the corresponding links entry for the database in the provider's index meta-database, outlined in `Links Endpoint JSON Response Schema`_. + The dictionary can contain the following OPTIONAL fields: -- **id**: the identifier of this database within those served by this provider, i.e., the ID under which this database is served in this provider's index meta-database. -- **name**: a human-readable name for the database, e.g., for use in clients. -- **version**: a string describing the version of the database. -- **description**: a human-readable description of the database, e.g., for use in clients. -- **homepage**: a `JSON API link `__, pointing to a homepage for the particular database. -- **maintainer**: a dictionary providing details about the maintainer of the database, which MUST contain the single field: + - **id**: the identifier of this database within those served by this provider, i.e., the ID under which this database is served in this provider's index meta-database. + - **name**: a human-readable name for the database, e.g., for use in clients. + - **version**: a string describing the version of the database. + - **description**: a human-readable description of the database, e.g., for use in clients. + - **homepage**: a `JSON API link `__, pointing to a homepage for the particular database. + - **maintainer**: a dictionary providing details about the maintainer of the database, which MUST contain the single field: -- **email** with the maintainer's email address. + - **email** with the maintainer's email address. -- **implementation**: a dictionary describing the server implementation, containing the OPTIONAL fields: + - **implementation**: a dictionary describing the server implementation, containing the OPTIONAL fields: -- **name**: name of the implementation. -- **version**: version string of the current implementation. -- **homepage**: a `JSON API link `__, pointing to the homepage of the implementation. -- **source\_url**: a `JSON API link `__ pointing to the implementation source, either downloadable archive or version control system. + - **name**: name of the implementation. + - **version**: version string of the current implementation. + - **homepage**: a `JSON API link `__, pointing to the homepage of the implementation. + - **source\_url**: a `JSON API link `__ pointing to the implementation source, either downloadable archive or version control system. -- **maintainer**: a dictionary providing details about the maintainer of the implementation, MUST contain the single field: + - **maintainer**: a dictionary providing details about the maintainer of the implementation, MUST contain the single field: -- **email** with the maintainer's email address. + - **email** with the maintainer's email address. -- **issue\_tracker**: a `JSON API link `__ pointing to the implementation's issue tracker. + - **issue\_tracker**: a `JSON API link `__ pointing to the implementation's issue tracker. -- **warnings**: a list of warning resource objects representing non-critical errors or warnings. -A warning resource object is defined similarly to a `JSON:API error object `__, but MUST also include the field :field:`type`, which MUST have the value :field-val:`"warning"`. -The field :field:`detail` MUST be present and SHOULD contain a non-critical message, e.g., reporting unrecognized search attributes or deprecated features. -The field :field:`status`, representing an HTTP response status code, MUST NOT be present for a warning resource object. -This is an exclusive field for error resource objects. + - **warnings**: a list of warning resource objects representing non-critical errors or warnings. + A warning resource object is defined similarly to a `JSON:API error object `__, but MUST also include the field :field:`type`, which MUST have the value :field-val:`"warning"`. + The field :field:`detail` MUST be present and SHOULD contain a non-critical message, e.g., reporting unrecognized search attributes or deprecated features. + The field :field:`status`, representing an HTTP response status code, MUST NOT be present for a warning resource object. + This is an exclusive field for error resource objects. -Example for a deprecation warning: + Example for a deprecation warning: -.. code:: jsonc + .. code:: jsonc -{ -"id": "dep_chemical_formula_01", -"type": "warning", -"code": "_exmpl_dep_chemical_formula", -"title": "Deprecation Warning", -"detail": "chemical_formula is deprecated, use instead chemical_formula_hill" -} + { + "id": "dep_chemical_formula_01", + "type": "warning", + "code": "_exmpl_dep_chemical_formula", + "title": "Deprecation Warning", + "detail": "chemical_formula is deprecated, use instead chemical_formula_hill" + } -**Note**: warning :field:`id`\ s MUST NOT be trusted to identify the exceptional situations (i.e., they are not error codes), use instead the field :field:`code` for this. -Warning :field:`id`\ s can *only* be trusted to be unique in the list of warning resource objects, i.e., together with the :field:`type`. + **Note**: warning :field:`id`\ s MUST NOT be trusted to identify the exceptional situations (i.e., they are not error codes), use instead the field :field:`code` for this. + Warning :field:`id`\ s can *only* be trusted to be unique in the list of warning resource objects, i.e., together with the :field:`type`. -General OPTIMADE warning codes are specified in section `Warnings`_. + General OPTIMADE warning codes are specified in section `Warnings`_. -- Other OPTIONAL additional information *global to the query* that is not specified in this document, MUST start with a database-provider-specific prefix (see section `Database-Provider-Specific Namespace Prefixes`_). + - Other OPTIONAL additional information *global to the query* that is not specified in this document, MUST start with a database-provider-specific prefix (see section `Database-Provider-Specific Namespace Prefixes`_). -- Example for a request made to :query-url:`http://example.com/optimade/v1/structures/?filter=a=1 AND b=2`: + - Example for a request made to :query-url:`http://example.com/optimade/v1/structures/?filter=a=1 AND b=2`: -.. code:: jsonc + .. code:: jsonc + + { + "meta": { + "query": { + "representation": "/structures/?filter=a=1 AND b=2" + }, + "api_version": "1.0.0", + "schema": "http://schemas.optimade.org/openapi/v1/optimade.json", + "time_stamp": "2007-04-05T14:30:20Z", + "data_returned": 10, + "data_available": 10, + "more_data_available": false, + "provider": { + "name": "Example provider", + "description": "Provider used for examples, not to be assigned to a real database", + "prefix": "exmpl", + "homepage": "http://example.com" + }, + "implementation": { + "name": "exmpl-optimade", + "version": "0.1.0", + "source_url": "http://git.example.com/exmpl-optimade", + "maintainer": { + "email": "admin@example.com" + }, + "issue_tracker": "http://tracker.example.com/exmpl-optimade" + }, + "database": { + "id": "example_db", + "name": "Example database 1 (of many)", + "description": "The first example database in a series hosted by the Example Provider.", + "homepage": "http://database_one.example.com", + "maintainer": { + "email": "science_lead@example.com" + } + } + } + // ... + } + + - **schema**: a `JSON:API links object `__ that points to a schema for the response. + If it is a string, or a dictionary containing no :field:`meta` field, the provided URL MUST point at an `OpenAPI `__ schema. + It is possible that future versions of this specification allow for alternative schema types. + Hence, if the :field:`meta` field of the JSON:API links object is provided and contains a field :field:`schema_type` that is not equal to the string :field-val:`OpenAPI` the client MUST NOT handle failures to parse the schema or to validate the response against the schema as errors. -{ -"meta": { -"query": { -"representation": "/structures/?filter=a=1 AND b=2" -}, -"api_version": "1.0.0", -"schema": "http://schemas.optimade.org/openapi/v1/optimade.json", -"time_stamp": "2007-04-05T14:30:20Z", -"data_returned": 10, -"data_available": 10, -"more_data_available": false, -"provider": { -"name": "Example provider", -"description": "Provider used for examples, not to be assigned to a real database", -"prefix": "exmpl", -"homepage": "http://example.com" -}, -"implementation": { -"name": "exmpl-optimade", -"version": "0.1.0", -"source_url": "http://git.example.com/exmpl-optimade", -"maintainer": { -"email": "admin@example.com" -}, -"issue_tracker": "http://tracker.example.com/exmpl-optimade" -}, -"database": { -"id": "example_db", -"name": "Example database 1 (of many)", -"description": "The first example database in a series hosted by the Example Provider.", -"homepage": "http://database_one.example.com", -"maintainer": { -"email": "science_lead@example.com" -} -} -} -// ... -} - -- **schema**: a `JSON:API links object `__ that points to a schema for the response. -If it is a string, or a dictionary containing no :field:`meta` field, the provided URL MUST point at an `OpenAPI `__ schema. -It is possible that future versions of this specification allow for alternative schema types. -Hence, if the :field:`meta` field of the JSON:API links object is provided and contains a field :field:`schema_type` that is not equal to the string :field-val:`OpenAPI` the client MUST NOT handle failures to parse the schema or to validate the response against the schema as errors. - -**Note**: The :field:`schema` field was previously RECOMMENDED in all responses, but is now demoted to being OPTIONAL since there now is a standard way of specifying a response schema in JSON:API through the :field:`describedby` subfield of the top-level :field:`links` field. + **Note**: The :field:`schema` field was previously RECOMMENDED in all responses, but is now demoted to being OPTIONAL since there now is a standard way of specifying a response schema in JSON:API through the :field:`describedby` subfield of the top-level :field:`links` field. - **data**: The schema of this value varies by endpoint, it can be either a *single* `JSON:API resource object `__ or a *list* of JSON:API resource objects. -Every resource object needs the :field:`type` and :field:`id` fields, and its attributes (described in section `API Endpoints`_) need to be in a dictionary corresponding to the :field:`attributes` field. + Every resource object needs the :field:`type` and :field:`id` fields, and its attributes (described in section `API Endpoints`_) need to be in a dictionary corresponding to the :field:`attributes` field. -Every resource object MAY also contain a :field:`meta` field which MAY contain the following keys: + Every resource object MAY also contain a :field:`meta` field which MAY contain the following keys: -- **property_metadata**: an object containing per-entry and per-property metadata. -The keys are the names of the fields in :field:`attributes` for which metadata is available. -The values belonging to these keys are dictionaries containing the relevant metadata fields. -See also `Metadata properties`_ + - **property_metadata**: an object containing per-entry and per-property metadata. + The keys are the names of the fields in :field:`attributes` for which metadata is available. + The values belonging to these keys are dictionaries containing the relevant metadata fields. + See also `Metadata properties`_ -- **partial_data_links**: an object used to list links which can be used to fetch data that has been omitted from the :field:`data` part of the response. -The keys are the names of the fields in :field:`attributes` for which partial data links are available. -Each value is a list of objects that MUST have the following keys: + - **partial_data_links**: an object used to list links which can be used to fetch data that has been omitted from the :field:`data` part of the response. + The keys are the names of the fields in :field:`attributes` for which partial data links are available. + Each value is a list of objects that MUST have the following keys: -- **format**: String. -The name of the format provided via this link. -For one of the objects this :field:`format` field SHOULD have the value "jsonlines", which refers to the format in `OPTIMADE JSON lines partial data format`_. + - **format**: String. + The name of the format provided via this link. + For one of the objects this :field:`format` field SHOULD have the value "jsonlines", which refers to the format in `OPTIMADE JSON lines partial data format`_. -- **link**: String. -A `JSON API link `__ that points to a location from which the omitted data can be fetched. -There is no requirement on the syntax or format for the link URL. + - **link**: String. + A `JSON API link `__ that points to a location from which the omitted data can be fetched. + There is no requirement on the syntax or format for the link URL. -For more information about the mechanism to transmit large property values, including an example of the format of :field:`partial_data_links`, see `Transmission of large property values`_. + For more information about the mechanism to transmit large property values, including an example of the format of :field:`partial_data_links`, see `Transmission of large property values`_. The response MAY also return resources related to the primary data in the field: - **links**: a `JSON API links object `__ is REQUIRED for implementing pagination. -(see section `Entry Listing URL Query Parameters`_.) -Each field of a links object, i.e., a "link", MUST be one of: + (see section `Entry Listing URL Query Parameters`_.) + Each field of a links object, i.e., a "link", MUST be one of: -- :field-val:`null` -- a string representing a URI, or -- a dictionary ("link object") with fields + - :field-val:`null` + - a string representing a URI, or + - a dictionary ("link object") with fields -- **href**: a string representing a URI -- **meta**: (OPTIONAL) a meta object containing non-standard meta-information about the link + - **href**: a string representing a URI + - **meta**: (OPTIONAL) a meta object containing non-standard meta-information about the link -Example links objects: + Example links objects: -- **base\_url**: a links object representing the base URL of the implementation. Example: + - **base\_url**: a links object representing the base URL of the implementation. Example: -.. code:: jsonc + .. code:: jsonc -{ -"links": { -"base_url": { -"href": "http://example.com/optimade", -"meta": { -"_exmpl_db_version": "3.2.1" -} -} -// ... -} -// ... -} + { + "links": { + "base_url": { + "href": "http://example.com/optimade", + "meta": { + "_exmpl_db_version": "3.2.1" + } + } + // ... + } + // ... + } -The :field:`links` field SHOULD include the following links objects: + The :field:`links` field SHOULD include the following links objects: -- **describedby**: a links object giving the URL for a schema that describes the response. -The URL SHOULD resolve into a JSON formatted response returning a JSON object with top level :field:`$schema` and/or :field:`$id` fields that can be used by the client to identify the schema format. + - **describedby**: a links object giving the URL for a schema that describes the response. + The URL SHOULD resolve into a JSON formatted response returning a JSON object with top level :field:`$schema` and/or :field:`$id` fields that can be used by the client to identify the schema format. -**Note**: This field is the standard facility in JSON:API to communicate a response schema. -It overlaps in function with the field :field:`schema` in the top level :field:`meta` field. + **Note**: This field is the standard facility in JSON:API to communicate a response schema. + It overlaps in function with the field :field:`schema` in the top level :field:`meta` field. -The following fields are REQUIRED for implementing pagination: + The following fields are REQUIRED for implementing pagination: -- **next**: represents a link to fetch the next set of results. -When the current response is the last page of data, this field MUST be either omitted or :field-val:`null`\ -valued. + - **next**: represents a link to fetch the next set of results. + When the current response is the last page of data, this field MUST be either omitted or :field-val:`null`\ -valued. -An implementation MAY also use the following reserved fields for pagination. -They represent links in a similar way as for :field:`next`. + An implementation MAY also use the following reserved fields for pagination. + They represent links in a similar way as for :field:`next`. -- **prev**: the previous page of data. :field-val:`null` or omitted when the current response is the first page of data. -- **last**: the last page of data. -- **first**: the first page of data. + - **prev**: the previous page of data. :field-val:`null` or omitted when the current response is the first page of data. + - **last**: the last page of data. + - **first**: the first page of data. -Finally, the :field:`links` field MAY also include the following links object: + Finally, the :field:`links` field MAY also include the following links object: -- **self**: a links object giving the URL from which the response was obtained. + - **self**: a links object giving the URL from which the response was obtained. - **included**: a list of `JSON:API resource objects `__ related to the primary data contained in :field:`data`. -Responses that contain related resources under :field:`included` are known as `compound documents `__ in the JSON:API. + Responses that contain related resources under :field:`included` are known as `compound documents `__ in the JSON:API. -The definition of this field is found in the `JSON:API specification `__. -Specifically, if the query parameter :query-param:`include` is included in the request, :field:`included` MUST NOT include unrequested resource objects. -For further information on the parameter :query-param:`include`, see section `Entry Listing URL Query Parameters`_. + The definition of this field is found in the `JSON:API specification `__. + Specifically, if the query parameter :query-param:`include` is included in the request, :field:`included` MUST NOT include unrequested resource objects. + For further information on the parameter :query-param:`include`, see section `Entry Listing URL Query Parameters`_. -This value MUST be either an empty array or an array of related resource objects. + This value MUST be either an empty array or an array of related resource objects. If there were errors in producing the response all other fields MAY be present, but the top-level :field:`data` field MUST be skipped, and the following field MUST be present: - **errors**: a list of `JSON:API error objects `__, where the field :field:`detail` MUST be present. -All other fields are OPTIONAL. + All other fields are OPTIONAL. An example of a full response: .. code:: jsonc -{ -"links": { -"next": null, -"base_url": { -"href": "http://example.com/optimade", -"meta": { -"_exmpl_db_version": "3.2.1" -} -} -}, -"meta": { -"query": { -"representation": "/structures?filter=a=1 AND b=2" -}, -"api_version": "1.0.0", -"time_stamp": "2007-04-05T14:30:20Z", -"data_returned": 10, -"data_available": 10, -"last_id": "xy10", -"more_data_available": false, -"provider": { -"name": "Example provider", -"description": "Provider used for examples, not to be assigned to a real database", -"prefix": "exmpl", -"homepage": { -"href": "http://example.com", -"meta": { -"_exmpl_title": "This is an example site" -} -} -}, -// -}, -"data": [ -// ... -], -"included": [ -// ... -] -} + { + "links": { + "next": null, + "base_url": { + "href": "http://example.com/optimade", + "meta": { + "_exmpl_db_version": "3.2.1" + } + } + }, + "meta": { + "query": { + "representation": "/structures?filter=a=1 AND b=2" + }, + "api_version": "1.0.0", + "time_stamp": "2007-04-05T14:30:20Z", + "data_returned": 10, + "data_available": 10, + "last_id": "xy10", + "more_data_available": false, + "provider": { + "name": "Example provider", + "description": "Provider used for examples, not to be assigned to a real database", + "prefix": "exmpl", + "homepage": { + "href": "http://example.com", + "meta": { + "_exmpl_title": "This is an example site" + } + } + }, + // + }, + "data": [ + // ... + ], + "included": [ + // ... + ] + } - **@context**: A JSON-LD context that enables interpretation of data in the response as linked data. -If provided, it SHOULD be one of the following: + If provided, it SHOULD be one of the following: -- An object conforming to a JSON-LD standard, which includes a :field:`@version` field specifying the version of the standard. -- A string containing a URL that resolves to such an object. + - An object conforming to a JSON-LD standard, which includes a :field:`@version` field specifying the version of the standard. + - A string containing a URL that resolves to such an object. - **jsonapi**: A `JSON:API object `__. -The :field:`version` subfield SHOULD be :field-val:`"1.1"`. -The :field:`meta` subfield SHOULD be included and contain the following subfields: + The :field:`version` subfield SHOULD be :field-val:`"1.1"`. + The :field:`meta` subfield SHOULD be included and contain the following subfields: -- **api**: A string with the value "OPTIMADE". -- **api-version**: A string with the full version of the OPTIMADE standard that the processing and response adheres to. -This MAY be the version indicated at the top of this document, but MAY also be another version if the client, e.g., has used the query parameter :query-param:`api_hint` to request processing according to another version. + - **api**: A string with the value "OPTIMADE". + - **api-version**: A string with the full version of the OPTIMADE standard that the processing and response adheres to. + This MAY be the version indicated at the top of this document, but MAY also be another version if the client, e.g., has used the query parameter :query-param:`api_hint` to request processing according to another version. -If the server is able to handle serialization in such a way that it can dictate the order of the top level object members in the response, it is RECOMMENDED to put the :field:`jsonapi` as the first top level member to simplify identification of the response. + If the server is able to handle serialization in such a way that it can dictate the order of the top level object members in the response, it is RECOMMENDED to put the :field:`jsonapi` as the first top level member to simplify identification of the response. HTTP Response Status Codes -------------------------- @@ -1160,9 +1160,9 @@ Example response: .. code:: CSV -version -1 -0 + version + 1 + 0 The above response means that the API versions 1 and 0 are served under the versioned base URLs :query-url:`/v1` and :query-url:`/v0`, respectively. The order of the versions indicates that the API implementation regards version 1 as preferred over version 0. @@ -1178,14 +1178,14 @@ Each entry in the list includes a set of properties and their corresponding valu The section `Entry list`_ specifies properties as belonging to one of three categories: 1. Properties marked as REQUIRED in the response. -These properties MUST always be present for all entries in the response. + These properties MUST always be present for all entries in the response. 2. Properties marked as REQUIRED only if the query parameter :query-param:`response_fields` is not part of the request, or if they are explicitly requested in :query-param:`response_fields`. -Otherwise they MUST NOT be included. -One can think of these properties as constituting a default value for :query-param:`response_fields` when that parameter is omitted. + Otherwise they MUST NOT be included. + One can think of these properties as constituting a default value for :query-param:`response_fields` when that parameter is omitted. 3. Properties not marked as REQUIRED in any case, MUST be included only if explicitly requested in the query parameter :query-param:`response_fields`. -Otherwise they SHOULD NOT be included. + Otherwise they SHOULD NOT be included. Examples of valid entry listing endpoint URLs: @@ -1214,65 +1214,65 @@ Standard OPTIONAL URL query parameters standardized by the JSON:API specificatio - **filter**: a filter string, in the format described below in section `API Filtering Format Specification`_. - **page\_limit**: sets a numerical limit on the number of entries returned. -See `JSON:API 1.1 `__. -The API implementation MUST return no more than the number specified. -It MAY return fewer. -The database MAY have a maximum limit and not accept larger numbers (in which case the :http-error:`403 Forbidden` error code MUST be returned). -The default limit value is up to the API implementation to decide. -Example: :query-url:`http://example.com/optimade/v1/structures?page_limit=100` + See `JSON:API 1.1 `__. + The API implementation MUST return no more than the number specified. + It MAY return fewer. + The database MAY have a maximum limit and not accept larger numbers (in which case the :http-error:`403 Forbidden` error code MUST be returned). + The default limit value is up to the API implementation to decide. + Example: :query-url:`http://example.com/optimade/v1/structures?page_limit=100` - **page\_{offset, number, cursor, above, below}**: A server MUST implement pagination in the case of no user-specified :query-param:`sort` parameter (via the :field:`links` response field, see section `JSON Response Schema: Common Fields`_). -A server MAY implement pagination in concert with :query-param:`sort`. -The following parameters, all prefixed by "page\_", are RECOMMENDED for use with pagination. -If an implementation chooses + A server MAY implement pagination in concert with :query-param:`sort`. + The following parameters, all prefixed by "page\_", are RECOMMENDED for use with pagination. + If an implementation chooses -- *offset-based pagination*: using :field:`page_offset` and :field:`page_limit` is RECOMMENDED. -- *cursor-based pagination*: using :field:`page_cursor` and :field:`page_limit` is RECOMMENDED. -- *page-based pagination*: using :field:`page_number` and :field:`page_limit` is RECOMMENDED. It is RECOMMENDED that the first page has number 1, i.e., that :field:`page_number` is 1-based. -- *value-based pagination*: using :field:`page_above`/:field:`page_below` and :field:`page_limit` is RECOMMENDED. + - *offset-based pagination*: using :field:`page_offset` and :field:`page_limit` is RECOMMENDED. + - *cursor-based pagination*: using :field:`page_cursor` and :field:`page_limit` is RECOMMENDED. + - *page-based pagination*: using :field:`page_number` and :field:`page_limit` is RECOMMENDED. It is RECOMMENDED that the first page has number 1, i.e., that :field:`page_number` is 1-based. + - *value-based pagination*: using :field:`page_above`/:field:`page_below` and :field:`page_limit` is RECOMMENDED. -Examples (all OPTIONAL behavior a server MAY implement): + Examples (all OPTIONAL behavior a server MAY implement): -- skip 50 structures and fetch up to 100: :query-url:`/structures?page_offset=50&page_limit=100`. -- fetch page 2 of up to 50 structures per page: :query-url:`/structures?page_number=2&page_limit=50`. -- fetch up to 100 structures above sort-field value 4000 (in this example, server chooses to fetch results sorted by increasing :field:`id`, so :field:`page_above` value refers to an :field:`id` value): :query-url:`/structures?page_above=4000&page_limit=100`. + - skip 50 structures and fetch up to 100: :query-url:`/structures?page_offset=50&page_limit=100`. + - fetch page 2 of up to 50 structures per page: :query-url:`/structures?page_number=2&page_limit=50`. + - fetch up to 100 structures above sort-field value 4000 (in this example, server chooses to fetch results sorted by increasing :field:`id`, so :field:`page_above` value refers to an :field:`id` value): :query-url:`/structures?page_above=4000&page_limit=100`. - **sort**: If supporting sortable queries, an implementation MUST use the :query-param:`sort` query parameter with format as specified by `JSON:API 1.1 `__. -An implementation MAY support multiple sort fields for a single query. -If it does, it again MUST conform to the JSON:API 1.1 specification. + An implementation MAY support multiple sort fields for a single query. + If it does, it again MUST conform to the JSON:API 1.1 specification. -If an implementation supports sorting for an `entry listing endpoint `_, then the :endpoint:`/info/` endpoint MUST include, for each field name :field:`` in its :field:`data.properties.` response value that can be used for sorting, the key :field:`sortable` with value :field-val:`true`. -If a field name under an entry listing endpoint supporting sorting cannot be used for sorting, the server MUST either leave out the :field:`sortable` key or set it equal to :field-val:`false` for the specific field name. -The set of field names, with :field:`sortable` equal to :field-val:`true` are allowed to be used in the "sort fields" list according to its definition in the JSON:API 1.1 specification. -The field :field:`sortable` is in addition to each property description and other OPTIONAL fields. -An example is shown in section `Entry Listing Info Endpoints`_. + If an implementation supports sorting for an `entry listing endpoint `_, then the :endpoint:`/info/` endpoint MUST include, for each field name :field:`` in its :field:`data.properties.` response value that can be used for sorting, the key :field:`sortable` with value :field-val:`true`. + If a field name under an entry listing endpoint supporting sorting cannot be used for sorting, the server MUST either leave out the :field:`sortable` key or set it equal to :field-val:`false` for the specific field name. + The set of field names, with :field:`sortable` equal to :field-val:`true` are allowed to be used in the "sort fields" list according to its definition in the JSON:API 1.1 specification. + The field :field:`sortable` is in addition to each property description and other OPTIONAL fields. + An example is shown in section `Entry Listing Info Endpoints`_. - **include**: A server MAY implement the JSON:API concept of returning `compound documents `__ by utilizing the :query-param:`include` query parameter as specified by `JSON:API 1.0 `__. -All related resource objects MUST be returned as part of an array value for the top-level :field:`included` field, see section `JSON Response Schema: Common Fields`_. + All related resource objects MUST be returned as part of an array value for the top-level :field:`included` field, see section `JSON Response Schema: Common Fields`_. -The value of :query-param:`include` MUST be a comma-separated list of "relationship paths", as defined in the `JSON:API `__. -If relationship paths are not supported, or a server is unable to identify a relationship path a :http-error:`400 Bad Request` response MUST be made. + The value of :query-param:`include` MUST be a comma-separated list of "relationship paths", as defined in the `JSON:API `__. + If relationship paths are not supported, or a server is unable to identify a relationship path a :http-error:`400 Bad Request` response MUST be made. -The **default value** for :query-param:`include` is :query-val:`references`. -This means :entry:`references` entries MUST always be included under the top-level field :field:`included` as default, since a server assumes if :query-param:`include` is not specified by a client in the request, it is still specified as :query-string:`include=references`. -Note, if a client explicitly specifies :query-param:`include` and leaves out :query-val:`references`, :entry:`references` resource objects MUST NOT be included under the top-level field :field:`included`, as per the definition of :field:`included`, see section `JSON Response Schema: Common Fields`_. + The **default value** for :query-param:`include` is :query-val:`references`. + This means :entry:`references` entries MUST always be included under the top-level field :field:`included` as default, since a server assumes if :query-param:`include` is not specified by a client in the request, it is still specified as :query-string:`include=references`. + Note, if a client explicitly specifies :query-param:`include` and leaves out :query-val:`references`, :entry:`references` resource objects MUST NOT be included under the top-level field :field:`included`, as per the definition of :field:`included`, see section `JSON Response Schema: Common Fields`_. -**Note**: A query with the parameter :query-param:`include` set to the empty string means no related resource objects are to be returned under the top-level field :field:`included`. + **Note**: A query with the parameter :query-param:`include` set to the empty string means no related resource objects are to be returned under the top-level field :field:`included`. Standard OPTIONAL URL query parameters not in the JSON:API specification: - **response\_format**: the output format requested (see section `Response Format`_). -Defaults to the format string 'json', which specifies the standard output format described in this specification. -Example: :query-url:`http://example.com/optimade/v1/structures?response_format=xml` + Defaults to the format string 'json', which specifies the standard output format described in this specification. + Example: :query-url:`http://example.com/optimade/v1/structures?response_format=xml` - **email\_address**: an email address of the user making the request. -The email SHOULD be that of a person and not an automatic system. -Example: :query-url:`http://example.com/optimade/v1/structures?email_address=user@example.com` + The email SHOULD be that of a person and not an automatic system. + Example: :query-url:`http://example.com/optimade/v1/structures?email_address=user@example.com` - **response\_fields**: a comma-delimited set of fields to be provided in the output. -If provided, these fields MUST be returned along with the REQUIRED fields. -Other OPTIONAL fields MUST NOT be returned when this parameter is present. -Example: :query-url:`http://example.com/optimade/v1/structures?response_fields=last_modified,nsites` + If provided, these fields MUST be returned along with the REQUIRED fields. + Other OPTIONAL fields MUST NOT be returned when this parameter is present. + Example: :query-url:`http://example.com/optimade/v1/structures?response_fields=last_modified,nsites` Additional OPTIONAL URL query parameters not described above are not considered to be part of this standard, and are instead considered to be "custom URL query parameters". These custom URL query parameters MUST be of the format "". @@ -1287,8 +1287,8 @@ Examples: - :query-url:`http://example.com/optimade/v1/structures?_exmpl_warning_verbosity=10` - :query-url:`http://example.com/optimade/v1/structures?_exmpl_filter="elements all in [Al, Si, Ga]"` -**Note**: the specification presently makes no attempt to standardize access control mechanisms. -There are security concerns with access control based on URL tokens, and the above example is not to be taken as a recommendation for such a mechanism. + **Note**: the specification presently makes no attempt to standardize access control mechanisms. + There are security concerns with access control based on URL tokens, and the above example is not to be taken as a recommendation for such a mechanism. Entry Listing JSON Response Schema ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -1301,51 +1301,51 @@ In the default JSON response format every dictionary (`resource object `__ can OPTIONALLY contain the field -- **self**: the entry's URL + - **self**: the entry's URL - **meta**: a `JSON API meta object `__ that is used to communicate metadata. -See `JSON Response Schema: Common Fields`_ for more information about this field. + See `JSON Response Schema: Common Fields`_ for more information about this field. - **relationships**: a dictionary containing references to other entries according to the description in section `Relationships`_ encoded as `JSON:API Relationships `__. -The OPTIONAL human-readable description of the relationship MAY be provided in the :field:`description` field inside the :field:`meta` dictionary of the JSON:API resource identifier object. -All relationships to entries of the same entry type MUST be grouped into the same JSON:API relationship object and placed in the relationships dictionary with the entry type name as key (e.g., :entry:`structures`). + The OPTIONAL human-readable description of the relationship MAY be provided in the :field:`description` field inside the :field:`meta` dictionary of the JSON:API resource identifier object. + All relationships to entries of the same entry type MUST be grouped into the same JSON:API relationship object and placed in the relationships dictionary with the entry type name as key (e.g., :entry:`structures`). Example: .. code:: jsonc -{ -"data": [ -{ -"type": "structures", -"id": "example.db:structs:0001", -"attributes": { -"chemical_formula_descriptive": "Es2 O3", -"url": "http://example.db/structs/0001", -"immutable_id": "http://example.db/structs/0001@123", -"last_modified": "2007-04-05T14:30:20Z" -} -}, -{ -"type": "structures", -"id": "example.db:structs:1234", -"attributes": { -"chemical_formula_descriptive": "Es2", -"url": "http://example.db/structs/1234", -"immutable_id": "http://example.db/structs/1234@123", -"last_modified": "2007-04-07T12:02:20Z" -} -} -// ... -] -// ... -} + { + "data": [ + { + "type": "structures", + "id": "example.db:structs:0001", + "attributes": { + "chemical_formula_descriptive": "Es2 O3", + "url": "http://example.db/structs/0001", + "immutable_id": "http://example.db/structs/0001@123", + "last_modified": "2007-04-05T14:30:20Z" + } + }, + { + "type": "structures", + "id": "example.db:structs:1234", + "attributes": { + "chemical_formula_descriptive": "Es2", + "url": "http://example.db/structs/1234", + "immutable_id": "http://example.db/structs/1234@123", + "last_modified": "2007-04-07T12:02:20Z" + } + } + // ... + ] + // ... + } Single Entry Endpoints ---------------------- @@ -1373,26 +1373,25 @@ The meaning of these URL query parameters are as defined above in section `Entry One additional query parameter :query-param:`property_slices` MUST be handled by the API implementation either as defined below or by returning the error :http-error:`501 Not Implemented`: -- **property\_slices**: A number of slice specifications to request only parts of array properties for the functionality described in `Slices of array properties`_. - - The query parameter contains a comma-separated (",", ASCII 44(0x2C)) list of slice specifications. - Each slice specification consists of an ordered sequence of four elements separated by the colon character (":", ASCII 58(0x3A)). - The elements in the sequence are the dimension name and the three components of the slice, i.e., the start, stop, and step values defined in the same way as for a `slice object`_. - Omitting the value for any of the components of the slice specifies a default value (however, all the colon separators MUST be included). - The start value specifies the first index in that dimension for which values should be returned (which is 0-based and inclusive). - The default is :val:`0`. - The stop value specifies the last index for which values should be returned (inclusive). - The default is the last index of the array along the specified dimension. - The step value specifies the step size in that dimension. - The default is :val:`1`. +- **property\_slices**: A number of slice specifications to specify parts of array properties for the functionality described in `Slices of array properties`_. - An empty value of the :query-param:`property_slices` query parameter MUST be interpreted as equivalent to the query parameter not being included in the request. + The query parameter contains a comma-separated (",", ASCII 44(0x2C)) list of slice specification. + Each slice specification consists of a dimension name and the start, stop and step values of the slice separated by the colon character (":", ASCII 58(0x3A)) + Omitting the value between two colons specifies a default value. + The components of the slice are defined in the same way as for a `slice object`_. + The first integer specifies the start of the slice, i.e. the first index in that dimension for which values should be returned. + The default is the index of the first value. + The second integer specifies the end/stop of the slice, i.e. the last index for which values should be returned. + The default is the index of the last value of the property. + The third integer specifies the step size in that dimension. + The default is :val:`1` + The slices are 0-based, i.e. the first value has index 0, and inclusive i.e. for the dimension :val:`dim_frames` the range :val:`dim_frames:10:20:1` the last value returned belongs to index 20. Requirements and conventions for the response when this query parameter is used are described in `Slices of array properties`_. Example: - - :query-url:`http://optimade.example.com/v1/trajectories/id_12345?response_fields=cartesian_site_positions&property_ranges=dim_frames::999:10,dim_sites:30:70:` + - :query-url:`http://optimade.example.com/v1/trajectories/id_12345?response_fields=cartesian_site_positions&property_ranges=dim_frames::999:10,dim_sites:30:70::` This query URL requests items from the array :field:`cartesian_site_positions` is requested only for the 31st to 71st sites (i.e., with indexes 30 through 70 inclusive) for 1 out of every 10 frames of the first 1000 frames (i.e., taking steps of 10 over indexes 0 through 999 inclusive, which requests the frames with index 0, 10, 20, 30, ..., 990) of a trajectory with ID :val:`id_12345`. From 315546a291920ff4f6c10aa464ea7f750f93fcb8 Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Fri, 14 Jun 2024 10:24:40 +0200 Subject: [PATCH 23/70] Edit property slice definition [WIP] --- optimade.rst | 29 +++++++++++++++-------------- 1 file changed, 15 insertions(+), 14 deletions(-) diff --git a/optimade.rst b/optimade.rst index 35c37ea7d..2de298162 100644 --- a/optimade.rst +++ b/optimade.rst @@ -1373,25 +1373,26 @@ The meaning of these URL query parameters are as defined above in section `Entry One additional query parameter :query-param:`property_slices` MUST be handled by the API implementation either as defined below or by returning the error :http-error:`501 Not Implemented`: -- **property\_slices**: A number of slice specifications to specify parts of array properties for the functionality described in `Slices of array properties`_. - - The query parameter contains a comma-separated (",", ASCII 44(0x2C)) list of slice specification. - Each slice specification consists of a dimension name and the start, stop and step values of the slice separated by the colon character (":", ASCII 58(0x3A)) - Omitting the value between two colons specifies a default value. - The components of the slice are defined in the same way as for a `slice object`_. - The first integer specifies the start of the slice, i.e. the first index in that dimension for which values should be returned. - The default is the index of the first value. - The second integer specifies the end/stop of the slice, i.e. the last index for which values should be returned. - The default is the index of the last value of the property. - The third integer specifies the step size in that dimension. - The default is :val:`1` - The slices are 0-based, i.e. the first value has index 0, and inclusive i.e. for the dimension :val:`dim_frames` the range :val:`dim_frames:10:20:1` the last value returned belongs to index 20. +- **property\_slices**: A number of slice specifications to request only parts of array properties for the functionality described in `Slices of array properties`_. + + The query parameter contains a comma-separated (",", ASCII 44(0x2C)) list of slice specifications. + Each slice specification consists of an ordered sequence of four elements separated by the colon character (":", ASCII 58(0x3A)). + The elements in the sequence are the dimension name and the three components of the slice, i.e., the start, stop, and step values defined in the same way as for a `slice object`_. + Omitting the value for any of the components of the slice specifies a default value (however, all the colon separators MUST be included). + The start value specifies the first index in that dimension for which values should be returned (which is 0-based and inclusive). + The default is :val:`0`. + The stop value specifies the last index for which values should be returned (inclusive). + The default is the last index of the array along the specified dimension. + The step value specifies the step size in that dimension. + The default is :val:`1`. + + An empty value of the :query-param:`property_slices` query parameter MUST be interpreted as equivalent to the query parameter not being included in the request. Requirements and conventions for the response when this query parameter is used are described in `Slices of array properties`_. Example: - - :query-url:`http://optimade.example.com/v1/trajectories/id_12345?response_fields=cartesian_site_positions&property_ranges=dim_frames::999:10,dim_sites:30:70::` + - :query-url:`http://optimade.example.com/v1/trajectories/id_12345?response_fields=cartesian_site_positions&property_ranges=dim_frames::999:10,dim_sites:30:70:` This query URL requests items from the array :field:`cartesian_site_positions` is requested only for the 31st to 71st sites (i.e., with indexes 30 through 70 inclusive) for 1 out of every 10 frames of the first 1000 frames (i.e., taking steps of 10 over indexes 0 through 999 inclusive, which requests the frames with index 0, 10, 20, 30, ..., 990) of a trajectory with ID :val:`id_12345`. From ce9661b629381fe198a9184cbbc0c6b10ef0ad72 Mon Sep 17 00:00:00 2001 From: Giovanni Pizzi Date: Fri, 14 Jun 2024 10:37:22 +0200 Subject: [PATCH 24/70] Small sentence fix --- optimade.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/optimade.rst b/optimade.rst index 2de298162..03625b7e2 100644 --- a/optimade.rst +++ b/optimade.rst @@ -1394,7 +1394,7 @@ One additional query parameter :query-param:`property_slices` MUST be handled by - :query-url:`http://optimade.example.com/v1/trajectories/id_12345?response_fields=cartesian_site_positions&property_ranges=dim_frames::999:10,dim_sites:30:70:` - This query URL requests items from the array :field:`cartesian_site_positions` is requested only for the 31st to 71st sites (i.e., with indexes 30 through 70 inclusive) for 1 out of every 10 frames of the first 1000 frames (i.e., taking steps of 10 over indexes 0 through 999 inclusive, which requests the frames with index 0, 10, 20, 30, ..., 990) of a trajectory with ID :val:`id_12345`. + This query URL requests items from the array :field:`cartesian_site_positions` only for the 31st to 71st sites (i.e., with indexes 30 through 70 inclusive) for 1 out of every 10 frames of the first 1000 frames (i.e., taking steps of 10 over indexes 0 through 999 inclusive, which requests the frames with index 0, 10, 20, 30, ..., 990) of a trajectory with ID :val:`id_12345`. Single Entry JSON Response Schema ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From 358157f8f82ed445e1eb94e7f6e9e12795cfff71 Mon Sep 17 00:00:00 2001 From: Giovanni Pizzi Date: Fri, 14 Jun 2024 10:49:41 +0200 Subject: [PATCH 25/70] Clarifying the part on dimension names. --- optimade.rst | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/optimade.rst b/optimade.rst index 03625b7e2..db3920b4a 100644 --- a/optimade.rst +++ b/optimade.rst @@ -2367,13 +2367,13 @@ A Property Definition MUST be composed according to the combination of the requi **REQUIRED keys:** - :field:`names`: List of Strings. - A list of names of the dimensions of the underlying one or multi-dimensionsional data represented as mutiple levels of lists. - The order is that the the first name applies to the outermost list, the next name to the lists embedded in that list, etc. + A list of names of the dimensions of the underlying one or multi-dimensionsional data represented as multiple levels of lists. + The order is such that the first name applies to the outermost list, the next name to the lists embedded in that list, etc. Dimension names defined by the OPTIMADE standard are prefixed by ``dim_``. - Dimension names defined by database or definition providers MUST be prefixed by the corresponding namespace prefix and SHOULD also be prefixed with ``dim_``, e.g., ``_exmpl_dim_particles``. - If, within one entry, two or more array dimensions in one or more properties share the same :field:`name`, those represent the same dimension. - For example, in a trajectory entry there is a property :property:`cartesian_site_positions` where one of the dimension names is :val:`dim_frames`. - If there is another one-dimensional array property :property:`_exmpl_energy` that specifies the same dimension name in its property definition, then this declares that the energy and site positions at index *i* pertain to the same frame. + Dimension names defined by database or definition providers MUST be prefixed by the corresponding database or namespace prefix, and SHOULD also be prefixed with ``dim_``, e.g., ``_exmpl_dim_particles``. + If, within one entry, two or more array axes in one or more properties share the same dimension :field:`name`, those represent the same dimension. + For example, let us consider the property :property:`cartesian_site_positions` of the trajectory entry, where the first dimension name is :val:`dim_frames`. + If there is another one-dimensional (i.e., with a single axis) array property :property:`_exmpl_energy` of the same trajectory entry that specifies in its property definition the same dimension name :val:`dim_frames` for its axis, then the values of :property:`_exmpl_energy` and of :property:`cartesian_site_positions` at index *i* pertain to the same frame. - :field:`sizes`: List of Integers or :val:`null`. A list of fixed length requirements on the underlying one or multi-dimensionsional data represented as mutiple levels of lists. From 7e29652ecf5a5503dbe5cfd9c571496b183ee621 Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Sat, 15 Jun 2024 15:30:46 +0200 Subject: [PATCH 26/70] Apply suggestions from review Co-authored-by: Andrius Merkys Co-authored-by: Antanas Vaitkus --- optimade.rst | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/optimade.rst b/optimade.rst index db3920b4a..04daee596 100644 --- a/optimade.rst +++ b/optimade.rst @@ -623,12 +623,12 @@ The field :field:`array_axes` is defined as follows: A list of dictionaries which provide information related to the axes of an array property, which is relevant for slicing. Each item, in order, represents the array axis as declared in the corresponding property definition. - Each item MUST be a dictionary with the following REQUIRED fields: + Each item MUST be a dictionary with the following REQUIRED field: - :field:`dimension_name`: String. The dimension name of the corresponding array axis. - If the request specifies the :query-param:`property_slices` query parameter for any of the array axes of this array property the following key MUST be present, otherwise it MUST be omitted or equal to :val:`null`: + If the request specifies the :query-param:`property_slices` query parameter for any of the array axes of this array property the following key MUST be present, otherwise the key MUST be omitted or equal to :val:`null`: - :field:`requested_slice`: Dictionary. A field that describes the requested slice that was provided via the query parameter :query-param:`property_slices`. @@ -685,7 +685,7 @@ The field :field:`array_axes` is defined as follows: - ``{"start": 3, "stop": 7, "step": 2}`` means the server certifies that values at indexes 0,1,2,4,6 and any index from 8 to the end of the array are :val:`null`. -Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format for a request to the trajectory endpoint with the query parameter :query-param:`property_slices=dim_frames:3:37:5` and :query-param:`response_fields=cartesian_site_positions,_exmpl_temperature` where the trajectory consists of 432934 frames (with indexes 0 to 432933) and where the :field:`cartisian_site_positions` contains 7 sites. Furthermore, the :field:`_exmpl_temperature` contains only :val:`null` values except for items with indexes 1000, 1030, 1060, ..., 4000 (where the values can be either numeric or :val:`null`). +Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format for a request to the trajectory endpoint with the query parameter :query-param:`property_slices=dim_frames:3:37:5` and :query-param:`response_fields=cartesian_site_positions,_exmpl_temperature` where the trajectory consists of 432934 frames (with indexes 0 to 432933) and where the :field:`cartesian_site_positions` contains 7 sites. Furthermore, the :field:`_exmpl_temperature` contains only :val:`null` values except for items with indexes 1000, 1030, 1060, ..., 4000 (where the values can be either numeric or :val:`null`). .. code:: jsonc @@ -715,7 +715,7 @@ Below follows an example of the :field:`data` and :field:`meta` parts of a respo "step": 1 }, "sliceable": true, - "length": 432934, + "length": 432934 }, { "dimension_name": "dim_sites", @@ -725,13 +725,13 @@ Below follows an example of the :field:`data` and :field:`meta` parts of a respo "step": 1 }, "sliceable": false, - "length": 7, + "length": 7 }, { "dimension_name": "dim_spatial", - "length": 3, + "length": 3 } - ], + ] }, "_exmpl_temperature": { "array_axes": [ @@ -748,7 +748,7 @@ Below follows an example of the :field:`data` and :field:`meta` parts of a respo "step": 30 }, "sliceable": true, - "length": 432934, + "length": 432934 } ] } @@ -1394,7 +1394,7 @@ One additional query parameter :query-param:`property_slices` MUST be handled by - :query-url:`http://optimade.example.com/v1/trajectories/id_12345?response_fields=cartesian_site_positions&property_ranges=dim_frames::999:10,dim_sites:30:70:` - This query URL requests items from the array :field:`cartesian_site_positions` only for the 31st to 71st sites (i.e., with indexes 30 through 70 inclusive) for 1 out of every 10 frames of the first 1000 frames (i.e., taking steps of 10 over indexes 0 through 999 inclusive, which requests the frames with index 0, 10, 20, 30, ..., 990) of a trajectory with ID :val:`id_12345`. + This query URL requests items from the array :field:`cartesian_site_positions` only for the 31st to 71st sites (i.e., with indexes 30 through 70 inclusive) for 1 out of every 10 frames of the first 1000 frames (i.e., taking steps of 10 over indexes 0 through 999 inclusive, which requests the frames with indexes 0, 10, 20, 30, ..., 990) of a trajectory with ID :val:`id_12345`. Single Entry JSON Response Schema ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -2370,7 +2370,7 @@ A Property Definition MUST be composed according to the combination of the requi A list of names of the dimensions of the underlying one or multi-dimensionsional data represented as multiple levels of lists. The order is such that the first name applies to the outermost list, the next name to the lists embedded in that list, etc. Dimension names defined by the OPTIMADE standard are prefixed by ``dim_``. - Dimension names defined by database or definition providers MUST be prefixed by the corresponding database or namespace prefix, and SHOULD also be prefixed with ``dim_``, e.g., ``_exmpl_dim_particles``. + Dimension names defined by database or definition providers MUST be prefixed by the corresponding database or namespace prefix, and SHOULD also be prefixed by ``dim_``, e.g., ``_exmpl_dim_particles``. If, within one entry, two or more array axes in one or more properties share the same dimension :field:`name`, those represent the same dimension. For example, let us consider the property :property:`cartesian_site_positions` of the trajectory entry, where the first dimension name is :val:`dim_frames`. If there is another one-dimensional (i.e., with a single axis) array property :property:`_exmpl_energy` of the same trajectory entry that specifies in its property definition the same dimension name :val:`dim_frames` for its axis, then the values of :property:`_exmpl_energy` and of :property:`cartesian_site_positions` at index *i* pertain to the same frame. From 085030d2cba15e35b7ce8c7a7f9883530984d25d Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Sat, 15 Jun 2024 16:06:08 +0200 Subject: [PATCH 27/70] Define the term array for OPTMADE data types --- optimade.rst | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/optimade.rst b/optimade.rst index 04daee596..e19681e0b 100644 --- a/optimade.rst +++ b/optimade.rst @@ -218,7 +218,10 @@ representation in all contexts. They are as follows: - Basic types: **string**, **integer**, **float**, **boolean**, **timestamp**. - **list**: an ordered collection of items, where all items are of the same type, unless they are unknown. A list can be empty, i.e., contain no items. -- **dictionary**: an associative array of **keys** and **values**, where **keys** are pre-determined strings, i.e., for the same entry property, the **keys** remain the same among different entries whereas the **values** change. + Multidimensional collections of items are represented as nested lists. + The specification uses **array** as a more general term for structures of nested lists representing single or multidimensional data, and the term **array axes** for the levels of nesting. + Note that arrays are represented using lists and not as a separate data type. +- **dictionary**: an associative set of **keys** and **values**, where **keys** are pre-determined strings, i.e., for the same entry property, the **keys** remain the same among different entries whereas the **values** change. The **values** of a dictionary can be any basic type, list, dictionary, or unknown. An entry property value that is not present in the database is **unknown**. @@ -683,7 +686,7 @@ The field :field:`array_axes` is defined as follows: Examples: - - ``{"start": 3, "stop": 7, "step": 2}`` means the server certifies that values at indexes 0,1,2,4,6 and any index from 8 to the end of the array are :val:`null`. + - ``{"start": 3, "stop": 7, "step": 2}`` means the server certifies that values at indexes 0, 1, 2, 4, 6 and any index from 8 to the end of the array are :val:`null`. Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format for a request to the trajectory endpoint with the query parameter :query-param:`property_slices=dim_frames:3:37:5` and :query-param:`response_fields=cartesian_site_positions,_exmpl_temperature` where the trajectory consists of 432934 frames (with indexes 0 to 432933) and where the :field:`cartesian_site_positions` contains 7 sites. Furthermore, the :field:`_exmpl_temperature` contains only :val:`null` values except for items with indexes 1000, 1030, 1060, ..., 4000 (where the values can be either numeric or :val:`null`). @@ -1392,9 +1395,9 @@ One additional query parameter :query-param:`property_slices` MUST be handled by Example: - - :query-url:`http://optimade.example.com/v1/trajectories/id_12345?response_fields=cartesian_site_positions&property_ranges=dim_frames::999:10,dim_sites:30:70:` + - :query-url:`http://optimade.example.com/v1/trajectories/id_12345?response_fields=frame_cartesian_site_positions&property_ranges=dim_frames::999:10,dim_sites:30:70:` - This query URL requests items from the array :field:`cartesian_site_positions` only for the 31st to 71st sites (i.e., with indexes 30 through 70 inclusive) for 1 out of every 10 frames of the first 1000 frames (i.e., taking steps of 10 over indexes 0 through 999 inclusive, which requests the frames with indexes 0, 10, 20, 30, ..., 990) of a trajectory with ID :val:`id_12345`. + This query URL requests items from the array :field:`frame_cartesian_site_positions` only for the 31st to 71st sites (i.e., with indexes 30 through 70 inclusive) for 1 out of every 10 frames of the first 1000 frames (i.e., taking steps of 10 over indexes 0 through 999 inclusive, which requests the frames with indexes 0, 10, 20, 30, ..., 990) of a trajectory with ID :val:`id_12345`. Single Entry JSON Response Schema ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From b3a96a0ac424324355b8b1f9373c97ae265de135 Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Sat, 15 Jun 2024 16:16:36 +0200 Subject: [PATCH 28/70] Fix examples for slicing assuming a trajectory cartesian_site_coordinates with different format than the structure one --- optimade.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/optimade.rst b/optimade.rst index e19681e0b..97f97723c 100644 --- a/optimade.rst +++ b/optimade.rst @@ -688,7 +688,7 @@ The field :field:`array_axes` is defined as follows: - ``{"start": 3, "stop": 7, "step": 2}`` means the server certifies that values at indexes 0, 1, 2, 4, 6 and any index from 8 to the end of the array are :val:`null`. -Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format for a request to the trajectory endpoint with the query parameter :query-param:`property_slices=dim_frames:3:37:5` and :query-param:`response_fields=cartesian_site_positions,_exmpl_temperature` where the trajectory consists of 432934 frames (with indexes 0 to 432933) and where the :field:`cartesian_site_positions` contains 7 sites. Furthermore, the :field:`_exmpl_temperature` contains only :val:`null` values except for items with indexes 1000, 1030, 1060, ..., 4000 (where the values can be either numeric or :val:`null`). +Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format for a request to the trajectory endpoint with the query parameter :query-param:`property_slices=dim_frames:3:37:5` and :query-param:`response_fields=frame_cartesian_site_positions,_exmpl_temperature` where the trajectory consists of 432934 frames (with indexes 0 to 432933) and where the :field:`frame_cartesian_site_positions` contains 7 sites. Furthermore, the :field:`_exmpl_temperature` contains only :val:`null` values except for items with indexes 1000, 1030, 1060, ..., 4000 (where the values can be either numeric or :val:`null`). .. code:: jsonc @@ -698,12 +698,12 @@ Below follows an example of the :field:`data` and :field:`meta` parts of a respo "type": "trajectories", "id": "2345678", "attributes": { - "cartesian_site_positions": null, + "frame_cartesian_site_positions": null, "_exmpl_temperature": null }, "meta": { "property_metadata": { - "cartesian_site_positions": { + "frame_cartesian_site_positions": { "array_axes": [ { "dimension_name": "dim_frames", @@ -2375,8 +2375,8 @@ A Property Definition MUST be composed according to the combination of the requi Dimension names defined by the OPTIMADE standard are prefixed by ``dim_``. Dimension names defined by database or definition providers MUST be prefixed by the corresponding database or namespace prefix, and SHOULD also be prefixed by ``dim_``, e.g., ``_exmpl_dim_particles``. If, within one entry, two or more array axes in one or more properties share the same dimension :field:`name`, those represent the same dimension. - For example, let us consider the property :property:`cartesian_site_positions` of the trajectory entry, where the first dimension name is :val:`dim_frames`. - If there is another one-dimensional (i.e., with a single axis) array property :property:`_exmpl_energy` of the same trajectory entry that specifies in its property definition the same dimension name :val:`dim_frames` for its axis, then the values of :property:`_exmpl_energy` and of :property:`cartesian_site_positions` at index *i* pertain to the same frame. + For example, let us consider the property :property:`frame_cartesian_site_positions` of the trajectory entry, where the first dimension name is :val:`dim_frames`. + If there is another one-dimensional (i.e., with a single axis) array property :property:`_exmpl_energy` of the same trajectory entry that specifies in its property definition the same dimension name :val:`dim_frames` for its axis, then the values of :property:`_exmpl_energy` and of :property:`frame_cartesian_site_positions` at index *i* pertain to the same frame. - :field:`sizes`: List of Integers or :val:`null`. A list of fixed length requirements on the underlying one or multi-dimensionsional data represented as mutiple levels of lists. From 6231d2a408b0140cbf7c82e2454f120e22e83c49 Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Sat, 15 Jun 2024 16:37:33 +0200 Subject: [PATCH 29/70] Adjust confusing phrasing about slicing based on review comment --- optimade.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/optimade.rst b/optimade.rst index 97f97723c..2e512801d 100644 --- a/optimade.rst +++ b/optimade.rst @@ -688,7 +688,8 @@ The field :field:`array_axes` is defined as follows: - ``{"start": 3, "stop": 7, "step": 2}`` means the server certifies that values at indexes 0, 1, 2, 4, 6 and any index from 8 to the end of the array are :val:`null`. -Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format for a request to the trajectory endpoint with the query parameter :query-param:`property_slices=dim_frames:3:37:5` and :query-param:`response_fields=frame_cartesian_site_positions,_exmpl_temperature` where the trajectory consists of 432934 frames (with indexes 0 to 432933) and where the :field:`frame_cartesian_site_positions` contains 7 sites. Furthermore, the :field:`_exmpl_temperature` contains only :val:`null` values except for items with indexes 1000, 1030, 1060, ..., 4000 (where the values can be either numeric or :val:`null`). +Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format for a request to the trajectory endpoint with the query parameter :query-param:`property_slices=dim_frames:3:37:5` and :query-param:`response_fields=frame_cartesian_site_positions,_exmpl_temperature` where the trajectory consists of 432934 frames (with indexes 0 to 432933) and where the :field:`frame_cartesian_site_positions` contains 7 sites. +Furthermore, the metadata subfield :field:`available_slice` in :field:`array_axes` in :field:`_exmpl_temperature` certifies that all values of the data field :field:`_exmpl_temperature` are :val:`null` except at indexes 1000, 1030, 1060, ..., 4000, where values can be either numeric or :val:`null`). .. code:: jsonc From 9f639d18577af27b9f47e8753f4f193b44a4ddb4 Mon Sep 17 00:00:00 2001 From: Giovanni Pizzi Date: Thu, 4 Jul 2024 09:13:03 +0200 Subject: [PATCH 30/70] Apply suggestions from code review Co-authored-by: Antanas Vaitkus --- optimade.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/optimade.rst b/optimade.rst index 2e512801d..064628bb6 100644 --- a/optimade.rst +++ b/optimade.rst @@ -639,11 +639,11 @@ The field :field:`array_axes` is defined as follows: The implementation MUST preserve the values as given in the query parameter, including the distinction between specific values and default values even when they are equivalent. It MAY contain the following subfields that are defined according to the specification of a `slice object`_. - - :field:`start`: Integer or :val:`null`. + - :field:`start`: Non-negative integer or :val:`null`. - - :field:`stop`: Integer or :val:`null`. + - :field:`stop`: Non-negative integer or :val:`null`. - - :field:`step`: Integer or :val:`null`. + - :field:`step`: Positive integer or :val:`null`. A :val:`null` value for any of these fields means it has the default value defined for a `slice object`_. A missing value is equivalent to a :val:`null` value. @@ -1379,8 +1379,8 @@ One additional query parameter :query-param:`property_slices` MUST be handled by - **property\_slices**: A number of slice specifications to request only parts of array properties for the functionality described in `Slices of array properties`_. - The query parameter contains a comma-separated (",", ASCII 44(0x2C)) list of slice specifications. - Each slice specification consists of an ordered sequence of four elements separated by the colon character (":", ASCII 58(0x3A)). + The query parameter contains a comma-separated (",", ASCII symbol 44 dec, 0x2C hex) list of slice specifications. + Each slice specification consists of an ordered sequence of four elements separated by the colon character (":", ASCII symbol 58 dec, 0x3A hex). The elements in the sequence are the dimension name and the three components of the slice, i.e., the start, stop, and step values defined in the same way as for a `slice object`_. Omitting the value for any of the components of the slice specifies a default value (however, all the colon separators MUST be included). The start value specifies the first index in that dimension for which values should be returned (which is 0-based and inclusive). From 46eafb55dc3f584270cad6a0cd990cc247b0cab7 Mon Sep 17 00:00:00 2001 From: Giovanni Pizzi Date: Thu, 4 Jul 2024 09:20:25 +0200 Subject: [PATCH 31/70] Apply suggestions from code review Co-authored-by: Antanas Vaitkus --- optimade.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/optimade.rst b/optimade.rst index 064628bb6..46aa40eaa 100644 --- a/optimade.rst +++ b/optimade.rst @@ -221,7 +221,7 @@ representation in all contexts. They are as follows: Multidimensional collections of items are represented as nested lists. The specification uses **array** as a more general term for structures of nested lists representing single or multidimensional data, and the term **array axes** for the levels of nesting. Note that arrays are represented using lists and not as a separate data type. -- **dictionary**: an associative set of **keys** and **values**, where **keys** are pre-determined strings, i.e., for the same entry property, the **keys** remain the same among different entries whereas the **values** change. +- **dictionary**: a collection of **key**-**value** pairs, where **keys** are pre-determined strings, i.e., for the same entry property the **keys** remain the same among different entries whereas the **values** change. The **values** of a dictionary can be any basic type, list, dictionary, or unknown. An entry property value that is not present in the database is **unknown**. From 657a696526f0f050be16d111d0cab402d6a362bb Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Tue, 18 Jun 2024 16:01:53 +0200 Subject: [PATCH 32/70] Change the format of property_slices to the syntax dim_xxx[start:stop:step] --- optimade.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/optimade.rst b/optimade.rst index 46aa40eaa..59a0e240c 100644 --- a/optimade.rst +++ b/optimade.rst @@ -1379,8 +1379,9 @@ One additional query parameter :query-param:`property_slices` MUST be handled by - **property\_slices**: A number of slice specifications to request only parts of array properties for the functionality described in `Slices of array properties`_. - The query parameter contains a comma-separated (",", ASCII symbol 44 dec, 0x2C hex) list of slice specifications. - Each slice specification consists of an ordered sequence of four elements separated by the colon character (":", ASCII symbol 58 dec, 0x3A hex). + The query parameter contains a comma-separated (",", ASCII symbol 44 dec) list of slice specifications. + Each slice specification consists of a dimension name followed by the left square bracket character ("[", ASCII symbol 91 dec), a slice, and the right square bracket character ("]", ASCII symbol 93 dec). + The slice is given as an ordered sequence of three components separated by the colon character (":", ASCII symbol 58 dec). The elements in the sequence are the dimension name and the three components of the slice, i.e., the start, stop, and step values defined in the same way as for a `slice object`_. Omitting the value for any of the components of the slice specifies a default value (however, all the colon separators MUST be included). The start value specifies the first index in that dimension for which values should be returned (which is 0-based and inclusive). From 919b83d3e9cdcd3d07942aeef1ff5a2a0ef0c3c3 Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Fri, 18 Oct 2024 11:15:25 +0200 Subject: [PATCH 33/70] Fix in formulation for property_slices format dim_xxx[start:stop:step] --- optimade.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/optimade.rst b/optimade.rst index 59a0e240c..b2e8a45f8 100644 --- a/optimade.rst +++ b/optimade.rst @@ -1382,7 +1382,7 @@ One additional query parameter :query-param:`property_slices` MUST be handled by The query parameter contains a comma-separated (",", ASCII symbol 44 dec) list of slice specifications. Each slice specification consists of a dimension name followed by the left square bracket character ("[", ASCII symbol 91 dec), a slice, and the right square bracket character ("]", ASCII symbol 93 dec). The slice is given as an ordered sequence of three components separated by the colon character (":", ASCII symbol 58 dec). - The elements in the sequence are the dimension name and the three components of the slice, i.e., the start, stop, and step values defined in the same way as for a `slice object`_. + The elements in the sequence are the start, stop, and step values defined in the same way as for a `slice object`_. Omitting the value for any of the components of the slice specifies a default value (however, all the colon separators MUST be included). The start value specifies the first index in that dimension for which values should be returned (which is 0-based and inclusive). The default is :val:`0`. From a623390c6b4073382eda72d2230f69c35b271570 Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Fri, 18 Oct 2024 16:47:43 +0200 Subject: [PATCH 34/70] Apply suggestions from review Co-authored-by: ndaelman-hu <107392603+ndaelman-hu@users.noreply.github.com> --- optimade.rst | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/optimade.rst b/optimade.rst index b2e8a45f8..66fea6426 100644 --- a/optimade.rst +++ b/optimade.rst @@ -612,8 +612,8 @@ Slices of array properties -------------------------- The OPTIMADE standard defines a way for a client to request only a subset of the items of an array, referred to as a slice. -The protocol for this functionality allows the server to allow slicing independently per entry, per array, and per array axis. -This functionality is separate from the protocol described in `Transmission of large property values`_. +The protocol for this functionality specifies how a server MAY support slicing arrays independently per entry, per array, and per array axis. +This functionality is separate from (but compatible with) the protocol described in `Transmission of large property values`_. Slices are used for a client to ask the server to only provide a subset of items of an array, which can result in a small or large set of items. In contrast, the protocol for large property values is used by the server implementation to transmit a set of items that it deems too large to provide inside the normal OPTIMADE response. @@ -623,7 +623,7 @@ When the client request includes the query parameter :query-param:`property_slic The field :field:`array_axes` is defined as follows: - :field:`array_axes`: List of Dictionary. - A list of dictionaries which provide information related to the axes of an array property, which is relevant for slicing. + A list of dictionaries which provide information related to the axes of an array property, including sliceable axes. Each item, in order, represents the array axis as declared in the corresponding property definition. Each item MUST be a dictionary with the following REQUIRED field: @@ -680,8 +680,9 @@ The field :field:`array_axes` is defined as follows: If the field is omitted or :val:`null`, it means the same thing as :val:`false`. - :field:`available_slice`: Dictionary or :val:`null`. - This field describes a `slice object`_ where there MAY be non-null values in the data. - By including this field, the server certifies that there are only :val:`null` values outside this slice. + This field describes a `slice object`_. + By including this field, the server certifies that there MUST only be :val:`null` values outside this slice. + Inside the slice there MAY be any combination of null and non-null values. If not provided, or equal to :val:`null` or an empty dictionary, the client cannot make any assumptions about what part of the array contains :val:`null` values. Examples: @@ -2375,7 +2376,7 @@ A Property Definition MUST be composed according to the combination of the requi A list of names of the dimensions of the underlying one or multi-dimensionsional data represented as multiple levels of lists. The order is such that the first name applies to the outermost list, the next name to the lists embedded in that list, etc. Dimension names defined by the OPTIMADE standard are prefixed by ``dim_``. - Dimension names defined by database or definition providers MUST be prefixed by the corresponding database or namespace prefix, and SHOULD also be prefixed by ``dim_``, e.g., ``_exmpl_dim_particles``. + Dimension names defined by database or definition providers MUST be prefixed by the corresponding database or namespace prefix, which SHOULD then be immediately followed by ``dim_``, e.g., ``_exmpl_dim_particles``. If, within one entry, two or more array axes in one or more properties share the same dimension :field:`name`, those represent the same dimension. For example, let us consider the property :property:`frame_cartesian_site_positions` of the trajectory entry, where the first dimension name is :val:`dim_frames`. If there is another one-dimensional (i.e., with a single axis) array property :property:`_exmpl_energy` of the same trajectory entry that specifies in its property definition the same dimension name :val:`dim_frames` for its axis, then the values of :property:`_exmpl_energy` and of :property:`frame_cartesian_site_positions` at index *i* pertain to the same frame. From b164c09e6ce1dc86a139bc88a9ada0dcec325f65 Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Mon, 15 Sep 2025 17:56:41 +0300 Subject: [PATCH 35/70] Apply suggestions from code review --- optimade.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/optimade.rst b/optimade.rst index 66fea6426..c49687533 100644 --- a/optimade.rst +++ b/optimade.rst @@ -614,8 +614,8 @@ Slices of array properties The OPTIMADE standard defines a way for a client to request only a subset of the items of an array, referred to as a slice. The protocol for this functionality specifies how a server MAY support slicing arrays independently per entry, per array, and per array axis. This functionality is separate from (but compatible with) the protocol described in `Transmission of large property values`_. -Slices are used for a client to ask the server to only provide a subset of items of an array, which can result in a small or large set of items. -In contrast, the protocol for large property values is used by the server implementation to transmit a set of items that it deems too large to provide inside the normal OPTIMADE response. +The protocol for large property values is used by the server implementation to transmit a set of items that it deems too large to provide inside the normal OPTIMADE response. +Slices, on the other hand, are used for a client to request a subset of any size of the items of an array, which can possibly (but not necessarily) result in such a large amount of values that the protocol for large property values is required to transmit them. The main mechanism is provided through the query parameter :query-param:`property_slices` defined in section `Single Entry URL Query Parameters`_. Information relating to the ability of the server to handle this query parameter and the relevant ranges of indexes is provided using metadata property field :field:`array_axes` (see `Metadata properties`_). From 628b4da92a33ce48adf2e2710f811c55a6e8849b Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Mon, 15 Sep 2025 18:01:02 +0300 Subject: [PATCH 36/70] Apply changes from review Co-authored-by: Nathan Daelman <107392603+ndaelman-hu@users.noreply.github.com> --- optimade.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/optimade.rst b/optimade.rst index c49687533..eecbb78f4 100644 --- a/optimade.rst +++ b/optimade.rst @@ -618,7 +618,7 @@ The protocol for large property values is used by the server implementation to t Slices, on the other hand, are used for a client to request a subset of any size of the items of an array, which can possibly (but not necessarily) result in such a large amount of values that the protocol for large property values is required to transmit them. The main mechanism is provided through the query parameter :query-param:`property_slices` defined in section `Single Entry URL Query Parameters`_. -Information relating to the ability of the server to handle this query parameter and the relevant ranges of indexes is provided using metadata property field :field:`array_axes` (see `Metadata properties`_). +Information relating to the ability of the server to handle this query parameter and the relevant ranges of indexes is provided using the metadata property field :field:`array_axes` (see `Metadata properties`_). When the client request includes the query parameter :query-param:`property_slices`, the server MUST provide metadata for all properties for which including the subfield :field:`requested_slice` of the :field:`array_axes` is MANDATORY (see below). The field :field:`array_axes` is defined as follows: From 010c348cb455e3a3547867ac7faed3d7a9fac2b5 Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Mon, 15 Sep 2025 18:11:04 +0300 Subject: [PATCH 37/70] Changes from code review --- optimade.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/optimade.rst b/optimade.rst index eecbb78f4..dd11985cd 100644 --- a/optimade.rst +++ b/optimade.rst @@ -631,7 +631,7 @@ The field :field:`array_axes` is defined as follows: - :field:`dimension_name`: String. The dimension name of the corresponding array axis. - If the request specifies the :query-param:`property_slices` query parameter for any of the array axes of this array property the following key MUST be present, otherwise the key MUST be omitted or equal to :val:`null`: + If the request specifies the :query-param:`property_slices` query parameter for any of the array axes of this array property the following key MUST be present. However, if that query parameter is not specified, the key MUST either be omitted or set equal to :val:`null`: - :field:`requested_slice`: Dictionary. A field that describes the requested slice that was provided via the query parameter :query-param:`property_slices`. From 3e49e5e2f2398e6578a46a3cb28213815e08f7b9 Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Mon, 15 Sep 2025 18:14:30 +0300 Subject: [PATCH 38/70] Update from code review --- optimade.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/optimade.rst b/optimade.rst index dd11985cd..75778de94 100644 --- a/optimade.rst +++ b/optimade.rst @@ -636,7 +636,7 @@ The field :field:`array_axes` is defined as follows: - :field:`requested_slice`: Dictionary. A field that describes the requested slice that was provided via the query parameter :query-param:`property_slices`. The subfields MUST reflect the values provided via the :query-param:`property_slices`. - The implementation MUST preserve the values as given in the query parameter, including the distinction between specific values and default values even when they are equivalent. + The implementation MUST preserve the values as given in the query parameter, including the distinction between specific values and default values even when they are equivalent (see example below). It MAY contain the following subfields that are defined according to the specification of a `slice object`_. - :field:`start`: Non-negative integer or :val:`null`. From 15205332cd24be346de12f1578a1c329676baf0a Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Mon, 15 Sep 2025 18:17:20 +0300 Subject: [PATCH 39/70] Update from code review Co-authored-by: Nathan Daelman <107392603+ndaelman-hu@users.noreply.github.com> --- optimade.rst | 27 +++++++++++---------------- 1 file changed, 11 insertions(+), 16 deletions(-) diff --git a/optimade.rst b/optimade.rst index 75778de94..abf9fded6 100644 --- a/optimade.rst +++ b/optimade.rst @@ -650,22 +650,17 @@ The field :field:`array_axes` is defined as follows: (Consequently, the dictionary MAY be empty if all fields take the default values.) Examples: - - - For :query-string:`property_slices=dim_frames:::` some of the equivalent valid representations are: - - - ``{}`` - - ``{"start": null, "stop": null, "step": null}`` - - ``{"start": null}`` - - ``{"stop": null, "step": null}`` - - Whereas, e.g., the following representation is not valid: - - - ``{"start": 0}`` (despite that :val:`0` is the default value of the :field:`start` field.) - But it is instead a valid representation if the query parameter was :query-string:`property_slices=dim_frames:0::`. - - - :query-string:`property_slices=dim_frames:3:5:2` the representation MUST be: - - - ``{"start": 3, "stop": 5, "step": 2}`` + Query Type | Query Formulation | Valid Representations + ----------------|---------------------------|------------------------------ + fully formed | :query-string:`property_slices=dim_frames:3:5:2` | ``{"start": 3, "stop": 5, "step": 2}`` + empty query with start | :query-string:`property_slices=dim_frames:0::` | ``{"start": 0}`` + empty query | :query-string:`property_slices=dim_frames:::` | ``{}`` + | | ``{"start": null, "stop": null, "step": null}`` + | | ``{"start": null}`` + | | ``{"stop": null, "step": null}`` + + Note that any of the representations for the "empty query" are valid. + Even though, semantically, it means the same as the "empty query with start" at index zero, both differ in representation. This is due to the starting index being explicitely specified now. The reason becomes apparent once the starting index takes on a value different from the default. The dictionary MAY contain the following fields: From d7d2a811640d6000dd9ee4bfef6854f7de2254ad Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Mon, 15 Sep 2025 18:20:09 +0300 Subject: [PATCH 40/70] Update from code review Co-authored-by: Giovanni Pizzi --- optimade.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/optimade.rst b/optimade.rst index abf9fded6..2bf059c58 100644 --- a/optimade.rst +++ b/optimade.rst @@ -1383,7 +1383,7 @@ One additional query parameter :query-param:`property_slices` MUST be handled by The start value specifies the first index in that dimension for which values should be returned (which is 0-based and inclusive). The default is :val:`0`. The stop value specifies the last index for which values should be returned (inclusive). - The default is the last index of the array along the specified dimension. + The default is :val:`null`, which represents the last index of the array along the specified dimension. The step value specifies the step size in that dimension. The default is :val:`1`. From 3f4f3a3b7c31548530a8c4a2034eef8daf396aff Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Mon, 15 Sep 2025 18:24:17 +0300 Subject: [PATCH 41/70] Changes from code review --- optimade.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/optimade.rst b/optimade.rst index 2bf059c58..62042e0ac 100644 --- a/optimade.rst +++ b/optimade.rst @@ -1388,6 +1388,7 @@ One additional query parameter :query-param:`property_slices` MUST be handled by The default is :val:`1`. An empty value of the :query-param:`property_slices` query parameter MUST be interpreted as equivalent to the query parameter not being included in the request. +As a consequence, the server will not slice any properties. Requirements and conventions for the response when this query parameter is used are described in `Slices of array properties`_. From 589ab543c37816d6df95ac27b46b83bf7c2d13f6 Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Mon, 15 Sep 2025 18:25:06 +0300 Subject: [PATCH 42/70] Update from code review --- optimade.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/optimade.rst b/optimade.rst index 62042e0ac..ac2411fad 100644 --- a/optimade.rst +++ b/optimade.rst @@ -1394,7 +1394,7 @@ As a consequence, the server will not slice any properties. Example: - - :query-url:`http://optimade.example.com/v1/trajectories/id_12345?response_fields=frame_cartesian_site_positions&property_ranges=dim_frames::999:10,dim_sites:30:70:` + - :query-url:`http://optimade.example.com/v1/trajectories/id_12345?response_fields=frame_cartesian_site_positions&property_slices=dim_frames::999:10,dim_sites:30:70:` This query URL requests items from the array :field:`frame_cartesian_site_positions` only for the 31st to 71st sites (i.e., with indexes 30 through 70 inclusive) for 1 out of every 10 frames of the first 1000 frames (i.e., taking steps of 10 over indexes 0 through 999 inclusive, which requests the frames with indexes 0, 10, 20, 30, ..., 990) of a trajectory with ID :val:`id_12345`. From 8ddf2604e7423de0a7bd0f469dac4a6df0f3358b Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Mon, 15 Sep 2025 18:29:05 +0300 Subject: [PATCH 43/70] Changes from code review --- optimade.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/optimade.rst b/optimade.rst index ac2411fad..6a8dfc077 100644 --- a/optimade.rst +++ b/optimade.rst @@ -2375,7 +2375,9 @@ A Property Definition MUST be composed according to the combination of the requi Dimension names defined by database or definition providers MUST be prefixed by the corresponding database or namespace prefix, which SHOULD then be immediately followed by ``dim_``, e.g., ``_exmpl_dim_particles``. If, within one entry, two or more array axes in one or more properties share the same dimension :field:`name`, those represent the same dimension. For example, let us consider the property :property:`frame_cartesian_site_positions` of the trajectory entry, where the first dimension name is :val:`dim_frames`. - If there is another one-dimensional (i.e., with a single axis) array property :property:`_exmpl_energy` of the same trajectory entry that specifies in its property definition the same dimension name :val:`dim_frames` for its axis, then the values of :property:`_exmpl_energy` and of :property:`frame_cartesian_site_positions` at index *i* pertain to the same frame. + Let the trajectory entry in this example have another, one-dimensional, array property :property:`_exmpl_energy`, which in its property definition specifies *the same name*, :val:`dim_frames`, as the name of the axis corresponding to its single dimension. + The joint dimension name means the values of :property:`_exmpl_energy` and of :property:`frame_cartesian_site_positions` at index *i* pertain to the same frame. + If slicing is used to request only parts of the data along the :val:`dim_frames` dimension, that is a request to slice both the properties according to the specified slice. - :field:`sizes`: List of Integers or :val:`null`. A list of fixed length requirements on the underlying one or multi-dimensionsional data represented as mutiple levels of lists. From f0f120a731b74f1cde580d1aa3dac43a69cc1723 Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Mon, 15 Sep 2025 18:44:01 +0300 Subject: [PATCH 44/70] Update from code review --- optimade.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/optimade.rst b/optimade.rst index 6a8dfc077..b2198ae75 100644 --- a/optimade.rst +++ b/optimade.rst @@ -1396,7 +1396,9 @@ As a consequence, the server will not slice any properties. - :query-url:`http://optimade.example.com/v1/trajectories/id_12345?response_fields=frame_cartesian_site_positions&property_slices=dim_frames::999:10,dim_sites:30:70:` - This query URL requests items from the array :field:`frame_cartesian_site_positions` only for the 31st to 71st sites (i.e., with indexes 30 through 70 inclusive) for 1 out of every 10 frames of the first 1000 frames (i.e., taking steps of 10 over indexes 0 through 999 inclusive, which requests the frames with indexes 0, 10, 20, 30, ..., 990) of a trajectory with ID :val:`id_12345`. + This query URL requests items from the trajectory with ID :val:`id_12345`. + It requests items from the array :field:`frame_cartesian_site_positions` for this trajectory. + The items that are requested are for only the 31st to 71st sites (i.e., with indexes 30 through 70 inclusive) for 1 out of every 10 frames of the first 1000 frames (i.e., taking steps of 10 over indexes 0 through 999 inclusive, which requests the frames with indexes 0, 10, 20, 30, ..., 990). Single Entry JSON Response Schema ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From aa644fc8a1c2a5021d579c575a32632ee9b5df6b Mon Sep 17 00:00:00 2001 From: Nathan Daelman <107392603+ndaelman-hu@users.noreply.github.com> Date: Tue, 16 Sep 2025 15:26:02 +0200 Subject: [PATCH 45/70] Apply suggestions from code review Add verbosity to slice metadata response --- optimade.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/optimade.rst b/optimade.rst index 2939e2092..38d28b13e 100644 --- a/optimade.rst +++ b/optimade.rst @@ -623,7 +623,7 @@ When the client request includes the query parameter :query-param:`property_slic The field :field:`array_axes` is defined as follows: - :field:`array_axes`: List of Dictionary. - A list of dictionaries which provide information related to the axes of an array property, including sliceable axes. + A list of dictionaries which provide descriptive information related to the axes of an array property, including sliceable axes. Each item, in order, represents the array axis as declared in the corresponding property definition. Each item MUST be a dictionary with the following REQUIRED field: @@ -634,7 +634,7 @@ The field :field:`array_axes` is defined as follows: If the request specifies the :query-param:`property_slices` query parameter for any of the array axes of this array property the following key MUST be present. However, if that query parameter is not specified, the key MUST either be omitted or set equal to :val:`null`: - :field:`requested_slice`: Dictionary. - A field that describes the requested slice that was provided via the query parameter :query-param:`property_slices`. + A metadata field that describes the requested slice that was provided via the query parameter :query-param:`property_slices`. The subfields MUST reflect the values provided via the :query-param:`property_slices`. The implementation MUST preserve the values as given in the query parameter, including the distinction between specific values and default values even when they are equivalent (see example below). It MAY contain the following subfields that are defined according to the specification of a `slice object`_. @@ -675,7 +675,7 @@ The field :field:`array_axes` is defined as follows: If the field is omitted or :val:`null`, it means the same thing as :val:`false`. - :field:`available_slice`: Dictionary or :val:`null`. - This field describes a `slice object`_. + This metadata field describes a `slice object`_. By including this field, the server certifies that there MUST only be :val:`null` values outside this slice. Inside the slice there MAY be any combination of null and non-null values. If not provided, or equal to :val:`null` or an empty dictionary, the client cannot make any assumptions about what part of the array contains :val:`null` values. From ea15d1fcb2558ee36a5c1220b5d738c853d1e42f Mon Sep 17 00:00:00 2001 From: Gian-Marco Rignanese Date: Wed, 17 Sep 2025 14:08:38 +0300 Subject: [PATCH 46/70] Update optimade.rst Co-authored-by: Nathan Daelman <107392603+ndaelman-hu@users.noreply.github.com> --- optimade.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/optimade.rst b/optimade.rst index 38d28b13e..360c62f2f 100644 --- a/optimade.rst +++ b/optimade.rst @@ -684,7 +684,7 @@ The field :field:`array_axes` is defined as follows: - ``{"start": 3, "stop": 7, "step": 2}`` means the server certifies that values at indexes 0, 1, 2, 4, 6 and any index from 8 to the end of the array are :val:`null`. -Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format for a request to the trajectory endpoint with the query parameter :query-param:`property_slices=dim_frames:3:37:5` and :query-param:`response_fields=frame_cartesian_site_positions,_exmpl_temperature` where the trajectory consists of 432934 frames (with indexes 0 to 432933) and where the :field:`frame_cartesian_site_positions` contains 7 sites. +Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format for a request to the trajectory endpoint with the query parameter :query-param:`property_slices=dim_frames[3:37:5]` and :query-param:`response_fields=frame_cartesian_site_positions,_exmpl_temperature` where the trajectory consists of 432934 frames (with indexes 0 to 432933) and where the :field:`frame_cartesian_site_positions` contains 7 sites. Furthermore, the metadata subfield :field:`available_slice` in :field:`array_axes` in :field:`_exmpl_temperature` certifies that all values of the data field :field:`_exmpl_temperature` are :val:`null` except at indexes 1000, 1030, 1060, ..., 4000, where values can be either numeric or :val:`null`). .. code:: jsonc From 21ca65eb07cf208c3869491de9ac04fdc1979d8c Mon Sep 17 00:00:00 2001 From: Gian-Marco Rignanese Date: Wed, 17 Sep 2025 14:22:55 +0300 Subject: [PATCH 47/70] Replace "array" with "list" following the discussion at the OPTIMADE workshop in Vilnius --- optimade.rst | 51 +++++++++++++++++++++++++-------------------------- 1 file changed, 25 insertions(+), 26 deletions(-) diff --git a/optimade.rst b/optimade.rst index 360c62f2f..bb45dcb67 100644 --- a/optimade.rst +++ b/optimade.rst @@ -219,8 +219,7 @@ representation in all contexts. They are as follows: - **list**: an ordered collection of items, where all items are of the same type, unless they are unknown. A list can be empty, i.e., contain no items. Multidimensional collections of items are represented as nested lists. - The specification uses **array** as a more general term for structures of nested lists representing single or multidimensional data, and the term **array axes** for the levels of nesting. - Note that arrays are represented using lists and not as a separate data type. + The term **list axes** refers to the levels of nesting. - **dictionary**: a collection of **key**-**value** pairs, where **keys** are pre-determined strings, i.e., for the same entry property the **keys** remain the same among different entries whereas the **values** change. The **values** of a dictionary can be any basic type, list, dictionary, or unknown. @@ -541,7 +540,7 @@ Database providers are allowed to define their own metadata properties in :field For example, the metadata property definition of the field :field:`_exmpl_example_field` MUST NOT define a metadata field named, e.g., :field:`accuracy`; the field rather needs to be named, e.g., :field:`_exmpl_accuracy`. The reason for this limitation is to avoid name collisions with metadata fields defined by the OPTIMADE standard in the future that apply also to database-specific data fields. -Implementation of the :field:`meta` field is OPTIONAL, unless the server implements slicing, in which case it is MANDATORY (see `Slices of array properties`_). +Implementation of the :field:`meta` field is OPTIONAL, unless the server implements slicing, in which case it is MANDATORY (see `Slices of list properties`_). When an implementation supports the :field:`property_metadata` field, it SHOULD include metadata fields for all properties which have metadata and are present in the data part of the response. If the client includes the string ``property_metadata`` in the query parameter :query-param:`response_fields`, the server MUST include metadata fields for all properties which have metadata. Furthermore, if the server returns metadata for a property, it must be included in its entirety, i.e., including all non-null fields. @@ -608,30 +607,30 @@ Example of the corresponding metadata property definition contained in the field } // ... -Slices of array properties +Slices of list properties -------------------------- -The OPTIMADE standard defines a way for a client to request only a subset of the items of an array, referred to as a slice. -The protocol for this functionality specifies how a server MAY support slicing arrays independently per entry, per array, and per array axis. +The OPTIMADE standard defines a way for a client to request only a subset of the items of a list, referred to as a slice. +The protocol for this functionality specifies how a server MAY support slicing lists independently per list axis. This functionality is separate from (but compatible with) the protocol described in `Transmission of large property values`_. The protocol for large property values is used by the server implementation to transmit a set of items that it deems too large to provide inside the normal OPTIMADE response. -Slices, on the other hand, are used for a client to request a subset of any size of the items of an array, which can possibly (but not necessarily) result in such a large amount of values that the protocol for large property values is required to transmit them. +Slices, on the other hand, are used for a client to request a subset of any size of the items of a list, which can possibly (but not necessarily) result in such a large amount of values that the protocol for large property values is required to transmit them. The main mechanism is provided through the query parameter :query-param:`property_slices` defined in section `Single Entry URL Query Parameters`_. -Information relating to the ability of the server to handle this query parameter and the relevant ranges of indexes is provided using the metadata property field :field:`array_axes` (see `Metadata properties`_). -When the client request includes the query parameter :query-param:`property_slices`, the server MUST provide metadata for all properties for which including the subfield :field:`requested_slice` of the :field:`array_axes` is MANDATORY (see below). -The field :field:`array_axes` is defined as follows: +Information relating to the ability of the server to handle this query parameter and the relevant ranges of indexes is provided using the metadata property field :field:`list_axes` (see `Metadata properties`_). +When the client request includes the query parameter :query-param:`property_slices`, the server MUST provide metadata for all properties for which including the subfield :field:`requested_slice` of the :field:`list_axes` is MANDATORY (see below). +The field :field:`list_axes` is defined as follows: -- :field:`array_axes`: List of Dictionary. - A list of dictionaries which provide descriptive information related to the axes of an array property, including sliceable axes. - Each item, in order, represents the array axis as declared in the corresponding property definition. +- :field:`list_axes`: List of Dictionary. + A list of dictionaries which provide descriptive information related to the axes of a list property, including sliceable axes. + Each item, in order, represents the list axis as declared in the corresponding property definition. Each item MUST be a dictionary with the following REQUIRED field: - :field:`dimension_name`: String. - The dimension name of the corresponding array axis. + The dimension name of the corresponding list axis. - If the request specifies the :query-param:`property_slices` query parameter for any of the array axes of this array property the following key MUST be present. However, if that query parameter is not specified, the key MUST either be omitted or set equal to :val:`null`: + If the request specifies the :query-param:`property_slices` query parameter for any of the list axes of this list property the following key MUST be present. However, if that query parameter is not specified, the key MUST either be omitted or set equal to :val:`null`: - :field:`requested_slice`: Dictionary. A metadata field that describes the requested slice that was provided via the query parameter :query-param:`property_slices`. @@ -665,7 +664,7 @@ The field :field:`array_axes` is defined as follows: The dictionary MAY contain the following fields: - :field:`length`: Integer. - The length of this array axis which MUST be the same as the length of the declared dimension for this axis in the corresponding property definition. + The length of this list axis which MUST be the same as the length of the declared dimension for this axis in the corresponding property definition. Note that the length of a dimension can be different for different entries if the length is not explicitly declared by the property definition. For example, the number of frames, i.e., the length of the dimension named ``dim_frames``, is generally different for different trajectories. @@ -678,11 +677,11 @@ The field :field:`array_axes` is defined as follows: This metadata field describes a `slice object`_. By including this field, the server certifies that there MUST only be :val:`null` values outside this slice. Inside the slice there MAY be any combination of null and non-null values. - If not provided, or equal to :val:`null` or an empty dictionary, the client cannot make any assumptions about what part of the array contains :val:`null` values. + If not provided, or equal to :val:`null` or an empty dictionary, the client cannot make any assumptions about what part of the list contains :val:`null` values. Examples: - - ``{"start": 3, "stop": 7, "step": 2}`` means the server certifies that values at indexes 0, 1, 2, 4, 6 and any index from 8 to the end of the array are :val:`null`. + - ``{"start": 3, "stop": 7, "step": 2}`` means the server certifies that values at indexes 0, 1, 2, 4, 6 and any index from 8 to the end of the list are :val:`null`. Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format for a request to the trajectory endpoint with the query parameter :query-param:`property_slices=dim_frames[3:37:5]` and :query-param:`response_fields=frame_cartesian_site_positions,_exmpl_temperature` where the trajectory consists of 432934 frames (with indexes 0 to 432933) and where the :field:`frame_cartesian_site_positions` contains 7 sites. Furthermore, the metadata subfield :field:`available_slice` in :field:`array_axes` in :field:`_exmpl_temperature` certifies that all values of the data field :field:`_exmpl_temperature` are :val:`null` except at indexes 1000, 1030, 1060, ..., 4000, where values can be either numeric or :val:`null`). @@ -701,7 +700,7 @@ Furthermore, the metadata subfield :field:`available_slice` in :field:`array_axe "meta": { "property_metadata": { "frame_cartesian_site_positions": { - "array_axes": [ + "list_axes": [ { "dimension_name": "dim_frames", "requested_slice": { @@ -734,7 +733,7 @@ Furthermore, the metadata subfield :field:`available_slice` in :field:`array_axe ] }, "_exmpl_temperature": { - "array_axes": [ + "list_axes": [ { "dimension_name": "dim_frames", "requested_slice": { @@ -1373,7 +1372,7 @@ The meaning of these URL query parameters are as defined above in section `Entry One additional query parameter :query-param:`property_slices` MUST be handled by the API implementation either as defined below or by returning the error :http-error:`501 Not Implemented`: -- **property\_slices**: A number of slice specifications to request only parts of array properties for the functionality described in `Slices of array properties`_. +- **property\_slices**: A number of slice specifications to request only parts of list properties for the functionality described in `Slices of list properties`_. The query parameter contains a comma-separated (",", ASCII symbol 44 dec) list of slice specifications. Each slice specification consists of a dimension name followed by the left square bracket character ("[", ASCII symbol 91 dec), a slice, and the right square bracket character ("]", ASCII symbol 93 dec). @@ -1383,21 +1382,21 @@ One additional query parameter :query-param:`property_slices` MUST be handled by The start value specifies the first index in that dimension for which values should be returned (which is 0-based and inclusive). The default is :val:`0`. The stop value specifies the last index for which values should be returned (inclusive). - The default is :val:`null`, which represents the last index of the array along the specified dimension. + The default is :val:`null`, which represents the last index of the list along the specified dimension. The step value specifies the step size in that dimension. The default is :val:`1`. An empty value of the :query-param:`property_slices` query parameter MUST be interpreted as equivalent to the query parameter not being included in the request. As a consequence, the server will not slice any properties. - Requirements and conventions for the response when this query parameter is used are described in `Slices of array properties`_. + Requirements and conventions for the response when this query parameter is used are described in `Slices of list properties`_. Example: - :query-url:`http://optimade.example.com/v1/trajectories/id_12345?response_fields=frame_cartesian_site_positions&property_slices=dim_frames::999:10,dim_sites:30:70:` This query URL requests items from the trajectory with ID :val:`id_12345`. - It requests items from the array :field:`frame_cartesian_site_positions` for this trajectory. + It requests items from the list :field:`frame_cartesian_site_positions` for this trajectory. The items that are requested are for only the 31st to 71st sites (i.e., with indexes 30 through 70 inclusive) for 1 out of every 10 frames of the first 1000 frames (i.e., taking steps of 10 over indexes 0 through 999 inclusive, which requests the frames with indexes 0, 10, 20, 30, ..., 990). Single Entry JSON Response Schema @@ -2377,9 +2376,9 @@ A Property Definition MUST be composed according to the combination of the requi The order is such that the first name applies to the outermost list, the next name to the lists embedded in that list, etc. Dimension names defined by the OPTIMADE standard are prefixed by ``dim_``. Dimension names defined by database or definition providers MUST be prefixed by the corresponding database or namespace prefix, which SHOULD then be immediately followed by ``dim_``, e.g., ``_exmpl_dim_particles``. - If, within one entry, two or more array axes in one or more properties share the same dimension :field:`name`, those represent the same dimension. + If, within one entry, two or more list axes in one or more properties share the same dimension :field:`name`, those represent the same dimension. For example, let us consider the property :property:`frame_cartesian_site_positions` of the trajectory entry, where the first dimension name is :val:`dim_frames`. - Let the trajectory entry in this example have another, one-dimensional, array property :property:`_exmpl_energy`, which in its property definition specifies *the same name*, :val:`dim_frames`, as the name of the axis corresponding to its single dimension. + Let the trajectory entry in this example have another, one-dimensional, list property :property:`_exmpl_energy`, which in its property definition specifies *the same name*, :val:`dim_frames`, as the name of the axis corresponding to its single dimension. The joint dimension name means the values of :property:`_exmpl_energy` and of :property:`frame_cartesian_site_positions` at index *i* pertain to the same frame. If slicing is used to request only parts of the data along the :val:`dim_frames` dimension, that is a request to slice both the properties according to the specified slice. From 2f14be94ba040f04b4978b3a792c04071e452886 Mon Sep 17 00:00:00 2001 From: Gian-Marco Rignanese Date: Wed, 17 Sep 2025 14:23:37 +0300 Subject: [PATCH 48/70] Update optimade.rst --- optimade.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/optimade.rst b/optimade.rst index bb45dcb67..ebd292458 100644 --- a/optimade.rst +++ b/optimade.rst @@ -684,7 +684,7 @@ The field :field:`list_axes` is defined as follows: - ``{"start": 3, "stop": 7, "step": 2}`` means the server certifies that values at indexes 0, 1, 2, 4, 6 and any index from 8 to the end of the list are :val:`null`. Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format for a request to the trajectory endpoint with the query parameter :query-param:`property_slices=dim_frames[3:37:5]` and :query-param:`response_fields=frame_cartesian_site_positions,_exmpl_temperature` where the trajectory consists of 432934 frames (with indexes 0 to 432933) and where the :field:`frame_cartesian_site_positions` contains 7 sites. -Furthermore, the metadata subfield :field:`available_slice` in :field:`array_axes` in :field:`_exmpl_temperature` certifies that all values of the data field :field:`_exmpl_temperature` are :val:`null` except at indexes 1000, 1030, 1060, ..., 4000, where values can be either numeric or :val:`null`). +Furthermore, the metadata subfield :field:`available_slice` in :field:`list_axes` in :field:`_exmpl_temperature` certifies that all values of the data field :field:`_exmpl_temperature` are :val:`null` except at indexes 1000, 1030, 1060, ..., 4000, where values can be either numeric or :val:`null`). .. code:: jsonc From fd3d2c14f80f11c0714eff8a330660bfd62ea31b Mon Sep 17 00:00:00 2001 From: Giovanni Pizzi Date: Wed, 17 Sep 2025 14:58:34 +0300 Subject: [PATCH 49/70] Fixing RST issues and remaining array/list ambiguity --- optimade.rst | 72 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 42 insertions(+), 30 deletions(-) diff --git a/optimade.rst b/optimade.rst index ebd292458..80440bc8d 100644 --- a/optimade.rst +++ b/optimade.rst @@ -649,17 +649,29 @@ The field :field:`list_axes` is defined as follows: (Consequently, the dictionary MAY be empty if all fields take the default values.) Examples: - Query Type | Query Formulation | Valid Representations - ----------------|---------------------------|------------------------------ - fully formed | :query-string:`property_slices=dim_frames:3:5:2` | ``{"start": 3, "stop": 5, "step": 2}`` - empty query with start | :query-string:`property_slices=dim_frames:0::` | ``{"start": 0}`` - empty query | :query-string:`property_slices=dim_frames:::` | ``{}`` - | | ``{"start": null, "stop": null, "step": null}`` - | | ``{"start": null}`` - | | ``{"stop": null, "step": null}`` - - Note that any of the representations for the "empty query" are valid. - Even though, semantically, it means the same as the "empty query with start" at index zero, both differ in representation. This is due to the starting index being explicitely specified now. The reason becomes apparent once the starting index takes on a value different from the default. + + +------------------+---------------------------------------------------+----------------------------------------------+ + | Query Type | Query Formulation | Valid Representations | + +==================+===================================================+==============================================+ + | fully formed | :query-string:`property_slices=dim_frames[3:5:2]` | ``{"start": 3, "stop": 5, "step": 2}`` | + +------------------+---------------------------------------------------+----------------------------------------------+ + | empty query with | :query-string:`property_slices=dim_frames[0::]` | ``{"start": 0}`` | + | start | | | + +------------------+---------------------------------------------------+----------------------------------------------+ + | empty query | :query-string:`property_slices=dim_frames[::]` | ``{}`` | + | | | | + | | | ``{"start": null, "stop": null, "step": | + | | | null}`` | + | | | | + | | | ``{"start": null}`` | + | | | | + | | | ``{"stop": null, "step": null}`` | + +------------------+---------------------------------------------------+----------------------------------------------+ + + Note that any of the representations displayed above for the "empty query" are valid. + Semantically, the two examples above "empty query" and the "empty query with start" are equivalent, but differ in representation. + This is due to the starting index being explicitely specified in the "empty query with start" example. + The need for this difference becomes apparent if the starting index takes on a value different from the default. The dictionary MAY contain the following fields: @@ -1387,7 +1399,7 @@ One additional query parameter :query-param:`property_slices` MUST be handled by The default is :val:`1`. An empty value of the :query-param:`property_slices` query parameter MUST be interpreted as equivalent to the query parameter not being included in the request. -As a consequence, the server will not slice any properties. + As a consequence, the server will not slice any properties. Requirements and conventions for the response when this query parameter is used are described in `Slices of list properties`_. @@ -3485,7 +3497,7 @@ fractional\_site\_positions - **Description**: fractional coordinates (positions) of each site in the structure. A site is usually used to describe positions of atoms; what atoms can be encountered at a given site is conveyed by the :property:`species_at_sites` property, and the species themselves are described in the :property:`species` property. Site coordinates MAY be given as `cartesian_site_positions`_, `fractional_site_positions`_, or both. - When symmetry operations given in `space_group_symmetry_operations_xyz`_ are applied, they MUST be applied to coordinates given in the `fractional_site_positions`_ array. + When symmetry operations given in `space_group_symmetry_operations_xyz`_ are applied, they MUST be applied to coordinates given in the `fractional_site_positions`_ list. - **Type**: list of list of floats - **Requirements/Conventions**: @@ -3494,7 +3506,7 @@ fractional\_site\_positions If supported, filters MAY support only a subset of comparison operators. - It MUST be a list of length equal to the number of sites in the structure, where every element is a list of the three fractional coordinates of a site expressed as float values in the fractions of the unit cell vectors given by the `lattice_vectors`_ property. - An entry MAY have multiple sites at the same site position (for a relevant use of this, see e.g., the property `assemblies`_). - - **Note**: Since both `cartesian_site_positions`_ and the `fractional_site_positions`_ always describe the same sites, they MUST always have the same number of elements, equal to the number of elements in the `species_at_sites`_ array. + - **Note**: Since both `cartesian_site_positions`_ and the `fractional_site_positions`_ always describe the same sites, they MUST always have the same number of elements, equal to the number of elements in the `species_at_sites`_ list. - **Examples**: @@ -4457,21 +4469,21 @@ Note: since the below definition references both JSON fields and OPTIMADE proper .. _slice object: -To aid the definition of the format below, we first define a "slice object" to be a JSON object describing slices of arrays. +To aid the definition of the format below, we first define a "slice object" to be a JSON object describing slices of lists. The dictionary has the following OPTIONAL fields: - :field:`"start"`: Integer. The slice starts at the value with the given index (inclusive). - The default is 0, i.e., the value at the start of the array. + The default is 0, i.e., the value at the start of the list. - :field:`"stop"`: Integer. The slice ends at the value with the given index (inclusive). - If omitted, the end of the slice is the last index of the array. + If omitted, the end of the slice is the last index of the list. - :field:`"step"`: Integer. The absolute difference in index between two subsequent values that are included in the slice. The default is 1, i.e., every value in the range indicated by :field:`start` and :field:`stop` is included in the slice. - Hence, a value of 2 denotes a slice of every second value in the array. + Hence, a value of 2 denotes a slice of every second value in the list. -For example, for the array :val:`["a", "b", "c", "d", "e", "f", "g", "h", "i", "j"]` the slice object :val:`{"start": 1, "stop": 7, "step": 3}` refers to the items :val:`["b", "e", "h"]`. +For example, for the list :val:`["a", "b", "c", "d", "e", "f", "g", "h", "i", "j"]` the slice object :val:`{"start": 1, "stop": 7, "step": 3}` refers to the items :val:`["b", "e", "h"]`. Furthermore, we also define the following special markers: @@ -4514,9 +4526,9 @@ The header object MUST contain the keys: The following key is RECOMMENDED in the header object: -- :field:`"returned_ranges"`: Array of Object. - For dense layout, and sparse layout of one dimensional list properties, the array contains a single element which is a `slice object`_ representing the range of data present in the response. - In the specific case of a hierarchy of list properties represented as a sparse multidimensional array, if the field :field:`"returned_ranges"` is given, it MUST contain one slice object per dimension of the multidimensional array, representing slices for each dimension that cover the data given in the response. +- :field:`"returned_ranges"`: Array of object. + For dense layout, and sparse layout of one dimensional list properties, the :field:`"returned_ranges"` array contains a single element which is a `slice object`_ representing the range of data present in the response. + In the specific case of a hierarchy of list properties represented as a sparse multidimensional list, if the field :field:`"returned_ranges"` is given, it MUST contain one slice object per dimension of the multidimensional list, representing slices for each dimension that cover the data given in the response. The header object MAY also contain the keys: @@ -4557,19 +4569,19 @@ The format of data lines of the response (i.e., all lines except the first and t - **Dense layout:** In the dense partial data layout, each data line reproduces one item from the OPTIMADE list property being transmitted in the JSON format. If OPTIMADE list properties are embedded inside the item, they can either be included in full or replaced with a reference-marker. If a list is replaced by a reference marker, the client MAY use the provided URL to obtain the list items. - If the field :field:`"returned_ranges"` is omitted, then the client MUST assume that the data is a continuous range of data from the start of the array up to the number of elements given until reaching the end-of-data-marker or next-marker. + If the field :field:`"returned_ranges"` is omitted, then the client MUST assume that the data is a continuous range of data from the start of the list up to the number of elements given until reaching the end-of-data-marker or next-marker. - **Sparse layout for one-dimensional list:** When the response sparsely communicates items for a one-dimensional OPTIMADE list property, each data line contains a JSON array of the format: - - The first item of the array is the zero-based index of list property item being provided by this line. - - The second item of the array is the list property item located at the indicated index, represented using the same format as each line in the dense layout. + - The first item of the array is the zero-based index of the list property item being provided by this line. + - The second item of the array is the value of the list property item located at the indicated index, represented using the same format as each line in the dense layout. In the same way as for the dense layout, reference-markers are allowed inside the item data for embedded lists that do not fit in the response (see example below). -- **Sparse layout for multidimensional lists:** the server MAY use a specific sparse layout for the case that the OPTIMADE property represents a series of directly hierarchically embedded lists (i.e., a multidimensional sparse array). +- **Sparse layout for multidimensional lists:** the server MAY use a specific sparse layout for the case that the OPTIMADE property represents a series of directly hierarchically embedded lists (i.e., a multidimensional sparse list). In this case, each data line contains a JSON array of the format: - - All array items except the last one are integer zero-based indices of the list property item being provided by this line; these indices refer to the aggregated dimensions in the order of outermost to innermost. - - The last item of the array is the list property item located at the indicated coordinates, represented using the same format as each line in the dense layout. + - All array items except the last one are zero-based integer indices of the list property item being provided by this line; these indices refer to the aggregated dimensions in the order of outermost to innermost. + - The last item of the array is the value of the list property item located at the indicated coordinates, represented using the same format as each line in the dense layout. In the same way as for the dense layout, reference-markers are allowed inside the item data for embedded lists that do not fit in the response (see example below). If the final line of the response is a next-marker, the client MAY continue fetching the data for the property by retrieving another partial data response from the provided URL. @@ -4583,7 +4595,7 @@ In that case the values shall remain as assigned, i.e., they are not overwritten Examples ~~~~~~~~ -Below follows an example of a dense response for a partial array data of integer values. +Below follows an example of a dense response for a partial list data of integer values. The request returns the first three items and provides the next-marker link to continue fetching data: .. code:: jsonl @@ -4594,7 +4606,7 @@ The request returns the first three items and provides the next-marker link to c -12.6 ["PARTIAL-DATA-NEXT", ["https://example.db.org/value4"]] -Below follows an example of a dense response for a list property as a partial array of multidimensional array values. +Below follows an example of a dense response for a list property as a partial list of multidimensional list values. The item with index 10 in the original list is provided explicitly in the response and is the first one provided in the response since start=10. The item with index 12 in the list, the second data item provided since start=10 and step=2, is not included only referenced. The third provided item (index 14 in the original list) is only partially returned: it is a list of three items, the first and last are explicitly provided, the second one is only referenced. From 116c3ad7abc9903e7d9b50c624f1125bff332614 Mon Sep 17 00:00:00 2001 From: Giovanni Pizzi Date: Wed, 17 Sep 2025 15:04:14 +0300 Subject: [PATCH 50/70] Fix spurious old syntax for slices --- optimade.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/optimade.rst b/optimade.rst index 80440bc8d..c099a8c4b 100644 --- a/optimade.rst +++ b/optimade.rst @@ -1405,7 +1405,7 @@ One additional query parameter :query-param:`property_slices` MUST be handled by Example: - - :query-url:`http://optimade.example.com/v1/trajectories/id_12345?response_fields=frame_cartesian_site_positions&property_slices=dim_frames::999:10,dim_sites:30:70:` + - :query-url:`http://optimade.example.com/v1/trajectories/id_12345?response_fields=frame_cartesian_site_positions&property_slices=dim_frames[:999:10],dim_sites[30:70:]` This query URL requests items from the trajectory with ID :val:`id_12345`. It requests items from the list :field:`frame_cartesian_site_positions` for this trajectory. From 52a88272ad0376d493634162e7b0a1d3893a3dac Mon Sep 17 00:00:00 2001 From: Giovanni Pizzi Date: Wed, 17 Sep 2025 15:27:07 +0300 Subject: [PATCH 51/70] Adding explicit discussion of valid lists --- optimade.rst | 31 ++++++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-) diff --git a/optimade.rst b/optimade.rst index c099a8c4b..6730eda6c 100644 --- a/optimade.rst +++ b/optimade.rst @@ -218,8 +218,37 @@ representation in all contexts. They are as follows: - Basic types: **string**, **integer**, **float**, **boolean**, **timestamp**. - **list**: an ordered collection of items, where all items are of the same type, unless they are unknown. A list can be empty, i.e., contain no items. - Multidimensional collections of items are represented as nested lists. + Multidimensional collections of items are represented as nested lists, and are still referred to simply as "lists" in this document. + + We note that the term "array" is often used in programming languages to refer to similar data structures. + However, the term "array" is avoided in this specification to prevent confusion with the meaning of the term "array" in the JSON specification. + The term **list axes** refers to the levels of nesting. + We note that it is valid for a list to have elements of different lengths along a given axis, i.e., the list does not need to be rectangular. + However, the dimensionality of a list, defined as the number of axes (i.e., the number of levels of nesting) must be the same for all entries of a given entry type. + + Valid examples: + + - ``[1.0, 2.0, 3.0]`` is a one-dimensional list of three floats (only one axis). + + - ``[]`` is a one-dimensional empty list. + + - ``[1.0, null, 3.0]`` is a one-dimensional list of floats. The ``null`` value is valid as it represents an unknown value. + + - ``[[1, 2], [3, 4], [5, 6]]`` is a two-dimensional list of three lists, each containing two integers. The first list axis has length 3 and the second list axis has length 2. + + - ``[[1, 2], [3, 4, 5], [6]]`` is a two-dimensional list of three lists, each containing a different number of integers. It can be interpreted as a 3x3 (or 3x4, 4x4, ...) list, where the missing elements have implicitly the value :val:`null`. + + Invalid examples: + + - ``[3.0, "string"]`` is invalid since the two elements have different types. + + Further invalid examples (since they cannot be described via an OPTIMADE property specification): + + - ``[[1.0, 2.0], ["string", "string"]]`` is invalid since, while each of the two sublists is a valid list ([1.0, 2.0] is a list of integers, and ["string", "string"] is a list of strings), the two sublists have different types. + + - ``[[1.0, 2.0], [3.0, [4.0]], [5.0]]`` is invalid since the second sublist contains a nested list, whereas the other two sublists do not, thus not allowing a consistent definition of the dimensions and axes of the list. + - **dictionary**: a collection of **key**-**value** pairs, where **keys** are pre-determined strings, i.e., for the same entry property the **keys** remain the same among different entries whereas the **values** change. The **values** of a dictionary can be any basic type, list, dictionary, or unknown. From ed5b790e10bb413fe2fade2cd31991fe99f7f322 Mon Sep 17 00:00:00 2001 From: Giovanni Pizzi Date: Wed, 17 Sep 2025 18:18:02 +0300 Subject: [PATCH 52/70] Several textual clarifications --- optimade.rst | 25 ++++++++++++++++++------- 1 file changed, 18 insertions(+), 7 deletions(-) diff --git a/optimade.rst b/optimade.rst index 6730eda6c..440a9b25d 100644 --- a/optimade.rst +++ b/optimade.rst @@ -569,7 +569,11 @@ Database providers are allowed to define their own metadata properties in :field For example, the metadata property definition of the field :field:`_exmpl_example_field` MUST NOT define a metadata field named, e.g., :field:`accuracy`; the field rather needs to be named, e.g., :field:`_exmpl_accuracy`. The reason for this limitation is to avoid name collisions with metadata fields defined by the OPTIMADE standard in the future that apply also to database-specific data fields. -Implementation of the :field:`meta` field is OPTIONAL, unless the server implements slicing, in which case it is MANDATORY (see `Slices of list properties`_). +Implementation of the :field:`meta` field is OPTIONAL. +However, it becomes MANDATORY when some of subfields included in the :field:`meta` field are MANDATORY according to other parts of the OPTIMADE specification. +This happens, for instance, if a server supports slicing and receives a request containing the :query-param:`property_slices` query parameter. +In this case, the inclusion of the :field:`requested_slices` subfield of the :field:`list_axes` field is MANDATORY, see `Slices of list properties`_. + When an implementation supports the :field:`property_metadata` field, it SHOULD include metadata fields for all properties which have metadata and are present in the data part of the response. If the client includes the string ``property_metadata`` in the query parameter :query-param:`response_fields`, the server MUST include metadata fields for all properties which have metadata. Furthermore, if the server returns metadata for a property, it must be included in its entirety, i.e., including all non-null fields. @@ -637,7 +641,7 @@ Example of the corresponding metadata property definition contained in the field // ... Slices of list properties --------------------------- +------------------------- The OPTIMADE standard defines a way for a client to request only a subset of the items of a list, referred to as a slice. The protocol for this functionality specifies how a server MAY support slicing lists independently per list axis. @@ -664,6 +668,7 @@ The field :field:`list_axes` is defined as follows: - :field:`requested_slice`: Dictionary. A metadata field that describes the requested slice that was provided via the query parameter :query-param:`property_slices`. The subfields MUST reflect the values provided via the :query-param:`property_slices`. + In particular, the field is MANDATORY if the client request includes the :query-param:`property_slices` query parameter for this list axis. The implementation MUST preserve the values as given in the query parameter, including the distinction between specific values and default values even when they are equivalent (see example below). It MAY contain the following subfields that are defined according to the specification of a `slice object`_. @@ -725,7 +730,7 @@ The field :field:`list_axes` is defined as follows: - ``{"start": 3, "stop": 7, "step": 2}`` means the server certifies that values at indexes 0, 1, 2, 4, 6 and any index from 8 to the end of the list are :val:`null`. Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format for a request to the trajectory endpoint with the query parameter :query-param:`property_slices=dim_frames[3:37:5]` and :query-param:`response_fields=frame_cartesian_site_positions,_exmpl_temperature` where the trajectory consists of 432934 frames (with indexes 0 to 432933) and where the :field:`frame_cartesian_site_positions` contains 7 sites. -Furthermore, the metadata subfield :field:`available_slice` in :field:`list_axes` in :field:`_exmpl_temperature` certifies that all values of the data field :field:`_exmpl_temperature` are :val:`null` except at indexes 1000, 1030, 1060, ..., 4000, where values can be either numeric or :val:`null`). +Furthermore, the metadata subfield :field:`available_slice` in :field:`list_axes` in :field:`_exmpl_temperature` certifies that all values of the data field :field:`_exmpl_temperature` are :val:`null` except at indexes 1000, 1030, 1060, ..., 4000, where values can be either numeric or :val:`null`. .. code:: jsonc @@ -2305,7 +2310,7 @@ Optional filter features ~~~~~~~~~~~~~~~~~~~~~~~~ Some features of the filtering language are marked OPTIONAL. -An implementation that encounters an OPTIONAL feature that it does not support MUST respond with error ``501 Not Implemented`` with an explanation of which OPTIONAL construct the error refers to. +An implementation that encounters an OPTIONAL feature that it does not support MUST respond with error :http-error:`501 Not Implemented` with an explanation of which OPTIONAL construct the error refers to. Property Definitions ==================== @@ -4502,13 +4507,18 @@ To aid the definition of the format below, we first define a "slice object" to b The dictionary has the following OPTIONAL fields: - :field:`"start"`: Integer. - The slice starts at the value with the given index (inclusive). + The slice starts at the value with the given index (inclusive, meaning the item at the start index is included). + It MUST be a non-negative integer. The default is 0, i.e., the value at the start of the list. - :field:`"stop"`: Integer. - The slice ends at the value with the given index (inclusive). + The slice ends at the value with the given index (inclusive, meaning the item at the stop index is included). + It MUST be a non-negative integer. If omitted, the end of the slice is the last index of the list. + + Note: The fact the stop index is inclusive differs from common slicing conventions in many programming languages, where the stop index is typically exclusive. Care should be taken to avoid off-by-one errors. - :field:`"step"`: Integer. The absolute difference in index between two subsequent values that are included in the slice. + It MUST be a non-zero positive integer. The default is 1, i.e., every value in the range indicated by :field:`start` and :field:`stop` is included in the slice. Hence, a value of 2 denotes a slice of every second value in the list. @@ -4625,7 +4635,8 @@ Examples ~~~~~~~~ Below follows an example of a dense response for a partial list data of integer values. -The request returns the first three items and provides the next-marker link to continue fetching data: +The request returns a slice of data starting from index 10 up to and including index 20, with a step of 2. +This response provides the first three values of that slice (corresponding to indices 10, 12, and 14) and includes a next-marker link to fetch subsequent data: .. code:: jsonl From 611174d1d9a0c2342e6f50227b3ad2d82c1b0e6e Mon Sep 17 00:00:00 2001 From: Giovanni Pizzi Date: Wed, 17 Sep 2025 23:50:51 +0300 Subject: [PATCH 53/70] Update optimade.rst MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Saulius Gražulis --- optimade.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/optimade.rst b/optimade.rst index 440a9b25d..f3e2137df 100644 --- a/optimade.rst +++ b/optimade.rst @@ -576,7 +576,7 @@ In this case, the inclusion of the :field:`requested_slices` subfield of the :fi When an implementation supports the :field:`property_metadata` field, it SHOULD include metadata fields for all properties which have metadata and are present in the data part of the response. If the client includes the string ``property_metadata`` in the query parameter :query-param:`response_fields`, the server MUST include metadata fields for all properties which have metadata. -Furthermore, if the server returns metadata for a property, it must be included in its entirety, i.e., including all non-null fields. +Furthermore, if the server returns metadata for a property, it MUST be included in its entirety, i.e., including all non-null fields. Example of a response in the JSON response format with two structure entries that each include a metadata property for the attribute field :field:`elements_ratios` and the database-specific per entry metadata field :field:`_exmpl_originates_from_project`: From dd03e2dcd14ffcb50d1a93aa4c8c7202536565b7 Mon Sep 17 00:00:00 2001 From: Giovanni Pizzi Date: Wed, 17 Sep 2025 23:51:29 +0300 Subject: [PATCH 54/70] Update optimade.rst MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Saulius Gražulis --- optimade.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/optimade.rst b/optimade.rst index f3e2137df..5f1cfa4dc 100644 --- a/optimade.rst +++ b/optimade.rst @@ -575,7 +575,7 @@ This happens, for instance, if a server supports slicing and receives a request In this case, the inclusion of the :field:`requested_slices` subfield of the :field:`list_axes` field is MANDATORY, see `Slices of list properties`_. When an implementation supports the :field:`property_metadata` field, it SHOULD include metadata fields for all properties which have metadata and are present in the data part of the response. -If the client includes the string ``property_metadata`` in the query parameter :query-param:`response_fields`, the server MUST include metadata fields for all properties which have metadata. +If the client includes the string ``property_metadata`` in the query parameter :query-param:`response_fields`, the server MUST include metadata fields for all properties which have metadata and are part of the response fields. Furthermore, if the server returns metadata for a property, it MUST be included in its entirety, i.e., including all non-null fields. Example of a response in the JSON response format with two structure entries that each include a metadata property for the attribute field :field:`elements_ratios` and the database-specific per entry metadata field :field:`_exmpl_originates_from_project`: From 01a87995ccd1eaaf2c7d698b539c7a7b0a6a16f8 Mon Sep 17 00:00:00 2001 From: Giovanni Pizzi Date: Wed, 17 Sep 2025 23:53:13 +0300 Subject: [PATCH 55/70] Reverting one change after discussion --- optimade.rst | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/optimade.rst b/optimade.rst index 5f1cfa4dc..923577df1 100644 --- a/optimade.rst +++ b/optimade.rst @@ -569,11 +569,7 @@ Database providers are allowed to define their own metadata properties in :field For example, the metadata property definition of the field :field:`_exmpl_example_field` MUST NOT define a metadata field named, e.g., :field:`accuracy`; the field rather needs to be named, e.g., :field:`_exmpl_accuracy`. The reason for this limitation is to avoid name collisions with metadata fields defined by the OPTIMADE standard in the future that apply also to database-specific data fields. -Implementation of the :field:`meta` field is OPTIONAL. -However, it becomes MANDATORY when some of subfields included in the :field:`meta` field are MANDATORY according to other parts of the OPTIMADE specification. -This happens, for instance, if a server supports slicing and receives a request containing the :query-param:`property_slices` query parameter. -In this case, the inclusion of the :field:`requested_slices` subfield of the :field:`list_axes` field is MANDATORY, see `Slices of list properties`_. - +Implementation of the :field:`meta` field is OPTIONAL, unless the server implements slicing, in which case it is MANDATORY (see `Slices of list properties`_). When an implementation supports the :field:`property_metadata` field, it SHOULD include metadata fields for all properties which have metadata and are present in the data part of the response. If the client includes the string ``property_metadata`` in the query parameter :query-param:`response_fields`, the server MUST include metadata fields for all properties which have metadata and are part of the response fields. Furthermore, if the server returns metadata for a property, it MUST be included in its entirety, i.e., including all non-null fields. From 987590d51d349712fd1dc6402f3a787f7ed74da4 Mon Sep 17 00:00:00 2001 From: Giovanni Pizzi Date: Wed, 17 Sep 2025 23:55:59 +0300 Subject: [PATCH 56/70] Renamed property_slices to dimension_slices for better clarity (Slices act on dimension names, not property names) --- optimade.rst | 56 ++++++++++++++++++++++++++-------------------------- 1 file changed, 28 insertions(+), 28 deletions(-) diff --git a/optimade.rst b/optimade.rst index 923577df1..d08d8b1bc 100644 --- a/optimade.rst +++ b/optimade.rst @@ -645,9 +645,9 @@ This functionality is separate from (but compatible with) the protocol described The protocol for large property values is used by the server implementation to transmit a set of items that it deems too large to provide inside the normal OPTIMADE response. Slices, on the other hand, are used for a client to request a subset of any size of the items of a list, which can possibly (but not necessarily) result in such a large amount of values that the protocol for large property values is required to transmit them. -The main mechanism is provided through the query parameter :query-param:`property_slices` defined in section `Single Entry URL Query Parameters`_. +The main mechanism is provided through the query parameter :query-param:`dimension_slices` defined in section `Single Entry URL Query Parameters`_. Information relating to the ability of the server to handle this query parameter and the relevant ranges of indexes is provided using the metadata property field :field:`list_axes` (see `Metadata properties`_). -When the client request includes the query parameter :query-param:`property_slices`, the server MUST provide metadata for all properties for which including the subfield :field:`requested_slice` of the :field:`list_axes` is MANDATORY (see below). +When the client request includes the query parameter :query-param:`dimension_slices`, the server MUST provide metadata for all properties for which including the subfield :field:`requested_slice` of the :field:`list_axes` is MANDATORY (see below). The field :field:`list_axes` is defined as follows: - :field:`list_axes`: List of Dictionary. @@ -659,12 +659,12 @@ The field :field:`list_axes` is defined as follows: - :field:`dimension_name`: String. The dimension name of the corresponding list axis. - If the request specifies the :query-param:`property_slices` query parameter for any of the list axes of this list property the following key MUST be present. However, if that query parameter is not specified, the key MUST either be omitted or set equal to :val:`null`: + If the request specifies the :query-param:`dimension_slices` query parameter for any of the list axes of this list property the following key MUST be present. However, if that query parameter is not specified, the key MUST either be omitted or set equal to :val:`null`: - :field:`requested_slice`: Dictionary. - A metadata field that describes the requested slice that was provided via the query parameter :query-param:`property_slices`. - The subfields MUST reflect the values provided via the :query-param:`property_slices`. - In particular, the field is MANDATORY if the client request includes the :query-param:`property_slices` query parameter for this list axis. + A metadata field that describes the requested slice that was provided via the query parameter :query-param:`dimension_slices`. + The subfields MUST reflect the values provided via the :query-param:`dimension_slices`. + In particular, the field is MANDATORY if the client request includes the :query-param:`dimension_slices` query parameter for this list axis. The implementation MUST preserve the values as given in the query parameter, including the distinction between specific values and default values even when they are equivalent (see example below). It MAY contain the following subfields that are defined according to the specification of a `slice object`_. @@ -680,23 +680,23 @@ The field :field:`list_axes` is defined as follows: Examples: - +------------------+---------------------------------------------------+----------------------------------------------+ - | Query Type | Query Formulation | Valid Representations | - +==================+===================================================+==============================================+ - | fully formed | :query-string:`property_slices=dim_frames[3:5:2]` | ``{"start": 3, "stop": 5, "step": 2}`` | - +------------------+---------------------------------------------------+----------------------------------------------+ - | empty query with | :query-string:`property_slices=dim_frames[0::]` | ``{"start": 0}`` | - | start | | | - +------------------+---------------------------------------------------+----------------------------------------------+ - | empty query | :query-string:`property_slices=dim_frames[::]` | ``{}`` | - | | | | - | | | ``{"start": null, "stop": null, "step": | - | | | null}`` | - | | | | - | | | ``{"start": null}`` | - | | | | - | | | ``{"stop": null, "step": null}`` | - +------------------+---------------------------------------------------+----------------------------------------------+ + +------------------+----------------------------------------------------+----------------------------------------------+ + | Query Type | Query Formulation | Valid Representations | + +==================+====================================================+==============================================+ + | fully formed | :query-string:`dimension_slices=dim_frames[3:5:2]` | ``{"start": 3, "stop": 5, "step": 2}`` | + +------------------+----------------------------------------------------+----------------------------------------------+ + | empty query with | :query-string:`dimension_slices=dim_frames[0::]` | ``{"start": 0}`` | + | start | | | + +------------------+----------------------------------------------------+----------------------------------------------+ + | empty query | :query-string:`dimension_slices=dim_frames[::]` | ``{}`` | + | | | | + | | | ``{"start": null, "stop": null, "step": | + | | | null}`` | + | | | | + | | | ``{"start": null}`` | + | | | | + | | | ``{"stop": null, "step": null}`` | + +------------------+----------------------------------------------------+----------------------------------------------+ Note that any of the representations displayed above for the "empty query" are valid. Semantically, the two examples above "empty query" and the "empty query with start" are equivalent, but differ in representation. @@ -725,7 +725,7 @@ The field :field:`list_axes` is defined as follows: - ``{"start": 3, "stop": 7, "step": 2}`` means the server certifies that values at indexes 0, 1, 2, 4, 6 and any index from 8 to the end of the list are :val:`null`. -Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format for a request to the trajectory endpoint with the query parameter :query-param:`property_slices=dim_frames[3:37:5]` and :query-param:`response_fields=frame_cartesian_site_positions,_exmpl_temperature` where the trajectory consists of 432934 frames (with indexes 0 to 432933) and where the :field:`frame_cartesian_site_positions` contains 7 sites. +Below follows an example of the :field:`data` and :field:`meta` parts of a response using the JSON response format for a request to the trajectory endpoint with the query parameter :query-param:`dimension_slices=dim_frames[3:37:5]` and :query-param:`response_fields=frame_cartesian_site_positions,_exmpl_temperature` where the trajectory consists of 432934 frames (with indexes 0 to 432933) and where the :field:`frame_cartesian_site_positions` contains 7 sites. Furthermore, the metadata subfield :field:`available_slice` in :field:`list_axes` in :field:`_exmpl_temperature` certifies that all values of the data field :field:`_exmpl_temperature` are :val:`null` except at indexes 1000, 1030, 1060, ..., 4000, where values can be either numeric or :val:`null`. .. code:: jsonc @@ -1412,9 +1412,9 @@ While the following URL query parameters are OPTIONAL for clients, API implement The URL query parameter :query-param:`include` is OPTIONAL for both clients and API implementations. The meaning of these URL query parameters are as defined above in section `Entry Listing URL Query Parameters`_. -One additional query parameter :query-param:`property_slices` MUST be handled by the API implementation either as defined below or by returning the error :http-error:`501 Not Implemented`: +One additional query parameter :query-param:`dimension_slices` MUST be handled by the API implementation either as defined below or by returning the error :http-error:`501 Not Implemented`: -- **property\_slices**: A number of slice specifications to request only parts of list properties for the functionality described in `Slices of list properties`_. +- **dimension\_slices**: A number of slice specifications to request only parts of list properties for the functionality described in `Slices of list properties`_. The query parameter contains a comma-separated (",", ASCII symbol 44 dec) list of slice specifications. Each slice specification consists of a dimension name followed by the left square bracket character ("[", ASCII symbol 91 dec), a slice, and the right square bracket character ("]", ASCII symbol 93 dec). @@ -1428,14 +1428,14 @@ One additional query parameter :query-param:`property_slices` MUST be handled by The step value specifies the step size in that dimension. The default is :val:`1`. - An empty value of the :query-param:`property_slices` query parameter MUST be interpreted as equivalent to the query parameter not being included in the request. + An empty value of the :query-param:`dimension_slices` query parameter MUST be interpreted as equivalent to the query parameter not being included in the request. As a consequence, the server will not slice any properties. Requirements and conventions for the response when this query parameter is used are described in `Slices of list properties`_. Example: - - :query-url:`http://optimade.example.com/v1/trajectories/id_12345?response_fields=frame_cartesian_site_positions&property_slices=dim_frames[:999:10],dim_sites[30:70:]` + - :query-url:`http://optimade.example.com/v1/trajectories/id_12345?response_fields=frame_cartesian_site_positions&dimension_slices=dim_frames[:999:10],dim_sites[30:70:]` This query URL requests items from the trajectory with ID :val:`id_12345`. It requests items from the list :field:`frame_cartesian_site_positions` for this trajectory. From 731cf0a2adc875974e8e7a4d360abc98efb299d7 Mon Sep 17 00:00:00 2001 From: Giovanni Pizzi Date: Thu, 18 Sep 2025 00:01:25 +0300 Subject: [PATCH 57/70] Adding client implementation note on discovering sliceable dimensions --- optimade.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/optimade.rst b/optimade.rst index d08d8b1bc..dd20a8ecd 100644 --- a/optimade.rst +++ b/optimade.rst @@ -645,6 +645,8 @@ This functionality is separate from (but compatible with) the protocol described The protocol for large property values is used by the server implementation to transmit a set of items that it deems too large to provide inside the normal OPTIMADE response. Slices, on the other hand, are used for a client to request a subset of any size of the items of a list, which can possibly (but not necessarily) result in such a large amount of values that the protocol for large property values is required to transmit them. +**Client implementation note**: To determine which properties expose sliceable dimensions (and, more generally, the names of such dimensions), clients can request the corresponding property metadata by including ``property_metadata`` in the ``response_fields`` query parameter, together with the names of the properties of interest (see comments in `Metadata properties`_). + The main mechanism is provided through the query parameter :query-param:`dimension_slices` defined in section `Single Entry URL Query Parameters`_. Information relating to the ability of the server to handle this query parameter and the relevant ranges of indexes is provided using the metadata property field :field:`list_axes` (see `Metadata properties`_). When the client request includes the query parameter :query-param:`dimension_slices`, the server MUST provide metadata for all properties for which including the subfield :field:`requested_slice` of the :field:`list_axes` is MANDATORY (see below). From ef816e812f6c4c571121f4a6caa350f9b4d2bf9c Mon Sep 17 00:00:00 2001 From: Giovanni Pizzi Date: Thu, 18 Sep 2025 00:07:39 +0300 Subject: [PATCH 58/70] Minor fixes --- optimade.rst | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/optimade.rst b/optimade.rst index dd20a8ecd..28bb58f1d 100644 --- a/optimade.rst +++ b/optimade.rst @@ -218,8 +218,7 @@ representation in all contexts. They are as follows: - Basic types: **string**, **integer**, **float**, **boolean**, **timestamp**. - **list**: an ordered collection of items, where all items are of the same type, unless they are unknown. A list can be empty, i.e., contain no items. - Multidimensional collections of items are represented as nested lists, and are still referred to simply as "lists" in this document. - + Multidimensional collections of items are represented as nested lists, and are still simply referred to as "lists" in this document. We note that the term "array" is often used in programming languages to refer to similar data structures. However, the term "array" is avoided in this specification to prevent confusion with the meaning of the term "array" in the JSON specification. @@ -245,9 +244,9 @@ representation in all contexts. They are as follows: Further invalid examples (since they cannot be described via an OPTIMADE property specification): - - ``[[1.0, 2.0], ["string", "string"]]`` is invalid since, while each of the two sublists is a valid list ([1.0, 2.0] is a list of integers, and ["string", "string"] is a list of strings), the two sublists have different types. + - ``[[1.0, 2.0], ["string", "string"]]`` is invalid since, while each of the two sublists is a valid list ([1.0, 2.0] is a list of floats, and ["string", "string"] is a list of strings), the two sublists have different types. - - ``[[1.0, 2.0], [3.0, [4.0]], [5.0]]`` is invalid since the second sublist contains a nested list, whereas the other two sublists do not, thus not allowing a consistent definition of the dimensions and axes of the list. + - ``[[1.0, 2.0], [[3.0], [4.0]], [5.0]]`` is invalid since the second sublist contains nested lists, whereas the other two sublists do not, thus not allowing a consistent definition of the dimensions and axes of the list. - **dictionary**: a collection of **key**-**value** pairs, where **keys** are pre-determined strings, i.e., for the same entry property the **keys** remain the same among different entries whereas the **values** change. The **values** of a dictionary can be any basic type, list, dictionary, or unknown. @@ -702,7 +701,7 @@ The field :field:`list_axes` is defined as follows: Note that any of the representations displayed above for the "empty query" are valid. Semantically, the two examples above "empty query" and the "empty query with start" are equivalent, but differ in representation. - This is due to the starting index being explicitely specified in the "empty query with start" example. + This is due to the starting index being explicitly specified in the "empty query with start" example. The need for this difference becomes apparent if the starting index takes on a value different from the default. The dictionary MAY contain the following fields: @@ -4563,7 +4562,7 @@ The header object MUST contain the keys: The following key is RECOMMENDED in the header object: -- :field:`"returned_ranges"`: Array of object. +- :field:`"returned_ranges"`: Array of Object. For dense layout, and sparse layout of one dimensional list properties, the :field:`"returned_ranges"` array contains a single element which is a `slice object`_ representing the range of data present in the response. In the specific case of a hierarchy of list properties represented as a sparse multidimensional list, if the field :field:`"returned_ranges"` is given, it MUST contain one slice object per dimension of the multidimensional list, representing slices for each dimension that cover the data given in the response. From 4b6bf5d3a4eb644e2e2115c8e089710ef22cb06f Mon Sep 17 00:00:00 2001 From: Giovanni Pizzi Date: Thu, 18 Sep 2025 11:48:03 +0300 Subject: [PATCH 59/70] Update optimade.rst Co-authored-by: Rickard Armiento --- optimade.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/optimade.rst b/optimade.rst index 28bb58f1d..07918ce88 100644 --- a/optimade.rst +++ b/optimade.rst @@ -222,9 +222,10 @@ representation in all contexts. They are as follows: We note that the term "array" is often used in programming languages to refer to similar data structures. However, the term "array" is avoided in this specification to prevent confusion with the meaning of the term "array" in the JSON specification. - The term **list axes** refers to the levels of nesting. - We note that it is valid for a list to have elements of different lengths along a given axis, i.e., the list does not need to be rectangular. + The term **list axes** refers to the levels of nesting of a list. + A list can have elements of different lengths along a given axis, i.e., the list does not need to be rectangular. However, the dimensionality of a list, defined as the number of axes (i.e., the number of levels of nesting) must be the same for all entries of a given entry type. + Hence, each item in the list can be indexed by a number of indices equal to the dimensionality of the list. Valid examples: From 293e68b160de4c0b780153449a46c061f2156dda Mon Sep 17 00:00:00 2001 From: Giovanni Pizzi Date: Thu, 18 Sep 2025 11:50:35 +0300 Subject: [PATCH 60/70] Adding comment by @rartino --- optimade.rst | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/optimade.rst b/optimade.rst index 07918ce88..295aaa45e 100644 --- a/optimade.rst +++ b/optimade.rst @@ -216,9 +216,8 @@ context-independent types that are assumed to have some form of representation in all contexts. They are as follows: - Basic types: **string**, **integer**, **float**, **boolean**, **timestamp**. -- **list**: an ordered collection of items, where all items are of the same type, unless they are unknown. +- **list**: a collection of items organized as nested ordered one-dimensional arrangements. All items are of the same type, unless unknown. A list can be empty, i.e., contain no items. - Multidimensional collections of items are represented as nested lists, and are still simply referred to as "lists" in this document. We note that the term "array" is often used in programming languages to refer to similar data structures. However, the term "array" is avoided in this specification to prevent confusion with the meaning of the term "array" in the JSON specification. From 70500e73c8a8e8d64f802aaa0155cf777f2bbfc3 Mon Sep 17 00:00:00 2001 From: Giovanni Pizzi Date: Thu, 18 Sep 2025 15:21:25 +0300 Subject: [PATCH 61/70] Improvement to clarify that all null values must be specified in lists. --- optimade.rst | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/optimade.rst b/optimade.rst index 295aaa45e..27ecb25f8 100644 --- a/optimade.rst +++ b/optimade.rst @@ -223,7 +223,7 @@ representation in all contexts. They are as follows: The term **list axes** refers to the levels of nesting of a list. A list can have elements of different lengths along a given axis, i.e., the list does not need to be rectangular. - However, the dimensionality of a list, defined as the number of axes (i.e., the number of levels of nesting) must be the same for all entries of a given entry type. + However, the dimensionality of a list, defined as the number of axes (i.e., the number of levels of nesting) MUST be the same for all entries of a given entry type. Hence, each item in the list can be indexed by a number of indices equal to the dimensionality of the list. Valid examples: @@ -236,18 +236,23 @@ representation in all contexts. They are as follows: - ``[[1, 2], [3, 4], [5, 6]]`` is a two-dimensional list of three lists, each containing two integers. The first list axis has length 3 and the second list axis has length 2. - - ``[[1, 2], [3, 4, 5], [6]]`` is a two-dimensional list of three lists, each containing a different number of integers. It can be interpreted as a 3x3 (or 3x4, 4x4, ...) list, where the missing elements have implicitly the value :val:`null`. + - ``[[1, 2], [3, 4, 5], [6]]`` is a two-dimensional list of three lists, each containing a different number of integers, and each sublist has a different length. + + - ``[[1, 2], [null], [3]]`` and ``[[1, 2], [], [3]]`` are both valid two-dimensional lists, since all items can be indexed by two list indices. Invalid examples: - ``[3.0, "string"]`` is invalid since the two elements have different types. - Further invalid examples (since they cannot be described via an OPTIMADE property specification): - - ``[[1.0, 2.0], ["string", "string"]]`` is invalid since, while each of the two sublists is a valid list ([1.0, 2.0] is a list of floats, and ["string", "string"] is a list of strings), the two sublists have different types. - ``[[1.0, 2.0], [[3.0], [4.0]], [5.0]]`` is invalid since the second sublist contains nested lists, whereas the other two sublists do not, thus not allowing a consistent definition of the dimensions and axes of the list. + - ``[[1, 2], null, [3]]`` is invalid since the second item is ``null`` and not a one-dimensional list. + Therefore, using two indices to address elements of the list is not always valid (i.e., if we call the list ``exmpl``, it is undefined what ``exmpl[1,2]`` represents). + We note however that there are situations where the OPTIMADE property definition could allow this datastructure to be valid (as a list of optional lists), even if this would not be a two-dimensional list for which dimensions and axes can be defined. + + - **dictionary**: a collection of **key**-**value** pairs, where **keys** are pre-determined strings, i.e., for the same entry property the **keys** remain the same among different entries whereas the **values** change. The **values** of a dictionary can be any basic type, list, dictionary, or unknown. From 47d84856422cdce5f509ab6cef04adae97c3a877 Mon Sep 17 00:00:00 2001 From: Gian-Marco Rignanese Date: Fri, 19 Sep 2025 09:12:11 +0300 Subject: [PATCH 62/70] Update optimade.rst Co-authored-by: Giovanni Pizzi --- optimade.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/optimade.rst b/optimade.rst index 27ecb25f8..436389f6d 100644 --- a/optimade.rst +++ b/optimade.rst @@ -1441,10 +1441,10 @@ One additional query parameter :query-param:`dimension_slices` MUST be handled b Example: - - :query-url:`http://optimade.example.com/v1/trajectories/id_12345?response_fields=frame_cartesian_site_positions&dimension_slices=dim_frames[:999:10],dim_sites[30:70:]` + - :query-url:`http://optimade.example.com/v1/trajectories/id_12345?response_fields=cartesian_site_positions&dimension_slices=dim_frames[:999:10],dim_sites[30:70:]` This query URL requests items from the trajectory with ID :val:`id_12345`. - It requests items from the list :field:`frame_cartesian_site_positions` for this trajectory. + It requests items from the list :field:`cartesian_site_positions` for this trajectory. The items that are requested are for only the 31st to 71st sites (i.e., with indexes 30 through 70 inclusive) for 1 out of every 10 frames of the first 1000 frames (i.e., taking steps of 10 over indexes 0 through 999 inclusive, which requests the frames with indexes 0, 10, 20, 30, ..., 990). Single Entry JSON Response Schema From d98ab4708d7a007a3f6e695dd3c101d701828278 Mon Sep 17 00:00:00 2001 From: Gian-Marco Rignanese Date: Fri, 19 Sep 2025 09:12:39 +0300 Subject: [PATCH 63/70] Update optimade.rst Co-authored-by: Giovanni Pizzi --- optimade.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/optimade.rst b/optimade.rst index 436389f6d..79bd2779b 100644 --- a/optimade.rst +++ b/optimade.rst @@ -2425,9 +2425,9 @@ A Property Definition MUST be composed according to the combination of the requi Dimension names defined by the OPTIMADE standard are prefixed by ``dim_``. Dimension names defined by database or definition providers MUST be prefixed by the corresponding database or namespace prefix, which SHOULD then be immediately followed by ``dim_``, e.g., ``_exmpl_dim_particles``. If, within one entry, two or more list axes in one or more properties share the same dimension :field:`name`, those represent the same dimension. - For example, let us consider the property :property:`frame_cartesian_site_positions` of the trajectory entry, where the first dimension name is :val:`dim_frames`. + For example, let us consider the property :property:`cartesian_site_positions` of the trajectory entry, where the first dimension name is :val:`dim_frames`. Let the trajectory entry in this example have another, one-dimensional, list property :property:`_exmpl_energy`, which in its property definition specifies *the same name*, :val:`dim_frames`, as the name of the axis corresponding to its single dimension. - The joint dimension name means the values of :property:`_exmpl_energy` and of :property:`frame_cartesian_site_positions` at index *i* pertain to the same frame. + The joint dimension name means the values of :property:`_exmpl_energy` and of :property:`cartesian_site_positions` at index *i* pertain to the same frame. If slicing is used to request only parts of the data along the :val:`dim_frames` dimension, that is a request to slice both the properties according to the specified slice. - :field:`sizes`: List of Integers or :val:`null`. From 9038665f9c0333f0d97d6e30c2848f4f68de22f9 Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Fri, 19 Sep 2025 09:17:58 +0300 Subject: [PATCH 64/70] Update list definition based on review discussions --- optimade.rst | 35 ++++++++++++++++++----------------- 1 file changed, 18 insertions(+), 17 deletions(-) diff --git a/optimade.rst b/optimade.rst index 79bd2779b..46145fe7d 100644 --- a/optimade.rst +++ b/optimade.rst @@ -218,15 +218,17 @@ representation in all contexts. They are as follows: - Basic types: **string**, **integer**, **float**, **boolean**, **timestamp**. - **list**: a collection of items organized as nested ordered one-dimensional arrangements. All items are of the same type, unless unknown. A list can be empty, i.e., contain no items. - We note that the term "array" is often used in programming languages to refer to similar data structures. - However, the term "array" is avoided in this specification to prevent confusion with the meaning of the term "array" in the JSON specification. - - The term **list axes** refers to the levels of nesting of a list. - A list can have elements of different lengths along a given axis, i.e., the list does not need to be rectangular. - However, the dimensionality of a list, defined as the number of axes (i.e., the number of levels of nesting) MUST be the same for all entries of a given entry type. - Hence, each item in the list can be indexed by a number of indices equal to the dimensionality of the list. - - Valid examples: +- **list**: an ordered indexed collection of items that are either (i) all of the same basic type, (ii) only dictionaries, or (iii) only lists. + In each case, (i)-(iii), unknown values (i.e., ``null``) are also allowed at any of the positions. + Furthermore, a list can also be empty, i.e., contain no items. + An empty list has a distinct meaning from ``null``: a list of no items (see `Properties with an unknown value`_). + + Multidimensional collections of items are represented as nested lists. + This specification uses the term **list axis** to refer to the levels of nesting in nested lists, and **dimensionality** as the deepest nesting level of the list and its sublists. + A list can have items of different lengths along a given axis, i.e., nested lists are **not** limited to representing rectangular arrangement of items. + While this definition allows for complex structures of nested lists of varying nesting level, only those variants that can be validated by OPTIMADE `Property definitions`_ can be used for properties. + + Examples of valid lists: - ``[1.0, 2.0, 3.0]`` is a one-dimensional list of three floats (only one axis). @@ -240,18 +242,17 @@ representation in all contexts. They are as follows: - ``[[1, 2], [null], [3]]`` and ``[[1, 2], [], [3]]`` are both valid two-dimensional lists, since all items can be indexed by two list indices. - Invalid examples: - - - ``[3.0, "string"]`` is invalid since the two elements have different types. + - ``[[1, 2], null, [3]]`` is valid since ``null`` can substitute items of any type, including sublists. - - ``[[1.0, 2.0], ["string", "string"]]`` is invalid since, while each of the two sublists is a valid list ([1.0, 2.0] is a list of floats, and ["string", "string"] is a list of strings), the two sublists have different types. + Examples of invalid lists: - - ``[[1.0, 2.0], [[3.0], [4.0]], [5.0]]`` is invalid since the second sublist contains nested lists, whereas the other two sublists do not, thus not allowing a consistent definition of the dimensions and axes of the list. + - ``[3.0, "string"]`` is not a valid list, since the two elements have different types. + + Examples of lists that are valid, but cannot be used for properties in OPTIMADE due to limits imposed by OPTIMADE property definitions: - - ``[[1, 2], null, [3]]`` is invalid since the second item is ``null`` and not a one-dimensional list. - Therefore, using two indices to address elements of the list is not always valid (i.e., if we call the list ``exmpl``, it is undefined what ``exmpl[1,2]`` represents). - We note however that there are situations where the OPTIMADE property definition could allow this datastructure to be valid (as a list of optional lists), even if this would not be a two-dimensional list for which dimensions and axes can be defined. + - ``[[1.0, 2.0], ["string", "string"]]``: while each of the two sublists is a valid list ([1.0, 2.0] is a list of floats, and ["string", "string"] is a list of strings), the two sublists cannot will not be able to be validated by the same item subschema in a Property definition. + - ``[[1.0, 2.0], [[3.0], [4.0]], [5.0]]``: the second sublist contains nested lists, whereas the other two sublists do not, thus not allowing a consistent definition of the dimensions and axes of the list. - **dictionary**: a collection of **key**-**value** pairs, where **keys** are pre-determined strings, i.e., for the same entry property the **keys** remain the same among different entries whereas the **values** change. The **values** of a dictionary can be any basic type, list, dictionary, or unknown. From d36092fc1db6ff635d6a4c4cf6e832941a7e2c1f Mon Sep 17 00:00:00 2001 From: Giovanni Pizzi Date: Fri, 19 Sep 2025 09:23:50 +0300 Subject: [PATCH 65/70] Removing leftover lines on fix over GitHub --- optimade.rst | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/optimade.rst b/optimade.rst index 0588f05f3..d56e6795f 100644 --- a/optimade.rst +++ b/optimade.rst @@ -216,13 +216,11 @@ context-independent types that are assumed to have some form of representation in all contexts. They are as follows: - Basic types: **string**, **integer**, **float**, **boolean**, **timestamp**. -- **list**: a collection of items organized as nested ordered one-dimensional arrangements. All items are of the same type, unless unknown. - A list can be empty, i.e., contain no items. - **list**: an ordered indexed collection of items that are either (i) all of the same basic type, (ii) only dictionaries, or (iii) only lists. In each case, (i)-(iii), unknown values (i.e., ``null``) are also allowed at any of the positions. Furthermore, a list can also be empty, i.e., contain no items. An empty list has a distinct meaning from ``null``: a list of no items (see `Properties with an unknown value`_). - + Multidimensional collections of items are represented as nested lists. This specification uses the term **list axis** to refer to the levels of nesting in nested lists, and **dimensionality** as the deepest nesting level of the list and its sublists. A list can have items of different lengths along a given axis, i.e., nested lists are **not** limited to representing rectangular arrangement of items. @@ -247,8 +245,8 @@ representation in all contexts. They are as follows: Examples of invalid lists: - ``[3.0, "string"]`` is not a valid list, since the two elements have different types. - - Examples of lists that are valid, but cannot be used for properties in OPTIMADE due to limits imposed by OPTIMADE property definitions: + + Examples of lists that are valid, but cannot be used for properties in OPTIMADE due to limits imposed by OPTIMADE property definitions: - ``[[1.0, 2.0], ["string", "string"]]``: while each of the two sublists is a valid list ([1.0, 2.0] is a list of floats, and ["string", "string"] is a list of strings), the two sublists cannot will not be able to be validated by the same item subschema in a Property definition. From da14ac20e40aec1e35a58820b9c239bb9ceb4c91 Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Fri, 19 Sep 2025 09:25:30 +0300 Subject: [PATCH 66/70] Minor grammar fix --- optimade.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/optimade.rst b/optimade.rst index d56e6795f..d1a5a3cb5 100644 --- a/optimade.rst +++ b/optimade.rst @@ -248,7 +248,7 @@ representation in all contexts. They are as follows: Examples of lists that are valid, but cannot be used for properties in OPTIMADE due to limits imposed by OPTIMADE property definitions: - - ``[[1.0, 2.0], ["string", "string"]]``: while each of the two sublists is a valid list ([1.0, 2.0] is a list of floats, and ["string", "string"] is a list of strings), the two sublists cannot will not be able to be validated by the same item subschema in a Property definition. + - ``[[1.0, 2.0], ["string", "string"]]``: while each of the two sublists is a valid list ([1.0, 2.0] is a list of floats, and ["string", "string"] is a list of strings), the two sublists will not be able to be validated by the same item subschema in a Property definition. - ``[[1.0, 2.0], [[3.0], [4.0]], [5.0]]``: the second sublist contains nested lists, whereas the other two sublists do not, thus not allowing a consistent definition of the dimensions and axes of the list. From d2f8644e10e987a3ad51767ad467e383bcd6d2d4 Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Fri, 19 Sep 2025 10:40:10 +0300 Subject: [PATCH 67/70] Apply suggestions from code review --- optimade.rst | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/optimade.rst b/optimade.rst index d1a5a3cb5..2a95d03d6 100644 --- a/optimade.rst +++ b/optimade.rst @@ -218,8 +218,8 @@ representation in all contexts. They are as follows: - Basic types: **string**, **integer**, **float**, **boolean**, **timestamp**. - **list**: an ordered indexed collection of items that are either (i) all of the same basic type, (ii) only dictionaries, or (iii) only lists. In each case, (i)-(iii), unknown values (i.e., ``null``) are also allowed at any of the positions. - Furthermore, a list can also be empty, i.e., contain no items. - An empty list has a distinct meaning from ``null``: a list of no items (see `Properties with an unknown value`_). + A list can also be empty, i.e., contain no items. + An empty list has a distinct meaning from ``null``, namely, it is a list with no items (see `Properties with an unknown value`_). Multidimensional collections of items are represented as nested lists. This specification uses the term **list axis** to refer to the levels of nesting in nested lists, and **dimensionality** as the deepest nesting level of the list and its sublists. @@ -251,6 +251,7 @@ representation in all contexts. They are as follows: - ``[[1.0, 2.0], ["string", "string"]]``: while each of the two sublists is a valid list ([1.0, 2.0] is a list of floats, and ["string", "string"] is a list of strings), the two sublists will not be able to be validated by the same item subschema in a Property definition. - ``[[1.0, 2.0], [[3.0], [4.0]], [5.0]]``: the second sublist contains nested lists, whereas the other two sublists do not, thus not allowing a consistent definition of the dimensions and axes of the list. + As of OPTIMADE v1.2, such lists are excluded by Property definitions. - **dictionary**: a collection of **key**-**value** pairs, where **keys** are pre-determined strings, i.e., for the same entry property the **keys** remain the same among different entries whereas the **values** change. The **values** of a dictionary can be any basic type, list, dictionary, or unknown. @@ -718,7 +719,8 @@ The field :field:`list_axes` is defined as follows: - :field:`sliceable`: Boolean. If :val:`true`, the server MUST handle slices for that dimension. - If :val:`false`, the server MAY handle slices for that dimension, or MAY return the error :http-error:`501 Not Implemented` when a client requests a slice involving this dimension. + If :val:`false`, the server MAY handle slices for that dimension, but if it receives a slice request it cannot handle, it MUST return the error :http-error:`501 Not Implemented` when a client requests a slice involving this dimension. + If a server is capable of handling slicing for a particular dimension, the server SHOULD indicate this by setting :field:`sliceable` to :val:`true` in its responses. If the field is omitted or :val:`null`, it means the same thing as :val:`false`. - :field:`available_slice`: Dictionary or :val:`null`. From 4578c0e4f8e9f9c9daae7cf9217169fd8a8a9650 Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Fri, 19 Sep 2025 10:42:46 +0300 Subject: [PATCH 68/70] Add suggestion from review --- optimade.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/optimade.rst b/optimade.rst index 2a95d03d6..574f49ddb 100644 --- a/optimade.rst +++ b/optimade.rst @@ -248,7 +248,7 @@ representation in all contexts. They are as follows: Examples of lists that are valid, but cannot be used for properties in OPTIMADE due to limits imposed by OPTIMADE property definitions: - - ``[[1.0, 2.0], ["string", "string"]]``: while each of the two sublists is a valid list ([1.0, 2.0] is a list of floats, and ["string", "string"] is a list of strings), the two sublists will not be able to be validated by the same item subschema in a Property definition. + - ``[[1.0, 2.0], ["string", "string"]]``: while each of the two sublists is a valid list ([1.0, 2.0] is a list of floats, and ["string", "string"] is a list of strings), the two sublists cannot be validated by the same item subschema in a Property definition. - ``[[1.0, 2.0], [[3.0], [4.0]], [5.0]]``: the second sublist contains nested lists, whereas the other two sublists do not, thus not allowing a consistent definition of the dimensions and axes of the list. As of OPTIMADE v1.2, such lists are excluded by Property definitions. From 697b2c1e4fc39c40f0100eebef86b6a072fd0b8d Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Fri, 19 Sep 2025 10:46:59 +0300 Subject: [PATCH 69/70] Remove trailing whitespace --- optimade.rst | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/optimade.rst b/optimade.rst index 574f49ddb..869b7ffee 100644 --- a/optimade.rst +++ b/optimade.rst @@ -719,7 +719,7 @@ The field :field:`list_axes` is defined as follows: - :field:`sliceable`: Boolean. If :val:`true`, the server MUST handle slices for that dimension. - If :val:`false`, the server MAY handle slices for that dimension, but if it receives a slice request it cannot handle, it MUST return the error :http-error:`501 Not Implemented` when a client requests a slice involving this dimension. + If :val:`false`, the server MAY handle slices for that dimension, but if it receives a slice request it cannot handle, it MUST return the error :http-error:`501 Not Implemented` when a client requests a slice involving this dimension. If a server is capable of handling slicing for a particular dimension, the server SHOULD indicate this by setting :field:`sliceable` to :val:`true` in its responses. If the field is omitted or :val:`null`, it means the same thing as :val:`false`. @@ -4936,4 +4936,3 @@ Example file {"type": "info", "id": "structures", ...} {"type": "references", "id": "2", "attributes": {...}} {"type": "structures", "id": "1", "attributes": {...}, "relationships": {"references": {"data": [{"id": "2", "type": "references"}]}}} - From 310ea6e7f6167497caa55c5f80bc6e3360beb23a Mon Sep 17 00:00:00 2001 From: Rickard Armiento Date: Fri, 19 Sep 2025 10:54:34 +0300 Subject: [PATCH 70/70] Fix formulation based on code reivew MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Daniel Beltrán <44979434+d-beltran@users.noreply.github.com> --- optimade.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/optimade.rst b/optimade.rst index 869b7ffee..4eb577644 100644 --- a/optimade.rst +++ b/optimade.rst @@ -677,7 +677,7 @@ The field :field:`list_axes` is defined as follows: - :field:`start`: Non-negative integer or :val:`null`. - - :field:`stop`: Non-negative integer or :val:`null`. + - :field:`stop`: Non-negative integer equal or greater to the :field:`start` value or :val:`null`. - :field:`step`: Positive integer or :val:`null`.