Skip to content

Add ranged properties #452

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
JPBergsma wants to merge 35 commits into from
Closed

Add ranged properties #452

Changes from 10 commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
c18cf0c
Intermediate state adding ranged properties.
JPBergsma Dec 31, 2022
7dc50f2
Merge branch 'develop' into JPBergsma/Add_ranged_properties
JPBergsma Jan 3, 2023
6afcf9c
first draft for ranged properties.
JPBergsma Jan 6, 2023
1a55c2a
Merge branch 'develop' into JPBergsma/Add_ranged_properties
JPBergsma Jan 6, 2023
b597a67
Removed average, set, min and max fields for now as these become quit…
JPBergsma Jan 9, 2023
37db878
Small corrections.
JPBergsma Jan 9, 2023
a832751
Added how to treat missing values for requested range.
JPBergsma Jan 11, 2023
16aba07
changed description field
JPBergsma Jan 12, 2023
b1d69a8
Apply suggestions from code review
JPBergsma Jan 12, 2023
edbfc25
Merge branch 'Materials-Consortia:develop' into JPBergsma/Add_ranged_…
JPBergsma Jan 12, 2023
14de45d
Apply suggestions from code review Vaitkus
JPBergsma Jan 17, 2023
906db81
intermediate state from implementing code review.
JPBergsma Jan 17, 2023
1feb4a9
Merge branch 'JPBergsma/Add_ranged_properties' of https://github.com/…
JPBergsma Jan 17, 2023
a96dffe
Processed comments rartino.
JPBergsma Jan 18, 2023
15f599c
Small corrections.
JPBergsma Feb 15, 2023
73905dc
Added returned range property.
JPBergsma Feb 15, 2023
c6834f3
Added extra explanation values field.
JPBergsma Feb 16, 2023
b0cc94c
Apply suggestions from code review
JPBergsma Feb 17, 2023
0cee1e6
Merge branch 'develop' into JPBergsma/Add_ranged_properties
JPBergsma Mar 6, 2023
d7c8a9c
Processed comments rickard and a few more small improvements.
JPBergsma Mar 9, 2023
d1e8d74
Further changes after proof reading.
JPBergsma Mar 9, 2023
139c70e
further refinements.
JPBergsma Mar 9, 2023
916d6f2
Changed 'n_' to 'n' for ranged metadata properties tio be consistent …
JPBergsma Mar 13, 2023
ee3651e
Changed wording of range_id field after suggestion Rickard.
JPBergsma Mar 13, 2023
1f794b6
placed subsequent sentences on seperate lines.
JPBergsma Mar 16, 2023
169a1f4
Processed points discussed with Rickard.
JPBergsma Mar 24, 2023
f513596
Added per entry next field + small corrections.
JPBergsma Mar 28, 2023
65b8ad1
Apply suggestions from code review
JPBergsma May 2, 2023
94db38c
Adjusted the description next fields for ranged properties.
JPBergsma May 2, 2023
033ea11
Corrected range name for _exmpl_ranged_thermostat.
JPBergsma May 25, 2023
3762d30
Added that querying on properties in the range dictionary is optional.
JPBergsma May 30, 2023
8332567
Specifically mention that support for queries directly on the values …
JPBergsma May 31, 2023
a87b301
Updated example to latest version metadata proposal and added more ex…
JPBergsma Jun 2, 2023
7e9b4f4
Improved explanation returned_range field.
JPBergsma Jun 2, 2023
4b298c5
Small corrections.
JPBergsma Jun 2, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
179 changes: 179 additions & 0 deletions optimade.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ OPTIMADE API specification v1.2.0~develop

entry : names of type of resources, served via OPTIMADE, pertaining to data in a database.
property : data item that belongs to an entry.
ranged_property : A property that can be returned in pieces and that supports slicing.
Comment thread
JPBergsma marked this conversation as resolved.
Outdated
val : value examples that properties can be.
:val: is ONLY used when referencing values of actual properties, i.e., information that belongs to the database.
type : data type of values.
Expand Down Expand Up @@ -67,6 +68,8 @@ OPTIMADE API specification v1.2.0~develop

.. role:: property(literal)

.. role:: ranged-property(literal)

Comment thread
JPBergsma marked this conversation as resolved.
Outdated
.. role:: val(literal)

.. role:: type(literal)
Expand Down Expand Up @@ -442,6 +445,163 @@ For example, the following query can be sent to API implementations `exmpl1` and

:filter:`filter=_exmpl1_band_gap<2.0 OR _exmpl2_band_gap<2.5`


Ranged Properties
-----------------

- **Description**: Ranged properties support slicing, so the client can request that only some of the values need to be returned.
Likewise, the server can use paging to return the property in multiple parts.
This can be useful for properties that are so large that it can be inconvenient to return them in a single response.
If an entry is too large to be returned in a single response a link is provided, as described in `JSON Response Schema: Common Fields`_ under the `links.next` field, from which the remainder of the requested data can be retrieved.
Ranged properties also provide a method to correlate the values of two ranged properties via a :property:`range_ids`.
The metadata is returned by default, the data is only returned when specifically requested via the :query-param:`ranged property` query parameter as described under `Entry Listing URL Query Parameters`_.

When a client does not use query parameters to select a range for the ranged property, the server returns a dictionary with meta data about the range property with the following format:
Comment thread
JPBergsma marked this conversation as resolved.
Outdated
- **Type**: dictionary with keys:
Comment thread
JPBergsma marked this conversation as resolved.
Outdated

- :property:`serialization_format`: string (REQUIRED)
- :property:`values`: List of any data type (OPTIONAL)
Copy link
Copy Markdown
Member

@rartino rartino Jan 13, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, do I understand correctly that the present design is for the member values in the "metadata dictionary" to also contain the actual data values? (So, in a sense, "metadata" is mislabeling it?)

So, correlated with the thoughts above about pagination: how about making responses always contain two fields for ranged properties, one metadata field with only metadata, and one for the data. That way, even the initial response that contain a ranged property can pass the first page of data the way it would if this was just a regular (non-ranged) property. This means that if the ranged property happens to be small, the client gets all the data it needs in the first response without having to do anything else. Only in the case all data does not fit in one response does the client need to engage with the specifics of accessing ranged properties. This will actually greatly help collections (#386): I felt we were going to end up with an ugly design to force clients to fiddle with range access for all collections, when in many cases of smaller collections this would be unnecessary.

So, I propose:

  • Split out values into its own field; i.e., edit the initial text to say that for ranged properties two fields are returned:
    <property_name>_ranged_meta: a dictionary with all these metadata fields.
    <property_name>_ranged_data: a list of values of any type.
    The server MAY support querying on either or both.

For the field <fieldname>_ranged_meta all the fields you defined here are kept with the same meaning, but two new ones are needed (and perhaps these should be the only ones MANDATORY):

  • 'links': a JSON:API object on the format described in the JSON:API specification section "Pagination", which is used for pagination of the ranged property.
  • 'more_data_available': boolean. True if the data does not fit in a single page and this is not the final page.

The field <fieldname>_ranged_data would then be defined as:

  • If no range-selecting query parameters are given, it contains the first page of data (which is all of the data if and only if more_data_available is false in <fieldname>_ranged_meta. If <fieldname>_ranged_meta is true, then further pages of data can be fetched by the client via the URL in next in <fieldname>_ranged_meta.
  • If any of the range-selecting query parameters defined below are given, then data matching that range selection is returned. The server MAY support paging also of the requested range, or MAY return an error if the requested range is too big to fit in a single response.

Copy link
Copy Markdown
Contributor Author

@JPBergsma JPBergsma Jan 18, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, do I understand correctly that the present design is for the member values in the "metadata dictionary" to also contain the actual data values? (So, in a sense, "metadata" is mislabeling it?)

Yes, the dictionary would also contain data when the property is listed within the property_range query parameter so the term "metadata dictionary" is misleading.

how about making responses always contain two fields for ranged properties, one metadata field with only metadata, and one for the data.

I do like that all the data is grouped together in a single field. We could also make a separate sub dictionary to group the metadata separate from the data. But overall, it is not that important how the fields are organized.

That way, even the initial response that contain a ranged property can pass the first page of data the way it would if this was just a regular (non-ranged) property.

I am not sure, I like your proposal to return the first “page” of data of a ranged property by default.
The first request will typically be for a client to check which entries are present. It can then select the most interesting entries. And retrieve the additional data for these entries.
With your proposal, the number of entries that a server can return in a single response would be much smaller, as most of the response would consist of the first “pages” of properties.

If a user wants the behaviour you described, they could simply use the property_range query parameter to specify that they want the first x values of a property returned.

If a client would retrieve only a single property, it would make more sense to already provide some of the data, but having a different behaviour for the single entry endpoint could also be confusing/complicate things.

I felt we were going to end up with an ugly design to force clients to fiddle with range access for all collections, when in many cases of smaller collections this would be unnecessary.

Another solution could perhaps be to define the collection field flexibly so that you could have a regular and a ranged version of this property. I already wrote that if the part after the ranged prefix is the name of a regular property, it should meet all the requirements of that property.

'links': a JSON:API object on the format described in the JSON:API specification section "Pagination", which is used for pagination of the ranged property.

In the JSON API v1.0 links is a reserved key word, so we would have to give this field a different name. (or drop support for json API 1.0) see section 5.2.3 of the JSON API specification.

It is quite a big change you are suggesting. Instead of retrieving the data per entry, you want to retrieve data per property. A bit as if each property is a resource in itself.

If any of the range-selecting query parameters defined below are given, then data matching that range selection is returned. The server MAY support paging also of the requested range, or MAY return an error if the requested range is too big to fit in a single response.

I do not think returning errors to the user is very nice when the user can’t know in advance that something is wrong.
So we should somehow specify how much data the server can return, and then the client should construct a request that fits between these bounds.
But this is difficult as the size of the properties differs from entry to entry and is unknown to the client.
So I do think that the server should implement pagination as it has all the data needed to do so while the client has not.

We should probably also think about creating streaming responses, so sending back large data items is less odd to the client. Now we first prepare the response and then send it. But for large responses this would be odd as the user does not get any data back for a while. If we do this, we can send larger responses, and we would not need ranged properties as often. But this is probably more of an issue for the optimade python tools rather than a change in the optimade specification.

Sara and Gian-Marco have asked me to prioritize writing a tutorial for the old trajectory proposal, so it may take a while before I get back to this topic.

Copy link
Copy Markdown
Member

@rartino rartino Jan 19, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the dictionary would also contain data when the property is listed within the property_range query parameter so the term "metadata dictionary" is misleading.

Where does the text of this PR presently say this, i.e., that the behavior of value depends on the query parameter property_range? What I see in the text is: "The property values MUST be present when serialization_format is not set to "linear"."

I do like that all the data is grouped together in a single field. We could also make a separate sub dictionary to group the metadata separate from the data. But overall, it is not that important how the fields are organized.

I think there is a point in separating data and metadata, especially when we consider the possibility of searching for data. But I agree that whether the data goes inside or outside the dictionary is not the most crucial design decision. Perhaps we can discuss it on the web meeting.

I am not sure, I like your proposal to return the first “page” of data of a ranged property by default. The first request will typically be for a client to check which entries are present. It can then select the most interesting entries. And retrieve the additional data for these entries.
With your proposal, the number of entries that a server can return in a single response would be much smaller, as most of the response would consist of the first “pages” of properties.

The first page doesn't need to be very large, it is up to the implementation - it could be, say, 20 values. If the client wants the data it asks for the field (and should then be happy to also get the data), if the client does not want the data it should not ask for the field.

If a user wants the behaviour you described, they could simply use the property_range query parameter to specify that they want the first x values of a property returned.

I'm trying to move this in the direction where implementations that do not support random access of the data does not have to implement property_range at all, which is fairly complex to parse the way it is currently defined.

I hadn't even considered that the way you've written this is meant to support a range selection over every entry that matches a filter. I do not think implementations should be forced to support that even if they do support range selection over a single entry.

In the JSON API v1.0 links is a reserved key word, so we would have to give this field a different name. (or drop support for json API 1.0) see section 5.2.3 of the JSON API specification.

Can you cite the part of the JSON:API specification that you read to say one cannot have links as a key in a dictionary inside attributes?

It is quite a big change you are suggesting. Instead of retrieving the data per entry, you want to retrieve data per property. A bit as if each property is a resource in itself.

No, that is not what I meant, sorry if I was unclear. This was meant to be a small adjustment in behavior - the URL in links is meant to give you a single entry response with the next page of data for every property using the same range id as the one your are paging in. You can implement it by a URL that adds "property_ranges" selecting the next range (but you don't have to). I see now that I had missed how you want to allow multidimensional indexing with multiple range_ids; one would in that case have to work out how paging works in the multidimensional case. (But, do we need to support multidimensional ranges?)

So we should somehow specify how much data the server can return, and then the client should construct a request that fits between these bounds.

Well, as you also conclude, that cannot generally be done given the degrees of freedom supported by serialization_format. So, sure, if you want to specify that IF property_range is supported and the client asks for more results than the server can/wants to return, then the server also MUST allow paging those ranged responses with the paging mechanism I proposed, I'd be ok with that. As long as there is an option not to support property_range at all.

Copy link
Copy Markdown
Contributor Author

@JPBergsma JPBergsma Feb 16, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where does the text of this PR presently say this, i.e., that the behavior of value depends on the query parameter.

On line 526/527 it is mentioned that the values are only returned when the property is listed in the property_ranges query parameter.
I have added an extra line to the description of the values field to make it a bit clearer.

I'm trying to move this in the direction where implementations that do not support random access of the data does not have to implement property_range at all, which is fairly complex to parse the way it is currently defined.

In that case, the client would still need a way to specify which fields it wants to have. So I think you would still need the property_range query parameter.
How would you make it easier to parse ?

I hadn't even considered that the way you've written this is meant to support a range selection over every entry that matches a filter. I do not think implementations should be forced to support that even if they do support range selection over a single entry.

Most of the code for the single entry endpoint and the multiple entry endpoint is the same. So I do not think this makes the implementation more difficult. It may actually be more work to create different behaviour for the single and multiple entry points.

Can you cite the part of the JSON:API specification that you read to say one cannot have links as a key in a dictionary inside attributes?

“Complex data structures involving JSON objects and arrays are allowed as attribute values. However, any object that constitutes or is contained in an attribute MUST NOT contain a relationships or links member, as those members are reserved by this specification for future use.”

No, that is not what I meant, sorry if I was unclear. This was meant to be a small adjustment in behaviour - the URL in links is meant to give you a single entry response with the next page of data for every property using the same range id as the one your are paging in. You can implement it by a URL that adds "property_ranges" selecting the next range (but you don't have to).

So, if I understand it correctly, you want to give each resource/entry its own next link to retrieve the remainder of this entry, and use the standard JSON API next link to point to the next entry that matches the filter.
I do not use the range_id for retrieving the data.
You use the name of the field in the query parameter property_ranges. So you can define a separate range for each property.

I see now that I had missed how you want to allow multidimensional indexing with multiple range_ids; one would in that case have to work out how paging works in the multidimensional case. (But, do we need to support multidimensional ranges?)

The current proposal already supports paging in multiple dimensions. (The range_ids have nothing to do with indexing. It only shows that two properties use the same range, so that values with the same index belong to each other.)
I mostly started out thinking how I could make sure that the proposal was compatible with a future extension to multiple dimensions. And when I had figured that out, I thought I could just as well write it out in the specification.
I could leave out the references to multiple dimensions, although in that case some people may wonder why there are some extra pairs of square brackets needed.
Some people have also asked me whether it would be possible to retrieve the cartesian_site_positions only for specific atoms. The current specification does not yet allow that, but I think the property_ranges query parameter could be extended to allow specific indexes. We could, for example, make a convention that the list that now contains the first last and step parameters could be replaced with a list of the indexes, placed between “(“ and “)”, for which data should be retrieved.

Copy link
Copy Markdown
Member

@rartino rartino Feb 17, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm trying to move this in the direction where implementations that do not support random access of the data does not have to implement property_range at all, which is fairly complex to parse the way it is currently defined.

In that case, the client would still need a way to specify which fields it wants to have. So I think you would still need the property_range query parameter.

I don't get what you mean here, sorry. Which fields to return in a response is requested via response_fields?

How would you make it easier to parse?

I'm trying to propose to make it so that the single mandatory feature to get all data in a ranged property is via a next-like paging system, which wouldn't require interpreting property_range at all.

Can you cite the part of the JSON:API specification that you read to say one cannot have links as a key in a dictionary inside attributes?

“Complex data structures involving JSON objects and arrays are allowed as attribute values. However, any object that constitutes or is contained in an attribute MUST NOT contain a relationships or links member, as those members are reserved by this specification for future use.”

Oh, wow, I had missed this completely. It is a really strange rule to have for attribute data at arbitrary depth, and it seems they removed it in v1.1. Well, ok then, you are right that we would have to name that inner field something different than links. Perhaps just place a URL in a member named next.

So, if I understand it correctly, you want to give each resource/entry its own next link to retrieve the remainder of this entry, and use the standard JSON API next link to point to the next entry that matches the filter. I do not use the range_id for retrieving the data. You use the name of the field in the query parameter property_ranges. So you can define a separate range for each property.

No, the way you describe it does not sound right. Lets me try again.

The standard JSON API next link would work as it always has: it pages over a range of returned entries. This is an absolute necessity to adhere to JSON:API.

A ranged property would, among its metadata, communicate a "special" next link. Following that next link would give you that entry (that whole entry with all attributes/properties as requested by response_fields) but where the "data" part of the ranged property contains the next range of values. This is the most straightforward way (and thus easiest to implement) I can imagine to communicate paged ranged data values within a fully valid OPTIMADE/JSON:API response.

Maybe the most clear way to express it is that this "feature" is meant to be completely compatible with your design of property_ranges. Someone who has implemented property_ranges gets precisely the right thing by simply constructing that special next link by adding a property_ranges parameter that specifies the next page of data.

Some people have also asked me whether it would be possible to retrieve the cartesian_site_positions only for specific atoms. The current specification does not yet allow that, but I think the property_ranges query parameter could be extended to allow specific indexes.

Why would this extension belong in property_ranges? If so, one could only use it for ranged properties? It seems to me it belongs in response_fields or as its own query parameter, where one could imagine various ways to extend it to ask for specific subsets of a property.

Copy link
Copy Markdown
Contributor Author

@JPBergsma JPBergsma Feb 17, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't get what you mean here, sorry. Which fields to return in a response is requested via response_fields?

In the discussion on the trajectory endpoint, it was concluded that it would be confusing to determine whether the values should be returned based on whether the property was present in the response_fields. If the whole property (metadata + data) is present under a single field, there would be no way to specify with the response_fields parameter that you only want the metadata.
So instead we decided to trigger returning the data by the presence of the query parameters first_frame, last_frame and frame_step.
It therefore made sense to me to do the same here with the property_ranges query parameter.

I now understand why you wanted to split the data and metadata into two separate fields. That way, you could add the data field to the response_fields to get the data. And not return it otherwise.
If we however add by default a next link to the metadata, it would not be necessary to have a separate field for the data.

No, the way you describe it does not sound right. Let me try again.
The standard JSON API next link would work as it always has: it pages over a range of returned entries. This is an absolute necessity to adhere to JSON:API.
A ranged property would, among its metadata, communicate a "special" next link. Following that next link would give you that entry (that whole entry with all attributes/properties as requested by response_fields) but where the "data" part of the ranged property contains the next range of values. This is the most straightforward way (and thus easiest to implement) I can imagine to communicate paged ranged data values within a fully valid OPTIMADE/JSON:API response.
Maybe the most clear way to express it is that this "feature" is meant to be completely compatible with your design of property_ranges. Someone who has implemented property_ranges gets precisely the right thing by simply constructing that special next link by adding a property_ranges parameter that specifies the next page of data.

Ok, I think I understand you now. I’ll adjust in the proposal.

Why would this extension belong in property_ranges? If so, one could only use it for ranged properties? It seems to me it belongs in response_fields or as its own query parameter, where one could imagine various ways to extend it to ask for specific subsets of a property.

The cartesian_site_positions field is an example of multidimensional data.
In a trajectory the first dimension would cover the frames, the second dimension the particles and the third the dimensions(x, y, z).
In a PDB file, the atoms belonging to the protein usually come first, while the solvent atoms come later. So one could make a query, selecting only a part of the range in the second dimension, so that only the positions of the atoms of the protein and not those of the solvent would be returned.

- :property:`nvalues`: integer (OPTIONAL)
- :property:`range_ids`: list of strings (OPTIONAL)
- :property:`n_dim`: integer (REQUIRED)
- :property:`dim_size`: list of integers (REQUIRED)
- :property:`offset_linear`: float (OPTIONAL)
- :property:`step_size_linear`: list of floats (OPTIONAL)
- :property:`offset_regular`: list of integers (OPTIONAL)
- :property:`step_size_regular`: list of integers (OPTIONAL)
- :property:`indexes`: list of lists of integers (OPTIONAL)

- **Requirements/Conventions**:

- **Support**: OPTIONAL support in implementations.
- Ranged properties can be identified by the prefix "_ranged_". If it is a database specific field, the prefix of the database comes first.
- If the part of the property name after the "_ranged_" prefix matches the name of a OPTIMADE field for the entry point, the values in the list of the :property:`values` MUST follow the rules of this property.
- By default, only the metadata SHOULD be returned (i.e. all the fields except :property:`values` and :property:`indexes`).
The :property:`values` and :property:`indexes` fields SHOULD only be returned when requested via the :query-param:`ranged property` as described under `Entry Listing URL Query Parameters`_.

- **Query**: Queries on the dictionary fields SHOULD be supported, except for the :property:`values` and :property:`indexes` fields for which querying is OPTIONAL.
- **Keys**:

- **serialization_format**: To improve the compactness of the data, there are several ways to show to which index a value belongs.
This is specified by the :property:`serialization_format`.

- **Type**: string

- **Requirements/Conventions**: This field MUST be present.
- **Possible values**:

- **linear**: The value is a linear function of the indexes.
This function is defined by :property:`offset_linear` and :property:`step_size_linear`.
- **regular**: The value is set for one out of every :property:`step_size_sparse` indexes, with :property:`offset_sparse` indicating the index of the first value.
- **custom**: A separate list with indexes is defined in the field :property:`indexes` to indicate to which index each value belongs.

- **values**: The values belonging to this property.
The format of this field depends on the property for which data is stored.

- **Type**: List of Any
- **Requirements/Conventions**: The property :property:`values` MUST be present when :property:`serialization_format` is not set to :val:`"linear"`.

- **nvalues**: The number of values in the field :property:`values`.

- **Type**: integer
- **Requirements/Conventions**: The value MUST be present when :property:`serialization_format` is not set to :val:`"linear"`.

- **range_ids**: A list with an identifier for each dimension of the range. It shows that that dimension correlates to the same dimension of another range.
For example, when data of an MD trajectory is shared, it could be used to indicate that the energies and the cartesian_site_positions of the index in a certain dimension are correlated. i.e. which energy belongs to which set of cartesian_site_positions.

- **Type**: list of strings
- **Requirements/Conventions**: OPTIONAL

- **n_dim**: The number of dimensions this property has.
- **Type**: integer
- **Requirements/Conventions**: REQUIRED
It MUST be equal to the length of the dim_size field.

- **dim_size**: The dimensions of the range in each dimension.

- **Type**: list of integers
- **Requirements/Conventions**: REQUIRED

- **offset_linear**: If :property:`serialization_format` is set to :val:`"linear"` this property gives the value at the origin, i.e. where the index in all dimensions is 1.

- **Type**: float
- **Requirements/Conventions**: The value MAY be present when :property:`serialization_format` is set to :val:`"linear"`, otherwise the value SHOULD NOT be present.
The default value is 0 in each dimension.

- **step_size_linear**: If :property:`serialization_format` is set to :val:`"linear"`, this value gives the change in the value of the property per step along each of the dimensions of the range.
e.g. If the value :property:`offset_linear` = 0.5 and the value of :property:`step_size_linear` = [0.2,0.3] than at index[3,4] the value of the property will be 1.8.

- **Type**: list of float
- **Requirements/Conventions**: The value MUST be present when :property:`serialization_format` is set to "linear".
Otherwise, it SHOULD NOT be present.

- **offset_regular**: If :property:`serialization_format` is set to :val:`"regular"` this property gives the indexes of the first value.

- **Type**: list of integers
- **Requirements/Conventions**: The value MAY be present when :property:`serialization_format` is set to :val:`"regular"`, otherwise the value SHOULD NOT be present.
The default value is 1 in every dimension.

- **step_size_regular**: If :property:`serialization_format` is set to :val:`"regular"`, this value indicates that a value is defined one out of every :property:`step_size_regular` steps in each dimension.

- **Type**: list of integers
- **Requirements/Conventions**: The value MUST be present when :property:`serialization_format` is set to :val:`"regular"`.
Otherwise, it SHOULD NOT be present.

- **indexes**: If :property:`serialization_format` is set to :val:`"custom"`, this field holds the indexes to which the values in the value field belong.

- **Type**: List of lists of integers
- **Requirements/Conventions**: The value MUST be present when :property:`serialization_format` is set to "custom".
Otherwise, it SHOULD NOT be present. The order of the values must be the same as those in :property:`values`.

- **Example**:

.. code:: jsonc
Comment thread
JPBergsma marked this conversation as resolved.
Outdated

{
"_ranged_cartesian_site_positions": {
"n_dim": 3,
"dim_size": [100, 3, 3],
"range_ids": ["mdsteps","particles","xyz"],
"serialization_format": "regular",
"offset_regular": [1, 1, 1],
"step_size_regular": [1, 1, 1],
"nvalues": 900,
"values": [[[2.36, 5.36, 9.56],[7.24, 3.58, 0.56],[8.12, 6.95, 4.56]],
[[2.38, 5.37, 9.56],[7.24, 3.57, 0.58],[8.11, 6.93, 4.58]],
[[2.39, 5.38, 9.55],[7.23, 3.57, 0.59],[8.10, 6.93, 4.57]],
// ...
],
Comment thread
JPBergsma marked this conversation as resolved.
Outdated
},
"_ranged_species_at_sites": {
"n_dim": 1,
"dim_size": [3],
"range_ids": ["particles"],
"serialization_format": "regular",
"offset_regular": [0],
"step_size_regular": [1],
"nvalues": 3,
"values": ["He", "Ne", "Ar"],
Comment thread
JPBergsma marked this conversation as resolved.
Outdated
},
"_exmpl_ranged_time":{
"n_dim": 1,
"dim_size": [100],
"range_ids": ["mdsteps"],
"serialization_format": "linear",
"step_size_linear": 0.2,
Comment thread
JPBergsma marked this conversation as resolved.
Outdated
},
"_exmpl_ranged_thermostat": {
"n_dim": 1,
"dim_size": [100],
"range_ids": ["mdsteps"],
"serialization_format": "custom",
"nvalues": 3,
"values": [20, 40, 60]
Comment thread
JPBergsma marked this conversation as resolved.
Outdated
"indexes": [[0], [20], [80]]
}
}



Responses
=========

Expand Down Expand Up @@ -880,6 +1040,25 @@ Standard OPTIONAL URL query parameters not in the JSON API specification:
If provided, these fields MUST be returned along with the REQUIRED fields.
Other OPTIONAL fields MUST NOT be returned when this parameter is present.
Example: :query-url:`http://example.com/optimade/v1/structures?response_fields=last_modified,nsites`
- **property\_ranges**: specifies which ranges should be returned for ranged properties.
It MUST be supported by databases having ranged properties.
It consists of a property name directly followed by the range that should be returned.
A range is written as a list with a list for each dimension.
Each dimensions list has three values.
The first value of the range specifies the first index in that dimension for which values should be returned.
The second value specifies the last index for which values should be returned.
The third value specifies the step size.
Ranges can be specified for multiple properties by separating them with a comma.
Comment thread
JPBergsma marked this conversation as resolved.
Outdated
Databases MUST return the values belonging to properties listed and SHOULD use the ranges in this query parameter.
For properties with :property:`serialization_format` :val:`custom` indexes that fall in the requested range but for which there is no value defined should not be returned.
For properties with :property:`serialization_format` :val:`regular` indexes that fall in the requested range but for which there is no value defined should have the value :val:`null`.
The ranges are 1 based, i.e. the first value has index 1, and inclusive i.e. for a the range :val:`[10,20,1]` the last value returned belongs to index 20.
Comment thread
JPBergsma marked this conversation as resolved.
Outdated
Example:

If there would be a structure with id: id_12345 and a property :ranged-property:`_ranged_test_field` with the values :val:`[[9.64, 7.52, 0.69, 5.69], [4.82, 8.35, 3.26, 3.25], [4.82, 2.78, 7.87, 7.42], [5.49, 3.48, 1.65, 0.75]` the query: :query-url:`http://example.com/optimade/v1/structures/id_12345?property_ranges=_ranged_test_field[[1, 3, 2], [2, 3, 1]]`
will return the value: :val:`[[7.52, 0.69], [2.78, 7.87]]`
Multiple ranges can be requested in one query. e.g. :query-param:`property_ranges=_ranged_test_field[[1, 3, 2], [2, 3, 1]], _ranged_other_field[[1,100,1]]`


Additional OPTIONAL URL query parameters not described above are not considered to be part of this standard, and are instead considered to be "custom URL query parameters".
These custom URL query parameters MUST be of the format "<database-provider-specific prefix><url\_query\_parameter\_name>".
Expand Down