Skip to content

Download SARAH data via API - #539

Open
jome1 wants to merge 26 commits into
PyPSA:masterfrom
jome1:sarah_api
Open

jome1 wants to merge 26 commits into
PyPSA:masterfrom
jome1:sarah_api

Conversation

@jome1

@jome1 jome1 commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Closes #447. Assisted by AI.

Changes proposed in this Pull Request

Download SARAH data via API (EUMDAC).

Details:

  • Adds parallel product downloads, ZIP extraction, retries, and cleanup of temporary files in sarah.py.
    max. 15 parallel connections (see here
  • Adds the sarah_from_api and sarah_api_workers options to the cutout documentation in cutout.py.
  • Calls module cleanup after preparation, including when preparation raises an error, in data.py.
  • Adds the optional sarah-api dependency, mocked API tests, and a release-note entry in pyproject.toml, test_sarah_api.py, and RELEASE_NOTES.rst.
    I have no experience with test. The tests were completely created by AI.

How to use:

  1. Install additional dependency: conda install -c conda-forge eumdac
  2. create an EUMETSAT user account here. Find API credentials here. Setup the API credentials in Terminal using the following command: eumdac set-credentials ConsumerKey ConsumerSecret
    (more info here)
  3. use sarah_from_api=True in atlite.Cutout() when specifying cutout. For example:
cutout = atlite.Cutout(
    path="sarah_test.nc",
    module=["sarah"],
    sarah_from_api=True,
    sarah_api_workers=15,
    x=slice(31.6, 33.3),
    y=slice(28.3, 29.5),
    time=slice("2019-01-01", "2019-01-05"),
)

cutout.prepare()

ToDo

Checklist

  • Code changes are sufficiently documented; i.e. new functions contain docstrings and further explanations may be given in doc.
  • Unit tests for new features were added (if applicable).
  • Newly introduced dependencies are added to environment.yaml, environment_docs.yaml and setup.py (if applicable).
  • A note for the release notes doc/release_notes.rst of the upcoming release is included.
  • I consent to the release of this PR's code under the MIT license.

@jome1

jome1 commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

I think we should have an option to not delete the raw SARAH files, in case someone want to create cutouts for multiple countries.

@jome1

jome1 commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

FYI: SARAH-3 products cannot be filtered by region for download.

Answer from EUMETSAT User Service Helpdesk:
"Currently, Data Tailor does not support the customisation of SARAH-3 products, which is why downloading a subset for your region of interest is not possible.
Unfortunately, we cannot provide an estimated timeline for when this functionality will become available. However, we will record your request as an enhancement for future consideration."

@euronion

euronion commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator

I think we should have an option to not delete the raw SARAH files, in case someone want to create cutouts for multiple countries.

If we can't filter through the API by region, then yes, that would make sense.

jome1 and others added 19 commits November 16, 2026 11:58
…files and select user defined storage location
@jome1

jome1 commented Nov 16, 2026

Copy link
Copy Markdown
Contributor Author

It is maybe not the most elegant implementation, but it seems to work for me.
Below, you can find the different approaches how to use the SARAH API.
Maybe you can have a look when time allows @euronion?

  1. One-time download (raw SARAH files get deleted automatically after cutout creation)
cutout = atlite.Cutout(
    path="result.nc",
    module=["sarah"],
    sarah_from_api=True,  # Enable API
    x=slice(5, 10),
    y=slice(35, 40),
    time=slice("2020-01-01", "2020-01-04"),
)
cutout.prepare()
  1. Custom storage location for temporary raw SARAH files (raw SARAH files get automatically deleted)
cutout = atlite.Cutout(
    ...,
    sarah_from_api=True,
    sarah_dir="/data/my_sarah",
)
  1. store raw SARAH files for reuse (e.g. you want to create multiple cutouts for different countries using the same timeframe)
cutout = atlite.Cutout(
    ...,
    sarah_from_api=True,
    keep_sarah_files=True,  # Files saved to ~/.atlite/sarah_raw/
)
  1. Custom folder to store raw SARAH files for reuse (same as above, but user selects storage location)
cutout = atlite.Cutout(
    ...,
    sarah_from_api=True,
    sarah_dir="/data/my_sarah",
    keep_sarah_files=True,
)
  1. Use existing files (no API call; the "old" approach already implemented in atlite)
cutout = atlite.Cutout(
    ...,
    sarah_dir="/data/my_sarah",  # Just reads, no download
)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Use API to download SARAH data

2 participants