Skip to content

743: Store fallback XSD unzipped & make updating it easier #766

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Simon-Will wants to merge 4 commits into
base: develop
Choose a base branch
from
Open

743: Store fallback XSD unzipped & make updating it easier #766

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
d9360c4
743: Update fallback XSD
Simon-Will Jul 14, 2026
a6fda50
743: Create script for updating fallback XSD
Simon-Will Jul 14, 2026
be91440
743: Add instruction about updating fallback XSD to release template
Simon-Will Jul 14, 2026
ae792b8
743: Update changelog
Simon-Will Jul 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions .github/ISSUE_TEMPLATE/issue_template_release.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,18 @@ assignees: ''
## Preparation
- [ ] 🐙 Create a `Draft GitHub Release` with the release version number `vx.x.x` as title
- [ ] Merge all open Pull Requests to `develop`
- [ ] Update the fallback XSD directory:
* Update the directory and create a pull request. E.g.:
```bash
if python3 scripts/update_fallback_xsd.py; then
BRANCHNAME="update-fallback-xsd-$(date -Idate)"
git switch -c "$BRANCHNAME"
git add open_mastr/resources
git commit -m 'Update fallback XSD'
git push origin "$BRANCHNAME"
fi
```
* Check and merge the pull request
- [ ] Run tests locally with `pytest` and apply linting with `pre-commit run -a`
## Create a `release` branch
- [ ] Checkout `develop` and branch with `git checkout -b release-vx.x.x`
Expand Down
4 changes: 4 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@ and the versioning aims to respect [Semantic Versioning](http://semver.org/spec/

## [v0.xx.x] Unreleased - 202x-xx-xx
### Added
- Add script for updating fallback XSD
[#766](https://github.com/OpenEnergyPlatform/open-MaStR/pull/766)
- Add `Mastr.generate_data_model` method that downloads the current MaStR documentation and generates SQLAlchemy tables from the XSD definitions; supports `english=True` for English column names
[#718](https://github.com/OpenEnergyPlatform/open-MaStR/pull/718)
- Add `mastr_table_to_db_table` argument to `Mastr.download` to pass a custom database schema
Expand All @@ -22,6 +24,8 @@ and the versioning aims to respect [Semantic Versioning](http://semver.org/spec/


### Changed
- Change fallback XSD from zipped docs to plain directory with XSD files
[#766](https://github.com/OpenEnergyPlatform/open-MaStR/pull/766)
- Fix race condition in parallel bulk XML import that resulted in silent data loss
[#765](https://github.com/OpenEnergyPlatform/open-MaStR/pull/765)
- Add support for reading XSD files from both zipped / unzipped documentation sources
Expand Down
21 changes: 13 additions & 8 deletions open_mastr/mastr.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,12 +45,6 @@
# setup logger
log = setup_logger()

FALLBACK_DOCS_PATH = (
Path(__file__).parent
/ "resources"
/ "Dokumentation-MaStR-Gesamtdatenexport-20260216-Fallback.zip"
)


class Mastr:
"""`Mastr` is used to download the MaStR database.
Expand Down Expand Up @@ -195,12 +189,13 @@ def generate_data_model(
english=english,
)
except Exception as e:
fallback_docs_path = _get_fallback_xsd()
log.exception(
f"Encountered {e!r} when downloading or processing MaStR documentation."
f" Falling back to stored docs at {FALLBACK_DOCS_PATH}"
f" Falling back to stored docs at {fallback_docs_path}"
)
return _generate_data_model_from_downloaded_docs(
zipped_docs_file_path=FALLBACK_DOCS_PATH,
zipped_docs_file_path=fallback_docs_path,
data=data,
catalog_value_as_str=catalog_value_as_str,
metadata=metadata,
Expand Down Expand Up @@ -519,3 +514,13 @@ def _generate_data_model_from_downloaded_docs(
] = sqlalchemy_model

return mastr_table_to_db_table


def _get_fallback_xsd() -> Path:
fallback_xsds = list((Path(__file__).parent / "resources").glob("fallback-xsd-*"))
if not fallback_xsds:
raise RuntimeError("Cannot find any fallback XSD directory")
fallback_xsd = fallback_xsds[0]
if len(fallback_xsds) > 0:
log.warning(f"Found multiple fallback XSD directories. Using {fallback_xsd}")
return fallback_xsd
Binary file removed open_mastr/resources/Dokumentation-MaStR-Gesamtdatenexport-20260216-Fallback.zip
View file
Open in desktop
Binary file not shown.
96 changes: 96 additions & 0 deletions open_mastr/resources/fallback-xsd-20260712/AnlagenEegBiomasse.xsd
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema attributeFormDefault="unqualified" elementFormDefault="qualified" xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:element name="AnlagenEegBiomasse">
<xs:complexType>
<xs:sequence>
<xs:element name="AnlageEegBiomasse" maxOccurs="unbounded" minOccurs="1">
<xs:complexType>
<xs:sequence>
<xs:element type="xs:date" name="Registrierungsdatum" minOccurs="0"/>
<xs:element type="xs:dateTime" name="DatumLetzteAktualisierung"/>
<xs:element type="xs:date" name="EegInbetriebnahmedatum" minOccurs="0"/>
<xs:element type="xs:string" name="EegMaStRNummer"/>
<xs:element type="xs:string" name="AnlagenschluesselEeg" minOccurs="0"/>
<xs:element type="xs:string" name="AnlagenkennzifferAnlagenregister" minOccurs="0"/>
<xs:element name="AnlagenkennzifferAnlagenregister_nv">
<xs:simpleType>
<xs:restriction base="xs:byte">
<xs:enumeration value="0"/>
<xs:enumeration value="1"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
<xs:element type="xs:float" name="InstallierteLeistung" minOccurs="0"/>
<xs:element name="AusschliesslicheVerwendungBiomasse" minOccurs="0">
<xs:simpleType>
<xs:restriction base="xs:byte">
<xs:enumeration value="1"/>
<xs:enumeration value="0"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
<xs:element name="AusschreibungZuschlag" minOccurs="0">
<xs:simpleType>
<xs:restriction base="xs:byte">
<xs:enumeration value="0"/>
<xs:enumeration value="1"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
<xs:element type="xs:string" name="Zuschlagsnummer" minOccurs="0"/>
<xs:element name="BiogasInanspruchnahmeFlexiPraemie" minOccurs="0">
<xs:simpleType>
<xs:restriction base="xs:byte">
<xs:enumeration value="1"/>
<xs:enumeration value="0"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
<xs:element type="xs:date" name="BiogasDatumInanspruchnahmeFlexiPraemie" minOccurs="0"/>
<xs:element name="BiogasLeistungserhoehung" minOccurs="0">
<xs:simpleType>
<xs:restriction base="xs:byte">
<xs:enumeration value="0"/>
<xs:enumeration value="1"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
<xs:element type="xs:date" name="BiogasDatumLeistungserhoehung" minOccurs="0"/>
<xs:element type="xs:float" name="BiogasUmfangLeistungserhoehung" minOccurs="0"/>
<xs:element type="xs:float" name="BiogasGaserzeugungskapazitaet" minOccurs="0"/>
<xs:element name="BiogasGaserzeugungskapazitaet_nv">
<xs:simpleType>
<xs:restriction base="xs:byte">
<xs:enumeration value="0"/>
<xs:enumeration value="1"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
<xs:element type="xs:float" name="BiogasHoechstbemessungsleistung" minOccurs="0"/>
<xs:element type="xs:date" name="BiomethanErstmaligerEinsatz" minOccurs="0"/>
<xs:element name="BiomethanErstmaligerEinsatz_nv">
<xs:simpleType>
<xs:restriction base="xs:byte">
<xs:enumeration value="0"/>
<xs:enumeration value="1"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
<xs:element name="AnlageBetriebsstatus">
<xs:simpleType>
<xs:restriction base="xs:byte">
<xs:enumeration value="35"/>
<xs:enumeration value="38"/>
<xs:enumeration value="37"/>
<xs:enumeration value="31"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
<xs:element type="xs:string" name="VerknuepfteEinheitenMaStRNummern" minOccurs="0"/>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:schema>
41 changes: 41 additions & 0 deletions open_mastr/resources/fallback-xsd-20260712/AnlagenEegGeothermieGrubengasDruckentspannung.xsd
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema attributeFormDefault="unqualified" elementFormDefault="qualified" xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:element name="AnlagenEegGeothermieGrubengasDruckentspannung">
<xs:complexType>
<xs:sequence>
<xs:element name="AnlageEegGeothermieGrubengasDruckentspannung" maxOccurs="unbounded" minOccurs="0">
<xs:complexType>
<xs:sequence>
<xs:element type="xs:date" name="Registrierungsdatum" minOccurs="0"/>
<xs:element type="xs:dateTime" name="DatumLetzteAktualisierung"/>
<xs:element type="xs:date" name="EegInbetriebnahmedatum" minOccurs="0"/>
<xs:element type="xs:string" name="EegMaStRNummer"/>
<xs:element type="xs:string" name="AnlagenschluesselEeg" minOccurs="0"/>
<xs:element type="xs:string" name="AnlagenkennzifferAnlagenregister" minOccurs="0"/>
<xs:element name="AnlagenkennzifferAnlagenregister_nv">
<xs:simpleType>
<xs:restriction base="xs:byte">
<xs:enumeration value="0"/>
<xs:enumeration value="1"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
<xs:element type="xs:float" name="InstallierteLeistung" minOccurs="0"/>
<xs:element name="AnlageBetriebsstatus">
<xs:simpleType>
<xs:restriction base="xs:byte">
<xs:enumeration value="35"/>
<xs:enumeration value="38"/>
<xs:enumeration value="37"/>
<xs:enumeration value="31"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
<xs:element type="xs:string" name="VerknuepfteEinheitenMaStRNummern" minOccurs="0"/>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:schema>
78 changes: 78 additions & 0 deletions open_mastr/resources/fallback-xsd-20260712/AnlagenEegSolar.xsd
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema attributeFormDefault="unqualified" elementFormDefault="qualified" xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:element name="AnlagenEegSolar">
<xs:complexType>
<xs:sequence>
<xs:element name="AnlageEegSolar" maxOccurs="unbounded" minOccurs="0">
<xs:complexType>
<xs:sequence>
<xs:element type="xs:date" name="Registrierungsdatum" minOccurs="0"/>
<xs:element type="xs:dateTime" name="DatumLetzteAktualisierung"/>
<xs:element type="xs:date" name="EegInbetriebnahmedatum" minOccurs="0"/>
<xs:element type="xs:string" name="EegMaStRNummer"/>
<xs:element name="InanspruchnahmeZahlungNachEeg" minOccurs="0">
<xs:simpleType>
<xs:restriction base="xs:byte">
<xs:enumeration value="1"/>
<xs:enumeration value="0"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
<xs:element type="xs:string" name="AnlagenschluesselEeg" minOccurs="0"/>
<xs:element type="xs:string" name="AnlagenkennzifferAnlagenregister" minOccurs="0"/>
<xs:element name="AnlagenkennzifferAnlagenregister_nv">
<xs:simpleType>
<xs:restriction base="xs:byte">
<xs:enumeration value="1"/>
<xs:enumeration value="0"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
<xs:element type="xs:float" name="InstallierteLeistung" minOccurs="0"/>
<xs:element type="xs:string" name="RegistrierungsnummerPvMeldeportal" minOccurs="0"/>
<xs:element name="RegistrierungsnummerPvMeldeportal_nv">
<xs:simpleType>
<xs:restriction base="xs:byte">
<xs:enumeration value="0"/>
<xs:enumeration value="1"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
<xs:element name="MieterstromZugeordnet" minOccurs="0">
<xs:simpleType>
<xs:restriction base="xs:byte">
<xs:enumeration value="0"/>
<xs:enumeration value="1"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
<xs:element type="xs:date" name="MieterstromMeldedatum" minOccurs="0"/>
<xs:element type="xs:date" name="MieterstromErsteZuordnungZuschlag" minOccurs="0"/>
<xs:element name="AusschreibungZuschlag" minOccurs="0">
<xs:simpleType>
<xs:restriction base="xs:byte">
<xs:enumeration value="0"/>
<xs:enumeration value="1"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
<xs:element type="xs:float" name="ZugeordneteGebotsmenge" minOccurs="0"/>
<xs:element type="xs:string" name="Zuschlagsnummer" minOccurs="0"/>
<xs:element name="AnlageBetriebsstatus">
<xs:simpleType>
<xs:restriction base="xs:byte">
<xs:enumeration value="35"/>
<xs:enumeration value="38"/>
<xs:enumeration value="37"/>
<xs:enumeration value="31"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
<xs:element type="xs:string" name="VerknuepfteEinheitenMaStRNummern"/>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:schema>
30 changes: 30 additions & 0 deletions open_mastr/resources/fallback-xsd-20260712/AnlagenEegSpeicher.xsd
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema attributeFormDefault="unqualified" elementFormDefault="qualified" xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:element name="AnlagenEegSpeicher">
<xs:complexType>
<xs:sequence>
<xs:element name="AnlageEegSpeicher" maxOccurs="unbounded" minOccurs="0">
<xs:complexType>
<xs:sequence>
<xs:element type="xs:date" name="Registrierungsdatum" minOccurs="0"/>
<xs:element type="xs:dateTime" name="DatumLetzteAktualisierung"/>
<xs:element type="xs:date" name="EegInbetriebnahmedatum" minOccurs="0"/>
<xs:element type="xs:string" name="EegMaStRNummer"/>
<xs:element type="xs:string" name="AnlagenschluesselEeg" minOccurs="0"/>
<xs:element name="AusschreibungZuschlag" minOccurs="0">
<xs:simpleType>
<xs:restriction base="xs:byte">
<xs:enumeration value="1"/>
<xs:enumeration value="0"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
<xs:element type="xs:string" name="Zuschlagsnummer" minOccurs="0"/>
<xs:element type="xs:string" name="VerknuepfteEinheitenMaStRNummern" minOccurs="0"/>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:schema>
42 changes: 42 additions & 0 deletions open_mastr/resources/fallback-xsd-20260712/AnlagenEegWasser.xsd
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema attributeFormDefault="unqualified" elementFormDefault="qualified" xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:element name="AnlagenEegWasser">
<xs:complexType>
<xs:sequence>
<xs:element name="AnlageEegWasser" maxOccurs="unbounded" minOccurs="0">
<xs:complexType>
<xs:sequence>
<xs:element type="xs:date" name="Registrierungsdatum" minOccurs="0"/>
<xs:element type="xs:dateTime" name="DatumLetzteAktualisierung"/>
<xs:element type="xs:date" name="EegInbetriebnahmedatum" minOccurs="0"/>
<xs:element type="xs:string" name="EegMaStRNummer"/>
<xs:element type="xs:string" name="AnlagenschluesselEeg" minOccurs="0"/>
<xs:element type="xs:string" name="AnlagenkennzifferAnlagenregister" minOccurs="0"/>
<xs:element name="AnlagenkennzifferAnlagenregister_nv">
<xs:simpleType>
<xs:restriction base="xs:byte">
<xs:enumeration value="0"/>
<xs:enumeration value="1"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
<xs:element type="xs:float" name="InstallierteLeistung" minOccurs="0"/>
<xs:element name="AnlageBetriebsstatus">
<xs:simpleType>
<xs:restriction base="xs:byte">
<xs:enumeration value="35"/>
<xs:enumeration value="38"/>
<xs:enumeration value="31"/>
<xs:enumeration value="37"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
<xs:element type="xs:string" name="ErtuechtigungIds" minOccurs="0"/>
<xs:element type="xs:string" name="VerknuepfteEinheitenMaStRNummern" minOccurs="0"/>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:schema>
Loading
Loading