feat(exercise-world): package Nacre and seed MISP #211
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| #!/usr/bin/env bash | ||
| # Refresh image-bundled MISP galaxy definitions in the persistent app/files | ||
| # volume. Docker initializes a new named volume from the image once, but does | ||
| # not update an existing volume when a newer image is deployed. | ||
| set -euo pipefail | ||
|
|
||
| SEED_DIR="${MISP_GALAXY_SEED_DIR:-/opt/misp-galaxy}" | ||
| TARGET_DIR="${MISP_GALAXY_TARGET_DIR:-/var/www/MISP/app/files/misp-galaxy}" | ||
| OWNER="${MISP_GALAXY_OWNER-www-data:www-data}" | ||
|
|
||
| log() { echo "[refresh-galaxy-files] $*"; } | ||
|
|
||
| if [ ! -d "${SEED_DIR}/clusters" ] || [ ! -d "${SEED_DIR}/galaxies" ]; then | ||
| log "ERROR: bundled galaxy seed is incomplete at ${SEED_DIR}" | ||
| exit 1 | ||
| fi | ||
|
|
||
| log "Refreshing bundled galaxy definitions in ${TARGET_DIR} …" | ||
| mkdir -p "${TARGET_DIR}" | ||
| cp -a "${SEED_DIR}/." "${TARGET_DIR}/" | ||
|
|
||
| if [ -n "${OWNER}" ]; then | ||
| chown -R "${OWNER}" "${TARGET_DIR}" | ||
| fi | ||
|
|
||
| log "Bundled galaxy definitions are current." |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,112 @@ | ||
| #!/usr/bin/env python3 | ||
| """Fail provisioning unless the pinned Synthetic Exercise World is loaded.""" | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import json | ||
| import os | ||
| import ssl | ||
| import sys | ||
| from pathlib import Path | ||
| from typing import Any | ||
| from urllib.request import Request, urlopen | ||
|
|
||
|
|
||
| EXPECTED_GALAXY_UUID = "3c3de5f0-5982-4c7f-88cf-8abf43b8d6c1" | ||
| EXPECTED_COLLECTION_UUID = "7d6d7f2f-b3d4-4bc5-9f27-43e12f7f4658" | ||
| EXPECTED_ENTITY_COUNT = 60 | ||
|
|
||
|
|
||
| class ExerciseWorldVerificationError(ValueError): | ||
| pass | ||
|
|
||
|
|
||
| def _list_items(payload: Any) -> list[Any]: | ||
| if isinstance(payload, list): | ||
| return payload | ||
| if not isinstance(payload, dict): | ||
| return [] | ||
| for key in ("response", "Galaxies", "galaxies"): | ||
| if isinstance(payload.get(key), list): | ||
| return payload[key] | ||
| if isinstance(payload.get("Galaxy"), list): | ||
| return payload["Galaxy"] | ||
| return [payload] | ||
|
|
||
|
|
||
| def find_exercise_world_galaxy(payload: Any) -> str: | ||
| for item in _list_items(payload): | ||
| galaxy = item.get("Galaxy", item) if isinstance(item, dict) else {} | ||
| if galaxy.get("type") != "exercise-world": | ||
| continue | ||
| if galaxy.get("uuid") != EXPECTED_GALAXY_UUID: | ||
| raise ExerciseWorldVerificationError( | ||
| f"exercise-world galaxy UUID mismatch: {galaxy.get('uuid')}" | ||
| ) | ||
| galaxy_id = galaxy.get("id") | ||
| if galaxy_id is None: | ||
| raise ExerciseWorldVerificationError("exercise-world galaxy is missing its database id") | ||
| return str(galaxy_id) | ||
| raise ExerciseWorldVerificationError("exercise-world galaxy is not loaded") | ||
|
|
||
|
|
||
| def verify_cluster_payload(payload: Any) -> None: | ||
| root = payload.get("Galaxy", payload) if isinstance(payload, dict) else {} | ||
| clusters = payload.get("GalaxyCluster") if isinstance(payload, dict) else None | ||
| if not isinstance(clusters, list) and isinstance(root, dict): | ||
| clusters = root.get("GalaxyCluster") | ||
| if not isinstance(clusters, list): | ||
| raise ExerciseWorldVerificationError("exercise-world response has no GalaxyCluster list") | ||
|
|
||
| normalized = [ | ||
| item.get("GalaxyCluster", item) if isinstance(item, dict) else {} | ||
| for item in clusters | ||
| ] | ||
| if len(normalized) != EXPECTED_ENTITY_COUNT: | ||
| raise ExerciseWorldVerificationError( | ||
| f"exercise-world entity count mismatch: expected {EXPECTED_ENTITY_COUNT}, got {len(normalized)}" | ||
| ) | ||
|
|
||
| entity_uuids = {item.get("uuid") for item in normalized} | ||
| if None in entity_uuids or len(entity_uuids) != EXPECTED_ENTITY_COUNT: | ||
| raise ExerciseWorldVerificationError("exercise-world entity UUIDs are missing or duplicated") | ||
| collection_uuids = {item.get("collection_uuid") for item in normalized} | ||
| if collection_uuids != {EXPECTED_COLLECTION_UUID}: | ||
| raise ExerciseWorldVerificationError( | ||
| f"exercise-world collection UUID mismatch: {sorted(str(value) for value in collection_uuids)}" | ||
| ) | ||
|
|
||
|
|
||
| def misp_get(base_url: str, auth_key: str, path: str) -> Any: | ||
| request = Request( | ||
| f"{base_url.rstrip('/')}{path}", | ||
| headers={"Authorization": auth_key, "Accept": "application/json"}, | ||
| ) | ||
| context = ssl._create_unverified_context() | ||
| with urlopen(request, context=context, timeout=30) as response: | ||
| return json.load(response) | ||
|
|
||
|
|
||
| def main() -> int: | ||
| base_url = os.environ.get("MISP_URL", "https://misp") | ||
| key_file = Path(os.environ.get("MISP_ADMIN_KEY_FILE", "/keys/admin-authkey")) | ||
| try: | ||
| auth_key = key_file.read_text(encoding="utf-8").strip() | ||
| if not auth_key: | ||
| raise ExerciseWorldVerificationError(f"empty admin auth key: {key_file}") | ||
| galaxy_id = find_exercise_world_galaxy(misp_get(base_url, auth_key, "/galaxies/index.json")) | ||
| verify_cluster_payload(misp_get(base_url, auth_key, f"/galaxies/view/{galaxy_id}.json")) | ||
| except (OSError, json.JSONDecodeError, ExerciseWorldVerificationError) as exc: | ||
| print(f"[verify-exercise-world] ERROR: {exc}", file=sys.stderr) | ||
| return 1 | ||
|
|
||
| print( | ||
| f"[verify-exercise-world] verified galaxy {EXPECTED_GALAXY_UUID} " | ||
| f"with {EXPECTED_ENTITY_COUNT} entities", | ||
| file=sys.stderr, | ||
| ) | ||
| return 0 | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| raise SystemExit(main()) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| __pycache__/ | ||
| *.py[cod] |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When upgrading an existing Compose deployment, the
misp-filesnamed volume mounted over/var/www/MISP/app/filesindocker-compose.yml:71-73retains the previous v2.5.37 contents and masks the v2.5.44 files checked during the image build. Consequently,provision-content.sh:64-65imports the old bundled galaxies and this newly mandatory verifier exits before sample-event provisioning because Nacre is absent. Copy or synchronize the new bundled galaxy data into the persistent volume before updating and verifying it.Useful? React with 👍 / 👎.