Skip to content

Latest commit

 

History

History
56 lines (46 loc) · 2.96 KB

data-flow.md

File metadata and controls

56 lines (46 loc) · 2.96 KB

Data flow

Ingestion

flowchart LR
    Sources["sources.json"] --> Collect["Fetch and map feeds"]
    Feeds["Upstream result files"] --> Collect
    Releases["GitHub releases and binary assets"] --> Collect
    Previous["Last published dataset"] --> Collect
    Collect --> Evidence["Associate ROCm packages"]
    Evidence --> Dataset["Versioned dataset"]
    Dataset -->|Main runs| Publish["Publish data/"]
    Collect --> Artifact["CI artifact"]
    Dataset --> Artifact
Loading
Input What it is and where it comes from
sources.json Configuration maintained here: source repositories, branches, files, adapters, and filters. Configuration guide.
Upstream result files llama.cpp's qualification index and Lemonade's benchmark leaderboard. Retained fields.
Last published dataset Our saved JSON from the last successful publication, restored using the latest.json pointer. It supplies the prior release inventory and upstream health history.

The JSON reference documents file contracts, field lineage, and pinned mapping examples.

The output is the published data directory, including its latest.json pointer.

PRs validate a fresh collection from producer feeds and upload an artifact. Main runs publish successful datasets. Failed runs update live latest.json and sync.json with the error while retaining the last good dataset pointer. CI artifacts contain source copies and diagnostics and are retained for 30 days. Producer build history for the selected tracks is grouped by calendar month.

llama.cpp reads stable builds from lemonade-sdk/llama.cpp and their stable ROCm model results from Lemonade's benchmark feed. See data pipeline for sources, adapters, channels, and how to add a feed.

Published data

flowchart LR
    Latest["latest.json"] -->|index_url| Index["Pinned index.json"]
    Index -->|inventory path| Inventory["Release inventory"]
    Index -->|overview path| Overview["ROCm overview"]
    Overview -->|history paths| History["history/YYYY-MM.json"]
    History -->|report_url| Report["Producer report"]
    Overview -->|performance path| Performance["Model measurements"]
Loading

index.default_version identifies the default ROCm version. Overviews contain latest build and performance summaries and links to their history. See file definitions and history grouping.