🇲🇾 Open Data Scraper Malaysia

A lightweight, extensible toolkit that scrapes publicly available Malaysian government data — fuel prices, foreign exchange rates, and weather warnings — and stores it as clean, versioned JSON. Built for developers, researchers, and hobbyists who want ready-to-use Malaysian open data without wrestling with inconsistent APIs.

Data refreshes automatically every day via GitHub Actions, so the data/ folder in this repo is always up to date.

Why this exists

Malaysian government agencies publish useful public data (fuel prices, exchange rates, weather warnings), but each source has its own API shape, quirks, and reliability. This project normalizes them into one consistent interface, with cached snapshots committed to the repo so you can consume the data without hitting the upstream APIs yourself.

Data sources

Source	Provider	Description
fuel_price	data.gov.my	Weekly retail prices for RON95, RON97, and Diesel
exchange_rate	Bank Negara Malaysia	Daily MYR buying/selling/middle rates against major currencies
weather_warning	data.gov.my (MET Malaysia)	Active weather warnings (strong wind, rough seas, thunderstorms, etc.)

Scraped output lands in data/ as one JSON file per source (e.g. data/fuel_price.json).

Installation

git clone https://github.com/mohdsyahid/open-data-scraper-malaysia.git
cd open-data-scraper-malaysia
python -m venv .venv
source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -r requirements.txt

Usage

Run every scraper and write JSON to data/:

python main.py

Run a specific source only:

python main.py --source fuel_price
python main.py --source exchange_rate --source weather_warning

Write to a custom directory, with verbose logging:

python main.py --output /tmp/msia-data --verbose

Using it as a library

from scrapers import FuelPriceScraper

scraper = FuelPriceScraper()
records = scraper.fetch()  # list[dict]
print(records[-1])
# {'date': '2026-08-01', 'ron95': 2.05, 'ron97': 3.47, 'diesel': 2.15, ...}

Project structure

open-data-scraper-malaysia/
├── main.py                  # CLI entrypoint
├── scrapers/
│   ├── base.py               # Shared fetch/save pipeline
│   ├── fuel_price.py
│   ├── exchange_rate.py
│   └── weather_warning.py
├── data/                     # Scraped JSON snapshots (auto-updated daily)
├── tests/                    # pytest suite with mocked HTTP responses
└── .github/workflows/
    ├── ci.yml                 # Lint + test on every push/PR
    └── scrape.yml              # Daily scheduled scrape + auto-commit

Adding a new scraper

1. Create scrapers/your_source.py with a class extending BaseScraper.
2. Implement fetch() to return a list of records.
3. Register it in scrapers/__init__.py and main.py's SCRAPERS dict.
4. Add tests in tests/test_scrapers.py using the responses library to mock HTTP calls.

from .base import BaseScraper

class YourSourceScraper(BaseScraper):
    name = "your_source"

    def fetch(self):
        response = self._get("https://api.example.gov.my/endpoint")
        return response.json()

Development

pip install -r requirements-dev.txt
ruff check .        # lint
pytest -v           # run tests

Automation

A scheduled GitHub Actions workflow runs daily at 09:00 MYT, re-scrapes every source, and commits any changes back to data/. You can also trigger it manually from the Actions tab (workflow_dispatch).

Contributing

Contributions are welcome — new data sources, bug fixes, or better error handling all help. Please open an issue first for larger changes, and make sure ruff check . and pytest pass before submitting a PR.

Disclaimer

This project scrapes publicly available government data for convenience. It is not affiliated with, endorsed by, or officially connected to data.gov.my, Bank Negara Malaysia, or MET Malaysia. Always verify critical decisions (e.g. financial, safety) against the official source.

License

MIT