lt-asr-lm

Lithuanian ASR language-model corpus curation + n-gram builder + rescoring — a reusable toolkit for building high-quality LT language models from multiple text sources and using them to rescore ASR hypotheses from different acoustic models (Gemma 4, Parakeet, Whisper, …).

Built because Lithuanian ASR is my thing now and I'd rather curate the corpus once and reuse it across every future fine-tune than rebuild it per project.

Why a separate project

Over the course of doing Lithuanian ASR fine-tunes on different architectures, the LM-building work keeps coming up:

• Parakeet fine-tune: token-level 4-gram from LT Wikipedia → −2.83 pp WER (lt-full-20260411 run at 10.76% WER)
• Gemma 4 multimodal fine-tune: Wikipedia word-level 4-gram → (TBD)
• Future models: Whisper LT fine-tune, whatever comes next

Each project re-downloaded, re-cleaned, and re-counted the same corpus. That's silly. This repo owns that work so future projects can pip install -e . it and call a one-liner to get a rescoring LM.

What it does

1. Downloads Lithuanian text from multiple sources (Wikipedia, OpenSubtitles, CC-100, literature, existing ASR training manifests).
2. Normalizes every line the same way (Unicode NFC, lowercase, strip punctuation, preserve all letters including Lithuanian diacritics). Matches the eval normalizer used in our WER scripts so LM queries line up with scored hypotheses.
3. Deduplicates across sources (subtitle corpora especially).
4. Builds a word-level 4-gram n-gram LM with Stupid Backoff scoring (Brants et al. 2007). No kenlm/C++ dependency — pure Python, ~75 MB output for ~36M Lithuanian words.
5. Rescores ASR n-best hypotheses as a post-hoc step, so it works with any decoder that can produce candidate outputs — beam search, sampling, greedy-with-variants.
6. Evaluates LM quality via perplexity on held-out text (e.g. CV25 LT test transcripts), so we can compare "corpus A" vs "corpus A + B" without re-running ASR.

Quick start

git clone <this repo>
cd lt-asr-lm
python -m venv .venv && source .venv/bin/activate
pip install -e .

# Download and normalize each source individually:
lt-asr-lm download wikipedia
lt-asr-lm download opensubtitles
lt-asr-lm download cc100

# Build a 4-gram LM from everything we have:
lt-asr-lm build-lm --sources wikipedia,opensubtitles,cc100 \
    --out data/lm/lt_4gram.counts.gz

# Sanity check: perplexity on CV25 LT test reference transcripts
lt-asr-lm eval-ppl --lm data/lm/lt_4gram.counts.gz \
    --refs path/to/cv25_test_transcripts.txt

Sources (planned / implemented)

Source	Est. size (LT)	Status	Notes
Wikipedia (20231101.lt)	~3.4M sentences, 36M words	✅ implemented	encyclopedic register
OpenSubtitles (OPUS)	~30M+ sentences	⏳	transcribed speech — closest to ASR output
CC-100 LT	~600M words raw	⏳	large, web-scale, noisy
Project Gutenberg LT	~tens of books	🤔	formal literary — small
OSCAR LT	~100M words	🤔	cleaned CommonCrawl
LT news archives	varies	🚫 paywall	skip for now
Existing ASR manifests	~30K sentences	⏳	exact domain match — highest per-token value

Legend: ✅ implemented · ⏳ planned · 🤔 maybe · 🚫 out of scope

Design principles

• Additive. Each source is a separate downloader / cleaner. Adding a new source = one new file + one entry in the CLI. Nothing else breaks.
• Normalization is shared. One normalizer ([src/lt_asr_lm/normalize.py]) used by every source, every LM query, and every downstream rescorer. If the normalizer changes, everything is rebuilt together.
• No heavy deps. Pure Python: gzip text format for LM, Stupid Backoff for scoring. No kenlm, no Boost, no C++. Running this on macOS or WSL2 or a fresh cloud VM should never require a C++ toolchain dance.
• Reusable across fine-tune projects. Other projects (gemma-tuner-multimodal, finetuneparakeet, …) pip install -e this and call from lt_asr_lm.ngram import StupidBackoffLM for post-hoc rescoring.

Related

• gemma-tuner-multimodal — Gemma 4 LoRA fine-tune on LT
• finetuneparakeet — NVIDIA Parakeet fine-tune on LT