Persian ASR Triple Threat 🔥 Benchmark

A same-test leaderboard for Persian speech recognition. Two real-world sets — VisualEars6669 (6,669-row / 10.49h noisy benchmark with clean, farfield & obstructed conditions) and FLEURS-fa Full (4,341-row public Persian FLEURS) — now scored three ways:

WER word error CER character error S³ ShenavaSanj Score — semantic, DHH-weighted 🔑 Ess Err essential-content error rate

S³ is a Persian-first Deaf/hard-of-hearing-focused semantic metric: it weights every ASR error by how much that word matters for understanding, so dropping a keyword costs far more than a filler. 🔑 Ess Err is the error rate on only the most meaning-critical words (importance ≥ 0.8) — the words a DHH reader can least afford to lose. Triple Threat 🔥 = 60% S³ + 20% WER + 20% CER. Each pillar is averaged 50/50 across VisualEars6669 and FLEURS-fa Full, so the two dataset splits remain equally weighted. Fair metrics ignore punctuation, spaces, half-spaces and diacritics, and normalize numbers (spelled-out ↔ digits) so a correctly-heard «صد» isn't penalized against «۱۰۰». Lower is better. 🔒 marks closed paid APIs. S³ is undergoing validation with Persian DHH annotators.

🔥 Best Triple Threat 🥇 gemini-3.5-flash · 3.01 60% S³ · 20% WER · 20% CER
🧠 Best S³ (semantic, DHH-weighted) 🥇 gemini-3.5-flash · 3.06 lower = fewer meaning-critical errors

Sortable leaderboard — sorted by Triple Threat

Sortable leaderboard — sorted by Triple Threat
1

handy-computer/nemotron-3.5-asr-streaming-0.6b-gguf
0.115
109.02
10.25
10.55
115.78
22.44
115.43
130.32
1916.5
Closed paid API / Google Gemini Transcribe
diagnostic
COMPOUND-FAIR re-score: WER & S³ & Ess under +both normalization (ZWNJ/compound-insensitive + homophone folding), applied uniformly to every model; CER is fair_text. Full g6669 (6669) + FLEURS (4341). Gemini & Chrome per-clip freshly regenerated (Gemini via OpenRouter reasoning=medium). | Google Chrome Web Speech API capture at locale fa-IR using BlackHole virtual audio routing. Full Golden6669 run: original ws6669_* capture overlaid with wsretry78_* retry rows; 78 retry rows, 41 blank-to-text recoveries, 37 genuine blank results remain. Scored with the official Persian normalizer on all 6,669 clips. Online-only browser/cloud recognition; no local weights. Decode: 1×RT (real-time streaming) — finalizes during playback; measured per-clip wall ~7.3s/6669, ~13.6s/FLEURS. | Params encoded as 0.0 because this is a closed browser/cloud API; decode encoded as 1000 ms/sec to avoid nulls while preserving the existing 1×RT provenance.