Persian ASR Triple Threat 🔥 Benchmark

A same-test leaderboard for Persian speech recognition. Two real-world sets — VisualEars6669 (6,669-row / 10.49h noisy benchmark with clean, farfield & obstructed conditions) and FLEURS-fa Full (4,341-row public Persian FLEURS) — now scored three ways:

WER word error CER character error S³ ShenavaSanj Score — semantic, DHH-weighted 🔑 Ess Err essential-content error rate

S³ is a Persian-first Deaf/hard-of-hearing-focused semantic metric: it weights every ASR error by how much that word matters for understanding, so dropping a keyword costs far more than a filler. 🔑 Ess Err is the error rate on only the most meaning-critical words (importance ≥ 0.8) — the words a DHH reader can least afford to lose. Triple Threat 🔥 = 60% S³ + 20% WER + 20% CER. Each pillar is averaged 50/50 across VisualEars6669 and FLEURS-fa Full, so the two dataset splits remain equally weighted. Fair metrics ignore punctuation, spaces, half-spaces and diacritics, and normalize numbers (spelled-out ↔ digits) so a correctly-heard «صد» isn't penalized against «۱۰۰». Lower is better. 🔒 marks closed paid APIs. S³ is undergoing validation with Persian DHH annotators.

🔥 Best Triple Threat 🥇 gemini-3.5-flash · 3.01 60% S³ · 20% WER · 20% CER
🧠 Best S³ (semantic, DHH-weighted) 🥇 gemini-3.5-flash · 3.06 lower = fewer meaning-critical errors

Sortable leaderboard — sorted by Triple Threat

Sortable leaderboard — sorted by Triple Threat
1

handy-computer/nemotron-3.5-asr-streaming-0.6b-gguf
0.115
109.02
10.21
10.55
115.78
22.44
115.43
130.32
1916.5
Closed paid API / Google Gemini Transcribe
diagnostic
OpenRouter dated request google/gemini-3.5-flash-lite-20260721, Google AI Studio only, no provider fallback. Full 11010 cached clips on Stallion. Verbatim ASR training prompt, high reasoning, temperature 0, seed 20261107, max_tokens 16384. Prompt/effort chosen exploratorily from 32 fixed clips x 4 settings; all 32 selected-arm pilot outputs reused without best-of selection, and pilot-excluded metrics attached. Unchanged score_muse.py and cached weights, exact Muse parity verified; spacing/compound scorer caveat same as recent API lanes. Empty outputs retained. Decode is mean successful HTTP request wall time including upload/provider/network, excluding queue/backoff, with concurrent requests; not local GPU or streaming decode. Safety diagnostic: selected prompt hallucinated Persian words on all four all-zero silence durations (1/5/15/30 seconds), reproduced on repeat; speech consistency is not correctness. Benchmark alone does not establish safe unattended 1000-hour labeling. FLEURS training exposure for this proprietary model is unknown. Receipt: gemini35_flash_lite_scores.json. Empty scored outputs: 5 (provider policy blocks: 3). Compound-aware diagnostic Triple Threat 6.4828 is attached and not used for rank.