Wrong words, per 100
In code-switched speech, stock Parakeet gets about 11 words in every 100 wrong (11.3%). Hebrish v1.0 gets about 5 (5.1%).
One square is one word; coloured squares are mistakes.
Benchmarks
How well it gets the Hebrew words right
Hebrish is scored two ways: on a frozen public benchmark that any speech model can be run against, and on the held-out test split of its own dataset. The latest round, on 2026-10-19, scored v1.0 and the Nemotron streaming fine-tune. It cost about 50 cents of GPU time.
73.3%Hebrew terms written exactly right by v1.0 (public benchmark)
70.1%The same, for terms it never saw in training
3.3%Word error rate on the benchmark (stock Parakeet: 9.5%)
2.2%Word error rate on plain English: no worse than stock (2.7%)
White paper
How v1.0 was trained, exported and benchmarked, with every figure on this page: Hebrish v1.0 white paper (PDF, 9 pages). CC-BY-4.0.
One speaker's voice. Every test clip, like every training clip, is the same person reading aloud. These numbers say how well Hebrish learned that voice. How it does on yours is untested: tell us.
Public benchmark
A frozen test set of 84 clean clips with 172 Hebrew term occurrences, scored the same way for every model. It never changes in place, so every row compares with every other. Hebrish models are highlighted.
Accuracy against mistakes
The same ten models on one map. Further right means more Hebrew terms written exactly right; lower means fewer wrong words overall. Every Hebrish fine-tune sits in the bottom-right corner.
Show as a table
| Model | Term recall | Unseen-term recall | WER | English control WER |
|---|---|---|---|---|
| Hebrish v1.0 | 73.3% | 70.1% | 3.3% | 2.2% |
| Hebrish PoC-3 | 67.4% | 63.2% | 4.0% | 2.5% |
| Hebrish PoC-2 | 65.7% | 60.7% | 4.2% | 2.4% |
| Hebrish PoC-1 | 62.2% | 61.5% | 4.6% | 2.5% |
| Hebrish Nemotron streaming (offline) | 62.2% | 54.7% | 4.7% | 2.5% |
| Whisper Hebrish v1 some overlap | 54.1% | 56.4% | 6.5% | 4.7% |
| Whisper Large V3 | 45.9% | 46.2% | 7.5% | 2.4% |
| Whisper Large V3 Turbo | 40.1% | 39.3% | 8.9% | 2.4% |
| Parakeet TDT 0.6B v2 (stock) | 32.6% | 32.5% | 9.5% | 2.7% |
| Parakeet TDT 0.6B v3 (stock) | 31.4% | 31.6% | 10.5% | 2.7% |
Test set v1, clean subset. Unseen-term recall counts 117 occurrences of 88 terms that appear nowhere in Hebrish's training data. Known issue: 22 of the 139 term labels are English tech words, which makes term recall 2 to 6 points higher than a Hebrew-only count. Leaderboard · test set · runner and scorer.
Held-out test split
The test split of the code-switching dataset: 387 clips at revision 550bfcaa, with 293 occurrences of held-out Hebrew terms that are never in training. Stock Parakeet and PoC-3 were re-scored on the same clips, because the split grows with the dataset and only same-revision numbers compare.
Hebrew words written right, per 100
For Hebrew words it heard in training, stock Parakeet writes 28 in every 100 in the exact spelling. Hebrish v1.0 writes 73.
One square is one Hebrew word; coloured squares are spelled right.
Stock Parakeet to v1.0, test by test
All four models
Show as a table
| Test | Stock v2 | PoC-3 | v1.0 | Nemotron |
|---|---|---|---|---|
| Code-switching WER | 11.3% | 6.0% | 5.1% | 5.8% |
| Held-out Hebrew terms correct | 25.9% | 45.7% | 47.8% | 44.4% |
| Test terms also seen in training, correct | 27.7% | 63.3% | 72.7% | 68.0% |
| Own English WER | 3.1% | 2.3% | 1.7% | 3.3% |
| LibriSpeech test-clean WER | 1.2% | 1.0% | 1.2% | 2.0% |
| Long-form WER (15 recordings) | 9.6% | 5.8% | 5.0% | 5.8% |
Normalised WER (lower case, no punctuation). On held-out terms, v1.0's gain over PoC-3 (+2.1 points, about 6 of 293) is within noise; its gain on terms seen in training (+9.4) is not. English did not get worse: own English improved and LibriSpeech stayed level. That is the "forgetting" check a release has to pass.
Every version, older test split
How the proof-of-concept versions compared when they were first scored, on the earlier and smaller test split (revision b47310621f79, 219 clips). PoC-4 and PoC-5 were never benchmarked.
Hollow points are targets for the next checkpoints, not results. v1.0 is not on this chart because it was scored on the newer split; its numbers are above.
How it is run
- On rented GPUs. Each model is scored in its own job on one NVIDIA L40S on Modal. The 2026-10-19 round ran six jobs in parallel, about two and a half minutes each, for about 50 cents in all.
- Only what is needed. A new checkpoint is scored with stock Parakeet and the last benchmarked release on the same clips. The public benchmark is frozen, so older rows never need re-running.
- Contamination check. Before a result is published, the benchmark's 145 clip IDs and texts are checked against the model's training manifest. For v1.0 and Nemotron none were found.
- Term matching is whole-word and case-insensitive, against the one canonical spelling in the lexicon. A near miss counts as wrong.
Known limits
- One speaker, read speech, mostly one phone microphone.
- Two terms on the held-out list (mandatim, Beit Safafa) also appear in a few training sentences. If they were trained on, a handful of "unseen" hits are not unseen. Being checked.
- Nemotron's public-benchmark row is scored offline. On the held-out split it was also scored while streaming, at 560 ms chunks: 5.65% code-switching WER against 5.81% offline, so streaming cost nothing measurable. Smaller chunks (80 and 160 ms) are not measured.
Figures to reuse
The same results as static images, free to reuse with credit (CC-BY-4.0). PNG at 1600 px; each also exists as SVG at the same address ending in .svg.
Public benchmark: term recall · all ten models
Public benchmark: WER · all ten models
Held-out split · four models
Forgetting check · plain English, stock vs v1.0
PoC-1 to PoC-3 · older test split