I still need to hit Khaivut from the Kupah and bring the Karti Sashrai to the MRI at Hadassah.
Benchmarks
How well it gets the Hebrew words right
Hebrish is scored on a frozen public benchmark that any speech model can be run against, and on the held-out test split of its own dataset. The latest round, on 2026-10-30, scored Sabich v2.3 on the domain-balanced test set v2, which on 2026-10-25 had scored stock Parakeet, Jachnun v1.0 and Sabich v1.0, v2.0 and v2.1, and on 2026-10-27 Sabich v2.2. Sabich v2.3, the model in the apps, scores slightly below v2.2 on it but better on the held-out split of its own dataset (changelog, 2026-10-30): the two test sets disagree, and both differences are small. Older rounds, on other test sets, follow as history; Shakshuka v2.0 is not benchmarked yet.
White paper
How Jachnun v1.0 and Sabich v1.0 were trained, exported and benchmarked, with every figure on this page: Model release 1 white paper (PDF, 13 pages), also in Hebrew. CC-BY-4.0.
Same recording. Two models.
Real output on recordings neither model was trained on.
I still need a hitchaivut from the kupa, and bring the kartis ashrai to the MRI at Hadassah.
hitchaivut התחייבות referral guaranteekupa קופה health fundkartis ashrai כרטיס אשראי credit card
Verbatim transcripts from the public benchmark, test set v2, scored 2026-10-30.
Public benchmark, test set v2
The current test set, released 2026-10-25: public and ungated, 129 code-switching clips (14.8 min) with about 10 from each of the 13 domains of the lexicon, so places, food, bureaucracy and the army count equally. They hold 221 Hebrew term occurrences, 137 of them of terms absent from the training split. No clip was trained on by any Hebrish model. Two checks sit beside it: 25 plain-English clips and 25 tech-dictation clips. Hebrish models are highlighted.
Show as a table
| Model | Code-switching WER | Term recall | Unseen-term recall | Seen-term recall | English control WER | Tech WER |
|---|---|---|---|---|---|---|
| Sabich v2.3 current | 4.96% | 67.0% | 59.9% | 78.6% | 2.42% | 3.02% |
| Sabich v2.2 | 4.91% | 70.1% | 62.0% | 83.3% | 2.12% | 2.76% |
| Sabich v2.1 | 5.21% | 65.6% | 56.2% | 81.0% | 2.42% | 3.02% |
| Sabich v2.0 | 5.60% | 65.2% | 54.0% | 83.3% | 2.12% | 4.02% |
| Sabich v1.0 | 5.73% | 63.4% | 53.3% | 79.8% | 2.12% | 4.77% |
| Jachnun v1.0 | 5.85% | 61.1% | 51.8% | 76.2% | 2.42% | 6.03% |
| Parakeet TDT 0.6B v2 (stock) | 13.72% | 19.9% | 16.1% | 26.2% | 2.73% | 7.54% |
Test set v2, clean subset, normalised WER; scored 2026-10-25 (Sabich v2.2: 2026-10-27; v2.3: 2026-10-30) on one NVIDIA L40S per model. Sabich v2.3 scores slightly below v2.2 on every measure here: code-switching WER 4.96% (v2.2: 4.91%), term recall 67.0% (70.1%; 7 fewer of 221 hits), unseen terms 59.9% (62.0%), seen terms 78.6% (83.3%), English control 2.42% (2.12%) and tech 3.02% (2.76%). On the held-out split of its own dataset it is ahead of v2.2 (code-switching WER 4.74% against 5.12%); the two test sets disagree, and both differences are small. From v2.1 to v2.2 every measure improved: code-switching WER 5.21% to 4.91%, term recall 65.6% to 70.1% (10 more of 221 hits), unseen terms 56.2% to 62.0%, seen terms 81.0% to 83.3%, English control 2.42% to 2.12% and tech 3.02% to 2.76%; on 129 clips treat these as a steady gain, not a leap. From v2.0 to v2.1 every difference is a few words or hits, within noise: code-switching errors went from 131 to 122 in 2,340 words and unseen terms from 74 to 77 of 137 hits; seen-term recall slipped from 70 to 68 of 84 hits and English control WER rose from 7 to 8 wrong words in 330. A gap of 2 to 3 points in term recall is about 5 hits. Not comparable with test set v1 below. More: Sabich v2.1 on the new benchmark · test set · result files · try Sabich (v2.1 in the demo) in your browser.
Test set v2 by domain
The same seven models, split by the lexicon domain each clip was chosen for. Switch between code-switching WER and Hebrew term recall, and click a cell to read that domain's clips below.
Every clip, two models side by side
Each point is one clip: model A's word error rate across, model B's up. Points under the diagonal are clips where B made fewer errors. Below it, every clip's reference and both transcripts as the models wrote them, with the words that don't match the reference marked, Hebrew terms highlighted (underlined when the term is absent from the training split) and the recording to play from the test set on Hugging Face. Per-clip WERs and term hits are recomputed from the published result files with the benchmark's own scorer, which reproduces every headline number above (bench-v2.json).
Public benchmark, test set v1 (history)
The first public test set (2026-10-15), kept unchanged so its rows still compare with each other: 84 clean clips with 172 Hebrew term occurrences, scored the same way for every model. Hebrish models are highlighted. No Sabich version after v1.0 was scored on it.
Accuracy against mistakes
The same eleven models on one map. Further right means more Hebrew terms written exactly right; lower means fewer wrong words overall. Every Hebrish fine-tune sits in the bottom-right corner.
Show as a table
| Model | Term recall | Unseen-term recall | WER | English control WER |
|---|---|---|---|---|
| Sabich v1.0 (code-switching + tech) | 73.8% | 68.4% | 3.3% | 2.0% |
| Jachnun v1.0 | 73.3% | 70.1% | 3.3% | 2.2% |
| Hebrish PoC-3 | 67.4% | 63.2% | 4.0% | 2.5% |
| Hebrish PoC-2 | 65.7% | 60.7% | 4.2% | 2.4% |
| Hebrish PoC-1 | 62.2% | 61.5% | 4.6% | 2.5% |
| Hebrish Shakshuka PoC-2, Nemotron streaming (offline) | 62.2% | 54.7% | 4.7% | 2.5% |
| Whisper Hebrish v1 some overlap | 54.1% | 56.4% | 6.5% | 4.7% |
| Whisper Large V3 | 45.9% | 46.2% | 7.5% | 2.4% |
| Whisper Large V3 Turbo | 40.1% | 39.3% | 8.9% | 2.4% |
| Parakeet TDT 0.6B v2 (stock) | 32.6% | 32.5% | 9.5% | 2.7% |
| Parakeet TDT 0.6B v3 (stock) | 31.4% | 31.6% | 10.5% | 2.7% |
Test set v1, clean subset. Unseen-term recall counts 117 occurrences of 88 terms that appear nowhere in Hebrish's training data. Known issue: 22 of the 139 term labels are English tech words, which makes term recall 2 to 6 points higher than a Hebrew-only count. Leaderboard · test set · runner and scorer.
Held-out test split
The test split of the code-switching dataset: 387 clips at revision 550bfcaa, with 293 occurrences of held-out Hebrew terms that are never in training. Stock Parakeet and PoC-3 were re-scored on the same clips, because the split grows with the dataset and only same-revision numbers compare.
Wrong words, per 100
In code-switched speech, stock Parakeet gets about 11 words in every 100 wrong (11.3%). Jachnun v1.0 gets about 5 (5.1%).
One square is one word; coloured squares are mistakes.
Hebrew words written right, per 100
For Hebrew words it heard in training, stock Parakeet writes 28 in every 100 in the exact spelling. Jachnun v1.0 writes 73.
One square is one Hebrew word; coloured squares are spelled right.
Stock Parakeet to v1.0, test by test
All four models
Show as a table
| Test | Stock v2 | PoC-3 | v1.0 | Shakshuka PoC-2 |
|---|---|---|---|---|
| Code-switching WER | 11.3% | 6.0% | 5.1% | 5.8% |
| Held-out Hebrew terms correct | 25.9% | 45.7% | 47.8% | 44.4% |
| Test terms also seen in training, correct | 27.7% | 63.3% | 72.7% | 68.0% |
| Own English WER | 3.1% | 2.3% | 1.7% | 3.3% |
| LibriSpeech test-clean WER | 1.2% | 1.0% | 1.2% | 2.0% |
| Long-form WER (15 recordings) | 9.6% | 5.8% | 5.0% | 5.8% |
Normalised WER (lower case, no punctuation). On held-out terms, v1.0's gain over PoC-3 (+2.1 points, about 6 of 293) is within noise; its gain on terms seen in training (+9.4) is not. English did not get worse: own English improved and LibriSpeech stayed level. That is the "forgetting" check a release has to pass.
Sabich v1.0 against Jachnun v1.0
Sabich v1.0 is trained on Jachnun's kind of recordings plus the tech and task recordings, from stock Parakeet (not on top of Jachnun). The question was whether the extra data costs code-switching accuracy. It does not: the two are level on code-switching, and Sabich is a little better on tech dictation. Scored on the newer held-out split (revision f956dbb1: 422 code-switched clips, 324 held-out term occurrences), with Jachnun v1.0 and stock Parakeet re-scored on the same clips. On the public benchmark above they are level too: 73.8% against 73.3% term recall, 3.3% WER each.
Show as a table
| Test | Stock v2 | Jachnun v1.0 | Sabich v1.0 |
|---|---|---|---|
| Code-switching WER (422 clips) | 11.82% | 5.42% | 5.30% |
| Held-out Hebrew terms correct | 23.77% | 46.91% | 48.15% |
| Test terms also seen in training, correct | 27.65% | 72.01% | 72.70% |
| Tech dictation WER (20 clips) | 2.51% | 1.74% | 1.54% |
| Own English WER (84 clips) | 2.81% | 1.56% | 1.62% |
| Own English without Hebrew (57 clips) | 1.79% | 1.23% | 1.23% |
| LibriSpeech test-clean WER (200 clips) | 1.23% | 1.20% | 1.30% |
| Long-form WER (17 recordings) | 10.34% | 5.84% | 5.77% |
Normalised WER. Every gap between Jachnun and Sabich is within noise at these sizes (1.2 points of held-out recall is about 4 of 324 terms). LibriSpeech is 0.07 points above stock for Sabich, about 3 words in 200 clips; accepted as noise. Not comparable with the 550bfcaa numbers above: the split grew between revisions. Source: ASR-Training data/evals/r7-*-summary.json.
Every version, older test split
How the proof-of-concept versions compared when they were first scored, on the earlier and smaller test split (revision b47310621f79, 219 clips). PoC-4 and PoC-5 were never benchmarked.
Hollow points are targets for the next checkpoints, not results. v1.0 is not on this chart because it was scored on the newer split; its numbers are above.
How it is run
- On rented GPUs. Each model is scored in its own job on one NVIDIA L40S on Modal. The 2026-10-19 round ran six jobs in parallel, about two and a half minutes each. The 2026-10-25 round on test set v2 scored five models the same way.
- Only what is needed. A new checkpoint is scored with stock Parakeet and the last benchmarked release on the same clips. The public benchmark is frozen, so older rows never need re-running.
- Contamination check. Test set v2 was built without any clip whose id or text is in the training or validation manifest of any Hebrish training run (22 runs, checked 2026-10-25). On test set v1, the 145 clip IDs and texts were checked against each model's training manifest; for v1.0 and Nemotron none were found.
- Term matching is whole-word and case-insensitive, against the one canonical spelling in the lexicon. A near miss counts as wrong.
Known limits
- One speaker, read speech, mostly one phone microphone.
- Two terms on the held-out list (mandatim, Beit Safafa) also appear in a few training sentences. If they were trained on, a handful of "unseen" hits are not unseen. Being checked.
- Nemotron's public-benchmark row is scored offline. On the held-out split it was also scored while streaming, at 560 ms chunks: 5.65% code-switching WER against 5.81% offline, so streaming cost nothing measurable. Smaller chunks (80 and 160 ms) are not measured.
Figures to reuse
The same results as static images, free to reuse with credit (CC-BY-4.0). PNG at 1600 px; each also exists as SVG at the same address ending in .svg.