Sabich v2.1 on the new benchmark: what changed, and what it scored
Sabich v2.1 went out on 2026-10-25 without a benchmark. Later the same day it was scored on a new public test set, together with every earlier Sabich version, Jachnun v1.0 and stock Parakeet. All five ran on the same clips. This post covers what changed in v2.1's training and what the scores show.
The new test set
Test set v2 is public and ungated. Its headline subset has 129 code-switching clips, about 10 from each of the 13 vocabulary domains on the lexicon: places, food, bureaucracy, the army and so on. That way no single kind of word dominates the score. It holds 221 Hebrew term occurrences, 137 of them of terms that aren't in the training split. No clip in it was trained on by any Hebrish model: every take whose id or text appears in a training run's manifest was removed. It also has two checks: 25 clips of plain English and 25 of English tech dictation. The older v1 set stays loadable, but its numbers don't compare with these.
What changed from v2.0 to v2.1
Same recipe as v2.0: retrained from stock Parakeet TDT 0.6B v2 (not on top of v2.0), 20 epochs on one GPU, with the training mix that includes everything. What changed was the data.
- More recordings: 80.6 more minutes of the speaker's takes (642.3 to 722.9 min) and a matching rise in LibriSpeech replay (71.6 to 80.3 min).
- Priority words: 288 clips with English words the speaker's dictation got wrong, each repeated three times (67.7 minutes per epoch).
- No
<unk>in the targets: characters the vocabulary can't spell were mapped or removed (the full story). - No paragraph passages, and code-switching and tech never share a sentence.
In all, 848.4 minutes per epoch against 713.9. The lexicon page charts the two mixes side by side and shows how often each word appears in training.
The results
| Model | Code-switching WER | Term recall | Unseen-term recall | English control WER | Tech WER |
|---|---|---|---|---|---|
| Stock Parakeet | 13.72% | 19.9% | 16.1% | 2.73% | 7.54% |
| Jachnun v1.0 | 5.85% | 61.1% | 51.8% | 2.42% | 6.03% |
| Sabich v1.0 | 5.73% | 63.4% | 53.3% | 2.12% | 4.77% |
| Sabich v2.0 | 5.60% | 65.2% | 54.0% | 2.12% | 4.02% |
| Sabich v2.1 | 5.21% | 65.6% | 56.2% | 2.42% | 3.02% |
The big step is still the first one. Stock Parakeet gets about 14 words in 100 wrong in English with Hebrew words in it and spells one Hebrew word in five right. Every fine-tune gets about 5 or 6 in 100 wrong and spells close to two in three right. From v1.0 to v2.1, each Sabich version has scored a little better than the last on code-switching WER, term recall and unseen-term recall, and tech WER fell from 4.77% to 3.02%.
What is within noise
The steps between Sabich versions are small, and the test set is small too: one speaker reading aloud. From v2.0 to v2.1, code-switching errors went from 131 to 122 in 2,340 words, and unseen terms from 74 to 77 hits out of 137. Two numbers went the other way:
- Seen-term recall, for Hebrew words that are also in training, slipped from 83.3% to 81.0%: 68 hits out of 84 instead of 70.
- English control WER rose from 2.12% to 2.42%: 8 wrong words in 330 instead of 7, across 25 clips.
Both are a difference of one or two words, well within noise at this size, and neither is a sign that English got worse: v2.1 is level with Jachnun v1.0 on English and still below stock. On the benchmark's own rule of thumb, a gap of 2 to 3 points in term recall is about 5 hits and shouldn't be read as real. So the honest reading of v2.1 against v2.0 is a small gain that points the same way on most measures.
Try it
Sabich v2.1 is the model the Hebrish apps download now. To hear it without installing anything, the Sabich demo runs it in your browser. Every result, including the older test sets, is on the benchmarks page, and the result files with every transcript are in the benchmark repository.
