What "word error rate" means, and why we also count Hebrew words
The model card reports results as percentages: a word error rate of 5.3%, Hebrew terms right 66% of the time. If you don't work in speech recognition, those numbers don't say much on their own. This post explains what each one measures, with the real figures for Hebrish v3 against stock Parakeet.
Word error rate: mistakes per 100 words
Word error rate (WER) compares what the model wrote with what was actually said, word by word. It counts three kinds of mistake: a wrong word, a missing word, and an extra word that was never said. Add them up and divide by the number of words that were said.
Take a real recording from the home page. The speaker said:
I want to go to the makolet today to buy my groceries.
Stock Parakeet wrote "Micollette" for makolet. That is one wrong word out of twelve, so the word error rate for that sentence is 1 ÷ 12, about 8%. Hebrish wrote makolet: no mistakes, 0%.
The easiest way to read a WER is as mistakes per 100 words. A WER of 5% means about five words in every hundred need fixing. Lower is better.
On held-out English sentences with Hebrew words in them, stock Parakeet has a WER of 10.9%, roughly eleven mistakes per hundred words. Hebrish v3 has 5.3%, roughly five. In a 300-word email, that is the difference between fixing about 33 words and about 16.
Why WER is not enough
A typical test sentence has twelve or fifteen words and one or two Hebrew words. A model can get every English word right, miss the Hebrew word every time, and still score a fairly low WER. But the Hebrew word is the one you care about: it is the reason to use Hebrish at all.
So we also count the Hebrew words directly. Term accuracy (called term recall on the benchmark) is the share of Hebrew words written in exactly the right spelling: teudat zehut, not "today at Zehut", and not teudat zehoot either. Here higher is better.
| Hebrew words written correctly | Stock Parakeet | Hebrish v3 |
|---|---|---|
| Words that appear in training | 29% | 66% |
| Words it never saw in training | 27% | 48% |
Read 66% as "about two out of every three Hebrew words come out right", against fewer than one in three for the stock model. The second row is the harder test: words the model never heard during training. It still gets 48% of those, against 27%, which means it has learned something about how Hebrew words sound and are spelled, not only a fixed list.
Did the English get worse?
Teaching a model new words can damage what it already knew. So every version is also tested on plain English. On the same speaker's plain English, WER went from 3.5% to 2.9%. On LibriSpeech, a standard set of other people reading English, it went from 1.2% to 1.0%. English got slightly better, not worse.
The public benchmark
The numbers above come from the project's own held-out test set (219 recordings). There is also a separate public benchmark that compares models side by side. On it, Hebrish v3 writes 67.4% of Hebrew terms correctly, against 45.9% for Whisper Large V3 and 32.6% for stock Parakeet.
What the numbers don't tell you
- One voice. Every training recording is one speaker's: an Irish-accented English speaker living in Jerusalem, and the test recordings are in the same voice. How Hebrish does on other voices has not been measured.
- Read-aloud speech. The test sentences were read from a script. Casual speech, background noise and other microphones may do worse.
- Only v1 to v3 are scored. Versions 4 and 5 are published but not benchmarked yet.
Look for the "What does this mean?" buttons next to the numbers on this site: they explain each one in the same terms. The full tables are on the model card.