Accent transfer in ASR: will a model trained on one voice work for yours?
Every Hebrish recording so far is one speaker's. What a speech-recognition fine-tune learns that should carry over to other accents and voices, what stays behind, and how it will be measured.
Every recording Hebrish has been trained on so far is one speaker's: Daniel, an Irish-accented English speaker in Jerusalem. That was deliberate. One voice was the quickest way to build a first dataset, check that the recipe works and set up a benchmark. The obvious next question is what happens when someone else speaks to it. For example, a woman with an American accent, a South African, or someone who grew up speaking Hebrew. This post sets out what to expect, why, and how it is going to be measured.
Not tested yet
No Hebrish model has been scored on another speaker's voice yet. Everything below about other voices is an expectation based on how fine-tuning works, not a result. The benchmarks are all on the training speaker's voice, on recordings the models never trained on.
What a fine-tune learns
Hebrish models start from a stock speech model, NVIDIA's Parakeet TDT 0.6B v2, which was trained on a large amount of English from many speakers. Fine-tuning on code-switched recordings teaches it two different things at once, and they don't transfer equally.
Should carry over
- Which Hebrew words exist and how they are spelled: mezuman, not "him as Uman".
- Where they turn up in an English sentence: after "pay with", before "office".
- Everything the base model already knew about English and about many voices. The fine-tune sits on top of it.
Probably won't
- How those words sound in one voice: one accent, one pitch range, one way of saying each Hebrew word.
- One set of equipment: the same microphones and rooms in almost every take.
With hours of one speaker, part of what the model has learned is "Hebrew words sound like this voice". The word knowledge should still help other speakers. The tuning to one voice won't.
Accent transfer: what to expect for other voices
- Hebrew words: better than stock, worse than on the training voice. Stock Parakeet doesn't know these words at all: on test set v2 it spells 19.9% of them right, against 70.1% for Sabich v2.2. Most of stock's mistakes come from missing vocabulary, which a different voice doesn't fix. So a fine-tuned model should still win on the Hebrew words for other speakers, probably by a clear margin, but below its 70%.
- Ordinary English: about the same as stock. That part is still mostly the base model. Every training run also replays some LibriSpeech, a standard set of English audiobook recordings from many speakers, so the model doesn't forget it.
- Accent matters more than gender. The base model has heard plenty of women's voices, so pitch on its own is not the main problem. Pronunciation is: an American r, different vowels and stress, or a native Israeli ch or r in a Hebrew word all sound different from the version the model learned.
- One risk to watch: Hebrew words that weren't said. Almost every code-switching sentence in training has a Hebrew word in it. When the model is unsure of an unfamiliar voice, it may lean towards writing one where the speaker said English. A test on one voice can't show this, which is why the test below includes English-only sentences.
Why more voices, and how many
You don't have to train a model on each user's voice for it to work. Training on one voice is the narrowest case, and the first few extra speakers usually give the biggest gain. Going from one voice to a handful of varied ones (different accents, men and women) should close most of the gap. After that, each new voice helps less. Training on one particular user's voice might come later as an optional extra, but it isn't required.
That is why the first major milestone for Hebrish is a model trained on many speakers. Until then the project is an early prototype, and commercial licences aren't offered.
What is happening now
- A cross-speaker test (planned). A few volunteers with different accents each read about 30 sentences from the public test set, plus some English-only control sentences. That is roughly 10 minutes per person. Stock Parakeet and Sabich are scored on their recordings. The results will replace the expectations above with numbers.
- Collecting more voices. The get involved page explains which recordings help most and has ready-made consent forms for a voice or a lecture series. The roadmap lists the voices and recording conditions the dataset is still missing.
If you speak English with Hebrew words in it, and your accent isn't Irish, your voice is exactly what the next model needs. Here's how to contribute recordings.
A companion post looks at the other big training question so far: whether one model can learn code-switching and tech vocabulary together.
