Hebrish
Updates

Blog · Training and data

How Meorav-Multi-Voice was trained, and why v1.0 is a proof of concept

Five designed voices, Hebrew learned from masked spans of Daniel's takes, one training run: the pipeline step by step, and why the Hebrew did not transfer as well as hoped.

PoC Meorav-Multi-Voice v1.0, published on 2026-11-16, is a proof of concept. It was not trained on enough data, so the Hebrew code-switching did not transfer to its five voices as well as hoped. This post goes through how it was made, from 2026-11-14 to 2026-11-16: how the voices were designed, how a voice is picked, what the training data was, the training run itself, what did not carry over and what would fix it. The numbers come from the run records and the model card.

The idea

Daniel's own voice reads Hebrew words well: Meorav-Daniel, trained on his recordings, gets the gutturals right. But his voice is his, and Irish-accented. The aim was to keep the Hebrew pronunciation that Meorav-Daniel learned from him and swap the voice for designed ones that belong to nobody. The voice becomes an input; the pronunciation stays in the weights.

1. Designing the voices

All five voices were designed with OmniVoice (k2-fsa, code at commit 08be0b4), which can make a voice from a text description alone, with no reference audio. Settings: 32 steps, guidance scale 2.0, denoise on, language English.

  • Josh (American male) is designed voice A3, picked on 2026-11-14 for Meorav v3.0. Four descriptions (young adult or middle-aged, moderate or low pitch, all "male, american accent") times five seeds made 20 candidates, each reading the same two narration sentences. Daniel picked A3 by ear: male, young adult, moderate pitch, american accent, seed 3.
  • Rivka, Nigel, Fiona and Bazza came from a sweep on 2026-11-16, after Daniel found one plain American voice a little grating and wanted something more fun: a male and a female voice for each of OmniVoice's native English accents (American, British, Australian, Canadian; it has no Irish or South African), <gender>, young adult, moderate pitch, <accent> accent, seeds 0 and 1, 16 clips of one lively line, on one L4. Seed 1 was kept for the American female, British male, British female and Australian male.

Fixing the seed. A designed voice changes from call to call, and fixing the seed does not bring the same voice back on new text: it only reproduces the clip that seed made. So each voice was frozen as a clip. The chosen render becomes the voice's reference, and everything after that is cloned from it. For A3 the reference was restored with Sidon v0.1 (8.0 s; Daniel preferred the restored clips throughout), and all ten freeze renders came back word-exact through Whisper. The other four use their sweep clip as the reference. These five clips are the files in the model's voices/ folder.

2. Picking a voice by x-vector

Meorav-Multi-Voice is a full fine-tune of Qwen3-TTS-12Hz-1.7B-Base. That model has a speaker encoder: from a reference clip it makes one vector for the voice (an x-vector). With x_vector_only_mode=True it uses only that vector and does not continue from the clip's audio or text. So the voice is picked by which clip you pass. Getting there took four small pilots on 2026-11-15 and 2026-11-16:

  1. Stock model, two kinds of conditioning. On the untrained base, with Daniel's clip or A3's, as a clip or as an x-vector only: "My achi called to say the chag starts earlier this year" lost the gutturals in every case. Conditioning carries the voice and nothing else; the model's own reading of the spelling decides the Hebrew. The Hebrew has to be trained in.
  2. Hebrew spans on the stock model. Daniel's takes, trained only on their Hebrew words, plus 45 minutes of LibriTTS-R English from 23 speakers; learning rate 1e-5, 3 epochs, one A100-80GB. Daniel's verdict: "the base is still problematic". Hebrew is not one of the base model's ten languages.
  3. Meorav-Daniel with the speaker encoder put back. Fine-tuning on Daniel alone had detached the encoder, so the base model's 76 speaker-encoder tensors were restored onto Meorav-Daniel's weights. With no further training, the Hebrew words were right in every voice, but every voice sounded like Daniel: the model had only ever seen his x-vector and ignored the others.
  4. One pass of multi-speaker training from there. First with LibriTTS-R speakers, then with A3's synthetic English (377 lines) plus every third of Daniel's span rows (154): one epoch, learning rate 2e-5. The x-vector steered the voice again. A3 kept the Hebrew better than the LibriTTS voices did, though not all the gutturals. Daniel then asked for a female voice too, and the sweep in step 1 grew that into five.

3. The training data

PartWhatRowsLoss on
Daniel's code-switching takes461 takes, the ones out of a 600-take set that contain Hebrew; 1,311 Hebrew words, roughly nine minutes of Hebrew-word audio. The words were labelled by Claude Sonnet, timed by forced alignment (torchaudio MMS_FA, no failures) and padded by one 80 ms codec frame each side. Each take is used twice.922Hebrew-word spans only; the English is masked out
Synthetic English, five voices245 lines per voice, the same texts for every voice (a random subset of 377 LibriTTS-R dev-clean texts), rendered by OmniVoice cloning each voice's reference clip, 24 kHz. Rivka 30.6 min, Nigel 30.5, Fiona 30.4, Bazza 26.9; Josh's lines are the A3 renders of the same texts from the pilot.1,225Everything
Total2,147

Every row is conditioned on its own speaker's x-vector, from a reference cropped or padded to 5 seconds. Training clips were 3 to 15 seconds long. The masking is the core of the recipe: outside the Hebrew spans, Daniel's frames get no loss on either the talker or the sub-talker, so the model is taught how he says chag and tachana without being taught his voice.

4. The run: mv1

Start fromMeorav-Daniel v1.0 weights with the base model's speaker encoder and config restored
Training codeQwen3-TTS's fine-tuning script at commit 022e286b, with the fixes the Meorav line has carried since 2026-10-21 (label shift, sub-talker input and loss, text projection, fp32 master weights) plus the span masking
OptimiserAdamW, learning rate 1e-5, weight decay 0.01
Batch4, gradient accumulation 4 (effective 16)
Epochs3; the epoch-3 checkpoint is the release
Precisionbf16 autocast with fp32 master weights and optimiser state; checkpoint saved in bf16
HardwareOne A100-80GB on Modal
Logged loss2.40 at the first batch, 1.93 at the end of epoch 1, 1.30 at the end of epoch 3 (logged every 10 batches; it mixes masked and full rows, so it is not a quality measure)

The run time was not recorded. After training, ten lines were rendered in each of the five voices (x-vector only, seed 0) for listening; they are the samples on the model page.

5. What came out, and what did not carry over

The voice swap works: one model, five distinct voices, each picked by its clip. Daniel listened to the renders and liked the voices. The Hebrew is where v1.0 falls short. Daniel's assessment is that it was not trained on enough data, so the code-switching did not transfer to the new voices as well as hoped. The clearest gap is the gutturals: the "ch" of chag and achi (the sound of ח) is not always produced. There is no benchmark and no listening score, so this is a judgement by ear, not a measurement.

One reading of why, from the numbers above (an interpretation, not tested): the Hebrew signal is small. Roughly nine minutes of Hebrew-word audio, from one speaker, sits against more than two hours of English in the five new voices, all trained in full. The English rows have no Hebrew in them. So the new voices learn English from their own lines, and their Hebrew has to come across from Daniel's spans through the shared weights.

Two smaller limits: training clips were at most 15 seconds, so long text should be generated sentence by sentence, and only the five voices were trained. Other reference clips can be tried, with no promises.

6. What would fix it

  • More data. The run used 461 of Daniel's takes. Meorav-Daniel itself was trained on 3,995 (about 5.5 hours), so far more Hebrew-span material exists than this run used.
  • A phonetic front end. A route worked out on 2026-11-14 for Meorav v3, agreed in principle and not started: find the Hebrew words in the text with the Hebrew-Latin token classifier, turn them into Hebrew script (a lexicon first, then a transliteration model), turn that into IPA with Phonikud, an open Hebrew grapheme-to-phoneme model built for TTS that marks stress and writes the gutturals as their own sounds, run the English through an English grapheme-to-phoneme tool, and give one phoneme string to a TTS model that takes phonemes. The pronunciation would then be decided before the TTS model, where it can be checked and corrected word by word, instead of hoped for from text. The open questions are which phoneme-input TTS model to use (none has been tested yet), mapping two different IPA conventions onto one symbol set, and how often the step from Latin letters to Hebrew script gets a word wrong.

Try it

The model is Hebrish/Meorav-Multi-Voice on Hugging Face (accept the terms; access is approved instantly), run with qwen-tts; the card has a short example. To try the voices without code, Meorav desktop 0.2.0 is a desktop tester for Windows and Linux with a picker for the five voices. What the voices are, samples and the audio drama are in the announcement post.

← All posts · More in Training and data · Archive · Search · Meorav-Multi-Voice model page · Announcement: one model, five designed voices · Meorav-Multi-Voice on Hugging Face · RSS feed