Cleaning up phone recordings for Meorav v2.2, and speaker isolation
How a new class of audio-to-audio models turned my speech-to-text recordings into usable text-to-speech data for Meorav v2.2, why it still sounds flat, and why speaker isolation has me rethinking the dictation pipeline as a chain of small models.
Meorav v2.2, the text-to-speech model, is trained on my speech-to-text recordings after cleaning them with NVIDIA's RE-USE.
Audio-to-audio models like RE-USE and Sidon rebuild the voice instead of subtracting noise, so the result doesn't sound robotic.
Meorav now pronounces the Hebrew words well but sounds flat. Short invented stories are my plan for more expressive recordings.
I'm recording Israeli email addresses, web domains and phone numbers, but Parakeet can't output the @ sign.
So I'm moving towards a chain of small models: speaker isolation in front of the ASR, normalisation and clean-up after it.
Recordings with background conversation are now useful test material for speaker isolation.
For the last couple of weeks I've been recording a lot of voice clips to train Sabich, my speech-to-text model for code-switching, mixing English and Hebrew. This episode is mostly about the other side: text-to-speech. Yesterday I found a class of audio models that turned those recordings into usable training data for Meorav v2.2, and, poking around next to them, something that has me rethinking how a dictation pipeline should be built.
Why speech-to-text recordings make poor text-to-speech data
6:44 My objective was never to make a text-to-speech model as such. But at around 30 hours of training data I thought: why not try the voice clone again? If you train on one speaker, you get a voice clone whether you want one or not.
The trouble is that the two want opposite things. Speech-to-text wants a messy audio corpus, because you're training it to recognise speech in all kinds of difficult audio environments. I've deliberately mixed it up: quiet rooms, traffic honking, a Jabra headset at my desk, my phone held up to my mouth.
Text-to-speech is a magnifying glass on your recording environment.
Claude Code found the phone takes are a better foundation for TTS than the headset ones: the headset's built-in filter gives you that call-centre sound. The first Meorav, a small test fine-tune of Qwen3-TTS, was very robotic and sounded like it was coming down a phone line. Classic denoisers like DeepFilterNet didn't fix it. They work by subtraction, you lose a lot of vocal tone, and because TTS magnifies your voice, that oddly digitised sound is exactly what transfers into the model.
Still, what I was listening for first was pronunciation: can it say "I need to pick up my teudat zehut at misrad hapnim" the Hebrew way, and not the way a stock model reads it? It basically worked.
Audio to audio: rebuilding instead of subtracting
11:30 Last night, with 40 or 50 hours recorded, I asked Claude whether a much more robust Meorav was possible. It dropped what turned out to be the golden lead: Sidon, a model I'd never heard of. I ran 20 clips through it on Modal from bed while putting my son to sleep, using Claude Code remotely. It sent a zip file to my phone, and I thought: whoa, that is really, really good.
These models sit under Hugging Face's audio-to-audio task. If traditional background-noise removal is subtractive, these are additive, or more accurately predictive. I went digging for alternatives and settled on NVIDIA's RE-USE, whose target use case seems to be cleaning up imperfect audio for creating TTS models. Everything else in the project is NVIDIA, so it fit the stack. Both are in my Audio restoration collection.
Start from the speech-to-text corpus everything I've recorded.
Keep the code-switching takes narrower than Sabich, which takes everything.
Drop the noisy ones leaving a few thousand clips.
Clean them with RE-USE rebuilding the voice rather than subtracting noise.
19:11 One defect remains: the prosody is very flat. The vocabulary, the most important part, is good. But the training material is synthetic sentences that are, frankly, as dull as dishwater to read, and that's about 99% of what I've recorded so far.
The plan is short, weird stories: Daniel goes to the supermarket to buy eggs, but make it weird. Reading a few sentences like that pulls me out of the monotone. Fine-tuning is about distributions: as I record more expressive audio, the average gets pulled up. For anyone who wonders: I'm not actually that flat emotionally.
Email addresses and the @ sign
25:41 On the speech-to-text side I've been recording Israeli email addresses, phone numbers and domains like .org.il and .gov.il. Then I hit a road bump: Parakeet does not emit the @ sign. The character isn't in the decoder, so those recordings are no use to the fine-tune for now.
I assume NVIDIA did that deliberately: it wants inverse text normalisation (ITN) as a separate pass after the ASR. So, working with the system rather than against it, I'm looking at a small model between the ASR and the app, usable with or without the post-processing model I've trained, a fine-tune of S1-mini.
Speaker isolation and a chain of small models
23:17 Poking around near RE-USE, I found speaker isolation, and said: what? No way. Picture dictating an email walking through the shuk, someone calling out "hey", or a bar blaring music and the ASR transcribing the lyrics. These are the first models I've seen explicitly trained to isolate the speaker; the ones I'm looking at are in my Speaker Isolation collection.
Picking up Ezra from daycare, it clicked: this could go in the input chain, alongside voice activity detection. Just as VAD keeps silent audio away from the model, a second bouncer stands in front of it. A seemingly simple dictation pipeline could be five models, one after the other, on a phone.
Last year I thought audio multimodal models would just take over from ASR. I've done a 180.
I think domain-specific small models are going to win the day.
Next I'll be trying the pipeline out, and recordings with a conversation in the background are now exactly the samples to test it on. The full episode has more, including why I went all out on recording after switching to Parakeet, and Claude repainting the walls when asked to vacuum the room. Have a listen.
Full transcript, cleaned up 3,910 words · everything said in the episode, written to be read
This is the recording written up to be read: everything I said is here, tidied into sections. The audio is lightly edited: most "um"s and "uh"s cut, long pauses shortened, EQ and level evened out (33:28 recorded, 31:56 published). Then the voice was restored with Sidon (University of Tokyo) to sound closer to a studio microphone. The intro and outro are voiced by Herman Poppleberry of My Weird Prompts, over “Happy Strummin’” by Jason Shaw (music by audionautix.com, CC BY 3.0); the recording starts at 0:15. Timestamps jump to that point in the recording. All episodes: podcast.
This is an edition of the podcast for the Hebrish project, where I share updates on what I'm working on and what I've done, and jot down some notes.
It started as an attempt to record a voice note while picking up my son, which did not end well. There are times you can multitask and times you can't, and that was one of the latter. I was trying to record it there on purpose, though, because what I'm going to talk about is exactly the kind of audio scenario where this technology helps.
What I want to share is some of what I discovered yesterday. This podcast is a very casual collection of clips of me sharing things I'm figuring out as I go along. I think that's what everyone in AI is doing at the moment, and I think it's a wonderful thing. It's certainly how you build voice technology: a lot of it is experimental, there's no clear answer, and you try things and see what works. This episode is very much in that category.
For the last couple of weeks I've been recording a lot of voice clips to train a speech-to-text model that supports code-switching, mixing English and Hebrew. (I will never stop thinking "code-switching" is a weird name, but that's what it's called.) This is by far the most time I've ever put into recording sample audio.
I'd tried a couple of small proofs of concept before, first with Whisper, pushing the boat out to see what I could get. My first objective was: can I fine-tune Whisper so that when I say "I'm going to the makolet", it writes "makolet"? When I got that, it was the whole achievement of the proof of concept for me. That might sound silly, but at the time it was extremely exciting to record some audio, run a training pipeline and end up with a model I could put side by side with stock Whisper and say: regular Whisper doesn't get this right, and my little fine-tune, trained in Jerusalem on a few dozen sentences of me speaking, does.
That was really motivating, but the model didn't run on my phone, so I wasn't using it and had less reason to iterate on it. When Parakeet came out and I switched to it a few weeks ago, local inference on the phone I use every day suddenly became viable, and I had a very strong incentive to get a fine-tune really working.
It didn't make sense to record just enough audio to get back to where I'd been; that would be going around in circles. The old training data wasn't well shaped for Parakeet anyway. So I decided to start from scratch and go all out. That's what the last couple of weeks have been: scrappy, recording when I can, with some marathon days. What's been insanely motivating is that with each iteration I can really hear it improving.
Today's topic is the text-to-speech side. But my primary goal, by a long margin, is a very capable speech-to-text model that has no trouble with anything, down to the niche tasks I was training on today: Israeli email addresses and web domains.
It takes a while to even think of all the small ways localisation matters. A model trained mostly on an American corpus may never have seen many email addresses ending in .org.il (the Israeli .org) or .gov.il (the government domain). And you think: I've come this far, I may as well record 100 sentences to put that in the training corpus. So I did that today, and phone numbers too. I'm tinkering at the edges now of what is a very useful model for my own purposes.
The open question is how well it transfers to other users. The training corpus is a single speaker for now: apart from a few token contributions from my wife and a friend, it's 99.9% me. If it transfers to any appreciable extent, or if the drop is modest, I think that's a very impressive achievement. Not mine: the achievement of ASR in general, that it works across accents. In the previous episode I talked about ways around that, such as synthetic data.
My objective was never to make a text-to-speech model as such. But at around 30 hours of training data I thought: why not try the voice clone again? I say "voice clone" because if you train on one speaker, you get a voice clone whether you want one or not. I have no particular desire to clone myself for its own sake, although for pranking friends alone it's quite worth it.
The real reason was to see whether a text-to-speech model could come out of what I'd recorded so far. My first, naive try was handing Claude the clips I had and asking whether we could train a TTS model on them.
6:44 What speech-to-text wants, and what text-to-speech wants
As anyone who has tried this already knows, the training objectives for speech-to-text and text-to-speech are very different.
Speech-to-text wants a messy audio corpus, because you're training it to recognise speech in all kinds of difficult audio environments.
Text-to-speech is a magnifying glass on your recording environment.
Where I'm recording now is probably the quietest space I have, and even here you might hear a faint noise floor from the air conditioning. Unless you're in a recording studio, there's some imperfection.
It was harder still because most of my speech-to-text recordings were made on my phone, and, more interestingly, the ones that were hardest for TTS were made on voice headsets. I've deliberately mixed up microphones for the speech-to-text data. Besides recording when it's quiet, when it's noisy and when there's traffic honking, sometimes I dictate at my desk on a Jabra headset and sometimes I hold my phone up to my mouth, like now. So why not get a mix of environments and a mix of microphones?
The recordings are labelled by device, and when I asked Claude Code to go through them it found that the phone ones are a better foundation for TTS than the headset ones. The headset has a built-in filter that helps for calls but gives you that call-centre headset sound; the phone does less processing in hardware.
8:33 The first Meorav: denoising and a robot voice
I didn't want to give up there. The first version, a small test fine-tune of Qwen3 [Qwen3-TTS] on my corpus, was very robotic and mechanical, and certainly sounded like it was coming down a phone line.
My next question to Claude was: what if we denoise the speech-to-text takes? We used the classic denoising programs, DeepFilterNet and the like, the ones that pull in PyTorch and other heavy dependencies. They're very effective, but you inevitably lose a lot of vocal tone, because they work by subtraction. In trying to take out the air-conditioning hum, the honking or the crying, it's very, very hard to keep the full range of your vocal expression. I'd say almost impossible. You usually end up with audio that sounds oddly digitised, and because TTS magnifies your voice, that's exactly what transfers into the model.
Besides the robot voice, it was still worth releasing, because my first objective is to test whether the model reproduces code-switched pronunciation accurately. Ideally it will sound nice, but first: can it say my little test sentence, "I want to go to the makolet to buy bread", the Hebrew way, and not the way a stock model would? Or the other classic: "I need to pick up my teudat zehut at misrad hapnim".
A stock model reads that as "hap-nim". Look at hapnim with English phonetics: the Hebrew pronunciation is "hap-neem", and an English speaker who didn't know it was Hebrew, and had only seen it written on a page, would say "hap-nim".
That's what I was testing and listening for. More important than the robot voice, it basically worked. It was at least a decent proof of concept.
Last night I was training the latest version of Sabich, the main project, which is now folded into one release model. I asked Claude: I've now got 40 or 50 hours, a lot more audio than the last time I tried. Is there any chance we can get a much more robust version of the TTS engine, Meorav?
It said: phonetically, yes. For speaking quality, it's still poor material, because it's mostly phone takes, and the headset takes are worse than the phone ones for this. And then Claude dropped what turned out to be the golden lead: a model called Sidon, which I'd never heard of.
It's an audio-to-audio model. I didn't have high expectations, but I said: let's run 20 clips through it on Modal. That takes a few minutes and a few cents of GPU time, and I can listen to how they come out. It was one of the first times I used Claude Code remotely: I was in bed putting my son to sleep, checking in. Claude Code remote works very nicely. It literally sent a zip file to my phone, I downloaded it, and I thought: whoa, that is really, really good.
These models fall under Hugging Face's audio-to-audio task. If you think of traditional background-noise removal as subtractive, these are additive, or more accurately predictive. Their architecture is very interesting and very different from their predecessors, and that difference is what sets them apart.
The pipeline I ended up using for Meorav v2.2 (on Hugging Face; release notes in the v2.2 post) has several stages. It starts from my speech-to-text corpus, everything I've recorded.
Code-switching takes only. The philosophy for training Sabich is to take everything: tech vocabulary and code-switching. This experiment is deliberately narrower and uses only the code-switching takes. That's the full set.
Score them for noise, using DNSMOS if I'm not mistaken [not quite: see the correction below]. It's hard to keep track of all these packages when Claude Code does much of the orchestration and I'm thinking about the approach and browsing Hugging Face categories.
Drop the noisy ones. The takes with an acceptable noise level became the candidates for cleaning. That still left a few thousand clips, about an hour of processing.
Clean them with an audio-to-audio model, and train Meorav v2.2 on the result.
Correction (2026-11-09). DNSMOS did not score the takes for noise. It was used only on its speech-quality scale, as an outlier cut (a score below 2.5 dropped 87 takes); its background-noise score was not a filter. The noisy takes were removed by noisy-room labels (3,638), a noise-floor check (320), a bandwidth or limiter check (299), a narrowband check (272) and PANNs sound-event detection (237). 3,449 code-switching takes passed; 2,924 of them (no digits, at most 15 seconds) went into training with 1,103 read takes, and a podcast voice chain followed RE-USE. The full pipeline: training pipelines.
Looking around before committing
Sidon didn't look like a huge model on Hugging Face, and I was extremely impressed by it. But before running it over everything, I wanted to see what else was out there, since this pipeline should be replicable. So I went digging through audio-to-audio.
This is a problem with a lot of machine-learning task categories: they're granular, but not granular enough. Audio to audio covers everything from cleaning up a music track to voice alteration with a neural network. There's a lot in there. The interesting part was models tagged as speech improvement, which isn't quite background-noise reduction.
To cut to the chase, I found a few candidates. I have a very high regard for Sidon; I thought it was fantastic. I believed it came out of a lab in Tokyo [it did: Saruwatari Lab, University of Tokyo]. The biggest one I came across next was NVIDIA's RE-USE, all capitals, which felt like it was made for exactly this new class of task. Its target use case seems to be cleaning up imperfect audio for creating TTS models.
I went with RE-USE because everything else in the project is NVIDIA: I'm training Parakeet and Nemotron. It felt the better fit for the rest of the stack.
So that's the recipe that ran in full: take my speech-to-text corpus, keep the code-switching part, drop the takes that are too noisy, run the rest through RE-USE. That's what Meorav v2.2 was trained on. Both models, and others I looked at, are in my Audio restoration collection on Hugging Face.
Listening to the cleaned clips was really impressive. Claude, at the time I'm recording this (I say that habitually now, because AI moves so fast that in a year this will sound like me talking about dinosaurs), is Claude Opus 5.5, the cutting edge. It's very proactive, and I have to rein it in with system prompts. It's insanely powerful, but it's like asking someone to vacuum the room and hearing back: I vacuumed the room, and I also repainted the walls and drilled some holes. Not really, but that's the approach.
In its eagerness, it went well beyond the five before-and-after clips I'd asked for, which were all I needed to say "great, run it over the whole corpus". It put together a whole composite of takes. One of them was a take where Ezra, my son, was crying in the background. I played the before clip and then the after clip, and I thought: oh my gosh, that is crazy good.
It removed the crying entirely, the classic background-noise use case. If I'd used DeepFilterNet at that strength, I'd have sounded like a robot. This both removed the noise and made me sound like I was speaking into a decent microphone. Extremely impressive, and extremely powerful.
There is so much exciting happening in AI at the moment, even within this little world of audio and TTS. It's crazy.
Before the next topic, one defect we didn't manage to iron out: the prosody is very flat. I told Claude: the quality is nice, and the vocabulary, the most important part, is good. My only remaining criticism is that while it no longer sounds like a phone call, it sounds very flat.
The reason is the training material: synthetic sentences that are, frankly, as dull as dishwater to read. "I need to pick up eggs and chicken at the supermarket." It's very hard to read that sentence with any life in it, so it naturally comes out flat. And that's about 99% of what I've recorded so far.
I'm trying to engineer my way around each of these obstacles, and the idea here was to generate artificial stories. I had two reasons. One was expression. The other was that the speech-to-text corpus is biased towards very short sentences, and I wanted longer ones.
It was frustrated from every direction:
First we tried building longer passages by stacking unrelated sentences. That was just weird: paragraphs that made no sense.
Then the made-up stories went too far the other way: sentences two paragraphs long.
Used for speech-to-text training, those long ones caused out-of-memory errors on the NVIDIA GPUs.
So the goal of one kind of training data that works for both models didn't pan out. But synthetic stories are still my main plan for TTS. I literally asked Claude for a short story: Daniel goes to the supermarket to buy eggs, but make it weird. Reading a few sentences like that pulls me out of the monotone. "He went and got the last tub of pancake flour and made the greatest pancake in the history of pancakes. Then he went to the makolet and said: I made pancakes." You know what I mean.
That's where Meorav stands: the flatness is a known quantity. We also tried tuning the temperature, fixing it at the technical level, and that didn't really work. Until the next run has enough expressive training data to displace the flat material, the phonetic reproduction will be decent but I'll sound really bored, or like I've taken some drug that makes me sound like this. It'll say "makolet" correctly, but sound like I've been lobotomised and lost any shred of emotion. For anyone who listens and wonders: I'm not actually that flat emotionally.
So right now it's a flat, imperfect, Irish-accented, code-switching TTS model, a wonderfully weird combination from every direction. The fascinating thing about fine-tuning is that it's about distributions: as I gather more data, and go out of my way to record expressive audio, the average will be pulled up in future training runs. That's where I'll leave the Meorav update for now.
Back to audio to audio, because yesterday I came across something absolutely mind-blowing in that Hugging Face task. Poking around near NVIDIA's RE-USE, I found speaker isolation. And I said: what? No way.
How to isolate the speaker is one of the training problems I've thought about most for ASR. It sounds trivial until you think about it, and then it's a world of complications. The examples I've given in earlier episodes: you're dictating an email walking through the shuk in Jerusalem, or whatever market or pharmacy is near you, holding your phone up to your mouth, and some guy calls out "hey", or someone cuts across you, there are conversations around you. A minute later you pass a bar blaring music and see the ASR transcribing the lyrics: wait, I didn't say that, that's just music. Or you're in an airport.
Part of what I wanted to find out with my fine-tune was whether that can be trained away by the simplest method imaginable: pairs where the ground truth is just what I said, recorded with music playing in a bar. Never training on the task deliberately, just giving it enough examples, can it learn the pattern? I don't know, because I don't have enough of those samples in my dataset, and I haven't tested it in those environments. They're not hard to find, in Jerusalem or anywhere: pull out your phone in a bar or a busy market, try to dictate an email, and you'll see the challenge.
This matters because of a challenge I hit today with email addresses. I expected to train them as a task in my Parakeet fine-tune, and ran into a road bump: Parakeet does not emit the @ sign [I said "ampersand" in the recording]. You can record samples of yourself dictating email addresses, which is what I'm doing, but those pairs are no use to the fine-tune for now, because the character isn't in the decoder. That really surprised me.
But Parakeet is fantastic, and NVIDIA's speech models are brilliant; I assume this was thought through and done deliberately. NVIDIA takes a modular approach and wants inverse text normalisation (ITN) as a separate pass after the ASR. So, working with the system rather than against it, I'm looking at adding a small model between the ASR and the point where the app releases the text. It can be used with or without the post-processing model I've also trained, a fine-tune of S1-mini.
The basic stack would then be: dictation, and if Parakeet outputs an email address, say "daniel at danielrosehill.com", the ITN layer normalises it with the @ sign Parakeet can't emit.
I mention this because it's pulling me in a direction I thought was far too complicated and would have done anything to avoid. Last year I thought audio multimodal models would just take over from ASR: we wouldn't need ASR if we could feed in mixed audio and text and get text out.
The total opposite says: no, ASR is the workhorse, and we go very niche and very modular, with small models for post-processing and small models for normalising email addresses. That's the direction I'm currently looking at, on the output side of the chain.
Today, picking up little Ezra from daycare, I thought about the speaker isolation models again. I didn't make the connection straight away, but then: hang on, this could go in the input chain. Speaker isolation before the model, alongside voice activity detection (VAD), maybe in parallel. I'm not sure exactly how that chain would work, but the output goes into the ASR.
Then telling the intended speaker apart from incidental noise isn't the ASR model's concern at all. It's the VAD approach again: just as VAD keeps silent audio away from the model, a second bouncer stands in front of it, and on the other side there might be two more layers.
So it's very plausible that a seemingly simple dictation pipeline could be five models, one after the other, on a phone, on low-capacity hardware. That sounds wild to me, but that's where I see it heading.
And there's a lot to be said for it. Maybe it's because I'm 37 and grew up when the floppy disk was high-capacity computing, but it still blows my mind that we can run ASR models on a smartphone at all, let alone five models back to back in the blink of an eye, every stage happening at a pace no human can perceive.
That's what I'm looking at and thinking about at the moment, and it's really fun to work out how these pieces can fit together. I'll be trying the pipeline out. Recordings I'm adding to the datasets on Hugging Face where there's a conversation in the background are now really useful, because they're exactly the samples to test speaker isolation on. I'm going to get more of those and see how well these models work.
These are all very new models in audio to audio, and very exciting. They'd probably make my training work on code-switching and tech vocabulary easier, too: I can drop tasks I'd been planning, like handling unintended audio, and say that's out of scope, that's what the isolation model does. The ASR model focuses purely on recognition, and post-processing is a separate model too.
If that's the way professional dictation is going, I've done a 180, and I'm very much in favour. It's just too hard to think about everything at once once you really look at dictation: where and how it can break, and what has to come together to go from 90% accurate to 100%. I think domain-specific small models are going to win the day.
That's a project update on all the little moving parts. A lot of fun, I have to say, and more updates are coming soon to this informal podcast for Hebrish.
Article and transcript written up from the recording by Opus 5.5 (Claude, Anthropic), working with Daniel. Nothing is added that wasn't said; corrections are marked.
Audio: lightly edited (fillers and long pauses cut) and restored with Sidon (University of Tokyo).
