Recording on Jaffa Street: accuracy, cleanup models and more voices
A voice note from a bench on Jaffa Street: where accuracy caps out, what belongs in the speech model and what in post-processing, and how to get more voices into the training data.
Accuracy is still improving. About 2% WER looks like the general ballpark for where it caps out.
I'm testing where ASR should end and post-processing begin: an S1-mini fine-tune, plus the small paragraph model. The latency between steps is a few milliseconds.
New training tasks: prices, currencies and dates, then false starts and self-corrections.
My own transcription history now shows what's being misheard, and Export CSV brings that to Android.
I'm recording in noisier places and while walking, and filling gaps in names and greetings.
An experiment with IPA and synthetic speech for more voices had mixed results.
Sabich v2.3 trains today. After that, expect fewer, more targeted releases.
I recorded this second update on a bench on Jaffa Street in Jerusalem, outside Powers Coffee, the day before Halloween. I was there to record voice samples: English and Hebrew conversations in the background, a nice hum of city noise, and the Jerusalem Light Rail doing its little bell thing every few minutes.
This week was less about new vocabulary and more about everything else in the training data: where accuracy tops out, which jobs belong in the speech model and which come after it, and how to get voices other than mine into the dataset.
Why I'm in this for the long term
1:47 November marks two years of daily voice typing for me, mostly on Parakeet these days rather than Whisper; I'd conservatively estimate about three hours a day. That's a turnaround for someone whose mum bought him Mavis Beacon to learn touch typing, and who types at 100 to 120 words per minute.
So I have a very real self-interest in getting the accuracy of these fine-tunes up. There's also an interesting question of where the WER caps out: so far, about 2% looks like the general ballpark, and some error always comes from the microphone and the background noise. What interests me more than tiny gains is that accuracy on words not seen in training is going up, which suggests the model is picking up a pattern in which words are Hebrew.
What goes in the speech model, and what comes after
5:22 I'm thinking about where the boundary lies between ASR and post-processing. I made a quick fine-tune of S1-mini, a post-processing model by Superwhisper, on a few hundred synthetic sentence pairs for paragraphs and light-touch editing. And the small paragraph model I released works: I've been using it.
It blows my mind a bit that inference can be so quick that you can have an ASR model feeding a paragraph-break model feeding post-processing.
I benchmarked what I call the junction, the latency between steps. It's a few milliseconds, not perceptible. You don't need to achieve these results in one model. If the S1-mini fine-tune works well, my preference would be to do as much as I can in the fine-tuning, like number formatting, and let the cleanup model do the paragraph inference.
Teaching it how I actually talk
8:13 I'm training a list of tasks I'd like baked into Parakeet itself. Prices: "the cake cost me five fifty" doesn't state the decimal, and native Parakeet writes out "dollars". Currencies, including shekels written as NIS. Dates like "the 18th", with the ordinal suffix I'd want in an email or on WhatsApp.
10:24 False starts are something I do all the time: "I need to go to the bakery to", and then off into my shopping list. Spotting a thought that never got finished probably needs the whole sentence in view, which is a lot to ask of the first stage, but it's an open experiment. Self-corrections ("I need to buy apples, wait, I mean oranges") are easy to generate, though the vocabulary is wider than you'd think: "wait", "scratch that", "I mean".
My philosophy for now: as much as possible in the first forward pass, post-processing as a fallback. Maybe in the end all that needs doing afterwards is paragraphs.
Learning from my own mistranscriptions
13:44 I export my transcription history and run a skill in Claude Code that amounts to "look at this and infer what's being mistranscribed". It finds not only the error but what it should have been. A missing word gets added to the training data; one already there gets a priority flag and more sentences.
That's why I built Export CSV into the Android app. I keep it as an option, because it means your transcription history is sitting there on disk, but it shows me whether Android gets things wrong that the desktop handles well.
15:02 I'm also recording in noisier places, like the shuk, and scoring the noise, because one open question is whether there's a level of noise at which training data stops being usable. I want more samples of me walking, and I'm buttressing Jewish and Israeli names and greetings.
More voices through IPA and synthetic speech
16:29 I have an Irish accent, and different speakers are going to have different needs. My wife recorded a few samples, but there's a lot of my own voice in there already. So I tried something:
Pronounce the target words, which is what my recordings already do.
Transcribe to IPA, so a word like makolet has a precise pronunciation.
Synthesise it with a TTS model that supports IPA, in different voices (I used Kokoro).
Review the clips and mark the correct ones, a few seconds each.
Blend me reading with the synthetic voices in the final dataset.
It had mixed success. Some clips and some TTS engines simply didn't work, and strangely, where it was weaker was getting the accent right. The job now is finding a TTS model with a lot of accent support and IPA recognition.
What's next
19:30 On the day I recorded this, the latest model was 2.2 and I was about to train Sabich v2.3. As the model gets better I'll train less often: use it for a week or two, pinpoint what it still doesn't recognise well, and train on a targeted short list. Smaller dataset deltas, releases more spread out.
Now I'm really curious how it does with other people's voices. If you've used Sabich, do drop me a line through feedback.
That's the written version. The full episode has the light rail bell, a move to a bench mid-sentence, and a Shabbat shalom from Yafo Street: give it a listen.
Full transcript, cleaned up 2,407 words · everything said in the episode, written to be read
This is the recording written up to be read: everything I said is here, tidied into sections. The audio is lightly edited: most "um"s and "uh"s cut, long pauses shortened, EQ and level evened out (22:30 recorded, 21:04 published). Then the voice was restored with Sidon (University of Tokyo) to sound closer to a studio microphone. The intro and outro are voiced by Herman Poppleberry of My Weird Prompts, over “Happy Strummin’” by Jason Shaw (music by audionautix.com, CC BY 3.0); the recording starts at 0:15. Timestamps jump to that point in the recording. All episodes: podcast.
This is the second edition of the informal Hebrish podcast, which is less a podcast than me giving periodic updates. It's 30 October, the day before Halloween, and I'm on a bench on Jaffa Street in Jerusalem, outside Powers Coffee.
I'm here to record voice samples. I tried a few in the shuk last night, which was very, very noisy. This spot sits somewhere in the middle: there's no blaring music, so as a challenging noise environment it's probably a bit too easy, but it's a popular place with English speakers, so there are English and Hebrew conversations in the background and a nice hum of city noise.
I also recorded three minutes of the raw, unprocessed noise here, as a reminder to myself of the scene: people walking in both directions, families talking, and above all the Jerusalem Light Rail, which comes by every few minutes and does its little bell thing. It's in most of today's samples.
What I mainly want to say about this project is that I intend to maintain it for the long term. November will mark two years of daily use. It was Thanksgiving two years ago, with my wife in the US, that I went all in on voice transcription and started using Whisper for everything I could. On any given day I'd conservatively estimate I use it for about three hours, on the desktop or the phone. These days it's mostly Parakeet rather than Whisper.
The bigger point goes back further. When I was growing up, my mum bought me Mavis Beacon so I could learn touch typing, which was a brilliant idea. If you're talking about digital literacy, you see people in their 50s pecking at the keyboard and realise what a great thing that was to learn. My typing speed is 100 to 120 words per minute, and could probably use some touch-ups.
Because I learned young and typed fast, I always assumed that was how I'd use a computer. I tried speech to text over the years and it wasn't that good. Then I found Whisper almost two years ago, and it was suddenly vastly better.
So I have a very real self-interest in getting the accuracy of these fine-tunes up. It doesn't have to be finished by tomorrow. I've done a couple of releases this week because I'm eager to see the WER moving in the right direction, and because there's an interesting question of where it caps out. Even with hundreds of hours of reading code-switching words, is there a point where you still get a WER above zero?
So far the answer looks like yes, with about 2% as the general ballpark. Real conditions are never perfect either. Some error always comes from the audio chain, the microphone and the background noise, so even a fantastic model makes mistakes when it isn't getting an ideal signal in the first place.
I'm not obsessed with chasing tiny gains. What does interest me is that accuracy on words not seen in training is going up, which suggests the model is picking up a pattern in which words are Hebrew, and even in tech acronyms. The WER is still falling a little, though I expect diminishing returns. To be honest, what's interested me more in this week's incremental updates is everything in the training data that isn't domain vocabulary, and that's a pretty long list.
I come up with tasks as I use this every day: that thing about Parakeet drives me mad, can we train it out? As I mentioned last time, I'm thinking about where the boundary lies between ASR and post-processing. Which tasks belong in the ASR step, and which should be left to post-processing?
Yesterday I created an experimental fine-tune of S1, a post-processing model by Superwhisper, using a few hundred synthetic sentence pairs just for adding paragraphs and some light-touch editing. I also noticed that the small paragraph model I released works: I was using it yesterday.
Separation of duties, and the junction
I think it's good to have both for now, because of the separation of duties. It blows my mind a bit that inference can be so quick that you can have an ASR model feeding a paragraph-break model feeding post-processing. You can really break tasks down.
I ran a couple of benchmarks of the inference and of what I call the junction, which isn't an official word: it's just the latency between steps. If I get text out of Parakeet, run it into my small paragraph model built on ModernBERT, then into Whisper, how long does that take? The answer is negligible: a few milliseconds, not perceptible. I'm in awe of what a luxury that is. You don't need to achieve these results in one model.
Keeping my options open
There's a good argument for thinking carefully about which way is best, but for now I'm in an experimental phase. That's why I released the small BERT model just for paragraph breaks, and why I tried a quick fine-tune of S1-mini yesterday. If that works well, it would probably be my preference: do as much as I can in the fine-tuning, like getting number formatting right, and let the cleanup model, S1-mini, do the paragraph inference, which to me is the key task.
Some things maybe just don't belong in this model, and that's what an experimental subset of tasks in this fine-tune is for. I'm training a model for my own purposes called [name redacted], on my personal Hugging Face rather than in the Hebrish project. I have one that's trained on tech vocabulary plus these tasks, and this one is just the tasks, with no vocabulary training: the list of improvements I'm trying to bake into Parakeet itself. That's my data plan for now: Parakeet plus all of those things.
Here's what those things look like:
Prices. "The cake cost me $5.50" should come out with the dollar symbol, 5, decimal, 50. That's actually a tough one. When you say "the cake cost me five fifty", the decimal isn't stated, so it could be rendered as "$5 50", and native Parakeet will write out "dollars".
Currencies. I'm training on the major currencies. In Israel we have shekels, typically written as NIS, as in "I bought a cake for 50 NIS." So I'm creating sentences to train that formatting.
Dates. In "I booked the appointment for the 18th", my preference for an email or WhatsApp is "18th": the ordinal number with the "th" suffix. Little things like that.
10:24 Disfluencies, false starts and self-corrections
The more challenging tasks are disfluencies. The classics are "um" and "ah", and those are actually the lowest priority, because they can be handled in the app: if the model emits them, a regex at inference can hide them. Ideally the model does it, which is why I'm training on the task, but there's an escape hatch if it doesn't work.
False starts
False starts and inappropriate capitalization have no such escape hatch. False starts are something I do all the time in dictation, so that's an important one. You start a sentence, "I need to go to the bakery to", and then go off with "pick up, my shopping list is grapes, apples". Maybe you're recording notes for the day and you leave the first thought hanging in the air. The task is to detect that for what it is.
To me that's probably a job for post-processing proper, because the attention mechanism needs to look at the whole sentence and realise this is a thought that never got finished, and that there was a sentence boundary. I think that's asking a lot of the first-stage model, but whether it can be done in the first pass is an open experiment.
Self-corrections
Self-corrections are easy to generate examples for: "I need to buy apples, wait, I mean oranges." What's unusual about them is that the correction can take the form of "wait", "scratch that" or "I mean", so the vocabulary is wider than you'd think. Again, I'm looking at whether this can be done in the first forward pass of the ASR model.
My philosophy for now
It really comes down to the technical makeup of these models, so I'm keeping an open mind. I've had very good success with NVIDIA so far, and I'm thinking about these questions as new models come out: what's the most appropriate base? My philosophy is to do as much as possible in the first forward pass, with post-processing as a fallback, plus paragraph inference. Maybe in the end all that needs doing afterwards is paragraphs, which is why a separate model could have a viable independent use if that's literally its only job.
I use my own fine-tune in the app every day, so one thing I'm doing is exporting my transcription history. I have a skill in Claude Code that amounts to "look at this and infer what's being mistranscribed", and it does a very good job. It can identify not only the error but what it should have been.
If the word isn't already in one of the training datasets, it gets added. If it is, it gets a priority flag and I add more sentences to mitigate it. So it's an adaptive approach based on my own history, and I'm seeing it work well.
That's a bit less easy on Android, which is why I built Export CSV into the app. Hopefully it's useful to other people too. It's something I always look for in apps, but I'm careful to keep it as an option, because it means your transcription history is sitting there on disk. For now I added it mostly for my own use: I can export and see whether the Android app is getting things wrong that the desktop handles well, which feeds the same corrective loop.
Big picture, accuracy is improving, and right now I'm gathering examples from different surroundings, like here on Yafo. It'll get noisier once I'm in the shuk, and day to day I'm often in much, much noisier places than this. One open question is whether there's a level of noise at which training data stops being usable, because it's training on audio the app doesn't stand a chance on. I'm using technical parameters to score the noise in the samples, and doing some annotations of the tech vocabulary.
I'm pacing around as I say this, which reminds me that I want more samples of me walking, because I often start walking when I dictate, in the house or elsewhere. I'm also buttressing people's names, Jewish names, Israeli names and greetings: anything underrepresented in the dataset gets trained on. (There's a bench, so I'm moving over to it. And there's the light rail.)
So that's where things are. There's a good representation of vocabulary, but I keep finding gaps. I also use a manual skill with Claude where I say, wait, why don't we have this?
I'm adding the gaps as I see them, and I also feel there's too much repetition of niche vocabulary, so I'm working on that too.
The other thread I pulled on this week is accents. I have an Irish accent, and for this code-switching dataset to be useful to other people, different speakers are going to have different needs. I got my wife to record a few samples, and I'm sure I'll get my buddies to do some, but there's a lot of my own data in there already. So I asked myself whether there was anything clever I could do.
The experiment
When I gather this data, what I'm really doing is pronouncing the target words. So what if I transcribe that to IPA, then use a TTS model that supports IPA to pronounce it, and create synthetic samples in different voices? Take the classic sentence, "I need to go to the makolet". Once makolet is in IPA, any TTS model with IPA support can do the whole sentence, or just generate "makolet" with the rest coming from an English model, ideally in the same voice. I was using Kokoro for that.
How it went
I had mixed success. It needs review: some clips, and some of the TTS engines, simply didn't work. But it wasn't hard to go through the synthetic samples and mark them correct, at a few seconds per clip, and then you have them in different voices and accents. Strangely, where it was weaker was getting the accent right.
So I think the job is finding a TTS model with a lot of accent support and IPA recognition. As a potential answer to the single-voice issue, the idea is to take my raw code-switching dataset and the other samples, run them through IPA, generate synthetic TTS in different voices, and make the final training dataset a blend of me reading plus synthetic voices. For my own personal fine-tunes, maybe I'll keep just my own voice.
Those are the miscellaneous updates for this week. The latest model I've trained is 2.2. I'm going to do 2.3 on Sabich, the main model, today and see how the benchmarks go; hopefully another bit of improvement. Going into next week I'm going to record longer stretches at a time.
I'll probably be training less often and more spaced apart as the model gets better. The more sustainable cadence I can see is using it for a week or two and, as accuracy gets very good, pinpointing over time the behaviours, vocabulary and words it's not recognizing well. Then a much more targeted short list gets trained on between incremental releases, so smaller dataset deltas and releases more spread out.
The validation so far has been good, and now I'm really curious how it does with other people's voices. If you've used it, do drop me a line. I'm sending it out this week to just a few people at organizations I think might find it really useful, and you're welcome to download it. On iOS, I still haven't figured anything out, but I can work on that next week.
That's it from Jerusalem, on Friday, by Yafo Street. Shabbat shalom.
Sources and links
Hebrish models and pages
Sabich: the speech-to-text model; v2.3 was trained the day of this recording
Article and transcript written up from the recording by Opus 5.5 (Claude, Anthropic), working with Daniel. Nothing is added that wasn't said; corrections are marked.
Audio: lightly edited (fillers and long pauses cut) and restored with Sidon (University of Tokyo).
