Recording on Jaffa Street: accuracy, cleanup models and more voices

The audio is lightly edited: most "um"s and "uh"s cut, long pauses shortened, EQ and level evened out (22:30 recorded, 21:04 published). The transcript below was made with AssemblyAI and lightly edited for clarity: filler words and false starts removed, misheard names corrected. Timestamps jump to that point in the recording. Older updates: all updates.

0:00 Recording on Jaffa Street

This is the second edition of the informal podcast for the Hebrish app. It's not really a podcast so much as me providing periodic updates. Today is the 30th of October. Halloween tomorrow. There you go.

I'm sitting here on a bench on Jaffa Street in Jerusalem, outside Powers Coffee, recording some voice samples. I tried a few last night in the shuk, the Jerusalem shuk, which was very, very noisy. The ones I'm recording here have a medium level of noise: a few conversations. As a challenging noise environment this is probably a bit too easy, in the sense that there's no blaring music. This is a very popular place with English speakers, so there are a few English conversations in the background, and Hebrew, and a nice little background hum of the city.

As a reminder to self: I recorded three minutes, unprocessed, of just the raw noise here, capturing the conversations. And I'm describing the scene on Jaffa Street: people walking back and forth in both directions, families talking. The most distinctive audio feature is the Jerusalem Light Rail. The train comes by every few minutes in either direction and does its little bell thing. That's in most of the samples I've recorded today.

1:32 Two years of dictation

In general, I wanted to share an update on this project, what I've been doing it for and what I've been doing with it for the past couple of weeks. The main thing I'll say is that my intention is very much to maintain this over the long term. It's going to be two years of daily use in November. It was actually Thanksgiving two years ago, when my wife was in the US, that I went all in on voice transcription and began using Whisper for everything I could. So I'm almost at my two-year anniversary, and on any given day I'd conservatively estimate I use this for about three hours, on the desktop or the phone.

I'm mostly using Parakeet now instead of Whisper. But I think that's less important than this: when I was growing up, my mum bought me Mavis Beacon at a young age to learn touch typing. A brilliant idea. If you're talking about digital literacy, you see people in their 50s pecking at the keyboard and you think, wow, this was a really great thing to learn.

My typing speed is 100, 120 words per minute, and could probably use some touch-ups. Because I learned typing young and was a proficient, fast typist, I always felt that's how I'm going to use a computer. I'm probably repeating what I said in the first episode, or something I've talked about many times, but I tried speech to text over the years and it wasn't that good. When I finally found Whisper almost two years ago, it was like, whoa, this is suddenly vastly better.

3:12 Where accuracy caps out

So I have a very real self-interest in getting the accuracy of these fine-tunes up. It's not something I have to finish by tomorrow. The reason I've done a couple of releases this week is that I'm eager to see, first, that the WER is moving in a positive direction. There's an interesting question of where it caps out. Even if you had hundreds of hours of just reading code-switching words, is there a point where you'd still get a WER greater than zero?

The answer I've seen so far is yes: about 2% is generally the ballpark. And of course you're going to have imperfect conditions. In the real world a certain amount of error comes from the audio chain: the microphone you're using, the background noise. So even with a fantastic model you'll get some errors, because you're not getting an ideal signal into the model in the first place. That's what I'm bearing in mind.

I'm not obsessed with chasing tiny gains. It is interesting to me that the [accuracy on] words not seen in training is increasing, which speaks to a pattern being recognized in which words are Hebrew, and even tech acronyms. And the WER is still falling. I expect diminishing returns, but it's still going down a little bit. What I've been more interested in, to be honest, in this week's incremental updates is everything in my training dataset that isn't domain vocabulary. That's actually a pretty long list.

5:07 ASR or post-processing?

I come up with tasks for it as I use this every day: oh yeah, that's something that drives me mad about the way Parakeet works. Is that something we can train out? I mentioned in the last episode that what I'm thinking about at the moment is where the boundary lies between ASR and post-processing. Which tasks are suitable to train into the ASR step, and which should properly be left for post-processing? Yesterday I created an experimental fine-tune of S1.

S1, by Superwhisper, is a post-processing model. I made a few hundred synthetic sentence pairs just for adding paragraphs and doing a bit of light-touch editing. I also noticed that the paragraph model I released, the small model, works: I was using it yesterday.

I think it's good to have both at the moment, because of the separation of duties. That's really the question I'm thinking about for audio. It blows my mind a bit that inference can be so quick that you can have an ASR model feeding into a paragraph-break model feeding into post-processing. You can really break down tasks. I did a couple of benchmarks of the inference and of what I call the junction.

That's not an official word, just: what's the latency between steps? If I get text out of Parakeet, then run it into my small paragraph model, which is built on ModernBERT, then into Whisper, how long does it take? The answer is negligible: a few milliseconds, not perceptible. So I'm experimenting with audio chains and I'm in awe of how much of a luxury it is: if you want to achieve these results, you don't need to do it all in one model.

There's a good argument to be made for thinking about which way is best. But for the moment I'm in an experimental phase and keeping my options open. That's why I released the small BERT model just for paragraph breaks, and why yesterday I said, let's try a quick fine-tune of S1-mini. If that works well, that would probably be my preference. I'm going to try to do as much as I can in the fine-tuning, like getting number formatting correct, and then the cleanup model, S1-mini, does the paragraph inference, which to me is the key task.

7:58 Numbers, currencies and dates

And maybe some things just aren't really proper in this model. I'll go into what I think those are: the experimental subset of tasks in this fine-tune. There's a model I'm training for my own purposes called [name redacted]. It's on my personal Hugging Face, as opposed to this Hebrish project.

What it is: no vocabulary training. I have one that's trained on tech vocabulary plus these things, and this one is just these things. By these things I mean the list of improvements I'm trying to bake into Parakeet itself. My idea for how this would work in the future is something like this, at a data level.

This is my data plan. Currently this model, [name redacted], is Parakeet plus all those things. I'll give an example: number formatting, which is actually a tough one. "The cake cost me $5.50." The task I'm training on is to render that as dollar symbol, 5, decimal, 50.

Why it's tricky: when you say "the cake cost me five fifty", the decimal isn't stated, and you could render that as "$5 50". Native Parakeet will render it as "dollars". So I'm training on that, with the major currencies. In Israel we have shekels, NIS. Typically that's written as, say, "I bought a cake for 50 NIS."

So I'm creating sentences to train currency formatting. The next one is date handling. For example, "I booked the appointment for the 18th." My preference, if that were going into an email or WhatsApp, is "18th": the ordinal number written with the "th" suffix. Little things like that.

10:09 Disfluencies, false starts and self-corrections

The more challenging tasks, the more deterministic ones, are in the category of disfluencies. The classic ones are the "um" and the "ah". Those are actually the least priority, because they're the ones that can be handled within the app you're using: if the model emits them, you can run a regex at inference and not show them. Ideally it should be done in the model, which is why I'm training on the task. But there is an escape hatch even if it doesn't work.

With false starts and inappropriate capitalization, not so much. False starts are something I do all the time in dictation, so that's an important one I'm training on. For example, you start a sentence, "I need to go to the bakery to", and then: "pick up, my shopping list is grapes, apples" or something like that. Maybe you're recording notes for the day and you just leave the first thought hanging in the air. The task is to detect that for what it is.

To me that's probably a job for post-processing proper, because the attention mechanism needs to look at the whole sentence to say, wait a second, this is a thought that never got finished; there was a sentence boundary. That's asking a lot of the first-stage model, I think, but it's an open experiment whether it can be done in the first pass. The other one is self-corrections, and again it's easy to generate examples: "I need to buy apples, wait, I mean oranges." What's unique about these features of speech is that the self-correction can take the form of "wait", "scratch that",

"I mean", so the vocabulary is wider than you'd think. Again, I'm looking at whether this can be done in the first forward pass of the ASR model. It's really a question of the technical makeup of these models, which is why I'm keeping an open mind. I've had very good success with NVIDIA so far, and I'm thinking about these questions as new models come out: what's the most appropriate base? My philosophy is to do as much as can be done in the first forward pass, with post-processing as a fallback, plus paragraph inference. Maybe in the end all that needs doing afterwards is paragraphs, which is why the separate model would have a viable independent use, if that's literally the only thing it's doing.

13:29 Learning from my own mistranscriptions

I'm doing a couple of things at the moment. One is exporting my transcription history. I have a skill I use with Claude Code: I'm using the app, my own fine-tune, every day, and the skill is something like "look at this and infer what's being mistranscribed".

It does a very good job. It can identify not only the error but what it should have been. If the word isn't already in one of the training datasets, it gets added; if it is, it gets a priority flag and I add more sentences to mitigate it. So that's an adaptive approach based on my own transcription history, and I'm seeing it work well. On Android it's a bit less easy, which is why I built Export CSV into the app. Hopefully it's a useful feature for people too.

It's something I always look for in apps, but I'm careful to have it as an option, because it means your transcription history is sitting there on disk. I added it mostly for my own use for now: I can export and see whether the Android app is getting stuff wrong that the desktop handles well. So I'm using that for the corrective loop too.

14:47 Noise, walking and names

Big picture, the accuracy is improving, and what I'm doing at the moment is gathering examples from different surroundings, like sitting here on Yafo. It's going to get noisier once I'm in the shuk. When I use these transcription apps day to day, I'm often in much, much noisier environments than this. One open question for me is whether there's a level of noise at which the training data isn't really usable, because it's training on audio the app doesn't stand a chance on. I'm using technical parameters to score the noise in the samples, and doing some annotations of the tech vocabulary. I'm just pacing around now.

A note to self: I want more samples where I'm walking, because I often start walking when I'm dictating, in the house or elsewhere. And I'm buttressing people's names, Jewish names, Israeli names, greetings: anything underrepresented in the dataset is something I'm training on. Ah, there's a bench I can sit down on, so I'm moving over to the bench. Oh, there's the light rail.

So that's where we are. I'd say there's a good representation of vocabulary, but I keep finding gaps. I'm also using a manual skill with Claude where I say, wait, why don't we have this?

16:14 More voices: IPA and synthetic speech

So I'm adding the gaps I see. I also feel there's too much repetition of niche vocabulary, so that's something I'm working on. And speaking of accents: one other thread I've pulled on this week. Obviously I have an Irish accent, and for this dataset, the Hebrew code-switching thing, to be useful to other people, different speakers are going to have different needs.

I mentioned that I'm working on it. I got my wife to record a few samples, and I'm sure I'll get my buddies to do some, but there's a lot of my own training data in there already. So this week I was asking myself what I could do.

Is there anything clever I could do? One idea I came up with was to think about what I'm actually doing when I gather this training data: I'm pronouncing things, pronouncing the target words, basically.

My experiment was: what if I transcribe that to IPA, then use a TTS model that supports IPA to pronounce it, and create synthetic samples in different voices? Say I've recorded a sentence, the classic example, "I need to go to the makolet". Once I've transcribed makolet to IPA from my sample, I can use any TTS model with IPA support to do either the whole sentence, or just generate "makolet" in the IPA model and the rest with an English model, ideally in the same voice. I was using Kokoro for that.

I had mixed success with it, but I was able to show it needs review: some of the clips, and some of the TTS engines, simply didn't work. It wasn't too hard to go through the synthetic training samples and mark them correct: a few seconds per clip, and then you have them in different voices and accents. Strangely, where it was less good was getting the accent right.

So I think it's about finding a TTS model with a lot of accent support and IPA recognition. That's another thing I'm looking at, as a potential solution to the single-voice issue: take my raw dataset for code-switching, and the other samples for that matter, run them through IPA, generate synthetic TTS in different voices, and the final training dataset is a blend of me reading plus synthetic voices.

19:15 Next releases and sign-off

And maybe for my own personal fine-tunes I'll keep just my own voice. So those are the miscellaneous updates for this week. I'll probably be doing fewer trainings, more spaced apart. The latest model as I've trained it is 2.2. I'm going to do 2.3 today on Sabich, the main model, and see how the benchmarks go; hopefully another bit of improvement. Coming into next week I'm going to record longer stretches at a time.

As the model gets better, the trainings will be less frequent. What I envision as a more sustainable cadence for this project is using it for a week or two, and then, as the accuracy becomes very good, pinpointing over time: these are behaviours, this is vocabulary, these are words it's not recognizing well. Then a much more targeted short list of words gets trained on between incremental releases. So it might be smaller dataset deltas between releases that are more spread out. I think that's probably the way it will go. The validation so far has been good, and now I'm really curious to see how it does with other people's voices. So if you've used it, do drop me a line.

I'm very interested to hear. I'm sending it out this week to just a few people, people at organizations I think might find this really useful. Feel free to download it. iOS: I still haven't figured anything out, but I can work on that next week.

That's it from Jerusalem, on Friday, by Yafo Street. Shabbat shalom.