How I record training data for Hebrish

I made this short video to show how the training data for my speech recognition fine-tunes actually gets collected. I've been using dictation almost around the clock for coming up on two years. It feels a bit weird to say I fell in love with dictation, but from the first time I used Whisper I thought it was going to be life-changing, and I've used it everywhere, all the time, ever since. This is the less glamorous side of that: the coal face of speech AI development. The video runs about 11 minutes, and this is the same tour in writing.

In this video

• 0:00 Intro
• 0:45 Why I started
• 1:55 The recording app
• 3:02 Recording with a gamepad
• 4:08 Recording and syncing takes
• 6:18 How the sentences are made
• 7:21 Longer reading passages
• 8:23 Small fine-tunes every few days
• 9:24 The models

Why I started

I work on several ideas in parallel, because I find it easier to collect the data, grow the datasets and then push them into different training pipelines. It started with code-switching, which, living in Israel, is the most everyday problem I have with dictation: Hebrew words in the middle of English sentences that it just doesn't get.

Then I realised there's a whole bunch of things that off-the-shelf dictation struggles with, even great models like NVIDIA's Parakeet. Tech brands are the next one: they come and go so quickly that a speech recognition corpus trained on older material can't keep current. And now I'm adding smaller things like formatting. If I say I have a meeting on August 9, I want it written as August 9. Where post-processing should begin and speech recognition should stop is very much the open question I'm thinking about, and I think other people in the voice space are too.

The recording app

I'm a real desktop fan, but for the past few months I've been on a laptop since we moved apartment, and I've leaned into it for recording. I did a lot of takes on my phone, but I have a hard time trusting Android to do a good job of capturing audio at the bitrate I record at. So this is my humble ThinkPad E15, running the recording app that Claude Code built for me. It works on the phone too, and it's meant to be really slick on both, which is a hard thing to get right with a responsive design.

The app says I've recorded 31,000 clips. I do sleep, I promise: it's all been done in batches. What I've learned about collecting this data is that, from my perspective at least, the interface is everything. If it's easy to record in batches, you get into a kind of flow state, some of the sentences are even quite interesting, and before you know it you've pushed up another 100 or 200. That's why I've put so much effort, with Claude of course, into making it really easy to use.

Recording with a gamepad

At my laptop the microphone sits in one place and the shortcuts are somewhere else, and I didn't want to spend the whole day leaning over the keyboard. So I asked myself what I could use instead, and landed on a gamepad: the Logitech F310, a blast from the past. I haven't used a game controller since I was a teenager, so it's funny to be rediscovering them. I went for a wired USB one, because when you do this for hours you really come to appreciate USB's reliability. I got Claude to print me a little mapping of the buttons, because I keep forgetting which is which: start recording, save, retake and skip. They'll be muscle memory soon. Just off screen are the other things an AI voice recorder needs in life, like plenty of Coca-Cola.

Recording and syncing takes

The app has a record button, an auto-next setting that moves on to the next sentence by itself after a short gap I can set, and a sync. Takes go up to a Cloudflare R2 bucket first, and are then pulled to Hugging Face periodically. When a fine-tuning run starts, it pulls its data from Hugging Face. There's also an urgent sync for when I want to skip the staging step.

A lot of the time I get a sentence in one go, and a lot of the time it takes two or three takes before the pronunciation is right. In the video you can see a few real ones, about server backups, Viterbi decoding with a larger beam width and trimming clips in LosslessCut. I keep a jug of water next to me and take a break every ten clips or so.

I also built a small part of the app for capturing ideas, because when I'm recording, especially out of the house, I keep thinking of vocabulary to add. Those notes go up to Hugging Face too.

How the sentences are made

The task view lets me choose what I'm training on. The sentences are grouped into tasks, priority words override them, and I can save mixes, so I can decide that today is a phonetics day. There's a vast amount there now.

A lot of it is generated by a language model, but with me steering. I have a Claude Code skill for vocabulary, where I tell Claude what's missing and what I want emphasised. Another one is for fixing things in retrospect: it goes through my own transcription history and finds the words that were mistranscribed. With the core vocabulary skill, Claude and I work out some candidate examples, we make word and context pairs, and then DeepSeek V4.1 Flash on OpenRouter writes a batch of sentences from them. You could call it supervised synthetic data generation.

Longer reading passages

There's a bit of real text as well, from fully open books, plus reading stories in order. One weakness of the training data is that it's mostly short sentences. That's good for learning vocabulary, but if I want the model to learn where one sentence ends and the next begins, it needs longer training pairs. So the app can serve two sentences at a time instead of one. It still has a bug or two (there are always bugs), but that's the idea. I asked Claude to write these as narratives so they're less mind-numbing to read: I read AI-written text that gives me something to actually think about, and that makes an hour or two at a time very doable.

Small fine-tunes every few days

It sounds a bit crazy to have recorded 65 hours of audio so far. But I train a new checkpoint every few days, download the weights and use them myself. I hate the expression "eat your own dog food", and I don't have a dog, but I eat my own very liberally. That's how I get real feedback on where the fine-tunes are improving and where they're still lagging behind, and it tells me what to record next.

I wanted to get to a strong baseline quickly, which is why I did a lot of recording in fairly quick succession. From now on I see myself recording a few hours a week, maybe more, finding what's failing and adding new vocabulary. The idea of very small incremental fine-tunes really appeals to me.

The models

• Sabich is Parakeet v2 fine-tuned on my Hebrew-English code-switching and tech vocabulary, updated on a rolling basis.
• Shakshuka is the streaming one, on an NVIDIA Nemotron base, with the code-switching training.
• Falafel is trained on short commands like "open tab" and "close tab", a subset of the bigger training data. The idea is a desktop voice agent.

Claude came up with all the names. There's also Fritter, just the tech side without code-switching, which isn't a Hebrish model. If you use Claude Code or similar tools, you're constantly dictating things like IP addresses, and that's a subset I've trained on, so I hope it's useful to other people as well.

The models are open weights on Hugging Face. Some are gated: the Hebrew ones are a bit more sensitive, and I don't want people misusing them.

It's been a lot of fun, and it's also a daily business tool for me: I dictate for maybe four hours a day across all my devices. It's a slightly weird and occasionally tedious process, but it's very worthwhile, and it's a really nice feeling to watch the vocabulary improve.

Get in touch

• Report a bug or ask for a feature, including words that come out wrong.
• Send me recordings in your own voice and accent.
• Contact me about anything else, including GPU time.
• Follow the Hebrish channel on YouTube for the next update.

Thanks for watching, and for reading.