Project update and next sprint

The audio is lightly cleaned: long pauses shortened, level evened out. The transcript below was made with AssemblyAI and lightly edited for clarity: filler words and false starts removed, misheard names corrected. Timestamps jump to that point in the recording. Corrections made after recording are listed at the end. The unedited transcript is in the project's planning folder on GitHub, with the requests and decisions drawn from it; the project notes are in the project context.

0:00 Why this voice note

This is the first installment of the Hebrish app podcast. I don't intend to launch a podcast for this project; I don't think a code-switching project needs one. But this voice note has several uses. On the website it's an update for anyone who wants to hear what the project is. I'm also using it to gather notes and context for the next few sprints for my AI agent, the esteemed Mr. Claude Code. It's Claude Opus 5.5, and it's starting to scare me: it's the first model that is just prodigiously good. It makes me feel very stupid. I can't keep up with it. It's so fast, and it's wonderful how quickly it has helped me iterate through this.

So I'll explain what this project is and why I wanted to do it, and then I'll get to what's on my mind in terms of features. The second part is instructions for Claude. I like to build in public. In my GitHub workspaces I usually have a folder for project management, and I almost always use this voice workflow: I record, then there are sprints, decisions and so on. In this project I'm putting that in public, including the voice notes in their original form. The first half is explaining to humans or AI agents, whoever you are, what this project is.

1:36 How I came to speech to text

I began using speech to text coming up on two years ago, with Whisper, OpenAI's open-source speech-to-text model: an exceptional ASR model. I'll try to de-jargonize the tech a bit: ASR is automatic speech recognition, the basic speech-to-text model. I've been using Linux for going on 20 years, and I'd always wanted to use speech to text.

I'm very open about having ADHD, and I mention it here because I feel like these tools, text to speech and speech to text, are for people with neurodivergence. I think we're overrepresented among the people who think these tools are the best things ever, and I'm definitely in that category. That's interesting to me because these technologies have such a deep assistive lineage: people with disabilities, people who can't type, the elderly. I would never have made the connection until recently that neurodivergence is part of that population. But maybe we are under that umbrella, along with people who just find it generally helpful.

I tried speech to text a few times on different projects, and until Whisper it was, I would say, virtually unusable. It worked, there were earlier versions, but it didn't cross the bar where it actually saves you time. And if it doesn't cross that bar, it's worse than useless, because you just go back to the keyboard.

Then Whisper came along. My wife was in the US around Thanksgiving two years ago, so I had the place to myself, and that was the week I became a speech-to-text convert. I told Hannah that when she came back she'd find me talking into my phone constantly, because I'd discovered this new app and decided I was never going to type again. It was great to have the house to myself, wander around the kitchen talking, getting all my ideas and to-do lists out, without feeling like anyone was listening.

When I saw that the technology had matured that rapidly, I began trying to use it across devices. On the Linux desktop, which has that weird feature of being where most of the prototypes happen and, paradoxically, often where the finished tech comes last because it's the smallest market. There were apps, and I began using Whisper on my phone. Over the past couple of years I've been developing prototypes for my own use, because however many great speech-to-text apps there are, there are always UI quirks that I want.

When I began using Whisper day to day, there was an app called voicenotes.com. I don't know if they still use Whisper under the hood, but it was the first one I got excited about. I began using it for everything: email drafts, and a lot of prompts for AI tools. That's where I began getting great results with AI. Voice is so much quicker than typing that you can record all the context data, all the background about yourself that's essential to getting a well-rounded answer.

6:36 Why Hebrew words go wrong

Anyone who's used speech to text in Israel will tell you it struggles with the Hebrew terminology. Some words do get transcribed, and that throws a lot of people off: you'll see Shabbat, kashrut. The big ASR models like Whisper, the first major open-source ASR model, are trained on a huge corpus of speech. The general understanding is that a lot of it is YouTube, huge amounts of YouTube video. That's why earlier models would sometimes hallucinate things like "thanks for watching": many hours of training audio ended like that.

If you pull a huge corpus of general English from a source like YouTube, you'll come across words like Shabbat. So how reliably a Hebrew word gets transcribed depends on how much it's in general circulation in English, biased towards American English and the content that winds up on the English-speaking internet. That distribution doesn't extend to a lot of the words English speakers in Israel use, like shuk and bituach leumi. The further you go down that long tail, the less likely the word is to be in a stock model that's trying to transcribe all of English.

Every Hebrew word you want recognized that sounds like something in English is fighting inside the model. The models are predictive tools. When you speak, the model looks at your phonemes, the sounds you make. It isn't literally understanding; it doesn't have a concept of words as such. It's finding matches between patterns of articulation and words. So a niche Hebrew word that isn't well represented will always be displaced by an English word that sounds like it. The additional challenge, of course, is that when you pronounce a Hebrew word, you're following Hebrew phonetics.

9:23 How the fine-tune works

The first time I did this, it was a fine-tune of Whisper. The training method is, and this is where it gets tedious, reading sentences like "I need to go to the makolet today." You create a dataset with your voice recordings and a ground truth: the model receives the correctly transcribed version, "I need to go to the makolet," spelled M-A-K-O-L-E-T. Then you train. Training means taking an existing model, not starting from scratch. You take Whisper, or the latest version of Parakeet, and change its internal weights, which is a way of exposing it to these new words.

The challenge is that you're a drop in the ocean. You're adding a drop of weights about makolet and bituach leumi to a model with hundreds of thousands of hours of audio in its training data. And the potential pitfall is overfitting, or catastrophic forgetting: the model gets so attuned to the niche words that you get the inverse problem. It starts predicting code-switched words for words that are in English. I'm trying to think of an English word that sounds like makolet; I don't have one off the top of my head.

That's why the training mixture contains neutral English sentences. I say "I'm moving the table lamp to the couch," which has no training target. It's just a stam sentence. Those are in the training data so that, while the model shifts towards learning the code-switched vocabulary, it doesn't forget how to process normal speech.

12:27 Why Parakeet, and the model plan

The fine-tune is based on NVIDIA's Parakeet, a lightweight ASR model of about 600 million parameters. I chose it as the base because my wife and I moved apartments a few months ago, and since then we've been moving boxes, and for the first time in my life I've been using my phone more than my desktop. Getting ASR models to run on a phone is still challenging: even the best flagship phone is fundamentally a small piece of hardware. So the question on the phone is: what's the smallest model that isn't rubbish? Parakeet is a good model for this. It's small, light and quite good.

NVIDIA has done something OpenAI hasn't really done with Whisper. People have tried to bring Whisper to the phone by quantizing it, and I've used my own Whisper fine-tune on the phone without great results. The first time I found an app with Parakeet, I thought: this is the one. It's fast and accurate enough, and I immediately saw it was a much more promising base for fine-tuning than Whisper. I think they both have value.

Qwen also has an ASR model, which I fine-tuned yesterday. The direction of the project, the training plan, is to have one model for maximum performance, the highest accuracy possible, for servers or desktops with GPU acceleration; one everyday driver for your phone; one for small devices; and one for streaming. For streaming, NVIDIA's Nemotron is a streaming-output model, and I'm going to update it with the dataset to get a first good version trained, because that's where I'll transition to the second part of this note.

14:57 One app, not two

The way I use these every day straddles a few use cases. A lot of the problem with speech interfaces and dictation tools to date is more a UI criticism than anything else: there's a split between long form and short form. People roll out voice keyboards, and people build transcription tools for long-form processing. I don't think they need to be two apps. In this case at least, I'd rather have one app that does several things than two apps.

15:57 Why now

Before the UI backlog, a little about the models. Claude actually coined the names; I wanted one name for each type of model. This past week has been a crazy, frenetic development period, and a large part of what's driving me to work in such a rapid burst is this: my wife and I have a son. He's a year and a few months old, and we found ourselves in the emergency room twice in the last month.

I find speech recognition more helpful every day. When you're holding a squirming child and trying to clean up the place and also working out how to respond to an email, voice to text is by far the most efficient way to do it. It's become even more indispensable since I became a parent. When we were in the emergency room and talking to our health fund, I realized this kind of tool could help a lot of people. In those situations you don't have time to go to your keyboard. If you're messaging someone with an update about your kid, or talking to the insurance or someone who speaks English, you don't have time to fix mistranscriptions.

So my motivation is primarily that I want a good tool for my own use. And I genuinely think, and hope, that it will be useful for other people.

17:44 Beyond Hebrew words: disfluencies, shekels, paragraphs

One more thing about the training composition; this is going to be a thorough voice note. When I began training on the code-switching sentences, and because this is primarily something I want for my own use (if other people find it helpful, great, and I'll try to accommodate different usage patterns), I made a list: what do Parakeet and Whisper get wrong? What stands between 90% good and 100% good?

That's the gap I feel ASR is in at the moment. Plugins and MCP are becoming very mature. In ChatGPT there's finally an email tool, a Drive tool, a Docs tool, so you can dictate into one of these apps and say "send that as an email to my boss." That's taken a while to reach consumer apps, and it's a very big deal for ASR.

The baseline of what an ASR model does is turn phonetics into letters on a screen. No rule says an ASR model will add punctuation, capital letters, sentence boundaries or paragraphs. None of that is in the fundamental logic of ASR; it's been integrated as the models improved. Right now there are bolt-ons, like dedicated post-processing layers in dictation apps such as SuperWhisper. That's one direction. There are also multimodal, or omnimodal, models that process both audio and text: you can write a prompt saying "the audio you've received is a shopping list; output it as a shopping list." That's a very appealing approach to me.

Until recently I'd have said that's where ASR is going, and classic ASR would be yesteryear's technology. But the other method is still going very strong: an ASR model paired with an LLM, where the second pass is a cleanup step that can also reformat.

What I'm trying with my model is an architectural decision more than anything else, and it's experimental. Not everyone will think it makes sense; it's what makes sense to me at the moment. We're doing the code-switching training on Parakeet: can we push that a bit further? What else do I see as problematic?

When I began using Parakeet, I noticed it output disfluencies: hesitations, false starts and word repetition. A false start is "I want to…" and never finishing the sentence. Repetition is "I'm going to put the tomato, tomahdo, in the fridge." You're out there, you're scattered, you're holding a phone to your face, and the transcription you want is "I want to put the tomatoes in the fridge."

Another one is both Parakeet-specific and Israel-specific. If you say "the falafel cost 20 shekels," and that sentence is going into a WhatsApp or an email, I can't think of a situation where I'd want "shekels" written out like that. I'd prefer "20 NIS." People might disagree and say they always write "shekels" out. But as the creator of this open-source project, I have the luxury of building it for my personal preferences, and that's one of them.

The lack of paragraph spacing is one I've always found strange. So I recorded a dataset pair for paragraph-spaced text: a long-form ground truth, say a few paragraphs about a DIY project, which Parakeet returns as one blob of text. This is where the post-processing argument comes in. One school says a small post-processing model does all these after-steps: ASR is responsible only for an accurate rendition of the words, and everything else, paragraph breaks included, comes next. My beef with that argument is that the moment you add sentence boundaries and punctuation, which Parakeet does, the model has already gone beyond rendering words: it's creating sentences. And I'd argue paragraph breaking has been a big blind spot in ASR. It's a fundamental property of readable text, arguably barely less essential than full stops.

That's a small dataset, and the question is whether a small dataset can nudge the model towards doing it. I don't know; there might be some architectural reason it's impossible that I haven't thought of. I'm an experimenter and a builder, and I don't know all the inner details of how these models work. It's very complicated stuff. So that's an experiment, and another small dataset.

25:40 Tech vocabulary, my own words, and Meorav

Another small dataset adds technical vocabulary: "I'm running a Docker Compose script," "I'm running a Kubernetes cluster." And I have a small dataset of my own quirky vocabulary, things I see the model getting wrong. I have a podcast called My Weird Prompts, an AI-generated podcast with hosts called Herman Poppleberry and Corn. Those names are commonly botched, so I made a very small dataset with just the things I see it get wrong, again for my own benefit.

That brings me to the restructure I did today. For a while I've wanted a voice clone, because I like to pull pranks on people and have a voice agent call up a friend. If Mike ever listens to this, he's probably spoken to my voice clone. Voice cloning of course has huge potential for black-hat use and fraud. I take the view that I have a YouTube channel and hundreds of hours of my speech on the internet, and you can clone anyone's voice from ten seconds of YouTube. So I want my voice available for my own occasional use.

While I was doing this, I thought: can I take the dataset I'm building for code switching and use it to create a TTS voice? The elephant in the room is that not many people in Israel have an Irish accent. A TTS that renders code-switched text correctly, but in an Irish accent, has niche appeal; an acquired taste, you might say. My voice isn't the ideal baseline, but it's worth having out there as an experimental TTS.

I was arguing with Claude that I needed more imaginative names for the TTS one, and we landed on Meorav. I thought that was clever: for those outside Israel, there's a famous dish called meorav yerushalmi, all sorts of organ meats thrown together. And that's kind of what code-switched speech is. Meorav is the TTS model; I trained a first test run yesterday.

29:07 The kitchen sink: Sabich and Jachnun

For anyone interested in how the datasets are structured: earlier this week I was recording the code-switching sentences, like "I need to go to the makolet to buy milk," and the tech sentences, like "I need to check the Docker memory use." My first idea was a Parakeet tech model and a Parakeet code-switching model. I assumed that if I mixed the tech sentences in with the Hebrew sentences, accuracy on both would suffer, and that the cleaner approach was to keep them split.

A few days ago we ran a test evaluation of what I initially called the kitchen sink. I told Claude: let's create a kitchen-sink dataset, lob it all together, train it and see how it does on everything. To my surprise, accuracy on neither the tech words nor the Hebrew words degraded. So the open question is whether the kitchen-sink model, which is called Sabich, is the right default. For people who work in tech and code-switch, it's the obvious choice. If you don't work in tech, do a few samples of vocabulary you won't use degrade it, or may it as well be in the model?

From a maintenance standpoint it's much easier to maintain and ship one model than two, so my long-term preference would be to collapse to one, for Hebrish at least. For people who aren't in Israel, a model with just the tech vocabulary is a more logical fit, so I also want to ship one of those under my personal account. It isn't a Hebrish model. I'm totally confused by all the names Claude has come up with.

The names are creative because each one stands for something. Sabich is the kitchen sink. Jachnun, if we maintain it, is pure code switching: its training doesn't include the tech dataset or my niche personal words. Whether the experimental datasets, like paragraph inference, go into Jachnun is an open question. My preference at the moment is that Sabich is really the kitchen sink: all the things and all the experiments baked in. Jachnun, just code switching, will be more conservative: focused on the code-switching vocabulary, perhaps leaving out everything else including paragraph inference. And whether Jachnun will have a life of its own is an open question.

I wanted a series name because each name stands for a dataset composition rather than a specific model. Within each family I'll experiment: can we have a desktop version? Can we run the fine-tuning dataset on Qwen, or on a different ASR model when one comes out? So the name refers to the series and what it contains, not the particular iteration, which depends on which ASR model is being trained.

33:35 Licensing, accents and noisy recordings

That's a lot of ground already. I need to get to the implementation part for Claude's context, but even this context will hopefully be useful for the agent. Licensing is free for non-commercial use, because perhaps someone will find a good commercial use for this.

As I've built up the dataset, it's occurred to me how much more it needs. We have Ashkenazi pronunciation and Sephardic pronunciation; we need different accents and different genders. There's so much religious vocabulary: all the masechtot of the Talmud, the major sefarim, the parshiot, major rabbis.

For those who haven't tried something like this before: when I want to add a word to the fine-tune, take makolet, the classic example I use over and over, I don't record one sentence, "I want to go to the makolet." Ideally it's ten sentences. And this is where data collection for the TTS model and for speech to text differ significantly. For speech to text you actually want a dataset that's a bit scrappy, because it should reflect real-world use.

People try to dictate something in the shuk with a bar blaring music in the background and other people talking around them in different languages. Each of those is a distinct challenge for ASR, because the model can't necessarily distinguish the intended speaker from everyone else; there's no built-in heuristic for that in the model. You might have wind noise, honking, a baby crying. So when I'm recording I think: great, this is noisy, let's record a few samples here. As you record a word in different conditions, the model gets more robust. It doesn't collapse when you say a code-switched word in a gust of wind and the audio is bad to begin with. That's the augmentation.

Back to licensing: this is a not-for-profit venture. It's something I wanted for my own use, and to benefit other people in Israel and the Jewish world. But time costs money. Training and fine-tuning all of these costs money, developing the app takes a lot of time, and so does recording the samples. So I wanted to leave the door open to people either supporting it or licensing it, if it's useful in a context like telephony.

For now it's open like that, and the datasets are access-gated on Hugging Face. I'm not unaware that someone could look for a dataset of code-switched English and Hebrew with bad intentions, to put it that way. So it's a voluntary security measure on my side more than anything else.

37:35 Apps and platforms

On apps and user interfaces: currently it's biased towards what I actually use. There's an Ubuntu app and an Android app, initially provided through F-Droid, and I'm looking into publishing on Google Play. For that I have to make a few small changes, get it reviewed and record a demo; they have a submission process, but I definitely want to do it. Windows is a thing as well.

iOS is a bit more complex, because inference on iOS works differently and has its own pathway. That's the more involved migration, but of course it's very important: it should be available on both platforms, 100%. It's a work in progress, and again something that costs money. I looked into it: it's a bit more expensive than Android to ship, and for the moment I have to keep my investment of time and money reasonably capped.

On other speakers: for this corpus and this use case, I'm thinking about all the ways people pronounce Hebrew, the different accents. Yeshivish, Haredi, Ashkenazi, Sephardic. I want all of them represented.

39:10 One recorder for speech recognition and speech synthesis

Something I noticed recording these samples: it's very hard to focus on tagging and recording at the same time. Claude built a custom recording app for me to record my samples in, which is fantastic. It has settings where I can say "baby crying," and that's how the annotation gets noted. But it's hard to do that reliably when you're just trying to read and pronounce things correctly and audibly.

On the dataset plumbing: there's huge overlap between the TTS and STT data. The difference is that for text to speech you want the opposite of the STT conditions: a clean noise floor, no background noise, because this is training data for the TTS. It's the same vocabulary I'm training on, but those samples are only good for the TTS set if they're quiet. Noisy ones are fine for the ASR dataset, but not for the TTS dataset.

As I said, Opus 5.5 is the genius I'm explaining all this to. I don't want to read all these mundane sentences twice over. So we're building a plumbing system for my voice: the app has a mode where I choose whether I'm recording for the TTS model or for ASR, and depending on that, the clips are routed into the right datasets, and training is done on the right clips.

41:04 Desktop: shortcuts and the macro pad

Finally, the UI itself. This is where I get to the actual instructions for Claude. Having used voice tech full time for a long time, here's what I'm currently thinking.

On the desktop, I think a macro pad is the best way to go. The simplest pattern is push-to-talk, or what I call tap-tap, and that works. But you also need buttons for stopping and retaking; these become evident when you use it all day, every day. I've narrowed it down to about six buttons, which is more than a traditional USB foot pedal provides: things like cancel, eject and retake. Something Claude pointed out today, very intelligently, is that the best design is for all of these, including the escape one, to be mappable onto custom keys, so you never conflict with global shortcuts. That's a big case for a macro pad. With system-level speech to text, which is what this is, rather than application-scoped, you have a greater chance of keyboard shortcut conflicts.

I usually use F13, because it's recognized as a valid key, and I map my USB devices onto it. In tap-tap mode I hit a little USB button, a $5 USB HID button from AliExpress that sends F13. So I think the settings and shortcuts should offer a basic setup, where the user uses Enter and Escape because that's easier on a laptop, and an advanced shortcut-binding option, so that someone with dedicated hardware like a macro pad can remap all of these onto whatever keys they want.

43:34 Android: Bluetooth and one button

Number two is Android. Bluetooth support is very important. If I'm in a noisy environment, I use a Bluetooth microphone; we added the ability to change the microphone input source, and that's key.

The other thing I'd find helpful is a button binding, and this is tricky. I have a Yealink Bluetooth headset with a button. On Android you can use an interceptor tool to intercept and map it. There's only one button on the device. There are things like a dedicated Bluetooth remote, which is probably the closest equivalent to a macro pad on Android. But for a first implementation, assume the user has one button: it would be great to use that button with the app, binding it to start recording and stop. That's the fundamental if there's nothing else. And I think it has to be tap-tap, because holding down a button on an earphone would look very weird. The fuller implementation can come later.

45:35 Usage patterns, streaming and Shakshuka

I'd like to document what you could call canonical usage patterns, and find a way to map them onto models, because each one favors a different model. One of the experimental datasets, by the way, is non-diarized speech: speech the model can tell wasn't intended for transcription.

The first usage pattern: I'm typing into Claude Code, using text injection at the cursor on Ubuntu. We added a few typing-speed settings: normal, slower, slowest. The most impressive implementation I've seen is actually the one in Claude Code; I think it uses Deepgram under the hood. What's impressive is that it streams, and it does one more thing: when you finish a sentence and it determines that what it previously wrote was wrong given the new context, it rewrites the previous text on the fly. It looks like magic.

I asked Claude what this technology is: just a streaming model, or an additional construct? It said it's an additional construct, but the building block is a streaming model. When it explained that, I realized this is the reason for the streaming model, which is based on Nemotron: Shakshuka. Shakshuka is the third model, the streaming fine-tune, and it's the kitchen-sink version. Another naming convention could be Jachnun streaming, the Jachnun code-switching-only dataset on Nemotron, and Sabich streaming. But for now it's Shakshuka, the third member of the family, just for streaming. We might revisit that. I feel the difference is significant enough that it's better to keep it as a separate model.

That means there isn't currently a code-switching-only Shakshuka. There could be a Shakshuka-all and a Shakshuka code-switching, a Nemotron variant trained only on the code-switching dataset. That may be the better way to go down the line, but it doesn't exist yet. Right now Shakshuka is everything, or as many things as can be included in that baseline.

For typing at the computer, the default would use Jachnun. [The recording says Jachnun here; the surrounding context is the streaming model. Still to be confirmed.] That also changes how setup should go. Right now it's "choose your model." I think the better presentation is "download models." We can still recommend models for your hardware, but the baseline assumption, on the desktop at least, is that people want both a streaming and a non-streaming model.

And we should keep the trainings in parallel. At the moment I'm experimenting, seeing what works, and that creates divergence. Whenever I have a significant amount of new training data, there should be a much more standardized process: a defined next version, released for Jachnun, Sabich and Shakshuka at about the same time on the same dataset, so they don't diverge and are all kept up to date in parallel. My personal tech-only model can be retrained at the same time for convenience, but it doesn't have to share the release cycle.

50:28 Long-form transcription

The second mode is where transcription and dictation need careful thought about how they're presented. There are two very different things you can do. One is to sit down at the computer or device and record something long, say a 15-minute note.

When I'm streaming into Claude Code and I have a big note, like this one, I don't want it just appearing there. That's where I lose confidence: there's a limit on input text in Claude Code in the terminal. If I'm going to spend an hour explaining all the ins and outs of an idea, I want to be confident that the source, the audio, which is what's valuable, won't be lost. That's why I'm recording this on my phone in a sound recorder app, which I do habitually.

For long-form transcription I see two user interfaces that make sense. The first is long-form transcription in the app itself. This is where model matching becomes really important: by default it locks onto the Parakeet fine-tune, and all the inference scaffolding around it is optimized for long form, meaning it reliably captures notes. The app could say "up to 90 minutes," or whatever maximum can be supported. That requires chunking and local storage, and offline it's not trivial. At that length you also need to think about API limits, upload limits, and doing things like VAD and downsampling as you capture the chunks. So that architecture is different.

The second is almost cleaner: file-based transcription. You upload a file and it runs inference over it.

Today we added local storage for the dictation history. Ideally it would support the cloud as well. I also want a way of seeding words. And this is more a project note: I'm going to mine my own dictation history, which is thousands of words a day at a minimum, to find what the model still gets wrong, and pinpoint words that need to go into the training dataset. The transcription history becomes a source of iterative improvement. Maybe in future people could share anonymized histories of their own. Probably no one will want to, but it could be an option. It's easy for me to see as I use the tool: great, it gets this right now; why is it still misspelling that? We need more data there. This will hopefully make that much easier.

On integration: right now, when we transcribe in the app, it adds to the history and puts the text on the clipboard. That's the text-injection workflow. For long form, I think the ideal would be something more like a document editor, but really a self-standing module. You go into dictation mode, and while that mode is running you can choose a file or send a recording. It transcribes and saves automatically. In future, with the cloud backup I want to add, I'll use it with my own account as a beta test, and potentially users could sign up for it as a SaaS tool. For now it's for my own testing.

The ideal UI for me: it transcribes and auto-saves, because the first concern is always data loss prevention. Great, it's auto-saved; now I might tidy it up a bit and hit save. One idea I had earlier for this mode: automatically create a new Google Doc from the transcription, after you give it a name. Like the Voicenotes feature.

56:00 Clipboard, text injection and output modes

I think the default should be recording and putting the text on the clipboard. In a previous implementation I had two modes you could choose. The default, on Ubuntu, put it on the clipboard and injected it. The rationale is that the clipboard is a backup: if you rely on text injection and accidentally move your mouse outside the target, that's a way to lose the text. What I've seen done on Android, and I think it's implemented in this app, is that if the transcription comes back and you weren't in a text field, the overlay shows a little text-insertion chip so you can pop it in.

So that was my default mode. I also added either-or modes: one was clipboard only, no text injection; the other was text injection only, no clipboard. I don't actually see uses for them. In general I'd like a fairly deterministic philosophy. That's just my philosophy: too many options for every little thing can be distracting and unhelpful. For the moment I'd rather make an executive call: we always use the clipboard as a backup.

One second, I'm going to retract that, which is why this voice note is different. We should actually have all the modes, and here's why. If it's always going to the clipboard, and we have a history recording all your recent transcriptions, searchable, showing the last ten, then you can always copy it out of the app; that's a reliable fallback. The clipboard can become annoying: you start dictating, copy something from another app, and before you paste it the transcription comes back and overwrites your clipboard. So there's an edge case where people might want the deterministic setting, or some detection for that race condition: the clipboard buffer has just been filled, so don't overwrite it, and send a notification saying "transcription held back," or something like that. That's an idea for the future. That's the general shape.

59:26 Storage, recordings and history

On desktop these are definitely viable. On Android I'm not sure to what extent a fully offline transcription mode can be achieved, or whether we have the local storage for it. I know the all-files access permission is a blocker for Google Play, so I don't know if the default permissions allow reliable local storage. I think they do, in which case it could just save to local storage. But on a phone you're more constrained.

A side question is audio recordings, and my strong preference is that they're never retained. The only retention of a recording, on any platform, is for failure prevention: if transcription fails, keep it. Maybe keep the last three, if a fallback ever needs a few. Otherwise it takes space, and it makes me uncomfortable to have voice recordings sitting around. Eventually the history should have settings for truncation too.

Those are the main ones. That's the project outline, the next, somewhat jumbled, sprint. On desktop, the UI needs to be closely integrated with the system tray icon. And for the text-editor mode for long-form dictation, there may be existing components we can use; the ideal is a very text-editor-like experience.

61:23 Voice activity detection and personal dictionaries

Two architectural things to tag on, to get everything out in one go. First, accessory layers: VAD. I'm not sure if it can be implemented on Android. If it's there on desktop, should it be a toggle? Fast inference is key.

The second open question is a personal dictionary. This is an interesting layer. On Android the personal dictionary is exposed quite liberally, and I've always been an advocate of using these systems. Most users aren't going to create their own fine-tunes. My method of baking my own dictionary into my own fine-tune may make sense, but it isn't replicable for most people.

I'm not sure how that would be integrated, or whether it could be part of the inference pass. On Android we'd integrate with the user's personal dictionary, or offer an option to; then on the desktop, syncing across devices. That's where cloud integration would be brilliant, because it's a huge productivity accelerator. If you use these tools every day, you add your personal dictionary on one app on Android, then you use another computer, the words aren't there, you create them again, and they're never quite in sync. That's one failure I've seen.

63:01 Report a word, and more voices

If anyone actually starts using this, which would be great, a brilliant feature that fits my workflow would be "report a word that didn't get transcribed correctly." Say you dictated a brand name and it didn't work: you send just the word, or the word and a note. That could feed a user-submitted corrections dataset, piped into my recording app, so I can record it.

The final thing is the recording app. I'm sure I can find a few friends who'd be happy to record a few sentences in different accents. Hannah has recorded a few sentences, and I have a little manual tagging for that. My general principle: the priority is always gathering more training data, and fixing up the annotations later, before the next training run.

So ideally the training app has multi-tenancy: a few users, and whoever you're logged in as is the annotator. Then we tag that speaker with their accent, and maybe a short bio for credits as a contributor to the project. We could probably build that in now, because I'm the only user. I think I have a manual drop-down for Hannah, but it would be better to add her address to the allowlist, so her speaker profile gets added: gender female, US accent, from Dallas, in Israel for ten years. The useful part is that the dataset could then be stratified by accent: how much is from female speakers, from code switching, and so on.

The quick-and-dirty fix for now, for the day a friend shows up at my apartment and says "the app is great, can I record a few sentences?": I'm not going to create a profile on the fly and get on to Claude. So, as a fallback, a custom entry where I can add a name, accent, gender and short description, so they can be added to the annotation pool that way. I can work out logins later if it becomes necessary. That's a good enough patch for now, because it would be fun to go out into the field and ask people to read a few sentences for this app. It's free, and they'd be helping English speakers get better dictation.

66:04 Sign-off

All right, I have to go pick up the little one very shortly, so that's my brain dump: all the ideas, all the progress updates. If you want to get in touch, my name is Daniel Rosehill. My personal site is danielrosehill.com, I have an AI consultancy, Carrot Cake AI, as in carrot cakes, and you can reach me through hebrish.app: H-E-B-R-I-S-H.

Oh, one final thing: a small dataset of our own identity. "hebrish.app" should be in it, and "Hebrish" should be in it, because ironically that doesn't get recognized currently. And the general principle: on Hugging Face, all the datasets and models for the project are in the organization, and all the GitHub repos for this project are in the organization, not my personal account, to keep it properly segregated.

Okay, I really have to go. That's it: over and out from Hebrish. Daniel Rosehill, progress update and vision for the next sprint, late October 2026.

Corrections after recording

• Desktop output mode (typed the same day): the output-mode toggle already exists. Make it easier to change: put it in the system tray icon's menu, and add shortcut bindings for toggling the output mode. Desktop only.