Why paragraph breaks come from a separate small model

Apps 1.002 added a Paragraphs setting to Transcription Mode: None, Time-based, or Paragraph model. Paragraph model is a separate, small model that runs after the speech model has finished. This post explains why paragraphs aren't trained into the speech models themselves, for now.

The speech models can't write a line break

A speech model writes from a fixed vocabulary of word pieces: 1,024 for Jachnun, Sabich and Shakshuka alike. None of those pieces is a line break. A model can only write what its vocabulary can spell, so no amount of training data with paragraph breaks in it would make these models start a new paragraph. The breaks would simply be lost from the training text (the same family of problem as the missing semicolon).

Putting breaks into the speech model would mean changing its vocabulary and retraining, and the new piece would have to be learned from audio, where a paragraph has no sound of its own. That may be worth doing one day. For now, a separate model does the job without touching the speech models.

The paragraph model

• What it is: a small English classifier, a fine-tune of MiniLM-L12 (33 million parameters), that reads the finished transcript and marks the words where a new paragraph should start, where the topic moves on. It is Daniel's own model, made as a separate project from Hebrish: danielrosehill/MiniLM-L12-ASR-Paragraph-Breaks.
• It only adds breaks. It classifies the gaps between words, so it can't change, add or drop a word.
• How well it does: it finds paragraph breaks with an F1 score of 0.85 on 26 held-out, hand-reviewed voice memos.
• In the apps: it runs on your device, once, when a transcription finishes. Choosing it the first time downloads it (34 MB). If it is missing or fails, that transcript gets time-based paragraphs and the app says so. Dictation is unchanged.

Other finishing steps

The same place in the apps, after the speech model, is where other text clean-up can run. Hesitations (um, uh, er) are already removed there in both apps. Next on the plan is an optional clean-up step on desktop using S1-mini by Superwhisper, a small model made for tidying dictated text, if it runs well on a laptop and keeps the romanized Hebrew words intact. It isn't in the apps yet.