Why the next training release drops some punctuation

Since Sabich v2.0, dictated text has sometimes had a stray <unk> in it, always mid-sentence where a short pause was: "described him<unk> he is", "manually<unk> we can". Daniel's desktop dictation history had six of them in 936 dictations, all from Sabich v2.0 and none from the other models. This post is about where they came from and what changed so the next training release doesn't learn them.

What <unk> is

A speech model doesn't write letters one at a time. It writes from a fixed vocabulary of word pieces: 1,024 of them for the Parakeet models (Jachnun and Sabich), the same number for Shakshuka. Piece number 0 is <unk>, "unknown". During training, every character in a transcript that no piece can spell is replaced by it.

Parakeet's vocabulary has the full stop, comma, question mark, exclamation mark, colon, hyphen and apostrophe. It has no semicolon, no dash (— or –), no double quotes, no curly apostrophe, no brackets, no & and no +. Shakshuka's vocabulary also has no accented letters, so "café" and the Irish "Dáil" can't be spelled either.

Where it came from

The main recordings dataset had 22 semicolons in its transcripts, in sentences like "…who Moshe Hillel Hirsch is; he is a rabbi and…". In training, each one became <unk> at a clause break, so the model learned to write <unk> at a clause break. The six in the dictation history are exactly that pattern. A model can't be fixed afterwards for something it has learned. The only real fix is to keep the character out of the training text.

What changed

• One list of allowed characters for every speech transcript: letters a–z and A–Z, digits, spaces, . , ? ! ' : - / % $, £ and €. It is the set every current model's vocabulary can spell.
• The datasets were cleaned. 211 transcripts in 15 Hugging Face datasets were rewritten. Semicolons and dashes became commas, & became "and", C++ became "C plus plus", quotes and brackets went, and accents were dropped (café to cafe). The audio wasn't touched, and each changed row keeps its old text alongside. The text-to-speech recordings were left as they are: Meorav reads any character, and quotes and dashes help it sound right.
• New sentences are checked where they are written. The tool that drafts recording prompts is told the rule, and its checker rejects sentences that break it. Anything published from the recorder is mapped to the allowed set, typed and corrected text included. The training code maps every transcript once more before a run.
• The apps hide it in the meantime. From the next app releases, desktop and Android remove <unk> from the model's output and leave a space. That covers Sabich v2.0 until it is retrained.

What it means for the next training release

The next release trains on transcripts that use only characters its vocabulary can spell, so it has nothing to learn <unk> from. The models will still write commas, full stops and question marks as before. They won't write semicolons or dashes, which they never could spell anyway.