Hebrish
Technical

Technical · Training

Training learnings

Non-obvious things learned while fine-tuning the Hebrish models, mostly on Parakeet TDT 0.6B v2. Each entry is specific to the model it names and dated the day it was found, with what it changed in how the models are trained.

A finding about one tokenizer may not hold for another: check the model before applying it. The notebook is also a public repository, Hebrish-ASR/ASR-Training-Learnings (CC-BY-4.0).

Test takes don't teach, so count only training takes toward a word's target

2026-11-18 · Model: nvidia/parakeet-tdt-0.6b-v2 · Affects: Sabich, Jachnun, Fritter, Falafel · Topic: data · Status: counts measured; split mechanism inferred · Source

What was seen

Of 21 takes of "Deepgram", 10 were in the test split, which is never trained on. The recording app counted all 21 toward the word's target of 20, so it stopped serving the word as done while the model had seen it only 11 times.

Why

The sentence bank's split is term-disjoint: a term picked for evaluation sends every sentence containing it to test. Two things moved takes into test that should have trained:

  • A take keeps the split its sentence had when it was recorded. Words later exempted from the hold-out (words the model is known to get wrong are exempt) kept their earlier test takes.
  • Compound terms held out for evaluation ("Deepgram Nova", "Deepgram Aura") take the shorter word with them.

Across all tracked terms, 15,910 takes of target words sat in test on 2026-11-18.

What changed in the training

Each word's target is now its training target plus its current test takes, so the app keeps serving it until the training split alone has enough. The audit is rerun before every training run, because new test takes keep arriving. Measured shortfall after the change: about 60,000 training takes across 13,000 terms, against 42,000 under the old count.

Case is decided on the first piece, so Deepgram comes out as "deepgram"

2026-11-18 · Model: nvidia/parakeet-tdt-0.6b-v2 · Affects: Sabich, Jachnun, Fritter, Falafel · Topic: tokenizer · Status: segmentation measured; cause inferred · Source

What was seen

The fine-tuned Sabich wrote "deepgram" and once "Degram" for the brand Deepgram, even when it was said slowly and clearly. Every training transcript wrote it as Deepgram, so the labels were not the problem.

Why (best explanation so far)

Parakeet TDT 0.6B v2 has a 1,024-piece SentencePiece vocabulary. It has no Deep piece, so the two spellings start differently:

WordCased piecesLowercase pieces
Deepgram▁D e ep gr am (5)▁de ep gr am (4)
DeepSeek▁D e ep S e e k (7)▁de ep se e k (5)
GitHub▁G it H ub (4)▁g ith ub (3)
Docker▁D ock er▁do ck er

The capital has to be chosen on the very first piece, as a bare ▁D, against ▁de, one of the commonest word starts in English (delete, deploy, decide). The decoder's general English habits favour the lowercase path, and a handful of examples does not outweigh them. "Degram" is the same path with the ep piece dropped.

How few examples: the word had 21 recorded takes, but only 11 were in the training split (see Test takes don't teach).

What changed in the training

Take targets now scale with how hard a word is for this tokenizer, not only with how important it is. A term earns a difficulty point for each of: a bare-capital first piece where the lowercase form starts with a longer piece (case), a capital inside the word (camel), five or more pieces or three or more single letters (frag), and being a brand that is also an ordinary word (homonym, Parakeet). Ordinary English words score nothing. Target = importance target x (1 + 0.25 x points): Deepgram went from 20 takes to 30.

Brands that are also ordinary words (Parakeet, Fritter) lose their capital

2026-11-18 · Model: nvidia/parakeet-tdt-0.6b-v2 · Affects: Sabich, Fritter · Topic: tokenizer · Status: observed in use; cause inferred · Source

What was seen

"Parakeet" is often written lowercase in dictation, although in this project it almost always means the AI model, not the bird. The same risk applies to model and product names such as Fritter, Blender, Fedora, Rhino and Svelte.

Why (inferred)

The model has heard the ordinary word, lowercase mid-sentence, far more often than the brand. Unlike Deepgram, the pieces are not the problem (▁P ar ake et either way, apart from the first letter); the prior for the common word is.

What changed in the training

A one-word capitalised term whose lowercase form is an ordinary English word gets a homonym difficulty point, so it needs more takes (Parakeet 20 -> 25, Fritter 15 -> 19). Multi-word title-case phrases made of ordinary words ("Feedback Panel") do not: their case is genuinely ambiguous. See Case is decided on the first piece.

Adding symbols to v2's tokenizer by surgery broke the model

2026-11-17 · Model: nvidia/parakeet-tdt-0.6b-v2 · Affects: Sabich · Topic: tokenizer · Status: confirmed (outcome); cause not investigated · Source

What was tried

Eight symbol pieces (@ & + # ( ) _ =) were appended to the v2 vocabulary after the existing 1,024, with the blank token moved after them and the new output rows trained in one rolling fine-tune, so the model could write email addresses and code symbols directly.

What happened

The resulting release (Sabich v4.8) produced "no speech" or invented sentences in real dictation. Its export metadata was consistent, so the fault is in the weights or the decoding, not the packaging. It was withdrawn the same evening and the previous release restored.

What changed in the training

Tokenizer surgery is not to be retried on these lines. Symbols stay out of the v2 targets: rows carry the spoken form ("at", "underscore", "plus") and a separate full-form target for bases whose tokenizer already has the symbol; a formatting step after decoding writes the symbol. See Parakeet v3 has the same gaps.

Parakeet v3's bigger tokenizer has the same missing symbols as v2

2026-11-13 · Model: nvidia/parakeet-tdt-0.6b-v2, nvidia/parakeet-tdt-0.6b-v3 · Affects: Sabich, Fritter · Topic: tokenizer · Status: confirmed · Source

Question

v3 is multilingual with an 8,192-piece vocabulary. Would moving to it let the model write + (for +972), * or brackets?

Finding

No. Measured on both tokenizers (each tokenizer.model read out of the .nemo by HTTP range requests; a character counts as writable if a<c>b encodes without <unk> and decodes back unchanged):

Parakeet TDT 0.6B v2Parakeet TDT 0.6B v3
Pieces1,0248,192
Writable of 699 candidates205327
ASCII symbols!$%',-./:?^!$%',-./:? (loses ^)

The extra pieces are European letters, not symbols.

What changed in the training

Nothing: v2 stays the base for the dictation lines (chosen over v3 earlier from real-use results). Symbols are handled by spoken-form targets plus formatting after decoding (see Missing characters become <unk>).

A character the tokenizer lacks becomes <unk>, and the model learns to write it

2026-10-25 · Model: nvidia/parakeet-tdt-0.6b-v2, nvidia/nemotron-speech-streaming-en-0.6b · Affects: Sabich, Jachnun, Fritter, Shakshuka, Falafel · Topic: tokenizer · Status: confirmed · Source

What was seen

An early Sabich wrote <unk> at clause breaks in real dictation. The training data held 22 semicolons.

Why

A character missing from the model's vocabulary tokenizes to <unk>, and fine-tuning teaches the model to emit it. Parakeet v2 cannot write ; " “ ” ’ — – ( ) & + @ _ or some accented capitals; the Nemotron streaming base cannot write any accented letter. Measured on the tokenizers themselves: of 699 candidate characters, v2 can write 205.

What changed in the training

Every training transcript uses only ASCII letters, digits, spaces, . , ? ! ' : - / % $, £ and €. Anything else is mapped before training: ; and dashes to a comma, & to "and", + to "plus", curly apostrophes to ', quotes and brackets dropped, accents folded. 211 already-published rows were rewritten to match. Where the real form matters (email addresses, filenames with _), a row keeps a spoken-form target for v2 ("dana underscore levi at tidewell.io") and the real form separately for bases that can write it, and a formatting step after decoding turns one into the other.