vocal_helper.bench module
vocal_helper.bench
Transcription accuracy, scored the way a multi-speaker recording actually fails.
Provenance
The French normalisation and the WER report are ported from
operator-assistance-streaming/streaming/score_wer.py, which is being absorbed into this
package. They are kept close to the original — the filler list and the repeat-collapsing are
calibrated on real French calls and there is no reason to re-derive them — with one coupling
removed: the original read its reference out of a Gladia JSON, so every number it produced
was agreement with Gladia rather than accuracy. A reference here is text, or turns, from
wherever the caller got them.
What plain WER cannot see
On a recording with more than one voice, the failure that destroys everything downstream is
not a misheard word, it is the right words attributed to the wrong person. WER is blind to
it: concatenate a whole episode and a diarizer that merged five panellists into two scores
exactly as well as one that separated them. DER is blind the other way — it never reads a
word. cpwer() prices both at once by scoring under the best possible speaker mapping, so
a transcript only wins if the words AND the speaker are right.
That is not theoretical here. A 2026-10-19 bench on three real sixty-four-minute six-speaker panels had the local diarization path emitting 103 to 124 clusters for six people. Plain WER on the concatenated text would barely have moved.
The other thing a WER average hides is which words were missed. Getting “le Figaro” wrong as
“le Fégaro” costs the same fraction of a point as a dropped “the”, while destroying any
corpus-level count of who discussed what. term_error_rate() scores only the terms that
carry that weight — the glossary, the proper nouns — and is the direct measurement of what a
glossary-correction stage exists to fix.
Credits
cpWER is the CHiME-6 formulation (Watanabe et al., 2020). Implemented here rather than imported so the package keeps its dependency surface; the optimal assignment uses SciPy’s Hungarian solver.
- vocal_helper.bench.FR_FILLERS = {'bah', 'ben', 'euh', 'euhh', 'hein', 'heu', 'hum', 'mh', 'mhh', 'mhmh', 'mmh', 'uh', 'uhm', 'um'}
a long list starts eating real content, and the cost of missing one filler is far smaller than the cost of deleting a word somebody said.
- Type:
French disfluencies. Deliberately short
- vocal_helper.bench.PUNCT_RE = re.compile("[^\\w\\s']")
Punctuation stripper that keeps apostrophes, so French clitics (“c’est”, “qu’il”) survive as one token instead of splitting into noise.
- vocal_helper.bench.cpwer(reference_turns, hypothesis_turns, *, style_tolerant=False)[source]
Concatenated minimum-permutation WER — words and speaker, priced together.
Each side is concatenated per speaker into one stream per voice, then speakers are matched one-to-one so the total word error is as low as it can be, and the WER of that best case is reported. A transcript therefore only scores well when the words are right and they are under the right person.
This is the number to quote for a multi-speaker recording. Plain WER on concatenated text cannot tell a diarizer that separated five panellists from one that merged them into two; DER can, but never reads a word; cpWER refuses both blind spots.
Speaker counts need not match. Unmatched speakers — on either side — are charged in full, which is the honest treatment: inventing a sixth voice and missing a real one are both errors, and a metric that quietly dropped the leftovers would reward over-splitting.
Returns the
cpwerfigure, the chosenmapping(hypothesis speaker → reference speaker), and both speaker counts so a bad score can be read as mis-transcription or as mis-attribution without re-running anything.n_ref_speakersvsn_hyp_speakersis often the whole story.
- vocal_helper.bench.der_report(reference_turns, hypothesis_turns, *, collar=0.0, skip_overlap=False)[source]
DER broken into its parts, plus JER and both speaker counts.
The defaults are collar 0 and overlap scored, which is not the convention. The usual 0.25 s collar with overlap skipped deletes exactly the regions an interruption study is about, and a system can look good on it precisely by ignoring simultaneous speech. Pass
collar=0.25, skip_overlap=Truefor a number comparable to published work, and read it as a second line rather than the verdict.Why the decomposition is not optional: miss, false alarm and confusion have different cures. A high false alarm is a voice-activity problem, a high confusion is an identity problem, and a single DER cannot tell you which one you have. On the 2026-10-19 AMI bench the two local paths had near-identical DERs and opposite failures.
jeris reported next to it because DER is time-weighted: one dominant speaker can hide the collapse of everyone else, which is the failure mode of a panel show. JER normalises per speaker and does not let that through.n_ref_speakers/n_hyp_speakersare there because a count is the bluntest and often the most informative signal — a model with an architectural ceiling cannot emit a fifth label, and that shows here before any error rate is read.Requires the
pyannoteextra for the metrics themselves.
- vocal_helper.bench.normalize_fr(text, *, drop_fillers=False, collapse_repeats=False)[source]
Normalise a French transcript for scoring.
Always: NFKC, lowercase, punctuation stripped (apostrophes kept), whitespace collapsed.
drop_fillersandcollapse_repeatsare the style-tolerant mode, and they must be applied to both sides or the comparison is meaningless. They exist because engines disagree about transcription style rather than about what was said: a verbatim model that writes every “euh” and every false start is not less accurate than one that tidies them away, and scoring them against each other without this punishes honesty.
- vocal_helper.bench.speaker_label_error(reference_turns, hypothesis_turns)[source]
How often the speaker label on a displayed turn is wrong.
Ported from
operator-assistance-streaming/streaming/score_der.pyand generalised off its Gladia-only reference. For each reference turn, the hypothesis speaker holding the most overlap with it wins the vote; labels are then matched one-to-one so the global error is as low as possible, and the remaining mismatches are counted.The point is what it ignores: boundary slop. DER charges every millisecond at the edge of a turn, which is right for comparing diarizers and wrong for a product — a reader never sees a boundary, they see a name against a paragraph. This asks their question instead: how often is that name wrong.
Returns
turns,errors,error_rate, the chosenmappingandunattributed(reference turns no hypothesis speaker overlaps at all, counted as errors because a missing label is a wrong label to the person reading it).
- vocal_helper.bench.term_error_rate(reference, hypothesis, terms)[source]
How many of terms that the reference contains the hypothesis also got.
A WER average treats every word alike, which is wrong in the direction that matters. A dropped article costs the same as a mangled proper noun, while only one of the two destroys a corpus-level count of who discussed what — “le Fégaro” for Le Figaro is a rounding error in WER and a lost record in the analysis.
Multi-word terms are matched as phrases on the normalised text. Only terms actually present in the reference count: scoring a hypothesis for missing a word nobody said would make the rate depend on the glossary’s length rather than on the transcription.
Returns
present(terms the reference uses),found,missed(the list, so the failure is inspectable and not just a number) anderror_ratein[0, 1].
- vocal_helper.bench.wer_report(reference, hypothesis, *, style_tolerant=False)[source]
WER, MER, WIL and CER between two transcripts, plus token counts.
Returns
{"wer": None, "error": ...}rather than raising on an empty reference: a sweep over many recordings must report the hole and carry on, not die on one bad file.