vocal_helper.rediar module

vocal_helper.rediar

Offline-quality diarization on a live stream, at a cost that does not grow.

The online diarizer is a latency-bound approximation: it embeds each voiced segment as it arrives and clusters by cosine distance, which cannot model overlapping speech and so sits three to four times above the offline error rate (measured, vocal_helper.router: 0.586 online against 0.122 offline). When the speaker labels matter as much as the words, that is not good enough, and no better online clusterer fixes it — the limitation is the latency, not the clusterer.

What fixes it is running the offline diarizer, periodically, on a bounded window, and correcting the labels already shown. The offline diarizer is cheap per second of audio; what makes it unaffordable live is re-running it on everything.

Not a new algorithm: the live expression of an existing one

The identity strategy here is ported from the native core’s block-wise diarizer (core/nh-core/src/blockwise.rs, same author), which solved this for long recordings. That code earned three details this module would otherwise have got wrong, and each of them is the difference between working and almost working:

  1. The primer is a per-speaker budget, not a lump. Each already-known speaker contributes a few seconds of their own harvested audio. A single block of “earlier audio” would simply miss whoever had not spoken yet, and their label would then drift at every refresh.

  2. The pieces are separated by harvested room tone, not silence. Real low-level ambience from the same recording, and the gap must exceed the segmentation model’s min_duration_off (0.5 s in the measured configuration, hence a 0.8 s default) or the model does not cut a turn boundary there at all. Digital silence is worse than useless: it is out of distribution, and the model behaves oddly around it.

  3. Identity is recovered by read-back, not by matching. The primer is prepended and [primer | window] is diarized in one call, so the engine’s own clustering puts each window turn in the same cluster as the matching reference. Then you simply read which local label covered each reference span. No embeddings, no centroids, no similarity threshold of our own — and exact, where an overlap heuristic is approximate.

Why the obvious version does not hold

“Re-diarize the whole buffer every thirty seconds” is the natural design. Its instantaneous cost is backend_rtf × buffer_s / period_s, and buffer_s grows with the session: at 0.067 and a thirty-second period, one hour reaches 0.067 × 3600 / 30 ≈ 8.0. It works beautifully for three minutes and collapses after twenty. RediarPolicy.unbounded_rtf() exists so that number can be looked at rather than assumed away.

Bounded, the cost is constant in the session length:

rtf = backend_rtf × (window_s + primer_s(speakers)) / period_s

And it grows with the speaker count, because the primer does — mildly, at the default settings: 12.8 s of primer per speaker against a 180 s window, so each extra voice costs 0.014 of real-time factor. The default policy carries eighteen speakers on a Metal machine before the budget breaks, which is well past any real panel. The dependency still belongs in the model rather than in a footnote: speakers arrive during a session, and a budget that silently assumed the room it started with is a budget that fails in the second half.

Usage example

>>> policy = RediarPolicy()
>>> policy.window_s, policy.ref_budget_s, policy.gap_s, policy.period_s
(180.0, 12.0, 0.8, 60.0)
>>> round(policy.primer_s(speakers=4), 1)
52.0
>>> round(policy.stage_rtf(0.067, speakers=4), 3)
0.259
>>> policy.max_speakers_within(asr_rtf=0.033, backend_rtf=0.067)
18
vocal_helper.rediar.DEFAULT_GAP_S = 0.8

Room tone between pieces. Must exceed the segmentation model’s ``min_duration_off`` (0.5 s).

vocal_helper.rediar.DEFAULT_MIN_SPEAKER_S = 5.0

In-window speech before a new cluster becomes a speaker.

Not the native block-wise diarizer’s 15 s, and the difference matters. That value was tuned for windows cut from a whole recording, where a real participant holds the floor a good stretch within some window. Here the window is bounded and short, and a genuine but quiet participant can legitimately hold it for ten seconds in three minutes — a 15 s floor then never promotes them, the speaker count never grows past one, and the refresh degenerates into agreeing with whatever it already believed. Measured on AMI: that is exactly what happened.

Five seconds still clears a spurious fragment, which lasts under two.

vocal_helper.rediar.DEFAULT_PERIOD_S = 60.0

New audio between refreshes. Also the worst case for how long a wrong label stays on screen.

vocal_helper.rediar.DEFAULT_REF_BUDGET_S = 12.0

enough for a strong embedder to cluster it, cheap enough to repeat.

Type:

Per-speaker reference audio

vocal_helper.rediar.DEFAULT_WINDOW_S = 180.0

Trailing audio per refresh. Balances per-call clustering cost against primer overhead.

class vocal_helper.rediar.RediarPolicy(window_s=180.0, ref_budget_s=12.0, gap_s=0.8, period_s=60.0, min_speaker_s=5.0, backend='pyannote', max_speakers=12, stable_rounds_before_pinning=1)[source]

Bases: object

How often to re-diarize, over how much audio, and what that costs.

Parameters:

  • window_s (float) – Trailing audio re-diarized at each refresh.

  • ref_budget_s (float) – Audio harvested per already-known speaker for the primer.

  • gap_s (float) – Room tone inserted between primer pieces and before the window. Below the segmentation model’s min_duration_off the model does not cut there, the references bleed into each other, and the read-back becomes ambiguous.

  • period_s (float) – New audio accumulated between refreshes.

  • min_speaker_s (float) – In-window speech a new cluster must clear to be promoted to a speaker.

  • backend (str) – The offline diarizer. "pyannote" is the measured default at an error rate of 0.122; "sherpa" is the torch-free option at 0.174.

  • max_speakers (int)

  • stable_rounds_before_pinning (int)

backend: str = 'pyannote'
gap_s: float = 0.8
holds(*, asr_rtf, backend_rtf, speakers=6, vad_rtf=0.01, margin=2.0)[source]

Whether the whole system stays under a real-time factor of 1.

Assumes the worst case for placement — transcription and diarization on the same compute unit, so their costs add. That is the truth on a single-GPU machine, and assuming otherwise would be optimism dressed as arithmetic.

Parameters:

  • speakers (int, keyword-only, optional) – Expected speaker count. Six by default, which is a panel: budget for the room you might get, not the room you started with.

  • margin (float, keyword-only, optional) – Required headroom. Two by default: crossing 1 is not a graceful degradation but a delay that grows for the rest of the session.

  • asr_rtf (float)

  • backend_rtf (float)

  • vad_rtf (float)

Return type:

bool

Examples

>>> RediarPolicy().holds(asr_rtf=0.033, backend_rtf=0.067)
True
>>> RediarPolicy(period_s=10.0).holds(asr_rtf=0.033, backend_rtf=0.067)
False
max_speakers: int = 12

Hard ceiling on discovered speakers.

The lesson of DIARIZATION-TROUBLES.md: an unbounded greedy clusterer minted 203 labels for a four-speaker podcast. A ceiling is not a quality knob, it is the difference between a transcript and a list of 203 strangers. Twelve is past any panel and short of any runaway.

max_speakers_within(*, asr_rtf, backend_rtf, margin=2.0, ceiling=24)[source]

How many speakers this policy can carry before the budget breaks.

The number to know before a session, not during: when the answer is four and a fifth person joins, the pipeline starts falling behind, and the honest response is to widen the period rather than to discover it live.

Returns:

The largest speaker count that still holds. 0 when even one speaker does not fit, which means the policy is wrong for this machine.

Return type:

int

Parameters:

Examples

>>> RediarPolicy().max_speakers_within(asr_rtf=0.033, backend_rtf=0.067)
18

Without a GPU, transcription costs five times more and the room shrinks — still a real panel, but no longer unlimited:

>>> RediarPolicy().max_speakers_within(asr_rtf=0.164, backend_rtf=0.067)
9

Refresh too often and nothing fits, because the primer is re-diarized every time: a twenty-second period spends more on repeating itself than on new audio.

>>> RediarPolicy(period_s=20.0).max_speakers_within(asr_rtf=0.033, backend_rtf=0.067)
0
min_speaker_s: float = 5.0
period_s: float = 60.0
primer_s(*, speakers)[source]

Primer length for a given number of known speakers.

One reference plus one gap each, and one more gap before the window so the model also cuts between the primer and the audio being diarized.

Examples

>>> RediarPolicy().primer_s(speakers=0)
0.0
>>> round(RediarPolicy().primer_s(speakers=1), 1)
13.6
>>> round(RediarPolicy().primer_s(speakers=6), 1)
77.6
Parameters:

speakers (int)

Return type:

float

ref_budget_s: float = 12.0
stable_rounds_before_pinning: int = 1

Consecutive refreshes with no new speaker before the count gets pinned.

Discovery and stability pull opposite ways, and both are needed. While the count is still growing, the diarizer must be free to find a voice nobody has heard yet, so it runs with a floor. Once the set has stopped growing, the count is pinned, and that is what makes cross-window identity clean: the engine returns exactly as many clusters as there are references, every one of them maps back to a reference, and there is nothing left over to mint a new name from.

Measured: without this, the identity drifted worse than the online stage it was meant to fix — seven labels for four speakers, long-range agreement 0.297 against the online stage’s 0.969.

stage_rtf(backend_rtf, *, speakers)[source]

What this stage costs as a real-time factor, at a given speaker count.

Constant in the session length. Not constant in the speaker count, and that dependency is the one a live budget has to carry: speakers arrive during a session.

Examples

>>> round(RediarPolicy().stage_rtf(0.067, speakers=2), 3)
0.23
>>> round(RediarPolicy().stage_rtf(0.067, speakers=8), 3)
0.316
Parameters:
Return type:

float

static unbounded_rtf(*, buffer_s, period_s, backend_rtf)[source]

What re-diarizing the whole buffer would cost, for comparison.

A function rather than a comment, because the number is the argument: small enough to look fine in a demo, impossible in use. That is exactly how this design error survives a review.

Examples

>>> RediarPolicy.unbounded_rtf(buffer_s=180.0, period_s=60.0, backend_rtf=0.067)
0.201
>>> round(RediarPolicy.unbounded_rtf(buffer_s=3600.0, period_s=60.0, backend_rtf=0.067), 1)
4.0
Parameters:
Return type:

float

window_s: float = 180.0
class vocal_helper.rediar.SpeakerReference(speaker, pcm, primer_t0=0.0, primer_t1=0.0)[source]

Bases: object

Harvested audio standing for one speaker, and where it sits in the primer.

primer_t0/primer_t1 are filled once the primer is assembled: they are what the read-back looks at, so a reference without them cannot anchor anything.

Parameters:
property duration_s: float

Length of the harvested audio, in the primer’s time base.

pcm: ndarray[tuple[int, ...], dtype[float32]]
primer_t0: float = 0.0
primer_t1: float = 0.0
speaker: str
vocal_helper.rediar.plan_refresh(*, now_s, last_refresh_s, policy)[source]

Decide whether a refresh is due, and over which window.

Returns:

(window_start_s, window_end_s) when due, else None. Clamped at zero, so an early refresh re-reads only the audio that exists.

Return type:

tuple of float, or None

Parameters:

Examples

>>> policy = RediarPolicy(window_s=180.0, period_s=60.0)
>>> plan_refresh(now_s=30.0, last_refresh_s=0.0, policy=policy) is None
True
>>> plan_refresh(now_s=60.0, last_refresh_s=0.0, policy=policy)
(0.0, 60.0)
>>> plan_refresh(now_s=600.0, last_refresh_s=540.0, policy=policy)
(420.0, 600.0)
vocal_helper.rediar.read_back_identities(segments, references, *, min_coverage=0.0)[source]

Recover global speaker identity by reading which local label covered each reference.

The engine diarized [primer | window] in one call, so its own clustering already placed each window turn with the matching reference. Identity therefore travels by time, not by embedding geometry: for each reference span, read which local label covered it.

This is the part ported from the native block-wise diarizer, and it is why that design needs no embeddings of its own.

The read-back is a bijection, so it is resolved by optimal assignment and not by a per-reference argmax. The distinction is not cosmetic. A greedy argmax lets one local label win two references — which happens whenever the engine merges two primer speakers into a single cluster — and the second write then overwrites the first, so one known speaker silently loses its label and is reported as absent. Optimal assignment cannot do that: the merged label takes one reference, the other reference stays unmatched, and the caller sees a speaker it could not place instead of a wrong answer.

Parameters:

  • segments (list of tuple) – (t0, t1, local_label) over the primer-plus-window time base.

  • references (list of SpeakerReference) – With their primer spans filled in.

  • min_coverage (float) – Fraction of a reference’s own duration that the matched label must cover for the pairing to count. 0.0 (default) accepts any shared time at all; raising it refuses a label that merely brushed the reference.

Returns:

{local_label: global_speaker}. A local label absent from the result covered no reference: it is a candidate new speaker, and the caller promotes it only if it clears min_speaker_s — guessing instead would merge a passer-by into a participant.

Return type:

dict of str

Examples

>>> import numpy as np
>>> silence = np.zeros(1, dtype=np.float32)
>>> refs = [SpeakerReference("S0", silence, 0.0, 12.0),
...         SpeakerReference("S1", silence, 12.8, 24.8)]
>>> segs = [(0.0, 12.0, "c1"), (12.8, 24.8, "c0"), (30.0, 40.0, "c2")]
>>> read_back_identities(segs, refs)
{'c0': 'S1', 'c1': 'S0'}

c2 is deliberately absent: it covered no reference, so it is a newcomer and not something to be guessed at.

A reference nobody covered yields nothing for that speaker, rather than a wrong answer:

>>> read_back_identities([(50.0, 60.0, "c9")], refs)
{}

One label straddling both references — the engine merged two primer speakers — is assigned to one of them, and the other is reported as unplaced. A per-reference argmax would instead have overwritten the first mapping and lost S0 without a word:

>>> read_back_identities([(0.0, 24.8, "c0")], refs)
{'c0': 'S0'}