vocal_helper.consistency module

vocal_helper.consistency

Is it still the same speaker forty minutes later?

The diarization error rate does not answer that. It is a time-weighted sum of missed speech, false alarm and speaker confusion, scored with a collar, and on a long recording it is dominated by boundaries: thousands of small edge errors swamp the one failure that actually ruins a transcript, which is a speaker losing their identity halfway through and finishing the meeting under a different label.

Two recordings can share a diarization error rate of 0.30 and be worlds apart. In one, every turn is attributed correctly and the boundaries are a little loose. In the other, the boundaries are perfect and the chair is called S0 for twenty minutes and S3 after the coffee break. The second is useless for anything that totals speaking time, builds an agreement network, or follows one person across episodes — and the error rate will not tell you which one you have.

So this module measures identity, and only identity.

Everything here is measured after an optimal one-to-one assignment between true speakers and hypothesis labels, solved by the Hungarian algorithm on the matrix of shared time. That is the same reconciliation the diarization error rate performs, and using it here is not a detail — it is what makes the numbers mean anything.

A first version of this module took, for each true speaker independently, whichever label they spent most time under. That is not an assignment: several speakers can claim the same label. The consequence was absurd and it was observed in the wild — a stage that collapsed four voices onto one label scored a drift of 0.000 and a long-range agreement of 0.97, because all four speakers shared one “dominant” label and every appearance therefore matched every other. The metric rewarded total collapse. With a one-to-one assignment the collapse is priced correctly: the label goes to one speaker and the other three are simply wrong everywhere.

The three numbers

Drift, per true speaker: one minus the share of their speech that landed on their assigned label. Zero is perfect. It answers “was this person held together, under their own name?”

Collision, per hypothesis label: one minus the share of that label’s time belonging to its assigned speaker. Zero is perfect. It answers “does this label mean one person?”

Both are needed, because each alone is still gameable at the edges, and together with the assignment they pin both failure directions: splitting one voice across labels, and merging several voices into one.

Long-range agreement, as a function of a gap: among all pairs of appearances by the same true speaker separated by at least that gap, the fraction where both appearances landed on that speaker’s assigned label. This is the number that answers the question in the title, and the one a windowed or block-wise diarizer has to be judged on: its whole identity machinery exists to make the curve flat. Requiring the assigned label, rather than merely the same label, is what stops a uniformly wrong answer from scoring as a consistent one.

Why a synthetic corpus is the right instrument here

Real meeting corpora give a reference, but they do not let you choose when a speaker comes back. To measure long-range consistency you want one voice appearing at minute one, minute ten and minute forty, with nothing of theirs in between — and you want to know, for certain, that it is the same voice.

The in-house bagarre fabricator (pdbms/scripts/fabricate_bagarre.py) builds exactly that: every turn is cut from a known mono-speaker clip, so the identity is exact everywhere by construction rather than by annotation, and the recipe records which speaker is in which interval. Annotation disagreement, the usual noise floor of this kind of measurement, simply does not exist.

Usage example

>>> reference = [(0.0, 10.0, "alice"), (10.0, 20.0, "bob"), (60.0, 70.0, "alice")]
>>> perfect = [(0.0, 10.0, "S0"), (10.0, 20.0, "S1"), (60.0, 70.0, "S0")]
>>> report = speaker_consistency(reference, perfect)
>>> report.worst_drift, report.worst_collision
(0.0, 0.0)
>>> report.long_range_agreement(gap_s=30.0)
1.0

The failure the error rate hides: alice is renamed after the gap.

>>> renamed = [(0.0, 10.0, "S0"), (10.0, 20.0, "S1"), (60.0, 70.0, "S2")]
>>> report = speaker_consistency(reference, renamed)
>>> round(report.worst_drift, 2)
0.5
>>> report.long_range_agreement(gap_s=30.0)
0.0

And the collapse that the first version of this module rewarded, now priced:

>>> collapsed = [(0.0, 20.0, "S0"), (60.0, 70.0, "S0")]
>>> report = speaker_consistency(reference, collapsed)
>>> report.worst_drift        # bob never gets a label of his own
1.0
>>> report.long_range_agreement(gap_s=30.0)
1.0

Alice keeps her label across the gap, so her long-range agreement is honest; it is worst_drift that reports bob’s disappearance. One number cannot carry both, which is why there are three.

class vocal_helper.consistency.ConsistencyReport(drift=<factory>, collision=<factory>, assigned_label=<factory>, appearances=<factory>, label_time=<factory>)[source]

Bases: object

Identity quality, independent of boundary quality.

Parameters:

drift (dict[str, float])
collision (dict[str, float])
assigned_label (dict[str, str])
appearances (dict[str, list[tuple[float, float, str]]])
label_time (dict[str, float])

appearances: dict[str, list[tuple[float, float, str]]]: Per true speaker, their reference intervals with the label each one received.

assigned_label: dict[str, str]

Per true speaker, the label the one-to-one assignment gave them.

A speaker absent from this mapping received no label of their own, which happens when the hypothesis has fewer labels than there are speakers — a collapse.

collision: dict[str, float]

one minus the share of its time on its ASSIGNED speaker.

Type:: Per hypothesis label

describe()[source]

One line, for a log or a study table.

Return type:: str

drift: dict[str, float]

one minus the share of their speech on their ASSIGNED label.

Type:: Per true speaker

label_time: dict[str, float]: Seconds of reference speech each hypothesis label covers.

long_range_agreement(*, gap_s)[source]

Share of same-speaker appearance pairs, at least gap_s apart, that kept a label.

Parameters:: gap_s (float, keyword-only) – Minimum silence between the end of one appearance and the start of the next, for the pair to count. A large value isolates exactly the failure this module exists for: identity lost across a long absence.
Returns:: In [0, 1], or 1.0 when no pair is that far apart — nothing was asked of the diarizer at that range, so nothing is held against it.
Return type:: float

Examples

>>> reference = [(0.0, 5.0, "a"), (100.0, 105.0, "a")]
>>> kept = speaker_consistency(reference, [(0.0, 5.0, "S0"), (100.0, 105.0, "S0")])
>>> kept.long_range_agreement(gap_s=60.0)
1.0
>>> kept.long_range_agreement(gap_s=600.0)   # nothing that far apart
1.0

property misattributed_share: float

Share of labelled speech time that sits on the wrong speaker.

The time-weighted companion to worst_collision, and the number to quote when comparing two diarizers. A handful of tiny orphan clusters barely moves it; half the audio under a wrong name sends it to 0.5.

Returns:: In [0, 1]. Zero means every labelled second belongs to the speaker its label was assigned to.
Return type:: float

Examples

A tiny spurious label next to an otherwise perfect answer: the maximum saturates, the share stays honest.

>>> reference = [(0.0, 100.0, "alice")]
>>> almost = [(0.0, 99.9, "S0"), (99.9, 100.0, "S1")]
>>> report = speaker_consistency(reference, almost)
>>> report.worst_collision
1.0
>>> round(report.misattributed_share, 4)
0.001

property worst_collision: float

The most ambiguous label.

Saturates, and that is a known limit of the maximum. One spurious label holding a tenth of a second is assigned to nobody, scores 1.0 on its own, and pins this number at 1.0 whatever the rest of the recording looks like. It answers “is any label meaningless?” — useful as a tripwire, useless as a measure of how bad things are. For that, read misattributed_share.

property worst_drift: float: The least well-held speaker. One bad voice ruins a transcript, so the maximum is the honest summary and the mean would hide it.

vocal_helper.consistency.speaker_consistency(reference, hypothesis, *, min_share=0.0)[source]

Measure whether identities were held, ignoring boundary accuracy.

Parameters:

reference (list of tuple) – (t0, t1, true_speaker). From a synthetic recipe, where identity is exact by construction, or from a hand annotation.
hypothesis (list of tuple) – (t0, t1, label) as produced by a diarizer.
min_share (float, keyword-only, optional) – Ignore a label that holds less than this share of a speaker’s time, as boundary spill rather than a genuine identity split. 0.0 counts everything, which is the strict reading and the default: a frame of spill is still an error, it is simply a small one, and the share-weighted drift already makes it small.

Return type:

ConsistencyReport

Examples

Two voices merged into one label. Both numbers now report it, because the assignment can only give S0 to one of them: the other is unassigned and therefore wrong everywhere.

>>> reference = [(0.0, 10.0, "alice"), (10.0, 20.0, "bob")]
>>> merged = [(0.0, 20.0, "S0")]
>>> report = speaker_consistency(reference, merged)
>>> report.worst_drift, report.worst_collision
(1.0, 0.5)
>>> sorted(report.assigned_label)        # only one of the two got a label
['alice']

One voice split in half: the mirror image. Alice keeps one of the two labels, so half her speech is lost to the other — and the orphan label belongs to nobody, which scores a collision of 1.

>>> split = [(0.0, 10.0, "S0"), (10.0, 20.0, "S1")]
>>> report = speaker_consistency([(0.0, 20.0, "alice")], split)
>>> report.worst_drift, report.worst_collision
(0.5, 1.0)

That an unowned label costs a full collision is deliberate, and it is what makes this a direct detector of over-segmentation: the repository’s open high-severity bug is a clusterer that minted 203 labels for four speakers, and 199 of them would be unowned here.