speaker_helper.narration module

Narration: a document read aloud, and the timed track that follows it.

Module summary

synthesize() answers “what does this sentence sound like”. This module answers the different question a document asks: read these blocks, in this order, with the pauses their structure calls for, and tell me where each one landed in the audio so a page can follow along while it plays.

That second half is the part a plain text -> wav call cannot give you, and the part that is easy to get subtly wrong. Three facts, each learned the expensive way:

What the voice says is not always what the page shows. A formula is notation on the page and a sentence in the ear; a footnote marker is a signpost on the page and a stray number in the ear. So a Segment carries both: text is the page’s words and goes into the cue, say is what the engine is handed. The invariant is that the cue keeps the page’s words — that is what lets a read-along highlight land on the right ones.

A segment with nothing speakable in it still takes its turn. A French page closing a quotation on its own line gives a “sentence” that is a single ». Kokoro refuses it — nothing to synthesize, produced no phonemes — and the obvious repair, dropping it, is worse than the crash: the page still shows that block, so every later sentence would light up one place out. It becomes a short silence that keeps its rank.

The timeline and the file must agree, and something has to check. The offsets are exact and already known while the audio is concatenated; the risk is a piece appended without being recorded. Timeline.total_seconds() is compared against the written file, and a mismatch is an error rather than a narration whose highlighting drifts further out with every sentence.

Usage example

>>> timeline = Timeline(1000)
>>> timeline.silence(500)
500
>>> timeline.speech(2000, index=0, text="Hello.")
>>> timeline.total_seconds
2.5
>>> print(cues_to_vtt(timeline.cues))
WEBVTT

0
00:00:00.500 --> 00:00:02.500
Hello.

Author

Warith HARCHAOUI — https://linkedin.com/in/warith-harchaoui

class speaker_helper.narration.NarrationResult(wav_bytes, sample_rate, duration_s, compute_s, cues=<factory>, voice_id='', language='', detected_language='', language_fallback=False, ignored_hints=<factory>)[source]

Bases: object

A narrated document: the audio, and where every block landed in it.

Parameters:

wav_bytes (bytes) – The complete WAV file (RIFF header + PCM).
sample_rate (int) – Sample rate in Hz.
duration_s (float) – Length of the audio.
compute_s (float) – Wall-clock seconds spent synthesising.
cues (list of SpeechCue) – Where each segment is spoken. Free to produce — the offsets were already computed to lay the audio out — and the only thing that lets a page follow a narration instead of merely playing it.
voice_id (str) – The voice that read it.
language (str) – The language it was phonemised in. Getting this wrong is not an accent: French read as en-us is an American reading French spelling aloud, letter patterns and all.
detected_language (str) – What the text turned out to be written in, when nobody named a voice. Empty when the caller chose one — this is the detector’s answer, not a restatement of the request.
language_fallback (bool) – True when the text is in a language the backend has no voice for, so it is being read by a voice that does not speak it. Said out loud, because the audio alone will not make it obvious to whoever generated it and does not speak the language either.
ignored_hints (list of str) – Dials the segments asked for and this backend does not have. Empty on a plain narration; non-empty means the caller should know which engine does have them.

compute_s: float

cues: list[SpeechCue]

detected_language: str = ''

duration_s: float

ignored_hints: list[str]

language: str = ''

language_fallback: bool = False

property rtf: float: Real-time factor compute_s / duration_s (inf if silent).

sample_rate: int

to_vtt()[source]

The timed track, as WebVTT.

Return type:: str

voice_id: str = ''

wav_bytes: bytes

speaker_helper.narration.PACE_SPEED: dict[str, float] = {'fast': 1.15, 'normal': 1.0, 'slow': 0.85}: Multipliers the engines are handed for each pace.

speaker_helper.narration.PUNCTUATION_PAUSE_MS: int = 120: How long a segment with nothing speakable in it is held for. Short enough to read as punctuation rather than as a gap, long enough to exist.

class speaker_helper.narration.Segment(text, say='', pause_before_ms=0.0, pause_after_ms=0.0, pace='normal', emotion='', intensity='')[source]

Bases: object

One block of a document, as the narrator receives it.

Parameters:

text (str) – The page’s words. This is what the cue carries, and therefore what a read-along highlights. Never substituted.
say (str) – What the engine should actually pronounce, when that differs — a formula read as a sentence, a footnote marker dropped. Empty means “say text”, which is the ordinary case.
pause_before_ms (float) – Silence the document’s structure asks for around this block — the gap that sets a heading apart from the paragraph before it. Requested by the caller, applied here.
pause_after_ms (float) – Silence the document’s structure asks for around this block — the gap that sets a heading apart from the paragraph before it. Requested by the caller, applied here.
pace (str) – "slow", "normal" or "fast". See PACE_SPEED.
emotion (str) – Asked for by some documents, honoured by some engines. A backend that has no such dial reports the name in NarrationResult.ignored_hints rather than pretending.
intensity (str) – Asked for by some documents, honoured by some engines. A backend that has no such dial reports the name in NarrationResult.ignored_hints rather than pretending.

Notes

pause_before_ms is skipped on the first segment: a “pause before” with nothing before it is dead air at the head of the file.

emotion: str = ''

classmethod from_mapping(raw)[source]

Build a segment from a plain dict, ignoring keys it does not know.

The wire format between a document tool and this one is JSON, and the document side is free to carry extra keys (block ids, heading levels) that are none of a narrator’s business. Unknown keys are dropped here rather than rejected, so adding one upstream is not a breaking change.

Parameters:: raw (dict[str, Any])
Return type:: Segment

intensity: str = ''

property is_speakable: bool

Whether anything here could be pronounced at all.

False for a segment that is only punctuation. Such a segment is not dropped — see this module’s docstring — it becomes a silence that keeps its rank.

pace: str = 'normal'

pause_after_ms: float = 0.0

pause_before_ms: float = 0.0

say: str = ''

property spoken: str

say when set, else text.

Type:: What to hand the engine

text: str

class speaker_helper.narration.SpeechCue(index, text, start_seconds, end_seconds)[source]

Bases: object

One narrated block, with the slice of audio that speaks it.

Parameters:

index (int) – Position in the original segment list, not in the audio. The two differ: a segment whose text is empty after cleaning is never synthesised, so counting emitted pieces drifts from counting segments. Carrying the original index is what lets a page map a cue back to the block it came from.
text (str) – The page’s words — see Segment.
start_seconds (float) – Where this block is spoken, from the head of the file.
end_seconds (float) – Where this block is spoken, from the head of the file.

end_seconds: float

index: int

start_seconds: float

text: str

class speaker_helper.narration.Timeline(sample_rate_hz)[source]

Bases: object

Accumulates durations while a backend accumulates audio.

Two jobs, one object: apply the pauses the segments ask for, and record where every segment landed. Both need the same running sample count, and keeping them apart is how they drift.

Deliberately numpy-free: it counts samples and the caller holds the arrays, so this module imports with nothing but the standard library — which is what lets a caller build and inspect a narration plan without the ONNX runtime anywhere near it.

Examples

>>> timeline = Timeline(1000)
>>> timeline.silence(500)
500
>>> timeline.speech(2000, index=0, text="Hello.")
>>> timeline.silence(250)
250
>>> [(c.start_seconds, c.end_seconds) for c in timeline.cues]
[(0.5, 2.5)]
>>> timeline.total_seconds
2.75

Parameters:: sample_rate_hz (int)

property cues: list[SpeechCue]: Every recorded cue, in playback order.

silence(milliseconds)[source]

Advance the clock by milliseconds; return the samples to append.

Returning the count rather than an array is what keeps this module numpy-free. A caller that ignores the return value desynchronises the timeline from its own audio, which is exactly what check_timeline_matches_audio() exists to catch.

Parameters:: milliseconds (float)
Return type:: int

speech(sample_count, *, index, text)[source]

Record that sample_count samples of audio speak text.

Parameters:

sample_count (int)
index (int)
text (str)

Return type:

None

property total_seconds: float: Length of the audio described so far, silences included.

speaker_helper.narration.check_timeline_matches_audio(timeline, duration_s)[source]

Refuse a narration whose cues do not describe the file that was written.

The timeline counted exactly what the backend said it appended, so a disagreement means a piece reached the audio without being recorded. The symptom of letting that through is not a crash: it is a narration that plays correctly and highlights further and further behind itself, which looks like a bug in whatever renders it.

Raises:

ValueError – When the two disagree by more than a sample’s worth of rounding.

Parameters:

timeline (Timeline)
duration_s (float)

Return type:

None

speaker_helper.narration.coerce_segments(raw)[source]

Accept segments as dicts, as Segment, or as bare strings.

Three shapes because three callers: a document tool writes JSON, a Python caller builds dataclasses, and somebody at a REPL has a list of sentences and no interest in any of this.

Parameters:: raw (list[Any])
Return type:: list[Segment]

speaker_helper.narration.cues_to_vtt(cues)[source]

Render narration cues as WebVTT.

Each cue carries its segment index as the WebVTT identifier — the line before the timings — which is what lets a page highlight the right block without matching on text. The text is collapsed onto one line because a blank line ends a cue, so a hard-wrapped payload would silently become two cues and shift every index after it.

Parameters:: cues (list[SpeechCue])
Return type:: str

speaker_helper.narration.format_timestamp(seconds)[source]

Format seconds as WebVTT’s HH:MM:SS.mmm.

Negative input clamps to zero: a negative timestamp is never meaningful and a renderer’s behaviour on one is undefined rather than obvious.

Parameters:: seconds (float)
Return type:: str

speaker_helper.narration.resolve_spoken_math(segments, lang, *, narrator=None)[source]

Fill each segment’s say with a readable form of its notation.

A convenience over speaker_helper.speech_text.speakable(), which is where the work actually happens. Separate from Segment on purpose: a segment whose say was set deliberately — a pronunciation override, a formula a human already phrased — must not be overwritten, so this only fills the empty ones.

Parameters:

segments (list of Segment) – Left untouched; a new list is returned.
lang (str) – The language the page is written in. Not optional and not defaulted: a French page read with English operator names is worse than one that says nothing, because the listener hears fluent words in the wrong language and assumes they misheard.
narrator (callable or None) – (latex, lang) -> str | None, asked about formulas too complex to spell out. Optional — without one every formula gets the deterministic reading, which is the baseline, not a degraded mode.

Returns:

The same blocks, with say filled where it was empty and the notation differs from the words.

Return type:

list of Segment