speaker_helper.backends.audio8 module
Audio8: a compact realism-tuned cloner, offline, from a reference recording.
Module summary
Edge0’s Audio8-TTS-Realistic (0.6B parameters, Apache-2.0, 23 languages
including French; arXiv 2610.14776) was trained SFT → GRPO on WER and speaker
similarity → GRPO on a realism critic that compares each output with a human
saying the same sentence in the same voice. It is the 2026-11 candidate for the
owner’s offline case, where the brief is a voice that sounds alive, not a fast
one.
It renders by cloning: a reference recording plus its exact transcript
conditions every call. Without a reference the model picks a speaker of its
own, and measured on 2026-11-04 that speaker is male (median F0 139 Hz in
French, 163 Hz in English). So this backend always passes a reference, and by
default the bundled female one (assets/ref-fr-female.wav + .txt,
CC BY 4.0): with it the French output measured F0 274 Hz and a TitaNet
similarity of 0.72 to the reference.
One runtime, said plainly
Only the MLX path is wired: mlx-audio (≥ 0.6.4 — 0.6.1 fails to
dequantise the tied 4-bit embedding) running the community 4-bit conversion
nativ-community/Audio8-TTS-Realistic-0.6b-4bit (~1.7 GB, cached once,
offline after). Upstream’s PyTorch path needs transformers<5 with
trust_remote_code and has not been exercised here, so off a Mac this
backend reports itself unavailable instead of shipping untested code.
The codec decodes at 44.1 kHz; every other backend lays audio out at 24 kHz, so the output is resampled — otherwise a narration mixing engines would carry wrong cue times after the first segment.
There is no phoneme input and no streaming: pronunciation-lexicon entries
reach it through the respelling tier applied by
Speaker (see speaker_helper.lexicon),
and each call returns one complete waveform.
Usage example
>>> from speaker_helper import Speaker, Settings
>>> # spk = Speaker(Settings.from_mapping({"backend": "audio8", "language": "fr"}))
>>> # await spk.say("Le rendez-vous est fixé au 14 novembre.")
- class speaker_helper.backends.audio8.Audio8Engine(settings)[source]
Bases:
objectThe Audio8 backend: text in, a cloned (female by default) voice out.
- Parameters:
settings (Settings) –
voice_sample(or a per-callvoice) names the reference recording; empty picks the bundled female reference. Theaudio8mapping supplies the knobs:repo— MLX model repository (defaultDEFAULT_REPO).temperature/top_p/top_k— sampling (upstream defaults0.7/0.9/50).seed— sampling seed (default0;nulldraws fresh).max_tokens— codec-frame budget (default2048, clamped by the runtime to what the context leaves).
- DEFAULT_REPO_ID: str = 'nativ-community/Audio8-TTS-Realistic-0.6b-4bit'
Default model repository and its rough download size (for the log line).
- async clone_voice(name, samples, *, language=None)[source]
Adopt the first sample, with its transcript, as the reference.
As with chatterbox there is no enrolment: the model conditions on the recording at generation time, so the honest return value is the path. Audio8 also needs what the recording says; an empty
reference_textfalls back toreference_transcript().- Parameters:
name (str)
samples (list[VoiceSample])
language (str | None)
- Return type:
- async health()[source]
Whether this engine could synthesise, without loading the model.
- Return type:
- async list_voices(engine=None)[source]
One entry: the reference this engine will actually clone.
Same contract as the chatterbox backend: the bundled reference is named with its provenance and its gender; somebody else’s recording is named by file, and its gender left to them.
- async synthesize(text, *, language=None, voice=None)[source]
Synthesise
textinto oneAudioResult.No
speedparameter on purpose: the model has no rate control, and resampling would shift pitch with pace.narrate()detects the missing keyword and withdraws the knob instead of failing.- Parameters:
text (str) – What to say (must be non-empty).
language (str or None) – Recorded on the result and used as the ASR hint for a reference transcript; the model itself reads the language from the text.
voice (str or None) – Path to a reference recording for this call; empty keeps the configured (or bundled female) reference.
- Raises:
ValueError – If
textis empty or whitespace-only.- Return type:
- speaker_helper.backends.audio8.DEFAULT_MAX_TOKENS: int = 2048
Generation budget in codec frames. The model shares 2,048 positions between text, reference and audio, and the runtime clamps to what is left; upstream’s 512 default stops at ~24 s (21.5 frames/s), shorter than one chunk can be.
- speaker_helper.backends.audio8.DEFAULT_REPO: str = 'nativ-community/Audio8-TTS-Realistic-0.6b-4bit'
the community MLX 4-bit conversion.
- Type:
The default model repository
- speaker_helper.backends.audio8.DEFAULT_SEED: int = 0
Fixed sampling seed, so a synthesis is reproducible run to run — the evaluation layer compares takes, and a fresh draw each time is noise there.
- speaker_helper.backends.audio8.MIN_MLX_AUDIO: tuple[int, int, int] = (0, 6, 4)
The oldest
mlx-audiothat loads the 4-bit checkpoint (0.6.1 raised a matmul shape error on the tied embedding, measured 2026-11-04).
- speaker_helper.backends.audio8.NATIVE_RATE_HZ: int = 44100
The codec’s native rate;
resample()bridges the two.
- speaker_helper.backends.audio8.QWEN3_LANGUAGE_NAMES: dict[str, str] = {'de': 'german', 'en': 'english', 'es': 'spanish', 'fr': 'french', 'it': 'italian', 'ja': 'japanese', 'ko': 'korean', 'pt': 'portuguese', 'ru': 'russian', 'zh': 'chinese'}
Qwen3-TTS names languages in words; anything unlisted is left to “auto”.
- class speaker_helper.backends.audio8.Qwen3TTSEngine(settings)[source]
Bases:
Audio8EngineQwen3-TTS 0.6B (Apache-2.0) through the same MLX cloning contract.
Measured on 2026-11-04 (M2 Max, 8-bit, the bundled female reference, a two-sentence French paragraph): the closest clone of any engine here (TitaNet similarity 0.82, F0 233 Hz, transcript exact), sustained streaming RTF 0.77–0.90 and first audio after ~0.7–0.9 s. That sits on the router’s 0.8 real-time ceiling rather than under it, and its public Elo is below kokoro’s, so it is a candidate for the owner’s ear, not a route.
Same knobs as
Audio8Engineundersettings.qwen3tts, except that sampling defaults are the model’s own (only what you set is sent). The codec decodes at 24 kHz, so nothing is resampled.Caveat met while measuring: on an exFAT/external Hugging Face cache, macOS AppleDouble files (
._model.safetensors) make mlx-audio’s loader fail with “Invalid json header length”;dot_cleanon the cache fixes it.- Parameters:
settings (Settings)
- speaker_helper.backends.audio8.SAMPLE_RATE_HZ: int = 24000
The rate this wrapper lays audio out at — the same as kokoro, chatterbox and kyutai, so a narration can mix engines.
- speaker_helper.backends.audio8.check_install()[source]
Return 0 when the MLX runtime for Audio8 is usable.
- Return type:
- speaker_helper.backends.audio8.install_problem()[source]
Why this engine cannot run here, or
""when it can.- Returns:
A one-line reason (not a Mac,
mlx-audiomissing or too old), empty when the MLX runtime is usable.- Return type:
- speaker_helper.backends.audio8.reference_transcript(audio, *, language)[source]
The transcript that must accompany a reference recording.
Audio8 aligns the reference audio to its text, so a wrong or missing transcript degrades the clone instead of failing. Lookup order: the bundled reference’s own transcript, a
<stem>.txtbeside the file, thenensure_transcript()(vocal-helper, cached as<file>.txt).- Parameters:
audio (Path) – The reference recording.
language (str) – ISO-639-1 hint for the ASR fallback.
- Returns:
The transcript.
- Return type:
- Raises:
RuntimeError – When no transcript is on disk and vocal-helper is not installed.
- speaker_helper.backends.audio8.resample(samples, source_hz, target_hz)[source]
Band-limited resampling of a whole mono signal through the FFT.
Offline synthesis hands over the complete waveform, so the exact spectral method costs nothing a streaming filter would save, and it keeps this backend free of a SciPy dependency.
- Parameters:
- Returns:
1-D float32 samples at
target_hz, the same duration.- Return type:
numpy.ndarray
Examples
>>> import numpy as np >>> resample(np.zeros(44_100), 44_100, 24_000).shape (24000,)