speaker_helper.backends.audio8 module

Audio8: a compact realism-tuned cloner, offline, from a reference recording.

Module summary

Edge0’s Audio8-TTS-Realistic (0.6B parameters, Apache-2.0, 23 languages including French; arXiv 2610.14776) was trained SFT → GRPO on WER and speaker similarity → GRPO on a realism critic that compares each output with a human saying the same sentence in the same voice. It is the 2026-11 candidate for the owner’s offline case, where the brief is a voice that sounds alive, not a fast one.

It renders by cloning: a reference recording plus its exact transcript conditions every call. Without a reference the model picks a speaker of its own, and measured on 2026-11-04 that speaker is male (median F0 139 Hz in French, 163 Hz in English). So this backend always passes a reference, and by default the bundled female one (assets/ref-fr-female.wav + .txt, CC BY 4.0): with it the French output measured F0 274 Hz and a TitaNet similarity of 0.72 to the reference.

One runtime, said plainly

Only the MLX path is wired: mlx-audio (≥ 0.6.4 — 0.6.1 fails to dequantise the tied 4-bit embedding) running the community 4-bit conversion nativ-community/Audio8-TTS-Realistic-0.6b-4bit (~1.7 GB, cached once, offline after). Upstream’s PyTorch path needs transformers<5 with trust_remote_code and has not been exercised here, so off a Mac this backend reports itself unavailable instead of shipping untested code.

The codec decodes at 44.1 kHz; every other backend lays audio out at 24 kHz, so the output is resampled — otherwise a narration mixing engines would carry wrong cue times after the first segment.

There is no phoneme input and no streaming: pronunciation-lexicon entries reach it through the respelling tier applied by Speaker (see speaker_helper.lexicon), and each call returns one complete waveform.

Usage example

>>> from speaker_helper import Speaker, Settings
>>> # spk = Speaker(Settings.from_mapping({"backend": "audio8", "language": "fr"}))
>>> # await spk.say("Le rendez-vous est fixé au 14 novembre.")

Author

Warith HARCHAOUI — https://linkedin.com/in/warith-harchaoui

class speaker_helper.backends.audio8.Audio8Engine(settings)[source]

Bases: object

The Audio8 backend: text in, a cloned (female by default) voice out.

Parameters:

settings (Settings) –

voice_sample (or a per-call voice) names the reference recording; empty picks the bundled female reference. The audio8 mapping supplies the knobs:

  • repo — MLX model repository (default DEFAULT_REPO).

  • temperature / top_p / top_k — sampling (upstream defaults 0.7 / 0.9 / 50).

  • seed — sampling seed (default 0; null draws fresh).

  • max_tokens — codec-frame budget (default 2048, clamped by the runtime to what the context leaves).

DEFAULT_REPO_ID: str = 'nativ-community/Audio8-TTS-Realistic-0.6b-4bit'

Default model repository and its rough download size (for the log line).

DOWNLOAD_SIZE: str = '~1.7 GB'
ENGINE: str = 'audio8'

Backend name, settings section and pip extra.

async aclose()[source]

Drop the implementation so MLX can free the weights.

Return type:

None

async clone_voice(name, samples, *, language=None)[source]

Adopt the first sample, with its transcript, as the reference.

As with chatterbox there is no enrolment: the model conditions on the recording at generation time, so the honest return value is the path. Audio8 also needs what the recording says; an empty reference_text falls back to reference_transcript().

Parameters:
Return type:

str

async health()[source]

Whether this engine could synthesise, without loading the model.

Return type:

dict

async list_voices(engine=None)[source]

One entry: the reference this engine will actually clone.

Same contract as the chatterbox backend: the bundled reference is named with its provenance and its gender; somebody else’s recording is named by file, and its gender left to them.

Parameters:

engine (str | None)

Return type:

VoiceList

async synthesize(text, *, language=None, voice=None)[source]

Synthesise text into one AudioResult.

No speed parameter on purpose: the model has no rate control, and resampling would shift pitch with pace. narrate() detects the missing keyword and withdraws the knob instead of failing.

Parameters:
  • text (str) – What to say (must be non-empty).

  • language (str or None) – Recorded on the result and used as the ASR hint for a reference transcript; the model itself reads the language from the text.

  • voice (str or None) – Path to a reference recording for this call; empty keeps the configured (or bundled female) reference.

Raises:

ValueError – If text is empty or whitespace-only.

Return type:

AudioResult

speaker_helper.backends.audio8.DEFAULT_MAX_TOKENS: int = 2048

Generation budget in codec frames. The model shares 2,048 positions between text, reference and audio, and the runtime clamps to what is left; upstream’s 512 default stops at ~24 s (21.5 frames/s), shorter than one chunk can be.

speaker_helper.backends.audio8.DEFAULT_REPO: str = 'nativ-community/Audio8-TTS-Realistic-0.6b-4bit'

the community MLX 4-bit conversion.

Type:

The default model repository

speaker_helper.backends.audio8.DEFAULT_SEED: int = 0

Fixed sampling seed, so a synthesis is reproducible run to run — the evaluation layer compares takes, and a fresh draw each time is noise there.

speaker_helper.backends.audio8.MIN_MLX_AUDIO: tuple[int, int, int] = (0, 6, 4)

The oldest mlx-audio that loads the 4-bit checkpoint (0.6.1 raised a matmul shape error on the tied embedding, measured 2026-11-04).

speaker_helper.backends.audio8.NATIVE_RATE_HZ: int = 44100

The codec’s native rate; resample() bridges the two.

speaker_helper.backends.audio8.QWEN3_LANGUAGE_NAMES: dict[str, str] = {'de': 'german', 'en': 'english', 'es': 'spanish', 'fr': 'french', 'it': 'italian', 'ja': 'japanese', 'ko': 'korean', 'pt': 'portuguese', 'ru': 'russian', 'zh': 'chinese'}

Qwen3-TTS names languages in words; anything unlisted is left to “auto”.

class speaker_helper.backends.audio8.Qwen3TTSEngine(settings)[source]

Bases: Audio8Engine

Qwen3-TTS 0.6B (Apache-2.0) through the same MLX cloning contract.

Measured on 2026-11-04 (M2 Max, 8-bit, the bundled female reference, a two-sentence French paragraph): the closest clone of any engine here (TitaNet similarity 0.82, F0 233 Hz, transcript exact), sustained streaming RTF 0.77–0.90 and first audio after ~0.7–0.9 s. That sits on the router’s 0.8 real-time ceiling rather than under it, and its public Elo is below kokoro’s, so it is a candidate for the owner’s ear, not a route.

Same knobs as Audio8Engine under settings.qwen3tts, except that sampling defaults are the model’s own (only what you set is sent). The codec decodes at 24 kHz, so nothing is resampled.

Caveat met while measuring: on an exFAT/external Hugging Face cache, macOS AppleDouble files (._model.safetensors) make mlx-audio’s loader fail with “Invalid json header length”; dot_clean on the cache fixes it.

Parameters:

settings (Settings)

DEFAULT_REPO_ID: str = 'mlx-community/Qwen3-TTS-12Hz-0.6B-Base-8bit'

Default model repository and its rough download size (for the log line).

DOWNLOAD_SIZE: str = '~2 GB'
ENGINE: str = 'qwen3tts'

Backend name, settings section and pip extra.

speaker_helper.backends.audio8.SAMPLE_RATE_HZ: int = 24000

The rate this wrapper lays audio out at — the same as kokoro, chatterbox and kyutai, so a narration can mix engines.

speaker_helper.backends.audio8.check_install()[source]

Return 0 when the MLX runtime for Audio8 is usable.

Return type:

int

speaker_helper.backends.audio8.install_problem()[source]

Why this engine cannot run here, or "" when it can.

Returns:

A one-line reason (not a Mac, mlx-audio missing or too old), empty when the MLX runtime is usable.

Return type:

str

speaker_helper.backends.audio8.reference_transcript(audio, *, language)[source]

The transcript that must accompany a reference recording.

Audio8 aligns the reference audio to its text, so a wrong or missing transcript degrades the clone instead of failing. Lookup order: the bundled reference’s own transcript, a <stem>.txt beside the file, then ensure_transcript() (vocal-helper, cached as <file>.txt).

Parameters:
  • audio (Path) – The reference recording.

  • language (str) – ISO-639-1 hint for the ASR fallback.

Returns:

The transcript.

Return type:

str

Raises:

RuntimeError – When no transcript is on disk and vocal-helper is not installed.

speaker_helper.backends.audio8.resample(samples, source_hz, target_hz)[source]

Band-limited resampling of a whole mono signal through the FFT.

Offline synthesis hands over the complete waveform, so the exact spectral method costs nothing a streaming filter would save, and it keeps this backend free of a SciPy dependency.

Parameters:
  • samples (numpy.ndarray) – 1-D float samples at source_hz.

  • source_hz (int) – The two rates.

  • target_hz (int) – The two rates.

Returns:

1-D float32 samples at target_hz, the same duration.

Return type:

numpy.ndarray

Examples

>>> import numpy as np
>>> resample(np.zeros(44_100), 44_100, 24_000).shape
(24000,)