Prerequisites for TTS and the Hard Technical Limits of Neural Speech Synthesis in 2026

Most people who build TTS pipelines for the first time make the same mistake: they treat the task as a single transformation from string to audio file, and they do not think about the intermediate representations until something breaks. The break usually arrives as flat, robotic output on emotionally charged text, or as a latency spike that makes real-time voice agents unusable, or as a mispronounced proper noun that no amount of prompt engineering will fix.

The mechanism underneath neural speech synthesis is more layered than it appears from the API surface. Knowing what that mechanism actually is — and where it structurally cannot succeed — determines whether you build something usable or something that needs a complete rearchitect six weeks after launch.

The Stack Before You Write a Single API Call

What do you need to understand before building a text-to-speech pipeline?

Before touching any model, the conceptual prerequisite is understanding that modern Text-to-Speech systems are not monolithic end-to-end transformations. They are, at minimum, a two-stage process: a text-to-acoustic-feature stage and an acoustic-feature-to-waveform stage. Some architectures collapse both into one network; others keep them separate. The distinction is not cosmetic — it determines latency profile, fine-tuning surface, and failure modes.

The first stage takes text as input and produces intermediate acoustic representations. The canonical intermediate is the Mel Spectrogram: a time-frequency representation of audio energy that captures how loudness is distributed across frequency bands over time. Tacotron 2, the foundational encoder-decoder architecture that established this pipeline pattern, trained a recurrent encoder-decoder with attention to map text to mel spectrograms, which a separate WaveNet Vocoder then converted to waveforms. This two-stage design became the template that most subsequent architectures either followed or reacted against.

The second stage — the vocoder — is where waveforms are synthesized from mel spectrograms. Early neural vocoders like WaveNet were autoregressive: they generated audio sample by sample, which gave high fidelity but made real-time generation computationally expensive. HiFi-GAN and BigVGAN replaced autoregressive generation with generative adversarial training, producing high-quality audio at speeds suitable for streaming. BigVGAN v2 now supports 44.1kHz output, a quality level that was impractical for real-time synthesis until recently.

The conceptual prerequisite list for building a TTS pipeline has five items:

1. Phoneme processing. Text is ambiguous in ways that are invisible to readers. “Read” can be /rɛd/ or /riːd/; “lead” maps to two different words with different pronunciations. The system resolves this ambiguity through Phoneme sequences — symbolic representations of sounds — derived either from a pronunciation dictionary or from a grapheme-to-phoneme (G2P) model. Out-of-vocabulary words (OOV), proper nouns, and domain-specific abbreviations require a working G2P fallback; without one, alignment failures are common (Smallest.ai).

2. Prosody modeling. Prosody — the rhythm, stress, and intonation of speech — is not encoded in text. A question mark signals rising intonation at the sentence level; it says nothing about where emphasis falls, how long pauses should be, or whether a word is being stressed for irony. Systems must predict duration and pitch contours at the token level from contextual signals that are often absent from the input.

3. Vocoder selection. The vocoder choice determines the quality ceiling and the inference speed floor. Neural vocoders trained on a specific speaker’s data generalize poorly; vocoders trained on diverse data tend to be slower or lower quality. This is a concrete engineering decision, not an implementation detail.

4. Streaming vs. batch generation. Many systems buffer several hundred milliseconds of audio before beginning playback, which is tolerable for audiobooks but unacceptable for voice agents in live conversations. Streaming synthesis requires generating audio chunks incrementally, which constrains architecture choices significantly.

5. Cloning surface vs. preset voices. Voice Cloning systems like XTTS (originally from Coqui AI, now maintained as a community fork at idiap/coqui-ai-TTS after Coqui’s shutdown in January 2024) and Fish Audio require reference audio of varying minimum lengths. Fish Audio S2 Pro requires at minimum 10 seconds of reference audio for voice cloning (Fish Audio Docs). Systems like Kokoro TTS take the opposite approach: no voice cloning at all, only 54 preset voices with blending capability. Both choices have engineering consequences that propagate through the rest of the pipeline.

Understanding all five before architecture selection is not excessive due diligence. It is the minimum required to make non-random decisions about model choice.

The Architecture Divide: Two Generations of Neural TTS

Two architectural generations define the current model space, and their failure modes are different in ways that matter for pipeline design.

Generation 1 (sequential two-stage): Tacotron 2 exemplifies this approach — text goes to mel spectrogram via an encoder-decoder, mel spectrogram goes to waveform via a vocoder. The advantage is modularity: you can swap the vocoder independently. The disadvantage is error accumulation across stages, plus the latency of running two models sequentially.

Generation 2 (end-to-end, flow-based): VITS collapsed the two stages into a single model using a variational autoencoder (VAE) with adversarial training — the mel spectrogram becomes an internal latent representation rather than an explicit output. This eliminates the vocoder error cascade and enables faster inference, but it also collapses the fine-tuning surface: end-to-end architecture trades modularity for speed. You cannot independently adjust the acoustic model without affecting waveform quality.

Cartesia Sonic represents the current commercial state of the art. Sonic 3.5 achieves a 90ms time-to-first-audio (TTFA) and supports 40+ languages (Cartesia Docs). As of the Artificial Analysis Speech Arena in June 2026, Sonic 3.5 held the top position by ELO score — but the ELO scores between the top five commercial models are close enough that rankings shift weekly. Treat any current leaderboard position as a snapshot, not a verdict.

ElevenLabs Flash v2.5 achieves approximately 75ms latency (excluding network overhead) with a 40,000 character per request limit (ElevenLabs Docs) — the Flash model optimizes for latency while the flagship Eleven v3 optimizes for expressiveness across 70+ languages with a tighter 5,000 character per request limit.

The open-weight tier is led by Fish Audio S2 Pro, which trained on 10 million+ hours of audio across 80+ languages. Kokoro TTS, at 82 million parameters and 300MB (Hugging Face), achieves 210× real-time speed on an RTX 4090 and carries an Apache 2.0 license, which allows unrestricted commercial use.

Where Neural TTS Structurally Cannot Succeed

What are the technical limitations of AI text-to-speech systems in 2026?

The correct frame here is not “which model has the most bugs” but rather “which limitations are structural.” A structural limitation is one that cannot be fixed by scaling the model or by switching to a different provider — it is built into the task formulation or the underlying physics of speech synthesis.

There are three.

The latency-expressiveness tradeoff. This is the central structural tension in TTS in 2026. Expressive synthesis — conveying emotion, emphasis, irony, or hesitation — requires predicting prosodic features that depend on long-range context within the utterance. A sentence that ends with an unexpected twist requires different intonation than the same sentence read straight. Computing that context takes time. Systems optimized for sub-100ms TTFA must begin generating audio before they have processed the full input, which forces them to produce prosodically naive output for the first chunks.

The measurement data is not theoretical. Google Chirp HD, optimized for expressiveness, has measured latency in the range of 2,000–3,400ms for emotionally expressive synthesis — rendering it unusable for real-time voice agents despite high quality scores (Deepgram). The tradeoff is not a tuning parameter. It is a consequence of the information dependency structure of prosody.

Flat prosody in parallel generation. FastSpeech and related parallel generation models achieved faster inference than autoregressive models by removing sequential dependencies — but they inherited their prosody from teacher alignment, meaning they cannot correct flat intonation after the fact (arXiv). Parallel speedup sacrifices prosodic information flow. This is not a training data problem. It is an architectural constraint: if you parallelize the generation, you lose the causal chain that intonation requires.

OOV and proper noun failure. Grapheme-to-phoneme coverage is finite. Any G2P model trained on a fixed pronunciation dictionary will encounter words it has not seen — new proper nouns, brand names, technical abbreviations, words from low-resource languages transliterated into the target script. When the G2P fallback fails, the resulting phoneme sequence contains alignment errors that the acoustic model propagates faithfully into the audio. The mispronunciation is not a random error; it is a systematic consequence of distribution mismatch between training vocabulary and deployment vocabulary.

A fourth limitation that is often overlooked: downstream codec distortion. Standard telephony codecs, when applied to TTS output for voice agent delivery, introduce measurable distortion to prosodic parameters — a phenomenon sometimes called “digital flat affect.” Measured deviation in prosodic parameters from codec compression has been documented at up to 20% in controlled analysis (Deepgram). This means that a system which sounds expressive in testing may sound flat in deployment, with the signal degradation happening not in the TTS model at all but in the audio transport layer.

What the Architecture Actually Predicts About Failure

If you understand the mechanism, the failure modes become predictable rather than surprising.

If you prioritize sub-100ms TTFA, expect prosodic flatness on emotionally complex text — the model cannot resolve long-range intonation dependencies before it must begin generating.

If you use a parallel generation model like FastSpeech variants, expect duration and pitch predictions to regress toward the training distribution mean on unusual inputs, even when the text signals clear prosodic intent.

If your deployment text contains domain-specific terminology, technical abbreviations, or proper nouns from underrepresented languages, expect systematic mispronunciation that scales with OOV rate, not with model size.

If your audio transport uses standard telephony codecs (G.711, G.722, or similar), expect measurable prosodic degradation independent of model quality.

A practical consequence: Kokoro TTS has a hard context limit of 510 tokens per pass, a ceiling noted in its documentation. Inputs exceeding this require segmentation, and segment boundaries introduce prosodic discontinuities — the model cannot attend across the boundary, so intonation resets rather than flows. This is not a Kokoro-specific bug; it is a manifestation of the general limitation that prosody requires long-range context.

Rule of thumb: The model quality ceiling visible in benchmarks measures clean, single-sentence synthesis. Production pipelines operate on paragraphs, with OOV words, under codec compression, with streaming constraints. Every one of these factors degrades benchmark quality. The gap between benchmark and production is significant and should be treated as a design input, not a surprise.

When it breaks: TTS pipeline failures are rarely random. They concentrate on prosodic boundaries (clause endings, emphasis markers, rhetorical questions), OOV terms, and audio segments that cross streaming chunk boundaries — if your model sounds robotic specifically on those patterns, the architecture, not the training data, is the cause.

Security & compatibility notes:

• XTTS v2 / Coqui TTS (frozen model): Coqui AI shut down January 2024. The original coqui-ai/TTS repository is unmaintained; the community fork (idiap/coqui-ai-TTS) is active but cannot retrain models and model weights are frozen at v2. Use for production requires accepting that security patches and architecture updates will not arrive. Consider Fish Audio S2 Pro or Kokoro TTS as maintained alternatives.

The Data Says

Neural TTS in 2026 has solved the fidelity problem for clean single-speaker synthesis. What remains unsolved is structural: the latency-expressiveness tradeoff, the prosodic flatness of parallel architectures, and the OOV mispronunciation cascade are all consequences of the underlying task formulation — not limitations of individual models. The systems that perform best in production are those architected with these constraints as first-class design inputs, not as edge cases to be addressed after launch.