RFC: MOSS-TTS-Local-v1.5 Serving Optimization

Background

MOSS-TTS-Local-v1.5 uses a Global Transformer + Local Transformer architecture. The Global Transformer is a Qwen3-style backbone that models text, reference-audio codes, and previous-frame audio feedback; the Local Transformer expands each global hidden state into one acoustic frame.

Each acoustic frame contains 12 RVQ codebooks, and codebooks inside the frame are generated sequentially. This means every audio frame requires not only one backbone step, but also a local continue/stop decision and 12 codebook sampling steps.

MOSS-Audio-Tokenizer-v2 handles both reference-audio encoding and waveform decoding, targets 48 kHz stereo output, and supports native incremental decoding. This gives MOSS Local high-quality audio and a natural streaming path, but it also makes the serving path heavier than many 24 kHz mono or lightweight-vocoder TTS models.

The current vLLM-Omni MOSS Local path is functionally usable, but it does not yet fully use batching, GPU-resident state, CUDA Graph, tensor payloads, or codec streaming. The goal is to reduce RTF, restore true low-TTFP streaming, and make high-concurrency behavior observable, recoverable, and tunable.

Goals

• Reduce single-request and concurrent RTF by removing Python hot-path work, CPU syncs, small-kernel launches, and repeated tensor allocations.
• Let Stage 1 consume codec frames before Stage 0 finishes the full utterance, reducing TTFP.
• Preserve offline quality, stop semantics, and voice-clone behavior while adding batched, graphed, and streaming fast paths.
• Keep the optimization path compatible with vLLM-Omni multi-stage serving, benchmark, stats, and the OpenAI-compatible speech API.

Proposed Optimizations

Model Initial Support

• https://github.com/vllm-project/vllm-omni/pull/4664
• https://github.com/vllm-project/vllm-omni/pull/3420

Reference Audio Preprocessing

• Add a content-addressed cache for reference-audio codes to avoid repeatedly encoding the same speaker audio. Cache values should be stored on CPU in a compact dtype and returned as clones to avoid accidental mutation by downstream code.

• Add single-flight for the same uncached reference audio so concurrent requests trigger only one real encoding job. This prevents bursts of voice-clone traffic from overloading the codec encoder with duplicate work.

• Batch reference-audio encoding with a small wait window. The MOSS codec encoder is heavy enough that modest batching can improve GPU utilization, while per-item failures must still be isolated.

AR Decode State and Local Transformer

• Support compile for local transformer https://github.com/vllm-project/vllm-omni/pull/5530
• Support CUDA Graph for local transformer #5197
• Batch local-transformer frame decode across active requests. The 12 codebooks still keep their within-frame sequential dependency, but each codebook step should run over the active batch. #5530
• Reuse local-transformer KV within a frame to avoid re-prefilling the current prefix for every codebook. This must preserve MOSS Local's GPT-J/interleaved RoPE semantics and should be validated against the current eager path for logits and codes.

Streaming Codec and Vocoder

• Use the native MOSS-Audio-Tokenizer-v2 streaming session. Each streaming request should acquire a slot, decode waveform incrementally by chunk, and release the slot on finish, error, or abort. @gcanlin https://github.com/vllm-project/vllm-omni/pull/4804

• Streaming slot exhaustion must not crash the request. The system should queue, fall back to offline decode, or explicitly reject new requests based on configuration, and the behavior must be observable. @gcanlin https://github.com/vllm-project/vllm-omni/pull/4804

• Add coalesced vocoder steps so when one request is due for decoding, other ready or near-ready slots can join the same codec call. All participating slots should use the same frame count plus an execution mask to amortize codec forward cost. https://github.com/vllm-project/vllm-omni/pull/4804

• Capture common streaming vocoder chunk sizes with CUDA Graph. The codec's internal streaming state must be updated in place so graph replay sees stable buffer addresses. https://github.com/vllm-project/vllm-omni/pull/4929

• Make vocoder streaming batch dynamic and catch the dimension of batch #5235

GPU-to-GPU Transfer

• Evaluate CUDA IPC for same-GPU cross-process tensor stream chunks. This should come after delta tensor payloads are implemented, otherwise the benefit for tiny frame rows may be hidden by Python and CPU-path overhead.

• Evaluate NCCL or another D2D relay for cross-GPU Stage0/Stage1 deployment. This should become a default path only when codec chunk tensors are large enough and CPU relay becomes a visible bottleneck.

Validation

Correctness validation should cover offline/streaming audio duration, WER/CER, speaker similarity, chunk continuity, stop frames, max-frame handling, abort cleanup, and slot release. Performance validation should report RTF, TTFP, E2E latency, request throughput, audio throughput, graph hit rate, cache hit rate, and per-stage timing by warm state, concurrency level, and stream/non-stream mode.

CC List

@linyueqian @Sy0307 @hsliuustc0106