[Bug] Qwen3-Omni talker garbles one-word answers (~13-27%/request, seed-deterministic, pre-existing on main); Whisper-small judge inflates test_one_word_prompt_001 failures ~2x

Summary

tests/e2e/online_serving/test_qwen3_omni_expansion.py::test_one_word_prompt_001 fails intermittently in nightly CI (e.g. vllm-omni-rebase build 2861, Omni · Function Test with H100, 10/10 retry attempts failed). We investigated this end-to-end — including 30-sample serial single-request measurements on both dev/vllm-align (vLLM dcfebf93f4) and main (vLLM v0.25.0), audio dumps, spectrogram inspection, and three independent ASR reads of every failed sample. The conclusion: this is not a rebase/branch regression. It is a combination of (a) a real, pre-existing talker weakness on one-word outputs and (b) a test-side ASR measurement problem that roughly doubles the apparent failure rate.

Measurements (30 serial single requests each, 2×L20X, seed=0)

Configuration	Whisper-small verdict (current CI judge)
dev/vllm-align (vLLM dcfebf93f4), --async-chunk	9/30 garbled
dev/vllm-align, sync (--no-async-chunk)	12/30 garbled
main + vLLM v0.25.0, --async-chunk	6/30 garbled

Statistically indistinguishable across branches and across sync/async — the "async_chunk regression" framing does not hold. Failing sample indices are byte-for-byte reproducible across server restarts (seed=0 makes the pipeline deterministic per request order), which also means the test's 10 retries re-roll much less randomness than intended.

Audio-level analysis of the 9 failed samples (rebase, async_chunk)

We saved all 30 WAVs and judged each failure with Whisper-small (CI's judge), Whisper large-v3 (auto and forced-English), and wav2vec2 CTC (English chars + espeak IPA phonemes, which has no language-model prior to "autocorrect" toward London):

sample	whisper-small	large-v3 (EN)	wav2vec2 phonemes	verdict
4	'Longan'	London	l ʌ ŋ d ʌ n	correct (ASR false alarm)
3	'Wondon'	London	w a ŋ d ʌ n	degraded: /l/ onset → [w]
11	'ランダン'	London	m a5 n d a5 n	degraded: [m] onset, off vowels
14	'런덴'	London	l ʌ d e	degraded: first /n/ dropped
15	'One then'	London?	w a5 n t əɜ n	degraded: [w] onset, [t] for /d/
5	'Long din'	Long Din	l ɔ ŋ d eː n	wrong word
8	'One day'	One day	w a5 n t iɛ5	wrong word
21	'Long time'	long time	l ʌ ŋ d a	wrong word
24	'Lender'	lender	l æ n d ɚ	wrong word ("lender", unambiguous)

So the honest per-request rate of genuinely wrong/degraded audio is ~13% (clearly wrong) to ~27% (including degraded articulation) — on main as well as on the rebase branch.

Additional observations:

• Spectrograms of the failed samples show fluent, artifact-free speech (normal levels, formants, syllable structure) — the talker cleanly speaks a phonetically adjacent wrong word. This is a sampling/decoding issue at talker temperature 0.9 on ultra-short utterances, not pipeline/audio corruption.
• The degraded samples systematically drift toward CJK phonology: the phoneme model emits Mandarin-tagged vowels (a5, iɛ5) exactly on the failed samples, matching Whisper's ja/ko/zh language detections on them.
• CI retry arithmetic: at per-request garble p≈0.2, one 5-concurrent-request attempt passes with probability 0.8⁵≈33%, so 10 retries pass ≈98% of nightlies — main "passing" this test is largely retry lottery, and a modest numeric shift (e.g. from a vLLM version bump) plus bad luck produces a 10/10-failure night like build 2861. This test previously went through the same cycle in #3385, which fixed an EOS-boundary issue and raised retries 3→10.

Test-side ASR problem (independent of the model issue)

The audio judge is Whisper small with auto language detection (tests/helpers/media.py, convert_audio_file_to_text(model_size="small")). On 0.6–0.9 s single-word clips this:

1. mishears clean audio ('Wondon', 'Longan' for textbook London — see sample 4), and
2. has unstable language ID — it transcribed correct London audio as 'ロンドン' and '런던', which the raw-string containment check counts as failures even though they are the right word in another script.

Whisper large-v3 with language="en" forced cuts the measured failure rate from 9/30 to 4/30 on the same corpus.

Suggested fixes

1. Test side (cheap, immediate): in the short-text containment path, transcribe with model_size="large-v3" (or ≥ medium.en) and pass language="en" to model.transcribe(...). Optionally vary the request seed across retries so retries actually re-roll the sampling.
2. Model side (the real ~13%): harden the talker for very short thinker outputs — e.g. lower talker temperature when the thinker output is below a length threshold, or revisit the one-word EOS handling from #3385.

Repro

30 serial single requests against vllm_omni/deploy/qwen3_omni_moe.yaml (either --async-chunk or --no-async-chunk), Qwen/Qwen3-Omni-30B-A3B-Instruct, prompt "What is the capital of UK? Answer in one word", checking the transcript with the same containment rule as the test. Reproduces on 2×L20X and matches the H100 CI signature. A self-contained diagnostic that dumps every WAV + transcript is available and can be PR'd if useful.

🤖 Generated with Claude Code