/v1/realtime has no tool/function-calling support
#5,522 opened on Jul 29, 2026
Repository metrics
- Stars
- (4,990 stars)
- PR merge metrics
- (PR metrics pending)
Description
Environment: vllm-omni 0.24.0, serving Qwen/Qwen3-Omni-30B-A3B-Instruct via vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --no-async-chunk.
What we're trying to do: Use the self-hosted /v1/realtime websocket as a full speech-to-speech "brain" for a LiveKit Agents voice worker, including tool/function calling (the model deciding to call e.g. a weather-lookup or inventory-check tool mid-conversation) — the same pattern OpenAI's and Gemini's realtime APIs support.
What we found: The realtime protocol has no wire mechanism for tools at all, at either the vLLM or vllm-omni layer:
SessionUpdate(vllm/entrypoints/speech_to_text/realtime/protocol.py) only declares amodelfield.RealtimeConnection.handle_event(vllm/entrypoints/speech_to_text/realtime/connection.py) only readsevent.get("model")fromsession.update— nothing else is parsed or threaded into generation.- vllm-omni's
RealtimeConnectionoverride (vllm_omni/entrypoints/openai/realtime_connection.py) only adds audio-delta emission on top of that; it doesn't add tool support either. - There's no server→client event for a function-call request, and no client→server event to submit a tool result.
Impact: A realtime session can never be given tool definitions, so the model has no way to know it has any tools available — it just verbally declines ("I can't provide real-time weather updates...") rather than invoking anything, even though the same checkpoint supports tool-calling fine via /v1/chat/completions with tools=[...].
Request: Add tool/function-calling support to the realtime protocol — e.g. a tools field on SessionUpdate, plus request/response event types for function calls (mirroring OpenAI's response.function_call_arguments.* / conversation.item.create shape, or vLLM's own tool_parsers used for chat completions). Happy to help test against a patch if useful.