[New Model]: Add microsoft/Mage-VL with end-to-end full-duplex multimodal streaming inference

The model to consider.

• Model: https://huggingface.co/microsoft/Mage-VL
• Repository: https://github.com/microsoft/Mage/tree/8c94a0ac905167f40b05b09332b78752b7f9fbef/mage_vl
• Paper: https://arxiv.org/abs/2607.24904

Mage-VL is a 4B codec-native, proactive-streaming multimodal foundation model for image and video understanding. It combines Mage-ViT with a Qwen3-4B causal decoder and a lightweight cognition gate. The same checkpoint supports offline image/video understanding and event-gated causal streaming.

The closest model vllm-omni already supports.

The closest model is AURA, which depends Qwen3-VL and other Qwen-based vision-language models supported by vLLM/vLLM-Omni. Mage-VL also relates to vLLM-Omni's streaming video and full-duplex serving work.

Mage-VL's language backbone is Qwen3-4B-Instruct-2507, but its visual path is different: it uses a custom codec-native Mage-ViT, variable-length visual tokens derived from I/P-frame structure, 3D RoPE, codec-window streaming memory, and a cognition gate for proactive event-triggered responses.

What's your difficulty of supporting the model you want?

Basic offline image/video understanding is conceptually within upstream vLLM's multimodal model-support scope. This RFC intentionally focuses on the capability that makes Mage-VL especially relevant to vLLM-Omni: end-to-end, full-duplex multimodal streaming inference.

The main challenges are:

• Codec-native visual preprocessing: support traditional H.264/HEVC metadata (motion vectors and residual energy) and, where feasible, neural codec inputs such as DCVC-RT rate maps.
• Variable-length visual tokenization: preserve Mage-ViT's I-frame/P-frame token selection and 3D positional information instead of falling back to dense uniform frame sampling.
• Stateful causal streaming: incrementally ingest non-overlapping codec/video segments while retaining bounded recurrent streaming memory and recent codec context across the session.
• Cognition-gate execution: run the lightweight System-1 gate for every rolling window, remain silent for routine content, and invoke the full System-2 VLM only when a response-worthy event is detected.
• Concurrent input and output: continue accepting and encoding live video/image/text updates while a prior response is being generated or streamed, rather than forcing a strict input -> response -> done request boundary.
• Full-duplex session semantics: support persistent sessions, asynchronous events, overlapping perception and generation, client updates or interruption, output cancellation/staleness handling, and session-scoped state cleanup.
• Scheduling and batching: batch ready codec windows and gate evaluations across sessions without destroying per-session temporal state or adding a full re-prefill for every segment.
• Streaming API integration: expose event-gated text responses and session lifecycle through vLLM-Omni's realtime/duplex serving path, aligned where possible with the ongoing full-duplex work in #3745 and #5179.

A useful phased implementation would be:

1. Bring up offline image and frame-sampled video understanding, reusing upstream vLLM support where appropriate.
2. Add codec-native offline video inference and validate parity with the reference implementation.
3. Add stateful causal segment ingestion and cognition-gate outputs.
4. Integrate Mage-VL with vLLM-Omni's persistent realtime/full-duplex session runtime so input ingestion can overlap response generation.
5. Add correctness, latency, long-running-session, interruption, and concurrent-session tests.

The support bar for this issue should therefore not stop at one-shot image/video chat. The target is the model's native proactive-streaming behavior served as an end-to-end multimodal full-duplex experience.

Use case and motivation

Mage-VL enables low-latency continuous visual understanding for live video assistants, sports commentary, surveillance and event monitoring, embodied agents, robotics, accessibility narration, and other applications where the model must observe an ongoing stream and respond only when something meaningful happens.

Although Mage-VL is fundamentally a multimodal understanding model—and its basic model execution may belong in upstream vLLM—the end-to-end product experience requires more than ordinary request/response VLM serving. A client should be able to keep pushing visual segments and optional text queries while the model is reasoning or emitting a prior response. The server must preserve temporal state, evaluate the cognition gate continuously, emit asynchronous event-triggered responses, and handle interruptions or newer observations without tearing down the session.

This makes Mage-VL a strong model-level validation target for vLLM-Omni's full-duplex architecture: unlike speech-only duplex systems, it exercises bidirectional, persistent, stateful serving with continuous visual input and streamed text output. Supporting it would also help ensure that the full-duplex runtime is modality-agnostic and reusable for online video understanding and embodied/world-model workloads.

Expected outcome:

• Preserve standard Mage-VL image/video understanding quality.
• Support codec-native and frame-based inputs where practical.
• Maintain a persistent causal streaming session across video segments.
• Run the cognition gate incrementally and emit responses only for response-worthy events.
• Accept new multimodal input while output generation is in progress.
• Support interruption/cancellation and prevent stale output after newer input arrives.
• Provide offline, online, and end-to-end full-duplex examples plus tests and benchmarks.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.