vllm-project/vllm-omni

[Bug]: Qwen3-Omni-30B-A3B-Instruct run with error `AssertionError: DP adjusted local rank 1 is out of bounds for 1 devices.`

Open

#5,003 opened on Jul 10, 2026

View on GitHub
 (9 comments) (0 reactions) (0 assignees)Python (1,067 forks)github user discovery
bughelp wantedmedium priorityomni

Repository metrics

Stars
 (4,990 stars)
PR merge metrics
 (PR metrics pending)

Description

Your current environment

Collecting environment information...
INFO 07-10 09:29:19 [patch.py:252] NVFP4 W4A4 weight_scale NaN-clamp: installed.
==============================
        System Info
==============================
OS                           : Ubuntu 22.04.5 LTS (x86_64)
GCC version                  : (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0
Clang version                : Could not collect
CMake version                : Could not collect
Libc version                 : glibc-2.35

==============================
       PyTorch Info
==============================
PyTorch version              : 2.11.0+cu130
Is debug build               : False
CUDA used to build PyTorch   : 13.0
ROCM used to build PyTorch   : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.13 (main, Mar  4 2026, 09:23:07) [GCC 11.4.0] (64-bit runtime)
Python platform              : Linux-5.15.0-171-generic-x86_64-with-glibc2.35

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 13.0.88
CUDA_MODULE_LOADING set to   : 
GPU models and configuration : 
GPU 0: NVIDIA GeForce RTX 3090
GPU 1: NVIDIA GeForce RTX 3090
GPU 2: NVIDIA GeForce RTX 3090
GPU 3: NVIDIA GeForce RTX 3090

Nvidia driver version        : 590.44.01
cuDNN version                : Could not collect
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                            x86_64
CPU op-mode(s):                          32-bit, 64-bit
Address sizes:                           46 bits physical, 48 bits virtual
Byte Order:                              Little Endian
CPU(s):                                  24
On-line CPU(s) list:                     0-23
Vendor ID:                               GenuineIntel
Model name:                              Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz
CPU family:                              6
Model:                                   63
Thread(s) per core:                      1
Core(s) per socket:                      12
Socket(s):                               2
Stepping:                                2
CPU max MHz:                             3000.0000
CPU min MHz:                             1200.0000
BogoMIPS:                                4799.71
Flags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts md_clear flush_l1d ibpb_exit_to_user
Virtualization:                          VT-x
L1d cache:                               768 KiB (24 instances)
L1i cache:                               768 KiB (24 instances)
L2 cache:                                6 MiB (24 instances)
L3 cache:                                60 MiB (2 instances)
NUMA node(s):                            2
NUMA node0 CPU(s):                       0-11
NUMA node1 CPU(s):                       12-23
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             KVM: Mitigation: VMX disabled
Vulnerability L1tf:                      Mitigation; PTE Inversion; VMX conditional cache flushes, SMT disabled
Vulnerability Mds:                       Mitigation; Clear CPU buffers; SMT disabled
Vulnerability Meltdown:                  Mitigation; PTI
Vulnerability Mmio stale data:           Mitigation; Clear CPU buffers; SMT disabled
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; Retpolines; IBPB conditional; IBRS_FW; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Mitigation; IBPB before exit to userspace

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.6.12
[pip3] numpy==2.2.6
[pip3] nvidia-cublas==13.1.0.3
[pip3] nvidia-cuda-cccl==13.3.3.3.1
[pip3] nvidia-cuda-crt==13.3.33
[pip3] nvidia-cuda-cupti==13.0.85
[pip3] nvidia-cuda-nvcc==13.2.78
[pip3] nvidia-cuda-nvrtc==13.0.88
[pip3] nvidia-cuda-runtime==13.0.96
[pip3] nvidia-cuda-tileiras==13.2.78
[pip3] nvidia-cudnn-cu13==9.19.0.56
[pip3] nvidia-cudnn-frontend==1.25.0
[pip3] nvidia-cufft==12.0.0.61
[pip3] nvidia-cufile==1.15.1.6
[pip3] nvidia-curand==10.4.0.35
[pip3] nvidia-cusolver==12.0.4.66
[pip3] nvidia-cusparse==12.6.3.3
[pip3] nvidia-cusparselt-cu13==0.8.0
[pip3] nvidia-cutlass-dsl==4.5.2
[pip3] nvidia-cutlass-dsl-libs-base==4.5.2
[pip3] nvidia-cutlass-dsl-libs-cu13==4.5.2
[pip3] nvidia-ml-py==13.610.43
[pip3] nvidia-nccl-cu13==2.28.9
[pip3] nvidia-nvjitlink==13.0.88
[pip3] nvidia-nvshmem-cu13==3.4.5
[pip3] nvidia-nvtx==13.0.85
[pip3] nvidia-nvvm==13.2.78
[pip3] onnxruntime==1.27.0
[pip3] pyzmq==27.1.0
[pip3] tokenspeed-triton==3.7.10.post20260531
[pip3] torch==2.11.0+cu130
[pip3] torch_c_dlpack_ext==0.1.5
[pip3] torch-einops-utils==0.1.6
[pip3] torchaudio==2.11.0+cu130
[pip3] torchsde==0.2.6
[pip3] torchvision==0.26.0+cu130
[pip3] transformers==5.12.1
[pip3] triton==3.6.0
[pip3] x-transformers==2.23.3
[conda] Could not collect

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.24.0
vLLM-Omni Version            : 0.24.0
vLLM Build Flags:
  CUDA Archs: 7.5 8.0 8.6 8.9 9.0 10.0 12.0; ROCm: Disabled
GPU Topology:
        GPU0    GPU1    GPU2    GPU3    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      PHB     SYS     SYS     0-11    0               N/A
GPU1    PHB      X      SYS     SYS     0-11    0               N/A
GPU2    SYS     SYS      X      PHB     12-23   1               N/A
GPU3    SYS     SYS     PHB      X      12-23   1               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

==============================
     Environment Variables
==============================
NVIDIA_VISIBLE_DEVICES=all
NVIDIA_REQUIRE_CUDA=cuda>=13.0 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 brand=unknown,driver>=565,driver<566 brand=grid,driver>=565,driver<566 brand=tesla,driver>=565,driver<566 brand=nvidia,driver>=565,driver<566 brand=quadro,driver>=565,driver<566 brand=quadrortx,driver>=565,driver<566 brand=nvidiartx,driver>=565,driver<566 brand=vapps,driver>=565,driver<566 brand=vpc,driver>=565,driver<566 brand=vcs,driver>=565,driver<566 brand=vws,driver>=565,driver<566 brand=cloudgaming,driver>=565,driver<566 brand=unknown,driver>=570,driver<571 brand=grid,driver>=570,driver<571 brand=tesla,driver>=570,driver<571 brand=nvidia,driver>=570,driver<571 brand=quadro,driver>=570,driver<571 brand=quadrortx,driver>=570,driver<571 brand=nvidiartx,driver>=570,driver<571 brand=vapps,driver>=570,driver<571 brand=vpc,driver>=570,driver<571 brand=vcs,driver>=570,driver<571 brand=vws,driver>=570,driver<571 brand=cloudgaming,driver>=570,driver<571 brand=unknown,driver>=575,driver<576 brand=grid,driver>=575,driver<576 brand=tesla,driver>=575,driver<576 brand=nvidia,driver>=575,driver<576 brand=quadro,driver>=575,driver<576 brand=quadrortx,driver>=575,driver<576 brand=nvidiartx,driver>=575,driver<576 brand=vapps,driver>=575,driver<576 brand=vpc,driver>=575,driver<576 brand=vcs,driver>=575,driver<576 brand=vws,driver>=575,driver<576 brand=cloudgaming,driver>=575,driver<576
TORCH_CUDA_ARCH_LIST=7.5 8.0 8.6 8.9 9.0 10.0 12.0
NVIDIA_DRIVER_CAPABILITIES=compute,utility
VLLM_USAGE_SOURCE=production-docker-image
CUDA_VERSION=13.0.2
VLLM_ENABLE_CUDA_COMPATIBILITY=0
VLLM_DISABLE_COMPILE_CACHE=1
LD_LIBRARY_PATH=/usr/local/lib/python3.12/dist-packages/cv2/../../lib64:/usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/cuda/lib64
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root

Your code version

# docker image
vllm/vllm-omni:v0.24.0

🐛 Describe the bug

When serving Qwen3-Omni-30B-A3B-Instruct, encountered the following error:

cli

# run docker container
docker run -itd --name qwen3-omni-30b-a3b-instruct \
    -v /data/model:/data/model \
    -e TZ=Asia/Shanghai \
    -e VLLM_DISABLE_COMPILE_CACHE=1 \
    --ipc=host \
    --network host \
    --shm-size 50G \
    --gpus all \
    --privileged \
    --cap-add=IPC_LOCK \
    --cap-add=SYS_ADMIN \
    --cap-add=SYS_NICE \
    --ulimit memlock=-1 \
    --ulimit stack=67108864 \
    --restart unless-stopped \
    --entrypoint /bin/bash \
    vllm/vllm-omni:v0.24.0


# launch model: cmd 1
vllm serve /data/model/Qwen3-Omni-30B-A3B-Instruct \
  --omni \
  --served-model-name qwen3-omni-30b-a3b-instruct \
  --enable-prefix-caching \
  --enable-prompt-tokens-details \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.9 \
  --trust-remote-code \
  --stage-overrides '{
    "0": {"tensor_parallel_size": 4, "devices": "0,1,2,3", "gpu_memory_utilization": 0.8},
    "1": {"devices": "0", "gpu_memory_utilization": 0.1},
    "2": {"devices": "1", "gpu_memory_utilization": 0.1}
  }' \
  --max-model-len 16384 \
  --host 0.0.0.0 \
  --port 8091

# launch model: cmd 2
vllm serve /data/model/Qwen3-Omni-30B-A3B-Instruct \
  --omni \
  --served-model-name qwen3-omni-30b-a3b-instruct \
  --enable-prefix-caching \
  --enable-prompt-tokens-details \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.9 \
  --trust-remote-code \
  --max-model-len 16384 \
  --host 0.0.0.0 \
  --port 8091

# launch model: cmd 3
vllm serve /data/model/Qwen3-Omni-30B-A3B-Instruct \
  --omni \
  --served-model-name qwen3-omni-30b-a3b-instruct \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.9 \
  --trust-remote-code \
  --stage-overrides '{
    "0": {"tensor_parallel_size": 4, "devices": "0,1,2,3", "gpu_memory_utilization": 0.8},
    "1": {"devices": "0", "gpu_memory_utilization": 0.1},
    "2": {"devices": "1", "gpu_memory_utilization": 0.1}
  }' \
  --max-model-len 16384 \
  --host 0.0.0.0 \
  --port 8091

Error Log

(StageEngineCoreProc_stage0_replica0 pid=635) INFO 07-10 09:16:29 [vllm.py:1006] Asynchronous scheduling is enabled.
(StageEngineCoreProc_stage0_replica0 pid=635) INFO 07-10 09:16:29 [kernel.py:276] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(APIServer pid=546) INFO 07-10 09:16:29 [stage_runtime.py:586] [StageRuntime] Stage 0 engine startup completed
(APIServer pid=546) INFO 07-10 09:16:29 [stage_engine_core_client.py:157] [StageEngineCoreClient] stage-0 [rep-0] initializing EngineCore
(APIServer pid=546) INFO 07-10 09:16:29 [stage_engine_core_client.py:213] [StageEngineCoreClient] stage-0 [rep-0] EngineCore running
(APIServer pid=546) INFO 07-10 09:16:29 [stage_runtime.py:600] [StageRuntime] Stage 0 initialized
INFO 07-10 09:16:39 [patch.py:252] NVFP4 W4A4 weight_scale NaN-clamp: installed.
(StageEngineCoreProc_stage1_replica0 pid=1580) INFO 07-10 09:16:42 [core.py:114] Initializing a V1 LLM engine (v0.24.0) with config: model='/data/model/Qwen3-Omni-30B-A3B-Instruct', speculative_config=None, tokenizer='/data/model/Qwen3-Omni-30B-A3B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=16384, download_dir=None, load_format=auto, tensor_parallel_size=4, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_verbose=False), seed=0, served_model_name=qwen3-omni-30b-a3b-instruct, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [32768], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 128, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto')
(StageEngineCoreProc_stage1_replica0 pid=1580) WARNING 07-10 09:16:42 [multiproc_executor.py:1063] Reducing Torch parallelism from 24 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
(StageEngineCoreProc_stage1_replica0 pid=1580) INFO 07-10 09:16:42 [multiproc_executor.py:140] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=10.10.249.75 (local), world_size=4, local_world_size=4
INFO 07-10 09:16:52 [patch.py:252] NVFP4 W4A4 weight_scale NaN-clamp: installed.
INFO 07-10 09:16:53 [patch.py:252] NVFP4 W4A4 weight_scale NaN-clamp: installed.
INFO 07-10 09:16:53 [patch.py:252] NVFP4 W4A4 weight_scale NaN-clamp: installed.
INFO 07-10 09:16:53 [patch.py:252] NVFP4 W4A4 weight_scale NaN-clamp: installed.
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'interleaved'}
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'interleaved'}
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'interleaved'}
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'interleaved'}
(Worker pid=1673) INFO 07-10 09:16:56 [parallel_state.py:1588] world_size=4 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:35103 backend=nccl
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'interleaved'}
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'interleaved', 'mrope_section'}
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_section', 'interleaved'}
[transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'interleaved', 'mrope_section'}
(Worker pid=1676) ERROR 07-10 09:16:57 [multiproc_executor.py:898] WorkerProc failed to start.
(Worker pid=1676) ERROR 07-10 09:16:57 [multiproc_executor.py:898] Traceback (most recent call last):
(Worker pid=1676) ERROR 07-10 09:16:57 [multiproc_executor.py:898]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 865, in worker_main
(Worker pid=1676) ERROR 07-10 09:16:57 [multiproc_executor.py:898]     worker = WorkerProc(*args, **kwargs)
(Worker pid=1676) ERROR 07-10 09:16:57 [multiproc_executor.py:898]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1676) ERROR 07-10 09:16:57 [multiproc_executor.py:898]   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker pid=1676) ERROR 07-10 09:16:57 [multiproc_executor.py:898]     return func(*args, **kwargs)
(Worker pid=1676) ERROR 07-10 09:16:57 [multiproc_executor.py:898]            ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1676) ERROR 07-10 09:16:57 [multiproc_executor.py:898]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 626, in __init__
(Worker pid=1676) ERROR 07-10 09:16:57 [multiproc_executor.py:898]     self.worker.init_device()
(Worker pid=1676) ERROR 07-10 09:16:57 [multiproc_executor.py:898]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/worker_base.py", line 331, in init_device
(Worker pid=1676) ERROR 07-10 09:16:57 [multiproc_executor.py:898]     self.worker.init_device()  # type: ignore
(Worker pid=1676) ERROR 07-10 09:16:57 [multiproc_executor.py:898]     ^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1676) ERROR 07-10 09:16:57 [multiproc_executor.py:898]   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker pid=1676) ERROR 07-10 09:16:57 [multiproc_executor.py:898]     return func(*args, **kwargs)
(Worker pid=1676) ERROR 07-10 09:16:57 [multiproc_executor.py:898]            ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1676) ERROR 07-10 09:16:57 [multiproc_executor.py:898]   File "/usr/local/lib/python3.12/dist-packages/vllm_omni/worker/gpu_ar_worker.py", line 72, in init_device
(Worker pid=1676) ERROR 07-10 09:16:57 [multiproc_executor.py:898]     assert self.local_rank < torch.accelerator.device_count(), (
(Worker pid=1676) ERROR 07-10 09:16:57 [multiproc_executor.py:898]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1676) ERROR 07-10 09:16:57 [multiproc_executor.py:898] AssertionError: DP adjusted local rank 3 is out of bounds for 1 devices.
(StageEngineCoreProc_stage1_replica0 pid=1580) INFO 07-10 09:16:57 [multiproc_executor.py:426] [shutdown] Executor: waiting for worker exit count=4
(Worker pid=1675) ERROR 07-10 09:16:57 [multiproc_executor.py:898] WorkerProc failed to start.
(Worker pid=1675) ERROR 07-10 09:16:57 [multiproc_executor.py:898] Traceback (most recent call last):
(Worker pid=1675) ERROR 07-10 09:16:57 [multiproc_executor.py:898]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 865, in worker_main
(Worker pid=1675) ERROR 07-10 09:16:57 [multiproc_executor.py:898]     worker = WorkerProc(*args, **kwargs)
(Worker pid=1675) ERROR 07-10 09:16:57 [multiproc_executor.py:898]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1675) ERROR 07-10 09:16:57 [multiproc_executor.py:898]   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker pid=1675) ERROR 07-10 09:16:57 [multiproc_executor.py:898]     return func(*args, **kwargs)
(Worker pid=1675) ERROR 07-10 09:16:57 [multiproc_executor.py:898]            ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1675) ERROR 07-10 09:16:57 [multiproc_executor.py:898]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 626, in __init__
(Worker pid=1675) ERROR 07-10 09:16:57 [multiproc_executor.py:898]     self.worker.init_device()
(Worker pid=1675) ERROR 07-10 09:16:57 [multiproc_executor.py:898]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/worker_base.py", line 331, in init_device
(Worker pid=1675) ERROR 07-10 09:16:57 [multiproc_executor.py:898]     self.worker.init_device()  # type: ignore
(Worker pid=1675) ERROR 07-10 09:16:57 [multiproc_executor.py:898]     ^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1675) ERROR 07-10 09:16:57 [multiproc_executor.py:898]   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker pid=1675) ERROR 07-10 09:16:57 [multiproc_executor.py:898]     return func(*args, **kwargs)
(Worker pid=1675) ERROR 07-10 09:16:57 [multiproc_executor.py:898]            ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1675) ERROR 07-10 09:16:57 [multiproc_executor.py:898]   File "/usr/local/lib/python3.12/dist-packages/vllm_omni/worker/gpu_ar_worker.py", line 72, in init_device
(Worker pid=1675) ERROR 07-10 09:16:57 [multiproc_executor.py:898]     assert self.local_rank < torch.accelerator.device_count(), (
(Worker pid=1675) ERROR 07-10 09:16:57 [multiproc_executor.py:898]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1675) ERROR 07-10 09:16:57 [multiproc_executor.py:898] AssertionError: DP adjusted local rank 2 is out of bounds for 1 devices.
(Worker pid=1674) ERROR 07-10 09:16:57 [multiproc_executor.py:898] WorkerProc failed to start.
(Worker pid=1674) ERROR 07-10 09:16:57 [multiproc_executor.py:898] Traceback (most recent call last):
(Worker pid=1674) ERROR 07-10 09:16:57 [multiproc_executor.py:898]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 865, in worker_main
(Worker pid=1674) ERROR 07-10 09:16:57 [multiproc_executor.py:898]     worker = WorkerProc(*args, **kwargs)
(Worker pid=1674) ERROR 07-10 09:16:57 [multiproc_executor.py:898]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1674) ERROR 07-10 09:16:57 [multiproc_executor.py:898]   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker pid=1674) ERROR 07-10 09:16:57 [multiproc_executor.py:898]     return func(*args, **kwargs)
(Worker pid=1674) ERROR 07-10 09:16:57 [multiproc_executor.py:898]            ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1674) ERROR 07-10 09:16:57 [multiproc_executor.py:898]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 626, in __init__
(Worker pid=1674) ERROR 07-10 09:16:57 [multiproc_executor.py:898]     self.worker.init_device()
(Worker pid=1674) ERROR 07-10 09:16:57 [multiproc_executor.py:898]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/worker_base.py", line 331, in init_device
(Worker pid=1674) ERROR 07-10 09:16:57 [multiproc_executor.py:898]     self.worker.init_device()  # type: ignore
(Worker pid=1674) ERROR 07-10 09:16:57 [multiproc_executor.py:898]     ^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1674) ERROR 07-10 09:16:57 [multiproc_executor.py:898]   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker pid=1674) ERROR 07-10 09:16:57 [multiproc_executor.py:898]     return func(*args, **kwargs)
(Worker pid=1674) ERROR 07-10 09:16:57 [multiproc_executor.py:898]            ^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1674) ERROR 07-10 09:16:57 [multiproc_executor.py:898]   File "/usr/local/lib/python3.12/dist-packages/vllm_omni/worker/gpu_ar_worker.py", line 72, in init_device
(Worker pid=1674) ERROR 07-10 09:16:57 [multiproc_executor.py:898]     assert self.local_rank < torch.accelerator.device_count(), (
(Worker pid=1674) ERROR 07-10 09:16:57 [multiproc_executor.py:898]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker pid=1674) ERROR 07-10 09:16:57 [multiproc_executor.py:898] AssertionError: DP adjusted local rank 1 is out of bounds for 1 devices.
(StageEngineCoreProc_stage1_replica0 pid=1580) WARNING 07-10 09:17:02 [multiproc_executor.py:438] [shutdown] Executor: workers still running after grace period; sending SIGTERM count=1
(StageEngineCoreProc_stage1_replica0 pid=1580) WARNING 07-10 09:17:06 [multiproc_executor.py:448] [shutdown] Executor: workers still running after SIGTERM; sending SIGKILL count=1
(StageEngineCoreProc_stage1_replica0 pid=1580) ERROR 07-10 09:17:06 [stage_engine_core_proc.py:176] StageEngineCoreProc failed to start.
(StageEngineCoreProc_stage1_replica0 pid=1580) ERROR 07-10 09:17:06 [stage_engine_core_proc.py:176] Traceback (most recent call last):
(StageEngineCoreProc_stage1_replica0 pid=1580) ERROR 07-10 09:17:06 [stage_engine_core_proc.py:176]   File "/usr/local/lib/python3.12/dist-packages/vllm_omni/engine/stage_engine_core_proc.py", line 127, in run_stage_core
(StageEngineCoreProc_stage1_replica0 pid=1580) ERROR 07-10 09:17:06 [stage_engine_core_proc.py:176]     engine_core = StageEngineCoreProc(
(StageEngineCoreProc_stage1_replica0 pid=1580) ERROR 07-10 09:17:06 [stage_engine_core_proc.py:176]                   ^^^^^^^^^^^^^^^^^^^^
(StageEngineCoreProc_stage1_replica0 pid=1580) ERROR 07-10 09:17:06 [stage_engine_core_proc.py:176]   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(StageEngineCoreProc_stage1_replica0 pid=1580) ERROR 07-10 09:17:06 [stage_engine_core_proc.py:176]     return func(*args, **kwargs)
(StageEngineCoreProc_stage1_replica0 pid=1580) ERROR 07-10 09:17:06 [stage_engine_core_proc.py:176]            ^^^^^^^^^^^^^^^^^^^^^
(StageEngineCoreProc_stage1_replica0 pid=1580) ERROR 07-10 09:17:06 [stage_engine_core_proc.py:176]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 966, in __init__
(StageEngineCoreProc_stage1_replica0 pid=1580) ERROR 07-10 09:17:06 [stage_engine_core_proc.py:176]     super().__init__(
(StageEngineCoreProc_stage1_replica0 pid=1580) ERROR 07-10 09:17:06 [stage_engine_core_proc.py:176]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 123, in __init__
(StageEngineCoreProc_stage1_replica0 pid=1580) ERROR 07-10 09:17:06 [stage_engine_core_proc.py:176]     self.model_executor = executor_class(vllm_config)
(StageEngineCoreProc_stage1_replica0 pid=1580) ERROR 07-10 09:17:06 [stage_engine_core_proc.py:176]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(StageEngineCoreProc_stage1_replica0 pid=1580) ERROR 07-10 09:17:06 [stage_engine_core_proc.py:176]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 108, in __init__
(StageEngineCoreProc_stage1_replica0 pid=1580) ERROR 07-10 09:17:06 [stage_engine_core_proc.py:176]     super().__init__(vllm_config)
(StageEngineCoreProc_stage1_replica0 pid=1580) ERROR 07-10 09:17:06 [stage_engine_core_proc.py:176]   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(StageEngineCoreProc_stage1_replica0 pid=1580) ERROR 07-10 09:17:06 [stage_engine_core_proc.py:176]     return func(*args, **kwargs)
(StageEngineCoreProc_stage1_replica0 pid=1580) ERROR 07-10 09:17:06 [stage_engine_core_proc.py:176]            ^^^^^^^^^^^^^^^^^^^^^
(StageEngineCoreProc_stage1_replica0 pid=1580) ERROR 07-10 09:17:06 [stage_engine_core_proc.py:176]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 109, in __init__
(StageEngineCoreProc_stage1_replica0 pid=1580) ERROR 07-10 09:17:06 [stage_engine_core_proc.py:176]     self._init_executor()
(StageEngineCoreProc_stage1_replica0 pid=1580) ERROR 07-10 09:17:06 [stage_engine_core_proc.py:176]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 201, in _init_executor
(StageEngineCoreProc_stage1_replica0 pid=1580) ERROR 07-10 09:17:06 [stage_engine_core_proc.py:176]     self.workers = WorkerProc.wait_for_ready(unready_workers)
(StageEngineCoreProc_stage1_replica0 pid=1580) ERROR 07-10 09:17:06 [stage_engine_core_proc.py:176]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(StageEngineCoreProc_stage1_replica0 pid=1580) ERROR 07-10 09:17:06 [stage_engine_core_proc.py:176]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 762, in wait_for_ready
(StageEngineCoreProc_stage1_replica0 pid=1580) ERROR 07-10 09:17:06 [stage_engine_core_proc.py:176]     raise e from None
(StageEngineCoreProc_stage1_replica0 pid=1580) ERROR 07-10 09:17:06 [stage_engine_core_proc.py:176] Exception: WorkerProc initialization failed due to an exception in a background process. See stack trace for root cause.
(StageEngineCoreProc_stage1_replica0 pid=1580) Process StageEngineCoreProc_stage1_replica0:
(StageEngineCoreProc_stage1_replica0 pid=1580) Traceback (most recent call last):
(StageEngineCoreProc_stage1_replica0 pid=1580)   File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(StageEngineCoreProc_stage1_replica0 pid=1580)     self.run()
(StageEngineCoreProc_stage1_replica0 pid=1580)   File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
(StageEngineCoreProc_stage1_replica0 pid=1580)     self._target(*self._args, **self._kwargs)
(StageEngineCoreProc_stage1_replica0 pid=1580)   File "/usr/local/lib/python3.12/dist-packages/vllm_omni/engine/stage_engine_core_proc.py", line 127, in run_stage_core
(StageEngineCoreProc_stage1_replica0 pid=1580)     engine_core = StageEngineCoreProc(
(StageEngineCoreProc_stage1_replica0 pid=1580)                   ^^^^^^^^^^^^^^^^^^^^
(StageEngineCoreProc_stage1_replica0 pid=1580)   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(StageEngineCoreProc_stage1_replica0 pid=1580)     return func(*args, **kwargs)
(StageEngineCoreProc_stage1_replica0 pid=1580)            ^^^^^^^^^^^^^^^^^^^^^
(StageEngineCoreProc_stage1_replica0 pid=1580)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 966, in __init__
(StageEngineCoreProc_stage1_replica0 pid=1580)     super().__init__(
(StageEngineCoreProc_stage1_replica0 pid=1580)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 123, in __init__
(StageEngineCoreProc_stage1_replica0 pid=1580)     self.model_executor = executor_class(vllm_config)
(StageEngineCoreProc_stage1_replica0 pid=1580)                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(StageEngineCoreProc_stage1_replica0 pid=1580)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 108, in __init__
(StageEngineCoreProc_stage1_replica0 pid=1580)     super().__init__(vllm_config)
(StageEngineCoreProc_stage1_replica0 pid=1580)   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(StageEngineCoreProc_stage1_replica0 pid=1580)     return func(*args, **kwargs)
(StageEngineCoreProc_stage1_replica0 pid=1580)            ^^^^^^^^^^^^^^^^^^^^^
(StageEngineCoreProc_stage1_replica0 pid=1580)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 109, in __init__
(StageEngineCoreProc_stage1_replica0 pid=1580)     self._init_executor()
(StageEngineCoreProc_stage1_replica0 pid=1580)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 201, in _init_executor
(StageEngineCoreProc_stage1_replica0 pid=1580)     self.workers = WorkerProc.wait_for_ready(unready_workers)
(StageEngineCoreProc_stage1_replica0 pid=1580)                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(StageEngineCoreProc_stage1_replica0 pid=1580)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 762, in wait_for_ready
(StageEngineCoreProc_stage1_replica0 pid=1580)     raise e from None
(StageEngineCoreProc_stage1_replica0 pid=1580) Exception: WorkerProc initialization failed due to an exception in a background process. See stack trace for root cause.
INFO 07-10 09:17:18 [patch.py:252] NVFP4 W4A4 weight_scale NaN-clamp: installed.

Before submitting a new issue...

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

Contributor guide

Research direction
Investigate the mismatch between DP (data parallel) ranks and the number of devices specified in stage overrides. Check how stage0 uses 4 devices but stage1 and stage2 each use 1 device, causing DP local rank 1 to exceed bounds. Look at vLLM's pipeline parallelism and device assignment logic in the multi stage configuration.
Tech stack
pythonpytorch
Domain
backendaidistributed systems
Issue type
Bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Needs investigation
Prerequisites
PythonPyTorchvLLM
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
70