vllm-project/semantic-router

[CI/Build] Model-gated multimodal tests never execute in CI: MULTIMODAL_MODEL_PATH is set nowhere (Go and Rust)

Closed

#2,319 opened on Jul 3, 2026

 (1 comment) (0 reactions) (0 assignees)Go (699 forks)github user discovery
area/corehelp wantedroadmapsignal-decision-engine

Repository metrics

Stars
 (4,293 stars)
PR merge metrics
 (PR metrics pending)

Description

What

Every multimodal test that needs the real model is gated on MULTIMODAL_MODEL_PATH and skips: on the Go side, TestEmbeddingClassifier_Integration* (pkg/classification/embedding_classifier_integration_test.go:54-56); on the candle-binding side, the Go binding tests (TestMultiModalEncodeImageFromBase64 and siblings in candle-binding/semantic-router_test.go) skip on the same variable, and the Rust unit tests in multimodal_embedding.rs are #[ignore]-gated and read it too. No workflow or makefile sets the variable, so these tests skip on every PR run and on main. #2186 added the CI-safe Rust lib lane (closing #2182) - that lane excludes model-dependent tests by design, so this gap is the complement of that fix, not a regression of it.

Why it matters

During the #2104 review, the FFI image-encode path had zero executed coverage anywhere - verification had to be manual against a local model. Green CI on image-touching PRs currently proves the validation and extraction layers only; an encode-path regression would ship invisibly. The tests exist and would fail for the right reasons - nothing runs them.

Proposal

Options by cost: (a) a nightly job that fetches multi-modal-embed-small via the router's --download-only path (make download-models with a config referencing it) and exports MULTIMODAL_MODEL_PATH; (b) a small fixture checkpoint for PR CI (fastest feedback, requires producing one); (c) at minimum, a documented manual-receipt convention for PRs touching the multimodal FFI. Happy to take this once there's a preference on lane cost.

H2 execution scope

Make multimodal model-gated tests either run in CI with a supported fixture or produce an explicit manual receipt policy. This is part of #2347 release validation.

Relevant code surfaces: Go/Rust multimodal tests, CI workflows, model artifact setup, and repo harness validation docs.

Acceptance criteria

  • CI no longer silently skips all multimodal tests because MULTIMODAL_MODEL_PATH is unset.
  • The project either supplies a lightweight fixture path or documents manual receipt requirements.
  • Skipped multimodal tests report why they are skipped and what validation replaces them.
  • Release validation can prove at least one meaningful multimodal encode path works.

Contributor guide