[RFC]: CI improvement plan — E2E layering, Acc, pinned deps, job matrix, pre-commit, CodeQL

Description

Amendment: adds pre-commit hardening (vLLM-aligned) and CodeQL / GitHub Code Scanning as CI quality tracks (Design §9–§10, PRs 11–12). Existing E2E / Acc / pin / coverage plan unchanged.

Motivation.

Issue	Current State	Target State
Split L2 tiny framework	#4197 partially landed under tests/model_tests/diffusion/ (DIFFUSION_TEST_SETTINGS, builders, offline/online/alignment), while most L2/L3/L4 e2e still lives as one-off modules under tests/e2e/{online_serving,offline_inference}/ (~100+ test_*.py)	One layered layout under tests/e2e/models/; shared launch / request / assert logic lives in tests/helpers/
Boilerplate explosion	Each model adds a near-copy of OmniServerParams / OmniRunner param lists, prompt builders, and send_*_request + assert glue (e.g. test_*_expansion.py)	L3: one online runner + one offline runner; cases are JSON. L2: one registry entry (DiffusionModelTestOpts / future Omni/TTS opts)
Unclear CI level ownership	L2 tiny smoke vs L3/L4 full-weight e2e are mixed by filename convention (test_foo.py vs test_foo_expansion.py) and markers, but not by directory	Directory + marker contract: tiny_model/ → L2; online_serving/ + offline_inference/ JSON → L3–L4 smoke
Helper drift	Assertions / runtime already centralize in tests/helpers/{runtime,assertions,mark}.py, but model-specific builders, step expansion, and case→fixture mapping still scatter in test modules	Canonical helpers for builders, runner/server param expansion, run_level-aware validation
Acc suites fragmented	tests/e2e/accuracy/ (and some benchmarks/accuracy/) use one-off scripts (pixel/CLIP/custom similarity, custom launches); hard to compare Nightly results across models	Unify on lmms-eval as the Acc harness: shared launch + task/metric config; repo keeps adapters, case JSON, and thresholds
Manual CI YAML churn for every model	Each new/retargeted E2E job requires editing test-ready.yml / test-merge.yml / test-nightly.yml (commands, mirror_hardwares, source_file_dependencies) even when the test itself is only a registry/JSON line	PR diff + case/registry metadata drive which Ready/Merge/Nightly E2E jobs upload; common path needs no hand-edited leaf step (open design — see Design §6)
No per-model coverage visibility	CI already collects repo-wide --cov=vllm_omni on some Ready/Merge CPU jobs, but there is no breakdown by model or entry mode; online serving vs offline inference (and different sampling params) exercise very different code paths with no visibility	Collect and dashboard per-model coverage from L2/L3 online + offline cases (optionally Acc); make online-only / offline-only gaps visible when adding or retargeting models
Floating CI deps hard to bisect	dev/CI image deps often use open lower bounds (>=); a Dockerfile/DEP_FILES cache bust silently resolves newer wheels (e.g. transfer engines, eval harnesses) and breaks Nightly/Ready with failures that look like product regressions	Align with vLLM: pin CI/test (and critical dev) deps to exact versions (== / locked requirements); treat upgrades as explicit, reviewable PRs
Thin / drifted pre-commit vs vLLM	Omni has ruff + basic hooks + typos/actionlint, but is missing several vLLM gates (markdownlint, shellcheck, SPDX, systematic forbidden-import / torch.cuda checks, wired mypy). Mergify tips and [tool.mypy] can disagree with .pre-commit-config.yaml; typos pin historically stale; suggestion hook order can be wrong	Align pre-commit with vLLM where it fits Omni (incl. default_stages + manual mypy); fix docs/CI drift; keep Omni-only hooks (e.g. test-mark check) documented
No CodeQL / Code Scanning	No CodeQL (or equivalent semantic SAST) workflow in-repo; security/defect analysis beyond lint is absent on PR/main	Enable GitHub CodeQL for Python (PR + main + scheduled); triage alerts; optionally require clean/new alerts after backlog is manageable

Relation to prior RFCs

• Extends / operationalizes #4197 (tiny diffusion registry) into a full E2E layering story for model smoke (L2 tiny + JSON-driven L3/L4 serving/offline).

• Orthogonal to #1807 (deploy YAML sole source of truth): this RFC owns test case shape and runners; stage/deploy defaults still follow #1807 (CLI overrides, no new CI overlay YAMLs).

• Accuracy track (this amendment): Acc is a parallel L3/L4 precision rail on the same tests/e2e/models/ layering. Smoke (tiny/JSON) and Acc share launch helpers and markers/run_level, but Acc asserts lmms-eval metrics instead of structural assert_response gates. Acc design deliberately references vLLM’s lm-eval accuracy evaluation pattern (harness + pinned eval deps + CI gates); Omni uses lmms-eval for multimodal coverage. Acc does not replace L2 tiny or JSON L3/L4 smoke.

• Per-model coverage track (this amendment): Keep repo-wide unit coverage, and add per-model coverage artifacts from model E2E (online vs offline, and distinct sampling profiles where useful). Goal is visibility into which model entry paths are actually exercised, not a hard gate on every PR in v1. Prompted by discussion on this RFC (https://github.com/vllm-project/vllm-omni/issues/5332#issuecomment-5091716066).

• Pinned-deps track (this amendment): Follow vLLM’s practice of exact version pins for CI/test stacks (see vLLM requirements/ / locked test files) so dependency bumps are intentional PRs, not silent resolver drift after image rebuilds.

• Pre-commit track (this amendment): Close the gap vs upstream vLLM’s hook set (mypy manual, markdownlint, shellcheck, SPDX, forbidden-import, broader torch.cuda policy), fix Mergify/[tool.mypy] drift, refresh typos, and keep suggestion last. Pre-commit remains the fast local + CI lint gate.

• CodeQL track (this amendment): Add GitHub CodeQL / Code Scanning for semantic security and defect findings that pre-commit cannot cover. Complementary to pre-commit — not a replacement (see Design §9–§10).

Problem statement (today)

1. Adding a diffusion pipeline to CI often means a new Python file with duplicated server/runner setup, even when the only novelty is model id + a few extra_body fields.

2. Tiny-weight smoke (tests/model_tests/diffusion) and full-weight e2e (tests/e2e/...) do not share the same “case → launch → request → assert” vocabulary, so contributors copy whichever neighbor file looks closest.

3. L2 / L3 / L4 ownership is encoded mainly in filenames and markers, which makes it hard to discover the intended entry point when adding a new model.

4. Accuracy gates for new models often reinvent test_*_accuracy.py (custom metrics/prompts/seeds), so Nightly Acc results are not comparable and do not share the smoke "case → launch → request → assert" vocabulary.

5. Contributors want lmms-eval as the community multimodal eval entry point, separating what to score from how to serve.

6. Diff-aware CI today only filters hand-written E2E steps (source_file_dependencies). Contributors still repeatedly edit test-*.yml when adding models; there is no first-class way to derive Ready/Merge/Nightly jobs from tiny/JSON/Acc case metadata.

7. Floating CI/dev dependencies make post-upgrade CI failures hard to attribute (product vs transitive wheel bump). Upstream vLLM pins test/CUDA requirements; vLLM-Omni should adopt the same discipline so Nightly/Ready are reproducible.

8. Repo-level coverage exists on some Ready/Merge CPU jobs, but there is no per-model breakdown. Online serving, offline inference, and different sampling parameters can cover very different code paths; today we cannot see that gap in CI dashboards.

9. Pre-commit is thinner than vLLM and partially inconsistent with contributor-facing docs (Mergify / mypy). Missing hooks leave style, license headers, shell/docs lint, and policy checks to human review.

10. There is no in-repo CodeQL (or similar SAST) pipeline; deep static security analysis is missing from the PR/main quality bar that vLLM already has via GitHub code scanning.

Proposed Change.

1. Introduce tests/e2e/models/ as the canonical E2E model-test tree with four buckets:

   • tiny_model/ — L2 tiny-weight smoke (registry-driven).

   • online_serving/ — L3–L4 HTTP serving (JSON cases + one template test).

   • offline_inference/ — L3–L4 in-process OmniRunner (JSON cases + one template test).

   • accuracy/ — L3–L4 precision rail (JSON/YAML cases + lmms-eval runner template).

2. Split responsibilities cleanly:

   • tests/helpers/ — cross-cutting primitives already used repo-wide: runtime.py, assertions.py (+ assert_response facade), mark.py, and tiny_models.py (tiny builders / builder registry only).

   • tests/e2e/models/tiny_model/helper.py — L2-only glue: settings → pytest params, steps_for_case, determinism helpers, alignment helpers.

   • tests/e2e/models/case_io.py — shared JSON load + case_to_server_params / case_to_runner_params + mark materialization for online/offline templates (avoids duplicating loaders in both packages).

3. L2 tiny_model: thin test modules; registry stays data-only:

   • cases.py: type aliases + DIFFUSION_TEST_SETTINGS (and later Omni/TTS registries).

   • helper.py: expand / steps / L2 assert wrappers (not builders).

   • test_offline.py / test_online.py / test_alignment.py: parametrize + loop steps; no model-specific launch/assert logic inline.

   • Adding a pipeline = one DiffusionModelTestOpts(...) block (+ builder in tests/helpers/tiny_models.py if new arch).

4. L3 (and simple L4 smoke): JSON-defined cases + unified runners:

   • tests/e2e/models/online_serving/test_online.py reads cases/*.json.

   • tests/e2e/models/offline_inference/test_offline.py reads cases/*.json.

   • Schema covers: name, mark, server/runner, input, output.

5. Acc via lmms-eval: add a unified accuracy template that launches via the same OmniServer / OmniRunner path as JSON L3, then runs lmms-eval tasks and gates on metric thresholds; migrate tests/e2e/accuracy/* incrementally (pixel/CLIP-only suites may stay as documented escape hatches).

6. Migrate incrementally; do not big-bang delete tests/e2e/online_serving/*_expansion.py, tests/model_tests/diffusion/, or legacy Acc modules until CI green and docs updated.

7. CI auto job selection (open design): extend the upload path so Ready/Merge/Nightly E2E jobs are selected (and ideally generated) from tiny/JSON/Acc metadata + PR diff, instead of requiring developers to keep hand-maintaining leaf steps in test-*.yml. Spike first; do not lock A–D in this RFC.

8. Pin CI/dev dependencies (vLLM-aligned): replace open lower bounds with exact pins for packages that affect CI reproducibility (eval harnesses, connectors, CUDA-adjacent wheels, etc.); document upgrade procedure (bump pin + rebuild image + targeted retest). Builds on / supersedes ad-hoc pins such as those in recent CI dep PRs.

9. Per-model code coverage: collect coverage scoped by model id and entry mode (online vs offline; optional sampling-profile tags); publish artifacts to a dashboard; use the signal to find untested entry paths when adding models. Complements repo-wide --cov=vllm_omni on unit/core_model suites; does not replace them.

10. Harden pre-commit (vLLM-aligned): add missing hooks (mypy as manual, markdownlint-cli2, shellcheck, SPDX, forbidden-import, systematic torch.cuda ban with allowlists); upgrade typos; fix hook order + default_stages; sync Mergify tip and CI --hook-stage manual with reality.

11. Enable CodeQL: add a Python CodeQL workflow (PR + main + schedule); publish to GitHub Code Scanning; start non-blocking if needed, then tighten after triage (Design §10).

Design

0) Directory Layout

Target (canonical) — revised from the planned tree:

tests/
├── helpers/                          # repo-wide primitives
│   ├── runtime.py                    # OmniServer / OmniRunner / clients (existing)
│   ├── assertions.py                 # modality asserts (existing) + assert_response facade
│   ├── mark.py                       # hardware_marks (existing)
│   └── tiny_models.py                # ★ tiny builders (+ optional DiffusionAccs/Opts types)
│
└── e2e/
    └── models/
        ├── helper.py                # ★ shared: load JSON, case→server/runner params, marks
        ├── tiny_model/               # L2
        │   ├── cases.py              # DIFFUSION_TEST_SETTINGS only
        │   ├── helper.py             # L2 expand / steps_for_case / determinism glue
        │   ├── test_offline.py
        │   ├── test_online.py
        ├── online_serving/           # L3–L4 HTTP
        │   ├── test_online.py        # single template
        │   └── cases/
        │       ├── moss_tts.json
        │       ├── qwen_image.json
        │       └── qwen3_omni.json
        └── offline_inference/        # L3–L4 OmniRunner
            ├── test_offline.py       # single template
            └── cases/
                ├── moss_voice.json
                ├── cosyvoice3.json
                └── ming_tts.json
        └── accuracy/                 # L3–L4 Acc (lmms-eval)
            ├── helper.py             # lmms-eval invoke + threshold gates
            ├── test_accuracy.py      # single template: case → launch → eval → gate
            └── cases/
                ├── qwen3_omni_<task>.json
                └── qwen3_tts_<task>.json

Why these tweaks vs the draft tree

Draft	Revision	Reason
Only helpers/{runtime,assertions,mark,tiny_models}	Keep that; do not put JSON loaders in helpers/	JSON case IO is e2e-models-specific; keep helpers/ for reusable primitives
tiny_model/helper.py for “all helpers”	helper.py = L2-only expand/steps; builders stay in helpers/tiny_models.py	Matches “builders in helpers / case logic next to tests”
No alignment module	Keep test_alignment.py under tiny_model/	Preserves #4197 fail-closed check for missing tiny builders
Online/offline each invent case loading	Add e2e/models/case_io.py	One schema parser + mark materializer for both templates
Name tiny_models.py next to package tiny_model/	Keep names as planned	File = builders module; dir = L2 test package — distinct enough

Deprecated / migrate-from (repo snapshot; confirm in PR)

Current	Fate
tests/model_tests/diffusion/diff_model_builders.py (+ related types)	→ tests/helpers/tiny_models.py (builders/types); expand/filtering → e2e/models/tiny_model/helper.py
tests/model_tests/diffusion/test_common_*.py, test_alignment.py	→ e2e/models/tiny_model/test_{offline,online,alignment}.py
tests/model_tests/diffusion/model_settings.py	→ e2e/models/tiny_model/cases.py
tests/e2e/online_serving/test_*.py (+ *_expansion.py)	Convert happy-path smoke to JSON + delete shims once covered; keep exceptional Python only when JSON cannot express control flow
tests/e2e/offline_inference/test_*.py	Same as online
tests/e2e/accuracy/ (+ bits of benchmarks/accuracy/)	Happy-path / task Acc → e2e/models/accuracy/ + lmms-eval; pixel/video similarity escape hatches may remain under _legacy/ with a TODO
tests/e2e/features/	Out of scope (feature/runtime tests, not model matrix)

1) Migration Scope

Based on current tree (approximate; re-glob in implementing PR):

• L2 already present: tests/model_tests/diffusion/{model_settings,config_types,diff_model_builders,case_filtering,task_runners,test_common_*,test_alignment}.py; CI: .buildkite/cuda/test-ready.yml + test-nightly.yml collect tests/model_tests/diffusion.

• L3/L4 Python e2e: tests/e2e/online_serving/ (~67 modules), tests/e2e/offline_inference/ (~50 modules) — migrate by modality / priority, not all at once.

• Shared helpers already reusable: tests/helpers/runtime.py (OmniServer, OmniRunner, OpenAIClientHandler, send_*_request), tests/helpers/assertions.py, tests/helpers/mark.py, run_level fixture in tests/helpers/fixtures/run_args.py.

2) Layer contracts

L2 — tiny_model (registry + package-local helper)

• Weights: tiny / random via builders in tests/helpers/tiny_models.py; run_level=core_model selects tiny path (existing tiny_model_paths pattern).

• Data: DIFFUSION_TEST_SETTINGS: dict[str, DiffusionModelTestOpts] in cases.py only.

• Logic: tiny_model/helper.py owns get_offline_runner_params / online equivalents, steps_for_case, determinism helpers; tests call assert_response from tests/helpers/assertions.py.

• Tests: one offline / one online template, subtests per step; test_alignment.py for registry completeness.

Illustrative offline template:

@pytest.mark.parametrize(
    "omni_runner,case",
    get_offline_runner_params(DIFFUSION_TEST_SETTINGS),
    indirect=["omni_runner"],
)
def test_tiny_diffusion_offline(omni_runner, case, run_level, subtests):
    for step in steps_for_case(case, mode="offline"):
        with subtests.test(step.name):
            if step.kind == "determinism":
                assert_diffusion_determinism_offline(
                    omni_runner, step.request, output=step.output, run_level=run_level
                )
            else:
                raw = send_generate_request(
                    omni_runner, mode="offline", request=step.request, api="diffusion"
                )
                resp = process_output(raw, modality=step.output["modality"], transport=step.request)
                assert_response(resp, step.request, output=step.output, run_level=run_level)

Registry entry (unchanged data shape from #4197 / current model_settings.py):

DIFFUSION_TEST_SETTINGS: dict[str, DiffusionModelTestOpts] = {
    "Flux2KleinPipeline": DiffusionModelTestOpts(
        model="black-forest-labs/FLUX.2-klein-4B",
        builder=tiny_flux2_klein_builder,
        supported_tasks=[DiffusionTasks.TEXT_TO_IMAGE, DiffusionTasks.IMAGE_TO_IMAGE],
        extra_test_groups=[
            [DiffusionAccs.HSDP, DiffusionAccs.TEA_CACHE],
            [DiffusionAccs.SEQUENCE_PARALLEL, DiffusionAccs.CACHE_DIT, DiffusionAccs.LAYERWISE_OFFLOAD],
            [DiffusionAccs.CFG_PARALLEL, DiffusionAccs.TENSOR_PARALLEL, DiffusionAccs.CPU_OFFLOAD],
        ],
    ),
    # New pipeline: add one DiffusionModelTestOpts(...) here only
}

L3 — JSON cases + unified online/offline runners

Online template (tests/e2e/models/online_serving/test_online.py):

"""Unified online E2E runner: JSON case → OmniServer → HTTP → assert_response."""
@pytest.mark.parametrize(
    "omni_server",
    [case_to_server_params(c) for c in CASES],
    ids=[c["name"] for c in CASES],
    indirect=True,
)
@pytest.mark.parametrize("resolved_case", CASES, indirect=True)
def test_online(resolved_case, omni_server, openai_client, run_level):
    case = resolved_case
    engine_req = build_prompt(case, mode="online")
    raw = send_generate_request(
        openai_client, engine_req, async_=case["input"].get("async", False)
    )
    response = process_output(
        raw,
        modality=case["output"]["modality"],
        transport=case["input"],
    )
    assert_response(
        response,
        case["input"],
        output=case["output"],
        run_level=run_level,
    )

case_to_server_params / build_prompt / case loading live in tests/e2e/models/case_io.py (or thin wrappers that call into tests/helpers/runtime.py).

JSON case schema (v1) — one object per case (files may be a list):

[
  {
    "name": "qwen_image_t2i_baseline",
    "mark": {
      "hardware_marks": { "res": { "cuda": "H100" }, "num_cards": 1 },
      "marks": ["full_model", "diffusion"]
    },
    "server": {
      "model": "Qwen/Qwen-Image",
      "server_args": []
    },
    "input": {
      "builder": "diffusion",
      "prompt": "A photo of a cat sitting on a laptop keyboard, digital art style.",
      "negative_prompt": "blurry, low quality",
      "width": 512,
      "height": 512,
      "num_inference_steps": 20
    },
    "output": {
      "modality": "image",
      "width": 512,
      "height": 512,
      "num_outputs": 1
    }
  }
]

Field	Role
name	pytest id
mark.hardware_marks	→ tests.helpers.mark.hardware_marks
mark.marks	pytest markers (core_model / advanced_model / full_model, diffusion / tts / omni, …)
server / runner	launch surface (OmniServerParams / OmniRunner kwargs + CLI)
input.builder	request family: diffusion | omni | tts | … → build_prompt / send_generate_request
output	expected shape / counts; assert_response gates stricter checks by run_level

Offline JSON mirrors online but uses runner instead of server (model, omni kwargs / stage CLI). Same input / output / mark keys.

Escape hatch: if a case needs multi-turn state, custom fixtures, or non-HTTP protocols that do not fit the schema, keep a dedicated Python module under the same directory (or under features/) and document why JSON was insufficient. Prefer extending the schema over proliferating one-off files.

L4 smoke note: feature-combination expansion matrices (per #1832) should prefer JSON cases with combined server_args / runner kwargs under the same online/offline templates, rather than a new Python file per combo — unless control flow cannot be expressed in JSON.

Acc — accuracy/ (lmms-eval)

• Goal: one template for precision regression; cases declare lmms-eval task + metric thresholds, not bespoke assert scripts.

• Precedent: Accuracy rail is intentionally analogous to vLLM’s lm-eval-based accuracy evaluation (shared harness + pinned eval deps + task/metric gates in CI). vLLM-Omni uses lmms-eval because Omni/diffusion/TTS workloads are multimodal; the process (harness-driven Acc, not one-off scripts) mirrors vLLM rather than inventing a parallel Acc framework.

• Launch: reuse case_to_server_params / case_to_runner_params (same as JSON L3) so Acc does not invent a third server stack.

• Eval: tests/e2e/models/accuracy/helper.py wraps lmms-eval (pinned in CI/dev deps); model adapters prefer OpenAI-compatible / Omni serving surfaces.

• CI: Merge (advanced_model) runs a small cheap Acc subset; Nightly (full_model) runs the full Acc matrix.

• Escape hatch: pixel golden / video similarity / ModelOpt quant suites that lmms-eval cannot express stay temporary under tests/e2e/accuracy/_legacy/ (or equivalent) with an explicit migrate TODO — do not start a parallel Acc framework.

Illustrative Acc case (v1 sketch):

[
  {
    "name": "qwen3_omni_mmmu_smoke",
    "mark": {
      "hardware_marks": { "res": { "cuda": "H100" }, "num_cards": 1 },
      "marks": ["full_model", "omni", "accuracy"]
    },
    "server": {
      "model": "Qwen/Qwen3-Omni-30B-A3B-Instruct",
      "server_args": []
    },
    "lmms_eval": {
      "task": "mmmu_val",
      "model_args": {},
      "batch_size": 1
    },
    "metrics": [
      { "name": "accuracy", "min": 0.0 }
    ]
  }
]

3) Update Strategy (phased)

Phase	Work	Exit criteria
A — Helpers + facades	assert_response on assertions.py; start helpers/tiny_models.py + e2e/models/case_io.py; no mass moves	Unit tests under tests/helpers/tests/; one pilot JSON case green locally
B — Relocate L2	Move model_tests/diffusion → helpers/tiny_models.py + e2e/models/tiny_model/{cases,helper,test_*}; update Buildkite paths	Ready+nightly tiny diffusion jobs green; alignment still fails closed on missing builders
C — JSON L3 pilots	Convert 2–3 representative online + offline cases (e.g. Qwen-Image t2i, one TTS, one omni chat) to JSON	Markers/--run-level still select correctly on merge/nightly
D — Bulk migrate smoke	Sweep happy-path test_*.py / simple expansion rows into JSON; keep exceptional Python only for escape-hatch cases	Net reduction in duplicated Python; CI time not regress
E — Docs	Update contributing / test skill paths to the new tree (incl. Acc vs smoke)	Docs + skill point at new tree
F — Acc harness	e2e/models/accuracy/ + lmms-eval facade + CI job; 1–2 pilot cases	Pilot Acc green on Nightly; deps pinned
G — Acc migrate	Sweep mappable tests/e2e/accuracy/* into lmms-eval cases; list legacy leftovers	Nightly Acc primary path; legacy documented

Phase A freeze rule: new model e2e must not add a new boilerplate Python module when a JSON case + existing builder would suffice; new tiny diffusion models must extend DIFFUSION_TEST_SETTINGS (+ builder in tiny_models.py) instead of a bespoke L2 file.

4) Backward Compatibility

• User-facing product: no change.

• Contributors: pytest node ids and file paths will change; update Buildkite source_file_dependencies and local copy-paste commands.

• Markers / --run-level: preserve semantics (core_model / advanced_model / full_model ↔ L2/L3/L4). JSON mark.marks must materialize the same markers today’s decorators use.

• Temporary shims: old test_foo_expansion.py may become a one-line pointer for one release window, then delete.

5) Risks and Mitigations

Risk	Mitigation
JSON schema too weak for omni multi-modal messages	Support input.messages / artifact refs; allow Python escape hatch
Helper layer becomes a second framework	Facades only; keep calling existing OmniServer / assert_diffusion_response
CI path churn breaks ready/merge/nightly	Phase B updates YAML in the same PR as the move; keep old path as thin re-export briefly if needed
Conflict with #1807 deploy cleanup	JSON server_args / runner CLI use documented overrides; no new deploy overlay YAMLs
lmms-eval / task churn or floating deps	Pin lmms-eval (+ related) in dev/CI image like vLLM pins lm-eval in test requirements; treat harness/task upgrades as explicit PRs (Design §7)
Acc too slow for Merge	Keep Merge Acc subset tiny; full matrix Nightly-only

6) CI auto job selection (open design)

Baseline (today). CUDA L2/L3 already have two skip layers (see docs/contributing/ci/ci_settings.md):

1. Bootstrap skip_ci.py — skip whole L2/L3 uploads for docs-only / CI-YAML-only diffs.
2. Step filter in upload_pipeline.py — omit E2E leaf jobs whose source_file_dependencies prefixes do not match the PR diff.

Adding or retargeting an E2E job still means hand-editing .buildkite/cuda/test-{ready,merge,nightly}.yml: label, commands, mirror_hardwares, and source_file_dependencies. That YAML churn grows as tiny registries and JSON cases multiply under this RFC.

Gap. Diff-aware filtering only drops existing YAML steps. It does not create jobs from tests/e2e/models/** registries/JSON, nor keep source_file_dependencies in sync when cases move.

Goal. Contributors add/change a tiny registry entry, JSON case, or Acc case; CI selects (and ideally materializes) the right Ready/Merge/Nightly jobs from that metadata + PR diff — without a parallel YAML edit for the common path.

Design options (pick via spike; not locked):

ID	Approach	Idea	Pros	Cons
A	Convention → codegen	Path/name conventions (or case front-matter) → generate leaf steps at upload time	Single source near tests; YAML becomes thin shells	Need stable conventions + codegen tests
B	Sidecar metadata	Per-case ci.yml / # ci: block listing level, HW preset, pytest path	Explicit; easy review	More files; risk of drift from case
C	Central matrix	One ci_matrix.yaml (or Python) listing all E2E jobs; test-*.yml only hosts groups/hooks	Clear inventory; easy audit	Still a second registry unless generated from cases
D	Smarter deps only	Keep hand-written steps; auto-derive / validate source_file_dependencies from case paths + model dirs	Smallest change; builds on today	Does not remove YAML job boilerplate

Recommended spike path: (1) inventory current E2E leaf jobs vs intended tiny/JSON/Acc cases; (2) prototype A or C for one diffusion + one omni job in Ready only; (3) decide whether Nightly stays hand-curated longer (cost/HW). Prefer extending upload_pipeline.py over a second uploader.

Constraints / non-goals for v1:

• Do not bypass label triggers (ready, merge-test, nightly-test) or bootstrap skip policy.
• Preserve mirror_hardwares presets; do not invent per-PR GPU strings in case JSON.
• Escape-hatch / expansion suites may remain hand-listed.
• AMD/NPU/Intel can lag CUDA until the CUDA matrix is stable.
• No silent job explosion: generated Ready/Merge sets must be bounded and unit-tested (tests/buildkite/).

Tie-in to this RFC: Phases that land tiny_model/ + JSON cases are the natural metadata source for auto-selection; Acc (lmms-eval) jobs should use the same mechanism once the smoke matrix lands.

7) Pinned CI dependencies (vLLM-aligned)

Problem. When CI image layers or DEP_FILES hashes change, floating specs (pkg>=x) re-resolve to newer releases. Failures then appear as model/connector flakes while the root cause is an unreviewed wheel bump — costly to localize on Ready/Merge/Nightly.

Policy (target). Match vLLM dependency management:

• Prefer exact pins (==) for CI/dev packages that influence runtime behavior or eval results.
• Prefer a locked or explicitly reviewed requirements set for the CUDA CI image (same spirit as vLLM requirements/test/*.txt), rather than open ranges in pyproject.toml optional dev.
• Dependency upgrades are dedicated PRs (or clearly scoped commits): pin bump + why + which jobs revalidated.
• lmms-eval (Acc) and other eval/tooling pins follow the same rule — task/harness upgrades must not ride along unnoticed with unrelated Dockerfile edits.

Non-goals: pinning every transitive leaf forever without tooling; blocking urgent security bumps (still pin the chosen fixed version after the bump).

8) Per-model code coverage

Goal: answer "which code paths does model X actually exercise in online vs offline (and under different sampling params)?" — visibility that repo-wide --cov=vllm_omni cannot provide.

Scope (v1):

1. Instrument L2 tiny and L3 online/offline model E2E with pytest-cov (or equivalent), tagged by:
   • model_id / pipeline class
   • entry mode: online | offline
   • optional case tags (e.g. sampling profile / CFG / streaming)
2. Emit per-run coverage XML/JSON artifacts; aggregate into a simple dashboard (Buildkite artifact index or external coverage UI).
3. Prefer scoping reports to model-relevant packages when possible (e.g. vllm_omni/diffusion/models/<arch>/, shared entrypoints/runtime), while still recording overall touched lines for comparison.
4. Pilot on a small set (one diffusion + one Omni or TTS) before rolling out broadly.

Non-goals (v1):

• Hard coverage floors / blocking gates on every model PR.
• Replacing existing unit/core_model repo-wide coverage jobs.
• Instrumenting every Nightly Acc job on day one (optional follow-up once smoke coverage is stable).

Depends on: stable case identity from L2/L3 templates (PR 3–4) so artifacts can be keyed by model + mode without one-off YAML per suite.

9) Pre-commit hardening (vLLM-aligned)

Problem. Omni’s .pre-commit-config.yaml is thinner than upstream vLLM, and a few existing knobs are inconsistent with docs/CI:

Gap	Today (Omni)	Target
Docs / Mergify drift	Mergify tip still mentions mypy-3.10 / markdownlint, but hooks may be missing or differently named; [tool.mypy] in pyproject.toml is unused if not wired into pre-commit	Tip + hooks + CI args stay in sync; mypy config is actually invoked
Missing hooks vs vLLM	No markdownlint-cli2, no shellcheck, no SPDX header check, no systematic forbidden-import / torch.cuda guard (beyond a few ruff TID251 bans and a pickle-only script)	Add those hooks (mypy as stages: [manual], same spirit as vLLM)
typos pin	Historically pinned to an obsolete typos-dict-v0.* tag; upstream uses v1.43.x	Pin a current v1.43.x (or newer) release
Hook order	Comment says keep suggestion last, but other local hooks can run after it	Enforce suggestion as the final hook
Stages	Missing default_stages: [pre-commit, manual] pattern used by vLLM so CI can run --hook-stage manual	Align CI workflow + local install with that pattern

Policy.

1. Local gate (pre-commit): fast format/lint/custom scripts on every commit; autofix where safe.
2. CI gate: re-run the same config with --all-files --hook-stage manual (so manual-only hooks such as mypy also run).
3. Do not put deep semantic security analysis into pre-commit (too slow / not autofixable) — that is CodeQL’s job (§10).
4. Gradual adoption is OK: allowlist existing torch.cuda / pickle call sites; SPDX can be backfilled in the same PR that enables the hook; mypy may start with non-strict / exclude model adapters and tighten later.
5. Keep Omni-specific hooks that vLLM lacks when useful (check-test-ci-coverage, DCO signoff), but document which are SKIP’d in CI and why.

Non-goals (v1): copying every vLLM hook (clang-format, rust, pip-compile sync) when Omni has no matching tree; making mypy fully strict on day one.

10) CodeQL (semantic security / defect scanning)

Why CodeQL is not a pre-commit substitute.

	pre-commit	CodeQL
Role	Commit-time style + light correctness + repo policy scripts	CI semantic static analysis (dataflow / taint / security queries)
When	Local git commit + GitHub Actions re-run of the same hooks	GitHub Actions / Code Scanning on PR + main (minutes, not seconds)
Depth	File/line rules, regex/custom scripts	Builds a CodeQL DB; cross-function queries
Typical finds	Trailing whitespace, ruff, typos, missing SPDX, banned imports	Injection, unsafe deserialization, path traversal, some resource bugs
Autofix	Often yes (ruff format, eof, etc.)	No — alerts only
Fit for every commit	Yes	No

Upstream vLLM already runs CodeQL via GitHub code scanning (workflow runs). Omni currently has no CodeQL workflow under .github/workflows/.

Proposed Change (v1).

1. Add .github/workflows/codeql.yml (or enable GitHub default / advanced setup) for Python with build-mode: none (interpreted language).
2. Trigger on: push to main, pull_request to main, and a weekly schedule (catch regressions on quiet branches).
3. Start with default query suite; optionally move to security-extended once alert volume is triageable.
4. Permissions: security-events: write, contents: read, actions: read (standard Code Scanning).
5. Gate policy (v1): upload results to GitHub Code Scanning; do not hard-block every PR on day one if the backlog is large — triage known issues, then tighten (required check or Mergify condition) once noise is under control.
6. Document in contributing / Mergify how to read CodeQL alerts vs pre-commit failures.

Non-goals (v1): custom QL packs for Omni-specific APIs; scanning vendored third-party trees as first-class; replacing Bandit/Semgrep (optional follow-ups, not required here).

Usage

Add a tiny diffusion pipeline (L2)

1. Implement tiny_<arch>_builder in tests/helpers/tiny_models.py.

2. Append one DiffusionModelTestOpts(...) in tests/e2e/models/tiny_model/cases.py.

3. Run:

pytest -sv tests/e2e/models/tiny_model -k Flux2KleinPipeline -m 'core_model and cuda' --run-level=core_model

Add a full-weight online smoke (L3)

1. Drop tests/e2e/models/online_serving/cases/<model>_<task>.json.

2. No new Python unless escape hatch.

3. Run:

pytest -sv tests/e2e/models/online_serving/test_online.py -k qwen_image_t2i_baseline -m 'full_model and diffusion' --run-level=full_model

Add an Acc case (lmms-eval)

1. Drop tests/e2e/models/accuracy/cases/<model>_<task>.json (task + metric thresholds).

2. Ensure a model adapter exists for the serving surface (or reuse the shared OpenAI-compatible adapter).

3. Run:

pytest -sv tests/e2e/models/accuracy/test_accuracy.py -k qwen3_omni_mmmu_smoke -m 'full_model and accuracy' --run-level=full_model

Assert depth vs run_level

• core_model: structural smoke (non-empty modality payload, basic counts).

• advanced_model / full_model: dimensions and other gates already encoded in tests/helpers/assertions.py.

Validation

1. Helpers: pytest -sv tests/helpers/tests -m core_model (facade routing / builder smoke as applicable).

2. L2: Buildkite ready step currently pointing at tests/model_tests/diffusion retargeted to tests/e2e/models/tiny_model with the same markers.

3. L3 pilots: one online + one offline JSON case on merge (advanced_model) and nightly (full_model) as applicable.

4. Parity: for each migrated module, keep a short checklist — same model id, cards, markers, and assert class — before deleting the old file.

5. Acc pilots: one Omni (or existing qwen3_omni Acc) + optional TTS/diffusion-mapped task on Nightly via tests/e2e/models/accuracy.

6. Acc parity: for each migrated Acc module, record old metric name → lmms-eval task/metric + threshold before deleting the old file.

7. Coverage pilots: one diffusion + one Omni (or TTS) model producing online and offline coverage artifacts; confirm the dashboard shows distinct path coverage across entry modes.

8. Pre-commit: pre-commit run --show-diff-on-failure --color=always --all-files --hook-stage manual is green (or documented SKIP only for known backlog hooks).

9. CodeQL: workflow green on a pilot PR; alerts visible under Security → Code scanning; triage note linked from this RFC.

10. Non-goals for v1: rewriting all *_expansion.py feature grids in one PR; WebSocket / fullduplex / ComfyUI (tests/e2e/features/); forcing every pixel/video similarity suite into lmms-eval in the first Acc PR.

Planned PR series.

Ordered delivery (titles indicative; open as separate PRs linked to this RFC):

#	Type	Title	Scope	Depends on	Status
1	Test	[Test] Diffusion tiny-model L2 registry; replace current L2 diffusion e2e	Land tests/e2e/models/tiny_model for diffusion; builders → helpers/tiny_models.py; retarget Ready/Nightly; retire/replace current diffusion L2 one-offs	#4197 / this RFC Phase B	⏳#5353
2	Test	[Test] Omni tiny-model L2 registry; replace current L2 omni e2e	Same pattern for Omni (and TTS stubs as needed); replace current omni L2	PR 1 conventions	⏳#5459
3	Test	[Test] Unify tiny-model templates (offline/online/alignment)	Freeze test_offline.py / test_online.py / test_alignment.py + helper.py APIs; docs + skill paths	PR 1–2	🙋
4	Test	[Test] JSON-driven L3/L4 E2E template (qwen3-tts pilot)	case_io.py + online/offline templates; qwen3-tts as first JSON case	PR 3	🙋
5	Test	[Test] Migrate all models' L3/L4 smoke to JSON templates	Bulk-migrate happy-path tests/e2e/{online_serving,offline_inference}; keep escape hatches	PR 4	🙋
6	Test	[Test] lmms-eval accuracy harness framework	e2e/models/accuracy/ + test_accuracy.py + helper + CI job + pinned deps; pattern aligned with vLLM lm-eval Acc; 1–2 pilot cases	Can parallelize with PR 4 once launch contract is stable	🙋
7	Test	[Test] Migrate existing accuracy suites to lmms-eval harness	Replace/retire mappable tests/e2e/accuracy/*; document legacy leftovers; switch Nightly Acc primary path	PR 6	🙋
8	CI	[CI] Auto-select / generate E2E jobs from case metadata (spike → CUDA Ready/Merge)	Spike Design §6 options A–D; land bounded codegen or matrix for CUDA L2/L3 E2E; unit tests under tests/buildkite/; docs in ci_settings.md; Nightly/other platforms optional follow-ups	—	🙋
9	CI	[CI] Pin CI/dev dependencies (vLLM-aligned)	Exact-pin critical dev/CI deps; document upgrade path; keep lmms-eval and connector pins in the same policy; optional move toward a locked requirements file like vLLM requirements/test	Can land early / in parallel with PR 1–8; Acc PR 6 must consume the pin policy	✅#5392
10	CI	[CI] Per-model code coverage collection	Collect coverage per model × online/offline from L2/L3 E2E; upload artifacts; document how new models opt in	—	⏳#5593
11	CI	[CI] Harden pre-commit (vLLM-aligned hooks + docs sync) + Enable CodeQL / GitHub Code Scanning (Python)	Add/fix mypy(manual), markdownlint, shellcheck, SPDX, forbidden-import, torch.cuda checks; upgrade typos; suggestion last; sync Mergify + pre-commit workflow, CodeQL enable		⏳@yenuo26

Milestones

1. M1 (L2): PR 1 → PR 2 → PR 3

2. M2 (L3/L4 smoke): PR 4 → PR 5

3. M3 (Acc): PR 6 → PR 7

4. M4 (CI matrix): PR 8 (can overlap M2 once tiny/JSON conventions are stable)

5. M5 (Pinned deps): PR 9 (can land anytime; required before Acc Nightly is trusted)

6. M6 (Per-model coverage): PR 10 (can overlap M2 once case identity is stable)

7. M7 (Static quality / security): PR 11 (pre-commit) + PR 12 (CodeQL); can land anytime and in parallel with M1–M6

Feedback Period.

1 week (align with #4197 discussion window).

CC List.

@alex-jw-brooks @NickCao @congw729

Any Other Things.

• lmms-eval is the Acc harness of record for new precision gates; pin versions in CI to avoid floating-eval regressions (same class of issue as floating dev deps).

• Diffusion-only / pixel-similarity Acc may use a metric plugin or _legacy/ path until an lmms-eval task exists — still registered under the Acc directory contract.

• Proposed helper API names (assert_response, send_generate_request, get_offline_runner_params, steps_for_case) are target facades; initial PR may thin-wrap today’s send_diffusion_request / assert_diffusion_response / get_parametrized_options / task_runners without behavior change.

• Omni / TTS tiny registries are follow-ups after diffusion L2 relocation (same pattern as #4197).

• This RFC does not change CI five-level meaning documented in contributing / vllm-omni-test skill; it only makes the filesystem match that meaning for model smoke tests.

• Existing suites outside this layout (e.g. tests/e2e/features/) are out of scope for this RFC.

• Per-model coverage was requested in RFC discussion (https://github.com/vllm-project/vllm-omni/issues/5332#issuecomment-5091716066). Repo-wide --cov=vllm_omni remains for unit/core_model suites; model E2E adds the per-model × online/offline slice for path visibility.

• Pre-commit vs CodeQL: pre-commit = fast style/policy on commit; CodeQL = slower semantic SAST in CI. Both belong in this RFC’s CI quality bar; neither replaces E2E/Acc/coverage tracks.

• openLiBing / org governance platforms (if used later) may orchestrate these checks but do not replace configuring pre-commit + CodeQL in this repo.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.