vllm-project/semantic-router

feat: add HaluGate-driven per-model per-domain reputation routing

Closed

#2,542 opened on Jul 15, 2026

 (0 comments) (0 reactions) (0 assignees)Go (699 forks)github user discovery
area/model-selectionarea/observabilityenhancementevaluationhelp wantedpriority/P1roadmapsignal-decision-engine

Repository metrics

Stars
 (4,293 stars)
PR merge metrics
 (PR metrics pending)

Description

Motivation

HaluGate already produces request-level signals that are captured in Router Replay. The missing capability is a trustworthy materialization pipeline that converts those observations into bounded model/domain reputation evidence and then exposes it to Router Learning.

Audited upstream baseline (2026-07-15)

  • HaluGate is registered in the model/config surfaces and replay records include HaluGate-related fields.
  • Router Replay records the selected logical model, but a durable model-version/config-version identity is not guaranteed across every record.
  • There is no production materializer that aggregates HaluGate outcomes into per-model/per-domain reputation.
  • Router Learning—not the generic model selection package—is the intended consumer for bounded adaptation.

Scope

  1. Define a versioned HaluGate outcome event with logical model, deployable model/version identity, domain/taxonomy version, config/policy version, score/verdict, and provenance.
  2. Add an asynchronous materializer with minimum sample thresholds, time decay, uncertainty, and bounded storage/cardinality.
  3. Separate observed evidence from the derived reputation snapshot.
  4. Make Router Learning consume reputation only as a bounded feature/action prior under protection.
  5. Provide shadow mode, replay diagnostics, rollback, and operator inspection.
  6. Specify behavior for missing versions, model aliases, delayed outcomes, and evaluator drift.

Safety and statistical boundaries

  • Sparse or stale evidence must not produce confident routing changes.
  • HaluGate evidence cannot override authorization, hard safety rules, or the validated candidate set.
  • Aggregate by privacy-safe keys; do not require raw prompt retention.
  • Correlation is not a causal quality estimate; evaluation must compare against fixed baselines.

Acceptance criteria

  • Event and snapshot schemas are versioned and migration-tested.
  • Model/config/taxonomy identity is sufficient to prevent cross-version contamination.
  • Decay, sample thresholds, uncertainty, and cold-start behavior are deterministic.
  • Shadow/replay evaluation precedes live Router Learning consumption.
  • Every applied effect is reconstructable from Router Replay and emits a bounded reason.
  • Restart, multi-replica, and delayed-outcome semantics are documented.
  • E2E tests prove hard safety constraints take precedence.

Likely change surfaces

src/semantic-router/pkg/routerreplay/, src/semantic-router/pkg/routerruntime/, src/semantic-router/pkg/extproc/, src/semantic-router/pkg/config/, observability, CLI diagnostics, docs, and tests.

Related: #2238, #2243.

Contributor guide