vLLM-Omni Q3 Config / Entrypoints Roadmap

Tracking some of the ongoing discussions around config cleanup and entrypoints, since we have had some relevant discussions around it in the English community syncs, and it would be helpful to link subissues to as we progress to ensure things are moving towards a better state + help coordinate PRs.

VllmOmniModelConfig

• See details in the RFC: https://github.com/vllm-project/vllm-omni/issues/4021; most PRs are linked there for now since its pretty comprehensive
  • Config resolver: https://github.com/vllm-project/vllm-omni/pull/5140 (@fake0fan)

Milestones:

• Initial OmniModelConfig implementation
• OmniModelConfig is cleanly integrated with composable parallel strategy config
• OmniModelConfig is active on the production path

Mostly being worked on by @zwhzzz0821 @Acerak01-fy @alex-jw-brooks @lishunyang12

Legacy Migration / Deprecation Policies

• Migrate all legacy state configs to deploy configs (@zwhzzz0821 @lishunyang12)
• Remove legacy stage configs path (@zwhzzz0821)
• Add more formal deprecation policies and timeline (i.e., analogous to this for vLLM).
  • For this, we should have consistent infrastructure for logging when deprecated things will be removed by so that we can ensure that they are actually removed.
• Remove OmegaConfig integration (@alex-jw-brooks)

Code Health

• Handle environment variables in a way that better aligns with vLLM's patterns
• Standardize naming conventions for pipelines with multiple variants
• Remove hardcoded kwarg string lists, since these are hard to maintain
• Move away from passing **kwargs / dicts in different places and add/reuse better data structures to prevent silently ignored options
• Remove overly defensive fallbacks (i.e., getattrs / .get defaults), add better type hints
  • https://github.com/vllm-project/vllm-omni/pull/5199 (@princepride)

Entrypoints / Config Auditing

• Test entrypoints to see which flags are actually being applied correctly in configs, and which aren't; some are silently ignored or are confusing (@akshatvishu)
• Check which deploy config settings are not passable as CLI flags and ensure behavior is consistent (@yzong-rh)
• Discuss blocking feature flags that don't work correctly as not implemented etc, with follow-ups to properly support them if needed
• Add utils for hitting all endpoints, e.g., through tiny model framework, and ensure that we can't send requests to invalid endpoints and crash the server (was previously possible with completions) (@zyforsure)

Feel free to add thoughts or suggestions as needed. For now this is mostly to track things around config and help coordinate on some of the more complex efforts, but will add some links for entrypoints refactoring etc later since they are probably overlapping a little.

CC @lishunyang12 @hsliuustc0106 @linyueqian @gcanlin @ieaves @akshatvishu @kushanam