Datapf 2407 vllm 0.15.1 - #8
Open
Yannick Lambruschi (ylambruschi) wants to merge 5 commits into
Open
Conversation
…M and RunAI streamer Enable serving models directly from S3, GCS, Azure Blob, and HTTP/HTTPS URIs using vLLM backend with RunAI Model Streamer for fast loading without downloading the entire model first. Changes: - Skip architecture validation for remote URIs (s3://, gs://, az://, http://, etc.) - Add force_vllm parameter to bypass validation when --backend=vllm is explicit - Maintain backward compatibility for HuggingFace model IDs and local paths - Add comprehensive documentation and examples for S3/RunAI usage This enables use cases like: --model_id=s3://bucket/model/ --backend=vllm --load-format=runai_streamer Benefits: - Faster model loading by streaming directly to GPU - Reduced storage requirements (no local model copy needed) - Flexible model deployment from various cloud storage providers Files modified: - huggingfaceserver/vllm/utils.py: Add remote URI detection and force_vllm parameter - huggingfaceserver/__main__.py: Pass force_vllm flag based on backend selection - README.md: Add "Using Remote Storage URIs with vLLM" section - New: example_s3_runai.yaml with complete InferenceService examples - New: CHANGELOG_S3_RUNAI.md with detailed implementation notes Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Yannick Lambruschi (ylambruschi)
force-pushed
the
DATAPF-2407_vllm_0.11.2
branch
4 times, most recently
from
February 9, 2026 21:21
dd58add to
caf0e1c
Compare
Upgrade HuggingFace server to use vLLM 0.14.1 for bug fixes, security improvements, and performance enhancements. Key Changes: - vLLM: 0.11.2 → 0.14.1 - PyTorch: 2.9.0 → 2.9.1 (required by vLLM 0.14) - Protobuf: 5.x → 6.33.5 (required by vLLM 0.14) - AsyncEngineArgs import path updated for vLLM 0.14 API Breaking Changes Addressed: 1. Updated protobuf constraint from <6.0.0 to <7.0.0 to accommodate vLLM 0.14.1 requirement of protobuf >= 6.30.0 2. Fixed AsyncEngineArgs import: from vllm → from vllm.engine.arg_utils 3. Updated lock files for all transitive dependency changes New Features Available: - Async scheduling enabled by default (performance improvement) - gRPC server entrypoint with HTTP/2 multiplexing - Automatic context length: --max-model-len auto - Enhanced /embeddings endpoint and new /server_info endpoint - Support for 8 new model architectures Dependency Updates: - Added: httpx-sse, mcp, sse-starlette, grpcio-reflection - Removed: scipy, xformers (no longer required by vLLM) - Updated: triton, grpcio, mistral-common, compressed-tensors, flashinfer Compatibility: ✅ S3/RunAI streamer feature fully compatible (no changes needed) ✅ Remote URI detection and force_vllm parameter preserved ✅ All existing functionality maintained Files Modified: - python/kserve/pyproject.toml: vLLM version and protobuf constraint - python/huggingfaceserver/huggingfaceserver/vllm/vllm_model.py: Import fix - python/kserve/uv.lock: Updated dependencies - python/huggingfaceserver/uv.lock: Updated dependencies Documentation: - Added MIGRATION_VLLM_0.14.1.md: Migration plan and requirements - Added VLLM_0.14.1_CHANGES.md: Comprehensive change summary References: - vLLM v0.14.1: https://github.com/vllm-project/vllm/releases/tag/v0.14.1 - KServe PR kserve#4990: kserve#4990 Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Yannick Lambruschi (ylambruschi)
force-pushed
the
DATAPF-2407_vllm_0.11.2
branch
3 times, most recently
from
February 10, 2026 17:35
be60434 to
a796245
Compare
Upgrade HuggingFace server to use vLLM 0.15.1 for significant performance improvements, bug fixes, and security updates. Key Changes: - vLLM: 0.14.1 → 0.15.1 - FlashInfer: 0.5.3 → 0.6.1 (4x faster torch.compile cold-start) - aiohttp: 3.11.16 → 3.13.3 (Security: CVE-2025-69223) - apache-tvm-ffi: 0.1.3 → 0.1.8.post2 - nvidia-cutlass-dsl: 4.3.0 → 4.3.5 Breaking Changes (vLLM 0.15.0+): ❌ OpenAI entrypoints reorganized into subdirectories - Protocol classes moved to subdirectories: - OLD: vllm.entrypoints.openai.protocol - NEW: chat_completion.protocol, completion.protocol, engine.protocol - Serving classes moved to subdirectories: - OLD: vllm.entrypoints.openai.serving_completion - NEW: vllm.entrypoints.openai.completion.serving - OLD: vllm.entrypoints.openai.serving_chat - NEW: vllm.entrypoints.openai.chat_completion.serving - IMPACT: Required import updates in types/__init__.py and vllm_model.py Performance Improvements: - torch.compile cold-start: 88s → 22s (4x faster for Llama3-70B) - FlashInfer 0.6.1: Better MoE kernel performance on RTX GPUs - V1 engine: Continued CPU/GPU overlap optimizations Bug Fixes (0.15.1): - Prefix cache hit rates improved - Speculative decoding metrics fixed - ROCm GEMM dispatch fixed Security Fixes: - CVE-2025-69223: aiohttp vulnerability patched Code Changes Required: 1. python/kserve/kserve/protocol/rest/openai/types/__init__.py - Updated protocol imports to use new subdirectories: - Chat: vllm.entrypoints.openai.chat_completion.protocol - Completion: vllm.entrypoints.openai.completion.protocol - Engine: vllm.entrypoints.openai.engine.protocol 2. python/huggingfaceserver/huggingfaceserver/vllm/vllm_model.py - Updated ErrorResponse import: engine.protocol - Updated OpenAIServingCompletion: completion.serving - Updated OpenAIServingChat: chat_completion.serving 3. python/huggingfaceserver/test_vllm_migration.py (NEW) - Added comprehensive test script to validate vLLM API compatibility - Tests imports, attributes, and method signatures - Tests both protocol and serving class imports - Enables fast local validation without Docker builds Compatibility: ✅ Pooling imports unchanged (embed.serving, score.serving) ✅ All parameters unchanged (enable_log_requests, disable_log_stats) ✅ S3/RunAI streamer still works ✅ Python 3.10, 3.11, 3.12 supported ✅ PyTorch 2.9.1+ compatible Files Modified: - python/kserve/pyproject.toml: vLLM version updated - python/kserve/kserve/protocol/rest/openai/types/__init__.py: Protocol imports - python/huggingface_server.Dockerfile: VLLM_VERSION, FLASHINFER_VERSION - python/huggingface_server_cpu.Dockerfile: VLLM_VERSION - python/huggingfaceserver/tests/setup_vllm.sh: VLLM_VERSION - python/huggingfaceserver/huggingfaceserver/vllm/vllm_model.py: All imports - python/huggingfaceserver/test_vllm_migration.py: NEW comprehensive test - python/kserve/uv.lock: Dependencies updated - python/huggingfaceserver/uv.lock: Dependencies updated Documentation: - Added MIGRATION_VLLM_0.15.1.md: Complete migration guide - Added QUICK_TEST.md: Local testing instructions Testing Recommendations: 1. Run test_vllm_migration.py for import validation 2. Test embeddings endpoint thoroughly 3. Test reranking endpoint thoroughly 4. Test chat completions endpoint 5. Test completions endpoint 6. Performance benchmarks (expect 4x improvement in cold-start) 7. S3/RunAI streamer verification Rollback Plan: Revert both dependency versions AND all import path changes References: - vLLM v0.15.1: https://github.com/vllm-project/vllm/releases/tag/v0.15.1 - vLLM v0.15.0: https://github.com/vllm-project/vllm/releases/tag/v0.15.0 - Module reorganization: https://github.com/vllm-project/vllm/tree/v0.15.1/vllm/entrypoints/openai Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Yannick Lambruschi (ylambruschi)
force-pushed
the
DATAPF-2407_vllm_0.11.2
branch
from
February 10, 2026 18:37
a796245 to
017a339
Compare
Yannick Lambruschi (ylambruschi)
changed the title
Jeanine Harb (jharb-mirakl)
approved these changes
Feb 13, 2026
Jeanine Harb (jharb-mirakl)
left a comment
Jeanine Harb (jharb-mirakl)
left a comment
There was a problem hiding this comment.
LGTM, with minor comments!
| if args.model_id: | ||
| return cast(str, args.model_id) | ||
| return Path(Storage.download(args.model_dir)) | ||
| # If --model_id is specified then pass model_id to HF API, otherwise load the model from /mnt/models |
There was a problem hiding this comment.
The diff shows an extra indentation here, is it the case?
| @@ -0,0 +1,165 @@ | |||
| # vLLM 0.11.2 → 0.14.1 Migration Plan | |||
There was a problem hiding this comment.
Need to adapt migration plan to vllm 0.15.1
There was a problem hiding this comment.
Just saw that there is a migration plan for 0.15.1 as well: do you need both, or can they be merged?
| @@ -0,0 +1,200 @@ | |||
| # vLLM 0.14.1 Migration - Changes Summary | |||
There was a problem hiding this comment.
Need to adapt it to vllm 0.15.1?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
Which issue(s) this PR fixes (optional, in
fixes #<issue number>(, fixes #<issue_number>, ...)format, will close the issue(s) when PR gets merged):Fixes #
Type of changes
Please delete options that are not relevant.
Feature/Issue validation/testing:
Please describe the tests that you ran to verify your changes and relevant result summary. Provide instructions so it can be reproduced.
Please also list any relevant details for your test configuration.
Test A
Test B
Logs
Special notes for your reviewer:
Checklist:
Release note:
Re-running failed tests
/rerun-all- rerun all failed workflows./rerun-workflow <workflow name>- rerun a specific failed workflow. Only one workflow name can be specified. Multiple /rerun-workflow commands are allowed per comment.