[XPU] Make checkpoint_engine worker device-agnostic - #32382
Draft
siju-samuel wants to merge 1 commit into
Draft
Conversation
The checkpoint_engine worker hardcoded torch.cuda for device UUID/id resolution, so the ZMQ handshake with checkpoint-engine's ParameterServer only worked on CUDA. Make it device-agnostic: - _accelerator_type() now delegates to SGLang's get_device() instead of assuming CUDA, and device-module lookups go through get_device_module(). - get_device_uuid() mirrors ps.py::_get_physical_gpu_id exactly: NPU keys as NPU-<uuid>, every other accelerator (CUDA, XPU) as GPU-<uuid>. This also fixes NPU, which the old cuda-fallback would have keyed as GPU-<uuid> and never matched the ParameterServer. Wire the checkpoint-engine optional dependency into pyproject_xpu.toml. XPU support is not yet in a released checkpoint-engine, so it tracks the upstream source (git+https://github.com/MoonshotAI/checkpoint-engine.git) until a release ships it, mirroring how sgl-kernel-xpu is referenced. Tests (test/registered/unit/checkpoint_engine/): CPU-mockable routing for cuda/xpu/npu key formats and the AssertionError->ValueError wrapping, plus a hardware-gated XPU case asserting the worker's real UUID matches the ParameterServer's independently derived key. Registered for CPU and XPU CI.
Contributor
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
| logger = logging.getLogger(__name__) | ||
|
|
||
|
|
||
| def _accelerator_type() -> str: |
Contributor
There was a problem hiding this comment.
why not get_device, directly ?
| ] | ||
|
|
||
| [project.optional-dependencies] | ||
| checkpoint-engine = ["checkpoint-engine @ git+https://github.com/MoonshotAI/checkpoint-engine.git"] |
Contributor
There was a problem hiding this comment.
after PR merge and any release let's use/pin it like CUDA
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Important
Depends on MoonshotAI/checkpoint-engine#96
(adds Intel XPU support to checkpoint-engine). That PR must merge first
pyproject_xpu.tomltracks checkpoint-engine's upstreammain, which onlyresolves to XPU-capable code once #96 lands.
Motivation
SGLang's checkpoint-engine integration enables fast in-place weight updates
for a running server (the RL / online-serving use case), but the worker
hardcoded
torch.cudafor device UUID/id resolution. As a result the ZMQhandshake with checkpoint-engine's
ParameterServeronly worked on CUDA.This makes the worker device-agnostic so the same path works on Intel XPU,
and along the way fixes a latent NPU bug: the ParameterServer keys NPU
devices as
NPU-<uuid>, but the old CUDA fallback would have keyed them asGPU-<uuid>— a key that never resolves.Modifications
srt/checkpoint_engine/checkpoint_engine_worker.py_accelerator_type()now delegates to SGLang's device-agnosticget_device()instead of assuming CUDA; device-module lookups go throughget_device_module()(removing thegetattr(torch, ...)indirection).get_device_uuid()mirrorscheckpoint_engine.ps::_get_physical_gpu_idexactly: NPU →
NPU-<uuid>, every other accelerator (CUDA, XPU) →GPU-<uuid>. This is the fix for the NPU key mismatch above.python/pyproject_xpu.tomlcheckpoint-engineoptional dependency for the XPU build. XPUsupport is not yet in a released checkpoint-engine, so it tracks upstream
source (
git+https://github.com/MoonshotAI/checkpoint-engine.git) until arelease ships it, mirroring how
sgl-kernel-xpuis referenced in the samefile.
test/registered/unit/checkpoint_engine/test_checkpoint_engine_worker.pyAssertionError→ValueErrorwrapping.the key the
ParameterServerindependently derives (the actual handshakecontract).
base-a-test-cpu) and XPU(
stage-b-test-1-gpu-xpu) CI.Accuracy Tests
Not applicable — this changes device-identity resolution for the weight-update
handshake, not model forward/kernels. Model outputs are unaffected.
Speed Tests and Profiling
Not applicable — no change to the inference hot path.
get_device()/get_device_module()are called only during the weight-update handshake, notper token.
Checklist
CI States
Latest PR Test (Base): ❌ Run #30145719133
Latest PR Test (Extra): ❌ Run #30145719088