kv-router: skip partial-prefix BlockStored events instead of exiting - #38
Draft
Thachnh wants to merge 1 commit into
Draft
kv-router: skip partial-prefix BlockStored events instead of exiting#38Thachnh wants to merge 1 commit into
Thachnh wants to merge 1 commit into
Conversation
vLLM (>= the DeepSeek-V4.1 day-0 image) hashes prefixes every prefix_match_unit (hash_block_size) tokens, which can be finer than the cache block size, and publishes the prompt tail that ends inside a cache block as a BlockStored whose block_size is the sub-block length (32/64/96 for a 128-token block). The standalone indexer treated any block_size mismatch on a main-attention event as a fatal --block-size misconfig and exited, so every V4.1 indexer crash-looped. Treat a proper divisor of the configured block size as a partial-prefix entry: drop it (rate-limited warn). Non-divisor mismatches stay fatal.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
vLLM (from the DeepSeek-V4.1-Flash day-0 image,
vllm/vllm-openai:deepseekv41-flash-0909) hashes prefixes everyprefix_match_unit(hash_block_size, 32 for V4.1) tokens, which is finer than the cache block size (128). It publishes the prompt tail that ends inside a cache block as aBlockStoredon the main MLA group whoseblock_sizeis the sub-block length (32/64/96).The standalone indexer treats any main-attention block-size mismatch as a fatal
--block-sizemisconfiguration and callsstd::process::exit(1), so all threekv-indexer:*flavors fordeepseek-ai/DeepSeek-V4.1-Flashcrash-looped (engine block size 32 != configured --block-size 128). No indexer--block-sizeworks: 128 dies on the partial events, 32 dies on the whole-block events.Observed event mix on a V4.1 pod (150 s tap): groups 0-3
sliding_window_mla@32 (already filtered as non-main), group 4mla_attention@128 (whole blocks) plus @32 partial entries.Fix
convert_event: whenblock_size < kv_block_sizetreat the event as a partial-prefix entry and drop it (rate-limited warn) instead of erroring. A larger event block size stays fatal (that can only be a misconfig). Whole-block routing information is unaffected; only sub-block tail entries are ignored.Unit test added;
cargo test --lib zmq_wire::tests= 19 passed.Same commit cherry-picked onto
24h-indexerasv41-partial-skip-h24for the h24 image.Refs DEE-640.