standalone-indexer: subscribe to every data-parallel rank of a discovered pod

[Shang-Pin]

Why

Pod discovery registers each engine pod once, as dp_rank 0 on tcp://<ip>:<zmq_port>. vLLM offsets both the KV-event ZMQ port and the /kv_recover port by data_parallel_rank, so on a --data-parallel-size N engine, ranks 1..N-1 publish to ports nobody subscribes to. Everything they cache is invisible to /query while the engine still hits it.

Measured on deepseek-ai/DeepSeek-V4.1-Flash (DP=2 in the main-0915-ds4-patches image): every prefill the scheduler placed on rank 1, about half of them, showed 0 in the standalone indexer and in rank 0's /kv_recover dump, and all of its blocks in rank 1's dump. deepapi's KV ladder read actual 0.78 > h24 0.65 > perfect 0.63 > reality 0.59 (engine's own hit ratio 0.782), i.e. the indexer under-reports the cache by about a third and KV-aware routing is blind to half of every pod.

What

• New --watch-dp-size N (default 1, so single-rank engines are unchanged) carried as KubeDiscoveryConfig::dp_size.
• The watcher registers one listener per rank under the pod's instance: dp_rank r, tcp://<ip>:<zmq_port + r>, http://<ip>:<recover_port + r> (rank_endpoints).
• register() rejects a rank that is already present, so a pod whose registration fails part-way is deregistered before the retry instead of being left half-subscribed forever.
• Unit tests for the per-rank endpoints; docs.md section.

Rollout

i models -n deepseek-ai/DeepSeek-V4.1-Flash services update kv-indexer:{routing,reality,h24} \
    --container-image localhost:30500/dynamo-indexer:kvtest-<sha> --extra-arg=--watch-dp-size=2
i models -n deepseek-ai/DeepSeek-V4.1-Flash services deploy

Verify: /workers lists two listeners per pod (ports 5557/5558); a fresh prefill matches in the indexer regardless of the rank that ran it; the ladder returns to h24 >= perfect >= actual.

Follow-up (backend): derive --watch-dp-size from the model config's --data-parallel-size at services deploy so the next DP model does not repeat this.

🤖 Generated with Claude Code

[Shang-Pin]

Built and pushed localhost:30500/dynamo-indexer:kvtest-bb6dcfd5c5 from this branch (container/build-indexer-image.sh). cargo test -p dynamo-kv-router --features kube-discovery pod_watcher: 11 passed (4 new), no warnings in the touched files. Rollout to deepseek-ai/DeepSeek-V4.1-Flash's three indexers pending.

[Shang-Pin]

Rolled out to deepseek-ai/DeepSeek-V4.1-Flash kv-indexer:routing and kv-indexer:reality (kvtest-bb6dcfd5c5, --watch-dp-size=2). /workers now shows listeners on 5557 and 5558 for all six pods, all active. Fresh 50k-token prefills on one pod: 3/3 matched in the reality indexer (1/3 before). KV ladder over the first 10 min: reality_best 0.80, perfect 0.79, actual 0.75 (was reality 0.59 / perfect 0.63 / actual 0.78); h24 stays 0.59 until #40 lands, since its image comes from the h24 branch.