0121 maxsim support

[FarmersWrap]

1. fully random chunking, passage level chunking for training.
2. encode chunks for evaluation.

For training, train

1. --passage_chunk_size 256 # fixed size chunk, will be disable when set 0.
2. --passage_chunk_size_range 32,256 # fixed size chunk within a passage, but a number selected btw 32-256
3. --passage_chunk_size_range 32,256
   --passage_chunk_size_variable # fully random chunks selected btw 32-256
4. --encode_use_pre_chunked # Not fully tested. use dataset with (doc_id, chunks) to train

For evaluation encoding, encode

1. fixed chunk size, 0 means no chunk
   --passage_chunk_size 0
2. pick a random size btw 32-64 tokens to chunk a passage
   --passage_chunk_size_range "32,64" \
3. need a prechunked jsonl
   --dataset_name json
   --dataset_path "${prechunked_corpus_jsonl}"
   --dataset_split train
   --encode_use_pre_chunked

prechunked.jsonl: {"docid": "q-en-0-pos-0", "chunks": ["a", "b", "c"]}

[MXueguang]

Thanks for the contribution — chunked passage encoding + MaxSim aggregation is a feature we'd genuinely like to have in Tevatron. That said, the branch in its current form is a research-experiment snapshot rather than a mergeable change, so I'm requesting changes. Main points:

Files that shouldn't be in the PR

• req.txt is a personal pip freeze, including -e git+ssh:// references to private forks — please remove.
• finetune.sh, finetune_with_chunk.sh, run_retrieval*.sh are personal experiment scripts (hardcoded output dirs etc.). If you want to keep runnable recipes, move a cleaned-up version under examples/.

Changes that break the default (non-chunked) path for all users

• driver/train.py / driver/encode.py: the unconditional tokenizer.eos_token_id = tokenizer.pad_token_id mutates EOS for every run. In train.py it's placed before the pad_token_id is None fallback, so models like Llama/Mistral end up with pad = eos = None, breaking --append_eos_token. This only happens to work for Qwen (pad == eos). Please drop both lines and derive the chunk separator from the tokenizer instead.
• Debug leftovers that fire on every step for everyone: print(target...) in EncoderModel.forward, print(eos_positions...) in encode_passage, the selected_negatives: Randomly selected!!! print in dataset.py, print(encoded) dumping full embedding arrays in encode.py, module-level torch.set_printoptions(threshold=inf) and from rich import print in collator.py (rich isn't a declared dependency), and the if True: Python double-loop in compute_maxsim_similarity that logs every Q×P pair per forward.
• EOS_TOKEN_ID = 151643 is hardcoded (Qwen vocab) and asserted in encode_passage — chunked mode crashes on any non-Qwen model.

Correctness within the feature

• Under DDP, q_reps/p_reps are dist-gathered but chunk_mask is not, so compute_maxsim_similarity sees p_reps of size P × world_size vs a chunk_mask of size P — multi-GPU chunked training is broken.
• Passing eos_positions via attribute mutation (model.eos_positions = ... in the trainer) is fragile; please thread it through forward() kwargs.

Staleness

• The branch is based on an October base and conflicts with the Megatron/SPLADE merge (84433fb): collator.py has two real conflicts, and a naive resolution would revert main's lazy multimodal imports and the add_special_tokens=self.data_args.add_special_tokens behavior.

Given the drift, I'd suggest re-implementing the (reasonably contained) core logic on top of current main rather than resolving conflicts in place, keeping the feature strictly additive: with chunking flags off, behavior must be byte-identical to main. The test suite here is a good start — please keep it, but make it hermetic (no Hub downloads in fixtures) and trim the duplicated golden cases.

Also note #215 appears to be an evolved superset of this PR — if that's the intended vehicle, let's close this one and consolidate the rework (including these tests) there.

[FarmersWrap]

Closing as superseded by #215, which is now the single vehicle for the MPE / chunked-MaxSim feature.