Skip to content

feat: add GLM-5 SFT loss mask support#1824

Closed
stevewx wants to merge 1 commit intoTHUDM:mainfrom
stevewx:feat/glm5-sft-loss-mask
Closed

feat: add GLM-5 SFT loss mask support#1824
stevewx wants to merge 1 commit intoTHUDM:mainfrom
stevewx:feat/glm5-sft-loss-mask

Conversation

@stevewx
Copy link
Copy Markdown
Contributor

@stevewx stevewx commented Apr 10, 2026

Summary

  • Add --loss-mask-type glm5 for multi-turn SFT on GLM-5 models
  • GLM-5 uses role-token delimiters (<|assistant|>, <|user|>, etc.) with no closing tags, unlike Qwen's ChatML <|im_start|>/<|im_end|> format
  • Handles both </think> (non-thinking) and <think>...</think> (thinking) modes
  • Supports step_loss_mask per-turn control and tool-calling via <|observation|> role

Changes

  • slime/utils/mask_utils.py — add gen_multi_turn_loss_mask_glm5() method using char-level offset mapping
  • slime/utils/arguments.py — add "glm5" to --loss-mask-type choices
  • tests/utils/test_loss_mask_type_glm5.py — 7 unit tests covering single-turn, multi-turn, step_loss_mask, tool calling, tool schemas, no-system-message, and length consistency

Test plan

  • 7 unit tests pass locally (pytest tests/utils/test_loss_mask_type_glm5.py)
  • Existing Qwen3.5 tests unaffected (pytest tests/utils/test_loss_mask_type_qwen35.py)
  • Verified against real GLM-5 tokenizer (zai-org/GLM-5)
  • Tested GLM-5 SFT training with OpenHermes-2.5

@stevewx stevewx marked this pull request as ready for review April 10, 2026 01:06
@stevewx stevewx mentioned this pull request Apr 20, 2026
@stevewx
Copy link
Copy Markdown
Contributor Author

stevewx commented Apr 21, 2026

closed in favor of #1844

@stevewx stevewx closed this Apr 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant