Skip to content

Add SwanLab training monitor#1256

Open
Lubenwei-nb123 wants to merge 3 commits into
ModelTC:mainfrom
Lubenwei-nb123:add_remote_logger
Open

Add SwanLab training monitor#1256
Lubenwei-nb123 wants to merge 3 commits into
ModelTC:mainfrom
Lubenwei-nb123:add_remote_logger

Conversation

@Lubenwei-nb123

Copy link
Copy Markdown
Contributor

add swanlab key in yaml to activate monitor

logging:
    rank_zero_only: true
    train_log_every_iters: 10
    infer_log_every_steps: 10
    swanlab:
        enable: true
        project: xxx
        name: xxx
        api_key: your_api_key
image

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new monitoring module supporting SwanLab logging and integrates it across the base and specific trainers (DMD, DoPSD, Flow) to log training metrics. The feedback highlights two key improvements: first, wrapping the .item() call on metrics in a try-except block to prevent crashes when dealing with multi-element PyTorch tensors; second, making the SwanLab API key optional in the configuration to avoid security risks associated with hardcoded secrets and to support environment-based authentication.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment on lines +63 to +66
for key, value in metrics.items():
if hasattr(value, "item"):
value = value.item()
values[key] = value

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Any PyTorch Tensor (regardless of its shape or number of elements) has the item() method. If a multi-element tensor is passed in metrics, calling value.item() will raise a ValueError: only one element tensors can be converted to Python scalars, which would crash the training loop.

To make the logging robust and prevent unexpected crashes, we should wrap the .item() call in a try-except block or check if the tensor has only a single element.

Suggested change
for key, value in metrics.items():
if hasattr(value, "item"):
value = value.item()
values[key] = value
for key, value in metrics.items():
if hasattr(value, "item"):
try:
value = value.item()
except (ValueError, RuntimeError):
pass
values[key] = value

Comment thread lightx2v_train/lightx2v_train/runtime/monitor.py
Lubenwei-nb123 and others added 2 commits July 15, 2026 21:32
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant