Skip to content

Add label-wise accuracy support for multilabel Accuracy metric - #3820

Open
danijimmy19 wants to merge 1 commit into
pytorch:masterfrom
danijimmy19:feature/labelwise-multi-label-metrics
Open

Add label-wise accuracy support for multilabel Accuracy metric#3820
danijimmy19 wants to merge 1 commit into
pytorch:masterfrom
danijimmy19:feature/labelwise-multi-label-metrics

Conversation

@danijimmy19

@danijimmy19 danijimmy19 commented Aug 7, 2026

Copy link
Copy Markdown

Description

Fixes #513.

Accuracy currently only supports subset accuracy (exact match across all labels) for multilabel classification — it collapses all C labels per sample with torch.all(..., dim=-1). This hides which specific labels a model is getting wrong, unlike Precision/Recall, which already support per-label output via average=False.

This PR adds an average="label-wise" option to Accuracy for is_multilabel=True, returning a (C,) tensor of per-label accuracy instead of a single scalar.

Root cause

Accuracy's _num_correct is a scalar with no average dispatch, unlike _BasePrecisionRecall, which was given tensor accumulators and dispatch when Precision/Recall were added in the same commit multilabel support landed in Accuracy (1a8ead8b). Accuracy was never brought to parity.

Change

In update(), when average="label-wise":

# Before: subset accuracy — all C labels must match per sample
correct = torch.all(y == y_pred.type_as(y), dim=-1)  # (N,) bool

# After: per-label accuracy — each column independently
correct_per_label = (y == y_pred.type_as(y)).to(dtype=torch.float64)  # (N, C)
self._num_correct = self._num_correct + correct_per_label.sum(dim=0)  # (C,)

compute() returns a (C,) tensor for average="label-wise", and preserves the existing scalar-float behavior otherwise.

Example

# y_pred / y_true: 5 samples, 3 labels

# Before — Accuracy(is_multilabel=True)
# Result: 0.4  (only 2/5 samples have all 3 labels correct)

# After — Accuracy(is_multilabel=True, average="label-wise")
# Result: [1.0, 0.8, 0.6]  — pinpoints that label 2 is the weakest

Edge cases handled

  • average="label-wise" with is_multilabel=False raises ValueError in __init__
  • Multi-batch updates with different batch sizes accumulate correctly
  • Spatial (N, C, H, W) multilabel inputs are flattened to (N*H*W, C) before the per-label sum
  • Distributed training: tensor _num_correct of shape (C,) is handled correctly by the existing @sync_all_reduce decorator
  • compute() before any update() raises NotComputableError via the existing guard

Testing

Added tests to tests/ignite/metrics/test_accuracy.py covering:

  • output shape (C,), all-correct/all-wrong cases
  • multi-batch consistency (matches single-batch result)
  • correctness against sklearn.metrics.accuracy_score computed per column
  • fallback behavior (average=None) unchanged
  • reset/epoch boundary behavior
  • invalid configuration (average="label-wise" + is_multilabel=False)

Check list

  • New tests are added (if a new feature is added)
  • New doc strings: description and/or example code are in RST format
  • Documentation is updated (if required)

@github-actions github-actions Bot added the module: metrics Metrics module label Aug 7, 2026
@danijimmy19

Copy link
Copy Markdown
Author

Hi @vfdev-5, this is my first contribution to this project. I've implemented label-wise accuracy support for issue #513. While working on this, I noticed PR #3810 is also open against the same issue with a similar approach, happy to differentiate my implementation or defer to that PR, whichever you'd prefer. Would appreciate a review when you have time!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

module: metrics Metrics module

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Label-wise metrics (Accuracy etc.) for multi-label problems

1 participant