PERF: bypass factorize in axis=1 EA reduction fastpath (GH-56903)#65597
Open
jbrockmendel wants to merge 2 commits into
Open
PERF: bypass factorize in axis=1 EA reduction fastpath (GH-56903)#65597jbrockmendel wants to merge 2 commits into
jbrockmendel wants to merge 2 commits into
Conversation
The EA-backed axis=1 reduction in DataFrame._reduce built a synthetic ser.groupby(row_index).agg(...), which re-factorized codes that were already in factorized form. Call the EA's _groupby_op directly with the pre-known ngroups/codes instead, skipping factorize and the surrounding groupby pipeline. ~2-3x faster on axis=1 reductions for nullable/Arrow extension dtypes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
axis=1reduction fastpath inDataFrame._reduceflattens the frame into a 1D EA and groups by row indices to simulate a transpose+reduce. The row indices passed toSeries.groupbywere already in factorized form (np.tile(np.arange(nrows), ncols)), but the standard groupby pipeline re-factorized them.arr._groupby_opdirectly with the pre-knownngroupsandids, skipping the redundantfactorizeand the surrounding groupby pipeline.idxmin/idxmaxget an inline conversion from flat positions to column positions plus the sameskipna=FalseNA-detection thatGroupBy._idxmax_idxminwas raising before.Benchmark (Apple M1 Pro)
Small-frame numbers (10k x 10, BooleanDtype) drop from ~1.1 ms to ~0.35 ms across
any/all/sum/min/max/mean.cProfile on the 250k x 100 case showed
factorizeat 43% of wall time pre-change; that line item is gone post-change.Test plan
pandas/tests/reductionspandas/tests/frame/test_reductions.pypandas/tests/extensionpandas/tests/arrays/maskedpandas/tests/groupby/aggregatepre-commit runon changed filesmypy/pyrightonpandas/core/frame.py🤖 Generated with Claude Code