[SPARK-56452][PYTHON] Fix pip install failure for prerelease versions#55311
Closed
kiyeonjeon21 wants to merge 3 commits intoapache:masterfrom
Closed
[SPARK-56452][PYTHON] Fix pip install failure for prerelease versions#55311kiyeonjeon21 wants to merge 3 commits intoapache:masterfrom
kiyeonjeon21 wants to merge 3 commits intoapache:masterfrom
Conversation
The version validation regex in install.py rejects prerelease version strings like "4.2.0.dev4" because it only matches X.Y.Z format. This adds an optional ".devN" suffix to both regexes in checked_versions() and convert_old_hadoop_version().
HyukjinKwon
approved these changes
Apr 12, 2026
Member
HyukjinKwon
left a comment
HyukjinKwon
left a comment
There was a problem hiding this comment.
Looks good but let's make the CI green to make sure.
Contributor
|
@HyukjinKwon do we plan to release any other pre-release versions in the future? For python semantics its |
Member
|
maybe but we will likely stop preview releases for now cuz we're going to frequently release |
Contributor
|
Yeah I'm asking because if we plan to do other pre-releases in the future we might just solve all of them together in this PR. If we won't do that, we can just detect |
Member
|
Merged to master. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Updated the version validation regexes in
python/pyspark/install.pyto accept prerelease version strings (e.g.4.2.0.dev4).Two regexes were updated:
checked_versions()(line 71): the regex that detects bare version strings and prependsspark-convert_old_hadoop_version()(line 109): the regex that extracts major/minor version partsBoth now include an optional
(?:\.dev[0-9]+)?suffix.Why are the changes needed?
When installing a PySpark prerelease via pip (e.g.
pip install pyspark==4.2.0.dev4), the version string4.2.0.dev4does not match the existing regex^[0-9]+\.[0-9]+\.[0-9]+$. This causes thespark-prefix to not be added, which then triggers aRuntimeError:GitHub issue: #55289
Does this PR introduce any user-facing change?
Yes. Previously,
pip install pyspark==4.2.0.dev4would fail with aRuntimeError. After this fix, prerelease versions install correctly.How was this patch tested?
Added two test cases in
test_checked_versionsfor prerelease versions:checked_versions("4.2.0.dev4", "3", "2.3")— bare version with dev suffixchecked_versions("spark-4.2.0.dev4", "hadoop3", "hive2.3")— already prefixed version with dev suffixWas this patch authored or co-authored using generative AI tooling?
No.