Repository navigation
Fix lossy str->bytes conversion that corrupts non-ASCII input - #78
Merged
Merged
Conversation
parse_email's str path mapped each code point to its low byte (chars().map(|c| c as u8)), truncating non-ASCII code points and silently corrupting the bytes handed to the parser (e.g. 'é' U+00E9 -> 0xE9, '日' U+65E5 -> 0xE5). Decode str losslessly as its UTF-8 bytes via text.as_bytes().to_vec() for all strings. ASCII is unaffected (ASCII == its own UTF-8); the bytes path and TypeError fallback are unchanged. Closes #19 Signed-off-by: yuriyryabikov <22548029+kurok@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
parse_emailacceptsstrorbytes. Forstrinput,to_bytes()insrc/fast_mail_parser.rsmapped each code point to its low byte (chars().map(|c| c as u8), with anis_ascii()fast path). For non-ASCII input this truncated code points (e.g. 'é' U+00E9 -> 0xE9, '日' U+65E5 -> 0xE5), silently corrupting the bytes handed to the parser.This fix decodes
strlosslessly as its UTF-8 bytes viatext.as_bytes().to_vec()for all strings, dropping both theis_ascii()branch and the lossy per-char cast. Thebytespath and theTypeErrorfallback are unchanged.Behavior change
This changes output for non-ASCII
strinput (now lossless UTF-8 instead of truncated). It is a correctness fix for a corruption bug; ASCII input andbytesinput are unchanged. Consumers should be unaffected unless they relied on the (buggy) truncation.Tests
Reworked
tests/test_contract.py::test__non_ascii_str_decodes_as_truncated_code_points->test__non_ascii_str_decodes_as_utf8to assert the new correct behavior: a non-ASCIIstrparses identically to its explicit UTF-8 encoding.test__accepts_str_and_bytes_equivalentlystill passes. Full suite: 70 passed.Closes #19