fix: skip RSA_MIN_PAD_SZ check for PSS padding in RsaPublicEncryptEx#10255
fix: skip RSA_MIN_PAD_SZ check for PSS padding in RsaPublicEncryptEx#10255MarkAtwood wants to merge 1 commit intowolfSSL:masterfrom
Conversation
The RSA_MIN_PAD_SZ guard (inLen > sz - 11) is a PKCS#1 v1.5 constraint. PSS has its own length check inside RsaPad_PSS (emLen >= hLen + sLen + 2 per RFC 8017) and does not need this guard. For keys in the range [hLen+2, hLen+10] bytes, the outer guard fires and returns RSA_BUFFER_E before RsaPad_PSS ever runs, even though PSS with saltLen=0 would be geometrically valid for those key sizes. Add a WC_RSA_PSS ifdef that skips the RSA_BUFFER_E return when pad_type == WC_RSA_PSS_PAD, mirroring the existing WC_RSA_NO_PADDING exception for raw (no-pad) mode.
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Adjusts RsaPublicEncryptEx so the RSA_MIN_PAD_SZ (PKCS#1 v1.5) input-length guard does not preempt PSS padding’s own size validation, enabling certain non-standard small key sizes to proceed to RsaPad_PSS.
Changes:
- Skips the
RSA_BUFFER_Eearly-return whenpad_type == WC_RSA_PSS_PAD(underWC_RSA_PSS). - Keeps the existing exception path for
WC_RSA_NO_PADbehavior.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| /* In the case that no padding is used the input length can and should | ||
| * be the same size as the RSA key. */ | ||
| if (pad_type != WC_RSA_NO_PAD) | ||
| #endif | ||
| #ifdef WC_RSA_PSS | ||
| /* PSS performs its own input-length check inside RsaPad_PSS; the | ||
| * RSA_MIN_PAD_SZ guard applies only to PKCS#1 v1.5 padding. */ | ||
| if (pad_type != WC_RSA_PSS_PAD) | ||
| #endif | ||
| return RSA_BUFFER_E; |
There was a problem hiding this comment.
This relies on an implicit nesting effect where the second if becomes the statement-body of the first if when both WC_RSA_NO_PADDING and WC_RSA_PSS are enabled. That’s fragile (any future added statement between them would change control flow) and hard to read. Make the intent explicit by restructuring (e.g., a single boolean/compound condition that returns RSA_BUFFER_E only when padding is neither WC_RSA_NO_PAD nor WC_RSA_PSS_PAD, or by using braces with an explicit combined condition under the relevant #ifdefs).
|
Summary
The
RSA_MIN_PAD_SZguard (inLen > sz - 11→RSA_BUFFER_E) is a PKCS#1 v1.5 concept. PSS has its own length check insideRsaPad_PSS(emLen >= hLen + sLen + 2per RFC 8017 §9.1.1) and the outer guard fires first, beforeRsaPad_PSSever runs.For keys in the range
[hLen+2, hLen+10]bytes, the outer guard incorrectly returnsRSA_BUFFER_Efor combinations where PSS withsaltLen=0would be geometrically valid. Keys in this range are non-standard but valid — they can be loaded from external DER.Fix: add a
WC_RSA_PSS#ifdefthat skips theRSA_BUFFER_Ereturn whenpad_type == WC_RSA_PSS_PAD, mirroring the existingWC_RSA_NO_PADDINGexception for raw mode.Test plan
/cc @wolfSSL-Fenrir-bot please review