Fix NULL derefs, buffer overflow, and i2d contract in EVP/OCSP/X509

[ColtonWilley]

Summary

Harden OpenSSL compatibility layer against NULL pointers, negative lengths,
and buffer overflows across EVP, OCSP, and X509 APIs. Fix DSA SignFinal
write-before-check overflow, add missing i2d_OCSP_RESPONSE allocation path,
and fix unaligned keyUsage access.

Test plan

• Existing CI passes
• OCSP single_get0_status test updated for new -1 return on NULL

[mattia-moffa]

Jenkins retest this please (Peer closed underlying transport)

[github-actions]

MemBrowse Memory Report

No memory changes detected for:

• gcc-arm-cortex-m4- gcc-arm-cortex-m4-baremetal- gcc-arm-cortex-m4-min-ecc- gcc-arm-cortex-m4-tls12

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #10217

Scan targets checked: wolfcrypt-api_misuse, wolfcrypt-bugs, wolfcrypt-compliance, wolfcrypt-concurrency, wolfcrypt-consttime, wolfcrypt-defaults, wolfcrypt-mutation, wolfcrypt-portability, wolfcrypt-proptest, wolfcrypt-src, wolfcrypt-zeroize, wolfssl-bugs, wolfssl-compliance, wolfssl-consttime, wolfssl-defaults, wolfssl-mutation, wolfssl-proptest, wolfssl-src, wolfssl-zeroize

No new issues found in the changed files. ✅

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Hardens the OpenSSL compatibility layer against NULL pointers, negative lengths, and overflow/contract issues across EVP, OCSP, and X509-related APIs.

Changes:

• Added input validation to prevent NULL dereferences and negative-length misuse in multiple EVP/OCSP/X509 entry points.
• Fixed a DSA SignFinal write-before-size-check overflow and improved safety around digest/sign/verify flows.
• Implemented/adjusted OCSP/X509 encoding and parsing behaviors (including i2d_OCSP_RESPONSE allocation path and unaligned keyUsage access).

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
wolfcrypt/src/pwdbased.c	Adds an overflow guard when combining PBKDF key/IV lengths.
wolfcrypt/src/evp.c	Adds NULL/negative length guards, fixes DSA SignFinal overflow ordering, and hardens digest sign/verify paths.
tests/api.c	Updates OCSP status test to expect -1 on NULL input.
src/x509.c	Fixes unaligned keyUsage access and adds serial number size/data validation.
src/ssl.c	Hardens wolfSSL_OCSP_parse_url by validating required input/output pointers.
src/ocsp.c	Adds NULL checks, adjusts return behavior for single status, and implements i2d_OCSP_RESPONSE allocation/increment behavior.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.