Support python-asn1 3.x - #63
Open
Bad3r wants to merge 4 commits into
Open
Conversation
python-asn1 3.0.0 changed behavior the parser relies on (andrivet/python-asn1#297): Decoder.read() now recursively decodes constructed elements into lists instead of returning their raw content octets, Encoder.write() with Types.Constructed re-encodes each item of the value instead of emitting pre-encoded octets verbatim, Decoder.eof() is relative to the whole input instead of the entered container, and Decoder.leave() no longer skips unread elements. With asn1 3.x installed, every parse failed with "asn1.core.Error: Expecting bytes or a subclass of io.RawIOBase or BufferedIOBase. Get <class 'list'> instead." Image4 handling requires byte-exact round-trips of nested DER (signed manifest bodies, X.509 certificates), and re-encoding decoded values is lossy (IA5String decodes to str and re-encodes as PrintableString), so decoded lists cannot replace raw octets. pyimg4/_asn1compat.py restores the 2.x raw-octet semantics on both majors: a behavioral probe at import distinguishes the installed API, the 2.x branch keeps the public calls, and the 3.x branch replicates read() and write() up to value decoding through python-asn1 internals whose presence is checked at import time so an incompatible future release fails loudly instead of producing corrupt output. Addresses m1stadev#62. Validation: uv run pytest (12 passed) with asn1 3.3.0 and 2.8.0; offline tests (7 passed) with asn1 2.7.0.
The asn1<3.0.0 pin was added in m1stadev#59 as a stopgap for the 3.0.0 API break, which pyimg4/_asn1compat.py now handles on both majors. Distributions that only package the latest python-asn1 (such as Arch Linux, currently on 3.3.0) could not satisfy the pin, which is what m1stadev#62 reports. No new upper bound is added: the compat layer fails at import with a clear error on an unsupported future release instead of resolving to a broken install. uv.lock is deliberately untouched. It already records asn1 as ">=2.7.0" because it was never regenerated after m1stadev#59, so dropping the pin makes the lock match pyproject.toml again, and the locked asn1 2.8.0 still satisfies the constraint. Regenerating it here would only bump the lockfile revision. CI installs each major explicitly instead. Validation: pytest with asn1 3.3.0 and 2.8.0.
pyimg4/_asn1compat.py has one branch per python-asn1 major: the 2.x branch uses the public raw-octet API and the 3.x branch replicates it through python-asn1 internals, so a regression in either can only be caught by running the suite against both. Constrain asn1 with uv pip install after uv sync, and run pytest with --no-sync so the override is not reverted by the locked version.
pyimg4/_asn1compat.py is the only place that depends on raw-octet reads and writes, and on 3.x it reaches into python-asn1 internals to get them, so a silent behavior change there would corrupt output rather than raise. On asn1 3.3.0 the unwrapped calls return [1] where the parser needs b"\x02\x01\x01", encode 3009020102020101020101 where it needs 3003020101, and report eof() False at the end of an entered container. These assert the three invariants directly on a two-level SEQUENCE, so they run identically on both majors and fail on a future release that changes them again, without needing an Image4 fixture to trip over the difference. Validation: pytest tests/test_asn1compat.py with asn1 3.3.0 and 2.8.0 (6 passed each).
Merged
9 tasks
Bad3r
added a commit
to Bad3r/nixpkgs
that referenced
this pull request
Aug 24, 2026
* python3Packages.asn1_2: init at 2.8.0 python-asn1 3.0.0 changed Decoder.read() to recursively decode constructed elements into lists instead of returning their raw content octets, Encoder.write() with Types.Constructed to re-encode each item of the value instead of emitting pre-encoded octets verbatim, and Decoder.eof() to be relative to the whole input instead of the entered container. Consumers written against the 2.x raw-octet API cannot be pointed at 3.x unchanged. pyimg4 is the only consumer in tree: it pins asn1<3.0.0 upstream (m1stadev/PyIMG4#59) and has been marked broken since asn1 was updated to 3.x. asn1 itself stays on 3.3.0 as the default. Validation: nix-build -A python3Packages.asn1_2 (upstream tests/test_asn1.py pass on 3.12, 3.13, 3.14). * python3Packages.pyimg4: unbreak by building against asn1_2 Marked broken in ef300f4 because asn1 is 3.3.0 and pyimg4 pins asn1<3.0.0. The pin is not stale metadata: with the constraint relaxed, 5 of 7 tests fail with "asn1.core.Error: Expecting bytes or a subclass of io.RawIOBase or BufferedIOBase. Get <class list> instead" and "UnexpectedTagError: Expected tag of type IA5String, got PrintableString", because Decoder.read() no longer returns the raw content octets that parser.py re-parses and re-emits. Upstream declined the port in m1stadev/PyIMG4#59 ("pyimg4 should be refactored around it first") and has not released since v0.8.8, so build against asn1_2 rather than patch. This unblocks ipsw-parser and pymobiledevice3, which fail to evaluate through pyimg4. A port is proposed in m1stadev/PyIMG4#63; once released, this can move back to asn1. Validation: nix-build -A python3Packages.pyimg4 (7 passed, 5 deselected), -A python3Packages.ipsw-parser, -A python3Packages.pymobiledevice3 (56 passed, 1 xfailed); nix-store -qR on the pymobiledevice3 output shows asn1 2.8.0 only, no collision with 3.3.0.
Bad3r
added a commit
to Bad3r/nixpkgs
that referenced
this pull request
Aug 24, 2026
Marked broken in ef300f4 because asn1 is 3.3.0 and pyimg4 pins asn1<3.0.0. The pin is not stale metadata: with the constraint relaxed, 5 of 7 tests fail with "asn1.core.Error: Expecting bytes or a subclass of io.RawIOBase or BufferedIOBase. Get <class list> instead" and "UnexpectedTagError: Expected tag of type IA5String, got PrintableString", because Decoder.read() no longer returns the raw content octets that parser.py re-parses and re-emits. Upstream declined the port in m1stadev/PyIMG4#59 ("pyimg4 should be refactored around it first") and has not released since v0.8.8, so build against asn1_2 rather than patch. This unblocks ipsw-parser and pymobiledevice3, which fail to evaluate through pyimg4. A port is proposed in m1stadev/PyIMG4#63; once released, this can move back to asn1. Validation: nix-build -A python3Packages.pyimg4 (7 passed, 5 deselected), -A python3Packages.ipsw-parser, -A python3Packages.pymobiledevice3 (56 passed, 1 xfailed); nix-store -qR on the pymobiledevice3 output shows asn1 2.8.0 only, no collision with 3.3.0. Assisted-by: Claude Code (claude-fable-5)
Bad3r
added a commit
to Bad3r/nixpkgs
that referenced
this pull request
Aug 24, 2026
Marked broken in ef300f4 because asn1 is 3.3.0 and pyimg4 pins asn1<3.0.0. The pin is not stale metadata: with the constraint relaxed, 5 of 7 tests fail with "asn1.core.Error: Expecting bytes or a subclass of io.RawIOBase or BufferedIOBase. Get <class list> instead" and "UnexpectedTagError: Expected tag of type IA5String, got PrintableString", because Decoder.read() no longer returns the raw content octets that parser.py re-parses and re-emits. Upstream declined the port in m1stadev/PyIMG4#59 ("pyimg4 should be refactored around it first") and has not released since v0.8.8, so build against asn1_2 rather than patch. This unblocks ipsw-parser and pymobiledevice3, which fail to evaluate through pyimg4. A port is proposed in m1stadev/PyIMG4#63; once released, this can move back to asn1. Validation: nix-build -A python3Packages.pyimg4 (7 passed, 5 deselected), -A python3Packages.ipsw-parser, -A python3Packages.pymobiledevice3 (56 passed, 1 xfailed); nix-store -qR on the pymobiledevice3 output shows asn1 2.8.0 only, no collision with 3.3.0. Assisted-by: Claude Code (claude-opus-5)
13 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #62.
python-asn1 3.0.0 changed four behaviors
parser.pydepends on (andrivet/python-asn1#297). With asn1 3.x installed, every parse fails immediately:The pin added in #59 keeps that from happening, but distributions that only package the latest python-asn1 (Arch, and nixpkgs, which is on 3.3.0) cannot satisfy it, so PyIMG4 is unavailable there and every downstream package with it.
What changed in python-asn1 3.0
On a two-level
SEQUENCE(30 08 30 03 02 01 01 02 01 02), against 3.3.0:Decoder.read()on the innerSEQUENCEb'\x02\x01\x01'[1]Encoder.write(b'\x02\x01\x01', Numbers.Sequence, Types.Constructed, ...)30030201013009020102020101020101Decoder.eof()at the end of an entered containerTrueFalseDecoder.leave()with unread elementsApproach
pyimg4/_asn1compat.pyrestores the 2.x raw-octet semantics on both majors, andparser.pycalls it at the 30-odd sites that relied on them. A behavioral probe at import picks the branch, so no version parsing and no new pin.The 3.x public API cannot read or emit raw pre-encoded content, so that branch replicates
read()andwrite()up to value decoding using python-asn1 internals. Their presence is asserted at import, so an incompatible future release raisesImportErrorwith a pointer to the issue tracker rather than silently emitting corrupt Image4 files.If you would rather not depend on those internals, the alternative is restructuring the parsers to walk containers in place with
enter()/leave()instead of re-parsing raw sub-slices. That is a much larger change, and it still would not coverIM4M.certificates, which has to be stored and re-emitted verbatim.uv.lockis deliberately untouched. It already recordsasn1 >= 2.7.0because it was never regenerated after #59.Verification
Tests pass on both majors, and CI now runs the matrix:
More importantly, this is not just "the tests are green". I dumped parsed structure plus SHA-256 of every re-encoded artifact for all four fixtures and compared three runs: this branch on asn1 3.3.0, this branch on asn1 2.8.0, and unmodified v0.8.8 on asn1 2.8.0. All three are byte-identical, including
IM4M.signature,IM4M.certificates, each of the 35 manifest images, and fullIM4M/IM4P/IM4R/IMG4round-trips.CLI round-trip on asn1 3.3.0 is byte-exact too:
tests/test_asn1compat.pypins the three invariants directly, so a future python-asn1 that changes them again fails the suite instead of the output. Lint and format are clean under the pinned ruff 0.9.7.The network-dependent tests (
lzss,lzfse,payp) were not run; they download full IPSWs. They exercise compression paths that this change does not touch.