Skip to content

python3Packages.{pyimg4,smart-meter-texas}: fix against asn1 3.x by adding asn1_2 - #555937

Draft
Bad3r wants to merge 3 commits into
NixOS:masterfrom
Bad3r:python-pyimg4-unbreak-asn1-master
Draft

python3Packages.{pyimg4,smart-meter-texas}: fix against asn1 3.x by adding asn1_2#555937
Bad3r wants to merge 3 commits into
NixOS:masterfrom
Bad3r:python-pyimg4-unbreak-asn1-master

Conversation

@Bad3r

@Bad3r Bad3r commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

python3Packages.pyimg4 has been marked broken since ef300f4, which set broken = lib.versionAtLeast asn1.version "3". Since asn1 is at 3.3.0, the guard always fires and takes ipsw-parser and pymobiledevice3 down with it:

$ nix shell nixpkgs#python3Packages.pymobiledevice3
error: Refusing to evaluate package 'python3.14-pyimg4-0.8.8' ... because it has problems:
       - broken: This package is broken.
note: trace involved the following derivations:
       derivation 'python3.14-pymobiledevice3-7.7.0'
       derivation 'python3.14-ipsw-parser-1.5.0'

The pin is not stale metadata. With the constraint relaxed and broken dropped, 5 of 7 tests fail against asn1 3.3.0:

FAILED tests/test_im4m.py::test_im4m  - asn1.core.Error: Expecting bytes or a subclass of io.RawIOBase or BufferedI...
FAILED tests/test_im4r.py::test_read  - asn1.core.Error: Expecting bytes or a subclass of io.RawIOBase or BufferedI...
FAILED tests/test_img4.py::test_read  - pyimg4.errors.UnexpectedTagError: Expected tag of type IA5String, got Print...

python-asn1 3.0 changed Decoder.read() to recursively decode constructed elements into nested lists instead of returning their raw content octets, which is what parser.py re-parses and re-emits throughout. It also changed Encoder.write(..., Types.Constructed) to iterate the value rather than emit pre-encoded octets verbatim, so pre-encoded DER silently becomes one Integer per byte. Upstream declined the port in m1stadev/PyIMG4#59 ("Version 3.0.0 is technically much better, but pyimg4 should be refactored around it first to support it") and has not released since v0.8.8 in April 2025.

So this adds asn1_2 at 2.8.0, the last 2.x, and builds pyimg4 against it, following the existing convention for versioned Python attributes (cython_0, chardet_5, django_5). asn1 stays at 3.3.0 as the default. No pythonRelaxDeps entry is needed, since 2.8.0 satisfies upstream's asn1>=2.7.0,<3.0.0 directly.

smart-meter-texas is the only other Nixpkgs consumer of python3Packages.asn1, and it is broken on 3.x for the same root cause: _find_ca_issuers_uri() recurses through Decoder.enter()/leave() while looping on Decoder.eof(), which 3.0 made relative to the whole input instead of the entered container. On a two-entry authorityInfoAccess (the common OCSP-then-caIssuers layout) it raises AttributeError: 'NoneType' object has no attribute 'typ' on 3.3.0 and returns the URI on 2.8.0. It goes unnoticed because upstream ships no tests and get_ca_issuers_uri() swallows the exception. The third commit moves it to asn1_2 too; details and the reproduction are in a comment below.

That leaves asn1 at 3.3.0 with no in-tree consumer, so downgrading it outright and dropping asn1_2 would be a smaller diff and breaks nothing in the tree. I avoided that because it would silently hand 2.x to out-of-tree users who asked for the current release. Glad to switch if you prefer it.

I have also proposed the actual port upstream in m1stadev/PyIMG4#63. Once that lands and is released, pyimg4 can move back to asn1 and asn1_2 can be dropped. Happy to go the other way instead and patch pyimg4 with that diff here if maintainers prefer not to carry a second asn1.

Verification

nix-build -A python3Packages.pyimg4           # 7 passed, 5 deselected
nix-build -A python3Packages.ipsw-parser      # ok
nix-build -A python3Packages.pymobiledevice3  # 56 passed, 113 deselected, 1 xfailed
nix-build -A python3Packages.asn1_2           # upstream tests/test_asn1.py pass
nix-instantiate -A python3Packages.{asn1_2,pyimg4,ipsw-parser,ldap3} \
                -A python312Packages.pyimg4 -A python313Packages.asn1_2

nix-store -qR on the pymobiledevice3 output contains python3.14-asn1-2.8.0 and no 3.3.0, so there is no module collision in the closure.

The affected set is closed and was built in full by hand: asn1_2 is new and only pyimg4 consumes it, pyimg4's only reverse dependency is ipsw-parser, whose only reverse dependency is pymobiledevice3, which nothing else in tree depends on. All four are built above. nixpkgs-review pr 555937 has since confirmed exactly that set: 8 packages built (asn1_2, pyimg4, ipsw-parser, pymobiledevice3 on python313 and python314), 0 failures. Report posted as a comment below.

Automation/AI disclosure

Per CONTRIBUTING.md: both commits carry an Assisted-by: Claude Code (claude-opus-5) trailer, and this pull request summary was likewise drafted with that tooling. I have reviewed the diff and the reasoning, and verified the outcome independently by building every affected package and inspecting the runtime closure, as listed above.

Things done

Bad3r added 2 commits August 24, 2026 07:07
python-asn1 3.0.0 changed Decoder.read() to recursively decode constructed elements into lists instead of returning
their raw content octets, Encoder.write() with Types.Constructed to re-encode each item of the value instead of
emitting pre-encoded octets verbatim, and Decoder.eof() to be relative to the whole input instead of the entered
container. Consumers written against the 2.x raw-octet API cannot be pointed at 3.x unchanged.

pyimg4 is the only consumer in tree: it pins asn1<3.0.0 upstream (m1stadev/PyIMG4#59) and has been marked broken since
asn1 was updated to 3.x. asn1 itself stays on 3.3.0 as the default.

Validation: nix-build -A python3Packages.asn1_2 (upstream tests/test_asn1.py pass on 3.12, 3.13, 3.14).

Assisted-by: Claude Code (claude-opus-5)
Marked broken in ef300f4 because asn1 is 3.3.0 and pyimg4 pins asn1<3.0.0. The pin is not stale metadata: with the
constraint relaxed, 5 of 7 tests fail with "asn1.core.Error: Expecting bytes or a subclass of io.RawIOBase or
BufferedIOBase. Get <class list> instead" and "UnexpectedTagError: Expected tag of type IA5String, got
PrintableString", because Decoder.read() no longer returns the raw content octets that parser.py re-parses and
re-emits.

Upstream declined the port in m1stadev/PyIMG4#59 ("pyimg4 should be refactored around it first") and has not released
since v0.8.8, so build against asn1_2 rather than patch. This unblocks ipsw-parser and pymobiledevice3, which fail to
evaluate through pyimg4. A port is proposed in m1stadev/PyIMG4#63; once released, this can move back to asn1.

Validation: nix-build -A python3Packages.pyimg4 (7 passed, 5 deselected), -A python3Packages.ipsw-parser,
-A python3Packages.pymobiledevice3 (56 passed, 1 xfailed); nix-store -qR on the pymobiledevice3 output shows asn1 2.8.0
only, no collision with 3.3.0.

Assisted-by: Claude Code (claude-opus-5)
@Bad3r
Bad3r requested a review from dotlambda August 24, 2026 04:11
@nixpkgs-ci nixpkgs-ci Bot added 8.has: package (new) This PR adds a new package 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. llm-assisted Contributions that include code generation with LLMs 6.topic: python Python is a high-level, general-purpose programming language. labels Aug 24, 2026
@Bad3r

This comment was marked as outdated.

@Bad3r

This comment was marked as outdated.

ClientSSLContext._find_ca_issuers_uri() walks an X.509 authorityInfoAccess extension by recursing through
Decoder.enter()/leave() while looping on Decoder.eof(). asn1 3.0.0 made eof() relative to the whole input rather than
the entered container, so the loop runs past the end of a container, peek() returns None, and the next tag.typ access
raises AttributeError.

It only shows up when authorityInfoAccess holds more than one AccessDescription. That is the common real-world layout
(OCSP first, then caIssuers), and it is why this was not caught: doCheck is false because upstream ships no tests, and
get_ca_issuers_uri() wraps the call in "except Exception", so on asn1 3.x the failure is swallowed and logged as
"Failed to lookup CA Issuers URI value" while the function silently returns None.

Reproduced on a hand-built DER authorityInfoAccess with two AccessDescriptions: asn1 2.8.0 returns the caIssuers URI,
asn1 3.3.0 raises "AttributeError: NoneType object has no attribute typ". Upstream requires asn1>=2.4.1, so 2.8.0
satisfies it without relaxing anything.

Validation: nix-build -A python3Packages.smart-meter-texas.

Assisted-by: Claude Code (claude-opus-5)
@Bad3r

Bad3r commented Aug 24, 2026

Copy link
Copy Markdown
Contributor Author

Follow-up: asn1 has no in-tree consumer that can actually use 3.x

I went looking for what else a second asn1 would affect, by scanning every pkgs/development/python-modules/*/default.nix for a function argument named asn1:

builtins.filter
  (n: let r = builtins.tryEval (import (dir + "/${n}/default.nix"));
      in r.success && builtins.isFunction r.value && (builtins.functionArgs r.value) ? asn1)
  names

On master that returns exactly two packages: pyimg4 and smart-meter-texas. Nothing else in Nixpkgs depends on python3Packages.asn1.

And smart-meter-texas turns out to be broken on 3.x too, for the same root cause.


Since neither in-tree consumer can use 3.x, downgrading python3Packages.asn1 to 2.8.0 and dropping asn1_2 entirely is a strictly smaller diff, and I confirmed it breaks nothing in the tree.

I did not do that, because the attribute is not only consumed in-tree: downgrading it would silently hand 2.x to out-of-tree users (overlays, flakes, dev shells) who asked for the current release, and 3.x is not defective, it is just an API break neither consumer has been ported to. Keeping asn1 at 3.3.0 and putting the two 2.x-bound consumers on asn1_2 avoids that regression. Happy to switch to the downgrade if maintainers prefer it, it is a two-line change.

I have proposed fix upstream for pyimg4 in m1stadev/PyIMG4#63. (note that smart-meter-texas haven't had a commit for over 2 years and seems inactive, considering open a fix PR but not confident it would be merged, should it be removed form nixpkgs?)

@Bad3r Bad3r changed the title python3Packages.pyimg4: unbreak by building against a new asn1_2 python3Packages.{pyimg4,smart-meter-texas}: fix against asn1 3.x by adding asn1_2 Aug 24, 2026
@nixpkgs-ci nixpkgs-ci Bot added 10.rebuild-linux: 11-100 This PR causes between 11 and 100 packages to rebuild on Linux. and removed 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. labels Aug 24, 2026
@Bad3r

This comment was marked as outdated.

@Bad3r

Bad3r commented Aug 24, 2026

Copy link
Copy Markdown
Contributor Author

nixpkgs-review result

Generated using nixpkgs-review-gha (b0ab82d)

Command: nixpkgs-review pr 555937
Commit: 0d127b7d8b5b2d71c1a6fd62075b9932a11e9afb (subsequent changes)
Merge: 81492164e921821453b9fe9389af1467a1d4b0c0

Logs: https://github.com/Bad3r/nixpkgs-review-gha/actions/runs/32696606997/attempts/1


x86_64-linux

✅ 11 packages built:
  • python313Packages.asn1_2
  • python313Packages.ipsw-parser
  • python313Packages.pyimg4
  • python313Packages.pymobiledevice3
  • python313Packages.smart-meter-texas
  • python314Packages.asn1_2
  • python314Packages.ipsw-parser
  • python314Packages.pyimg4
  • python314Packages.pymobiledevice3
  • python314Packages.smart-meter-texas
  • tests.home-assistant-components.smart_meter_texas

aarch64-linux

✅ 11 packages built:
  • python313Packages.asn1_2
  • python313Packages.ipsw-parser
  • python313Packages.pyimg4
  • python313Packages.pymobiledevice3
  • python313Packages.smart-meter-texas
  • python314Packages.asn1_2
  • python314Packages.ipsw-parser
  • python314Packages.pyimg4
  • python314Packages.pymobiledevice3
  • python314Packages.smart-meter-texas
  • tests.home-assistant-components.smart_meter_texas

aarch64-darwin (sandbox = relaxed)

✅ 10 packages built:
  • python313Packages.asn1_2
  • python313Packages.ipsw-parser
  • python313Packages.pyimg4
  • python313Packages.pymobiledevice3
  • python313Packages.smart-meter-texas
  • python314Packages.asn1_2
  • python314Packages.ipsw-parser
  • python314Packages.pyimg4
  • python314Packages.pymobiledevice3
  • python314Packages.smart-meter-texas

@wolfgangwalther
wolfgangwalther removed their request for review August 24, 2026 06:45

@dotlambda dotlambda left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Versioned attributes are not allowed in dependencies (they must not be propagated by python3Packages). You can downgrade asn1 instead.

@Bad3r
Bad3r marked this pull request as draft August 24, 2026 14:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

6.topic: python Python is a high-level, general-purpose programming language. 8.has: package (new) This PR adds a new package 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. 10.rebuild-linux: 11-100 This PR causes between 11 and 100 packages to rebuild on Linux. llm-assisted Contributions that include code generation with LLMs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants