Do witness resolution & forwarding in Converter - #3049
Open
plajjan wants to merge 6 commits into
Open
Conversation
added 6 commits
July 29, 2026 00:27
Golden files (Boxing output and generated C) for a small test
program, boxparam.act.
Base is a generic class whose methods take an argument of type T.
Deriv inherits from Base[int], so its versions of those methods take
an int. Because the methods can also be called through Base, where
the argument type is not known, the argument is always passed boxed
(a B_int), never as a raw machine integer -- even when the Deriv
version is called directly.
The goldens cover such an argument being compared, reassigned, and
passed on to another method. Two of the recorded outputs are plain
bugs, committed on purpose so that the diff of the next commit shows
exactly what the fix changes:
bool ...cmp (... self, B_int x) {
bool N_tmp = (x > 3LL); // compares the pointer
B_int ...bump (... self, B_int x) {
x += 1LL; // moves the pointer
return toB_int(((B_int)x)->val); // reads through it
A method slot's C calling convention is decided by the oldest
declaration of the slot in the hierarchy (rtypeOf). When that
declaration is generic, the slot passes the argument as a boxed
object, because callers may invoke it through the base class's method
table, where every argument is a pointer-sized word:
class Base[T](object):
def f(self, x: T) -> bool:
return True
An override in a class that narrows T to a scalar type inherits that
convention: Deriv(Base[int]).f receives a boxed B_int, not a raw
int64_t, even though its body computes with raw machine integers.
(Only this shape is affected. A slot whose oldest declaration already
names a scalar type passes it raw everywhere.)
The Boxing pass, however, assumed that every scalar-typed local holds
a raw value. It wrapped reads of such parameters in Box like raw
locals, and the Box/UnBox cancellation in the primitive operator
reduction then dropped the representation conversion entirely, so
class Deriv(Base[int]):
def f(self, x: int) -> bool:
return x > 3
Deriv().f(1) # returned True: compared the B_int pointer to 3
Track boxed parameters in the Boxing env and leave their reads
unwrapped. Where the body needs the raw value, a real UnBox now
survives to CodeGen and renders as ->val: the parameter is unboxed at
each use inside the callee.
Assignments cannot follow the read convention: an augmented
assignment would update the caller's box in place, and a plain
rebinding would store a raw value that later reads would unbox again.
A boxed parameter assigned in the body is therefore renamed aside and
copied into an ordinary raw local at function entry, after which
every read and store follows the standard raw-local convention and
the caller's box is never written.
Three build-failure projects capture how witness resolution behaves when extensions of one class are spread over modules. witness_rival_direct__bf: two modules, neither importing the other, each declare a conditional Pick extension for Box. Both witnesses are genuine rival implementations, so resolving Pick[Box[Both]] is ambiguous. This must keep failing. witness_rival_ancestry__bf: the same rivalry, except each module implements Pick through its own child protocol, so both witnesses reach Pick through their inheritance ancestry rather than declaring it directly. Equally ambiguous, and it must also keep failing, in either import order. witness_canonical_import__bf: module points declares Pt and extends it with Ord, which covers Eq; module hashpt extends Pt with Hashable, whose inherited Eq slots are thereby finalized, so hashpt implements none of them. There is exactly one Eq[Pt] implementation, yet declaring the Hashable extension makes every == involving Pt fail to resolve, both inside hashpt and in any module importing both. This failure is a deficiency; the next commit makes resolution find the one implementation, and flips this project into a running test.
An extension covers its protocol's whole inheritance ancestry:
extending Pt with Ord also makes the extension a witness for Eq,
because Ord inherits Eq. The first extension to cover a protocol owns
it: the type checker requires it to implement the protocol's methods,
and every later extension covering the same protocol is forbidden
from implementing them again (checkAttributes calls these methods
finalized). So for each protocol and type there is exactly one
implementation, held by whichever extension came first.
Witness registration did not follow that rule. An extension
registered a witness for every protocol in its ancestry, including
the finalized ones it holds no implementation for. Such an entry is
an empty shell: resolution that lands on it finds methods the
extension was not allowed to write. Within one module the shells were
harmless, because registration deduplicates against the local table
and the owning extension's earlier entry wins. Across modules there
was no deduplication at all, since the local table and the imported
interfaces are enumerated separately:
# module points
extension Pt (Ord):
def __eq__(a, b): ... # the one Eq[Pt] implementation
...
# module hashpt
extension points.Pt (Hashable): # Hashable inherits Eq
def hash(self, h): ...
hashpt's extension implements nothing of Eq, which is finalized by
points' Ord extension, yet it registered a second Eq witness for Pt.
The constraint solver requires exactly one candidate per witness
lookup, so every == involving Pt stopped compiling, inside hashpt and
in every module importing both. In practice, extending an imported
class with any protocol that inherits something was unusable.
Nothing about a registration itself distinguishes a shell from a
genuine implementation declared in an unrelated module; the
distinction exists only at the moment the extension is checked, when
hasWitness tells us whether an earlier witness covers the protocol.
So record that answer: NExt gains a field listing the extension's
finalized protocols (bumping the .tydb interface version), and those
protocols are skipped everywhere witnesses are registered: the local
type table (setupWits), the extension's own self-witnesses used while
checking its body (tydefineInst), and the entries importers read from
its interface (extWitnesses).
Every remaining registration is backed by an implementation. A
protocol with one implementation resolves to it from any module, in
any import order. And nothing is silently legalized: if two
independent modules, neither importing the other, both implement the
same protocol for the same type, then neither was finalized, both
register, and every use remains ambiguous and rejected in either
import order, whether they cover the protocol directly or through
their ancestry.
Of the three projects pinned by the previous commit, the two rival
ones keep failing unchanged; witness_canonical_import, previously
rejected despite its single Eq implementation, now builds and runs.
A fixture, witness_forward.act, with golden files for the converted AST (Pass 3) and the generated C (Pass 9). Protocol PA declares a static slot (same) and an instance slot (total); protocols PB and PC both inherit PA. The extension Thing (PB) implements everything; the extension Thing (PC) implements only its own method, since PA's methods are finalized by the PB extension. The goldens pin what currently comes out for PCD_Thing, the witness class of the second extension. In the converted AST, same and total are bare signatures; nothing in the AST fills them. In the generated C, CodeGen repairs the method table behind the compiler's back with $forward wrappers that search out PBD_Thing as a provider, but the class still counts as abstract, so no PCD_ThingG_new constructor is emitted; any program whose witness resolution instantiates this class fails to compile. Both outputs are about to change.
A generated witness class can inherit an abstract protocol slot whose
implementation lives in another witness: with Ord and Hashable both
inheriting Eq, Hashable[int] gets __eq__ from the Ord[int] extension.
CodeGen repaired such method tables by searching all classes for a
compatible provider and emitting a C forwarding wrapper, behavior
absent from the converted AST that every later pass had to work
around. It also only fixed the table: the witness class itself stayed
abstract, so no constructor was emitted and any user-level occurrence
of the shape failed to compile.
Synthesize the forwarding at extension checking instead. A slot is
left unimplemented exactly when it is final, i.e. an earlier witness
covers its protocol (checkAttributes), so for each missing final slot
the extension body gains an ordinary method
def total(self): return Coll.total(self)
built like the NotImplemented stubs (fromTEnv). Selecting the slot
through the covering protocol makes the type checker route the call
to that canonical witness, following any witness fields on the way
(sibling providers such as Collection reached through
Sequence$list.W_Collection), instantiating provider type arguments,
and padding cyclic-witness opts. Since the first witness covering a
protocol must implement it directly, forwarding chains are one hop
and cannot cycle. A slot inherited through several agreeing parents
gets one forwarding method (its signatures must agree; conflicting
inherited signatures are now rejected explicitly). Converted witness
classes are complete, ordinary classes for all later passes,
selective compilation sees the provider dependency in the AST, and
the forwarding also works where the old ABI-equality check silently
gave up (generic extensions with differing type variables, providers
with constructor arguments).
Remove the CodeGen machinery, the never-called genDirectMethodCall
provider search next to it, and the dead hand-written builtin C
implementations whose symbols the generated forwarding methods now
provide. The builtin method tables come out complete without any
$forward wrappers, with __eq__ forwarders compiling down to direct
value comparisons.
Contributor
Author
|
@nordlander I felt I had a bit of an epiphany during #3045, an insight upon which this branch is then based. simple and elegant 😃 the branch is actually a net-negative in terms of lines of code, removing more code than it adds (if we do not count the new test cases). It is also quite possible that I'm missing something central on how this needs to work, rendering the PR useless 😜 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This builds on #3043 and #3045 (perhaps easier to review them individually), so it's just the last 2 commits in this PR that is the real change in this PR.
We currently have some code running in CodeGen to fill in unpopulated witness slots, like when a protocol inherits from some other protocol but a particular extension doesn't concretely specify all methods. Our classic example is in builtins where both the Ord and Hashable protocol inherit from Eq. Some instance implement Ord, which define
__eq__to cover the Eq part, and then the later Hashable protocol should have its__eq__resolved to the available method. The first extension to implement the methods for a protocol "owns" it and can be used by later extensions.Doing this in CodeGen is really rather late. For the work on selective back passes, we want to collect this information about witnesses and we can't do this from CodeGen, so we repeated the effort. This PR does a more proper move of this witness resolution functionality into Converter so it is available in the AST just after the type checking pass. The change itself is structured into 2 commits, the first which adds a snapshot test so we can see in the second commit the effect it has on the AST.