Skip to content

Carry a value in and out of a stream field the way the catalog states - #200

Closed
estebanzimanyi wants to merge 7 commits into
MobilityDB:mainfrom
estebanzimanyi:codegen/value-hexwkb-marshalling
Closed

Carry a value in and out of a stream field the way the catalog states#200
estebanzimanyi wants to merge 7 commits into
MobilityDB:mainfrom
estebanzimanyi:codegen/value-hexwkb-marshalling

Conversation

@estebanzimanyi

@estebanzimanyi estebanzimanyi commented Sep 4, 2026

Copy link
Copy Markdown
Member

A shape names one arity and one operand vocabulary, so a function differing from
a covered one only in the type of its second argument falls out of the streamable
surface entirely. value_marshalled asks the catalog the same question of every
argument and of the result -- can a stream field carry a value of this type, and
can the answer be handed back -- and emits whenever all of them say yes. It is
last in SHAPES, so every named shape keeps its own naming, ordering and
comments and this answers only what none of them claimed.

A span, a set and a spanset travel in hex-WKB because that is the only encoding
that round-trips one: the text form of a Span * states no variety, and the
catalog's own in for it is bigintspan_in, one of many. The
typeEncodings[T].decoders.wkb entry names a decoder for the type itself, so one
decoder answers every variety, and the result comes back through the _as_hexwkb
sibling -- the same exchange form a temporal already travels in, so the output of
one operator is the input of the next. A type carrying a text parser named for
itself keeps it, which leaves the operators already emitted byte-identical.

Every codec is called from its own catalogued signature rather than from the
arity of the one the templates were written against: interval_in takes a typmod
after the string, raster_as_hexwkb takes only a length, and pcpoint_as_hexwkb
neither. A pointer-to-pointer marshals as nothing, since rstrip("*") reads
Cbuffer ** as a Cbuffer, which is an array or an answer written back through
the argument and not one value.

The operand that gets built is the first argument that needs building, and
primary_index records where it goes back in the call, which is what lets an
operand sit in the middle of a longer argument list.

structural_residue() is asked before the shapes. What a function IS -- an
aggregate's state transition, a value's own text form, an answer returned through
its argument list -- decides that no per-event operator exists for it whatever a
shape makes of its signature. The --out descriptor path applies the same gate,
so the ledger and the descriptor name one set.

The ledger reads 1695 GENERATED, 774 GAP, 868 RESIDUE and 172 DEFERRED over the
3509 public functions the catalog carries, and every one of the 1695 emitted
operator cores compiles against a -DMEOS=ON -DALL=ON MEOS with 0 errors and 0
warnings.

An operator whose answer is serialized in a form the operand side cannot parse
composes with nothing, and a circular buffer left in hex-WKB was exactly that: a
Cbuffer operand is read from the text its own _in parses, so the answer has to
leave as that text. result_marshalling asks operand_marshalling which
encoding the type travels in and takes that encoding's encoder, which is what
keeps the two sides from disagreeing. A span stays hex-WKB, because hex-WKB is
the only form stating which variety it is.

The encoder is typeEncodings[T].encoders.text rather than a composed <t>_out,
which is what answers a geometry: its text encoder is geo_as_ewkt. The
arguments after the value are what out_aux names, counted against the encoder's
own parameters, so a geometry is written at the precision the catalog states and
a type whose encoder says something else declines.

A geometry is also an operand a function can be built around, read with geom_in
-- the parser the binding already uses for a geometry argument, and the one that
accepts the SRID-carrying EWKT the result side writes. It is parsed directly
rather than through StaticGeometry so the value is owned by the lambda, which
is what the single free at the end of the call expects.

The ledger reads 1826 GENERATED, 662 GAP, 868 RESIDUE and 153 DEFERRED, and every
one of the 1826 emitted operator cores compiles against a -DMEOS=ON -DALL=ON
MEOS with 0 errors and 0 warnings.

A shape names one arity and one operand vocabulary, so a function differing from
a covered one only in the type of its second argument falls out of the streamable
surface entirely. `value_marshalled` asks the catalog the same question of every
argument and of the result -- can a stream field carry a value of this type, and
can the answer be handed back -- and emits whenever all of them say yes. It is
last in `SHAPES`, so every named shape keeps its own naming, ordering and
comments and this answers only what none of them claimed.

A span, a set and a spanset travel in hex-WKB because that is the only encoding
that round-trips one: the text form of a `Span *` states no variety, and the
catalog's own `in` for it is `bigintspan_in`, one of many. The
`typeEncodings[T].decoders.wkb` entry names a decoder for the type itself, so one
decoder answers every variety, and the result comes back through the `_as_hexwkb`
sibling -- the same exchange form a temporal already travels in, so the output of
one operator is the input of the next. A type carrying a text parser named for
itself keeps it, which leaves the operators already emitted byte-identical.

Every codec is called from its own catalogued signature rather than from the
arity of the one the templates were written against: `interval_in` takes a typmod
after the string, `raster_as_hexwkb` takes only a length, and `pcpoint_as_hexwkb`
neither. A pointer-to-pointer marshals as nothing, since `rstrip("*")` reads
`Cbuffer **` as a `Cbuffer`, which is an array or an answer written back through
the argument and not one value.

The operand that gets built is the first argument that needs building, and
`primary_index` records where it goes back in the call, which is what lets an
operand sit in the middle of a longer argument list.

`structural_residue()` is asked before the shapes. What a function IS -- an
aggregate's state transition, a value's own text form, an answer returned through
its argument list -- decides that no per-event operator exists for it whatever a
shape makes of its signature. The `--out` descriptor path applies the same gate,
so the ledger and the descriptor name one set.

The ledger reads 1695 GENERATED, 774 GAP, 868 RESIDUE and 172 DEFERRED over the
3509 public functions the catalog carries, and every one of the 1695 emitted
operator cores compiles against a `-DMEOS=ON -DALL=ON` MEOS with 0 errors and 0
warnings.
An operator whose answer is serialized in a form the operand side cannot parse
composes with nothing, and a circular buffer left in hex-WKB was exactly that: a
Cbuffer operand is read from the text its own `_in` parses, so the answer has to
leave as that text. `result_marshalling` asks `operand_marshalling` which
encoding the type travels in and takes that encoding's encoder, which is what
keeps the two sides from disagreeing. A span stays hex-WKB, because hex-WKB is
the only form stating which variety it is.

The encoder is `typeEncodings[T].encoders.text` rather than a composed `<t>_out`,
which is what answers a geometry: its text encoder is `geo_as_ewkt`. The
arguments after the value are what `out_aux` names, counted against the encoder's
own parameters, so a geometry is written at the precision the catalog states and
a type whose encoder says something else declines.

A geometry is also an operand a function can be built around, read with `geom_in`
-- the parser the binding already uses for a geometry argument, and the one that
accepts the SRID-carrying EWKT the result side writes. It is parsed directly
rather than through `StaticGeometry` so the value is owned by the lambda, which
is what the single `free` at the end of the call expects.

The ledger reads 1826 GENERATED, 662 GAP, 868 RESIDUE and 153 DEFERRED, and every
one of the 1826 emitted operator cores compiles against a `-DMEOS=ON -DALL=ON`
MEOS with 0 errors and 0 warnings.
A TInstant, a TSequence and a TSequenceSet each begin with the Temporal they are
a kind of, so `temporal_as_hexwkb` serializes one and the answer travels in the
same exchange form every other temporal uses. The operand side already reads this
off the catalog -- `scalar_base_input` casts a constructor's `TInstant *` for
exactly that reason -- and a function ANSWERING one states the same fact from the
other side, so 31 nearest-approach instants, sequence accessors and stop
detectors compose with the operators that read a temporal.

The local keeps the type the function answers and the cast sits at the encoder
call, so neither the declaration nor the call misstates what MEOS returned.

Every one of the 56 subtype-returning operators calls a function answering a
fresh copy, so the `free` after serializing is right. The view-returning
`*_inst_p` family answers a pointer INTO the temporal, which a caller must not
free, and every one of those is declared in an internal header the
public-surface gate already refuses.

The ledger reads 1882 GENERATED, 632 GAP, 868 RESIDUE and 127 DEFERRED, and every
one of the 1882 emitted operator cores compiles against a `-DMEOS=ON -DALL=ON`
MEOS with 0 errors and 0 warnings.
`typeEncodings` describes the types whose in/out pair the catalog's encodings
scan registers, and a type absent from it can still declare that pair itself.
`text` is the case: the catalog states its marshalling in the `wire` block as a
plain JSON string -- `always_eq_ttext_text` carries `{"name": "txt", "kind":
"json", "json": "string"}` -- while `text_in(const char *) -> text *` and
`text_out(const text *) -> char *` are the pair that builds one and reads it
back. Reading the catalog's own function list answers a `text *` operand where
the encodings map has nothing to say.

The EXACT signatures are what keep this from admitting a variety-locked parser.
`span_in` takes a MeosType alongside the string, so a Span declines here for the
same reason it declines against the encodings map, and travels as the hex-WKB
that states which variety it is.

The ledger reads 1952 GENERATED, 567 GAP, 868 RESIDUE and 122 DEFERRED, and every
one of the 1952 emitted operator cores compiles against a `-DMEOS=ON -DALL=ON`
MEOS with 0 errors and 0 warnings.
Every operand builder makes its instant from a `uint64_t` epoch column, through
`MEOS::Meos::convertEpochToTimestamp` and `timestamptz_in`. A function taking a
`TimestampTz` ARGUMENT wants the same value by the same route, so the argument is
an epoch column too and the conversion sits where the call is assembled. MEOS
owns the timezone resolution, as it owns the SRID: the binding hands it the text
and parses nothing itself.

Nothing here states the width of the typedef. The catalog resolves no typedefs,
which is why an `S2CellId` operand declines rather than narrow in silence; a
timestamp needs no such guess, because the value reaches MEOS as the text
`timestamptz_in` parses.

The ledger reads 2002 GENERATED, 523 GAP, 868 RESIDUE and 116 DEFERRED, and every
one of the 2002 emitted operator cores compiles against a `-DMEOS=ON -DALL=ON`
MEOS with 0 errors and 0 warnings.
A `TimestampTz` is a count of microseconds, and both sibling bindings hand one
to MEOS as the number it is: GoMEOS calls `C.TimestampTz(t)` on an int64, and
MobilityDuck writes `static_cast<TimestampTz>(timestamp.value)`. Neither
converts through text.

`MEOSWrapper::convertEpochToTimestamp` cannot stand in for that. It infers the
unit from the magnitude of the number it is given -- nanoseconds above 1e19,
microseconds above 1e16, milliseconds above 1e13, seconds otherwise -- then
truncates to whole seconds and clamps anything beyond 2100-01-01. An argument
routed through it carries a guessed unit, loses everything below a second, and
reports a value it was never given.

So the operand declines, and the functions taking one stay in the gap with the
rest of the surface the generator owes. The ledger reads 1952 GENERATED, 567
GAP, 868 RESIDUE and 122 DEFERRED.
… of it

`stbox_tmax(const STBox *, TimestampTz *result)` answers two things: whether the
box has a T dimension at all, and the value when it does. GoMEOS hands both back,
`func STBOXTmax(box *STBox) (bool, int64)`, and the catalog says the same in
`shape.outParams`. A per-event operator yields ONE field, so it can carry the flag
or the value and not both, and returning the value alone reads "absent" as zero.

So the 41 functions of that shape are reason-marked `RESIDUE:out-param-pair`
rather than generated. The gap is the work the generator owes; a shape the
operator model cannot express is not work, and counting it as such invites
exactly the half-answer this refuses.

The ledger reads 1952 GENERATED, 526 GAP, 909 RESIDUE and 122 DEFERRED. The
emitted descriptor is byte-identical to the one before this change: nothing is
generated differently, only classified honestly.
@estebanzimanyi
estebanzimanyi force-pushed the codegen/value-hexwkb-marshalling branch from 08369fd to d712af9 Compare September 4, 2026 11:37
@estebanzimanyi

Copy link
Copy Markdown
Member Author

Closing: the change is not decidable on the evidence this repository can produce for it.

The branch carries a 59-line delta to codegen_nebula.py, and running the pristine and the
modified generator against tools/codegen/trgeo-descriptor.json -- the only descriptor the
repository commits, byte-identical on both arms -- emits output that is identical file for file,
0 differing lines over 140 emitted paths. The rig discriminates: injecting one line into an
emitted .cpp moves it to 1 differing file and restoring it returns 0. So that descriptor does
not reach the paths this change touches, and a green run over it says nothing about the change.

Deciding it needs a descriptor covering the value marshalling and timestamp-argument shapes the
delta actually alters, built from a freshly derived meos-idl.json.

@estebanzimanyi
estebanzimanyi deleted the codegen/value-hexwkb-marshalling branch September 4, 2026 19:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant