Store PC protocols as deltas from the previous version

[u9g]

What

Protocols are now stored as deltas from the previous version instead of full copies.

A version's proto.yml can start with !base: <version> (resolved through dataPaths.json). Semantics, per section:

• a type present in both base and delta is replaced wholesale by the delta's definition — deliberately no deep merge, since mapper values and packet IDs are positional and partial merges would be unsafe
• a type only in the base is inherited unchanged
• a type only in the delta is added
• TypeName: !delete removes an inherited type (hard error if it doesn't exist); ^section: !delete removes a whole section

Versions whose protocol didn't change keep sharing data dirs via dataPaths.json exactly as before. npm run build resolves !base chains transparently — full protos compile exactly as they did.

Commits

1. Render protocol.json with object keys in sorted order — pure reordering of all 109 protocol.json files (verified content-identical), so generated files are byte-stable regardless of how a protocol is assembled. Top-level section order is preserved: protodef-validator registers the global types namespace before the per-state namespaces, so types must stay first.
2. Store PC protocols as deltas — the resolver, the one-time backfill tool, and the 51 converted deltas (net −70k lines; e.g. 1.21.8 goes from ~3.2k lines to 42). Thanks to sorted rendering, the regenerated protocol.json files are identical except for the position of the configuration section (new in 1.20.2) in the 12 versions that have it.
3. Speed up the test suite — linear uniqueItems instead of Ajv's O(n²) pairwise deep-equal, and one shared protodef Validator instead of one per version.

Safety

Each converted delta was verified to compile to a protocol deep-equal to the previously committed protocol.json. The backfill tool (tools/js/backfillDeltas.js) bakes this golden check in, so it can be re-run for future versions. Full suite passes with the currently published protodef-validator (1820 tests).

Bedrock keeps full protos for now (its generated packet_map.yml import needs handling first); the resolver is edition-agnostic.

[rom1504]

We discussed this many times in the past and always concluded full protos are better. What changed now ?

[u9g]

Can you share with me the conversations or reasons from these conversations? I don’t believe I’ve ever heard of this being discussed.

[rom1504]

Having the full proto have these benefits

• Easy to change one version without impacting others
• Easy to use by any tools in any language without reimplementing the diff logic everywhere
• Easy to understand a proto of a given version by just looking at the relevant file

What does diffing brings besides saving a few bytes ?

[u9g]

Easy to change one version without impacting others

Why would you want this? All protocol versions are version bumps upon a previous version and almost always build on eachother.

Easy to use by any tools in any language without reimplementing the diff logic everywhere

You still have the opportunity to render the files locally every time you make a change.

Easy to understand a proto of a given version by just looking at the relevant file

you can still do this by looking at the rendered version.

What does diffing brings besides saving a few bytes ?

1. Most protocol changes are minimal changes upon a previous iteration of the protocol. When every protocol version lives as a full copy in the repo, you lose this context, and it becomes impossible to easily tell what changed between two versions, which makes any cross-version work much harder.
2. This makes the workflow for backporting a protocol fix easy, you just add the change to the lowest version with that change. This is a very common workflow that takes a lot of work for contributors now.

The advantages of this change are mostly in lowering the barrier for contribution that requires backporting protocol changes that have been wrong across many versions, and easily scanning the protocol as an implementer to know where to expect to make changes in the protocol implementations that is powered by these files. No part of the benefit is saving bytes, because we still have the rendered versions, just not in the repo.

[rom1504]

Are you sure in practice the "propagate to all later versions" property work as expected? How can we check?

How do we make it so all non js dependency don't break with this change?

[rom1504]

This makes the workflow for backporting a protocol fix easy, you just add the change to the lowest version with that change. This is a very common workflow that takes a lot of work for contributors now.

Why can't we already do that with full files if that property of the diff works ? This would be just a script to run

My guess is actually the protocols diff do not compose that well

[u9g]

Are you sure in practice the "propagate to all later versions" property work as expected? How can we check?

The PR shows that the changes properly work as described, it is the proof. After running the updated compilation step, we have generated the same json as before, but with the protocol-delta yaml, rather than the full protocol yaml.

How do we make it so all non js dependency don't break with this change?

We will leave the .json files in the repo that they depend on. Other than that, I'm not sure what we can do. If you have a list of who depended on the yaml versions, we can open an issue that they should now depend on the json version, but otherwise I'm not sure what to do there. We could also generate yaml files from the deltas, but I think it's a step backward if we just add a new format, unless we add a # This is generated, see the protocol yaml. comment to the json and the yaml files, but I wouldn't do that unless we know there is someone for whom the change of the yaml format will be a breaking change.

[u9g]

This makes the workflow for backporting a protocol fix easy, you just add the change to the lowest version with that change. This is a very common workflow that takes a lot of work for contributors now.

Why can't we already do that with full files if that property of the diff works ? This would be just a script to run

Look, you can totally make a script, formats materially change the viewing experience.

My guess is actually the protocols diff do not compose that well

the last 5 pc protocol fixes:

#1230 - changes 10 files before, changes 1 after
#1225 - changes 6 files before, changes 1 after
#1216 - changes 1 file before, changes 1 file after - but this pr was done wrong because it updated the json and not the latest/proto.yml and nobody noticed
#1201 - changes 16 files before, changes 2 files after
#1200 - changes 34 files before, changes 2 files after

You can be sure no files will be missed by adding in CI a bot that says what the range for a change is, and any missing in between, and when the protocol next diverges from that change, but this PR will make it easier to make those changes for both the person making the change and the person reviewing it.

I don't mind if we close this PR though, we can also just make a ci workflow that shows the range of versions effectred by the protocol change made, any forgotten files, any inbetween versions missed, and where the changes end (and whether that's before latest).

This format just makes that process slightly easier by only needing to know at what version the change you just made stops applying to newer versions.

[AnonymoDGH]

I'm going to try to fix all the issues one by one.

[rom1504]

Ok I took another look at the content of the PR and now I get it, you store the diff in the yml file but keep the whole content per version in the json file.

This is actually better than what I had in mind and does compose relatively nicely with the overall process.

[rom1504]

@extremeheat you made that proto yml, what do you think about this change?

[extremeheat]

I commented on the discord, but it can be pretty hard to follow packets when the relevant types (packets have types which can have their own subtypes) are dispersed throughout several versions.

For example if SlotComponent changes (which is used in multiple packets), that type itself has many subtypes inside of it. If you want to follow the logic and fix an issue in it, that means a manual process of figuring out where the last change to the data type was and then from there looking at a subtype, checking if it's in the current file, it was updated in a newer version (yes a newer version, as old -> new -> old can happen when digging through subtypes), an older version, etc and then building a model of how the final result is supposed to look

There is a search all feature yes but that gets quite messy as you have to walk back in reverse order every time you update the latest protocol. Comparatively back porting a correction (uncommon) is quite easy as you just apply same patch to n previous versions

[u9g]

Closing based in @extremeheat ’s thoughts. If we decide to, we can reopen later.