fix(calldata): reject malformed and non-canonical encodings

[maho0638]

Fixes #227

What

• fail immediately on truncated calldata reads
• reject overlong/non-canonical ULEB128 encodings
• reject invalid UTF-8 in strings and map keys
• enforce canonical strictly-increasing map-key order, rejecting duplicate keys
• add regression coverage for malformed inputs and a canonical nested round-trip

Why

The GenVM reference decoder rejects these malformed or non-canonical representations, while genlayer-js previously accepted some of them. Matching the reference decoder removes ambiguous wire representations and prevents malformed container counts from continuing after input exhaustion.

Testing done

• added focused regression coverage for EOF/truncation, non-canonical ULEB128, invalid UTF-8, map ordering/duplicates, and canonical nested round-trips
• upstream GitHub Actions are currently awaiting approval to run for this fork PR
• CodeRabbit review was re-requested after the earlier rate-limit response

Decisions made

• reject malformed encodings rather than normalize them silently
• enforce the same canonical boundary as the GenVM reference decoder
• keep valid canonical encoder/decoder behavior unchanged

Checks

• I have tested this code through upstream GitHub Actions (workflow approval is still pending)
• I have reviewed my own PR
• I have created an issue for this PR
• I have set a descriptive PR title compliant with conventional commits

Reviewing tips

The highest-value review points are the decoder EOF guard, ULEB128 canonicality check, strict UTF-8 decoding, and map-key ordering enforcement.

User facing release notes

calldata.decode now rejects malformed and non-canonical calldata that GenVM's reference decoder already rejects.

[coderabbitai]

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 59 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

• Configuration used: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: bcf2434d-5d40-48be-a4e8-1d1ce37d855d

📥 Commits

Reviewing files that changed from the base of the PR and between 4dabdf2 and 73d4f36.

📒 Files selected for processing (2)

• src/abi/calldata/decoder.ts
• tests/calldata.test.ts

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[maho0638]

@coderabbitai review

[coderabbitai]

⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[maho0638]

@coderabbitai review

[coderabbitai]

⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[maho0638]

Manual fallback review while CodeRabbit is rate-limited: I compared the decoder changes against the GenVM Python calldata reference and the calldata spec. The EOF guard, canonical ULEB128 rejection, strict UTF-8 handling, and strict map-key ordering match the reference semantics, and the regression tests cover each boundary plus a canonical nested round-trip. I did not find an additional discrepancy in this pass. Upstream Actions are still awaiting fork-workflow approval.