feat(fts)!: make v2 the default index format

[BubbleCal]

Feature

What is the new feature?

This makes FTS v2 the default format for newly-created inverted / full-text indexes and replaces the previous environment-variable switch with an explicit format_version index creation parameter.

Why do we need this feature?

FTS v2 is the latest format, but format selection should be part of the index creation API instead of process-wide environment state. Existing v1 indexes must remain queryable and must continue to be maintained as v1 after append, incremental indexing, and optimize.

How does it work?

• Defaults new FTS index creation to v2.
• Adds explicit format_version handling in Rust params and exposes it through Python and Java index creation APIs.
• Preserves the stored FTS format when deriving params from existing indexes.
• Ensures mem-wal / maintained-index paths use the resolved index format instead of the default.
• Restores FTS format from IndexMetadata.index_version when rebuilding mem-wal FTS config, mapping legacy 0 and v1 1 to v1 and 2 to v2.
• Updates compatibility tests so old wheels can still use LANCE_FTS_FORMAT_VERSION while new code uses explicit format_version.

Breaking Change

BREAKING CHANGE: Newly-created FTS / inverted indexes now default to v2 instead of v1. Workflows that require the v1 layout, including compatibility with older Lance readers, must pass format_version=1; LANCE_FTS_FORMAT_VERSION no longer controls new Lance index creation.

Compatibility

Existing v1 FTS indexes remain readable. The regression coverage includes explicit v1 append + optimize_indices(OptimizeOptions::append()) and verifies the resulting FTS index metadata remains v1.

Verification

• GitHub Actions: 30/30 checks passed on commit e6a2ca759.
• git diff --check passed locally.
• Required local Rust / Python / Java checks could not run in this environment because cargo, uv, and a Java Runtime are not installed locally; CI covered the PR checks.

[codecov]

Codecov Report

❌ Patch coverage is 90.86957% with 21 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance-index/src/scalar/inverted/tokenizer.rs	82.45%	9 Missing and 1 partial ⚠️
rust/lance/src/dataset/mem_wal/memtable/flush.rs	79.16%	2 Missing and 3 partials ⚠️
rust/lance-index/src/scalar/inverted/builder.rs	83.33%	1 Missing and 1 partial ⚠️
rust/lance/src/dataset/mem_wal/index/fts.rs	83.33%	2 Missing ⚠️
rust/lance-index/src/scalar/inverted/index.rs	92.85%	1 Missing ⚠️
rust/lance/src/dataset/mem_wal/index.rs	98.24%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[BubbleCal]

@claude review

[Xuanwo]

The mem-wal maintained-index path should have end-to-end v1 coverage before this lands. The key compatibility contract is that an existing v1 FTS index remains v1 after maintenance; if this path accidentally flushes the maintained index as v2, FTS queries may still pass while older readers lose compatibility.

[Xuanwo]

Thank you for working on this. We will need a breaking change upgrade note. Can be a follow up PR.