Indexes 4: Adds Indexes for SPK repositories

[dcookspi]

Note: for info on benefits of indexing for spk solves see #1340 (5 of 5). Maybe start there and work back down to this PR if you prefer to review PRs top down.

This adds a new Indexed repository type, RepositoryIndex traits, a RepoIndex enum, and a flatbuffer index implementation in FlatBufferRepoIndex, along with some configuration. The indexing is designed to help the solvers. An index only contains information useful for solving. It does not contain enough information to do other things, such as build or test the packages.

The hierarchy of new types is roughly:

RepositoryHandle (existing spk repos)
   Indexed 
       RepoIndex
           FlatBufferRepoIndex  (makes SolverPackageSpecs, a v0 package alternate, see PR 3 of 5 below)
                flatbuffers schema (see PR 3 of 5 below)

This also updates the solver and repo based tests to include the new Indexed repo. This does not add command line tools or solver interaction for indexes. Those are in the last PR in this chain.

This is the change that adds indexes and index based packages to Spk repositories.

This is 4 of 5 chained PRs for adding indexes to spk solves:

1. Indexes 1: Change Package and related traits to not return references to fields #1336
2. Indexes 2: Add new_unchecked() constructors to spk schema objects #1337
3. Indexes 3: Adds flatbuffers schema and SolverPackageSpec for indexes to spk #1338
4. this PR
5. Indexes 5: Adds spk repo index subcommand for index generation and updates #1340
6. Indexes 6: Changes version_filter field in index schema #1344
7. Indexes 7: Adds a lock file around index generation and updates #1354
8. Indexes 8: Fixes the spk build or mkb crash with indexes enabled #1355
9. Indexes 9: Adds messaging on package events to kafka #1356

[codecov]

Codecov Report

❌ Patch coverage is 60.01864% with 429 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
crates/spk-storage/src/storage/flatbuffer_index.rs	55.77%	314 Missing ⚠️
crates/spk-storage/src/storage/indexed.rs	57.36%	55 Missing ⚠️
crates/spk-schema/src/v0/indexed_package.rs	0.00%	29 Missing ⚠️
...s/spk-storage/src/storage/flatbuffer_index_test.rs	83.72%	21 Missing ⚠️
crates/spk-storage/src/storage/repository_index.rs	85.71%	4 Missing ⚠️
crates/spk-solve/src/solvers/solver_test.rs	80.00%	2 Missing ⚠️
crates/spk-storage/src/storage/handle.rs	66.66%	2 Missing ⚠️
crates/spk-storage/src/storage/runtime.rs	0.00%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[dcookspi]

I can't replicate the test failures locally at the moment. I'll dig into them further tomororw.

[dcookspi]

I can't replicate the test failures locally at the moment. I'll dig into them further tomororw.

The issue was related to deprecated packages and the migration-to-components feature. I've put in a temporary fix so the tests pass for now. But it should not get merged as is. A direction needs to be decided with regard to deprecated packages, solves, and what to index in order to replace the temp fix with a better solution. See #1339 (comment) comment above.

[jrray]

Okay but if removing the temp file fails the ? usage here will mask the original error. Any errors in cleanup could be logged but the caller should get the first error as the return value.

[dcookspi]

I've update all the places like this to log and return the first error.