Indexes 10: Adds index update listeners and Indexer#1363
Open
dcookspi wants to merge 2 commits into
Open
Conversation
dcookspi
added
enhancement
dcookspi
force-pushed
the
index-9-messaging2-publish-listener-and-indexer
branch
from
e6efe8c to
e327dd8
Compare
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
This was referenced May 11, 2026
dcookspi
force-pushed
the
index-9-messaging2-publish-listener-and-indexer
branch
from
93e8121 to
c26413c
Compare
dcookspi
force-pushed
the
index-9-messaging2-publish-listener-and-indexer
branch
from
c26413c to
d910642
Compare
dcookspi
commented
May 13, 2026
| let new_embedded_providers = self.get_embedded_providers(package)?; | ||
| // No change case #1: no embedded packages involved. | ||
| if original_embedded_providers.is_empty() && new_embedded_providers.is_empty() { | ||
| announce_package_event( |
Collaborator
Author
There was a problem hiding this comment.
I've realised the additional announce_package_event calls in this method (update_publish) are not correct. They all need to be closer to the points where the methods that update the storage are called, so I'll be moving these around abit.
- Fix this.
Collaborator
Author
There was a problem hiding this comment.
I've moved these calls around to be closer to the underlying storage updating calls.
Base automatically changed from
index-9-messaging-to-kafka-feature
to
index-8-spk-build-fix
May 14, 2026 19:00
dcookspi
force-pushed
the
index-8-spk-build-fix
branch
from
cfe91ed to
19faa81
Compare
…package changes and updating the index. Signed-off-by: David Gilligan-Cook <dcook@imageworks.com>
…ecate commands in update_publish. Adds debug logging for message sending. Signed-off-by: David Gilligan-Cook <dcook@imageworks.com>
dcookspi
force-pushed
the
index-9-messaging2-publish-listener-and-indexer
branch
from
e196230 to
acc943d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds an index update listening step to spk commands that update packages:
publish,deprecate,undeprecate, andremove. After finishing their current processing, these comands will wait and listen to messages on an index updates topic to verify their changes have made it into the repository's index before finishing.This also adds the
spk repo indexer ...command to start an index updating process (an Indexer) that will listen for package update messages for a given repo, and kick off index updates for that repo's index as packages are changed. The Indexer will send index status event messages to the index updates topic for other spk commands to listen to. Thepublish,deprecate,undeprecate, andremovecommands will use those messages to determine when the index the care about has been updated.This also extends the spk configuration for messaging channels, indexes, and indexers to include the index status topic, naming channels and indexers, and various new timeouts and settings.
The idea is to support automatic updating of an index when packages change. A site would need an external messaging system set up. But then the site could configure a message channel and an indexer, and run an indexer as a service (e.g. for the origin repo). Then all the spk commands that modify packages in that repo, would send messages, via the message channel, to the indexer for it to update the index automatically, and the indexer would send messages back to the commands to let them know the index now contains their changes.
This is another PR in the chain for adding indexes to spk solves:
new_unchecked()constructors to spk schema objects #1337spk repo indexsubcommand for index generation and updates #1340version_filterfield in index schema #1344