Remove deleted vacancies from the index

[berkes]

Intent

• Deleted Status Updates (toots) that were indexed as vacancy should be removed from the index eventually.

Problem

When a user deletes a toot (update), or delete and redrafts it, we are showing results that are no longer available in the fediverse.

We need to remove these from our index too.

Discussion

When deleting a toot, a server may send out a delete event over the fediverse. The mastodon streaming API then emits this as a delete event. We may capture this event, but could miss it.

Structure

Options:

• Rely only on the the delete event of the streaming api. This requires us to have solid uptime and ensure we don't miss any delete events otherwise.
• A cron or timed run that re-checks a list of updates to see if it still available. Could be done over the web relying on HTTP status codes. Delete when no longer fetchable/timeout etc.
• Combination of both.

The timed-recheck is more solid and will capture servers going down or toots otherwise dissapearing without sending delete events. It is heavier on server and client.

Example

Toot re-drafted several times:

curl 'https://meili.flockingbird.social/indexes/vacancies/search' -H 'Content-Type: application/json' -H 'X-Meili-API-Key: 3365f08f0ddf01bf8d5f89d21a73b570f11d28e21818bb99d9c849f9c0d2ac05' --data-raw '{"q":"toothpaste","offset":0,"limit":5}'

Check list

• Redrafted toots should appear only once in our index after X time
• Deleted toots should dissapear from our index after X time
• X time should be evaulated after a while and finetuned. For now, 1 day is fine.