Support ML async job cancellation, fail jobs on redis errors

[carlosgjs]

Summary

This pull request builds on RolnickLab#1150 and it's based off carlos/redisatomic

This pull request introduces a new chaos testing management command for fault injection and refactors async job cleanup logic to improve reliability and resilience. The most important changes include the addition of a manual chaos testing utility, improved job log handling to prevent lost logs, and a more robust cleanup of async resources for jobs using Redis and NATS. The cleanup logic is now more consistent and reliable, especially in failure and cancellation scenarios.

• Added chaos_monkey.py management command for manual fault injection of Redis and NATS, allowing developers to flush or pause these services to simulate outages and test job resilience.
• Refactored cleanup_async_job_resources to accept job ID and logger instead of a Job instance, ensuring cleanup can occur even if the Job object is unavailable and improving logging consistency.
• Introduced _fail_job helper to mark jobs as failed and trigger async resource cleanup when Redis state is missing, improving failure handling in NATS pipeline results.
• Updated job cancellation logic to always trigger async cleanup and correctly set status to REVOKED for async jobs.
• Improved job log handler to refresh logs from the database before writing, reducing lost logs due to concurrent updates.
• Ensured logger handler always references the current job instance, preventing stale log writes in worker processes.
• Added _stream_exists check in NATS queue orchestration to avoid unnecessary stream creation and improve error handling when reserving tasks.

How to Test the Changes

Start a job with e.g.:

docker compose run --rm django python manage.py test_ml_job_e2e --collection "ami-1000" --pipeline quebec_vermont_moths_2023 --dispatch-mode async_api --project 1

Then either cancel it in the UI or flush/stop Redis

docker compose run --rm django python manage.py chaos_monkey flush redis # must run in container
docker compose down redis

Screenshots

Known Issues

Occasionally the Error logs get overwritten by another worker and hence the error won't be displayed, which is a known issue with the job logger.

Deployment Notes

Include instructions if this PR requires specific steps for its deployment (database migrations, config changes, etc.)

Checklist

• I have tested these changes appropriately.
• I have added and/or modified relevant tests.
• I updated relevant documentation or comments.
• I have verified that this PR follows the project's coding standards.
• Any dependent changes have already been merged to main.

[mihow]

Works! I am canceling and retrying on a 1000 image job. Very cool!

I made one fix to keep /tasks from returning tasks while canceling and added a new active_states group that includes all job states that count as job still in motion.