You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[FEA] Delete cached Spark event logs when they are no longer needed #281
Is your feature request related to a problem? Please describe.
The tool saves event logs so a timed-out attempt can pick up where it left off. Once it has finished calculating a job's cost, those saved logs usually serve no further purpose, but they stay on disk.
Describe the solution you'd like
Keep the files while a job still needs retries, then remove them after a successful calculation. Do not remove a file that another calculation is using. Test retries, successful cleanup, and calculating the same job again later.
Describe alternatives you've considered
Operators can clear the cache while callers are stopped, but that also removes useful progress for other jobs still awaiting retries.
Additional context
Follow-up to #278 and the discussion about event-log cache lifetime: #278 (comment)
Activity