Skip to content

[FEA] Delete cached Spark event logs when they are no longer needed #281

Open

Description

@gerashegalov

Is your feature request related to a problem? Please describe.

The tool saves event logs so a timed-out attempt can pick up where it left off. Once it has finished calculating a job's cost, those saved logs usually serve no further purpose, but they stay on disk.

Describe the solution you'd like

Keep the files while a job still needs retries, then remove them after a successful calculation. Do not remove a file that another calculation is using. Test retries, successful cleanup, and calculating the same job again later.

Describe alternatives you've considered

Operators can clear the cache while callers are stopped, but that also removes useful progress for other jobs still awaiting retries.

Additional context

Follow-up to #278 and the discussion about event-log cache lifetime: #278 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions