Skip to content

[FEA] Keep the YARN log cache from growing forever #280

Open

Description

@gerashegalov

Is your feature request related to a problem? Please describe.

On a long-running cluster, the cache keeps information from old ResourceManager logs and can grow without a limit.

Describe the solution you'd like

Add a way to remove old data or limit how far back the tool looks. It must still calculate costs correctly for recent jobs. If someone asks about an older job, the tool should either reload what it needs or clearly say that the job is outside the configured history window.

Test this with many rotated logs, a ResourceManager restart, and both recent and older jobs.

Describe alternatives you've considered

Operators can clear the disposable cache while callers are stopped, but that loses all saved work and does not prevent it from growing again.

Additional context

Follow-up to #278 and the review discussion: #278 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions