Auto Tuning Changes in Dr Elephant #1
Open
mkumar1984 wants to merge 475 commits into
Open
Conversation
* adding more metrics for monitoring process jobs count and latency * Fix metrics null ptr API usage issues * Add jobs processed in last 24 hours metric
Includes resource usage, resource wastage and runtime metrics for Spark.
…e is to publish the metrics to other applications. The property is disabled by default and users who wish to make use of this specify their own agent jar.
LIHADOOP-23967: Setup JobHistory, FlowHistory and help pages for new UI
Other fixes:
* Additional information to workflow, workflow-summaries rest end point
* Added an rest end point for the dashboard summary
Integration of the Dr. Elephant ember app with play framework
…inkedin#158) * Adding tests for SparkAggregated Metrics * Fix code review comments * Make Memory Buffer Percentage Configurable. Disable Schedule Delay for spark * Fix code to not use getOrDefault * use parseDouble instead for faster performance * return resource usage by user in GB Hours * Handle NumberFormatException when map/reduce memory config is not defined * build fix
* Added not-found page * Search panel on pages served by play * Piwik integration
LIHADOOP-24687 Dr Elephant shows more than 100% waittime for some jobs
LIHADOOP-24688 Dr Elephant shows NAN when the resource usage and resource wasted is zero
LIHADOOP-24695 Dr Elephant should show loading panel on search page
…inkedin#169) * logger actually returned number of application types, not job types * also log the appType size
…before running heuristic (linkedin#168)
…ows (linkedin#167) also fixed testDepthCalculation - our test workflow only has one parent so Depth should be 1.
* Rewrite Spark fetcher/heuristics. The purpose of this update is to: - rewrite the Spark data fetcher to use Spark event logs minimally, since it can be expensive to download and process these fully as done before - rewrite the Spark data fetcher to use the [Spark monitoring REST API](https://spark.apache.org/docs/1.4.1/monitoring.html#rest-api), which provides almost all of the information Spark heuristics need - update the Spark heuristics to provide hopefully more useful information and avoid being arbitrarily restrictive The new Spark-related code is provided in parallel to the old Spark-related code. To enable it: - Uncomment and swap in the appropriate fragments in `AggregatorConf.xml`, `FetcherConf.xml`, and `HeuristicConf.xml`. - Set `SPARK_CONF_DIR` (or `SPARK_HOME`) to an appropriate location so that Dr. Elephant can find `spark-defaults.conf`. Heuristics added: - "Executor shuffle read bytes distribution": We now provide a distribution with min/25p/median/75p/max, with severity based on a max-to-median ratio. - "Executor shuffle write bytes distribution": We now provide a distribution with min/25p/median/75p/max, with severity based on a max-to-median ratio. Heuristics changed: - "Average input size" -> "Executor input bytes distribution": Instead of providing an average along with min/max, we now provide a distribution with min/25p/median/75p/max, with severity based on a max-to-median ratio. - "Average peak storage memory" -> "Executor storage memory used distribution": Instead of providing an average along with min/max, we now provide a distribution with min/25p/median/75p/max, with severity based on a max-to-median ratio. - "Average runtime" -> "Executor task time distribution": Instead of providing an average along with min/max, we now provide a distribution with min/25p/median/75p/max, with severity based on a max-to-median ratio. - "Memory utilization rate" -> "Executor storage memory utilization rate": This seemed to imply total memory but it is just the utilization rate for storage memory, so has been relabeled to indicate that. Shuffle memory is important too (but we don't seem to have access to shuffle memory utilization metrics). - "Total memory used at peak" -> "Total executor storage memory used": This also refers to storage memory. It has been relabeled to indicate that. - "Spark problematic stages" -> ("Spark stages with high task failure rates", "Spark stages with long average executor runtimes"): This was a combination of stages with high task failure rates and those with long runtimes. Those have been separated. Heuristics removed: - spark.executor.cores: I think this is somewhat discretionary. At the very least, our internal recommendation stopped matching the one in Dr. Elephant. - spark.shuffle.manager: This was changed to "sort" by default as of Spark 1.2, so there is no current use for checking this setting. - "Average output size": Metrics related to output size appear to be deprecated or non-existent, so there is no current use for checking this setting. Finally, overall waste metrics are calculated based on allocation [app runtime * # of executors * executor memory] vs. usage [total executor run time * executor memory]. They were previously calculated based only on storage memory and some 50% buffer, which I didn't understand. Added unit tests and also tested against our internal cluster as much as practically I could. Will need help to fully validate.
* Fix default (mb) and verify memory size >= 0. * Parse the string that's already been retrieved from the config.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR contains auto tuning changes for Dr Elephant. It contains API for auto tuning. It also contains daemon for job complete detector, baseline computation, fitness computation and param generator.