Dr-Elephant with Tez support#264

Open

harishnvh wants to merge 252 commits into

linkedin:masterfrom

qubole:q-master

This pull request is big! We're only showing the most recent 250 commits

Commits on Sep 16, 2014

HADOOP-6642: Enforce code style conventions for Dr. Elephant
fli
committed

Commits on Sep 18, 2014

HADOOP-6635: Mapper Spill Heuristic added
nntnag17
committed

Commits on Sep 19, 2014

HADOOP-5675: Dr. Elephant Search trims leading and trailing whitespaces
Akshay Rai
committed

Commits on Sep 26, 2014

HADOOP-6565: Pluggable heuristics feature added
nntnag17
committed

Commits on Oct 6, 2014

HADOOP-7151: Dr. Elephant - Updating 'Mapper Speed' and 'Mapper Input Size' help pages
Akshay Rai
committed

Commits on Oct 7, 2014

Commits on Oct 8, 2014

Commits on Oct 14, 2014

HADOOP-5357: Logging for Dr. Elephant
Anant Nag
committed

Commits on Oct 15, 2014

HADOOP-7474: Dr. Elephant successfully filters HadoopJava jobs
Akshay Rai
committed

Commits on Oct 16, 2014

HADOOP-7420: Filtering pig jobs using pig.script property
Akshay Rai
committed

Commits on Oct 23, 2014

Commits on Nov 6, 2014

HADOOP-7810: Update Dr.Elephant version number to 0.6.1-SNAPSHOT
fli
committed

Commits on Nov 11, 2014

HADOOP-7581: Fix for filtering jobs by Start Date in Hadoop 2
Akshay Rai
committed

Commits on Nov 12, 2014

Commits on Nov 13, 2014

Commits on Nov 26, 2014

Fixed the build script.
mcvsubbu
committed

Commits on Dec 1, 2014

Commits on Dec 2, 2014

Commits on Dec 11, 2014

HADOOP-8316: Update Dr.Elephant version number to 0.6.4-SNAPSHOT
fli
committed

Commits on Dec 13, 2014

HADOOP-8096: Property 'username' is converted to lowercase before supplying it to the query instead of asking the query to perform the case-insensitive comparison.
Akshay Rai
committed

Commits on Dec 15, 2014

[HADOOP-7814] Prepare to add metrics library to dr-elephant
mcvsubbu
committed

Commits on Dec 23, 2014

[HADOOP-7814] Publish counters via dr-elephant
mcvsubbu
committed

Commits on Jan 16, 2015

[HADOOP-8640] Fix hang in dr-elephant due to h1 and h2 incompatibility
mcvsubbu
committed

Commits on Jan 21, 2015

Commits on Jan 29, 2015

HADOOP-8294: Improved slow performance of join queries
Akshay Rai
committed

Commits on Feb 2, 2015

HADOOP-5369: Paginate search results of Dr. Elephant
Akshay Rai
committed

Commits on Feb 3, 2015

HADOOP-7948: Make Dr. Elephant's Reducer Time Moderate Severity cut-off 51 reducers instead of 50
Fangshi Li
committed
HADOOP-8320: Add detailed suggestions in Dr. Elephant help page
Fangshi Li
committed
HADOOP-8320: Add detailed suggestions in Dr. Elephant help page
Fangshi Li
committed
HADOOP-8856: Fix a broken Dr.Elephant test case for reducer time heuristic
Fangshi Li
committed

Commits on Feb 4, 2015

HADOOP-8859: Add ideal task time suggestion in Dr. Elephant 's reducer time help page
Fangshi Li
committed
HADOOP-8846: Release Dr. Elephant v0.6.5
Fangshi Li
committed

Commits on Feb 9, 2015

HADOOP-8929: Update Dr.Elephant version number to 0.6.6-SNAPSHOT
Fangshi Li
committed

Commits on Mar 30, 2015

HADOOP-9685: Dr. Elephant fails to fetch task counter data on hadoop2
Fangshi Li
committed

Commits on Mar 31, 2015

HADOOP-8941: Further improve search performance for Dr. Elephant
Akshay Rai
committed

Commits on Apr 2, 2015

HADOOP-9714: Add Pluggable jobtype to Dr. Elephant
Fangshi Li
committed
HADOOP-9716: Dr. Elephant rest interface needs search endpoint
Fangshi Li
committed
HADOOP-9853: Release Dr. Elephant v0.6.6
Fangshi Li
committed

Commits on Apr 6, 2015

HADOOP-9886: Fix for making Dr. Elephant's pagination thread safe
Akshay Rai
committed

Commits on Apr 20, 2015

HADOOP-10289: Update Dr. Elephant version number to 0.6.7-SNAPSHOT
Fangshi Li
committed
HADOOP-10089: Dr. Elephant misses on type of Voldemort bnp job
Fangshi Li
committed
HADOOP-10301: Dr. Elephant occasionally misses a job
Fangshi Li
committed

Commits on Apr 21, 2015

Commits on Apr 25, 2015

[HADOOP-10405] Fix one-off error in getting cluster name from NN hostname
mcvsubbu
committed

Commits on Jun 12, 2015

HADOOP-11768: Update Dr. Elephant version to v1.0.0-SNAPSHOT
Fangshi Li
committed

Commits on Jun 15, 2015

Commits on Jun 17, 2015

HADOOP-11876: Separate Dr. Elephant configs from deployable zip
Fangshi Li
committed

Commits on Jun 18, 2015

HDOOP-11456: Change Dr. Elephant's heuristic from MapperInputSize to MapperTime
Fangshi Li
committed
HADOOP-11922: Fix Dr. Elephant search by heuristic field
Fangshi Li
committed
HADOOP-11912: Release Dr. Elephant v1.0.0
Fangshi Li
committed

Commits on Jun 22, 2015

HADOOP-12013: Dr. Elephant has a corner case to analyze duplicate entries
Fangshi Li
committed

Commits on Jun 29, 2015

HADOOP-12247 - Update Dr. Elephant version to v1.0.1-SNAPSHOT
Fangshi Li
committed

Commits on Jul 9, 2015

HADOOP-11913: Fix Dr. Elephant broken tests
Fangshi Li
committed
HADOOP-12067: Dr. Elephant heuristic is too lenient to jobs with long running tasks
Fangshi Li
committed

Commits on Jul 10, 2015

HADOOP-12023: Dr. Elephagent new heuristics: memory util ratio
Fangshi Li
committed
HADOOP-12593: Release Dr. Elephant v1.0.1
Fangshi Li
committed

Commits on Jul 13, 2015

HADOOP-12632: Enable Dr. Elephant 's mapper spill heuristics
Fangshi Li
committed

Commits on Jul 14, 2015

HADOOP-12669: Dr. Elephant's mapper spill heuristic is too harsh
Fangshi Li
committed
HADOOP-12674: Release Dr. Elephant v1.0.2
Fangshi Li
committed
HADOOP-12603: Dr. Elephant has a failed test in Mapper Speed Heuristic
Fangshi Li
committed
HADOOP-12603: Update Mapper Spill Heuristic help page
Fangshi Li
committed

Commits on Jul 15, 2015

Commits on Jul 27, 2015

LIHADOOP-13019, LIHADOOP-13020: Upgrade to Spark 1.4, and add event log limit for Spark log fetcher
timyitong
committed

Commits on Jul 28, 2015

LIHADOOP-13018: Dr. Elephant 's Spill heuristic should be lenient to small number of tasks
Fangshi Li
committed

Commits on Jul 29, 2015

LIHADOOP-13094: Fix Dr. Elephant build on Jenkins after spark1.4 upgrade
Fangshi Li
committed
LIHADOOP-13016: Release Dr. Elephant v1.0.3
Fangshi Li
committed

Commits on Aug 21, 2015

Adjust StageRuntime heuristics so that it's the real duration time instead of total aggregated executor time
timyitong
committed

Commits on Aug 24, 2015

LIHADOOP-13747 Update Spark driver name to be consistent with Spark 1.4.
Victsm
committed

Commits on Sep 1, 2015

LIHADOOP-13953 The start script should take arguments to enable evolutions
Akshay Rai
committed
LIHADOOP-5177 Update documentation of Dr. Elephant
Akshay Rai
committed

Commits on Sep 9, 2015

LIHADOOP-12802 Occasional Concurrent Modification Exception during Dr. Elephant start
Akshay Rai
committed

Commits on Sep 11, 2015

LIHADOOP-8114 [Opensource Dr. E] Apache License and LinkedIn Copyright Notices

akshayrai
authored and
Akshay Rai
committed
Minor formatting change in Readme License
Akshay Rai
committed

Commits on Sep 14, 2015

Commits on Sep 29, 2015

LIHADOOP-14322 Dr. Elephant feature request. The ability to compare result for two flow executions
Akshay Rai
committed

Commits on Sep 30, 2015

LIHADOOP-14722: Adjust some Spark heuristic parameters of Dr Elephant
timyitong
committed

Commits on Oct 8, 2015

LIHADOOP-14688 Adding a timestamp to Dr. Elephant search result page
Akshay Rai
committed

Commits on Oct 14, 2015

LIHADOOP-13819: Dr. Elephant heuristic request. GC efficiency
Fangshi Li
committed
LIHADOOP-13711: Dr. Elephant should add virtual memory info in its memory heuristics
Fangshi Li
committed
LIHADOOP-15134: Dr. Elephant should provide exception stacktrace from failed jobs
Fangshi Li
committed
LIHADOOP-14342: Dr. Elephant's memory heuristic has a corner case to throw NumberFormatException
Fangshi Li
committed
LIHADOOP-15136: Release Dr. Elephant v1.0.5
Fangshi Li
committed

Commits on Oct 22, 2015

LIHADOOP-15080 Clean up and organize the views in Dr. Elephant
Akshay Rai
committed
LIHADOOP-15252: Update Dr. Elephant version to v1.0.6-SNAPSHOT
Fangshi Li
committed

Commits on Oct 26, 2015

LIHADOOP-15372 Update the Heuristic Conf for helpGC with the proper package name
akshayrai
committed

Commits on Oct 28, 2015

LIHADOOP-15377 Job execution URLs on Dr. Elephant's Flow Execution page is incorrect
akshayrai
committed

Commits on Dec 18, 2015

LIHADOOP-16125 Other executions of a job should list the recent jobs from history firBt
Akshay Rai
committed

Commits on Dec 23, 2015

LIHADOOP-15217 Implement comparison of historical flows/jobs in Dr. Elephant
Akshay Rai
committed
LIHADOOP-16102 Discuss and remove Hadoop 1 support from Dr. Elephant
Akshay Rai
committed

Commits on Jan 6, 2016

LIHADOOP-9004 Support Hadoop 2 properties in Dr. Elephant
Akshay Rai
committed

Commits on Jan 14, 2016

LIHADOOP-16687 Update Dr. Elephant to use the new Hadoop API
Akshay Rai
committed

Commits on Jan 29, 2016

LIHADOOP-16576 Make severity threshold values configurable for the heuristics.

Akshay Rai
authored and
akshayrai
committed

Commits on Feb 1, 2016

LIHADOOP-17131 Prepare Readme for Dr. Elephant
akshayrai
committed

Commits on Feb 12, 2016

LIHADOOP-16587 Dr. Elephant should provide search by start time
akshayrai
committed

Commits on Feb 15, 2016

Commits on Feb 22, 2016

LIHADOOP-16100: Dr. Elephant should show timestamp specific to timezones or at least UTC.
akshayrai
committed

Commits on Feb 25, 2016

LIHADOOP-15268: Task sampling in Dr. Elephant
Fangshi Li
committed

Commits on Mar 1, 2016

Commits on Mar 7, 2016

Commits on Mar 8, 2016

Commits on Mar 14, 2016

Commits on Apr 7, 2016

Commits on Apr 11, 2016

Commits on Apr 19, 2016

Get the active resource manager URL when HA is enabled
Krishna Puttaswamy
committed
fix start script to redirect output to a file instead of /dev/null
Krishna Puttaswamy
committed
addressed code review comments on RM HA support PR
Krishna Puttaswamy
committed
Merge pull request #15 from krishnap/kp_fix_start_script
akshayrai
committed
Merge pull request #14 from krishnap/kp_fix_rm_issue
akshayrai
committed

Commits on Apr 22, 2016

LIHADOOP-19325: [DR-Elephant] Add integration tests for the rest API's in Dr. Elephant

rajagopr
authored and
akshayrai
committed

Commits on Apr 26, 2016

Added a link to the mailing list.
akshayrai
committed

Commits on Apr 27, 2016

Adding commons codec to the dependency (#26)

tglstory
authored and
akshayrai
committed
Spark dependency resolution based on version (#25)

tglstory
authored and
akshayrai
committed
LIHADOOP-18925: Expose additional metrics through Dr Elephant ( Job Execution time, Job wait time, resource usage, wasted resources )
Anant Nag
committed

Commits on Apr 28, 2016

LIHADOOP-19491: [DR-Elephant] Add code coverage to Dr. Elephant

rajagopr
authored and
akshayrai
committed
LIHADOOP-19495 Make sampling configurable in Dr elephant
Anant Nag
committed

Commits on May 2, 2016

LIHADOOP-19610: Metrics should not be shown if runtime and resourceused are 0
Anant Nag
committed

Commits on May 4, 2016

LIHADOOP-19609: Make Spark Event Log Directory configurable
akshayrai
committed

Commits on May 5, 2016

Fix an cast error in SparkDataCollection.scala (#37)

stiga-huang
authored and
akshayrai
committed

Commits on May 6, 2016

Separate configuration logic from RM address update logic (#38)

tglstory
authored and
akshayrai
committed

Commits on May 10, 2016

LIHADOOP-19850: Test for AnalyticJob (#42)

rajagopr
authored and
akshayrai
committed

Commits on May 18, 2016

LIHADOOP-20088: Add organization name to Dr. Elephant to generate ivy file (#46)

rajagopr
authored and
akshayrai
committed

Commits on May 20, 2016

LIHADOOP-18924: Improve Job History to show runtime, waittime, resource used, resource wasted.
Anant Nag
committed

Commits on May 23, 2016

LIHADOOP-20170: Write help pages for new metrics in Dr. Elephant
Anant Nag
committed

Commits on May 26, 2016

LIHADOOP-19666: totalDelay is negative for zero tasks
Anant Nag
committed
LIHADOOP-19841: Enable Analyzing Spark jobs for Namenode HA
Anant Nag
committed
Merge remote-tracking branch 'origin/master' into AggregatedMetrics
Anant Nag
committed
Minor fixup on AnalyticJobTest
Anant Nag
committed
LIHADOOP-20350: Release dr-elephant v. 2.0.3
Anant Nag
committed

Commits on May 27, 2016

Commits on May 28, 2016

Added engineering blog link
akshayrai
committed

Commits on Jun 1, 2016

LIHADOOP-20476: Generating code coverage report in xml format too. This will be consumed by the product dashboard. (#65)

rajagopr
authored and
akshayrai
committed

Commits on Jun 6, 2016

Commits on Jun 7, 2016

LIHADOOP-20626 Ignore slow start for considering reducer delay
Anant Nag
committed
LIHADOOP-20629 Minor UI fixes for Dr. Elephant
Anant Nag
committed
Merge branch 'master' of https://github.com/linkedin/dr-elephant into AggregatedMetrics
Anant Nag
committed
LIHADOOP-20630: Release Dr.Elephant v. 2.0.4
Anant Nag
committed

Commits on Jun 8, 2016

Removing some un-necessary logic. (#70)

rajagopr
authored and
akshayrai
committed

Commits on Jun 9, 2016

Added support to search by queue-name (#40)

chetnachaudhari
authored and
akshayrai
committed

Commits on Jun 18, 2016

Try alternate resource manager if one is down (#78)

paulbramsen
authored and
akshayrai
committed
Ignore and remove .DS_Store files (#80)

paulbramsen
authored and
akshayrai
committed

Commits on Jun 22, 2016

Add GeneralConf.xml to make number of executors configurable (#75)

stiga-huang
authored and
akshayrai
committed

Commits on Jun 23, 2016

Add spark log ext conf param (#79)

paulbramsen
authored and
akshayrai
committed
Fix fallback to default `SPARK_LOG_EXT` value (#82)

ljank
authored and
akshayrai
committed

Commits on Jun 25, 2016

Add Travis CI support (#86)

paulbramsen
authored and
akshayrai
committed

Commits on Jul 1, 2016

Add Airflow scheduler support to Dr. Elephant. Includes Tests. (#83)

paulbramsen
authored and
akshayrai
committed
Make default container mem configurable (#90)

paulbramsen
authored and
akshayrai
committed

Commits on Jul 5, 2016

Web analytics for Dr. Elephant. (#94)

rajagopr
authored and
akshayrai
committed

Commits on Jul 7, 2016

Fix job history page tab highlighting (#98)

paulbramsen
authored and
akshayrai
committed
dr-elephant#95: copy conf files to zip package automatically (#96)

YannByron
authored and
akshayrai
committed

Commits on Jul 14, 2016

Add Gitter badge (#102)

gitter-badger
authored and
akshayrai
committed
Put build and gitter badges on the same line (#105)

paulbramsen
authored and
akshayrai
committed

Commits on Jul 15, 2016

Update wiki images for aggregated metrics feature.
Anant Nag
committed
Merge remote-tracking branch 'origin/AggregatedMetrics'
Anant Nag
committed

Commits on Jul 18, 2016

Add 3.sql and revert modifications in 1.sql
Anant Nag
committed

Commits on Jul 20, 2016

Merge pull request #109 from nntnag17/master
nntnag17
authored

Commits on Jul 21, 2016

Make flow search more powerful (#99)

paulbramsen
authored and
akshayrai
committed
Add thread to all log output for better debugging (#108)

paulbramsen
authored and
akshayrai
committed

Commits on Jul 22, 2016

Fix Flow/Job ID/Url mixups (#100)

paulbramsen
authored and
akshayrai
committed

Commits on Jul 25, 2016

fix a log error when throttling spark event log (#114)

stiga-huang
authored and
akshayrai
committed

Commits on Jul 27, 2016

LIHADOOP-21568: Add Dropwizard metrics to Dr. Elephant. This change enables metrics by default. (#115)

rajagopr
authored and
akshayrai
committed

Commits on Jul 28, 2016

Add default for boolean unboxing (#119)

paulbramsen
authored and
akshayrai
committed

Commits on Aug 1, 2016

Fix sync issue in info extractor (#118)

paulbramsen
authored and
akshayrai
committed
Modify main run loop to be inline with thread pool executor paradigms (#121)

paulbramsen
authored and
akshayrai
committed
Faster fetcher for MapReduce apps (#67)

stiga-huang
authored and
akshayrai
committed

Commits on Aug 2, 2016

Show result count in search (#120)

paulbramsen
authored and
akshayrai
committed
LIHADOOP-21657: Release Dr. Elephant v. 2.0.5 (#117)

rajagopr
authored and
akshayrai
committed

Commits on Aug 8, 2016

Add rest interface for getting resource usage by user (#125)

shankar37
authored and
akshayrai
committed
Improved metrics (#126)

paulbramsen
authored and
akshayrai
committed

Commits on Aug 29, 2016

Add Metrics feature for Spark Jobs (#135)

shankar37
authored and
akshayrai
committed

Commits on Sep 14, 2016

Changes to load an agent jar along with Dr. Elephant. Current use-case is to publish the metrics
rajagopr
committed

Commits on Oct 3, 2016

Add rest end point for new user interface
Anant Nag
committed

Commits on Oct 11, 2016

LIHADOOP-23923: Add rest end points for search results and search options
Anant Nag
committed

Commits on Oct 18, 2016

Commits on Oct 19, 2016

Added License to all the new files
annag
committed

Commits on Oct 24, 2016

Integration of the ember app with play framework
Anant Nag
committed

Commits on Oct 26, 2016

Commits on Nov 7, 2016

Commits on Nov 9, 2016

Multiple fixes: (#161)
nntnag17
authored

Commits on Dec 29, 2016

Build Dr. Elephant with Qubole Jars. (#3)
vrajat
authored

Commits on Jan 12, 2017

new:dev:#5:Support publish to Qubole's maven repository (#8)
vrajat
authored
Setting version to 2.0.6-qds-0.4.0
Rajat Venkatesh
committed
Setting version to 2.0.6-qds-0.4.1-SNAPSHOT
Rajat Venkatesh
committed
new:dev:Add support for Qubole's internal packaging tools
Rajat Venkatesh
committed

Commits on Jan 19, 2017

new: dev: Added preliminary support for Tez(#10)

abhishekdas99
authored and
Abhishek Das
committed

Commits on Mar 10, 2017

chg:usr:Setting version to 2.0.6-qds-0.5.0
Rajat Venkatesh
committed
Setting version to 2.0.6-qds-0.5.1-SNAPSHOT
Rajat Venkatesh
committed
chg:dev:Set version for next release of QDS
Rajat Venkatesh
committed