-
Notifications
You must be signed in to change notification settings - Fork 843
POC of code level optimization recommendation for Hive #618
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
pralabhkumar
wants to merge
10
commits into
linkedin:tuning_20190221
Choose a base branch
from
pralabhkumar:poc_code_level_optimization
base: tuning_20190221
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 2 commits
Commits
Show all changes
10 commits
Select commit
Hold shift + click to select a range
4009da6
First Version (MVP) of code level optimization for Hive
pralabhkumar 1497c4f
Changes to details page of code level heuristic
pralabhkumar 07a6c7d
Restructruting of the code and unit test cases for Code Extractor
pralabhkumar 9c20e27
Unit test cases added for Code Optimizer
pralabhkumar ddd407a
Unit test cases added for whole level optimization, Restructing done
pralabhkumar f10b8f5
Indentation changes
pralabhkumar 8d31614
Removed unnecessary from AnalyticJobGeneratorHadoop2
pralabhkumar b6fd45c
Restructured and removal of unnecessary code
pralabhkumar 07d1e4e
Added 10.sql
pralabhkumar ef0e609
Changes in exception handling
pralabhkumar File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| package com.linkedin.drelephant.analysis.code; | ||
|
|
||
| import java.util.ArrayList; | ||
| import java.util.List; | ||
|
|
||
|
|
||
| public class Code { | ||
| private List<Statement> statements = null; | ||
| public Code(){ | ||
| statements = new ArrayList<>(); | ||
| } | ||
|
|
||
| public List<Statement> getStatements() { | ||
| return statements; | ||
| } | ||
|
|
||
| public void setStatements(List<Statement> statements) { | ||
| this.statements = statements; | ||
| } | ||
|
|
||
| @Override | ||
| public String toString() { | ||
| return "Code{" + "statements=" + statements + '}'; | ||
| } | ||
| } |
28 changes: 28 additions & 0 deletions
28
app/com/linkedin/drelephant/analysis/code/CodeAnalyzerException.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| package com.linkedin.drelephant.analysis.code; | ||
|
|
||
| import java.io.IOException; | ||
| import org.apache.log4j.Logger; | ||
| import org.codehaus.jettison.json.JSONException; | ||
|
|
||
|
|
||
| /** | ||
| * This is wrapper for exceptions | ||
| */ | ||
| public class CodeAnalyzerException extends Exception { | ||
| private static final Logger logger = Logger.getLogger(CodeAnalyzerException.class); | ||
|
|
||
| public CodeAnalyzerException(IOException exception) { | ||
| super(exception); | ||
| logger.error("Unable to get code ", exception); | ||
| } | ||
|
|
||
| public CodeAnalyzerException(JSONException exception) { | ||
| super(exception); | ||
| logger.error("Unable to parse JSON ", exception); | ||
| } | ||
|
|
||
| public CodeAnalyzerException(Exception exception) { | ||
| super(exception); | ||
| logger.error("Unknown exception have come , catching this , to not halt the system because of unknown exception ", exception); | ||
| } | ||
| } |
17 changes: 17 additions & 0 deletions
17
app/com/linkedin/drelephant/analysis/code/CodeExtractor.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| package com.linkedin.drelephant.analysis.code; | ||
|
|
||
| import java.io.IOException; | ||
| import java.net.MalformedURLException; | ||
| import models.AppResult; | ||
| import org.codehaus.jettison.json.JSONException; | ||
|
|
||
|
|
||
| public interface CodeExtractor { | ||
| boolean isCodeNameExtractible(AppResult appResult); | ||
|
|
||
| String codeFileNameExtractor(AppResult appResult) throws MalformedURLException; | ||
|
|
||
| JobCodeInfoDataSet codeInfoExtractor(String codeFileName) throws IOException, JSONException; | ||
|
|
||
| JobCodeInfoDataSet execute(AppResult appResult) throws CodeAnalyzerException; | ||
| } |
10 changes: 10 additions & 0 deletions
10
app/com/linkedin/drelephant/analysis/code/CodeOptimizationRule.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| package com.linkedin.drelephant.analysis.code; | ||
|
|
||
| import com.linkedin.drelephant.analysis.Severity; | ||
|
|
||
|
|
||
| public interface CodeOptimizationRule { | ||
| void processRule(Script script); | ||
| String getRuleName(); | ||
| String getSeverity(); | ||
| } |
10 changes: 10 additions & 0 deletions
10
app/com/linkedin/drelephant/analysis/code/CodeOptimizer.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| package com.linkedin.drelephant.analysis.code; | ||
|
|
||
| import com.linkedin.drelephant.analysis.Severity; | ||
|
|
||
|
|
||
| public interface CodeOptimizer { | ||
| Code generateCode (String inputFileName); | ||
| Script execute(String inputFileName); | ||
| String getSeverity(); | ||
| } |
83 changes: 83 additions & 0 deletions
83
app/com/linkedin/drelephant/analysis/code/JobCodeInfoDataSet.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,83 @@ | ||
| package com.linkedin.drelephant.analysis.code; | ||
|
|
||
| public class JobCodeInfoDataSet { | ||
| private String jobExecutionID = null; | ||
| private String jobName = null; | ||
| private String flowName = null; | ||
| private String projectName = null; | ||
| private String fileName = null; | ||
| private String sourceCode = null; | ||
| private String scmType = null; | ||
| private String repoName = null; | ||
|
|
||
| public String getRepoName() { | ||
| return repoName; | ||
| } | ||
|
|
||
| public void setRepoName(String repoName) { | ||
| this.repoName = repoName; | ||
| } | ||
|
|
||
| public String getScmType() { | ||
| return scmType; | ||
| } | ||
|
|
||
| public void setScmType(String scmType) { | ||
| this.scmType = scmType; | ||
| } | ||
|
|
||
| public String getJobName() { | ||
| return jobName; | ||
| } | ||
|
|
||
| public void setJobName(String jobName) { | ||
| this.jobName = jobName; | ||
| } | ||
|
|
||
| public String getFlowName() { | ||
| return flowName; | ||
| } | ||
|
|
||
| public void setFlowName(String flowName) { | ||
| this.flowName = flowName; | ||
| } | ||
|
|
||
| public String getProjectName() { | ||
| return projectName; | ||
| } | ||
|
|
||
| public void setProjectName(String projectName) { | ||
| this.projectName = projectName; | ||
| } | ||
|
|
||
| public String getFileName() { | ||
| return fileName; | ||
| } | ||
|
|
||
| public void setFileName(String fileName) { | ||
| this.fileName = fileName.replaceAll("src/",""); | ||
| } | ||
|
|
||
| public String getSourceCode() { | ||
| return sourceCode; | ||
| } | ||
|
|
||
| public void setSourceCode(String sourceCode) { | ||
| this.sourceCode = sourceCode; | ||
| } | ||
|
|
||
| public String getJobExecutionID() { | ||
| return jobExecutionID; | ||
| } | ||
|
|
||
| public void setJobExecutionID(String jobExecutionID) { | ||
| this.jobExecutionID = jobExecutionID; | ||
| } | ||
|
|
||
| @Override | ||
| public String toString() { | ||
| return "JobCodeInfoDataSet{" + "jobExecutionID='" + jobExecutionID + '\'' + ", jobName='" + jobName + '\'' | ||
| + ", flowName='" + flowName + '\'' + ", projectName='" + projectName + '\'' + ", fileName='" + fileName + '\'' | ||
| + ", scmType='" + scmType + '\'' + '}'; | ||
| } | ||
| } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Giving out internal queue configuration. At a minimum make this configurable. Better model the conditional execution as an interface? Applies elsewhere too