Search in sources :

Example 1 with AzkabanJobCompleteDetector

use of com.linkedin.drelephant.tuning.AzkabanJobCompleteDetector in project dr-elephant by linkedin.

the class AutoTuner method run.

public void run() {
    logger.info("Starting Auto Tuning thread");
    HDFSContext.load();
    Configuration configuration = ElephantContext.instance().getAutoTuningConf();
    Long interval = Utils.getNonNegativeLong(configuration, AUTO_TUNING_DAEMON_WAIT_INTERVAL, DEFAULT_METRICS_COMPUTATION_INTERVAL);
    try {
        AutoTuningMetricsController.init();
        BaselineComputeUtil baselineComputeUtil = new BaselineComputeUtil();
        FitnessComputeUtil fitnessComputeUtil = new FitnessComputeUtil();
        ParamGenerator paramGenerator = new PSOParamGenerator();
        JobCompleteDetector jobCompleteDetector = new AzkabanJobCompleteDetector();
        while (!Thread.currentThread().isInterrupted()) {
            try {
                baselineComputeUtil.computeBaseline();
                jobCompleteDetector.updateCompletedExecutions();
                fitnessComputeUtil.updateFitness();
                paramGenerator.getParams();
            } catch (Exception e) {
                logger.error("Error in auto tuner thread ", e);
            }
            Thread.sleep(interval);
        }
    } catch (Exception e) {
        logger.error("Error in auto tuner thread ", e);
    }
    logger.info("Auto tuning thread shutting down");
}
Also used : BaselineComputeUtil(com.linkedin.drelephant.tuning.BaselineComputeUtil) Configuration(org.apache.hadoop.conf.Configuration) AzkabanJobCompleteDetector(com.linkedin.drelephant.tuning.AzkabanJobCompleteDetector) FitnessComputeUtil(com.linkedin.drelephant.tuning.FitnessComputeUtil) PSOParamGenerator(com.linkedin.drelephant.tuning.PSOParamGenerator) AzkabanJobCompleteDetector(com.linkedin.drelephant.tuning.AzkabanJobCompleteDetector) JobCompleteDetector(com.linkedin.drelephant.tuning.JobCompleteDetector) PSOParamGenerator(com.linkedin.drelephant.tuning.PSOParamGenerator) ParamGenerator(com.linkedin.drelephant.tuning.ParamGenerator)

Aggregations

AzkabanJobCompleteDetector (com.linkedin.drelephant.tuning.AzkabanJobCompleteDetector)1 BaselineComputeUtil (com.linkedin.drelephant.tuning.BaselineComputeUtil)1 FitnessComputeUtil (com.linkedin.drelephant.tuning.FitnessComputeUtil)1 JobCompleteDetector (com.linkedin.drelephant.tuning.JobCompleteDetector)1 PSOParamGenerator (com.linkedin.drelephant.tuning.PSOParamGenerator)1 ParamGenerator (com.linkedin.drelephant.tuning.ParamGenerator)1 Configuration (org.apache.hadoop.conf.Configuration)1