Search in sources :

Example 21 with PipeliteException

use of pipelite.exception.PipeliteException in project pipelite by enasequence.

the class ProcessRunnerPoolManager method createPools.

public synchronized void createPools() {
    log.atInfo().log("Creating process runner pools");
    try {
        _createPipelineRunners();
        _createScheduleRunner();
        log.atInfo().log("Created process runner pools");
        _createServiceManager();
    } catch (Exception ex) {
        log.atSevere().withCause(ex).log("Unexpected exception when creating process runner pools");
        clear();
        throw new PipeliteException(ex);
    }
}
Also used : PipeliteException(pipelite.exception.PipeliteException) TimeoutException(java.util.concurrent.TimeoutException) PipeliteException(pipelite.exception.PipeliteException)

Example 22 with PipeliteException

use of pipelite.exception.PipeliteException in project pipelite by enasequence.

the class ProcessRunnerPoolManager method _createServiceManager.

/**
 * Should not be called directly. Called by {@link #createPools()}.
 */
public synchronized void _createServiceManager() {
    log.atInfo().log("Creating service manager for process runner pools");
    if (pools.isEmpty()) {
        log.atSevere().log("Failed to create service manager for process runner pools because none exist");
        return;
    }
    try {
        serviceManager = new ServiceManager(pools);
        serviceManager.addListener(new ServiceManager.Listener() {

            public void stopped() {
            }

            public void healthy() {
            }

            public void failure(Service service) {
                String serviceName = ((ProcessRunnerPool) service).serviceName();
                log.atSevere().withCause(service.failureCause()).log("Process runner pool has failed: " + serviceName);
                pipeliteServices.internalError().saveInternalError(serviceName, this.getClass(), service.failureCause());
                stopPools();
            }
        }, MoreExecutors.directExecutor());
    } catch (Exception ex) {
        log.atSevere().withCause(ex).log("Unexpected exception when creating service manager for process runner pools");
        clear();
        throw new PipeliteException(ex);
    }
    state = State.INITIALISED;
    log.atInfo().log("Created service manager for process runner pools");
}
Also used : ServiceManager(com.google.common.util.concurrent.ServiceManager) Service(com.google.common.util.concurrent.Service) PipeliteException(pipelite.exception.PipeliteException) TimeoutException(java.util.concurrent.TimeoutException) PipeliteException(pipelite.exception.PipeliteException)

Example 23 with PipeliteException

use of pipelite.exception.PipeliteException in project pipelite by enasequence.

the class AwsBatchExecutor method submit.

@Override
protected SubmitResult submit(StageExecutorRequest request) {
    logContext(log.atFine(), request).log("Submitting AWSBatch job.");
    AwsBatchExecutorParameters params = getExecutorParams();
    region = params.getRegion();
    SubmitJobRequest submitJobRequest = new SubmitJobRequest().withJobName(awsBatchJobName(request)).withJobQueue(params.getQueue()).withJobDefinition(params.getDefinition()).withParameters(params.getParameters()).withTimeout(new JobTimeout().withAttemptDurationSeconds((int) params.getTimeout().getSeconds()));
    // TODO: .withContainerOverrides()
    AWSBatch awsBatch = awsBatchClient(region);
    SubmitJobResult submitJobResult = RetryTask.DEFAULT.execute(r -> awsBatch.submitJob(submitJobRequest));
    if (submitJobResult == null || submitJobResult.getJobId() == null) {
        throw new PipeliteException("Missing AWSBatch submit job id.");
    }
    String jobId = submitJobResult.getJobId();
    logContext(log.atInfo(), request).log("Submitted AWSBatch job " + getJobId());
    return new SubmitResult(jobId, StageExecutorResult.submitted());
}
Also used : AWSBatch(com.amazonaws.services.batch.AWSBatch) AwsBatchExecutorParameters(pipelite.stage.parameters.AwsBatchExecutorParameters) PipeliteException(pipelite.exception.PipeliteException)

Aggregations

PipeliteException (pipelite.exception.PipeliteException)23 StageExecutorResult (pipelite.stage.executor.StageExecutorResult)6 KubernetesClientException (io.fabric8.kubernetes.client.KubernetesClientException)5 DefaultKubernetesClient (io.fabric8.kubernetes.client.DefaultKubernetesClient)4 KubernetesClient (io.fabric8.kubernetes.client.KubernetesClient)4 Pod (io.fabric8.kubernetes.api.model.Pod)2 ByteArrayOutputStream (java.io.ByteArrayOutputStream)2 IOException (java.io.IOException)2 OutputStream (java.io.OutputStream)2 Duration (java.time.Duration)2 ZonedDateTime (java.time.ZonedDateTime)2 TimeoutException (java.util.concurrent.TimeoutException)2 Pipeline (pipelite.Pipeline)2 Schedule (pipelite.Schedule)2 ProcessEntity (pipelite.entity.ProcessEntity)2 ScheduleEntity (pipelite.entity.ScheduleEntity)2 PipeliteMetrics (pipelite.metrics.PipeliteMetrics)2 ProcessBuilder (pipelite.process.builder.ProcessBuilder)2 PipeliteServices (pipelite.service.PipeliteServices)2 AWSBatch (com.amazonaws.services.batch.AWSBatch)1