Search in sources :

Example 1 with WorkerContext

use of org.bf2.srs.fleetmanager.execution.manager.WorkerContext in project srs-fleet-manager by bf2fc6cc711aee1a0c2a.

the class DeprovisionRegistryWorker method finallyExecute.

@Transactional
@Override
public void finallyExecute(Task aTask, WorkerContext ctl, Optional<Exception> error) throws RegistryNotFoundException, RegistryStorageConflictException {
    DeprovisionRegistryTask task = (DeprovisionRegistryTask) aTask;
    Optional<RegistryData> registry = storage.getRegistryById(task.getRegistryId());
    if (registry.isPresent()) {
        var reg = registry.get();
        // Failure - Could not delete tenant or update status
        // Try updating status to failed, otherwise user can retry.
        reg.setStatus(RegistryStatusValueDto.FAILED.value());
        // TODO Add failed_reason
        storage.createOrUpdateRegistry(reg);
        log.warn("Failed to deprovision Registry: {}", registry);
    } else {
        // SUCCESS
        log.debug("Registry (ID = {}) has been deleted.", task.getRegistryId());
    }
}
Also used : DeprovisionRegistryTask(org.bf2.srs.fleetmanager.execution.impl.tasks.deprovision.DeprovisionRegistryTask) RegistryData(org.bf2.srs.fleetmanager.storage.sqlPanacheImpl.model.RegistryData) Transactional(javax.transaction.Transactional)

Example 2 with WorkerContext

use of org.bf2.srs.fleetmanager.execution.manager.WorkerContext in project srs-fleet-manager by bf2fc6cc711aee1a0c2a.

the class DeprovisionRegistryWorker method execute.

@Transactional
@Override
public void execute(Task aTask, WorkerContext ctl) throws RegistryStorageConflictException, RegistryNotFoundException, AccountManagementServiceException, TenantManagerServiceException {
    var task = (DeprovisionRegistryTask) aTask;
    var registryOptional = storage.getRegistryById(task.getRegistryId());
    if (registryOptional.isPresent()) {
        // FAILURE POINT 1
        var registry = registryOptional.get();
        RegistryDeploymentData registryDeployment = registry.getRegistryDeployment();
        // FAILURE POINT 2
        if (task.getRegistryTenantId() == null) {
            final var tenantId = registry.getId();
            TenantManagerConfig tenantManagerConfig = Utils.createTenantManagerConfig(registryDeployment);
            try {
                tms.deleteTenant(tenantManagerConfig, tenantId);
                log.debug("Tenant id='{}' delete request send.", tenantId);
            } catch (TenantNotFoundServiceException ex) {
                log.info("Tenant id='{}' does not exist (already deleted?).", tenantId);
            }
            task.setRegistryTenantId(tenantId);
        }
        /* Return AMS entitlement
             * FAILURE POINT 3
             * Recovery: We recover by setting the registry status to failed so we don't lose information
             *   and the process can be initiated again.
             * Reentrancy: If the registry was already returned, (i.e. failed in #3)
             *   we need to continue without raising an error, otherwise we will keep retrying.
             */
        if (!task.isAmsSuccess()) {
            final String subscriptionId = registry.getSubscriptionId();
            // TODO Workaround: Remove this once we have RHOSRTrial working.
            if (subscriptionId != null && RegistryInstanceTypeValueDto.of(registry.getInstanceType()) != RegistryInstanceTypeValueDto.EVAL) {
                try {
                    ams.deleteSubscription(subscriptionId);
                } catch (SubscriptionNotFoundServiceException ex) {
                    log.info("Subscription ID '{}' for tenant ID '{}' does not exist (already deleted?).", subscriptionId, task.getRegistryTenantId());
                }
            } else {
                log.debug("Deleting an eval instance {} without calling AMS.", registry.getId());
            }
            task.setAmsSuccess(true);
            log.debug("Subscription (id='{}') for Registry (id='{}') deleted.", subscriptionId, registry.getId());
        }
        /* Delete the registry from DB
             * FAILURE POINT 4
             * Recovery: We set the status to failed so it can be retried.
             * Reentrancy: This is the last step, so nothing to do.
             */
        storage.deleteRegistry(registry.getId());
    } else {
        log.warn("Registry id='{}' not found. Stopping.", task.getRegistryId());
        ctl.stop();
    }
}
Also used : DeprovisionRegistryTask(org.bf2.srs.fleetmanager.execution.impl.tasks.deprovision.DeprovisionRegistryTask) TenantNotFoundServiceException(org.bf2.srs.fleetmanager.spi.tenants.TenantNotFoundServiceException) SubscriptionNotFoundServiceException(org.bf2.srs.fleetmanager.spi.ams.SubscriptionNotFoundServiceException) RegistryDeploymentData(org.bf2.srs.fleetmanager.storage.sqlPanacheImpl.model.RegistryDeploymentData) TenantManagerConfig(org.bf2.srs.fleetmanager.spi.tenants.model.TenantManagerConfig) Transactional(javax.transaction.Transactional)

Example 3 with WorkerContext

use of org.bf2.srs.fleetmanager.execution.manager.WorkerContext in project srs-fleet-manager by bf2fc6cc711aee1a0c2a.

the class EvalInstanceExpirationRegistryWorker method execute.

@Transactional
@Override
public void execute(Task aTask, WorkerContext ctx) {
    EvalInstanceExpirationRegistryTask task = (EvalInstanceExpirationRegistryTask) aTask;
    // Nothing to do, the heavy lifting is done by the deprovisioning task.
    log.debug("Initiating deprovisioning of an eval instance, registry id = '{}'", task.getRegistryId());
    ctx.delay(() -> tasks.submit(DeprovisionRegistryTask.builder().registryId(task.getRegistryId()).build()));
}
Also used : EvalInstanceExpirationRegistryTask(org.bf2.srs.fleetmanager.execution.impl.tasks.deprovision.EvalInstanceExpirationRegistryTask) Transactional(javax.transaction.Transactional)

Example 4 with WorkerContext

use of org.bf2.srs.fleetmanager.execution.manager.WorkerContext in project srs-fleet-manager by bf2fc6cc711aee1a0c2a.

the class StartDeprovisionRegistryWorker method finallyExecute.

@Transactional
@Override
public void finallyExecute(Task aTask, WorkerContext ctl, Optional<Exception> error) throws RegistryNotFoundException, RegistryStorageConflictException {
    StartDeprovisionRegistryTask task = (StartDeprovisionRegistryTask) aTask;
    Optional<RegistryData> registryOptional = storage.getRegistryById(task.getRegistryId());
    if (registryOptional.isPresent()) {
        var registry = registryOptional.get();
        // SUCCESS STATE
        if (RegistryStatusValueDto.DEPROVISIONING_DELETING.value().equals(registry.getStatus()))
            return;
        // FAILURE
        // Nothing to do, user can retry
        log.warn("Failed to start deprovisioning of Registry '{}'. Check the status to see if the instance is stuck.", registry);
    } else {
        log.warn("Could not find Registry (ID = {}).", task.getRegistryId());
    }
}
Also used : StartDeprovisionRegistryTask(org.bf2.srs.fleetmanager.execution.impl.tasks.deprovision.StartDeprovisionRegistryTask) RegistryData(org.bf2.srs.fleetmanager.storage.sqlPanacheImpl.model.RegistryData) Transactional(javax.transaction.Transactional)

Example 5 with WorkerContext

use of org.bf2.srs.fleetmanager.execution.manager.WorkerContext in project srs-fleet-manager by bf2fc6cc711aee1a0c2a.

the class JobWrapper method execute.

@Override
@SneakyThrows
@ActivateRequestContext
public void execute(JobExecutionContext quartzJobContext) {
    Task task = loadTask(quartzJobContext);
    List<Worker> selectedWorkers = workers.stream().filter(w -> w.supports(task) && !workerExclusions.contains(w.getClass())).collect(toList());
    for (Worker worker : selectedWorkers) {
        WorkerContextImpl wCtx = loadWorkerContext(quartzJobContext, worker, task);
        Instant next = null;
        Exception lastException = null;
        try {
            log.debug("Task Manager (task = {}, worker = {}, workerContext = {}): Executing task.", task, worker, wCtx);
            worker.execute(task, wCtx);
            wCtx.getDelayedActions().forEach(Runnable::run);
            // OK vvv
            // Reset retry counter
            wCtx.setRetryAttempts(0);
            // Reset min retry counter
            wCtx.setMinRetries(task.getSchedule().getMinRetries());
            // Normal rescheduling
            next = nextExecution(task);
        } catch (Exception anEx) {
            // TODO Throwable?
            lastException = anEx;
            if (anEx instanceof RetryExecutionControlException) {
                log.debug("Task Manager (task = {}, worker = {}, workerContext = {}): Task requested a retry.", task, worker, wCtx, anEx);
                RetryExecutionControlException ex = (RetryExecutionControlException) anEx;
                if (ex.isForce() && wCtx.getMinRetries() < Integer.MAX_VALUE) {
                    // Make space for forced retry, no more than Integer.MAX_VALUE
                    wCtx.setMinRetries(wCtx.getMinRetries() + 1);
                    next = Instant.now().plus(Duration.ofSeconds(1));
                }
                if (ex.getMinRetries() > wCtx.getMinRetries()) {
                    wCtx.setMinRetries(ex.getMinRetries());
                }
                lastException = null;
            }
            if (wCtx.getRetryAttempts() < wCtx.getMinRetries() && (next == null)) {
                // Reschedule if the minRetries is not reached
                next = Instant.now().plus(backoff(wCtx.getRetryAttempts()));
            }
            if (anEx instanceof StopExecutionControlException) {
                log.debug("Task Manager (task = {}, worker = {}, workerContext = {}): Task requested a stop.", task, worker, wCtx, anEx);
                // Unschedule
                next = null;
                lastException = null;
            }
            if (lastException != null) {
                log.warn("Task Manager (task = {}, worker = {}, workerContext = {}, nextExecution = {}): Task threw an exception during execution: {}", task, worker, wCtx, next, anEx);
            }
            wCtx.setRetryAttempts(wCtx.getRetryAttempts() + 1);
        } finally {
            // Unlikely used
            wCtx.setDelayedActions(new ArrayList<>(0));
            saveWorkerContext(quartzJobContext, wCtx, worker);
            saveTask(quartzJobContext, task);
            // Scheduling
            if (next != null) {
                if (wCtx.getRetryAttempts() == wCtx.getMinRetries()) {
                    log.info("Task Manager (task = {}, worker = {}, workerContext = {}): Last rescheduling at {}.", task, worker, wCtx, next);
                } else {
                    log.debug("Task Manager (task = {}, worker = {}, workerContext = {}): Rescheduling task at {}.", task, worker, wCtx, next);
                }
                taskManager.rerigger(task, next);
            } else {
                try {
                    log.debug("Task Manager (task = {}, worker = {}, workerContext = {}): Executing finallyExecute. Last exception = {}", task, worker, wCtx, lastException);
                    worker.finallyExecute(task, wCtx, ofNullable(lastException));
                    wCtx.getDelayedActions().forEach(Runnable::run);
                } catch (Exception ex) {
                    log.warn("Task Manager (task = {}, worker = {}, workerContext = {}): Ignoring an exception thrown in finallyExecute: {}", task, worker, wCtx, ex);
                } finally {
                    log.debug("Task Manager (task = {}, worker = {}, workerContext = {}): Removing task.", task, worker, wCtx);
                    taskManager.remove(task);
                }
            }
        }
    }
}
Also used : Task(org.bf2.srs.fleetmanager.execution.manager.Task) Instant.now(java.time.Instant.now) SneakyThrows(lombok.SneakyThrows) OperationContext(org.bf2.srs.fleetmanager.operation.OperationContext) LoggerFactory(org.slf4j.LoggerFactory) Job(org.quartz.Job) Duration.ofSeconds(java.time.Duration.ofSeconds) ArrayList(java.util.ArrayList) Inject(javax.inject.Inject) WorkerContext(org.bf2.srs.fleetmanager.execution.manager.WorkerContext) DisallowConcurrentExecution(org.quartz.DisallowConcurrentExecution) Duration(java.time.Duration) Objects.requireNonNull(java.util.Objects.requireNonNull) PersistJobDataAfterExecution(org.quartz.PersistJobDataAfterExecution) SerDesObjectMapperProducer(org.bf2.srs.fleetmanager.common.SerDesObjectMapperProducer) Instance(javax.enterprise.inject.Instance) JobExecutionContext(org.quartz.JobExecutionContext) Logger(org.slf4j.Logger) Optional.ofNullable(java.util.Optional.ofNullable) ObjectMapper(com.fasterxml.jackson.databind.ObjectMapper) ConcurrentHashMap(java.util.concurrent.ConcurrentHashMap) Set(java.util.Set) QuartzIDs.jobDetailKeyForTask(org.bf2.srs.fleetmanager.execution.manager.impl.QuartzIDs.jobDetailKeyForTask) Instant(java.time.Instant) QuartzIDs.jobDetailKeyForWorker(org.bf2.srs.fleetmanager.execution.manager.impl.QuartzIDs.jobDetailKeyForWorker) ActivateRequestContext(javax.enterprise.context.control.ActivateRequestContext) List(java.util.List) Collectors.toList(java.util.stream.Collectors.toList) Worker(org.bf2.srs.fleetmanager.execution.manager.Worker) ApplicationScoped(javax.enterprise.context.ApplicationScoped) ConfigProperty(org.eclipse.microprofile.config.inject.ConfigProperty) Task(org.bf2.srs.fleetmanager.execution.manager.Task) QuartzIDs.jobDetailKeyForTask(org.bf2.srs.fleetmanager.execution.manager.impl.QuartzIDs.jobDetailKeyForTask) Instant(java.time.Instant) QuartzIDs.jobDetailKeyForWorker(org.bf2.srs.fleetmanager.execution.manager.impl.QuartzIDs.jobDetailKeyForWorker) Worker(org.bf2.srs.fleetmanager.execution.manager.Worker) ActivateRequestContext(javax.enterprise.context.control.ActivateRequestContext) SneakyThrows(lombok.SneakyThrows)

Aggregations

Transactional (javax.transaction.Transactional)8 RegistryData (org.bf2.srs.fleetmanager.storage.sqlPanacheImpl.model.RegistryData)7 RegistryDeploymentData (org.bf2.srs.fleetmanager.storage.sqlPanacheImpl.model.RegistryDeploymentData)4 ProvisionRegistryTenantTask (org.bf2.srs.fleetmanager.execution.impl.tasks.ProvisionRegistryTenantTask)3 List (java.util.List)2 Collectors.toList (java.util.stream.Collectors.toList)2 ApplicationScoped (javax.enterprise.context.ApplicationScoped)2 Inject (javax.inject.Inject)2 ScheduleRegistryTask (org.bf2.srs.fleetmanager.execution.impl.tasks.ScheduleRegistryTask)2 TestTask (org.bf2.srs.fleetmanager.execution.impl.tasks.TestTask)2 Command (org.bf2.srs.fleetmanager.execution.impl.tasks.TestTask.Command)2 DeprovisionRegistryTask (org.bf2.srs.fleetmanager.execution.impl.tasks.deprovision.DeprovisionRegistryTask)2 StartDeprovisionRegistryTask (org.bf2.srs.fleetmanager.execution.impl.tasks.deprovision.StartDeprovisionRegistryTask)2 Task (org.bf2.srs.fleetmanager.execution.manager.Task)2 WorkerContext (org.bf2.srs.fleetmanager.execution.manager.WorkerContext)2 TenantNotFoundServiceException (org.bf2.srs.fleetmanager.spi.tenants.TenantNotFoundServiceException)2 TenantManagerConfig (org.bf2.srs.fleetmanager.spi.tenants.model.TenantManagerConfig)2 Logger (org.slf4j.Logger)2 LoggerFactory (org.slf4j.LoggerFactory)2 ObjectMapper (com.fasterxml.jackson.databind.ObjectMapper)1