Search in sources :

Example 1 with Job

use of net.greghaines.jesque.Job in project box-c by UNC-Libraries.

the class DepositSupervisorTest method setup.

@Before
public void setup() throws Exception {
    when(jobFactory.materializeJob(any())).thenAnswer(new Answer<Object>() {

        @Override
        public Object answer(InvocationOnMock invocation) throws Throwable {
            Job job = invocation.getArgumentAt(0, Job.class);
            if (job == null) {
                return null;
            }
            String uuid = (String) job.getArgs()[0];
            String depositUUID = (String) job.getArgs()[1];
            return new TestDepositJob(uuid, depositUUID);
        }
    });
    depositDestination = pidMinter.mintContentPid();
    agent = new AgentPrincipalsImpl("user", new AccessGroupSetImpl());
    pipelineStatusFactory.setPipelineState(DepositPipelineState.active);
    supervisor = new DepositSupervisor();
    supervisor.setJesqueConfig(jesqueConfig);
    supervisor.setDepositStatusFactory(depositStatusFactory);
    supervisor.setJobStatusFactory(jobStatusFactory);
    setField(supervisor, "pipelineStatusFactory", pipelineStatusFactory);
    setField(supervisor, "depositWorkerPools", depositWorkerPools);
    setField(supervisor, "queueDAO", queueDAO);
    supervisor.setUnavailableDelaySeconds(60);
    supervisor.setCleanupDelaySeconds(60);
    // Turn up monitoring speed so tests are shorter
    supervisor.setActionMonitorDelay(25l);
    actionMonitor = supervisor.actionMonitoringTask;
}
Also used : AccessGroupSetImpl(edu.unc.lib.boxc.auth.fcrepo.models.AccessGroupSetImpl) InvocationOnMock(org.mockito.invocation.InvocationOnMock) PackageIntegrityCheckJob(edu.unc.lib.boxc.deposit.validate.PackageIntegrityCheckJob) Job(net.greghaines.jesque.Job) BagIt2N3BagJob(edu.unc.lib.boxc.deposit.normalize.BagIt2N3BagJob) AgentPrincipalsImpl(edu.unc.lib.boxc.auth.fcrepo.models.AgentPrincipalsImpl) Before(org.junit.Before)

Example 2 with Job

use of net.greghaines.jesque.Job in project box-c by UNC-Libraries.

the class DepositSupervisor method queueNewDeposit.

private void queueNewDeposit(String uuid, Map<String, String> fields) {
    LOG.info("Queuing first job for deposit {}", uuid);
    Job job = makeJob(PackageIntegrityCheckJob.class, uuid);
    depositStatusFactory.setState(uuid, DepositState.queued);
    depositStatusFactory.clearActionRequest(uuid);
    enqueueJob(job, fields, 0);
}
Also used : UnpackDepositJob(edu.unc.lib.boxc.deposit.normalize.UnpackDepositJob) ValidateFileAvailabilityJob(edu.unc.lib.boxc.deposit.validate.ValidateFileAvailabilityJob) ValidateContentModelJob(edu.unc.lib.boxc.deposit.validate.ValidateContentModelJob) ValidateDescriptionJob(edu.unc.lib.boxc.deposit.validate.ValidateDescriptionJob) PackageIntegrityCheckJob(edu.unc.lib.boxc.deposit.validate.PackageIntegrityCheckJob) CleanupDepositJob(edu.unc.lib.boxc.deposit.CleanupDepositJob) Job(net.greghaines.jesque.Job) FixityCheckJob(edu.unc.lib.boxc.deposit.validate.FixityCheckJob) StaffOnlyPermissionJob(edu.unc.lib.boxc.deposit.fcrepo4.StaffOnlyPermissionJob) Simple2N3BagJob(edu.unc.lib.boxc.deposit.normalize.Simple2N3BagJob) ValidateDestinationJob(edu.unc.lib.boxc.deposit.validate.ValidateDestinationJob) VocabularyEnforcementJob(edu.unc.lib.boxc.deposit.normalize.VocabularyEnforcementJob) TransferBinariesToStorageJob(edu.unc.lib.boxc.deposit.transfer.TransferBinariesToStorageJob) IngestContentObjectsJob(edu.unc.lib.boxc.deposit.fcrepo4.IngestContentObjectsJob) VirusScanJob(edu.unc.lib.boxc.deposit.validate.VirusScanJob) NormalizeFileObjectsJob(edu.unc.lib.boxc.deposit.normalize.NormalizeFileObjectsJob) BagIt2N3BagJob(edu.unc.lib.boxc.deposit.normalize.BagIt2N3BagJob) CDRMETS2N3BagJob(edu.unc.lib.boxc.deposit.normalize.CDRMETS2N3BagJob) IngestDepositRecordJob(edu.unc.lib.boxc.deposit.fcrepo4.IngestDepositRecordJob) PreconstructedDepositJob(edu.unc.lib.boxc.deposit.normalize.PreconstructedDepositJob) DirectoryToBagJob(edu.unc.lib.boxc.deposit.normalize.DirectoryToBagJob) AssignStorageLocationsJob(edu.unc.lib.boxc.deposit.normalize.AssignStorageLocationsJob) ExtractTechnicalMetadataJob(edu.unc.lib.boxc.deposit.validate.ExtractTechnicalMetadataJob)

Example 3 with Job

use of net.greghaines.jesque.Job in project box-c by UNC-Libraries.

the class DepositSupervisor method queueNextJob.

private void queueNextJob(Job job, String depositUUID, Map<String, String> status, List<String> successfulJobs, long delay) throws DepositFailedException {
    Job nextJob = getNextJob(depositUUID, status, successfulJobs);
    if (nextJob != null) {
        LOG.info("Queuing next job {} for deposit {}", nextJob.getClassName(), depositUUID);
        enqueueJob(nextJob, status, delay);
    } else {
        depositStatusFactory.setState(depositUUID, DepositState.finished);
        final Counter finished = CounterFactory.createCounter(job.getClass(), "finished-deposits");
        finished.inc();
        depositDuration(depositUUID, status);
        depositEmailHandler.sendDepositResults(depositUUID);
        // Send message indicating the deposit has completed
        sendDepositCompleteEvent(depositUUID);
        // schedule cleanup job after the configured delay
        Job cleanJob = makeJob(CleanupDepositJob.class, depositUUID);
        LOG.info("Queuing {} for deposit {}", cleanJob.getClassName(), depositUUID);
        enqueueJob(cleanJob, status, 1000 * this.getCleanupDelaySeconds());
    }
}
Also used : Counter(io.dropwizard.metrics5.Counter) UnpackDepositJob(edu.unc.lib.boxc.deposit.normalize.UnpackDepositJob) ValidateFileAvailabilityJob(edu.unc.lib.boxc.deposit.validate.ValidateFileAvailabilityJob) ValidateContentModelJob(edu.unc.lib.boxc.deposit.validate.ValidateContentModelJob) ValidateDescriptionJob(edu.unc.lib.boxc.deposit.validate.ValidateDescriptionJob) PackageIntegrityCheckJob(edu.unc.lib.boxc.deposit.validate.PackageIntegrityCheckJob) CleanupDepositJob(edu.unc.lib.boxc.deposit.CleanupDepositJob) Job(net.greghaines.jesque.Job) FixityCheckJob(edu.unc.lib.boxc.deposit.validate.FixityCheckJob) StaffOnlyPermissionJob(edu.unc.lib.boxc.deposit.fcrepo4.StaffOnlyPermissionJob) Simple2N3BagJob(edu.unc.lib.boxc.deposit.normalize.Simple2N3BagJob) ValidateDestinationJob(edu.unc.lib.boxc.deposit.validate.ValidateDestinationJob) VocabularyEnforcementJob(edu.unc.lib.boxc.deposit.normalize.VocabularyEnforcementJob) TransferBinariesToStorageJob(edu.unc.lib.boxc.deposit.transfer.TransferBinariesToStorageJob) IngestContentObjectsJob(edu.unc.lib.boxc.deposit.fcrepo4.IngestContentObjectsJob) VirusScanJob(edu.unc.lib.boxc.deposit.validate.VirusScanJob) NormalizeFileObjectsJob(edu.unc.lib.boxc.deposit.normalize.NormalizeFileObjectsJob) BagIt2N3BagJob(edu.unc.lib.boxc.deposit.normalize.BagIt2N3BagJob) CDRMETS2N3BagJob(edu.unc.lib.boxc.deposit.normalize.CDRMETS2N3BagJob) IngestDepositRecordJob(edu.unc.lib.boxc.deposit.fcrepo4.IngestDepositRecordJob) PreconstructedDepositJob(edu.unc.lib.boxc.deposit.normalize.PreconstructedDepositJob) DirectoryToBagJob(edu.unc.lib.boxc.deposit.normalize.DirectoryToBagJob) AssignStorageLocationsJob(edu.unc.lib.boxc.deposit.normalize.AssignStorageLocationsJob) ExtractTechnicalMetadataJob(edu.unc.lib.boxc.deposit.validate.ExtractTechnicalMetadataJob)

Example 4 with Job

use of net.greghaines.jesque.Job in project box-c by UNC-Libraries.

the class DepositSupervisor method addQueuedDeposits.

private void addQueuedDeposits(String queueName, Map<String, Set<String>> depositMap) {
    QueueInfo info = queueDAO.getQueueInfo(queueName, 0, 0);
    for (Job job : info.getJobs()) {
        String depositId = (String) job.getArgs()[1];
        Set<String> jobs = depositMap.get(depositId);
        if (jobs == null) {
            jobs = new HashSet<>();
            depositMap.put(depositId, jobs);
        }
        jobs.add(job.getClassName());
    }
}
Also used : QueueInfo(net.greghaines.jesque.meta.QueueInfo) UnpackDepositJob(edu.unc.lib.boxc.deposit.normalize.UnpackDepositJob) ValidateFileAvailabilityJob(edu.unc.lib.boxc.deposit.validate.ValidateFileAvailabilityJob) ValidateContentModelJob(edu.unc.lib.boxc.deposit.validate.ValidateContentModelJob) ValidateDescriptionJob(edu.unc.lib.boxc.deposit.validate.ValidateDescriptionJob) PackageIntegrityCheckJob(edu.unc.lib.boxc.deposit.validate.PackageIntegrityCheckJob) CleanupDepositJob(edu.unc.lib.boxc.deposit.CleanupDepositJob) Job(net.greghaines.jesque.Job) FixityCheckJob(edu.unc.lib.boxc.deposit.validate.FixityCheckJob) StaffOnlyPermissionJob(edu.unc.lib.boxc.deposit.fcrepo4.StaffOnlyPermissionJob) Simple2N3BagJob(edu.unc.lib.boxc.deposit.normalize.Simple2N3BagJob) ValidateDestinationJob(edu.unc.lib.boxc.deposit.validate.ValidateDestinationJob) VocabularyEnforcementJob(edu.unc.lib.boxc.deposit.normalize.VocabularyEnforcementJob) TransferBinariesToStorageJob(edu.unc.lib.boxc.deposit.transfer.TransferBinariesToStorageJob) IngestContentObjectsJob(edu.unc.lib.boxc.deposit.fcrepo4.IngestContentObjectsJob) VirusScanJob(edu.unc.lib.boxc.deposit.validate.VirusScanJob) NormalizeFileObjectsJob(edu.unc.lib.boxc.deposit.normalize.NormalizeFileObjectsJob) BagIt2N3BagJob(edu.unc.lib.boxc.deposit.normalize.BagIt2N3BagJob) CDRMETS2N3BagJob(edu.unc.lib.boxc.deposit.normalize.CDRMETS2N3BagJob) IngestDepositRecordJob(edu.unc.lib.boxc.deposit.fcrepo4.IngestDepositRecordJob) PreconstructedDepositJob(edu.unc.lib.boxc.deposit.normalize.PreconstructedDepositJob) DirectoryToBagJob(edu.unc.lib.boxc.deposit.normalize.DirectoryToBagJob) AssignStorageLocationsJob(edu.unc.lib.boxc.deposit.normalize.AssignStorageLocationsJob) ExtractTechnicalMetadataJob(edu.unc.lib.boxc.deposit.validate.ExtractTechnicalMetadataJob)

Example 5 with Job

use of net.greghaines.jesque.Job in project castled by castledio.

the class JesqueTasksClient method batchEnqueueTasks.

public void batchEnqueueTasks(List<JesqueTaskParams> jesqueTaskParamsList) {
    Map<TaskPriority, List<JesqueTaskParams>> paramsPerPriority = jesqueTaskParamsList.stream().collect(Collectors.groupingBy(JesqueTaskParams::getTaskPriority));
    for (TaskPriority priority : paramsPerPriority.keySet()) {
        List<Job> jobs = paramsPerPriority.get(priority).stream().map(this::buildJesqueJob).collect(Collectors.toList());
        this.client.delayedBatchEnqueue(priority.name(), jobs, System.currentTimeMillis() + PUBLISH_DELAY_MS);
    }
}
Also used : TaskPriority(io.castled.jarvis.taskmanager.models.TaskPriority) List(java.util.List) Job(net.greghaines.jesque.Job)

Aggregations

Job (net.greghaines.jesque.Job)6 BagIt2N3BagJob (edu.unc.lib.boxc.deposit.normalize.BagIt2N3BagJob)5 PackageIntegrityCheckJob (edu.unc.lib.boxc.deposit.validate.PackageIntegrityCheckJob)5 CleanupDepositJob (edu.unc.lib.boxc.deposit.CleanupDepositJob)3 IngestContentObjectsJob (edu.unc.lib.boxc.deposit.fcrepo4.IngestContentObjectsJob)3 IngestDepositRecordJob (edu.unc.lib.boxc.deposit.fcrepo4.IngestDepositRecordJob)3 StaffOnlyPermissionJob (edu.unc.lib.boxc.deposit.fcrepo4.StaffOnlyPermissionJob)3 AssignStorageLocationsJob (edu.unc.lib.boxc.deposit.normalize.AssignStorageLocationsJob)3 CDRMETS2N3BagJob (edu.unc.lib.boxc.deposit.normalize.CDRMETS2N3BagJob)3 DirectoryToBagJob (edu.unc.lib.boxc.deposit.normalize.DirectoryToBagJob)3 NormalizeFileObjectsJob (edu.unc.lib.boxc.deposit.normalize.NormalizeFileObjectsJob)3 PreconstructedDepositJob (edu.unc.lib.boxc.deposit.normalize.PreconstructedDepositJob)3 Simple2N3BagJob (edu.unc.lib.boxc.deposit.normalize.Simple2N3BagJob)3 UnpackDepositJob (edu.unc.lib.boxc.deposit.normalize.UnpackDepositJob)3 VocabularyEnforcementJob (edu.unc.lib.boxc.deposit.normalize.VocabularyEnforcementJob)3 TransferBinariesToStorageJob (edu.unc.lib.boxc.deposit.transfer.TransferBinariesToStorageJob)3 ExtractTechnicalMetadataJob (edu.unc.lib.boxc.deposit.validate.ExtractTechnicalMetadataJob)3 FixityCheckJob (edu.unc.lib.boxc.deposit.validate.FixityCheckJob)3 ValidateContentModelJob (edu.unc.lib.boxc.deposit.validate.ValidateContentModelJob)3 ValidateDescriptionJob (edu.unc.lib.boxc.deposit.validate.ValidateDescriptionJob)3