Examples with JobletRequest io.datarouter.joblet.storage.jobletrequest.JobletRequest used on opensource projects

Example 1 with JobletRequest

use of io.datarouter.joblet.storage.jobletrequest.JobletRequest in project datarouter by hotpads.

the class TableSamplerTestTool method executeJobletsAndCollectSamples.

public static List<TableSample> executeJobletsAndCollectSamples(DatarouterInjector injector, List<JobletPackage> jobletPackages) {
    List<TableSample> results = new ArrayList<>();
    for (JobletData jobletData : JobletPackage.getJobletDatas(jobletPackages)) {
        TableSpanSamplerJoblet joblet = injector.getInstance(TableSpanSamplerJoblet.class);
        JobletRequest jobletRequest = new JobletRequest();
        // this field referenced by the joblet
        jobletRequest.getKey().setCreated(System.currentTimeMillis());
        joblet.setJobletRequest(jobletRequest);
        TableSpanSamplerJobletParams params = new TableSpanSamplerJobletCodec().unmarshallData(jobletData.getData());
        joblet.setJobletParams(params);
        joblet.process();
        results.addAll(joblet.getSamples());
    }
    return results;
}

Example 2 with JobletRequest

use of io.datarouter.joblet.storage.jobletrequest.JobletRequest in project datarouter by hotpads.

the class MysqlLockForUpdateJobletRequestSelector method getJobletRequestForProcessing.

@Override
public Optional<JobletRequest> getJobletRequestForProcessing(PhaseTimer timer, JobletType<?> type, String reservedBy) {
    while (true) {
        var mysqlOp = new GetJobletRequest(reservedBy, type, datarouter, jobletRequestDao, mysqlFieldCodecFactory, mysqlSqlFactory, jobletRequestSqlBuilder, jobletService);
        JobletRequest jobletRequest = sessionExecutor.runWithoutRetries(mysqlOp);
        timer.add("GetJobletRequest");
        if (jobletRequest == null) {
            // for back-off
            jobletRequestQueueManager.onJobletRequestMissForAllPriorities(type);
            return Optional.empty();
        }
        if (!jobletRequest.getStatus().isRunning()) {
            // weird flow. it was probably just marked as timedOut, so skip it
            continue;
        }
        var queueKey = new JobletRequestQueueKey(type, jobletRequest.getKey().getPriority());
        datarouterJobletCounters.incQueueHit(queueKey.getQueueName());
        return Optional.of(jobletRequest);
    }
}

Example 3 with JobletRequest

use of io.datarouter.joblet.storage.jobletrequest.JobletRequest in project datarouter by hotpads.

the class MysqlUpdateAndScanJobletRequestSelector method getJobletRequestForProcessing.

@Override
public Optional<JobletRequest> getJobletRequestForProcessing(PhaseTimer timer, JobletType<?> type, String reservedBy) {
    ReserveJobletRequest mysqlOp = new ReserveJobletRequest(reservedBy, type, datarouter, jobletRequestDao, mysqlSqlFactory, jobletRequestSqlBuilder);
    while (sessionExecutor.runWithoutRetries(mysqlOp)) {
        // returns false if no joblet found
        JobletRequest jobletRequest = jobletRequestDao.getReservedRequest(type, reservedBy, Isolation.readUncommitted);
        if (JobletStatus.CREATED == jobletRequest.getStatus()) {
            jobletRequest.setReservedBy(reservedBy);
            jobletRequest.setReservedAt(System.currentTimeMillis());
            jobletRequest.setStatus(JobletStatus.RUNNING);
            jobletRequestDao.put(jobletRequest);
            return Optional.of(jobletRequest);
        }
        // we got a previously timed-out joblet
        jobletRequest.incrementNumTimeouts();
        if (jobletRequest.getNumTimeouts() <= JobletService.MAX_JOBLET_RETRIES) {
            jobletRequestDao.put(jobletRequest);
            JobletRequestQueueKey queueKey = new JobletRequestQueueKey(type, jobletRequest.getKey().getPriority());
            datarouterJobletCounters.incQueueHit(queueKey.getQueueName());
            return Optional.of(jobletRequest);
        }
        jobletRequest.setStatus(JobletStatus.TIMED_OUT);
        jobletRequestDao.put(jobletRequest);
    // loop around for another
    }
    // for back-off
    jobletRequestQueueManager.onJobletRequestMissForAllPriorities(type);
    return Optional.empty();
}

Example 4 with JobletRequest

use of io.datarouter.joblet.storage.jobletrequest.JobletRequest in project datarouter by hotpads.

the class GetJobletRequest method runOnce.

@Override
public JobletRequest runOnce() {
    PreparedStatement selectStatement = makeSelectStatement().prepare(getConnection());
    JobletRequest jobletRequest = MysqlTool.selectDatabeans(mysqlFieldCodecFactory, fieldInfo.getDatabeanSupplier(), fieldInfo.getFields(), selectStatement).stream().findFirst().orElse(null);
    if (jobletRequest == null) {
        return null;
    }
    jobletService.updateStatusToRunning(jobletRequest, reservedBy);
    return jobletRequest;
}

Example 5 with JobletRequest

use of io.datarouter.joblet.storage.jobletrequest.JobletRequest in project datarouter by hotpads.

the class JobletExceptionHandler method makeContent.

private ContainerTag<?> makeContent(List<JobletRequest> rows) {
    var title = h4(TITLE).withClass("mt-2");
    var table = new J2HtmlTable<JobletRequest>().withClasses("sortable table table-sm table-striped border").withHtmlColumn("Exception ID", row -> {
        String id = row.getExceptionRecordId();
        return externalLinkBuilder.get().exception(request.getContextPath(), id).map(href -> td(a(id).withHref(href))).orElse(td(id));
    }).withColumn("Type", row -> row.getKey().getType()).withColumn("Execution order", row -> row.getKey().getExecutionOrder()).withColumn("Batch sequence", row -> row.getKey().getBatchSequence()).withColumn("Data ID", JobletRequest::getJobletDataId).withColumn("Reserved by", JobletRequest::getReservedBy).withColumn("Created ago", JobletRequest::getCreatedAgo).withColumn("Restartable", JobletRequest::getRestartable).withColumn("Num items", JobletRequest::getNumItems).withColumn("Queue ID", JobletRequest::getQueueId).build(rows);
    return div(title, table).withClass("container-fluid");
}