Search in sources :

Example 6 with TaskSummary

use of com.netflix.conductor.common.run.TaskSummary in project conductor by Netflix.

the class ElasticSearchDAOV5 method indexTask.

@Override
public void indexTask(Task task) {
    try {
        long startTime = Instant.now().toEpochMilli();
        String id = task.getTaskId();
        TaskSummary summary = new TaskSummary(task);
        byte[] doc = objectMapper.writeValueAsBytes(summary);
        UpdateRequest req = new UpdateRequest(indexName, TASK_DOC_TYPE, id);
        req.doc(doc, XContentType.JSON);
        req.upsert(doc, XContentType.JSON);
        logger.debug("Indexing task document: {} for workflow: {}" + id, task.getWorkflowInstanceId());
        indexObject(req, TASK_DOC_TYPE);
        long endTime = Instant.now().toEpochMilli();
        logger.debug("Time taken {} for  indexing task:{} in workflow: {}", endTime - startTime, task.getTaskId(), task.getWorkflowInstanceId());
        Monitors.recordESIndexTime("index_task", TASK_DOC_TYPE, endTime - startTime);
        Monitors.recordWorkerQueueSize("indexQueue", ((ThreadPoolExecutor) executorService).getQueue().size());
    } catch (Exception e) {
        logger.error("Failed to index task: {}", task.getTaskId(), e);
    }
}
Also used : UpdateRequest(org.elasticsearch.action.update.UpdateRequest) TaskSummary(com.netflix.conductor.common.run.TaskSummary) ThreadPoolExecutor(java.util.concurrent.ThreadPoolExecutor) IndexNotFoundException(org.elasticsearch.index.IndexNotFoundException) ParserException(com.netflix.conductor.elasticsearch.query.parser.ParserException) ResourceAlreadyExistsException(org.elasticsearch.ResourceAlreadyExistsException) ApplicationException(com.netflix.conductor.core.execution.ApplicationException) IOException(java.io.IOException)

Example 7 with TaskSummary

use of com.netflix.conductor.common.run.TaskSummary in project conductor by Netflix.

the class ExecutionService method searchWorkflowByTasks.

public SearchResult<WorkflowSummary> searchWorkflowByTasks(String query, String freeText, int start, int size, List<String> sortOptions) {
    SearchResult<TaskSummary> taskSummarySearchResult = searchTasks(query, freeText, start, size, sortOptions);
    List<WorkflowSummary> workflowSummaries = taskSummarySearchResult.getResults().stream().parallel().map(taskSummary -> {
        try {
            String workflowId = taskSummary.getWorkflowId();
            return new WorkflowSummary(executionDAOFacade.getWorkflowById(workflowId, false));
        } catch (Exception e) {
            logger.error("Error fetching workflow by id: {}", taskSummary.getWorkflowId(), e);
            return null;
        }
    }).filter(Objects::nonNull).distinct().collect(Collectors.toList());
    int missing = taskSummarySearchResult.getResults().size() - workflowSummaries.size();
    long totalHits = taskSummarySearchResult.getTotalHits() - missing;
    return new SearchResult<>(totalHits, workflowSummaries);
}
Also used : QueueUtils(com.netflix.conductor.core.utils.QueueUtils) Status(com.netflix.conductor.common.metadata.tasks.Task.Status) LoggerFactory(org.slf4j.LoggerFactory) HashMap(java.util.HashMap) MetadataDAO(com.netflix.conductor.dao.MetadataDAO) Task(com.netflix.conductor.common.metadata.tasks.Task) Singleton(javax.inject.Singleton) ExecutionDAOFacade(com.netflix.conductor.core.orchestration.ExecutionDAOFacade) TaskResult(com.netflix.conductor.common.metadata.tasks.TaskResult) ServiceUtils(com.netflix.conductor.service.utils.ServiceUtils) ArrayList(java.util.ArrayList) WorkflowSummary(com.netflix.conductor.common.run.WorkflowSummary) Inject(javax.inject.Inject) Workflow(com.netflix.conductor.common.run.Workflow) Map(java.util.Map) PollData(com.netflix.conductor.common.metadata.tasks.PollData) ExternalPayloadStorage(com.netflix.conductor.common.utils.ExternalPayloadStorage) Max(javax.validation.constraints.Max) Operation(com.netflix.conductor.common.utils.ExternalPayloadStorage.Operation) WorkflowExecutor(com.netflix.conductor.core.execution.WorkflowExecutor) EventExecution(com.netflix.conductor.common.metadata.events.EventExecution) LinkedList(java.util.LinkedList) TaskDef(com.netflix.conductor.common.metadata.tasks.TaskDef) ApplicationException(com.netflix.conductor.core.execution.ApplicationException) Message(com.netflix.conductor.core.events.queue.Message) SearchResult(com.netflix.conductor.common.run.SearchResult) Logger(org.slf4j.Logger) Trace(com.netflix.conductor.annotations.Trace) TaskExecLog(com.netflix.conductor.common.metadata.tasks.TaskExecLog) TaskSummary(com.netflix.conductor.common.run.TaskSummary) WorkflowDef(com.netflix.conductor.common.metadata.workflow.WorkflowDef) Collectors(java.util.stream.Collectors) Monitors(com.netflix.conductor.metrics.Monitors) ExternalStorageLocation(com.netflix.conductor.common.run.ExternalStorageLocation) Objects(java.util.Objects) QueueDAO(com.netflix.conductor.dao.QueueDAO) List(java.util.List) SystemTaskType(com.netflix.conductor.core.execution.SystemTaskType) Optional(java.util.Optional) Configuration(com.netflix.conductor.core.config.Configuration) Comparator(java.util.Comparator) PayloadType(com.netflix.conductor.common.utils.ExternalPayloadStorage.PayloadType) Collections(java.util.Collections) WorkflowSummary(com.netflix.conductor.common.run.WorkflowSummary) TaskSummary(com.netflix.conductor.common.run.TaskSummary) Objects(java.util.Objects) SearchResult(com.netflix.conductor.common.run.SearchResult) ApplicationException(com.netflix.conductor.core.execution.ApplicationException)

Example 8 with TaskSummary

use of com.netflix.conductor.common.run.TaskSummary in project conductor by Netflix.

the class ElasticSearchRestDAOV5 method indexTask.

@Override
public void indexTask(Task task) {
    try {
        long startTime = Instant.now().toEpochMilli();
        String taskId = task.getTaskId();
        TaskSummary summary = new TaskSummary(task);
        indexObject(indexName, TASK_DOC_TYPE, taskId, summary);
        long endTime = Instant.now().toEpochMilli();
        logger.debug("Time taken {} for  indexing task:{} in workflow: {}", endTime - startTime, taskId, task.getWorkflowInstanceId());
        Monitors.recordESIndexTime("index_task", TASK_DOC_TYPE, endTime - startTime);
        Monitors.recordWorkerQueueSize("indexQueue", ((ThreadPoolExecutor) executorService).getQueue().size());
    } catch (Exception e) {
        logger.error("Failed to index task: {}", task.getTaskId(), e);
    }
}
Also used : TaskSummary(com.netflix.conductor.common.run.TaskSummary) ThreadPoolExecutor(java.util.concurrent.ThreadPoolExecutor) ResponseException(org.elasticsearch.client.ResponseException) ApplicationException(com.netflix.conductor.core.execution.ApplicationException) JsonProcessingException(com.fasterxml.jackson.core.JsonProcessingException) IOException(java.io.IOException)

Example 9 with TaskSummary

use of com.netflix.conductor.common.run.TaskSummary in project conductor by Netflix.

the class ElasticSearchRestDAOV6 method indexTask.

@Override
public void indexTask(Task task) {
    try {
        long startTime = Instant.now().toEpochMilli();
        String taskId = task.getTaskId();
        TaskSummary summary = new TaskSummary(task);
        String docType = StringUtils.isBlank(docTypeOverride) ? TASK_DOC_TYPE : docTypeOverride;
        indexObject(taskIndexName, docType, taskId, summary);
        long endTime = Instant.now().toEpochMilli();
        logger.debug("Time taken {} for  indexing task:{} in workflow: {}", endTime - startTime, taskId, task.getWorkflowInstanceId());
        Monitors.recordESIndexTime("index_task", TASK_DOC_TYPE, endTime - startTime);
        Monitors.recordWorkerQueueSize("indexQueue", ((ThreadPoolExecutor) executorService).getQueue().size());
    } catch (Exception e) {
        logger.error("Failed to index task: {}", task.getTaskId(), e);
    }
}
Also used : TaskSummary(com.netflix.conductor.common.run.TaskSummary) ThreadPoolExecutor(java.util.concurrent.ThreadPoolExecutor) ResponseException(org.elasticsearch.client.ResponseException) ParserException(com.netflix.conductor.elasticsearch.query.parser.ParserException) ApplicationException(com.netflix.conductor.core.execution.ApplicationException) JsonProcessingException(com.fasterxml.jackson.core.JsonProcessingException) IOException(java.io.IOException)

Example 10 with TaskSummary

use of com.netflix.conductor.common.run.TaskSummary in project conductor by Netflix.

the class TestElasticSearchDAOV6 method shouldIndexTaskAsync.

@Test
public void shouldIndexTaskAsync() throws Exception {
    Workflow workflow = TestUtils.loadWorkflowSnapshot("workflow");
    Task task = workflow.getTasks().get(0);
    TaskSummary summary = new TaskSummary(task);
    indexDAO.asyncIndexTask(task).get();
    List<String> tasks = tryFindResults(() -> searchTasks(workflow));
    assertEquals(summary.getTaskId(), tasks.get(0));
}
Also used : Task(com.netflix.conductor.common.metadata.tasks.Task) TaskSummary(com.netflix.conductor.common.run.TaskSummary) Workflow(com.netflix.conductor.common.run.Workflow) Test(org.junit.Test)

Aggregations

TaskSummary (com.netflix.conductor.common.run.TaskSummary)13 Task (com.netflix.conductor.common.metadata.tasks.Task)8 Workflow (com.netflix.conductor.common.run.Workflow)7 Test (org.junit.Test)7 ApplicationException (com.netflix.conductor.core.execution.ApplicationException)6 IOException (java.io.IOException)5 ThreadPoolExecutor (java.util.concurrent.ThreadPoolExecutor)5 ParserException (com.netflix.conductor.elasticsearch.query.parser.ParserException)4 JsonProcessingException (com.fasterxml.jackson.core.JsonProcessingException)3 SearchResult (com.netflix.conductor.common.run.SearchResult)2 ArrayList (java.util.ArrayList)2 ResourceAlreadyExistsException (org.elasticsearch.ResourceAlreadyExistsException)2 UpdateRequest (org.elasticsearch.action.update.UpdateRequest)2 ResponseException (org.elasticsearch.client.ResponseException)2 IndexNotFoundException (org.elasticsearch.index.IndexNotFoundException)2 Trace (com.netflix.conductor.annotations.Trace)1 EventExecution (com.netflix.conductor.common.metadata.events.EventExecution)1 PollData (com.netflix.conductor.common.metadata.tasks.PollData)1 Status (com.netflix.conductor.common.metadata.tasks.Task.Status)1 TaskDef (com.netflix.conductor.common.metadata.tasks.TaskDef)1