Search in sources :

Example 1 with TaskIdentifier

use of org.flyte.api.v1.TaskIdentifier in project flytekit-java by flyteorg.

the class IdentifierRewrite method getLatestTaskVersion.

private String getLatestTaskVersion(String project, String domain, String name) {
    TaskIdentifier latestTaskId = adminClient().fetchLatestTaskId(NamedEntityIdentifier.builder().domain(domain).project(project).name(name).build());
    Verify.verifyNotNull(latestTaskId, "task not found domain=[%s], project=[%s], name=[%s]", domain, project, name);
    return latestTaskId.version();
}
Also used : TaskIdentifier(org.flyte.api.v1.TaskIdentifier) PartialTaskIdentifier(org.flyte.api.v1.PartialTaskIdentifier)

Example 2 with TaskIdentifier

use of org.flyte.api.v1.TaskIdentifier in project flytekit-java by flyteorg.

the class FlyteAdminClientTest method fetchLatestTaskIdShouldReturnFirstTaskFromList.

@Test
public void fetchLatestTaskIdShouldReturnFirstTaskFromList() {
    stubService.taskLists = Arrays.asList(TaskOuterClass.Task.newBuilder().setId(newIdentifier(ResourceType.TASK, TASK_NAME, TASK_VERSION)).build(), TaskOuterClass.Task.newBuilder().setId(newIdentifier(ResourceType.TASK, TASK_NAME, TASK_OLD_VERSION)).build());
    TaskIdentifier taskId = client.fetchLatestTaskId(NamedEntityIdentifier.builder().project(PROJECT).domain(DOMAIN).name(TASK_NAME).build());
    assertThat(taskId, equalTo(TaskIdentifier.builder().project(PROJECT).domain(DOMAIN).name(TASK_NAME).version(TASK_VERSION).build()));
}
Also used : TaskIdentifier(org.flyte.api.v1.TaskIdentifier) PartialTaskIdentifier(org.flyte.api.v1.PartialTaskIdentifier) Test(org.junit.Test)

Example 3 with TaskIdentifier

use of org.flyte.api.v1.TaskIdentifier in project flytekit-java by flyteorg.

the class FlyteAdminClientTest method fetchLatestTaskIdShouldReturnNullWhenEmptyList.

@Test
public void fetchLatestTaskIdShouldReturnNullWhenEmptyList() {
    stubService.taskLists = Collections.emptyList();
    TaskIdentifier taskId = client.fetchLatestTaskId(NamedEntityIdentifier.builder().project(PROJECT).domain(DOMAIN).name(TASK_NAME).build());
    assertThat(taskId, nullValue());
}
Also used : TaskIdentifier(org.flyte.api.v1.TaskIdentifier) PartialTaskIdentifier(org.flyte.api.v1.PartialTaskIdentifier) Test(org.junit.Test)

Example 4 with TaskIdentifier

use of org.flyte.api.v1.TaskIdentifier in project flytekit-java by flyteorg.

the class FlyteAdminClientTest method shouldPropagateCreateTaskToStub.

@Test
public void shouldPropagateCreateTaskToStub() {
    TaskIdentifier identifier = TaskIdentifier.builder().domain(DOMAIN).project(PROJECT).name(TASK_NAME).version(TASK_VERSION).build();
    TypedInterface interface_ = TypedInterface.builder().inputs(ImmutableMap.of("x", createVar(SimpleType.STRING))).outputs(ImmutableMap.of("y", createVar(SimpleType.INTEGER))).build();
    Container container = Container.builder().command(ImmutableList.of(COMMAND)).args(ImmutableList.of()).image(IMAGE_NAME).env(ImmutableList.of(KeyValuePair.of("key", "value"))).build();
    RetryStrategy retries = RetryStrategy.builder().retries(4).build();
    TaskTemplate template = TaskTemplate.builder().container(container).type("custom-task").interface_(interface_).custom(Struct.of(emptyMap())).retries(retries).build();
    client.createTask(identifier, template);
    assertThat(stubService.createTaskRequest, equalTo(TaskOuterClass.TaskCreateRequest.newBuilder().setId(newIdentifier(ResourceType.TASK, TASK_NAME, TASK_VERSION)).setSpec(newTaskSpec()).build()));
}
Also used : TaskTemplate(org.flyte.api.v1.TaskTemplate) TypedInterface(org.flyte.api.v1.TypedInterface) Container(org.flyte.api.v1.Container) TaskIdentifier(org.flyte.api.v1.TaskIdentifier) PartialTaskIdentifier(org.flyte.api.v1.PartialTaskIdentifier) RetryStrategy(org.flyte.api.v1.RetryStrategy) Test(org.junit.Test)

Example 5 with TaskIdentifier

use of org.flyte.api.v1.TaskIdentifier in project flytekit-java by flyteorg.

the class ExecuteDynamicWorkflow method execute.

private void execute() {
    Config config = Config.load();
    ExecutionConfig executionConfig = ExecutionConfig.load();
    Collection<ClassLoader> modules = ClassLoaders.forModuleDir(config.moduleDir()).values();
    Map<String, FileSystem> fileSystems = FileSystemLoader.loadFileSystems(modules);
    FileSystem outputFs = FileSystemLoader.getFileSystem(fileSystems, outputPrefix);
    ProtoWriter protoWriter = new ProtoWriter(outputPrefix, outputFs);
    try {
        FileSystem inputFs = FileSystemLoader.getFileSystem(fileSystems, inputs);
        ProtoReader protoReader = new ProtoReader(inputFs);
        TaskTemplate taskTemplate = protoReader.getTaskTemplate(taskTemplatePath);
        ClassLoader packageClassLoader = PackageLoader.load(fileSystems, taskTemplate);
        Map<String, String> env = getEnv();
        Map<WorkflowIdentifier, WorkflowTemplate> workflowTemplates = ClassLoaders.withClassLoader(packageClassLoader, () -> Registrars.loadAll(WorkflowTemplateRegistrar.class, env));
        Map<TaskIdentifier, RunnableTask> runnableTasks = ClassLoaders.withClassLoader(packageClassLoader, () -> Registrars.loadAll(RunnableTaskRegistrar.class, env));
        Map<TaskIdentifier, DynamicWorkflowTask> dynamicWorkflowTasks = ClassLoaders.withClassLoader(packageClassLoader, () -> Registrars.loadAll(DynamicWorkflowTaskRegistrar.class, env));
        // before we run anything, switch class loader, otherwise,
        // ServiceLoaders and other things wouldn't work, for instance,
        // FileSystemRegister in Apache Beam
        // we don't take the whole "custom" field, but only jflyte part, for that we ser-de it
        Struct custom = JFlyteCustom.deserializeFromStruct(taskTemplate.custom()).serializeToStruct();
        // all tasks already have staged jars, we can reuse 'jflyte' custom from current task to get
        // it
        Map<TaskIdentifier, TaskTemplate> taskTemplates = mapValues(ProjectClosure.createTaskTemplates(executionConfig, runnableTasks, dynamicWorkflowTasks), template -> template.toBuilder().custom(ProjectClosure.merge(template.custom(), custom)).build());
        DynamicJobSpec futures = withClassLoader(packageClassLoader, () -> {
            Map<String, Literal> input = protoReader.getInput(inputs);
            DynamicWorkflowTask task = getDynamicWorkflowTask(this.task);
            return task.run(input);
        });
        DynamicJobSpec rewrittenFutures = rewrite(executionConfig, futures, taskTemplates, workflowTemplates);
        if (rewrittenFutures.nodes().isEmpty()) {
            Map<String, Literal> outputs = getLiteralMap(rewrittenFutures.outputs());
            protoWriter.writeOutputs(outputs);
        } else {
            protoWriter.writeFutures(rewrittenFutures);
        }
    } catch (ContainerError e) {
        LOG.error("failed to run dynamic workflow", e);
        protoWriter.writeError(ProtoUtil.serializeContainerError(e));
    } catch (Throwable e) {
        LOG.error("failed to run dynamic workflow", e);
        protoWriter.writeError(ProtoUtil.serializeThrowable(e));
    }
}
Also used : DynamicWorkflowTaskRegistrar(org.flyte.api.v1.DynamicWorkflowTaskRegistrar) WorkflowTemplate(org.flyte.api.v1.WorkflowTemplate) TaskIdentifier(org.flyte.api.v1.TaskIdentifier) PartialTaskIdentifier(org.flyte.api.v1.PartialTaskIdentifier) Struct(org.flyte.api.v1.Struct) WorkflowIdentifier(org.flyte.api.v1.WorkflowIdentifier) PartialWorkflowIdentifier(org.flyte.api.v1.PartialWorkflowIdentifier) RunnableTaskRegistrar(org.flyte.api.v1.RunnableTaskRegistrar) FileSystem(org.flyte.jflyte.api.FileSystem) Literal(org.flyte.api.v1.Literal) ClassLoaders.withClassLoader(org.flyte.jflyte.ClassLoaders.withClassLoader) TaskTemplate(org.flyte.api.v1.TaskTemplate) WorkflowTemplateRegistrar(org.flyte.api.v1.WorkflowTemplateRegistrar) DynamicWorkflowTask(org.flyte.api.v1.DynamicWorkflowTask) RunnableTask(org.flyte.api.v1.RunnableTask) DynamicJobSpec(org.flyte.api.v1.DynamicJobSpec) ContainerError(org.flyte.api.v1.ContainerError)

Aggregations

TaskIdentifier (org.flyte.api.v1.TaskIdentifier)11 PartialTaskIdentifier (org.flyte.api.v1.PartialTaskIdentifier)8 HashMap (java.util.HashMap)4 RunnableTask (org.flyte.api.v1.RunnableTask)3 TaskTemplate (org.flyte.api.v1.TaskTemplate)3 Test (org.junit.Test)3 Test (org.junit.jupiter.api.Test)3 DynamicWorkflowTask (org.flyte.api.v1.DynamicWorkflowTask)2 RetryStrategy (org.flyte.api.v1.RetryStrategy)2 TypedInterface (org.flyte.api.v1.TypedInterface)2 ParameterizedTest (org.junit.jupiter.params.ParameterizedTest)2 IdentifierOuterClass (flyteidl.core.IdentifierOuterClass)1 Container (org.flyte.api.v1.Container)1 ContainerError (org.flyte.api.v1.ContainerError)1 DynamicJobSpec (org.flyte.api.v1.DynamicJobSpec)1 DynamicWorkflowTaskRegistrar (org.flyte.api.v1.DynamicWorkflowTaskRegistrar)1 Literal (org.flyte.api.v1.Literal)1 PartialWorkflowIdentifier (org.flyte.api.v1.PartialWorkflowIdentifier)1 Resources (org.flyte.api.v1.Resources)1 RunnableTaskRegistrar (org.flyte.api.v1.RunnableTaskRegistrar)1