Search in sources :

Example 21 with WorkflowTemplate

use of com.google.cloud.dataproc.v1.WorkflowTemplate in project flytekit-java by flyteorg.

the class LocalEngineTest method testBindingCollection.

@Test
public void testBindingCollection() {
    String workflowName = new ListWorkflow().getName();
    Map<String, WorkflowTemplate> workflows = loadWorkflows();
    Map<String, RunnableTask> tasks = loadTasks();
    WorkflowTemplate workflow = workflows.get(workflowName);
    Map<String, Literal> outputs = LocalEngine.compileAndExecute(workflow, tasks, emptyMap(), ImmutableMap.of());
    // 3 = 1 + 2, 7 = 3 + 4
    Literal i3 = Literal.ofScalar(Scalar.ofPrimitive(Primitive.ofIntegerValue(3)));
    Literal i7 = Literal.ofScalar(Scalar.ofPrimitive(Primitive.ofIntegerValue(7)));
    assertEquals(ImmutableMap.of("list", Literal.ofCollection(ImmutableList.of(i3, i7))), outputs);
}
Also used : WorkflowTemplate(org.flyte.api.v1.WorkflowTemplate) Literal(org.flyte.api.v1.Literal) RunnableTask(org.flyte.api.v1.RunnableTask) ListWorkflow(org.flyte.localengine.examples.ListWorkflow) Test(org.junit.jupiter.api.Test)

Example 22 with WorkflowTemplate

use of com.google.cloud.dataproc.v1.WorkflowTemplate in project flytekit-java by flyteorg.

the class SdkWorkflowBuilderTest method testTimes2WorkflowIdl.

@Test
void testTimes2WorkflowIdl() {
    SdkWorkflowBuilder builder = new SdkWorkflowBuilder();
    new Times2Workflow().expand(builder);
    Node expectedNode = Node.builder().id("square").taskNode(TaskNode.builder().referenceId(PartialTaskIdentifier.builder().name("org.flyte.flytekit.SdkWorkflowBuilderTest$MultiplicationTask").build()).build()).inputs(asList(Binding.builder().var_("a").binding(BindingData.ofOutputReference(OutputReference.builder().var("in").nodeId(Node.START_NODE_ID).build())).build(), Binding.builder().var_("b").binding(BindingData.ofScalar(Scalar.ofPrimitive(Primitive.ofIntegerValue(2L)))).build())).upstreamNodeIds(emptyList()).build();
    WorkflowTemplate expected = WorkflowTemplate.builder().metadata(WorkflowMetadata.builder().build()).interface_(expectedInterface()).outputs(expectedOutputs()).nodes(singletonList(expectedNode)).build();
    assertEquals(expected, builder.toIdlTemplate());
}
Also used : WorkflowTemplate(org.flyte.api.v1.WorkflowTemplate) TaskNode(org.flyte.api.v1.TaskNode) BranchNode(org.flyte.api.v1.BranchNode) Node(org.flyte.api.v1.Node) Test(org.junit.jupiter.api.Test) ParameterizedTest(org.junit.jupiter.params.ParameterizedTest)

Example 23 with WorkflowTemplate

use of com.google.cloud.dataproc.v1.WorkflowTemplate in project flytekit-java by flyteorg.

the class SdkWorkflowTemplateRegistrar method load.

Map<WorkflowIdentifier, WorkflowTemplate> load(SdkConfig sdkConfig, List<SdkWorkflow> sdkWorkflows) {
    LOG.fine("Discovering SdkWorkflow");
    Map<WorkflowIdentifier, WorkflowTemplate> workflows = new HashMap<>();
    for (SdkWorkflow sdkWorkflow : sdkWorkflows) {
        String name = sdkWorkflow.getName();
        WorkflowIdentifier workflowId = WorkflowIdentifier.builder().domain(sdkConfig.domain()).project(sdkConfig.project()).name(name).version(sdkConfig.version()).build();
        LOG.fine(String.format("Discovered [%s]", name));
        SdkWorkflowBuilder builder = new SdkWorkflowBuilder();
        sdkWorkflow.expand(builder);
        WorkflowTemplate workflow = WorkflowTemplateIdl.ofBuilder(builder);
        WorkflowTemplate previous = workflows.put(workflowId, workflow);
        if (previous != null) {
            throw new IllegalArgumentException(String.format("Discovered a duplicate workflow [%s] [%s] [%s]", name, workflow, previous));
        }
    }
    return workflows;
}
Also used : WorkflowIdentifier(org.flyte.api.v1.WorkflowIdentifier) WorkflowTemplate(org.flyte.api.v1.WorkflowTemplate) HashMap(java.util.HashMap)

Aggregations

WorkflowTemplate (org.flyte.api.v1.WorkflowTemplate)18 Test (org.junit.jupiter.api.Test)12 Node (org.flyte.api.v1.Node)8 Literal (org.flyte.api.v1.Literal)7 RunnableTask (org.flyte.api.v1.RunnableTask)7 WorkflowIdentifier (org.flyte.api.v1.WorkflowIdentifier)6 PartialWorkflowIdentifier (org.flyte.api.v1.PartialWorkflowIdentifier)5 TaskNode (org.flyte.api.v1.TaskNode)5 WorkflowNode (org.flyte.api.v1.WorkflowNode)5 Test (org.junit.Test)5 BranchNode (org.flyte.api.v1.BranchNode)4 TypedInterface (org.flyte.api.v1.TypedInterface)4 ListWorkflowTemplatesPagedResponse (com.google.cloud.dataproc.v1.WorkflowTemplateServiceClient.ListWorkflowTemplatesPagedResponse)3 AbstractMessage (com.google.protobuf.AbstractMessage)3 List (java.util.List)3 WorkflowMetadata (org.flyte.api.v1.WorkflowMetadata)3 ParameterizedTest (org.junit.jupiter.params.ParameterizedTest)3 WorkflowTemplate (com.google.cloud.dataproc.v1.WorkflowTemplate)2 HashMap (java.util.HashMap)2 DynamicWorkflowTask (org.flyte.api.v1.DynamicWorkflowTask)2