Search in sources :

Example 1 with TezMergedInputContextImpl

use of org.apache.tez.runtime.api.impl.TezMergedInputContextImpl in project tez by apache.

the class LogicalIOProcessorRuntimeTask method initializeGroupInputs.

private void initializeGroupInputs() throws TezException {
    if (groupInputSpecs != null && !groupInputSpecs.isEmpty()) {
        groupInputsMap = new ConcurrentHashMap<String, MergedLogicalInput>(groupInputSpecs.size());
        for (GroupInputSpec groupInputSpec : groupInputSpecs) {
            LOG.debug("Initializing GroupInput using GroupInputSpec: {}", groupInputSpec);
            MergedInputContext mergedInputContext = new TezMergedInputContextImpl(groupInputSpec.getMergedInputDescriptor().getUserPayload(), groupInputSpec.getGroupName(), groupInputsMap, inputReadyTracker, localDirs, this);
            List<Input> inputs = Lists.newArrayListWithCapacity(groupInputSpec.getGroupVertices().size());
            for (String groupVertex : groupInputSpec.getGroupVertices()) {
                inputs.add(inputsMap.get(groupVertex));
            }
            MergedLogicalInput groupInput = (MergedLogicalInput) createMergedInput(groupInputSpec.getMergedInputDescriptor(), mergedInputContext, inputs);
            groupInputsMap.put(groupInputSpec.getGroupName(), groupInput);
        }
    }
}
Also used : LogicalInput(org.apache.tez.runtime.api.LogicalInput) Input(org.apache.tez.runtime.api.Input) MergedLogicalInput(org.apache.tez.runtime.api.MergedLogicalInput) TezMergedInputContextImpl(org.apache.tez.runtime.api.impl.TezMergedInputContextImpl) MergedLogicalInput(org.apache.tez.runtime.api.MergedLogicalInput) GroupInputSpec(org.apache.tez.runtime.api.impl.GroupInputSpec) MergedInputContext(org.apache.tez.runtime.api.MergedInputContext)

Example 2 with TezMergedInputContextImpl

use of org.apache.tez.runtime.api.impl.TezMergedInputContextImpl in project tez by apache.

the class TestInputReadyTracker method testGrouped.

@Test(timeout = 20000)
public void testGrouped() throws InterruptedException {
    InputReadyTracker inputReadyTracker = new InputReadyTracker();
    ImmediatelyReadyInputForTest input1 = new ImmediatelyReadyInputForTest(inputReadyTracker);
    ControlledReadyInputForTest input2 = new ControlledReadyInputForTest(inputReadyTracker);
    ImmediatelyReadyInputForTest input3 = new ImmediatelyReadyInputForTest(inputReadyTracker);
    ControlledReadyInputForTest input4 = new ControlledReadyInputForTest(inputReadyTracker);
    List<Input> group1Inputs = new ArrayList<Input>();
    group1Inputs.add(input1);
    group1Inputs.add(input2);
    List<Input> group2Inputs = new ArrayList<Input>();
    group2Inputs.add(input3);
    group2Inputs.add(input4);
    Map<String, MergedLogicalInput> mergedInputMap = new HashMap<String, MergedLogicalInput>();
    MergedInputContext mergedInputContext1 = new TezMergedInputContextImpl(null, "group1", mergedInputMap, inputReadyTracker, null, null);
    MergedInputContext mergedInputContext2 = new TezMergedInputContextImpl(null, "group2", mergedInputMap, inputReadyTracker, null, null);
    AnyOneMergedInputForTest group1 = new AnyOneMergedInputForTest(mergedInputContext1, group1Inputs);
    AllMergedInputForTest group2 = new AllMergedInputForTest(mergedInputContext2, group2Inputs);
    mergedInputMap.put("group1", group1);
    mergedInputMap.put("group2", group2);
    // Register groups with tracker
    List<MergedLogicalInput> groups = Lists.newArrayList(group1, group2);
    inputReadyTracker.setGroupedInputs(groups);
    // Test for simple inputs
    List<Input> requestList;
    long startTime = 0l;
    long readyTime = 0l;
    requestList = new ArrayList<Input>();
    requestList.add(group1);
    Input readyInput = inputReadyTracker.waitForAnyInputReady(requestList);
    assertTrue(group1.isReady);
    assertTrue(input1.isReady);
    assertFalse(input2.isReady);
    assertEquals(group1, readyInput);
    requestList = new ArrayList<Input>();
    requestList.add(group2);
    startTime = System.nanoTime();
    setDelayedInputReady(input4);
    inputReadyTracker.waitForAllInputsReady(requestList);
    readyTime = System.nanoTime();
    // Should have moved into ready state - only happens when the setReady function is invoked.
    // Ensure the method returned only after the specific Input was told it is ready
    assertTrue(group2.isReady);
    assertTrue(input3.isReady);
    assertTrue(input4.isReady);
    assertTrue(readyTime >= startTime + SLEEP_TIME);
}
Also used : HashMap(java.util.HashMap) TezMergedInputContextImpl(org.apache.tez.runtime.api.impl.TezMergedInputContextImpl) MergedLogicalInput(org.apache.tez.runtime.api.MergedLogicalInput) ArrayList(java.util.ArrayList) MergedInputContext(org.apache.tez.runtime.api.MergedInputContext) Input(org.apache.tez.runtime.api.Input) AbstractLogicalInput(org.apache.tez.runtime.api.AbstractLogicalInput) MergedLogicalInput(org.apache.tez.runtime.api.MergedLogicalInput) Test(org.junit.Test)

Aggregations

Input (org.apache.tez.runtime.api.Input)2 MergedInputContext (org.apache.tez.runtime.api.MergedInputContext)2 MergedLogicalInput (org.apache.tez.runtime.api.MergedLogicalInput)2 TezMergedInputContextImpl (org.apache.tez.runtime.api.impl.TezMergedInputContextImpl)2 ArrayList (java.util.ArrayList)1 HashMap (java.util.HashMap)1 AbstractLogicalInput (org.apache.tez.runtime.api.AbstractLogicalInput)1 LogicalInput (org.apache.tez.runtime.api.LogicalInput)1 GroupInputSpec (org.apache.tez.runtime.api.impl.GroupInputSpec)1 Test (org.junit.Test)1