Search in sources :

Example 21 with OutputDescriptor

use of org.apache.tez.dag.api.OutputDescriptor in project tez by apache.

the class TestMemoryDistributor method testReserveFractionConfigured.

@Test(timeout = 5000)
public void testReserveFractionConfigured() throws TezException {
    Configuration conf = new Configuration(this.conf);
    conf.setDouble(TezConfiguration.TEZ_TASK_SCALE_MEMORY_RESERVE_FRACTION, 0.5d);
    MemoryDistributor dist = new MemoryDistributor(2, 1, conf);
    dist.setJvmMemory(10000l);
    // First request
    MemoryUpdateCallbackForTest e1Callback = new MemoryUpdateCallbackForTest();
    InputContext e1InputContext1 = createTestInputContext();
    InputDescriptor e1InDesc1 = createTestInputDescriptor();
    dist.requestMemory(10000, e1Callback, e1InputContext1, e1InDesc1);
    // Second request
    MemoryUpdateCallbackForTest e2Callback = new MemoryUpdateCallbackForTest();
    InputContext e2InputContext2 = createTestInputContext();
    InputDescriptor e2InDesc2 = createTestInputDescriptor();
    dist.requestMemory(10000, e2Callback, e2InputContext2, e2InDesc2);
    // Third request - output
    MemoryUpdateCallbackForTest e3Callback = new MemoryUpdateCallbackForTest();
    OutputContext e3OutputContext1 = createTestOutputContext();
    OutputDescriptor e3OutDesc2 = createTestOutputDescriptor();
    dist.requestMemory(5000, e3Callback, e3OutputContext1, e3OutDesc2);
    dist.makeInitialAllocations();
    // Total available: 50% of 10K = 7000
    // 3 requests - 10K, 10K, 5K
    // Scale down to - 2000, 2000, 1000
    assertEquals(2000, e1Callback.assigned);
    assertEquals(2000, e2Callback.assigned);
    assertEquals(1000, e3Callback.assigned);
}
Also used : InputDescriptor(org.apache.tez.dag.api.InputDescriptor) Configuration(org.apache.hadoop.conf.Configuration) TezConfiguration(org.apache.tez.dag.api.TezConfiguration) OutputDescriptor(org.apache.tez.dag.api.OutputDescriptor) InputContext(org.apache.tez.runtime.api.InputContext) OutputContext(org.apache.tez.runtime.api.OutputContext) Test(org.junit.Test)

Example 22 with OutputDescriptor

use of org.apache.tez.dag.api.OutputDescriptor in project tez by apache.

the class TestMemoryDistributor method createTestOutputDescriptor.

protected OutputDescriptor createTestOutputDescriptor() {
    OutputDescriptor desc = mock(OutputDescriptor.class);
    doReturn("OutputClass").when(desc).getClassName();
    return desc;
}
Also used : OutputDescriptor(org.apache.tez.dag.api.OutputDescriptor)

Example 23 with OutputDescriptor

use of org.apache.tez.dag.api.OutputDescriptor in project tez by apache.

the class TestMemoryDistributor method testScalingNoProcessor.

@Test(timeout = 5000)
public void testScalingNoProcessor() throws TezException {
    MemoryDistributor dist = new MemoryDistributor(2, 1, conf);
    dist.setJvmMemory(10000l);
    // First request
    MemoryUpdateCallbackForTest e1Callback = new MemoryUpdateCallbackForTest();
    InputContext e1InputContext1 = createTestInputContext();
    InputDescriptor e1InDesc1 = createTestInputDescriptor();
    dist.requestMemory(10000, e1Callback, e1InputContext1, e1InDesc1);
    // Second request
    MemoryUpdateCallbackForTest e2Callback = new MemoryUpdateCallbackForTest();
    InputContext e2InputContext2 = createTestInputContext();
    InputDescriptor e2InDesc2 = createTestInputDescriptor();
    dist.requestMemory(10000, e2Callback, e2InputContext2, e2InDesc2);
    // Third request - output
    MemoryUpdateCallbackForTest e3Callback = new MemoryUpdateCallbackForTest();
    OutputContext e3OutputContext1 = createTestOutputContext();
    OutputDescriptor e3OutDesc2 = createTestOutputDescriptor();
    dist.requestMemory(5000, e3Callback, e3OutputContext1, e3OutDesc2);
    dist.makeInitialAllocations();
    // Total available: 70% of 10K = 7000
    // 3 requests - 10K, 10K, 5K
    // Scale down to - 2800, 2800, 1400
    assertEquals(2800, e1Callback.assigned);
    assertEquals(2800, e2Callback.assigned);
    assertEquals(1400, e3Callback.assigned);
}
Also used : InputDescriptor(org.apache.tez.dag.api.InputDescriptor) OutputDescriptor(org.apache.tez.dag.api.OutputDescriptor) InputContext(org.apache.tez.runtime.api.InputContext) OutputContext(org.apache.tez.runtime.api.OutputContext) Test(org.junit.Test)

Example 24 with OutputDescriptor

use of org.apache.tez.dag.api.OutputDescriptor in project tez by apache.

the class TestLogicalIOProcessorRuntimeTask method createOutputSpecList.

private List<OutputSpec> createOutputSpecList() {
    OutputDescriptor outputtDesc = OutputDescriptor.create(TestOutput.class.getName());
    OutputSpec outputSpec = new OutputSpec("outedge", outputtDesc, 1);
    return Lists.newArrayList(outputSpec);
}
Also used : OutputDescriptor(org.apache.tez.dag.api.OutputDescriptor) OutputSpec(org.apache.tez.runtime.api.impl.OutputSpec)

Example 25 with OutputDescriptor

use of org.apache.tez.dag.api.OutputDescriptor in project tez by apache.

the class OutputSpec method readFields.

@Override
public void readFields(DataInput in) throws IOException {
    destinationVertexName = StringInterner.weakIntern(in.readUTF());
    physicalEdgeCount = in.readInt();
    outputDescriptor = new OutputDescriptor();
    outputDescriptor.readFields(in);
}
Also used : OutputDescriptor(org.apache.tez.dag.api.OutputDescriptor)

Aggregations

OutputDescriptor (org.apache.tez.dag.api.OutputDescriptor)32 Test (org.junit.Test)14 OutputContext (org.apache.tez.runtime.api.OutputContext)13 InputDescriptor (org.apache.tez.dag.api.InputDescriptor)10 UserPayload (org.apache.tez.dag.api.UserPayload)10 Configuration (org.apache.hadoop.conf.Configuration)9 OutputCommitterDescriptor (org.apache.tez.dag.api.OutputCommitterDescriptor)9 TezConfiguration (org.apache.tez.dag.api.TezConfiguration)8 InputContext (org.apache.tez.runtime.api.InputContext)8 DAG (org.apache.tez.dag.api.DAG)7 Resource (org.apache.hadoop.yarn.api.records.Resource)6 Path (org.apache.hadoop.fs.Path)5 DataSinkDescriptor (org.apache.tez.dag.api.DataSinkDescriptor)5 WeightedScalingMemoryDistributor (org.apache.tez.runtime.library.resources.WeightedScalingMemoryDistributor)5 GroupInputEdge (org.apache.tez.dag.api.GroupInputEdge)4 ProcessorDescriptor (org.apache.tez.dag.api.ProcessorDescriptor)4 Vertex (org.apache.tez.dag.api.Vertex)4 OutputSpec (org.apache.tez.runtime.api.impl.OutputSpec)4 ByteString (com.google.protobuf.ByteString)3 JobConf (org.apache.hadoop.mapred.JobConf)3