Search in sources :

Example 6 with StopWatch

use of org.apache.tez.util.StopWatch in project tez by apache.

the class TezUtilsInternal method uncompressBytes.

public static byte[] uncompressBytes(byte[] inBytes) throws IOException {
    StopWatch sw = new StopWatch().start();
    byte[] uncompressed = uncompressBytesInflateDeflate(inBytes);
    sw.stop();
    if (LOG.isDebugEnabled()) {
        LOG.debug("CompressedSize: " + inBytes.length + ", UncompressedSize: " + uncompressed.length + ", UncompressTimeTaken: " + sw.now(TimeUnit.MILLISECONDS));
    }
    return uncompressed;
}
Also used : StopWatch(org.apache.tez.util.StopWatch)

Example 7 with StopWatch

use of org.apache.tez.util.StopWatch in project tez by apache.

the class TestMemoryWithEvents method testMemory.

private void testMemory(DAG dag, boolean sendDMEvents) throws Exception {
    StopWatch stopwatch = new StopWatch();
    stopwatch.start();
    TezConfiguration tezconf = new TezConfiguration(defaultConf);
    MockTezClient tezClient = new MockTezClient("testMockAM", tezconf, true, null, null, null, null, false, false, numThreads, 1000);
    tezClient.start();
    MockDAGAppMaster mockApp = tezClient.getLocalClient().getMockApp();
    MockContainerLauncher mockLauncher = mockApp.getContainerLauncher();
    mockLauncher.startScheduling(false);
    mockApp.eventsDelegate = new TestMockDAGAppMaster.TestEventsDelegate();
    mockApp.doSleep = false;
    DAGClient dagClient = tezClient.submitDAG(dag);
    mockLauncher.waitTillContainersLaunched();
    mockLauncher.startScheduling(true);
    DAGStatus status = dagClient.waitForCompletion();
    Assert.assertEquals(DAGStatus.State.SUCCEEDED, status.getState());
    checkMemory(dag.getName(), mockApp);
    stopwatch.stop();
    System.out.println("Time taken(ms): " + stopwatch.now(TimeUnit.MILLISECONDS));
    tezClient.stop();
}
Also used : DAGClient(org.apache.tez.dag.api.client.DAGClient) DAGStatus(org.apache.tez.dag.api.client.DAGStatus) MockContainerLauncher(org.apache.tez.dag.app.MockDAGAppMaster.MockContainerLauncher) StopWatch(org.apache.tez.util.StopWatch) TezConfiguration(org.apache.tez.dag.api.TezConfiguration)

Aggregations

StopWatch (org.apache.tez.util.StopWatch)7 Configuration (org.apache.hadoop.conf.Configuration)2 JobConf (org.apache.hadoop.mapred.JobConf)2 DAGClient (org.apache.tez.dag.api.client.DAGClient)2 Preconditions (com.google.common.base.Preconditions)1 IOException (java.io.IOException)1 InterruptedIOException (java.io.InterruptedIOException)1 StringWriter (java.io.StringWriter)1 ByteBuffer (java.nio.ByteBuffer)1 EnumSet (java.util.EnumSet)1 HashMap (java.util.HashMap)1 LinkedList (java.util.LinkedList)1 List (java.util.List)1 Map (java.util.Map)1 Set (java.util.Set)1 TimeUnit (java.util.concurrent.TimeUnit)1 Collectors (java.util.stream.Collectors)1 ExceptionUtils (org.apache.commons.lang3.exception.ExceptionUtils)1 InPlaceUpdate (org.apache.hadoop.hive.common.log.InPlaceUpdate)1 ProgressMonitor (org.apache.hadoop.hive.common.log.ProgressMonitor)1