Examples with RuntimeMetric com.facebook.presto.common.RuntimeMetric used on opensource projects

Search in sources :

Example 1 with RuntimeMetric

use of com.facebook.presto.common.RuntimeMetric in project presto by prestodb.

the class StatusPrinter method printFinalInfo.

public void printFinalInfo() {
    Duration wallTime = nanosSince(start);
    QueryStatusInfo results = client.finalStatusInfo();
    StatementStats stats = results.getStats();
    int nodes = stats.getNodes();
    if ((nodes == 0) || (stats.getTotalSplits() == 0)) {
        return;
    }
    // blank line
    out.println();
    // Query 12, FINISHED, 1 node
    String querySummary = format("Query %s, %s, %,d %s", results.getId(), stats.getState(), nodes, pluralize("node", nodes));
    out.println(querySummary);
    if (debug) {
        out.println(results.getInfoUri().toString());
    }
    // Splits: 1000 total, 842 done (84.20%)
    String splitsSummary = format("Splits: %,d total, %,d done (%.2f%%)", stats.getTotalSplits(), stats.getCompletedSplits(), stats.getProgressPercentage().orElse(0.0));
    out.println(splitsSummary);
    if (debug) {
        // CPU Time: 565.2s total,   26K rows/s, 3.85MB/s
        Duration cpuTime = millis(stats.getCpuTimeMillis());
        String cpuTimeSummary = format("CPU Time: %.1fs total, %5s rows/s, %8s, %d%% active", cpuTime.getValue(SECONDS), formatCountRate(stats.getProcessedRows(), cpuTime, false), formatDataRate(bytes(stats.getProcessedBytes()), cpuTime, true), (int) percentage(stats.getCpuTimeMillis(), stats.getWallTimeMillis()));
        out.println(cpuTimeSummary);
        double parallelism = cpuTime.getValue(MILLISECONDS) / wallTime.getValue(MILLISECONDS);
        // Per Node: 3.5 parallelism, 83.3K rows/s, 0.7 MB/s
        String perNodeSummary = format("Per Node: %.1f parallelism, %5s rows/s, %8s", parallelism / nodes, formatCountRate((double) stats.getProcessedRows() / nodes, wallTime, false), formatDataRate(bytes(stats.getProcessedBytes() / nodes), wallTime, true));
        reprintLine(perNodeSummary);
        // Parallelism: 5.3
        out.println(format("Parallelism: %.1f", parallelism));
        // Peak User Memory: 1.97GB
        reprintLine("Peak User Memory: " + formatDataSize(bytes(stats.getPeakMemoryBytes()), true));
        // Peak Total Memory: 1.98GB
        reprintLine("Peak Total Memory: " + formatDataSize(bytes(stats.getPeakTotalMemoryBytes()), true));
        // Peak Task Total Memory: 1.99GB
        reprintLine("Peak Task Total Memory: " + formatDataSize(bytes(stats.getPeakTaskTotalMemoryBytes()), true));
        // Spilled Data: 20GB
        if (stats.getSpilledBytes() > 0) {
            reprintLine("Spilled: " + formatDataSize(bytes(stats.getSpilledBytes()), true));
        }
        // bytesFromCache: sum=2K count=2 min=1K max=1K
        if (stats.getRuntimeStats() != null) {
            stats.getRuntimeStats().getMetrics().values().stream().sorted(Comparator.comparing(RuntimeMetric::getName)).forEach(metric -> reprintLine(format("%s: sum=%s count=%s min=%s max=%s", metric.getName(), autoFormatMetricValue(metric.getName(), metric.getSum()), formatCount(metric.getCount()), autoFormatMetricValue(metric.getName(), metric.getMin()), autoFormatMetricValue(metric.getName(), metric.getMax()))));
        }
    }
    // 0:32 [2.12GB, 15M rows] [67MB/s, 463K rows/s]
    String statsLine = format("%s [%s rows, %s] [%s rows/s, %s]", formatTime(wallTime), formatCount(stats.getProcessedRows()), formatDataSize(bytes(stats.getProcessedBytes()), true), formatCountRate(stats.getProcessedRows(), wallTime, false), formatDataRate(bytes(stats.getProcessedBytes()), wallTime, true));
    out.println(statsLine);
    // blank line
    out.println();
}
Also used : StatementStats(com.facebook.presto.client.StatementStats) Duration(io.airlift.units.Duration) QueryStatusInfo(com.facebook.presto.client.QueryStatusInfo) RuntimeMetric(com.facebook.presto.common.RuntimeMetric)

Example 2 with RuntimeMetric

use of com.facebook.presto.common.RuntimeMetric in project presto by prestodb.

the class TestOperatorStats method testAdd.

@Test
public void testAdd() {
    OperatorStats actual = EXPECTED.add(ImmutableList.of(EXPECTED, EXPECTED));
    assertEquals(actual.getStageId(), 0);
    assertEquals(actual.getStageExecutionId(), 10);
    assertEquals(actual.getOperatorId(), 41);
    assertEquals(actual.getOperatorType(), "test");
    assertEquals(actual.getTotalDrivers(), 3 * 1);
    assertEquals(actual.getAddInputCalls(), 3 * 2);
    assertEquals(actual.getAddInputWall(), new Duration(3 * 3, NANOSECONDS));
    assertEquals(actual.getAddInputCpu(), new Duration(3 * 4, NANOSECONDS));
    assertEquals(actual.getAddInputAllocation(), new DataSize(3 * 123, BYTE));
    assertEquals(actual.getRawInputDataSize(), new DataSize(3 * 5, BYTE));
    assertEquals(actual.getInputDataSize(), new DataSize(3 * 6, BYTE));
    assertEquals(actual.getInputPositions(), 3 * 7);
    assertEquals(actual.getSumSquaredInputPositions(), 3 * 8.0);
    assertEquals(actual.getGetOutputCalls(), 3 * 9);
    assertEquals(actual.getGetOutputWall(), new Duration(3 * 10, NANOSECONDS));
    assertEquals(actual.getGetOutputCpu(), new Duration(3 * 11, NANOSECONDS));
    assertEquals(actual.getGetOutputAllocation(), new DataSize(3 * 234, BYTE));
    assertEquals(actual.getOutputDataSize(), new DataSize(3 * 12, BYTE));
    assertEquals(actual.getOutputPositions(), 3 * 13);
    assertEquals(actual.getPhysicalWrittenDataSize(), new DataSize(3 * 14, BYTE));
    assertEquals(actual.getAdditionalCpu(), new Duration(3 * 100, NANOSECONDS));
    assertEquals(actual.getBlockedWall(), new Duration(3 * 15, NANOSECONDS));
    assertEquals(actual.getFinishCalls(), 3 * 16);
    assertEquals(actual.getFinishWall(), new Duration(3 * 17, NANOSECONDS));
    assertEquals(actual.getFinishCpu(), new Duration(3 * 18, NANOSECONDS));
    assertEquals(actual.getFinishAllocation(), new DataSize(3 * 345, BYTE));
    assertEquals(actual.getUserMemoryReservation(), new DataSize(3 * 19, BYTE));
    assertEquals(actual.getRevocableMemoryReservation(), new DataSize(3 * 20, BYTE));
    assertEquals(actual.getSystemMemoryReservation(), new DataSize(3 * 21, BYTE));
    assertEquals(actual.getPeakUserMemoryReservation(), new DataSize(22, BYTE));
    assertEquals(actual.getPeakSystemMemoryReservation(), new DataSize(23, BYTE));
    assertEquals(actual.getPeakTotalMemoryReservation(), new DataSize(24, BYTE));
    assertEquals(actual.getSpilledDataSize(), new DataSize(3 * 25, BYTE));
    assertNull(actual.getInfo());
    RuntimeMetric expectedMetric = RuntimeMetric.merge(TEST_RUNTIME_METRIC_1, TEST_RUNTIME_METRIC_1);
    expectedMetric.mergeWith(TEST_RUNTIME_METRIC_1);
    assertRuntimeMetricEquals(actual.getRuntimeStats().getMetric(TEST_METRIC_NAME), expectedMetric);
}
Also used : DataSize(io.airlift.units.DataSize) Duration(io.airlift.units.Duration) RuntimeMetric(com.facebook.presto.common.RuntimeMetric) Test(org.testng.annotations.Test)

Example 3 with RuntimeMetric

use of com.facebook.presto.common.RuntimeMetric in project presto by prestodb.

the class StatusPrinter method printQueryInfo.

private void printQueryInfo(QueryStatusInfo results, WarningsPrinter warningsPrinter) {
    StatementStats stats = results.getStats();
    Duration wallTime = nanosSince(start);
    // cap progress at 99%, otherwise it looks weird when the query is still running and it says 100%
    int progressPercentage = (int) min(99, stats.getProgressPercentage().orElse(0.0));
    if (console.isRealTerminal()) {
        // blank line
        reprintLine("");
        int terminalWidth = console.getWidth();
        if (terminalWidth < 75) {
            reprintLine("WARNING: Terminal");
            reprintLine("must be at least");
            reprintLine("80 characters wide");
            reprintLine("");
            reprintLine(stats.getState());
            reprintLine(format("%s %d%%", formatTime(wallTime), progressPercentage));
            return;
        }
        int nodes = stats.getNodes();
        // Query 10, RUNNING, 1 node, 778 splits
        String querySummary = format("Query %s, %s, %,d %s, %,d splits", results.getId(), stats.getState(), nodes, pluralize("node", nodes), stats.getTotalSplits());
        reprintLine(querySummary);
        String url = results.getInfoUri().toString();
        if (debug && (url.length() < terminalWidth)) {
            reprintLine(url);
        }
        if ((nodes == 0) || (stats.getTotalSplits() == 0)) {
            return;
        }
        if (debug) {
            // Splits:   620 queued, 34 running, 124 done
            String splitsSummary = format("Splits:   %,d queued, %,d running, %,d done", stats.getQueuedSplits(), stats.getRunningSplits(), stats.getCompletedSplits());
            reprintLine(splitsSummary);
            // CPU Time: 56.5s total, 36.4K rows/s, 4.44MB/s, 60% active
            Duration cpuTime = millis(stats.getCpuTimeMillis());
            String cpuTimeSummary = format("CPU Time: %.1fs total, %5s rows/s, %8s, %d%% active", cpuTime.getValue(SECONDS), formatCountRate(stats.getProcessedRows(), cpuTime, false), formatDataRate(bytes(stats.getProcessedBytes()), cpuTime, true), (int) percentage(stats.getCpuTimeMillis(), stats.getWallTimeMillis()));
            reprintLine(cpuTimeSummary);
            double parallelism = cpuTime.getValue(MILLISECONDS) / wallTime.getValue(MILLISECONDS);
            // Per Node: 3.5 parallelism, 83.3K rows/s, 0.7 MB/s
            String perNodeSummary = format("Per Node: %.1f parallelism, %5s rows/s, %8s", parallelism / nodes, formatCountRate((double) stats.getProcessedRows() / nodes, wallTime, false), formatDataRate(bytes(stats.getProcessedBytes() / nodes), wallTime, true));
            reprintLine(perNodeSummary);
            // Parallelism: 5.3
            reprintLine(format("Parallelism: %.1f", parallelism));
            // Peak Memory: 1.97GB
            reprintLine("Peak Memory: " + formatDataSize(bytes(stats.getPeakMemoryBytes()), true));
            // Peak Total Memory: 1.98GB
            reprintLine("Peak Total Memory: " + formatDataSize(bytes(stats.getPeakTotalMemoryBytes()), true));
            // Peak Task Total Memory: 1.99GB
            reprintLine("Peak Task Total Memory: " + formatDataSize(bytes(stats.getPeakTaskTotalMemoryBytes()), true));
            // Spilled Data: 20GB
            if (stats.getSpilledBytes() > 0) {
                reprintLine("Spilled: " + formatDataSize(bytes(stats.getSpilledBytes()), true));
            }
            if (stats.getRuntimeStats() != null) {
                stats.getRuntimeStats().getMetrics().values().stream().sorted(Comparator.comparing(RuntimeMetric::getName)).forEach(metric -> reprintLine(format("%s: sum=%s count=%s min=%s max=%s", metric.getName(), autoFormatMetricValue(metric.getName(), metric.getSum()), formatCount(metric.getCount()), autoFormatMetricValue(metric.getName(), metric.getMin()), autoFormatMetricValue(metric.getName(), metric.getMax()))));
            }
        }
        // otherwise handled above
        verify(terminalWidth >= 75);
        // progress bar is 17-42 characters wide
        int progressWidth = (min(terminalWidth, 100) - 75) + 17;
        String formattedWallTime = formatTime(wallTime);
        if (formattedWallTime.length() > 5) {
            // fix overflowed progress bar for queries running over 100 minutes
            progressWidth -= formattedWallTime.length() - 5;
        }
        if (stats.isScheduled()) {
            String progressBar = formatProgressBar(progressWidth, stats.getCompletedSplits(), max(0, stats.getRunningSplits()), stats.getTotalSplits());
            // 0:17 [ 103MB,  802K rows] [5.74MB/s, 44.9K rows/s] [=====>>                                   ] 10%
            String progressLine = format("%s [%5s rows, %6s] [%5s rows/s, %8s] [%s] %d%%", formattedWallTime, formatCount(stats.getProcessedRows()), formatDataSize(bytes(stats.getProcessedBytes()), true), formatCountRate(stats.getProcessedRows(), wallTime, false), formatDataRate(bytes(stats.getProcessedBytes()), wallTime, true), progressBar, progressPercentage);
            reprintLine(progressLine);
        } else {
            String progressBar = formatProgressBar(progressWidth, Ints.saturatedCast(nanosSince(start).roundTo(SECONDS)));
            // 0:17 [ 103MB,  802K rows] [5.74MB/s, 44.9K rows/s] [    <=>                                  ]
            String progressLine = format("%s [%5s rows, %6s] [%5s rows/s, %8s] [%s]", formattedWallTime, formatCount(stats.getProcessedRows()), formatDataSize(bytes(stats.getProcessedBytes()), true), formatCountRate(stats.getProcessedRows(), wallTime, false), formatDataRate(bytes(stats.getProcessedBytes()), wallTime, true), progressBar);
            reprintLine(progressLine);
        }
        // blank line
        reprintLine("");
        // STAGE  S    ROWS    RPS  BYTES    BPS   QUEUED    RUN   DONE
        String stagesHeader = format("%10s%1s  %5s  %6s  %5s  %7s  %6s  %5s  %5s", "STAGE", "S", "ROWS", "ROWS/s", "BYTES", "BYTES/s", "QUEUED", "RUN", "DONE");
        reprintLine(stagesHeader);
        printStageTree(stats.getRootStage(), "", new AtomicInteger());
    } else {
        // Query 31 [S] i[2.7M 67.3MB 62.7MBps] o[35 6.1KB 1KBps] splits[252/16/380]
        String querySummary = format("Query %s [%s] i[%s %s %s] o[%s %s %s] splits[%,d/%,d/%,d]", results.getId(), stats.getState(), formatCount(stats.getProcessedRows()), formatDataSize(bytes(stats.getProcessedBytes()), false), formatDataRate(bytes(stats.getProcessedBytes()), wallTime, false), formatCount(stats.getProcessedRows()), formatDataSize(bytes(stats.getProcessedBytes()), false), formatDataRate(bytes(stats.getProcessedBytes()), wallTime, false), stats.getQueuedSplits(), stats.getRunningSplits(), stats.getCompletedSplits());
        reprintLine(querySummary);
    }
    warningsPrinter.print(results.getWarnings(), false, false);
}
Also used : StatementStats(com.facebook.presto.client.StatementStats) AtomicInteger(java.util.concurrent.atomic.AtomicInteger) Duration(io.airlift.units.Duration) RuntimeMetric(com.facebook.presto.common.RuntimeMetric)

Example 4 with RuntimeMetric

use of com.facebook.presto.common.RuntimeMetric in project presto by prestodb.

the class TestOperatorStats method testAddMergeable.

@Test
public void testAddMergeable() {
    OperatorStats actual = MERGEABLE.add(ImmutableList.of(MERGEABLE, MERGEABLE));
    assertEquals(actual.getStageId(), 0);
    assertEquals(actual.getStageExecutionId(), 10);
    assertEquals(actual.getOperatorId(), 41);
    assertEquals(actual.getOperatorType(), "test");
    assertEquals(actual.getTotalDrivers(), 3 * 1);
    assertEquals(actual.getAddInputCalls(), 3 * 2);
    assertEquals(actual.getAddInputWall(), new Duration(3 * 3, NANOSECONDS));
    assertEquals(actual.getAddInputCpu(), new Duration(3 * 4, NANOSECONDS));
    assertEquals(actual.getAddInputAllocation(), new DataSize(3 * 123, BYTE));
    assertEquals(actual.getRawInputDataSize(), new DataSize(3 * 5, BYTE));
    assertEquals(actual.getInputDataSize(), new DataSize(3 * 6, BYTE));
    assertEquals(actual.getInputPositions(), 3 * 7);
    assertEquals(actual.getSumSquaredInputPositions(), 3 * 8.0);
    assertEquals(actual.getGetOutputCalls(), 3 * 9);
    assertEquals(actual.getGetOutputWall(), new Duration(3 * 10, NANOSECONDS));
    assertEquals(actual.getGetOutputCpu(), new Duration(3 * 11, NANOSECONDS));
    assertEquals(actual.getGetOutputAllocation(), new DataSize(3 * 234, BYTE));
    assertEquals(actual.getOutputDataSize(), new DataSize(3 * 12, BYTE));
    assertEquals(actual.getOutputPositions(), 3 * 13);
    assertEquals(actual.getPhysicalWrittenDataSize(), new DataSize(3 * 14, BYTE));
    assertEquals(actual.getAdditionalCpu(), new Duration(3 * 100, NANOSECONDS));
    assertEquals(actual.getBlockedWall(), new Duration(3 * 15, NANOSECONDS));
    assertEquals(actual.getFinishCalls(), 3 * 16);
    assertEquals(actual.getFinishWall(), new Duration(3 * 17, NANOSECONDS));
    assertEquals(actual.getFinishCpu(), new Duration(3 * 18, NANOSECONDS));
    assertEquals(actual.getFinishAllocation(), new DataSize(3 * 345, BYTE));
    assertEquals(actual.getUserMemoryReservation(), new DataSize(3 * 19, BYTE));
    assertEquals(actual.getRevocableMemoryReservation(), new DataSize(3 * 20, BYTE));
    assertEquals(actual.getSystemMemoryReservation(), new DataSize(3 * 21, BYTE));
    assertEquals(actual.getPeakUserMemoryReservation(), new DataSize(22, BYTE));
    assertEquals(actual.getPeakSystemMemoryReservation(), new DataSize(23, BYTE));
    assertEquals(actual.getPeakTotalMemoryReservation(), new DataSize(24, BYTE));
    assertEquals(actual.getSpilledDataSize(), new DataSize(3 * 25, BYTE));
    assertEquals(actual.getInfo().getClass(), PartitionedOutputInfo.class);
    assertEquals(((PartitionedOutputInfo) actual.getInfo()).getPagesAdded(), 3 * MERGEABLE_INFO.getPagesAdded());
    RuntimeMetric expectedMetric = RuntimeMetric.merge(TEST_RUNTIME_METRIC_2, TEST_RUNTIME_METRIC_2);
    expectedMetric.mergeWith(TEST_RUNTIME_METRIC_2);
    assertRuntimeMetricEquals(actual.getRuntimeStats().getMetric(TEST_METRIC_NAME), expectedMetric);
}
Also used : DataSize(io.airlift.units.DataSize) Duration(io.airlift.units.Duration) RuntimeMetric(com.facebook.presto.common.RuntimeMetric) Test(org.testng.annotations.Test)

Aggregations

RuntimeMetric (com.facebook.presto.common.RuntimeMetric)4 Duration (io.airlift.units.Duration)4 StatementStats (com.facebook.presto.client.StatementStats)2 DataSize (io.airlift.units.DataSize)2 Test (org.testng.annotations.Test)2 QueryStatusInfo (com.facebook.presto.client.QueryStatusInfo)1 AtomicInteger (java.util.concurrent.atomic.AtomicInteger)1
About Me
UseOf

Java Tutorials :

Simple Java RabbitMQ Example with Spring
Simple Springboot JPA Eclipeslink Example
Simple SpringBoot JPA Hibernate Example
Jackson JSON @JsonIgnore and @JsonIgnoreProperties Examples
Java Infinite recursion (StackOverflowError) Jackson solutions
Simple Java Jackson Serialize Objects to JSON and convert them back To Object
ElasticSearch 5 - Upload files using Java API in binary field Example
Java JAXB marshall unmarshall Examples from String and Stream
Java 8 forEach List and Map examples
ElasticSearch 5 Java API SearchRequestBuilder examples
Java ElasticSearch 5 examples with node index and mapping - running local
Convert String to int or Integer Java 8
Java 9 in action - JShell - Ubuntu
Java - removing invalid characters from a file name
Hello world!? or Hello Tutorial