Examples with BeamTableStatistics org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics used on opensource projects

Example 1 with BeamTableStatistics

use of org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics in project beam by apache.

the class BigQueryRowCountIT method testEmptyTable.

@Test
public void testEmptyTable() {
    BigQueryTableProvider provider = new BigQueryTableProvider();
    Table table = getTable("testTable", bigQuery.tableSpec());
    BeamSqlTable sqlTable = provider.buildBeamSqlTable(table);
    BeamTableStatistics size = sqlTable.getTableStatistics(TestPipeline.testingPipelineOptions());
    assertNotNull(size);
    assertEquals(0d, size.getRowCount(), 0.1);
}

Example 2 with BeamTableStatistics

use of org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics in project beam by apache.

the class BeamKafkaTableStatisticsTest method testOrderedArrivalSinglePartitionRate.

@Test
public void testOrderedArrivalSinglePartitionRate() {
    KafkaTestTable table = testTable(1);
    for (int i = 0; i < 100; i++) {
        table.addRecord(createKafkaTestRecord("k" + i, i, 500L * i));
    }
    BeamTableStatistics stats = table.getTableStatistics(null);
    Assert.assertEquals(2d, stats.getRate(), 0.001);
}

Example 3 with BeamTableStatistics

use of org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics in project beam by apache.

the class BeamKafkaTableStatisticsTest method testOnePartitionAheadRate.

@Test
public void testOnePartitionAheadRate() {
    KafkaTestTable table = testTable(3);
    for (int i = 0; i < 100; i++) {
        table.addRecord(createKafkaTestRecord("1", i, 1000L * i));
        table.addRecord(createKafkaTestRecord("2", i, 500L * i));
    }
    table.setNumberOfRecordsForRate(20);
    BeamTableStatistics stats = table.getTableStatistics(null);
    Assert.assertEquals(1d, stats.getRate(), 0.001);
}

Example 4 with BeamTableStatistics

use of org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics in project beam by apache.

the class BeamKafkaTableStatisticsTest method allTheRecordsSameTimeRate.

@Test
public void allTheRecordsSameTimeRate() {
    KafkaTestTable table = testTable(3);
    for (int i = 0; i < 100; i++) {
        table.addRecord(createKafkaTestRecord("key" + i, i, 1000L));
    }
    BeamTableStatistics stats = table.getTableStatistics(null);
    Assert.assertTrue(stats.isUnknown());
}

Example 5 with BeamTableStatistics

use of org.apache.beam.sdk.extensions.sql.impl.BeamTableStatistics in project beam by apache.

the class BeamIOSourceRel method estimateNodeStats.

@Override
public NodeStats estimateNodeStats(BeamRelMetadataQuery mq) {
    BeamTableStatistics rowCountStatistics = calciteTable.getStatistic();
    double window = (beamTable.isBounded() == PCollection.IsBounded.BOUNDED) ? rowCountStatistics.getRowCount() : CONSTANT_WINDOW_SIZE;
    return NodeStats.create(rowCountStatistics.getRowCount(), rowCountStatistics.getRate(), window);
}