Examples with LocalExchangeSinkFactoryId io.prestosql.operator.exchange.LocalExchange.LocalExchangeSinkFactoryId used on opensource projects

Search in sources :

Example 1 with LocalExchangeSinkFactoryId

use of io.prestosql.operator.exchange.LocalExchange.LocalExchangeSinkFactoryId in project hetu-core by openlookeng.

the class TestLocalExchange method testBroadcast.

@Test(dataProvider = "executionStrategy")
public void testBroadcast(PipelineExecutionStrategy executionStrategy) {
    LocalExchangeFactory localExchangeFactory = new LocalExchangeFactory(FIXED_BROADCAST_DISTRIBUTION, 2, TYPES, ImmutableList.of(), Optional.empty(), executionStrategy, LOCAL_EXCHANGE_MAX_BUFFERED_BYTES);
    LocalExchangeSinkFactoryId localExchangeSinkFactoryId = localExchangeFactory.newSinkFactoryId();
    localExchangeFactory.noMoreSinkFactories();
    run(localExchangeFactory, executionStrategy, exchange -> {
        assertEquals(exchange.getBufferCount(), 2);
        assertExchangeTotalBufferedBytes(exchange, 0);
        LocalExchangeSinkFactory sinkFactory = exchange.getSinkFactory(localExchangeSinkFactoryId);
        String sinkAId = "sinkA";
        String sinkBId = "sinkB";
        LocalExchangeSink sinkA = sinkFactory.createSink(sinkAId);
        assertSinkCanWrite(sinkA);
        LocalExchangeSink sinkB = sinkFactory.createSink(sinkBId);
        assertSinkCanWrite(sinkB);
        sinkFactory.close();
        sinkFactory.noMoreSinkFactories();
        LocalExchangeSource sourceA = exchange.getSource(0);
        assertSource(sourceA, 0);
        LocalExchangeSource sourceB = exchange.getSource(1);
        assertSource(sourceB, 0);
        sinkA.addPage(createPage(0), sinkAId);
        assertSource(sourceA, 1);
        assertSource(sourceB, 1);
        assertExchangeTotalBufferedBytes(exchange, 1);
        sinkA.addPage(createPage(0), sinkAId);
        assertSource(sourceA, 2);
        assertSource(sourceB, 2);
        assertExchangeTotalBufferedBytes(exchange, 2);
        assertRemovePage(sourceA, createPage(0));
        assertSource(sourceA, 1);
        assertSource(sourceB, 2);
        assertExchangeTotalBufferedBytes(exchange, 2);
        assertRemovePage(sourceA, createPage(0));
        assertSource(sourceA, 0);
        assertSource(sourceB, 2);
        assertExchangeTotalBufferedBytes(exchange, 2);
        sinkA.finish();
        assertSinkFinished(sinkA);
        assertExchangeTotalBufferedBytes(exchange, 2);
        sinkB.addPage(createPage(0), sinkBId);
        assertSource(sourceA, 1);
        assertSource(sourceB, 3);
        assertExchangeTotalBufferedBytes(exchange, 3);
        sinkB.finish();
        assertSinkFinished(sinkB);
        assertSource(sourceA, 1);
        assertSource(sourceB, 3);
        assertExchangeTotalBufferedBytes(exchange, 3);
        assertRemovePage(sourceA, createPage(0));
        assertSourceFinished(sourceA);
        assertSource(sourceB, 3);
        assertExchangeTotalBufferedBytes(exchange, 3);
        assertRemovePage(sourceB, createPage(0));
        assertRemovePage(sourceB, createPage(0));
        assertSourceFinished(sourceA);
        assertSource(sourceB, 1);
        assertExchangeTotalBufferedBytes(exchange, 1);
        assertRemovePage(sourceB, createPage(0));
        assertSourceFinished(sourceA);
        assertSourceFinished(sourceB);
        assertExchangeTotalBufferedBytes(exchange, 0);
    });
}
Also used : LocalExchangeSinkFactory(io.prestosql.operator.exchange.LocalExchange.LocalExchangeSinkFactory) LocalExchangeFactory(io.prestosql.operator.exchange.LocalExchange.LocalExchangeFactory) LocalExchangeSinkFactoryId(io.prestosql.operator.exchange.LocalExchange.LocalExchangeSinkFactoryId) Test(org.testng.annotations.Test)

Example 2 with LocalExchangeSinkFactoryId

use of io.prestosql.operator.exchange.LocalExchange.LocalExchangeSinkFactoryId in project hetu-core by openlookeng.

the class TestLocalExchange method writeUnblockWhenAllReadersFinishAndPagesConsumed.

@Test(dataProvider = "executionStrategy")
public void writeUnblockWhenAllReadersFinishAndPagesConsumed(PipelineExecutionStrategy executionStrategy) {
    LocalExchangeFactory localExchangeFactory = new LocalExchangeFactory(FIXED_BROADCAST_DISTRIBUTION, 2, TYPES, ImmutableList.of(), Optional.empty(), executionStrategy, new DataSize(1, BYTE));
    LocalExchangeSinkFactoryId localExchangeSinkFactoryId = localExchangeFactory.newSinkFactoryId();
    localExchangeFactory.noMoreSinkFactories();
    run(localExchangeFactory, executionStrategy, exchange -> {
        assertEquals(exchange.getBufferCount(), 2);
        assertExchangeTotalBufferedBytes(exchange, 0);
        LocalExchangeSinkFactory sinkFactory = exchange.getSinkFactory(localExchangeSinkFactoryId);
        LocalExchangeSink sinkA = sinkFactory.createSink("sinkA");
        assertSinkCanWrite(sinkA);
        LocalExchangeSink sinkB = sinkFactory.createSink("sinkB");
        assertSinkCanWrite(sinkB);
        sinkFactory.close();
        sinkFactory.noMoreSinkFactories();
        LocalExchangeSource sourceA = exchange.getSource(0);
        assertSource(sourceA, 0);
        LocalExchangeSource sourceB = exchange.getSource(1);
        assertSource(sourceB, 0);
        sinkA.addPage(createPage(0), null);
        ListenableFuture<?> sinkAFuture = assertSinkWriteBlocked(sinkA);
        ListenableFuture<?> sinkBFuture = assertSinkWriteBlocked(sinkB);
        assertSource(sourceA, 1);
        assertSource(sourceB, 1);
        assertExchangeTotalBufferedBytes(exchange, 1);
        sourceA.finish();
        assertSource(sourceA, 1);
        assertRemovePage(sourceA, createPage(0));
        assertSourceFinished(sourceA);
        assertExchangeTotalBufferedBytes(exchange, 1);
        assertSource(sourceB, 1);
        assertSinkWriteBlocked(sinkA);
        assertSinkWriteBlocked(sinkB);
        sourceB.finish();
        assertSource(sourceB, 1);
        assertRemovePage(sourceB, createPage(0));
        assertSourceFinished(sourceB);
        assertExchangeTotalBufferedBytes(exchange, 0);
        assertTrue(sinkAFuture.isDone());
        assertTrue(sinkBFuture.isDone());
        assertSinkFinished(sinkA);
        assertSinkFinished(sinkB);
    });
}
Also used : LocalExchangeSinkFactory(io.prestosql.operator.exchange.LocalExchange.LocalExchangeSinkFactory) DataSize(io.airlift.units.DataSize) LocalExchangeFactory(io.prestosql.operator.exchange.LocalExchange.LocalExchangeFactory) LocalExchangeSinkFactoryId(io.prestosql.operator.exchange.LocalExchange.LocalExchangeSinkFactoryId) Test(org.testng.annotations.Test)

Example 3 with LocalExchangeSinkFactoryId

use of io.prestosql.operator.exchange.LocalExchange.LocalExchangeSinkFactoryId in project hetu-core by openlookeng.

the class TestLocalExchange method testRandom.

@Test(dataProvider = "executionStrategy")
public void testRandom(PipelineExecutionStrategy executionStrategy) {
    LocalExchangeFactory localExchangeFactory = new LocalExchangeFactory(FIXED_ARBITRARY_DISTRIBUTION, 2, TYPES, ImmutableList.of(), Optional.empty(), executionStrategy, LOCAL_EXCHANGE_MAX_BUFFERED_BYTES);
    LocalExchangeSinkFactoryId localExchangeSinkFactoryId = localExchangeFactory.newSinkFactoryId();
    localExchangeFactory.noMoreSinkFactories();
    run(localExchangeFactory, executionStrategy, exchange -> {
        assertEquals(exchange.getBufferCount(), 2);
        assertExchangeTotalBufferedBytes(exchange, 0);
        LocalExchangeSinkFactory sinkFactory = exchange.getSinkFactory(localExchangeSinkFactoryId);
        LocalExchangeSink sink = sinkFactory.createSink("");
        assertSinkCanWrite(sink);
        sinkFactory.close();
        sinkFactory.noMoreSinkFactories();
        LocalExchangeSource sourceA = exchange.getSource(0);
        assertSource(sourceA, 0);
        LocalExchangeSource sourceB = exchange.getSource(1);
        assertSource(sourceB, 0);
        for (int i = 0; i < 100; i++) {
            Page page = createPage(0);
            sink.addPage(page, null);
            assertExchangeTotalBufferedBytes(exchange, i + 1);
            LocalExchangeBufferInfo bufferInfoA = sourceA.getBufferInfo();
            LocalExchangeBufferInfo bufferInfoB = sourceB.getBufferInfo();
            assertEquals(bufferInfoA.getBufferedBytes() + bufferInfoB.getBufferedBytes(), retainedSizeOfPages(i + 1));
            assertEquals(bufferInfoA.getBufferedPages() + bufferInfoB.getBufferedPages(), i + 1);
        }
        // we should get ~50 pages per source, but we should get at least some pages in each buffer
        assertTrue(sourceA.getBufferInfo().getBufferedPages() > 0);
        assertTrue(sourceB.getBufferInfo().getBufferedPages() > 0);
        assertExchangeTotalBufferedBytes(exchange, 100);
    });
}
Also used : LocalExchangeSinkFactory(io.prestosql.operator.exchange.LocalExchange.LocalExchangeSinkFactory) MarkerPage(io.prestosql.spi.snapshot.MarkerPage) Page(io.prestosql.spi.Page) LocalExchangeFactory(io.prestosql.operator.exchange.LocalExchange.LocalExchangeFactory) LocalExchangeSinkFactoryId(io.prestosql.operator.exchange.LocalExchange.LocalExchangeSinkFactoryId) Test(org.testng.annotations.Test)

Example 4 with LocalExchangeSinkFactoryId

use of io.prestosql.operator.exchange.LocalExchange.LocalExchangeSinkFactoryId in project hetu-core by openlookeng.

the class TestLocalExchange method testMarkerBroadcast.

@Test(dataProvider = "markerExecutions")
public void testMarkerBroadcast(PartitioningHandle partitioningHandle, PipelineExecutionStrategy executionStrategy) {
    List<Integer> partitionChannels = partitioningHandle == FIXED_HASH_DISTRIBUTION ? ImmutableList.of(0) : ImmutableList.of();
    LocalExchangeFactory localExchangeFactory = new LocalExchangeFactory(partitioningHandle, 2, TYPES, partitionChannels, Optional.empty(), executionStrategy, LOCAL_EXCHANGE_MAX_BUFFERED_BYTES);
    LocalExchangeSinkFactoryId localExchangeSinkFactoryId = localExchangeFactory.newSinkFactoryId();
    localExchangeFactory.noMoreSinkFactories();
    run(localExchangeFactory, executionStrategy, exchange -> {
        assertEquals(exchange.getBufferCount(), 2);
        assertEquals(exchange.getBufferedBytes(), 0);
        final String sinkAId = "sinkA";
        final String sinkBId = "sinkB";
        LocalExchangeSinkFactory sinkFactory = exchange.getSinkFactory(localExchangeSinkFactoryId);
        LocalExchangeSink sinkA = sinkFactory.createSink(sinkAId);
        assertSinkCanWrite(sinkA);
        LocalExchangeSink sinkB = sinkFactory.createSink(sinkBId);
        assertSinkCanWrite(sinkB);
        sinkFactory.close();
        sinkFactory.noMoreSinkFactories();
        LocalExchangeSource sourceA = exchange.getSource(0);
        assertSource(sourceA, 0);
        LocalExchangeSource sourceB = exchange.getSource(1);
        assertSource(sourceB, 0);
        MarkerPage marker1 = MarkerPage.snapshotPage(1);
        MarkerPage marker2 = MarkerPage.snapshotPage(2);
        MarkerPage resume1 = MarkerPage.resumePage(1);
        long markerSize = marker1.getRetainedSizeInBytes();
        sinkA.addPage(marker1, sinkAId);
        assertSource(sourceA, 1);
        assertSource(sourceB, 1);
        assertEquals(exchange.getBufferedBytes(), markerSize * 2);
        sinkB.addPage(marker2, sinkBId);
        assertSource(sourceA, 2);
        assertSource(sourceB, 2);
        assertEquals(exchange.getBufferedBytes(), markerSize * 4);
        assertRemovePage(sourceA, marker1, sinkAId);
        assertSource(sourceA, 1);
        assertSource(sourceB, 2);
        assertEquals(exchange.getBufferedBytes(), markerSize * 3);
        assertRemovePage(sourceA, marker2, sinkBId);
        assertSource(sourceA, 0);
        assertSource(sourceB, 2);
        assertEquals(exchange.getBufferedBytes(), markerSize * 2);
        sinkA.finish();
        assertSinkFinished(sinkA);
        assertEquals(exchange.getBufferedBytes(), markerSize * 2);
        sinkB.addPage(resume1, sinkBId);
        assertSource(sourceA, 1);
        assertSource(sourceB, 3);
        assertEquals(exchange.getBufferedBytes(), markerSize * 4);
        sinkB.finish();
        assertSinkFinished(sinkB);
        assertSource(sourceA, 1);
        assertSource(sourceB, 3);
        assertEquals(exchange.getBufferedBytes(), markerSize * 4);
        assertRemovePage(sourceA, resume1, sinkBId);
        assertSourceFinished(sourceA);
        assertSource(sourceB, 3);
        assertEquals(exchange.getBufferedBytes(), markerSize * 3);
        assertRemovePage(sourceB, marker1, sinkAId);
        assertRemovePage(sourceB, marker2, sinkBId);
        assertSourceFinished(sourceA);
        assertSource(sourceB, 1);
        assertEquals(exchange.getBufferedBytes(), markerSize * 1);
        assertRemovePage(sourceB, resume1, sinkBId);
        assertSourceFinished(sourceA);
        assertSourceFinished(sourceB);
        assertEquals(exchange.getBufferedBytes(), markerSize * 0);
    });
}
Also used : LocalExchangeSinkFactory(io.prestosql.operator.exchange.LocalExchange.LocalExchangeSinkFactory) MarkerPage(io.prestosql.spi.snapshot.MarkerPage) LocalExchangeFactory(io.prestosql.operator.exchange.LocalExchange.LocalExchangeFactory) LocalExchangeSinkFactoryId(io.prestosql.operator.exchange.LocalExchange.LocalExchangeSinkFactoryId) Test(org.testng.annotations.Test)

Example 5 with LocalExchangeSinkFactoryId

use of io.prestosql.operator.exchange.LocalExchange.LocalExchangeSinkFactoryId in project hetu-core by openlookeng.

the class TestLocalExchange method testPassthrough.

@Test(dataProvider = "executionStrategy")
public void testPassthrough(PipelineExecutionStrategy executionStrategy) {
    LocalExchangeFactory localExchangeFactory = new LocalExchangeFactory(FIXED_PASSTHROUGH_DISTRIBUTION, 2, TYPES, ImmutableList.of(), Optional.empty(), executionStrategy, new DataSize(retainedSizeOfPages(1), BYTE));
    LocalExchangeSinkFactoryId localExchangeSinkFactoryId = localExchangeFactory.newSinkFactoryId();
    localExchangeFactory.noMoreSinkFactories();
    run(localExchangeFactory, executionStrategy, exchange -> {
        assertEquals(exchange.getBufferCount(), 2);
        assertExchangeTotalBufferedBytes(exchange, 0);
        LocalExchangeSinkFactory sinkFactory = exchange.getSinkFactory(localExchangeSinkFactoryId);
        LocalExchangeSink sinkA = sinkFactory.createSink("sinkA");
        LocalExchangeSink sinkB = sinkFactory.createSink("sinkB");
        assertSinkCanWrite(sinkA);
        assertSinkCanWrite(sinkB);
        sinkFactory.close();
        sinkFactory.noMoreSinkFactories();
        LocalExchangeSource sourceA = exchange.getSource(0);
        assertSource(sourceA, 0);
        LocalExchangeSource sourceB = exchange.getSource(1);
        assertSource(sourceB, 0);
        sinkA.addPage(createPage(0), null);
        assertSource(sourceA, 1);
        assertSource(sourceB, 0);
        assertSinkWriteBlocked(sinkA);
        assertSinkCanWrite(sinkB);
        sinkB.addPage(createPage(1), null);
        assertSource(sourceA, 1);
        assertSource(sourceB, 1);
        assertSinkWriteBlocked(sinkA);
        assertExchangeTotalBufferedBytes(exchange, 2);
        assertRemovePage(sourceA, createPage(0));
        assertSource(sourceA, 0);
        assertSinkCanWrite(sinkA);
        assertSinkWriteBlocked(sinkB);
        assertExchangeTotalBufferedBytes(exchange, 1);
        sinkA.finish();
        assertSinkFinished(sinkA);
        assertSource(sourceB, 1);
        sourceA.finish();
        sourceB.finish();
        assertRemovePage(sourceB, createPage(1));
        assertSourceFinished(sourceA);
        assertSourceFinished(sourceB);
        assertSinkFinished(sinkB);
        assertExchangeTotalBufferedBytes(exchange, 0);
    });
}
Also used : LocalExchangeSinkFactory(io.prestosql.operator.exchange.LocalExchange.LocalExchangeSinkFactory) DataSize(io.airlift.units.DataSize) LocalExchangeFactory(io.prestosql.operator.exchange.LocalExchange.LocalExchangeFactory) LocalExchangeSinkFactoryId(io.prestosql.operator.exchange.LocalExchange.LocalExchangeSinkFactoryId) Test(org.testng.annotations.Test)

Aggregations

LocalExchangeFactory (io.prestosql.operator.exchange.LocalExchange.LocalExchangeFactory)9 LocalExchangeSinkFactoryId (io.prestosql.operator.exchange.LocalExchange.LocalExchangeSinkFactoryId)9 LocalExchangeSinkFactory (io.prestosql.operator.exchange.LocalExchange.LocalExchangeSinkFactory)8 Test (org.testng.annotations.Test)8 DataSize (io.airlift.units.DataSize)4 MarkerPage (io.prestosql.spi.snapshot.MarkerPage)2 HashBuilderOperatorFactory (io.prestosql.operator.HashBuilderOperator.HashBuilderOperatorFactory)1 ValuesOperatorFactory (io.prestosql.operator.ValuesOperator.ValuesOperatorFactory)1 LocalExchangeSinkOperatorFactory (io.prestosql.operator.exchange.LocalExchangeSinkOperator.LocalExchangeSinkOperatorFactory)1 LocalExchangeSourceOperatorFactory (io.prestosql.operator.exchange.LocalExchangeSourceOperator.LocalExchangeSourceOperatorFactory)1 Page (io.prestosql.spi.Page)1 PlanNodeId (io.prestosql.spi.plan.PlanNodeId)1 Type (io.prestosql.spi.type.Type)1 JoinFilterFunctionFactory (io.prestosql.sql.gen.JoinFilterFunctionCompiler.JoinFilterFunctionFactory)1 OptionalInt (java.util.OptionalInt)1
About Me
UseOf

Java Tutorials :

Simple Java RabbitMQ Example with Spring
Simple Springboot JPA Eclipeslink Example
Simple SpringBoot JPA Hibernate Example
Jackson JSON @JsonIgnore and @JsonIgnoreProperties Examples
Java Infinite recursion (StackOverflowError) Jackson solutions
Simple Java Jackson Serialize Objects to JSON and convert them back To Object
ElasticSearch 5 - Upload files using Java API in binary field Example
Java JAXB marshall unmarshall Examples from String and Stream
Java 8 forEach List and Map examples
ElasticSearch 5 Java API SearchRequestBuilder examples
Java ElasticSearch 5 examples with node index and mapping - running local
Convert String to int or Integer Java 8
Java 9 in action - JShell - Ubuntu
Java - removing invalid characters from a file name
Hello world!? or Hello Tutorial