Search in sources :

Example 11 with SerializationDelegate

use of org.apache.flink.runtime.plugable.SerializationDelegate in project flink by apache.

the class OutputEmitterTest method testBroadcast.

@Test
public void testBroadcast() {
    // Test for IntValue
    @SuppressWarnings({ "unchecked", "rawtypes" }) final TypeComparator<Record> intComp = new RecordComparatorFactory(new int[] { 0 }, new Class[] { IntValue.class }).createComparator();
    final ChannelSelector<SerializationDelegate<Record>> oe1 = new OutputEmitter<Record>(ShipStrategyType.BROADCAST, intComp);
    final SerializationDelegate<Record> delegate = new SerializationDelegate<Record>(new RecordSerializerFactory().getSerializer());
    int numChannels = 100;
    int numRecords = 50000;
    int[] hit = new int[numChannels];
    for (int i = 0; i < numRecords; i++) {
        IntValue k = new IntValue(i);
        Record rec = new Record(k);
        delegate.setInstance(rec);
        int[] chans = oe1.selectChannels(delegate, hit.length);
        for (int chan : chans) {
            hit[chan]++;
        }
    }
    for (int aHit : hit) {
        assertTrue(aHit + "", aHit == numRecords);
    }
    // Test for StringValue
    @SuppressWarnings({ "unchecked", "rawtypes" }) final TypeComparator<Record> stringComp = new RecordComparatorFactory(new int[] { 0 }, new Class[] { StringValue.class }).createComparator();
    final ChannelSelector<SerializationDelegate<Record>> oe2 = new OutputEmitter<Record>(ShipStrategyType.BROADCAST, stringComp);
    numChannels = 100;
    numRecords = 5000;
    hit = new int[numChannels];
    for (int i = 0; i < numRecords; i++) {
        StringValue k = new StringValue(i + "");
        Record rec = new Record(k);
        delegate.setInstance(rec);
        int[] chans = oe2.selectChannels(delegate, hit.length);
        for (int chan : chans) {
            hit[chan]++;
        }
    }
    for (int aHit : hit) {
        assertTrue(aHit + "", aHit == numRecords);
    }
}
Also used : RecordComparatorFactory(org.apache.flink.runtime.testutils.recordutils.RecordComparatorFactory) RecordSerializerFactory(org.apache.flink.runtime.testutils.recordutils.RecordSerializerFactory) SerializationDelegate(org.apache.flink.runtime.plugable.SerializationDelegate) OutputEmitter(org.apache.flink.runtime.operators.shipping.OutputEmitter) Record(org.apache.flink.types.Record) StringValue(org.apache.flink.types.StringValue) IntValue(org.apache.flink.types.IntValue) Test(org.junit.Test)

Aggregations

SerializationDelegate (org.apache.flink.runtime.plugable.SerializationDelegate)11 OutputEmitter (org.apache.flink.runtime.operators.shipping.OutputEmitter)8 RecordSerializerFactory (org.apache.flink.runtime.testutils.recordutils.RecordSerializerFactory)8 IntValue (org.apache.flink.types.IntValue)8 Record (org.apache.flink.types.Record)8 Test (org.junit.Test)8 RecordComparatorFactory (org.apache.flink.runtime.testutils.recordutils.RecordComparatorFactory)7 StringValue (org.apache.flink.types.StringValue)5 DoubleValue (org.apache.flink.types.DoubleValue)2 IOException (java.io.IOException)1 PipedInputStream (java.io.PipedInputStream)1 PipedOutputStream (java.io.PipedOutputStream)1 ArrayList (java.util.ArrayList)1 DataDistribution (org.apache.flink.api.common.distributions.DataDistribution)1 TypeSerializer (org.apache.flink.api.common.typeutils.TypeSerializer)1 IntSerializer (org.apache.flink.api.common.typeutils.base.IntSerializer)1 DataInputView (org.apache.flink.core.memory.DataInputView)1 DataInputViewStreamWrapper (org.apache.flink.core.memory.DataInputViewStreamWrapper)1 DataOutputView (org.apache.flink.core.memory.DataOutputView)1 DataOutputViewStreamWrapper (org.apache.flink.core.memory.DataOutputViewStreamWrapper)1