Search in sources :

Example 31 with BinaryEncoder

use of co.cask.cdap.common.io.BinaryEncoder in project cdap by caskdata.

the class QueueEntry method serializeHashKeys.

public static byte[] serializeHashKeys(Map<String, Integer> hashKeys) throws IOException {
    // many entries will have no hash keys. Reuse a static value for that
    if (hashKeys == null || hashKeys.isEmpty()) {
        return SERIALIZED_EMPTY_HASH_KEYS;
    }
    ByteArrayOutputStream bos = new ByteArrayOutputStream();
    Encoder encoder = new BinaryEncoder(bos);
    encoder.writeInt(hashKeys.size());
    for (Map.Entry<String, Integer> entry : hashKeys.entrySet()) {
        encoder.writeString(entry.getKey()).writeInt(entry.getValue());
    }
    // per Avro spec, end with a (block of length) zero
    encoder.writeInt(0);
    return bos.toByteArray();
}
Also used : BinaryEncoder(co.cask.cdap.common.io.BinaryEncoder) BinaryEncoder(co.cask.cdap.common.io.BinaryEncoder) Encoder(co.cask.cdap.common.io.Encoder) ByteArrayOutputStream(java.io.ByteArrayOutputStream) ImmutableMap(com.google.common.collect.ImmutableMap) Map(java.util.Map)

Example 32 with BinaryEncoder

use of co.cask.cdap.common.io.BinaryEncoder in project cdap by caskdata.

the class DatumOutputEmitter method emit.

@Override
public void emit(T data, Map<String, Object> partitions) {
    try {
        ByteArrayOutputStream output = new ByteArrayOutputStream();
        output.write(schemaHash);
        writer.encode(data, new BinaryEncoder(output));
        producerSupplier.get().enqueue(new QueueEntry(Maps.transformValues(partitions, PARTITION_MAP_TRANSFORMER), output.toByteArray()));
    } catch (IOException e) {
        throw Throwables.propagate(e);
    }
}
Also used : BinaryEncoder(co.cask.cdap.common.io.BinaryEncoder) ByteArrayOutputStream(java.io.ByteArrayOutputStream) IOException(java.io.IOException) QueueEntry(co.cask.cdap.data2.queue.QueueEntry)

Aggregations

BinaryEncoder (co.cask.cdap.common.io.BinaryEncoder)32 Test (org.junit.Test)24 BinaryDecoder (co.cask.cdap.common.io.BinaryDecoder)22 PipedInputStream (java.io.PipedInputStream)21 PipedOutputStream (java.io.PipedOutputStream)21 ReflectionDatumReader (co.cask.cdap.internal.io.ReflectionDatumReader)18 TypeToken (com.google.common.reflect.TypeToken)18 ByteArrayOutputStream (java.io.ByteArrayOutputStream)11 Encoder (co.cask.cdap.common.io.Encoder)7 Schema (co.cask.cdap.api.data.schema.Schema)6 ReflectionSchemaGenerator (co.cask.cdap.internal.io.ReflectionSchemaGenerator)6 ImmutableList (com.google.common.collect.ImmutableList)5 List (java.util.List)5 IOException (java.io.IOException)4 ReflectionDatumWriter (co.cask.cdap.internal.io.ReflectionDatumWriter)3 ImmutableMap (com.google.common.collect.ImmutableMap)3 Map (java.util.Map)3 ByteBuffer (java.nio.ByteBuffer)2 DataSetException (co.cask.cdap.api.dataset.DataSetException)1 StreamEvent (co.cask.cdap.api.flow.flowlet.StreamEvent)1