Search in sources :

Example 1 with Decoder

use of co.cask.cdap.common.io.Decoder in project cdap by caskdata.

the class QueueToStreamConsumer method poll.

@Override
public DequeueResult<StreamEvent> poll(int maxEvents, long timeout, TimeUnit timeoutUnit) throws IOException, InterruptedException {
    final DequeueResult<byte[]> result = consumer.dequeue(maxEvents);
    // Decode byte array into stream event
    ImmutableList.Builder<StreamEvent> builder = ImmutableList.builder();
    for (byte[] content : result) {
        try {
            builder.add(STREAM_EVENT_CODEC.decodePayload(content));
        } catch (Throwable t) {
            // If failed to decode, it maybe using old (pre 2.1) stream codec. Try to decode with old one.
            ByteBuffer buffer = ByteBuffer.wrap(content);
            SchemaHash schemaHash = new SchemaHash(buffer);
            Preconditions.checkArgument(schemaHash.equals(StreamEventDataCodec.STREAM_DATA_SCHEMA.getSchemaHash()), "Schema from payload not matching with StreamEventData schema.");
            Decoder decoder = new BinaryDecoder(new ByteBufferInputStream(buffer));
            // In old schema, timestamp is not recorded.
            builder.add(new StreamEvent(StreamEventDataCodec.decode(decoder), 0));
        }
    }
    final List<StreamEvent> events = builder.build();
    return new DequeueResult<StreamEvent>() {

        @Override
        public boolean isEmpty() {
            return events.isEmpty();
        }

        @Override
        public void reclaim() {
            result.reclaim();
        }

        @Override
        public int size() {
            return events.size();
        }

        @Override
        public Iterator<StreamEvent> iterator() {
            return events.iterator();
        }
    };
}
Also used : SchemaHash(co.cask.cdap.api.data.schema.SchemaHash) ImmutableList(com.google.common.collect.ImmutableList) StreamEvent(co.cask.cdap.api.flow.flowlet.StreamEvent) ByteBufferInputStream(co.cask.common.io.ByteBufferInputStream) BinaryDecoder(co.cask.cdap.common.io.BinaryDecoder) Decoder(co.cask.cdap.common.io.Decoder) ByteBuffer(java.nio.ByteBuffer) BinaryDecoder(co.cask.cdap.common.io.BinaryDecoder) DequeueResult(co.cask.cdap.data2.queue.DequeueResult)

Example 2 with Decoder

use of co.cask.cdap.common.io.Decoder in project cdap by caskdata.

the class KeyIdentifierCodec method decode.

@Override
public KeyIdentifier decode(byte[] data) throws IOException {
    ByteArrayInputStream bis = new ByteArrayInputStream(data);
    Decoder decoder = new BinaryDecoder(bis);
    DatumReader<KeyIdentifier> reader = readerFactory.create(KEY_IDENTIFIER_TYPE, KeyIdentifier.Schemas.getCurrentSchema());
    int readVersion = decoder.readInt();
    Schema readSchema = KeyIdentifier.Schemas.getSchemaVersion(readVersion);
    if (readSchema == null) {
        throw new IOException("Unknown schema version for KeyIdentifier: " + readVersion);
    }
    return reader.read(decoder, readSchema);
}
Also used : ByteArrayInputStream(java.io.ByteArrayInputStream) Schema(co.cask.cdap.api.data.schema.Schema) IOException(java.io.IOException) BinaryDecoder(co.cask.cdap.common.io.BinaryDecoder) Decoder(co.cask.cdap.common.io.Decoder) BinaryDecoder(co.cask.cdap.common.io.BinaryDecoder)

Example 3 with Decoder

use of co.cask.cdap.common.io.Decoder in project cdap by caskdata.

the class StreamEventCodec method decodePayload.

/**
 * Reverse of {@link #encodePayload(StreamEvent)}.
 *
 * @param payload The byte array containing the queue payload.
 * @return A {@link StreamEvent} reconstructed from payload.
 */
public StreamEvent decodePayload(byte[] payload) {
    ByteBuffer buffer = ByteBuffer.wrap(payload);
    SchemaHash schemaHash = new SchemaHash(buffer);
    Preconditions.checkArgument(schemaHash.equals(STREAM_EVENT_SCHEMA.getSchemaHash()), "Schema from payload not matching StreamEvent schema.");
    Decoder decoder = new BinaryDecoder(new ByteBufferInputStream(buffer));
    try {
        StreamEventData data = StreamEventDataCodec.decode(decoder);
        // Read the timestamp
        long timestamp = decoder.readLong();
        return new StreamEvent(data, timestamp);
    } catch (IOException e) {
        // It should never happens, otherwise something very wrong.
        throw Throwables.propagate(e);
    }
}
Also used : SchemaHash(co.cask.cdap.api.data.schema.SchemaHash) StreamEvent(co.cask.cdap.api.flow.flowlet.StreamEvent) ByteBufferInputStream(co.cask.common.io.ByteBufferInputStream) IOException(java.io.IOException) BinaryDecoder(co.cask.cdap.common.io.BinaryDecoder) Decoder(co.cask.cdap.common.io.Decoder) StreamEventData(co.cask.cdap.api.stream.StreamEventData) ByteBuffer(java.nio.ByteBuffer) BinaryDecoder(co.cask.cdap.common.io.BinaryDecoder)

Example 4 with Decoder

use of co.cask.cdap.common.io.Decoder in project cdap by caskdata.

the class AccessTokenCodec method decode.

@Override
public AccessToken decode(byte[] data) throws IOException {
    ByteArrayInputStream bis = new ByteArrayInputStream(data);
    Decoder decoder = new BinaryDecoder(bis);
    DatumReader<AccessToken> reader = readerFactory.create(ACCESS_TOKEN_TYPE, AccessToken.Schemas.getCurrentSchema());
    int readVersion = decoder.readInt();
    Schema readSchema = AccessToken.Schemas.getSchemaVersion(readVersion);
    if (readSchema == null) {
        throw new IOException("Unknown schema version for AccessToken: " + readVersion);
    }
    return reader.read(decoder, readSchema);
}
Also used : ByteArrayInputStream(java.io.ByteArrayInputStream) Schema(co.cask.cdap.api.data.schema.Schema) IOException(java.io.IOException) BinaryDecoder(co.cask.cdap.common.io.BinaryDecoder) Decoder(co.cask.cdap.common.io.Decoder) BinaryDecoder(co.cask.cdap.common.io.BinaryDecoder)

Example 5 with Decoder

use of co.cask.cdap.common.io.Decoder in project cdap by caskdata.

the class AccessTokenIdentifierCodec method decode.

@Override
public AccessTokenIdentifier decode(byte[] data) throws IOException {
    ByteArrayInputStream bis = new ByteArrayInputStream(data);
    Decoder decoder = new BinaryDecoder(bis);
    DatumReader<AccessTokenIdentifier> reader = readerFactory.create(ACCESS_TOKEN_IDENTIFIER_TYPE, AccessTokenIdentifier.Schemas.getCurrentSchema());
    int readVersion = decoder.readInt();
    Schema readSchema = AccessTokenIdentifier.Schemas.getSchemaVersion(readVersion);
    if (readSchema == null) {
        throw new IOException("Unknown schema version for AccessTokenIdentifier: " + readVersion);
    }
    return reader.read(decoder, readSchema);
}
Also used : ByteArrayInputStream(java.io.ByteArrayInputStream) Schema(co.cask.cdap.api.data.schema.Schema) IOException(java.io.IOException) BinaryDecoder(co.cask.cdap.common.io.BinaryDecoder) Decoder(co.cask.cdap.common.io.Decoder) BinaryDecoder(co.cask.cdap.common.io.BinaryDecoder)

Aggregations

BinaryDecoder (co.cask.cdap.common.io.BinaryDecoder)6 Decoder (co.cask.cdap.common.io.Decoder)6 IOException (java.io.IOException)4 Schema (co.cask.cdap.api.data.schema.Schema)3 ByteArrayInputStream (java.io.ByteArrayInputStream)3 ByteBuffer (java.nio.ByteBuffer)3 SchemaHash (co.cask.cdap.api.data.schema.SchemaHash)2 StreamEvent (co.cask.cdap.api.flow.flowlet.StreamEvent)2 ByteBufferInputStream (co.cask.common.io.ByteBufferInputStream)2 StreamEventData (co.cask.cdap.api.stream.StreamEventData)1 BinaryEncoder (co.cask.cdap.common.io.BinaryEncoder)1 Encoder (co.cask.cdap.common.io.Encoder)1 DequeueResult (co.cask.cdap.data2.queue.DequeueResult)1 ImmutableList (com.google.common.collect.ImmutableList)1 PipedInputStream (java.io.PipedInputStream)1 PipedOutputStream (java.io.PipedOutputStream)1 IntBuffer (java.nio.IntBuffer)1 Test (org.junit.Test)1