Search in sources :

Example 1 with CompressionCodec

use of com.yahoo.pulsar.common.compression.CompressionCodec in project pulsar by yahoo.

the class PersistentTopics method peekNthMessage.

@GET
@Path("/{property}/{cluster}/{namespace}/{destination}/subscription/{subName}/position/{messagePosition}")
@ApiOperation(value = "Peek nth message on a topic subscription.")
@ApiResponses(value = { @ApiResponse(code = 403, message = "Don't have admin permission"), @ApiResponse(code = 404, message = "Topic, subscription or the message position does not exist") })
public Response peekNthMessage(@PathParam("property") String property, @PathParam("cluster") String cluster, @PathParam("namespace") String namespace, @PathParam("destination") @Encoded String destination, @PathParam("subName") String subName, @PathParam("messagePosition") int messagePosition, @QueryParam("authoritative") @DefaultValue("false") boolean authoritative) {
    destination = decode(destination);
    DestinationName dn = DestinationName.get(domain(), property, cluster, namespace, destination);
    PartitionedTopicMetadata partitionMetadata = getPartitionedTopicMetadata(property, cluster, namespace, destination, authoritative);
    if (partitionMetadata.partitions > 0) {
        throw new RestException(Status.METHOD_NOT_ALLOWED, "Peek messages on a partitioned topic is not allowed");
    }
    validateAdminOperationOnDestination(dn, authoritative);
    PersistentTopic topic = getTopicReference(dn);
    PersistentReplicator repl = null;
    PersistentSubscription sub = null;
    Entry entry = null;
    if (subName.startsWith(topic.replicatorPrefix)) {
        repl = getReplicatorReference(subName, topic);
    } else {
        sub = getSubscriptionReference(subName, topic);
    }
    try {
        if (subName.startsWith(topic.replicatorPrefix)) {
            entry = repl.peekNthMessage(messagePosition).get();
        } else {
            entry = sub.peekNthMessage(messagePosition).get();
        }
        checkNotNull(entry);
        PositionImpl pos = (PositionImpl) entry.getPosition();
        ByteBuf metadataAndPayload = entry.getDataBuffer();
        // moves the readerIndex to the payload
        MessageMetadata metadata = Commands.parseMessageMetadata(metadataAndPayload);
        ResponseBuilder responseBuilder = Response.ok();
        responseBuilder.header("X-Pulsar-Message-ID", pos.toString());
        for (KeyValue keyValue : metadata.getPropertiesList()) {
            responseBuilder.header("X-Pulsar-PROPERTY-" + keyValue.getKey(), keyValue.getValue());
        }
        if (metadata.hasPublishTime()) {
            responseBuilder.header("X-Pulsar-publish-time", DATE_FORMAT.format(Instant.ofEpochMilli(metadata.getPublishTime())));
        }
        // Decode if needed
        CompressionCodec codec = CompressionCodecProvider.getCompressionCodec(metadata.getCompression());
        ByteBuf uncompressedPayload = codec.decode(metadataAndPayload, metadata.getUncompressedSize());
        // Copy into a heap buffer for output stream compatibility
        ByteBuf data = PooledByteBufAllocator.DEFAULT.heapBuffer(uncompressedPayload.readableBytes(), uncompressedPayload.readableBytes());
        data.writeBytes(uncompressedPayload);
        uncompressedPayload.release();
        StreamingOutput stream = new StreamingOutput() {

            @Override
            public void write(OutputStream output) throws IOException, WebApplicationException {
                output.write(data.array(), data.arrayOffset(), data.readableBytes());
                data.release();
            }
        };
        return responseBuilder.entity(stream).build();
    } catch (NullPointerException npe) {
        throw new RestException(Status.NOT_FOUND, "Message not found");
    } catch (Exception exception) {
        log.error("[{}] Failed to get message at position {} from {} {}", clientAppId(), messagePosition, dn, subName, exception);
        throw new RestException(exception);
    } finally {
        if (entry != null) {
            entry.release();
        }
    }
}
Also used : PersistentReplicator(com.yahoo.pulsar.broker.service.persistent.PersistentReplicator) KeyValue(com.yahoo.pulsar.common.api.proto.PulsarApi.KeyValue) PositionImpl(org.apache.bookkeeper.mledger.impl.PositionImpl) OutputStream(java.io.OutputStream) RestException(com.yahoo.pulsar.broker.web.RestException) StreamingOutput(javax.ws.rs.core.StreamingOutput) ByteBuf(io.netty.buffer.ByteBuf) PersistentSubscription(com.yahoo.pulsar.broker.service.persistent.PersistentSubscription) RestException(com.yahoo.pulsar.broker.web.RestException) TopicBusyException(com.yahoo.pulsar.broker.service.BrokerServiceException.TopicBusyException) WebApplicationException(javax.ws.rs.WebApplicationException) PulsarClientException(com.yahoo.pulsar.client.api.PulsarClientException) PreconditionFailedException(com.yahoo.pulsar.client.admin.PulsarAdminException.PreconditionFailedException) SubscriptionBusyException(com.yahoo.pulsar.broker.service.BrokerServiceException.SubscriptionBusyException) NotFoundException(com.yahoo.pulsar.client.admin.PulsarAdminException.NotFoundException) NotAllowedException(com.yahoo.pulsar.broker.service.BrokerServiceException.NotAllowedException) KeeperException(org.apache.zookeeper.KeeperException) IOException(java.io.IOException) Entry(org.apache.bookkeeper.mledger.Entry) MessageMetadata(com.yahoo.pulsar.common.api.proto.PulsarApi.MessageMetadata) PersistentTopic(com.yahoo.pulsar.broker.service.persistent.PersistentTopic) DestinationName(com.yahoo.pulsar.common.naming.DestinationName) CompressionCodec(com.yahoo.pulsar.common.compression.CompressionCodec) ResponseBuilder(javax.ws.rs.core.Response.ResponseBuilder) PartitionedTopicMetadata(com.yahoo.pulsar.common.partition.PartitionedTopicMetadata) Path(javax.ws.rs.Path) GET(javax.ws.rs.GET) ApiOperation(io.swagger.annotations.ApiOperation) ApiResponses(io.swagger.annotations.ApiResponses)

Example 2 with CompressionCodec

use of com.yahoo.pulsar.common.compression.CompressionCodec in project pulsar by yahoo.

the class ConsumerImpl method uncompressPayloadIfNeeded.

private ByteBuf uncompressPayloadIfNeeded(MessageIdData messageId, MessageMetadata msgMetadata, ByteBuf payload, ClientCnx currentCnx) {
    CompressionType compressionType = msgMetadata.getCompression();
    CompressionCodec codec = codecProvider.getCodec(compressionType);
    int uncompressedSize = msgMetadata.getUncompressedSize();
    if (uncompressedSize > PulsarDecoder.MaxMessageSize) {
        // Uncompressed size is itself corrupted since it cannot be bigger than the MaxMessageSize
        log.error("[{}][{}] Got corrupted uncompressed message size {} at {}", topic, subscription, uncompressedSize, messageId);
        discardCorruptedMessage(messageId, currentCnx, ValidationError.UncompressedSizeCorruption);
        return null;
    }
    try {
        ByteBuf uncompressedPayload = codec.decode(payload, uncompressedSize);
        return uncompressedPayload;
    } catch (IOException e) {
        log.error("[{}][{}] Failed to decompress message with {} at {}: {}", topic, subscription, compressionType, messageId, e.getMessage(), e);
        discardCorruptedMessage(messageId, currentCnx, ValidationError.DecompressionError);
        return null;
    }
}
Also used : CompressionCodec(com.yahoo.pulsar.common.compression.CompressionCodec) IOException(java.io.IOException) ByteBuf(io.netty.buffer.ByteBuf) CompressionType(com.yahoo.pulsar.common.api.proto.PulsarApi.CompressionType)

Aggregations

CompressionCodec (com.yahoo.pulsar.common.compression.CompressionCodec)2 ByteBuf (io.netty.buffer.ByteBuf)2 IOException (java.io.IOException)2 NotAllowedException (com.yahoo.pulsar.broker.service.BrokerServiceException.NotAllowedException)1 SubscriptionBusyException (com.yahoo.pulsar.broker.service.BrokerServiceException.SubscriptionBusyException)1 TopicBusyException (com.yahoo.pulsar.broker.service.BrokerServiceException.TopicBusyException)1 PersistentReplicator (com.yahoo.pulsar.broker.service.persistent.PersistentReplicator)1 PersistentSubscription (com.yahoo.pulsar.broker.service.persistent.PersistentSubscription)1 PersistentTopic (com.yahoo.pulsar.broker.service.persistent.PersistentTopic)1 RestException (com.yahoo.pulsar.broker.web.RestException)1 NotFoundException (com.yahoo.pulsar.client.admin.PulsarAdminException.NotFoundException)1 PreconditionFailedException (com.yahoo.pulsar.client.admin.PulsarAdminException.PreconditionFailedException)1 PulsarClientException (com.yahoo.pulsar.client.api.PulsarClientException)1 CompressionType (com.yahoo.pulsar.common.api.proto.PulsarApi.CompressionType)1 KeyValue (com.yahoo.pulsar.common.api.proto.PulsarApi.KeyValue)1 MessageMetadata (com.yahoo.pulsar.common.api.proto.PulsarApi.MessageMetadata)1 DestinationName (com.yahoo.pulsar.common.naming.DestinationName)1 PartitionedTopicMetadata (com.yahoo.pulsar.common.partition.PartitionedTopicMetadata)1 ApiOperation (io.swagger.annotations.ApiOperation)1 ApiResponses (io.swagger.annotations.ApiResponses)1