Search in sources :

Example 16 with RemotingCommandException

use of org.apache.rocketmq.remoting.exception.RemotingCommandException in project rocketmq by apache.

the class ClientManageProcessor method checkClientConfig.

public RemotingCommand checkClientConfig(ChannelHandlerContext ctx, RemotingCommand request) throws RemotingCommandException {
    final RemotingCommand response = RemotingCommand.createResponseCommand(null);
    CheckClientRequestBody requestBody = CheckClientRequestBody.decode(request.getBody(), CheckClientRequestBody.class);
    if (requestBody != null && requestBody.getSubscriptionData() != null) {
        SubscriptionData subscriptionData = requestBody.getSubscriptionData();
        if (ExpressionType.isTagType(subscriptionData.getExpressionType())) {
            response.setCode(ResponseCode.SUCCESS);
            response.setRemark(null);
            return response;
        }
        if (!this.brokerController.getBrokerConfig().isEnablePropertyFilter()) {
            response.setCode(ResponseCode.SYSTEM_ERROR);
            response.setRemark("The broker does not support consumer to filter message by " + subscriptionData.getExpressionType());
            return response;
        }
        try {
            FilterFactory.INSTANCE.get(subscriptionData.getExpressionType()).compile(subscriptionData.getSubString());
        } catch (Exception e) {
            log.warn("Client {}@{} filter message, but failed to compile expression! sub={}, error={}", requestBody.getClientId(), requestBody.getGroup(), requestBody.getSubscriptionData(), e.getMessage());
            response.setCode(ResponseCode.SUBSCRIPTION_PARSE_FAILED);
            response.setRemark(e.getMessage());
            return response;
        }
    }
    response.setCode(ResponseCode.SUCCESS);
    response.setRemark(null);
    return response;
}
Also used : RemotingCommand(org.apache.rocketmq.remoting.protocol.RemotingCommand) SubscriptionData(org.apache.rocketmq.common.protocol.heartbeat.SubscriptionData) CheckClientRequestBody(org.apache.rocketmq.common.protocol.body.CheckClientRequestBody) RemotingCommandException(org.apache.rocketmq.remoting.exception.RemotingCommandException)

Example 17 with RemotingCommandException

use of org.apache.rocketmq.remoting.exception.RemotingCommandException in project rocketmq by apache.

the class QueryMessageProcessor method viewMessageById.

public RemotingCommand viewMessageById(ChannelHandlerContext ctx, RemotingCommand request) throws RemotingCommandException {
    final RemotingCommand response = RemotingCommand.createResponseCommand(null);
    final ViewMessageRequestHeader requestHeader = (ViewMessageRequestHeader) request.decodeCommandCustomHeader(ViewMessageRequestHeader.class);
    response.setOpaque(request.getOpaque());
    final SelectMappedBufferResult selectMappedBufferResult = this.brokerController.getMessageStore().selectOneMessageByOffset(requestHeader.getOffset());
    if (selectMappedBufferResult != null) {
        response.setCode(ResponseCode.SUCCESS);
        response.setRemark(null);
        try {
            FileRegion fileRegion = new OneMessageTransfer(response.encodeHeader(selectMappedBufferResult.getSize()), selectMappedBufferResult);
            ctx.channel().writeAndFlush(fileRegion).addListener(new ChannelFutureListener() {

                @Override
                public void operationComplete(ChannelFuture future) throws Exception {
                    selectMappedBufferResult.release();
                    if (!future.isSuccess()) {
                        log.error("Transfer one message from page cache failed, ", future.cause());
                    }
                }
            });
        } catch (Throwable e) {
            log.error("", e);
            selectMappedBufferResult.release();
        }
        return null;
    } else {
        response.setCode(ResponseCode.SYSTEM_ERROR);
        response.setRemark("can not find message by the offset, " + requestHeader.getOffset());
    }
    return response;
}
Also used : RemotingCommand(org.apache.rocketmq.remoting.protocol.RemotingCommand) ViewMessageRequestHeader(org.apache.rocketmq.common.protocol.header.ViewMessageRequestHeader) ChannelFuture(io.netty.channel.ChannelFuture) OneMessageTransfer(org.apache.rocketmq.broker.pagecache.OneMessageTransfer) FileRegion(io.netty.channel.FileRegion) SelectMappedBufferResult(org.apache.rocketmq.store.SelectMappedBufferResult) ChannelFutureListener(io.netty.channel.ChannelFutureListener) RemotingCommandException(org.apache.rocketmq.remoting.exception.RemotingCommandException)

Example 18 with RemotingCommandException

use of org.apache.rocketmq.remoting.exception.RemotingCommandException in project rocketmq by apache.

the class RemotingCommand method decodeCommandCustomHeader.

public CommandCustomHeader decodeCommandCustomHeader(Class<? extends CommandCustomHeader> classHeader) throws RemotingCommandException {
    CommandCustomHeader objectHeader;
    try {
        objectHeader = classHeader.newInstance();
    } catch (InstantiationException e) {
        return null;
    } catch (IllegalAccessException e) {
        return null;
    }
    if (this.extFields != null) {
        Field[] fields = getClazzFields(classHeader);
        for (Field field : fields) {
            if (!Modifier.isStatic(field.getModifiers())) {
                String fieldName = field.getName();
                if (!fieldName.startsWith("this")) {
                    try {
                        String value = this.extFields.get(fieldName);
                        if (null == value) {
                            if (!isFieldNullable(field)) {
                                throw new RemotingCommandException("the custom field <" + fieldName + "> is null");
                            }
                            continue;
                        }
                        field.setAccessible(true);
                        String type = getCanonicalName(field.getType());
                        Object valueParsed;
                        if (type.equals(STRING_CANONICAL_NAME)) {
                            valueParsed = value;
                        } else if (type.equals(INTEGER_CANONICAL_NAME_1) || type.equals(INTEGER_CANONICAL_NAME_2)) {
                            valueParsed = Integer.parseInt(value);
                        } else if (type.equals(LONG_CANONICAL_NAME_1) || type.equals(LONG_CANONICAL_NAME_2)) {
                            valueParsed = Long.parseLong(value);
                        } else if (type.equals(BOOLEAN_CANONICAL_NAME_1) || type.equals(BOOLEAN_CANONICAL_NAME_2)) {
                            valueParsed = Boolean.parseBoolean(value);
                        } else if (type.equals(DOUBLE_CANONICAL_NAME_1) || type.equals(DOUBLE_CANONICAL_NAME_2)) {
                            valueParsed = Double.parseDouble(value);
                        } else {
                            throw new RemotingCommandException("the custom field <" + fieldName + "> type is not supported");
                        }
                        field.set(objectHeader, valueParsed);
                    } catch (Throwable e) {
                        log.error("Failed field [{}] decoding", fieldName, e);
                    }
                }
            }
        }
        objectHeader.checkFields();
    }
    return objectHeader;
}
Also used : JSONField(com.alibaba.fastjson.annotation.JSONField) Field(java.lang.reflect.Field) CommandCustomHeader(org.apache.rocketmq.remoting.CommandCustomHeader) RemotingCommandException(org.apache.rocketmq.remoting.exception.RemotingCommandException)

Example 19 with RemotingCommandException

use of org.apache.rocketmq.remoting.exception.RemotingCommandException in project rocketmq-rocketmq-all-4.1.0-incubating by lirenzuo.

the class MQAdminImpl method queryMessage.

/**
 * 通过断点查看rocketmq-console项目查询MessageId我们可以发现,只有offsetMsgId才是真正的这种规则的key,而msgId其实仅仅知识生成的UNIQ_KEY
 * 这也是为什么用MessageDecoder.decodeMessageId端口号超限问题的原因
 */
protected QueryResult queryMessage(String topic, String key, int maxNum, long begin, long end, boolean isUniqKey) throws MQClientException, InterruptedException {
    TopicRouteData topicRouteData = this.mQClientFactory.getAnExistTopicRouteData(topic);
    if (null == topicRouteData) {
        this.mQClientFactory.updateTopicRouteInfoFromNameServer(topic);
        topicRouteData = this.mQClientFactory.getAnExistTopicRouteData(topic);
    }
    if (topicRouteData != null) {
        List<String> brokerAddrs = new LinkedList<String>();
        for (BrokerData brokerData : topicRouteData.getBrokerDatas()) {
            String addr = brokerData.selectBrokerAddr();
            if (addr != null) {
                brokerAddrs.add(addr);
            }
        }
        if (!brokerAddrs.isEmpty()) {
            final CountDownLatch countDownLatch = new CountDownLatch(brokerAddrs.size());
            final List<QueryResult> queryResultList = new LinkedList<QueryResult>();
            final ReadWriteLock lock = new ReentrantReadWriteLock(false);
            for (String addr : brokerAddrs) {
                try {
                    // 构建查询请求头信息
                    QueryMessageRequestHeader requestHeader = new QueryMessageRequestHeader();
                    requestHeader.setTopic(topic);
                    requestHeader.setKey(key);
                    requestHeader.setMaxNum(maxNum);
                    requestHeader.setBeginTimestamp(begin);
                    requestHeader.setEndTimestamp(end);
                    this.mQClientFactory.getMQClientAPIImpl().queryMessage(addr, requestHeader, timeoutMillis * 3, new InvokeCallback() {

                        @Override
                        public void operationComplete(ResponseFuture responseFuture) {
                            try {
                                RemotingCommand response = responseFuture.getResponseCommand();
                                if (response != null) {
                                    switch(response.getCode()) {
                                        case ResponseCode.SUCCESS:
                                            {
                                                QueryMessageResponseHeader responseHeader = null;
                                                try {
                                                    responseHeader = (QueryMessageResponseHeader) response.decodeCommandCustomHeader(QueryMessageResponseHeader.class);
                                                } catch (RemotingCommandException e) {
                                                    log.error("decodeCommandCustomHeader exception", e);
                                                    return;
                                                }
                                                List<MessageExt> wrappers = MessageDecoder.decodes(ByteBuffer.wrap(response.getBody()), true);
                                                QueryResult qr = new QueryResult(responseHeader.getIndexLastUpdateTimestamp(), wrappers);
                                                try {
                                                    lock.writeLock().lock();
                                                    queryResultList.add(qr);
                                                } finally {
                                                    lock.writeLock().unlock();
                                                }
                                                break;
                                            }
                                        default:
                                            log.warn("getResponseCommand failed, {} {}", response.getCode(), response.getRemark());
                                            break;
                                    }
                                } else {
                                    log.warn("getResponseCommand return null");
                                }
                            } finally {
                                countDownLatch.countDown();
                            }
                        }
                    }, isUniqKey);
                } catch (Exception e) {
                    log.warn("queryMessage exception", e);
                }
            }
            boolean ok = countDownLatch.await(timeoutMillis * 4, TimeUnit.MILLISECONDS);
            if (!ok) {
                log.warn("queryMessage, maybe some broker failed");
            }
            long indexLastUpdateTimestamp = 0;
            List<MessageExt> messageList = new LinkedList<MessageExt>();
            for (QueryResult qr : queryResultList) {
                if (qr.getIndexLastUpdateTimestamp() > indexLastUpdateTimestamp) {
                    indexLastUpdateTimestamp = qr.getIndexLastUpdateTimestamp();
                }
                for (MessageExt msgExt : qr.getMessageList()) {
                    if (isUniqKey) {
                        if (msgExt.getMsgId().equals(key)) {
                            if (messageList.size() > 0) {
                                if (messageList.get(0).getStoreTimestamp() > msgExt.getStoreTimestamp()) {
                                    messageList.clear();
                                    messageList.add(msgExt);
                                }
                            } else {
                                messageList.add(msgExt);
                            }
                        } else {
                            log.warn("queryMessage by uniqKey, find message key not matched, maybe hash duplicate {}", msgExt.toString());
                        }
                    } else {
                        String keys = msgExt.getKeys();
                        if (keys != null) {
                            boolean matched = false;
                            String[] keyArray = keys.split(MessageConst.KEY_SEPARATOR);
                            if (keyArray != null) {
                                for (String k : keyArray) {
                                    if (key.equals(k)) {
                                        matched = true;
                                        break;
                                    }
                                }
                            }
                            if (matched) {
                                messageList.add(msgExt);
                            } else {
                                log.warn("queryMessage, find message key not matched, maybe hash duplicate {}", msgExt.toString());
                            }
                        }
                    }
                }
            }
            if (!messageList.isEmpty()) {
                return new QueryResult(indexLastUpdateTimestamp, messageList);
            } else {
                throw new MQClientException(ResponseCode.NO_MESSAGE, "query message by key finished, but no message.");
            }
        }
    }
    throw new MQClientException(ResponseCode.TOPIC_NOT_EXIST, "The topic[" + topic + "] not matched route info");
}
Also used : InvokeCallback(org.apache.rocketmq.remoting.InvokeCallback) QueryMessageRequestHeader(org.apache.rocketmq.common.protocol.header.QueryMessageRequestHeader) BrokerData(org.apache.rocketmq.common.protocol.route.BrokerData) RemotingCommandException(org.apache.rocketmq.remoting.exception.RemotingCommandException) ResponseFuture(org.apache.rocketmq.remoting.netty.ResponseFuture) CountDownLatch(java.util.concurrent.CountDownLatch) ReentrantReadWriteLock(java.util.concurrent.locks.ReentrantReadWriteLock) LinkedList(java.util.LinkedList) RemotingCommandException(org.apache.rocketmq.remoting.exception.RemotingCommandException) MQClientException(org.apache.rocketmq.client.exception.MQClientException) MQBrokerException(org.apache.rocketmq.client.exception.MQBrokerException) RemotingException(org.apache.rocketmq.remoting.exception.RemotingException) TopicRouteData(org.apache.rocketmq.common.protocol.route.TopicRouteData) RemotingCommand(org.apache.rocketmq.remoting.protocol.RemotingCommand) MessageExt(org.apache.rocketmq.common.message.MessageExt) QueryResult(org.apache.rocketmq.client.QueryResult) QueryMessageResponseHeader(org.apache.rocketmq.common.protocol.header.QueryMessageResponseHeader) ReentrantReadWriteLock(java.util.concurrent.locks.ReentrantReadWriteLock) ReadWriteLock(java.util.concurrent.locks.ReadWriteLock) MQClientException(org.apache.rocketmq.client.exception.MQClientException)

Example 20 with RemotingCommandException

use of org.apache.rocketmq.remoting.exception.RemotingCommandException in project rocketmq-rocketmq-all-4.1.0-incubating by lirenzuo.

the class PullMessageProcessor method processRequest.

private RemotingCommand processRequest(final Channel channel, RemotingCommand request, boolean brokerAllowSuspend) throws RemotingCommandException {
    RemotingCommand response = RemotingCommand.createResponseCommand(PullMessageResponseHeader.class);
    final PullMessageResponseHeader responseHeader = (PullMessageResponseHeader) response.readCustomHeader();
    final PullMessageRequestHeader requestHeader = (PullMessageRequestHeader) request.decodeCommandCustomHeader(PullMessageRequestHeader.class);
    // 由于使用sendfile,所以必须要设置
    response.setOpaque(request.getOpaque());
    if (LOG.isDebugEnabled()) {
        LOG.debug("receive PullMessage request command, {}", request);
    }
    // 检查Broker权限
    if (!PermName.isReadable(this.brokerController.getBrokerConfig().getBrokerPermission())) {
        response.setCode(ResponseCode.NO_PERMISSION);
        response.setRemark(String.format("the broker[%s] pulling message is forbidden", this.brokerController.getBrokerConfig().getBrokerIP1()));
        return response;
    }
    // 确保订阅组存在
    SubscriptionGroupConfig subscriptionGroupConfig = this.brokerController.getSubscriptionGroupManager().findSubscriptionGroupConfig(requestHeader.getConsumerGroup());
    if (null == subscriptionGroupConfig) {
        response.setCode(ResponseCode.SUBSCRIPTION_GROUP_NOT_EXIST);
        response.setRemark(String.format("subscription group [%s] does not exist, %s", requestHeader.getConsumerGroup(), FAQUrl.suggestTodo(FAQUrl.SUBSCRIPTION_GROUP_NOT_EXIST)));
        return response;
    }
    // 这个订阅组是否可以消费消息
    if (!subscriptionGroupConfig.isConsumeEnable()) {
        response.setCode(ResponseCode.NO_PERMISSION);
        response.setRemark("subscription group no permission, " + requestHeader.getConsumerGroup());
        return response;
    }
    final boolean hasSuspendFlag = PullSysFlag.hasSuspendFlag(requestHeader.getSysFlag());
    final boolean hasCommitOffsetFlag = PullSysFlag.hasCommitOffsetFlag(requestHeader.getSysFlag());
    final boolean hasSubscriptionFlag = PullSysFlag.hasSubscriptionFlag(requestHeader.getSysFlag());
    final long suspendTimeoutMillisLong = hasSuspendFlag ? requestHeader.getSuspendTimeoutMillis() : 0;
    // 检查topic是否存在
    TopicConfig topicConfig = this.brokerController.getTopicConfigManager().selectTopicConfig(requestHeader.getTopic());
    if (null == topicConfig) {
        LOG.error("The topic {} not exist, consumer: {} ", requestHeader.getTopic(), RemotingHelper.parseChannelRemoteAddr(channel));
        response.setCode(ResponseCode.TOPIC_NOT_EXIST);
        response.setRemark(String.format("topic[%s] not exist, apply first please! %s", requestHeader.getTopic(), FAQUrl.suggestTodo(FAQUrl.APPLY_TOPIC_URL)));
        return response;
    }
    // 检查topic权限
    if (!PermName.isReadable(topicConfig.getPerm())) {
        response.setCode(ResponseCode.NO_PERMISSION);
        response.setRemark("the topic[" + requestHeader.getTopic() + "] pulling message is forbidden");
        return response;
    }
    // 检查队列有效性
    if (requestHeader.getQueueId() < 0 || requestHeader.getQueueId() >= topicConfig.getReadQueueNums()) {
        String errorInfo = String.format("queueId[%d] is illegal, topic:[%s] topicConfig.readQueueNums:[%d] consumer:[%s]", requestHeader.getQueueId(), requestHeader.getTopic(), topicConfig.getReadQueueNums(), channel.remoteAddress());
        LOG.warn(errorInfo);
        response.setCode(ResponseCode.SYSTEM_ERROR);
        response.setRemark(errorInfo);
        return response;
    }
    // 订阅关系处理
    SubscriptionData subscriptionData = null;
    ConsumerFilterData consumerFilterData = null;
    if (hasSubscriptionFlag) {
        try {
            subscriptionData = FilterAPI.build(requestHeader.getTopic(), requestHeader.getSubscription(), requestHeader.getExpressionType());
            if (!ExpressionType.isTagType(subscriptionData.getExpressionType())) {
                consumerFilterData = ConsumerFilterManager.build(requestHeader.getTopic(), requestHeader.getConsumerGroup(), requestHeader.getSubscription(), requestHeader.getExpressionType(), requestHeader.getSubVersion());
                assert consumerFilterData != null;
            }
        } catch (Exception e) {
            // 
            LOG.warn(// 
            "Parse the consumer's subscription[{}] failed, group: {}", // 
            requestHeader.getSubscription(), requestHeader.getConsumerGroup());
            response.setCode(ResponseCode.SUBSCRIPTION_PARSE_FAILED);
            response.setRemark("parse the consumer's subscription failed");
            return response;
        }
    } else {
        ConsumerGroupInfo consumerGroupInfo = this.brokerController.getConsumerManager().getConsumerGroupInfo(requestHeader.getConsumerGroup());
        if (null == consumerGroupInfo) {
            LOG.warn("The consumer's group info not exist, group: {}", requestHeader.getConsumerGroup());
            response.setCode(ResponseCode.SUBSCRIPTION_NOT_EXIST);
            response.setRemark("the consumer's group info not exist" + FAQUrl.suggestTodo(FAQUrl.SAME_GROUP_DIFFERENT_TOPIC));
            return response;
        }
        if (// 
        !subscriptionGroupConfig.isConsumeBroadcastEnable() && consumerGroupInfo.getMessageModel() == MessageModel.BROADCASTING) {
            response.setCode(ResponseCode.NO_PERMISSION);
            response.setRemark("the consumer group[" + requestHeader.getConsumerGroup() + "] can not consume by broadcast way");
            return response;
        }
        subscriptionData = consumerGroupInfo.findSubscriptionData(requestHeader.getTopic());
        if (null == subscriptionData) {
            LOG.warn("The consumer's subscription not exist, group: {}, topic:{}", requestHeader.getConsumerGroup(), requestHeader.getTopic());
            response.setCode(ResponseCode.SUBSCRIPTION_NOT_EXIST);
            response.setRemark("the consumer's subscription not exist" + FAQUrl.suggestTodo(FAQUrl.SAME_GROUP_DIFFERENT_TOPIC));
            return response;
        }
        // 判断Broker的订阅关系版本是否最新
        if (subscriptionData.getSubVersion() < requestHeader.getSubVersion()) {
            LOG.warn("The broker's subscription is not latest, group: {} {}", requestHeader.getConsumerGroup(), subscriptionData.getSubString());
            response.setCode(ResponseCode.SUBSCRIPTION_NOT_LATEST);
            response.setRemark("the consumer's subscription not latest");
            return response;
        }
        if (!ExpressionType.isTagType(subscriptionData.getExpressionType())) {
            consumerFilterData = this.brokerController.getConsumerFilterManager().get(requestHeader.getTopic(), requestHeader.getConsumerGroup());
            if (consumerFilterData == null) {
                response.setCode(ResponseCode.FILTER_DATA_NOT_EXIST);
                response.setRemark("The broker's consumer filter data is not exist!Your expression may be wrong!");
                return response;
            }
            if (consumerFilterData.getClientVersion() < requestHeader.getSubVersion()) {
                LOG.warn("The broker's consumer filter data is not latest, group: {}, topic: {}, serverV: {}, clientV: {}", requestHeader.getConsumerGroup(), requestHeader.getTopic(), consumerFilterData.getClientVersion(), requestHeader.getSubVersion());
                response.setCode(ResponseCode.FILTER_DATA_NOT_LATEST);
                response.setRemark("the consumer's consumer filter data not latest");
                return response;
            }
        }
    }
    if (!ExpressionType.isTagType(subscriptionData.getExpressionType()) && !this.brokerController.getBrokerConfig().isEnablePropertyFilter()) {
        response.setCode(ResponseCode.SYSTEM_ERROR);
        response.setRemark("The broker does not support consumer to filter message by " + subscriptionData.getExpressionType());
        return response;
    }
    MessageFilter messageFilter;
    if (this.brokerController.getBrokerConfig().isFilterSupportRetry()) {
        messageFilter = new ExpressionForRetryMessageFilter(subscriptionData, consumerFilterData, this.brokerController.getConsumerFilterManager());
    } else {
        messageFilter = new ExpressionMessageFilter(subscriptionData, consumerFilterData, this.brokerController.getConsumerFilterManager());
    }
    // 这个地方体现了零拷贝。消息落地磁盘的时候,是从file map出来的内存buffer,此时消费消息的时候无需再读取文件,而是直接读取map出来的buffer!
    final GetMessageResult getMessageResult = this.brokerController.getMessageStore().getMessage(requestHeader.getConsumerGroup(), requestHeader.getTopic(), requestHeader.getQueueId(), requestHeader.getQueueOffset(), requestHeader.getMaxMsgNums(), messageFilter);
    if (getMessageResult != null) {
        response.setRemark(getMessageResult.getStatus().name());
        responseHeader.setNextBeginOffset(getMessageResult.getNextBeginOffset());
        responseHeader.setMinOffset(getMessageResult.getMinOffset());
        responseHeader.setMaxOffset(getMessageResult.getMaxOffset());
        // 消费较慢,重定向到另外一台机器
        if (getMessageResult.isSuggestPullingFromSlave()) {
            responseHeader.setSuggestWhichBrokerId(subscriptionGroupConfig.getWhichBrokerWhenConsumeSlowly());
        } else // 消费正常,按照订阅组配置重定向
        {
            responseHeader.setSuggestWhichBrokerId(MixAll.MASTER_ID);
        }
        switch(this.brokerController.getMessageStoreConfig().getBrokerRole()) {
            case ASYNC_MASTER:
            case SYNC_MASTER:
                break;
            case SLAVE:
                if (!this.brokerController.getBrokerConfig().isSlaveReadEnable()) {
                    response.setCode(ResponseCode.PULL_RETRY_IMMEDIATELY);
                    responseHeader.setSuggestWhichBrokerId(MixAll.MASTER_ID);
                }
                break;
        }
        if (this.brokerController.getBrokerConfig().isSlaveReadEnable()) {
            // consume too slow ,redirect to another machine
            if (getMessageResult.isSuggestPullingFromSlave()) {
                responseHeader.setSuggestWhichBrokerId(subscriptionGroupConfig.getWhichBrokerWhenConsumeSlowly());
            } else // consume ok
            {
                responseHeader.setSuggestWhichBrokerId(subscriptionGroupConfig.getBrokerId());
            }
        } else {
            responseHeader.setSuggestWhichBrokerId(MixAll.MASTER_ID);
        }
        switch(getMessageResult.getStatus()) {
            case FOUND:
                response.setCode(ResponseCode.SUCCESS);
                break;
            case MESSAGE_WAS_REMOVING:
                response.setCode(ResponseCode.PULL_RETRY_IMMEDIATELY);
                break;
            // 这两个返回值都表示服务器暂时没有这个队列,应该立刻将客户端Offset重置为0
            case NO_MATCHED_LOGIC_QUEUE:
            case NO_MESSAGE_IN_QUEUE:
                if (0 != requestHeader.getQueueOffset()) {
                    response.setCode(ResponseCode.PULL_OFFSET_MOVED);
                    // XXX: warn and notify me
                    // 
                    LOG.info(// 
                    "the broker store no queue data, fix the request offset {} to {}, Topic: {} QueueId: {} Consumer Group: {}", // 
                    requestHeader.getQueueOffset(), // 
                    getMessageResult.getNextBeginOffset(), // 
                    requestHeader.getTopic(), // 
                    requestHeader.getQueueId(), // 
                    requestHeader.getConsumerGroup());
                } else {
                    response.setCode(ResponseCode.PULL_NOT_FOUND);
                }
                break;
            case NO_MATCHED_MESSAGE:
                response.setCode(ResponseCode.PULL_RETRY_IMMEDIATELY);
                break;
            case OFFSET_FOUND_NULL:
                response.setCode(ResponseCode.PULL_NOT_FOUND);
                break;
            case OFFSET_OVERFLOW_BADLY:
                response.setCode(ResponseCode.PULL_OFFSET_MOVED);
                // XXX: warn and notify me
                LOG.info("The request offset:{} over flow badly, broker max offset:{} , consumer: {}", requestHeader.getQueueOffset(), getMessageResult.getMaxOffset(), channel.remoteAddress());
                break;
            case OFFSET_OVERFLOW_ONE:
                response.setCode(ResponseCode.PULL_NOT_FOUND);
                break;
            case OFFSET_TOO_SMALL:
                response.setCode(ResponseCode.PULL_OFFSET_MOVED);
                LOG.info("The request offset is too small. group={}, topic={}, requestOffset={}, brokerMinOffset={}, clientIp={}", requestHeader.getConsumerGroup(), requestHeader.getTopic(), requestHeader.getQueueOffset(), getMessageResult.getMinOffset(), channel.remoteAddress());
                break;
            default:
                assert false;
                break;
        }
        // 消息轨迹:记录客户端拉取的消息记录(不表示消费成功)
        if (this.hasConsumeMessageHook()) {
            // 执行hook
            ConsumeMessageContext context = new ConsumeMessageContext();
            context.setConsumerGroup(requestHeader.getConsumerGroup());
            context.setTopic(requestHeader.getTopic());
            context.setQueueId(requestHeader.getQueueId());
            String owner = request.getExtFields().get(BrokerStatsManager.COMMERCIAL_OWNER);
            switch(response.getCode()) {
                case ResponseCode.SUCCESS:
                    int commercialBaseCount = brokerController.getBrokerConfig().getCommercialBaseCount();
                    int incValue = getMessageResult.getMsgCount4Commercial() * commercialBaseCount;
                    context.setCommercialRcvStats(BrokerStatsManager.StatsType.RCV_SUCCESS);
                    context.setCommercialRcvTimes(incValue);
                    context.setCommercialRcvSize(getMessageResult.getBufferTotalSize());
                    context.setCommercialOwner(owner);
                    break;
                case ResponseCode.PULL_NOT_FOUND:
                    if (!brokerAllowSuspend) {
                        context.setCommercialRcvStats(BrokerStatsManager.StatsType.RCV_EPOLLS);
                        context.setCommercialRcvTimes(1);
                        context.setCommercialOwner(owner);
                    }
                    break;
                case ResponseCode.PULL_RETRY_IMMEDIATELY:
                case ResponseCode.PULL_OFFSET_MOVED:
                    context.setCommercialRcvStats(BrokerStatsManager.StatsType.RCV_EPOLLS);
                    context.setCommercialRcvTimes(1);
                    context.setCommercialOwner(owner);
                    break;
                default:
                    assert false;
                    break;
            }
            this.executeConsumeMessageHookBefore(context);
        }
        switch(response.getCode()) {
            case ResponseCode.SUCCESS:
                // 统计
                this.brokerController.getBrokerStatsManager().incGroupGetNums(requestHeader.getConsumerGroup(), requestHeader.getTopic(), getMessageResult.getMessageCount());
                this.brokerController.getBrokerStatsManager().incGroupGetSize(requestHeader.getConsumerGroup(), requestHeader.getTopic(), getMessageResult.getBufferTotalSize());
                this.brokerController.getBrokerStatsManager().incBrokerGetNums(getMessageResult.getMessageCount());
                if (this.brokerController.getBrokerConfig().isTransferMsgByHeap()) {
                    final long beginTimeMills = this.brokerController.getMessageStore().now();
                    final byte[] r = this.readGetMessageResult(getMessageResult, requestHeader.getConsumerGroup(), requestHeader.getTopic(), requestHeader.getQueueId());
                    this.brokerController.getBrokerStatsManager().incGroupGetLatency(requestHeader.getConsumerGroup(), requestHeader.getTopic(), requestHeader.getQueueId(), (int) (this.brokerController.getMessageStore().now() - beginTimeMills));
                    response.setBody(r);
                } else {
                    try {
                        FileRegion fileRegion = new ManyMessageTransfer(response.encodeHeader(getMessageResult.getBufferTotalSize()), getMessageResult);
                        channel.writeAndFlush(fileRegion).addListener(new ChannelFutureListener() {

                            @Override
                            public void operationComplete(ChannelFuture future) throws Exception {
                                getMessageResult.release();
                                if (!future.isSuccess()) {
                                    LOG.error("Fail to transfer messages from page cache to {}", channel.remoteAddress(), future.cause());
                                }
                            }
                        });
                    } catch (Throwable e) {
                        LOG.error("Error occurred when transferring messages from page cache", e);
                        getMessageResult.release();
                    }
                    response = null;
                }
                break;
            case ResponseCode.PULL_NOT_FOUND:
                // 长轮询
                if (brokerAllowSuspend && hasSuspendFlag) {
                    long pollingTimeMills = suspendTimeoutMillisLong;
                    if (!this.brokerController.getBrokerConfig().isLongPollingEnable()) {
                        pollingTimeMills = this.brokerController.getBrokerConfig().getShortPollingTimeMills();
                    }
                    String topic = requestHeader.getTopic();
                    long offset = requestHeader.getQueueOffset();
                    int queueId = requestHeader.getQueueId();
                    PullRequest pullRequest = new PullRequest(request, channel, pollingTimeMills, this.brokerController.getMessageStore().now(), offset, subscriptionData, messageFilter);
                    this.brokerController.getPullRequestHoldService().suspendPullRequest(topic, queueId, pullRequest);
                    response = null;
                    break;
                }
            // 向Consumer返回应答
            case ResponseCode.PULL_RETRY_IMMEDIATELY:
                break;
            case ResponseCode.PULL_OFFSET_MOVED:
                if (this.brokerController.getMessageStoreConfig().getBrokerRole() != BrokerRole.SLAVE || this.brokerController.getMessageStoreConfig().isOffsetCheckInSlave()) {
                    MessageQueue mq = new MessageQueue();
                    mq.setTopic(requestHeader.getTopic());
                    mq.setQueueId(requestHeader.getQueueId());
                    mq.setBrokerName(this.brokerController.getBrokerConfig().getBrokerName());
                    OffsetMovedEvent event = new OffsetMovedEvent();
                    event.setConsumerGroup(requestHeader.getConsumerGroup());
                    event.setMessageQueue(mq);
                    event.setOffsetRequest(requestHeader.getQueueOffset());
                    event.setOffsetNew(getMessageResult.getNextBeginOffset());
                    this.generateOffsetMovedEvent(event);
                    LOG.warn("PULL_OFFSET_MOVED:correction offset. topic={}, groupId={}, requestOffset={}, newOffset={}, suggestBrokerId={}", requestHeader.getTopic(), requestHeader.getConsumerGroup(), event.getOffsetRequest(), event.getOffsetNew(), responseHeader.getSuggestWhichBrokerId());
                } else {
                    responseHeader.setSuggestWhichBrokerId(subscriptionGroupConfig.getBrokerId());
                    response.setCode(ResponseCode.PULL_RETRY_IMMEDIATELY);
                    LOG.warn("PULL_OFFSET_MOVED:none correction. topic={}, groupId={}, requestOffset={}, suggestBrokerId={}", requestHeader.getTopic(), requestHeader.getConsumerGroup(), requestHeader.getQueueOffset(), responseHeader.getSuggestWhichBrokerId());
                }
                break;
            default:
                assert false;
        }
    } else {
        response.setCode(ResponseCode.SYSTEM_ERROR);
        response.setRemark("store getMessage return null");
    }
    // 存储Consumer消费进度
    // 说明是首次调用,相对于长轮询通知
    boolean storeOffsetEnable = brokerAllowSuspend;
    // 说明Consumer设置了标志位
    storeOffsetEnable = storeOffsetEnable && hasCommitOffsetFlag;
    storeOffsetEnable = // 只有Master支持存储offset
    storeOffsetEnable && this.brokerController.getMessageStoreConfig().getBrokerRole() != BrokerRole.SLAVE;
    if (storeOffsetEnable) {
        this.brokerController.getConsumerOffsetManager().commitOffset(RemotingHelper.parseChannelRemoteAddr(channel), requestHeader.getConsumerGroup(), requestHeader.getTopic(), requestHeader.getQueueId(), requestHeader.getCommitOffset());
    }
    return response;
}
Also used : PullRequest(org.apache.rocketmq.broker.longpolling.PullRequest) ExpressionForRetryMessageFilter(org.apache.rocketmq.broker.filter.ExpressionForRetryMessageFilter) ConsumerGroupInfo(org.apache.rocketmq.broker.client.ConsumerGroupInfo) SubscriptionGroupConfig(org.apache.rocketmq.common.subscription.SubscriptionGroupConfig) RemotingCommand(org.apache.rocketmq.remoting.protocol.RemotingCommand) ManyMessageTransfer(org.apache.rocketmq.broker.pagecache.ManyMessageTransfer) OffsetMovedEvent(org.apache.rocketmq.common.protocol.topic.OffsetMovedEvent) FileRegion(io.netty.channel.FileRegion) ConsumerFilterData(org.apache.rocketmq.broker.filter.ConsumerFilterData) ChannelFuture(io.netty.channel.ChannelFuture) GetMessageResult(org.apache.rocketmq.store.GetMessageResult) ExpressionMessageFilter(org.apache.rocketmq.broker.filter.ExpressionMessageFilter) TopicConfig(org.apache.rocketmq.common.TopicConfig) ChannelFutureListener(io.netty.channel.ChannelFutureListener) RemotingCommandException(org.apache.rocketmq.remoting.exception.RemotingCommandException) PullMessageResponseHeader(org.apache.rocketmq.common.protocol.header.PullMessageResponseHeader) PullMessageRequestHeader(org.apache.rocketmq.common.protocol.header.PullMessageRequestHeader) ConsumeMessageContext(org.apache.rocketmq.broker.mqtrace.ConsumeMessageContext) MessageQueue(org.apache.rocketmq.common.message.MessageQueue) SubscriptionData(org.apache.rocketmq.common.protocol.heartbeat.SubscriptionData) ExpressionForRetryMessageFilter(org.apache.rocketmq.broker.filter.ExpressionForRetryMessageFilter) MessageFilter(org.apache.rocketmq.store.MessageFilter) ExpressionMessageFilter(org.apache.rocketmq.broker.filter.ExpressionMessageFilter)

Aggregations

RemotingCommandException (org.apache.rocketmq.remoting.exception.RemotingCommandException)26 RemotingCommand (org.apache.rocketmq.remoting.protocol.RemotingCommand)22 ChannelFuture (io.netty.channel.ChannelFuture)8 ChannelFutureListener (io.netty.channel.ChannelFutureListener)8 FileRegion (io.netty.channel.FileRegion)6 UnsupportedEncodingException (java.io.UnsupportedEncodingException)4 UnknownHostException (java.net.UnknownHostException)4 MQClientException (org.apache.rocketmq.client.exception.MQClientException)4 TopicConfig (org.apache.rocketmq.common.TopicConfig)4 QueryMessageRequestHeader (org.apache.rocketmq.common.protocol.header.QueryMessageRequestHeader)4 QueryMessageResponseHeader (org.apache.rocketmq.common.protocol.header.QueryMessageResponseHeader)4 SubscriptionData (org.apache.rocketmq.common.protocol.heartbeat.SubscriptionData)4 TopicRouteData (org.apache.rocketmq.common.protocol.route.TopicRouteData)4 CommandCustomHeader (org.apache.rocketmq.remoting.CommandCustomHeader)4 RemotingTimeoutException (org.apache.rocketmq.remoting.exception.RemotingTimeoutException)4 JSONField (com.alibaba.fastjson.annotation.JSONField)2 IOException (java.io.IOException)2 Field (java.lang.reflect.Field)2 LinkedList (java.util.LinkedList)2 CountDownLatch (java.util.concurrent.CountDownLatch)2