Search in sources :

Example 31 with DefaultKeyValue

use of io.openmessaging.internal.DefaultKeyValue in project rocketmq-externals by apache.

the class DivideTaskByQueue method divide.

@Override
public List<KeyValue> divide(Map<String, Set<TaskTopicInfo>> topicRouteMap, TaskDivideConfig tdc) {
    List<KeyValue> config = new ArrayList<KeyValue>();
    int parallelism = tdc.getTaskParallelism();
    Map<Integer, List<TaskTopicInfo>> queueTopicList = new HashMap<Integer, List<TaskTopicInfo>>();
    int id = -1;
    for (String t : topicRouteMap.keySet()) {
        for (TaskTopicInfo taskTopicInfo : topicRouteMap.get(t)) {
            int ind = ++id % parallelism;
            if (!queueTopicList.containsKey(ind)) {
                queueTopicList.put(ind, new ArrayList<TaskTopicInfo>());
            }
            queueTopicList.get(ind).add(taskTopicInfo);
        }
    }
    for (int i = 0; i < parallelism; i++) {
        KeyValue keyValue = new DefaultKeyValue();
        keyValue.put(TaskConfigEnum.TASK_STORE_ROCKETMQ.getKey(), tdc.getStoreTopic());
        keyValue.put(TaskConfigEnum.TASK_SOURCE_ROCKETMQ.getKey(), tdc.getSourceNamesrvAddr());
        keyValue.put(TaskConfigEnum.TASK_DATA_TYPE.getKey(), DataType.COMMON_MESSAGE.ordinal());
        keyValue.put(TaskConfigEnum.TASK_TOPIC_INFO.getKey(), JSONObject.toJSONString(queueTopicList.get(i)));
        keyValue.put(TaskConfigEnum.TASK_SOURCE_RECORD_CONVERTER.getKey(), tdc.getSrcRecordConverter());
        keyValue.put(TaskConfigEnum.TASK_SOURCE_ACL_ENABLE.getKey(), String.valueOf(tdc.isSrcAclEnable()));
        keyValue.put(TaskConfigEnum.TASK_SOURCE_ACCESS_KEY.getKey(), tdc.getSrcAccessKey());
        keyValue.put(TaskConfigEnum.TASK_SOURCE_SECRET_KEY.getKey(), tdc.getSrcSecretKey());
        config.add(keyValue);
    }
    return config;
}
Also used : DefaultKeyValue(io.openmessaging.internal.DefaultKeyValue) DefaultKeyValue(io.openmessaging.internal.DefaultKeyValue) KeyValue(io.openmessaging.KeyValue) HashMap(java.util.HashMap) TaskTopicInfo(org.apache.rocketmq.replicator.config.TaskTopicInfo) ArrayList(java.util.ArrayList) List(java.util.List) ArrayList(java.util.ArrayList)

Example 32 with DefaultKeyValue

use of io.openmessaging.internal.DefaultKeyValue in project rocketmq-externals by apache.

the class RmqSourceReplicatorTest method testGenerateTopic.

@Test
public void testGenerateTopic() throws NoSuchFieldException {
    RmqSourceReplicator rmqSourceReplicator = Mockito.spy(RmqSourceReplicator.class);
    RmqConnectorConfig config = new RmqConnectorConfig();
    KeyValue kv = new DefaultKeyValue();
    kv.put(ConfigDefine.CONN_TOPIC_RENAME_FMT, "${topic}.replica");
    config.validate(kv);
    Field field = RmqSourceReplicator.class.getDeclaredField("replicatorConfig");
    FieldSetter.setField(rmqSourceReplicator, field, config);
    String dstTopic = rmqSourceReplicator.generateTargetTopic("dest");
    assertThat(dstTopic).isEqualTo("dest.replica");
}
Also used : DefaultKeyValue(io.openmessaging.internal.DefaultKeyValue) Field(java.lang.reflect.Field) RmqConnectorConfig(org.apache.rocketmq.replicator.config.RmqConnectorConfig) KeyValue(io.openmessaging.KeyValue) DefaultKeyValue(io.openmessaging.internal.DefaultKeyValue) Test(org.junit.Test)

Aggregations

DefaultKeyValue (io.openmessaging.internal.DefaultKeyValue)32 KeyValue (io.openmessaging.KeyValue)27 Test (org.junit.Test)14 ArrayList (java.util.ArrayList)8 List (java.util.List)7 HashMap (java.util.HashMap)4 Field (java.lang.reflect.Field)3 SourceDataEntry (io.openmessaging.connector.api.data.SourceDataEntry)2 Map (java.util.Map)2 Set (java.util.Set)2 MongoSourceTask (org.apache.connect.mongo.connector.MongoSourceTask)2 ReplicaSet (org.apache.connect.mongo.replicator.ReplicaSet)2 ReplicaSetConfig (org.apache.connect.mongo.replicator.ReplicaSetConfig)2 ReplicaSetsContext (org.apache.connect.mongo.replicator.ReplicaSetsContext)2 TaskTopicInfo (org.apache.rocketmq.replicator.config.TaskTopicInfo)2 Before (org.junit.Before)2 RedisURI (com.moilioncircle.redis.replicator.RedisURI)1 InvocationTargetException (java.lang.reflect.InvocationTargetException)1 ByteBuffer (java.nio.ByteBuffer)1 LinkedList (java.util.LinkedList)1