Examples with Communication com.alibaba.datax.core.statistics.communication.Communication used on opensource projects

Example 16 with Communication

use of com.alibaba.datax.core.statistics.communication.Communication in project DataX by alibaba.

the class TaskGroupContainerTest method testRealTransformer.

@Test
public void testRealTransformer() {
    LoadUtil.bind(configurationRealTransformer);
    this.configurationRealTransformer.set("plugin.writer.fakewriter.class", FakeOneReader.class.getName());
    this.configurationRealTransformer.set("plugin.writer.fakewriter.class", FakeLongTimeWriter.class.getName());
    this.configurationRealTransformer.set(CoreConstant.DATAX_CORE_CONTAINER_TASKGROUP_CHANNEL, 1);
    Configuration jobContent = this.configurationRealTransformer.getListConfiguration(CoreConstant.DATAX_JOB_CONTENT).get(0);
    List<Configuration> jobContents = new ArrayList<Configuration>();
    jobContents.add(jobContent);
    this.configurationRealTransformer.set(CoreConstant.DATAX_JOB_CONTENT, jobContents);
    TaskGroupContainer taskGroupContainer = new TaskGroupContainer(this.configurationRealTransformer);
    taskGroupContainer.start();
    Assert.assertTrue(State.SUCCEEDED == taskGroupContainer.getContainerCommunicator().collect().getState());
    Communication res = null;
    try {
        Method com = TaskGroupContainer.class.getDeclaredMethod("reportTaskGroupCommunication", Communication.class, int.class);
        com.setAccessible(true);
        res = (Communication) com.invoke(taskGroupContainer, new Communication(), 1);
        System.out.println("TaskGroup => " + CommunicationTool.Stringify.getSnapshot(res));
    } catch (Exception e) {
        e.printStackTrace();
    }
    Assert.assertTrue(res != null);
    Assert.assertEquals(res.getLongCounter(CommunicationTool.TOTAL_READ_RECORDS).longValue(), 30);
    Assert.assertEquals(res.getLongCounter(CommunicationTool.TRANSFORMER_SUCCEED_RECORDS).longValue(), 10);
    Assert.assertEquals(res.getLongCounter(CommunicationTool.TRANSFORMER_FAILED_RECORDS).longValue(), 0);
    Assert.assertEquals(res.getLongCounter(CommunicationTool.TRANSFORMER_FILTER_RECORDS).longValue(), 10);
    Assert.assertTrue(res.getLongCounter(CommunicationTool.TRANSFORMER_USED_TIME).longValue() > 0);
}

Example 17 with Communication

use of com.alibaba.datax.core.statistics.communication.Communication in project DataX by alibaba.

the class AbstractCollector method registerTGCommunication.

public void registerTGCommunication(List<Configuration> taskGroupConfigurationList) {
    for (Configuration config : taskGroupConfigurationList) {
        int taskGroupId = config.getInt(CoreConstant.DATAX_CORE_CONTAINER_TASKGROUP_ID);
        LocalTGCommunicationManager.registerTaskGroupCommunication(taskGroupId, new Communication());
    }
}

Example 18 with Communication

use of com.alibaba.datax.core.statistics.communication.Communication in project DataX by alibaba.

the class AbstractCollector method registerTaskCommunication.

public void registerTaskCommunication(List<Configuration> taskConfigurationList) {
    for (Configuration taskConfig : taskConfigurationList) {
        int taskId = taskConfig.getInt(CoreConstant.TASK_ID);
        this.taskCommunicationMap.put(taskId, new Communication());
    }
}

Example 19 with Communication

use of com.alibaba.datax.core.statistics.communication.Communication in project DataX by alibaba.

the class AbstractCollector method collectFromTask.

public Communication collectFromTask() {
    Communication communication = new Communication();
    communication.setState(State.SUCCEEDED);
    for (Communication taskCommunication : this.taskCommunicationMap.values()) {
        communication.mergeFrom(taskCommunication);
    }
    return communication;
}

Example 20 with Communication

use of com.alibaba.datax.core.statistics.communication.Communication in project DataX by alibaba.

the class AbstractContainerCommunicator method resetCommunication.

public void resetCommunication(Integer id) {
    Map<Integer, Communication> map = getCommunicationMap();
    map.put(id, new Communication());
}