Search in sources :

Example 1 with TerminEventData

use of com.alibaba.otter.shared.arbitrate.model.TerminEventData in project otter by alibaba.

the class TerminMemoryArbitrateEvent method await.

public TerminEventData await(Long pipelineId) throws InterruptedException {
    Assert.notNull(pipelineId);
    MemoryStageController stageController = ArbitrateFactory.getInstance(pipelineId, MemoryStageController.class);
    TerminEventData eventData = stageController.waitTermin();
    if (logger.isDebugEnabled()) {
        logger.debug("## await pipeline[{}] processId[{}] is termin", pipelineId, eventData.getProcessId());
    }
    return eventData;
}
Also used : TerminEventData(com.alibaba.otter.shared.arbitrate.model.TerminEventData)

Example 2 with TerminEventData

use of com.alibaba.otter.shared.arbitrate.model.TerminEventData in project otter by alibaba.

the class TerminMemoryArbitrateEvent method exhaust.

public void exhaust(Long pipelineId) {
    Assert.notNull(pipelineId);
    MemoryStageController stageController = ArbitrateFactory.getInstance(pipelineId, MemoryStageController.class);
    int size = stageController.sizeTermin();
    try {
        for (int i = 0; i < size; i++) {
            TerminEventData data = stageController.waitTermin();
            ack(data);
        }
    } catch (InterruptedException e) {
        throw new ArbitrateException(e);
    }
}
Also used : TerminEventData(com.alibaba.otter.shared.arbitrate.model.TerminEventData) ArbitrateException(com.alibaba.otter.shared.arbitrate.exception.ArbitrateException)

Example 3 with TerminEventData

use of com.alibaba.otter.shared.arbitrate.model.TerminEventData in project otter by alibaba.

the class TerminZooKeeperArbitrateEvent method exhaust.

/**
 * 消耗掉所有的termin信号
 */
public void exhaust(Long pipelineId) {
    Assert.notNull(pipelineId);
    TerminMonitor terminMonitor = ArbitrateFactory.getInstance(pipelineId, TerminMonitor.class);
    int size = terminMonitor.size();
    try {
        for (int i = 0; i < size; i++) {
            Long processId;
            processId = terminMonitor.waitForProcess();
            TerminEventData data = new TerminEventData();
            data.setPipelineId(pipelineId);
            data.setProcessId(processId);
            ack(data);
        }
    } catch (InterruptedException e) {
        throw new ArbitrateException(e);
    }
}
Also used : TerminEventData(com.alibaba.otter.shared.arbitrate.model.TerminEventData) TerminMonitor(com.alibaba.otter.shared.arbitrate.impl.setl.monitor.TerminMonitor) ArbitrateException(com.alibaba.otter.shared.arbitrate.exception.ArbitrateException)

Example 4 with TerminEventData

use of com.alibaba.otter.shared.arbitrate.model.TerminEventData in project otter by alibaba.

the class OtterDownStreamHandler method notifyFailed.

private void notifyFailed() {
    detectingSuccessedCount.set(0);
    long failedCount = detectingFailedCount.incrementAndGet();
    if (failedCount == 1) {
        // 系数重置
        detectingExpCount = 1;
        notifyMainstemStatus(MainStemEventData.Status.TAKEING);
    }
    if (failedCount >= detectingThresoldCount * detectingExpCount * detectingExpCount) {
        notifyMainstemStatus(MainStemEventData.Status.TAKEING);
        // 系数增大一次
        detectingExpCount++;
        // 并且发送一次报警信息,系统不太正常了,超过一定时间一次都没有拿到对应的数据
        // 可能出现的情况:
        // 1. 主备发生切换,定位position花费了过久的时间
        // 2. MysqlEventParser工作不正常,一直拿不到数据,比如数据库挂了,但是又没通知其进行主备切换
        TerminEventData errorEventData = new TerminEventData();
        errorEventData.setPipelineId(pipelineId);
        errorEventData.setType(TerminType.WARNING);
        errorEventData.setCode("mainstem");
        errorEventData.setDesc(String.format(DETECTING_FAILED_MESSAGE, pipelineId, String.valueOf(detectingIntervalInSeconds * failedCount)));
        arbitrateEventService.terminEvent().single(errorEventData);
    }
}
Also used : TerminEventData(com.alibaba.otter.shared.arbitrate.model.TerminEventData)

Example 5 with TerminEventData

use of com.alibaba.otter.shared.arbitrate.model.TerminEventData in project otter by alibaba.

the class SelectTask method processTermin.

private boolean processTermin(boolean lastStatus, Long batchId, Long processId) throws InterruptedException {
    int retry = 0;
    SelectException exception = null;
    TerminEventData terminData = null;
    while (retry++ < 30) {
        // 因为存在网络因素,而且在Load进行termin处理时,因为是异步处理,有一定的概率会出现termin不按顺序过来
        terminData = arbitrateEventService.terminEvent().await(pipelineId);
        Long terminBatchId = terminData.getBatchId();
        Long terminProcessId = terminData.getProcessId();
        if (terminBatchId == null && processId != -1L && !processId.equals(terminProcessId)) {
            // 针对manager发起rollback,terminBatchId可能为null,需要特殊处理下
            exception = new SelectException("unmatched processId, SelectTask batchId = " + batchId + " processId = " + processId + " and Termin Event: " + terminData.toString());
            // sleep 1秒,等新的数据包
            Thread.sleep(1000);
        } else if (terminBatchId != null && batchId != -1L && !batchId.equals(terminBatchId)) {
            exception = new SelectException("unmatched terminId, SelectTask batchId = " + batchId + " processId = " + processId + " and Termin Event: " + terminData.toString());
            // sleep 1秒,等新的数据包
            Thread.sleep(1000);
        } else {
            // batchId/processId对上了,退出
            exception = null;
            break;
        }
    }
    if (exception != null) {
        throw exception;
    }
    if (needCheck) {
        checkContinueWork();
    }
    boolean status = terminData.getType().isNormal();
    if (lastStatus == false && status == true) {
        // 上一批失败,这一批成功,说明调度有问题
        throw new SelectException(String.format("last status is rollback , but now [batchId:%d , processId:%d] is ack", batchId, terminData.getProcessId()));
    }
    if (terminData.getType().isNormal()) {
        ack(batchId);
        sendDelayStat(pipelineId, terminData.getEndTime(), terminData.getFirstTime());
    } else {
        rollback(batchId);
    }
    // 先发送对应的数据
    arbitrateEventService.terminEvent().ack(terminData);
    return status;
}
Also used : TerminEventData(com.alibaba.otter.shared.arbitrate.model.TerminEventData) SelectException(com.alibaba.otter.node.etl.select.exceptions.SelectException)

Aggregations

TerminEventData (com.alibaba.otter.shared.arbitrate.model.TerminEventData)32 Test (org.testng.annotations.Test)12 PermitMonitor (com.alibaba.otter.shared.arbitrate.impl.setl.monitor.PermitMonitor)9 BaseArbitrateEventTest (com.alibaba.otter.shared.arbitrate.setl.event.BaseArbitrateEventTest)9 ArrayList (java.util.ArrayList)7 EtlEventData (com.alibaba.otter.shared.arbitrate.model.EtlEventData)6 Mock (mockit.Mock)5 BaseEventTest (com.alibaba.otter.shared.arbitrate.BaseEventTest)3 ArbitrateException (com.alibaba.otter.shared.arbitrate.exception.ArbitrateException)3 Date (java.util.Date)3 ExtractMemoryArbitrateEvent (com.alibaba.otter.shared.arbitrate.impl.setl.memory.ExtractMemoryArbitrateEvent)2 LoadMemoryArbitrateEvent (com.alibaba.otter.shared.arbitrate.impl.setl.memory.LoadMemoryArbitrateEvent)2 SelectMemoryArbitrateEvent (com.alibaba.otter.shared.arbitrate.impl.setl.memory.SelectMemoryArbitrateEvent)2 TransformMemoryArbitrateEvent (com.alibaba.otter.shared.arbitrate.impl.setl.memory.TransformMemoryArbitrateEvent)2 ExtractZooKeeperArbitrateEvent (com.alibaba.otter.shared.arbitrate.impl.setl.zookeeper.ExtractZooKeeperArbitrateEvent)2 LoadZooKeeperArbitrateEvent (com.alibaba.otter.shared.arbitrate.impl.setl.zookeeper.LoadZooKeeperArbitrateEvent)2 SelectZooKeeperArbitrateEvent (com.alibaba.otter.shared.arbitrate.impl.setl.zookeeper.SelectZooKeeperArbitrateEvent)2 TransformZooKeeperArbitrateEvent (com.alibaba.otter.shared.arbitrate.impl.setl.zookeeper.TransformZooKeeperArbitrateEvent)2 IOException (java.io.IOException)2 SelectException (com.alibaba.otter.node.etl.select.exceptions.SelectException)1