Search in sources :

Example 1 with OutgoingBatches

use of org.jumpmind.symmetric.model.OutgoingBatches in project symmetric-ds by JumpMind.

the class DataExtractorService method extract.

public List<OutgoingBatch> extract(ProcessInfo processInfo, Node targetNode, String queue, IOutgoingTransport transport) {
    /*
         * make sure that data is routed before extracting if the route job is
         * not configured to start automatically
         */
    if (!parameterService.is(ParameterConstants.START_ROUTE_JOB) && parameterService.is(ParameterConstants.ROUTE_ON_EXTRACT)) {
        routerService.routeData(true);
    }
    OutgoingBatches batches = null;
    if (queue != null) {
        NodeGroupLinkAction defaultAction = configurationService.getNodeGroupLinkFor(nodeService.findIdentity().getNodeGroupId(), targetNode.getNodeGroupId(), false).getDataEventAction();
        ProcessInfoKey.ProcessType processType = processInfo.getKey().getProcessType();
        NodeGroupLinkAction action = null;
        if (processType.equals(ProcessInfoKey.ProcessType.PUSH_JOB)) {
            action = NodeGroupLinkAction.P;
        } else if (processType.equals(ProcessInfoKey.ProcessType.PULL_HANDLER)) {
            action = NodeGroupLinkAction.W;
        }
        batches = outgoingBatchService.getOutgoingBatches(targetNode.getNodeId(), queue, action, defaultAction, false);
    } else {
        batches = outgoingBatchService.getOutgoingBatches(targetNode.getNodeId(), false);
    }
    if (batches.containsBatches()) {
        ChannelMap channelMap = transport.getSuspendIgnoreChannelLists(configurationService, queue, targetNode);
        List<OutgoingBatch> activeBatches = filterBatchesForExtraction(batches, channelMap);
        if (activeBatches.size() > 0) {
            BufferedWriter writer = transport.openWriter();
            IDataWriter dataWriter = new ProtocolDataWriter(nodeService.findIdentityNodeId(), writer, targetNode.requires13Compatiblity());
            return extract(processInfo, targetNode, activeBatches, dataWriter, writer, ExtractMode.FOR_SYM_CLIENT);
        }
    }
    return Collections.emptyList();
}
Also used : ChannelMap(org.jumpmind.symmetric.model.ChannelMap) NodeGroupLinkAction(org.jumpmind.symmetric.model.NodeGroupLinkAction) ProtocolDataWriter(org.jumpmind.symmetric.io.data.writer.ProtocolDataWriter) ProcessType(org.jumpmind.symmetric.model.ProcessInfoKey.ProcessType) ProcessInfoKey(org.jumpmind.symmetric.model.ProcessInfoKey) OutgoingBatches(org.jumpmind.symmetric.model.OutgoingBatches) OutgoingBatch(org.jumpmind.symmetric.model.OutgoingBatch) BufferedWriter(java.io.BufferedWriter) BatchBufferedWriter(org.jumpmind.symmetric.transport.BatchBufferedWriter) IDataWriter(org.jumpmind.symmetric.io.data.IDataWriter)

Example 2 with OutgoingBatches

use of org.jumpmind.symmetric.model.OutgoingBatches in project symmetric-ds by JumpMind.

the class MonitorTypeBatchError method check.

@Override
public long check(Monitor monitor) {
    int outgoingErrorCount = 0;
    OutgoingBatches outgoingBatches = outgoingBatchService.getOutgoingBatchErrors(1000);
    for (OutgoingBatch batch : outgoingBatches.getBatches()) {
        int batchErrorMinutes = (int) (System.currentTimeMillis() - batch.getCreateTime().getTime()) / 60000;
        if (batchErrorMinutes >= monitor.getThreshold()) {
            outgoingErrorCount++;
        }
    }
    int incomingErrorCount = 0;
    List<IncomingBatch> incomingBatches = incomingBatchService.findIncomingBatchErrors(1000);
    for (IncomingBatch batch : incomingBatches) {
        int batchErrorMinutes = (int) (System.currentTimeMillis() - batch.getCreateTime().getTime()) / 60000;
        if (batchErrorMinutes >= monitor.getThreshold()) {
            incomingErrorCount++;
        }
    }
    return outgoingErrorCount + incomingErrorCount;
}
Also used : OutgoingBatches(org.jumpmind.symmetric.model.OutgoingBatches) OutgoingBatch(org.jumpmind.symmetric.model.OutgoingBatch) IBuiltInExtensionPoint(org.jumpmind.extension.IBuiltInExtensionPoint) IncomingBatch(org.jumpmind.symmetric.model.IncomingBatch)

Example 3 with OutgoingBatches

use of org.jumpmind.symmetric.model.OutgoingBatches in project symmetric-ds by JumpMind.

the class OutgoingBatchService method getOutgoingBatchByLoad.

public OutgoingBatches getOutgoingBatchByLoad(long loadId) {
    OutgoingBatches batches = new OutgoingBatches();
    batches.setBatches(sqlTemplate.query(getSql("selectOutgoingBatchPrefixSql", "selectOutgoingBatchLoadSql"), new OutgoingBatchMapper(true), loadId));
    return batches;
}
Also used : OutgoingBatches(org.jumpmind.symmetric.model.OutgoingBatches)

Example 4 with OutgoingBatches

use of org.jumpmind.symmetric.model.OutgoingBatches in project symmetric-ds by JumpMind.

the class DataExtractorService method extract.

public List<OutgoingBatch> extract(ProcessInfo processInfo, Node targetNode, IOutgoingTransport transport) {
    /*
         * make sure that data is routed before extracting if the route job is
         * not configured to start automatically
         */
    if (!parameterService.is(ParameterConstants.START_ROUTE_JOB)) {
        routerService.routeData(true);
    }
    OutgoingBatches batches = outgoingBatchService.getOutgoingBatches(targetNode.getNodeId(), false);
    if (batches.containsBatches()) {
        ChannelMap channelMap = transport.getSuspendIgnoreChannelLists(configurationService, targetNode);
        List<OutgoingBatch> activeBatches = filterBatchesForExtraction(batches, channelMap);
        if (activeBatches.size() > 0) {
            IDataWriter dataWriter = new ProtocolDataWriter(nodeService.findIdentityNodeId(), transport.openWriter(), targetNode.requires13Compatiblity());
            return extract(processInfo, targetNode, activeBatches, dataWriter, ExtractMode.FOR_SYM_CLIENT);
        }
    }
    return Collections.emptyList();
}
Also used : ChannelMap(org.jumpmind.symmetric.model.ChannelMap) ProtocolDataWriter(org.jumpmind.symmetric.io.data.writer.ProtocolDataWriter) OutgoingBatches(org.jumpmind.symmetric.model.OutgoingBatches) OutgoingBatch(org.jumpmind.symmetric.model.OutgoingBatch) IDataWriter(org.jumpmind.symmetric.io.data.IDataWriter)

Example 5 with OutgoingBatches

use of org.jumpmind.symmetric.model.OutgoingBatches in project symmetric-ds by JumpMind.

the class DataExtractorService method extractBatchRange.

public boolean extractBatchRange(Writer writer, String nodeId, Date startBatchTime, Date endBatchTime, String... channelIds) {
    boolean foundBatch = false;
    Node sourceNode = nodeService.findIdentity();
    OutgoingBatches batches = outgoingBatchService.getOutgoingBatchRange(nodeId, startBatchTime, endBatchTime, channelIds);
    List<OutgoingBatch> list = batches.getBatches();
    for (OutgoingBatch outgoingBatch : list) {
        Node targetNode = nodeService.findNode(nodeId);
        if (targetNode == null && Constants.UNROUTED_NODE_ID.equals(nodeId)) {
            targetNode = new Node();
            targetNode.setNodeId("-1");
        }
        if (targetNode != null) {
            IDataReader dataReader = new ExtractDataReader(symmetricDialect.getPlatform(), new SelectFromSymDataSource(outgoingBatch, sourceNode, targetNode, new ProcessInfo()));
            DataContext ctx = new DataContext();
            ctx.put(Constants.DATA_CONTEXT_TARGET_NODE, targetNode);
            ctx.put(Constants.DATA_CONTEXT_SOURCE_NODE, nodeService.findIdentity());
            new DataProcessor(dataReader, createTransformDataWriter(nodeService.findIdentity(), targetNode, new ProtocolDataWriter(nodeService.findIdentityNodeId(), writer, targetNode.requires13Compatiblity())), "extract range").process(ctx);
            foundBatch = true;
        }
    }
    return foundBatch;
}
Also used : IDataReader(org.jumpmind.symmetric.io.data.IDataReader) DataContext(org.jumpmind.symmetric.io.data.DataContext) ProtocolDataWriter(org.jumpmind.symmetric.io.data.writer.ProtocolDataWriter) Node(org.jumpmind.symmetric.model.Node) OutgoingBatches(org.jumpmind.symmetric.model.OutgoingBatches) OutgoingBatch(org.jumpmind.symmetric.model.OutgoingBatch) ProcessInfo(org.jumpmind.symmetric.model.ProcessInfo) DataProcessor(org.jumpmind.symmetric.io.data.DataProcessor) ExtractDataReader(org.jumpmind.symmetric.io.data.reader.ExtractDataReader)

Aggregations

OutgoingBatches (org.jumpmind.symmetric.model.OutgoingBatches)34 OutgoingBatch (org.jumpmind.symmetric.model.OutgoingBatch)15 NodeChannel (org.jumpmind.symmetric.model.NodeChannel)14 IOutgoingBatchService (org.jumpmind.symmetric.service.IOutgoingBatchService)8 TriggerRouter (org.jumpmind.symmetric.model.TriggerRouter)7 ArrayList (java.util.ArrayList)6 Test (org.junit.Test)6 Date (java.util.Date)5 IConfigurationService (org.jumpmind.symmetric.service.IConfigurationService)5 FixMethodOrder (org.junit.FixMethodOrder)5 File (java.io.File)3 ProtocolDataWriter (org.jumpmind.symmetric.io.data.writer.ProtocolDataWriter)3 ChannelMap (org.jumpmind.symmetric.model.ChannelMap)3 BigDecimal (java.math.BigDecimal)2 List (java.util.List)2 IOutgoingBatchFilter (org.jumpmind.symmetric.ext.IOutgoingBatchFilter)2 IDataWriter (org.jumpmind.symmetric.io.data.IDataWriter)2 Channel (org.jumpmind.symmetric.model.Channel)2 FileTriggerRouter (org.jumpmind.symmetric.model.FileTriggerRouter)2 IFileSyncService (org.jumpmind.symmetric.service.IFileSyncService)2