Search in sources :

Example 11 with JetException

use of com.hazelcast.jet.JetException in project hazelcast by hazelcast.

the class WriteMapP method init.

@Override
public void init(@Nonnull Outbox outbox, @Nonnull Context context) {
    map = instance().getMap(mapName);
    boolean hasCustomSerializers = serializationService instanceof DelegatingSerializationService && ((DelegatingSerializationService) serializationService).hasAddedSerializers();
    boolean hasNearCache = map instanceof NearCachedMapProxyImpl;
    if (hasNearCache && hasCustomSerializers) {
        // See https://github.com/hazelcast/hazelcast-jet/issues/3046
        throw new JetException("Writing into IMap with both near cache and custom serializers not supported");
    }
    if (!hasCustomSerializers) {
        addToBuffer = item -> buffer.add(new SimpleEntry<>(key(item), value(item)));
    } else if (map instanceof MapProxyImpl) {
        PartitioningStrategy<?> partitionStrategy = ((MapProxyImpl<K, V>) map).getPartitionStrategy();
        addToBuffer = item -> {
            Data key = serializationService.toData(key(item), partitionStrategy);
            Data value = serializationService.toData(value(item));
            buffer.add(new SimpleEntry<>(key, value));
        };
    } else if (map instanceof ClientMapProxy) {
        // TODO: add strategy/unify after https://github.com/hazelcast/hazelcast/issues/13950 is fixed
        addToBuffer = item -> {
            Data key = serializationService.toData(key(item));
            Data value = serializationService.toData(value(item));
            buffer.add(new SimpleEntry<>(key, value));
        };
    } else {
        throw new RuntimeException("Unexpected map class: " + map.getClass().getName());
    }
}
Also used : HazelcastInstance(com.hazelcast.core.HazelcastInstance) FunctionEx(com.hazelcast.function.FunctionEx) EdgeConfig(com.hazelcast.jet.config.EdgeConfig) DelegatingSerializationService(com.hazelcast.jet.impl.serialization.DelegatingSerializationService) NearCachedMapProxyImpl(com.hazelcast.map.impl.proxy.NearCachedMapProxyImpl) PartitioningStrategy(com.hazelcast.partition.PartitioningStrategy) Outbox(com.hazelcast.jet.core.Outbox) Data(com.hazelcast.internal.serialization.Data) Processor(com.hazelcast.jet.core.Processor) Collections.singletonList(java.util.Collections.singletonList) JetException(com.hazelcast.jet.JetException) Consumer(java.util.function.Consumer) ArrayMap(com.hazelcast.jet.impl.connector.HazelcastWriters.ArrayMap) MapProxyImpl(com.hazelcast.map.impl.proxy.MapProxyImpl) PermissionsUtil.mapPutPermission(com.hazelcast.security.PermissionsUtil.mapPutPermission) Integer.max(java.lang.Integer.max) List(java.util.List) Permission(java.security.Permission) SerializationService(com.hazelcast.internal.serialization.SerializationService) ClientMapProxy(com.hazelcast.client.impl.proxy.ClientMapProxy) Inbox(com.hazelcast.jet.core.Inbox) Nonnull(javax.annotation.Nonnull) SimpleEntry(java.util.AbstractMap.SimpleEntry) IMap(com.hazelcast.map.IMap) DelegatingSerializationService(com.hazelcast.jet.impl.serialization.DelegatingSerializationService) SimpleEntry(java.util.AbstractMap.SimpleEntry) NearCachedMapProxyImpl(com.hazelcast.map.impl.proxy.NearCachedMapProxyImpl) Data(com.hazelcast.internal.serialization.Data) JetException(com.hazelcast.jet.JetException) ClientMapProxy(com.hazelcast.client.impl.proxy.ClientMapProxy) NearCachedMapProxyImpl(com.hazelcast.map.impl.proxy.NearCachedMapProxyImpl) MapProxyImpl(com.hazelcast.map.impl.proxy.MapProxyImpl) PartitioningStrategy(com.hazelcast.partition.PartitioningStrategy)

Example 12 with JetException

use of com.hazelcast.jet.JetException in project hazelcast by hazelcast.

the class XaSinkProcessorBase method recoverTransaction.

private void recoverTransaction(Xid xid) throws InterruptedException {
    if (!snapshotUtility.usesTransactionLifecycle()) {
        return;
    }
    for (; ; ) {
        try {
            xaResource.commit(xid, false);
            context.logger().info("Successfully committed restored transaction ID: " + xid);
        } catch (XAException e) {
            switch(e.errorCode) {
                case XA_RETRY:
                    LoggingUtil.logFine(context.logger(), "Commit failed with XA_RETRY, will retry in %s ms. XID: %s", COMMIT_RETRY_DELAY_MS, xid);
                    Thread.sleep(COMMIT_RETRY_DELAY_MS);
                    LoggingUtil.logFine(context.logger(), "Retrying commit %s", xid);
                    continue;
                case XAException.XA_HEURCOM:
                    context.logger().info("Due to a heuristic decision, the work done on behalf of " + "the specified transaction branch was already committed. Transaction ID: " + xid);
                    break;
                case XAException.XA_HEURRB:
                    context.logger().warning("Due to a heuristic decision, the work done on behalf of the restored " + "transaction ID was rolled back. Messages written in that transaction are lost. " + "Ignoring the problem and will continue the job. Transaction ID: " + xid, handleXAException(e, xid));
                    break;
                case XAException.XAER_NOTA:
                    LoggingUtil.logFine(context.logger(), "Failed to commit XID restored from snapshot: The " + "specified XID is not known to the resource manager. This happens normally when the " + "transaction was committed in phase 2 of the snapshot and can be ignored, but can " + "happen also if the transaction wasn't committed in phase 2 and the RM lost it (in " + "this case data written in it is lost). Transaction ID: %s", xid);
                    break;
                default:
                    throw new JetException("Failed to commit XID restored from the snapshot, XA error code: " + e.errorCode + ". Data loss is possible. Transaction ID: " + xid + ", cause: " + e, handleXAException(e, xid));
            }
        }
        break;
    }
}
Also used : XAException(javax.transaction.xa.XAException) JetException(com.hazelcast.jet.JetException)

Example 13 with JetException

use of com.hazelcast.jet.JetException in project hazelcast by hazelcast.

the class StreamEventJournalP method init.

@Override
protected void init(@Nonnull Context context) throws Exception {
    @SuppressWarnings("unchecked") CompletableFuture<EventJournalInitialSubscriberState>[] futures = new CompletableFuture[partitionIds.length];
    Arrays.setAll(futures, i -> eventJournalReader.subscribeToEventJournal(partitionIds[i]));
    for (int i = 0; i < futures.length; i++) {
        emitOffsets[i] = readOffsets[i] = getSequence(futures[i].get());
    }
    if (!isRemoteReader) {
        // try to serde projection/predicate to fail fast if they aren't known to IMDG
        HazelcastInstanceImpl hzInstance = Util.getHazelcastInstanceImpl(context.hazelcastInstance());
        InternalSerializationService ss = hzInstance.getSerializationService();
        try {
            deserializeWithCustomClassLoader(ss, hzInstance.getClass().getClassLoader(), ss.toData(predicate));
            deserializeWithCustomClassLoader(ss, hzInstance.getClass().getClassLoader(), ss.toData(projection));
        } catch (HazelcastSerializationException e) {
            throw new JetException("The projection or predicate classes are not known to IMDG. It's not enough to " + "add them to the job class path, they must be deployed using User code deployment: " + e, e);
        }
    }
}
Also used : HazelcastInstanceImpl(com.hazelcast.instance.impl.HazelcastInstanceImpl) CompletableFuture(java.util.concurrent.CompletableFuture) HazelcastSerializationException(com.hazelcast.nio.serialization.HazelcastSerializationException) InternalSerializationService(com.hazelcast.internal.serialization.InternalSerializationService) JetException(com.hazelcast.jet.JetException)

Example 14 with JetException

use of com.hazelcast.jet.JetException in project hazelcast by hazelcast.

the class WriteBufferedP method init.

@Override
public void init(@Nonnull Outbox outbox, @Nonnull Context context) {
    PermissionsUtil.checkPermission(createFn, context);
    B localBuff = createFn.apply(context);
    if (localBuff == null) {
        throw new JetException("Null buffer created");
    }
    ManagedContext managedContext = context.managedContext();
    buffer = (B) managedContext.initialize(localBuff);
}
Also used : ManagedContext(com.hazelcast.core.ManagedContext) JetException(com.hazelcast.jet.JetException)

Example 15 with JetException

use of com.hazelcast.jet.JetException in project hazelcast by hazelcast.

the class StoreSnapshotTasklet method stateMachineStep.

private void stateMachineStep() {
    switch(state) {
        case DRAIN:
            progTracker.notDone();
            if (pendingEntry != null) {
                if (!ssWriter.offer(pendingEntry)) {
                    return;
                }
                progTracker.madeProgress();
            }
            pendingEntry = null;
            ProgressState result = inboundEdgeStream.drainTo(addToInboxFunction);
            if (result.isDone()) {
                assert ssWriter.isEmpty() : "input is done, but we had some entries and not the barrier";
                snapshotContext.storeSnapshotTaskletDone(pendingSnapshotId - 1, isHigherPrioritySource);
                state = DONE;
                progTracker.reset();
            }
            progTracker.madeProgress(result.isMadeProgress());
            if (hasReachedBarrier) {
                state = FLUSH;
                stateMachineStep();
            }
            break;
        case FLUSH:
            progTracker.notDone();
            if (ssWriter.flushAndResetMap()) {
                progTracker.madeProgress();
                state = REACHED_BARRIER;
            }
            break;
        case REACHED_BARRIER:
            if (ssWriter.hasPendingAsyncOps()) {
                progTracker.notDone();
                return;
            }
            // check for writing error
            Throwable error = ssWriter.getError();
            if (error != null) {
                logger.severe("Error writing to snapshot map", error);
                snapshotContext.reportError(error);
            }
            progTracker.madeProgress();
            long bytes = ssWriter.getTotalPayloadBytes();
            long keys = ssWriter.getTotalKeys();
            long chunks = ssWriter.getTotalChunks();
            snapshotContext.phase1DoneForTasklet(bytes, keys, chunks);
            metrics.set(new LongLongAccumulator(bytes, keys));
            ssWriter.resetStats();
            pendingSnapshotId++;
            hasReachedBarrier = false;
            state = DRAIN;
            progTracker.notDone();
            break;
        default:
            // note State.DONE also goes here
            throw new JetException("Unexpected state: " + state);
    }
}
Also used : LongLongAccumulator(com.hazelcast.jet.accumulator.LongLongAccumulator) ProgressState(com.hazelcast.jet.impl.util.ProgressState) JetException(com.hazelcast.jet.JetException)

Aggregations

JetException (com.hazelcast.jet.JetException)52 IOException (java.io.IOException)8 Nonnull (javax.annotation.Nonnull)8 ILogger (com.hazelcast.logging.ILogger)7 List (java.util.List)7 Map (java.util.Map)6 Util.idToString (com.hazelcast.jet.Util.idToString)5 JobConfig (com.hazelcast.jet.config.JobConfig)5 CompletableFuture (java.util.concurrent.CompletableFuture)5 BroadcastKey (com.hazelcast.jet.core.BroadcastKey)4 JobStatus (com.hazelcast.jet.core.JobStatus)4 Path (java.nio.file.Path)4 HashMap (java.util.HashMap)4 ConcurrentHashMap (java.util.concurrent.ConcurrentHashMap)4 FunctionEx (com.hazelcast.function.FunctionEx)3 Job (com.hazelcast.jet.Job)3 DAG (com.hazelcast.jet.core.DAG)3 Watermark (com.hazelcast.jet.core.Watermark)3 Tuple2 (com.hazelcast.jet.datamodel.Tuple2)3 Pipeline (com.hazelcast.jet.pipeline.Pipeline)3