Search in sources :

Example 26 with IPartitionService

use of com.hazelcast.internal.partition.IPartitionService in project hazelcast by hazelcast.

the class XATransaction method putTransactionInfoRemote.

private void putTransactionInfoRemote() throws ExecutionException, InterruptedException {
    PutRemoteTransactionOperation operation = new PutRemoteTransactionOperation(transactionLog.getRecords(), txnId, xid, txOwnerUuid, timeoutMillis, startTime);
    OperationService operationService = nodeEngine.getOperationService();
    IPartitionService partitionService = nodeEngine.getPartitionService();
    int partitionId = partitionService.getPartitionId(xid);
    InternalCompletableFuture<Object> future = operationService.invokeOnPartition(SERVICE_NAME, operation, partitionId);
    future.get();
}
Also used : IPartitionService(com.hazelcast.internal.partition.IPartitionService) OperationService(com.hazelcast.spi.impl.operationservice.OperationService) PutRemoteTransactionOperation(com.hazelcast.transaction.impl.xa.operations.PutRemoteTransactionOperation)

Example 27 with IPartitionService

use of com.hazelcast.internal.partition.IPartitionService in project hazelcast by hazelcast.

the class ExecutionPlanBuilder method getPartitionAssignment.

/**
 * Assign the partitions to their owners. Partitions whose owner isn't in
 * the {@code memberList}, are assigned to one of the members in a
 * round-robin way.
 */
public static Map<MemberInfo, int[]> getPartitionAssignment(NodeEngine nodeEngine, List<MemberInfo> memberList) {
    IPartitionService partitionService = nodeEngine.getPartitionService();
    Map<Address, MemberInfo> membersByAddress = memberList.stream().collect(toMap(MemberInfo::getAddress, identity()));
    final MemberInfo[] partitionOwners = new MemberInfo[partitionService.getPartitionCount()];
    int memberIndex = 0;
    for (int partitionId = 0; partitionId < partitionOwners.length; partitionId++) {
        Address address = partitionService.getPartitionOwnerOrWait(partitionId);
        MemberInfo member = membersByAddress.get(address);
        if (member == null) {
            // if the partition owner isn't in the current memberList, assign to one of the other members in
            // round-robin fashion
            member = memberList.get(memberIndex++ % memberList.size());
        }
        partitionOwners[partitionId] = member;
    }
    return IntStream.range(0, partitionOwners.length).mapToObj(i -> tuple2(partitionOwners[i], i)).collect(Collectors.groupingBy(Tuple2::f0, Collectors.mapping(Tuple2::f1, Collectors.collectingAndThen(Collectors.<Integer>toList(), intList -> intList.stream().mapToInt(Integer::intValue).toArray()))));
}
Also used : IntStream(java.util.stream.IntStream) Address(com.hazelcast.cluster.Address) EdgeConfig(com.hazelcast.jet.config.EdgeConfig) Util.checkSerializable(com.hazelcast.jet.impl.util.Util.checkSerializable) Util.doWithClassLoader(com.hazelcast.jet.impl.util.Util.doWithClassLoader) HashMap(java.util.HashMap) ExceptionUtil.sneakyThrow(com.hazelcast.jet.impl.util.ExceptionUtil.sneakyThrow) Function(java.util.function.Function) ArrayList(java.util.ArrayList) PrefixedLogger.prefixedLogger(com.hazelcast.jet.impl.util.PrefixedLogger.prefixedLogger) LinkedHashMap(java.util.LinkedHashMap) Collectors.toMap(java.util.stream.Collectors.toMap) ILogger(com.hazelcast.logging.ILogger) MemberInfo(com.hazelcast.internal.cluster.MemberInfo) Map(java.util.Map) DAG(com.hazelcast.jet.core.DAG) Edge(com.hazelcast.jet.core.Edge) ProcessorSupplier(com.hazelcast.jet.core.ProcessorSupplier) Tuple2(com.hazelcast.jet.datamodel.Tuple2) NodeEngineImpl(com.hazelcast.spi.impl.NodeEngineImpl) PrefixedLogger.prefix(com.hazelcast.jet.impl.util.PrefixedLogger.prefix) NodeEngine(com.hazelcast.spi.impl.NodeEngine) Util.toList(com.hazelcast.jet.impl.util.Util.toList) ProcessorMetaSupplier(com.hazelcast.jet.core.ProcessorMetaSupplier) IPartitionService(com.hazelcast.internal.partition.IPartitionService) JobConfig(com.hazelcast.jet.config.JobConfig) MetaSupplierCtx(com.hazelcast.jet.impl.execution.init.Contexts.MetaSupplierCtx) Collectors(java.util.stream.Collectors) Subject(javax.security.auth.Subject) Vertex(com.hazelcast.jet.core.Vertex) List(java.util.List) Tuple2.tuple2(com.hazelcast.jet.datamodel.Tuple2.tuple2) FunctionEx.identity(com.hazelcast.function.FunctionEx.identity) Entry(java.util.Map.Entry) JobClassLoaderService(com.hazelcast.jet.impl.JobClassLoaderService) JetServiceBackend(com.hazelcast.jet.impl.JetServiceBackend) Address(com.hazelcast.cluster.Address) MemberInfo(com.hazelcast.internal.cluster.MemberInfo) IPartitionService(com.hazelcast.internal.partition.IPartitionService) Tuple2(com.hazelcast.jet.datamodel.Tuple2)

Example 28 with IPartitionService

use of com.hazelcast.internal.partition.IPartitionService in project hazelcast by hazelcast.

the class ExecutionPlan method initDag.

private void initDag(InternalSerializationService jobSerializationService) {
    final Map<Integer, VertexDef> vMap = vertices.stream().collect(toMap(VertexDef::vertexId, v -> v));
    for (VertexDef v : vertices) {
        v.inboundEdges().forEach(e -> e.initTransientFields(vMap, v, false));
        v.outboundEdges().forEach(e -> e.initTransientFields(vMap, v, true));
    }
    final IPartitionService partitionService = nodeEngine.getPartitionService();
    vertices.stream().map(VertexDef::outboundEdges).flatMap(List::stream).map(EdgeDef::partitioner).filter(Objects::nonNull).forEach(partitioner -> {
        if (partitioner instanceof SerializationServiceAware) {
            ((SerializationServiceAware) partitioner).setSerializationService(jobSerializationService);
        }
        partitioner.init(object -> partitionService.getPartitionId(jobSerializationService.toData(object)));
    });
}
Also used : Address(com.hazelcast.cluster.Address) ImdgUtil.getMemberConnection(com.hazelcast.jet.impl.util.ImdgUtil.getMemberConnection) Arrays(java.util.Arrays) SnapshotContext(com.hazelcast.jet.impl.execution.SnapshotContext) Collections.unmodifiableList(java.util.Collections.unmodifiableList) ConcurrentConveyor.concurrentConveyor(com.hazelcast.internal.util.concurrent.ConcurrentConveyor.concurrentConveyor) Processor(com.hazelcast.jet.core.Processor) OutboundCollector.compositeCollector(com.hazelcast.jet.impl.execution.OutboundCollector.compositeCollector) ObjectWithPartitionId(com.hazelcast.jet.impl.util.ObjectWithPartitionId) ProcessorTasklet(com.hazelcast.jet.impl.execution.ProcessorTasklet) ImdgUtil(com.hazelcast.jet.impl.util.ImdgUtil) Collectors.toMap(java.util.stream.Collectors.toMap) ConcurrentConveyor(com.hazelcast.internal.util.concurrent.ConcurrentConveyor) Map(java.util.Map) Util.memoize(com.hazelcast.jet.impl.util.Util.memoize) SerializationServiceAware(com.hazelcast.internal.serialization.SerializationServiceAware) DISTRIBUTE_TO_ALL(com.hazelcast.jet.core.Edge.DISTRIBUTE_TO_ALL) ObjectDataInput(com.hazelcast.nio.ObjectDataInput) TASKLET_INIT_CLOSE_EXECUTOR_NAME(com.hazelcast.jet.impl.execution.TaskletExecutionService.TASKLET_INIT_CLOSE_EXECUTOR_NAME) InboundEdgeStream(com.hazelcast.jet.impl.execution.InboundEdgeStream) PrefixedLogger.prefix(com.hazelcast.jet.impl.util.PrefixedLogger.prefix) Collection(java.util.Collection) IPartitionService(com.hazelcast.internal.partition.IPartitionService) JobConfig(com.hazelcast.jet.config.JobConfig) ConcurrentHashMap(java.util.concurrent.ConcurrentHashMap) Set(java.util.Set) ConcurrentInboundEdgeStream(com.hazelcast.jet.impl.execution.ConcurrentInboundEdgeStream) Collectors(java.util.stream.Collectors) Objects(java.util.Objects) List(java.util.List) Stream(java.util.stream.Stream) DEFAULT_QUEUE_SIZE(com.hazelcast.jet.config.EdgeConfig.DEFAULT_QUEUE_SIZE) StoreSnapshotTasklet(com.hazelcast.jet.impl.execution.StoreSnapshotTasklet) ObjectDataOutput(com.hazelcast.nio.ObjectDataOutput) TopologyChangedException(com.hazelcast.jet.core.TopologyChangedException) IntStream(java.util.stream.IntStream) ComparatorEx(com.hazelcast.function.ComparatorEx) IdentifiedDataSerializable(com.hazelcast.nio.serialization.IdentifiedDataSerializable) ImdgUtil.writeList(com.hazelcast.jet.impl.util.ImdgUtil.writeList) OutboundEdgeStream(com.hazelcast.jet.impl.execution.OutboundEdgeStream) RoutingPolicy(com.hazelcast.jet.core.Edge.RoutingPolicy) Util.doWithClassLoader(com.hazelcast.jet.impl.util.Util.doWithClassLoader) SenderTasklet(com.hazelcast.jet.impl.execution.SenderTasklet) HashMap(java.util.HashMap) Supplier(java.util.function.Supplier) ProcSupplierCtx(com.hazelcast.jet.impl.execution.init.Contexts.ProcSupplierCtx) ArrayList(java.util.ArrayList) PrefixedLogger.prefixedLogger(com.hazelcast.jet.impl.util.PrefixedLogger.prefixedLogger) JetException(com.hazelcast.jet.JetException) ConveyorCollector(com.hazelcast.jet.impl.execution.ConveyorCollector) ReceiverTasklet(com.hazelcast.jet.impl.execution.ReceiverTasklet) ILogger(com.hazelcast.logging.ILogger) InternalSerializationService(com.hazelcast.internal.serialization.InternalSerializationService) Nonnull(javax.annotation.Nonnull) ProcessorSupplier(com.hazelcast.jet.core.ProcessorSupplier) QueuedPipe(com.hazelcast.internal.util.concurrent.QueuedPipe) IntFunction(java.util.function.IntFunction) JetConfig(com.hazelcast.jet.config.JetConfig) NodeEngineImpl(com.hazelcast.spi.impl.NodeEngineImpl) OneToOneConcurrentArrayQueue(com.hazelcast.internal.util.concurrent.OneToOneConcurrentArrayQueue) Connection(com.hazelcast.internal.nio.Connection) Tasklet(com.hazelcast.jet.impl.execution.Tasklet) ProcCtx(com.hazelcast.jet.impl.execution.init.Contexts.ProcCtx) AsyncSnapshotWriterImpl(com.hazelcast.jet.impl.util.AsyncSnapshotWriterImpl) IOException(java.io.IOException) ConveyorCollectorWithPartition(com.hazelcast.jet.impl.execution.ConveyorCollectorWithPartition) Subject(javax.security.auth.Subject) File(java.io.File) ImdgUtil.readList(com.hazelcast.jet.impl.util.ImdgUtil.readList) Collectors.toList(java.util.stream.Collectors.toList) OutboundCollector(com.hazelcast.jet.impl.execution.OutboundCollector) JobClassLoaderService(com.hazelcast.jet.impl.JobClassLoaderService) ProcessingGuarantee(com.hazelcast.jet.config.ProcessingGuarantee) JetServiceBackend(com.hazelcast.jet.impl.JetServiceBackend) IPartitionService(com.hazelcast.internal.partition.IPartitionService) SerializationServiceAware(com.hazelcast.internal.serialization.SerializationServiceAware)

Example 29 with IPartitionService

use of com.hazelcast.internal.partition.IPartitionService in project hazelcast by hazelcast.

the class ScheduledExecutorServiceProxy method accumulateTaskHandlersAsScheduledFutures.

@SuppressWarnings("unchecked")
private <V> void accumulateTaskHandlersAsScheduledFutures(Map<Member, List<IScheduledFuture<V>>> accumulator, Map<?, ?> taskHandlersMap) {
    ClusterService clusterService = getNodeEngine().getClusterService();
    IPartitionService partitionService = getNodeEngine().getPartitionService();
    for (Map.Entry<?, ?> entry : taskHandlersMap.entrySet()) {
        Member owner;
        Object key = entry.getKey();
        if (key instanceof Number) {
            owner = clusterService.getMember(partitionService.getPartitionOwner((Integer) key));
        } else {
            owner = (Member) key;
        }
        List<ScheduledTaskHandler> handlers = (List<ScheduledTaskHandler>) entry.getValue();
        List<IScheduledFuture<V>> futures = new ArrayList<>();
        for (ScheduledTaskHandler handler : handlers) {
            IScheduledFuture future = new ScheduledFutureProxy(handler, this);
            futures.add(initializeManagedContext(future));
        }
        if (accumulator.containsKey(owner)) {
            List<IScheduledFuture<V>> memberFutures = accumulator.get(owner);
            memberFutures.addAll(futures);
        } else {
            accumulator.put(owner, futures);
        }
    }
}
Also used : IPartitionService(com.hazelcast.internal.partition.IPartitionService) ArrayList(java.util.ArrayList) IScheduledFuture(com.hazelcast.scheduledexecutor.IScheduledFuture) ClusterService(com.hazelcast.internal.cluster.ClusterService) ScheduledTaskHandler(com.hazelcast.scheduledexecutor.ScheduledTaskHandler) AbstractDistributedObject(com.hazelcast.spi.impl.AbstractDistributedObject) ArrayList(java.util.ArrayList) LinkedList(java.util.LinkedList) List(java.util.List) MapUtil.createHashMap(com.hazelcast.internal.util.MapUtil.createHashMap) LinkedHashMap(java.util.LinkedHashMap) Map(java.util.Map) Member(com.hazelcast.cluster.Member)

Example 30 with IPartitionService

use of com.hazelcast.internal.partition.IPartitionService in project hazelcast by hazelcast.

the class MapExecuteOnKeysMessageTask method getPartitions.

@Override
public PartitionIdSet getPartitions() {
    IPartitionService partitionService = nodeEngine.getPartitionService();
    int partitions = partitionService.getPartitionCount();
    PartitionIdSet partitionIds = new PartitionIdSet(partitions);
    Iterator<Data> iterator = parameters.keys.iterator();
    int addedPartitions = 0;
    while (iterator.hasNext() && addedPartitions < partitions) {
        Data key = iterator.next();
        if (partitionIds.add(partitionService.getPartitionId(key))) {
            addedPartitions++;
        }
    }
    return partitionIds;
}
Also used : IPartitionService(com.hazelcast.internal.partition.IPartitionService) PartitionIdSet(com.hazelcast.internal.util.collection.PartitionIdSet) Data(com.hazelcast.internal.serialization.Data)

Aggregations

IPartitionService (com.hazelcast.internal.partition.IPartitionService)36 Data (com.hazelcast.internal.serialization.Data)10 NodeEngine (com.hazelcast.spi.impl.NodeEngine)10 Address (com.hazelcast.cluster.Address)8 List (java.util.List)7 RecordStore (com.hazelcast.map.impl.recordstore.RecordStore)6 OperationService (com.hazelcast.spi.impl.operationservice.OperationService)6 ArrayList (java.util.ArrayList)6 Map (java.util.Map)6 MapServiceContext (com.hazelcast.map.impl.MapServiceContext)5 IPartition (com.hazelcast.internal.partition.IPartition)4 PartitionIdSet (com.hazelcast.internal.util.collection.PartitionIdSet)4 MapService (com.hazelcast.map.impl.MapService)4 ClusterService (com.hazelcast.internal.cluster.ClusterService)3 MapUtil.createHashMap (com.hazelcast.internal.util.MapUtil.createHashMap)3 ICacheService (com.hazelcast.cache.impl.ICacheService)2 Connection (com.hazelcast.internal.nio.Connection)2 JobConfig (com.hazelcast.jet.config.JobConfig)2 ProcessorSupplier (com.hazelcast.jet.core.ProcessorSupplier)2 JetServiceBackend (com.hazelcast.jet.impl.JetServiceBackend)2