Search in sources :

Example 1 with StartProcessingJobOperation

use of com.hazelcast.mapreduce.impl.operation.StartProcessingJobOperation in project hazelcast by hazelcast.

the class AbstractMapReduceTask method startSupervisionTask.

private void startSupervisionTask(JobTracker jobTracker) {
    final MapReduceService mapReduceService = getService(MapReduceService.SERVICE_NAME);
    final JobTrackerConfig config = ((AbstractJobTracker) jobTracker).getJobTrackerConfig();
    final boolean communicateStats = config.isCommunicateStats();
    final int chunkSize = getChunkSizeOrConfigChunkSize(config);
    final TopologyChangedStrategy topologyChangedStrategy = getTopologyChangedStrategyOrConfigTopologyChangedStrategy(config);
    final String name = getDistributedObjectName();
    final String jobId = getJobId();
    final KeyValueSource keyValueSource = getKeyValueSource();
    final Mapper mapper = getMapper();
    final CombinerFactory combinerFactory = getCombinerFactory();
    final ReducerFactory reducerFactory = getReducerFactory();
    final Collection keys = getKeys();
    final Collection<Object> keyObjects = getKeyObjects(keys);
    final KeyPredicate predicate = getPredicate();
    final ClusterService clusterService = nodeEngine.getClusterService();
    for (Member member : clusterService.getMembers(KeyValueJobOperation.MEMBER_SELECTOR)) {
        Operation operation = new KeyValueJobOperation(name, jobId, chunkSize, keyValueSource, mapper, combinerFactory, reducerFactory, communicateStats, topologyChangedStrategy);
        executeOperation(operation, member.getAddress(), mapReduceService, nodeEngine);
    }
    // After we prepared all the remote systems we can now start the processing
    for (Member member : clusterService.getMembers(DATA_MEMBER_SELECTOR)) {
        Operation operation = new StartProcessingJobOperation(name, jobId, keyObjects, predicate);
        executeOperation(operation, member.getAddress(), mapReduceService, nodeEngine);
    }
}
Also used : JobTrackerConfig(com.hazelcast.config.JobTrackerConfig) KeyValueJobOperation(com.hazelcast.mapreduce.impl.operation.KeyValueJobOperation) ReducerFactory(com.hazelcast.mapreduce.ReducerFactory) AbstractJobTracker(com.hazelcast.mapreduce.impl.AbstractJobTracker) StartProcessingJobOperation(com.hazelcast.mapreduce.impl.operation.StartProcessingJobOperation) Operation(com.hazelcast.spi.Operation) MapReduceUtil.executeOperation(com.hazelcast.mapreduce.impl.MapReduceUtil.executeOperation) KeyValueJobOperation(com.hazelcast.mapreduce.impl.operation.KeyValueJobOperation) KeyPredicate(com.hazelcast.mapreduce.KeyPredicate) KeyValueSource(com.hazelcast.mapreduce.KeyValueSource) CombinerFactory(com.hazelcast.mapreduce.CombinerFactory) Mapper(com.hazelcast.mapreduce.Mapper) TopologyChangedStrategy(com.hazelcast.mapreduce.TopologyChangedStrategy) ClusterService(com.hazelcast.internal.cluster.ClusterService) MapReduceService(com.hazelcast.mapreduce.impl.MapReduceService) Collection(java.util.Collection) StartProcessingJobOperation(com.hazelcast.mapreduce.impl.operation.StartProcessingJobOperation) Member(com.hazelcast.core.Member)

Example 2 with StartProcessingJobOperation

use of com.hazelcast.mapreduce.impl.operation.StartProcessingJobOperation in project hazelcast by hazelcast.

the class KeyValueJob method startSupervisionTask.

private <T> JobCompletableFuture<T> startSupervisionTask(TrackableJobFuture<T> jobFuture, String jobId) {
    AbstractJobTracker jobTracker = (AbstractJobTracker) this.jobTracker;
    JobTrackerConfig config = jobTracker.getJobTrackerConfig();
    boolean communicateStats = config.isCommunicateStats();
    if (chunkSize == -1) {
        chunkSize = config.getChunkSize();
    }
    if (topologyChangedStrategy == null) {
        topologyChangedStrategy = config.getTopologyChangedStrategy();
    }
    ClusterService clusterService = nodeEngine.getClusterService();
    for (Member member : clusterService.getMembers(KeyValueJobOperation.MEMBER_SELECTOR)) {
        Operation operation = new KeyValueJobOperation<KeyIn, ValueIn>(name, jobId, chunkSize, keyValueSource, mapper, combinerFactory, reducerFactory, communicateStats, topologyChangedStrategy);
        executeOperation(operation, member.getAddress(), mapReduceService, nodeEngine);
    }
    // After we prepared all the remote systems we can now start the processing
    for (Member member : clusterService.getMembers(DATA_MEMBER_SELECTOR)) {
        Operation operation = new StartProcessingJobOperation<KeyIn>(name, jobId, keys, predicate);
        executeOperation(operation, member.getAddress(), mapReduceService, nodeEngine);
    }
    return jobFuture;
}
Also used : ClusterService(com.hazelcast.internal.cluster.ClusterService) JobTrackerConfig(com.hazelcast.config.JobTrackerConfig) KeyValueJobOperation(com.hazelcast.mapreduce.impl.operation.KeyValueJobOperation) AbstractJobTracker(com.hazelcast.mapreduce.impl.AbstractJobTracker) StartProcessingJobOperation(com.hazelcast.mapreduce.impl.operation.StartProcessingJobOperation) Operation(com.hazelcast.spi.Operation) StartProcessingJobOperation(com.hazelcast.mapreduce.impl.operation.StartProcessingJobOperation) MapReduceUtil.executeOperation(com.hazelcast.mapreduce.impl.MapReduceUtil.executeOperation) KeyValueJobOperation(com.hazelcast.mapreduce.impl.operation.KeyValueJobOperation) Member(com.hazelcast.core.Member)

Example 3 with StartProcessingJobOperation

use of com.hazelcast.mapreduce.impl.operation.StartProcessingJobOperation in project hazelcast by hazelcast.

the class MapReduceDataSerializerHook method createFactory.

@Override
public DataSerializableFactory createFactory() {
    ConstructorFunction<Integer, IdentifiedDataSerializable>[] constructors = new ConstructorFunction[LEN];
    constructors[KEY_VALUE_SOURCE_MAP] = new ConstructorFunction<Integer, IdentifiedDataSerializable>() {

        @Override
        public IdentifiedDataSerializable createNew(Integer arg) {
            return new MapKeyValueSource();
        }
    };
    constructors[KEY_VALUE_SOURCE_MULTIMAP] = new ConstructorFunction<Integer, IdentifiedDataSerializable>() {

        @Override
        public IdentifiedDataSerializable createNew(Integer arg) {
            return new MultiMapKeyValueSource();
        }
    };
    constructors[REDUCER_CHUNK_MESSAGE] = new ConstructorFunction<Integer, IdentifiedDataSerializable>() {

        @Override
        public IdentifiedDataSerializable createNew(Integer arg) {
            return new IntermediateChunkNotification();
        }
    };
    constructors[REDUCER_LAST_CHUNK_MESSAGE] = new ConstructorFunction<Integer, IdentifiedDataSerializable>() {

        @Override
        public IdentifiedDataSerializable createNew(Integer arg) {
            return new LastChunkNotification();
        }
    };
    constructors[TRACKED_JOB_OPERATION] = new ConstructorFunction<Integer, IdentifiedDataSerializable>() {

        @Override
        public IdentifiedDataSerializable createNew(Integer arg) {
            return new KeyValueJobOperation();
        }
    };
    constructors[REQUEST_PARTITION_MAPPING] = new ConstructorFunction<Integer, IdentifiedDataSerializable>() {

        @Override
        public IdentifiedDataSerializable createNew(Integer arg) {
            return new RequestPartitionMapping();
        }
    };
    constructors[REQUEST_PARTITION_REDUCING] = new ConstructorFunction<Integer, IdentifiedDataSerializable>() {

        @Override
        public IdentifiedDataSerializable createNew(Integer arg) {
            return new RequestPartitionReducing();
        }
    };
    constructors[REQUEST_PARTITION_PROCESSED] = new ConstructorFunction<Integer, IdentifiedDataSerializable>() {

        @Override
        public IdentifiedDataSerializable createNew(Integer arg) {
            return new RequestPartitionProcessed();
        }
    };
    constructors[GET_RESULT_OPERATION] = new ConstructorFunction<Integer, IdentifiedDataSerializable>() {

        @Override
        public IdentifiedDataSerializable createNew(Integer arg) {
            return new GetResultOperation();
        }
    };
    constructors[START_PROCESSING_OPERATION] = new ConstructorFunction<Integer, IdentifiedDataSerializable>() {

        @Override
        public IdentifiedDataSerializable createNew(Integer arg) {
            return new StartProcessingJobOperation();
        }
    };
    constructors[REQUEST_PARTITION_RESULT] = new ConstructorFunction<Integer, IdentifiedDataSerializable>() {

        @Override
        public IdentifiedDataSerializable createNew(Integer arg) {
            return new RequestPartitionResult();
        }
    };
    constructors[REDUCING_FINISHED_MESSAGE] = new ConstructorFunction<Integer, IdentifiedDataSerializable>() {

        @Override
        public IdentifiedDataSerializable createNew(Integer arg) {
            return new ReducingFinishedNotification();
        }
    };
    constructors[FIRE_NOTIFICATION_OPERATION] = new ConstructorFunction<Integer, IdentifiedDataSerializable>() {

        @Override
        public IdentifiedDataSerializable createNew(Integer arg) {
            return new FireNotificationOperation();
        }
    };
    constructors[REQUEST_MEMBERID_ASSIGNMENT] = new ConstructorFunction<Integer, IdentifiedDataSerializable>() {

        @Override
        public IdentifiedDataSerializable createNew(Integer arg) {
            return new RequestMemberIdAssignment();
        }
    };
    constructors[PROCESS_STATS_UPDATE_OPERATION] = new ConstructorFunction<Integer, IdentifiedDataSerializable>() {

        @Override
        public IdentifiedDataSerializable createNew(Integer arg) {
            return new ProcessStatsUpdateOperation();
        }
    };
    constructors[NOTIFY_REMOTE_EXCEPTION_OPERATION] = new ConstructorFunction<Integer, IdentifiedDataSerializable>() {

        @Override
        public IdentifiedDataSerializable createNew(Integer arg) {
            return new NotifyRemoteExceptionOperation();
        }
    };
    constructors[CANCEL_JOB_SUPERVISOR_OPERATION] = new ConstructorFunction<Integer, IdentifiedDataSerializable>() {

        @Override
        public IdentifiedDataSerializable createNew(Integer arg) {
            return new CancelJobSupervisorOperation();
        }
    };
    constructors[KEY_VALUE_SOURCE_LIST] = new ConstructorFunction<Integer, IdentifiedDataSerializable>() {

        @Override
        public IdentifiedDataSerializable createNew(Integer arg) {
            return new ListKeyValueSource();
        }
    };
    constructors[KEY_VALUE_SOURCE_SET] = new ConstructorFunction<Integer, IdentifiedDataSerializable>() {

        @Override
        public IdentifiedDataSerializable createNew(Integer arg) {
            return new SetKeyValueSource();
        }
    };
    constructors[KEYS_ASSIGNMENT_RESULT] = new ConstructorFunction<Integer, IdentifiedDataSerializable>() {

        @Override
        public IdentifiedDataSerializable createNew(Integer arg) {
            return new KeysAssignmentResult();
        }
    };
    constructors[KEYS_ASSIGNMENT_OPERATION] = new ConstructorFunction<Integer, IdentifiedDataSerializable>() {

        @Override
        public IdentifiedDataSerializable createNew(Integer arg) {
            return new KeysAssignmentOperation();
        }
    };
    constructors[POSTPONE_PARTITION_PROCESSING_OPERATION] = new ConstructorFunction<Integer, IdentifiedDataSerializable>() {

        @Override
        public IdentifiedDataSerializable createNew(Integer arg) {
            return new PostPonePartitionProcessing();
        }
    };
    constructors[HASH_MAP_ADAPTER] = new ConstructorFunction<Integer, IdentifiedDataSerializable>() {

        @Override
        public IdentifiedDataSerializable createNew(Integer arg) {
            return new HashMapAdapter();
        }
    };
    constructors[COMBINER_RESULT_LIST] = new ConstructorFunction<Integer, IdentifiedDataSerializable>() {

        @Override
        public IdentifiedDataSerializable createNew(Integer arg) {
            return new CombinerResultList();
        }
    };
    return new ArrayDataSerializableFactory(constructors);
}
Also used : GetResultOperation(com.hazelcast.mapreduce.impl.operation.GetResultOperation) KeyValueJobOperation(com.hazelcast.mapreduce.impl.operation.KeyValueJobOperation) ProcessStatsUpdateOperation(com.hazelcast.mapreduce.impl.operation.ProcessStatsUpdateOperation) LastChunkNotification(com.hazelcast.mapreduce.impl.notification.LastChunkNotification) KeysAssignmentResult(com.hazelcast.mapreduce.impl.operation.KeysAssignmentResult) ReducingFinishedNotification(com.hazelcast.mapreduce.impl.notification.ReducingFinishedNotification) RequestPartitionProcessed(com.hazelcast.mapreduce.impl.operation.RequestPartitionProcessed) IdentifiedDataSerializable(com.hazelcast.nio.serialization.IdentifiedDataSerializable) KeysAssignmentOperation(com.hazelcast.mapreduce.impl.operation.KeysAssignmentOperation) PostPonePartitionProcessing(com.hazelcast.mapreduce.impl.operation.PostPonePartitionProcessing) IntermediateChunkNotification(com.hazelcast.mapreduce.impl.notification.IntermediateChunkNotification) ConstructorFunction(com.hazelcast.util.ConstructorFunction) RequestPartitionReducing(com.hazelcast.mapreduce.impl.operation.RequestPartitionReducing) FireNotificationOperation(com.hazelcast.mapreduce.impl.operation.FireNotificationOperation) RequestPartitionResult(com.hazelcast.mapreduce.impl.operation.RequestPartitionResult) RequestPartitionMapping(com.hazelcast.mapreduce.impl.operation.RequestPartitionMapping) StartProcessingJobOperation(com.hazelcast.mapreduce.impl.operation.StartProcessingJobOperation) RequestMemberIdAssignment(com.hazelcast.mapreduce.impl.operation.RequestMemberIdAssignment) ArrayDataSerializableFactory(com.hazelcast.internal.serialization.impl.ArrayDataSerializableFactory) CancelJobSupervisorOperation(com.hazelcast.mapreduce.impl.operation.CancelJobSupervisorOperation) NotifyRemoteExceptionOperation(com.hazelcast.mapreduce.impl.operation.NotifyRemoteExceptionOperation)

Aggregations

KeyValueJobOperation (com.hazelcast.mapreduce.impl.operation.KeyValueJobOperation)3 StartProcessingJobOperation (com.hazelcast.mapreduce.impl.operation.StartProcessingJobOperation)3 JobTrackerConfig (com.hazelcast.config.JobTrackerConfig)2 Member (com.hazelcast.core.Member)2 ClusterService (com.hazelcast.internal.cluster.ClusterService)2 AbstractJobTracker (com.hazelcast.mapreduce.impl.AbstractJobTracker)2 MapReduceUtil.executeOperation (com.hazelcast.mapreduce.impl.MapReduceUtil.executeOperation)2 Operation (com.hazelcast.spi.Operation)2 ArrayDataSerializableFactory (com.hazelcast.internal.serialization.impl.ArrayDataSerializableFactory)1 CombinerFactory (com.hazelcast.mapreduce.CombinerFactory)1 KeyPredicate (com.hazelcast.mapreduce.KeyPredicate)1 KeyValueSource (com.hazelcast.mapreduce.KeyValueSource)1 Mapper (com.hazelcast.mapreduce.Mapper)1 ReducerFactory (com.hazelcast.mapreduce.ReducerFactory)1 TopologyChangedStrategy (com.hazelcast.mapreduce.TopologyChangedStrategy)1 MapReduceService (com.hazelcast.mapreduce.impl.MapReduceService)1 IntermediateChunkNotification (com.hazelcast.mapreduce.impl.notification.IntermediateChunkNotification)1 LastChunkNotification (com.hazelcast.mapreduce.impl.notification.LastChunkNotification)1 ReducingFinishedNotification (com.hazelcast.mapreduce.impl.notification.ReducingFinishedNotification)1 CancelJobSupervisorOperation (com.hazelcast.mapreduce.impl.operation.CancelJobSupervisorOperation)1