Search in sources :

Example 1 with KeyValueSource

use of com.hazelcast.mapreduce.KeyValueSource in project hazelcast by hazelcast.

the class AbstractMapReduceTask method startSupervisionTask.

private void startSupervisionTask(JobTracker jobTracker) {
    final MapReduceService mapReduceService = getService(MapReduceService.SERVICE_NAME);
    final JobTrackerConfig config = ((AbstractJobTracker) jobTracker).getJobTrackerConfig();
    final boolean communicateStats = config.isCommunicateStats();
    final int chunkSize = getChunkSizeOrConfigChunkSize(config);
    final TopologyChangedStrategy topologyChangedStrategy = getTopologyChangedStrategyOrConfigTopologyChangedStrategy(config);
    final String name = getDistributedObjectName();
    final String jobId = getJobId();
    final KeyValueSource keyValueSource = getKeyValueSource();
    final Mapper mapper = getMapper();
    final CombinerFactory combinerFactory = getCombinerFactory();
    final ReducerFactory reducerFactory = getReducerFactory();
    final Collection keys = getKeys();
    final Collection<Object> keyObjects = getKeyObjects(keys);
    final KeyPredicate predicate = getPredicate();
    final ClusterService clusterService = nodeEngine.getClusterService();
    for (Member member : clusterService.getMembers(KeyValueJobOperation.MEMBER_SELECTOR)) {
        Operation operation = new KeyValueJobOperation(name, jobId, chunkSize, keyValueSource, mapper, combinerFactory, reducerFactory, communicateStats, topologyChangedStrategy);
        executeOperation(operation, member.getAddress(), mapReduceService, nodeEngine);
    }
    // After we prepared all the remote systems we can now start the processing
    for (Member member : clusterService.getMembers(DATA_MEMBER_SELECTOR)) {
        Operation operation = new StartProcessingJobOperation(name, jobId, keyObjects, predicate);
        executeOperation(operation, member.getAddress(), mapReduceService, nodeEngine);
    }
}
Also used : JobTrackerConfig(com.hazelcast.config.JobTrackerConfig) KeyValueJobOperation(com.hazelcast.mapreduce.impl.operation.KeyValueJobOperation) ReducerFactory(com.hazelcast.mapreduce.ReducerFactory) AbstractJobTracker(com.hazelcast.mapreduce.impl.AbstractJobTracker) StartProcessingJobOperation(com.hazelcast.mapreduce.impl.operation.StartProcessingJobOperation) Operation(com.hazelcast.spi.Operation) MapReduceUtil.executeOperation(com.hazelcast.mapreduce.impl.MapReduceUtil.executeOperation) KeyValueJobOperation(com.hazelcast.mapreduce.impl.operation.KeyValueJobOperation) KeyPredicate(com.hazelcast.mapreduce.KeyPredicate) KeyValueSource(com.hazelcast.mapreduce.KeyValueSource) CombinerFactory(com.hazelcast.mapreduce.CombinerFactory) Mapper(com.hazelcast.mapreduce.Mapper) TopologyChangedStrategy(com.hazelcast.mapreduce.TopologyChangedStrategy) ClusterService(com.hazelcast.internal.cluster.ClusterService) MapReduceService(com.hazelcast.mapreduce.impl.MapReduceService) Collection(java.util.Collection) StartProcessingJobOperation(com.hazelcast.mapreduce.impl.operation.StartProcessingJobOperation) Member(com.hazelcast.core.Member)

Aggregations

JobTrackerConfig (com.hazelcast.config.JobTrackerConfig)1 Member (com.hazelcast.core.Member)1 ClusterService (com.hazelcast.internal.cluster.ClusterService)1 CombinerFactory (com.hazelcast.mapreduce.CombinerFactory)1 KeyPredicate (com.hazelcast.mapreduce.KeyPredicate)1 KeyValueSource (com.hazelcast.mapreduce.KeyValueSource)1 Mapper (com.hazelcast.mapreduce.Mapper)1 ReducerFactory (com.hazelcast.mapreduce.ReducerFactory)1 TopologyChangedStrategy (com.hazelcast.mapreduce.TopologyChangedStrategy)1 AbstractJobTracker (com.hazelcast.mapreduce.impl.AbstractJobTracker)1 MapReduceService (com.hazelcast.mapreduce.impl.MapReduceService)1 MapReduceUtil.executeOperation (com.hazelcast.mapreduce.impl.MapReduceUtil.executeOperation)1 KeyValueJobOperation (com.hazelcast.mapreduce.impl.operation.KeyValueJobOperation)1 StartProcessingJobOperation (com.hazelcast.mapreduce.impl.operation.StartProcessingJobOperation)1 Operation (com.hazelcast.spi.Operation)1 Collection (java.util.Collection)1