Search in sources :

Example 6 with AbstractJobTracker

use of com.hazelcast.mapreduce.impl.AbstractJobTracker in project hazelcast by hazelcast.

the class KeyValueJob method startSupervisionTask.

private <T> JobCompletableFuture<T> startSupervisionTask(TrackableJobFuture<T> jobFuture, String jobId) {
    AbstractJobTracker jobTracker = (AbstractJobTracker) this.jobTracker;
    JobTrackerConfig config = jobTracker.getJobTrackerConfig();
    boolean communicateStats = config.isCommunicateStats();
    if (chunkSize == -1) {
        chunkSize = config.getChunkSize();
    }
    if (topologyChangedStrategy == null) {
        topologyChangedStrategy = config.getTopologyChangedStrategy();
    }
    ClusterService clusterService = nodeEngine.getClusterService();
    for (Member member : clusterService.getMembers(KeyValueJobOperation.MEMBER_SELECTOR)) {
        Operation operation = new KeyValueJobOperation<KeyIn, ValueIn>(name, jobId, chunkSize, keyValueSource, mapper, combinerFactory, reducerFactory, communicateStats, topologyChangedStrategy);
        executeOperation(operation, member.getAddress(), mapReduceService, nodeEngine);
    }
    // After we prepared all the remote systems we can now start the processing
    for (Member member : clusterService.getMembers(DATA_MEMBER_SELECTOR)) {
        Operation operation = new StartProcessingJobOperation<KeyIn>(name, jobId, keys, predicate);
        executeOperation(operation, member.getAddress(), mapReduceService, nodeEngine);
    }
    return jobFuture;
}
Also used : ClusterService(com.hazelcast.internal.cluster.ClusterService) JobTrackerConfig(com.hazelcast.config.JobTrackerConfig) KeyValueJobOperation(com.hazelcast.mapreduce.impl.operation.KeyValueJobOperation) AbstractJobTracker(com.hazelcast.mapreduce.impl.AbstractJobTracker) StartProcessingJobOperation(com.hazelcast.mapreduce.impl.operation.StartProcessingJobOperation) Operation(com.hazelcast.spi.Operation) StartProcessingJobOperation(com.hazelcast.mapreduce.impl.operation.StartProcessingJobOperation) MapReduceUtil.executeOperation(com.hazelcast.mapreduce.impl.MapReduceUtil.executeOperation) KeyValueJobOperation(com.hazelcast.mapreduce.impl.operation.KeyValueJobOperation) Member(com.hazelcast.core.Member)

Example 7 with AbstractJobTracker

use of com.hazelcast.mapreduce.impl.AbstractJobTracker in project hazelcast by hazelcast.

the class GetResultOperation method run.

@Override
public void run() throws Exception {
    MapReduceService mapReduceService = getService();
    JobSupervisor supervisor = mapReduceService.getJobSupervisor(getName(), getJobId());
    if (supervisor != null) {
        result = supervisor.getJobResults();
        // This is the final call so cleanup on all nodes that are not job owners
        if (!supervisor.isOwnerNode()) {
            mapReduceService.destroyJobSupervisor(supervisor);
            AbstractJobTracker jobTracker = (AbstractJobTracker) mapReduceService.getJobTracker(getName());
            jobTracker.unregisterTrackableJob(getJobId());
            jobTracker.unregisterMapCombineTask(getJobId());
            jobTracker.unregisterReducerTask(getJobId());
        }
    }
}
Also used : MapReduceService(com.hazelcast.mapreduce.impl.MapReduceService) AbstractJobTracker(com.hazelcast.mapreduce.impl.AbstractJobTracker) JobSupervisor(com.hazelcast.mapreduce.impl.task.JobSupervisor)

Aggregations

AbstractJobTracker (com.hazelcast.mapreduce.impl.AbstractJobTracker)7 MapReduceService (com.hazelcast.mapreduce.impl.MapReduceService)5 ClusterService (com.hazelcast.internal.cluster.ClusterService)4 JobSupervisor (com.hazelcast.mapreduce.impl.task.JobSupervisor)3 TrackableJobFuture (com.hazelcast.mapreduce.impl.task.TrackableJobFuture)3 JobTrackerConfig (com.hazelcast.config.JobTrackerConfig)2 Member (com.hazelcast.core.Member)2 MapReduceUtil.executeOperation (com.hazelcast.mapreduce.impl.MapReduceUtil.executeOperation)2 KeyValueJobOperation (com.hazelcast.mapreduce.impl.operation.KeyValueJobOperation)2 StartProcessingJobOperation (com.hazelcast.mapreduce.impl.operation.StartProcessingJobOperation)2 Operation (com.hazelcast.spi.Operation)2 IOException (java.io.IOException)2 CancellationException (java.util.concurrent.CancellationException)2 CombinerFactory (com.hazelcast.mapreduce.CombinerFactory)1 KeyPredicate (com.hazelcast.mapreduce.KeyPredicate)1 KeyValueSource (com.hazelcast.mapreduce.KeyValueSource)1 Mapper (com.hazelcast.mapreduce.Mapper)1 ReducerFactory (com.hazelcast.mapreduce.ReducerFactory)1 TopologyChangedStrategy (com.hazelcast.mapreduce.TopologyChangedStrategy)1 JobTaskConfiguration (com.hazelcast.mapreduce.impl.task.JobTaskConfiguration)1