Search in sources :

Example 1 with KeyedDirectTLink

use of edu.iu.dsc.tws.tset.links.batch.KeyedDirectTLink in project twister2 by DSC-SPIDAL.

the class KeyedOperationsExample method execute.

@Override
public void execute(WorkerEnvironment workerEnv) {
    BatchEnvironment env = TSetEnvironment.initBatch(workerEnv);
    KeyedSourceTSet<String, Integer> kSource = dummyKeyedSource(env, COUNT, PARALLELISM);
    KeyedDirectTLink<String, Integer> kDirect = kSource.keyedDirect();
    kDirect.forEach(i -> LOG.info("d_fe: " + i.toString()));
    KeyedCachedTSet<String, Integer> cache = kDirect.cache();
    cache.keyedDirect().forEach(i -> LOG.info("c_d_fe: " + i.toString()));
    cache.keyedReduce(Integer::sum).forEach(i -> LOG.info("c_r_fe: " + i.toString()));
    cache.keyedGather().forEach((ApplyFunc<Tuple<String, Iterator<Integer>>>) data -> {
        StringBuilder sb = new StringBuilder();
        sb.append("c_g_fe: key ").append(data.getKey()).append("->");
        while (data.getValue().hasNext()) {
            sb.append(" ").append(data.getValue().next());
        }
        LOG.info(sb.toString());
    });
}
Also used : Tuple(edu.iu.dsc.tws.api.comms.structs.Tuple) Iterator(java.util.Iterator) ResourceAllocator(edu.iu.dsc.tws.rsched.core.ResourceAllocator) BatchEnvironment(edu.iu.dsc.tws.tset.env.BatchEnvironment) HashMap(java.util.HashMap) Config(edu.iu.dsc.tws.api.config.Config) KeyedDirectTLink(edu.iu.dsc.tws.tset.links.batch.KeyedDirectTLink) Logger(java.util.logging.Logger) KeyedCachedTSet(edu.iu.dsc.tws.tset.sets.batch.KeyedCachedTSet) JobConfig(edu.iu.dsc.tws.api.JobConfig) KeyedSourceTSet(edu.iu.dsc.tws.tset.sets.batch.KeyedSourceTSet) WorkerEnvironment(edu.iu.dsc.tws.api.resource.WorkerEnvironment) TSetEnvironment(edu.iu.dsc.tws.tset.env.TSetEnvironment) ApplyFunc(edu.iu.dsc.tws.api.tset.fn.ApplyFunc) BatchEnvironment(edu.iu.dsc.tws.tset.env.BatchEnvironment) Tuple(edu.iu.dsc.tws.api.comms.structs.Tuple)

Aggregations

JobConfig (edu.iu.dsc.tws.api.JobConfig)1 Tuple (edu.iu.dsc.tws.api.comms.structs.Tuple)1 Config (edu.iu.dsc.tws.api.config.Config)1 WorkerEnvironment (edu.iu.dsc.tws.api.resource.WorkerEnvironment)1 ApplyFunc (edu.iu.dsc.tws.api.tset.fn.ApplyFunc)1 ResourceAllocator (edu.iu.dsc.tws.rsched.core.ResourceAllocator)1 BatchEnvironment (edu.iu.dsc.tws.tset.env.BatchEnvironment)1 TSetEnvironment (edu.iu.dsc.tws.tset.env.TSetEnvironment)1 KeyedDirectTLink (edu.iu.dsc.tws.tset.links.batch.KeyedDirectTLink)1 KeyedCachedTSet (edu.iu.dsc.tws.tset.sets.batch.KeyedCachedTSet)1 KeyedSourceTSet (edu.iu.dsc.tws.tset.sets.batch.KeyedSourceTSet)1 HashMap (java.util.HashMap)1 Iterator (java.util.Iterator)1 Logger (java.util.logging.Logger)1