Search in sources :

Example 6 with OutputCollector

use of org.apache.heron.api.bolt.OutputCollector in project heron by twitter.

the class JoinOperatorTest method getJoinOperator.

@SuppressWarnings({ "rawtypes", "unchecked" })
private JoinOperator<String, KeyValue<String, String>, KeyValue<String, String>, String> getJoinOperator(JoinType type) {
    SerializableFunction<KeyValue<String, String>, String> f = x -> x == null ? "null" : x.getKey();
    JoinOperator<String, KeyValue<String, String>, KeyValue<String, String>, String> joinOperator = new JoinOperator(type, "leftComponent", "rightComponent", f, f, (SerializableBiFunction<KeyValue<String, String>, KeyValue<String, String>, String>) (o, o2) -> (o == null ? "null" : o.getValue()) + (o2 == null ? "null" : o2.getValue()));
    joinOperator.prepare(new Config(), PowerMockito.mock(TopologyContext.class), new OutputCollector(new IOutputCollector() {

        @Override
        public void reportError(Throwable error) {
        }

        @Override
        public List<Integer> emit(String streamId, Collection<Tuple> anchors, List<Object> tuple) {
            emittedTuples.addAll(tuple);
            return null;
        }

        @Override
        public void emitDirect(int taskId, String streamId, Collection<Tuple> anchors, List<Object> tuple) {
        }

        @Override
        public void ack(Tuple input) {
        }

        @Override
        public void fail(Tuple input) {
        }
    }));
    return joinOperator;
}
Also used : TopologyContext(org.apache.heron.api.topology.TopologyContext) OutputCollector(org.apache.heron.api.bolt.OutputCollector) TopologyAPI(org.apache.heron.api.generated.TopologyAPI) JoinType(org.apache.heron.streamlet.JoinType) HashMap(java.util.HashMap) HashSet(java.util.HashSet) SerializableBiFunction(org.apache.heron.streamlet.SerializableBiFunction) Map(java.util.Map) Tuple(org.apache.heron.api.tuple.Tuple) IOutputCollector(org.apache.heron.api.bolt.IOutputCollector) LinkedList(java.util.LinkedList) PowerMockito(org.powermock.api.mockito.PowerMockito) Before(org.junit.Before) KeyValue(org.apache.heron.streamlet.KeyValue) KeyedWindow(org.apache.heron.streamlet.KeyedWindow) Collection(java.util.Collection) Set(java.util.Set) TopologyContextImpl(org.apache.heron.common.utils.topology.TopologyContextImpl) Test(org.junit.Test) TupleImpl(org.apache.heron.common.utils.tuple.TupleImpl) TupleWindowImpl(org.apache.heron.api.windowing.TupleWindowImpl) SerializableFunction(org.apache.heron.streamlet.SerializableFunction) Mockito(org.mockito.Mockito) List(java.util.List) Fields(org.apache.heron.api.tuple.Fields) Config(org.apache.heron.api.Config) Values(org.apache.heron.api.tuple.Values) Assert(org.junit.Assert) TupleWindow(org.apache.heron.api.windowing.TupleWindow) Collections(java.util.Collections) TopologyBuilder(org.apache.heron.api.topology.TopologyBuilder) OutputCollector(org.apache.heron.api.bolt.OutputCollector) IOutputCollector(org.apache.heron.api.bolt.IOutputCollector) KeyValue(org.apache.heron.streamlet.KeyValue) Config(org.apache.heron.api.Config) IOutputCollector(org.apache.heron.api.bolt.IOutputCollector) Collection(java.util.Collection) LinkedList(java.util.LinkedList) List(java.util.List) TopologyContext(org.apache.heron.api.topology.TopologyContext) Tuple(org.apache.heron.api.tuple.Tuple)

Example 7 with OutputCollector

use of org.apache.heron.api.bolt.OutputCollector in project heron by twitter.

the class KeyByOperatorTest method getKeyByOperator.

@SuppressWarnings({ "rawtypes", "unchecked" })
private KeyByOperator<String, String, Integer> getKeyByOperator() {
    KeyByOperator<String, String, Integer> keyByOperator = new KeyByOperator<String, String, Integer>(x -> (Integer.valueOf(x) % 2 == 0) ? "even" : "odd", x -> Integer.valueOf(x));
    keyByOperator.prepare(new Config(), PowerMockito.mock(TopologyContext.class), new OutputCollector(new IOutputCollector() {

        @Override
        public void reportError(Throwable error) {
        }

        @Override
        public List<Integer> emit(String streamId, Collection<Tuple> anchors, List<Object> tuple) {
            emittedTuples.addAll(tuple);
            return null;
        }

        @Override
        public void emitDirect(int taskId, String streamId, Collection<Tuple> anchors, List<Object> tuple) {
        }

        @Override
        public void ack(Tuple input) {
        }

        @Override
        public void fail(Tuple input) {
        }
    }));
    return keyByOperator;
}
Also used : OutputCollector(org.apache.heron.api.bolt.OutputCollector) IOutputCollector(org.apache.heron.api.bolt.IOutputCollector) Config(org.apache.heron.api.Config) IOutputCollector(org.apache.heron.api.bolt.IOutputCollector) Collection(java.util.Collection) List(java.util.List) LinkedList(java.util.LinkedList) TopologyContext(org.apache.heron.api.topology.TopologyContext) Tuple(org.apache.heron.api.tuple.Tuple)

Example 8 with OutputCollector

use of org.apache.heron.api.bolt.OutputCollector in project incubator-heron by apache.

the class IntegrationTestBolt method prepare.

@Override
public void prepare(Map<String, Object> map, TopologyContext context, OutputCollector outputCollector) {
    update(context);
    this.collector = new OutputCollector(new IntegrationTestBoltCollector(outputCollector));
    this.delegateBolt.prepare(map, new TestTopologyContext(context), collector);
}
Also used : OutputCollector(org.apache.heron.api.bolt.OutputCollector) IOutputCollector(org.apache.heron.api.bolt.IOutputCollector)

Example 9 with OutputCollector

use of org.apache.heron.api.bolt.OutputCollector in project incubator-heron by apache.

the class BoltInstance method init.

@SuppressWarnings("unchecked")
@Override
public void init(State<Serializable, Serializable> state) {
    TopologyContextImpl topologyContext = helper.getTopologyContext();
    // Initialize the GlobalMetrics
    GlobalMetrics.init(topologyContext, systemConfig.getHeronMetricsExportInterval());
    boltMetrics.registerMetrics(topologyContext);
    // Initialize the instanceState if the topology is stateful and bolt is a stateful component
    if (isTopologyStateful && bolt instanceof IStatefulComponent) {
        this.instanceState = state;
        ((IStatefulComponent<Serializable, Serializable>) bolt).initState(instanceState);
        if (spillState) {
            if (FileUtils.isDirectoryExists(spillStateLocation)) {
                FileUtils.cleanDir(spillStateLocation);
            } else {
                FileUtils.createDirectory(spillStateLocation);
            }
        }
    }
    // Delegate
    bolt.prepare(topologyContext.getTopologyConfig(), topologyContext, new OutputCollector(collector));
    // Invoke user-defined prepare task hook
    topologyContext.invokeHookPrepare();
    // Init the CustomStreamGrouping
    helper.prepareForCustomStreamGrouping();
}
Also used : OutputCollector(org.apache.heron.api.bolt.OutputCollector) TopologyContextImpl(org.apache.heron.common.utils.topology.TopologyContextImpl) IStatefulComponent(org.apache.heron.api.topology.IStatefulComponent)

Example 10 with OutputCollector

use of org.apache.heron.api.bolt.OutputCollector in project incubator-heron by apache.

the class JoinOperatorTest method getJoinOperator.

@SuppressWarnings({ "rawtypes", "unchecked" })
private JoinOperator<String, KeyValue<String, String>, KeyValue<String, String>, String> getJoinOperator(JoinType type) {
    SerializableFunction<KeyValue<String, String>, String> f = x -> x == null ? "null" : x.getKey();
    JoinOperator<String, KeyValue<String, String>, KeyValue<String, String>, String> joinOperator = new JoinOperator(type, "leftComponent", "rightComponent", f, f, (SerializableBiFunction<KeyValue<String, String>, KeyValue<String, String>, String>) (o, o2) -> (o == null ? "null" : o.getValue()) + (o2 == null ? "null" : o2.getValue()));
    joinOperator.prepare(new Config(), PowerMockito.mock(TopologyContext.class), new OutputCollector(new IOutputCollector() {

        @Override
        public void reportError(Throwable error) {
        }

        @Override
        public List<Integer> emit(String streamId, Collection<Tuple> anchors, List<Object> tuple) {
            emittedTuples.addAll(tuple);
            return null;
        }

        @Override
        public void emitDirect(int taskId, String streamId, Collection<Tuple> anchors, List<Object> tuple) {
        }

        @Override
        public void ack(Tuple input) {
        }

        @Override
        public void fail(Tuple input) {
        }
    }));
    return joinOperator;
}
Also used : TopologyContext(org.apache.heron.api.topology.TopologyContext) OutputCollector(org.apache.heron.api.bolt.OutputCollector) TopologyAPI(org.apache.heron.api.generated.TopologyAPI) JoinType(org.apache.heron.streamlet.JoinType) HashMap(java.util.HashMap) HashSet(java.util.HashSet) SerializableBiFunction(org.apache.heron.streamlet.SerializableBiFunction) Map(java.util.Map) Tuple(org.apache.heron.api.tuple.Tuple) IOutputCollector(org.apache.heron.api.bolt.IOutputCollector) LinkedList(java.util.LinkedList) PowerMockito(org.powermock.api.mockito.PowerMockito) Before(org.junit.Before) KeyValue(org.apache.heron.streamlet.KeyValue) KeyedWindow(org.apache.heron.streamlet.KeyedWindow) Collection(java.util.Collection) Set(java.util.Set) TopologyContextImpl(org.apache.heron.common.utils.topology.TopologyContextImpl) Test(org.junit.Test) TupleImpl(org.apache.heron.common.utils.tuple.TupleImpl) TupleWindowImpl(org.apache.heron.api.windowing.TupleWindowImpl) SerializableFunction(org.apache.heron.streamlet.SerializableFunction) Mockito(org.mockito.Mockito) List(java.util.List) Fields(org.apache.heron.api.tuple.Fields) Config(org.apache.heron.api.Config) Values(org.apache.heron.api.tuple.Values) Assert(org.junit.Assert) TupleWindow(org.apache.heron.api.windowing.TupleWindow) Collections(java.util.Collections) TopologyBuilder(org.apache.heron.api.topology.TopologyBuilder) OutputCollector(org.apache.heron.api.bolt.OutputCollector) IOutputCollector(org.apache.heron.api.bolt.IOutputCollector) KeyValue(org.apache.heron.streamlet.KeyValue) Config(org.apache.heron.api.Config) IOutputCollector(org.apache.heron.api.bolt.IOutputCollector) Collection(java.util.Collection) LinkedList(java.util.LinkedList) List(java.util.List) TopologyContext(org.apache.heron.api.topology.TopologyContext) Tuple(org.apache.heron.api.tuple.Tuple)

Aggregations

OutputCollector (org.apache.heron.api.bolt.OutputCollector)12 IOutputCollector (org.apache.heron.api.bolt.IOutputCollector)10 Collection (java.util.Collection)8 LinkedList (java.util.LinkedList)8 List (java.util.List)8 Config (org.apache.heron.api.Config)8 TopologyContext (org.apache.heron.api.topology.TopologyContext)8 Tuple (org.apache.heron.api.tuple.Tuple)8 TopologyContextImpl (org.apache.heron.common.utils.topology.TopologyContextImpl)4 KeyValue (org.apache.heron.streamlet.KeyValue)4 Collections (java.util.Collections)2 HashMap (java.util.HashMap)2 HashSet (java.util.HashSet)2 Map (java.util.Map)2 Set (java.util.Set)2 TopologyAPI (org.apache.heron.api.generated.TopologyAPI)2 IStatefulComponent (org.apache.heron.api.topology.IStatefulComponent)2 TopologyBuilder (org.apache.heron.api.topology.TopologyBuilder)2 Fields (org.apache.heron.api.tuple.Fields)2 Values (org.apache.heron.api.tuple.Values)2