Search in sources :

Example 6 with EcoTopologyDefinition

use of org.apache.heron.eco.definition.EcoTopologyDefinition in project heron by twitter.

the class Eco method printTopologyInfo.

static void printTopologyInfo(EcoExecutionContext ctx) {
    EcoTopologyDefinition t = ctx.getTopologyDefinition();
    LOG.info("---------- TOPOLOGY DETAILS ----------");
    LOG.info(String.format("Topology Name: %s", t.getName()));
    LOG.info("--------------- SPOUTS ---------------");
    for (SpoutDefinition s : t.getSpouts()) {
        LOG.info(String.format("%s [%d] (%s)", s.getId(), s.getParallelism(), s.getClassName()));
    }
    LOG.info("---------------- BOLTS ---------------");
    for (BoltDefinition b : t.getBolts()) {
        LOG.info(String.format("%s [%d] (%s)", b.getId(), b.getParallelism(), b.getClassName()));
    }
    LOG.info("--------------- STREAMS ---------------");
    for (StreamDefinition sd : t.getStreams()) {
        LOG.info(String.format("%s --%s--> %s", sd.getFrom(), sd.getGrouping().getType(), sd.getTo()));
    }
    LOG.info("--------------------------------------");
}
Also used : BoltDefinition(org.apache.heron.eco.definition.BoltDefinition) StreamDefinition(org.apache.heron.eco.definition.StreamDefinition) EcoTopologyDefinition(org.apache.heron.eco.definition.EcoTopologyDefinition) SpoutDefinition(org.apache.heron.eco.definition.SpoutDefinition)

Example 7 with EcoTopologyDefinition

use of org.apache.heron.eco.definition.EcoTopologyDefinition in project heron by twitter.

the class StreamBuilder method buildStreams.

protected <K extends Serializable, V extends Serializable> void buildStreams(EcoExecutionContext executionContext, TopologyBuilder builder, ObjectBuilder objectBuilder) throws IllegalAccessException, InstantiationException, ClassNotFoundException, NoSuchFieldException, InvocationTargetException {
    EcoTopologyDefinition topologyDefinition = executionContext.getTopologyDefinition();
    Map<String, ComponentStream> componentStreams = new HashMap<>();
    HashMap<String, BoltDeclarer> declarers = new HashMap<>();
    for (StreamDefinition stream : topologyDefinition.getStreams()) {
        Object boltObj = executionContext.getBolt(stream.getTo());
        BoltDeclarer declarer = declarers.get(stream.getTo());
        if (boltObj instanceof IRichBolt) {
            if (declarer == null) {
                declarer = builder.setBolt(stream.getTo(), (IRichBolt) boltObj, topologyDefinition.parallelismForBolt(stream.getTo()));
                declarers.put(stream.getTo(), declarer);
            }
        } else if (boltObj instanceof IBasicBolt) {
            if (declarer == null) {
                declarer = builder.setBolt(stream.getTo(), (IBasicBolt) boltObj, topologyDefinition.parallelismForBolt(stream.getTo()));
                declarers.put(stream.getTo(), declarer);
            }
        } else if (boltObj instanceof IStatefulWindowedBolt) {
            if (declarer == null) {
                // noinspection unchecked
                declarer = builder.setBolt(stream.getTo(), (IStatefulWindowedBolt<K, V>) boltObj, topologyDefinition.parallelismForBolt(stream.getTo()));
                declarers.put(stream.getTo(), declarer);
            }
        } else if (boltObj instanceof IWindowedBolt) {
            if (declarer == null) {
                declarer = builder.setBolt(stream.getTo(), (IWindowedBolt) boltObj, topologyDefinition.parallelismForBolt(stream.getTo()));
                declarers.put(stream.getTo(), declarer);
            }
        } else {
            throw new IllegalArgumentException("Class does not appear to be a bolt: " + boltObj.getClass().getName());
        }
        GroupingDefinition grouping = stream.getGrouping();
        // if the streamId is defined, use it for the grouping,
        // otherwise assume default stream
        String streamId = grouping.getStreamId() == null ? Utils.DEFAULT_STREAM_ID : grouping.getStreamId();
        switch(grouping.getType()) {
            case SHUFFLE:
                declarer.shuffleGrouping(stream.getFrom(), streamId);
                break;
            case FIELDS:
                List<String> groupingArgs = grouping.getArgs();
                if (groupingArgs == null) {
                    throw new IllegalArgumentException("You must supply arguments for Fields grouping");
                }
                declarer.fieldsGrouping(stream.getFrom(), streamId, new Fields(groupingArgs));
                break;
            case ALL:
                declarer.allGrouping(stream.getFrom(), streamId);
                break;
            case GLOBAL:
                declarer.globalGrouping(stream.getFrom(), streamId);
                break;
            case NONE:
                declarer.noneGrouping(stream.getFrom(), streamId);
                break;
            case CUSTOM:
                declarer.customGrouping(stream.getFrom(), streamId, buildCustomStreamGrouping(stream.getGrouping().getCustomClass(), executionContext, objectBuilder));
                break;
            default:
                throw new UnsupportedOperationException("unsupported grouping type: " + grouping);
        }
    }
    executionContext.setStreams(componentStreams);
}
Also used : IRichBolt(org.apache.heron.api.bolt.IRichBolt) ComponentStream(org.apache.heron.eco.definition.ComponentStream) StreamDefinition(org.apache.heron.eco.definition.StreamDefinition) HashMap(java.util.HashMap) IStatefulWindowedBolt(org.apache.heron.api.bolt.IStatefulWindowedBolt) EcoTopologyDefinition(org.apache.heron.eco.definition.EcoTopologyDefinition) GroupingDefinition(org.apache.heron.eco.definition.GroupingDefinition) IBasicBolt(org.apache.heron.api.bolt.IBasicBolt) Fields(org.apache.heron.api.tuple.Fields) BoltDeclarer(org.apache.heron.api.topology.BoltDeclarer) IWindowedBolt(org.apache.heron.api.bolt.IWindowedBolt)

Example 8 with EcoTopologyDefinition

use of org.apache.heron.eco.definition.EcoTopologyDefinition in project heron by twitter.

the class SpoutBuilder method buildSpouts.

protected void buildSpouts(EcoExecutionContext executionContext, TopologyBuilder builder, ObjectBuilder objectBuilder) throws ClassNotFoundException, InstantiationException, IllegalAccessException, NoSuchFieldException, InvocationTargetException {
    EcoTopologyDefinition topologyDefinition = executionContext.getTopologyDefinition();
    for (ObjectDefinition def : topologyDefinition.getSpouts()) {
        Object obj = objectBuilder.buildObject(def, executionContext);
        builder.setSpout(def.getId(), (IRichSpout) obj, def.getParallelism());
        executionContext.addSpout(def.getId(), obj);
    }
}
Also used : ObjectDefinition(org.apache.heron.eco.definition.ObjectDefinition) EcoTopologyDefinition(org.apache.heron.eco.definition.EcoTopologyDefinition)

Example 9 with EcoTopologyDefinition

use of org.apache.heron.eco.definition.EcoTopologyDefinition in project heron by twitter.

the class StreamBuilder method buildStreams.

protected void buildStreams(EcoExecutionContext executionContext, TopologyBuilder builder, ObjectBuilder objectBuilder) throws IllegalAccessException, InstantiationException, ClassNotFoundException, NoSuchFieldException, InvocationTargetException {
    EcoTopologyDefinition topologyDefinition = executionContext.getTopologyDefinition();
    Map<String, ComponentStream> componentStreams = new HashMap<>();
    HashMap<String, BoltDeclarer> declarers = new HashMap<>();
    for (StreamDefinition stream : topologyDefinition.getStreams()) {
        Object boltObj = executionContext.getBolt(stream.getTo());
        BoltDeclarer declarer = declarers.get(stream.getTo());
        if (boltObj instanceof IRichBolt) {
            if (declarer == null) {
                declarer = builder.setBolt(stream.getTo(), (IRichBolt) boltObj, topologyDefinition.parallelismForBolt(stream.getTo()));
                declarers.put(stream.getTo(), declarer);
            }
        } else if (boltObj instanceof IBasicBolt) {
            if (declarer == null) {
                declarer = builder.setBolt(stream.getTo(), (IBasicBolt) boltObj, topologyDefinition.parallelismForBolt(stream.getTo()));
                declarers.put(stream.getTo(), declarer);
            }
        } else if (boltObj instanceof IWindowedBolt) {
            if (declarer == null) {
                declarer = builder.setBolt(stream.getTo(), (IWindowedBolt) boltObj, topologyDefinition.parallelismForBolt(stream.getTo()));
                declarers.put(stream.getTo(), declarer);
            }
        } else {
            throw new IllegalArgumentException("Class does not appear to be a bolt: " + boltObj.getClass().getName());
        }
        GroupingDefinition grouping = stream.getGrouping();
        // if the streamId is defined, use it for the grouping,
        // otherwise assume default stream
        String streamId = grouping.getStreamId() == null ? Utils.DEFAULT_STREAM_ID : grouping.getStreamId();
        switch(grouping.getType()) {
            case SHUFFLE:
                declarer.shuffleGrouping(stream.getFrom(), streamId);
                break;
            case FIELDS:
                List<String> groupingArgs = grouping.getArgs();
                if (groupingArgs == null) {
                    throw new IllegalArgumentException("You must supply arguments for Fields grouping");
                }
                declarer.fieldsGrouping(stream.getFrom(), streamId, new Fields(groupingArgs));
                break;
            case ALL:
                declarer.allGrouping(stream.getFrom(), streamId);
                break;
            case GLOBAL:
                declarer.globalGrouping(stream.getFrom(), streamId);
                break;
            case NONE:
                declarer.noneGrouping(stream.getFrom(), streamId);
                break;
            case CUSTOM:
                declarer.customGrouping(stream.getFrom(), streamId, buildCustomStreamGrouping(stream.getGrouping().getCustomClass(), executionContext, objectBuilder));
                break;
            default:
                throw new UnsupportedOperationException("unsupported grouping type: " + grouping);
        }
    }
    executionContext.setStreams(componentStreams);
}
Also used : IRichBolt(org.apache.storm.topology.IRichBolt) ComponentStream(org.apache.heron.eco.definition.ComponentStream) StreamDefinition(org.apache.heron.eco.definition.StreamDefinition) HashMap(java.util.HashMap) EcoTopologyDefinition(org.apache.heron.eco.definition.EcoTopologyDefinition) GroupingDefinition(org.apache.heron.eco.definition.GroupingDefinition) IBasicBolt(org.apache.storm.topology.IBasicBolt) Fields(org.apache.storm.tuple.Fields) BoltDeclarer(org.apache.storm.topology.BoltDeclarer) IWindowedBolt(org.apache.storm.topology.IWindowedBolt)

Example 10 with EcoTopologyDefinition

use of org.apache.heron.eco.definition.EcoTopologyDefinition in project heron by twitter.

the class BoltBuilder method buildBolts.

public void buildBolts(EcoExecutionContext executionContext, ObjectBuilder objectBuilder) throws IllegalAccessException, InstantiationException, ClassNotFoundException, NoSuchFieldException, InvocationTargetException {
    EcoTopologyDefinition topologyDefinition = executionContext.getTopologyDefinition();
    for (ObjectDefinition def : topologyDefinition.getBolts()) {
        Object obj = objectBuilder.buildObject(def, executionContext);
        executionContext.addBolt(def.getId(), obj);
    }
}
Also used : ObjectDefinition(org.apache.heron.eco.definition.ObjectDefinition) EcoTopologyDefinition(org.apache.heron.eco.definition.EcoTopologyDefinition)

Aggregations

EcoTopologyDefinition (org.apache.heron.eco.definition.EcoTopologyDefinition)31 Test (org.junit.Test)21 FileInputStream (java.io.FileInputStream)15 ByteArrayInputStream (java.io.ByteArrayInputStream)13 InputStream (java.io.InputStream)13 Config (org.apache.heron.api.Config)13 EcoParser (org.apache.heron.eco.parser.EcoParser)8 ArrayList (java.util.ArrayList)4 BoltDefinition (org.apache.heron.eco.definition.BoltDefinition)4 StreamDefinition (org.apache.heron.eco.definition.StreamDefinition)4 HashMap (java.util.HashMap)3 GroupingDefinition (org.apache.heron.eco.definition.GroupingDefinition)3 ObjectDefinition (org.apache.heron.eco.definition.ObjectDefinition)3 SpoutDefinition (org.apache.heron.eco.definition.SpoutDefinition)3 BeanDefinition (org.apache.heron.eco.definition.BeanDefinition)2 ComponentStream (org.apache.heron.eco.definition.ComponentStream)2 Before (org.junit.Before)2 Matchers.anyObject (org.mockito.Matchers.anyObject)2 ByteArrayOutputStream (java.io.ByteArrayOutputStream)1 Properties (java.util.Properties)1