Examples with MISTEdge edu.snu.mist.common.graph.MISTEdge used on opensource projects

Example 31 with MISTEdge

use of edu.snu.mist.common.graph.MISTEdge in project mist by snuspl.

the class MISTQueryImpl method getAvroOperatorDag.

@Override
public Tuple<List<AvroVertex>, List<Edge>> getAvroOperatorDag() {
    final LogicalDagOptimizer logicalDagOptimizer = new LogicalDagOptimizer(dag);
    final DAG<MISTStream, MISTEdge> optimizedDag = logicalDagOptimizer.getOptimizedDAG();
    final Queue<MISTStream> queue = new LinkedList<>();
    final List<MISTStream> vertices = new ArrayList<>();
    final List<Edge> edges = new ArrayList<>();
    // Put all vertices into a queue
    final Iterator<MISTStream> iterator = GraphUtils.topologicalSort(optimizedDag);
    while (iterator.hasNext()) {
        final MISTStream vertex = iterator.next();
        queue.add(vertex);
        vertices.add(vertex);
    }
    // Visit each vertex and serialize its edges
    while (!queue.isEmpty()) {
        final MISTStream vertex = queue.remove();
        final int fromIndex = vertices.indexOf(vertex);
        final Map<MISTStream, MISTEdge> neighbors = optimizedDag.getEdges(vertex);
        for (final Map.Entry<MISTStream, MISTEdge> neighbor : neighbors.entrySet()) {
            final int toIndex = vertices.indexOf(neighbor.getKey());
            final MISTEdge edgeInfo = neighbor.getValue();
            final Edge.Builder edgeBuilder = Edge.newBuilder().setFrom(fromIndex).setTo(toIndex).setDirection(edgeInfo.getDirection()).setBranchIndex(edgeInfo.getIndex());
            edges.add(edgeBuilder.build());
        }
    }
    final Set<MISTStream> rootVertices = optimizedDag.getRootVertices();
    // Serialize each vertex via avro.
    final List<AvroVertex> serializedVertices = new ArrayList<>();
    for (final MISTStream vertex : vertices) {
        final AvroVertex.Builder vertexBuilder = AvroVertex.newBuilder();
        vertexBuilder.setConfiguration(vertex.getConfiguration());
        vertexBuilder.setVertexId(String.valueOf(vertexIdIndex));
        // Set vertex type
        if (rootVertices.contains(vertex)) {
            // this is a source
            vertexBuilder.setAvroVertexType(AvroVertexTypeEnum.SOURCE);
        } else if (optimizedDag.getEdges(vertex).size() == 0) {
            // this is a sink
            vertexBuilder.setAvroVertexType(AvroVertexTypeEnum.SINK);
        } else {
            vertexBuilder.setAvroVertexType(AvroVertexTypeEnum.OPERATOR);
        }
        serializedVertices.add(vertexBuilder.build());
        vertexIdIndex++;
    }
    return new Tuple<>(serializedVertices, edges);
}

Example 32 with MISTEdge

use of edu.snu.mist.common.graph.MISTEdge in project mist by snuspl.

the class DefaultDagGeneratorImpl method dfsCreation.

private void dfsCreation(final ExecutionVertex parent, final MISTEdge parentEdge, final ConfigVertex currVertex, final Map<ConfigVertex, ExecutionVertex> created, final DAG<ConfigVertex, MISTEdge> configDag, final ExecutionDag executionDag, final URL[] urls, final ClassLoader classLoader) throws IOException, ClassNotFoundException {
    final ExecutionVertex currExecutionVertex;
    final DAG<ExecutionVertex, MISTEdge> dag = executionDag.getDag();
    if (created.get(currVertex) == null) {
        currExecutionVertex = executionVertexGenerator.generate(currVertex, urls, classLoader);
        created.put(currVertex, currExecutionVertex);
        dag.addVertex(currExecutionVertex);
        // do dfs creation
        for (final Map.Entry<ConfigVertex, MISTEdge> edges : configDag.getEdges(currVertex).entrySet()) {
            final ConfigVertex childVertex = edges.getKey();
            final MISTEdge edge = edges.getValue();
            dfsCreation(currExecutionVertex, edge, childVertex, created, configDag, executionDag, urls, classLoader);
        }
    } else {
        currExecutionVertex = created.get(currVertex);
    }
    dag.addEdge(parent, currExecutionVertex, parentEdge);
}

Example 33 with MISTEdge

use of edu.snu.mist.common.graph.MISTEdge in project mist by snuspl.

the class DefaultDagGeneratorImpl method generate.

/**
 * This generates the logical and physical plan from the avro operator chain dag.
 * Note that the avro operator chain dag is already partitioned,
 * so we need to rewind the partition to generate the logical dag.
 * @param configDag the tuple of queryId and avro operator chain dag
 * @return the logical and execution dag
 */
@Override
public ExecutionDag generate(final DAG<ConfigVertex, MISTEdge> configDag, final List<String> jarFilePaths) throws IOException, ClassNotFoundException {
    // For execution dag
    final ExecutionDag executionDag = new ExecutionDag(new AdjacentListConcurrentMapDAG<>());
    // Get a class loader
    final URL[] urls = SerializeUtils.getJarFileURLs(jarFilePaths);
    final ClassLoader classLoader = classLoaderProvider.newInstance(urls);
    final Map<ConfigVertex, ExecutionVertex> created = new HashMap<>(configDag.numberOfVertices());
    for (final ConfigVertex source : configDag.getRootVertices()) {
        final ExecutionVertex currExecutionVertex = executionVertexGenerator.generate(source, urls, classLoader);
        executionDag.getDag().addVertex(currExecutionVertex);
        created.put(source, currExecutionVertex);
        // do dfs creation
        for (final Map.Entry<ConfigVertex, MISTEdge> edges : configDag.getEdges(source).entrySet()) {
            final ConfigVertex childVertex = edges.getKey();
            final MISTEdge edge = edges.getValue();
            dfsCreation(currExecutionVertex, edge, childVertex, created, configDag, executionDag, urls, classLoader);
        }
    }
    return executionDag;
}

Example 34 with MISTEdge

use of edu.snu.mist.common.graph.MISTEdge in project mist by snuspl.

the class NonBlockingQueueSourceOutputEmitter method processAllEvent.

@Override
public int processAllEvent() {
    int numProcessedEvent = 0;
    MistEvent event = queue.poll();
    while (event != null) {
        numEvents.decrementAndGet();
        for (final Map.Entry<ExecutionVertex, MISTEdge> entry : nextOperators.entrySet()) {
            process(event, entry.getValue().getDirection(), (PhysicalOperator) entry.getKey());
        }
        numProcessedEvent += 1;
        event = queue.poll();
    }
    return numProcessedEvent;
}

Example 35 with MISTEdge

use of edu.snu.mist.common.graph.MISTEdge in project mist by snuspl.

the class OperatorOutputEmitter method emitCheckpoint.

@Override
public void emitCheckpoint(final MistCheckpointEvent checkpoint) {
    // Checkpoint is not changed, so we just forward it to the next operator chains.
    for (final Map.Entry<ExecutionVertex, MISTEdge> nextQuery : nextOperators.entrySet()) {
        final Direction direction = nextQuery.getValue().getDirection();
        sendCheckpoint(checkpoint, direction, nextQuery.getKey());
    }
}