Examples with SourceConfiguration edu.snu.mist.client.datastreams.configurations.SourceConfiguration used on opensource projects

Example 11 with SourceConfiguration

use of edu.snu.mist.client.datastreams.configurations.SourceConfiguration in project mist by snuspl.

the class QueryDeletion method submitQuery.

/**
 * Submit a query containing reduce-by-key operator.
 * The query reads strings from a source server, filters alphabetical words,
 * counts words using reduce-by-key operator, and sends them to a sink server.
 * @return result of the submission
 * @throws IOException
 * @throws InjectionException
 */
public static APIQueryControlResult submitQuery(final Configuration configuration) throws IOException, InjectionException, URISyntaxException {
    final String sourceSocket = Tang.Factory.getTang().newInjector(configuration).getNamedInstance(NettySourceAddress.class);
    final SourceConfiguration localTextSocketSourceConf = MISTExampleUtils.getLocalTextSocketSourceConf(sourceSocket);
    // Simple reduce function.
    final MISTBiFunction<Integer, Integer, Integer> reduceFunction = (v1, v2) -> {
        return v1 + v2;
    };
    final MISTQueryBuilder queryBuilder = new MISTQueryBuilder();
    queryBuilder.socketTextStream(localTextSocketSourceConf).filter(s -> isAlpha(s)).map(s -> new Tuple2(s, 1)).reduceByKey(0, String.class, reduceFunction).map(s -> s.toString()).textSocketOutput(MISTExampleUtils.SINK_HOSTNAME, MISTExampleUtils.SINK_PORT);
    return MISTExampleUtils.submit(queryBuilder, configuration);
}

Example 12 with SourceConfiguration

use of edu.snu.mist.client.datastreams.configurations.SourceConfiguration in project mist by snuspl.

the class GroupRecoveryTest method buildQuery.

/**
 * Builds the query for this test.
 * @return the built MISTQuery
 */
private MISTQuery buildQuery() {
    /**
     * Build the query which consists of:
     * SRC1 -> toTuple -> reduceByKey -> sort -> toString -> sink1
     */
    final SourceConfiguration localTextSocketSource1Conf = TextSocketSourceConfiguration.newBuilder().setHostAddress("localhost").setHostPort(16118).build();
    final MISTQueryBuilder queryBuilder = new MISTQueryBuilder();
    queryBuilder.setApplicationId("test-group");
    final int defaultWatermarkPeriod = 100;
    final WatermarkConfiguration testConf = PeriodicWatermarkConfiguration.newBuilder().setWatermarkPeriod(defaultWatermarkPeriod).build();
    queryBuilder.socketTextStream(localTextSocketSource1Conf, testConf).map(toTupleMapFunc).reduceByKey(0, String.class, countFunc).map(m -> new TreeMap<>(m)).map(toStringMapFunc).textSocketOutput("localhost", SINK_PORT);
    return queryBuilder.build();
}