Examples with Pipeline com.hazelcast.jet.pipeline.Pipeline used on opensource projects

Example 66 with Pipeline

use of com.hazelcast.jet.pipeline.Pipeline in project hazelcast by hazelcast.

the class CommonElasticSourcesTest method given_sourceCreatedByFactoryMethod3_whenReadFromElasticSource_thenFinishWithOneResult.

@Test
public void given_sourceCreatedByFactoryMethod3_whenReadFromElasticSource_thenFinishWithOneResult() {
    indexDocument("my-index-1", of("name", "Frantisek"));
    indexDocument("my-index-2", of("name", "Vladimir"));
    Pipeline p = Pipeline.create();
    BatchSource<String> source = ElasticSources.elastic(elasticClientSupplier(), () -> new SearchRequest("my-index-1"), hit -> (String) hit.getSourceAsMap().get("name"));
    p.readFrom(source).writeTo(Sinks.list(results));
    submitJob(p);
    assertThat(results).containsExactly("Frantisek");
}

Example 67 with Pipeline

use of com.hazelcast.jet.pipeline.Pipeline in project hazelcast by hazelcast.

the class CommonElasticSourcesTest method given_documents_when_readFromElasticSourceWithQuery_then_resultHasMatchingDocuments.

@Test
public void given_documents_when_readFromElasticSourceWithQuery_then_resultHasMatchingDocuments() {
    indexDocument("my-index", of("name", "Frantisek"));
    indexDocument("my-index", of("name", "Vladimir"));
    Pipeline p = Pipeline.create();
    BatchSource<String> source = new ElasticSourceBuilder<>().clientFn(elasticClientSupplier()).searchRequestFn(() -> new SearchRequest("my-index").source(new SearchSourceBuilder().query(QueryBuilders.matchQuery("name", "Frantisek")))).mapToItemFn(hit -> (String) hit.getSourceAsMap().get("name")).build();
    p.readFrom(source).writeTo(Sinks.list(results));
    submitJob(p);
    assertThat(results).containsOnlyOnce("Frantisek");
}

Example 68 with Pipeline

use of com.hazelcast.jet.pipeline.Pipeline in project hazelcast by hazelcast.

the class CommonElasticSourcesTest method given_multipleDocuments_when_readFromElasticSourceWithScroll_then_resultHasAllDocuments.

@Test
public void given_multipleDocuments_when_readFromElasticSourceWithScroll_then_resultHasAllDocuments() throws IOException {
    indexBatchOfDocuments("my-index");
    Pipeline p = Pipeline.create();
    BatchSource<String> source = new ElasticSourceBuilder<>().clientFn(elasticClientSupplier()).searchRequestFn(() -> {
        SearchRequest sr = new SearchRequest("my-index");
        // needs to scroll 5 times
        sr.source().size(10).query(matchAllQuery());
        return sr;
    }).mapToItemFn(SearchHit::getSourceAsString).build();
    p.readFrom(source).writeTo(Sinks.list(results));
    submitJob(p);
    assertThat(results).hasSize(BATCH_SIZE);
}

Example 69 with Pipeline

use of com.hazelcast.jet.pipeline.Pipeline in project hazelcast by hazelcast.

the class CommonElasticSourcesTest method given_multipleIndexes_when_readFromElasticSourceWithIndexWildcard_then_resultDocumentsFromAllIndexes.

@Test
public void given_multipleIndexes_when_readFromElasticSourceWithIndexWildcard_then_resultDocumentsFromAllIndexes() {
    indexDocument("my-index-1", of("name", "Frantisek"));
    indexDocument("my-index-2", of("name", "Vladimir"));
    Pipeline p = Pipeline.create();
    BatchSource<String> source = new ElasticSourceBuilder<>().clientFn(elasticClientSupplier()).searchRequestFn(() -> new SearchRequest("my-index-*")).mapToItemFn(hit -> (String) hit.getSourceAsMap().get("name")).build();
    p.readFrom(source).writeTo(Sinks.list(results));
    submitJob(p);
    assertThat(results).containsOnlyOnce("Frantisek", "Vladimir");
}

Example 70 with Pipeline

use of com.hazelcast.jet.pipeline.Pipeline in project hazelcast by hazelcast.

the class CommonElasticSourcesTest method given_nonExistingIndex_whenReadFromElasticSource_thenThrowException.

@Test
public void given_nonExistingIndex_whenReadFromElasticSource_thenThrowException() {
    Pipeline p = Pipeline.create();
    BatchSource<String> source = new ElasticSourceBuilder<>().clientFn(elasticClientSupplier()).searchRequestFn(() -> new SearchRequest("non-existing-index")).mapToItemFn(SearchHit::getSourceAsString).retries(// we expect the exception -> faster test
    0).build();
    p.readFrom(source).writeTo(Sinks.list(results));
    assertThatThrownBy(() -> submitJob(p)).hasRootCauseInstanceOf(ResponseException.class).hasStackTraceContaining("no such index [non-existing-index]");
}