Examples with Pipeline com.hazelcast.jet.pipeline.Pipeline used on opensource projects

Example 71 with Pipeline

use of com.hazelcast.jet.pipeline.Pipeline in project hazelcast by hazelcast.

the class CommonElasticSourcesTest method given_indexWithOneDocument_whenReadFromElasticSource_thenFinishWithOneResult.

@Test
public void given_indexWithOneDocument_whenReadFromElasticSource_thenFinishWithOneResult() {
    indexDocument("my-index", of("name", "Frantisek"));
    Pipeline p = Pipeline.create();
    BatchSource<String> source = new ElasticSourceBuilder<>().clientFn(elasticClientSupplier()).searchRequestFn(() -> new SearchRequest("my-index")).mapToItemFn(hit -> (String) hit.getSourceAsMap().get("name")).build();
    p.readFrom(source).writeTo(Sinks.list(results));
    submitJob(p);
    assertThat(results).containsExactly("Frantisek");
}

Example 72 with Pipeline

use of com.hazelcast.jet.pipeline.Pipeline in project hazelcast by hazelcast.

the class CommonElasticSourcesTest method given_emptyIndex_when_readFromElasticSource_then_finishWithNoResults.

@Test
public void given_emptyIndex_when_readFromElasticSource_then_finishWithNoResults() throws IOException {
    // Ideally we would just create the index but it gives "field _id not found" when there are no documents
    // in the index, not sure if it is an Elastic bug or wrong setup
    // 
    // elasticClient.indices().create(new CreateIndexRequest("my-index"), DEFAULT);
    // Instead we index a document and delete it, ending up with index with correct settings applied
    indexDocument("my-index", of("name", "Frantisek"));
    deleteDocuments();
    Pipeline p = Pipeline.create();
    BatchSource<String> source = new ElasticSourceBuilder<>().clientFn(elasticClientSupplier()).searchRequestFn(() -> new SearchRequest("my-index")).mapToItemFn(SearchHit::getSourceAsString).build();
    p.readFrom(source).writeTo(Sinks.list(results));
    submitJob(p);
    assertThat(results).isEmpty();
}

Example 73 with Pipeline

use of com.hazelcast.jet.pipeline.Pipeline in project hazelcast by hazelcast.

the class ElasticClientsTest method given_clientAsString_whenReadFromElasticSource_thenFinishSuccessfully.

@Test
public void given_clientAsString_whenReadFromElasticSource_thenFinishSuccessfully() {
    ElasticsearchContainer container = ElasticSupport.elastic.get();
    String httpHostAddress = container.getHttpHostAddress();
    indexDocument("my-index", ImmutableMap.of("name", "Frantisek"));
    Pipeline p = Pipeline.create();
    p.readFrom(ElasticSources.elastic(() -> ElasticClients.client(httpHostAddress), SearchHit::getSourceAsString)).writeTo(Sinks.list(results));
    submitJob(p);
    assertThat(results).hasSize(1);
}

Example 74 with Pipeline

use of com.hazelcast.jet.pipeline.Pipeline in project hazelcast by hazelcast.

the class BaseFileFormatTest method assertItemsInSource.

protected <T> void assertItemsInSource(int memberCount, FileSourceBuilder<T> source, ConsumerEx<List<T>> assertion) {
    if (useHadoop) {
        source.useHadoopForLocalFiles(true);
    }
    Pipeline p = Pipeline.create();
    p.readFrom(source.build()).apply(Assertions.assertCollected(assertion));
    HazelcastInstance[] instances = createHazelcastInstances(memberCount);
    try {
        instances[0].getJet().newJob(p).join();
    } finally {
        for (HazelcastInstance instance : instances) {
            instance.shutdown();
        }
    }
}

Example 75 with Pipeline

use of com.hazelcast.jet.pipeline.Pipeline in project hazelcast by hazelcast.

the class AuthElasticSourcesTest method given_authenticatedClient_whenReadFromElasticSource_thenFinishSuccessfully.

@Test
public void given_authenticatedClient_whenReadFromElasticSource_thenFinishSuccessfully() {
    indexDocument("my-index", ImmutableMap.of("name", "Frantisek"));
    Pipeline p = Pipeline.create();
    p.readFrom(elasticSource(elasticClientSupplier())).writeTo(Sinks.list(results));
    submitJob(p);
}