Examples with Pipeline com.hazelcast.jet.pipeline.Pipeline used on opensource projects

Example 16 with Pipeline

use of com.hazelcast.jet.pipeline.Pipeline in project hazelcast-jet by hazelcast.

the class WriteFilePTest method when_twoMembers_then_twoFiles.

@Test
// the test keeps failing on Jenkins, even though it runs without failure hundreds of times locally
@Ignore
public void when_twoMembers_then_twoFiles() throws Exception {
    // Given
    Pipeline p = buildPipeline(null, null, false);
    addItemsToList(0, 10);
    createJetMember();
    // When
    instance.newJob(p).join();
    // Then
    try (DirectoryStream<Path> stream = Files.newDirectoryStream(directory)) {
        int[] count = { 0 };
        stream.forEach(path -> count[0]++);
        assertEquals(2, count[0]);
    }
}

Example 17 with Pipeline

use of com.hazelcast.jet.pipeline.Pipeline in project hazelcast-jet by hazelcast.

the class WriteFilePTest method when_overwrite_then_previousContentsOverwritten.

@Test
public void when_overwrite_then_previousContentsOverwritten() throws Exception {
    // Given
    Pipeline p = buildPipeline(null, null, false);
    addItemsToList(0, 10);
    try (BufferedWriter writer = Files.newBufferedWriter(file)) {
        writer.write("bla bla");
        writer.newLine();
    }
    // When
    instance.newJob(p).join();
    // Then
    checkFileContents(StandardCharsets.UTF_8, 10);
}

Example 18 with Pipeline

use of com.hazelcast.jet.pipeline.Pipeline in project hazelcast-jet by hazelcast.

the class WriteFilePTest method buildPipeline.

private Pipeline buildPipeline(DistributedFunction<String, String> toStringFn, Charset charset, boolean append) {
    if (toStringFn == null) {
        toStringFn = Object::toString;
    }
    if (charset == null) {
        charset = StandardCharsets.UTF_8;
    }
    Pipeline p = Pipeline.create();
    p.drawFrom(Sources.<String>list(list.getName())).drainTo(Sinks.files(directory.toString(), toStringFn, charset, append));
    return p;
}

Example 19 with Pipeline

use of com.hazelcast.jet.pipeline.Pipeline in project hazelcast-jet by hazelcast.

the class WriteFilePTest method when_append_then_previousContentsOfFileIsKept.

@Test
public void when_append_then_previousContentsOfFileIsKept() throws Exception {
    // Given
    Pipeline p = buildPipeline(null, null, true);
    addItemsToList(1, 10);
    try (BufferedWriter writer = Files.newBufferedWriter(file)) {
        writer.write("0");
        writer.newLine();
    }
    // When
    instance.newJob(p).join();
    // Then
    checkFileContents(StandardCharsets.UTF_8, 10);
}

Example 20 with Pipeline

use of com.hazelcast.jet.pipeline.Pipeline in project hazelcast-jet by hazelcast.

the class WriteFilePTest method when_localParallelismMoreThan1_then_multipleFiles.

@Test
public void when_localParallelismMoreThan1_then_multipleFiles() throws Exception {
    // Given
    Pipeline p = Pipeline.create();
    p.drawFrom(Sources.<String>list(list.getName())).drainTo(Sinks.files(directory.toString())).setLocalParallelism(2);
    addItemsToList(0, 10);
    // When
    instance.newJob(p).join();
    // Then
    try (DirectoryStream<Path> stream = Files.newDirectoryStream(directory)) {
        int[] count = { 0 };
        stream.forEach(path -> count[0]++);
        assertEquals(2, count[0]);
    }
}