Search in sources :

Example 86 with Configuration

use of org.apache.flink.configuration.Configuration in project flink by apache.

the class PrintSinkFunctionTest method testPrintSinkWithPrefix.

@Test
public void testPrintSinkWithPrefix() {
    ByteArrayOutputStream baos = new ByteArrayOutputStream();
    PrintStream stream = new PrintStream(baos);
    System.setOut(stream);
    final StreamingRuntimeContext ctx = Mockito.mock(StreamingRuntimeContext.class);
    Mockito.when(ctx.getNumberOfParallelSubtasks()).thenReturn(2);
    Mockito.when(ctx.getIndexOfThisSubtask()).thenReturn(1);
    PrintSinkFunction<String> printSink = new PrintSinkFunction<>();
    printSink.setRuntimeContext(ctx);
    try {
        printSink.open(new Configuration());
    } catch (Exception e) {
        Assert.fail();
    }
    printSink.setTargetToStandardErr();
    printSink.invoke("hello world!");
    assertEquals("Print to System.err", printSink.toString());
    assertEquals("2> hello world!" + line, baos.toString());
    printSink.close();
    stream.close();
}
Also used : PrintStream(java.io.PrintStream) StreamingRuntimeContext(org.apache.flink.streaming.api.operators.StreamingRuntimeContext) Configuration(org.apache.flink.configuration.Configuration) PrintSinkFunction(org.apache.flink.streaming.api.functions.sink.PrintSinkFunction) ByteArrayOutputStream(java.io.ByteArrayOutputStream) Test(org.junit.Test)

Example 87 with Configuration

use of org.apache.flink.configuration.Configuration in project flink by apache.

the class PrintSinkFunctionTest method testPrintSinkStdOut.

@Test
public void testPrintSinkStdOut() {
    ByteArrayOutputStream baos = new ByteArrayOutputStream();
    PrintStream stream = new PrintStream(baos);
    System.setOut(stream);
    final StreamingRuntimeContext ctx = Mockito.mock(StreamingRuntimeContext.class);
    PrintSinkFunction<String> printSink = new PrintSinkFunction<>();
    printSink.setRuntimeContext(ctx);
    try {
        printSink.open(new Configuration());
    } catch (Exception e) {
        Assert.fail();
    }
    printSink.setTargetToStandardOut();
    printSink.invoke("hello world!");
    assertEquals("Print to System.out", printSink.toString());
    assertEquals("hello world!" + line, baos.toString());
    printSink.close();
    stream.close();
}
Also used : PrintStream(java.io.PrintStream) StreamingRuntimeContext(org.apache.flink.streaming.api.operators.StreamingRuntimeContext) Configuration(org.apache.flink.configuration.Configuration) PrintSinkFunction(org.apache.flink.streaming.api.functions.sink.PrintSinkFunction) ByteArrayOutputStream(java.io.ByteArrayOutputStream) Test(org.junit.Test)

Example 88 with Configuration

use of org.apache.flink.configuration.Configuration in project flink by apache.

the class PrintSinkFunctionTest method testPrintSinkStdErr.

@Test
public void testPrintSinkStdErr() {
    ByteArrayOutputStream baos = new ByteArrayOutputStream();
    PrintStream stream = new PrintStream(baos);
    System.setOut(stream);
    final StreamingRuntimeContext ctx = Mockito.mock(StreamingRuntimeContext.class);
    PrintSinkFunction<String> printSink = new PrintSinkFunction<>();
    printSink.setRuntimeContext(ctx);
    try {
        printSink.open(new Configuration());
    } catch (Exception e) {
        Assert.fail();
    }
    printSink.setTargetToStandardErr();
    printSink.invoke("hello world!");
    assertEquals("Print to System.err", printSink.toString());
    assertEquals("hello world!" + line, baos.toString());
    printSink.close();
    stream.close();
}
Also used : PrintStream(java.io.PrintStream) StreamingRuntimeContext(org.apache.flink.streaming.api.operators.StreamingRuntimeContext) Configuration(org.apache.flink.configuration.Configuration) PrintSinkFunction(org.apache.flink.streaming.api.functions.sink.PrintSinkFunction) ByteArrayOutputStream(java.io.ByteArrayOutputStream) Test(org.junit.Test)

Example 89 with Configuration

use of org.apache.flink.configuration.Configuration in project flink by apache.

the class SocketClientSinkTest method testSocketSink.

@Test
public void testSocketSink() throws Exception {
    final ServerSocket server = new ServerSocket(0);
    final int port = server.getLocalPort();
    final AtomicReference<Throwable> error = new AtomicReference<Throwable>();
    Thread sinkRunner = new Thread("Test sink runner") {

        @Override
        public void run() {
            try {
                SocketClientSink<String> simpleSink = new SocketClientSink<>(host, port, simpleSchema, 0);
                simpleSink.open(new Configuration());
                simpleSink.invoke(TEST_MESSAGE + '\n');
                simpleSink.close();
            } catch (Throwable t) {
                error.set(t);
            }
        }
    };
    sinkRunner.start();
    Socket sk = server.accept();
    BufferedReader rdr = new BufferedReader(new InputStreamReader(sk.getInputStream()));
    String value = rdr.readLine();
    sinkRunner.join();
    server.close();
    if (error.get() != null) {
        Throwable t = error.get();
        t.printStackTrace();
        fail("Error in spawned thread: " + t.getMessage());
    }
    assertEquals(TEST_MESSAGE, value);
}
Also used : Configuration(org.apache.flink.configuration.Configuration) InputStreamReader(java.io.InputStreamReader) ServerSocket(java.net.ServerSocket) AtomicReference(java.util.concurrent.atomic.AtomicReference) BufferedReader(java.io.BufferedReader) Socket(java.net.Socket) ServerSocket(java.net.ServerSocket) Test(org.junit.Test)

Example 90 with Configuration

use of org.apache.flink.configuration.Configuration in project flink by apache.

the class SocketClientSinkTest method testSinkAutoFlush.

@Test
public void testSinkAutoFlush() throws Exception {
    final ServerSocket server = new ServerSocket(0);
    final int port = server.getLocalPort();
    final SocketClientSink<String> simpleSink = new SocketClientSink<>(host, port, simpleSchema, 0, true);
    simpleSink.open(new Configuration());
    final AtomicReference<Throwable> error = new AtomicReference<Throwable>();
    Thread sinkRunner = new Thread("Test sink runner") {

        @Override
        public void run() {
            try {
                // need two messages here: send a fin to cancel the client state:FIN_WAIT_2 while the server is CLOSE_WAIT
                simpleSink.invoke(TEST_MESSAGE + '\n');
            } catch (Throwable t) {
                error.set(t);
            }
        }
    };
    sinkRunner.start();
    Socket sk = server.accept();
    BufferedReader rdr = new BufferedReader(new InputStreamReader(sk.getInputStream()));
    String value = rdr.readLine();
    sinkRunner.join();
    simpleSink.close();
    server.close();
    if (error.get() != null) {
        Throwable t = error.get();
        t.printStackTrace();
        fail("Error in spawned thread: " + t.getMessage());
    }
    assertEquals(TEST_MESSAGE, value);
}
Also used : Configuration(org.apache.flink.configuration.Configuration) InputStreamReader(java.io.InputStreamReader) ServerSocket(java.net.ServerSocket) AtomicReference(java.util.concurrent.atomic.AtomicReference) BufferedReader(java.io.BufferedReader) Socket(java.net.Socket) ServerSocket(java.net.ServerSocket) Test(org.junit.Test)

Aggregations

Configuration (org.apache.flink.configuration.Configuration)630 Test (org.junit.Test)452 IOException (java.io.IOException)137 FileInputSplit (org.apache.flink.core.fs.FileInputSplit)93 File (java.io.File)92 JobID (org.apache.flink.api.common.JobID)74 ExecutionConfig (org.apache.flink.api.common.ExecutionConfig)68 JobVertex (org.apache.flink.runtime.jobgraph.JobVertex)49 ActorGateway (org.apache.flink.runtime.instance.ActorGateway)46 JobGraph (org.apache.flink.runtime.jobgraph.JobGraph)45 Path (org.apache.flink.core.fs.Path)44 ActorRef (akka.actor.ActorRef)43 ArrayList (java.util.ArrayList)43 Tuple2 (org.apache.flink.api.java.tuple.Tuple2)39 FiniteDuration (scala.concurrent.duration.FiniteDuration)38 LocalFlinkMiniCluster (org.apache.flink.runtime.minicluster.LocalFlinkMiniCluster)36 BeforeClass (org.junit.BeforeClass)35 AkkaActorGateway (org.apache.flink.runtime.instance.AkkaActorGateway)33 MetricRegistry (org.apache.flink.runtime.metrics.MetricRegistry)33 JobVertexID (org.apache.flink.runtime.jobgraph.JobVertexID)32