Examples with SimpleStringSchema org.apache.flink.api.common.serialization.SimpleStringSchema used on opensource projects

Example 61 with SimpleStringSchema

use of org.apache.flink.api.common.serialization.SimpleStringSchema in project flink by apache.

the class SinkIntoKinesis method main.

public static void main(String[] args) throws Exception {
    ObjectMapper mapper = new ObjectMapper();
    final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
    env.enableCheckpointing(10_000);
    DataStream<String> fromGen = env.fromSequence(1, 10_000_000L).map(Object::toString).returns(String.class).map(data -> mapper.writeValueAsString(ImmutableMap.of("data", data)));
    Properties sinkProperties = new Properties();
    sinkProperties.put(AWSConfigConstants.AWS_REGION, "your-region-here");
    KinesisDataStreamsSink<String> kdsSink = KinesisDataStreamsSink.<String>builder().setSerializationSchema(new SimpleStringSchema()).setPartitionKeyGenerator(element -> String.valueOf(element.hashCode())).setStreamName("your-stream-name").setMaxBatchSize(20).setKinesisClientProperties(sinkProperties).build();
    fromGen.sinkTo(kdsSink);
    env.execute("KDS Async Sink Example Program");
}

Example 62 with SimpleStringSchema

use of org.apache.flink.api.common.serialization.SimpleStringSchema in project flink by apache.

the class KinesisFirehoseSinkElementConverterTest method elementConverterUsesProvidedSchemaToSerializeRecord.

@Test
public void elementConverterUsesProvidedSchemaToSerializeRecord() {
    ElementConverter<String, Record> elementConverter = KinesisFirehoseSinkElementConverter.<String>builder().setSerializationSchema(new SimpleStringSchema()).build();
    String testString = "{many hands make light work;";
    Record serializedRecord = elementConverter.apply(testString, null);
    byte[] serializedString = (new SimpleStringSchema()).serialize(testString);
    assertThat(serializedRecord.data()).isEqualTo(SdkBytes.fromByteArray(serializedString));
}

Example 63 with SimpleStringSchema

use of org.apache.flink.api.common.serialization.SimpleStringSchema in project flink by apache.

the class KinesisFirehoseSinkTest method firehoseSinkFailsWithAppropriateMessageWhenInitialConditionsAreMisconfigured.

private void firehoseSinkFailsWithAppropriateMessageWhenInitialConditionsAreMisconfigured(Properties properties, String errorMessage) {
    StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
    KinesisFirehoseSink<String> kdsSink = KinesisFirehoseSink.<String>builder().setSerializationSchema(new SimpleStringSchema()).setDeliveryStreamName("non-existent-stream").setMaxBatchSize(1).setFirehoseClientProperties(properties).build();
    KinesisFirehoseTestUtils.getSampleDataGenerator(env, 10).sinkTo(kdsSink);
    Assertions.assertThatExceptionOfType(JobExecutionException.class).isThrownBy(() -> env.execute("Integration Test")).havingCause().havingCause().withMessageContaining(errorMessage);
}