Search in sources :

Example 1 with GenericRecordImpl

use of com.datastax.oss.sink.pulsar.GenericRecordImpl in project pulsar-sink by datastax.

the class RawDataEndToEndCCMIT method null_to_unset_false.

@Test
void null_to_unset_false() {
    // Make a row with some value for textcol to start with.
    session.execute("INSERT INTO types (bigintcol, textcol) VALUES (1234567, 'got here')");
    taskConfigs.add(makeConnectorProperties("bigintcol=key, textcol=value.text", ImmutableMap.of(String.format("topic.mytopic.%s.types.nullToUnset", keyspaceName), "false")));
    PulsarRecordImpl record = new PulsarRecordImpl("persistent://tenant/namespace/mytopic", "1234567", new GenericRecordImpl().put("text", null), recordType);
    runTaskWithRecords(record);
    // Verify that the record was inserted properly in the database; textcol should be unchanged.
    List<Row> results = session.execute("SELECT bigintcol, textcol FROM types").all();
    assertThat(results.size()).isEqualTo(1);
    Row row = results.get(0);
    assertThat(row.getLong("bigintcol")).isEqualTo(1234567L);
    assertThat(row.getString("textcol")).isNull();
}
Also used : PulsarRecordImpl(com.datastax.oss.sink.pulsar.PulsarRecordImpl) GenericRecordImpl(com.datastax.oss.sink.pulsar.GenericRecordImpl) Row(com.datastax.oss.driver.api.core.cql.Row) Test(org.junit.jupiter.api.Test)

Example 2 with GenericRecordImpl

use of com.datastax.oss.sink.pulsar.GenericRecordImpl in project pulsar-sink by datastax.

the class RawDataEndToEndCCMIT method should_insert_from_topic_with_complex_name.

@Test
void should_insert_from_topic_with_complex_name() {
    taskConfigs.add(makeConnectorProperties("bigintcol=value.bigint", "types", null, "this.is.complex_topic-name"));
    PulsarRecordImpl record = new PulsarRecordImpl("persistent://tenant/namespace/this.is.complex_topic-name", null, new GenericRecordImpl().put("bigint", 5725368L), recordType);
    runTaskWithRecords(record);
    // Verify that the record was inserted properly in the database.
    List<Row> results = session.execute("SELECT bigintcol FROM types").all();
    assertThat(results.size()).isEqualTo(1);
    Row row = results.get(0);
    assertThat(row.getLong("bigintcol")).isEqualTo(5725368L);
}
Also used : PulsarRecordImpl(com.datastax.oss.sink.pulsar.PulsarRecordImpl) GenericRecordImpl(com.datastax.oss.sink.pulsar.GenericRecordImpl) Row(com.datastax.oss.driver.api.core.cql.Row) Test(org.junit.jupiter.api.Test)

Example 3 with GenericRecordImpl

use of com.datastax.oss.sink.pulsar.GenericRecordImpl in project pulsar-sink by datastax.

the class RawDataEndToEndCCMIT method null_to_unset_true.

@Test
void null_to_unset_true() {
    // Make a row with some value for textcol to start with.
    session.execute("INSERT INTO types (bigintcol, textcol) VALUES (1234567, 'got here')");
    taskConfigs.add(makeConnectorProperties("bigintcol=key, textcol=value.text"));
    PulsarRecordImpl record = new PulsarRecordImpl("persistent://tenant/namespace/mytopic", "1234567", new GenericRecordImpl().put("text", null), recordType);
    runTaskWithRecords(record);
    // Verify that the record was inserted properly in the database; textcol should be unchanged.
    List<Row> results = session.execute("SELECT bigintcol, textcol FROM types").all();
    assertThat(results.size()).isEqualTo(1);
    Row row = results.get(0);
    assertThat(row.getLong("bigintcol")).isEqualTo(1234567L);
    assertThat(row.getString("textcol")).isEqualTo("got here");
}
Also used : PulsarRecordImpl(com.datastax.oss.sink.pulsar.PulsarRecordImpl) GenericRecordImpl(com.datastax.oss.sink.pulsar.GenericRecordImpl) Row(com.datastax.oss.driver.api.core.cql.Row) Test(org.junit.jupiter.api.Test)

Example 4 with GenericRecordImpl

use of com.datastax.oss.sink.pulsar.GenericRecordImpl in project pulsar-sink by datastax.

the class RawDataEndToEndCCMIT method raw_bigint_value.

@Test
void raw_bigint_value() {
    taskConfigs.add(makeConnectorProperties("bigintcol=value.bigint"));
    PulsarRecordImpl record = new PulsarRecordImpl("persistent://tenant/namespace/mytopic", null, new GenericRecordImpl().put("bigint", 5725368L), recordType);
    runTaskWithRecords(record);
    // Verify that the record was inserted properly in the database.
    List<Row> results = session.execute("SELECT bigintcol FROM types").all();
    assertThat(results.size()).isEqualTo(1);
    Row row = results.get(0);
    assertThat(row.getLong("bigintcol")).isEqualTo(5725368L);
}
Also used : PulsarRecordImpl(com.datastax.oss.sink.pulsar.PulsarRecordImpl) GenericRecordImpl(com.datastax.oss.sink.pulsar.GenericRecordImpl) Row(com.datastax.oss.driver.api.core.cql.Row) Test(org.junit.jupiter.api.Test)

Example 5 with GenericRecordImpl

use of com.datastax.oss.sink.pulsar.GenericRecordImpl in project pulsar-sink by datastax.

the class StructEndToEndCCMIT method raw_udt_value_and_cherry_pick_from_struct.

@Test
void raw_udt_value_and_cherry_pick_from_struct() {
    taskConfigs.add(makeConnectorProperties("bigintcol=key, udtcol=value, intcol=value.udtmem1"));
    RecordSchemaBuilder builder = org.apache.pulsar.client.api.schema.SchemaBuilder.record("MyBean");
    builder.field("udtmem1").type(SchemaType.INT32);
    builder.field("udtmem2").type(SchemaType.STRING);
    Schema recordTypeUtd = org.apache.pulsar.client.api.Schema.generic(builder.build(SchemaType.AVRO));
    GenericRecordImpl value = new GenericRecordImpl().put("udtmem1", 42).put("udtmem2", "the answer");
    PulsarRecordImpl record = new PulsarRecordImpl("persistent://tenant/namespace/mytopic", "98761234", value, recordTypeUtd);
    runTaskWithRecords(record);
    // Verify that the record was inserted properly in the database.
    List<Row> results = session.execute("SELECT bigintcol, udtcol, intcol FROM types").all();
    assertThat(results.size()).isEqualTo(1);
    Row row = results.get(0);
    assertThat(row.getLong("bigintcol")).isEqualTo(98761234L);
    UserDefinedType udt = new UserDefinedTypeBuilder(keyspaceName, "myudt").withField("udtmem1", DataTypes.INT).withField("udtmem2", DataTypes.TEXT).build();
    udt.attach(session.getContext());
    assertThat(row.getUdtValue("udtcol")).isEqualTo(udt.newValue(42, "the answer"));
    assertThat(row.getInt("intcol")).isEqualTo(42);
}
Also used : RecordSchemaBuilder(org.apache.pulsar.client.api.schema.RecordSchemaBuilder) PulsarRecordImpl(com.datastax.oss.sink.pulsar.PulsarRecordImpl) GenericSchema(org.apache.pulsar.client.api.schema.GenericSchema) Schema(org.apache.pulsar.client.api.Schema) UserDefinedType(com.datastax.oss.driver.api.core.type.UserDefinedType) GenericRecordImpl(com.datastax.oss.sink.pulsar.GenericRecordImpl) Row(com.datastax.oss.driver.api.core.cql.Row) UserDefinedTypeBuilder(com.datastax.oss.driver.internal.core.type.UserDefinedTypeBuilder) Test(org.junit.jupiter.api.Test)

Aggregations

GenericRecordImpl (com.datastax.oss.sink.pulsar.GenericRecordImpl)56 PulsarRecordImpl (com.datastax.oss.sink.pulsar.PulsarRecordImpl)56 Test (org.junit.jupiter.api.Test)50 Row (com.datastax.oss.driver.api.core.cql.Row)46 RecordSchemaBuilder (org.apache.pulsar.client.api.schema.RecordSchemaBuilder)16 Schema (org.apache.pulsar.client.api.Schema)12 GenericSchema (org.apache.pulsar.client.api.schema.GenericSchema)9 ParameterizedTest (org.junit.jupiter.params.ParameterizedTest)8 UserDefinedType (com.datastax.oss.driver.api.core.type.UserDefinedType)5 UserDefinedTypeBuilder (com.datastax.oss.driver.internal.core.type.UserDefinedTypeBuilder)5 GenericRecord (org.apache.pulsar.client.api.schema.GenericRecord)4 Query (com.datastax.oss.simulacron.common.request.Query)3 MethodSource (org.junit.jupiter.params.provider.MethodSource)3 LineString (com.datastax.dse.driver.api.core.data.geometry.LineString)2 DefaultLineString (com.datastax.dse.driver.internal.core.data.geometry.DefaultLineString)2 InstanceState (com.datastax.oss.common.sink.state.InstanceState)2 InetSocketAddress (java.net.InetSocketAddress)2 HashMap (java.util.HashMap)2 Point (com.datastax.dse.driver.api.core.data.geometry.Point)1 Polygon (com.datastax.dse.driver.api.core.data.geometry.Polygon)1