Search in sources :

Example 41 with GenericRecordImpl

use of com.datastax.oss.sink.pulsar.GenericRecordImpl in project pulsar-sink by datastax.

the class DeleteCCMIT method insert_with_nulls_when_delete_disabled.

@Test
void insert_with_nulls_when_delete_disabled() {
    taskConfigs.add(makeConnectorProperties("bigintcol=value.bigint, booleancol=value.boolean, intcol=value.int", "small_simple", ImmutableMap.of(String.format("topic.mytopic.%s.small_simple.deletesEnabled", keyspaceName), "false")));
    // Set up records for "mytopic"
    GenericRecordImpl value = new GenericRecordImpl().put("bigint", 1234567L).put("int", null).put("boolean", null);
    PulsarRecordImpl record = new PulsarRecordImpl("persistent://tenant/namespace/mytopic", null, value, recordType);
    runTaskWithRecords(record);
    // Verify that the record was inserted into the database with null non-pk values.
    List<Row> results = session.execute("SELECT * FROM small_simple").all();
    assertThat(results.size()).isEqualTo(1);
    Row row = results.get(0);
    assertThat(row.get("bigintcol", GenericType.LONG)).isEqualTo(1234567L);
    assertThat(row.get("booleancol", GenericType.BOOLEAN)).isNull();
    assertThat(row.get("intcol", GenericType.INTEGER)).isNull();
}
Also used : PulsarRecordImpl(com.datastax.oss.sink.pulsar.PulsarRecordImpl) GenericRecordImpl(com.datastax.oss.sink.pulsar.GenericRecordImpl) Row(com.datastax.oss.driver.api.core.cql.Row) Test(org.junit.jupiter.api.Test)

Example 42 with GenericRecordImpl

use of com.datastax.oss.sink.pulsar.GenericRecordImpl in project pulsar-sink by datastax.

the class DeleteCCMIT method delete_compound_decode_key_json.

@Test
void delete_compound_decode_key_json() {
    // First insert a row...
    session.execute("INSERT INTO small_compound (bigintcol, booleancol, intcol) VALUES (1234567, true, 42)");
    List<Row> results = session.execute("SELECT * FROM small_compound").all();
    assertThat(results.size()).isEqualTo(1);
    taskConfigs.add(makeConnectorProperties("bigintcol=key.bigint, booleancol=key.boolean, intcol=key.int", "small_compound", null));
    // Set up records for "mytopic", the key contains a json value
    String json = "{\"bigint\": 1234567, \"boolean\": true, \"int\": null}";
    PulsarRecordImpl record = new PulsarRecordImpl("persistent://tenant/namespace/mytopic", json, new GenericRecordImpl(), recordType);
    runTaskWithRecords(record);
    // Verify that the record was deleted from the database.
    results = session.execute("SELECT * FROM small_compound").all();
    assertThat(results.size()).isEqualTo(0);
}
Also used : PulsarRecordImpl(com.datastax.oss.sink.pulsar.PulsarRecordImpl) GenericRecordImpl(com.datastax.oss.sink.pulsar.GenericRecordImpl) Row(com.datastax.oss.driver.api.core.cql.Row) Test(org.junit.jupiter.api.Test)

Example 43 with GenericRecordImpl

use of com.datastax.oss.sink.pulsar.GenericRecordImpl in project pulsar-sink by datastax.

the class ConnectorSettingsCCMIT method should_insert_when_using_java_driver_contact_points_setting.

@Test
void should_insert_when_using_java_driver_contact_points_setting() {
    Map<String, Object> connectorProperties = makeConnectorPropertiesWithoutContactPointsAndPort("bigintcol=key, listcol=value.field1");
    // use single datastax-java-driver prefixed property that carry host:port
    connectorProperties.put(withDriverPrefix(DefaultDriverOption.CONTACT_POINTS), getContactPoints().stream().map(a -> {
        InetSocketAddress inetSocketAddress = (InetSocketAddress) a.resolve();
        return String.format("%s:%d", inetSocketAddress.getHostString(), inetSocketAddress.getPort());
    }).collect(Collectors.joining(",")));
    taskConfigs.add(connectorProperties);
    PulsarRecordImpl record = new PulsarRecordImpl("persistent://tenant/namespace/mytopic", "98761234", new GenericRecordImpl().put("field1", "[42, 37]"), recordType);
    record.setRecordSequence(1234L);
    runTaskWithRecords(record);
    // Verify that the record was inserted properly in the database.
    List<Row> results = session.execute("SELECT bigintcol, listcol FROM types").all();
    assertThat(results.size()).isEqualTo(1);
    Row row = results.get(0);
    assertThat(row.getLong("bigintcol")).isEqualTo(98761234L);
    assertThat(row.getList("listcol", Integer.class)).isEqualTo(Arrays.asList(42, 37));
}
Also used : PulsarRecordImpl(com.datastax.oss.sink.pulsar.PulsarRecordImpl) InetSocketAddress(java.net.InetSocketAddress) GenericRecordImpl(com.datastax.oss.sink.pulsar.GenericRecordImpl) Row(com.datastax.oss.driver.api.core.cql.Row) Test(org.junit.jupiter.api.Test)

Example 44 with GenericRecordImpl

use of com.datastax.oss.sink.pulsar.GenericRecordImpl in project pulsar-sink by datastax.

the class NowFunctionCCMIT method should_insert_value_using_now_function_avro.

@Test
void should_insert_value_using_now_function_avro() {
    taskConfigs.add(makeConnectorProperties("bigintcol=value.bigint, loaded_at=now(), loaded_at2=now()"));
    PulsarRecordImpl record = new PulsarRecordImpl("persistent://tenant/namespace/mytopic", null, new GenericRecordImpl().put("bigint", 1234567), recordType);
    runTaskWithRecords(record);
    // Verify that the record was inserted properly in the database.
    List<Row> results = session.execute("SELECT bigintcol, loaded_at, loaded_at2 FROM types").all();
    assertThat(results.size()).isEqualTo(1);
    Row row = results.get(0);
    assertThat(row.getLong("bigintcol")).isEqualTo(1234567L);
    UUID loadedAt = row.get("loaded_at", TypeCodecs.TIMEUUID);
    UUID loadedAt2 = row.get("loaded_at2", TypeCodecs.TIMEUUID);
    assertThat(loadedAt).isNotEqualTo(loadedAt2);
}
Also used : PulsarRecordImpl(com.datastax.oss.sink.pulsar.PulsarRecordImpl) GenericRecordImpl(com.datastax.oss.sink.pulsar.GenericRecordImpl) Row(com.datastax.oss.driver.api.core.cql.Row) UUID(java.util.UUID) Test(org.junit.jupiter.api.Test)

Example 45 with GenericRecordImpl

use of com.datastax.oss.sink.pulsar.GenericRecordImpl in project pulsar-sink by datastax.

the class NowFunctionCCMIT method should_insert_value_using_now_function_json.

@Test
void should_insert_value_using_now_function_json() {
    // given
    taskConfigs.add(makeConnectorProperties("bigintcol=value.bigint, loaded_at=now()"));
    // when
    PulsarRecordImpl record = new PulsarRecordImpl("persistent://tenant/namespace/mytopic", null, new GenericRecordImpl().put("bigint", 1234567), recordTypeJson);
    runTaskWithRecords(record);
    // then
    List<Row> results = session.execute("SELECT bigintcol, loaded_at FROM types").all();
    assertThat(results.size()).isEqualTo(1);
    Row row = results.get(0);
    assertThat(row.getLong("bigintcol")).isEqualTo(1234567L);
    assertThat(row.get("loaded_at", TypeCodecs.TIMEUUID)).isLessThanOrEqualTo(Uuids.timeBased());
}
Also used : PulsarRecordImpl(com.datastax.oss.sink.pulsar.PulsarRecordImpl) GenericRecordImpl(com.datastax.oss.sink.pulsar.GenericRecordImpl) Row(com.datastax.oss.driver.api.core.cql.Row) Test(org.junit.jupiter.api.Test)

Aggregations

GenericRecordImpl (com.datastax.oss.sink.pulsar.GenericRecordImpl)56 PulsarRecordImpl (com.datastax.oss.sink.pulsar.PulsarRecordImpl)56 Test (org.junit.jupiter.api.Test)50 Row (com.datastax.oss.driver.api.core.cql.Row)46 RecordSchemaBuilder (org.apache.pulsar.client.api.schema.RecordSchemaBuilder)16 Schema (org.apache.pulsar.client.api.Schema)12 GenericSchema (org.apache.pulsar.client.api.schema.GenericSchema)9 ParameterizedTest (org.junit.jupiter.params.ParameterizedTest)8 UserDefinedType (com.datastax.oss.driver.api.core.type.UserDefinedType)5 UserDefinedTypeBuilder (com.datastax.oss.driver.internal.core.type.UserDefinedTypeBuilder)5 GenericRecord (org.apache.pulsar.client.api.schema.GenericRecord)4 Query (com.datastax.oss.simulacron.common.request.Query)3 MethodSource (org.junit.jupiter.params.provider.MethodSource)3 LineString (com.datastax.dse.driver.api.core.data.geometry.LineString)2 DefaultLineString (com.datastax.dse.driver.internal.core.data.geometry.DefaultLineString)2 InstanceState (com.datastax.oss.common.sink.state.InstanceState)2 InetSocketAddress (java.net.InetSocketAddress)2 HashMap (java.util.HashMap)2 Point (com.datastax.dse.driver.api.core.data.geometry.Point)1 Polygon (com.datastax.dse.driver.api.core.data.geometry.Polygon)1