Search in sources :

Example 1 with CosmosPagedIterable

use of com.azure.cosmos.util.CosmosPagedIterable in project azure-cosmos-java-sql-api-samples by Azure-Samples.

the class CosmosDiagnosticsQuickStart method queryDocuments.

private void queryDocuments() throws Exception {
    logger.info("Query documents in the container : {}", containerName);
    String sql = "SELECT * FROM c WHERE c.lastName = 'Witherspoon'";
    CosmosPagedIterable<Family> filteredFamilies = container.queryItems(sql, new CosmosQueryRequestOptions(), Family.class);
    // Add handler to capture diagnostics
    filteredFamilies = filteredFamilies.handle(familyFeedResponse -> {
        logger.info("Query Item diagnostics through handler : {}", familyFeedResponse.getCosmosDiagnostics());
    });
    // Or capture diagnostics through iterableByPage() APIs.
    filteredFamilies.iterableByPage().forEach(familyFeedResponse -> {
        logger.info("Query item diagnostics through iterableByPage : {}", familyFeedResponse.getCosmosDiagnostics());
    });
    logger.info("Done.");
}
Also used : CosmosQueryRequestOptions(com.azure.cosmos.models.CosmosQueryRequestOptions) CosmosContainerProperties(com.azure.cosmos.models.CosmosContainerProperties) Logger(org.slf4j.Logger) CosmosDatabaseRequestOptions(com.azure.cosmos.models.CosmosDatabaseRequestOptions) CosmosDatabase(com.azure.cosmos.CosmosDatabase) CosmosContainerResponse(com.azure.cosmos.models.CosmosContainerResponse) PartitionKey(com.azure.cosmos.models.PartitionKey) LoggerFactory(org.slf4j.LoggerFactory) CosmosQueryRequestOptions(com.azure.cosmos.models.CosmosQueryRequestOptions) UUID(java.util.UUID) CosmosItemResponse(com.azure.cosmos.models.CosmosItemResponse) CosmosClientBuilder(com.azure.cosmos.CosmosClientBuilder) Family(com.azure.cosmos.examples.common.Family) CosmosClient(com.azure.cosmos.CosmosClient) CosmosPagedIterable(com.azure.cosmos.util.CosmosPagedIterable) CosmosContainer(com.azure.cosmos.CosmosContainer) CosmosDatabaseResponse(com.azure.cosmos.models.CosmosDatabaseResponse) ThroughputProperties(com.azure.cosmos.models.ThroughputProperties) ConsistencyLevel(com.azure.cosmos.ConsistencyLevel) CosmosException(com.azure.cosmos.CosmosException) CosmosDiagnostics(com.azure.cosmos.CosmosDiagnostics) AccountSettings(com.azure.cosmos.examples.common.AccountSettings) CosmosItemRequestOptions(com.azure.cosmos.models.CosmosItemRequestOptions) Family(com.azure.cosmos.examples.common.Family)

Example 2 with CosmosPagedIterable

use of com.azure.cosmos.util.CosmosPagedIterable in project kafka-connect-cosmosdb by microsoft.

the class SinkConnectorIT method testPostAvroMessageWithTemplateIdStrategy.

@Test
public void testPostAvroMessageWithTemplateIdStrategy() throws InterruptedException, ExecutionException {
    // Configure Kafka Config for AVRO message
    Properties kafkaProperties = createKafkaProducerProperties();
    kafkaProperties.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, KafkaAvroSerializer.class.getName());
    kafkaProperties.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, KafkaAvroSerializer.class.getName());
    kafkaProperties.put("schema.registry.url", SCHEMA_REGISTRY_URL);
    avroProducer = new KafkaProducer<>(kafkaProperties);
    addAvroConfigs();
    // Create sink connector with template ID strategy
    connectConfig.withConfig("id.strategy", TemplateStrategy.class.getName()).withConfig("id.strategy.template", "${topic}-${key}");
    connectClient.addConnector(connectConfig.build());
    // Send Kafka message to topic
    logger.debug("Sending Kafka message to " + kafkaProperties.getProperty("bootstrap.servers"));
    String id = RandomUtils.nextLong(1L, 9999999L) + "";
    Person person = new Person("Lucy Ferr", id + "");
    String keySchema = "{\"type\": \"record\",\"name\": \"key\",\"fields\":[{\"type\": \"string\",\"name\": \"key\"}]}}";
    String valueSchema = "{\"type\": \"record\",\"fields\": " + " [{\"type\": \"string\",\"name\": \"id\"}, " + " {\"type\": \"string\",\"name\": \"name\"}], " + " \"optional\": false,\"name\": \"record\"}";
    Schema.Parser parserKey = new Schema.Parser();
    Schema schemaKey = parserKey.parse(keySchema);
    GenericRecord avroKeyRecord = new GenericData.Record(schemaKey);
    avroKeyRecord.put("key", person.getId() + "");
    Schema.Parser parser = new Schema.Parser();
    Schema schemaValue = parser.parse(valueSchema);
    GenericRecord avroValueRecord = new GenericData.Record(schemaValue);
    avroValueRecord.put("id", person.getId() + "");
    avroValueRecord.put("name", person.getName());
    ProducerRecord<GenericRecord, GenericRecord> personRecord = new ProducerRecord<>(KAFKA_TOPIC_AVRO, avroKeyRecord, avroValueRecord);
    avroProducer.send(personRecord).get();
    // Wait a few seconds for the sink connector to push data to Cosmos DB
    sleep(8000);
    // Query Cosmos DB for data
    String cosmosId = KAFKA_TOPIC_AVRO + "-{\"key\":\"" + person.getId() + "\"}";
    String sql = String.format("SELECT * FROM c where c.id = '%s'", cosmosId);
    CosmosPagedIterable<Person> readResponse = targetContainer.queryItems(sql, new CosmosQueryRequestOptions(), Person.class);
    Optional<Person> retrievedPerson = readResponse.stream().filter(p -> p.getId().equals(cosmosId)).findFirst();
    Assert.assertNotNull("Person could not be retrieved", retrievedPerson.orElse(null));
}
Also used : RandomUtils(org.apache.commons.lang3.RandomUtils) ProducerRecord(org.apache.kafka.clients.producer.ProducerRecord) KafkaAvroSerializer(io.confluent.kafka.serializers.KafkaAvroSerializer) URL(java.net.URL) URISyntaxException(java.net.URISyntaxException) CosmosDatabase(com.azure.cosmos.CosmosDatabase) LoggerFactory(org.slf4j.LoggerFactory) CosmosQueryRequestOptions(com.azure.cosmos.models.CosmosQueryRequestOptions) StringUtils(org.apache.commons.lang3.StringUtils) GenericData(org.apache.avro.generic.GenericData) KafkaProducer(org.apache.kafka.clients.producer.KafkaProducer) NewConnectorDefinition(org.sourcelab.kafka.connect.apiclient.request.dto.NewConnectorDefinition) CosmosClient(com.azure.cosmos.CosmosClient) ProvidedInKeyStrategy(com.azure.cosmos.kafka.connect.sink.id.strategy.ProvidedInKeyStrategy) ThroughputProperties(com.azure.cosmos.models.ThroughputProperties) After(org.junit.After) StringSerializer(org.apache.kafka.common.serialization.StringSerializer) JsonNode(com.fasterxml.jackson.databind.JsonNode) ProducerConfig(org.apache.kafka.clients.producer.ProducerConfig) TemplateStrategy(com.azure.cosmos.kafka.connect.sink.id.strategy.TemplateStrategy) Before(org.junit.Before) CosmosContainerProperties(com.azure.cosmos.models.CosmosContainerProperties) GenericRecord(org.apache.avro.generic.GenericRecord) Schema(org.apache.avro.Schema) Configuration(org.sourcelab.kafka.connect.apiclient.Configuration) Logger(org.slf4j.Logger) Properties(java.util.Properties) KafkaConnectClient(org.sourcelab.kafka.connect.apiclient.KafkaConnectClient) ObjectMapper(com.fasterxml.jackson.databind.ObjectMapper) ProvidedInValueStrategy(com.azure.cosmos.kafka.connect.sink.id.strategy.ProvidedInValueStrategy) JsonProcessingException(com.fasterxml.jackson.core.JsonProcessingException) Test(org.junit.Test) IOException(java.io.IOException) Category(org.junit.experimental.categories.Category) CosmosClientBuilder(com.azure.cosmos.CosmosClientBuilder) JsonSerializer(org.apache.kafka.connect.json.JsonSerializer) ExecutionException(java.util.concurrent.ExecutionException) IntegrationTest(com.azure.cosmos.kafka.connect.IntegrationTest) CosmosPagedIterable(com.azure.cosmos.util.CosmosPagedIterable) CosmosContainer(com.azure.cosmos.CosmosContainer) Optional(java.util.Optional) Utils.sleep(org.apache.kafka.common.utils.Utils.sleep) JsonMappingException(com.fasterxml.jackson.databind.JsonMappingException) Builder(org.sourcelab.kafka.connect.apiclient.request.dto.NewConnectorDefinition.Builder) Assert(org.junit.Assert) Schema(org.apache.avro.Schema) KafkaAvroSerializer(io.confluent.kafka.serializers.KafkaAvroSerializer) ThroughputProperties(com.azure.cosmos.models.ThroughputProperties) CosmosContainerProperties(com.azure.cosmos.models.CosmosContainerProperties) Properties(java.util.Properties) CosmosQueryRequestOptions(com.azure.cosmos.models.CosmosQueryRequestOptions) ProducerRecord(org.apache.kafka.clients.producer.ProducerRecord) ProducerRecord(org.apache.kafka.clients.producer.ProducerRecord) GenericRecord(org.apache.avro.generic.GenericRecord) GenericRecord(org.apache.avro.generic.GenericRecord) Test(org.junit.Test) IntegrationTest(com.azure.cosmos.kafka.connect.IntegrationTest)

Example 3 with CosmosPagedIterable

use of com.azure.cosmos.util.CosmosPagedIterable in project kafka-connect-cosmosdb by microsoft.

the class SinkConnectorIT method testPostJsonMessageWithTemplateIdStrategy.

@Test
public void testPostJsonMessageWithTemplateIdStrategy() throws InterruptedException, ExecutionException {
    // Configure Kafka Config
    Properties kafkaProperties = createKafkaProducerProperties();
    kafkaProperties.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JsonSerializer.class.getName());
    producer = new KafkaProducer<>(kafkaProperties);
    // Create sink connector with template ID strategy
    connectConfig.withConfig("id.strategy", TemplateStrategy.class.getName()).withConfig("id.strategy.template", "${topic}-${key}");
    connectClient.addConnector(connectConfig.build());
    // Send Kafka message to topic
    logger.debug("Sending Kafka message to " + kafkaProperties.getProperty("bootstrap.servers"));
    Person person = new Person("Lucy Ferr", RandomUtils.nextLong(1L, 9999999L) + "");
    ObjectMapper om = new ObjectMapper();
    ProducerRecord<String, JsonNode> personRecord = new ProducerRecord<>(kafkaTopicJson, person.getId() + "", om.valueToTree(person));
    producer.send(personRecord).get();
    // Wait a few seconds for the sink connector to push data to Cosmos DB
    sleep(8000);
    // Query Cosmos DB for data
    String id = kafkaTopicJson + "-" + person.getId();
    String sql = String.format("SELECT * FROM c where c.id = '%s'", id);
    CosmosPagedIterable<Person> readResponse = targetContainer.queryItems(sql, new CosmosQueryRequestOptions(), Person.class);
    Optional<Person> retrievedPerson = readResponse.stream().filter(p -> p.getId().equals(id)).findFirst();
    Assert.assertNotNull("Person could not be retrieved", retrievedPerson.orElse(null));
}
Also used : RandomUtils(org.apache.commons.lang3.RandomUtils) ProducerRecord(org.apache.kafka.clients.producer.ProducerRecord) KafkaAvroSerializer(io.confluent.kafka.serializers.KafkaAvroSerializer) URL(java.net.URL) URISyntaxException(java.net.URISyntaxException) CosmosDatabase(com.azure.cosmos.CosmosDatabase) LoggerFactory(org.slf4j.LoggerFactory) CosmosQueryRequestOptions(com.azure.cosmos.models.CosmosQueryRequestOptions) StringUtils(org.apache.commons.lang3.StringUtils) GenericData(org.apache.avro.generic.GenericData) KafkaProducer(org.apache.kafka.clients.producer.KafkaProducer) NewConnectorDefinition(org.sourcelab.kafka.connect.apiclient.request.dto.NewConnectorDefinition) CosmosClient(com.azure.cosmos.CosmosClient) ProvidedInKeyStrategy(com.azure.cosmos.kafka.connect.sink.id.strategy.ProvidedInKeyStrategy) ThroughputProperties(com.azure.cosmos.models.ThroughputProperties) After(org.junit.After) StringSerializer(org.apache.kafka.common.serialization.StringSerializer) JsonNode(com.fasterxml.jackson.databind.JsonNode) ProducerConfig(org.apache.kafka.clients.producer.ProducerConfig) TemplateStrategy(com.azure.cosmos.kafka.connect.sink.id.strategy.TemplateStrategy) Before(org.junit.Before) CosmosContainerProperties(com.azure.cosmos.models.CosmosContainerProperties) GenericRecord(org.apache.avro.generic.GenericRecord) Schema(org.apache.avro.Schema) Configuration(org.sourcelab.kafka.connect.apiclient.Configuration) Logger(org.slf4j.Logger) Properties(java.util.Properties) KafkaConnectClient(org.sourcelab.kafka.connect.apiclient.KafkaConnectClient) ObjectMapper(com.fasterxml.jackson.databind.ObjectMapper) ProvidedInValueStrategy(com.azure.cosmos.kafka.connect.sink.id.strategy.ProvidedInValueStrategy) JsonProcessingException(com.fasterxml.jackson.core.JsonProcessingException) Test(org.junit.Test) IOException(java.io.IOException) Category(org.junit.experimental.categories.Category) CosmosClientBuilder(com.azure.cosmos.CosmosClientBuilder) JsonSerializer(org.apache.kafka.connect.json.JsonSerializer) ExecutionException(java.util.concurrent.ExecutionException) IntegrationTest(com.azure.cosmos.kafka.connect.IntegrationTest) CosmosPagedIterable(com.azure.cosmos.util.CosmosPagedIterable) CosmosContainer(com.azure.cosmos.CosmosContainer) Optional(java.util.Optional) Utils.sleep(org.apache.kafka.common.utils.Utils.sleep) JsonMappingException(com.fasterxml.jackson.databind.JsonMappingException) Builder(org.sourcelab.kafka.connect.apiclient.request.dto.NewConnectorDefinition.Builder) Assert(org.junit.Assert) JsonNode(com.fasterxml.jackson.databind.JsonNode) JsonSerializer(org.apache.kafka.connect.json.JsonSerializer) ThroughputProperties(com.azure.cosmos.models.ThroughputProperties) CosmosContainerProperties(com.azure.cosmos.models.CosmosContainerProperties) Properties(java.util.Properties) CosmosQueryRequestOptions(com.azure.cosmos.models.CosmosQueryRequestOptions) ProducerRecord(org.apache.kafka.clients.producer.ProducerRecord) ObjectMapper(com.fasterxml.jackson.databind.ObjectMapper) Test(org.junit.Test) IntegrationTest(com.azure.cosmos.kafka.connect.IntegrationTest)

Example 4 with CosmosPagedIterable

use of com.azure.cosmos.util.CosmosPagedIterable in project kafka-connect-cosmosdb by microsoft.

the class SinkConnectorIT method testPostJsonWithSchemaMessage.

/**
 * Post a valid JSON with schema message that should go through to CosmosDB.
 * Then read the result from CosmosDB.
 */
@Test
public void testPostJsonWithSchemaMessage() throws InterruptedException, ExecutionException, JsonMappingException, JsonProcessingException {
    // Configure Kafka Config for JSON with Schema message
    Properties kafkaProperties = createKafkaProducerProperties();
    kafkaProperties.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JsonSerializer.class.getName());
    producer = new KafkaProducer<>(kafkaProperties);
    // Create sink connector with Schema enabled JSON config
    connectClient.addConnector(connectConfig.withConfig("value.converter.schemas.enable", "true").withConfig("topics", KAFKA_TOPIC_JSON_SCHEMA).withConfig("connect.cosmos.containers.topicmap", KAFKA_TOPIC_JSON_SCHEMA + "#kafka").build());
    // Send Kafka message to topic
    logger.debug("Sending Kafka message to " + kafkaProperties.getProperty("bootstrap.servers"));
    Person person = new Person("Lucy Ferr", RandomUtils.nextLong(1L, 9999999L) + "");
    String jsonSchemaString = "{\"schema\":{\"type\":\"struct\",\"fields\":[{" + "\"type\": \"string\",\"field\": \"id\"}," + "{\"type\": \"string\",\"field\": \"name\"}]," + "\"name\": \"records\"},\"payload\": {" + "\"id\":\"" + person.getId() + "\",\"name\":\"" + person.getName() + "\"}}";
    ObjectMapper om = new ObjectMapper();
    JsonNode jsonSchemaNode = om.readTree(jsonSchemaString);
    ProducerRecord<String, JsonNode> personRecord = new ProducerRecord<>(KAFKA_TOPIC_JSON_SCHEMA, person.getId(), jsonSchemaNode);
    producer.send(personRecord).get();
    // Wait a few seconds for the sink connector to push data to Cosmos DB
    sleep(8000);
    // Query Cosmos DB for data
    String sql = String.format("SELECT * FROM c where c.id = '%s'", person.getId() + "");
    CosmosPagedIterable<Person> readResponse = targetContainer.queryItems(sql, new CosmosQueryRequestOptions(), Person.class);
    Optional<Person> retrievedPerson = readResponse.stream().filter(p -> p.getId().equals(person.getId())).findFirst();
    Assert.assertNotNull("Person could not be retrieved", retrievedPerson.orElse(null));
}
Also used : RandomUtils(org.apache.commons.lang3.RandomUtils) ProducerRecord(org.apache.kafka.clients.producer.ProducerRecord) KafkaAvroSerializer(io.confluent.kafka.serializers.KafkaAvroSerializer) URL(java.net.URL) URISyntaxException(java.net.URISyntaxException) CosmosDatabase(com.azure.cosmos.CosmosDatabase) LoggerFactory(org.slf4j.LoggerFactory) CosmosQueryRequestOptions(com.azure.cosmos.models.CosmosQueryRequestOptions) StringUtils(org.apache.commons.lang3.StringUtils) GenericData(org.apache.avro.generic.GenericData) KafkaProducer(org.apache.kafka.clients.producer.KafkaProducer) NewConnectorDefinition(org.sourcelab.kafka.connect.apiclient.request.dto.NewConnectorDefinition) CosmosClient(com.azure.cosmos.CosmosClient) ProvidedInKeyStrategy(com.azure.cosmos.kafka.connect.sink.id.strategy.ProvidedInKeyStrategy) ThroughputProperties(com.azure.cosmos.models.ThroughputProperties) After(org.junit.After) StringSerializer(org.apache.kafka.common.serialization.StringSerializer) JsonNode(com.fasterxml.jackson.databind.JsonNode) ProducerConfig(org.apache.kafka.clients.producer.ProducerConfig) TemplateStrategy(com.azure.cosmos.kafka.connect.sink.id.strategy.TemplateStrategy) Before(org.junit.Before) CosmosContainerProperties(com.azure.cosmos.models.CosmosContainerProperties) GenericRecord(org.apache.avro.generic.GenericRecord) Schema(org.apache.avro.Schema) Configuration(org.sourcelab.kafka.connect.apiclient.Configuration) Logger(org.slf4j.Logger) Properties(java.util.Properties) KafkaConnectClient(org.sourcelab.kafka.connect.apiclient.KafkaConnectClient) ObjectMapper(com.fasterxml.jackson.databind.ObjectMapper) ProvidedInValueStrategy(com.azure.cosmos.kafka.connect.sink.id.strategy.ProvidedInValueStrategy) JsonProcessingException(com.fasterxml.jackson.core.JsonProcessingException) Test(org.junit.Test) IOException(java.io.IOException) Category(org.junit.experimental.categories.Category) CosmosClientBuilder(com.azure.cosmos.CosmosClientBuilder) JsonSerializer(org.apache.kafka.connect.json.JsonSerializer) ExecutionException(java.util.concurrent.ExecutionException) IntegrationTest(com.azure.cosmos.kafka.connect.IntegrationTest) CosmosPagedIterable(com.azure.cosmos.util.CosmosPagedIterable) CosmosContainer(com.azure.cosmos.CosmosContainer) Optional(java.util.Optional) Utils.sleep(org.apache.kafka.common.utils.Utils.sleep) JsonMappingException(com.fasterxml.jackson.databind.JsonMappingException) Builder(org.sourcelab.kafka.connect.apiclient.request.dto.NewConnectorDefinition.Builder) Assert(org.junit.Assert) JsonNode(com.fasterxml.jackson.databind.JsonNode) JsonSerializer(org.apache.kafka.connect.json.JsonSerializer) ThroughputProperties(com.azure.cosmos.models.ThroughputProperties) CosmosContainerProperties(com.azure.cosmos.models.CosmosContainerProperties) Properties(java.util.Properties) CosmosQueryRequestOptions(com.azure.cosmos.models.CosmosQueryRequestOptions) ProducerRecord(org.apache.kafka.clients.producer.ProducerRecord) ObjectMapper(com.fasterxml.jackson.databind.ObjectMapper) Test(org.junit.Test) IntegrationTest(com.azure.cosmos.kafka.connect.IntegrationTest)

Example 5 with CosmosPagedIterable

use of com.azure.cosmos.util.CosmosPagedIterable in project kafka-connect-cosmosdb by microsoft.

the class SinkConnectorIT method testPostAvroMessageWithJsonPathInProvidedInKeyStrategy.

@Test
public void testPostAvroMessageWithJsonPathInProvidedInKeyStrategy() throws InterruptedException, ExecutionException {
    // Configure Kafka Config for AVRO message
    Properties kafkaProperties = createKafkaProducerProperties();
    kafkaProperties.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, KafkaAvroSerializer.class.getName());
    kafkaProperties.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, KafkaAvroSerializer.class.getName());
    kafkaProperties.put("schema.registry.url", SCHEMA_REGISTRY_URL);
    avroProducer = new KafkaProducer<>(kafkaProperties);
    addAvroConfigs();
    // Create sink connector with template ID strategy
    connectConfig.withConfig("id.strategy", ProvidedInKeyStrategy.class.getName()).withConfig("id.strategy.jsonPath", "$.key");
    connectClient.addConnector(connectConfig.build());
    // Send Kafka message to topic
    logger.debug("Sending Kafka message to " + kafkaProperties.getProperty("bootstrap.servers"));
    String id = RandomUtils.nextLong(1L, 9999999L) + "";
    Person person = new Person("Lucy Ferr", id + "");
    String keySchema = "{\"type\": \"record\",\"name\": \"key\",\"fields\":[{\"type\": \"string\",\"name\": \"key\"}]}}";
    String valueSchema = "{\"type\": \"record\",\"fields\": " + " [{\"type\": \"string\",\"name\": \"id\"}, " + " {\"type\": \"string\",\"name\": \"name\"}], " + " \"optional\": false,\"name\": \"record\"}";
    Schema.Parser parserKey = new Schema.Parser();
    Schema schemaKey = parserKey.parse(keySchema);
    GenericRecord avroKeyRecord = new GenericData.Record(schemaKey);
    avroKeyRecord.put("key", person.getId() + "");
    Schema.Parser parser = new Schema.Parser();
    Schema schemaValue = parser.parse(valueSchema);
    GenericRecord avroValueRecord = new GenericData.Record(schemaValue);
    avroValueRecord.put("id", person.getId() + "");
    avroValueRecord.put("name", person.getName());
    ProducerRecord<GenericRecord, GenericRecord> personRecord = new ProducerRecord<>(KAFKA_TOPIC_AVRO, avroKeyRecord, avroValueRecord);
    avroProducer.send(personRecord).get();
    // Wait a few seconds for the sink connector to push data to Cosmos DB
    sleep(8000);
    // Query Cosmos DB for data
    String cosmosId = person.getId();
    String sql = String.format("SELECT * FROM c where c.id = '%s'", cosmosId);
    CosmosPagedIterable<Person> readResponse = targetContainer.queryItems(sql, new CosmosQueryRequestOptions(), Person.class);
    Optional<Person> retrievedPerson = readResponse.stream().filter(p -> p.getId().equals(cosmosId)).findFirst();
    Assert.assertNotNull("Person could not be retrieved", retrievedPerson.orElse(null));
}
Also used : RandomUtils(org.apache.commons.lang3.RandomUtils) ProducerRecord(org.apache.kafka.clients.producer.ProducerRecord) KafkaAvroSerializer(io.confluent.kafka.serializers.KafkaAvroSerializer) URL(java.net.URL) URISyntaxException(java.net.URISyntaxException) CosmosDatabase(com.azure.cosmos.CosmosDatabase) LoggerFactory(org.slf4j.LoggerFactory) CosmosQueryRequestOptions(com.azure.cosmos.models.CosmosQueryRequestOptions) StringUtils(org.apache.commons.lang3.StringUtils) GenericData(org.apache.avro.generic.GenericData) KafkaProducer(org.apache.kafka.clients.producer.KafkaProducer) NewConnectorDefinition(org.sourcelab.kafka.connect.apiclient.request.dto.NewConnectorDefinition) CosmosClient(com.azure.cosmos.CosmosClient) ProvidedInKeyStrategy(com.azure.cosmos.kafka.connect.sink.id.strategy.ProvidedInKeyStrategy) ThroughputProperties(com.azure.cosmos.models.ThroughputProperties) After(org.junit.After) StringSerializer(org.apache.kafka.common.serialization.StringSerializer) JsonNode(com.fasterxml.jackson.databind.JsonNode) ProducerConfig(org.apache.kafka.clients.producer.ProducerConfig) TemplateStrategy(com.azure.cosmos.kafka.connect.sink.id.strategy.TemplateStrategy) Before(org.junit.Before) CosmosContainerProperties(com.azure.cosmos.models.CosmosContainerProperties) GenericRecord(org.apache.avro.generic.GenericRecord) Schema(org.apache.avro.Schema) Configuration(org.sourcelab.kafka.connect.apiclient.Configuration) Logger(org.slf4j.Logger) Properties(java.util.Properties) KafkaConnectClient(org.sourcelab.kafka.connect.apiclient.KafkaConnectClient) ObjectMapper(com.fasterxml.jackson.databind.ObjectMapper) ProvidedInValueStrategy(com.azure.cosmos.kafka.connect.sink.id.strategy.ProvidedInValueStrategy) JsonProcessingException(com.fasterxml.jackson.core.JsonProcessingException) Test(org.junit.Test) IOException(java.io.IOException) Category(org.junit.experimental.categories.Category) CosmosClientBuilder(com.azure.cosmos.CosmosClientBuilder) JsonSerializer(org.apache.kafka.connect.json.JsonSerializer) ExecutionException(java.util.concurrent.ExecutionException) IntegrationTest(com.azure.cosmos.kafka.connect.IntegrationTest) CosmosPagedIterable(com.azure.cosmos.util.CosmosPagedIterable) CosmosContainer(com.azure.cosmos.CosmosContainer) Optional(java.util.Optional) Utils.sleep(org.apache.kafka.common.utils.Utils.sleep) JsonMappingException(com.fasterxml.jackson.databind.JsonMappingException) Builder(org.sourcelab.kafka.connect.apiclient.request.dto.NewConnectorDefinition.Builder) Assert(org.junit.Assert) Schema(org.apache.avro.Schema) KafkaAvroSerializer(io.confluent.kafka.serializers.KafkaAvroSerializer) ThroughputProperties(com.azure.cosmos.models.ThroughputProperties) CosmosContainerProperties(com.azure.cosmos.models.CosmosContainerProperties) Properties(java.util.Properties) CosmosQueryRequestOptions(com.azure.cosmos.models.CosmosQueryRequestOptions) ProducerRecord(org.apache.kafka.clients.producer.ProducerRecord) ProducerRecord(org.apache.kafka.clients.producer.ProducerRecord) GenericRecord(org.apache.avro.generic.GenericRecord) GenericRecord(org.apache.avro.generic.GenericRecord) Test(org.junit.Test) IntegrationTest(com.azure.cosmos.kafka.connect.IntegrationTest)

Aggregations

CosmosClient (com.azure.cosmos.CosmosClient)9 CosmosClientBuilder (com.azure.cosmos.CosmosClientBuilder)9 CosmosContainer (com.azure.cosmos.CosmosContainer)9 CosmosDatabase (com.azure.cosmos.CosmosDatabase)9 CosmosContainerProperties (com.azure.cosmos.models.CosmosContainerProperties)9 CosmosQueryRequestOptions (com.azure.cosmos.models.CosmosQueryRequestOptions)9 ThroughputProperties (com.azure.cosmos.models.ThroughputProperties)9 CosmosPagedIterable (com.azure.cosmos.util.CosmosPagedIterable)9 Logger (org.slf4j.Logger)9 IntegrationTest (com.azure.cosmos.kafka.connect.IntegrationTest)8 ProvidedInKeyStrategy (com.azure.cosmos.kafka.connect.sink.id.strategy.ProvidedInKeyStrategy)8 ProvidedInValueStrategy (com.azure.cosmos.kafka.connect.sink.id.strategy.ProvidedInValueStrategy)8 TemplateStrategy (com.azure.cosmos.kafka.connect.sink.id.strategy.TemplateStrategy)8 JsonProcessingException (com.fasterxml.jackson.core.JsonProcessingException)8 JsonMappingException (com.fasterxml.jackson.databind.JsonMappingException)8 JsonNode (com.fasterxml.jackson.databind.JsonNode)8 ObjectMapper (com.fasterxml.jackson.databind.ObjectMapper)8 KafkaAvroSerializer (io.confluent.kafka.serializers.KafkaAvroSerializer)8 IOException (java.io.IOException)8 URISyntaxException (java.net.URISyntaxException)8