Examples with TypedProperties org.apache.hudi.common.config.TypedProperties used on opensource projects

Example 56 with TypedProperties

use of org.apache.hudi.common.config.TypedProperties in project hudi by apache.

the class TestCloudObjectsSelector method setUp.

@BeforeEach
void setUp() {
    initSparkContexts();
    initPath();
    initFileSystem();
    MockitoAnnotations.initMocks(this);
    props = new TypedProperties();
    sqsUrl = "test-queue";
    props.setProperty(S3_SOURCE_QUEUE_URL, sqsUrl);
    props.setProperty(S3_SOURCE_QUEUE_REGION, REGION_NAME);
}

Example 57 with TypedProperties

use of org.apache.hudi.common.config.TypedProperties in project hudi by apache.

the class TestDatePartitionPathSelector method getProps.

private static TypedProperties getProps(String basePath, String dateFormat, int datePartitionDepth, int numDaysToList, String currentDate) {
    TypedProperties properties = new TypedProperties();
    properties.put(ROOT_INPUT_PATH_PROP, basePath);
    properties.put(DATE_FORMAT, dateFormat);
    properties.put(DATE_PARTITION_DEPTH, "" + datePartitionDepth);
    properties.put(LOOKBACK_DAYS, "" + numDaysToList);
    properties.put(CURRENT_DATE, currentDate);
    return properties;
}

Example 58 with TypedProperties

use of org.apache.hudi.common.config.TypedProperties in project hudi by apache.

the class TestKafkaOffsetGen method testCheckTopicExists.

@Test
public void testCheckTopicExists() {
    TypedProperties props = getConsumerConfigs("latest", "string");
    KafkaOffsetGen kafkaOffsetGen = new KafkaOffsetGen(props);
    testUtils.createTopic(TEST_TOPIC_NAME, 1);
    boolean topicExists = kafkaOffsetGen.checkTopicExists(new KafkaConsumer(props));
    assertTrue(topicExists);
    props.put("hoodie.deltastreamer.source.kafka.topic", "random");
    kafkaOffsetGen = new KafkaOffsetGen(props);
    topicExists = kafkaOffsetGen.checkTopicExists(new KafkaConsumer(props));
    assertFalse(topicExists);
}

Example 59 with TypedProperties

use of org.apache.hudi.common.config.TypedProperties in project hudi by apache.

the class TestSchemaRegistryProvider method testGetSourceSchemaShouldRequestSchemaWithoutCreds.

@Test
public void testGetSourceSchemaShouldRequestSchemaWithoutCreds() throws IOException {
    TypedProperties props = getProps();
    props.put("hoodie.deltastreamer.schemaprovider.registry.url", "http://localhost");
    SchemaRegistryProvider spyUnderTest = getUnderTest(props);
    Schema actual = spyUnderTest.getSourceSchema();
    assertNotNull(actual);
    assertEquals(actual, getExpectedSchema(json));
    verify(spyUnderTest, times(0)).setAuthorizationHeader(Mockito.any(), Mockito.any());
}

Example 60 with TypedProperties

use of org.apache.hudi.common.config.TypedProperties in project hudi by apache.

the class TestSchemaRegistryProvider method testGetTargetSchemaShouldRequestSchemaWithoutCreds.

@Test
public void testGetTargetSchemaShouldRequestSchemaWithoutCreds() throws IOException {
    TypedProperties props = getProps();
    props.put("hoodie.deltastreamer.schemaprovider.registry.url", "http://localhost");
    SchemaRegistryProvider spyUnderTest = getUnderTest(props);
    Schema actual = spyUnderTest.getTargetSchema();
    assertNotNull(actual);
    assertEquals(actual, getExpectedSchema(json));
    verify(spyUnderTest, times(0)).setAuthorizationHeader(Mockito.any(), Mockito.any());
}