Search in sources :

Example 16 with TypedProperties

use of org.apache.hudi.common.config.TypedProperties in project hudi by apache.

the class TestGlobalDeleteKeyGenerator method getWrongRecordKeyFieldProps.

private TypedProperties getWrongRecordKeyFieldProps() {
    TypedProperties properties = new TypedProperties();
    properties.put(KeyGeneratorOptions.RECORDKEY_FIELD_NAME.key(), "_wrong_key");
    return properties;
}
Also used : TypedProperties(org.apache.hudi.common.config.TypedProperties)

Example 17 with TypedProperties

use of org.apache.hudi.common.config.TypedProperties in project hudi by apache.

the class TestNonpartitionedKeyGenerator method getPropertiesWithoutRecordKeyProp.

private TypedProperties getPropertiesWithoutRecordKeyProp() {
    TypedProperties properties = new TypedProperties();
    properties.put(KeyGeneratorOptions.PARTITIONPATH_FIELD_NAME.key(), "timestamp");
    return properties;
}
Also used : TypedProperties(org.apache.hudi.common.config.TypedProperties)

Example 18 with TypedProperties

use of org.apache.hudi.common.config.TypedProperties in project hudi by apache.

the class TestSimpleKeyGenerator method getWrongRecordKeyFieldProps.

private TypedProperties getWrongRecordKeyFieldProps() {
    TypedProperties properties = new TypedProperties();
    properties.put(KeyGeneratorOptions.PARTITIONPATH_FIELD_NAME.key(), "timestamp");
    properties.put(KeyGeneratorOptions.RECORDKEY_FIELD_NAME.key(), "_wrong_key");
    return properties;
}
Also used : TypedProperties(org.apache.hudi.common.config.TypedProperties)

Example 19 with TypedProperties

use of org.apache.hudi.common.config.TypedProperties in project hudi by apache.

the class TestSimpleKeyGenerator method getPropsWithNestedPartitionPathField.

private TypedProperties getPropsWithNestedPartitionPathField() {
    TypedProperties properties = getCommonProps();
    properties.put(KeyGeneratorOptions.PARTITIONPATH_FIELD_NAME.key(), "nested_col.prop1");
    return properties;
}
Also used : TypedProperties(org.apache.hudi.common.config.TypedProperties)

Example 20 with TypedProperties

use of org.apache.hudi.common.config.TypedProperties in project hudi by apache.

the class TestSimpleKeyGenerator method getComplexRecordKeyProp.

private TypedProperties getComplexRecordKeyProp() {
    TypedProperties properties = new TypedProperties();
    properties.put(KeyGeneratorOptions.PARTITIONPATH_FIELD_NAME.key(), "timestamp");
    properties.put(KeyGeneratorOptions.RECORDKEY_FIELD_NAME.key(), "_row_key,pii_col");
    return properties;
}
Also used : TypedProperties(org.apache.hudi.common.config.TypedProperties)

Aggregations

TypedProperties (org.apache.hudi.common.config.TypedProperties)143 Test (org.junit.jupiter.api.Test)47 HoodieTestDataGenerator (org.apache.hudi.common.testutils.HoodieTestDataGenerator)22 JavaRDD (org.apache.spark.api.java.JavaRDD)16 ParameterizedTest (org.junit.jupiter.params.ParameterizedTest)15 IOException (java.io.IOException)14 Path (org.apache.hadoop.fs.Path)14 Properties (java.util.Properties)13 GenericRecord (org.apache.avro.generic.GenericRecord)13 SourceFormatAdapter (org.apache.hudi.utilities.deltastreamer.SourceFormatAdapter)12 Row (org.apache.spark.sql.Row)12 BeforeEach (org.junit.jupiter.api.BeforeEach)11 ArrayList (java.util.ArrayList)10 HoodieTableMetaClient (org.apache.hudi.common.table.HoodieTableMetaClient)10 HoodieKey (org.apache.hudi.common.model.HoodieKey)9 DFSPropertiesConfiguration (org.apache.hudi.common.config.DFSPropertiesConfiguration)8 HoodieWriteConfig (org.apache.hudi.config.HoodieWriteConfig)8 HoodieIOException (org.apache.hudi.exception.HoodieIOException)8 Dataset (org.apache.spark.sql.Dataset)8 HoodieRecord (org.apache.hudi.common.model.HoodieRecord)7