Search in sources :

Example 11 with StandardizationAndValidationResult

use of com.thinkbiganalytics.spark.datavalidator.StandardizationAndValidationResult in project kylo by Teradata.

the class CleanseAndValidateRowTest method standardizeAndValidate.

@Test
public void standardizeAndValidate() {
    String fieldName = "field1";
    List<BaseFieldPolicy> policies = new ArrayList<>();
    policies.add(new SimpleRegexReplacer("(?i)foo", "bar"));
    policies.add(new LookupValidator("aabaraa"));
    policies.add(new SimpleRegexReplacer("(?i)bar", "test"));
    policies.add(new LookupValidator("aatestaa"));
    FieldPolicy fieldPolicy = FieldPolicyBuilder.newBuilder().addPolicies(policies).tableName("emp").fieldName(fieldName).feedFieldName(fieldName).build();
    HCatDataType fieldDataType = HCatDataType.createFromDataType(fieldName, "string");
    StandardizationAndValidationResult result = validator.standardizeAndValidateField(fieldPolicy, "aafooaa", fieldDataType, new HashMap<Class, Class>());
    assertEquals(result.getFieldValue(), "aatestaa");
    assertEquals(StandardDataValidator.VALID_RESULT, result.getFinalValidationResult());
}
Also used : FieldPolicy(com.thinkbiganalytics.policy.FieldPolicy) BaseFieldPolicy(com.thinkbiganalytics.policy.BaseFieldPolicy) HCatDataType(com.thinkbiganalytics.spark.validation.HCatDataType) ArrayList(java.util.ArrayList) LookupValidator(com.thinkbiganalytics.policy.validation.LookupValidator) BaseFieldPolicy(com.thinkbiganalytics.policy.BaseFieldPolicy) SimpleRegexReplacer(com.thinkbiganalytics.policy.standardization.SimpleRegexReplacer) StandardizationAndValidationResult(com.thinkbiganalytics.spark.datavalidator.StandardizationAndValidationResult) Test(org.junit.Test)

Aggregations

BaseFieldPolicy (com.thinkbiganalytics.policy.BaseFieldPolicy)11 StandardizationAndValidationResult (com.thinkbiganalytics.spark.datavalidator.StandardizationAndValidationResult)11 FieldPolicy (com.thinkbiganalytics.policy.FieldPolicy)10 ArrayList (java.util.ArrayList)9 HCatDataType (com.thinkbiganalytics.spark.validation.HCatDataType)8 Test (org.junit.Test)7 SimpleRegexReplacer (com.thinkbiganalytics.policy.standardization.SimpleRegexReplacer)4 LookupValidator (com.thinkbiganalytics.policy.validation.LookupValidator)3 StandardizationPolicy (com.thinkbiganalytics.policy.standardization.StandardizationPolicy)2 NotNullValidator (com.thinkbiganalytics.policy.validation.NotNullValidator)2 ValidationResult (com.thinkbiganalytics.policy.validation.ValidationResult)2 HashMap (java.util.HashMap)2 AcceptsEmptyValues (com.thinkbiganalytics.policy.standardization.AcceptsEmptyValues)1 CharacterValidator (com.thinkbiganalytics.policy.validation.CharacterValidator)1 RangeValidator (com.thinkbiganalytics.policy.validation.RangeValidator)1 ValidationPolicy (com.thinkbiganalytics.policy.validation.ValidationPolicy)1 CleansedRowResult (com.thinkbiganalytics.spark.datavalidator.CleansedRowResult)1 InvalidFormatException (com.thinkbiganalytics.spark.util.InvalidFormatException)1