Search in sources :

Example 1 with DataValidatorResult

use of com.thinkbiganalytics.spark.datavalidator.DataValidatorResult in project kylo by Teradata.

the class ValidationStage method apply.

@Nonnull
@Override
public TransformResult apply(@Nullable final TransformResult result) {
    Preconditions.checkNotNull(result);
    // Validate the data set
    final DataValidatorResult validatorResult = validator.validate(result.getDataSet(), getPolicyMap(result.getDataSet().schema()));
    // Add the validation result to the transform result
    final List<List<TransformValidationResult>> rows = validatorResult.getCleansedRowResultRDD().map(new ListTransformValidationResults()).collect();
    result.setValidationResults(rows);
    // Add the profile to the transform result
    final List<OutputRow> profile = (result.getProfile() != null) ? new ArrayList<>(result.getProfile()) : new ArrayList<OutputRow>();
    profile.addAll(validator.getProfileStats(validatorResult));
    result.setProfile(profile);
    return result;
}
Also used : DataValidatorResult(com.thinkbiganalytics.spark.datavalidator.DataValidatorResult) OutputRow(com.thinkbiganalytics.spark.dataprofiler.output.OutputRow) ArrayList(java.util.ArrayList) List(java.util.List) Nonnull(javax.annotation.Nonnull)

Aggregations

OutputRow (com.thinkbiganalytics.spark.dataprofiler.output.OutputRow)1 DataValidatorResult (com.thinkbiganalytics.spark.datavalidator.DataValidatorResult)1 ArrayList (java.util.ArrayList)1 List (java.util.List)1 Nonnull (javax.annotation.Nonnull)1