Search in sources :

Example 1 with RowConverterImpl

use of org.apache.carbondata.processing.newflow.converter.impl.RowConverterImpl in project carbondata by apache.

the class DataConverterProcessorStepImpl method initialize.

@Override
public void initialize() throws IOException {
    child.initialize();
    converters = new ArrayList<>();
    badRecordLogger = createBadRecordLogger();
    RowConverter converter = new RowConverterImpl(child.getOutput(), configuration, badRecordLogger);
    configuration.setCardinalityFinder(converter);
    converters.add(converter);
    converter.initialize();
}
Also used : RowConverterImpl(org.apache.carbondata.processing.newflow.converter.impl.RowConverterImpl) RowConverter(org.apache.carbondata.processing.newflow.converter.RowConverter)

Example 2 with RowConverterImpl

use of org.apache.carbondata.processing.newflow.converter.impl.RowConverterImpl in project carbondata by apache.

the class DataConverterProcessorWithBucketingStepImpl method initialize.

@Override
public void initialize() throws IOException {
    child.initialize();
    converters = new ArrayList<>();
    badRecordLogger = createBadRecordLogger();
    RowConverter converter = new RowConverterImpl(child.getOutput(), configuration, badRecordLogger);
    configuration.setCardinalityFinder(converter);
    converters.add(converter);
    converter.initialize();
    List<Integer> indexes = new ArrayList<>();
    List<ColumnSchema> columnSchemas = new ArrayList<>();
    DataField[] inputDataFields = getOutput();
    BucketingInfo bucketingInfo = configuration.getBucketingInfo();
    for (int i = 0; i < inputDataFields.length; i++) {
        for (int j = 0; j < bucketingInfo.getListOfColumns().size(); j++) {
            if (inputDataFields[i].getColumn().getColName().equals(bucketingInfo.getListOfColumns().get(j).getColumnName())) {
                indexes.add(i);
                columnSchemas.add(inputDataFields[i].getColumn().getColumnSchema());
                break;
            }
        }
    }
    partitioner = new HashPartitionerImpl(indexes, columnSchemas, bucketingInfo.getNumberOfBuckets());
}
Also used : RowConverterImpl(org.apache.carbondata.processing.newflow.converter.impl.RowConverterImpl) HashPartitionerImpl(org.apache.carbondata.processing.newflow.partition.impl.HashPartitionerImpl) ArrayList(java.util.ArrayList) ColumnSchema(org.apache.carbondata.core.metadata.schema.table.column.ColumnSchema) RowConverter(org.apache.carbondata.processing.newflow.converter.RowConverter) BucketingInfo(org.apache.carbondata.core.metadata.schema.BucketingInfo) DataField(org.apache.carbondata.processing.newflow.DataField)

Aggregations

RowConverter (org.apache.carbondata.processing.newflow.converter.RowConverter)2 RowConverterImpl (org.apache.carbondata.processing.newflow.converter.impl.RowConverterImpl)2 ArrayList (java.util.ArrayList)1 BucketingInfo (org.apache.carbondata.core.metadata.schema.BucketingInfo)1 ColumnSchema (org.apache.carbondata.core.metadata.schema.table.column.ColumnSchema)1 DataField (org.apache.carbondata.processing.newflow.DataField)1 HashPartitionerImpl (org.apache.carbondata.processing.newflow.partition.impl.HashPartitionerImpl)1