Examples with HoodieRecordSizeEstimator org.apache.hudi.common.util.HoodieRecordSizeEstimator used on opensource projects

Example 11 with HoodieRecordSizeEstimator

use of org.apache.hudi.common.util.HoodieRecordSizeEstimator in project hudi by apache.

the class BufferedConnectWriter method init.

private void init() {
    try {
        // Load and batch all incoming records in a map
        long memoryForMerge = IOUtils.getMaxMemoryPerPartitionMerge(context.getTaskContextSupplier(), config);
        LOG.info("MaxMemoryPerPartitionMerge => " + memoryForMerge);
        this.bufferedRecords = new ExternalSpillableMap<>(memoryForMerge, config.getSpillableMapBasePath(), new DefaultSizeEstimator(), new HoodieRecordSizeEstimator(new Schema.Parser().parse(config.getSchema())), config.getCommonConfig().getSpillableDiskMapType(), config.getCommonConfig().isBitCaskDiskMapCompressionEnabled());
    } catch (IOException io) {
        throw new HoodieIOException("Cannot instantiate an ExternalSpillableMap", io);
    }
}