Examples with SerializableConfiguration org.apache.hudi.common.config.SerializableConfiguration used on opensource projects

Example 26 with SerializableConfiguration

use of org.apache.hudi.common.config.SerializableConfiguration in project hudi by apache.

the class FSUtils method parallelizeFilesProcess.

public static <T> Map<String, T> parallelizeFilesProcess(HoodieEngineContext hoodieEngineContext, FileSystem fs, int parallelism, SerializableFunction<Pair<String, SerializableConfiguration>, T> pairFunction, List<String> subPaths) {
    Map<String, T> result = new HashMap<>();
    if (subPaths.size() > 0) {
        SerializableConfiguration conf = new SerializableConfiguration(fs.getConf());
        int actualParallelism = Math.min(subPaths.size(), parallelism);
        result = hoodieEngineContext.mapToPair(subPaths, subPath -> new ImmutablePair<>(subPath, pairFunction.apply(new ImmutablePair<>(subPath, conf))), actualParallelism);
    }
    return result;
}

Example 27 with SerializableConfiguration

use of org.apache.hudi.common.config.SerializableConfiguration in project hudi by apache.

the class FSUtils method parallelizeSubPathProcess.

/**
 * Processes sub-path in parallel.
 *
 * @param hoodieEngineContext {@code HoodieEngineContext} instance
 * @param fs file system
 * @param dirPath directory path
 * @param parallelism parallelism to use for sub-paths
 * @param subPathPredicate predicate to use to filter sub-paths for processing
 * @param pairFunction actual processing logic for each sub-path
 * @param <T> type of result to return for each sub-path
 * @return a map of sub-path to result of the processing
 */
public static <T> Map<String, T> parallelizeSubPathProcess(HoodieEngineContext hoodieEngineContext, FileSystem fs, Path dirPath, int parallelism, Predicate<FileStatus> subPathPredicate, SerializableFunction<Pair<String, SerializableConfiguration>, T> pairFunction) {
    Map<String, T> result = new HashMap<>();
    try {
        FileStatus[] fileStatuses = fs.listStatus(dirPath);
        List<String> subPaths = Arrays.stream(fileStatuses).filter(subPathPredicate).map(fileStatus -> fileStatus.getPath().toString()).collect(Collectors.toList());
        result = parallelizeFilesProcess(hoodieEngineContext, fs, parallelism, pairFunction, subPaths);
    } catch (IOException ioe) {
        throw new HoodieIOException(ioe.getMessage(), ioe);
    }
    return result;
}

Example 28 with SerializableConfiguration

use of org.apache.hudi.common.config.SerializableConfiguration in project hudi by apache.

the class TestFSUtils method testDeleteSubDirectoryRecursively.

@Test
public void testDeleteSubDirectoryRecursively() throws IOException {
    String rootDir = basePath + "/.hoodie/.temp";
    String subPathStr = rootDir + "/subdir1";
    FileSystem fileSystem = metaClient.getFs();
    prepareTestDirectory(fileSystem, rootDir);
    assertTrue(FSUtils.deleteSubPath(subPathStr, new SerializableConfiguration(fileSystem.getConf()), true));
}

Example 29 with SerializableConfiguration

use of org.apache.hudi.common.config.SerializableConfiguration in project hudi by apache.

the class TestFSUtils method testDeleteNonExistingSubDirectory.

@Test
public void testDeleteNonExistingSubDirectory() throws IOException {
    String rootDir = basePath + "/.hoodie/.temp";
    String subPathStr = rootDir + "/subdir10";
    FileSystem fileSystem = metaClient.getFs();
    cleanUpTestDirectory(fileSystem, rootDir);
    assertFalse(FSUtils.deleteSubPath(subPathStr, new SerializableConfiguration(fileSystem.getConf()), true));
}

Example 30 with SerializableConfiguration

use of org.apache.hudi.common.config.SerializableConfiguration in project hudi by apache.

the class TestFSUtils method testDeleteSubDirectoryNonRecursively.

@Test
public void testDeleteSubDirectoryNonRecursively() throws IOException {
    String rootDir = basePath + "/.hoodie/.temp";
    String subPathStr = rootDir + "/subdir1";
    FileSystem fileSystem = metaClient.getFs();
    prepareTestDirectory(fileSystem, rootDir);
    assertThrows(HoodieIOException.class, () -> FSUtils.deleteSubPath(subPathStr, new SerializableConfiguration(fileSystem.getConf()), false));
}