Examples with MARKERS_FILENAME_PREFIX org.apache.hudi.common.util.MarkerUtils.MARKERS_FILENAME_PREFIX used on opensource projects

Example 1 with MARKERS_FILENAME_PREFIX

use of org.apache.hudi.common.util.MarkerUtils.MARKERS_FILENAME_PREFIX in project hudi by apache.

the class TestUpgradeDowngrade method getTimelineServerBasedMarkerFileCount.

private long getTimelineServerBasedMarkerFileCount(String markerDir, FileSystem fileSystem) throws IOException {
    FileStatus[] fileStatuses = fileSystem.listStatus(new Path(markerDir));
    Predicate<String> prefixFilter = pathStr -> pathStr.contains(MARKERS_FILENAME_PREFIX);
    return Arrays.stream(fileStatuses).map(fileStatus -> fileStatus.getPath().toString()).filter(prefixFilter).collect(Collectors.toList()).stream().count();
}

Example 2 with MARKERS_FILENAME_PREFIX

use of org.apache.hudi.common.util.MarkerUtils.MARKERS_FILENAME_PREFIX in project hudi by apache.

the class TwoToOneDowngradeHandler method deleteTimelineBasedMarkerFiles.

private void deleteTimelineBasedMarkerFiles(HoodieEngineContext context, String markerDir, FileSystem fileSystem, int parallelism) throws IOException {
    // Deletes timeline based marker files if any.
    Predicate<FileStatus> prefixFilter = fileStatus -> fileStatus.getPath().getName().startsWith(MARKERS_FILENAME_PREFIX);
    FSUtils.parallelizeSubPathProcess(context, fileSystem, new Path(markerDir), parallelism, prefixFilter, pairOfSubPathAndConf -> FSUtils.deleteSubPath(pairOfSubPathAndConf.getKey(), pairOfSubPathAndConf.getValue(), false));
}