Examples with HoodieLocalEngineContext org.apache.hudi.common.engine.HoodieLocalEngineContext used on opensource projects

Example 6 with HoodieLocalEngineContext

use of org.apache.hudi.common.engine.HoodieLocalEngineContext in project hudi by apache.

the class AbstractSyncHoodieClient method getPartitionsWrittenToSince.

@SuppressWarnings("OptionalUsedAsFieldOrParameterType")
public List<String> getPartitionsWrittenToSince(Option<String> lastCommitTimeSynced) {
    if (!lastCommitTimeSynced.isPresent()) {
        LOG.info("Last commit time synced is not known, listing all partitions in " + basePath + ",FS :" + fs);
        HoodieLocalEngineContext engineContext = new HoodieLocalEngineContext(metaClient.getHadoopConf());
        return FSUtils.getAllPartitionPaths(engineContext, basePath, useFileListingFromMetadata, assumeDatePartitioning);
    } else {
        LOG.info("Last commit time synced is " + lastCommitTimeSynced.get() + ", Getting commits since then");
        return TimelineUtils.getPartitionsWritten(metaClient.getActiveTimeline().getCommitsTimeline().findInstantsAfter(lastCommitTimeSynced.get(), Integer.MAX_VALUE));
    }
}

Example 7 with HoodieLocalEngineContext

use of org.apache.hudi.common.engine.HoodieLocalEngineContext in project hudi by apache.

the class TimelineService method main.

public static void main(String[] args) throws Exception {
    final Config cfg = new Config();
    JCommander cmd = new JCommander(cfg, null, args);
    if (cfg.help) {
        cmd.usage();
        System.exit(1);
    }
    Configuration conf = FSUtils.prepareHadoopConf(new Configuration());
    FileSystemViewManager viewManager = buildFileSystemViewManager(cfg, new SerializableConfiguration(conf));
    TimelineService service = new TimelineService(new HoodieLocalEngineContext(FSUtils.prepareHadoopConf(new Configuration())), new Configuration(), cfg, FileSystem.get(new Configuration()), viewManager);
    service.run();
}

Example 8 with HoodieLocalEngineContext

use of org.apache.hudi.common.engine.HoodieLocalEngineContext in project hudi by apache.

the class TestFSUtils method testDeleteNonExistingDir.

@Test
public void testDeleteNonExistingDir() throws IOException {
    String rootDir = basePath + "/.hoodie/.temp";
    FileSystem fileSystem = metaClient.getFs();
    cleanUpTestDirectory(fileSystem, rootDir);
    assertFalse(FSUtils.deleteDir(new HoodieLocalEngineContext(metaClient.getHadoopConf()), fileSystem, new Path(rootDir), 2));
}

Example 9 with HoodieLocalEngineContext

use of org.apache.hudi.common.engine.HoodieLocalEngineContext in project hudi by apache.

the class TestFSUtils method testDeleteExistingDir.

@Test
public void testDeleteExistingDir() throws IOException {
    String rootDir = basePath + "/.hoodie/.temp";
    FileSystem fileSystem = metaClient.getFs();
    prepareTestDirectory(fileSystem, rootDir);
    Path rootDirPath = new Path(rootDir);
    assertTrue(fileSystem.exists(rootDirPath));
    assertTrue(FSUtils.deleteDir(new HoodieLocalEngineContext(metaClient.getHadoopConf()), fileSystem, rootDirPath, 2));
    assertFalse(fileSystem.exists(rootDirPath));
}

Example 10 with HoodieLocalEngineContext

use of org.apache.hudi.common.engine.HoodieLocalEngineContext in project hudi by apache.

the class TestFSUtils method testGetFileStatusAtLevel.

@Test
public void testGetFileStatusAtLevel() throws IOException {
    String rootDir = basePath + "/.hoodie/.temp";
    FileSystem fileSystem = metaClient.getFs();
    prepareTestDirectory(fileSystem, rootDir);
    List<FileStatus> fileStatusList = FSUtils.getFileStatusAtLevel(new HoodieLocalEngineContext(fileSystem.getConf()), fileSystem, new Path(basePath), 3, 2);
    assertEquals(CollectionUtils.createImmutableSet(basePath + "/.hoodie/.temp/subdir1/file1.txt", basePath + "/.hoodie/.temp/subdir2/file2.txt"), fileStatusList.stream().map(fileStatus -> fileStatus.getPath().toString()).filter(filePath -> filePath.endsWith(".txt")).collect(Collectors.toSet()));
}