Examples with ConsistencyGuard org.apache.hudi.common.fs.ConsistencyGuard used on opensource projects

Example 6 with ConsistencyGuard

use of org.apache.hudi.common.fs.ConsistencyGuard in project hudi by apache.

the class TestConsistencyGuard method testCheckFailingDisappearTimedWait.

@Test
public void testCheckFailingDisappearTimedWait() throws Exception {
    FileCreateUtils.createBaseFile(basePath, "partition/path", "000", "f1");
    ConsistencyGuard passing = new OptimisticConsistencyGuard(fs, getConsistencyGuardConfig());
    passing.waitTillAllFilesDisappear(basePath + "/partition/path", Arrays.asList(basePath + "/partition/path/f1_1-0-1_000" + BASE_FILE_EXTENSION, basePath + "/partition/path/f2_1-0-2_000" + BASE_FILE_EXTENSION));
}

Example 7 with ConsistencyGuard

use of org.apache.hudi.common.fs.ConsistencyGuard in project hudi by apache.

the class TestConsistencyGuard method testCheckPassingAppearAndDisAppear.

@ParameterizedTest
@MethodSource("consistencyGuardType")
public void testCheckPassingAppearAndDisAppear(String consistencyGuardType) throws Exception {
    FileCreateUtils.createBaseFile(basePath, "partition/path", "000", "f1");
    FileCreateUtils.createBaseFile(basePath, "partition/path", "000", "f2");
    FileCreateUtils.createBaseFile(basePath, "partition/path", "000", "f3");
    ConsistencyGuardConfig config = getConsistencyGuardConfig(1, 1000, 1000);
    ConsistencyGuard passing = consistencyGuardType.equals(FailSafeConsistencyGuard.class.getName()) ? new FailSafeConsistencyGuard(fs, config) : new OptimisticConsistencyGuard(fs, config);
    passing.waitTillFileAppears(new Path(basePath + "/partition/path/f1_1-0-1_000" + BASE_FILE_EXTENSION));
    passing.waitTillFileAppears(new Path(basePath + "/partition/path/f2_1-0-1_000" + BASE_FILE_EXTENSION));
    passing.waitTillAllFilesAppear(basePath + "/partition/path", Arrays.asList(basePath + "/partition/path/f1_1-0-1_000" + BASE_FILE_EXTENSION, basePath + "/partition/path/f2_1-0-1_000" + BASE_FILE_EXTENSION));
    fs.delete(new Path(basePath + "/partition/path/f1_1-0-1_000" + BASE_FILE_EXTENSION), false);
    fs.delete(new Path(basePath + "/partition/path/f2_1-0-1_000" + BASE_FILE_EXTENSION), false);
    passing.waitTillFileDisappears(new Path(basePath + "/partition/path/f1_1-0-1_000" + BASE_FILE_EXTENSION));
    passing.waitTillFileDisappears(new Path(basePath + "/partition/path/f2_1-0-1_000" + BASE_FILE_EXTENSION));
    passing.waitTillAllFilesDisappear(basePath + "/partition/path", Arrays.asList(basePath + "/partition/path/f1_1-0-1_000" + BASE_FILE_EXTENSION, basePath + "/partition/path/f2_1-0-1_000" + BASE_FILE_EXTENSION));
}

Example 8 with ConsistencyGuard

use of org.apache.hudi.common.fs.ConsistencyGuard in project hudi by apache.

the class TestConsistencyGuard method testCheckFailingDisappearsTimedWait.

@Test
public void testCheckFailingDisappearsTimedWait() throws Exception {
    FileCreateUtils.createBaseFile(basePath, "partition/path", "000", "f1");
    FileCreateUtils.createBaseFile(basePath, "partition/path", "000", "f1");
    ConsistencyGuard passing = new OptimisticConsistencyGuard(fs, getConsistencyGuardConfig());
    passing.waitTillFileDisappears(new Path(basePath + "/partition/path/f1_1-0-1_000" + BASE_FILE_EXTENSION));
}

Example 9 with ConsistencyGuard

use of org.apache.hudi.common.fs.ConsistencyGuard in project hudi by apache.

the class TestConsistencyGuard method testCheckFailingAppearsTimedWait.

@Test
public void testCheckFailingAppearsTimedWait() throws Exception {
    FileCreateUtils.createBaseFile(basePath, "partition/path", "000", "f1");
    ConsistencyGuard passing = new OptimisticConsistencyGuard(fs, getConsistencyGuardConfig());
    passing.waitTillFileAppears(new Path(basePath + "/partition/path/f1_1-0-2_000" + BASE_FILE_EXTENSION));
}