Search in sources :

Example 1 with FailSafeConsistencyGuard

use of org.apache.hudi.common.fs.FailSafeConsistencyGuard in project hudi by apache.

the class TestConsistencyGuard method testCheckFailingAppearFailSafe.

@Test
public void testCheckFailingAppearFailSafe() throws Exception {
    FileCreateUtils.createBaseFile(basePath, "partition/path", "000", "f1");
    ConsistencyGuard passing = new FailSafeConsistencyGuard(fs, getConsistencyGuardConfig());
    assertThrows(TimeoutException.class, () -> {
        passing.waitTillAllFilesAppear(basePath + "/partition/path", Arrays.asList(basePath + "/partition/path/f1_1-0-2_000" + BASE_FILE_EXTENSION, basePath + "/partition/path/f2_1-0-2_000" + BASE_FILE_EXTENSION));
    });
}
Also used : ConsistencyGuard(org.apache.hudi.common.fs.ConsistencyGuard) FailSafeConsistencyGuard(org.apache.hudi.common.fs.FailSafeConsistencyGuard) OptimisticConsistencyGuard(org.apache.hudi.common.fs.OptimisticConsistencyGuard) FailSafeConsistencyGuard(org.apache.hudi.common.fs.FailSafeConsistencyGuard) Test(org.junit.jupiter.api.Test) ParameterizedTest(org.junit.jupiter.params.ParameterizedTest)

Example 2 with FailSafeConsistencyGuard

use of org.apache.hudi.common.fs.FailSafeConsistencyGuard in project hudi by apache.

the class TestConsistencyGuard method testCheckFailingDisappearFailSafe.

@Test
public void testCheckFailingDisappearFailSafe() throws Exception {
    FileCreateUtils.createBaseFile(basePath, "partition/path", "000", "f1");
    ConsistencyGuard passing = new FailSafeConsistencyGuard(fs, getConsistencyGuardConfig());
    assertThrows(TimeoutException.class, () -> {
        passing.waitTillAllFilesDisappear(basePath + "/partition/path", Arrays.asList(basePath + "/partition/path/f1_1-0-1_000" + BASE_FILE_EXTENSION, basePath + "/partition/path/f2_1-0-2_000" + BASE_FILE_EXTENSION));
    });
}
Also used : ConsistencyGuard(org.apache.hudi.common.fs.ConsistencyGuard) FailSafeConsistencyGuard(org.apache.hudi.common.fs.FailSafeConsistencyGuard) OptimisticConsistencyGuard(org.apache.hudi.common.fs.OptimisticConsistencyGuard) FailSafeConsistencyGuard(org.apache.hudi.common.fs.FailSafeConsistencyGuard) Test(org.junit.jupiter.api.Test) ParameterizedTest(org.junit.jupiter.params.ParameterizedTest)

Example 3 with FailSafeConsistencyGuard

use of org.apache.hudi.common.fs.FailSafeConsistencyGuard in project hudi by apache.

the class TestConsistencyGuard method testCheckFailingAppearsFailSafe.

@Test
public void testCheckFailingAppearsFailSafe() throws Exception {
    FileCreateUtils.createBaseFile(basePath, "partition/path", "000", "f1");
    ConsistencyGuard passing = new FailSafeConsistencyGuard(fs, getConsistencyGuardConfig());
    assertThrows(TimeoutException.class, () -> {
        passing.waitTillFileAppears(new Path(basePath + "/partition/path/f1_1-0-2_000" + BASE_FILE_EXTENSION));
    });
}
Also used : Path(org.apache.hadoop.fs.Path) ConsistencyGuard(org.apache.hudi.common.fs.ConsistencyGuard) FailSafeConsistencyGuard(org.apache.hudi.common.fs.FailSafeConsistencyGuard) OptimisticConsistencyGuard(org.apache.hudi.common.fs.OptimisticConsistencyGuard) FailSafeConsistencyGuard(org.apache.hudi.common.fs.FailSafeConsistencyGuard) Test(org.junit.jupiter.api.Test) ParameterizedTest(org.junit.jupiter.params.ParameterizedTest)

Example 4 with FailSafeConsistencyGuard

use of org.apache.hudi.common.fs.FailSafeConsistencyGuard in project hudi by apache.

the class TestConsistencyGuard method testCheckFailingDisappearsFailSafe.

@Test
public void testCheckFailingDisappearsFailSafe() throws Exception {
    FileCreateUtils.createBaseFile(basePath, "partition/path", "000", "f1");
    FileCreateUtils.createBaseFile(basePath, "partition/path", "000", "f1");
    ConsistencyGuard passing = new FailSafeConsistencyGuard(fs, getConsistencyGuardConfig());
    assertThrows(TimeoutException.class, () -> {
        passing.waitTillFileDisappears(new Path(basePath + "/partition/path/f1_1-0-1_000" + BASE_FILE_EXTENSION));
    });
}
Also used : Path(org.apache.hadoop.fs.Path) ConsistencyGuard(org.apache.hudi.common.fs.ConsistencyGuard) FailSafeConsistencyGuard(org.apache.hudi.common.fs.FailSafeConsistencyGuard) OptimisticConsistencyGuard(org.apache.hudi.common.fs.OptimisticConsistencyGuard) FailSafeConsistencyGuard(org.apache.hudi.common.fs.FailSafeConsistencyGuard) Test(org.junit.jupiter.api.Test) ParameterizedTest(org.junit.jupiter.params.ParameterizedTest)

Example 5 with FailSafeConsistencyGuard

use of org.apache.hudi.common.fs.FailSafeConsistencyGuard in project hudi by apache.

the class TestConsistencyGuard method testCheckPassingAppearAndDisAppear.

@ParameterizedTest
@MethodSource("consistencyGuardType")
public void testCheckPassingAppearAndDisAppear(String consistencyGuardType) throws Exception {
    FileCreateUtils.createBaseFile(basePath, "partition/path", "000", "f1");
    FileCreateUtils.createBaseFile(basePath, "partition/path", "000", "f2");
    FileCreateUtils.createBaseFile(basePath, "partition/path", "000", "f3");
    ConsistencyGuardConfig config = getConsistencyGuardConfig(1, 1000, 1000);
    ConsistencyGuard passing = consistencyGuardType.equals(FailSafeConsistencyGuard.class.getName()) ? new FailSafeConsistencyGuard(fs, config) : new OptimisticConsistencyGuard(fs, config);
    passing.waitTillFileAppears(new Path(basePath + "/partition/path/f1_1-0-1_000" + BASE_FILE_EXTENSION));
    passing.waitTillFileAppears(new Path(basePath + "/partition/path/f2_1-0-1_000" + BASE_FILE_EXTENSION));
    passing.waitTillAllFilesAppear(basePath + "/partition/path", Arrays.asList(basePath + "/partition/path/f1_1-0-1_000" + BASE_FILE_EXTENSION, basePath + "/partition/path/f2_1-0-1_000" + BASE_FILE_EXTENSION));
    fs.delete(new Path(basePath + "/partition/path/f1_1-0-1_000" + BASE_FILE_EXTENSION), false);
    fs.delete(new Path(basePath + "/partition/path/f2_1-0-1_000" + BASE_FILE_EXTENSION), false);
    passing.waitTillFileDisappears(new Path(basePath + "/partition/path/f1_1-0-1_000" + BASE_FILE_EXTENSION));
    passing.waitTillFileDisappears(new Path(basePath + "/partition/path/f2_1-0-1_000" + BASE_FILE_EXTENSION));
    passing.waitTillAllFilesDisappear(basePath + "/partition/path", Arrays.asList(basePath + "/partition/path/f1_1-0-1_000" + BASE_FILE_EXTENSION, basePath + "/partition/path/f2_1-0-1_000" + BASE_FILE_EXTENSION));
}
Also used : Path(org.apache.hadoop.fs.Path) ConsistencyGuard(org.apache.hudi.common.fs.ConsistencyGuard) FailSafeConsistencyGuard(org.apache.hudi.common.fs.FailSafeConsistencyGuard) OptimisticConsistencyGuard(org.apache.hudi.common.fs.OptimisticConsistencyGuard) ConsistencyGuardConfig(org.apache.hudi.common.fs.ConsistencyGuardConfig) FailSafeConsistencyGuard(org.apache.hudi.common.fs.FailSafeConsistencyGuard) OptimisticConsistencyGuard(org.apache.hudi.common.fs.OptimisticConsistencyGuard) ParameterizedTest(org.junit.jupiter.params.ParameterizedTest) MethodSource(org.junit.jupiter.params.provider.MethodSource)

Aggregations

FailSafeConsistencyGuard (org.apache.hudi.common.fs.FailSafeConsistencyGuard)6 ConsistencyGuard (org.apache.hudi.common.fs.ConsistencyGuard)5 OptimisticConsistencyGuard (org.apache.hudi.common.fs.OptimisticConsistencyGuard)5 ParameterizedTest (org.junit.jupiter.params.ParameterizedTest)5 Test (org.junit.jupiter.api.Test)4 Path (org.apache.hadoop.fs.Path)3 FileSystem (org.apache.hadoop.fs.FileSystem)1 ConsistencyGuardConfig (org.apache.hudi.common.fs.ConsistencyGuardConfig)1 HoodieRetryWrapperFileSystem (org.apache.hudi.common.fs.HoodieRetryWrapperFileSystem)1 HoodieWrapperFileSystem (org.apache.hudi.common.fs.HoodieWrapperFileSystem)1 NoOpConsistencyGuard (org.apache.hudi.common.fs.NoOpConsistencyGuard)1 MethodSource (org.junit.jupiter.params.provider.MethodSource)1