Search in sources :

Example 41 with FileSystem

use of org.apache.flink.core.fs.FileSystem in project flink by apache.

the class YarnPreConfiguredMasterHaServicesTest method testCloseAndCleanup.

@Test
public void testCloseAndCleanup() throws Exception {
    final Configuration flinkConfig = new Configuration();
    flinkConfig.setString(YarnConfigOptions.APP_MASTER_RPC_ADDRESS, "localhost");
    flinkConfig.setInteger(YarnConfigOptions.APP_MASTER_RPC_PORT, 1427);
    // create the services
    YarnHighAvailabilityServices services = new YarnPreConfiguredMasterNonHaServices(flinkConfig, hadoopConfig);
    services.closeAndCleanupAllData();
    final FileSystem fileSystem = HDFS_ROOT_PATH.getFileSystem();
    final Path workDir = new Path(HDFS_CLUSTER.getFileSystem().getWorkingDirectory().toString());
    try {
        fileSystem.getFileStatus(new Path(workDir, YarnHighAvailabilityServices.FLINK_RECOVERY_DATA_DIR));
        fail("Flink recovery data directory still exists");
    } catch (FileNotFoundException e) {
    // expected, because the directory should have been cleaned up
    }
    assertTrue(services.isClosed());
    // doing another cleanup when the services are closed should fail
    try {
        services.closeAndCleanupAllData();
        fail("should fail with an IllegalStateException");
    } catch (IllegalStateException e) {
    // expected
    }
}
Also used : Path(org.apache.flink.core.fs.Path) Configuration(org.apache.flink.configuration.Configuration) FileSystem(org.apache.flink.core.fs.FileSystem) FileNotFoundException(java.io.FileNotFoundException) Test(org.junit.Test)

Aggregations

FileSystem (org.apache.flink.core.fs.FileSystem)41 Path (org.apache.flink.core.fs.Path)34 IOException (java.io.IOException)18 FileStatus (org.apache.flink.core.fs.FileStatus)13 ArrayList (java.util.ArrayList)8 Test (org.junit.Test)8 FSDataInputStream (org.apache.flink.core.fs.FSDataInputStream)6 FSDataOutputStream (org.apache.flink.core.fs.FSDataOutputStream)6 File (java.io.File)5 URI (java.net.URI)5 URISyntaxException (java.net.URISyntaxException)4 FileNotFoundException (java.io.FileNotFoundException)3 FileInputSplit (org.apache.flink.core.fs.FileInputSplit)3 DataInputViewStreamWrapper (org.apache.flink.core.memory.DataInputViewStreamWrapper)3 FileStateHandle (org.apache.flink.runtime.state.filesystem.FileStateHandle)3 DataOutputStream (java.io.DataOutputStream)2 InputStream (java.io.InputStream)2 Field (java.lang.reflect.Field)2 Map (java.util.Map)2 FileBaseStatistics (org.apache.flink.api.common.io.FileInputFormat.FileBaseStatistics)2