Examples with HadoopFileSystem edu.iu.dsc.tws.data.hdfs.HadoopFileSystem used on opensource projects

Example 1 with HadoopFileSystem

use of edu.iu.dsc.tws.data.hdfs.HadoopFileSystem in project twister2 by DSC-SPIDAL.

the class HdfsUtils method createHDFSFileSystem.

/**
 * This method create and return the file system respective to the configuration file.
 *
 * @return hadoopFileSystem
 */
public HadoopFileSystem createHDFSFileSystem() {
    Configuration conf = new Configuration(false);
    conf.addResource(new org.apache.hadoop.fs.Path(HdfsDataContext.getHdfsConfigDirectory(config)));
    conf.addResource(new org.apache.hadoop.fs.Path(HdfsDataContext.getHdfsDataDirectory(config)));
    conf.set("fs.defaultFS", getHdfsURL());
    HadoopFileSystem hadoopFileSystem;
    try {
        hadoopFileSystem = new HadoopFileSystem(conf, org.apache.hadoop.fs.FileSystem.get(conf));
    } catch (IOException ioe) {
        throw new RuntimeException("Hadoop File System Creation Error:", ioe);
    }
    return hadoopFileSystem;
}