Search in sources :

Example 1 with IDSequence

use of org.apache.sysml.runtime.controlprogram.parfor.util.IDSequence in project incubator-systemml by apache.

the class ResultMergeLocalFile method copyAllFiles.

private void copyAllFiles(String fnameNew, ArrayList<MatrixObject> inMO) throws CacheException, IOException {
    JobConf job = new JobConf(ConfigurationManager.getCachedJobConf());
    Path path = new Path(fnameNew);
    FileSystem fs = IOUtilFunctions.getFileSystem(path, job);
    //create output dir
    fs.mkdirs(path);
    //merge in all input matrix objects
    IDSequence seq = new IDSequence();
    for (MatrixObject in : inMO) {
        LOG.trace("ResultMerge (local, file): Merge input " + in.getVarName() + " (fname=" + in.getFileName() + ") via file rename.");
        //copy over files (just rename file or entire dir)
        Path tmpPath = new Path(in.getFileName());
        String lname = tmpPath.getName();
        fs.rename(tmpPath, new Path(fnameNew + "/" + lname + seq.getNextID()));
    }
}
Also used : Path(org.apache.hadoop.fs.Path) MatrixObject(org.apache.sysml.runtime.controlprogram.caching.MatrixObject) IDSequence(org.apache.sysml.runtime.controlprogram.parfor.util.IDSequence) FileSystem(org.apache.hadoop.fs.FileSystem) JobConf(org.apache.hadoop.mapred.JobConf)

Aggregations

FileSystem (org.apache.hadoop.fs.FileSystem)1 Path (org.apache.hadoop.fs.Path)1 JobConf (org.apache.hadoop.mapred.JobConf)1 MatrixObject (org.apache.sysml.runtime.controlprogram.caching.MatrixObject)1 IDSequence (org.apache.sysml.runtime.controlprogram.parfor.util.IDSequence)1