Examples with Path org.apache.flink.core.fs.Path used on opensource projects

Example 6 with Path

use of org.apache.flink.core.fs.Path in project flink by apache.

the class TaskTestBase method registerFileOutputTask.

public void registerFileOutputTask(AbstractInvokable outTask, FileOutputFormat<Record> outputFormat, String outPath) {
    TaskConfig dsConfig = new TaskConfig(this.mockEnv.getTaskConfiguration());
    outputFormat.setOutputFilePath(new Path(outPath));
    outputFormat.setWriteMode(WriteMode.OVERWRITE);
    dsConfig.setStubWrapper(new UserCodeObjectWrapper<>(outputFormat));
    outTask.setEnvironment(this.mockEnv);
}

Example 7 with Path

use of org.apache.flink.core.fs.Path in project flink by apache.

the class PythonPlanBinderTest method findTestFiles.

private static List<String> findTestFiles() throws Exception {
    List<String> files = new ArrayList<>();
    FileSystem fs = FileSystem.getLocalFileSystem();
    FileStatus[] status = fs.listStatus(new Path(fs.getWorkingDirectory().toString() + "/src/test/python/org/apache/flink/python/api"));
    for (FileStatus f : status) {
        String file = f.getPath().toString();
        if (file.endsWith(".py")) {
            files.add(file);
        }
    }
    return files;
}

Example 8 with Path

use of org.apache.flink.core.fs.Path in project flink by apache.

the class PythonPlanBinder method close.

private void close() {
    try {
        //prevent throwing exception so that previous exceptions aren't hidden.
        FileSystem hdfs = FileSystem.get(new URI(FLINK_HDFS_PATH));
        hdfs.delete(new Path(FLINK_HDFS_PATH), true);
        FileSystem local = FileSystem.getLocalFileSystem();
        local.delete(new Path(FLINK_PYTHON_FILE_PATH), true);
        local.delete(new Path(FLINK_TMP_DATA_DIR), true);
        streamer.close();
    } catch (NullPointerException npe) {
    } catch (IOException ioe) {
        LOG.error("PythonAPI file cleanup failed. " + ioe.getMessage());
    } catch (URISyntaxException use) {
    // can't occur
    }
}

Example 9 with Path

use of org.apache.flink.core.fs.Path in project flink by apache.

the class PythonPlanBinder method createCsvSource.

@SuppressWarnings("unchecked")
private void createCsvSource(PythonOperationInfo info) throws IOException {
    if (!(info.types instanceof TupleTypeInfo)) {
        throw new RuntimeException("The output type of a csv source has to be a tuple. The derived type is " + info);
    }
    Path path = new Path(info.path);
    String lineD = info.lineDelimiter;
    String fieldD = info.fieldDelimiter;
    TupleTypeInfo<?> types = (TupleTypeInfo) info.types;
    sets.put(info.setID, env.createInput(new TupleCsvInputFormat(path, lineD, fieldD, types), info.types).setParallelism(getParallelism(info)).name("CsvSource").map(new SerializerMap<>()).setParallelism(getParallelism(info)).name("CsvSourcePostStep"));
}

Example 10 with Path

use of org.apache.flink.core.fs.Path in project flink by apache.

the class PythonPlanBinder method copyFile.

private static void copyFile(String path, String target, String name) throws IOException, URISyntaxException {
    if (path.endsWith("/")) {
        path = path.substring(0, path.length() - 1);
    }
    String identifier = name == null ? path.substring(path.lastIndexOf("/")) : name;
    String tmpFilePath = target + "/" + identifier;
    clearPath(tmpFilePath);
    Path p = new Path(path);
    FileCache.copy(p.makeQualified(FileSystem.get(p.toUri())), new Path(tmpFilePath), true);
}