Examples with Location org.apache.metron.dataloads.nonbulk.flatfile.location.Location used on opensource projects

Example 1 with Location

use of org.apache.metron.dataloads.nonbulk.flatfile.location.Location in project metron by apache.

the class AbstractLocalImporter method extractWholeFiles.

public void extractWholeFiles(List<String> inputs, FileSystem fs, ThreadLocal<STATE_T> state, boolean quiet) throws IOException {
    final Progress progress = new Progress();
    final List<Location> locations = getLocationsRecursive(inputs, fs);
    locations.parallelStream().forEach(loc -> {
        try (BufferedReader br = loc.openReader()) {
            String s = br.lines().collect(Collectors.joining());
            extract(state.get(), s);
            if (!quiet) {
                progress.update();
            }
        } catch (IOException e) {
            throw new IllegalStateException("Unable to read " + loc + ": " + e.getMessage(), e);
        }
    });
}