Examples with FileInputInputStream org.embulk.spi.util.FileInputInputStream used on opensource projects

Example 1 with FileInputInputStream

use of org.embulk.spi.util.FileInputInputStream in project embulk by embulk.

the class JsonParserPlugin method run.

@Override
public void run(TaskSource taskSource, Schema schema, FileInput input, PageOutput output) {
    PluginTask task = taskSource.loadTask(PluginTask.class);
    final boolean stopOnInvalidRecord = task.getStopOnInvalidRecord();
    // record column
    final Column column = schema.getColumn(0);
    try (PageBuilder pageBuilder = newPageBuilder(schema, output);
        FileInputInputStream in = new FileInputInputStream(input)) {
        while (in.nextFile()) {
            boolean evenOneJsonParsed = false;
            try (JsonParser.Stream stream = newJsonStream(in, task)) {
                Value value;
                while ((value = stream.next()) != null) {
                    try {
                        if (!value.isMapValue()) {
                            throw new JsonRecordValidateException(String.format("A Json record must not represent map value but it's %s", value.getValueType().name()));
                        }
                        pageBuilder.setJson(column, value);
                        pageBuilder.addRecord();
                        evenOneJsonParsed = true;
                    } catch (JsonRecordValidateException e) {
                        if (stopOnInvalidRecord) {
                            throw new DataException(String.format("Invalid record: %s", value.toJson()), e);
                        }
                        log.warn(String.format("Skipped record (%s): %s", e.getMessage(), value.toJson()));
                    }
                }
            } catch (IOException | JsonParseException e) {
                if (Exec.isPreview() && evenOneJsonParsed) {
                    // ignore in preview if at least one JSON is already parsed.
                    break;
                }
                throw new DataException(e);
            }
        }
        pageBuilder.finish();
    }
}

Example 2 with FileInputInputStream

use of org.embulk.spi.util.FileInputInputStream in project embulk by embulk.

the class TestFileInputInputStream method testSkipReturnsZeroForNoData.

@Test
public void testSkipReturnsZeroForNoData() {
    FileInputInputStream in = new FileInputInputStream(new MockFileInput());
    assertEquals("Verify skip() returns 0 when there is no data.", 0L, in.skip(1));
}

Example 3 with FileInputInputStream

use of org.embulk.spi.util.FileInputInputStream in project embulk by embulk.

the class TestFileInputInputStream method newInputStream.

private void newInputStream() {
    fileInput = new ListFileInput(fileOutput.getFiles());
    in = new FileInputInputStream(fileInput);
}

Example 4 with FileInputInputStream

use of org.embulk.spi.util.FileInputInputStream in project embulk by embulk.

the class Bzip2FileDecoderPlugin method open.

@Override
public FileInput open(TaskSource taskSource, FileInput fileInput) {
    PluginTask task = taskSource.loadTask(PluginTask.class);
    final FileInputInputStream files = new FileInputInputStream(fileInput);
    return new InputStreamFileInput(task.getBufferAllocator(), new InputStreamFileInput.Provider() {

        public InputStream openNext() throws IOException {
            if (!files.nextFile()) {
                return null;
            }
            return new BZip2CompressorInputStream(files, true);
        }

        public void close() throws IOException {
            files.close();
        }
    });
}

Example 5 with FileInputInputStream

use of org.embulk.spi.util.FileInputInputStream in project embulk by embulk.

the class GzipFileDecoderPlugin method open.

@Override
public FileInput open(TaskSource taskSource, FileInput fileInput) {
    PluginTask task = taskSource.loadTask(PluginTask.class);
    final FileInputInputStream files = new FileInputInputStream(fileInput);
    return new InputStreamFileInput(task.getBufferAllocator(), new InputStreamFileInput.Provider() {

        public InputStream openNext() throws IOException {
            if (!files.nextFile()) {
                return null;
            }
            return new GZIPInputStream(files, 8 * 1024);
        }

        public void close() throws IOException {
            files.close();
        }
    });
}