Examples with Buffer org.embulk.spi.Buffer used on opensource projects

Example 1 with Buffer

use of org.embulk.spi.Buffer in project embulk by embulk.

the class TestLineDecoder method bufferList.

private static List<Buffer> bufferList(Charset charset, String... sources) throws UnsupportedCharsetException {
    List<Buffer> buffers = new ArrayList<Buffer>();
    for (String source : sources) {
        ByteBuffer buffer = charset.encode(source);
        buffers.add(Buffer.wrap(buffer.array(), 0, buffer.limit()));
    }
    return buffers;
}

Example 2 with Buffer

use of org.embulk.spi.Buffer in project embulk by embulk.

the class FileInputInputStream method nextBuffer.

private boolean nextBuffer() {
    releaseBuffer();
    Buffer b = in.poll();
    if (b == null) {
        return false;
    }
    buffer = b;
    return true;
}

Example 3 with Buffer

use of org.embulk.spi.Buffer in project embulk by embulk.

the class ListFileInput method close.

public void close() {
    do {
        while (true) {
            Buffer b = poll();
            if (b == null) {
                break;
            }
            b.release();
        }
    } while (nextFile());
}

Example 4 with Buffer

use of org.embulk.spi.Buffer in project embulk by embulk.

the class LocalFileOutputPlugin method open.

@Override
public TransactionalFileOutput open(TaskSource taskSource, final int taskIndex) {
    PluginTask task = taskSource.loadTask(PluginTask.class);
    final String pathPrefix = task.getPathPrefix();
    final String pathSuffix = task.getFileNameExtension();
    final String sequenceFormat = task.getSequenceFormat();
    return new TransactionalFileOutput() {

        private final List<String> fileNames = new ArrayList<>();

        private int fileIndex = 0;

        private FileOutputStream output = null;

        public void nextFile() {
            closeFile();
            String path = pathPrefix + String.format(sequenceFormat, taskIndex, fileIndex) + pathSuffix;
            log.info("Writing local file '{}'", path);
            fileNames.add(path);
            try {
                output = new FileOutputStream(new File(path));
            } catch (FileNotFoundException ex) {
                // TODO exception class
                throw new RuntimeException(ex);
            }
            fileIndex++;
        }

        private void closeFile() {
            if (output != null) {
                try {
                    output.close();
                } catch (IOException ex) {
                    throw new RuntimeException(ex);
                }
            }
        }

        public void add(Buffer buffer) {
            try {
                output.write(buffer.array(), buffer.offset(), buffer.limit());
            } catch (IOException ex) {
                throw new RuntimeException(ex);
            } finally {
                buffer.release();
            }
        }

        public void finish() {
            closeFile();
        }

        public void close() {
            closeFile();
        }

        public void abort() {
        }

        public TaskReport commit() {
            TaskReport report = Exec.newTaskReport();
            // report.set("file_sizes", fileSizes);
            return report;
        }
    };
}

Example 5 with Buffer

use of org.embulk.spi.Buffer in project embulk by embulk.

the class SamplingParserPlugin method run.

@Override
public void run(TaskSource taskSource, Schema schema, FileInput input, PageOutput output) {
    PluginTask task = taskSource.loadTask(PluginTask.class);
    Buffer buffer = readSample(input, task.getSampleBufferBytes());
    if (!taskSource.get(boolean.class, "force", false)) {
        if (buffer.limit() < minSampleBufferBytes) {
            throw new NotEnoughSampleError(buffer.limit());
        }
    }
    throw new SampledNoticeError(buffer);
}