Search in sources :

Example 1 with GoogleByteSource

use of io.druid.storage.google.GoogleByteSource in project druid by druid-io.

the class StaticGoogleBlobStoreFirehoseFactory method connect.

@Override
public Firehose connect(StringInputRowParser stringInputRowParser) throws IOException {
    Preconditions.checkNotNull(storage, "null storage");
    final LinkedList<GoogleBlob> objectQueue = Lists.newLinkedList(blobs);
    return new FileIteratingFirehose(new Iterator<LineIterator>() {

        @Override
        public boolean hasNext() {
            return !objectQueue.isEmpty();
        }

        @Override
        public LineIterator next() {
            final GoogleBlob nextURI = objectQueue.poll();
            final String bucket = nextURI.getBucket();
            final String path = nextURI.getPath().startsWith("/") ? nextURI.getPath().substring(1) : nextURI.getPath();
            try {
                final InputStream innerInputStream = new GoogleByteSource(storage, bucket, path).openStream();
                final InputStream outerInputStream = path.endsWith(".gz") ? CompressionUtils.gzipInputStream(innerInputStream) : innerInputStream;
                return IOUtils.lineIterator(new BufferedReader(new InputStreamReader(outerInputStream, Charsets.UTF_8)));
            } catch (Exception e) {
                LOG.error(e, "Exception opening bucket[%s] blob[%s]", bucket, path);
                throw Throwables.propagate(e);
            }
        }

        @Override
        public void remove() {
            throw new UnsupportedOperationException();
        }
    }, stringInputRowParser);
}
Also used : InputStreamReader(java.io.InputStreamReader) InputStream(java.io.InputStream) LineIterator(org.apache.commons.io.LineIterator) IOException(java.io.IOException) BufferedReader(java.io.BufferedReader) FileIteratingFirehose(io.druid.data.input.impl.FileIteratingFirehose) GoogleByteSource(io.druid.storage.google.GoogleByteSource)

Aggregations

FileIteratingFirehose (io.druid.data.input.impl.FileIteratingFirehose)1 GoogleByteSource (io.druid.storage.google.GoogleByteSource)1 BufferedReader (java.io.BufferedReader)1 IOException (java.io.IOException)1 InputStream (java.io.InputStream)1 InputStreamReader (java.io.InputStreamReader)1 LineIterator (org.apache.commons.io.LineIterator)1