Search in sources :

Example 6 with LocalFSInputStream

use of org.apache.asterix.external.input.stream.LocalFSInputStream in project asterixdb by apache.

the class LocalFSInputStreamFactory method createInputStream.

@Override
public synchronized AsterixInputStream createInputStream(IHyracksTaskContext ctx, int partition) throws HyracksDataException {
    if (watcher == null) {
        String nodeName = ctx.getJobletContext().getServiceContext().getNodeId();
        ArrayList<Path> inputResources = new ArrayList<>();
        for (int i = 0; i < inputFileSplits.length; i++) {
            if (inputFileSplits[i].getNodeName().equals(nodeName)) {
                inputResources.add(inputFileSplits[i].getFile().toPath());
            }
        }
        watcher = new FileSystemWatcher(inputResources, expression, isFeed);
    }
    return new LocalFSInputStream(watcher);
}
Also used : Path(java.nio.file.Path) FileSystemWatcher(org.apache.asterix.external.util.FileSystemWatcher) ArrayList(java.util.ArrayList) AlgebricksAbsolutePartitionConstraint(org.apache.hyracks.algebricks.common.constraints.AlgebricksAbsolutePartitionConstraint) LocalFSInputStream(org.apache.asterix.external.input.stream.LocalFSInputStream)

Aggregations

Path (java.nio.file.Path)6 ArrayList (java.util.ArrayList)6 LocalFSInputStream (org.apache.asterix.external.input.stream.LocalFSInputStream)6 FileSystemWatcher (org.apache.asterix.external.util.FileSystemWatcher)6 HashMap (java.util.HashMap)5 SemiStructuredRecordReader (org.apache.asterix.external.input.record.reader.stream.SemiStructuredRecordReader)4 CaseInsensitiveString (org.apache.asterix.external.classad.CaseInsensitiveString)3 ClassAdObjectPool (org.apache.asterix.external.classad.object.pool.ClassAdObjectPool)3 ClassAdParser (org.apache.asterix.external.library.ClassAdParser)3 File (java.io.File)2 PrintStream (java.io.PrintStream)2 CharArrayLexerSource (org.apache.asterix.external.classad.CharArrayLexerSource)2 ClassAd (org.apache.asterix.external.classad.ClassAd)2 Value (org.apache.asterix.external.classad.Value)2 ARecordType (org.apache.asterix.om.types.ARecordType)2 IAType (org.apache.asterix.om.types.IAType)2 IPrinter (org.apache.hyracks.algebricks.data.IPrinter)2 IPrinterFactory (org.apache.hyracks.algebricks.data.IPrinterFactory)2 ISerializerDeserializer (org.apache.hyracks.api.dataflow.value.ISerializerDeserializer)2 HyracksDataException (org.apache.hyracks.api.exceptions.HyracksDataException)2