Search in sources :

Example 6 with AbstractUnaryOutputSourceOperatorNodePushable

use of org.apache.hyracks.dataflow.std.base.AbstractUnaryOutputSourceOperatorNodePushable in project asterixdb by apache.

the class ExternalScanOperatorDescriptor method createPushRuntime.

@Override
public IOperatorNodePushable createPushRuntime(final IHyracksTaskContext ctx, IRecordDescriptorProvider recordDescProvider, final int partition, final int nPartitions) throws HyracksDataException {
    return new AbstractUnaryOutputSourceOperatorNodePushable() {

        @Override
        public void initialize() throws HyracksDataException {
            IDataSourceAdapter adapter = null;
            try {
                writer.open();
                adapter = adapterFactory.createAdapter(ctx, partition);
                adapter.start(partition, writer);
            } catch (Exception e) {
                writer.fail();
                throw new HyracksDataException(e);
            } finally {
                writer.close();
            }
        }
    };
}
Also used : IDataSourceAdapter(org.apache.asterix.external.api.IDataSourceAdapter) AbstractUnaryOutputSourceOperatorNodePushable(org.apache.hyracks.dataflow.std.base.AbstractUnaryOutputSourceOperatorNodePushable) HyracksDataException(org.apache.hyracks.api.exceptions.HyracksDataException) HyracksDataException(org.apache.hyracks.api.exceptions.HyracksDataException)

Example 7 with AbstractUnaryOutputSourceOperatorNodePushable

use of org.apache.hyracks.dataflow.std.base.AbstractUnaryOutputSourceOperatorNodePushable in project asterixdb by apache.

the class AlgebricksMetaOperatorDescriptor method createSourceInputPushRuntime.

private IOperatorNodePushable createSourceInputPushRuntime(final IHyracksTaskContext ctx) {
    return new AbstractUnaryOutputSourceOperatorNodePushable() {

        @Override
        public void initialize() throws HyracksDataException {
            IFrameWriter startOfPipeline;
            RecordDescriptor pipelineOutputRecordDescriptor = outputArity > 0 ? AlgebricksMetaOperatorDescriptor.this.outRecDescs[0] : null;
            PipelineAssembler pa = new PipelineAssembler(pipeline, inputArity, outputArity, null, pipelineOutputRecordDescriptor);
            startOfPipeline = pa.assemblePipeline(writer, ctx);
            try {
                startOfPipeline.open();
            } catch (Exception e) {
                startOfPipeline.fail();
                throw e;
            } finally {
                startOfPipeline.close();
            }
        }
    };
}
Also used : IFrameWriter(org.apache.hyracks.api.comm.IFrameWriter) AbstractUnaryOutputSourceOperatorNodePushable(org.apache.hyracks.dataflow.std.base.AbstractUnaryOutputSourceOperatorNodePushable) RecordDescriptor(org.apache.hyracks.api.dataflow.value.RecordDescriptor) HyracksDataException(org.apache.hyracks.api.exceptions.HyracksDataException)

Aggregations

HyracksDataException (org.apache.hyracks.api.exceptions.HyracksDataException)7 AbstractUnaryOutputSourceOperatorNodePushable (org.apache.hyracks.dataflow.std.base.AbstractUnaryOutputSourceOperatorNodePushable)7 RecordDescriptor (org.apache.hyracks.api.dataflow.value.RecordDescriptor)2 IKeyValueParser (org.apache.hyracks.hdfs.api.IKeyValueParser)2 DataOutput (java.io.DataOutput)1 File (java.io.File)1 FileInputStream (java.io.FileInputStream)1 FileNotFoundException (java.io.FileNotFoundException)1 InputStream (java.io.InputStream)1 HashSet (java.util.HashSet)1 List (java.util.List)1 Random (java.util.Random)1 IDataSourceAdapter (org.apache.asterix.external.api.IDataSourceAdapter)1 InputFormat (org.apache.hadoop.mapred.InputFormat)1 InputSplit (org.apache.hadoop.mapred.InputSplit)1 JobConf (org.apache.hadoop.mapred.JobConf)1 RecordReader (org.apache.hadoop.mapred.RecordReader)1 InputFormat (org.apache.hadoop.mapreduce.InputFormat)1 Job (org.apache.hadoop.mapreduce.Job)1 RecordReader (org.apache.hadoop.mapreduce.RecordReader)1