Search in sources :

Example 6 with ExternalScanOperatorDescriptor

use of org.apache.asterix.external.operators.ExternalScanOperatorDescriptor in project asterixdb by apache.

the class ExternalIndexingOperations method getIndexingOperator.

/**
     * This method create an indexing operator that index records in HDFS
     *
     * @param jobSpec
     * @param itemType
     * @param dataset
     * @param files
     * @param indexerDesc
     * @return
     * @throws AlgebricksException
     * @throws HyracksDataException
     * @throws Exception
     */
private static Pair<ExternalScanOperatorDescriptor, AlgebricksPartitionConstraint> getIndexingOperator(MetadataProvider metadataProvider, JobSpecification jobSpec, IAType itemType, Dataset dataset, List<ExternalFile> files, RecordDescriptor indexerDesc) throws HyracksDataException, AlgebricksException {
    ExternalDatasetDetails externalDatasetDetails = (ExternalDatasetDetails) dataset.getDatasetDetails();
    Map<String, String> configuration = externalDatasetDetails.getProperties();
    IAdapterFactory adapterFactory = AdapterFactoryProvider.getIndexingAdapterFactory(metadataProvider.getApplicationContext().getServiceContext(), externalDatasetDetails.getAdapter(), configuration, (ARecordType) itemType, files, true, null);
    return new Pair<>(new ExternalScanOperatorDescriptor(jobSpec, indexerDesc, adapterFactory), adapterFactory.getPartitionConstraint());
}
Also used : ExternalScanOperatorDescriptor(org.apache.asterix.external.operators.ExternalScanOperatorDescriptor) ExternalDatasetDetails(org.apache.asterix.metadata.entities.ExternalDatasetDetails) IAdapterFactory(org.apache.asterix.external.api.IAdapterFactory) Pair(org.apache.hyracks.algebricks.common.utils.Pair)

Aggregations

ExternalScanOperatorDescriptor (org.apache.asterix.external.operators.ExternalScanOperatorDescriptor)6 CompilationException (org.apache.asterix.common.exceptions.CompilationException)3 AlgebricksPartitionConstraint (org.apache.hyracks.algebricks.common.constraints.AlgebricksPartitionConstraint)3 AlgebricksException (org.apache.hyracks.algebricks.common.exceptions.AlgebricksException)3 Pair (org.apache.hyracks.algebricks.common.utils.Pair)3 RecordDescriptor (org.apache.hyracks.api.dataflow.value.RecordDescriptor)3 IOException (java.io.IOException)2 AsterixException (org.apache.asterix.common.exceptions.AsterixException)2 JobId (org.apache.asterix.common.transactions.JobId)2 MetadataException (org.apache.asterix.metadata.MetadataException)2 ConnectorPolicyAssignmentPolicy (org.apache.hyracks.algebricks.core.jobgen.impl.ConnectorPolicyAssignmentPolicy)2 SinkRuntimeFactory (org.apache.hyracks.algebricks.runtime.operators.base.SinkRuntimeFactory)2 AlgebricksMetaOperatorDescriptor (org.apache.hyracks.algebricks.runtime.operators.meta.AlgebricksMetaOperatorDescriptor)2 IOperatorDescriptor (org.apache.hyracks.api.dataflow.IOperatorDescriptor)2 JobSpecification (org.apache.hyracks.api.job.JobSpecification)2 AbstractOperatorDescriptor (org.apache.hyracks.dataflow.std.base.AbstractOperatorDescriptor)2 AbstractSingleActivityOperatorDescriptor (org.apache.hyracks.dataflow.std.base.AbstractSingleActivityOperatorDescriptor)2 OneToOneConnectorDescriptor (org.apache.hyracks.dataflow.std.connectors.OneToOneConnectorDescriptor)2 ExternalSortOperatorDescriptor (org.apache.hyracks.dataflow.std.sort.ExternalSortOperatorDescriptor)2 IIndexDataflowHelperFactory (org.apache.hyracks.storage.am.common.dataflow.IIndexDataflowHelperFactory)2