Search in sources :

Example 16 with AlgebricksAbsolutePartitionConstraint

use of org.apache.hyracks.algebricks.common.constraints.AlgebricksAbsolutePartitionConstraint in project asterixdb by apache.

the class MetadataProvider method getWriteFileRuntime.

@Override
public Pair<IPushRuntimeFactory, AlgebricksPartitionConstraint> getWriteFileRuntime(IDataSink sink, int[] printColumns, IPrinterFactory[] printerFactories, RecordDescriptor inputDesc) {
    FileSplitDataSink fsds = (FileSplitDataSink) sink;
    FileSplitSinkId fssi = fsds.getId();
    FileSplit fs = fssi.getFileSplit();
    File outFile = new File(fs.getPath());
    String nodeId = fs.getNodeName();
    SinkWriterRuntimeFactory runtime = new SinkWriterRuntimeFactory(printColumns, printerFactories, outFile, getWriterFactory(), inputDesc);
    AlgebricksPartitionConstraint apc = new AlgebricksAbsolutePartitionConstraint(new String[] { nodeId });
    return new Pair<>(runtime, apc);
}
Also used : SinkWriterRuntimeFactory(org.apache.hyracks.algebricks.runtime.operators.std.SinkWriterRuntimeFactory) AlgebricksAbsolutePartitionConstraint(org.apache.hyracks.algebricks.common.constraints.AlgebricksAbsolutePartitionConstraint) AlgebricksPartitionConstraint(org.apache.hyracks.algebricks.common.constraints.AlgebricksPartitionConstraint) FileSplit(org.apache.hyracks.api.io.FileSplit) ExternalFile(org.apache.asterix.external.indexing.ExternalFile) File(java.io.File) Pair(org.apache.hyracks.algebricks.common.utils.Pair)

Example 17 with AlgebricksAbsolutePartitionConstraint

use of org.apache.hyracks.algebricks.common.constraints.AlgebricksAbsolutePartitionConstraint in project asterixdb by apache.

the class StoragePathUtil method splitProviderAndPartitionConstraints.

public static Pair<IFileSplitProvider, AlgebricksPartitionConstraint> splitProviderAndPartitionConstraints(FileSplit[] splits) {
    IFileSplitProvider splitProvider = new ConstantFileSplitProvider(splits);
    String[] loc = new String[splits.length];
    for (int p = 0; p < splits.length; p++) {
        loc[p] = splits[p].getNodeName();
    }
    AlgebricksPartitionConstraint pc = new AlgebricksAbsolutePartitionConstraint(loc);
    return new Pair<>(splitProvider, pc);
}
Also used : AlgebricksAbsolutePartitionConstraint(org.apache.hyracks.algebricks.common.constraints.AlgebricksAbsolutePartitionConstraint) IFileSplitProvider(org.apache.hyracks.dataflow.std.file.IFileSplitProvider) ConstantFileSplitProvider(org.apache.hyracks.dataflow.std.file.ConstantFileSplitProvider) AlgebricksPartitionConstraint(org.apache.hyracks.algebricks.common.constraints.AlgebricksPartitionConstraint) AlgebricksPartitionConstraint(org.apache.hyracks.algebricks.common.constraints.AlgebricksPartitionConstraint) AlgebricksAbsolutePartitionConstraint(org.apache.hyracks.algebricks.common.constraints.AlgebricksAbsolutePartitionConstraint) Pair(org.apache.hyracks.algebricks.common.utils.Pair)

Example 18 with AlgebricksAbsolutePartitionConstraint

use of org.apache.hyracks.algebricks.common.constraints.AlgebricksAbsolutePartitionConstraint in project asterixdb by apache.

the class LocalFSInputStreamFactory method configurePartitionConstraint.

private void configurePartitionConstraint() throws AsterixException {
    Set<String> locs = new TreeSet<>();
    for (int i = 0; i < inputFileSplits.length; i++) {
        locs.add(inputFileSplits[i].getNodeName());
    }
    constraints = new AlgebricksAbsolutePartitionConstraint(locs.toArray(new String[locs.size()]));
}
Also used : AlgebricksAbsolutePartitionConstraint(org.apache.hyracks.algebricks.common.constraints.AlgebricksAbsolutePartitionConstraint) TreeSet(java.util.TreeSet) AlgebricksAbsolutePartitionConstraint(org.apache.hyracks.algebricks.common.constraints.AlgebricksAbsolutePartitionConstraint)

Aggregations

AlgebricksAbsolutePartitionConstraint (org.apache.hyracks.algebricks.common.constraints.AlgebricksAbsolutePartitionConstraint)18 AlgebricksPartitionConstraint (org.apache.hyracks.algebricks.common.constraints.AlgebricksPartitionConstraint)8 ArrayList (java.util.ArrayList)7 AlgebricksException (org.apache.hyracks.algebricks.common.exceptions.AlgebricksException)6 Pair (org.apache.hyracks.algebricks.common.utils.Pair)5 NodeControllerInfo (org.apache.hyracks.api.client.NodeControllerInfo)4 FileSplit (org.apache.hyracks.api.io.FileSplit)4 ILangCompilationProvider (org.apache.asterix.compiler.provider.ILangCompilationProvider)3 IOException (java.io.IOException)2 HashMap (java.util.HashMap)2 TreeSet (java.util.TreeSet)2 SinkWriterRuntimeFactory (org.apache.hyracks.algebricks.runtime.operators.std.SinkWriterRuntimeFactory)2 ISerializerDeserializer (org.apache.hyracks.api.dataflow.value.ISerializerDeserializer)2 RecordDescriptor (org.apache.hyracks.api.dataflow.value.RecordDescriptor)2 HyracksDataException (org.apache.hyracks.api.exceptions.HyracksDataException)2 JobSpecification (org.apache.hyracks.api.job.JobSpecification)2 IFileSplitProvider (org.apache.hyracks.dataflow.std.file.IFileSplitProvider)2 ObjectMapper (com.fasterxml.jackson.databind.ObjectMapper)1 File (java.io.File)1 RemoteException (java.rmi.RemoteException)1