Search in sources :

Example 1 with IterativeSVMDataObjectDirectSink

use of edu.iu.dsc.tws.examples.ml.svm.data.IterativeSVMDataObjectDirectSink in project twister2 by DSC-SPIDAL.

the class TGUtils method generateGenericDataPointLoader.

public static ComputeGraph generateGenericDataPointLoader(int samples, int parallelism, int numOfFeatures, String dataSourcePathStr, String dataObjectSourceStr, String dataObjectComputeStr, String dataObjectSinkStr, String graphName, Config config, OperationMode opMode) {
    SVMDataObjectSource<String, TextInputSplit> sourceTask = new SVMDataObjectSource(Context.TWISTER2_DIRECT_EDGE, dataSourcePathStr, samples);
    IterativeSVMDataObjectCompute dataObjectCompute = new IterativeSVMDataObjectCompute(Context.TWISTER2_DIRECT_EDGE, parallelism, samples, numOfFeatures, DELIMITER);
    IterativeSVMDataObjectDirectSink iterativeSVMPrimaryDataObjectDirectSink = new IterativeSVMDataObjectDirectSink();
    ComputeGraphBuilder datapointsComputeGraphBuilder = ComputeGraphBuilder.newBuilder(config);
    datapointsComputeGraphBuilder.addSource(dataObjectSourceStr, sourceTask, parallelism);
    ComputeConnection datapointComputeConnection = datapointsComputeGraphBuilder.addCompute(dataObjectComputeStr, dataObjectCompute, parallelism);
    ComputeConnection computeConnectionSink = datapointsComputeGraphBuilder.addCompute(dataObjectSinkStr, iterativeSVMPrimaryDataObjectDirectSink, parallelism);
    datapointComputeConnection.direct(dataObjectSourceStr).viaEdge(Context.TWISTER2_DIRECT_EDGE).withDataType(MessageTypes.OBJECT);
    computeConnectionSink.direct(dataObjectComputeStr).viaEdge(Context.TWISTER2_DIRECT_EDGE).withDataType(MessageTypes.OBJECT);
    datapointsComputeGraphBuilder.setMode(opMode);
    datapointsComputeGraphBuilder.setTaskGraphName(graphName);
    // Build the first taskgraph
    return datapointsComputeGraphBuilder.build();
}
Also used : TextInputSplit(edu.iu.dsc.tws.data.api.splits.TextInputSplit) IterativeSVMDataObjectDirectSink(edu.iu.dsc.tws.examples.ml.svm.data.IterativeSVMDataObjectDirectSink) IterativeSVMDataObjectCompute(edu.iu.dsc.tws.examples.ml.svm.data.IterativeSVMDataObjectCompute) ComputeGraphBuilder(edu.iu.dsc.tws.task.impl.ComputeGraphBuilder) SVMDataObjectSource(edu.iu.dsc.tws.examples.ml.svm.data.SVMDataObjectSource) ComputeConnection(edu.iu.dsc.tws.task.impl.ComputeConnection)

Example 2 with IterativeSVMDataObjectDirectSink

use of edu.iu.dsc.tws.examples.ml.svm.data.IterativeSVMDataObjectDirectSink in project twister2 by DSC-SPIDAL.

the class SvmSgdIterativeRunner method generateGenericDataPointLoader.

private ComputeGraph generateGenericDataPointLoader(int samples, int parallelism, int numOfFeatures, String dataSourcePathStr, String dataObjectSourceStr, String dataObjectComputeStr, String dataObjectSinkStr, String graphName) {
    SVMDataObjectSource<String, TextInputSplit> sourceTask = new SVMDataObjectSource(Context.TWISTER2_DIRECT_EDGE, dataSourcePathStr, samples);
    IterativeSVMDataObjectCompute dataObjectCompute = new IterativeSVMDataObjectCompute(Context.TWISTER2_DIRECT_EDGE, parallelism, samples, numOfFeatures, DELIMITER);
    IterativeSVMDataObjectDirectSink iterativeSVMPrimaryDataObjectDirectSink = new IterativeSVMDataObjectDirectSink();
    ComputeGraphBuilder datapointsComputeGraphBuilder = ComputeGraphBuilder.newBuilder(config);
    datapointsComputeGraphBuilder.addSource(dataObjectSourceStr, sourceTask, parallelism);
    ComputeConnection datapointComputeConnection = datapointsComputeGraphBuilder.addCompute(dataObjectComputeStr, dataObjectCompute, parallelism);
    ComputeConnection computeConnectionSink = datapointsComputeGraphBuilder.addCompute(dataObjectSinkStr, iterativeSVMPrimaryDataObjectDirectSink, parallelism);
    datapointComputeConnection.direct(dataObjectSourceStr).viaEdge(Context.TWISTER2_DIRECT_EDGE).withDataType(MessageTypes.OBJECT);
    computeConnectionSink.direct(dataObjectComputeStr).viaEdge(Context.TWISTER2_DIRECT_EDGE).withDataType(MessageTypes.OBJECT);
    datapointsComputeGraphBuilder.setMode(this.operationMode);
    datapointsComputeGraphBuilder.setTaskGraphName(graphName);
    // Build the first taskgraph
    return datapointsComputeGraphBuilder.build();
}
Also used : TextInputSplit(edu.iu.dsc.tws.data.api.splits.TextInputSplit) IterativeSVMDataObjectDirectSink(edu.iu.dsc.tws.examples.ml.svm.data.IterativeSVMDataObjectDirectSink) IterativeSVMDataObjectCompute(edu.iu.dsc.tws.examples.ml.svm.data.IterativeSVMDataObjectCompute) ComputeGraphBuilder(edu.iu.dsc.tws.task.impl.ComputeGraphBuilder) SVMDataObjectSource(edu.iu.dsc.tws.examples.ml.svm.data.SVMDataObjectSource) ComputeConnection(edu.iu.dsc.tws.task.impl.ComputeConnection)

Aggregations

TextInputSplit (edu.iu.dsc.tws.data.api.splits.TextInputSplit)2 IterativeSVMDataObjectCompute (edu.iu.dsc.tws.examples.ml.svm.data.IterativeSVMDataObjectCompute)2 IterativeSVMDataObjectDirectSink (edu.iu.dsc.tws.examples.ml.svm.data.IterativeSVMDataObjectDirectSink)2 SVMDataObjectSource (edu.iu.dsc.tws.examples.ml.svm.data.SVMDataObjectSource)2 ComputeConnection (edu.iu.dsc.tws.task.impl.ComputeConnection)2 ComputeGraphBuilder (edu.iu.dsc.tws.task.impl.ComputeGraphBuilder)2