Examples with Row edu.iu.dsc.tws.common.table.Row used on opensource projects

Example 6 with Row

use of edu.iu.dsc.tws.common.table.Row in project twister2 by DSC-SPIDAL.

the class RowBatchTLinkImpl method persist.

/*
   * Similar to cache, but stores data in disk rather than in memory.
   */
public StorableTBase<Row> persist() {
    // handling checkpointing
    if (getTSetEnv().isCheckpointingEnabled()) {
        String persistVariableName = this.getId() + "-persisted";
        BatchChkPntEnvironment chkEnv = (BatchChkPntEnvironment) getTSetEnv();
        Boolean persisted = chkEnv.initVariable(persistVariableName, false);
        if (persisted) {
            // create a source function with the capability to read from disk
            DiskPartitionBackedSource<Row> sourceFn = new DiskPartitionBackedSource<>(this.getId());
            // pass the source fn to the checkpointed tset (that would create a source tset from the
            // source function, the same way as a persisted tset. This preserves the order of tsets
            // that are being created in the checkpointed env)
            CheckpointedTSet<Row> checkTSet = new CheckpointedTSet<>(getTSetEnv(), sourceFn, this.getTargetParallelism(), getSchema());
            // adding checkpointed tset to the graph, so that the IDs would not change
            addChildToGraph(checkTSet);
            // run only the checkpointed tset so that it would populate the inputs in the executor
            getTSetEnv().runOne(checkTSet);
            return checkTSet;
        } else {
            StorableTBase<Row> storable = this.doPersist();
            chkEnv.updateVariable(persistVariableName, true);
            chkEnv.commit();
            return storable;
        }
    }
    return doPersist();
}

Example 7 with Row

use of edu.iu.dsc.tws.common.table.Row in project twister2 by DSC-SPIDAL.

the class RowSourceOp method execute.

@Override
public void execute() {
    if (source.hasNext()) {
        // todo:: change source function to accept a row, so we don't have to allocate new
        // row every time
        Row tuple = source.next();
        builder.add(tuple);
        if (builder.currentSize() > tableMaxSize) {
            multiEdgeOpAdapter.writeToEdges(builder.build());
            builder = new ArrowTableBuilder(schema.toArrowSchema(), runtime.getRootAllocator());
        }
    } else {
        multiEdgeOpAdapter.writeToEdges(builder.build());
        builder = null;
        multiEdgeOpAdapter.writeEndToEdges();
    }
}