Search in sources :

Example 1 with TableBuilder

use of edu.iu.dsc.tws.common.table.TableBuilder in project twister2 by DSC-SPIDAL.

the class STPartition method isComplete.

@Override
public boolean isComplete() {
    for (Map.Entry<Integer, Queue<Table>> e : inputs.entrySet()) {
        if (e.getValue().isEmpty()) {
            continue;
        }
        // partition the table, default
        Table t = e.getValue().poll();
        List<ArrowColumn> columns = t.getColumns();
        ArrowColumn col = columns.get(indexes[0]);
        for (int i = 0; i < col.getVector().getValueCount(); i++) {
            Row row = new OneRow(col.get(i));
            int target = selector.next(e.getKey(), row);
            TableBuilder builder = partitionedTables.get(target);
            if (builder == null) {
                builder = new ArrowTableBuilder(schema, allocator);
                this.partitionedTables.put(target, builder);
            }
            for (int j = 0; j < columns.size(); j++) {
                builder.getColumns().get(j).addValue(columns.get(j).get(i));
            }
        }
    }
    if (finished) {
        for (Map.Entry<Integer, TableBuilder> e : partitionedTables.entrySet()) {
            Table t = e.getValue().build();
            allToAll.insert(t, e.getKey());
        }
        // clear the tables, so we won't build the tables again
        partitionedTables.clear();
        for (int s : finishedSources) {
            allToAll.finish(s);
        }
        // clear so, we won't call finish again
        finishedSources.clear();
        return allToAll.isComplete();
    }
    return false;
}
Also used : Table(edu.iu.dsc.tws.common.table.Table) ArrowTableBuilder(edu.iu.dsc.tws.common.table.ArrowTableBuilder) ArrowTableBuilder(edu.iu.dsc.tws.common.table.ArrowTableBuilder) TableBuilder(edu.iu.dsc.tws.common.table.TableBuilder) ArrowColumn(edu.iu.dsc.tws.common.table.ArrowColumn) OneRow(edu.iu.dsc.tws.common.table.OneRow) Row(edu.iu.dsc.tws.common.table.Row) OneRow(edu.iu.dsc.tws.common.table.OneRow) HashMap(java.util.HashMap) Map(java.util.Map) Queue(java.util.Queue)

Aggregations

ArrowColumn (edu.iu.dsc.tws.common.table.ArrowColumn)1 ArrowTableBuilder (edu.iu.dsc.tws.common.table.ArrowTableBuilder)1 OneRow (edu.iu.dsc.tws.common.table.OneRow)1 Row (edu.iu.dsc.tws.common.table.Row)1 Table (edu.iu.dsc.tws.common.table.Table)1 TableBuilder (edu.iu.dsc.tws.common.table.TableBuilder)1 HashMap (java.util.HashMap)1 Map (java.util.Map)1 Queue (java.util.Queue)1