Search in sources :

Example 1 with MutableInt

use of org.apache.commons.lang3.mutable.MutableInt in project hive by apache.

the class DynamicPartitionPruner method initialize.

private void initialize() throws SerDeException {
    this.clear();
    Map<String, SourceInfo> columnMap = new HashMap<String, SourceInfo>();
    // sources represent vertex names
    Set<String> sources = work.getEventSourceTableDescMap().keySet();
    sourcesWaitingForEvents.addAll(sources);
    for (String s : sources) {
        // Set to 0 to start with. This will be decremented for all columns for which events
        // are generated by this source - which is eventually used to determine number of expected
        // events for the source. #colums X #tasks
        numExpectedEventsPerSource.put(s, new MutableInt(0));
        numEventsSeenPerSource.put(s, new MutableInt(0));
        // Virtual relation generated by the reduce sync
        List<TableDesc> tables = work.getEventSourceTableDescMap().get(s);
        // Real column name - on which the operation is being performed
        List<String> columnNames = work.getEventSourceColumnNameMap().get(s);
        // Column type
        List<String> columnTypes = work.getEventSourceColumnTypeMap().get(s);
        // Expression for the operation. e.g. N^2 > 10
        List<ExprNodeDesc> partKeyExprs = work.getEventSourcePartKeyExprMap().get(s);
        // eventSourceTableDesc, eventSourceColumnName, evenSourcePartKeyExpr move in lock-step.
        // One entry is added to each at the same time
        Iterator<String> cit = columnNames.iterator();
        Iterator<String> typit = columnTypes.iterator();
        Iterator<ExprNodeDesc> pit = partKeyExprs.iterator();
        // A single source can process multiple columns, and will send an event for each of them.
        for (TableDesc t : tables) {
            numExpectedEventsPerSource.get(s).decrement();
            ++sourceInfoCount;
            String columnName = cit.next();
            String columnType = typit.next();
            ExprNodeDesc partKeyExpr = pit.next();
            SourceInfo si = createSourceInfo(t, partKeyExpr, columnName, columnType, jobConf);
            if (!sourceInfoMap.containsKey(s)) {
                sourceInfoMap.put(s, new ArrayList<SourceInfo>());
            }
            List<SourceInfo> sis = sourceInfoMap.get(s);
            sis.add(si);
            // the union of the values in that case.
            if (columnMap.containsKey(columnName)) {
                // All Sources are initialized up front. Events from different sources will end up getting added to the same list.
                // Pruning is disabled if either source sends in an event which causes pruning to be skipped
                si.values = columnMap.get(columnName).values;
                si.skipPruning = columnMap.get(columnName).skipPruning;
            }
            columnMap.put(columnName, si);
        }
    }
}
Also used : HashMap(java.util.HashMap) MutableInt(org.apache.commons.lang3.mutable.MutableInt) TableDesc(org.apache.hadoop.hive.ql.plan.TableDesc) ExprNodeDesc(org.apache.hadoop.hive.ql.plan.ExprNodeDesc)

Example 2 with MutableInt

use of org.apache.commons.lang3.mutable.MutableInt in project hive by apache.

the class DynamicPartitionPruner method processVertex.

public void processVertex(String name) {
    LOG.info("Vertex succeeded: " + name);
    synchronized (sourcesWaitingForEvents) {
        // Get a deterministic count of number of tasks for the vertex.
        MutableInt prevVal = numExpectedEventsPerSource.get(name);
        int prevValInt = prevVal.intValue();
        Preconditions.checkState(prevValInt < 0, "Invalid value for numExpectedEvents for source: " + name + ", oldVal=" + prevValInt);
        prevVal.setValue((-1) * prevValInt * context.getVertexNumTasks(name));
        checkForSourceCompletion(name);
    }
}
Also used : MutableInt(org.apache.commons.lang3.mutable.MutableInt)

Example 3 with MutableInt

use of org.apache.commons.lang3.mutable.MutableInt in project neo4j by neo4j.

the class LockingStatementOperations method nodeDetachDelete.

@Override
public int nodeDetachDelete(final KernelStatement state, final long nodeId) throws KernelException {
    final MutableInt count = new MutableInt();
    TwoPhaseNodeForRelationshipLocking locking = new TwoPhaseNodeForRelationshipLocking(entityReadDelegate, relId -> {
        state.assertOpen();
        try {
            entityWriteDelegate.relationshipDelete(state, relId);
            count.increment();
        } catch (EntityNotFoundException e) {
        }
    });
    locking.lockAllNodesAndConsumeRelationships(nodeId, state);
    state.assertOpen();
    entityWriteDelegate.nodeDetachDelete(state, nodeId);
    return count.intValue();
}
Also used : MutableInt(org.apache.commons.lang3.mutable.MutableInt) EntityNotFoundException(org.neo4j.kernel.api.exceptions.EntityNotFoundException)

Example 4 with MutableInt

use of org.apache.commons.lang3.mutable.MutableInt in project neo4j by neo4j.

the class FullCheckIntegrationTest method createPropertyKey.

private int createPropertyKey() throws Exception {
    final MutableInt id = new MutableInt(-1);
    fixture.apply(new GraphStoreFixture.Transaction() {

        @Override
        protected void transactionData(GraphStoreFixture.TransactionDataBuilder tx, GraphStoreFixture.IdGenerator next) {
            int propertyKeyId = next.propertyKey();
            tx.propertyKey(propertyKeyId, "property");
            id.setValue(propertyKeyId);
        }
    });
    return id.intValue();
}
Also used : TransactionDataBuilder(org.neo4j.consistency.checking.GraphStoreFixture.TransactionDataBuilder) MutableInt(org.apache.commons.lang3.mutable.MutableInt) IdGenerator(org.neo4j.consistency.checking.GraphStoreFixture.IdGenerator) GraphStoreFixture(org.neo4j.consistency.checking.GraphStoreFixture)

Example 5 with MutableInt

use of org.apache.commons.lang3.mutable.MutableInt in project neo4j by neo4j.

the class FullCheckIntegrationTest method createRelType.

private int createRelType() throws Exception {
    final MutableInt id = new MutableInt(-1);
    fixture.apply(new GraphStoreFixture.Transaction() {

        @Override
        protected void transactionData(GraphStoreFixture.TransactionDataBuilder tx, GraphStoreFixture.IdGenerator next) {
            int relTypeId = next.relationshipType();
            tx.relationshipType(relTypeId, "relType");
            id.setValue(relTypeId);
        }
    });
    return id.intValue();
}
Also used : TransactionDataBuilder(org.neo4j.consistency.checking.GraphStoreFixture.TransactionDataBuilder) MutableInt(org.apache.commons.lang3.mutable.MutableInt) IdGenerator(org.neo4j.consistency.checking.GraphStoreFixture.IdGenerator) GraphStoreFixture(org.neo4j.consistency.checking.GraphStoreFixture)

Aggregations

MutableInt (org.apache.commons.lang3.mutable.MutableInt)28 Test (org.junit.Test)10 HashMap (java.util.HashMap)5 MutableBoolean (org.apache.commons.lang3.mutable.MutableBoolean)5 Map (java.util.Map)4 TimeoutException (java.util.concurrent.TimeoutException)4 File (java.io.File)3 ArrayList (java.util.ArrayList)3 List (java.util.List)2 ILogicalOperator (org.apache.hyracks.algebricks.core.algebra.base.ILogicalOperator)2 LogManager (org.apache.logging.log4j.LogManager)2 Logger (org.apache.logging.log4j.Logger)2 AllelicCount (org.broadinstitute.hellbender.tools.exome.alleliccount.AllelicCount)2 AllelicCountCollection (org.broadinstitute.hellbender.tools.exome.alleliccount.AllelicCountCollection)2 SimpleInterval (org.broadinstitute.hellbender.utils.SimpleInterval)2 Utils (org.broadinstitute.hellbender.utils.Utils)2 IOUtils (org.broadinstitute.hellbender.utils.io.IOUtils)2 ParamUtils (org.broadinstitute.hellbender.utils.param.ParamUtils)2 GraphStoreFixture (org.neo4j.consistency.checking.GraphStoreFixture)2 IdGenerator (org.neo4j.consistency.checking.GraphStoreFixture.IdGenerator)2