Search in sources :

Example 21 with IntOpenHashSet

use of it.unimi.dsi.fastutil.ints.IntOpenHashSet in project angel by Tencent.

the class InitParam method split.

@Override
public List<PartitionUpdateParam> split() {
    List<PartitionKey> pkeys = PSAgentContext.get().getMatrixMetaManager().getPartitions(matrixId);
    IntOpenHashSet serverIds = new IntOpenHashSet();
    List<PartitionUpdateParam> params = new ArrayList<>();
    for (PartitionKey pkey : pkeys) {
        int serverId = PSAgentContext.get().getMatrixMetaManager().getMasterPS(pkey).getIndex();
        if (!serverIds.contains(serverId)) {
            serverIds.add(serverId);
            params.add(new InitPartitionParam(matrixId, pkey, numPartitions, maxIndex, maxLength, negative, order, partDim, window));
        }
    }
    return params;
}
Also used : IntOpenHashSet(it.unimi.dsi.fastutil.ints.IntOpenHashSet) PartitionUpdateParam(com.tencent.angel.ml.matrix.psf.update.base.PartitionUpdateParam) ArrayList(java.util.ArrayList) PartitionKey(com.tencent.angel.PartitionKey)

Example 22 with IntOpenHashSet

use of it.unimi.dsi.fastutil.ints.IntOpenHashSet in project angel by Tencent.

the class LINESecondOrderModel method dot.

@Override
public float[] dot(ByteBuf edges) {
    Random negativeSeed = new Random(seed);
    IntOpenHashSet numInputs = new IntOpenHashSet();
    IntOpenHashSet numOutputs = new IntOpenHashSet();
    int batchSize = edges.readInt();
    float[] partialDots = new float[batchSize * (1 + negative)];
    int dotInc = 0;
    for (int position = 0; position < batchSize; position++) {
        int src = edges.readInt();
        // Skip-Gram model
        float[] inputs = layers[src / numNodeOneRow];
        int l1 = (src % numNodeOneRow) * dim * 2;
        numInputs.add(src);
        // Negative sampling
        int target;
        for (int a = 0; a < negative + 1; a++) {
            if (a == 0)
                target = edges.readInt();
            else
                do {
                    target = negativeSeed.nextInt(maxIndex);
                } while (target == src);
            numOutputs.add(target);
            float[] outputs = layers[target / numNodeOneRow];
            int l2 = (target % numNodeOneRow) * dim * 2 + dim;
            float f = 0.0f;
            for (int b = 0; b < dim; b++) f += inputs[l1 + b] * outputs[l2 + b];
            partialDots[dotInc++] = f;
        }
    }
    this.numInputsToUpdate = numInputs.size();
    this.numOutputsToUpdate = numOutputs.size();
    return partialDots;
}
Also used : IntOpenHashSet(it.unimi.dsi.fastutil.ints.IntOpenHashSet) Random(java.util.Random)

Example 23 with IntOpenHashSet

use of it.unimi.dsi.fastutil.ints.IntOpenHashSet in project pinot by linkedin.

the class DistinctCountMVAggregationFunction method getOrCreateHashSetForGroupKey.

/**
   * Returns the hash-set for the given group-key. Creates and returns one, if it does not exist.
   *
   * @param groupByResultHolder Result Holder
   * @param groupKey Group key for which to get the hash set.
   * @return Hash-set for the group key
   */
private IntOpenHashSet getOrCreateHashSetForGroupKey(@Nonnull GroupByResultHolder groupByResultHolder, int groupKey) {
    IntOpenHashSet valueSet = groupByResultHolder.getResult(groupKey);
    if (valueSet == null) {
        valueSet = new IntOpenHashSet();
        groupByResultHolder.setValueForKey(groupKey, valueSet);
    }
    return valueSet;
}
Also used : IntOpenHashSet(it.unimi.dsi.fastutil.ints.IntOpenHashSet)

Example 24 with IntOpenHashSet

use of it.unimi.dsi.fastutil.ints.IntOpenHashSet in project druid by druid-io.

the class InDimFilter method getFloatPredicateSupplier.

private Supplier<DruidFloatPredicate> getFloatPredicateSupplier() {
    return new Supplier<DruidFloatPredicate>() {

        private final Object initLock = new Object();

        private DruidFloatPredicate predicate;

        private void initFloatValues() {
            if (predicate != null) {
                return;
            }
            synchronized (initLock) {
                if (predicate != null) {
                    return;
                }
                IntArrayList floatBits = new IntArrayList(values.size());
                for (String value : values) {
                    Float floatValue = Floats.tryParse(value);
                    if (floatValue != null) {
                        floatBits.add(Float.floatToIntBits(floatValue));
                    }
                }
                if (floatBits.size() > NUMERIC_HASHING_THRESHOLD) {
                    final IntOpenHashSet floatBitsHashSet = new IntOpenHashSet(floatBits);
                    predicate = new DruidFloatPredicate() {

                        @Override
                        public boolean applyFloat(float input) {
                            return floatBitsHashSet.contains(Float.floatToIntBits(input));
                        }
                    };
                } else {
                    final int[] floatBitsArray = floatBits.toIntArray();
                    Arrays.sort(floatBitsArray);
                    predicate = new DruidFloatPredicate() {

                        @Override
                        public boolean applyFloat(float input) {
                            return Arrays.binarySearch(floatBitsArray, Float.floatToIntBits(input)) >= 0;
                        }
                    };
                }
            }
        }

        @Override
        public DruidFloatPredicate get() {
            initFloatValues();
            return predicate;
        }
    };
}
Also used : IntOpenHashSet(it.unimi.dsi.fastutil.ints.IntOpenHashSet) Supplier(com.google.common.base.Supplier) IntArrayList(it.unimi.dsi.fastutil.ints.IntArrayList)

Example 25 with IntOpenHashSet

use of it.unimi.dsi.fastutil.ints.IntOpenHashSet in project geode by apache.

the class GatewaySenderEventCallbackArgument method fromData.

@Override
public void fromData(DataInput in) throws IOException, ClassNotFoundException {
    super.fromData(in);
    this.originatingDSId = DataSerializer.readInteger(in);
    this.receipientDSIds = new IntOpenHashSet(2);
    int numberOfRecipientGateways = in.readInt();
    for (int i = 0; i < numberOfRecipientGateways; i++) {
        this.receipientDSIds.add(in.readInt());
    }
}
Also used : IntOpenHashSet(it.unimi.dsi.fastutil.ints.IntOpenHashSet)

Aggregations

IntOpenHashSet (it.unimi.dsi.fastutil.ints.IntOpenHashSet)32 IntSet (it.unimi.dsi.fastutil.ints.IntSet)11 IntIterator (it.unimi.dsi.fastutil.ints.IntIterator)4 PartitionKey (com.tencent.angel.PartitionKey)3 ArrayList (java.util.ArrayList)3 Random (java.util.Random)3 Supplier (com.google.common.base.Supplier)2 FieldSpec (com.linkedin.pinot.common.data.FieldSpec)2 IntArrayList (it.unimi.dsi.fastutil.ints.IntArrayList)2 IAE (org.apache.druid.java.util.common.IAE)2 BitmapBackedSelection (tech.tablesaw.selection.BitmapBackedSelection)2 Selection (tech.tablesaw.selection.Selection)2 JsonCreator (com.fasterxml.jackson.annotation.JsonCreator)1 JsonIgnore (com.fasterxml.jackson.annotation.JsonIgnore)1 JsonInclude (com.fasterxml.jackson.annotation.JsonInclude)1 JsonProperty (com.fasterxml.jackson.annotation.JsonProperty)1 VisibleForTesting (com.google.common.annotations.VisibleForTesting)1 Joiner (com.google.common.base.Joiner)1 Preconditions (com.google.common.base.Preconditions)1 Predicate (com.google.common.base.Predicate)1