Examples with VoidAggregation org.nd4j.parameterserver.distributed.messages.VoidAggregation used on opensource projects

Example 1 with VoidAggregation

use of org.nd4j.parameterserver.distributed.messages.VoidAggregation in project nd4j by deeplearning4j.

the class ClipboardTest method testPin2.

@Test
public void testPin2() throws Exception {
    Clipboard clipboard = new Clipboard();
    Random rng = new Random(12345L);
    Long validId = 123L;
    short shardIdx = 0;
    for (int i = 0; i < 300; i++) {
        VectorAggregation aggregation = new VectorAggregation(rng.nextLong(), (short) 100, (short) 1, Nd4j.create(5));
        // imitating valid
        if (i % 2 == 0 && shardIdx < 100) {
            aggregation.setTaskId(validId);
            aggregation.setShardIndex(shardIdx++);
        }
        clipboard.pin(aggregation);
    }
    VoidAggregation aggregation = clipboard.getStackFromClipboard(0L, validId);
    assertNotEquals(null, aggregation);
    assertEquals(0, aggregation.getMissingChunks());
    assertEquals(true, clipboard.hasCandidates());
    assertEquals(1, clipboard.getNumberOfCompleteStacks());
}

Example 2 with VoidAggregation

use of org.nd4j.parameterserver.distributed.messages.VoidAggregation in project nd4j by deeplearning4j.

the class Clipboard method pin.

/**
 * This method places incoming VoidAggregation into clipboard, for further tracking
 *
 * @param aggregation
 * @return TRUE, if given VoidAggregation was the last chunk, FALSE otherwise
 */
public boolean pin(@NonNull VoidAggregation aggregation) {
    RequestDescriptor descriptor = RequestDescriptor.createDescriptor(aggregation.getOriginatorId(), aggregation.getTaskId());
    VoidAggregation existing = clipboard.get(descriptor);
    if (existing == null) {
        existing = aggregation;
        trackingCounter.incrementAndGet();
        clipboard.put(descriptor, aggregation);
    }
    existing.accumulateAggregation(aggregation);
    // if (counter.incrementAndGet() % 10000 == 0)
    // log.info("Clipboard stats: Totals: {}; Completed: {};", clipboard.size(), completedQueue.size());
    int missing = existing.getMissingChunks();
    if (missing == 0) {
        // completedQueue.add(existing);
        completedCounter.incrementAndGet();
        return true;
    } else
        return false;
}

Example 3 with VoidAggregation

use of org.nd4j.parameterserver.distributed.messages.VoidAggregation in project nd4j by deeplearning4j.

the class Clipboard method nextCandidate.

/**
 * This method returns one of available aggregations, if there's at least 1 ready.
 *
 * @return
 */
public VoidAggregation nextCandidate() {
    VoidAggregation result = completedQueue.poll();
    // removing aggregation from tracking table
    if (result != null) {
        completedCounter.decrementAndGet();
        unpin(result.getOriginatorId(), result.getTaskId());
    }
    return result;
}

Example 4 with VoidAggregation

use of org.nd4j.parameterserver.distributed.messages.VoidAggregation in project nd4j by deeplearning4j.

the class Clipboard method isReady.

public boolean isReady(long originatorId, long taskId) {
    RequestDescriptor descriptor = RequestDescriptor.createDescriptor(originatorId, taskId);
    VoidAggregation aggregation = clipboard.get(descriptor);
    if (aggregation == null)
        return false;
    return aggregation.getMissingChunks() == 0;
}

Example 5 with VoidAggregation

use of org.nd4j.parameterserver.distributed.messages.VoidAggregation in project nd4j by deeplearning4j.

the class VectorAggregation method processMessage.

/**
 * Vector aggregations are saved only by Shards started aggregation process. All other Shards are ignoring this meesage
 */
@Override
public void processMessage() {
    if (clipboard.isTracking(this.originatorId, this.getTaskId())) {
        clipboard.pin(this);
        if (clipboard.isReady(this.originatorId, taskId)) {
            VoidAggregation aggregation = clipboard.unpin(this.originatorId, taskId);
            // FIXME: probably there's better solution, then "screw-and-forget" one
            if (aggregation == null)
                return;
            VectorCompleteMessage msg = new VectorCompleteMessage(taskId, aggregation.getAccumulatedResult());
            msg.setOriginatorId(aggregation.getOriginatorId());
            transport.sendMessage(msg);
        }
    }
}