Search in sources :

Example 6 with LongConsumer

use of java.util.function.LongConsumer in project flink by apache.

the class MemoryManager method getSharedMemoryResourceForManagedMemory.

// ------------------------------------------------------------------------
// Shared opaque memory resources
// ------------------------------------------------------------------------
/**
 * Acquires a shared memory resource, identified by a type string. If the resource already
 * exists, this returns a descriptor to the resource. If the resource does not yet exist, the
 * given memory fraction is reserved and the resource is initialized with that size.
 *
 * <p>The memory for the resource is reserved from the memory budget of this memory manager
 * (thus determining the size of the resource), but resource itself is opaque, meaning the
 * memory manager does not understand its structure.
 *
 * <p>The OpaqueMemoryResource object returned from this method must be closed once not used any
 * further. Once all acquisitions have closed the object, the resource itself is closed.
 *
 * <p><b>Important:</b> The failure semantics are as follows: If the memory manager fails to
 * reserve the memory, the external resource initializer will not be called. If an exception is
 * thrown when the opaque resource is closed (last lease is released), the memory manager will
 * still un-reserve the memory to make sure its own accounting is clean. The exception will need
 * to be handled by the caller of {@link OpaqueMemoryResource#close()}. For example, if this
 * indicates that native memory was not released and the process might thus have a memory leak,
 * the caller can decide to kill the process as a result.
 */
public <T extends AutoCloseable> OpaqueMemoryResource<T> getSharedMemoryResourceForManagedMemory(String type, LongFunctionWithException<T, Exception> initializer, double fractionToInitializeWith) throws Exception {
    // if we need to allocate the resource (no shared resource allocated, yet), this would be
    // the size to use
    final long numBytes = computeMemorySize(fractionToInitializeWith);
    // initializer and releaser as functions that are pushed into the SharedResources,
    // so that the SharedResources can decide in (thread-safely execute) when initialization
    // and release should happen
    final LongFunctionWithException<T, Exception> reserveAndInitialize = (size) -> {
        try {
            reserveMemory(type, size);
        } catch (MemoryReservationException e) {
            throw new MemoryAllocationException("Could not created the shared memory resource of size " + size + ". Not enough memory left to reserve from the slot's managed memory.", e);
        }
        try {
            return initializer.apply(size);
        } catch (Throwable t) {
            releaseMemory(type, size);
            throw t;
        }
    };
    final LongConsumer releaser = (size) -> releaseMemory(type, size);
    // This object identifies the lease in this request. It is used only to identify the release
    // operation.
    // Using the object to represent the lease is a bit nicer safer than just using a reference
    // counter.
    final Object leaseHolder = new Object();
    final SharedResources.ResourceAndSize<T> resource = sharedResources.getOrAllocateSharedResource(type, leaseHolder, reserveAndInitialize, numBytes);
    // the actual size may theoretically be different from what we requested, if allocated it
    // was by
    // someone else before with a different value for fraction (should not happen in practice,
    // though).
    final long size = resource.size();
    final ThrowingRunnable<Exception> disposer = () -> sharedResources.release(type, leaseHolder, releaser);
    return new OpaqueMemoryResource<>(resource.resourceHandle(), size, disposer);
}
Also used : MemorySegment(org.apache.flink.core.memory.MemorySegment) Logger(org.slf4j.Logger) ThrowingRunnable(org.apache.flink.util.function.ThrowingRunnable) Iterator(java.util.Iterator) MemorySegmentFactory.allocateOffHeapUnsafeMemory(org.apache.flink.core.memory.MemorySegmentFactory.allocateOffHeapUnsafeMemory) Collection(java.util.Collection) LoggerFactory(org.slf4j.LoggerFactory) ConcurrentHashMap(java.util.concurrent.ConcurrentHashMap) Set(java.util.Set) Preconditions(org.apache.flink.util.Preconditions) AtomicReference(java.util.concurrent.atomic.AtomicReference) ArrayList(java.util.ArrayList) LongConsumer(java.util.function.LongConsumer) VisibleForTesting(org.apache.flink.annotation.VisibleForTesting) MathUtils(org.apache.flink.util.MathUtils) HashSet(java.util.HashSet) LongFunctionWithException(org.apache.flink.util.function.LongFunctionWithException) TaskManagerOptions(org.apache.flink.configuration.TaskManagerOptions) List(java.util.List) Map(java.util.Map) NoSuchElementException(java.util.NoSuchElementException) Nonnull(javax.annotation.Nonnull) ConcurrentModificationException(java.util.ConcurrentModificationException) LongFunctionWithException(org.apache.flink.util.function.LongFunctionWithException) NoSuchElementException(java.util.NoSuchElementException) ConcurrentModificationException(java.util.ConcurrentModificationException) LongConsumer(java.util.function.LongConsumer)

Example 7 with LongConsumer

use of java.util.function.LongConsumer in project j2objc by google.

the class OptionalLongTest method testIfPresent.

public void testIfPresent() {
    LongConsumer alwaysFails = value -> fail();
    OptionalLong.empty().ifPresent(alwaysFails);
    final AtomicLong reference = new AtomicLong();
    LongConsumer recorder = (long value) -> reference.set(value);
    OptionalLong.of(56).ifPresent(recorder);
    assertEquals(56, reference.get());
}
Also used : OptionalLong(java.util.OptionalLong) AtomicLong(java.util.concurrent.atomic.AtomicLong) LongSupplier(java.util.function.LongSupplier) IOException(java.io.IOException) TestCase(junit.framework.TestCase) NoSuchElementException(java.util.NoSuchElementException) Supplier(java.util.function.Supplier) LongConsumer(java.util.function.LongConsumer) LongConsumer(java.util.function.LongConsumer) AtomicLong(java.util.concurrent.atomic.AtomicLong)

Example 8 with LongConsumer

use of java.util.function.LongConsumer in project j2objc by google.

the class LongConsumerTest method testAndThen.

public void testAndThen() throws Exception {
    StringBuilder sb = new StringBuilder();
    LongConsumer one = l -> sb.append("one:" + l + ",");
    LongConsumer two = l -> sb.append("two:" + l);
    one.andThen(two).accept(1L);
    assertEquals("one:1,two:1", sb.toString());
}
Also used : TestCase(junit.framework.TestCase) LongConsumer(java.util.function.LongConsumer) LongConsumer(java.util.function.LongConsumer)

Example 9 with LongConsumer

use of java.util.function.LongConsumer in project j2objc by google.

the class LongConsumerTest method testAndThen_null.

public void testAndThen_null() throws Exception {
    LongConsumer one = s -> {
    };
    try {
        one.andThen(null);
        fail();
    } catch (NullPointerException expected) {
    }
}
Also used : TestCase(junit.framework.TestCase) LongConsumer(java.util.function.LongConsumer) LongConsumer(java.util.function.LongConsumer)

Example 10 with LongConsumer

use of java.util.function.LongConsumer in project elasticsearch by elastic.

the class ShardStateActionTests method testMasterChannelException.

public void testMasterChannelException() throws InterruptedException {
    final String index = "test";
    setState(clusterService, ClusterStateCreationUtils.stateWithActivePrimary(index, true, randomInt(5)));
    CountDownLatch latch = new CountDownLatch(1);
    AtomicInteger retries = new AtomicInteger();
    AtomicBoolean success = new AtomicBoolean();
    AtomicReference<Throwable> throwable = new AtomicReference<>();
    LongConsumer retryLoop = requestId -> {
        if (randomBoolean()) {
            transport.handleRemoteError(requestId, randomFrom(new NotMasterException("simulated"), new Discovery.FailedToCommitClusterStateException("simulated")));
        } else {
            if (randomBoolean()) {
                transport.handleLocalError(requestId, new NodeNotConnectedException(null, "simulated"));
            } else {
                transport.handleError(requestId, new NodeDisconnectedException(null, ShardStateAction.SHARD_FAILED_ACTION_NAME));
            }
        }
    };
    final int numberOfRetries = randomIntBetween(1, 256);
    setUpMasterRetryVerification(numberOfRetries, retries, latch, retryLoop);
    ShardRouting failedShard = getRandomShardRouting(index);
    shardStateAction.localShardFailed(failedShard, "test", getSimulatedFailure(), new ShardStateAction.Listener() {

        @Override
        public void onSuccess() {
            success.set(true);
            latch.countDown();
        }

        @Override
        public void onFailure(Exception e) {
            success.set(false);
            throwable.set(e);
            latch.countDown();
            assert false;
        }
    });
    final CapturingTransport.CapturedRequest[] capturedRequests = transport.getCapturedRequestsAndClear();
    assertThat(capturedRequests.length, equalTo(1));
    assertFalse(success.get());
    assertThat(retries.get(), equalTo(0));
    retryLoop.accept(capturedRequests[0].requestId);
    latch.await();
    assertNull(throwable.get());
    assertThat(retries.get(), equalTo(numberOfRetries));
    assertTrue(success.get());
}
Also used : ShardRouting(org.elasticsearch.cluster.routing.ShardRouting) BeforeClass(org.junit.BeforeClass) ClusterServiceUtils.createClusterService(org.elasticsearch.test.ClusterServiceUtils.createClusterService) CoreMatchers.equalTo(org.hamcrest.CoreMatchers.equalTo) NotMasterException(org.elasticsearch.cluster.NotMasterException) ClusterService(org.elasticsearch.cluster.service.ClusterService) AllocationService(org.elasticsearch.cluster.routing.allocation.AllocationService) AtomicBoolean(java.util.concurrent.atomic.AtomicBoolean) AtomicReference(java.util.concurrent.atomic.AtomicReference) CorruptIndexException(org.apache.lucene.index.CorruptIndexException) CapturingTransport(org.elasticsearch.test.transport.CapturingTransport) CoreMatchers.instanceOf(org.hamcrest.CoreMatchers.instanceOf) ClusterState(org.elasticsearch.cluster.ClusterState) Settings(org.elasticsearch.common.settings.Settings) AtomicInteger(java.util.concurrent.atomic.AtomicInteger) After(org.junit.After) ThreadPool(org.elasticsearch.threadpool.ThreadPool) TransportResponse(org.elasticsearch.transport.TransportResponse) ESTestCase(org.elasticsearch.test.ESTestCase) TransportService(org.elasticsearch.transport.TransportService) ClusterStateObserver(org.elasticsearch.cluster.ClusterStateObserver) ClusterStateCreationUtils(org.elasticsearch.action.support.replication.ClusterStateCreationUtils) ShardsIterator(org.elasticsearch.cluster.routing.ShardsIterator) NodeDisconnectedException(org.elasticsearch.transport.NodeDisconnectedException) Before(org.junit.Before) DiscoveryNodes(org.elasticsearch.cluster.node.DiscoveryNodes) TestThreadPool(org.elasticsearch.threadpool.TestThreadPool) AfterClass(org.junit.AfterClass) Matchers.greaterThanOrEqualTo(org.hamcrest.Matchers.greaterThanOrEqualTo) Discovery(org.elasticsearch.discovery.Discovery) Predicate(java.util.function.Predicate) LongConsumer(java.util.function.LongConsumer) TimeUnit(java.util.concurrent.TimeUnit) CountDownLatch(java.util.concurrent.CountDownLatch) IndexRoutingTable(org.elasticsearch.cluster.routing.IndexRoutingTable) RoutingTable(org.elasticsearch.cluster.routing.RoutingTable) NodeNotConnectedException(org.elasticsearch.transport.NodeNotConnectedException) RoutingService(org.elasticsearch.cluster.routing.RoutingService) Matchers.is(org.hamcrest.Matchers.is) TransportException(org.elasticsearch.transport.TransportException) ClusterServiceUtils.setState(org.elasticsearch.test.ClusterServiceUtils.setState) NodeDisconnectedException(org.elasticsearch.transport.NodeDisconnectedException) Discovery(org.elasticsearch.discovery.Discovery) AtomicReference(java.util.concurrent.atomic.AtomicReference) CountDownLatch(java.util.concurrent.CountDownLatch) NotMasterException(org.elasticsearch.cluster.NotMasterException) CorruptIndexException(org.apache.lucene.index.CorruptIndexException) NodeDisconnectedException(org.elasticsearch.transport.NodeDisconnectedException) NodeNotConnectedException(org.elasticsearch.transport.NodeNotConnectedException) TransportException(org.elasticsearch.transport.TransportException) AtomicBoolean(java.util.concurrent.atomic.AtomicBoolean) LongConsumer(java.util.function.LongConsumer) NodeNotConnectedException(org.elasticsearch.transport.NodeNotConnectedException) AtomicInteger(java.util.concurrent.atomic.AtomicInteger) NotMasterException(org.elasticsearch.cluster.NotMasterException) ShardRouting(org.elasticsearch.cluster.routing.ShardRouting)

Aggregations

LongConsumer (java.util.function.LongConsumer)14 Spliterator (java.util.Spliterator)4 ObjLongConsumer (java.util.function.ObjLongConsumer)3 TestCase (junit.framework.TestCase)3 IOException (java.io.IOException)2 ArrayList (java.util.ArrayList)2 NoSuchElementException (java.util.NoSuchElementException)2 AtomicLong (java.util.concurrent.atomic.AtomicLong)2 AtomicReference (java.util.concurrent.atomic.AtomicReference)2 Field (java.lang.reflect.Field)1 URL (java.net.URL)1 Path (java.nio.file.Path)1 Collection (java.util.Collection)1 ConcurrentModificationException (java.util.ConcurrentModificationException)1 HashSet (java.util.HashSet)1 Iterator (java.util.Iterator)1 List (java.util.List)1 Map (java.util.Map)1 OptionalLong (java.util.OptionalLong)1 Set (java.util.Set)1