Search in sources :

Example 1 with BooleanSupplier

use of java.util.function.BooleanSupplier in project crate by crate.

the class RowsBatchIteratorBenchmark method measureConsumeNestedLoopLeftJoin.

@Benchmark
public void measureConsumeNestedLoopLeftJoin(Blackhole blackhole) throws Exception {
    BatchIterator leftJoin = NestedLoopBatchIterator.leftJoin(RowsBatchIterator.newInstance(oneThousandRows, 1), RowsBatchIterator.newInstance(tenThousandRows, 1), columns -> new BooleanSupplier() {

        Input<?> col1 = columns.get(0);

        Input<?> col2 = columns.get(1);

        @Override
        public boolean getAsBoolean() {
            return Objects.equals(col1.value(), col2.value());
        }
    });
    final Input<?> input = leftJoin.rowData().get(0);
    while (leftJoin.moveNext()) {
        blackhole.consume(input.value());
    }
    leftJoin.moveToStart();
}
Also used : NestedLoopBatchIterator(io.crate.data.join.NestedLoopBatchIterator) BooleanSupplier(java.util.function.BooleanSupplier)

Example 2 with BooleanSupplier

use of java.util.function.BooleanSupplier in project neo4j by neo4j.

the class BackupServiceStressTesting method shouldBehaveCorrectlyUnderStress.

@Test
public void shouldBehaveCorrectlyUnderStress() throws Exception {
    long durationInMinutes = parseLong(fromEnv("BACKUP_SERVICE_STRESS_DURATION", DEFAULT_DURATION_IN_MINUTES));
    String directory = fromEnv("BACKUP_SERVICE_STRESS_WORKING_DIRECTORY", DEFAULT_WORKING_DIR);
    String backupHostname = fromEnv("BACKUP_SERVICE_STRESS_BACKUP_HOSTNAME", DEFAULT_HOSTNAME);
    int backupPort = parseInt(fromEnv("BACKUP_SERVICE_STRESS_BACKUP_PORT", DEFAULT_PORT));
    String txPrune = fromEnv("BACKUP_SERVICE_STRESS_TX_PRUNE", DEFAULT_TX_PRUNE);
    boolean enableIndexes = parseBoolean(fromEnv("BACKUP_SERVICE_STRESS_ENABLE_INDEXES", DEFAULT_ENABLE_INDEXES));
    File store = new File(directory, "store");
    File work = new File(directory, "work");
    FileUtils.deleteRecursively(store);
    FileUtils.deleteRecursively(work);
    File storeDirectory = ensureExistsAndEmpty(store);
    File workDirectory = ensureExistsAndEmpty(work);
    final Map<String, String> config = configureBackup(configureTxLogRotationAndPruning(new HashMap<>(), txPrune), backupHostname, backupPort);
    GraphDatabaseBuilder graphDatabaseBuilder = new GraphDatabaseFactory().newEmbeddedDatabaseBuilder(storeDirectory.getAbsoluteFile()).setConfig(config);
    final AtomicBoolean stopTheWorld = new AtomicBoolean();
    BooleanSupplier notExpired = untilTimeExpired(durationInMinutes, MINUTES);
    Runnable onFailure = () -> stopTheWorld.set(true);
    BooleanSupplier keepGoingSupplier = () -> !stopTheWorld.get() && notExpired.getAsBoolean();
    AtomicReference<GraphDatabaseService> dbRef = new AtomicReference<>();
    ExecutorService service = Executors.newFixedThreadPool(3);
    try {
        dbRef.set(graphDatabaseBuilder.newGraphDatabase());
        if (enableIndexes) {
            WorkLoad.setupIndexes(dbRef.get());
        }
        Future<Throwable> workload = service.submit(new WorkLoad(keepGoingSupplier, onFailure, dbRef::get));
        Future<Throwable> backupWorker = service.submit(new BackupLoad(keepGoingSupplier, onFailure, backupHostname, backupPort, workDirectory));
        Future<Throwable> startStopWorker = service.submit(new StartStop(keepGoingSupplier, onFailure, graphDatabaseBuilder::newGraphDatabase, dbRef));
        long expirationTime = currentTimeMillis() + TimeUnit.MINUTES.toMillis(durationInMinutes + 5);
        assertSuccessfulExecution(workload, maxWaitTime(expirationTime), expirationTime);
        assertSuccessfulExecution(backupWorker, maxWaitTime(expirationTime), expirationTime);
        assertSuccessfulExecution(startStopWorker, maxWaitTime(expirationTime), expirationTime);
        service.shutdown();
        if (!service.awaitTermination(30, TimeUnit.SECONDS)) {
            ThreadTestUtils.dumpAllStackTraces();
            fail("Didn't manage to shut down the workers correctly, dumped threads for forensic purposes");
        }
    } finally {
        dbRef.get().shutdown();
        service.shutdown();
    }
    // let's cleanup disk space when everything went well
    FileUtils.deleteRecursively(storeDirectory);
    FileUtils.deleteRecursively(workDirectory);
}
Also used : GraphDatabaseService(org.neo4j.graphdb.GraphDatabaseService) HashMap(java.util.HashMap) AtomicReference(java.util.concurrent.atomic.AtomicReference) AtomicBoolean(java.util.concurrent.atomic.AtomicBoolean) GraphDatabaseFactory(org.neo4j.graphdb.factory.GraphDatabaseFactory) ExecutorService(java.util.concurrent.ExecutorService) File(java.io.File) BooleanSupplier(java.util.function.BooleanSupplier) GraphDatabaseBuilder(org.neo4j.graphdb.factory.GraphDatabaseBuilder) Test(org.junit.Test)

Example 3 with BooleanSupplier

use of java.util.function.BooleanSupplier in project neo4j by neo4j.

the class BackupStoreCopyInteractionStressTesting method shouldBehaveCorrectlyUnderStress.

@Test
public void shouldBehaveCorrectlyUnderStress() throws Exception {
    int numberOfCores = parseInt(fromEnv("BACKUP_STORE_COPY_INTERACTION_STRESS_NUMBER_OF_CORES", DEFAULT_NUMBER_OF_CORES));
    int numberOfEdges = parseInt(fromEnv("BACKUP_STORE_COPY_INTERACTION_STRESS_NUMBER_OF_EDGES", DEFAULT_NUMBER_OF_EDGES));
    long durationInMinutes = parseLong(fromEnv("BACKUP_STORE_COPY_INTERACTION_STRESS_DURATION", DEFAULT_DURATION_IN_MINUTES));
    String workingDirectory = fromEnv("BACKUP_STORE_COPY_INTERACTION_STRESS_WORKING_DIRECTORY", DEFAULT_WORKING_DIR);
    int baseCoreBackupPort = parseInt(fromEnv("BACKUP_STORE_COPY_INTERACTION_STRESS_BASE_CORE_BACKUP_PORT", DEFAULT_BASE_CORE_BACKUP_PORT));
    int baseEdgeBackupPort = parseInt(fromEnv("BACKUP_STORE_COPY_INTERACTION_STRESS_BASE_EDGE_BACKUP_PORT", DEFAULT_BASE_EDGE_BACKUP_PORT));
    boolean enableIndexes = parseBoolean(fromEnv("BACKUP_STORE_COPY_INTERACTION_STRESS_ENABLE_INDEXES", DEFAULT_ENABLE_INDEXES));
    String txPrune = fromEnv("BACKUP_STORE_COPY_INTERACTION_STRESS_TX_PRUNE", DEFAULT_TX_PRUNE);
    File clusterDirectory = ensureExistsAndEmpty(new File(workingDirectory, "cluster"));
    File backupDirectory = ensureExistsAndEmpty(new File(workingDirectory, "backups"));
    BiFunction<Boolean, Integer, SocketAddress> backupAddress = (isCore, id) -> new AdvertisedSocketAddress("localhost", (isCore ? baseCoreBackupPort : baseEdgeBackupPort) + id);
    Map<String, String> coreParams = enableRaftMessageLogging(configureRaftLogRotationAndPruning(configureTxLogRotationAndPruning(new HashMap<>(), txPrune)));
    Map<String, String> readReplicaParams = configureTxLogRotationAndPruning(new HashMap<>(), txPrune);
    Map<String, IntFunction<String>> instanceCoreParams = configureBackup(new HashMap<>(), id -> backupAddress.apply(true, id));
    Map<String, IntFunction<String>> instanceReadReplicaParams = configureBackup(new HashMap<>(), id -> backupAddress.apply(false, id));
    HazelcastDiscoveryServiceFactory discoveryServiceFactory = new HazelcastDiscoveryServiceFactory();
    Cluster cluster = new Cluster(clusterDirectory, numberOfCores, numberOfEdges, discoveryServiceFactory, coreParams, instanceCoreParams, readReplicaParams, instanceReadReplicaParams, Standard.LATEST_NAME);
    AtomicBoolean stopTheWorld = new AtomicBoolean();
    BooleanSupplier notExpired = untilTimeExpired(durationInMinutes, MINUTES);
    BooleanSupplier keepGoing = () -> !stopTheWorld.get() && notExpired.getAsBoolean();
    Runnable onFailure = () -> stopTheWorld.set(true);
    ExecutorService service = Executors.newFixedThreadPool(3);
    try {
        cluster.start();
        if (enableIndexes) {
            Workload.setupIndexes(cluster);
        }
        Future<Throwable> workload = service.submit(new Workload(keepGoing, onFailure, cluster));
        Future<Throwable> startStopWorker = service.submit(new StartStopLoad(fs, pageCache, keepGoing, onFailure, cluster, numberOfCores, numberOfEdges));
        Future<Throwable> backupWorker = service.submit(new BackupLoad(keepGoing, onFailure, cluster, numberOfCores, numberOfEdges, backupDirectory, backupAddress));
        long timeout = durationInMinutes + 5;
        assertNull(Exceptions.stringify(workload.get()), workload.get(timeout, MINUTES));
        assertNull(Exceptions.stringify(startStopWorker.get()), startStopWorker.get(timeout, MINUTES));
        assertNull(Exceptions.stringify(backupWorker.get()), backupWorker.get(timeout, MINUTES));
    } finally {
        cluster.shutdown();
        service.shutdown();
    }
    // let's cleanup disk space when everything went well
    FileUtils.deleteRecursively(clusterDirectory);
    FileUtils.deleteRecursively(backupDirectory);
}
Also used : StressTestingHelper.ensureExistsAndEmpty(org.neo4j.helper.StressTestingHelper.ensureExistsAndEmpty) Suppliers.untilTimeExpired(org.neo4j.function.Suppliers.untilTimeExpired) BiFunction(java.util.function.BiFunction) Exceptions(org.neo4j.helpers.Exceptions) StressTestingHelper.fromEnv(org.neo4j.helper.StressTestingHelper.fromEnv) AtomicBoolean(java.util.concurrent.atomic.AtomicBoolean) HashMap(java.util.HashMap) MINUTES(java.util.concurrent.TimeUnit.MINUTES) BooleanSupplier(java.util.function.BooleanSupplier) SocketAddress(org.neo4j.helpers.SocketAddress) Future(java.util.concurrent.Future) AdvertisedSocketAddress(org.neo4j.helpers.AdvertisedSocketAddress) ClusterConfiguration.configureRaftLogRotationAndPruning(org.neo4j.causalclustering.stresstests.ClusterConfiguration.configureRaftLogRotationAndPruning) HazelcastDiscoveryServiceFactory(org.neo4j.causalclustering.discovery.HazelcastDiscoveryServiceFactory) PageCacheRule(org.neo4j.test.rule.PageCacheRule) Map(java.util.Map) ClusterConfiguration.configureBackup(org.neo4j.causalclustering.stresstests.ClusterConfiguration.configureBackup) System.getProperty(java.lang.System.getProperty) ExecutorService(java.util.concurrent.ExecutorService) Before(org.junit.Before) IntFunction(java.util.function.IntFunction) Standard(org.neo4j.kernel.impl.store.format.standard.Standard) PageCache(org.neo4j.io.pagecache.PageCache) ClusterConfiguration.enableRaftMessageLogging(org.neo4j.causalclustering.stresstests.ClusterConfiguration.enableRaftMessageLogging) FileUtils(org.neo4j.io.fs.FileUtils) Test(org.junit.Test) Integer.parseInt(java.lang.Integer.parseInt) File(java.io.File) Executors(java.util.concurrent.Executors) Cluster(org.neo4j.causalclustering.discovery.Cluster) RuleChain(org.junit.rules.RuleChain) Rule(org.junit.Rule) DefaultFileSystemRule(org.neo4j.test.rule.fs.DefaultFileSystemRule) Boolean.parseBoolean(java.lang.Boolean.parseBoolean) Assert.assertNull(org.junit.Assert.assertNull) DatabaseConfiguration.configureTxLogRotationAndPruning(org.neo4j.helper.DatabaseConfiguration.configureTxLogRotationAndPruning) Long.parseLong(java.lang.Long.parseLong) FileSystemAbstraction(org.neo4j.io.fs.FileSystemAbstraction) AdvertisedSocketAddress(org.neo4j.helpers.AdvertisedSocketAddress) HazelcastDiscoveryServiceFactory(org.neo4j.causalclustering.discovery.HazelcastDiscoveryServiceFactory) AtomicBoolean(java.util.concurrent.atomic.AtomicBoolean) Boolean.parseBoolean(java.lang.Boolean.parseBoolean) SocketAddress(org.neo4j.helpers.SocketAddress) AdvertisedSocketAddress(org.neo4j.helpers.AdvertisedSocketAddress) BooleanSupplier(java.util.function.BooleanSupplier) Cluster(org.neo4j.causalclustering.discovery.Cluster) AtomicBoolean(java.util.concurrent.atomic.AtomicBoolean) IntFunction(java.util.function.IntFunction) ExecutorService(java.util.concurrent.ExecutorService) File(java.io.File) Test(org.junit.Test)

Example 4 with BooleanSupplier

use of java.util.function.BooleanSupplier in project Main by SpartanRefactoring.

the class PropositionTest method hasCycles.

private boolean hasCycles(final BooleanSupplier s) {
    final Stack<BooleanSupplier> path = new Stack<>();
    path.add(s);
    final Queue<BooleanSupplier> todo = new LinkedList<>();
    do {
        final BooleanSupplier current = todo.isEmpty() ? path.pop() : todo.remove();
        if (path.contains(current))
            return true;
        if (current instanceof Proposition.Some) {
            todo.addAll(((Proposition.Some) current).inner);
            continue;
        }
        if (current instanceof Proposition.Singleton)
            path.push(((Proposition.Singleton) current).inner);
    } while (!path.isEmpty());
    return false;
}
Also used : Proposition(il.org.spartan.utils.Proposition) BooleanSupplier(java.util.function.BooleanSupplier) LinkedList(java.util.LinkedList) Stack(java.util.Stack)

Example 5 with BooleanSupplier

use of java.util.function.BooleanSupplier in project elasticsearch by elastic.

the class ESIntegTestCase method waitForDocs.

/**
     * Waits until at least a give number of document is visible for searchers
     *
     * @param numDocs         number of documents to wait for
     * @param maxWaitTime     if not progress have been made during this time, fail the test
     * @param maxWaitTimeUnit the unit in which maxWaitTime is specified
     * @param indexer         a {@link org.elasticsearch.test.BackgroundIndexer}. If supplied it will be first checked for documents indexed.
     *                        This saves on unneeded searches.
     * @return the actual number of docs seen.
     */
public long waitForDocs(final long numDocs, int maxWaitTime, TimeUnit maxWaitTimeUnit, @Nullable final BackgroundIndexer indexer) throws InterruptedException {
    final AtomicLong lastKnownCount = new AtomicLong(-1);
    long lastStartCount = -1;
    BooleanSupplier testDocs = () -> {
        if (indexer != null) {
            lastKnownCount.set(indexer.totalIndexedDocs());
        }
        if (lastKnownCount.get() >= numDocs) {
            try {
                long count = client().prepareSearch().setSize(0).setQuery(matchAllQuery()).execute().actionGet().getHits().getTotalHits();
                if (count == lastKnownCount.get()) {
                    // no progress - try to refresh for the next time
                    client().admin().indices().prepareRefresh().get();
                }
                lastKnownCount.set(count);
            } catch (Exception e) {
                // count now acts like search and barfs if all shards failed...
                logger.debug("failed to executed count", e);
                return false;
            }
            logger.debug("[{}] docs visible for search. waiting for [{}]", lastKnownCount.get(), numDocs);
        } else {
            logger.debug("[{}] docs indexed. waiting for [{}]", lastKnownCount.get(), numDocs);
        }
        return lastKnownCount.get() >= numDocs;
    };
    while (!awaitBusy(testDocs, maxWaitTime, maxWaitTimeUnit)) {
        if (lastStartCount == lastKnownCount.get()) {
            // we didn't make any progress
            fail("failed to reach " + numDocs + "docs");
        }
        lastStartCount = lastKnownCount.get();
    }
    return lastKnownCount.get();
}
Also used : AtomicLong(java.util.concurrent.atomic.AtomicLong) BooleanSupplier(java.util.function.BooleanSupplier) IOException(java.io.IOException) ExecutionException(java.util.concurrent.ExecutionException) EsRejectedExecutionException(org.elasticsearch.common.util.concurrent.EsRejectedExecutionException) ElasticsearchException(org.elasticsearch.ElasticsearchException) ShardOperationFailedException(org.elasticsearch.action.ShardOperationFailedException)

Aggregations

BooleanSupplier (java.util.function.BooleanSupplier)8 File (java.io.File)3 ExecutorService (java.util.concurrent.ExecutorService)3 AtomicBoolean (java.util.concurrent.atomic.AtomicBoolean)3 Test (org.junit.Test)3 HashMap (java.util.HashMap)2 Cluster (org.neo4j.causalclustering.discovery.Cluster)2 HazelcastDiscoveryServiceFactory (org.neo4j.causalclustering.discovery.HazelcastDiscoveryServiceFactory)2 Proposition (il.org.spartan.utils.Proposition)1 NestedLoopBatchIterator (io.crate.data.join.NestedLoopBatchIterator)1 IOException (java.io.IOException)1 Boolean.parseBoolean (java.lang.Boolean.parseBoolean)1 Integer.parseInt (java.lang.Integer.parseInt)1 Long.parseLong (java.lang.Long.parseLong)1 System.getProperty (java.lang.System.getProperty)1 ArrayList (java.util.ArrayList)1 LinkedList (java.util.LinkedList)1 List (java.util.List)1 Map (java.util.Map)1 Stack (java.util.Stack)1