Search in sources :

Example 51 with LongSupplier

use of java.util.function.LongSupplier in project neo4j by neo4j.

the class NodeStoreTest method newNodeStore.

private NodeStore newNodeStore(FileSystemAbstraction fs, PageCache pageCache) {
    IdGeneratorFactory idGeneratorFactory = spy(new DefaultIdGeneratorFactory(fs, immediate(), databaseLayout.getDatabaseName()) {

        @Override
        protected IndexedIdGenerator instantiate(FileSystemAbstraction fs, PageCache pageCache, RecoveryCleanupWorkCollector recoveryCleanupWorkCollector, Path fileName, LongSupplier highIdSupplier, long maxValue, IdType idType, DatabaseReadOnlyChecker readOnlyChecker, Config config, CursorContext cursorContext, String databaseName, ImmutableSet<OpenOption> openOptions) {
            return spy(super.instantiate(fs, pageCache, recoveryCleanupWorkCollector, fileName, highIdSupplier, maxValue, idType, readOnlyChecker, config, cursorContext, databaseName, openOptions));
        }
    });
    StoreFactory factory = new StoreFactory(databaseLayout, Config.defaults(), idGeneratorFactory, pageCache, fs, NullLogProvider.getInstance(), PageCacheTracer.NULL, writable());
    neoStores = factory.openAllNeoStores(true);
    nodeStore = neoStores.getNodeStore();
    return nodeStore;
}
Also used : Path(java.nio.file.Path) EphemeralFileSystemAbstraction(org.neo4j.io.fs.EphemeralFileSystemAbstraction) FileSystemAbstraction(org.neo4j.io.fs.FileSystemAbstraction) IndexedIdGenerator(org.neo4j.internal.id.indexed.IndexedIdGenerator) Config(org.neo4j.configuration.Config) DefaultIdGeneratorFactory(org.neo4j.internal.id.DefaultIdGeneratorFactory) DefaultIdGeneratorFactory(org.neo4j.internal.id.DefaultIdGeneratorFactory) IdGeneratorFactory(org.neo4j.internal.id.IdGeneratorFactory) CursorContext(org.neo4j.io.pagecache.context.CursorContext) StringContains.containsString(org.hamcrest.core.StringContains.containsString) RecoveryCleanupWorkCollector(org.neo4j.index.internal.gbptree.RecoveryCleanupWorkCollector) IdType(org.neo4j.internal.id.IdType) OpenOption(java.nio.file.OpenOption) DatabaseReadOnlyChecker(org.neo4j.configuration.helpers.DatabaseReadOnlyChecker) LongSupplier(java.util.function.LongSupplier) PageCache(org.neo4j.io.pagecache.PageCache) DelegatingPageCache(org.neo4j.io.pagecache.DelegatingPageCache)

Example 52 with LongSupplier

use of java.util.function.LongSupplier in project neo4j by neo4j.

the class RelationshipModifierTest method shouldCreateAndDelete.

// ... other known cases ...
@RepeatedTest(20)
void shouldCreateAndDelete() {
    // given and initial state
    long node = createEmptyNode();
    IntSupplier typeStrategy = randomTypes(10);
    Supplier<RelationshipDirection> directionStrategy = RANDOM_DIRECTION;
    LongSupplier otherNodeStrategy = this::createEmptyNode;
    int maxRelationships = 100;
    List<RelationshipData> expectedRelationships = generateRelationshipData(random.nextInt(0, maxRelationships), node, typeStrategy, otherNodeStrategy, directionStrategy);
    createRelationships(expectedRelationships);
    // ... a set of relationships to create and delete
    List<RelationshipData> relationshipsToCreate = generateRelationshipData((int) random.among(new long[] { 0, 1, 10, 100 }), node, typeStrategy, otherNodeStrategy, directionStrategy);
    int numRelationshipsToDelete = min((int) random.among(new long[] { 0, 1, 10, maxRelationships }), expectedRelationships.size());
    RelationshipData[] relationshipsToDelete = random.selection(expectedRelationships.toArray(RelationshipData[]::new), numRelationshipsToDelete, numRelationshipsToDelete, false);
    // ... and rules for how the world changes "concurrently" while we perform these modifications
    // on locked
    monitors.addMonitorListener(new ResourceTypeLockOrderVerifier());
    // on read
    monitors.addMonitorListener(new ChangeWorldOnReadMonitor(node, typeStrategy, otherNodeStrategy, directionStrategy, expectedRelationships));
    // when
    RelationshipModifications modifications = modifications(relationshipsToCreate.toArray(RelationshipData[]::new), relationshipsToDelete);
    modify(modifications);
    applyModificationsToExpectedRelationships(modifications, expectedRelationships);
    // then
    assertThat(readRelationshipsFromStore(node, store)).isEqualTo(asSet(expectedRelationships));
}
Also used : RelationshipData(org.neo4j.internal.recordstorage.FlatRelationshipModifications.RelationshipData) RelationshipModifications(org.neo4j.storageengine.api.txstate.RelationshipModifications) RelationshipDirection(org.neo4j.storageengine.api.RelationshipDirection) IntSupplier(java.util.function.IntSupplier) LongSupplier(java.util.function.LongSupplier) RepeatedTest(org.junit.jupiter.api.RepeatedTest)

Example 53 with LongSupplier

use of java.util.function.LongSupplier in project crate by crate.

the class TranslogWriter method create.

public static TranslogWriter create(ShardId shardId, String translogUUID, long fileGeneration, Path file, ChannelFactory channelFactory, ByteSizeValue bufferSize, final long initialMinTranslogGen, long initialGlobalCheckpoint, final LongSupplier globalCheckpointSupplier, final LongSupplier minTranslogGenerationSupplier, final long primaryTerm, TragicExceptionHolder tragedy, LongConsumer persistedSequenceNumberConsumer) throws IOException {
    final Path checkpointFile = file.getParent().resolve(Translog.CHECKPOINT_FILE_NAME);
    final FileChannel channel = channelFactory.open(file);
    FileChannel checkpointChannel = null;
    try {
        checkpointChannel = channelFactory.open(checkpointFile, StandardOpenOption.WRITE);
        final TranslogHeader header = new TranslogHeader(translogUUID, primaryTerm);
        header.write(channel);
        final Checkpoint checkpoint = Checkpoint.emptyTranslogCheckpoint(header.sizeInBytes(), fileGeneration, initialGlobalCheckpoint, initialMinTranslogGen);
        writeCheckpoint(checkpointChannel, checkpointFile, checkpoint);
        final LongSupplier writerGlobalCheckpointSupplier;
        if (Assertions.ENABLED) {
            writerGlobalCheckpointSupplier = () -> {
                long gcp = globalCheckpointSupplier.getAsLong();
                assert gcp >= initialGlobalCheckpoint : "global checkpoint [" + gcp + "] lower than initial gcp [" + initialGlobalCheckpoint + "]";
                return gcp;
            };
        } else {
            writerGlobalCheckpointSupplier = globalCheckpointSupplier;
        }
        return new TranslogWriter(shardId, checkpoint, channel, checkpointChannel, file, checkpointFile, bufferSize, writerGlobalCheckpointSupplier, minTranslogGenerationSupplier, header, tragedy, persistedSequenceNumberConsumer);
    } catch (Exception exception) {
        // if we fail to bake the file-generation into the checkpoint we stick with the file and once we recover and that
        // file exists we remove it. We only apply this logic to the checkpoint.generation+1 any other file with a higher generation
        // is an error condition
        IOUtils.closeWhileHandlingException(channel, checkpointChannel);
        throw exception;
    }
}
Also used : Path(java.nio.file.Path) FileChannel(java.nio.channels.FileChannel) LongSupplier(java.util.function.LongSupplier) AlreadyClosedException(org.apache.lucene.store.AlreadyClosedException) IOException(java.io.IOException)

Example 54 with LongSupplier

use of java.util.function.LongSupplier in project crate by crate.

the class SysOperationsLogTableInfoTest method test_job_id_returns_job_id_of_operation_context_log.

@Test
public void test_job_id_returns_job_id_of_operation_context_log() {
    var table = SysOperationsLogTableInfo.create();
    var expressionFactory = table.expressions().get(new ColumnIdent("job_id"));
    var expression = expressionFactory.create();
    int id = 1;
    UUID jobId = UUID.randomUUID();
    String name = "Dummy";
    long started = 1;
    LongSupplier bytesUsed = () -> 10;
    String errorMessage = null;
    expression.setNextRow(new OperationContextLog(new OperationContext(id, jobId, name, started, bytesUsed), errorMessage));
    Object value = (String) expression.value();
    assertThat(value, Matchers.is(jobId.toString()));
}
Also used : OperationContext(io.crate.expression.reference.sys.operation.OperationContext) ColumnIdent(io.crate.metadata.ColumnIdent) OperationContextLog(io.crate.expression.reference.sys.operation.OperationContextLog) UUID(java.util.UUID) LongSupplier(java.util.function.LongSupplier) Test(org.junit.jupiter.api.Test)

Example 55 with LongSupplier

use of java.util.function.LongSupplier in project crate by crate.

the class InternalEngineTests method testOutOfOrderSequenceNumbersWithVersionConflict.

@Test
public void testOutOfOrderSequenceNumbersWithVersionConflict() throws IOException {
    final List<Engine.Operation> operations = new ArrayList<>();
    final int numberOfOperations = randomIntBetween(16, 32);
    final AtomicLong sequenceNumber = new AtomicLong();
    final Engine.Operation.Origin origin = randomFrom(LOCAL_TRANSLOG_RECOVERY, PEER_RECOVERY, PRIMARY, REPLICA);
    final LongSupplier sequenceNumberSupplier = origin == PRIMARY ? () -> UNASSIGNED_SEQ_NO : sequenceNumber::getAndIncrement;
    final Supplier<ParsedDocument> doc = () -> {
        final Document document = testDocumentWithTextField();
        document.add(new Field(SourceFieldMapper.NAME, BytesReference.toBytes(B_1), SourceFieldMapper.Defaults.FIELD_TYPE));
        return testParsedDocument("1", null, document, B_1, null);
    };
    final Term uid = newUid("1");
    final BiFunction<String, Engine.SearcherScope, Searcher> searcherFactory = engine::acquireSearcher;
    for (int i = 0; i < numberOfOperations; i++) {
        if (randomBoolean()) {
            final Engine.Index index = new Engine.Index(uid, doc.get(), sequenceNumberSupplier.getAsLong(), 1, i, origin == PRIMARY ? VersionType.EXTERNAL : null, origin, System.nanoTime(), Translog.UNSET_AUTO_GENERATED_TIMESTAMP, false, UNASSIGNED_SEQ_NO, 0);
            operations.add(index);
        } else {
            final Engine.Delete delete = new Engine.Delete("1", uid, sequenceNumberSupplier.getAsLong(), 1, i, origin == PRIMARY ? VersionType.EXTERNAL : null, origin, System.nanoTime(), UNASSIGNED_SEQ_NO, 0);
            operations.add(delete);
        }
    }
    final boolean exists = operations.get(operations.size() - 1) instanceof Engine.Index;
    Randomness.shuffle(operations);
    for (final Engine.Operation operation : operations) {
        if (operation instanceof Engine.Index) {
            engine.index((Engine.Index) operation);
        } else {
            engine.delete((Engine.Delete) operation);
        }
    }
    final long expectedLocalCheckpoint;
    if (origin == PRIMARY) {
        // we can only advance as far as the number of operations that did not conflict
        int count = 0;
        // each time the version increments as we walk the list, that counts as a successful operation
        long version = -1;
        for (int i = 0; i < numberOfOperations; i++) {
            if (operations.get(i).version() >= version) {
                count++;
                version = operations.get(i).version();
            }
        }
        // sequence numbers start at zero, so the expected local checkpoint is the number of successful operations minus one
        expectedLocalCheckpoint = count - 1;
    } else {
        expectedLocalCheckpoint = numberOfOperations - 1;
    }
    assertThat(engine.getProcessedLocalCheckpoint(), equalTo(expectedLocalCheckpoint));
    try (Engine.GetResult result = engine.get(new Engine.Get("2", uid), searcherFactory)) {
        assertThat(result.docIdAndVersion() != null, equalTo(exists));
    }
}
Also used : ArrayList(java.util.ArrayList) Matchers.containsString(org.hamcrest.Matchers.containsString) ParsedDocument(org.elasticsearch.index.mapper.ParsedDocument) Document(org.elasticsearch.index.mapper.ParseContext.Document) NumericDocValuesField(org.apache.lucene.document.NumericDocValuesField) TextField(org.apache.lucene.document.TextField) IndexableField(org.apache.lucene.index.IndexableField) StoredField(org.apache.lucene.document.StoredField) Field(org.apache.lucene.document.Field) ParsedDocument(org.elasticsearch.index.mapper.ParsedDocument) Searcher(org.elasticsearch.index.engine.Engine.Searcher) IndexSearcher(org.apache.lucene.search.IndexSearcher) Term(org.apache.lucene.index.Term) LongPoint(org.apache.lucene.document.LongPoint) AtomicLong(java.util.concurrent.atomic.AtomicLong) LongSupplier(java.util.function.LongSupplier) Test(org.junit.Test)

Aggregations

LongSupplier (java.util.function.LongSupplier)59 Random (java.util.Random)21 Test (org.junit.Test)9 IOException (java.io.IOException)7 Path (java.nio.file.Path)7 ArrayList (java.util.ArrayList)7 AtomicBoolean (java.util.concurrent.atomic.AtomicBoolean)7 AtomicLong (java.util.concurrent.atomic.AtomicLong)5 CircuitBreaker (org.elasticsearch.common.breaker.CircuitBreaker)5 BytesArray (org.elasticsearch.common.bytes.BytesArray)5 Tuple (io.crate.common.collections.Tuple)4 Streams (io.crate.common.io.Streams)4 TimeValue (io.crate.common.unit.TimeValue)4 List (java.util.List)4 Predicate (java.util.function.Predicate)4 Supplier (java.util.function.Supplier)4 StoredField (org.apache.lucene.document.StoredField)4 Version (org.elasticsearch.Version)4 NoopCircuitBreaker (org.elasticsearch.common.breaker.NoopCircuitBreaker)4 BytesReference (org.elasticsearch.common.bytes.BytesReference)4