Search in sources :

Example 1 with AbortingVisitor

use of com.palantir.common.base.AbortingVisitor in project atlasdb by palantir.

the class TransactionRangeMigrator method internalCopyRange.

private byte[] internalCopyRange(BatchingVisitable<RowResult<byte[]>> bv, final long maxBytes, final Transaction txn) {
    final Mutable<byte[]> lastRowName = Mutables.newMutable(null);
    final MutableLong bytesPut = new MutableLong(0L);
    bv.batchAccept(readBatchSize, AbortingVisitors.batching(// even though no exception can be thrown :-(
    new AbortingVisitor<RowResult<byte[]>, RuntimeException>() {

        @Override
        public boolean visit(RowResult<byte[]> rr) throws RuntimeException {
            return TransactionRangeMigrator.this.internalCopyRow(rr, maxBytes, txn, bytesPut, lastRowName);
        }
    }));
    return lastRowName.get();
}
Also used : RowResult(com.palantir.atlasdb.keyvalue.api.RowResult) MutableLong(org.apache.commons.lang3.mutable.MutableLong) AbortingVisitor(com.palantir.common.base.AbortingVisitor)

Example 2 with AbortingVisitor

use of com.palantir.common.base.AbortingVisitor in project atlasdb by palantir.

the class SnapshotTransactionTest method testTransactionAtomicity.

@Test
public void testTransactionAtomicity() throws Exception {
    // This test runs multiple transactions in parallel, with KeyValueService.put calls throwing
    // a RuntimeException from time to time and hanging other times. which effectively kills the
    // thread. We ensure that every transaction either adds 5 rows to the table or adds 0 rows
    // by checking at the end that the number of rows is a multiple of 5.
    final TableReference tableRef = TABLE;
    Random random = new Random(1);
    final UnstableKeyValueService unstableKvs = new UnstableKeyValueService(keyValueService, random);
    final TestTransactionManager unstableTransactionManager = new TestTransactionManagerImpl(unstableKvs, timestampService, lockClient, lockService, transactionService, conflictDetectionManager, sweepStrategyManager, sweepQueue);
    ScheduledExecutorService service = PTExecutors.newScheduledThreadPool(20);
    for (int i = 0; i < 30; i++) {
        final int threadNumber = i;
        service.schedule((Callable<Void>) () -> {
            if (threadNumber == 10) {
                unstableKvs.setRandomlyThrow(true);
            }
            if (threadNumber == 20) {
                unstableKvs.setRandomlyHang(true);
            }
            Transaction transaction = unstableTransactionManager.createNewTransaction();
            BatchingVisitable<RowResult<byte[]>> results = transaction.getRange(tableRef, RangeRequest.builder().build());
            final MutableInt nextIndex = new MutableInt(0);
            results.batchAccept(1, AbortingVisitors.batching((AbortingVisitor<RowResult<byte[]>, Exception>) row -> {
                byte[] dataBytes = row.getColumns().get(PtBytes.toBytes("data"));
                BigInteger dataValue = new BigInteger(dataBytes);
                nextIndex.setValue(Math.max(nextIndex.toInteger(), dataValue.intValue() + 1));
                return true;
            }));
            // rows to the table.
            for (int j = 0; j < 5; j++) {
                int rowNumber = nextIndex.toInteger() + j;
                Cell cell = Cell.create(PtBytes.toBytes("row" + rowNumber), PtBytes.toBytes("data"));
                transaction.put(tableRef, ImmutableMap.of(cell, BigInteger.valueOf(rowNumber).toByteArray()));
                Thread.yield();
            }
            transaction.commit();
            return null;
        }, i * 20, TimeUnit.MILLISECONDS);
    }
    service.shutdown();
    service.awaitTermination(1, TimeUnit.SECONDS);
    // Verify each table has a number of rows that's a multiple of 5
    Transaction verifyTransaction = txManager.createNewTransaction();
    BatchingVisitable<RowResult<byte[]>> results = verifyTransaction.getRange(tableRef, RangeRequest.builder().build());
    final MutableInt numRows = new MutableInt(0);
    results.batchAccept(1, AbortingVisitors.batching((AbortingVisitor<RowResult<byte[]>, Exception>) row -> {
        numRows.increment();
        return true;
    }));
    Assert.assertEquals(0, numRows.toInteger() % 5);
}
Also used : ColumnMetadataDescription(com.palantir.atlasdb.table.description.ColumnMetadataDescription) Matchers.not(org.hamcrest.Matchers.not) Future(java.util.concurrent.Future) Pair(org.apache.commons.lang3.tuple.Pair) MutableLong(org.apache.commons.lang3.mutable.MutableLong) Map(java.util.Map) TableReference(com.palantir.atlasdb.keyvalue.api.TableReference) BigInteger(java.math.BigInteger) AtlasDbConstants(com.palantir.atlasdb.AtlasDbConstants) Matchers.notNullValue(org.hamcrest.Matchers.notNullValue) Cell(com.palantir.atlasdb.keyvalue.api.Cell) Set(java.util.Set) LockMode(com.palantir.lock.LockMode) Executors(java.util.concurrent.Executors) Matchers.any(org.mockito.Matchers.any) Transaction(com.palantir.atlasdb.transaction.api.Transaction) Matchers.is(org.hamcrest.Matchers.is) TransactionLockTimeoutNonRetriableException(com.palantir.atlasdb.transaction.api.TransactionLockTimeoutNonRetriableException) Matchers.containsString(org.hamcrest.Matchers.containsString) NoOpCleaner(com.palantir.atlasdb.cleaner.NoOpCleaner) Mockito.mock(org.mockito.Mockito.mock) Joiner(com.google.common.base.Joiner) ColumnRangeSelection(com.palantir.atlasdb.keyvalue.api.ColumnRangeSelection) Iterables(com.google.common.collect.Iterables) ConflictHandler(com.palantir.atlasdb.transaction.api.ConflictHandler) Expectations(org.jmock.Expectations) CachePriority(com.palantir.atlasdb.protos.generated.TableMetadataPersistence.CachePriority) TransactionFailedRetriableException(com.palantir.atlasdb.transaction.api.TransactionFailedRetriableException) Callable(java.util.concurrent.Callable) SimpleTimeDuration(com.palantir.lock.SimpleTimeDuration) AtlasDbTestCase(com.palantir.atlasdb.AtlasDbTestCase) PtBytes(com.palantir.atlasdb.encoding.PtBytes) Mockito.verifyZeroInteractions(org.mockito.Mockito.verifyZeroInteractions) Multimaps(com.google.common.collect.Multimaps) LockAwareTransactionTask(com.palantir.atlasdb.transaction.api.LockAwareTransactionTask) Lists(com.google.common.collect.Lists) LegacyTimelockService(com.palantir.lock.impl.LegacyTimelockService) ScheduledExecutorService(java.util.concurrent.ScheduledExecutorService) AbortingVisitors(com.palantir.common.base.AbortingVisitors) LockRequest(com.palantir.lock.LockRequest) ImmutableSortedMap(com.google.common.collect.ImmutableSortedMap) Before(org.junit.Before) LockCollections(com.palantir.lock.LockCollections) Assert.assertTrue(org.junit.Assert.assertTrue) Throwables(com.google.common.base.Throwables) Test(org.junit.Test) AtlasRowLockDescriptor(com.palantir.lock.AtlasRowLockDescriptor) Mockery(org.jmock.Mockery) ExecutionException(java.util.concurrent.ExecutionException) RowResult(com.palantir.atlasdb.keyvalue.api.RowResult) AtomicLong(java.util.concurrent.atomic.AtomicLong) TimeDuration(com.palantir.lock.TimeDuration) Matchers.hasItem(org.hamcrest.Matchers.hasItem) LockService(com.palantir.lock.LockService) LockClient(com.palantir.lock.LockClient) Assert(org.junit.Assert) Assert.assertEquals(org.junit.Assert.assertEquals) MutableInt(org.apache.commons.lang3.mutable.MutableInt) BatchColumnRangeSelection(com.palantir.atlasdb.keyvalue.api.BatchColumnRangeSelection) Sequence(org.jmock.Sequence) Random(java.util.Random) BatchingVisitable(com.palantir.common.base.BatchingVisitable) CompletionService(java.util.concurrent.CompletionService) Assert.assertThat(org.junit.Assert.assertThat) PTExecutors(com.palantir.common.concurrent.PTExecutors) Mockito.verifyNoMoreInteractions(org.mockito.Mockito.verifyNoMoreInteractions) Matchers.eq(org.mockito.Matchers.eq) Assert.fail(org.junit.Assert.fail) TrackingKeyValueService(com.palantir.atlasdb.keyvalue.impl.TrackingKeyValueService) LockDescriptor(com.palantir.lock.LockDescriptor) TableMetadata(com.palantir.atlasdb.table.description.TableMetadata) ImmutableSet(com.google.common.collect.ImmutableSet) ImmutableMap(com.google.common.collect.ImmutableMap) TransactionReadSentinelBehavior(com.palantir.atlasdb.transaction.api.TransactionReadSentinelBehavior) Collection(java.util.Collection) SweepStrategy(com.palantir.atlasdb.protos.generated.TableMetadataPersistence.SweepStrategy) Collectors(java.util.stream.Collectors) ColumnSelection(com.palantir.atlasdb.keyvalue.api.ColumnSelection) RangeRequest(com.palantir.atlasdb.keyvalue.api.RangeRequest) TransactionConflictException(com.palantir.atlasdb.transaction.api.TransactionConflictException) List(java.util.List) MultiDelegateProxy(com.palantir.common.proxy.MultiDelegateProxy) Optional(java.util.Optional) ForwardingKeyValueService(com.palantir.atlasdb.keyvalue.impl.ForwardingKeyValueService) SortedMap(java.util.SortedMap) ExecutorCompletionService(java.util.concurrent.ExecutorCompletionService) HeldLocksToken(com.palantir.lock.HeldLocksToken) Supplier(com.google.common.base.Supplier) Multimap(com.google.common.collect.Multimap) LockRefreshToken(com.palantir.lock.LockRefreshToken) AtlasDbConstraintCheckingMode(com.palantir.atlasdb.transaction.api.AtlasDbConstraintCheckingMode) ImmutableList(com.google.common.collect.ImmutableList) Suppliers(com.google.common.base.Suppliers) WriteInfo(com.palantir.atlasdb.sweep.queue.WriteInfo) ExecutorService(java.util.concurrent.ExecutorService) Matchers.hasEntry(org.hamcrest.Matchers.hasEntry) EncodingUtils(com.palantir.atlasdb.ptobject.EncodingUtils) PreCommitCondition(com.palantir.atlasdb.transaction.api.PreCommitCondition) Matchers(org.hamcrest.Matchers) TimestampCache(com.palantir.atlasdb.cache.TimestampCache) Mockito.when(org.mockito.Mockito.when) Mockito.verify(org.mockito.Mockito.verify) TimeUnit(java.util.concurrent.TimeUnit) Ignore(org.junit.Ignore) KeyValueService(com.palantir.atlasdb.keyvalue.api.KeyValueService) TransactionFailedNonRetriableException(com.palantir.atlasdb.transaction.api.TransactionFailedNonRetriableException) NameMetadataDescription(com.palantir.atlasdb.table.description.NameMetadataDescription) Collections(java.util.Collections) AbortingVisitor(com.palantir.common.base.AbortingVisitor) ScheduledExecutorService(java.util.concurrent.ScheduledExecutorService) AbortingVisitor(com.palantir.common.base.AbortingVisitor) TransactionLockTimeoutNonRetriableException(com.palantir.atlasdb.transaction.api.TransactionLockTimeoutNonRetriableException) TransactionFailedRetriableException(com.palantir.atlasdb.transaction.api.TransactionFailedRetriableException) ExecutionException(java.util.concurrent.ExecutionException) TransactionConflictException(com.palantir.atlasdb.transaction.api.TransactionConflictException) TransactionFailedNonRetriableException(com.palantir.atlasdb.transaction.api.TransactionFailedNonRetriableException) BatchingVisitable(com.palantir.common.base.BatchingVisitable) RowResult(com.palantir.atlasdb.keyvalue.api.RowResult) TableReference(com.palantir.atlasdb.keyvalue.api.TableReference) Random(java.util.Random) Transaction(com.palantir.atlasdb.transaction.api.Transaction) MutableInt(org.apache.commons.lang3.mutable.MutableInt) BigInteger(java.math.BigInteger) Cell(com.palantir.atlasdb.keyvalue.api.Cell) Test(org.junit.Test)

Example 3 with AbortingVisitor

use of com.palantir.common.base.AbortingVisitor in project atlasdb by palantir.

the class TableMigratorTest method testMigrationToDifferentKvs.

// Table/IndexDefinition syntax
@SuppressWarnings({ "checkstyle:Indentation", "checkstyle:RightCurly" })
@Test
public void testMigrationToDifferentKvs() throws TableMappingNotFoundException {
    final TableReference tableRef = TableReference.create(Namespace.DEFAULT_NAMESPACE, "table");
    final TableReference namespacedTableRef = TableReference.createFromFullyQualifiedName("namespace." + tableRef.getTablename());
    TableDefinition definition = new TableDefinition() {

        {
            rowName();
            rowComponent("r", ValueType.BLOB);
            columns();
            column("c", "c", ValueType.BLOB);
        }
    };
    keyValueService.createTable(tableRef, definition.toTableMetadata().persistToBytes());
    keyValueService.createTable(namespacedTableRef, definition.toTableMetadata().persistToBytes());
    keyValueService.putMetadataForTable(namespacedTableRef, definition.toTableMetadata().persistToBytes());
    final Cell theCell = Cell.create(PtBytes.toBytes("r1"), PtBytes.toBytes("c"));
    final byte[] theValue = PtBytes.toBytes("v1");
    txManager.runTaskWithRetry((TransactionTask<Void, RuntimeException>) txn -> {
        Map<Cell, byte[]> values = ImmutableMap.of(theCell, theValue);
        txn.put(tableRef, values);
        txn.put(namespacedTableRef, values);
        return null;
    });
    final InMemoryKeyValueService kvs2 = new InMemoryKeyValueService(false);
    final ConflictDetectionManager cdm2 = ConflictDetectionManagers.createWithNoConflictDetection();
    final SweepStrategyManager ssm2 = SweepStrategyManagers.completelyConservative(kvs2);
    final TestTransactionManagerImpl txManager2 = new TestTransactionManagerImpl(kvs2, timestampService, lockClient, lockService, transactionService, cdm2, ssm2, MultiTableSweepQueueWriter.NO_OP);
    kvs2.createTable(tableRef, definition.toTableMetadata().persistToBytes());
    kvs2.createTable(namespacedTableRef, definition.toTableMetadata().persistToBytes());
    TableReference checkpointTable = TableReference.create(Namespace.DEFAULT_NAMESPACE, "checkpoint");
    GeneralTaskCheckpointer checkpointer = new GeneralTaskCheckpointer(checkpointTable, kvs2, txManager2);
    for (final TableReference name : Lists.newArrayList(tableRef, namespacedTableRef)) {
        TransactionRangeMigrator rangeMigrator = new TransactionRangeMigratorBuilder().srcTable(name).readTxManager(txManager).txManager(txManager2).checkpointer(checkpointer).build();
        TableMigratorBuilder builder = new TableMigratorBuilder().srcTable(name).partitions(1).executor(PTExecutors.newSingleThreadExecutor()).checkpointer(checkpointer).rangeMigrator(rangeMigrator);
        TableMigrator migrator = builder.build();
        migrator.migrate();
    }
    checkpointer.deleteCheckpoints();
    final ConflictDetectionManager verifyCdm = ConflictDetectionManagers.createWithNoConflictDetection();
    final SweepStrategyManager verifySsm = SweepStrategyManagers.completelyConservative(kvs2);
    final TestTransactionManagerImpl verifyTxManager = new TestTransactionManagerImpl(kvs2, timestampService, lockClient, lockService, transactionService, verifyCdm, verifySsm, MultiTableSweepQueueWriter.NO_OP);
    final MutableLong count = new MutableLong();
    for (final TableReference name : Lists.newArrayList(tableRef, namespacedTableRef)) {
        verifyTxManager.runTaskReadOnly((TransactionTask<Void, RuntimeException>) txn -> {
            BatchingVisitable<RowResult<byte[]>> bv = txn.getRange(name, RangeRequest.all());
            bv.batchAccept(1000, AbortingVisitors.batching(new AbortingVisitor<RowResult<byte[]>, RuntimeException>() {

                @Override
                public boolean visit(RowResult<byte[]> item) throws RuntimeException {
                    Iterable<Entry<Cell, byte[]>> cells = item.getCells();
                    Entry<Cell, byte[]> entry = Iterables.getOnlyElement(cells);
                    Assert.assertEquals(theCell, entry.getKey());
                    Assert.assertArrayEquals(theValue, entry.getValue());
                    count.increment();
                    return true;
                }
            }));
            return null;
        });
    }
    Assert.assertEquals(2L, count.longValue());
}
Also used : ConflictDetectionManager(com.palantir.atlasdb.transaction.impl.ConflictDetectionManager) Iterables(com.google.common.collect.Iterables) SweepStrategyManager(com.palantir.atlasdb.transaction.impl.SweepStrategyManager) BatchingVisitable(com.palantir.common.base.BatchingVisitable) InMemoryKeyValueService(com.palantir.atlasdb.keyvalue.impl.InMemoryKeyValueService) AtlasDbTestCase(com.palantir.atlasdb.AtlasDbTestCase) PtBytes(com.palantir.atlasdb.encoding.PtBytes) Lists(com.google.common.collect.Lists) PTExecutors(com.palantir.common.concurrent.PTExecutors) MutableLong(org.apache.commons.lang3.mutable.MutableLong) Map(java.util.Map) TableReference(com.palantir.atlasdb.keyvalue.api.TableReference) AbortingVisitors(com.palantir.common.base.AbortingVisitors) ConflictDetectionManagers(com.palantir.atlasdb.transaction.impl.ConflictDetectionManagers) SweepStrategyManagers(com.palantir.atlasdb.transaction.impl.SweepStrategyManagers) Namespace(com.palantir.atlasdb.keyvalue.api.Namespace) ImmutableMap(com.google.common.collect.ImmutableMap) Cell(com.palantir.atlasdb.keyvalue.api.Cell) Test(org.junit.Test) TableDefinition(com.palantir.atlasdb.table.description.TableDefinition) RangeRequest(com.palantir.atlasdb.keyvalue.api.RangeRequest) RowResult(com.palantir.atlasdb.keyvalue.api.RowResult) ValueType(com.palantir.atlasdb.table.description.ValueType) TransactionTask(com.palantir.atlasdb.transaction.api.TransactionTask) TableMappingNotFoundException(com.palantir.atlasdb.keyvalue.impl.TableMappingNotFoundException) Entry(java.util.Map.Entry) TestTransactionManagerImpl(com.palantir.atlasdb.transaction.impl.TestTransactionManagerImpl) MultiTableSweepQueueWriter(com.palantir.atlasdb.sweep.queue.MultiTableSweepQueueWriter) Assert(org.junit.Assert) AbortingVisitor(com.palantir.common.base.AbortingVisitor) InMemoryKeyValueService(com.palantir.atlasdb.keyvalue.impl.InMemoryKeyValueService) BatchingVisitable(com.palantir.common.base.BatchingVisitable) TableReference(com.palantir.atlasdb.keyvalue.api.TableReference) Entry(java.util.Map.Entry) TableDefinition(com.palantir.atlasdb.table.description.TableDefinition) Cell(com.palantir.atlasdb.keyvalue.api.Cell) SweepStrategyManager(com.palantir.atlasdb.transaction.impl.SweepStrategyManager) TestTransactionManagerImpl(com.palantir.atlasdb.transaction.impl.TestTransactionManagerImpl) RowResult(com.palantir.atlasdb.keyvalue.api.RowResult) MutableLong(org.apache.commons.lang3.mutable.MutableLong) ConflictDetectionManager(com.palantir.atlasdb.transaction.impl.ConflictDetectionManager) Map(java.util.Map) ImmutableMap(com.google.common.collect.ImmutableMap) Test(org.junit.Test)

Example 4 with AbortingVisitor

use of com.palantir.common.base.AbortingVisitor in project atlasdb by palantir.

the class KvsRangeMigrator method internalCopyRange.

private byte[] internalCopyRange(BatchingVisitable<RowResult<byte[]>> bv, final long maxBytes, @Output final Map<Cell, byte[]> writeMap) {
    final Mutable<byte[]> lastRowName = Mutables.newMutable(null);
    final MutableLong bytesPut = new MutableLong(0L);
    bv.batchAccept(readBatchSize, AbortingVisitors.batching(// even though no exception can be thrown :-(
    new AbortingVisitor<RowResult<byte[]>, RuntimeException>() {

        @Override
        public boolean visit(RowResult<byte[]> rr) throws RuntimeException {
            return KvsRangeMigrator.this.internalCopyRow(rr, maxBytes, writeMap, bytesPut, lastRowName);
        }
    }));
    return lastRowName.get();
}
Also used : RowResult(com.palantir.atlasdb.keyvalue.api.RowResult) MutableLong(org.apache.commons.lang3.mutable.MutableLong) AbortingVisitor(com.palantir.common.base.AbortingVisitor)

Aggregations

RowResult (com.palantir.atlasdb.keyvalue.api.RowResult)4 AbortingVisitor (com.palantir.common.base.AbortingVisitor)4 MutableLong (org.apache.commons.lang3.mutable.MutableLong)4 ImmutableMap (com.google.common.collect.ImmutableMap)2 Iterables (com.google.common.collect.Iterables)2 Lists (com.google.common.collect.Lists)2 AtlasDbTestCase (com.palantir.atlasdb.AtlasDbTestCase)2 PtBytes (com.palantir.atlasdb.encoding.PtBytes)2 Cell (com.palantir.atlasdb.keyvalue.api.Cell)2 RangeRequest (com.palantir.atlasdb.keyvalue.api.RangeRequest)2 TableReference (com.palantir.atlasdb.keyvalue.api.TableReference)2 AbortingVisitors (com.palantir.common.base.AbortingVisitors)2 BatchingVisitable (com.palantir.common.base.BatchingVisitable)2 PTExecutors (com.palantir.common.concurrent.PTExecutors)2 Map (java.util.Map)2 Assert (org.junit.Assert)2 Test (org.junit.Test)2 Joiner (com.google.common.base.Joiner)1 Supplier (com.google.common.base.Supplier)1 Suppliers (com.google.common.base.Suppliers)1