Search in sources :

Example 71 with Index

use of herddb.model.Index in project herddb by diennea.

the class TableManager method applyUpdate.

private void applyUpdate(Bytes key, Bytes value) throws DataStorageManagerException {
    // do not want to retain shared buffers as keys
    key = key.nonShared();
    /*
         * New record to be updated, it will always updated if there aren't errors thus is simpler to create
         * the record now
         */
    final Record record = new Record(key, value);
    /* This could be a normal or a temporary modifiable page */
    final Long prevPageId = keyToPage.get(key);
    if (prevPageId == null) {
        throw new IllegalStateException("corrupted transaction log: key " + key + " is not present in table " + table.tablespace + "." + table.name);
    }
    /*
         * We'll try to replace the record if in a writable page, otherwise we'll simply set the old page
         * as dirty and continue like a normal insertion
         */
    final Map<String, AbstractIndexManager> indexes = tableSpaceManager.getIndexesOnTable(table.name);
    /*
         * When index is enabled we need the old value to update them, we'll force the page load only if that
         * record is really needed.
         */
    final DataPage prevPage;
    final Record previous;
    boolean insertedInSamePage = false;
    if (indexes == null) {
        /* We don't need the page if isn't loaded or isn't a mutable new page*/
        prevPage = newPages.get(prevPageId);
        if (prevPage != null) {
            pageReplacementPolicy.pageHit(prevPage);
            previous = prevPage.get(key);
            if (previous == null) {
                throw new IllegalStateException("corrupted PK: old page " + prevPageId + " for updated record at " + key + " was not found in table " + table.tablespace + "." + table.name);
            }
        } else {
            previous = null;
        }
    } else {
        /* We really need the page for update index old values */
        prevPage = loadPageToMemory(prevPageId, false);
        previous = prevPage.get(key);
        if (previous == null) {
            throw new IllegalStateException("corrupted PK: old page " + prevPageId + " for updated record at " + key + " was not found in table" + table.tablespace + "." + table.name);
        }
    }
    if (prevPage == null || prevPage.immutable) {
        /* Unloaded or immutable, set it as dirty */
        pageSet.setPageDirty(prevPageId, previous);
    } else {
        /* Mutable page, need to check if still modifiable or already unloaded */
        final Lock lock = prevPage.pageLock.readLock();
        lock.lock();
        try {
            if (prevPage.writable) {
                /* We can try to modify the page directly */
                insertedInSamePage = prevPage.put(record);
            } else {
                /* Unfortunately is not writable (anymore), set it as dirty */
                pageSet.setPageDirty(prevPageId, previous);
            }
        } finally {
            lock.unlock();
        }
    }
    /* Insertion page */
    Long insertionPageId;
    if (insertedInSamePage) {
        /* Inserted in temporary mutable previous page, no need to alter keyToPage too: no record page change */
        insertionPageId = prevPageId;
    } else {
        /* Do real insertion */
        insertionPageId = currentDirtyRecordsPage.get();
        while (true) {
            final DataPage newPage = newPages.get(insertionPageId);
            if (newPage != null) {
                pageReplacementPolicy.pageHit(newPage);
                /* The temporary memory page could have been unloaded and loaded again in meantime */
                if (!newPage.immutable) {
                    /* Mutable page, need to check if still modifiable or already unloaded */
                    final Lock lock = newPage.pageLock.readLock();
                    lock.lock();
                    try {
                        if (newPage.writable) {
                            /* We can try to modify the page directly */
                            if (newPage.put(record)) {
                                break;
                            }
                        }
                    } finally {
                        lock.unlock();
                    }
                }
            }
            /* Try allocate a new page if no already done */
            insertionPageId = allocateLivePage(insertionPageId);
        }
        /* Update the value on keyToPage */
        keyToPage.put(key, insertionPageId);
    }
    if (LOGGER.isLoggable(Level.FINEST)) {
        LOGGER.log(Level.FINEST, "Updated key " + key + " from page " + prevPageId + " to page " + insertionPageId + " on table " + table.tablespace + "." + table.name);
    }
    if (indexes != null) {
        /* If there are indexes e have already forced a page load and previous record has been loaded */
        DataAccessor prevValues = previous.getDataAccessor(table);
        DataAccessor newValues = record.getDataAccessor(table);
        for (AbstractIndexManager index : indexes.values()) {
            Index indexDef = index.getIndex();
            String[] indexColumnNames = index.getColumnNames();
            Bytes indexKeyRemoved = RecordSerializer.serializeIndexKey(prevValues, indexDef, indexColumnNames);
            Bytes indexKeyAdded = RecordSerializer.serializeIndexKey(newValues, indexDef, indexColumnNames);
            index.recordUpdated(key, indexKeyRemoved, indexKeyAdded);
        }
    }
}
Also used : DataAccessor(herddb.utils.DataAccessor) KeyToPageIndex(herddb.index.KeyToPageIndex) Index(herddb.model.Index) Lock(java.util.concurrent.locks.Lock) StampedLock(java.util.concurrent.locks.StampedLock) ReentrantLock(java.util.concurrent.locks.ReentrantLock) Bytes(herddb.utils.Bytes) AtomicLong(java.util.concurrent.atomic.AtomicLong) Record(herddb.model.Record)

Example 72 with Index

use of herddb.model.Index in project herddb by diennea.

the class TableManager method executeDeleteAsync.

private CompletableFuture<StatementExecutionResult> executeDeleteAsync(DeleteStatement delete, Transaction transaction, StatementEvaluationContext context) {
    AtomicInteger updateCount = new AtomicInteger();
    Holder<Bytes> lastKey = new Holder<>();
    Holder<Bytes> lastValue = new Holder<>();
    long transactionId = transaction != null ? transaction.transactionId : 0;
    Predicate predicate = delete.getPredicate();
    List<CompletableFuture<PendingLogEntryWork>> writes = new ArrayList<>();
    Map<String, AbstractIndexManager> indexes = tableSpaceManager.getIndexesOnTable(table.name);
    ScanStatement scan = new ScanStatement(table.tablespace, table, predicate);
    try {
        accessTableData(scan, context, new ScanResultOperation() {

            @Override
            public void accept(Record current, LockHandle lockHandle) throws StatementExecutionException, LogNotAvailableException, DataStorageManagerException {
                // ensure we are holding the write locks on every unique index
                List<UniqueIndexLockReference> uniqueIndexes = null;
                try {
                    if (indexes != null || childrenTables != null) {
                        DataAccessor dataAccessor = current.getDataAccessor(table);
                        if (childrenTables != null) {
                            for (Table childTable : childrenTables) {
                                executeForeignKeyConstraintsAsParentTable(childTable, dataAccessor, context, transaction, true);
                            }
                        }
                        if (indexes != null) {
                            for (AbstractIndexManager index : indexes.values()) {
                                if (index.isUnique()) {
                                    Bytes indexKey = RecordSerializer.serializeIndexKey(dataAccessor, index.getIndex(), index.getColumnNames());
                                    if (uniqueIndexes == null) {
                                        uniqueIndexes = new ArrayList<>(1);
                                    }
                                    UniqueIndexLockReference uniqueIndexLock = new UniqueIndexLockReference(index, indexKey);
                                    uniqueIndexes.add(uniqueIndexLock);
                                    LockHandle lockForIndex = lockForWrite(uniqueIndexLock.key, transaction, index.getIndexName(), index.getLockManager());
                                    if (transaction == null) {
                                        uniqueIndexLock.lockHandle = lockForIndex;
                                    }
                                }
                            }
                        }
                    }
                } catch (IllegalArgumentException | herddb.utils.IllegalDataAccessException | StatementExecutionException err) {
                    locksManager.releaseLock(lockHandle);
                    StatementExecutionException finalError;
                    if (!(err instanceof StatementExecutionException)) {
                        finalError = new StatementExecutionException(err.getMessage(), err);
                    } else {
                        finalError = (StatementExecutionException) err;
                    }
                    CompletableFuture<PendingLogEntryWork> res = Futures.exception(finalError);
                    if (uniqueIndexes != null) {
                        for (UniqueIndexLockReference lock : uniqueIndexes) {
                            res = releaseWriteLock(res, lockHandle, lock.indexManager.getLockManager());
                        }
                    }
                    writes.add(res);
                    return;
                }
                LogEntry entry = LogEntryFactory.delete(table, current.key, transaction);
                CommitLogResult pos = log.log(entry, entry.transactionId <= 0);
                final List<UniqueIndexLockReference> _uniqueIndexes = uniqueIndexes;
                writes.add(pos.logSequenceNumber.thenApply(lsn -> new PendingLogEntryWork(entry, pos, lockHandle, _uniqueIndexes)));
                lastKey.value = current.key;
                lastValue.value = current.value;
                updateCount.incrementAndGet();
            }
        }, transaction, true, true);
    } catch (HerdDBInternalException err) {
        LOGGER.log(Level.SEVERE, "bad error during a delete", err);
        return Futures.exception(err);
    }
    if (writes.isEmpty()) {
        return CompletableFuture.completedFuture(new DMLStatementExecutionResult(transactionId, 0, null, null));
    }
    if (writes.size() == 1) {
        return writes.get(0).whenCompleteAsync((pending, error) -> {
            try {
                // apply any of the DML operations
                if (error == null) {
                    apply(pending.pos, pending.entry, false);
                }
            } finally {
                releaseMultiplePendingLogEntryWorks(writes);
            }
        }, tableSpaceManager.getCallbacksExecutor()).thenApply((pending) -> {
            return new DMLStatementExecutionResult(transactionId, updateCount.get(), lastKey.value, delete.isReturnValues() ? lastValue.value : null);
        });
    } else {
        return Futures.collect(writes).whenCompleteAsync((pendings, error) -> {
            try {
                // apply any of the DML operations
                if (error == null) {
                    for (PendingLogEntryWork pending : pendings) {
                        apply(pending.pos, pending.entry, false);
                    }
                }
            } finally {
                releaseMultiplePendingLogEntryWorks(writes);
            }
        }, tableSpaceManager.getCallbacksExecutor()).thenApply((pendings) -> {
            return new DMLStatementExecutionResult(transactionId, updateCount.get(), lastKey.value, delete.isReturnValues() ? lastValue.value : null);
        });
    }
}
Also used : Arrays(java.util.Arrays) NullLockManager(herddb.utils.NullLockManager) Table(herddb.model.Table) TruncateTableStatement(herddb.model.commands.TruncateTableStatement) DuplicatePrimaryKeyException(herddb.model.DuplicatePrimaryKeyException) TableStatus(herddb.storage.TableStatus) Map(java.util.Map) DataAccessor(herddb.utils.DataAccessor) LogNotAvailableException(herddb.log.LogNotAvailableException) CommitLogResult(herddb.log.CommitLogResult) LogSequenceNumber(herddb.log.LogSequenceNumber) UniqueIndexContraintViolationException(herddb.model.UniqueIndexContraintViolationException) Set(java.util.Set) RecordSerializer(herddb.codec.RecordSerializer) JSQLParserPlanner.delimit(herddb.sql.JSQLParserPlanner.delimit) DataPageMetaData(herddb.core.PageSet.DataPageMetaData) ScanStatement(herddb.model.commands.ScanStatement) Stream(java.util.stream.Stream) StatsLogger(org.apache.bookkeeper.stats.StatsLogger) Bytes(herddb.utils.Bytes) Holder(herddb.utils.Holder) LockHandle(herddb.utils.LockHandle) ForeignKeyViolationException(herddb.model.ForeignKeyViolationException) LogEntry(herddb.log.LogEntry) ArrayList(java.util.ArrayList) TransactionContext(herddb.model.TransactionContext) Transaction(herddb.model.Transaction) ThreadLocalRandom(java.util.concurrent.ThreadLocalRandom) Projection(herddb.model.Projection) ForeignKeyDef(herddb.model.ForeignKeyDef) EnsureLongIncrementAccumulator(herddb.utils.EnsureLongIncrementAccumulator) LogEntryType(herddb.log.LogEntryType) Record(herddb.model.Record) LogEntryFactory(herddb.log.LogEntryFactory) KeyToPageIndex(herddb.index.KeyToPageIndex) DataStorageManager(herddb.storage.DataStorageManager) ColumnTypes(herddb.model.ColumnTypes) ILocalLockManager(herddb.utils.ILocalLockManager) AtomicLong(java.util.concurrent.atomic.AtomicLong) Lock(java.util.concurrent.locks.Lock) IndexOperation(herddb.index.IndexOperation) Column(herddb.model.Column) StampedLock(java.util.concurrent.locks.StampedLock) UpdateStatement(herddb.model.commands.UpdateStatement) ScanLimitsImpl(herddb.model.ScanLimitsImpl) TupleComparator(herddb.model.TupleComparator) ServerConfiguration(herddb.server.ServerConfiguration) AtomicInteger(java.util.concurrent.atomic.AtomicInteger) RecordTooBigException(herddb.model.RecordTooBigException) DMLStatementExecutionResult(herddb.model.DMLStatementExecutionResult) LocalLockManager(herddb.utils.LocalLockManager) Futures(herddb.utils.Futures) DataStorageManagerException(herddb.storage.DataStorageManagerException) Index(herddb.model.Index) InsertStatement(herddb.model.commands.InsertStatement) DataScanner(herddb.model.DataScanner) DDLException(herddb.model.DDLException) RecordFunction(herddb.model.RecordFunction) StatementExecutionException(herddb.model.StatementExecutionException) TableContext(herddb.model.TableContext) Collection(java.util.Collection) ConcurrentHashMap(java.util.concurrent.ConcurrentHashMap) Logger(java.util.logging.Logger) Collectors(java.util.stream.Collectors) List(java.util.List) FullTableScanConsumer(herddb.storage.FullTableScanConsumer) GetStatement(herddb.model.commands.GetStatement) Entry(java.util.Map.Entry) Statement(herddb.model.Statement) LongAdder(java.util.concurrent.atomic.LongAdder) DataScannerException(herddb.model.DataScannerException) GetResult(herddb.model.GetResult) PrimaryIndexSeek(herddb.index.PrimaryIndexSeek) HashMap(java.util.HashMap) CompletableFuture(java.util.concurrent.CompletableFuture) Function(java.util.function.Function) BatchOrderedExecutor(herddb.utils.BatchOrderedExecutor) ConcurrentMap(java.util.concurrent.ConcurrentMap) Level(java.util.logging.Level) HashSet(java.util.HashSet) BooleanHolder(herddb.utils.BooleanHolder) ScanLimits(herddb.model.ScanLimits) DeleteStatement(herddb.model.commands.DeleteStatement) Iterator(java.util.Iterator) ReentrantLock(java.util.concurrent.locks.ReentrantLock) Semaphore(java.util.concurrent.Semaphore) DataPageDoesNotExistException(herddb.storage.DataPageDoesNotExistException) Counter(org.apache.bookkeeper.stats.Counter) StatementExecutionResult(herddb.model.StatementExecutionResult) TimeUnit(java.util.concurrent.TimeUnit) Consumer(java.util.function.Consumer) CommitLog(herddb.log.CommitLog) AbstractMap(java.util.AbstractMap) TableConsistencyCheckStatement(herddb.model.commands.TableConsistencyCheckStatement) Predicate(herddb.model.Predicate) StatementEvaluationContext(herddb.model.StatementEvaluationContext) Comparator(java.util.Comparator) Collections(java.util.Collections) SECONDS(java.util.concurrent.TimeUnit.SECONDS) TableManagerStats(herddb.core.stats.TableManagerStats) SystemProperties(herddb.utils.SystemProperties) DataStorageManagerException(herddb.storage.DataStorageManagerException) DataAccessor(herddb.utils.DataAccessor) ArrayList(java.util.ArrayList) CommitLogResult(herddb.log.CommitLogResult) StatementExecutionException(herddb.model.StatementExecutionException) Predicate(herddb.model.Predicate) Bytes(herddb.utils.Bytes) CompletableFuture(java.util.concurrent.CompletableFuture) Record(herddb.model.Record) ArrayList(java.util.ArrayList) List(java.util.List) LogEntry(herddb.log.LogEntry) ScanStatement(herddb.model.commands.ScanStatement) LockHandle(herddb.utils.LockHandle) Table(herddb.model.Table) Holder(herddb.utils.Holder) BooleanHolder(herddb.utils.BooleanHolder) AtomicInteger(java.util.concurrent.atomic.AtomicInteger) DMLStatementExecutionResult(herddb.model.DMLStatementExecutionResult) LogNotAvailableException(herddb.log.LogNotAvailableException)

Example 73 with Index

use of herddb.model.Index in project herddb by diennea.

the class TableSpaceManager method apply.

void apply(CommitLogResult position, LogEntry entry, boolean recovery) throws DataStorageManagerException, DDLException {
    if (!position.deferred || position.sync) {
        // this will wait for the write to be acknowledged by the log
        // it can throw LogNotAvailableException
        this.actualLogSequenceNumber = position.getLogSequenceNumber();
        if (LOGGER.isLoggable(Level.FINEST)) {
            LOGGER.log(Level.FINEST, "apply {0} {1}", new Object[] { position.getLogSequenceNumber(), entry });
        }
    } else {
        if (LOGGER.isLoggable(Level.FINEST)) {
            LOGGER.log(Level.FINEST, "apply {0} {1}", new Object[] { position, entry });
        }
    }
    switch(entry.type) {
        case LogEntryType.NOOP:
            {
            // NOOP
            }
            break;
        case LogEntryType.BEGINTRANSACTION:
            {
                long id = entry.transactionId;
                Transaction transaction = new Transaction(id, tableSpaceName, position);
                transactions.put(id, transaction);
            }
            break;
        case LogEntryType.ROLLBACKTRANSACTION:
            {
                long id = entry.transactionId;
                Transaction transaction = transactions.get(id);
                if (transaction == null) {
                    throw new DataStorageManagerException("invalid transaction id " + id + ", only " + transactions.keySet());
                }
                List<AbstractIndexManager> indexManagers = new ArrayList<>(indexes.values());
                for (AbstractIndexManager indexManager : indexManagers) {
                    if (indexManager.getCreatedInTransaction() == 0 || indexManager.getCreatedInTransaction() == id) {
                        indexManager.onTransactionRollback(transaction);
                    }
                }
                List<AbstractTableManager> managers = new ArrayList<>(tables.values());
                for (AbstractTableManager manager : managers) {
                    if (manager.getCreatedInTransaction() == 0 || manager.getCreatedInTransaction() == id) {
                        Table table = manager.getTable();
                        if (transaction.isNewTable(table.name)) {
                            LOGGER.log(Level.INFO, "rollback CREATE TABLE " + table.tablespace + "." + table.name);
                            disposeTable(manager);
                            Map<String, AbstractIndexManager> indexes = indexesByTable.remove(manager.getTable().name);
                            if (indexes != null) {
                                for (AbstractIndexManager indexManager : indexes.values()) {
                                    disposeIndexManager(indexManager);
                                }
                            }
                        } else {
                            manager.onTransactionRollback(transaction);
                        }
                    }
                }
                transactions.remove(transaction.transactionId);
            }
            break;
        case LogEntryType.COMMITTRANSACTION:
            {
                long id = entry.transactionId;
                Transaction transaction = transactions.get(id);
                if (transaction == null) {
                    throw new DataStorageManagerException("invalid transaction id " + id);
                }
                LogSequenceNumber commit = position.getLogSequenceNumber();
                transaction.sync(commit);
                List<AbstractTableManager> managers = new ArrayList<>(tables.values());
                for (AbstractTableManager manager : managers) {
                    if (manager.getCreatedInTransaction() == 0 || manager.getCreatedInTransaction() == id) {
                        manager.onTransactionCommit(transaction, recovery);
                    }
                }
                List<AbstractIndexManager> indexManagers = new ArrayList<>(indexes.values());
                for (AbstractIndexManager indexManager : indexManagers) {
                    if (indexManager.getCreatedInTransaction() == 0 || indexManager.getCreatedInTransaction() == id) {
                        indexManager.onTransactionCommit(transaction, recovery);
                    }
                }
                if ((transaction.droppedTables != null && !transaction.droppedTables.isEmpty()) || (transaction.droppedIndexes != null && !transaction.droppedIndexes.isEmpty())) {
                    if (transaction.droppedTables != null) {
                        for (String dropped : transaction.droppedTables) {
                            for (AbstractTableManager manager : managers) {
                                if (manager.getTable().name.equals(dropped)) {
                                    disposeTable(manager);
                                }
                            }
                        }
                    }
                    if (transaction.droppedIndexes != null) {
                        for (String dropped : transaction.droppedIndexes) {
                            for (AbstractIndexManager manager : indexManagers) {
                                if (manager.getIndex().name.equals(dropped)) {
                                    disposeIndexManager(manager);
                                }
                            }
                        }
                    }
                }
                if ((transaction.newTables != null && !transaction.newTables.isEmpty()) || (transaction.droppedTables != null && !transaction.droppedTables.isEmpty()) || (transaction.newIndexes != null && !transaction.newIndexes.isEmpty()) || (transaction.droppedIndexes != null && !transaction.droppedIndexes.isEmpty())) {
                    writeTablesOnDataStorageManager(position, false);
                    dbmanager.getPlanner().clearCache();
                }
                transactions.remove(transaction.transactionId);
            }
            break;
        case LogEntryType.CREATE_TABLE:
            {
                Table table = Table.deserialize(entry.value.to_array());
                if (entry.transactionId > 0) {
                    long id = entry.transactionId;
                    Transaction transaction = transactions.get(id);
                    transaction.registerNewTable(table, position);
                }
                bootTable(table, entry.transactionId, null, true);
                if (entry.transactionId <= 0) {
                    writeTablesOnDataStorageManager(position, false);
                }
            }
            break;
        case LogEntryType.CREATE_INDEX:
            {
                Index index = Index.deserialize(entry.value.to_array());
                if (entry.transactionId > 0) {
                    long id = entry.transactionId;
                    Transaction transaction = transactions.get(id);
                    transaction.registerNewIndex(index, position);
                }
                AbstractTableManager tableManager = tables.get(index.table);
                if (tableManager == null) {
                    throw new RuntimeException("table " + index.table + " does not exists");
                }
                bootIndex(index, tableManager, true, entry.transactionId, true, false);
                if (entry.transactionId <= 0) {
                    writeTablesOnDataStorageManager(position, false);
                }
            }
            break;
        case LogEntryType.DROP_TABLE:
            {
                String tableName = entry.tableName;
                if (entry.transactionId > 0) {
                    long id = entry.transactionId;
                    Transaction transaction = transactions.get(id);
                    transaction.registerDropTable(tableName, position);
                } else {
                    AbstractTableManager manager = tables.get(tableName);
                    if (manager != null) {
                        disposeTable(manager);
                        Map<String, AbstractIndexManager> indexes = indexesByTable.get(tableName);
                        if (indexes != null && !indexes.isEmpty()) {
                            LOGGER.log(Level.SEVERE, "It looks like we are dropping a table " + tableName + " with these indexes " + indexes);
                        }
                    }
                }
                if (entry.transactionId <= 0) {
                    writeTablesOnDataStorageManager(position, false);
                }
            }
            break;
        case LogEntryType.DROP_INDEX:
            {
                String indexName = entry.value.to_string();
                if (entry.transactionId > 0) {
                    long id = entry.transactionId;
                    Transaction transaction = transactions.get(id);
                    transaction.registerDropIndex(indexName, position);
                } else {
                    AbstractIndexManager manager = indexes.get(indexName);
                    if (manager != null) {
                        disposeIndexManager(manager);
                    }
                }
                if (entry.transactionId <= 0) {
                    writeTablesOnDataStorageManager(position, false);
                    dbmanager.getPlanner().clearCache();
                }
            }
            break;
        case LogEntryType.ALTER_TABLE:
            {
                Table table = Table.deserialize(entry.value.to_array());
                alterTable(table, null);
                writeTablesOnDataStorageManager(position, false);
            }
            break;
        case LogEntryType.TABLE_CONSISTENCY_CHECK:
            {
                /*
                    In recovery mode, we need to skip the consistency check.
                    The tablespace may not be avaible yet and therefore calcite will not able to performed the select query.
                */
                if (recovery) {
                    LOGGER.log(Level.INFO, "skip {0} consistency check LogEntry {1}", new Object[] { tableSpaceName, entry });
                    break;
                }
                try {
                    TableChecksum check = MAPPER.readValue(entry.value.to_array(), TableChecksum.class);
                    String tableSpace = check.getTableSpaceName();
                    String query = check.getQuery();
                    String tableName = entry.tableName;
                    // In the entry type = 14, the follower will have to run the query on the transaction log
                    if (!isLeader()) {
                        AbstractTableManager tablemanager = this.getTableManager(tableName);
                        DBManager manager = this.getDbmanager();
                        if (tablemanager == null || tablemanager.getCreatedInTransaction() > 0) {
                            throw new TableDoesNotExistException(String.format("Table %s does not exist.", tablemanager));
                        }
                        /*
                            scan = true
                            allowCache = false
                            returnValues = false
                            maxRows = -1
                        */
                        TranslatedQuery translated = manager.getPlanner().translate(tableSpace, query, Collections.emptyList(), true, false, false, -1);
                        TableChecksum scanResult = TableDataChecksum.createChecksum(manager, translated, this, tableSpace, tableName);
                        long followerDigest = scanResult.getDigest();
                        long leaderDigest = check.getDigest();
                        long leaderNumRecords = check.getNumRecords();
                        long followerNumRecords = scanResult.getNumRecords();
                        // the necessary condition to pass the check is to have exactly the same digest and the number of records processed
                        if (followerDigest == leaderDigest && leaderNumRecords == followerNumRecords) {
                            LOGGER.log(Level.INFO, "Data consistency check PASS for table {0}  tablespace {1} with  Checksum {2}", new Object[] { tableName, tableSpace, followerDigest });
                        } else {
                            LOGGER.log(Level.SEVERE, "Data consistency check FAILED for table {0} in tablespace {1} with Checksum {2}", new Object[] { tableName, tableSpace, followerDigest });
                        }
                    } else {
                        long digest = check.getDigest();
                        LOGGER.log(Level.INFO, "Created checksum {0}  for table {1} in tablespace {2} on node {3}", new Object[] { digest, entry.tableName, tableSpace, this.getDbmanager().getNodeId() });
                    }
                } catch (IOException | DataScannerException ex) {
                    LOGGER.log(Level.SEVERE, "Error during table consistency check ", ex);
                }
            }
            break;
        default:
            // other entry types are not important for the tablespacemanager
            break;
    }
    if (entry.tableName != null && entry.type != LogEntryType.CREATE_TABLE && entry.type != LogEntryType.CREATE_INDEX && entry.type != LogEntryType.ALTER_TABLE && entry.type != LogEntryType.DROP_TABLE && entry.type != LogEntryType.TABLE_CONSISTENCY_CHECK) {
        AbstractTableManager tableManager = tables.get(entry.tableName);
        tableManager.apply(position, entry, recovery);
    }
}
Also used : DataStorageManagerException(herddb.storage.DataStorageManagerException) Table(herddb.model.Table) TranslatedQuery(herddb.sql.TranslatedQuery) LogSequenceNumber(herddb.log.LogSequenceNumber) Index(herddb.model.Index) TableDoesNotExistException(herddb.model.TableDoesNotExistException) Transaction(herddb.model.Transaction) CopyOnWriteArrayList(java.util.concurrent.CopyOnWriteArrayList) ArrayList(java.util.ArrayList) List(java.util.List) TableChecksum(herddb.data.consistency.TableChecksum) Map(java.util.Map) ConcurrentHashMap(java.util.concurrent.ConcurrentHashMap) HashMap(java.util.HashMap)

Example 74 with Index

use of herddb.model.Index in project herddb by diennea.

the class TableSpaceManager method recover.

void recover(TableSpace tableSpaceInfo) throws DataStorageManagerException, LogNotAvailableException, MetadataStorageManagerException {
    if (recoveryInProgress) {
        throw new HerdDBInternalException("Cannot run recovery twice");
    }
    recoveryInProgress = true;
    LogSequenceNumber logSequenceNumber = dataStorageManager.getLastcheckpointSequenceNumber(tableSpaceUUID);
    actualLogSequenceNumber = logSequenceNumber;
    LOGGER.log(Level.INFO, "{0} recover {1}, logSequenceNumber from DataStorage: {2}", new Object[] { nodeId, tableSpaceName, logSequenceNumber });
    List<Table> tablesAtBoot = dataStorageManager.loadTables(logSequenceNumber, tableSpaceUUID);
    List<Index> indexesAtBoot = dataStorageManager.loadIndexes(logSequenceNumber, tableSpaceUUID);
    String tableNames = tablesAtBoot.stream().map(t -> {
        return t.name;
    }).collect(Collectors.joining(","));
    String indexNames = indexesAtBoot.stream().map(t -> {
        return t.name + " on table " + t.table;
    }).collect(Collectors.joining(","));
    if (!tableNames.isEmpty()) {
        LOGGER.log(Level.INFO, "{0} {1} tablesAtBoot: {2}, indexesAtBoot: {3}", new Object[] { nodeId, tableSpaceName, tableNames, indexNames });
    }
    for (Table table : tablesAtBoot) {
        TableManager tableManager = bootTable(table, 0, null, false);
        for (Index index : indexesAtBoot) {
            if (index.table.equals(table.name)) {
                bootIndex(index, tableManager, false, 0, false, false);
            }
        }
    }
    dataStorageManager.loadTransactions(logSequenceNumber, tableSpaceUUID, t -> {
        transactions.put(t.transactionId, t);
        LOGGER.log(Level.FINER, "{0} {1} tx {2} at boot lsn {3}", new Object[] { nodeId, tableSpaceName, t.transactionId, t.lastSequenceNumber });
        try {
            if (t.newTables != null) {
                for (Table table : t.newTables.values()) {
                    if (!tables.containsKey(table.name)) {
                        bootTable(table, t.transactionId, null, false);
                    }
                }
            }
            if (t.newIndexes != null) {
                for (Index index : t.newIndexes.values()) {
                    if (!indexes.containsKey(index.name)) {
                        AbstractTableManager tableManager = tables.get(index.table);
                        bootIndex(index, tableManager, false, t.transactionId, false, false);
                    }
                }
            }
        } catch (Exception err) {
            LOGGER.log(Level.SEVERE, "error while booting tmp tables " + err, err);
            throw new RuntimeException(err);
        }
    });
    if (LogSequenceNumber.START_OF_TIME.equals(logSequenceNumber) && dbmanager.getServerConfiguration().getBoolean(ServerConfiguration.PROPERTY_BOOT_FORCE_DOWNLOAD_SNAPSHOT, ServerConfiguration.PROPERTY_BOOT_FORCE_DOWNLOAD_SNAPSHOT_DEFAULT)) {
        LOGGER.log(Level.SEVERE, nodeId + " full recovery of data is forced (" + ServerConfiguration.PROPERTY_BOOT_FORCE_DOWNLOAD_SNAPSHOT + "=true) for tableSpace " + tableSpaceName);
        downloadTableSpaceData();
        log.recovery(actualLogSequenceNumber, new ApplyEntryOnRecovery(), false);
    } else {
        try {
            log.recovery(logSequenceNumber, new ApplyEntryOnRecovery(), false);
        } catch (FullRecoveryNeededException fullRecoveryNeeded) {
            LOGGER.log(Level.SEVERE, nodeId + " full recovery of data is needed for tableSpace " + tableSpaceName, fullRecoveryNeeded);
            downloadTableSpaceData();
            log.recovery(actualLogSequenceNumber, new ApplyEntryOnRecovery(), false);
        }
    }
    recoveryInProgress = false;
    if (!LogSequenceNumber.START_OF_TIME.equals(actualLogSequenceNumber)) {
        LOGGER.log(Level.INFO, "Recovery finished for {0} seqNum {1}", new Object[] { tableSpaceName, actualLogSequenceNumber });
        checkpoint(false, false, false);
    }
}
Also used : HDBException(herddb.client.HDBException) SystablesTableManager(herddb.core.system.SystablesTableManager) CommitTransactionStatement(herddb.model.commands.CommitTransactionStatement) TableCheckpoint(herddb.core.AbstractTableManager.TableCheckpoint) Table(herddb.model.Table) ClientConfiguration(herddb.client.ClientConfiguration) SyslogstatusManager(herddb.core.system.SyslogstatusManager) OpStatsLogger(org.apache.bookkeeper.stats.OpStatsLogger) IndexAlreadyExistsException(herddb.model.IndexAlreadyExistsException) Map(java.util.Map) DDLStatementExecutionResult(herddb.model.DDLStatementExecutionResult) LogNotAvailableException(herddb.log.LogNotAvailableException) PduCodec(herddb.proto.PduCodec) CommitLogResult(herddb.log.CommitLogResult) ClientSideMetadataProviderException(herddb.client.ClientSideMetadataProviderException) LogSequenceNumber(herddb.log.LogSequenceNumber) Set(java.util.Set) ScanStatement(herddb.model.commands.ScanStatement) CountDownLatch(java.util.concurrent.CountDownLatch) Stream(java.util.stream.Stream) HDBClient(herddb.client.HDBClient) TranslatedQuery(herddb.sql.TranslatedQuery) StatsLogger(org.apache.bookkeeper.stats.StatsLogger) TableDataChecksum(herddb.data.consistency.TableDataChecksum) DropTableStatement(herddb.model.commands.DropTableStatement) CopyOnWriteArrayList(java.util.concurrent.CopyOnWriteArrayList) Bytes(herddb.utils.Bytes) SysdualTableManager(herddb.core.system.SysdualTableManager) LogEntry(herddb.log.LogEntry) SysnodesTableManager(herddb.core.system.SysnodesTableManager) ArrayList(java.util.ArrayList) CreateIndexStatement(herddb.model.commands.CreateIndexStatement) TableSpaceDoesNotExistException(herddb.model.TableSpaceDoesNotExistException) TransactionContext(herddb.model.TransactionContext) Transaction(herddb.model.Transaction) BRINIndexManager(herddb.index.brin.BRINIndexManager) SystablestatsTableManager(herddb.core.system.SystablestatsTableManager) BiConsumer(java.util.function.BiConsumer) CommitLogListener(herddb.log.CommitLogListener) ForeignKeyDef(herddb.model.ForeignKeyDef) IndexDoesNotExistException(herddb.model.IndexDoesNotExistException) TableSpaceManagerStats(herddb.core.stats.TableSpaceManagerStats) LogEntryType(herddb.log.LogEntryType) LogEntryFactory(herddb.log.LogEntryFactory) IOException(java.io.IOException) DataStorageManager(herddb.storage.DataStorageManager) DropIndexStatement(herddb.model.commands.DropIndexStatement) ColumnTypes(herddb.model.ColumnTypes) ExecutionException(java.util.concurrent.ExecutionException) AtomicLong(java.util.concurrent.atomic.AtomicLong) Column(herddb.model.Column) KeyValue(herddb.utils.KeyValue) FullRecoveryNeededException(herddb.log.FullRecoveryNeededException) StampedLock(java.util.concurrent.locks.StampedLock) ServerHostData(herddb.network.ServerHostData) TableAlreadyExistsException(herddb.model.TableAlreadyExistsException) RollbackTransactionStatement(herddb.model.commands.RollbackTransactionStatement) CreateTableStatement(herddb.model.commands.CreateTableStatement) TimeoutException(java.util.concurrent.TimeoutException) JMXUtils(herddb.jmx.JMXUtils) TransactionResult(herddb.model.TransactionResult) MetadataStorageManager(herddb.metadata.MetadataStorageManager) Channel(herddb.network.Channel) ServerConfiguration(herddb.server.ServerConfiguration) Futures(herddb.utils.Futures) DataStorageManagerException(herddb.storage.DataStorageManagerException) Index(herddb.model.Index) TableDoesNotExistException(herddb.model.TableDoesNotExistException) AlterTableStatement(herddb.model.commands.AlterTableStatement) SysindexcolumnsTableManager(herddb.core.system.SysindexcolumnsTableManager) DataScanner(herddb.model.DataScanner) DDLException(herddb.model.DDLException) StatementExecutionException(herddb.model.StatementExecutionException) Collection(java.util.Collection) SysstatementsTableManager(herddb.core.system.SysstatementsTableManager) BeginTransactionStatement(herddb.model.commands.BeginTransactionStatement) ConcurrentHashMap(java.util.concurrent.ConcurrentHashMap) CompletionException(java.util.concurrent.CompletionException) TableAwareStatement(herddb.model.TableAwareStatement) Logger(java.util.logging.Logger) EOFException(java.io.EOFException) Collectors(java.util.stream.Collectors) Objects(java.util.Objects) HDBConnection(herddb.client.HDBConnection) List(java.util.List) FullTableScanConsumer(herddb.storage.FullTableScanConsumer) SystransactionsTableManager(herddb.core.system.SystransactionsTableManager) NodeMetadata(herddb.model.NodeMetadata) Entry(java.util.Map.Entry) Optional(java.util.Optional) TableSpace(herddb.model.TableSpace) SysconfigTableManager(herddb.core.system.SysconfigTableManager) SuppressFBWarnings(edu.umd.cs.findbugs.annotations.SuppressFBWarnings) SysindexesTableManager(herddb.core.system.SysindexesTableManager) Statement(herddb.model.Statement) MetadataStorageManagerException(herddb.metadata.MetadataStorageManagerException) DataScannerException(herddb.model.DataScannerException) SystablespacesTableManager(herddb.core.system.SystablespacesTableManager) HashMap(java.util.HashMap) CompletableFuture(java.util.concurrent.CompletableFuture) Pdu(herddb.proto.Pdu) Level(java.util.logging.Level) HashSet(java.util.HashSet) SysclientsTableManager(herddb.core.system.SysclientsTableManager) TableChecksum(herddb.data.consistency.TableChecksum) ExecutorService(java.util.concurrent.ExecutorService) DumpedLogEntry(herddb.backup.DumpedLogEntry) SysforeignkeysTableManager(herddb.core.system.SysforeignkeysTableManager) ObjectMapper(com.fasterxml.jackson.databind.ObjectMapper) StatementExecutionResult(herddb.model.StatementExecutionResult) TimeUnit(java.util.concurrent.TimeUnit) CommitLog(herddb.log.CommitLog) ClientSideMetadataProvider(herddb.client.ClientSideMetadataProvider) ConcurrentSkipListSet(java.util.concurrent.ConcurrentSkipListSet) SystablespacereplicastateTableManager(herddb.core.system.SystablespacereplicastateTableManager) SQLPlannedOperationStatement(herddb.model.commands.SQLPlannedOperationStatement) StatementEvaluationContext(herddb.model.StatementEvaluationContext) SyscolumnsTableManager(herddb.core.system.SyscolumnsTableManager) Comparator(java.util.Comparator) Collections(java.util.Collections) TableManagerStats(herddb.core.stats.TableManagerStats) MemoryHashIndexManager(herddb.index.MemoryHashIndexManager) SystemProperties(herddb.utils.SystemProperties) Table(herddb.model.Table) LogSequenceNumber(herddb.log.LogSequenceNumber) Index(herddb.model.Index) HDBException(herddb.client.HDBException) IndexAlreadyExistsException(herddb.model.IndexAlreadyExistsException) LogNotAvailableException(herddb.log.LogNotAvailableException) ClientSideMetadataProviderException(herddb.client.ClientSideMetadataProviderException) TableSpaceDoesNotExistException(herddb.model.TableSpaceDoesNotExistException) IndexDoesNotExistException(herddb.model.IndexDoesNotExistException) IOException(java.io.IOException) ExecutionException(java.util.concurrent.ExecutionException) FullRecoveryNeededException(herddb.log.FullRecoveryNeededException) TableAlreadyExistsException(herddb.model.TableAlreadyExistsException) TimeoutException(java.util.concurrent.TimeoutException) DataStorageManagerException(herddb.storage.DataStorageManagerException) TableDoesNotExistException(herddb.model.TableDoesNotExistException) DDLException(herddb.model.DDLException) StatementExecutionException(herddb.model.StatementExecutionException) CompletionException(java.util.concurrent.CompletionException) EOFException(java.io.EOFException) MetadataStorageManagerException(herddb.metadata.MetadataStorageManagerException) DataScannerException(herddb.model.DataScannerException) SystablesTableManager(herddb.core.system.SystablesTableManager) SysdualTableManager(herddb.core.system.SysdualTableManager) SysnodesTableManager(herddb.core.system.SysnodesTableManager) SystablestatsTableManager(herddb.core.system.SystablestatsTableManager) SysindexcolumnsTableManager(herddb.core.system.SysindexcolumnsTableManager) SysstatementsTableManager(herddb.core.system.SysstatementsTableManager) SystransactionsTableManager(herddb.core.system.SystransactionsTableManager) SysconfigTableManager(herddb.core.system.SysconfigTableManager) SysindexesTableManager(herddb.core.system.SysindexesTableManager) SystablespacesTableManager(herddb.core.system.SystablespacesTableManager) SysclientsTableManager(herddb.core.system.SysclientsTableManager) SysforeignkeysTableManager(herddb.core.system.SysforeignkeysTableManager) SystablespacereplicastateTableManager(herddb.core.system.SystablespacereplicastateTableManager) SyscolumnsTableManager(herddb.core.system.SyscolumnsTableManager) FullRecoveryNeededException(herddb.log.FullRecoveryNeededException)

Example 75 with Index

use of herddb.model.Index in project herddb by diennea.

the class ReplicaFullTableDataDumpReceiver method beginTable.

@Override
public void beginTable(DumpedTableMetadata dumpedTable, Map<String, Object> stats) throws DataStorageManagerException {
    Table table = dumpedTable.table;
    LOGGER.log(Level.INFO, "dumpReceiver " + tableSpaceName + ", beginTable " + table.name + ", stats:" + stats + ", dumped at " + dumpedTable.logSequenceNumber + " (general dump at " + logSequenceNumber + ")");
    currentTable = tableSpaceManager.bootTable(table, 0, dumpedTable.logSequenceNumber, false);
    for (Index index : dumpedTable.indexes) {
        tableSpaceManager.bootIndex(index, currentTable, false, 0, false, true);
    }
}
Also used : Table(herddb.model.Table) Index(herddb.model.Index)

Aggregations

Index (herddb.model.Index)101 Table (herddb.model.Table)92 CreateIndexStatement (herddb.model.commands.CreateIndexStatement)71 CreateTableStatement (herddb.model.commands.CreateTableStatement)71 DataScanner (herddb.model.DataScanner)67 ScanStatement (herddb.model.commands.ScanStatement)66 TranslatedQuery (herddb.sql.TranslatedQuery)61 Test (org.junit.Test)61 CreateTableSpaceStatement (herddb.model.commands.CreateTableSpaceStatement)52 SecondaryIndexSeek (herddb.index.SecondaryIndexSeek)51 MemoryCommitLogManager (herddb.mem.MemoryCommitLogManager)39 MemoryDataStorageManager (herddb.mem.MemoryDataStorageManager)39 MemoryMetadataStorageManager (herddb.mem.MemoryMetadataStorageManager)39 InsertStatement (herddb.model.commands.InsertStatement)31 DBManager (herddb.core.DBManager)24 GetStatement (herddb.model.commands.GetStatement)23 GetResult (herddb.model.GetResult)22 TransactionContext (herddb.model.TransactionContext)22 Bytes (herddb.utils.Bytes)22 ServerConfiguration (herddb.server.ServerConfiguration)21