Search in sources :

Example 1 with PostCheckpointAction

use of herddb.core.PostCheckpointAction in project herddb by diennea.

the class MemoryDataStorageManager method indexCheckpoint.

@Override
public List<PostCheckpointAction> indexCheckpoint(String tableSpace, String indexName, IndexStatus indexStatus, boolean pin) throws DataStorageManagerException {
    /* Checkpoint pinning */
    final Map<Long, Integer> pins = pinIndexAndGetPages(tableSpace, indexName, indexStatus, pin);
    final Set<LogSequenceNumber> checkpoints = pinIndexAndGetCheckpoints(tableSpace, indexName, indexStatus, pin);
    List<Long> pagesForIndex = new ArrayList<>();
    String prefix = tableSpace + "." + indexName + "_";
    for (String key : indexpages.keySet()) {
        if (key.startsWith(prefix)) {
            long pageId = Long.parseLong(key.substring(prefix.length()));
            if (!pins.containsKey(pageId)) {
                pagesForIndex.add(pageId);
            }
        }
    }
    pagesForIndex.removeAll(indexStatus.activePages);
    List<PostCheckpointAction> result = new ArrayList<>();
    for (long pageId : pagesForIndex) {
        result.add(new PostCheckpointAction(indexName, "drop page " + pageId) {

            @Override
            public void run() {
                // remove only after checkpoint completed
                indexpages.remove(prefix + pageId);
            }
        });
    }
    for (String oldStatus : indexStatuses.keySet()) {
        if (oldStatus.startsWith(prefix)) {
            /* Check for checkpoint skip only if match expected structure */
            final LogSequenceNumber log = evaluateLogSequenceNumber(prefix.substring(0, prefix.length()));
            if (log != null) {
                /* If is pinned skip this status*/
                if (checkpoints.contains(log)) {
                    continue;
                }
            }
            result.add(new PostCheckpointAction(indexName, "drop index checkpoint " + oldStatus) {

                @Override
                public void run() {
                    // remove only after checkpoint completed
                    indexStatuses.remove(oldStatus);
                }
            });
        }
    }
    VisibleByteArrayOutputStream oo = new VisibleByteArrayOutputStream(1024);
    try (ExtendedDataOutputStream dataOutputKeys = new ExtendedDataOutputStream(oo)) {
        indexStatus.serialize(dataOutputKeys);
        dataOutputKeys.flush();
        oo.write(oo.xxhash64());
    } catch (IOException err) {
        throw new DataStorageManagerException(err);
    }
    /* Uses a copy to limit byte[] size at the min needed */
    indexStatuses.put(checkpointName(tableSpace, indexName, indexStatus.sequenceNumber), oo.toByteArray());
    return result;
}
Also used : DataStorageManagerException(herddb.storage.DataStorageManagerException) ArrayList(java.util.ArrayList) LogSequenceNumber(herddb.log.LogSequenceNumber) VisibleByteArrayOutputStream(herddb.utils.VisibleByteArrayOutputStream) IOException(java.io.IOException) PostCheckpointAction(herddb.core.PostCheckpointAction) ExtendedDataOutputStream(herddb.utils.ExtendedDataOutputStream)

Example 2 with PostCheckpointAction

use of herddb.core.PostCheckpointAction in project herddb by diennea.

the class MemoryDataStorageManager method tableCheckpoint.

@Override
public List<PostCheckpointAction> tableCheckpoint(String tableSpace, String tableName, TableStatus tableStatus, boolean pin) throws DataStorageManagerException {
    /* Checkpoint pinning */
    final Map<Long, Integer> pins = pinTableAndGetPages(tableSpace, tableName, tableStatus, pin);
    final Set<LogSequenceNumber> checkpoints = pinTableAndGetCheckpoints(tableSpace, tableName, tableStatus, pin);
    List<Long> pagesForTable = new ArrayList<>();
    String prefix = tableSpace + "." + tableName + "_";
    for (String key : pages.keySet()) {
        if (key.startsWith(prefix)) {
            long pageId = Long.parseLong(key.substring(prefix.length()));
            if (!pins.containsKey(pageId)) {
                pagesForTable.add(pageId);
            }
        }
    }
    pagesForTable.removeAll(tableStatus.activePages.keySet());
    List<PostCheckpointAction> result = new ArrayList<>();
    for (long pageId : pagesForTable) {
        result.add(new PostCheckpointAction(tableName, "drop page " + pageId) {

            @Override
            public void run() {
                // remove only after checkpoint completed
                pages.remove(prefix + pageId);
                LOGGER.log(Level.SEVERE, "removing " + (prefix + pageId));
            }
        });
    }
    for (String oldStatus : tableStatuses.keySet()) {
        if (oldStatus.startsWith(prefix)) {
            /* Check for checkpoint skip only if match expected structure */
            final LogSequenceNumber log = evaluateLogSequenceNumber(prefix.substring(0, prefix.length()));
            if (log != null) {
                /* If is pinned skip this status*/
                if (checkpoints.contains(log)) {
                    continue;
                }
            }
            result.add(new PostCheckpointAction(tableName, "drop table checkpoint " + oldStatus) {

                @Override
                public void run() {
                    // remove only after checkpoint completed
                    tableStatuses.remove(oldStatus);
                }
            });
        }
    }
    VisibleByteArrayOutputStream oo = new VisibleByteArrayOutputStream(1024);
    try (ExtendedDataOutputStream dataOutputKeys = new ExtendedDataOutputStream(oo)) {
        tableStatus.serialize(dataOutputKeys);
        dataOutputKeys.flush();
        oo.write(oo.xxhash64());
    } catch (IOException err) {
        throw new DataStorageManagerException(err);
    }
    /* Uses a copy to limit byte[] size at the min needed */
    tableStatuses.put(checkpointName(tableSpace, tableName, tableStatus.sequenceNumber), oo.toByteArray());
    return result;
}
Also used : DataStorageManagerException(herddb.storage.DataStorageManagerException) ArrayList(java.util.ArrayList) LogSequenceNumber(herddb.log.LogSequenceNumber) VisibleByteArrayOutputStream(herddb.utils.VisibleByteArrayOutputStream) IOException(java.io.IOException) PostCheckpointAction(herddb.core.PostCheckpointAction) ExtendedDataOutputStream(herddb.utils.ExtendedDataOutputStream)

Example 3 with PostCheckpointAction

use of herddb.core.PostCheckpointAction in project herddb by diennea.

the class FileDataStorageManager method writeTransactionsAtCheckpoint.

@Override
public Collection<PostCheckpointAction> writeTransactionsAtCheckpoint(String tableSpace, LogSequenceNumber sequenceNumber, Collection<Transaction> transactions) throws DataStorageManagerException {
    if (sequenceNumber.isStartOfTime() && !transactions.isEmpty()) {
        throw new DataStorageManagerException("impossible to write a non empty transactions list at start-of-time");
    }
    Path tableSpaceDirectory = getTablespaceDirectory(tableSpace);
    try {
        Files.createDirectories(tableSpaceDirectory);
        Path checkPointFile = getTablespaceTransactionsFile(tableSpace, sequenceNumber);
        Path parent = getParent(checkPointFile);
        Files.createDirectories(parent);
        Path checkpointFileTemp = parent.resolve(checkPointFile.getFileName() + ".tmp");
        LOGGER.log(Level.FINE, "writeTransactionsAtCheckpoint for tableSpace {0} sequenceNumber {1} to {2}, active transactions {3}", new Object[] { tableSpace, sequenceNumber, checkPointFile.toAbsolutePath().toString(), transactions.size() });
        try (ManagedFile file = ManagedFile.open(checkpointFileTemp);
            SimpleBufferedOutputStream buffer = new SimpleBufferedOutputStream(file.getOutputStream(), COPY_BUFFERS_SIZE);
            ExtendedDataOutputStream dout = new ExtendedDataOutputStream(buffer)) {
            // version
            dout.writeVLong(1);
            // flags for future implementations
            dout.writeVLong(0);
            dout.writeUTF(tableSpace);
            dout.writeZLong(sequenceNumber.ledgerId);
            dout.writeZLong(sequenceNumber.offset);
            dout.writeInt(transactions.size());
            for (Transaction t : transactions) {
                t.serialize(dout);
            }
            dout.flush();
            file.sync();
        } catch (IOException err) {
            throw new DataStorageManagerException(err);
        }
        try {
            Files.move(checkpointFileTemp, checkPointFile, StandardCopyOption.ATOMIC_MOVE, StandardCopyOption.REPLACE_EXISTING);
        } catch (IOException err) {
            throw new DataStorageManagerException(err);
        }
    } catch (IOException err) {
        throw new DataStorageManagerException(err);
    }
    Collection<PostCheckpointAction> result = new ArrayList<>();
    try (DirectoryStream<Path> stream = Files.newDirectoryStream(tableSpaceDirectory)) {
        for (Path p : stream) {
            if (isTransactionsFile(p)) {
                try {
                    LogSequenceNumber logPositionInFile = readLogSequenceNumberFromTransactionsFile(tableSpace, p);
                    if (sequenceNumber.after(logPositionInFile)) {
                        LOGGER.log(Level.FINEST, "transactions metadata file " + p.toAbsolutePath() + ". will be deleted after checkpoint end");
                        result.add(new DeleteFileAction("transactions", "delete transactions file " + p.toAbsolutePath(), p));
                    }
                } catch (DataStorageManagerException ignore) {
                    LOGGER.log(Level.SEVERE, "Unparsable transactions file " + p.toAbsolutePath(), ignore);
                    result.add(new DeleteFileAction("transactions", "delete unparsable transactions file " + p.toAbsolutePath(), p));
                }
            }
        }
    } catch (IOException err) {
        LOGGER.log(Level.SEVERE, "Could not list dir " + tableSpaceDirectory, err);
    }
    return result;
}
Also used : Path(java.nio.file.Path) DataStorageManagerException(herddb.storage.DataStorageManagerException) ArrayList(java.util.ArrayList) LogSequenceNumber(herddb.log.LogSequenceNumber) IOException(java.io.IOException) ManagedFile(herddb.utils.ManagedFile) ExtendedDataOutputStream(herddb.utils.ExtendedDataOutputStream) PostCheckpointAction(herddb.core.PostCheckpointAction) SimpleBufferedOutputStream(herddb.utils.SimpleBufferedOutputStream) Transaction(herddb.model.Transaction)

Example 4 with PostCheckpointAction

use of herddb.core.PostCheckpointAction in project herddb by diennea.

the class FileDataStorageManager method indexCheckpoint.

@Override
public List<PostCheckpointAction> indexCheckpoint(String tableSpace, String indexName, IndexStatus indexStatus, boolean pin) throws DataStorageManagerException {
    Path dir = getIndexDirectory(tableSpace, indexName);
    LogSequenceNumber logPosition = indexStatus.sequenceNumber;
    Path checkpointFile = getTableCheckPointsFile(dir, logPosition);
    Path parent = getParent(checkpointFile);
    Path checkpointFileTemp = parent.resolve(checkpointFile.getFileName() + ".tmp");
    try {
        Files.createDirectories(dir);
        if (Files.isRegularFile(checkpointFile)) {
            IndexStatus actualStatus = readIndexStatusFromFile(checkpointFile);
            if (actualStatus != null && actualStatus.equals(indexStatus)) {
                LOGGER.log(Level.SEVERE, "indexCheckpoint " + tableSpace + ", " + indexName + ": " + indexStatus + " already saved on" + checkpointFile);
                return Collections.emptyList();
            }
        }
    } catch (IOException err) {
        throw new DataStorageManagerException(err);
    }
    LOGGER.log(Level.FINE, "indexCheckpoint " + tableSpace + ", " + indexName + ": " + indexStatus + " to file " + checkpointFile);
    try (ManagedFile file = ManagedFile.open(checkpointFileTemp);
        SimpleBufferedOutputStream buffer = new SimpleBufferedOutputStream(file.getOutputStream(), COPY_BUFFERS_SIZE);
        XXHash64Utils.HashingOutputStream oo = new XXHash64Utils.HashingOutputStream(buffer);
        ExtendedDataOutputStream dataOutputKeys = new ExtendedDataOutputStream(oo)) {
        // version
        dataOutputKeys.writeVLong(1);
        // flags for future implementations
        dataOutputKeys.writeVLong(0);
        indexStatus.serialize(dataOutputKeys);
        dataOutputKeys.writeLong(oo.hash());
        dataOutputKeys.flush();
        file.sync();
    } catch (IOException err) {
        throw new DataStorageManagerException(err);
    }
    try {
        Files.move(checkpointFileTemp, checkpointFile, StandardCopyOption.REPLACE_EXISTING, StandardCopyOption.ATOMIC_MOVE);
    } catch (IOException err) {
        throw new DataStorageManagerException(err);
    }
    /* Checkpoint pinning */
    final Map<Long, Integer> pins = pinIndexAndGetPages(tableSpace, indexName, indexStatus, pin);
    final Set<LogSequenceNumber> checkpoints = pinIndexAndGetCheckpoints(tableSpace, indexName, indexStatus, pin);
    long maxPageId = indexStatus.activePages.stream().max(Comparator.naturalOrder()).orElse(Long.MAX_VALUE);
    List<PostCheckpointAction> result = new ArrayList<>();
    // we can drop old page files now
    List<Path> pageFiles = getIndexPageFiles(tableSpace, indexName);
    for (Path p : pageFiles) {
        long pageId = getPageId(p);
        LOGGER.log(Level.FINEST, "checkpoint file {0} pageId {1}", new Object[] { p.toAbsolutePath(), pageId });
        if (pageId > 0 && !pins.containsKey(pageId) && !indexStatus.activePages.contains(pageId) && pageId < maxPageId) {
            LOGGER.log(Level.FINEST, "checkpoint file " + p.toAbsolutePath() + " pageId " + pageId + ". will be deleted after checkpoint end");
            result.add(new DeleteFileAction(indexName, "delete page " + pageId + " file " + p.toAbsolutePath(), p));
        }
    }
    try (DirectoryStream<Path> stream = Files.newDirectoryStream(dir)) {
        for (Path p : stream) {
            if (isTableOrIndexCheckpointsFile(p) && !p.equals(checkpointFile)) {
                IndexStatus status = readIndexStatusFromFile(p);
                if (logPosition.after(status.sequenceNumber) && !checkpoints.contains(status.sequenceNumber)) {
                    LOGGER.log(Level.FINEST, "checkpoint metadata file " + p.toAbsolutePath() + ". will be deleted after checkpoint end");
                    result.add(new DeleteFileAction(indexName, "delete checkpoint metadata file " + p.toAbsolutePath(), p));
                }
            }
        }
    } catch (IOException err) {
        LOGGER.log(Level.SEVERE, "Could not list indexName dir " + dir, err);
    }
    return result;
}
Also used : Path(java.nio.file.Path) DataStorageManagerException(herddb.storage.DataStorageManagerException) ArrayList(java.util.ArrayList) LogSequenceNumber(herddb.log.LogSequenceNumber) IOException(java.io.IOException) XXHash64Utils(herddb.utils.XXHash64Utils) ManagedFile(herddb.utils.ManagedFile) ExtendedDataOutputStream(herddb.utils.ExtendedDataOutputStream) PostCheckpointAction(herddb.core.PostCheckpointAction) SimpleBufferedOutputStream(herddb.utils.SimpleBufferedOutputStream) IndexStatus(herddb.storage.IndexStatus)

Example 5 with PostCheckpointAction

use of herddb.core.PostCheckpointAction in project herddb by diennea.

the class FileDataStorageManager method writeCheckpointSequenceNumber.

@Override
public Collection<PostCheckpointAction> writeCheckpointSequenceNumber(String tableSpace, LogSequenceNumber sequenceNumber) throws DataStorageManagerException {
    Path tableSpaceDirectory = getTablespaceDirectory(tableSpace);
    try {
        Files.createDirectories(tableSpaceDirectory);
        Path checkPointFile = getTablespaceCheckPointInfoFile(tableSpace, sequenceNumber);
        Path parent = getParent(checkPointFile);
        Files.createDirectories(parent);
        Path checkpointFileTemp = parent.resolve(checkPointFile.getFileName() + ".tmp");
        LOGGER.log(Level.INFO, "checkpoint for " + tableSpace + " at " + sequenceNumber + " to " + checkPointFile.toAbsolutePath().toString());
        try (ManagedFile file = ManagedFile.open(checkpointFileTemp);
            SimpleBufferedOutputStream buffer = new SimpleBufferedOutputStream(file.getOutputStream(), COPY_BUFFERS_SIZE);
            ExtendedDataOutputStream dout = new ExtendedDataOutputStream(buffer)) {
            // version
            dout.writeVLong(1);
            // flags for future implementations
            dout.writeVLong(0);
            dout.writeUTF(tableSpace);
            dout.writeZLong(sequenceNumber.ledgerId);
            dout.writeZLong(sequenceNumber.offset);
            dout.flush();
            file.sync();
        } catch (IOException err) {
            throw new DataStorageManagerException(err);
        }
        // write file atomically
        Files.move(checkpointFileTemp, checkPointFile, StandardCopyOption.ATOMIC_MOVE, StandardCopyOption.REPLACE_EXISTING);
    } catch (IOException err) {
        throw new DataStorageManagerException(err);
    }
    Collection<PostCheckpointAction> result = new ArrayList<>();
    try (DirectoryStream<Path> stream = Files.newDirectoryStream(tableSpaceDirectory)) {
        for (Path p : stream) {
            if (isTablespaceCheckPointInfoFile(p)) {
                try {
                    LogSequenceNumber logPositionInFile = readLogSequenceNumberFromCheckpointInfoFile(tableSpace, p);
                    if (sequenceNumber.after(logPositionInFile)) {
                        LOGGER.log(Level.FINEST, "checkpoint info file " + p.toAbsolutePath() + ". will be deleted after checkpoint end");
                        result.add(new DeleteFileAction("checkpoint", "delete checkpoint info file " + p.toAbsolutePath(), p));
                    }
                } catch (DataStorageManagerException ignore) {
                    LOGGER.log(Level.SEVERE, "unparsable checkpoint info file " + p.toAbsolutePath(), ignore);
                // do not auto-delete checkpoint files
                }
            }
        }
    } catch (IOException err) {
        LOGGER.log(Level.SEVERE, "Could not list dir " + tableSpaceDirectory, err);
    }
    return result;
}
Also used : Path(java.nio.file.Path) SimpleBufferedOutputStream(herddb.utils.SimpleBufferedOutputStream) DataStorageManagerException(herddb.storage.DataStorageManagerException) ArrayList(java.util.ArrayList) LogSequenceNumber(herddb.log.LogSequenceNumber) IOException(java.io.IOException) ManagedFile(herddb.utils.ManagedFile) ExtendedDataOutputStream(herddb.utils.ExtendedDataOutputStream) PostCheckpointAction(herddb.core.PostCheckpointAction)

Aggregations

PostCheckpointAction (herddb.core.PostCheckpointAction)10 ArrayList (java.util.ArrayList)10 DataStorageManagerException (herddb.storage.DataStorageManagerException)9 IOException (java.io.IOException)9 LogSequenceNumber (herddb.log.LogSequenceNumber)7 ExtendedDataOutputStream (herddb.utils.ExtendedDataOutputStream)7 ManagedFile (herddb.utils.ManagedFile)5 SimpleBufferedOutputStream (herddb.utils.SimpleBufferedOutputStream)5 Path (java.nio.file.Path)5 IndexStatus (herddb.storage.IndexStatus)4 Bytes (herddb.utils.Bytes)3 AtomicLong (java.util.concurrent.atomic.AtomicLong)3 VisibleByteArrayOutputStream (herddb.utils.VisibleByteArrayOutputStream)2 XXHash64Utils (herddb.utils.XXHash64Utils)2 HashSet (java.util.HashSet)2 KeyToPageIndex (herddb.index.KeyToPageIndex)1 BLinkKeyToPageIndex (herddb.index.blink.BLinkKeyToPageIndex)1 Index (herddb.model.Index)1 Table (herddb.model.Table)1 Transaction (herddb.model.Transaction)1