Search in sources :

Example 66 with DataStorageManagerException

use of herddb.storage.DataStorageManagerException in project herddb by diennea.

the class FileDataStorageManager method cleanupAfterBoot.

@Override
public void cleanupAfterBoot(String tableSpace, String tableName, Set<Long> activePagesAtBoot) throws DataStorageManagerException {
    // we have to drop old page files or page files partially written by checkpoint interrupted at JVM crash/reboot
    List<Path> pageFiles = getTablePageFiles(tableSpace, tableName);
    for (Path p : pageFiles) {
        long pageId = getPageId(p);
        LOGGER.log(Level.FINER, "cleanupAfterBoot file " + p.toAbsolutePath() + " pageId " + pageId);
        if (pageId > 0 && !activePagesAtBoot.contains(pageId)) {
            LOGGER.log(Level.SEVERE, "cleanupAfterBoot file " + p.toAbsolutePath() + " pageId " + pageId + ". will be deleted");
            try {
                Files.deleteIfExists(p);
            } catch (IOException err) {
                throw new DataStorageManagerException(err);
            }
        }
    }
}
Also used : Path(java.nio.file.Path) DataStorageManagerException(herddb.storage.DataStorageManagerException) IOException(java.io.IOException)

Example 67 with DataStorageManagerException

use of herddb.storage.DataStorageManagerException in project herddb by diennea.

the class FileDataStorageManager method writeIndexPage.

@Override
public void writeIndexPage(String tableSpace, String indexName, long pageId, DataWriter writer) throws DataStorageManagerException {
    long _start = System.currentTimeMillis();
    Path tableDir = getIndexDirectory(tableSpace, indexName);
    try {
        Files.createDirectories(tableDir);
    } catch (IOException err) {
        throw new DataStorageManagerException(err);
    }
    Path pageFile = getPageFile(tableDir, pageId);
    long size;
    try (ManagedFile file = ManagedFile.open(pageFile, StandardOpenOption.WRITE, StandardOpenOption.CREATE);
        SimpleBufferedOutputStream buffer = new SimpleBufferedOutputStream(file.getOutputStream(), COPY_BUFFERS_SIZE);
        XXHash64Utils.HashingOutputStream oo = new XXHash64Utils.HashingOutputStream(buffer);
        ExtendedDataOutputStream dataOutput = new ExtendedDataOutputStream(oo)) {
        // version
        dataOutput.writeVLong(1);
        // flags for future implementations
        dataOutput.writeVLong(0);
        size = oo.size();
        writer.write(dataOutput);
        size = oo.size() - size;
        // footer
        dataOutput.writeLong(oo.hash());
        dataOutput.flush();
        file.sync();
    } catch (IOException err) {
        throw new DataStorageManagerException(err);
    }
    long now = System.currentTimeMillis();
    if (LOGGER.isLoggable(Level.FINER)) {
        LOGGER.log(Level.FINER, "writePage {0} KBytes, time {2} ms", new Object[] { (size / 1024) + "", (now - _start) + "" });
    }
}
Also used : Path(java.nio.file.Path) SimpleBufferedOutputStream(herddb.utils.SimpleBufferedOutputStream) DataStorageManagerException(herddb.storage.DataStorageManagerException) IOException(java.io.IOException) XXHash64Utils(herddb.utils.XXHash64Utils) ManagedFile(herddb.utils.ManagedFile) ExtendedDataOutputStream(herddb.utils.ExtendedDataOutputStream)

Example 68 with DataStorageManagerException

use of herddb.storage.DataStorageManagerException in project herddb by diennea.

the class FileDataStorageManager method fullTableScan.

@Override
public void fullTableScan(String tableSpace, String tableUuid, LogSequenceNumber sequenceNumber, FullTableScanConsumer consumer) throws DataStorageManagerException {
    try {
        TableStatus status = getTableStatus(tableSpace, tableUuid, sequenceNumber);
        fullTableScan(tableSpace, tableUuid, status, consumer);
    } catch (HerdDBInternalException err) {
        throw new DataStorageManagerException(err);
    }
}
Also used : DataStorageManagerException(herddb.storage.DataStorageManagerException) HerdDBInternalException(herddb.core.HerdDBInternalException) TableStatus(herddb.storage.TableStatus)

Example 69 with DataStorageManagerException

use of herddb.storage.DataStorageManagerException in project herddb by diennea.

the class FileDataStorageManager method writeTables.

@Override
public Collection<PostCheckpointAction> writeTables(String tableSpace, LogSequenceNumber sequenceNumber, List<Table> tables, List<Index> indexlist) throws DataStorageManagerException {
    if (sequenceNumber.isStartOfTime() && !tables.isEmpty()) {
        throw new DataStorageManagerException("impossible to write a non empty table list at start-of-time");
    }
    Path tableSpaceDirectory = getTablespaceDirectory(tableSpace);
    try {
        Files.createDirectories(tableSpaceDirectory);
        Path fileTables = getTablespaceTablesMetadataFile(tableSpace, sequenceNumber);
        Path fileIndexes = getTablespaceIndexesMetadataFile(tableSpace, sequenceNumber);
        Path parent = getParent(fileTables);
        Files.createDirectories(parent);
        LOGGER.log(Level.FINE, "writeTables for tableSpace " + tableSpace + " sequenceNumber " + sequenceNumber + " to " + fileTables.toAbsolutePath().toString());
        try (ManagedFile file = ManagedFile.open(fileTables);
            SimpleBufferedOutputStream buffer = new SimpleBufferedOutputStream(file.getOutputStream(), COPY_BUFFERS_SIZE);
            ExtendedDataOutputStream dout = new ExtendedDataOutputStream(buffer)) {
            // version
            dout.writeVLong(1);
            // flags for future implementations
            dout.writeVLong(0);
            dout.writeUTF(tableSpace);
            dout.writeZLong(sequenceNumber.ledgerId);
            dout.writeZLong(sequenceNumber.offset);
            dout.writeInt(tables.size());
            for (Table t : tables) {
                byte[] tableSerialized = t.serialize();
                dout.writeArray(tableSerialized);
            }
            dout.flush();
            file.sync();
        } catch (IOException err) {
            throw new DataStorageManagerException(err);
        }
        try (ManagedFile file = ManagedFile.open(fileIndexes);
            SimpleBufferedOutputStream buffer = new SimpleBufferedOutputStream(file.getOutputStream(), COPY_BUFFERS_SIZE);
            ExtendedDataOutputStream dout = new ExtendedDataOutputStream(buffer)) {
            // version
            dout.writeVLong(1);
            // flags for future implementations
            dout.writeVLong(0);
            dout.writeUTF(tableSpace);
            dout.writeZLong(sequenceNumber.ledgerId);
            dout.writeZLong(sequenceNumber.offset);
            if (indexlist != null) {
                dout.writeInt(indexlist.size());
                for (Index t : indexlist) {
                    byte[] indexSerialized = t.serialize();
                    dout.writeArray(indexSerialized);
                }
            } else {
                dout.writeInt(0);
            }
            dout.flush();
            file.sync();
        } catch (IOException err) {
            throw new DataStorageManagerException(err);
        }
    } catch (IOException err) {
        throw new DataStorageManagerException(err);
    }
    Collection<PostCheckpointAction> result = new ArrayList<>();
    try (DirectoryStream<Path> stream = Files.newDirectoryStream(tableSpaceDirectory)) {
        for (Path p : stream) {
            if (isTablespaceIndexesMetadataFile(p)) {
                try {
                    LogSequenceNumber logPositionInFile = readLogSequenceNumberFromIndexMetadataFile(tableSpace, p);
                    if (sequenceNumber.after(logPositionInFile)) {
                        LOGGER.log(Level.FINEST, "indexes metadata file " + p.toAbsolutePath() + ". will be deleted after checkpoint end");
                        result.add(new DeleteFileAction("indexes", "delete indexesmetadata file " + p.toAbsolutePath(), p));
                    }
                } catch (DataStorageManagerException ignore) {
                    LOGGER.log(Level.SEVERE, "Unparsable indexesmetadata file " + p.toAbsolutePath(), ignore);
                    result.add(new DeleteFileAction("indexes", "delete unparsable indexesmetadata file " + p.toAbsolutePath(), p));
                }
            } else if (isTablespaceTablesMetadataFile(p)) {
                try {
                    LogSequenceNumber logPositionInFile = readLogSequenceNumberFromTablesMetadataFile(tableSpace, p);
                    if (sequenceNumber.after(logPositionInFile)) {
                        LOGGER.log(Level.FINEST, "tables metadata file " + p.toAbsolutePath() + ". will be deleted after checkpoint end");
                        result.add(new DeleteFileAction("tables", "delete tablesmetadata file " + p.toAbsolutePath(), p));
                    }
                } catch (DataStorageManagerException ignore) {
                    LOGGER.log(Level.SEVERE, "Unparsable tablesmetadata file " + p.toAbsolutePath(), ignore);
                    result.add(new DeleteFileAction("transactions", "delete unparsable tablesmetadata file " + p.toAbsolutePath(), p));
                }
            }
        }
    } catch (IOException err) {
        LOGGER.log(Level.SEVERE, "Could not list dir " + tableSpaceDirectory, err);
    }
    return result;
}
Also used : Path(java.nio.file.Path) DataStorageManagerException(herddb.storage.DataStorageManagerException) Table(herddb.model.Table) ArrayList(java.util.ArrayList) LogSequenceNumber(herddb.log.LogSequenceNumber) Index(herddb.model.Index) BLinkKeyToPageIndex(herddb.index.blink.BLinkKeyToPageIndex) KeyToPageIndex(herddb.index.KeyToPageIndex) IOException(java.io.IOException) ManagedFile(herddb.utils.ManagedFile) ExtendedDataOutputStream(herddb.utils.ExtendedDataOutputStream) PostCheckpointAction(herddb.core.PostCheckpointAction) SimpleBufferedOutputStream(herddb.utils.SimpleBufferedOutputStream)

Example 70 with DataStorageManagerException

use of herddb.storage.DataStorageManagerException in project herddb by diennea.

the class FileDataStorageManager method getIndexPageFiles.

public List<Path> getIndexPageFiles(String tableSpace, String indexName) throws DataStorageManagerException {
    Path indexDir = getIndexDirectory(tableSpace, indexName);
    try {
        Files.createDirectories(indexDir);
    } catch (IOException err) {
        throw new DataStorageManagerException(err);
    }
    try (DirectoryStream<Path> files = Files.newDirectoryStream(indexDir, new DirectoryStream.Filter<Path>() {

        @Override
        public boolean accept(Path entry) throws IOException {
            return isPageFile(entry);
        }
    })) {
        List<Path> result = new ArrayList<>();
        files.forEach(result::add);
        return result;
    } catch (IOException err) {
        throw new DataStorageManagerException(err);
    }
}
Also used : Path(java.nio.file.Path) DataStorageManagerException(herddb.storage.DataStorageManagerException) DirectoryStream(java.nio.file.DirectoryStream) ArrayList(java.util.ArrayList) IOException(java.io.IOException)

Aggregations

DataStorageManagerException (herddb.storage.DataStorageManagerException)80 IOException (java.io.IOException)46 ArrayList (java.util.ArrayList)28 LogSequenceNumber (herddb.log.LogSequenceNumber)23 Path (java.nio.file.Path)22 Bytes (herddb.utils.Bytes)19 Record (herddb.model.Record)15 StatementExecutionException (herddb.model.StatementExecutionException)15 ExtendedDataInputStream (herddb.utils.ExtendedDataInputStream)14 SimpleByteArrayInputStream (herddb.utils.SimpleByteArrayInputStream)14 InputStream (java.io.InputStream)14 AtomicLong (java.util.concurrent.atomic.AtomicLong)14 LogEntry (herddb.log.LogEntry)13 HashMap (java.util.HashMap)13 LogNotAvailableException (herddb.log.LogNotAvailableException)12 ConcurrentHashMap (java.util.concurrent.ConcurrentHashMap)12 BufferedInputStream (java.io.BufferedInputStream)11 CommitLogResult (herddb.log.CommitLogResult)10 Table (herddb.model.Table)10 ExtendedDataOutputStream (herddb.utils.ExtendedDataOutputStream)10