Search in sources :

Example 11 with WriteBatch

use of org.iq80.leveldb.WriteBatch in project EventHub by Codecademy.

the class MigrateIdMapToUseLevelDB method main.

public static void main(String[] args) throws Exception {
    String userStorageDirectory = args[0];
    String filename = userStorageDirectory + "/id_map.ser";
    File file = new File(filename);
    try (ObjectInputStream ois = new ObjectInputStream(new FileInputStream(file))) {
        //noinspection unchecked
        Map<String, Integer> idMap = (Map<String, Integer>) ois.readObject();
        int currentId = ois.readInt();
        Options options = new Options();
        options.createIfMissing(true);
        try (DB idMapDb = JniDBFactory.factory.open(new File(userStorageDirectory + "/id_map.db"), options)) {
            try (WriteBatch batch = idMapDb.createWriteBatch()) {
                for (Map.Entry<String, Integer> entry : idMap.entrySet()) {
                    batch.put(bytes(entry.getKey()), bytes("" + entry.getValue()));
                }
                batch.put(bytes("__eventtracker__id"), bytes("" + currentId));
                idMapDb.write(batch);
            }
        }
    }
}
Also used : Options(org.iq80.leveldb.Options) File(java.io.File) Map(java.util.Map) WriteBatch(org.iq80.leveldb.WriteBatch) FileInputStream(java.io.FileInputStream) DB(org.iq80.leveldb.DB) ObjectInputStream(java.io.ObjectInputStream)

Example 12 with WriteBatch

use of org.iq80.leveldb.WriteBatch in project cdap by caskdata.

the class LevelDBMessageTable method rollback.

@Override
protected void rollback(byte[] startKey, byte[] stopKey, byte[] txWritePtr) throws IOException {
    WriteBatch writeBatch = levelDB.createWriteBatch();
    try (CloseableIterator<Map.Entry<byte[], byte[]>> rowIterator = new DBScanIterator(levelDB, startKey, stopKey)) {
        while (rowIterator.hasNext()) {
            Map.Entry<byte[], byte[]> rowValue = rowIterator.next();
            byte[] value = rowValue.getValue();
            Map<String, byte[]> columns = decodeValue(value);
            writeBatch.put(rowValue.getKey(), encodeValue(txWritePtr, columns.get(PAYLOAD_COL)));
        }
    }
    try {
        levelDB.write(writeBatch, WRITE_OPTIONS);
    } catch (DBException ex) {
        throw new IOException(ex);
    }
}
Also used : DBException(org.iq80.leveldb.DBException) ImmutableMessageTableEntry(co.cask.cdap.messaging.store.ImmutableMessageTableEntry) RawMessageTableEntry(co.cask.cdap.messaging.store.RawMessageTableEntry) IOException(java.io.IOException) WriteBatch(org.iq80.leveldb.WriteBatch) HashMap(java.util.HashMap) Map(java.util.Map)

Example 13 with WriteBatch

use of org.iq80.leveldb.WriteBatch in project cdap by caskdata.

the class LevelDBMessageTable method persist.

@Override
protected void persist(Iterator<RawMessageTableEntry> entries) throws IOException {
    try (WriteBatch writeBatch = levelDB.createWriteBatch()) {
        while (entries.hasNext()) {
            RawMessageTableEntry entry = entries.next();
            byte[] rowKey = entry.getKey();
            // LevelDB doesn't make copies, and since we reuse RawMessageTableEntry object, we need to create copies.
            writeBatch.put(Arrays.copyOf(rowKey, rowKey.length), encodeValue(entry.getTxPtr(), entry.getPayload()));
        }
        levelDB.write(writeBatch, WRITE_OPTIONS);
    } catch (DBException ex) {
        throw new IOException(ex);
    }
}
Also used : DBException(org.iq80.leveldb.DBException) RawMessageTableEntry(co.cask.cdap.messaging.store.RawMessageTableEntry) IOException(java.io.IOException) WriteBatch(org.iq80.leveldb.WriteBatch)

Example 14 with WriteBatch

use of org.iq80.leveldb.WriteBatch in project cdap by caskdata.

the class LevelDBPayloadTable method persist.

@Override
public void persist(Iterator<RawPayloadTableEntry> entries) throws IOException {
    try (WriteBatch writeBatch = levelDB.createWriteBatch()) {
        while (entries.hasNext()) {
            RawPayloadTableEntry entry = entries.next();
            byte[] key = entry.getKey();
            byte[] value = entry.getValue();
            // LevelDB doesn't make copies, and since we reuse RawPayloadTableEntry object, we need to create copies.
            writeBatch.put(Arrays.copyOf(key, key.length), Arrays.copyOf(value, value.length));
        }
        levelDB.write(writeBatch, WRITE_OPTIONS);
    } catch (DBException ex) {
        throw new IOException(ex);
    }
}
Also used : DBException(org.iq80.leveldb.DBException) RawPayloadTableEntry(co.cask.cdap.messaging.store.RawPayloadTableEntry) IOException(java.io.IOException) WriteBatch(org.iq80.leveldb.WriteBatch)

Example 15 with WriteBatch

use of org.iq80.leveldb.WriteBatch in project cdap by caskdata.

the class LevelDBTableCore method deleteColumn.

public void deleteColumn(byte[] row, byte[] column) throws IOException {
    DB db = getDB();
    WriteBatch batch = db.createWriteBatch();
    try (DBIterator iterator = db.iterator()) {
        addToDeleteBatch(batch, iterator, row, column);
        db.write(batch);
    }
}
Also used : DBIterator(org.iq80.leveldb.DBIterator) WriteBatch(org.iq80.leveldb.WriteBatch) DB(org.iq80.leveldb.DB)

Aggregations

WriteBatch (org.iq80.leveldb.WriteBatch)30 IOException (java.io.IOException)21 DBException (org.iq80.leveldb.DBException)17 JniDBFactory.asString (org.fusesource.leveldbjni.JniDBFactory.asString)12 Map (java.util.Map)9 DB (org.iq80.leveldb.DB)8 DBIterator (org.iq80.leveldb.DBIterator)5 ImmutableMap (com.google.common.collect.ImmutableMap)4 ImmutableSortedMap (com.google.common.collect.ImmutableSortedMap)4 NavigableMap (java.util.NavigableMap)4 RawMessageTableEntry (co.cask.cdap.messaging.store.RawMessageTableEntry)3 RollingWriteBatch (org.apache.hadoop.yarn.server.timeline.RollingLevelDB.RollingWriteBatch)3 ImmutableMessageTableEntry (co.cask.cdap.messaging.store.ImmutableMessageTableEntry)2 RawPayloadTableEntry (co.cask.cdap.messaging.store.RawPayloadTableEntry)2 ByteArrayOutputStream (java.io.ByteArrayOutputStream)2 DataOutputStream (java.io.DataOutputStream)2 HashMap (java.util.HashMap)2 Row (co.cask.cdap.api.dataset.table.Row)1 Scanner (co.cask.cdap.api.dataset.table.Scanner)1 AbstractMessageTable (co.cask.cdap.messaging.store.AbstractMessageTable)1