Search in sources :

Example 6 with StringBinding

use of com.sleepycat.bind.tuple.StringBinding in project heritrix3 by internetarchive.

the class PersistProcessor method populatePersistEnv.

/**
 * Populates a new environment db from an old environment db or a persist
 * log. If path to new environment is not provided, only logs the entries
 * that would have been populated.
 *
 * @param sourcePath
 *            source of old entries: can be a path to an existing
 *            environment db, or a URL or path to a persist log
 * @param envFile
 *            path to new environment db (or null for a dry run)
 * @return number of records
 * @throws DatabaseException
 * @throws IOException
 */
public static int populatePersistEnv(String sourcePath, File envFile) throws IOException {
    int count = 0;
    StoredSortedMap<String, Map> historyMap = null;
    EnhancedEnvironment targetEnv = null;
    StoredClassCatalog classCatalog = null;
    Database historyDB = null;
    if (envFile != null) {
        // set up target environment
        FileUtils.ensureWriteableDirectory(envFile);
        targetEnv = setupCopyEnvironment(envFile);
        classCatalog = targetEnv.getClassCatalog();
        historyDB = targetEnv.openDatabase(null, URI_HISTORY_DBNAME, HISTORY_DB_CONFIG.toDatabaseConfig());
        historyMap = new StoredSortedMap<String, Map>(historyDB, new StringBinding(), new SerialBinding<Map>(classCatalog, Map.class), true);
    }
    try {
        count = copyPersistSourceToHistoryMap(new File(sourcePath), historyMap);
    } finally {
        // failed to populate it
        if (envFile != null) {
            logger.info(count + " records imported from " + sourcePath + " to BDB env " + envFile);
            historyDB.sync();
            historyDB.close();
            targetEnv.close();
        } else {
            logger.info(count + " records found in " + sourcePath);
        }
    }
    return count;
}
Also used : StringBinding(com.sleepycat.bind.tuple.StringBinding) StoredClassCatalog(com.sleepycat.bind.serial.StoredClassCatalog) Database(com.sleepycat.je.Database) SerialBinding(com.sleepycat.bind.serial.SerialBinding) Map(java.util.Map) StoredSortedMap(com.sleepycat.collections.StoredSortedMap) EnhancedEnvironment(org.archive.util.bdbje.EnhancedEnvironment) File(java.io.File)

Example 7 with StringBinding

use of com.sleepycat.bind.tuple.StringBinding in project heritrix3 by internetarchive.

the class PersistOnlineProcessor method start.

@SuppressWarnings("unchecked")
public void start() {
    if (isRunning()) {
        return;
    }
    StoredSortedMap<String, Map> historyMap;
    try {
        StoredClassCatalog classCatalog = bdb.getClassCatalog();
        BdbModule.BdbConfig dbConfig = HISTORY_DB_CONFIG;
        historyDb = bdb.openDatabase(getHistoryDbName(), dbConfig, true);
        historyMap = new StoredSortedMap<String, Map>(historyDb, new StringBinding(), new SerialBinding<Map>(classCatalog, Map.class), true);
    } catch (DatabaseException e) {
        throw new RuntimeException(e);
    }
    store = historyMap;
}
Also used : BdbModule(org.archive.bdb.BdbModule) StringBinding(com.sleepycat.bind.tuple.StringBinding) StoredClassCatalog(com.sleepycat.bind.serial.StoredClassCatalog) SerialBinding(com.sleepycat.bind.serial.SerialBinding) Map(java.util.Map) StoredSortedMap(com.sleepycat.collections.StoredSortedMap) DatabaseException(com.sleepycat.je.DatabaseException)

Aggregations

StringBinding (com.sleepycat.bind.tuple.StringBinding)7 StoredSortedMap (com.sleepycat.collections.StoredSortedMap)7 StoredClassCatalog (com.sleepycat.bind.serial.StoredClassCatalog)6 SerialBinding (com.sleepycat.bind.serial.SerialBinding)5 Map (java.util.Map)5 Database (com.sleepycat.je.Database)4 DatabaseException (com.sleepycat.je.DatabaseException)3 File (java.io.File)3 EnhancedEnvironment (org.archive.util.bdbje.EnhancedEnvironment)3 DatabaseConfig (com.sleepycat.je.DatabaseConfig)2 HashMap (java.util.HashMap)2 BdbModule (org.archive.bdb.BdbModule)2 Environment (com.sleepycat.je.Environment)1 EnvironmentConfig (com.sleepycat.je.EnvironmentConfig)1 BufferedReader (java.io.BufferedReader)1 Entry (java.util.Map.Entry)1 LineReadingIterator (org.archive.util.iterator.LineReadingIterator)1 JSONObject (org.json.JSONObject)1