Search in sources :

Example 26 with LeveldbIterator

use of org.apache.hadoop.yarn.server.utils.LeveldbIterator in project hadoop by apache.

the class ShuffleHandler method recoverState.

private void recoverState(Configuration conf) throws IOException {
    Path recoveryRoot = getRecoveryPath();
    if (recoveryRoot != null) {
        startStore(recoveryRoot);
        Pattern jobPattern = Pattern.compile(JobID.JOBID_REGEX);
        LeveldbIterator iter = null;
        try {
            iter = new LeveldbIterator(stateDb);
            iter.seek(bytes(JobID.JOB));
            while (iter.hasNext()) {
                Map.Entry<byte[], byte[]> entry = iter.next();
                String key = asString(entry.getKey());
                if (!jobPattern.matcher(key).matches()) {
                    break;
                }
                recoverJobShuffleInfo(key, entry.getValue());
            }
        } catch (DBException e) {
            throw new IOException("Database error during recovery", e);
        } finally {
            if (iter != null) {
                iter.close();
            }
        }
    }
}
Also used : Path(org.apache.hadoop.fs.Path) Pattern(java.util.regex.Pattern) DBException(org.iq80.leveldb.DBException) LeveldbIterator(org.apache.hadoop.yarn.server.utils.LeveldbIterator) JniDBFactory.asString(org.fusesource.leveldbjni.JniDBFactory.asString) ByteString(com.google.protobuf.ByteString) IOException(java.io.IOException) Map(java.util.Map) ConcurrentHashMap(java.util.concurrent.ConcurrentHashMap) HashMap(java.util.HashMap)

Aggregations

LeveldbIterator (org.apache.hadoop.yarn.server.utils.LeveldbIterator)26 IOException (java.io.IOException)25 DBException (org.iq80.leveldb.DBException)18 JniDBFactory.asString (org.fusesource.leveldbjni.JniDBFactory.asString)15 VisibleForTesting (com.google.common.annotations.VisibleForTesting)3 KeyParser (org.apache.hadoop.yarn.server.timeline.util.LeveldbUtils.KeyParser)3 ContainerId (org.apache.hadoop.yarn.api.records.ContainerId)2 GenericObjectMapper.readReverseOrderedLong (org.apache.hadoop.yarn.server.timeline.GenericObjectMapper.readReverseOrderedLong)2 GenericObjectMapper.writeReverseOrderedLong (org.apache.hadoop.yarn.server.timeline.GenericObjectMapper.writeReverseOrderedLong)2 KeyBuilder (org.apache.hadoop.yarn.server.timeline.util.LeveldbUtils.KeyBuilder)2 ByteString (com.google.protobuf.ByteString)1 ArrayList (java.util.ArrayList)1 HashMap (java.util.HashMap)1 Map (java.util.Map)1 ConcurrentHashMap (java.util.concurrent.ConcurrentHashMap)1 Pattern (java.util.regex.Pattern)1 Path (org.apache.hadoop.fs.Path)1 DelegationKey (org.apache.hadoop.security.token.delegation.DelegationKey)1 ApplicationAttemptId (org.apache.hadoop.yarn.api.records.ApplicationAttemptId)1 ApplicationId (org.apache.hadoop.yarn.api.records.ApplicationId)1