Examples with NewEpochResponseProto org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos.NewEpochResponseProto used on opensource projects

Example 1 with NewEpochResponseProto

use of org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos.NewEpochResponseProto in project hadoop by apache.

the class TestJournalNode method testReturnsSegmentInfoAtEpochTransition.

@Test(timeout = 100000)
public void testReturnsSegmentInfoAtEpochTransition() throws Exception {
    ch.newEpoch(1).get();
    ch.setEpoch(1);
    ch.startLogSegment(1, NameNodeLayoutVersion.CURRENT_LAYOUT_VERSION).get();
    ch.sendEdits(1L, 1, 2, QJMTestUtil.createTxnData(1, 2)).get();
    // Switch to a new epoch without closing earlier segment
    NewEpochResponseProto response = ch.newEpoch(2).get();
    ch.setEpoch(2);
    assertEquals(1, response.getLastSegmentTxId());
    ch.finalizeLogSegment(1, 2).get();
    // Switch to a new epoch after just closing the earlier segment.
    response = ch.newEpoch(3).get();
    ch.setEpoch(3);
    assertEquals(1, response.getLastSegmentTxId());
    // Start a segment but don't write anything, check newEpoch segment info
    ch.startLogSegment(3, NameNodeLayoutVersion.CURRENT_LAYOUT_VERSION).get();
    response = ch.newEpoch(4).get();
    ch.setEpoch(4);
    // Because the new segment is empty, it is equivalent to not having
    // started writing it. Hence, we should return the prior segment txid.
    assertEquals(1, response.getLastSegmentTxId());
}

Example 2 with NewEpochResponseProto

use of org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos.NewEpochResponseProto in project hadoop by apache.

the class QuorumJournalManager method recoverUnfinalizedSegments.

@Override
public void recoverUnfinalizedSegments() throws IOException {
    Preconditions.checkState(!isActiveWriter, "already active writer");
    LOG.info("Starting recovery process for unclosed journal segments...");
    Map<AsyncLogger, NewEpochResponseProto> resps = createNewUniqueEpoch();
    LOG.info("Successfully started new epoch " + loggers.getEpoch());
    if (LOG.isDebugEnabled()) {
        LOG.debug("newEpoch(" + loggers.getEpoch() + ") responses:\n" + QuorumCall.mapToString(resps));
    }
    long mostRecentSegmentTxId = Long.MIN_VALUE;
    for (NewEpochResponseProto r : resps.values()) {
        if (r.hasLastSegmentTxId()) {
            mostRecentSegmentTxId = Math.max(mostRecentSegmentTxId, r.getLastSegmentTxId());
        }
    }
    // segments, so there's nothing to recover.
    if (mostRecentSegmentTxId != Long.MIN_VALUE) {
        recoverUnclosedSegment(mostRecentSegmentTxId);
    }
    isActiveWriter = true;
}

Example 3 with NewEpochResponseProto

use of org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos.NewEpochResponseProto in project hadoop by apache.

the class QuorumJournalManager method createNewUniqueEpoch.

/**
   * Fence any previous writers, and obtain a unique epoch number
   * for write-access to the journal nodes.
   *
   * @return the new, unique epoch number
   */
Map<AsyncLogger, NewEpochResponseProto> createNewUniqueEpoch() throws IOException {
    Preconditions.checkState(!loggers.isEpochEstablished(), "epoch already created");
    Map<AsyncLogger, GetJournalStateResponseProto> lastPromises = loggers.waitForWriteQuorum(loggers.getJournalState(), getJournalStateTimeoutMs, "getJournalState()");
    long maxPromised = Long.MIN_VALUE;
    for (GetJournalStateResponseProto resp : lastPromises.values()) {
        maxPromised = Math.max(maxPromised, resp.getLastPromisedEpoch());
    }
    assert maxPromised >= 0;
    long myEpoch = maxPromised + 1;
    Map<AsyncLogger, NewEpochResponseProto> resps = loggers.waitForWriteQuorum(loggers.newEpoch(nsInfo, myEpoch), newEpochTimeoutMs, "newEpoch(" + myEpoch + ")");
    loggers.setEpoch(myEpoch);
    return resps;
}

Example 4 with NewEpochResponseProto

use of org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos.NewEpochResponseProto in project hadoop by apache.

the class Journal method newEpoch.

/**
   * Try to create a new epoch for this journal.
   * @param nsInfo the namespace, which is verified for consistency or used to
   * format, if the Journal has not yet been written to.
   * @param epoch the epoch to start
   * @return the status information necessary to begin recovery
   * @throws IOException if the node has already made a promise to another
   * writer with a higher epoch number, if the namespace is inconsistent,
   * or if a disk error occurs.
   */
synchronized NewEpochResponseProto newEpoch(NamespaceInfo nsInfo, long epoch) throws IOException {
    checkFormatted();
    storage.checkConsistentNamespace(nsInfo);
    // any other that we've promised. 
    if (epoch <= getLastPromisedEpoch()) {
        throw new IOException("Proposed epoch " + epoch + " <= last promise " + getLastPromisedEpoch());
    }
    updateLastPromisedEpoch(epoch);
    abortCurSegment();
    NewEpochResponseProto.Builder builder = NewEpochResponseProto.newBuilder();
    EditLogFile latestFile = scanStorageForLatestEdits();
    if (latestFile != null) {
        builder.setLastSegmentTxId(latestFile.getFirstTxId());
    }
    return builder.build();
}

Example 5 with NewEpochResponseProto

use of org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos.NewEpochResponseProto in project hadoop by apache.

the class TestJournal method testNewEpochAtBeginningOfSegment.

/**
   * Test that, if the writer crashes at the very beginning of a segment,
   * before any transactions are written, that the next newEpoch() call
   * returns the prior segment txid as its most recent segment.
   */
@Test(timeout = 10000)
public void testNewEpochAtBeginningOfSegment() throws Exception {
    journal.newEpoch(FAKE_NSINFO, 1);
    journal.startLogSegment(makeRI(1), 1, NameNodeLayoutVersion.CURRENT_LAYOUT_VERSION);
    journal.journal(makeRI(2), 1, 1, 2, QJMTestUtil.createTxnData(1, 2));
    journal.finalizeLogSegment(makeRI(3), 1, 2);
    journal.startLogSegment(makeRI(4), 3, NameNodeLayoutVersion.CURRENT_LAYOUT_VERSION);
    NewEpochResponseProto resp = journal.newEpoch(FAKE_NSINFO, 2);
    assertEquals(1, resp.getLastSegmentTxId());
}