Search in sources :

Example 16 with Version

use of org.apache.hadoop.yarn.server.records.Version in project hadoop by apache.

the class NMLeveldbStateStoreService method checkVersion.

/**
   * 1) Versioning scheme: major.minor. For e.g. 1.0, 1.1, 1.2...1.25, 2.0 etc.
   * 2) Any incompatible change of state-store is a major upgrade, and any
   *    compatible change of state-store is a minor upgrade.
   * 3) Within a minor upgrade, say 1.1 to 1.2:
   *    overwrite the version info and proceed as normal.
   * 4) Within a major upgrade, say 1.2 to 2.0:
   *    throw exception and indicate user to use a separate upgrade tool to
   *    upgrade NM state or remove incompatible old state.
   */
protected void checkVersion() throws IOException {
    Version loadedVersion = loadVersion();
    LOG.info("Loaded NM state version info " + loadedVersion);
    if (loadedVersion.equals(getCurrentVersion())) {
        return;
    }
    if (loadedVersion.isCompatibleTo(getCurrentVersion())) {
        LOG.info("Storing NM state version info " + getCurrentVersion());
        storeVersion();
    } else {
        throw new IOException("Incompatible version for NM state: expecting NM state version " + getCurrentVersion() + ", but loading version " + loadedVersion);
    }
}
Also used : Version(org.apache.hadoop.yarn.server.records.Version) IOException(java.io.IOException)

Example 17 with Version

use of org.apache.hadoop.yarn.server.records.Version in project hadoop by apache.

the class NMLeveldbStateStoreService method loadVersion.

Version loadVersion() throws IOException {
    byte[] data = db.get(bytes(DB_SCHEMA_VERSION_KEY));
    // if version is not stored previously, treat it as CURRENT_VERSION_INFO.
    if (data == null || data.length == 0) {
        return getCurrentVersion();
    }
    Version version = new VersionPBImpl(VersionProto.parseFrom(data));
    return version;
}
Also used : VersionPBImpl(org.apache.hadoop.yarn.server.records.impl.pb.VersionPBImpl) Version(org.apache.hadoop.yarn.server.records.Version)

Example 18 with Version

use of org.apache.hadoop.yarn.server.records.Version in project hadoop by apache.

the class TestNMLeveldbStateStoreService method testCheckVersion.

@Test
public void testCheckVersion() throws IOException {
    // default version
    Version defaultVersion = stateStore.getCurrentVersion();
    Assert.assertEquals(defaultVersion, stateStore.loadVersion());
    // compatible version
    Version compatibleVersion = Version.newInstance(defaultVersion.getMajorVersion(), defaultVersion.getMinorVersion() + 2);
    stateStore.storeVersion(compatibleVersion);
    Assert.assertEquals(compatibleVersion, stateStore.loadVersion());
    restartStateStore();
    // overwrite the compatible version
    Assert.assertEquals(defaultVersion, stateStore.loadVersion());
    // incompatible version
    Version incompatibleVersion = Version.newInstance(defaultVersion.getMajorVersion() + 1, defaultVersion.getMinorVersion());
    stateStore.storeVersion(incompatibleVersion);
    try {
        restartStateStore();
        Assert.fail("Incompatible version, should expect fail here.");
    } catch (ServiceStateException e) {
        Assert.assertTrue("Exception message mismatch", e.getMessage().contains("Incompatible version for NM state:"));
    }
}
Also used : Version(org.apache.hadoop.yarn.server.records.Version) ServiceStateException(org.apache.hadoop.service.ServiceStateException) Test(org.junit.Test)

Example 19 with Version

use of org.apache.hadoop.yarn.server.records.Version in project hadoop by apache.

the class RMStateStore method checkVersion.

/**
   * 1) Versioning scheme: major.minor. For e.g. 1.0, 1.1, 1.2...1.25, 2.0 etc.
   * 2) Any incompatible change of state-store is a major upgrade, and any
   *    compatible change of state-store is a minor upgrade.
   * 3) If theres's no version, treat it as CURRENT_VERSION_INFO.
   * 4) Within a minor upgrade, say 1.1 to 1.2:
   *    overwrite the version info and proceed as normal.
   * 5) Within a major upgrade, say 1.2 to 2.0:
   *    throw exception and indicate user to use a separate upgrade tool to
   *    upgrade RM state.
   */
public void checkVersion() throws Exception {
    Version loadedVersion = loadVersion();
    LOG.info("Loaded RM state version info " + loadedVersion);
    if (loadedVersion != null && loadedVersion.equals(getCurrentVersion())) {
        return;
    }
    // if there is no version info, treat it as CURRENT_VERSION_INFO;
    if (loadedVersion == null) {
        loadedVersion = getCurrentVersion();
    }
    if (loadedVersion.isCompatibleTo(getCurrentVersion())) {
        LOG.info("Storing RM state version info " + getCurrentVersion());
        storeVersion();
    } else {
        throw new RMStateVersionIncompatibleException("Expecting RM state version " + getCurrentVersion() + ", but loading version " + loadedVersion);
    }
}
Also used : Version(org.apache.hadoop.yarn.server.records.Version)

Example 20 with Version

use of org.apache.hadoop.yarn.server.records.Version in project hadoop by apache.

the class HistoryServerLeveldbStateStoreService method loadVersion.

Version loadVersion() throws IOException {
    byte[] data = db.get(bytes(DB_SCHEMA_VERSION_KEY));
    // if version is not stored previously, treat it as 1.0.
    if (data == null || data.length == 0) {
        return Version.newInstance(1, 0);
    }
    Version version = new VersionPBImpl(VersionProto.parseFrom(data));
    return version;
}
Also used : VersionPBImpl(org.apache.hadoop.yarn.server.records.impl.pb.VersionPBImpl) Version(org.apache.hadoop.yarn.server.records.Version)

Aggregations

Version (org.apache.hadoop.yarn.server.records.Version)23 IOException (java.io.IOException)10 Test (org.junit.Test)8 VersionPBImpl (org.apache.hadoop.yarn.server.records.impl.pb.VersionPBImpl)7 ServiceStateException (org.apache.hadoop.service.ServiceStateException)6 YarnConfiguration (org.apache.hadoop.yarn.conf.YarnConfiguration)3 VisibleForTesting (com.google.common.annotations.VisibleForTesting)2 Path (org.apache.hadoop.fs.Path)2 File (java.io.File)1 Configuration (org.apache.hadoop.conf.Configuration)1 FileStatus (org.apache.hadoop.fs.FileStatus)1 HdfsConfiguration (org.apache.hadoop.hdfs.HdfsConfiguration)1 MiniDFSCluster (org.apache.hadoop.hdfs.MiniDFSCluster)1 DataOutputBuffer (org.apache.hadoop.io.DataOutputBuffer)1 Text (org.apache.hadoop.io.Text)1 JobTokenIdentifier (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier)1 AccessControlException (org.apache.hadoop.security.AccessControlException)1 Token (org.apache.hadoop.security.token.Token)1 ApplicationId (org.apache.hadoop.yarn.api.records.ApplicationId)1 ApplicationInitializationContext (org.apache.hadoop.yarn.server.api.ApplicationInitializationContext)1