Search in sources :

Example 6 with Version

use of org.apache.hadoop.yarn.server.records.Version in project hadoop by apache.

the class LeveldbTimelineStateStore method checkVersion.

/**
   * 1) Versioning timeline state store:
   * major.minor. For e.g. 1.0, 1.1, 1.2...1.25, 2.0 etc.
   * 2) Any incompatible change of TS-store is a major upgrade, and any
   * compatible change of TS-store is a minor upgrade.
   * 3) Within a minor upgrade, say 1.1 to 1.2:
   * overwrite the version info and proceed as normal.
   * 4) Within a major upgrade, say 1.2 to 2.0:
   * throw exception and indicate user to use a separate upgrade tool to
   * upgrade timeline store or remove incompatible old state.
   */
private void checkVersion() throws IOException {
    Version loadedVersion = loadVersion();
    LOG.info("Loaded timeline state store version info " + loadedVersion);
    if (loadedVersion.equals(getCurrentVersion())) {
        return;
    }
    if (loadedVersion.isCompatibleTo(getCurrentVersion())) {
        LOG.info("Storing timeline state store version info " + getCurrentVersion());
        storeVersion(CURRENT_VERSION_INFO);
    } else {
        String incompatibleMessage = "Incompatible version for timeline state store: expecting version " + getCurrentVersion() + ", but loading version " + loadedVersion;
        LOG.fatal(incompatibleMessage);
        throw new IOException(incompatibleMessage);
    }
}
Also used : Version(org.apache.hadoop.yarn.server.records.Version) IOException(java.io.IOException)

Example 7 with Version

use of org.apache.hadoop.yarn.server.records.Version in project hadoop by apache.

the class RollingLevelDBTimelineStore method checkVersion.

/**
   * 1) Versioning timeline store: major.minor. For e.g. 1.0, 1.1, 1.2...1.25,
   * 2.0 etc. 2) Any incompatible change of TS-store is a major upgrade, and any
   * compatible change of TS-store is a minor upgrade. 3) Within a minor
   * upgrade, say 1.1 to 1.2: overwrite the version info and proceed as normal.
   * 4) Within a major upgrade, say 1.2 to 2.0: throw exception and indicate
   * user to use a separate upgrade tool to upgrade timeline store or remove
   * incompatible old state.
   */
private void checkVersion() throws IOException {
    Version loadedVersion = loadVersion();
    LOG.info("Loaded timeline store version info " + loadedVersion);
    if (loadedVersion.equals(getCurrentVersion())) {
        return;
    }
    if (loadedVersion.isCompatibleTo(getCurrentVersion())) {
        LOG.info("Storing timeline store version info " + getCurrentVersion());
        dbStoreVersion(CURRENT_VERSION_INFO);
    } else {
        String incompatibleMessage = "Incompatible version for timeline store: " + "expecting version " + getCurrentVersion() + ", but loading version " + loadedVersion;
        LOG.fatal(incompatibleMessage);
        throw new IOException(incompatibleMessage);
    }
}
Also used : Version(org.apache.hadoop.yarn.server.records.Version) IOException(java.io.IOException)

Example 8 with Version

use of org.apache.hadoop.yarn.server.records.Version in project hadoop by apache.

the class LeveldbTimelineStore method loadVersion.

Version loadVersion() throws IOException {
    try {
        byte[] data = db.get(bytes(TIMELINE_STORE_VERSION_KEY));
        // if version is not stored previously, treat it as CURRENT_VERSION_INFO.
        if (data == null || data.length == 0) {
            return getCurrentVersion();
        }
        Version version = new VersionPBImpl(VersionProto.parseFrom(data));
        return version;
    } catch (DBException e) {
        throw new IOException(e);
    }
}
Also used : VersionPBImpl(org.apache.hadoop.yarn.server.records.impl.pb.VersionPBImpl) Version(org.apache.hadoop.yarn.server.records.Version) IOException(java.io.IOException)

Example 9 with Version

use of org.apache.hadoop.yarn.server.records.Version in project hadoop by apache.

the class LeveldbTimelineStore method checkVersion.

/**
   * 1) Versioning timeline store: major.minor. For e.g. 1.0, 1.1, 1.2...1.25, 2.0 etc.
   * 2) Any incompatible change of TS-store is a major upgrade, and any
   *    compatible change of TS-store is a minor upgrade.
   * 3) Within a minor upgrade, say 1.1 to 1.2:
   *    overwrite the version info and proceed as normal.
   * 4) Within a major upgrade, say 1.2 to 2.0:
   *    throw exception and indicate user to use a separate upgrade tool to
   *    upgrade timeline store or remove incompatible old state.
   */
private void checkVersion() throws IOException {
    Version loadedVersion = loadVersion();
    LOG.info("Loaded timeline store version info " + loadedVersion);
    if (loadedVersion.equals(getCurrentVersion())) {
        return;
    }
    if (loadedVersion.isCompatibleTo(getCurrentVersion())) {
        LOG.info("Storing timeline store version info " + getCurrentVersion());
        dbStoreVersion(CURRENT_VERSION_INFO);
    } else {
        String incompatibleMessage = "Incompatible version for timeline store: expecting version " + getCurrentVersion() + ", but loading version " + loadedVersion;
        LOG.fatal(incompatibleMessage);
        throw new IOException(incompatibleMessage);
    }
}
Also used : Version(org.apache.hadoop.yarn.server.records.Version) IOException(java.io.IOException)

Example 10 with Version

use of org.apache.hadoop.yarn.server.records.Version in project hadoop by apache.

the class RMStateStoreTestBase method testCheckVersion.

public void testCheckVersion(RMStateStoreHelper stateStoreHelper) throws Exception {
    RMStateStore store = stateStoreHelper.getRMStateStore();
    store.setRMDispatcher(new TestDispatcher());
    // default version
    Version defaultVersion = stateStoreHelper.getCurrentVersion();
    store.checkVersion();
    Assert.assertEquals(defaultVersion, store.loadVersion());
    // compatible version
    Version compatibleVersion = Version.newInstance(defaultVersion.getMajorVersion(), defaultVersion.getMinorVersion() + 2);
    stateStoreHelper.writeVersion(compatibleVersion);
    Assert.assertEquals(compatibleVersion, store.loadVersion());
    store.checkVersion();
    // overwrite the compatible version
    Assert.assertEquals(defaultVersion, store.loadVersion());
    // incompatible version
    Version incompatibleVersion = Version.newInstance(defaultVersion.getMajorVersion() + 2, defaultVersion.getMinorVersion());
    stateStoreHelper.writeVersion(incompatibleVersion);
    try {
        store.checkVersion();
        Assert.fail("Invalid version, should fail.");
    } catch (Throwable t) {
        Assert.assertTrue(t instanceof RMStateVersionIncompatibleException);
    }
}
Also used : Version(org.apache.hadoop.yarn.server.records.Version)

Aggregations

Version (org.apache.hadoop.yarn.server.records.Version)23 IOException (java.io.IOException)10 Test (org.junit.Test)8 VersionPBImpl (org.apache.hadoop.yarn.server.records.impl.pb.VersionPBImpl)7 ServiceStateException (org.apache.hadoop.service.ServiceStateException)6 YarnConfiguration (org.apache.hadoop.yarn.conf.YarnConfiguration)3 VisibleForTesting (com.google.common.annotations.VisibleForTesting)2 Path (org.apache.hadoop.fs.Path)2 File (java.io.File)1 Configuration (org.apache.hadoop.conf.Configuration)1 FileStatus (org.apache.hadoop.fs.FileStatus)1 HdfsConfiguration (org.apache.hadoop.hdfs.HdfsConfiguration)1 MiniDFSCluster (org.apache.hadoop.hdfs.MiniDFSCluster)1 DataOutputBuffer (org.apache.hadoop.io.DataOutputBuffer)1 Text (org.apache.hadoop.io.Text)1 JobTokenIdentifier (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier)1 AccessControlException (org.apache.hadoop.security.AccessControlException)1 Token (org.apache.hadoop.security.token.Token)1 ApplicationId (org.apache.hadoop.yarn.api.records.ApplicationId)1 ApplicationInitializationContext (org.apache.hadoop.yarn.server.api.ApplicationInitializationContext)1