Examples with ConfigProperty org.apache.hudi.common.config.ConfigProperty used on opensource projects

Example 1 with ConfigProperty

use of org.apache.hudi.common.config.ConfigProperty in project hudi by apache.

the class UpgradeDowngrade method run.

/**
 * Perform Upgrade or Downgrade steps if required and updated table version if need be.
 * <p>
 * Starting from version 0.6.0, this upgrade/downgrade step will be added in all write paths.
 * <p>
 * Essentially, if a dataset was created using an previous table version in an older release,
 * and Hoodie version was upgraded to a new release with new table version supported,
 * Hoodie table version gets bumped to the new version and there are some upgrade steps need
 * to be executed before doing any writes.
 * <p>
 * Similarly, if a dataset was created using an newer table version in an newer release,
 * and then hoodie was downgraded to an older release or to older Hoodie table version,
 * then some downgrade steps need to be executed before proceeding w/ any writes.
 * <p>
 * Below shows the table version corresponding to the Hudi release:
 * Hudi release -> table version
 * pre 0.6.0 -> v0
 * 0.6.0 to 0.8.0 -> v1
 * 0.9.0 -> v2
 * 0.10.0 to current -> v3
 * <p>
 * On a high level, these are the steps performed
 * <p>
 * Step1 : Understand current hoodie table version and table version from hoodie.properties file
 * Step2 : Delete any left over .updated from previous upgrade/downgrade
 * Step3 : If version are different, perform upgrade/downgrade.
 * Step4 : Copy hoodie.properties -> hoodie.properties.updated with the version updated
 * Step6 : Rename hoodie.properties.updated to hoodie.properties
 * </p>
 *
 * @param toVersion   version to which upgrade or downgrade has to be done.
 * @param instantTime current instant time that should not be touched.
 */
public void run(HoodieTableVersion toVersion, String instantTime) {
    // Fetch version from property file and current version
    HoodieTableVersion fromVersion = metaClient.getTableConfig().getTableVersion();
    if (!needsUpgradeOrDowngrade(toVersion)) {
        return;
    }
    // Perform the actual upgrade/downgrade; this has to be idempotent, for now.
    LOG.info("Attempting to move table from version " + fromVersion + " to " + toVersion);
    Map<ConfigProperty, String> tableProps = new Hashtable<>();
    if (fromVersion.versionCode() < toVersion.versionCode()) {
        // upgrade
        while (fromVersion.versionCode() < toVersion.versionCode()) {
            HoodieTableVersion nextVersion = HoodieTableVersion.versionFromCode(fromVersion.versionCode() + 1);
            tableProps.putAll(upgrade(fromVersion, nextVersion, instantTime));
            fromVersion = nextVersion;
        }
    } else {
        // downgrade
        while (fromVersion.versionCode() > toVersion.versionCode()) {
            HoodieTableVersion prevVersion = HoodieTableVersion.versionFromCode(fromVersion.versionCode() - 1);
            tableProps.putAll(downgrade(fromVersion, prevVersion, instantTime));
            fromVersion = prevVersion;
        }
    }
    // Write out the current version in hoodie.properties.updated file
    for (Map.Entry<ConfigProperty, String> entry : tableProps.entrySet()) {
        metaClient.getTableConfig().setValue(entry.getKey(), entry.getValue());
    }
    metaClient.getTableConfig().setTableVersion(toVersion);
    HoodieTableConfig.update(metaClient.getFs(), new Path(metaClient.getMetaPath()), metaClient.getTableConfig().getProps());
}