Search in sources :

Example 36 with ConfigValue

use of com.typesafe.config.ConfigValue in project incubator-gobblin by apache.

the class StateStores method getStateStoreConfig.

private static Config getStateStoreConfig(Config config, String rootDir, String dbTableKey) {
    Config fallbackConfig = ConfigFactory.empty().withFallback(config).withValue(ConfigurationKeys.STATE_STORE_ROOT_DIR_KEY, ConfigValueFactory.fromAnyRef(rootDir)).withValue(ConfigurationKeys.STATE_STORE_DB_TABLE_KEY, ConfigValueFactory.fromAnyRef(dbTableKey));
    Config scopedConfig = ConfigFactory.empty();
    for (Map.Entry<String, ConfigValue> entry : config.withOnlyPath(ConfigurationKeys.INTERMEDIATE_STATE_STORE_PREFIX).entrySet()) {
        scopedConfig.withValue(entry.getKey().substring(ConfigurationKeys.INTERMEDIATE_STATE_STORE_PREFIX.length()), entry.getValue());
    }
    return scopedConfig.withFallback(fallbackConfig);
}
Also used : ConfigValue(com.typesafe.config.ConfigValue) Config(com.typesafe.config.Config) Map(java.util.Map)

Example 37 with ConfigValue

use of com.typesafe.config.ConfigValue in project incubator-gobblin by apache.

the class HiveDataset method resolveConfig.

/**
 * Replace various tokens (DB, TABLE, LOGICAL_DB, LOGICAL_TABLE) with their values.
 *
 * @param datasetConfig       The config object that needs to be resolved with final values.
 * @param realDbAndTable      Real DB and Table .
 * @param logicalDbAndTable   Logical DB and Table.
 * @return Resolved config object.
 */
@VisibleForTesting
protected static Config resolveConfig(Config datasetConfig, DbAndTable realDbAndTable, DbAndTable logicalDbAndTable) {
    Preconditions.checkNotNull(datasetConfig, "Dataset config should not be null");
    Preconditions.checkNotNull(realDbAndTable, "Real DB and table should not be null");
    Preconditions.checkNotNull(logicalDbAndTable, "Logical DB and table should not be null");
    ImmutableMap.Builder<String, Object> immutableMapBuilder = ImmutableMap.builder();
    Config resolvedConfig = datasetConfig.resolve();
    for (Map.Entry<String, ConfigValue> entry : resolvedConfig.entrySet()) {
        if (ConfigValueType.LIST.equals(entry.getValue().valueType())) {
            List<String> rawValueList = resolvedConfig.getStringList(entry.getKey());
            List<String> resolvedValueList = Lists.newArrayList();
            for (String rawValue : rawValueList) {
                String resolvedValue = StringUtils.replaceEach(rawValue, new String[] { DATABASE_TOKEN, TABLE_TOKEN, LOGICAL_DB_TOKEN, LOGICAL_TABLE_TOKEN }, new String[] { realDbAndTable.getDb(), realDbAndTable.getTable(), logicalDbAndTable.getDb(), logicalDbAndTable.getTable() });
                resolvedValueList.add(resolvedValue);
            }
            StringBuilder listToStringWithQuotes = new StringBuilder();
            for (String resolvedValueStr : resolvedValueList) {
                if (listToStringWithQuotes.length() > 0) {
                    listToStringWithQuotes.append(",");
                }
                listToStringWithQuotes.append("\"").append(resolvedValueStr).append("\"");
            }
            immutableMapBuilder.put(entry.getKey(), listToStringWithQuotes.toString());
        } else {
            String resolvedValue = StringUtils.replaceEach(resolvedConfig.getString(entry.getKey()), new String[] { DATABASE_TOKEN, TABLE_TOKEN, LOGICAL_DB_TOKEN, LOGICAL_TABLE_TOKEN }, new String[] { realDbAndTable.getDb(), realDbAndTable.getTable(), logicalDbAndTable.getDb(), logicalDbAndTable.getTable() });
            immutableMapBuilder.put(entry.getKey(), resolvedValue);
        }
    }
    return ConfigFactory.parseMap(immutableMapBuilder.build());
}
Also used : ConfigValue(com.typesafe.config.ConfigValue) Config(com.typesafe.config.Config) AutoReturnableObject(org.apache.gobblin.util.AutoReturnableObject) ToString(lombok.ToString) Map(java.util.Map) ImmutableMap(com.google.common.collect.ImmutableMap) ImmutableMap(com.google.common.collect.ImmutableMap) VisibleForTesting(com.google.common.annotations.VisibleForTesting)

Aggregations

ConfigValue (com.typesafe.config.ConfigValue)37 Map (java.util.Map)20 Config (com.typesafe.config.Config)11 ConfigException (com.typesafe.config.ConfigException)10 ArrayList (java.util.ArrayList)8 HashMap (java.util.HashMap)6 ConfigList (com.typesafe.config.ConfigList)5 ConfigObject (com.typesafe.config.ConfigObject)5 ImmutableMap (com.google.common.collect.ImmutableMap)4 Configuration (org.apache.hadoop.conf.Configuration)4 IOException (java.io.IOException)3 BigInteger (java.math.BigInteger)3 FileSystem (org.apache.hadoop.fs.FileSystem)3 Properties (java.util.Properties)2 ParaObject (com.erudika.para.core.ParaObject)1 Sysprop (com.erudika.para.core.Sysprop)1 Pager (com.erudika.para.core.utils.Pager)1 VisibleForTesting (com.google.common.annotations.VisibleForTesting)1 Gson (com.google.gson.Gson)1 ConfigOrigin (com.typesafe.config.ConfigOrigin)1