Search in sources :

Example 1 with HierarchicalPrioritizer

use of org.apache.gobblin.util.request_allocation.HierarchicalPrioritizer in project incubator-gobblin by apache.

the class CompactionSource method initRequestAllocator.

private void initRequestAllocator(State state) {
    try {
        ResourceEstimator estimator = GobblinConstructorUtils.<ResourceEstimator>invokeLongestConstructor(new ClassAliasResolver(ResourceEstimator.class).resolveClass(state.getProp(ConfigurationKeys.COMPACTION_ESTIMATOR, SimpleDatasetRequest.SimpleDatasetCountEstimator.class.getName())));
        RequestAllocatorConfig.Builder<SimpleDatasetRequest> configBuilder = RequestAllocatorConfig.builder(estimator).allowParallelization(1).withLimitedScopeConfig(ConfigBuilder.create().loadProps(state.getProperties(), ConfigurationKeys.COMPACTION_PRIORITIZATION_PREFIX).build());
        if (!state.contains(ConfigurationKeys.COMPACTION_PRIORITIZER_ALIAS)) {
            allocator = new GreedyAllocator<>(configBuilder.build());
            return;
        }
        Comparator<SimpleDatasetRequest> prioritizer = GobblinConstructorUtils.<Comparator>invokeLongestConstructor(new ClassAliasResolver(Comparator.class).resolveClass(state.getProp(ConfigurationKeys.COMPACTION_PRIORITIZER_ALIAS)), state);
        configBuilder.withPrioritizer(prioritizer);
        if (prioritizer instanceof HierarchicalPrioritizer) {
            allocator = new HierarchicalAllocator.Factory().createRequestAllocator(configBuilder.build());
        } else {
            allocator = RequestAllocatorUtils.inferFromConfig(configBuilder.build());
        }
    } catch (RuntimeException e) {
        throw e;
    } catch (Exception e) {
        throw new RuntimeException("Cannot initialize allocator", e);
    }
}
Also used : RequestAllocatorConfig(org.apache.gobblin.util.request_allocation.RequestAllocatorConfig) MRCompactionTaskFactory(org.apache.gobblin.compaction.mapreduce.MRCompactionTaskFactory) ResourceEstimator(org.apache.gobblin.util.request_allocation.ResourceEstimator) NoSuchElementException(java.util.NoSuchElementException) IOException(java.io.IOException) ExecutionException(java.util.concurrent.ExecutionException) SimpleDatasetRequest(org.apache.gobblin.data.management.dataset.SimpleDatasetRequest) Comparator(java.util.Comparator) ClassAliasResolver(org.apache.gobblin.util.ClassAliasResolver) HierarchicalPrioritizer(org.apache.gobblin.util.request_allocation.HierarchicalPrioritizer)

Aggregations

IOException (java.io.IOException)1 Comparator (java.util.Comparator)1 NoSuchElementException (java.util.NoSuchElementException)1 ExecutionException (java.util.concurrent.ExecutionException)1 MRCompactionTaskFactory (org.apache.gobblin.compaction.mapreduce.MRCompactionTaskFactory)1 SimpleDatasetRequest (org.apache.gobblin.data.management.dataset.SimpleDatasetRequest)1 ClassAliasResolver (org.apache.gobblin.util.ClassAliasResolver)1 HierarchicalPrioritizer (org.apache.gobblin.util.request_allocation.HierarchicalPrioritizer)1 RequestAllocatorConfig (org.apache.gobblin.util.request_allocation.RequestAllocatorConfig)1 ResourceEstimator (org.apache.gobblin.util.request_allocation.ResourceEstimator)1