Examples with RateBasedLimiter org.apache.gobblin.util.limiter.RateBasedLimiter used on opensource projects

Example 1 with RateBasedLimiter

use of org.apache.gobblin.util.limiter.RateBasedLimiter in project incubator-gobblin by apache.

the class MRStressTest method createLimiter.

static Limiter createLimiter(Configuration configuration, SharedResourcesBroker<SimpleScopeType> broker) {
    try {
        Limiter limiter = new NoopLimiter();
        long localQps = configuration.getLong(LOCALLY_ENFORCED_QPS, 0);
        if (localQps > 0) {
            log.info("Setting up local qps " + localQps);
            limiter = new MultiLimiter(limiter, new RateBasedLimiter(localQps));
        }
        if (configuration.getBoolean(USE_THROTTLING_SERVER, false)) {
            log.info("Setting up remote throttling.");
            String resourceId = configuration.get(RESOURCE_ID);
            Limiter globalLimiter = broker.getSharedResource(new RestliLimiterFactory<SimpleScopeType>(), new SharedLimiterKey(resourceId));
            limiter = new MultiLimiter(limiter, globalLimiter);
        }
        return limiter;
    } catch (NotConfiguredException nce) {
        throw new RuntimeException(nce);
    }
}

Example 2 with RateBasedLimiter

use of org.apache.gobblin.util.limiter.RateBasedLimiter in project incubator-gobblin by apache.

the class RateControlledFileSystem method getRateLimiter.

protected Limiter getRateLimiter() {
    try {
        String key = getUri().toString();
        RateBasedLimiter limiter = FS_URI_TO_RATE_LIMITER_CACHE.get(key, this.callableLimiter);
        if (limiter.getRateLimitPerSecond() < this.limitPerSecond) {
            try {
                limiter = this.callableLimiter.call();
                FS_URI_TO_RATE_LIMITER_CACHE.put(key, limiter);
            } catch (Exception exc) {
                throw new ExecutionException(exc);
            }
        }
        return limiter;
    } catch (ExecutionException ee) {
        throw new RuntimeException(ee);
    }
}

Example 3 with RateBasedLimiter

use of org.apache.gobblin.util.limiter.RateBasedLimiter in project incubator-gobblin by apache.

the class RatedControlledFileSystemTest method setUp.

@BeforeClass
public void setUp() throws IOException, ExecutionException {
    Limiter limiter = new RateBasedLimiter(20);
    this.rateControlledFs = new TestRateControlledFileSystem(FileSystem.getLocal(new Configuration()), 20, limiter);
    this.rateControlledFs.startRateControl();
}

Example 4 with RateBasedLimiter

use of org.apache.gobblin.util.limiter.RateBasedLimiter in project incubator-gobblin by apache.

the class RateBasedLimiterTest method setUp.

@BeforeClass
public void setUp() {
    this.limiter = new RateBasedLimiter(20, TimeUnit.SECONDS);
    this.limiter.start();
}