Search in sources :

Example 1 with QuotaManager

use of com.github.ambry.quota.QuotaManager in project ambry by linkedin.

the class FrontendRestRequestService method securityPostProcessRequestCallback.

/**
 * Build a callback to use for {@link SecurityService#postProcessRequest}. This callback forwards request to the
 * {@link Router} once ID conversion is completed. In the case of some sub-resources
 * (e.g., {@link SubResource#Replicas}), the request is completed and not forwarded to the {@link Router}.
 * @param convertedId the converted blob ID to use in router requests.
 * @param restRequest the {@link RestRequest}.
 * @param restResponseChannel the {@link RestResponseChannel}.
 * @param getCallback the {@link GetCallback} to use if this is a {@link RestMethod#GET} request, or null for other
 *                    request types.
 * @param headCallback the {@link HeadCallback} to use if this is a {@link RestMethod#HEAD} request, or null for other
 *                    request types.
 * @param deleteCallback the {@link DeleteCallback} to use if this is a {@link RestMethod#DELETE} request, or null for
 *                       other request types.
 * @return the {@link Callback} to use.
 */
private Callback<Void> securityPostProcessRequestCallback(String convertedId, RestRequest restRequest, RestResponseChannel restResponseChannel, GetCallback getCallback, HeadCallback headCallback, DeleteCallback deleteCallback) {
    Callback<ReadableStreamChannel> completionCallback = (result, exception) -> submitResponse(restRequest, restResponseChannel, result, exception);
    RestMethod restMethod = restRequest.getRestMethod();
    AsyncOperationTracker.Metrics metrics;
    switch(restMethod) {
        case GET:
            metrics = frontendMetrics.getSecurityPostProcessRequestMetrics;
            break;
        case HEAD:
            metrics = frontendMetrics.headSecurityPostProcessRequestMetrics;
            break;
        case DELETE:
            metrics = frontendMetrics.deleteSecurityPostProcessRequestMetrics;
            break;
        default:
            throw new IllegalStateException("Unrecognized RestMethod: " + restMethod);
    }
    return FrontendUtils.buildCallback(metrics, result -> {
        ReadableStreamChannel response = null;
        switch(restMethod) {
            case GET:
                SubResource subResource = getRequestPath(restRequest).getSubResource();
                // inject encryption metrics if need be
                if (BlobId.isEncrypted(convertedId)) {
                    RestRequestMetrics restRequestMetrics = getMetricsGroupForGet(frontendMetrics, subResource).getRestRequestMetrics(restRequest.isSslUsed(), true);
                    restRequest.getMetricsTracker().injectMetrics(restRequestMetrics);
                }
                if (subResource == null) {
                    getCallback.markStartTime();
                    router.getBlob(convertedId, getCallback.options, getCallback, QuotaUtils.buildQuotaChargeCallback(restRequest, quotaManager, true));
                } else {
                    switch(subResource) {
                        case BlobInfo:
                        case UserMetadata:
                        case Segment:
                            getCallback.markStartTime();
                            router.getBlob(convertedId, getCallback.options, getCallback, QuotaUtils.buildQuotaChargeCallback(restRequest, quotaManager, true));
                            break;
                        case Replicas:
                            response = getReplicasHandler.getReplicas(convertedId, restResponseChannel);
                            break;
                    }
                }
                break;
            case HEAD:
                GetOption getOption = getGetOption(restRequest, frontendConfig.defaultRouterGetOption);
                // inject encryption metrics if need be
                if (BlobId.isEncrypted(convertedId)) {
                    RestRequestMetrics requestMetrics = frontendMetrics.headBlobMetricsGroup.getRestRequestMetrics(restRequest.isSslUsed(), true);
                    restRequest.getMetricsTracker().injectMetrics(requestMetrics);
                }
                headCallback.markStartTime();
                router.getBlob(convertedId, new GetBlobOptionsBuilder().operationType(GetBlobOptions.OperationType.BlobInfo).getOption(getOption).restRequest(restRequest).build(), headCallback, QuotaUtils.buildQuotaChargeCallback(restRequest, quotaManager, false));
                break;
            case DELETE:
                deleteCallback.markStartTime();
                router.deleteBlob(convertedId, getHeader(restRequest.getArgs(), Headers.SERVICE_ID, false), deleteCallback, QuotaUtils.buildQuotaChargeCallback(restRequest, quotaManager, false));
                break;
            default:
                throw new IllegalStateException("Unrecognized RestMethod: " + restMethod);
        }
        if (response != null) {
            completionCallback.onCompletion(response, null);
        }
    }, restRequest.getUri(), logger, completionCallback);
}
Also used : Histogram(com.codahale.metrics.Histogram) GetOption(com.github.ambry.protocol.GetOption) FrontendConfig(com.github.ambry.config.FrontendConfig) ResponseStatus(com.github.ambry.rest.ResponseStatus) LoggerFactory(org.slf4j.LoggerFactory) ByteBufferReadableStreamChannel(com.github.ambry.commons.ByteBufferReadableStreamChannel) AccountService(com.github.ambry.account.AccountService) QuotaManager(com.github.ambry.quota.QuotaManager) ByteBuffer(java.nio.ByteBuffer) ThrowingConsumer(com.github.ambry.utils.ThrowingConsumer) RequestPath(com.github.ambry.rest.RequestPath) NamedBlobDb(com.github.ambry.named.NamedBlobDb) RestRequestService(com.github.ambry.rest.RestRequestService) RestRequestMetrics(com.github.ambry.rest.RestRequestMetrics) SystemTime(com.github.ambry.utils.SystemTime) Router(com.github.ambry.router.Router) RouterErrorCode(com.github.ambry.router.RouterErrorCode) RestResponseHandler(com.github.ambry.rest.RestResponseHandler) ReadableStreamChannel(com.github.ambry.router.ReadableStreamChannel) Logger(org.slf4j.Logger) GregorianCalendar(java.util.GregorianCalendar) RestMethod(com.github.ambry.rest.RestMethod) RestResponseChannel(com.github.ambry.rest.RestResponseChannel) RestServiceErrorCode(com.github.ambry.rest.RestServiceErrorCode) InternalKeys(com.github.ambry.rest.RestUtils.InternalKeys) ClusterMap(com.github.ambry.clustermap.ClusterMap) Utils(com.github.ambry.utils.Utils) IOException(java.io.IOException) GetBlobOptions(com.github.ambry.router.GetBlobOptions) RouterException(com.github.ambry.router.RouterException) BlobInfo(com.github.ambry.messageformat.BlobInfo) QuotaUtils(com.github.ambry.quota.QuotaUtils) AccountStatsStore(com.github.ambry.accountstats.AccountStatsStore) RestServiceException(com.github.ambry.rest.RestServiceException) GetBlobResult(com.github.ambry.router.GetBlobResult) Callback(com.github.ambry.commons.Callback) RestUtils(com.github.ambry.rest.RestUtils) AsyncOperationTracker(com.github.ambry.utils.AsyncOperationTracker) RestRequest(com.github.ambry.rest.RestRequest) GetBlobOptionsBuilder(com.github.ambry.router.GetBlobOptionsBuilder) BlobId(com.github.ambry.commons.BlobId) GetBlobOptionsBuilder(com.github.ambry.router.GetBlobOptionsBuilder) ByteBufferReadableStreamChannel(com.github.ambry.commons.ByteBufferReadableStreamChannel) ReadableStreamChannel(com.github.ambry.router.ReadableStreamChannel) RestRequestMetrics(com.github.ambry.rest.RestRequestMetrics) GetOption(com.github.ambry.protocol.GetOption) AsyncOperationTracker(com.github.ambry.utils.AsyncOperationTracker) RestMethod(com.github.ambry.rest.RestMethod)

Example 2 with QuotaManager

use of com.github.ambry.quota.QuotaManager in project ambry by linkedin.

the class FrontendRestRequestServiceFactory method getRestRequestService.

/**
 * Returns a new instance of {@link FrontendRestRequestService}.
 * @return a new instance of {@link FrontendRestRequestService}.
 */
@Override
public RestRequestService getRestRequestService() {
    try {
        IdSigningService idSigningService = Utils.<IdSigningServiceFactory>getObj(frontendConfig.idSigningServiceFactory, verifiableProperties, clusterMap.getMetricRegistry()).getIdSigningService();
        NamedBlobDb namedBlobDb = Utils.isNullOrEmpty(frontendConfig.namedBlobDbFactory) ? null : Utils.<NamedBlobDbFactory>getObj(frontendConfig.namedBlobDbFactory, verifiableProperties, clusterMap.getMetricRegistry(), accountService).getNamedBlobDb();
        IdConverterFactory idConverterFactory = Utils.getObj(frontendConfig.idConverterFactory, verifiableProperties, clusterMap.getMetricRegistry(), idSigningService, namedBlobDb);
        UrlSigningService urlSigningService = Utils.<UrlSigningServiceFactory>getObj(frontendConfig.urlSigningServiceFactory, verifiableProperties, clusterMap.getMetricRegistry()).getUrlSigningService();
        AccountAndContainerInjector accountAndContainerInjector = new AccountAndContainerInjector(accountService, frontendMetrics, frontendConfig);
        AccountStatsStore accountStatsStore = Utils.<AccountStatsStoreFactory>getObj(frontendConfig.accountStatsStoreFactory, verifiableProperties, clusterMapConfig, clusterMap.getMetricRegistry()).getAccountStatsStore();
        QuotaConfig quotaConfig = new QuotaConfig(verifiableProperties);
        QuotaManager quotaManager = ((QuotaManagerFactory) Utils.getObj(quotaConfig.quotaManagerFactory, quotaConfig, new SimpleQuotaRecommendationMergePolicy(quotaConfig), accountService, accountStatsStore, clusterMap.getMetricRegistry())).getQuotaManager();
        SecurityServiceFactory securityServiceFactory = Utils.getObj(frontendConfig.securityServiceFactory, verifiableProperties, clusterMap, accountService, urlSigningService, idSigningService, accountAndContainerInjector, quotaManager);
        return new FrontendRestRequestService(frontendConfig, frontendMetrics, router, clusterMap, idConverterFactory, securityServiceFactory, urlSigningService, idSigningService, namedBlobDb, accountService, accountAndContainerInjector, clusterMapConfig.clusterMapDatacenterName, clusterMapConfig.clusterMapHostName, clusterMapConfig.clusterMapClusterName, accountStatsStore, quotaManager);
    } catch (Exception e) {
        throw new IllegalStateException("Could not instantiate FrontendRestRequestService", e);
    }
}
Also used : NamedBlobDb(com.github.ambry.named.NamedBlobDb) AccountStatsStore(com.github.ambry.accountstats.AccountStatsStore) QuotaManagerFactory(com.github.ambry.quota.QuotaManagerFactory) NamedBlobDbFactory(com.github.ambry.named.NamedBlobDbFactory) QuotaConfig(com.github.ambry.config.QuotaConfig) SimpleQuotaRecommendationMergePolicy(com.github.ambry.quota.SimpleQuotaRecommendationMergePolicy) QuotaManager(com.github.ambry.quota.QuotaManager)

Example 3 with QuotaManager

use of com.github.ambry.quota.QuotaManager in project ambry by linkedin.

the class NonBlockingRouterQuotaCallbackTest method testRouterWithDefaultQuotaCallback.

/**
 * Test default {@link QuotaChargeCallback} doesn't charge anything and doesn't error out when throttling is disabled.
 */
@Test
public void testRouterWithDefaultQuotaCallback() throws Exception {
    try {
        setRouter();
        assertExpectedThreadCounts(2, 1);
        AtomicInteger listenerCalledCount = new AtomicInteger(0);
        QuotaConfig quotaConfig = new QuotaConfig(new VerifiableProperties(new Properties()));
        QuotaManager quotaManager = new ChargeTesterQuotaManager(quotaConfig, new SimpleQuotaRecommendationMergePolicy(quotaConfig), accountService, null, new QuotaMetrics(new MetricRegistry()), listenerCalledCount);
        QuotaChargeCallback quotaChargeCallback = QuotaUtils.buildQuotaChargeCallback(null, quotaManager, true);
        int blobSize = 3000;
        setOperationParams(blobSize, TTL_SECS);
        String compositeBlobId = router.putBlob(putBlobProperties, putUserMetadata, putChannel, PutBlobOptions.DEFAULT, null, quotaChargeCallback).get();
        assertEquals(0, listenerCalledCount.get());
        RetainingAsyncWritableChannel retainingAsyncWritableChannel = new RetainingAsyncWritableChannel();
        router.getBlob(compositeBlobId, new GetBlobOptionsBuilder().build(), null, quotaChargeCallback).get().getBlobDataChannel().readInto(retainingAsyncWritableChannel, null).get();
        // read out all the chunks.
        retainingAsyncWritableChannel.consumeContentAsInputStream().close();
        assertEquals(0, listenerCalledCount.get());
    } finally {
        router.close();
        assertExpectedThreadCounts(0, 0);
        // submission after closing should return a future that is already done.
        assertClosed();
    }
}
Also used : VerifiableProperties(com.github.ambry.config.VerifiableProperties) RetainingAsyncWritableChannel(com.github.ambry.commons.RetainingAsyncWritableChannel) MetricRegistry(com.codahale.metrics.MetricRegistry) Properties(java.util.Properties) VerifiableProperties(com.github.ambry.config.VerifiableProperties) QuotaMetrics(com.github.ambry.quota.QuotaMetrics) AtomicInteger(java.util.concurrent.atomic.AtomicInteger) QuotaConfig(com.github.ambry.config.QuotaConfig) SimpleQuotaRecommendationMergePolicy(com.github.ambry.quota.SimpleQuotaRecommendationMergePolicy) QuotaChargeCallback(com.github.ambry.quota.QuotaChargeCallback) QuotaManager(com.github.ambry.quota.QuotaManager) AmbryQuotaManager(com.github.ambry.quota.AmbryQuotaManager) Test(org.junit.Test)

Aggregations

QuotaManager (com.github.ambry.quota.QuotaManager)3 AccountStatsStore (com.github.ambry.accountstats.AccountStatsStore)2 QuotaConfig (com.github.ambry.config.QuotaConfig)2 NamedBlobDb (com.github.ambry.named.NamedBlobDb)2 SimpleQuotaRecommendationMergePolicy (com.github.ambry.quota.SimpleQuotaRecommendationMergePolicy)2 Histogram (com.codahale.metrics.Histogram)1 MetricRegistry (com.codahale.metrics.MetricRegistry)1 AccountService (com.github.ambry.account.AccountService)1 ClusterMap (com.github.ambry.clustermap.ClusterMap)1 BlobId (com.github.ambry.commons.BlobId)1 ByteBufferReadableStreamChannel (com.github.ambry.commons.ByteBufferReadableStreamChannel)1 Callback (com.github.ambry.commons.Callback)1 RetainingAsyncWritableChannel (com.github.ambry.commons.RetainingAsyncWritableChannel)1 FrontendConfig (com.github.ambry.config.FrontendConfig)1 VerifiableProperties (com.github.ambry.config.VerifiableProperties)1 BlobInfo (com.github.ambry.messageformat.BlobInfo)1 NamedBlobDbFactory (com.github.ambry.named.NamedBlobDbFactory)1 GetOption (com.github.ambry.protocol.GetOption)1 AmbryQuotaManager (com.github.ambry.quota.AmbryQuotaManager)1 QuotaChargeCallback (com.github.ambry.quota.QuotaChargeCallback)1