Search in sources :

Example 26 with CallMetadata

use of com.netflix.titus.api.model.callmetadata.CallMetadata in project titus-control-plane by Netflix.

the class JobManagementResource method findJobs.

@GET
@ApiOperation("Find jobs")
@Path("/jobs")
public JobQueryResult findJobs(@Context UriInfo info) {
    MultivaluedMap<String, String> queryParameters = info.getQueryParameters(true);
    JobQuery.Builder queryBuilder = JobQuery.newBuilder();
    Page page = RestUtil.createPage(queryParameters);
    CallMetadata callMetadata = resolveCallMetadata();
    logPageNumberUsage(systemLog, callMetadata, getClass().getSimpleName(), "findJobs", page);
    queryBuilder.setPage(page);
    queryBuilder.putAllFilteringCriteria(RestUtil.getFilteringCriteria(queryParameters));
    queryBuilder.addAllFields(RestUtil.getFieldsParameter(queryParameters));
    return Responses.fromSingleValueObservable(jobServiceGateway.findJobs(queryBuilder.build(), callMetadata));
}
Also used : CallMetadata(com.netflix.titus.api.model.callmetadata.CallMetadata) JobQuery(com.netflix.titus.grpc.protogen.JobQuery) Page(com.netflix.titus.grpc.protogen.Page) Path(javax.ws.rs.Path) GET(javax.ws.rs.GET) ApiOperation(io.swagger.annotations.ApiOperation)

Example 27 with CallMetadata

use of com.netflix.titus.api.model.callmetadata.CallMetadata in project titus-control-plane by Netflix.

the class GatewayJobServiceGateway method findTasks.

@Override
public Observable<TaskQueryResult> findTasks(TaskQuery taskQuery, CallMetadata callMetadata) {
    Map<String, String> filteringCriteriaMap = taskQuery.getFilteringCriteriaMap();
    Set<String> v3JobIds = new HashSet<>(StringExt.splitByComma(filteringCriteriaMap.getOrDefault("jobIds", "")));
    boolean needsMigrationFilter = "true".equalsIgnoreCase(filteringCriteriaMap.getOrDefault("needsMigration", "false"));
    // "needsMigration" query is served from the local job and relocation cache.
    if (needsMigrationFilter) {
        PageResult<Task> pageResult = needsMigrationQueryHandler.findTasks(GrpcJobQueryModelConverters.toJobQueryCriteria(taskQuery), toPage(taskQuery.getPage()));
        TaskQueryResult taskQueryResult = TaskQueryResult.newBuilder().setPagination(toGrpcPagination(pageResult.getPagination())).addAllItems(taskDataInjector.injectIntoTasks(pageResult.getItems())).build();
        return Observable.just(taskQueryResult);
    }
    Set<String> taskStates = Sets.newHashSet(StringExt.splitByComma(taskQuery.getFilteringCriteriaMap().getOrDefault("taskStates", "")));
    Observable<TaskQueryResult> observable;
    if (v3JobIds.isEmpty()) {
        // Active task set only
        observable = newActiveTaskQueryAction(taskQuery, callMetadata);
    } else {
        if (!taskStates.contains(TaskState.Finished.name())) {
            // Active task set only
            observable = newActiveTaskQueryAction(taskQuery, callMetadata);
        } else {
            Page page = taskQuery.getPage();
            boolean nextPageByNumber = StringExt.isEmpty(page.getCursor()) && page.getPageNumber() > 0;
            if (nextPageByNumber) {
                // In this case we ask for active and archived tasks using a page number > 0. Because of that
                // we have to fetch as much tasks from master as we can. Tasks that we do not fetch, will not be
                // visible to the client.
                TaskQuery largePageQuery = taskQuery.toBuilder().setPage(taskQuery.getPage().toBuilder().setPageNumber(0).setPageSize(gatewayConfiguration.getMaxTaskPageSize())).build();
                observable = newActiveTaskQueryAction(largePageQuery, callMetadata);
            } else {
                observable = newActiveTaskQueryAction(taskQuery, callMetadata);
            }
            observable = observable.flatMap(result -> retrieveArchivedTasksForJobs(v3JobIds, taskQuery).map(archivedTasks -> combineTaskResults(taskQuery, result, archivedTasks)));
        }
    }
    return observable.timeout(tunablesConfiguration.getRequestTimeoutMs(), TimeUnit.MILLISECONDS).map(queryResult -> taskDataInjector.injectIntoTaskQueryResult(queryResult));
}
Also used : JobStoreException(com.netflix.titus.api.jobmanager.store.JobStoreException) ObserveJobsQuery(com.netflix.titus.grpc.protogen.ObserveJobsQuery) TaskId(com.netflix.titus.grpc.protogen.TaskId) LoggerFactory(org.slf4j.LoggerFactory) StringExt(com.netflix.titus.common.util.StringExt) JobQueryResult(com.netflix.titus.grpc.protogen.JobQueryResult) GrpcUtil(com.netflix.titus.runtime.endpoint.common.grpc.GrpcUtil) StreamObserver(io.grpc.stub.StreamObserver) TitusServiceException(com.netflix.titus.api.service.TitusServiceException) GrpcJobQueryModelConverters(com.netflix.titus.runtime.endpoint.v3.grpc.GrpcJobQueryModelConverters) Map(java.util.Map) SanitizingJobServiceGateway(com.netflix.titus.runtime.jobmanager.gateway.SanitizingJobServiceGateway) Status(io.grpc.Status) JobStore(com.netflix.titus.api.jobmanager.store.JobStore) PaginationUtil(com.netflix.titus.api.model.PaginationUtil) CallMetadata(com.netflix.titus.api.model.callmetadata.CallMetadata) JobDescriptor(com.netflix.titus.grpc.protogen.JobDescriptor) Predicate(java.util.function.Predicate) TaskQueryResult(com.netflix.titus.grpc.protogen.TaskQueryResult) SECURITY_GROUPS_REQUIRED_FEATURE(com.netflix.titus.api.FeatureRolloutPlans.SECURITY_GROUPS_REQUIRED_FEATURE) Task(com.netflix.titus.grpc.protogen.Task) Set(java.util.Set) GrpcUtil.createRequestObservable(com.netflix.titus.runtime.endpoint.common.grpc.GrpcUtil.createRequestObservable) JobFunctions(com.netflix.titus.api.jobmanager.model.job.JobFunctions) Collectors(java.util.stream.Collectors) TaskState(com.netflix.titus.api.jobmanager.model.job.TaskState) Sets(com.google.common.collect.Sets) Page(com.netflix.titus.grpc.protogen.Page) GrpcJobQueryModelConverters.toPage(com.netflix.titus.runtime.endpoint.v3.grpc.GrpcJobQueryModelConverters.toPage) PageResult(com.netflix.titus.api.model.PageResult) List(java.util.List) GrpcJobQueryModelConverters.toGrpcPagination(com.netflix.titus.runtime.endpoint.v3.grpc.GrpcJobQueryModelConverters.toGrpcPagination) GrpcUtil.createSimpleClientResponseObserver(com.netflix.titus.runtime.endpoint.common.grpc.GrpcUtil.createSimpleClientResponseObserver) JobManagementServiceStub(com.netflix.titus.grpc.protogen.JobManagementServiceGrpc.JobManagementServiceStub) Clock(com.netflix.titus.common.util.time.Clock) JobChangeNotification(com.netflix.titus.grpc.protogen.JobChangeNotification) JobId(com.netflix.titus.grpc.protogen.JobId) EntitySanitizer(com.netflix.titus.common.model.sanitizer.EntitySanitizer) Job(com.netflix.titus.grpc.protogen.Job) JobManagerConfiguration(com.netflix.titus.runtime.jobmanager.JobManagerConfiguration) Singleton(javax.inject.Singleton) Function(java.util.function.Function) Observable(rx.Observable) HashSet(java.util.HashSet) Inject(javax.inject.Inject) JobQuery(com.netflix.titus.grpc.protogen.JobQuery) Pair(com.netflix.titus.common.util.tuple.Pair) TaskStatus(com.netflix.titus.grpc.protogen.TaskStatus) AdmissionSanitizer(com.netflix.titus.common.model.admission.AdmissionSanitizer) ExceptionExt(com.netflix.titus.common.util.ExceptionExt) JobServiceGateway(com.netflix.titus.runtime.jobmanager.gateway.JobServiceGateway) AdmissionValidator(com.netflix.titus.common.model.admission.AdmissionValidator) Named(javax.inject.Named) TaskQuery(com.netflix.titus.grpc.protogen.TaskQuery) JobServiceGatewayDelegate(com.netflix.titus.runtime.jobmanager.gateway.JobServiceGatewayDelegate) Logger(org.slf4j.Logger) JobAssertions(com.netflix.titus.api.jobmanager.model.job.sanitizer.JobAssertions) LogStorageInfo(com.netflix.titus.api.jobmanager.model.job.LogStorageInfo) GrpcJobManagementModelConverters(com.netflix.titus.runtime.endpoint.v3.grpc.GrpcJobManagementModelConverters) GrpcUtil.createWrappedStub(com.netflix.titus.runtime.endpoint.common.grpc.GrpcUtil.createWrappedStub) JOB_STRICT_SANITIZER(com.netflix.titus.api.jobmanager.model.job.sanitizer.JobSanitizerBuilder.JOB_STRICT_SANITIZER) TimeUnit(java.util.concurrent.TimeUnit) StatusRuntimeException(io.grpc.StatusRuntimeException) JobQueryCriteria(com.netflix.titus.runtime.endpoint.JobQueryCriteria) JobManagerCursors(com.netflix.titus.runtime.jobmanager.JobManagerCursors) GrpcJobQueryModelConverters.toJobQueryCriteria(com.netflix.titus.runtime.endpoint.v3.grpc.GrpcJobQueryModelConverters.toJobQueryCriteria) GrpcRequestConfiguration(com.netflix.titus.runtime.connector.GrpcRequestConfiguration) GrpcJobServiceGateway(com.netflix.titus.runtime.jobmanager.gateway.GrpcJobServiceGateway) VisibleForTesting(com.google.common.annotations.VisibleForTesting) Pagination(com.netflix.titus.api.model.Pagination) TitusRuntime(com.netflix.titus.common.runtime.TitusRuntime) Collections(java.util.Collections) ENVIRONMENT_VARIABLE_NAMES_STRICT_VALIDATION_FEATURE(com.netflix.titus.api.FeatureRolloutPlans.ENVIRONMENT_VARIABLE_NAMES_STRICT_VALIDATION_FEATURE) Task(com.netflix.titus.grpc.protogen.Task) TaskQuery(com.netflix.titus.grpc.protogen.TaskQuery) Page(com.netflix.titus.grpc.protogen.Page) GrpcJobQueryModelConverters.toPage(com.netflix.titus.runtime.endpoint.v3.grpc.GrpcJobQueryModelConverters.toPage) TaskQueryResult(com.netflix.titus.grpc.protogen.TaskQueryResult) HashSet(java.util.HashSet)

Example 28 with CallMetadata

use of com.netflix.titus.api.model.callmetadata.CallMetadata in project titus-control-plane by Netflix.

the class AggregatingAutoScalingService method updateAutoScalingPolicy.

@Override
public Completable updateAutoScalingPolicy(UpdatePolicyRequest request, CallMetadata callMetadata) {
    ScalingPolicyID policyId = request.getPolicyId();
    Observable<Empty> observable = getScalingPolicyInAllCells(policyId, callMetadata).flatMap(response -> singleCellCall(response.getCell(), (client, responseObserver) -> client.updateAutoScalingPolicy(request, responseObserver), callMetadata));
    return observable.toCompletable();
}
Also used : Completable(rx.Completable) UpdatePolicyRequest(com.netflix.titus.grpc.protogen.UpdatePolicyRequest) Cell(com.netflix.titus.api.federation.model.Cell) DeletePolicyRequest(com.netflix.titus.grpc.protogen.DeletePolicyRequest) GrpcUtil.createWrappedStub(com.netflix.titus.runtime.endpoint.common.grpc.GrpcUtil.createWrappedStub) Singleton(javax.inject.Singleton) AutoScalingServiceStub(com.netflix.titus.grpc.protogen.AutoScalingServiceGrpc.AutoScalingServiceStub) PutPolicyRequest(com.netflix.titus.grpc.protogen.PutPolicyRequest) Observable(rx.Observable) Inject(javax.inject.Inject) Empty(com.google.protobuf.Empty) GetPolicyResult(com.netflix.titus.grpc.protogen.GetPolicyResult) CellConnectorUtil.callToCell(com.netflix.titus.federation.service.CellConnectorUtil.callToCell) StreamObserver(io.grpc.stub.StreamObserver) GrpcConfiguration(com.netflix.titus.federation.startup.GrpcConfiguration) ScalingPolicyID(com.netflix.titus.grpc.protogen.ScalingPolicyID) AbstractStub(io.grpc.stub.AbstractStub) AutoScalingServiceGrpc(com.netflix.titus.grpc.protogen.AutoScalingServiceGrpc) AutoScalingService(com.netflix.titus.runtime.service.AutoScalingService) BiConsumer(java.util.function.BiConsumer) CallMetadata(com.netflix.titus.api.model.callmetadata.CallMetadata) JobId(com.netflix.titus.grpc.protogen.JobId) Empty(com.google.protobuf.Empty) ScalingPolicyID(com.netflix.titus.grpc.protogen.ScalingPolicyID)

Example 29 with CallMetadata

use of com.netflix.titus.api.model.callmetadata.CallMetadata in project titus-control-plane by Netflix.

the class AggregatingAutoScalingService method deleteAutoScalingPolicy.

@Override
public Completable deleteAutoScalingPolicy(DeletePolicyRequest request, CallMetadata callMetadata) {
    ScalingPolicyID policyId = request.getId();
    Observable<Empty> observable = getScalingPolicyInAllCells(policyId, callMetadata).flatMap(response -> singleCellCall(response.getCell(), (client, responseObserver) -> client.deleteAutoScalingPolicy(request, responseObserver), callMetadata));
    return observable.toCompletable();
}
Also used : Completable(rx.Completable) UpdatePolicyRequest(com.netflix.titus.grpc.protogen.UpdatePolicyRequest) Cell(com.netflix.titus.api.federation.model.Cell) DeletePolicyRequest(com.netflix.titus.grpc.protogen.DeletePolicyRequest) GrpcUtil.createWrappedStub(com.netflix.titus.runtime.endpoint.common.grpc.GrpcUtil.createWrappedStub) Singleton(javax.inject.Singleton) AutoScalingServiceStub(com.netflix.titus.grpc.protogen.AutoScalingServiceGrpc.AutoScalingServiceStub) PutPolicyRequest(com.netflix.titus.grpc.protogen.PutPolicyRequest) Observable(rx.Observable) Inject(javax.inject.Inject) Empty(com.google.protobuf.Empty) GetPolicyResult(com.netflix.titus.grpc.protogen.GetPolicyResult) CellConnectorUtil.callToCell(com.netflix.titus.federation.service.CellConnectorUtil.callToCell) StreamObserver(io.grpc.stub.StreamObserver) GrpcConfiguration(com.netflix.titus.federation.startup.GrpcConfiguration) ScalingPolicyID(com.netflix.titus.grpc.protogen.ScalingPolicyID) AbstractStub(io.grpc.stub.AbstractStub) AutoScalingServiceGrpc(com.netflix.titus.grpc.protogen.AutoScalingServiceGrpc) AutoScalingService(com.netflix.titus.runtime.service.AutoScalingService) BiConsumer(java.util.function.BiConsumer) CallMetadata(com.netflix.titus.api.model.callmetadata.CallMetadata) JobId(com.netflix.titus.grpc.protogen.JobId) Empty(com.google.protobuf.Empty) ScalingPolicyID(com.netflix.titus.grpc.protogen.ScalingPolicyID)

Example 30 with CallMetadata

use of com.netflix.titus.api.model.callmetadata.CallMetadata in project titus-control-plane by Netflix.

the class SpringCallMetadataInterceptorTest method expectDebug.

private void expectDebug(boolean expectedDebug) throws Exception {
    interceptor.preHandle(request, response, new Object());
    CallMetadata callMetadata = expectCallMetadataAuthentication();
    assertThat(callMetadata.isDebug()).isEqualTo(expectedDebug);
}
Also used : CallMetadata(com.netflix.titus.api.model.callmetadata.CallMetadata)

Aggregations

CallMetadata (com.netflix.titus.api.model.callmetadata.CallMetadata)35 List (java.util.List)16 ArrayList (java.util.ArrayList)14 Task (com.netflix.titus.api.jobmanager.model.job.Task)13 TitusRuntime (com.netflix.titus.common.runtime.TitusRuntime)12 Completable (rx.Completable)12 TaskState (com.netflix.titus.api.jobmanager.model.job.TaskState)11 Collections (java.util.Collections)11 Collectors (java.util.stream.Collectors)11 JobManagerException (com.netflix.titus.api.jobmanager.service.JobManagerException)10 Test (org.junit.Test)10 TaskStatus (com.netflix.titus.api.jobmanager.model.job.TaskStatus)9 V3JobOperations (com.netflix.titus.api.jobmanager.service.V3JobOperations)9 CollectionsExt (com.netflix.titus.common.util.CollectionsExt)9 Pair (com.netflix.titus.common.util.tuple.Pair)9 Set (java.util.Set)9 TimeUnit (java.util.concurrent.TimeUnit)9 TaskAttributes (com.netflix.titus.api.jobmanager.TaskAttributes)7 ServiceJobTask (com.netflix.titus.api.jobmanager.model.job.ServiceJobTask)7 JobManagerEvent (com.netflix.titus.api.jobmanager.model.job.event.JobManagerEvent)7