Search in sources :

Example 6 with Key

use of org.hypertrace.core.documentstore.Key in project document-store by hypertrace.

the class MongoCollection method bulkUpdateImpl.

private BulkWriteResult bulkUpdateImpl(List<BulkUpdateRequest> bulkUpdateRequests) throws JsonProcessingException {
    List<UpdateOneModel<BasicDBObject>> bulkCollection = new ArrayList<>();
    for (BulkUpdateRequest bulkUpdateRequest : bulkUpdateRequests) {
        Key key = bulkUpdateRequest.getKey();
        Map<String, Object> conditionMap = bulkUpdateRequest.getFilter() == null ? new HashMap<>() : MongoQueryParser.parseFilter(bulkUpdateRequest.getFilter());
        conditionMap.put(ID_KEY, key.toString());
        BasicDBObject conditionObject = new BasicDBObject(conditionMap);
        // update if filter condition is satisfied
        bulkCollection.add(new UpdateOneModel<>(conditionObject, prepareUpsert(key, bulkUpdateRequest.getDocument()), new UpdateOptions().upsert(false)));
    }
    return Failsafe.with(bulkWriteRetryPolicy).get(() -> collection.bulkWrite(bulkCollection, new BulkWriteOptions().ordered(false)));
}
Also used : ArrayList(java.util.ArrayList) BulkUpdateRequest(org.hypertrace.core.documentstore.BulkUpdateRequest) UpdateOptions(com.mongodb.client.model.UpdateOptions) FindOneAndUpdateOptions(com.mongodb.client.model.FindOneAndUpdateOptions) BasicDBObject(com.mongodb.BasicDBObject) UpdateOneModel(com.mongodb.client.model.UpdateOneModel) BulkWriteOptions(com.mongodb.client.model.BulkWriteOptions) BasicDBObject(com.mongodb.BasicDBObject) Key(org.hypertrace.core.documentstore.Key)

Example 7 with Key

use of org.hypertrace.core.documentstore.Key in project entity-service by hypertrace.

the class EntityQueryServiceImpl method doUpdate.

private void doUpdate(RequestContext requestContext, EntityUpdateRequest request) throws Exception {
    if (request.getOperation().hasSetAttribute()) {
        SetAttribute setAttribute = request.getOperation().getSetAttribute();
        String attributeId = setAttribute.getAttribute().getColumnName();
        String subDocPath = entityAttributeMapping.getDocStorePathByAttributeId(requestContext, attributeId).orElseThrow(() -> new IllegalArgumentException("Unknown attribute FQN " + attributeId));
        JSONDocument jsonDocument = convertToJsonDocument(setAttribute.getValue());
        Map<Key, Map<String, Document>> entitiesUpdateMap = new HashMap<>();
        for (String entityId : request.getEntityIdsList()) {
            SingleValueKey key = new SingleValueKey(requestContext.getTenantId().orElseThrow(), entityId);
            if (entitiesUpdateMap.containsKey(key)) {
                entitiesUpdateMap.get(key).put(subDocPath, jsonDocument);
            } else {
                Map<String, Document> subDocument = new HashMap<>();
                subDocument.put(subDocPath, jsonDocument);
                entitiesUpdateMap.put(key, subDocument);
            }
        }
        try {
            entitiesCollection.bulkUpdateSubDocs(entitiesUpdateMap);
        } catch (Exception e) {
            LOG.error("Failed to update entities {}, subDocPath {}, with new doc {}.", entitiesUpdateMap, subDocPath, jsonDocument, e);
            throw e;
        }
    }
}
Also used : SingleValueKey(org.hypertrace.core.documentstore.SingleValueKey) HashMap(java.util.HashMap) JSONDocument(org.hypertrace.core.documentstore.JSONDocument) Document(org.hypertrace.core.documentstore.Document) JSONDocument(org.hypertrace.core.documentstore.JSONDocument) Map(java.util.Map) HashMap(java.util.HashMap) SetAttribute(org.hypertrace.entity.query.service.v1.SetAttribute) SingleValueKey(org.hypertrace.core.documentstore.SingleValueKey) Key(org.hypertrace.core.documentstore.Key) ServiceException(com.google.protobuf.ServiceException) ConversionException(org.hypertrace.entity.query.service.converter.ConversionException)

Example 8 with Key

use of org.hypertrace.core.documentstore.Key in project entity-service by hypertrace.

the class EntityDataServiceImpl method delete.

/**
 * Deletes an Entity by the EntityId and EntityType
 *
 * @param request ID of the entity to be deleted
 * @param responseObserver Observer to be notified on about the Entity delete request
 */
@Override
public void delete(ByIdRequest request, StreamObserver<Empty> responseObserver) {
    if (StringUtils.isEmpty(request.getEntityId())) {
        LOG.info("{}. Invalid delete request:{}", request, ErrorMessages.ENTITY_ID_EMPTY);
        responseObserver.onError(new RuntimeException(ErrorMessages.ENTITY_ID_EMPTY));
        return;
    }
    Optional<String> tenantId = RequestContext.CURRENT.get().getTenantId();
    if (tenantId.isEmpty()) {
        responseObserver.onError(new ServiceException("Tenant id is missing in the request."));
        return;
    }
    Optional<Entity> existingEntity = getExistingEntity(tenantId.get(), request.getEntityType(), request.getEntityId());
    Key key = this.entityNormalizer.getEntityDocKey(tenantId.get(), request.getEntityType(), request.getEntityId());
    if (entitiesCollection.delete(key)) {
        responseObserver.onNext(Empty.newBuilder().build());
        responseObserver.onCompleted();
        existingEntity.ifPresent(entity -> entityChangeEventGenerator.sendDeleteNotification(RequestContext.CURRENT.get(), List.of(entity)));
    } else {
        responseObserver.onError(new RuntimeException("Could not delete the entity."));
    }
}
Also used : EnrichedEntity(org.hypertrace.entity.data.service.v1.EnrichedEntity) Entity(org.hypertrace.entity.data.service.v1.Entity) ServiceException(com.google.protobuf.ServiceException) Key(org.hypertrace.core.documentstore.Key)

Example 9 with Key

use of org.hypertrace.core.documentstore.Key in project entity-service by hypertrace.

the class EntityDataServiceImpl method upsertEntities.

@Override
public void upsertEntities(Entities request, StreamObserver<Empty> responseObserver) {
    String tenantId = RequestContext.CURRENT.get().getTenantId().orElse(null);
    if (tenantId == null) {
        responseObserver.onError(new ServiceException("Tenant id is missing in the request."));
        return;
    }
    try {
        Map<Key, Entity> entities = request.getEntityList().stream().map(entity -> this.entityNormalizer.normalize(tenantId, entity)).collect(Collectors.toUnmodifiableMap(entity -> this.entityNormalizer.getEntityDocKey(tenantId, entity), Function.identity()));
        java.util.Collection<Entity> existingEntities = getExistingEntities(entities.values());
        upsertEntities(entities, entitiesCollection, responseObserver);
        entityChangeEventGenerator.sendChangeNotification(RequestContext.CURRENT.get(), existingEntities, entities.values());
    } catch (Throwable throwable) {
        LOG.warn("Failed to upsert: {}", request, throwable);
        responseObserver.onError(throwable);
    }
}
Also used : ByIdRequest(org.hypertrace.entity.data.service.v1.ByIdRequest) ServiceException(com.google.protobuf.ServiceException) LoggerFactory(org.slf4j.LoggerFactory) Channel(io.grpc.Channel) EntityTypeClient(org.hypertrace.entity.type.service.rxclient.EntityTypeClient) StreamObserver(io.grpc.stub.StreamObserver) Map(java.util.Map) Objects.isNull(java.util.Objects.isNull) ENRICHED_ENTITIES_COLLECTION(org.hypertrace.entity.service.constants.EntityCollectionConstants.ENRICHED_ENTITIES_COLLECTION) DocStoreConverter(org.hypertrace.entity.service.util.DocStoreConverter) InvalidRequestException(org.hypertrace.entity.service.exception.InvalidRequestException) StringUtils(org.hypertrace.entity.service.util.StringUtils) Streams(com.google.common.collect.Streams) Empty(org.hypertrace.entity.data.service.v1.Empty) Collectors(java.util.stream.Collectors) List(java.util.List) Stream(java.util.stream.Stream) EntityRelationship(org.hypertrace.entity.data.service.v1.EntityRelationship) EntityRelationships(org.hypertrace.entity.data.service.v1.EntityRelationships) MergeAndUpsertEntityRequest(org.hypertrace.entity.data.service.v1.MergeAndUpsertEntityRequest) Optional(java.util.Optional) EntityChangeEventGenerator(org.hypertrace.entity.service.change.event.api.EntityChangeEventGenerator) EnrichedEntity(org.hypertrace.entity.data.service.v1.EnrichedEntity) Builder(org.hypertrace.entity.data.service.v1.Entity.Builder) Document(org.hypertrace.core.documentstore.Document) RequestContext(org.hypertrace.core.grpcutils.context.RequestContext) ByTypeAndIdentifyingAttributes(org.hypertrace.entity.data.service.v1.ByTypeAndIdentifyingAttributes) DocStoreJsonFormat(org.hypertrace.entity.service.util.DocStoreJsonFormat) Descriptors(com.google.protobuf.Descriptors) Filter(org.hypertrace.core.documentstore.Filter) HashMap(java.util.HashMap) Function(java.util.function.Function) Collection(org.hypertrace.core.documentstore.Collection) Entities(org.hypertrace.entity.data.service.v1.Entities) ArrayList(java.util.ArrayList) Datastore(org.hypertrace.core.documentstore.Datastore) Key(org.hypertrace.core.documentstore.Key) EntityServiceConstants(org.hypertrace.entity.service.constants.EntityServiceConstants) Entity(org.hypertrace.entity.data.service.v1.Entity) EntityDataServiceImplBase(org.hypertrace.entity.data.service.v1.EntityDataServiceGrpc.EntityDataServiceImplBase) Logger(org.slf4j.Logger) Iterator(java.util.Iterator) JSONDocument(org.hypertrace.core.documentstore.JSONDocument) IOException(java.io.IOException) ENTITY_RELATIONSHIPS_COLLECTION(org.hypertrace.entity.service.constants.EntityCollectionConstants.ENTITY_RELATIONSHIPS_COLLECTION) EnrichedEntities(org.hypertrace.entity.data.service.v1.EnrichedEntities) Query(org.hypertrace.entity.data.service.v1.Query) RAW_ENTITIES_COLLECTION(org.hypertrace.entity.service.constants.EntityCollectionConstants.RAW_ENTITIES_COLLECTION) Message(com.google.protobuf.Message) GeneratedMessageV3(com.google.protobuf.GeneratedMessageV3) MergeAndUpsertEntityResponse(org.hypertrace.entity.data.service.v1.MergeAndUpsertEntityResponse) Collections(java.util.Collections) RelationshipsQuery(org.hypertrace.entity.data.service.v1.RelationshipsQuery) EnrichedEntity(org.hypertrace.entity.data.service.v1.EnrichedEntity) Entity(org.hypertrace.entity.data.service.v1.Entity) ServiceException(com.google.protobuf.ServiceException) Key(org.hypertrace.core.documentstore.Key)

Example 10 with Key

use of org.hypertrace.core.documentstore.Key in project entity-service by hypertrace.

the class EntityDataServiceImpl method upsertEnrichedEntities.

@Override
public void upsertEnrichedEntities(EnrichedEntities request, StreamObserver<Empty> responseObserver) {
    String tenantId = RequestContext.CURRENT.get().getTenantId().orElse(null);
    if (isNull(tenantId)) {
        responseObserver.onError(new ServiceException("Tenant id is missing in the request."));
        return;
    }
    for (EnrichedEntity entity : request.getEntitiesList()) {
        if (StringUtils.isEmpty(entity.getEntityType())) {
            LOG.info("{}. Invalid upsertEnrichedEntities request:{}", entity, ErrorMessages.ENTITY_TYPE_EMPTY);
            responseObserver.onError(new RuntimeException(ErrorMessages.ENTITY_TYPE_EMPTY));
            return;
        }
        if (StringUtils.isEmpty(entity.getEntityId())) {
            LOG.info("{}. Invalid upsertEnrichedEntities request:{}", entity, ErrorMessages.ENTITY_ID_EMPTY);
            responseObserver.onError(new RuntimeException(ErrorMessages.ENTITY_ID_EMPTY));
            return;
        }
    }
    Map<Key, EnrichedEntity> entities = request.getEntitiesList().stream().collect(Collectors.toUnmodifiableMap(entity -> this.entityNormalizer.getEntityDocKey(tenantId, entity.getEntityType(), entity.getEntityId()), Function.identity()));
    upsertEntities(entities, enrichedEntitiesCollection, responseObserver);
}
Also used : ByIdRequest(org.hypertrace.entity.data.service.v1.ByIdRequest) ServiceException(com.google.protobuf.ServiceException) LoggerFactory(org.slf4j.LoggerFactory) Channel(io.grpc.Channel) EntityTypeClient(org.hypertrace.entity.type.service.rxclient.EntityTypeClient) StreamObserver(io.grpc.stub.StreamObserver) Map(java.util.Map) Objects.isNull(java.util.Objects.isNull) ENRICHED_ENTITIES_COLLECTION(org.hypertrace.entity.service.constants.EntityCollectionConstants.ENRICHED_ENTITIES_COLLECTION) DocStoreConverter(org.hypertrace.entity.service.util.DocStoreConverter) InvalidRequestException(org.hypertrace.entity.service.exception.InvalidRequestException) StringUtils(org.hypertrace.entity.service.util.StringUtils) Streams(com.google.common.collect.Streams) Empty(org.hypertrace.entity.data.service.v1.Empty) Collectors(java.util.stream.Collectors) List(java.util.List) Stream(java.util.stream.Stream) EntityRelationship(org.hypertrace.entity.data.service.v1.EntityRelationship) EntityRelationships(org.hypertrace.entity.data.service.v1.EntityRelationships) MergeAndUpsertEntityRequest(org.hypertrace.entity.data.service.v1.MergeAndUpsertEntityRequest) Optional(java.util.Optional) EntityChangeEventGenerator(org.hypertrace.entity.service.change.event.api.EntityChangeEventGenerator) EnrichedEntity(org.hypertrace.entity.data.service.v1.EnrichedEntity) Builder(org.hypertrace.entity.data.service.v1.Entity.Builder) Document(org.hypertrace.core.documentstore.Document) RequestContext(org.hypertrace.core.grpcutils.context.RequestContext) ByTypeAndIdentifyingAttributes(org.hypertrace.entity.data.service.v1.ByTypeAndIdentifyingAttributes) DocStoreJsonFormat(org.hypertrace.entity.service.util.DocStoreJsonFormat) Descriptors(com.google.protobuf.Descriptors) Filter(org.hypertrace.core.documentstore.Filter) HashMap(java.util.HashMap) Function(java.util.function.Function) Collection(org.hypertrace.core.documentstore.Collection) Entities(org.hypertrace.entity.data.service.v1.Entities) ArrayList(java.util.ArrayList) Datastore(org.hypertrace.core.documentstore.Datastore) Key(org.hypertrace.core.documentstore.Key) EntityServiceConstants(org.hypertrace.entity.service.constants.EntityServiceConstants) Entity(org.hypertrace.entity.data.service.v1.Entity) EntityDataServiceImplBase(org.hypertrace.entity.data.service.v1.EntityDataServiceGrpc.EntityDataServiceImplBase) Logger(org.slf4j.Logger) Iterator(java.util.Iterator) JSONDocument(org.hypertrace.core.documentstore.JSONDocument) IOException(java.io.IOException) ENTITY_RELATIONSHIPS_COLLECTION(org.hypertrace.entity.service.constants.EntityCollectionConstants.ENTITY_RELATIONSHIPS_COLLECTION) EnrichedEntities(org.hypertrace.entity.data.service.v1.EnrichedEntities) Query(org.hypertrace.entity.data.service.v1.Query) RAW_ENTITIES_COLLECTION(org.hypertrace.entity.service.constants.EntityCollectionConstants.RAW_ENTITIES_COLLECTION) Message(com.google.protobuf.Message) GeneratedMessageV3(com.google.protobuf.GeneratedMessageV3) MergeAndUpsertEntityResponse(org.hypertrace.entity.data.service.v1.MergeAndUpsertEntityResponse) Collections(java.util.Collections) RelationshipsQuery(org.hypertrace.entity.data.service.v1.RelationshipsQuery) ServiceException(com.google.protobuf.ServiceException) EnrichedEntity(org.hypertrace.entity.data.service.v1.EnrichedEntity) Key(org.hypertrace.core.documentstore.Key)

Aggregations

Key (org.hypertrace.core.documentstore.Key)22 Document (org.hypertrace.core.documentstore.Document)19 JSONDocument (org.hypertrace.core.documentstore.JSONDocument)17 HashMap (java.util.HashMap)14 ArrayList (java.util.ArrayList)12 Map (java.util.Map)12 Collection (org.hypertrace.core.documentstore.Collection)9 SingleValueKey (org.hypertrace.core.documentstore.SingleValueKey)9 ServiceException (com.google.protobuf.ServiceException)8 IOException (java.io.IOException)6 BulkUpdateResult (org.hypertrace.core.documentstore.BulkUpdateResult)6 BulkArrayValueUpdateRequest (org.hypertrace.core.documentstore.BulkArrayValueUpdateRequest)5 Query (org.hypertrace.core.documentstore.Query)5 Entity (org.hypertrace.entity.data.service.v1.Entity)5 JsonNode (com.fasterxml.jackson.databind.JsonNode)4 MongoCollection (com.mongodb.client.MongoCollection)4 List (java.util.List)4 Collectors (java.util.stream.Collectors)4 Datastore (org.hypertrace.core.documentstore.Datastore)4 Test (org.junit.jupiter.api.Test)4