Search in sources :

Example 56 with StreamId

use of co.cask.cdap.proto.id.StreamId in project cdap by caskdata.

the class LineageAdmin method getRollupRelations.

private Multimap<RelationKey, Relation> getRollupRelations(Multimap<RelationKey, Relation> relations, Map<ProgramRunId, RunRecordMeta> runRecordMap, Map<String, ProgramRunId> workflowIdMap) throws NotFoundException {
    Multimap<RelationKey, Relation> relationsNew = HashMultimap.create();
    for (Map.Entry<RelationKey, Collection<Relation>> entry : relations.asMap().entrySet()) {
        for (Relation relation : entry.getValue()) {
            ProgramRunId workflowProgramRunId = getWorkflowProgramRunid(relation, runRecordMap, workflowIdMap);
            if (workflowProgramRunId == null) {
                relationsNew.put(entry.getKey(), relation);
            } else {
                ProgramId workflowProgramId = new ProgramId(workflowProgramRunId.getNamespace(), workflowProgramRunId.getApplication(), workflowProgramRunId.getType(), workflowProgramRunId.getProgram());
                Relation workflowRelation;
                NamespacedEntityId data = relation.getData();
                if (data instanceof DatasetId) {
                    workflowRelation = new Relation((DatasetId) data, workflowProgramId, relation.getAccess(), RunIds.fromString(workflowProgramRunId.getRun()));
                } else {
                    workflowRelation = new Relation((StreamId) data, workflowProgramId, relation.getAccess(), RunIds.fromString(workflowProgramRunId.getRun()));
                }
                relationsNew.put(entry.getKey(), workflowRelation);
            }
        }
    }
    return relationsNew;
}
Also used : Relation(co.cask.cdap.data2.metadata.lineage.Relation) NamespacedEntityId(co.cask.cdap.proto.id.NamespacedEntityId) StreamId(co.cask.cdap.proto.id.StreamId) Collection(java.util.Collection) ProgramRunId(co.cask.cdap.proto.id.ProgramRunId) ProgramId(co.cask.cdap.proto.id.ProgramId) HashMap(java.util.HashMap) Map(java.util.Map) DatasetId(co.cask.cdap.proto.id.DatasetId)

Example 57 with StreamId

use of co.cask.cdap.proto.id.StreamId in project cdap by caskdata.

the class LineageHandler method streamLineage.

@GET
@Path("/namespaces/{namespace-id}/streams/{stream-id}/lineage")
public void streamLineage(HttpRequest request, HttpResponder responder, @PathParam("namespace-id") String namespaceId, @PathParam("stream-id") String stream, @QueryParam("start") String startStr, @QueryParam("end") String endStr, @QueryParam("levels") @DefaultValue("10") int levels, @QueryParam("collapse") List<String> collapse, @QueryParam("rollup") String rollup) throws Exception {
    checkLevels(levels);
    TimeRange range = parseRange(startStr, endStr);
    StreamId streamId = new StreamId(namespaceId, stream);
    Lineage lineage = lineageAdmin.computeLineage(streamId, range.getStart(), range.getEnd(), levels, rollup);
    responder.sendJson(HttpResponseStatus.OK, LineageSerializer.toLineageRecord(TimeUnit.MILLISECONDS.toSeconds(range.getStart()), TimeUnit.MILLISECONDS.toSeconds(range.getEnd()), lineage, getCollapseTypes(collapse)), LineageRecord.class, GSON);
}
Also used : StreamId(co.cask.cdap.proto.id.StreamId) Lineage(co.cask.cdap.data2.metadata.lineage.Lineage) Path(javax.ws.rs.Path) GET(javax.ws.rs.GET)

Example 58 with StreamId

use of co.cask.cdap.proto.id.StreamId in project cdap by caskdata.

the class MetadataHttpHandler method addStreamTags.

@POST
@Path("/namespaces/{namespace-id}/streams/{stream-id}/metadata/tags")
@AuditPolicy(AuditDetail.REQUEST_BODY)
public void addStreamTags(HttpRequest request, HttpResponder responder, @PathParam("namespace-id") String namespaceId, @PathParam("stream-id") String streamId) throws BadRequestException, NotFoundException {
    StreamId stream = new StreamId(namespaceId, streamId);
    metadataAdmin.addTags(stream, readArray(request));
    responder.sendString(HttpResponseStatus.OK, String.format("Added tags to stream %s successfully.", stream));
}
Also used : StreamId(co.cask.cdap.proto.id.StreamId) Path(javax.ws.rs.Path) AuditPolicy(co.cask.cdap.common.security.AuditPolicy) POST(javax.ws.rs.POST)

Example 59 with StreamId

use of co.cask.cdap.proto.id.StreamId in project cdap by caskdata.

the class MetadataHttpHandler method removeStreamMetadata.

@DELETE
@Path("/namespaces/{namespace-id}/streams/{stream-id}/metadata")
public void removeStreamMetadata(HttpRequest request, HttpResponder responder, @PathParam("namespace-id") String namespaceId, @PathParam("stream-id") String streamId) throws NotFoundException {
    StreamId stream = new StreamId(namespaceId, streamId);
    metadataAdmin.removeMetadata(stream);
    responder.sendString(HttpResponseStatus.OK, String.format("Metadata for stream %s deleted successfully.", stream));
}
Also used : StreamId(co.cask.cdap.proto.id.StreamId) Path(javax.ws.rs.Path) DELETE(javax.ws.rs.DELETE)

Example 60 with StreamId

use of co.cask.cdap.proto.id.StreamId in project cdap by caskdata.

the class MetadataHttpHandler method removeStreamProperty.

@DELETE
@Path("/namespaces/{namespace-id}/streams/{stream-id}/metadata/properties/{property}")
public void removeStreamProperty(HttpRequest request, HttpResponder responder, @PathParam("namespace-id") String namespaceId, @PathParam("stream-id") String streamId, @PathParam("property") String property) throws NotFoundException {
    StreamId stream = new StreamId(namespaceId, streamId);
    metadataAdmin.removeProperties(stream, property);
    responder.sendString(HttpResponseStatus.OK, String.format("Metadata property %s for stream %s deleted successfully.", property, stream));
}
Also used : StreamId(co.cask.cdap.proto.id.StreamId) Path(javax.ws.rs.Path) DELETE(javax.ws.rs.DELETE)

Aggregations

StreamId (co.cask.cdap.proto.id.StreamId)166 Test (org.junit.Test)88 DatasetId (co.cask.cdap.proto.id.DatasetId)33 ProgramId (co.cask.cdap.proto.id.ProgramId)30 NamespaceId (co.cask.cdap.proto.id.NamespaceId)27 Path (javax.ws.rs.Path)27 StreamEvent (co.cask.cdap.api.flow.flowlet.StreamEvent)24 ApplicationId (co.cask.cdap.proto.id.ApplicationId)22 IOException (java.io.IOException)20 StreamProperties (co.cask.cdap.proto.StreamProperties)17 FormatSpecification (co.cask.cdap.api.data.format.FormatSpecification)16 StreamViewId (co.cask.cdap.proto.id.StreamViewId)16 Location (org.apache.twill.filesystem.Location)15 StreamConfig (co.cask.cdap.data2.transaction.stream.StreamConfig)12 NamespaceMeta (co.cask.cdap.proto.NamespaceMeta)12 StreamAdmin (co.cask.cdap.data2.transaction.stream.StreamAdmin)11 ViewSpecification (co.cask.cdap.proto.ViewSpecification)10 MetadataSearchResultRecord (co.cask.cdap.proto.metadata.MetadataSearchResultRecord)10 Action (co.cask.cdap.proto.security.Action)10 GET (javax.ws.rs.GET)10