Search in sources :

Example 1 with EventLogView

use of org.apache.inlong.manager.common.pojo.workflow.EventLogView in project incubator-inlong by apache.

the class InlongGroupImpl method generateSnapshot.

private InlongGroupContext generateSnapshot() {
    // Fetch current group
    InlongGroupResponse groupResponse = managerClient.getGroupInfo(groupContext.getGroupId());
    InlongGroupInfo currentGroupInfo = CommonBeanUtils.copyProperties(groupResponse, InlongGroupInfo::new);
    groupContext.setGroupInfo(currentGroupInfo);
    String inlongGroupId = currentGroupInfo.getInlongGroupId();
    // Fetch stream in group
    List<InlongStream> dataStreams = fetchDataStreams(inlongGroupId);
    if (CollectionUtils.isNotEmpty(dataStreams)) {
        dataStreams.forEach(dataStream -> groupContext.setStream(dataStream));
    }
    // Create group context
    InlongGroupContext inlongGroupContext = new InlongGroupContext(groupContext, groupConf);
    // Fetch group logs
    List<EventLogView> logViews = managerClient.getInlongGroupError(inlongGroupId);
    if (CollectionUtils.isNotEmpty(logViews)) {
        Map<String, List<String>> errMsgs = Maps.newHashMap();
        Map<String, List<String>> groupLogs = Maps.newHashMap();
        logViews.stream().filter(x -> StringUtils.isNotEmpty(x.getElementName())).forEach(eventLogView -> {
            String taskName = eventLogView.getElementName();
            if (StringUtils.isNotEmpty(eventLogView.getException())) {
                errMsgs.computeIfAbsent(taskName, Lists::newArrayList).add(eventLogView.getException());
            }
            if (StringUtils.isNotEmpty(eventLogView.getRemark())) {
                groupLogs.computeIfAbsent(taskName, Lists::newArrayList).add(eventLogView.getRemark());
            }
        });
        inlongGroupContext.setErrMsgs(errMsgs);
        inlongGroupContext.setGroupLogs(groupLogs);
    }
    // Fetch stream logs
    Map<String, InlongStream> streams = inlongGroupContext.getInlongStreamMap();
    streams.keySet().stream().forEach(streamName -> {
        String inlongStreamId = "b_" + streamName;
        List<InlongStreamConfigLogListResponse> logList = managerClient.getStreamLogs(inlongGroupId, inlongStreamId);
        if (CollectionUtils.isNotEmpty(logList)) {
            Map<String, List<String>> streamLogs = Maps.newHashMap();
            logList.stream().filter(x -> StringUtils.isNotEmpty(x.getComponentName())).forEach(streamLog -> {
                String componentName = streamLog.getComponentName();
                String log = GsonUtil.toJson(streamLog);
                streamLogs.computeIfAbsent(componentName, Lists::newArrayList).add(log);
            });
            inlongGroupContext.getStreamLogs().put(streamName, streamLogs);
        }
    });
    return inlongGroupContext;
}
Also used : InlongParser(org.apache.inlong.manager.client.api.util.InlongParser) GsonUtil(org.apache.inlong.manager.client.api.util.GsonUtil) AssertUtil(org.apache.inlong.manager.client.api.util.AssertUtil) ProcessStatus(org.apache.inlong.manager.common.enums.ProcessStatus) InlongGroupInfo(org.apache.inlong.manager.common.pojo.group.InlongGroupInfo) TaskResponse(org.apache.inlong.manager.common.pojo.workflow.TaskResponse) StringUtils(org.apache.commons.lang3.StringUtils) InlongGroupContext(org.apache.inlong.manager.client.api.InlongGroupContext) Lists(com.google.common.collect.Lists) Pair(org.apache.commons.lang3.tuple.Pair) CollectionUtils(org.apache.commons.collections.CollectionUtils) InlongStream(org.apache.inlong.manager.client.api.InlongStream) Map(java.util.Map) InnerInlongManagerClient(org.apache.inlong.manager.client.api.inner.InnerInlongManagerClient) ProcessResponse(org.apache.inlong.manager.common.pojo.workflow.ProcessResponse) InlongStreamBuilder(org.apache.inlong.manager.client.api.InlongStreamBuilder) GroupState(org.apache.inlong.manager.common.enums.GroupState) InlongStreamApproveRequest(org.apache.inlong.manager.common.pojo.stream.InlongStreamApproveRequest) InlongStreamConfigLogListResponse(org.apache.inlong.manager.common.pojo.stream.InlongStreamConfigLogListResponse) InlongGroup(org.apache.inlong.manager.client.api.InlongGroup) InlongGroupRequest(org.apache.inlong.manager.common.pojo.group.InlongGroupRequest) Maps(com.google.common.collect.Maps) Collectors(java.util.stream.Collectors) InlongGroupTransfer(org.apache.inlong.manager.client.api.util.InlongGroupTransfer) EventLogView(org.apache.inlong.manager.common.pojo.workflow.EventLogView) InlongGroupResponse(org.apache.inlong.manager.common.pojo.group.InlongGroupResponse) List(java.util.List) InnerGroupContext(org.apache.inlong.manager.client.api.inner.InnerGroupContext) InlongGroupConf(org.apache.inlong.manager.client.api.InlongGroupConf) InlongStreamConf(org.apache.inlong.manager.client.api.InlongStreamConf) CommonBeanUtils(org.apache.inlong.manager.common.util.CommonBeanUtils) FullStreamResponse(org.apache.inlong.manager.common.pojo.stream.FullStreamResponse) WorkflowResult(org.apache.inlong.manager.common.pojo.workflow.WorkflowResult) InlongGroupApproveRequest(org.apache.inlong.manager.common.pojo.group.InlongGroupApproveRequest) InlongGroupState(org.apache.inlong.manager.client.api.InlongGroupContext.InlongGroupState) EventLogView(org.apache.inlong.manager.common.pojo.workflow.EventLogView) InlongStreamConfigLogListResponse(org.apache.inlong.manager.common.pojo.stream.InlongStreamConfigLogListResponse) InlongGroupContext(org.apache.inlong.manager.client.api.InlongGroupContext) InlongGroupInfo(org.apache.inlong.manager.common.pojo.group.InlongGroupInfo) InlongStream(org.apache.inlong.manager.client.api.InlongStream) List(java.util.List) InlongGroupResponse(org.apache.inlong.manager.common.pojo.group.InlongGroupResponse)

Example 2 with EventLogView

use of org.apache.inlong.manager.common.pojo.workflow.EventLogView in project incubator-inlong by apache.

the class InnerInlongManagerClient method getInlongGroupError.

/**
 * get inlong group error messages
 */
public List<EventLogView> getInlongGroupError(String inlongGroupId) {
    final String path = HTTP_PATH + "/workflow/event/list";
    String url = formatUrl(path);
    url = url + "&inlongGroupId=" + inlongGroupId + "&status=-1";
    Request request = new Request.Builder().get().url(url).build();
    Call call = httpClient.newCall(request);
    try {
        Response response = call.execute();
        assert response.body() != null;
        String body = response.body().string();
        AssertUtil.isTrue(response.isSuccessful(), String.format("Inlong request failed: %s", body));
        org.apache.inlong.manager.common.beans.Response responseBody = InlongParser.parseResponse(body);
        AssertUtil.isTrue(responseBody.getErrMsg() == null, String.format("Inlong request failed: %s", responseBody.getErrMsg()));
        PageInfo<EventLogView> pageInfo = InlongParser.parseEventLogViewList(responseBody);
        return pageInfo.getList();
    } catch (Exception e) {
        throw new RuntimeException(String.format("Get inlong group error messages failed: %s", e.getMessage()), e);
    }
}
Also used : Call(okhttp3.Call) EventLogView(org.apache.inlong.manager.common.pojo.workflow.EventLogView) SinkRequest(org.apache.inlong.manager.common.pojo.sink.SinkRequest) Request(okhttp3.Request) InlongStreamApproveRequest(org.apache.inlong.manager.common.pojo.stream.InlongStreamApproveRequest) InlongGroupRequest(org.apache.inlong.manager.common.pojo.group.InlongGroupRequest) SourceRequest(org.apache.inlong.manager.common.pojo.source.SourceRequest) InlongGroupApproveRequest(org.apache.inlong.manager.common.pojo.group.InlongGroupApproveRequest) Response(okhttp3.Response) InlongStreamConfigLogListResponse(org.apache.inlong.manager.common.pojo.stream.InlongStreamConfigLogListResponse) SinkListResponse(org.apache.inlong.manager.common.pojo.sink.SinkListResponse) SourceListResponse(org.apache.inlong.manager.common.pojo.source.SourceListResponse) InlongGroupListResponse(org.apache.inlong.manager.common.pojo.group.InlongGroupListResponse) InlongGroupResponse(org.apache.inlong.manager.common.pojo.group.InlongGroupResponse) FullStreamResponse(org.apache.inlong.manager.common.pojo.stream.FullStreamResponse)

Example 3 with EventLogView

use of org.apache.inlong.manager.common.pojo.workflow.EventLogView in project incubator-inlong by apache.

the class InlongParser method parseEventLogViewList.

public static PageInfo<EventLogView> parseEventLogViewList(Response response) {
    Object data = response.getData();
    String pageInfoJson = GsonUtil.toJson(data);
    return GsonUtil.fromJson(pageInfoJson, new TypeToken<PageInfo<EventLogView>>() {
    }.getType());
}
Also used : EventLogView(org.apache.inlong.manager.common.pojo.workflow.EventLogView) TypeToken(com.google.common.reflect.TypeToken) JsonObject(com.google.gson.JsonObject)

Aggregations

EventLogView (org.apache.inlong.manager.common.pojo.workflow.EventLogView)3 InlongGroupApproveRequest (org.apache.inlong.manager.common.pojo.group.InlongGroupApproveRequest)2 InlongGroupRequest (org.apache.inlong.manager.common.pojo.group.InlongGroupRequest)2 InlongGroupResponse (org.apache.inlong.manager.common.pojo.group.InlongGroupResponse)2 FullStreamResponse (org.apache.inlong.manager.common.pojo.stream.FullStreamResponse)2 InlongStreamApproveRequest (org.apache.inlong.manager.common.pojo.stream.InlongStreamApproveRequest)2 InlongStreamConfigLogListResponse (org.apache.inlong.manager.common.pojo.stream.InlongStreamConfigLogListResponse)2 Lists (com.google.common.collect.Lists)1 Maps (com.google.common.collect.Maps)1 TypeToken (com.google.common.reflect.TypeToken)1 JsonObject (com.google.gson.JsonObject)1 List (java.util.List)1 Map (java.util.Map)1 Collectors (java.util.stream.Collectors)1 Call (okhttp3.Call)1 Request (okhttp3.Request)1 Response (okhttp3.Response)1 CollectionUtils (org.apache.commons.collections.CollectionUtils)1 StringUtils (org.apache.commons.lang3.StringUtils)1 Pair (org.apache.commons.lang3.tuple.Pair)1