Search in sources :

Example 1 with SinkResponse

use of org.apache.inlong.manager.common.pojo.sink.SinkResponse in project incubator-inlong by apache.

the class HiveStreamSinkOperation method getById.

@Override
public SinkResponse getById(@NotNull String sinkType, @NotNull Integer id) {
    StreamSinkEntity entity = sinkMapper.selectByPrimaryKey(id);
    Preconditions.checkNotNull(entity, ErrorCodeEnum.SINK_INFO_NOT_FOUND.getMessage());
    String existType = entity.getSinkType();
    Preconditions.checkTrue(Constant.SINK_HIVE.equals(existType), String.format(Constant.SINK_TYPE_NOT_SAME, Constant.SINK_HIVE, existType));
    SinkResponse response = this.getFromEntity(entity, HiveSinkResponse::new);
    List<StreamSinkFieldEntity> entities = sinkFieldMapper.selectBySinkId(id);
    List<SinkFieldResponse> infos = CommonBeanUtils.copyListProperties(entities, SinkFieldResponse::new);
    response.setFieldList(infos);
    return response;
}
Also used : SinkResponse(org.apache.inlong.manager.common.pojo.sink.SinkResponse) HiveSinkResponse(org.apache.inlong.manager.common.pojo.sink.hive.HiveSinkResponse) HiveSinkResponse(org.apache.inlong.manager.common.pojo.sink.hive.HiveSinkResponse) StreamSinkEntity(org.apache.inlong.manager.dao.entity.StreamSinkEntity) SinkFieldResponse(org.apache.inlong.manager.common.pojo.sink.SinkFieldResponse) StreamSinkFieldEntity(org.apache.inlong.manager.dao.entity.StreamSinkFieldEntity)

Example 2 with SinkResponse

use of org.apache.inlong.manager.common.pojo.sink.SinkResponse in project incubator-inlong by apache.

the class IcebergStreamSinkOperation method getById.

@Override
public SinkResponse getById(String sinkType, Integer id) {
    StreamSinkEntity entity = sinkMapper.selectByPrimaryKey(id);
    Preconditions.checkNotNull(entity, ErrorCodeEnum.SINK_INFO_NOT_FOUND.getMessage());
    String existType = entity.getSinkType();
    Preconditions.checkTrue(Constant.SINK_ICEBERG.equals(existType), String.format(Constant.SINK_TYPE_NOT_SAME, Constant.SINK_ICEBERG, existType));
    SinkResponse response = this.getFromEntity(entity, IcebergSinkResponse::new);
    List<StreamSinkFieldEntity> entities = sinkFieldMapper.selectBySinkId(id);
    List<SinkFieldResponse> infos = CommonBeanUtils.copyListProperties(entities, SinkFieldResponse::new);
    response.setFieldList(infos);
    return response;
}
Also used : IcebergSinkResponse(org.apache.inlong.manager.common.pojo.sink.iceberg.IcebergSinkResponse) SinkResponse(org.apache.inlong.manager.common.pojo.sink.SinkResponse) StreamSinkEntity(org.apache.inlong.manager.dao.entity.StreamSinkEntity) SinkFieldResponse(org.apache.inlong.manager.common.pojo.sink.SinkFieldResponse) StreamSinkFieldEntity(org.apache.inlong.manager.dao.entity.StreamSinkFieldEntity) IcebergSinkResponse(org.apache.inlong.manager.common.pojo.sink.iceberg.IcebergSinkResponse)

Example 3 with SinkResponse

use of org.apache.inlong.manager.common.pojo.sink.SinkResponse in project incubator-inlong by apache.

the class CreateSortConfigListener method listen.

@Override
public ListenerResult listen(WorkflowContext context) throws Exception {
    LOGGER.info("Create sort config for context={}", context);
    ProcessForm form = context.getProcessForm();
    if (form instanceof UpdateGroupProcessForm) {
        UpdateGroupProcessForm updateGroupProcessForm = (UpdateGroupProcessForm) form;
        OperateType operateType = updateGroupProcessForm.getOperateType();
        if (operateType == OperateType.SUSPEND || operateType == OperateType.DELETE) {
            return ListenerResult.success();
        }
    }
    InlongGroupInfo groupInfo = this.getGroupInfo(form);
    String groupId = groupInfo.getInlongGroupId();
    if (StringUtils.isEmpty(groupId)) {
        LOGGER.warn("GroupId is null for context={}", context);
        return ListenerResult.success();
    }
    List<SinkResponse> sinkResponseList = streamSinkService.listSink(groupId, null);
    if (CollectionUtils.isEmpty(sinkResponseList)) {
        LOGGER.warn("Sink not found by groupId={}", groupId);
        return ListenerResult.success();
    }
    Map<String, DataFlowInfo> dataFlowInfoMap = sinkResponseList.stream().map(sink -> {
        DataFlowInfo flowInfo = commonOperateService.createDataFlow(groupInfo, sink);
        return Pair.of(sink.getInlongStreamId(), flowInfo);
    }).collect(Collectors.toMap(Pair::getKey, Pair::getValue));
    String dataFlows = OBJECT_MAPPER.writeValueAsString(dataFlowInfoMap);
    InlongGroupExtInfo extInfo = new InlongGroupExtInfo();
    extInfo.setInlongGroupId(groupId);
    extInfo.setKeyName(InlongGroupSettings.DATA_FLOW);
    extInfo.setKeyValue(dataFlows);
    if (groupInfo.getExtList() == null) {
        groupInfo.setExtList(Lists.newArrayList());
    }
    upsertDataFlow(groupInfo, extInfo);
    return ListenerResult.success();
}
Also used : ProcessForm(org.apache.inlong.manager.common.pojo.workflow.form.ProcessForm) InlongGroupInfo(org.apache.inlong.manager.common.pojo.group.InlongGroupInfo) LoggerFactory(org.slf4j.LoggerFactory) Autowired(org.springframework.beans.factory.annotation.Autowired) StringUtils(org.apache.commons.lang3.StringUtils) GroupResourceProcessForm(org.apache.inlong.manager.common.pojo.workflow.form.GroupResourceProcessForm) Lists(com.google.common.collect.Lists) Pair(org.apache.commons.lang3.tuple.Pair) CollectionUtils(org.apache.commons.collections.CollectionUtils) CommonOperateService(org.apache.inlong.manager.service.CommonOperateService) StreamSinkService(org.apache.inlong.manager.service.sink.StreamSinkService) TaskEvent(org.apache.inlong.manager.workflow.event.task.TaskEvent) ObjectMapper(org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper) Map(java.util.Map) OperateType(org.apache.inlong.manager.common.pojo.workflow.form.UpdateGroupProcessForm.OperateType) WorkflowContext(org.apache.inlong.manager.workflow.WorkflowContext) InlongGroupSettings(org.apache.inlong.manager.common.settings.InlongGroupSettings) Logger(org.slf4j.Logger) SinkResponse(org.apache.inlong.manager.common.pojo.sink.SinkResponse) Collectors(java.util.stream.Collectors) WorkflowListenerException(org.apache.inlong.manager.common.exceptions.WorkflowListenerException) ListenerResult(org.apache.inlong.manager.workflow.event.ListenerResult) Component(org.springframework.stereotype.Component) List(java.util.List) UpdateGroupProcessForm(org.apache.inlong.manager.common.pojo.workflow.form.UpdateGroupProcessForm) DataFlowInfo(org.apache.inlong.sort.protocol.DataFlowInfo) InlongGroupExtInfo(org.apache.inlong.manager.common.pojo.group.InlongGroupExtInfo) SortOperateListener(org.apache.inlong.manager.workflow.event.task.SortOperateListener) UpdateGroupProcessForm(org.apache.inlong.manager.common.pojo.workflow.form.UpdateGroupProcessForm) SinkResponse(org.apache.inlong.manager.common.pojo.sink.SinkResponse) ProcessForm(org.apache.inlong.manager.common.pojo.workflow.form.ProcessForm) GroupResourceProcessForm(org.apache.inlong.manager.common.pojo.workflow.form.GroupResourceProcessForm) UpdateGroupProcessForm(org.apache.inlong.manager.common.pojo.workflow.form.UpdateGroupProcessForm) InlongGroupInfo(org.apache.inlong.manager.common.pojo.group.InlongGroupInfo) OperateType(org.apache.inlong.manager.common.pojo.workflow.form.UpdateGroupProcessForm.OperateType) InlongGroupExtInfo(org.apache.inlong.manager.common.pojo.group.InlongGroupExtInfo) DataFlowInfo(org.apache.inlong.sort.protocol.DataFlowInfo)

Example 4 with SinkResponse

use of org.apache.inlong.manager.common.pojo.sink.SinkResponse in project incubator-inlong by apache.

the class KafkaStreamSinkServiceTest method testListByIdentifier.

@Test
public void testListByIdentifier() {
    SinkResponse sink = sinkService.get(kafkaSinkId, Constant.SINK_KAFKA);
    Assert.assertEquals(globalGroupId, sink.getInlongGroupId());
}
Also used : SinkResponse(org.apache.inlong.manager.common.pojo.sink.SinkResponse) KafkaSinkResponse(org.apache.inlong.manager.common.pojo.sink.kafka.KafkaSinkResponse) Test(org.junit.Test) ServiceBaseTest(org.apache.inlong.manager.service.ServiceBaseTest) InlongStreamServiceTest(org.apache.inlong.manager.service.core.impl.InlongStreamServiceTest)

Example 5 with SinkResponse

use of org.apache.inlong.manager.common.pojo.sink.SinkResponse in project incubator-inlong by apache.

the class KafkaStreamSinkOperation method getById.

@Override
public SinkResponse getById(@NotNull String sinkType, @NotNull Integer id) {
    StreamSinkEntity entity = sinkMapper.selectByPrimaryKey(id);
    Preconditions.checkNotNull(entity, ErrorCodeEnum.SINK_INFO_NOT_FOUND.getMessage());
    String existType = entity.getSinkType();
    Preconditions.checkTrue(Constant.SINK_KAFKA.equals(existType), String.format(Constant.SINK_TYPE_NOT_SAME, Constant.SINK_KAFKA, existType));
    SinkResponse response = this.getFromEntity(entity, KafkaSinkResponse::new);
    List<StreamSinkFieldEntity> entities = sinkFieldMapper.selectBySinkId(id);
    List<SinkFieldResponse> infos = CommonBeanUtils.copyListProperties(entities, SinkFieldResponse::new);
    response.setFieldList(infos);
    return response;
}
Also used : KafkaSinkResponse(org.apache.inlong.manager.common.pojo.sink.kafka.KafkaSinkResponse) SinkResponse(org.apache.inlong.manager.common.pojo.sink.SinkResponse) KafkaSinkResponse(org.apache.inlong.manager.common.pojo.sink.kafka.KafkaSinkResponse) StreamSinkEntity(org.apache.inlong.manager.dao.entity.StreamSinkEntity) SinkFieldResponse(org.apache.inlong.manager.common.pojo.sink.SinkFieldResponse) StreamSinkFieldEntity(org.apache.inlong.manager.dao.entity.StreamSinkFieldEntity)

Aggregations

SinkResponse (org.apache.inlong.manager.common.pojo.sink.SinkResponse)17 ServiceBaseTest (org.apache.inlong.manager.service.ServiceBaseTest)8 InlongStreamServiceTest (org.apache.inlong.manager.service.core.impl.InlongStreamServiceTest)8 Test (org.junit.Test)8 SinkFieldResponse (org.apache.inlong.manager.common.pojo.sink.SinkFieldResponse)4 ClickHouseSinkResponse (org.apache.inlong.manager.common.pojo.sink.ck.ClickHouseSinkResponse)4 HiveSinkResponse (org.apache.inlong.manager.common.pojo.sink.hive.HiveSinkResponse)4 IcebergSinkResponse (org.apache.inlong.manager.common.pojo.sink.iceberg.IcebergSinkResponse)4 KafkaSinkResponse (org.apache.inlong.manager.common.pojo.sink.kafka.KafkaSinkResponse)4 StreamSinkEntity (org.apache.inlong.manager.dao.entity.StreamSinkEntity)4 StreamSinkFieldEntity (org.apache.inlong.manager.dao.entity.StreamSinkFieldEntity)4 WorkflowListenerException (org.apache.inlong.manager.common.exceptions.WorkflowListenerException)2 InlongGroupInfo (org.apache.inlong.manager.common.pojo.group.InlongGroupInfo)2 SourceResponse (org.apache.inlong.manager.common.pojo.source.SourceResponse)2 FullStreamResponse (org.apache.inlong.manager.common.pojo.stream.FullStreamResponse)2 GroupResourceProcessForm (org.apache.inlong.manager.common.pojo.workflow.form.GroupResourceProcessForm)2 DataFlowInfo (org.apache.inlong.sort.protocol.DataFlowInfo)2 Page (com.github.pagehelper.Page)1 PageInfo (com.github.pagehelper.PageInfo)1 Lists (com.google.common.collect.Lists)1