Search in sources :

Example 1 with HiveSinkRequest

use of org.apache.inlong.manager.common.pojo.sink.hive.HiveSinkRequest in project incubator-inlong by apache.

the class HiveStreamSinkOperation method saveOpt.

@Override
public Integer saveOpt(SinkRequest request, String operator) {
    String sinkType = request.getSinkType();
    Preconditions.checkTrue(Constant.SINK_HIVE.equals(sinkType), ErrorCodeEnum.SINK_TYPE_NOT_SUPPORT.getMessage() + ": " + sinkType);
    HiveSinkRequest hiveRequest = (HiveSinkRequest) request;
    StreamSinkEntity entity = CommonBeanUtils.copyProperties(hiveRequest, StreamSinkEntity::new);
    entity.setStatus(EntityStatus.SINK_NEW.getCode());
    entity.setIsDeleted(EntityStatus.UN_DELETED.getCode());
    entity.setCreator(operator);
    entity.setModifier(operator);
    Date now = new Date();
    entity.setCreateTime(now);
    entity.setModifyTime(now);
    // get the ext params
    HiveSinkDTO dto = HiveSinkDTO.getFromRequest(hiveRequest);
    try {
        entity.setExtParams(objectMapper.writeValueAsString(dto));
    } catch (Exception e) {
        throw new BusinessException(ErrorCodeEnum.SINK_SAVE_FAILED);
    }
    sinkMapper.insert(entity);
    Integer sinkId = entity.getId();
    request.setId(sinkId);
    this.saveFieldOpt(request);
    return sinkId;
}
Also used : BusinessException(org.apache.inlong.manager.common.exceptions.BusinessException) HiveSinkRequest(org.apache.inlong.manager.common.pojo.sink.hive.HiveSinkRequest) HiveSinkDTO(org.apache.inlong.manager.common.pojo.sink.hive.HiveSinkDTO) StreamSinkEntity(org.apache.inlong.manager.dao.entity.StreamSinkEntity) Date(java.util.Date) BusinessException(org.apache.inlong.manager.common.exceptions.BusinessException)

Example 2 with HiveSinkRequest

use of org.apache.inlong.manager.common.pojo.sink.hive.HiveSinkRequest in project incubator-inlong by apache.

the class HiveStreamSinkOperation method updateOpt.

@Override
public void updateOpt(SinkRequest request, String operator) {
    String sinkType = request.getSinkType();
    Preconditions.checkTrue(Constant.SINK_HIVE.equals(sinkType), String.format(Constant.SINK_TYPE_NOT_SAME, Constant.SINK_HIVE, sinkType));
    StreamSinkEntity entity = sinkMapper.selectByPrimaryKey(request.getId());
    Preconditions.checkNotNull(entity, ErrorCodeEnum.SINK_INFO_NOT_FOUND.getMessage());
    HiveSinkRequest hiveRequest = (HiveSinkRequest) request;
    CommonBeanUtils.copyProperties(hiveRequest, entity, true);
    try {
        HiveSinkDTO dto = HiveSinkDTO.getFromRequest(hiveRequest);
        entity.setExtParams(objectMapper.writeValueAsString(dto));
    } catch (Exception e) {
        throw new BusinessException(ErrorCodeEnum.SINK_INFO_INCORRECT.getMessage());
    }
    entity.setPreviousStatus(entity.getStatus());
    entity.setStatus(EntityStatus.GROUP_CONFIG_ING.getCode());
    entity.setModifier(operator);
    entity.setModifyTime(new Date());
    sinkMapper.updateByPrimaryKeySelective(entity);
    boolean onlyAdd = EntityStatus.SINK_CONFIG_SUCCESSFUL.getCode().equals(entity.getPreviousStatus());
    this.updateFieldOpt(onlyAdd, hiveRequest);
    LOGGER.info("success to update sink of type={}", sinkType);
}
Also used : BusinessException(org.apache.inlong.manager.common.exceptions.BusinessException) HiveSinkRequest(org.apache.inlong.manager.common.pojo.sink.hive.HiveSinkRequest) HiveSinkDTO(org.apache.inlong.manager.common.pojo.sink.hive.HiveSinkDTO) StreamSinkEntity(org.apache.inlong.manager.dao.entity.StreamSinkEntity) BusinessException(org.apache.inlong.manager.common.exceptions.BusinessException) Date(java.util.Date)

Example 3 with HiveSinkRequest

use of org.apache.inlong.manager.common.pojo.sink.hive.HiveSinkRequest in project incubator-inlong by apache.

the class DisableZkForSortTest method createHiveSink.

public HiveSinkRequest createHiveSink(InlongStreamInfo streamInfo) {
    HiveSinkRequest hiveSinkRequest = new HiveSinkRequest();
    hiveSinkRequest.setInlongGroupId(streamInfo.getInlongGroupId());
    hiveSinkRequest.setSinkType("HIVE");
    hiveSinkRequest.setSinkName("HIVE");
    hiveSinkRequest.setInlongStreamId(streamInfo.getInlongStreamId());
    List<SinkFieldRequest> sinkFieldRequests = createStreamFields(streamInfo.getInlongGroupId(), streamInfo.getInlongStreamId()).stream().map(streamFieldInfo -> {
        SinkFieldRequest fieldInfo = new SinkFieldRequest();
        fieldInfo.setFieldName(streamFieldInfo.getFieldName());
        fieldInfo.setFieldType(streamFieldInfo.getFieldType());
        fieldInfo.setFieldComment(streamFieldInfo.getFieldComment());
        return fieldInfo;
    }).collect(Collectors.toList());
    hiveSinkRequest.setFieldList(sinkFieldRequests);
    hiveSinkRequest.setEnableCreateTable(0);
    hiveSinkRequest.setUsername(OPERATOR);
    hiveSinkRequest.setPassword("password");
    hiveSinkRequest.setDbName("default");
    hiveSinkRequest.setTableName("kip_test");
    hiveSinkRequest.setJdbcUrl("jdbc:hive2://172.17.12.135:7001");
    hiveSinkRequest.setFileFormat("TextFile");
    hiveSinkRequest.setHdfsDefaultFs("hdfs://172.17.12.235:4007");
    hiveSinkRequest.setWarehouseDir("/user/hive/warehouse");
    hiveSinkRequest.setFileFormat(StandardCharsets.UTF_8.name());
    hiveSinkRequest.setDataSeparator("124");
    streamSinkService.save(hiveSinkRequest, OPERATOR);
    return hiveSinkRequest;
}
Also used : ProcessStatus(org.apache.inlong.manager.common.enums.ProcessStatus) ProcessForm(org.apache.inlong.manager.common.pojo.workflow.form.ProcessForm) InlongGroupInfo(org.apache.inlong.manager.common.pojo.group.InlongGroupInfo) TaskEventListener(org.apache.inlong.manager.workflow.event.task.TaskEventListener) ServiceTask(org.apache.inlong.manager.workflow.definition.ServiceTask) Autowired(org.springframework.beans.factory.annotation.Autowired) InlongStreamInfo(org.apache.inlong.manager.common.pojo.stream.InlongStreamInfo) WorkflowServiceImplTest(org.apache.inlong.manager.service.workflow.WorkflowServiceImplTest) GroupResourceProcessForm(org.apache.inlong.manager.common.pojo.workflow.form.GroupResourceProcessForm) Lists(com.google.common.collect.Lists) StreamSinkService(org.apache.inlong.manager.service.sink.StreamSinkService) MockPlugin(org.apache.inlong.manager.service.mocks.MockPlugin) KafkaSourceRequest(org.apache.inlong.manager.common.pojo.source.kafka.KafkaSourceRequest) OperateType(org.apache.inlong.manager.common.pojo.workflow.form.UpdateGroupProcessForm.OperateType) StreamSourceService(org.apache.inlong.manager.service.source.StreamSourceService) WorkflowContext(org.apache.inlong.manager.workflow.WorkflowContext) ProcessName(org.apache.inlong.manager.service.workflow.ProcessName) ProcessResponse(org.apache.inlong.manager.common.pojo.workflow.ProcessResponse) GroupState(org.apache.inlong.manager.common.enums.GroupState) WorkflowBeanUtils(org.apache.inlong.manager.workflow.util.WorkflowBeanUtils) Test(org.junit.Test) Collectors(java.util.stream.Collectors) StandardCharsets(java.nio.charset.StandardCharsets) WorkflowProcess(org.apache.inlong.manager.workflow.definition.WorkflowProcess) InlongStreamService(org.apache.inlong.manager.service.core.InlongStreamService) List(java.util.List) WorkflowTask(org.apache.inlong.manager.workflow.definition.WorkflowTask) UpdateGroupProcessForm(org.apache.inlong.manager.common.pojo.workflow.form.UpdateGroupProcessForm) HiveSinkRequest(org.apache.inlong.manager.common.pojo.sink.hive.HiveSinkRequest) SinkFieldRequest(org.apache.inlong.manager.common.pojo.sink.SinkFieldRequest) WorkflowResult(org.apache.inlong.manager.common.pojo.workflow.WorkflowResult) Assert(org.junit.Assert) HiveSinkRequest(org.apache.inlong.manager.common.pojo.sink.hive.HiveSinkRequest) SinkFieldRequest(org.apache.inlong.manager.common.pojo.sink.SinkFieldRequest)

Example 4 with HiveSinkRequest

use of org.apache.inlong.manager.common.pojo.sink.hive.HiveSinkRequest in project incubator-inlong by apache.

the class HiveStreamSinkServiceTest method testGetAndUpdate.

@Test
public void testGetAndUpdate() {
    Integer id = this.saveSink();
    SinkResponse response = sinkService.get(id, Constant.SINK_HIVE);
    Assert.assertEquals(globalGroupId, response.getInlongGroupId());
    HiveSinkResponse hiveResponse = (HiveSinkResponse) response;
    hiveResponse.setEnableCreateResource(Constant.DISABLE_CREATE_RESOURCE);
    HiveSinkRequest request = CommonBeanUtils.copyProperties(hiveResponse, HiveSinkRequest::new);
    boolean result = sinkService.update(request, globalOperator);
    Assert.assertTrue(result);
}
Also used : SinkResponse(org.apache.inlong.manager.common.pojo.sink.SinkResponse) HiveSinkResponse(org.apache.inlong.manager.common.pojo.sink.hive.HiveSinkResponse) HiveSinkResponse(org.apache.inlong.manager.common.pojo.sink.hive.HiveSinkResponse) HiveSinkRequest(org.apache.inlong.manager.common.pojo.sink.hive.HiveSinkRequest) ServiceBaseTest(org.apache.inlong.manager.service.ServiceBaseTest) InlongStreamServiceTest(org.apache.inlong.manager.service.core.impl.InlongStreamServiceTest) Test(org.junit.Test)

Example 5 with HiveSinkRequest

use of org.apache.inlong.manager.common.pojo.sink.hive.HiveSinkRequest in project incubator-inlong by apache.

the class HiveStreamSinkServiceTest method saveSink.

public Integer saveSink() {
    streamServiceTest.saveInlongStream(globalGroupId, globalStreamId, globalOperator);
    HiveSinkRequest sinkInfo = new HiveSinkRequest();
    sinkInfo.setInlongGroupId(globalGroupId);
    sinkInfo.setInlongStreamId(globalStreamId);
    sinkInfo.setSinkType(Constant.SINK_HIVE);
    sinkInfo.setEnableCreateResource(Constant.DISABLE_CREATE_RESOURCE);
    sinkInfo.setSinkName(sinkName);
    return sinkService.save(sinkInfo, globalOperator);
}
Also used : HiveSinkRequest(org.apache.inlong.manager.common.pojo.sink.hive.HiveSinkRequest)

Aggregations

HiveSinkRequest (org.apache.inlong.manager.common.pojo.sink.hive.HiveSinkRequest)6 Date (java.util.Date)2 BusinessException (org.apache.inlong.manager.common.exceptions.BusinessException)2 SinkFieldRequest (org.apache.inlong.manager.common.pojo.sink.SinkFieldRequest)2 HiveSinkDTO (org.apache.inlong.manager.common.pojo.sink.hive.HiveSinkDTO)2 StreamSinkEntity (org.apache.inlong.manager.dao.entity.StreamSinkEntity)2 Test (org.junit.Test)2 Lists (com.google.common.collect.Lists)1 StandardCharsets (java.nio.charset.StandardCharsets)1 List (java.util.List)1 Collectors (java.util.stream.Collectors)1 DefaultAuthentication (org.apache.inlong.manager.client.api.auth.DefaultAuthentication)1 HiveSink (org.apache.inlong.manager.client.api.sink.HiveSink)1 GroupState (org.apache.inlong.manager.common.enums.GroupState)1 ProcessStatus (org.apache.inlong.manager.common.enums.ProcessStatus)1 InlongGroupInfo (org.apache.inlong.manager.common.pojo.group.InlongGroupInfo)1 SinkResponse (org.apache.inlong.manager.common.pojo.sink.SinkResponse)1 HiveSinkResponse (org.apache.inlong.manager.common.pojo.sink.hive.HiveSinkResponse)1 KafkaSourceRequest (org.apache.inlong.manager.common.pojo.source.kafka.KafkaSourceRequest)1 InlongStreamInfo (org.apache.inlong.manager.common.pojo.stream.InlongStreamInfo)1