Search in sources :

Example 1 with DataFlowInfo

use of org.apache.inlong.sort.protocol.DataFlowInfo in project incubator-inlong by apache.

the class CreateSortConfigListener method listen.

@Override
public ListenerResult listen(WorkflowContext context) throws Exception {
    LOGGER.info("Create sort config for context={}", context);
    ProcessForm form = context.getProcessForm();
    if (form instanceof UpdateGroupProcessForm) {
        UpdateGroupProcessForm updateGroupProcessForm = (UpdateGroupProcessForm) form;
        OperateType operateType = updateGroupProcessForm.getOperateType();
        if (operateType == OperateType.SUSPEND || operateType == OperateType.DELETE) {
            return ListenerResult.success();
        }
    }
    InlongGroupInfo groupInfo = this.getGroupInfo(form);
    String groupId = groupInfo.getInlongGroupId();
    if (StringUtils.isEmpty(groupId)) {
        LOGGER.warn("GroupId is null for context={}", context);
        return ListenerResult.success();
    }
    List<SinkResponse> sinkResponseList = streamSinkService.listSink(groupId, null);
    if (CollectionUtils.isEmpty(sinkResponseList)) {
        LOGGER.warn("Sink not found by groupId={}", groupId);
        return ListenerResult.success();
    }
    Map<String, DataFlowInfo> dataFlowInfoMap = sinkResponseList.stream().map(sink -> {
        DataFlowInfo flowInfo = commonOperateService.createDataFlow(groupInfo, sink);
        return Pair.of(sink.getInlongStreamId(), flowInfo);
    }).collect(Collectors.toMap(Pair::getKey, Pair::getValue));
    String dataFlows = OBJECT_MAPPER.writeValueAsString(dataFlowInfoMap);
    InlongGroupExtInfo extInfo = new InlongGroupExtInfo();
    extInfo.setInlongGroupId(groupId);
    extInfo.setKeyName(InlongGroupSettings.DATA_FLOW);
    extInfo.setKeyValue(dataFlows);
    if (groupInfo.getExtList() == null) {
        groupInfo.setExtList(Lists.newArrayList());
    }
    upsertDataFlow(groupInfo, extInfo);
    return ListenerResult.success();
}
Also used : ProcessForm(org.apache.inlong.manager.common.pojo.workflow.form.ProcessForm) InlongGroupInfo(org.apache.inlong.manager.common.pojo.group.InlongGroupInfo) LoggerFactory(org.slf4j.LoggerFactory) Autowired(org.springframework.beans.factory.annotation.Autowired) StringUtils(org.apache.commons.lang3.StringUtils) GroupResourceProcessForm(org.apache.inlong.manager.common.pojo.workflow.form.GroupResourceProcessForm) Lists(com.google.common.collect.Lists) Pair(org.apache.commons.lang3.tuple.Pair) CollectionUtils(org.apache.commons.collections.CollectionUtils) CommonOperateService(org.apache.inlong.manager.service.CommonOperateService) StreamSinkService(org.apache.inlong.manager.service.sink.StreamSinkService) TaskEvent(org.apache.inlong.manager.workflow.event.task.TaskEvent) ObjectMapper(org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper) Map(java.util.Map) OperateType(org.apache.inlong.manager.common.pojo.workflow.form.UpdateGroupProcessForm.OperateType) WorkflowContext(org.apache.inlong.manager.workflow.WorkflowContext) InlongGroupSettings(org.apache.inlong.manager.common.settings.InlongGroupSettings) Logger(org.slf4j.Logger) SinkResponse(org.apache.inlong.manager.common.pojo.sink.SinkResponse) Collectors(java.util.stream.Collectors) WorkflowListenerException(org.apache.inlong.manager.common.exceptions.WorkflowListenerException) ListenerResult(org.apache.inlong.manager.workflow.event.ListenerResult) Component(org.springframework.stereotype.Component) List(java.util.List) UpdateGroupProcessForm(org.apache.inlong.manager.common.pojo.workflow.form.UpdateGroupProcessForm) DataFlowInfo(org.apache.inlong.sort.protocol.DataFlowInfo) InlongGroupExtInfo(org.apache.inlong.manager.common.pojo.group.InlongGroupExtInfo) SortOperateListener(org.apache.inlong.manager.workflow.event.task.SortOperateListener) UpdateGroupProcessForm(org.apache.inlong.manager.common.pojo.workflow.form.UpdateGroupProcessForm) SinkResponse(org.apache.inlong.manager.common.pojo.sink.SinkResponse) ProcessForm(org.apache.inlong.manager.common.pojo.workflow.form.ProcessForm) GroupResourceProcessForm(org.apache.inlong.manager.common.pojo.workflow.form.GroupResourceProcessForm) UpdateGroupProcessForm(org.apache.inlong.manager.common.pojo.workflow.form.UpdateGroupProcessForm) InlongGroupInfo(org.apache.inlong.manager.common.pojo.group.InlongGroupInfo) OperateType(org.apache.inlong.manager.common.pojo.workflow.form.UpdateGroupProcessForm.OperateType) InlongGroupExtInfo(org.apache.inlong.manager.common.pojo.group.InlongGroupExtInfo) DataFlowInfo(org.apache.inlong.sort.protocol.DataFlowInfo)

Example 2 with DataFlowInfo

use of org.apache.inlong.sort.protocol.DataFlowInfo in project incubator-inlong by apache.

the class MultiTenancyInLongMsgMixedDeserializerTest method testIsInLongMsgDataFlow.

@Test
public void testIsInLongMsgDataFlow() {
    final TubeSourceInfo tubeSourceInfo = new TubeSourceInfo("topic", "address", null, new InLongMsgCsvDeserializationInfo("tid", ',', false), new FieldInfo[0]);
    final EmptySinkInfo sinkInfo = new EmptySinkInfo();
    final DataFlowInfo dataFlowInfo = new DataFlowInfo(1L, tubeSourceInfo, sinkInfo);
    assertTrue(MultiTenancyInLongMsgMixedDeserializer.isInLongMsgDataFlow(dataFlowInfo));
    final DataFlowInfo nonInLongMsgDataFlow = new DataFlowInfo(2L, new EmptySourceInfo(), sinkInfo);
    assertFalse(MultiTenancyInLongMsgMixedDeserializer.isInLongMsgDataFlow(nonInLongMsgDataFlow));
}
Also used : EmptySinkInfo(org.apache.inlong.sort.util.TestingUtils.EmptySinkInfo) InLongMsgCsvDeserializationInfo(org.apache.inlong.sort.protocol.deserialization.InLongMsgCsvDeserializationInfo) TubeSourceInfo(org.apache.inlong.sort.protocol.source.TubeSourceInfo) EmptySourceInfo(org.apache.inlong.sort.util.TestingUtils.EmptySourceInfo) DataFlowInfo(org.apache.inlong.sort.protocol.DataFlowInfo) Test(org.junit.Test)

Example 3 with DataFlowInfo

use of org.apache.inlong.sort.protocol.DataFlowInfo in project incubator-inlong by apache.

the class RecordTransformerTest method testRecordMatchSerializer.

@Test
public void testRecordMatchSerializer() throws Exception {
    final int bufferSize = 1024;
    final RecordTransformer transformer = new RecordTransformer(bufferSize);
    final FieldInfo field1 = new FieldInfo("field1", new LongFormatInfo());
    final FieldInfo field2 = new FieldInfo("field2", new StringFormatInfo());
    final TestingSinkInfo sinkInfo = new TestingSinkInfo(new FieldInfo[] { field1, field2 });
    final DataFlowInfo dataFlowInfo = new DataFlowInfo(1L, new EmptySourceInfo(), sinkInfo);
    transformer.addDataFlow(dataFlowInfo);
    Map<Long, RowSerializer> rowSerializers = transformer.getRowSerializers();
    final Row row = new Row(2);
    row.setField(0, 1024L);
    row.setField(1, "9527");
    final Record record = new Record(1L, System.currentTimeMillis(), row);
    assertSame(record, transformer.matchRecordAndSerializerField(record, rowSerializers.get(1L)));
}
Also used : RowSerializer(org.apache.flink.api.java.typeutils.runtime.RowSerializer) SerializedRecord(org.apache.inlong.sort.flink.SerializedRecord) Record(org.apache.inlong.sort.flink.Record) LongFormatInfo(org.apache.inlong.sort.formats.common.LongFormatInfo) TestingSinkInfo(org.apache.inlong.sort.util.TestingUtils.TestingSinkInfo) Row(org.apache.flink.types.Row) EmptySourceInfo(org.apache.inlong.sort.util.TestingUtils.EmptySourceInfo) StringFormatInfo(org.apache.inlong.sort.formats.common.StringFormatInfo) FieldInfo(org.apache.inlong.sort.protocol.FieldInfo) DataFlowInfo(org.apache.inlong.sort.protocol.DataFlowInfo) Test(org.junit.Test)

Example 4 with DataFlowInfo

use of org.apache.inlong.sort.protocol.DataFlowInfo in project incubator-inlong by apache.

the class RecordTransformerTest method testRecordNotMatchSerializer.

@Test
public void testRecordNotMatchSerializer() throws Exception {
    final int bufferSize = 1024;
    final RecordTransformer transformer = new RecordTransformer(bufferSize);
    final FieldInfo field1 = new FieldInfo("field1", new LongFormatInfo());
    final FieldInfo field2 = new FieldInfo("field2", new StringFormatInfo());
    final TestingSinkInfo sinkInfo = new TestingSinkInfo(new FieldInfo[] { field1, field2 });
    final DataFlowInfo dataFlowInfo = new DataFlowInfo(1L, new EmptySourceInfo(), sinkInfo);
    transformer.addDataFlow(dataFlowInfo);
    Map<Long, RowSerializer> rowSerializers = transformer.getRowSerializers();
    final Row oneFieldRow = new Row(1);
    oneFieldRow.setField(0, 1024L);
    final Record oneFieldRecord = new Record(1L, System.currentTimeMillis(), oneFieldRow);
    assertEquals(2, transformer.matchRecordAndSerializerField(oneFieldRecord, rowSerializers.get(1L)).getRow().getArity());
    final Row threeFieldRow = new Row(3);
    threeFieldRow.setField(0, 1024L);
    threeFieldRow.setField(1, "9527");
    threeFieldRow.setField(2, 2048);
    final Record threeFieldRecord = new Record(1L, System.currentTimeMillis(), threeFieldRow);
    assertEquals(2, transformer.matchRecordAndSerializerField(threeFieldRecord, rowSerializers.get(1L)).getRow().getArity());
}
Also used : RowSerializer(org.apache.flink.api.java.typeutils.runtime.RowSerializer) SerializedRecord(org.apache.inlong.sort.flink.SerializedRecord) Record(org.apache.inlong.sort.flink.Record) LongFormatInfo(org.apache.inlong.sort.formats.common.LongFormatInfo) TestingSinkInfo(org.apache.inlong.sort.util.TestingUtils.TestingSinkInfo) Row(org.apache.flink.types.Row) EmptySourceInfo(org.apache.inlong.sort.util.TestingUtils.EmptySourceInfo) StringFormatInfo(org.apache.inlong.sort.formats.common.StringFormatInfo) FieldInfo(org.apache.inlong.sort.protocol.FieldInfo) DataFlowInfo(org.apache.inlong.sort.protocol.DataFlowInfo) Test(org.junit.Test)

Example 5 with DataFlowInfo

use of org.apache.inlong.sort.protocol.DataFlowInfo in project incubator-inlong by apache.

the class RecordTransformerTest method testSerializerNotMatchRecord.

@Test
public void testSerializerNotMatchRecord() throws Exception {
    final Row row = new Row(2);
    row.setField(0, 1024L);
    row.setField(1, "9527");
    final Record record = new Record(1L, System.currentTimeMillis(), row);
    final int bufferSize = 1024;
    final RecordTransformer transformer = new RecordTransformer(bufferSize);
    final FieldInfo field1 = new FieldInfo("field1", new LongFormatInfo());
    final TestingSinkInfo sinkInfo = new TestingSinkInfo(new FieldInfo[] { field1 });
    final DataFlowInfo dataFlowInfo = new DataFlowInfo(1L, new EmptySourceInfo(), sinkInfo);
    transformer.addDataFlow(dataFlowInfo);
    Map<Long, RowSerializer> rowSerializers = transformer.getRowSerializers();
    assertEquals(1, transformer.matchRecordAndSerializerField(record, rowSerializers.get(1L)).getRow().getArity());
    final FieldInfo newField1 = new FieldInfo("field1", new LongFormatInfo());
    final FieldInfo newField2 = new FieldInfo("field2", new LongFormatInfo());
    final FieldInfo newField3 = new FieldInfo("field3", new LongFormatInfo());
    final TestingSinkInfo newSinkInfo = new TestingSinkInfo(new FieldInfo[] { newField1, newField2, newField3 });
    final DataFlowInfo newDataFlowInfo = new DataFlowInfo(1L, new EmptySourceInfo(), newSinkInfo);
    transformer.addDataFlow(newDataFlowInfo);
    Map<Long, RowSerializer> newRowSerializers = transformer.getRowSerializers();
    assertEquals(3, transformer.matchRecordAndSerializerField(record, newRowSerializers.get(1L)).getRow().getArity());
}
Also used : RowSerializer(org.apache.flink.api.java.typeutils.runtime.RowSerializer) SerializedRecord(org.apache.inlong.sort.flink.SerializedRecord) Record(org.apache.inlong.sort.flink.Record) Row(org.apache.flink.types.Row) LongFormatInfo(org.apache.inlong.sort.formats.common.LongFormatInfo) TestingSinkInfo(org.apache.inlong.sort.util.TestingUtils.TestingSinkInfo) EmptySourceInfo(org.apache.inlong.sort.util.TestingUtils.EmptySourceInfo) FieldInfo(org.apache.inlong.sort.protocol.FieldInfo) DataFlowInfo(org.apache.inlong.sort.protocol.DataFlowInfo) Test(org.junit.Test)

Aggregations

DataFlowInfo (org.apache.inlong.sort.protocol.DataFlowInfo)18 Test (org.junit.Test)11 FieldInfo (org.apache.inlong.sort.protocol.FieldInfo)10 Row (org.apache.flink.types.Row)8 Record (org.apache.inlong.sort.flink.Record)8 LongFormatInfo (org.apache.inlong.sort.formats.common.LongFormatInfo)8 TestingSinkInfo (org.apache.inlong.sort.util.TestingUtils.TestingSinkInfo)7 StringFormatInfo (org.apache.inlong.sort.formats.common.StringFormatInfo)6 EmptySourceInfo (org.apache.inlong.sort.util.TestingUtils.EmptySourceInfo)6 Configuration (org.apache.inlong.sort.configuration.Configuration)5 SerializedRecord (org.apache.inlong.sort.flink.SerializedRecord)5 HashMap (java.util.HashMap)3 RowSerializer (org.apache.flink.api.java.typeutils.runtime.RowSerializer)3 WorkflowListenerException (org.apache.inlong.manager.common.exceptions.WorkflowListenerException)3 SinkInfo (org.apache.inlong.sort.protocol.sink.SinkInfo)3 SourceInfo (org.apache.inlong.sort.protocol.source.SourceInfo)3 Timestamp (java.sql.Timestamp)2 InlongGroupInfo (org.apache.inlong.manager.common.pojo.group.InlongGroupInfo)2 SinkResponse (org.apache.inlong.manager.common.pojo.sink.SinkResponse)2 GroupResourceProcessForm (org.apache.inlong.manager.common.pojo.workflow.form.GroupResourceProcessForm)2