Search in sources :

Example 1 with BinlogSourceResponse

use of org.apache.inlong.manager.common.pojo.source.binlog.BinlogSourceResponse in project incubator-inlong by apache.

the class StreamSourceServiceTest method testGetAndUpdate.

@Test
public void testGetAndUpdate() {
    Integer id = this.saveSource();
    SourceResponse response = sourceService.get(id, Constant.SOURCE_BINLOG);
    Assert.assertEquals(globalGroupId, response.getInlongGroupId());
    BinlogSourceResponse binlogResponse = (BinlogSourceResponse) response;
    BinlogSourceRequest request = CommonBeanUtils.copyProperties(binlogResponse, BinlogSourceRequest::new);
    boolean result = sourceService.update(request, globalOperator);
    Assert.assertTrue(result);
    sourceService.delete(id, Constant.SOURCE_BINLOG, globalOperator);
}
Also used : BinlogSourceResponse(org.apache.inlong.manager.common.pojo.source.binlog.BinlogSourceResponse) SourceResponse(org.apache.inlong.manager.common.pojo.source.SourceResponse) BinlogSourceResponse(org.apache.inlong.manager.common.pojo.source.binlog.BinlogSourceResponse) BinlogSourceRequest(org.apache.inlong.manager.common.pojo.source.binlog.BinlogSourceRequest) ServiceBaseTest(org.apache.inlong.manager.service.ServiceBaseTest) InlongStreamServiceTest(org.apache.inlong.manager.service.core.impl.InlongStreamServiceTest) Test(org.junit.Test)

Example 2 with BinlogSourceResponse

use of org.apache.inlong.manager.common.pojo.source.binlog.BinlogSourceResponse in project incubator-inlong by apache.

the class SerializationUtils method serializeForKafka.

/**
 * Get serialization info for Kafka
 */
private static SerializationInfo serializeForKafka(SourceResponse sourceResponse, KafkaSinkResponse sinkResponse) {
    String serializationType = sinkResponse.getSerializationType();
    DataTypeEnum dataType = DataTypeEnum.forName(serializationType);
    switch(dataType) {
        case AVRO:
            return new AvroSerializationInfo();
        case JSON:
            return new JsonSerializationInfo();
        case CANAL:
            return new CanalSerializationInfo();
        case DEBEZIUM_JSON:
            Assert.isInstanceOf(BinlogSourceResponse.class, sourceResponse, "Unsupported serializationType for Kafka");
            BinlogSourceResponse binlogSource = (BinlogSourceResponse) sourceResponse;
            return new DebeziumSerializationInfo(binlogSource.getTimestampFormatStandard(), "FAIL", "", false);
        default:
            throw new IllegalArgumentException(String.format("Unsupported serializationType for Kafka sink: %s", serializationType));
    }
}
Also used : DebeziumSerializationInfo(org.apache.inlong.sort.protocol.serialization.DebeziumSerializationInfo) DataTypeEnum(org.apache.inlong.common.enums.DataTypeEnum) CanalSerializationInfo(org.apache.inlong.sort.protocol.serialization.CanalSerializationInfo) BinlogSourceResponse(org.apache.inlong.manager.common.pojo.source.binlog.BinlogSourceResponse) AvroSerializationInfo(org.apache.inlong.sort.protocol.serialization.AvroSerializationInfo) JsonSerializationInfo(org.apache.inlong.sort.protocol.serialization.JsonSerializationInfo)

Example 3 with BinlogSourceResponse

use of org.apache.inlong.manager.common.pojo.source.binlog.BinlogSourceResponse in project incubator-inlong by apache.

the class InlongParser method parseStreamList.

public static List<FullStreamResponse> parseStreamList(Response response) {
    Object data = response.getData();
    JsonObject pageInfoJson = GsonUtil.fromJson(GsonUtil.toJson(data), JsonObject.class);
    JsonArray fullStreamArray = pageInfoJson.getAsJsonArray("list");
    List<FullStreamResponse> list = Lists.newArrayList();
    for (int i = 0; i < fullStreamArray.size(); i++) {
        JsonObject fullStreamJson = (JsonObject) fullStreamArray.get(i);
        FullStreamResponse fullStreamResponse = GsonUtil.fromJson(fullStreamJson.toString(), FullStreamResponse.class);
        list.add(fullStreamResponse);
        // Parse sourceResponse in each stream
        JsonArray sourceJsonArr = fullStreamJson.getAsJsonArray(SOURCE_INFO);
        List<SourceResponse> sourceResponses = Lists.newArrayList();
        fullStreamResponse.setSourceInfo(sourceResponses);
        for (int j = 0; j < sourceJsonArr.size(); j++) {
            JsonObject sourceJson = (JsonObject) sourceJsonArr.get(i);
            String type = sourceJson.get(SOURCE_TYPE).getAsString();
            SourceType sourceType = SourceType.forType(type);
            switch(sourceType) {
                case BINLOG:
                    BinlogSourceResponse binlogSourceResponse = GsonUtil.fromJson(sourceJson.toString(), BinlogSourceResponse.class);
                    sourceResponses.add(binlogSourceResponse);
                    break;
                case KAFKA:
                    KafkaSourceResponse kafkaSourceResponse = GsonUtil.fromJson(sourceJson.toString(), KafkaSourceResponse.class);
                    sourceResponses.add(kafkaSourceResponse);
                    break;
                default:
                    throw new RuntimeException(String.format("Unsupport sourceType=%s for Inlong", sourceType));
            }
        }
        // Parse sinkResponse in each stream
        JsonArray sinkJsonArr = fullStreamJson.getAsJsonArray(SINK_INFO);
        List<SinkResponse> sinkResponses = Lists.newArrayList();
        fullStreamResponse.setSinkInfo(sinkResponses);
        for (int j = 0; j < sinkJsonArr.size(); j++) {
            JsonObject sinkJson = (JsonObject) sinkJsonArr.get(i);
            String type = sinkJson.get(SINK_TYPE).getAsString();
            SinkType sinkType = SinkType.forType(type);
            switch(sinkType) {
                case HIVE:
                    HiveSinkResponse hiveSinkResponse = GsonUtil.fromJson(sinkJson.toString(), HiveSinkResponse.class);
                    sinkResponses.add(hiveSinkResponse);
                    break;
                case KAFKA:
                    KafkaSinkResponse kafkaSinkResponse = GsonUtil.fromJson(sinkJson.toString(), KafkaSinkResponse.class);
                    sinkResponses.add(kafkaSinkResponse);
                    break;
                case ICEBERG:
                    IcebergSinkResponse icebergSinkResponse = GsonUtil.fromJson(sinkJson.toString(), IcebergSinkResponse.class);
                    sinkResponses.add(icebergSinkResponse);
                    break;
                case CLICKHOUSE:
                    ClickHouseSinkResponse clickHouseSinkResponse = GsonUtil.fromJson(sinkJson.toString(), ClickHouseSinkResponse.class);
                    sinkResponses.add(clickHouseSinkResponse);
                    break;
                default:
                    throw new RuntimeException(String.format("Unsupport sinkType=%s for Inlong", sinkType));
            }
        }
    }
    return list;
}
Also used : BinlogSourceResponse(org.apache.inlong.manager.common.pojo.source.binlog.BinlogSourceResponse) SourceResponse(org.apache.inlong.manager.common.pojo.source.SourceResponse) KafkaSourceResponse(org.apache.inlong.manager.common.pojo.source.kafka.KafkaSourceResponse) SourceType(org.apache.inlong.manager.common.enums.SourceType) JsonObject(com.google.gson.JsonObject) BinlogSourceResponse(org.apache.inlong.manager.common.pojo.source.binlog.BinlogSourceResponse) ClickHouseSinkResponse(org.apache.inlong.manager.common.pojo.sink.ck.ClickHouseSinkResponse) JsonArray(com.google.gson.JsonArray) KafkaSourceResponse(org.apache.inlong.manager.common.pojo.source.kafka.KafkaSourceResponse) ClickHouseSinkResponse(org.apache.inlong.manager.common.pojo.sink.ck.ClickHouseSinkResponse) KafkaSinkResponse(org.apache.inlong.manager.common.pojo.sink.kafka.KafkaSinkResponse) IcebergSinkResponse(org.apache.inlong.manager.common.pojo.sink.iceberg.IcebergSinkResponse) SinkResponse(org.apache.inlong.manager.common.pojo.sink.SinkResponse) HiveSinkResponse(org.apache.inlong.manager.common.pojo.sink.hive.HiveSinkResponse) KafkaSinkResponse(org.apache.inlong.manager.common.pojo.sink.kafka.KafkaSinkResponse) HiveSinkResponse(org.apache.inlong.manager.common.pojo.sink.hive.HiveSinkResponse) FullStreamResponse(org.apache.inlong.manager.common.pojo.stream.FullStreamResponse) JsonObject(com.google.gson.JsonObject) SinkType(org.apache.inlong.manager.common.enums.SinkType) IcebergSinkResponse(org.apache.inlong.manager.common.pojo.sink.iceberg.IcebergSinkResponse)

Example 4 with BinlogSourceResponse

use of org.apache.inlong.manager.common.pojo.source.binlog.BinlogSourceResponse in project incubator-inlong by apache.

the class InlongStreamSourceTransfer method parseStreamSource.

public static StreamSource parseStreamSource(SourceResponse sourceResponse) {
    String type = sourceResponse.getSourceType();
    SourceType sourceType = SourceType.forType(type);
    if (sourceType == SourceType.KAFKA && sourceResponse instanceof KafkaSourceResponse) {
        return parseKafkaSource((KafkaSourceResponse) sourceResponse);
    }
    if (sourceType == SourceType.BINLOG && sourceResponse instanceof BinlogSourceResponse) {
        return parseMySQLBinlogSource((BinlogSourceResponse) sourceResponse);
    }
    throw new IllegalArgumentException(String.format("Unsupported source type : %s for Inlong", sourceType));
}
Also used : KafkaSourceResponse(org.apache.inlong.manager.common.pojo.source.kafka.KafkaSourceResponse) SourceType(org.apache.inlong.manager.common.enums.SourceType) BinlogSourceResponse(org.apache.inlong.manager.common.pojo.source.binlog.BinlogSourceResponse)

Aggregations

BinlogSourceResponse (org.apache.inlong.manager.common.pojo.source.binlog.BinlogSourceResponse)4 SourceType (org.apache.inlong.manager.common.enums.SourceType)2 SourceResponse (org.apache.inlong.manager.common.pojo.source.SourceResponse)2 KafkaSourceResponse (org.apache.inlong.manager.common.pojo.source.kafka.KafkaSourceResponse)2 JsonArray (com.google.gson.JsonArray)1 JsonObject (com.google.gson.JsonObject)1 DataTypeEnum (org.apache.inlong.common.enums.DataTypeEnum)1 SinkType (org.apache.inlong.manager.common.enums.SinkType)1 SinkResponse (org.apache.inlong.manager.common.pojo.sink.SinkResponse)1 ClickHouseSinkResponse (org.apache.inlong.manager.common.pojo.sink.ck.ClickHouseSinkResponse)1 HiveSinkResponse (org.apache.inlong.manager.common.pojo.sink.hive.HiveSinkResponse)1 IcebergSinkResponse (org.apache.inlong.manager.common.pojo.sink.iceberg.IcebergSinkResponse)1 KafkaSinkResponse (org.apache.inlong.manager.common.pojo.sink.kafka.KafkaSinkResponse)1 BinlogSourceRequest (org.apache.inlong.manager.common.pojo.source.binlog.BinlogSourceRequest)1 FullStreamResponse (org.apache.inlong.manager.common.pojo.stream.FullStreamResponse)1 ServiceBaseTest (org.apache.inlong.manager.service.ServiceBaseTest)1 InlongStreamServiceTest (org.apache.inlong.manager.service.core.impl.InlongStreamServiceTest)1 AvroSerializationInfo (org.apache.inlong.sort.protocol.serialization.AvroSerializationInfo)1 CanalSerializationInfo (org.apache.inlong.sort.protocol.serialization.CanalSerializationInfo)1 DebeziumSerializationInfo (org.apache.inlong.sort.protocol.serialization.DebeziumSerializationInfo)1