Examples with ChangelogMode org.apache.flink.table.connector.ChangelogMode used on opensource projects

Example 1 with ChangelogMode

use of org.apache.flink.table.connector.ChangelogMode in project flink by apache.

the class CsvFileFormatFactory method createEncodingFormat.

@Override
public EncodingFormat<Factory<RowData>> createEncodingFormat(DynamicTableFactory.Context context, ReadableConfig formatOptions) {
    return new EncodingFormat<BulkWriter.Factory<RowData>>() {

        @Override
        public BulkWriter.Factory<RowData> createRuntimeEncoder(DynamicTableSink.Context context, DataType physicalDataType) {
            final RowType rowType = (RowType) physicalDataType.getLogicalType();
            final CsvSchema schema = buildCsvSchema(rowType, formatOptions);
            final RowDataToCsvConverter converter = RowDataToCsvConverters.createRowConverter(rowType);
            final CsvMapper mapper = new CsvMapper();
            final ObjectNode container = mapper.createObjectNode();
            final RowDataToCsvConverter.RowDataToCsvFormatConverterContext converterContext = new RowDataToCsvConverter.RowDataToCsvFormatConverterContext(mapper, container);
            return out -> CsvBulkWriter.forSchema(mapper, schema, converter, converterContext, out);
        }

        @Override
        public ChangelogMode getChangelogMode() {
            return ChangelogMode.insertOnly();
        }
    };
}

Example 2 with ChangelogMode

use of org.apache.flink.table.connector.ChangelogMode in project flink by apache.

the class TestManagedTableFactory method createDynamicTableSource.

@Override
public DynamicTableSource createDynamicTableSource(Context context) {
    FactoryUtil.TableFactoryHelper helper = FactoryUtil.createTableFactoryHelper(this, context);
    ChangelogMode changelogMode = parseChangelogMode(helper.getOptions().get(CHANGELOG_MODE));
    CompactPartitions compactPartitions = deserializeCompactPartitions(context.getCatalogTable().getOptions().getOrDefault(COMPACT_FILE_ENTRIES.key(), "")).orElse(CompactPartitions.from(Collections.emptyList()));
    return new TestManagedTableSource(context, compactPartitions, changelogMode);
}

Example 3 with ChangelogMode

use of org.apache.flink.table.connector.ChangelogMode in project flink by apache.

the class DynamicSourceUtils method validateScanSource.

private static void validateScanSource(String tableDebugName, ResolvedSchema schema, ScanTableSource scanSource, boolean isBatchMode, ReadableConfig config) {
    final ScanRuntimeProvider provider = scanSource.getScanRuntimeProvider(ScanRuntimeProviderContext.INSTANCE);
    final ChangelogMode changelogMode = scanSource.getChangelogMode();
    validateWatermarks(tableDebugName, schema);
    if (isBatchMode) {
        validateScanSourceForBatch(tableDebugName, changelogMode, provider);
    } else {
        validateScanSourceForStreaming(tableDebugName, schema, scanSource, changelogMode, config);
    }
}

Example 4 with ChangelogMode

use of org.apache.flink.table.connector.ChangelogMode in project flink by apache.

the class ChangelogModeJsonDeserializer method deserialize.

@Override
public ChangelogMode deserialize(JsonParser jsonParser, DeserializationContext deserializationContext) throws IOException {
    ChangelogMode.Builder builder = ChangelogMode.newBuilder();
    JsonNode rowKindsNode = jsonParser.readValueAsTree();
    for (JsonNode rowKindNode : rowKindsNode) {
        RowKind rowKind = RowKind.valueOf(rowKindNode.asText().toUpperCase());
        builder.addContainedKind(rowKind);
    }
    return builder.build();
}

Example 5 with ChangelogMode

use of org.apache.flink.table.connector.ChangelogMode in project flink by apache.

the class AbstractStreamTableEnvironmentImpl method toStreamInternal.

protected <T> DataStream<T> toStreamInternal(Table table, SchemaTranslator.ProducingResult schemaTranslationResult, @Nullable ChangelogMode changelogMode) {
    final CatalogManager catalogManager = getCatalogManager();
    final OperationTreeBuilder operationTreeBuilder = getOperationTreeBuilder();
    final QueryOperation projectOperation = schemaTranslationResult.getProjections().map(projections -> operationTreeBuilder.project(projections.stream().map(ApiExpressionUtils::unresolvedRef).collect(Collectors.toList()), table.getQueryOperation())).orElseGet(table::getQueryOperation);
    final ResolvedCatalogTable resolvedCatalogTable = catalogManager.resolveCatalogTable(new ExternalCatalogTable(schemaTranslationResult.getSchema()));
    final ExternalModifyOperation modifyOperation = new ExternalModifyOperation(ContextResolvedTable.anonymous("datastream_sink", resolvedCatalogTable), projectOperation, changelogMode, schemaTranslationResult.getPhysicalDataType().orElseGet(() -> resolvedCatalogTable.getResolvedSchema().toPhysicalRowDataType()));
    return toStreamInternal(table, modifyOperation);
}