Search in sources :

Example 1 with DataGenerator

use of org.apache.flink.streaming.api.functions.source.datagen.DataGenerator in project flink by apache.

the class DataGenTableSourceFactory method createDynamicTableSource.

@Override
public DynamicTableSource createDynamicTableSource(Context context) {
    Configuration options = new Configuration();
    context.getCatalogTable().getOptions().forEach(options::setString);
    DataType rowDataType = context.getPhysicalRowDataType();
    DataGenerator<?>[] fieldGenerators = new DataGenerator[DataType.getFieldCount(rowDataType)];
    Set<ConfigOption<?>> optionalOptions = new HashSet<>();
    List<String> fieldNames = DataType.getFieldNames(rowDataType);
    List<DataType> fieldDataTypes = DataType.getFieldDataTypes(rowDataType);
    for (int i = 0; i < fieldGenerators.length; i++) {
        String name = fieldNames.get(i);
        DataType type = fieldDataTypes.get(i);
        ConfigOption<String> kind = key(DataGenConnectorOptionsUtil.FIELDS + "." + name + "." + DataGenConnectorOptionsUtil.KIND).stringType().defaultValue(DataGenConnectorOptionsUtil.RANDOM);
        DataGeneratorContainer container = createContainer(name, type, options.get(kind), options);
        fieldGenerators[i] = container.getGenerator();
        optionalOptions.add(kind);
        optionalOptions.addAll(container.getOptions());
    }
    FactoryUtil.validateFactoryOptions(requiredOptions(), optionalOptions, options);
    Set<String> consumedOptionKeys = new HashSet<>();
    consumedOptionKeys.add(CONNECTOR.key());
    consumedOptionKeys.add(DataGenConnectorOptions.ROWS_PER_SECOND.key());
    consumedOptionKeys.add(DataGenConnectorOptions.NUMBER_OF_ROWS.key());
    optionalOptions.stream().map(ConfigOption::key).forEach(consumedOptionKeys::add);
    FactoryUtil.validateUnconsumedKeys(factoryIdentifier(), options.keySet(), consumedOptionKeys);
    String name = context.getObjectIdentifier().toString();
    return new DataGenTableSource(fieldGenerators, name, rowDataType, options.get(DataGenConnectorOptions.ROWS_PER_SECOND), options.get(DataGenConnectorOptions.NUMBER_OF_ROWS));
}
Also used : Configuration(org.apache.flink.configuration.Configuration) ConfigOption(org.apache.flink.configuration.ConfigOption) DataGenerator(org.apache.flink.streaming.api.functions.source.datagen.DataGenerator) DataType(org.apache.flink.table.types.DataType) HashSet(java.util.HashSet)

Example 2 with DataGenerator

use of org.apache.flink.streaming.api.functions.source.datagen.DataGenerator in project flink by apache.

the class RandomGeneratorVisitor method visit.

@Override
public DataGeneratorContainer visit(RowType rowType) {
    List<DataGeneratorContainer> fieldContainers = rowType.getFields().stream().map(field -> {
        String fieldName = name + "." + field.getName();
        return field.getType().accept(new RandomGeneratorVisitor(fieldName, config));
    }).collect(Collectors.toList());
    ConfigOption<?>[] options = fieldContainers.stream().flatMap(container -> container.getOptions().stream()).toArray(ConfigOption[]::new);
    DataGenerator[] generators = fieldContainers.stream().map(DataGeneratorContainer::getGenerator).toArray(DataGenerator[]::new);
    return DataGeneratorContainer.of(new RowDataGenerator(generators, rowType.getFieldNames()), options);
}
Also used : ConfigOptions(org.apache.flink.configuration.ConfigOptions) GenericArrayData(org.apache.flink.table.data.GenericArrayData) TinyIntType(org.apache.flink.table.types.logical.TinyIntType) IntType(org.apache.flink.table.types.logical.IntType) FloatType(org.apache.flink.table.types.logical.FloatType) MapType(org.apache.flink.table.types.logical.MapType) RowType(org.apache.flink.table.types.logical.RowType) CharType(org.apache.flink.table.types.logical.CharType) TimestampType(org.apache.flink.table.types.logical.TimestampType) DecimalType(org.apache.flink.table.types.logical.DecimalType) DoubleType(org.apache.flink.table.types.logical.DoubleType) ReadableConfig(org.apache.flink.configuration.ReadableConfig) BooleanType(org.apache.flink.table.types.logical.BooleanType) RowDataGenerator(org.apache.flink.connector.datagen.table.types.RowDataGenerator) DecimalDataRandomGenerator(org.apache.flink.connector.datagen.table.types.DecimalDataRandomGenerator) GenericMapData(org.apache.flink.table.data.GenericMapData) Duration(java.time.Duration) Map(java.util.Map) DataGeneratorMapper(org.apache.flink.connector.datagen.table.types.DataGeneratorMapper) RandomGenerator(org.apache.flink.streaming.api.functions.source.datagen.RandomGenerator) ConfigOption(org.apache.flink.configuration.ConfigOption) YearMonthIntervalType(org.apache.flink.table.types.logical.YearMonthIntervalType) MultisetType(org.apache.flink.table.types.logical.MultisetType) LocalZonedTimestampType(org.apache.flink.table.types.logical.LocalZonedTimestampType) DataGenerator(org.apache.flink.streaming.api.functions.source.datagen.DataGenerator) TimestampData(org.apache.flink.table.data.TimestampData) ConfigOptions.key(org.apache.flink.configuration.ConfigOptions.key) BigIntType(org.apache.flink.table.types.logical.BigIntType) DayTimeIntervalType(org.apache.flink.table.types.logical.DayTimeIntervalType) Set(java.util.Set) VarCharType(org.apache.flink.table.types.logical.VarCharType) ArrayType(org.apache.flink.table.types.logical.ArrayType) Collectors(java.util.stream.Collectors) SmallIntType(org.apache.flink.table.types.logical.SmallIntType) StringData(org.apache.flink.table.data.StringData) List(java.util.List) LogicalType(org.apache.flink.table.types.logical.LogicalType) ValidationException(org.apache.flink.table.api.ValidationException) Internal(org.apache.flink.annotation.Internal) ZonedTimestampType(org.apache.flink.table.types.logical.ZonedTimestampType) ConfigOption(org.apache.flink.configuration.ConfigOption) RowDataGenerator(org.apache.flink.connector.datagen.table.types.RowDataGenerator) DataGenerator(org.apache.flink.streaming.api.functions.source.datagen.DataGenerator) RowDataGenerator(org.apache.flink.connector.datagen.table.types.RowDataGenerator)

Aggregations

ConfigOption (org.apache.flink.configuration.ConfigOption)2 DataGenerator (org.apache.flink.streaming.api.functions.source.datagen.DataGenerator)2 Duration (java.time.Duration)1 HashSet (java.util.HashSet)1 List (java.util.List)1 Map (java.util.Map)1 Set (java.util.Set)1 Collectors (java.util.stream.Collectors)1 Internal (org.apache.flink.annotation.Internal)1 ConfigOptions (org.apache.flink.configuration.ConfigOptions)1 ConfigOptions.key (org.apache.flink.configuration.ConfigOptions.key)1 Configuration (org.apache.flink.configuration.Configuration)1 ReadableConfig (org.apache.flink.configuration.ReadableConfig)1 DataGeneratorMapper (org.apache.flink.connector.datagen.table.types.DataGeneratorMapper)1 DecimalDataRandomGenerator (org.apache.flink.connector.datagen.table.types.DecimalDataRandomGenerator)1 RowDataGenerator (org.apache.flink.connector.datagen.table.types.RowDataGenerator)1 RandomGenerator (org.apache.flink.streaming.api.functions.source.datagen.RandomGenerator)1 ValidationException (org.apache.flink.table.api.ValidationException)1 GenericArrayData (org.apache.flink.table.data.GenericArrayData)1 GenericMapData (org.apache.flink.table.data.GenericMapData)1