Search in sources :

Example 11 with CatalogManager

use of org.apache.flink.table.catalog.CatalogManager in project zeppelin by apache.

the class TableEnvFactory method createScalaFlinkStreamTableEnvironment.

public TableEnvironment createScalaFlinkStreamTableEnvironment(EnvironmentSettings settings, ClassLoader classLoader) {
    try {
        ImmutablePair<Object, Object> pair = flinkShims.createPlannerAndExecutor(classLoader, settings, senv.getJavaEnv(), oldPlannerStreamTableConfig, functionCatalog, catalogManager);
        Planner planner = (Planner) pair.left;
        Executor executor = (Executor) pair.right;
        Class clazz = Class.forName("org.apache.flink.table.api.bridge.scala.internal.StreamTableEnvironmentImpl");
        try {
            Constructor constructor = clazz.getConstructor(CatalogManager.class, ModuleManager.class, FunctionCatalog.class, TableConfig.class, org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.class, Planner.class, Executor.class, boolean.class);
            return (TableEnvironment) constructor.newInstance(oldPlannerCatalogManager, moduleManager, oldPlannerFunctionCatalog, oldPlannerStreamTableConfig, senv, planner, executor, settings.isStreamingMode());
        } catch (NoSuchMethodException e) {
            // Flink 1.11.1 change the constructor signature, FLINK-18419
            Constructor constructor = clazz.getConstructor(CatalogManager.class, ModuleManager.class, FunctionCatalog.class, TableConfig.class, org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.class, Planner.class, Executor.class, boolean.class, ClassLoader.class);
            return (TableEnvironment) constructor.newInstance(oldPlannerCatalogManager, moduleManager, oldPlannerFunctionCatalog, oldPlannerStreamTableConfig, senv, planner, executor, settings.isStreamingMode(), classLoader);
        }
    } catch (Exception e) {
        throw new TableException("Fail to createScalaFlinkStreamTableEnvironment", e);
    }
}
Also used : FunctionCatalog(org.apache.flink.table.catalog.FunctionCatalog) TableException(org.apache.flink.table.api.TableException) Constructor(java.lang.reflect.Constructor) TableEnvironment(org.apache.flink.table.api.TableEnvironment) ModuleManager(org.apache.flink.table.module.ModuleManager) CatalogManager(org.apache.flink.table.catalog.CatalogManager) TableException(org.apache.flink.table.api.TableException) Executor(org.apache.flink.table.delegation.Executor) TableConfig(org.apache.flink.table.api.TableConfig) Planner(org.apache.flink.table.delegation.Planner) StreamExecutionEnvironment(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment)

Example 12 with CatalogManager

use of org.apache.flink.table.catalog.CatalogManager in project zeppelin by apache.

the class TableEnvFactory method createJavaBlinkStreamTableEnvironment.

public TableEnvironment createJavaBlinkStreamTableEnvironment(EnvironmentSettings settings, ClassLoader classLoader) {
    try {
        ImmutablePair<Object, Object> pair = flinkShims.createPlannerAndExecutor(classLoader, settings, senv.getJavaEnv(), streamTableConfig, functionCatalog, catalogManager);
        Planner planner = (Planner) pair.left;
        Executor executor = (Executor) pair.right;
        Class clazz = Class.forName("org.apache.flink.table.api.bridge.java.internal.StreamTableEnvironmentImpl");
        try {
            Constructor constructor = clazz.getConstructor(CatalogManager.class, ModuleManager.class, FunctionCatalog.class, TableConfig.class, org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.class, Planner.class, Executor.class, boolean.class);
            return (TableEnvironment) constructor.newInstance(catalogManager, moduleManager, functionCatalog, streamTableConfig, senv.getJavaEnv(), planner, executor, settings.isStreamingMode());
        } catch (NoSuchMethodException e) {
            // Flink 1.11.1 change the constructor signature, FLINK-18419
            Constructor constructor = clazz.getConstructor(CatalogManager.class, ModuleManager.class, FunctionCatalog.class, TableConfig.class, org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.class, Planner.class, Executor.class, boolean.class, ClassLoader.class);
            return (TableEnvironment) constructor.newInstance(catalogManager, moduleManager, functionCatalog, streamTableConfig, senv.getJavaEnv(), planner, executor, settings.isStreamingMode(), classLoader);
        }
    } catch (Exception e) {
        throw new TableException("Fail to createJavaBlinkStreamTableEnvironment", e);
    }
}
Also used : FunctionCatalog(org.apache.flink.table.catalog.FunctionCatalog) TableException(org.apache.flink.table.api.TableException) Constructor(java.lang.reflect.Constructor) TableEnvironment(org.apache.flink.table.api.TableEnvironment) ModuleManager(org.apache.flink.table.module.ModuleManager) CatalogManager(org.apache.flink.table.catalog.CatalogManager) TableException(org.apache.flink.table.api.TableException) Executor(org.apache.flink.table.delegation.Executor) TableConfig(org.apache.flink.table.api.TableConfig) Planner(org.apache.flink.table.delegation.Planner) StreamExecutionEnvironment(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment)

Example 13 with CatalogManager

use of org.apache.flink.table.catalog.CatalogManager in project flink by apache.

the class CatalogManagerMocks method createCatalogManager.

public static CatalogManager createCatalogManager(@Nullable Catalog catalog) {
    final CatalogManager.Builder builder = preparedCatalogManager();
    if (catalog != null) {
        builder.defaultCatalog(DEFAULT_CATALOG, catalog);
    }
    final CatalogManager catalogManager = builder.build();
    catalogManager.initSchemaResolver(true, ExpressionResolverMocks.dummyResolver());
    return catalogManager;
}
Also used : CatalogManager(org.apache.flink.table.catalog.CatalogManager)

Example 14 with CatalogManager

use of org.apache.flink.table.catalog.CatalogManager in project flink by apache.

the class TableEnvironmentMock method getInstance.

private static TableEnvironmentMock getInstance(boolean isStreamingMode) {
    final TableConfig tableConfig = createTableConfig();
    final CatalogManager catalogManager = CatalogManagerMocks.createEmptyCatalogManager();
    final ModuleManager moduleManager = new ModuleManager();
    return new TableEnvironmentMock(catalogManager, moduleManager, tableConfig, createExecutor(), createFunctionCatalog(tableConfig, catalogManager, moduleManager), createPlanner(), isStreamingMode);
}
Also used : TableConfig(org.apache.flink.table.api.TableConfig) ModuleManager(org.apache.flink.table.module.ModuleManager) CatalogManager(org.apache.flink.table.catalog.CatalogManager)

Example 15 with CatalogManager

use of org.apache.flink.table.catalog.CatalogManager in project flink by apache.

the class AbstractStreamTableEnvironmentImpl method toStreamInternal.

protected <T> DataStream<T> toStreamInternal(Table table, SchemaTranslator.ProducingResult schemaTranslationResult, @Nullable ChangelogMode changelogMode) {
    final CatalogManager catalogManager = getCatalogManager();
    final OperationTreeBuilder operationTreeBuilder = getOperationTreeBuilder();
    final QueryOperation projectOperation = schemaTranslationResult.getProjections().map(projections -> operationTreeBuilder.project(projections.stream().map(ApiExpressionUtils::unresolvedRef).collect(Collectors.toList()), table.getQueryOperation())).orElseGet(table::getQueryOperation);
    final ResolvedCatalogTable resolvedCatalogTable = catalogManager.resolveCatalogTable(new ExternalCatalogTable(schemaTranslationResult.getSchema()));
    final ExternalModifyOperation modifyOperation = new ExternalModifyOperation(ContextResolvedTable.anonymous("datastream_sink", resolvedCatalogTable), projectOperation, changelogMode, schemaTranslationResult.getPhysicalDataType().orElseGet(() -> resolvedCatalogTable.getResolvedSchema().toPhysicalRowDataType()));
    return toStreamInternal(table, modifyOperation);
}
Also used : DataType(org.apache.flink.table.types.DataType) CatalogManager(org.apache.flink.table.catalog.CatalogManager) ModifyOperation(org.apache.flink.table.operations.ModifyOperation) QueryOperation(org.apache.flink.table.operations.QueryOperation) Schema(org.apache.flink.table.api.Schema) ObjectIdentifier(org.apache.flink.table.catalog.ObjectIdentifier) DataStreamQueryOperation(org.apache.flink.table.operations.DataStreamQueryOperation) Tuple2(org.apache.flink.api.java.tuple.Tuple2) UnresolvedIdentifier(org.apache.flink.table.catalog.UnresolvedIdentifier) ChangelogMode(org.apache.flink.table.connector.ChangelogMode) ExecutorFactory(org.apache.flink.table.delegation.ExecutorFactory) TupleTypeInfo(org.apache.flink.api.java.typeutils.TupleTypeInfo) Types(org.apache.flink.table.api.Types) FunctionCatalog(org.apache.flink.table.catalog.FunctionCatalog) Planner(org.apache.flink.table.delegation.Planner) ExternalQueryOperation(org.apache.flink.table.operations.ExternalQueryOperation) Expression(org.apache.flink.table.expressions.Expression) TableEnvironmentImpl(org.apache.flink.table.api.internal.TableEnvironmentImpl) OperationTreeBuilder(org.apache.flink.table.operations.utils.OperationTreeBuilder) TypeInformation(org.apache.flink.api.common.typeinfo.TypeInformation) ExternalModifyOperation(org.apache.flink.table.operations.ExternalModifyOperation) ResolvedCatalogTable(org.apache.flink.table.catalog.ResolvedCatalogTable) Nullable(javax.annotation.Nullable) ContextResolvedTable(org.apache.flink.table.catalog.ContextResolvedTable) TimeCharacteristic(org.apache.flink.streaming.api.TimeCharacteristic) SchemaTranslator(org.apache.flink.table.catalog.SchemaTranslator) ModuleManager(org.apache.flink.table.module.ModuleManager) TableConfig(org.apache.flink.table.api.TableConfig) TableException(org.apache.flink.table.api.TableException) Table(org.apache.flink.table.api.Table) StreamExecutorFactory(org.apache.flink.table.delegation.StreamExecutorFactory) Preconditions(org.apache.flink.util.Preconditions) Collectors(java.util.stream.Collectors) FieldInfoUtils(org.apache.flink.table.typeutils.FieldInfoUtils) DataStream(org.apache.flink.streaming.api.datastream.DataStream) ExternalCatalogTable(org.apache.flink.table.catalog.ExternalCatalogTable) List(java.util.List) TypeExtractor(org.apache.flink.api.java.typeutils.TypeExtractor) FactoryUtil(org.apache.flink.table.factories.FactoryUtil) ValidationException(org.apache.flink.table.api.ValidationException) Executor(org.apache.flink.table.delegation.Executor) ApiExpressionUtils(org.apache.flink.table.expressions.ApiExpressionUtils) Optional(java.util.Optional) Internal(org.apache.flink.annotation.Internal) TypeConversions(org.apache.flink.table.types.utils.TypeConversions) Transformation(org.apache.flink.api.dag.Transformation) Collections(java.util.Collections) StreamExecutionEnvironment(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment) ExternalModifyOperation(org.apache.flink.table.operations.ExternalModifyOperation) ResolvedCatalogTable(org.apache.flink.table.catalog.ResolvedCatalogTable) ExternalCatalogTable(org.apache.flink.table.catalog.ExternalCatalogTable) OperationTreeBuilder(org.apache.flink.table.operations.utils.OperationTreeBuilder) ApiExpressionUtils(org.apache.flink.table.expressions.ApiExpressionUtils) CatalogManager(org.apache.flink.table.catalog.CatalogManager) QueryOperation(org.apache.flink.table.operations.QueryOperation) DataStreamQueryOperation(org.apache.flink.table.operations.DataStreamQueryOperation) ExternalQueryOperation(org.apache.flink.table.operations.ExternalQueryOperation)

Aggregations

CatalogManager (org.apache.flink.table.catalog.CatalogManager)26 TableConfig (org.apache.flink.table.api.TableConfig)13 ModuleManager (org.apache.flink.table.module.ModuleManager)13 FunctionCatalog (org.apache.flink.table.catalog.FunctionCatalog)12 Executor (org.apache.flink.table.delegation.Executor)8 Planner (org.apache.flink.table.delegation.Planner)8 StreamExecutionEnvironment (org.apache.flink.streaming.api.environment.StreamExecutionEnvironment)6 TableException (org.apache.flink.table.api.TableException)6 GenericInMemoryCatalog (org.apache.flink.table.catalog.GenericInMemoryCatalog)6 ObjectIdentifier (org.apache.flink.table.catalog.ObjectIdentifier)6 ParameterizedTest (org.junit.jupiter.params.ParameterizedTest)6 Constructor (java.lang.reflect.Constructor)5 Configuration (org.apache.flink.configuration.Configuration)5 TableEnvironment (org.apache.flink.table.api.TableEnvironment)5 ValidationException (org.apache.flink.table.api.ValidationException)5 JsonSerdeTestUtil.configuredSerdeContext (org.apache.flink.table.planner.plan.nodes.exec.serde.JsonSerdeTestUtil.configuredSerdeContext)5 HashMap (java.util.HashMap)4 CatalogTable (org.apache.flink.table.catalog.CatalogTable)4 ContextResolvedTable (org.apache.flink.table.catalog.ContextResolvedTable)4 ResolvedCatalogTable (org.apache.flink.table.catalog.ResolvedCatalogTable)4