Search in sources :

Example 11 with Executor

use of org.apache.flink.table.delegation.Executor in project zeppelin by apache.

the class TableEnvFactory method createJavaBlinkStreamTableEnvironment.

public TableEnvironment createJavaBlinkStreamTableEnvironment(EnvironmentSettings settings, ClassLoader classLoader) {
    try {
        ImmutablePair<Object, Object> pair = flinkShims.createPlannerAndExecutor(classLoader, settings, senv.getJavaEnv(), streamTableConfig, functionCatalog, catalogManager);
        Planner planner = (Planner) pair.left;
        Executor executor = (Executor) pair.right;
        Class clazz = Class.forName("org.apache.flink.table.api.bridge.java.internal.StreamTableEnvironmentImpl");
        try {
            Constructor constructor = clazz.getConstructor(CatalogManager.class, ModuleManager.class, FunctionCatalog.class, TableConfig.class, org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.class, Planner.class, Executor.class, boolean.class);
            return (TableEnvironment) constructor.newInstance(catalogManager, moduleManager, functionCatalog, streamTableConfig, senv.getJavaEnv(), planner, executor, settings.isStreamingMode());
        } catch (NoSuchMethodException e) {
            // Flink 1.11.1 change the constructor signature, FLINK-18419
            Constructor constructor = clazz.getConstructor(CatalogManager.class, ModuleManager.class, FunctionCatalog.class, TableConfig.class, org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.class, Planner.class, Executor.class, boolean.class, ClassLoader.class);
            return (TableEnvironment) constructor.newInstance(catalogManager, moduleManager, functionCatalog, streamTableConfig, senv.getJavaEnv(), planner, executor, settings.isStreamingMode(), classLoader);
        }
    } catch (Exception e) {
        throw new TableException("Fail to createJavaBlinkStreamTableEnvironment", e);
    }
}
Also used : FunctionCatalog(org.apache.flink.table.catalog.FunctionCatalog) TableException(org.apache.flink.table.api.TableException) Constructor(java.lang.reflect.Constructor) TableEnvironment(org.apache.flink.table.api.TableEnvironment) ModuleManager(org.apache.flink.table.module.ModuleManager) CatalogManager(org.apache.flink.table.catalog.CatalogManager) TableException(org.apache.flink.table.api.TableException) Executor(org.apache.flink.table.delegation.Executor) TableConfig(org.apache.flink.table.api.TableConfig) Planner(org.apache.flink.table.delegation.Planner) StreamExecutionEnvironment(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment)

Example 12 with Executor

use of org.apache.flink.table.delegation.Executor in project flink by apache.

the class ExecutionContext method lookupExecutor.

private Executor lookupExecutor(String executorIdentifier, StreamExecutionEnvironment executionEnvironment) {
    try {
        final ExecutorFactory executorFactory = FactoryUtil.discoverFactory(classLoader, ExecutorFactory.class, executorIdentifier);
        final Method createMethod = executorFactory.getClass().getMethod("create", StreamExecutionEnvironment.class);
        return (Executor) createMethod.invoke(executorFactory, executionEnvironment);
    } catch (Exception e) {
        throw new TableException("Could not instantiate the executor. Make sure a planner module is on the classpath", e);
    }
}
Also used : TableException(org.apache.flink.table.api.TableException) Executor(org.apache.flink.table.delegation.Executor) ExecutorFactory(org.apache.flink.table.delegation.ExecutorFactory) Method(java.lang.reflect.Method) TableException(org.apache.flink.table.api.TableException)

Example 13 with Executor

use of org.apache.flink.table.delegation.Executor in project flink by apache.

the class TableEnvironmentImpl method create.

private static TableEnvironmentImpl create(EnvironmentSettings settings, Configuration configuration) {
    // temporary solution until FLINK-15635 is fixed
    final ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
    // use configuration to init table config
    final TableConfig tableConfig = new TableConfig();
    tableConfig.addConfiguration(configuration);
    final ModuleManager moduleManager = new ModuleManager();
    final CatalogManager catalogManager = CatalogManager.newBuilder().classLoader(classLoader).config(tableConfig.getConfiguration()).defaultCatalog(settings.getBuiltInCatalogName(), new GenericInMemoryCatalog(settings.getBuiltInCatalogName(), settings.getBuiltInDatabaseName())).build();
    final FunctionCatalog functionCatalog = new FunctionCatalog(tableConfig, catalogManager, moduleManager);
    final ExecutorFactory executorFactory = FactoryUtil.discoverFactory(classLoader, ExecutorFactory.class, settings.getExecutor());
    final Executor executor = executorFactory.create(configuration);
    final Planner planner = PlannerFactoryUtil.createPlanner(settings.getPlanner(), executor, tableConfig, moduleManager, catalogManager, functionCatalog);
    return new TableEnvironmentImpl(catalogManager, moduleManager, tableConfig, executor, functionCatalog, planner, settings.isStreamingMode(), classLoader);
}
Also used : FunctionCatalog(org.apache.flink.table.catalog.FunctionCatalog) Executor(org.apache.flink.table.delegation.Executor) ExecutorFactory(org.apache.flink.table.delegation.ExecutorFactory) TableConfig(org.apache.flink.table.api.TableConfig) Planner(org.apache.flink.table.delegation.Planner) ModuleManager(org.apache.flink.table.module.ModuleManager) CatalogManager(org.apache.flink.table.catalog.CatalogManager) GenericInMemoryCatalog(org.apache.flink.table.catalog.GenericInMemoryCatalog)

Example 14 with Executor

use of org.apache.flink.table.delegation.Executor in project flink by apache.

the class StreamTableEnvironmentImpl method create.

public static StreamTableEnvironment create(StreamExecutionEnvironment executionEnvironment, EnvironmentSettings settings, TableConfig tableConfig) {
    // temporary solution until FLINK-15635 is fixed
    final ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
    final ModuleManager moduleManager = new ModuleManager();
    final CatalogManager catalogManager = CatalogManager.newBuilder().classLoader(classLoader).config(tableConfig.getConfiguration()).defaultCatalog(settings.getBuiltInCatalogName(), new GenericInMemoryCatalog(settings.getBuiltInCatalogName(), settings.getBuiltInDatabaseName())).executionConfig(executionEnvironment.getConfig()).build();
    final FunctionCatalog functionCatalog = new FunctionCatalog(tableConfig, catalogManager, moduleManager);
    final Executor executor = lookupExecutor(classLoader, settings.getExecutor(), executionEnvironment);
    final Planner planner = PlannerFactoryUtil.createPlanner(settings.getPlanner(), executor, tableConfig, moduleManager, catalogManager, functionCatalog);
    return new StreamTableEnvironmentImpl(catalogManager, moduleManager, functionCatalog, tableConfig, executionEnvironment, planner, executor, settings.isStreamingMode(), classLoader);
}
Also used : FunctionCatalog(org.apache.flink.table.catalog.FunctionCatalog) Executor(org.apache.flink.table.delegation.Executor) Planner(org.apache.flink.table.delegation.Planner) ModuleManager(org.apache.flink.table.module.ModuleManager) AbstractStreamTableEnvironmentImpl(org.apache.flink.table.api.bridge.internal.AbstractStreamTableEnvironmentImpl) CatalogManager(org.apache.flink.table.catalog.CatalogManager) GenericInMemoryCatalog(org.apache.flink.table.catalog.GenericInMemoryCatalog)

Example 15 with Executor

use of org.apache.flink.table.delegation.Executor in project flink by apache.

the class DefaultExecutorTest method testJobName.

@Test
public void testJobName() {
    final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
    final Executor executor = new DefaultExecutor(env);
    final List<Transformation<?>> dummyTransformations = Collections.singletonList(env.fromElements(1, 2, 3).addSink(new DiscardingSink<>()).getTransformation());
    final Configuration configuration = new Configuration();
    configuration.set(PipelineOptions.NAME, "Custom Name");
    // default
    testJobName(executor.createPipeline(dummyTransformations, new Configuration(), "Default Name"), "Default Name");
    // Table API specific
    testJobName(executor.createPipeline(dummyTransformations, configuration, "Default Name"), "Custom Name");
    // DataStream API specific
    env.configure(configuration);
    testJobName(executor.createPipeline(dummyTransformations, new Configuration(), "Default Name"), "Custom Name");
}
Also used : Transformation(org.apache.flink.api.dag.Transformation) Executor(org.apache.flink.table.delegation.Executor) Configuration(org.apache.flink.configuration.Configuration) StreamExecutionEnvironment(org.apache.flink.streaming.api.environment.StreamExecutionEnvironment) Test(org.junit.Test)

Aggregations

Executor (org.apache.flink.table.delegation.Executor)15 Planner (org.apache.flink.table.delegation.Planner)10 StreamExecutionEnvironment (org.apache.flink.streaming.api.environment.StreamExecutionEnvironment)9 TableException (org.apache.flink.table.api.TableException)8 TableConfig (org.apache.flink.table.api.TableConfig)7 CatalogManager (org.apache.flink.table.catalog.CatalogManager)7 FunctionCatalog (org.apache.flink.table.catalog.FunctionCatalog)7 ModuleManager (org.apache.flink.table.module.ModuleManager)7 Constructor (java.lang.reflect.Constructor)5 EnvironmentSettings (org.apache.flink.table.api.EnvironmentSettings)5 TableEnvironment (org.apache.flink.table.api.TableEnvironment)5 ExecutorFactory (org.apache.flink.table.delegation.ExecutorFactory)3 AttributedString (org.jline.utils.AttributedString)3 Method (java.lang.reflect.Method)2 Transformation (org.apache.flink.api.dag.Transformation)2 Configuration (org.apache.flink.configuration.Configuration)2 GenericInMemoryCatalog (org.apache.flink.table.catalog.GenericInMemoryCatalog)2 PlannerFactory (org.apache.flink.table.delegation.PlannerFactory)2 Test (org.junit.Test)2 IOException (java.io.IOException)1