Search in sources :

Example 1 with HoodieFlinkEngineContext

use of org.apache.hudi.client.common.HoodieFlinkEngineContext in project hudi by apache.

the class BucketAssignFunction method open.

@Override
public void open(Configuration parameters) throws Exception {
    super.open(parameters);
    HoodieWriteConfig writeConfig = StreamerUtil.getHoodieClientConfig(this.conf, true);
    HoodieFlinkEngineContext context = new HoodieFlinkEngineContext(new SerializableConfiguration(StreamerUtil.getHadoopConf()), new FlinkTaskContextSupplier(getRuntimeContext()));
    this.bucketAssigner = BucketAssigners.create(getRuntimeContext().getIndexOfThisSubtask(), getRuntimeContext().getMaxNumberOfParallelSubtasks(), getRuntimeContext().getNumberOfParallelSubtasks(), ignoreSmallFiles(), HoodieTableType.valueOf(conf.getString(FlinkOptions.TABLE_TYPE)), context, writeConfig);
    this.payloadCreation = PayloadCreation.instance(this.conf);
}
Also used : SerializableConfiguration(org.apache.hudi.common.config.SerializableConfiguration) HoodieWriteConfig(org.apache.hudi.config.HoodieWriteConfig) HoodieFlinkEngineContext(org.apache.hudi.client.common.HoodieFlinkEngineContext) FlinkTaskContextSupplier(org.apache.hudi.client.FlinkTaskContextSupplier)

Example 2 with HoodieFlinkEngineContext

use of org.apache.hudi.client.common.HoodieFlinkEngineContext in project hudi by apache.

the class HoodieFlinkClientTestHarness method initFileSystem.

protected void initFileSystem() {
    hadoopConf = new Configuration();
    initFileSystemWithConfiguration(hadoopConf);
    context = new HoodieFlinkEngineContext(supplier);
}
Also used : MiniClusterResourceConfiguration(org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration) Configuration(org.apache.hadoop.conf.Configuration) HoodieFlinkEngineContext(org.apache.hudi.client.common.HoodieFlinkEngineContext)

Example 3 with HoodieFlinkEngineContext

use of org.apache.hudi.client.common.HoodieFlinkEngineContext in project hudi by apache.

the class StreamerUtil method createWriteClient.

/**
 * Creates the Flink write client.
 *
 * <p>This expects to be used by client, set flag {@code loadFsViewStorageConfig} to use
 * remote filesystem view storage config, or an in-memory filesystem view storage is used.
 */
@SuppressWarnings("rawtypes")
public static HoodieFlinkWriteClient createWriteClient(Configuration conf, RuntimeContext runtimeContext, boolean loadFsViewStorageConfig) {
    HoodieFlinkEngineContext context = new HoodieFlinkEngineContext(new SerializableConfiguration(getHadoopConf()), new FlinkTaskContextSupplier(runtimeContext));
    HoodieWriteConfig writeConfig = getHoodieClientConfig(conf, loadFsViewStorageConfig);
    return new HoodieFlinkWriteClient<>(context, writeConfig);
}
Also used : SerializableConfiguration(org.apache.hudi.common.config.SerializableConfiguration) HoodieWriteConfig(org.apache.hudi.config.HoodieWriteConfig) HoodieFlinkWriteClient(org.apache.hudi.client.HoodieFlinkWriteClient) HoodieFlinkEngineContext(org.apache.hudi.client.common.HoodieFlinkEngineContext) FlinkTaskContextSupplier(org.apache.hudi.client.FlinkTaskContextSupplier)

Example 4 with HoodieFlinkEngineContext

use of org.apache.hudi.client.common.HoodieFlinkEngineContext in project hudi by apache.

the class FlinkTables method createTable.

/**
 * Creates the hoodie flink table.
 *
 * <p>This expects to be used by client.
 */
public static HoodieFlinkTable<?> createTable(Configuration conf, RuntimeContext runtimeContext) {
    HoodieFlinkEngineContext context = new HoodieFlinkEngineContext(new SerializableConfiguration(getHadoopConf()), new FlinkTaskContextSupplier(runtimeContext));
    HoodieWriteConfig writeConfig = getHoodieClientConfig(conf, true);
    return HoodieFlinkTable.create(writeConfig, context);
}
Also used : SerializableConfiguration(org.apache.hudi.common.config.SerializableConfiguration) HoodieWriteConfig(org.apache.hudi.config.HoodieWriteConfig) HoodieFlinkEngineContext(org.apache.hudi.client.common.HoodieFlinkEngineContext) FlinkTaskContextSupplier(org.apache.hudi.client.FlinkTaskContextSupplier)

Example 5 with HoodieFlinkEngineContext

use of org.apache.hudi.client.common.HoodieFlinkEngineContext in project hudi by apache.

the class TestBucketAssigner method before.

@BeforeEach
public void before() throws IOException {
    final String basePath = tempFile.getAbsolutePath();
    conf = TestConfigurations.getDefaultConf(basePath);
    writeConfig = StreamerUtil.getHoodieClientConfig(conf);
    context = new HoodieFlinkEngineContext(new SerializableConfiguration(StreamerUtil.getHadoopConf()), new FlinkTaskContextSupplier(null));
    StreamerUtil.initTableIfNotExists(conf);
}
Also used : SerializableConfiguration(org.apache.hudi.common.config.SerializableConfiguration) HoodieFlinkEngineContext(org.apache.hudi.client.common.HoodieFlinkEngineContext) FlinkTaskContextSupplier(org.apache.hudi.client.FlinkTaskContextSupplier) BeforeEach(org.junit.jupiter.api.BeforeEach)

Aggregations

HoodieFlinkEngineContext (org.apache.hudi.client.common.HoodieFlinkEngineContext)5 FlinkTaskContextSupplier (org.apache.hudi.client.FlinkTaskContextSupplier)4 SerializableConfiguration (org.apache.hudi.common.config.SerializableConfiguration)4 HoodieWriteConfig (org.apache.hudi.config.HoodieWriteConfig)3 MiniClusterResourceConfiguration (org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration)1 Configuration (org.apache.hadoop.conf.Configuration)1 HoodieFlinkWriteClient (org.apache.hudi.client.HoodieFlinkWriteClient)1 BeforeEach (org.junit.jupiter.api.BeforeEach)1