Examples with HoodieFlinkEngineContext org.apache.hudi.client.common.HoodieFlinkEngineContext used on opensource projects

Example 1 with HoodieFlinkEngineContext

use of org.apache.hudi.client.common.HoodieFlinkEngineContext in project hudi by apache.

the class BucketAssignFunction method open.

@Override
public void open(Configuration parameters) throws Exception {
    super.open(parameters);
    HoodieWriteConfig writeConfig = StreamerUtil.getHoodieClientConfig(this.conf, true);
    HoodieFlinkEngineContext context = new HoodieFlinkEngineContext(new SerializableConfiguration(StreamerUtil.getHadoopConf()), new FlinkTaskContextSupplier(getRuntimeContext()));
    this.bucketAssigner = BucketAssigners.create(getRuntimeContext().getIndexOfThisSubtask(), getRuntimeContext().getMaxNumberOfParallelSubtasks(), getRuntimeContext().getNumberOfParallelSubtasks(), ignoreSmallFiles(), HoodieTableType.valueOf(conf.getString(FlinkOptions.TABLE_TYPE)), context, writeConfig);
    this.payloadCreation = PayloadCreation.instance(this.conf);
}

Example 2 with HoodieFlinkEngineContext

use of org.apache.hudi.client.common.HoodieFlinkEngineContext in project hudi by apache.

the class HoodieFlinkClientTestHarness method initFileSystem.

protected void initFileSystem() {
    hadoopConf = new Configuration();
    initFileSystemWithConfiguration(hadoopConf);
    context = new HoodieFlinkEngineContext(supplier);
}

Example 3 with HoodieFlinkEngineContext

use of org.apache.hudi.client.common.HoodieFlinkEngineContext in project hudi by apache.

the class StreamerUtil method createWriteClient.

/**
 * Creates the Flink write client.
 *
 * <p>This expects to be used by client, set flag {@code loadFsViewStorageConfig} to use
 * remote filesystem view storage config, or an in-memory filesystem view storage is used.
 */
@SuppressWarnings("rawtypes")
public static HoodieFlinkWriteClient createWriteClient(Configuration conf, RuntimeContext runtimeContext, boolean loadFsViewStorageConfig) {
    HoodieFlinkEngineContext context = new HoodieFlinkEngineContext(new SerializableConfiguration(getHadoopConf()), new FlinkTaskContextSupplier(runtimeContext));
    HoodieWriteConfig writeConfig = getHoodieClientConfig(conf, loadFsViewStorageConfig);
    return new HoodieFlinkWriteClient<>(context, writeConfig);
}

Example 4 with HoodieFlinkEngineContext

use of org.apache.hudi.client.common.HoodieFlinkEngineContext in project hudi by apache.

the class FlinkTables method createTable.

/**
 * Creates the hoodie flink table.
 *
 * <p>This expects to be used by client.
 */
public static HoodieFlinkTable<?> createTable(Configuration conf, RuntimeContext runtimeContext) {
    HoodieFlinkEngineContext context = new HoodieFlinkEngineContext(new SerializableConfiguration(getHadoopConf()), new FlinkTaskContextSupplier(runtimeContext));
    HoodieWriteConfig writeConfig = getHoodieClientConfig(conf, true);
    return HoodieFlinkTable.create(writeConfig, context);
}

Example 5 with HoodieFlinkEngineContext

use of org.apache.hudi.client.common.HoodieFlinkEngineContext in project hudi by apache.

the class TestBucketAssigner method before.

@BeforeEach
public void before() throws IOException {
    final String basePath = tempFile.getAbsolutePath();
    conf = TestConfigurations.getDefaultConf(basePath);
    writeConfig = StreamerUtil.getHoodieClientConfig(conf);
    context = new HoodieFlinkEngineContext(new SerializableConfiguration(StreamerUtil.getHadoopConf()), new FlinkTaskContextSupplier(null));
    StreamerUtil.initTableIfNotExists(conf);
}