Examples with StreamWriteFunctionWrapper org.apache.hudi.sink.utils.StreamWriteFunctionWrapper used on opensource projects

Example 1 with StreamWriteFunctionWrapper

use of org.apache.hudi.sink.utils.StreamWriteFunctionWrapper in project hudi by apache.

the class TestData method writeDataAsBatch.

/**
 * Write a list of row data with Hoodie format base on the given configuration.
 *
 * <p>The difference with {@link #writeData} is that it flush data using #endInput, and it
 * does not generate inflight instant.
 *
 * @param dataBuffer The data buffer to write
 * @param conf       The flink configuration
 * @throws Exception if error occurs
 */
public static void writeDataAsBatch(List<RowData> dataBuffer, Configuration conf) throws Exception {
    StreamWriteFunctionWrapper<RowData> funcWrapper = new StreamWriteFunctionWrapper<>(conf.getString(FlinkOptions.PATH), conf);
    funcWrapper.openFunction();
    for (RowData rowData : dataBuffer) {
        funcWrapper.invoke(rowData);
    }
    // this triggers the data write and event send
    funcWrapper.endInput();
    final OperatorEvent nextEvent = funcWrapper.getNextEvent();
    funcWrapper.getCoordinator().handleEventFromOperator(0, nextEvent);
    funcWrapper.close();
}

Example 2 with StreamWriteFunctionWrapper

use of org.apache.hudi.sink.utils.StreamWriteFunctionWrapper in project hudi by apache.

the class TestData method writeData.

/**
 * Write a list of row data with Hoodie format base on the given configuration.
 *
 * @param dataBuffer The data buffer to write
 * @param conf       The flink configuration
 * @throws Exception if error occurs
 */
public static void writeData(List<RowData> dataBuffer, Configuration conf) throws Exception {
    StreamWriteFunctionWrapper<RowData> funcWrapper = new StreamWriteFunctionWrapper<>(conf.getString(FlinkOptions.PATH), conf);
    funcWrapper.openFunction();
    for (RowData rowData : dataBuffer) {
        funcWrapper.invoke(rowData);
    }
    // this triggers the data write and event send
    funcWrapper.checkpointFunction(1);
    final OperatorEvent nextEvent = funcWrapper.getNextEvent();
    funcWrapper.getCoordinator().handleEventFromOperator(0, nextEvent);
    funcWrapper.checkpointComplete(1);
    funcWrapper.close();
}