Search in sources :

Example 1 with CommitterMetrics

use of io.mantisrx.connector.iceberg.sink.committer.metrics.CommitterMetrics in project mantis by Netflix.

the class IcebergCommitterStage method newTransformer.

/**
 * Use this to instantiate a new transformer from a given {@link Context}.
 */
public static Transformer newTransformer(Context context) {
    CommitterConfig config = new CommitterConfig(context.getParameters());
    CommitterMetrics metrics = new CommitterMetrics();
    Catalog catalog = context.getServiceLocator().service(Catalog.class);
    TableIdentifier id = TableIdentifier.of(config.getCatalog(), config.getDatabase(), config.getTable());
    Table table = catalog.loadTable(id);
    IcebergCommitter committer = new IcebergCommitter(table);
    return new Transformer(config, metrics, committer, Schedulers.computation());
}
Also used : CommitterConfig(io.mantisrx.connector.iceberg.sink.committer.config.CommitterConfig) TableIdentifier(org.apache.iceberg.catalog.TableIdentifier) Table(org.apache.iceberg.Table) CommitterMetrics(io.mantisrx.connector.iceberg.sink.committer.metrics.CommitterMetrics) Catalog(org.apache.iceberg.catalog.Catalog)

Example 2 with CommitterMetrics

use of io.mantisrx.connector.iceberg.sink.committer.metrics.CommitterMetrics in project mantis by Netflix.

the class IcebergCommitterStageTest method setUp.

@BeforeEach
void setUp() {
    this.scheduler = new TestScheduler();
    this.subscriber = new TestSubscriber<>();
    Parameters parameters = StageOverrideParameters.newParameters();
    CommitterConfig config = new CommitterConfig(parameters);
    CommitterMetrics metrics = new CommitterMetrics();
    this.committer = mock(IcebergCommitter.class);
    transformer = new IcebergCommitterStage.Transformer(config, metrics, committer, scheduler);
    ServiceLocator serviceLocator = mock(ServiceLocator.class);
    when(serviceLocator.service(Configuration.class)).thenReturn(mock(Configuration.class));
    this.catalog = mock(Catalog.class);
    Table table = mock(Table.class);
    when(table.spec()).thenReturn(PartitionSpec.unpartitioned());
    when(this.catalog.loadTable(any())).thenReturn(table);
    when(serviceLocator.service(Catalog.class)).thenReturn(this.catalog);
    this.context = mock(Context.class);
    when(this.context.getParameters()).thenReturn(parameters);
    when(this.context.getServiceLocator()).thenReturn(serviceLocator);
}
Also used : Context(io.mantisrx.runtime.Context) Parameters(io.mantisrx.runtime.parameter.Parameters) StageOverrideParameters(io.mantisrx.connector.iceberg.sink.StageOverrideParameters) Table(org.apache.iceberg.Table) Configuration(org.apache.hadoop.conf.Configuration) Catalog(org.apache.iceberg.catalog.Catalog) CommitterConfig(io.mantisrx.connector.iceberg.sink.committer.config.CommitterConfig) ServiceLocator(io.mantisrx.runtime.lifecycle.ServiceLocator) TestScheduler(rx.schedulers.TestScheduler) CommitterMetrics(io.mantisrx.connector.iceberg.sink.committer.metrics.CommitterMetrics) BeforeEach(org.junit.jupiter.api.BeforeEach)

Aggregations

CommitterConfig (io.mantisrx.connector.iceberg.sink.committer.config.CommitterConfig)2 CommitterMetrics (io.mantisrx.connector.iceberg.sink.committer.metrics.CommitterMetrics)2 Table (org.apache.iceberg.Table)2 Catalog (org.apache.iceberg.catalog.Catalog)2 StageOverrideParameters (io.mantisrx.connector.iceberg.sink.StageOverrideParameters)1 Context (io.mantisrx.runtime.Context)1 ServiceLocator (io.mantisrx.runtime.lifecycle.ServiceLocator)1 Parameters (io.mantisrx.runtime.parameter.Parameters)1 Configuration (org.apache.hadoop.conf.Configuration)1 TableIdentifier (org.apache.iceberg.catalog.TableIdentifier)1 BeforeEach (org.junit.jupiter.api.BeforeEach)1 TestScheduler (rx.schedulers.TestScheduler)1