Search in sources :

Example 1 with MetadataSupervisorManager

use of org.apache.druid.metadata.MetadataSupervisorManager in project druid by druid-io.

the class MaterializedViewSupervisorTest method setUp.

@Before
public void setUp() {
    TestDerbyConnector derbyConnector = derbyConnectorRule.getConnector();
    derbyConnector.createDataSourceTable();
    derbyConnector.createSegmentTable();
    taskStorage = EasyMock.createMock(TaskStorage.class);
    taskMaster = EasyMock.createMock(TaskMaster.class);
    indexerMetadataStorageCoordinator = new IndexerSQLMetadataStorageCoordinator(objectMapper, derbyConnectorRule.metadataTablesConfigSupplier().get(), derbyConnector);
    metadataSupervisorManager = EasyMock.createMock(MetadataSupervisorManager.class);
    sqlSegmentsMetadataManager = EasyMock.createMock(SqlSegmentsMetadataManager.class);
    taskQueue = EasyMock.createMock(TaskQueue.class);
    taskQueue.start();
    objectMapper.registerSubtypes(new NamedType(HashBasedNumberedShardSpec.class, "hashed"));
    spec = new MaterializedViewSupervisorSpec("base", new DimensionsSpec(Collections.singletonList(new StringDimensionSchema("dim"))), new AggregatorFactory[] { new LongSumAggregatorFactory("m1", "m1") }, HadoopTuningConfig.makeDefaultTuningConfig(), null, null, null, null, null, false, objectMapper, taskMaster, taskStorage, metadataSupervisorManager, sqlSegmentsMetadataManager, indexerMetadataStorageCoordinator, new MaterializedViewTaskConfig(), EasyMock.createMock(AuthorizerMapper.class), EasyMock.createMock(ChatHandlerProvider.class), new SupervisorStateManagerConfig());
    derivativeDatasourceName = spec.getDataSourceName();
    supervisor = (MaterializedViewSupervisor) spec.createSupervisor();
}
Also used : IndexerSQLMetadataStorageCoordinator(org.apache.druid.metadata.IndexerSQLMetadataStorageCoordinator) HashBasedNumberedShardSpec(org.apache.druid.timeline.partition.HashBasedNumberedShardSpec) NamedType(com.fasterxml.jackson.databind.jsontype.NamedType) LongSumAggregatorFactory(org.apache.druid.query.aggregation.LongSumAggregatorFactory) TestDerbyConnector(org.apache.druid.metadata.TestDerbyConnector) LongSumAggregatorFactory(org.apache.druid.query.aggregation.LongSumAggregatorFactory) AggregatorFactory(org.apache.druid.query.aggregation.AggregatorFactory) StringDimensionSchema(org.apache.druid.data.input.impl.StringDimensionSchema) TaskStorage(org.apache.druid.indexing.overlord.TaskStorage) SupervisorStateManagerConfig(org.apache.druid.indexing.overlord.supervisor.SupervisorStateManagerConfig) TaskQueue(org.apache.druid.indexing.overlord.TaskQueue) DimensionsSpec(org.apache.druid.data.input.impl.DimensionsSpec) MetadataSupervisorManager(org.apache.druid.metadata.MetadataSupervisorManager) TaskMaster(org.apache.druid.indexing.overlord.TaskMaster) SqlSegmentsMetadataManager(org.apache.druid.metadata.SqlSegmentsMetadataManager) Before(org.junit.Before)

Example 2 with MetadataSupervisorManager

use of org.apache.druid.metadata.MetadataSupervisorManager in project druid by druid-io.

the class SeekableStreamSupervisor method changeTaskCountInIOConfig.

private void changeTaskCountInIOConfig(int desiredActiveTaskCount) {
    ioConfig.setTaskCount(desiredActiveTaskCount);
    try {
        Optional<SupervisorManager> supervisorManager = taskMaster.getSupervisorManager();
        if (supervisorManager.isPresent()) {
            MetadataSupervisorManager metadataSupervisorManager = supervisorManager.get().getMetadataSupervisorManager();
            metadataSupervisorManager.insert(dataSource, spec);
        } else {
            log.error("supervisorManager is null in taskMaster, skipping scale action for dataSource [%s].", dataSource);
        }
    } catch (Exception e) {
        log.error(e, "Failed to sync taskCount to MetaStorage for dataSource [%s].", dataSource);
    }
}
Also used : SupervisorManager(org.apache.druid.indexing.overlord.supervisor.SupervisorManager) MetadataSupervisorManager(org.apache.druid.metadata.MetadataSupervisorManager) MetadataSupervisorManager(org.apache.druid.metadata.MetadataSupervisorManager) EntryExistsException(org.apache.druid.metadata.EntryExistsException) IOException(java.io.IOException) ExecutionException(java.util.concurrent.ExecutionException) TimeoutException(java.util.concurrent.TimeoutException) StreamException(org.apache.druid.indexing.seekablestream.common.StreamException) JsonProcessingException(com.fasterxml.jackson.core.JsonProcessingException)

Aggregations

MetadataSupervisorManager (org.apache.druid.metadata.MetadataSupervisorManager)2 JsonProcessingException (com.fasterxml.jackson.core.JsonProcessingException)1 NamedType (com.fasterxml.jackson.databind.jsontype.NamedType)1 IOException (java.io.IOException)1 ExecutionException (java.util.concurrent.ExecutionException)1 TimeoutException (java.util.concurrent.TimeoutException)1 DimensionsSpec (org.apache.druid.data.input.impl.DimensionsSpec)1 StringDimensionSchema (org.apache.druid.data.input.impl.StringDimensionSchema)1 TaskMaster (org.apache.druid.indexing.overlord.TaskMaster)1 TaskQueue (org.apache.druid.indexing.overlord.TaskQueue)1 TaskStorage (org.apache.druid.indexing.overlord.TaskStorage)1 SupervisorManager (org.apache.druid.indexing.overlord.supervisor.SupervisorManager)1 SupervisorStateManagerConfig (org.apache.druid.indexing.overlord.supervisor.SupervisorStateManagerConfig)1 StreamException (org.apache.druid.indexing.seekablestream.common.StreamException)1 EntryExistsException (org.apache.druid.metadata.EntryExistsException)1 IndexerSQLMetadataStorageCoordinator (org.apache.druid.metadata.IndexerSQLMetadataStorageCoordinator)1 SqlSegmentsMetadataManager (org.apache.druid.metadata.SqlSegmentsMetadataManager)1 TestDerbyConnector (org.apache.druid.metadata.TestDerbyConnector)1 AggregatorFactory (org.apache.druid.query.aggregation.AggregatorFactory)1 LongSumAggregatorFactory (org.apache.druid.query.aggregation.LongSumAggregatorFactory)1