Search in sources :

Example 11 with DruidCoordinatorRuntimeParams

use of org.apache.druid.server.coordinator.DruidCoordinatorRuntimeParams in project druid by druid-io.

the class UnloadUnusedSegmentsTest method test_unloadUnusedSegmentsFromAllServers.

@Test
public void test_unloadUnusedSegmentsFromAllServers() {
    mockDruidServer(historicalServer, ServerType.HISTORICAL, "historical", DruidServer.DEFAULT_TIER, 30L, 100L, segments, dataSources);
    mockDruidServer(historicalServerTier2, ServerType.HISTORICAL, "historicalTier2", "tier2", 30L, 100L, segments, dataSources);
    mockDruidServer(brokerServer, ServerType.BROKER, "broker", DruidServer.DEFAULT_TIER, 30L, 100L, segments, dataSources);
    mockDruidServer(indexerServer, ServerType.INDEXER_EXECUTOR, "indexer", DruidServer.DEFAULT_TIER, 30L, 100L, segmentsForRealtime, dataSourcesForRealtime);
    // Mock stuff that the coordinator needs
    mockCoordinator(coordinator);
    mockRuleManager(databaseRuleManager);
    // We keep datasource2 segments only, drop datasource1 and broadcastDatasource from all servers
    // realtimeSegment is intentionally missing from the set, to match how a realtime tasks's unpublished segments
    // will not appear in the coordinator's view of used segments.
    Set<DataSegment> usedSegments = ImmutableSet.of(segment2);
    DruidCoordinatorRuntimeParams params = CoordinatorRuntimeParamsTestHelpers.newBuilder().withDruidCluster(DruidClusterBuilder.newBuilder().addTier(DruidServer.DEFAULT_TIER, new ServerHolder(historicalServer, historicalPeon, false)).addTier("tier2", new ServerHolder(historicalServerTier2, historicalTier2Peon, false)).withBrokers(new ServerHolder(brokerServer, brokerPeon, false)).withRealtimes(new ServerHolder(indexerServer, indexerPeon, false)).build()).withLoadManagementPeons(ImmutableMap.of("historical", historicalPeon, "historicalTier2", historicalTier2Peon, "broker", brokerPeon, "indexer", indexerPeon)).withUsedSegmentsInTest(usedSegments).withBroadcastDatasources(broadcastDatasourceNames).withDatabaseRuleManager(databaseRuleManager).build();
    params = new UnloadUnusedSegments().run(params);
    CoordinatorStats stats = params.getCoordinatorStats();
    // We drop segment1 and broadcast1 from all servers, realtimeSegment is not dropped by the indexer
    Assert.assertEquals(5, stats.getTieredStat("unneededCount", DruidServer.DEFAULT_TIER));
    Assert.assertEquals(2, stats.getTieredStat("unneededCount", "tier2"));
}
Also used : DruidCoordinatorRuntimeParams(org.apache.druid.server.coordinator.DruidCoordinatorRuntimeParams) CoordinatorStats(org.apache.druid.server.coordinator.CoordinatorStats) ServerHolder(org.apache.druid.server.coordinator.ServerHolder) DataSegment(org.apache.druid.timeline.DataSegment) Test(org.junit.Test)

Aggregations

DruidCoordinatorRuntimeParams (org.apache.druid.server.coordinator.DruidCoordinatorRuntimeParams)11 CoordinatorStats (org.apache.druid.server.coordinator.CoordinatorStats)9 DataSegment (org.apache.druid.timeline.DataSegment)8 ServerHolder (org.apache.druid.server.coordinator.ServerHolder)7 Map (java.util.Map)5 Collectors (java.util.stream.Collectors)5 DruidCluster (org.apache.druid.server.coordinator.DruidCluster)5 ImmutableDruidServer (org.apache.druid.client.ImmutableDruidServer)4 Logger (org.apache.druid.java.util.common.logger.Logger)4 Inject (com.google.inject.Inject)3 Set (java.util.Set)3 ServiceEmitter (org.apache.druid.java.util.emitter.service.ServiceEmitter)3 ServiceMetricEvent (org.apache.druid.java.util.emitter.service.ServiceMetricEvent)3 DruidCoordinator (org.apache.druid.server.coordinator.DruidCoordinator)3 DruidCoordinatorConfig (org.apache.druid.server.coordinator.DruidCoordinatorConfig)3 LoadQueuePeon (org.apache.druid.server.coordinator.LoadQueuePeon)3 Test (org.junit.Test)3 Preconditions (com.google.common.base.Preconditions)2 Object2LongMap (it.unimi.dsi.fastutil.objects.Object2LongMap)2 HashMap (java.util.HashMap)2