Search in sources :

Example 1 with AllocationService

use of org.opensearch.cluster.routing.allocation.AllocationService in project OpenSearch by opensearch-project.

the class TransportResizeActionTests method testPassNumRoutingShardsAndFail.

public void testPassNumRoutingShardsAndFail() {
    int numShards = randomIntBetween(2, 100);
    ClusterState clusterState = ClusterState.builder(createClusterState("source", numShards, 0, numShards * 4, Settings.builder().put("index.blocks.write", true).build())).nodes(DiscoveryNodes.builder().add(newNode("node1"))).build();
    AllocationService service = new AllocationService(new AllocationDeciders(Collections.singleton(new MaxRetryAllocationDecider())), new TestGatewayAllocator(), new BalancedShardsAllocator(Settings.EMPTY), EmptyClusterInfoService.INSTANCE, EmptySnapshotsInfoService.INSTANCE);
    RoutingTable routingTable = service.reroute(clusterState, "reroute").routingTable();
    clusterState = ClusterState.builder(clusterState).routingTable(routingTable).build();
    // now we start the shard
    routingTable = OpenSearchAllocationTestCase.startInitializingShardsAndReroute(service, clusterState, "source").routingTable();
    clusterState = ClusterState.builder(clusterState).routingTable(routingTable).build();
    ResizeRequest resizeRequest = new ResizeRequest("target", "source");
    resizeRequest.setResizeType(ResizeType.SPLIT);
    resizeRequest.getTargetIndexRequest().settings(Settings.builder().put("index.number_of_shards", numShards * 2).build());
    TransportResizeAction.prepareCreateIndexRequest(resizeRequest, clusterState, null, "source", "target");
    resizeRequest.getTargetIndexRequest().settings(Settings.builder().put("index.number_of_shards", numShards * 2).put("index.number_of_routing_shards", numShards * 2).build());
    ClusterState finalState = clusterState;
    IllegalArgumentException iae = expectThrows(IllegalArgumentException.class, () -> TransportResizeAction.prepareCreateIndexRequest(resizeRequest, finalState, null, "source", "target"));
    assertEquals("cannot provide index.number_of_routing_shards on resize", iae.getMessage());
}
Also used : TestGatewayAllocator(org.opensearch.test.gateway.TestGatewayAllocator) ClusterState(org.opensearch.cluster.ClusterState) RoutingTable(org.opensearch.cluster.routing.RoutingTable) BalancedShardsAllocator(org.opensearch.cluster.routing.allocation.allocator.BalancedShardsAllocator) AllocationDeciders(org.opensearch.cluster.routing.allocation.decider.AllocationDeciders) MaxRetryAllocationDecider(org.opensearch.cluster.routing.allocation.decider.MaxRetryAllocationDecider) AllocationService(org.opensearch.cluster.routing.allocation.AllocationService)

Example 2 with AllocationService

use of org.opensearch.cluster.routing.allocation.AllocationService in project OpenSearch by opensearch-project.

the class TransportResizeActionTests method testPassNumRoutingShards.

public void testPassNumRoutingShards() {
    ClusterState clusterState = ClusterState.builder(createClusterState("source", 1, 0, Settings.builder().put("index.blocks.write", true).build())).nodes(DiscoveryNodes.builder().add(newNode("node1"))).build();
    AllocationService service = new AllocationService(new AllocationDeciders(Collections.singleton(new MaxRetryAllocationDecider())), new TestGatewayAllocator(), new BalancedShardsAllocator(Settings.EMPTY), EmptyClusterInfoService.INSTANCE, EmptySnapshotsInfoService.INSTANCE);
    RoutingTable routingTable = service.reroute(clusterState, "reroute").routingTable();
    clusterState = ClusterState.builder(clusterState).routingTable(routingTable).build();
    // now we start the shard
    routingTable = OpenSearchAllocationTestCase.startInitializingShardsAndReroute(service, clusterState, "source").routingTable();
    clusterState = ClusterState.builder(clusterState).routingTable(routingTable).build();
    ResizeRequest resizeRequest = new ResizeRequest("target", "source");
    resizeRequest.setResizeType(ResizeType.SPLIT);
    resizeRequest.getTargetIndexRequest().settings(Settings.builder().put("index.number_of_shards", 2).build());
    TransportResizeAction.prepareCreateIndexRequest(resizeRequest, clusterState, null, "source", "target");
    resizeRequest.getTargetIndexRequest().settings(Settings.builder().put("index.number_of_routing_shards", randomIntBetween(2, 10)).put("index.number_of_shards", 2).build());
    TransportResizeAction.prepareCreateIndexRequest(resizeRequest, clusterState, null, "source", "target");
}
Also used : TestGatewayAllocator(org.opensearch.test.gateway.TestGatewayAllocator) ClusterState(org.opensearch.cluster.ClusterState) RoutingTable(org.opensearch.cluster.routing.RoutingTable) BalancedShardsAllocator(org.opensearch.cluster.routing.allocation.allocator.BalancedShardsAllocator) AllocationDeciders(org.opensearch.cluster.routing.allocation.decider.AllocationDeciders) MaxRetryAllocationDecider(org.opensearch.cluster.routing.allocation.decider.MaxRetryAllocationDecider) AllocationService(org.opensearch.cluster.routing.allocation.AllocationService)

Example 3 with AllocationService

use of org.opensearch.cluster.routing.allocation.AllocationService in project OpenSearch by opensearch-project.

the class JoinTaskExecutorTests method testUpdatesNodeWithNewRoles.

public void testUpdatesNodeWithNewRoles() throws Exception {
    // Node roles vary by version, and new roles are suppressed for BWC. This means we can receive a join from a node that's already
    // in the cluster but with a different set of roles: the node didn't change roles, but the cluster state came via an older master.
    // In this case we must properly process its join to ensure that the roles are correct.
    final AllocationService allocationService = mock(AllocationService.class);
    when(allocationService.adaptAutoExpandReplicas(any())).then(invocationOnMock -> invocationOnMock.getArguments()[0]);
    final RerouteService rerouteService = (reason, priority, listener) -> listener.onResponse(null);
    final JoinTaskExecutor joinTaskExecutor = new JoinTaskExecutor(Settings.EMPTY, allocationService, logger, rerouteService, null);
    final DiscoveryNode masterNode = new DiscoveryNode(UUIDs.base64UUID(), buildNewFakeTransportAddress(), Version.CURRENT);
    final DiscoveryNode actualNode = new DiscoveryNode(UUIDs.base64UUID(), buildNewFakeTransportAddress(), Version.CURRENT);
    final DiscoveryNode bwcNode = new DiscoveryNode(actualNode.getName(), actualNode.getId(), actualNode.getEphemeralId(), actualNode.getHostName(), actualNode.getHostAddress(), actualNode.getAddress(), actualNode.getAttributes(), new HashSet<>(randomSubsetOf(actualNode.getRoles())), actualNode.getVersion());
    final ClusterState clusterState = ClusterState.builder(ClusterName.DEFAULT).nodes(DiscoveryNodes.builder().add(masterNode).localNodeId(masterNode.getId()).masterNodeId(masterNode.getId()).add(bwcNode)).build();
    final ClusterStateTaskExecutor.ClusterTasksResult<JoinTaskExecutor.Task> result = joinTaskExecutor.execute(clusterState, List.of(new JoinTaskExecutor.Task(actualNode, "test")));
    assertThat(result.executionResults.entrySet(), hasSize(1));
    final ClusterStateTaskExecutor.TaskResult taskResult = result.executionResults.values().iterator().next();
    assertTrue(taskResult.isSuccess());
    assertThat(result.resultingState.getNodes().get(actualNode.getId()).getRoles(), equalTo(actualNode.getRoles()));
}
Also used : DiscoveryNodes(org.opensearch.cluster.node.DiscoveryNodes) Metadata(org.opensearch.cluster.metadata.Metadata) IndexMetadata(org.opensearch.cluster.metadata.IndexMetadata) VersionUtils.maxCompatibleVersion(org.opensearch.test.VersionUtils.maxCompatibleVersion) VersionUtils.randomCompatibleVersion(org.opensearch.test.VersionUtils.randomCompatibleVersion) AllocationService(org.opensearch.cluster.routing.allocation.AllocationService) Version(org.opensearch.Version) HashMap(java.util.HashMap) VersionUtils.randomVersionBetween(org.opensearch.test.VersionUtils.randomVersionBetween) HashSet(java.util.HashSet) ClusterState(org.opensearch.cluster.ClusterState) DiscoveryNode(org.opensearch.cluster.node.DiscoveryNode) LegacyESVersion(org.opensearch.LegacyESVersion) VersionUtils(org.opensearch.test.VersionUtils) Mockito.anyBoolean(org.mockito.Mockito.anyBoolean) Map(java.util.Map) RerouteService(org.opensearch.cluster.routing.RerouteService) Matchers.hasSize(org.hamcrest.Matchers.hasSize) UUIDs(org.opensearch.common.UUIDs) OpenSearchTestCase(org.opensearch.test.OpenSearchTestCase) VersionUtils.randomVersion(org.opensearch.test.VersionUtils.randomVersion) ClusterStateTaskExecutor(org.opensearch.cluster.ClusterStateTaskExecutor) Settings(org.opensearch.common.settings.Settings) Mockito.when(org.mockito.Mockito.when) TransportService(org.opensearch.transport.TransportService) List(org.opensearch.common.collect.List) Matchers.equalTo(org.hamcrest.Matchers.equalTo) ClusterName(org.opensearch.cluster.ClusterName) Mockito.any(org.mockito.Mockito.any) Mockito.mock(org.mockito.Mockito.mock) ClusterState(org.opensearch.cluster.ClusterState) DiscoveryNode(org.opensearch.cluster.node.DiscoveryNode) ClusterStateTaskExecutor(org.opensearch.cluster.ClusterStateTaskExecutor) RerouteService(org.opensearch.cluster.routing.RerouteService) AllocationService(org.opensearch.cluster.routing.allocation.AllocationService)

Example 4 with AllocationService

use of org.opensearch.cluster.routing.allocation.AllocationService in project OpenSearch by opensearch-project.

the class JoinTaskExecutorTests method testUpdatesNodeWithOpenSearchVersionForExistingAndNewNodes.

public void testUpdatesNodeWithOpenSearchVersionForExistingAndNewNodes() throws Exception {
    // During the upgrade from Elasticsearch, OpenSearch node send their version as 7.10.2 to Elasticsearch master
    // in order to successfully join the cluster. But as soon as OpenSearch node becomes the master, cluster state
    // should show the OpenSearch nodes version as 1.x. As the cluster state was carry forwarded from ES master,
    // version in DiscoveryNode is stale 7.10.2.
    final AllocationService allocationService = mock(AllocationService.class);
    when(allocationService.adaptAutoExpandReplicas(any())).then(invocationOnMock -> invocationOnMock.getArguments()[0]);
    when(allocationService.disassociateDeadNodes(any(), anyBoolean(), any())).then(invocationOnMock -> invocationOnMock.getArguments()[0]);
    final RerouteService rerouteService = (reason, priority, listener) -> listener.onResponse(null);
    Map<String, Version> channelVersions = new HashMap<>();
    // OpenSearch node running BWC version
    String node_1 = UUIDs.base64UUID();
    // OpenSearch node running BWC version
    String node_2 = UUIDs.base64UUID();
    // OpenSearch node running BWC version, sending new join request and no active channel
    String node_3 = UUIDs.base64UUID();
    // ES node 7.10.2
    String node_4 = UUIDs.base64UUID();
    // ES node 7.10.2 in cluster but missing channel version
    String node_5 = UUIDs.base64UUID();
    // ES node 7.9.0
    String node_6 = UUIDs.base64UUID();
    // ES node 7.9.0 in cluster but missing channel version
    String node_7 = UUIDs.base64UUID();
    channelVersions.put(node_1, Version.CURRENT);
    channelVersions.put(node_2, Version.CURRENT);
    channelVersions.put(node_4, LegacyESVersion.V_7_10_2);
    channelVersions.put(node_6, LegacyESVersion.V_7_10_0);
    final TransportService transportService = mock(TransportService.class);
    when(transportService.getChannelVersion(any())).thenReturn(channelVersions);
    DiscoveryNodes.Builder nodes = new DiscoveryNodes.Builder().localNodeId(node_1);
    nodes.add(new DiscoveryNode(node_1, buildNewFakeTransportAddress(), LegacyESVersion.V_7_10_2));
    nodes.add(new DiscoveryNode(node_2, buildNewFakeTransportAddress(), LegacyESVersion.V_7_10_2));
    nodes.add(new DiscoveryNode(node_3, buildNewFakeTransportAddress(), LegacyESVersion.V_7_10_2));
    nodes.add(new DiscoveryNode(node_4, buildNewFakeTransportAddress(), LegacyESVersion.V_7_10_2));
    nodes.add(new DiscoveryNode(node_5, buildNewFakeTransportAddress(), LegacyESVersion.V_7_10_2));
    nodes.add(new DiscoveryNode(node_6, buildNewFakeTransportAddress(), LegacyESVersion.V_7_10_1));
    nodes.add(new DiscoveryNode(node_7, buildNewFakeTransportAddress(), LegacyESVersion.V_7_10_0));
    final ClusterState clusterState = ClusterState.builder(ClusterName.DEFAULT).nodes(nodes).build();
    final JoinTaskExecutor joinTaskExecutor = new JoinTaskExecutor(Settings.EMPTY, allocationService, logger, rerouteService, transportService);
    final DiscoveryNode existing_node_3 = clusterState.nodes().get(node_3);
    final DiscoveryNode node_3_new_join = new DiscoveryNode(existing_node_3.getName(), existing_node_3.getId(), existing_node_3.getEphemeralId(), existing_node_3.getHostName(), existing_node_3.getHostAddress(), existing_node_3.getAddress(), existing_node_3.getAttributes(), existing_node_3.getRoles(), Version.CURRENT);
    final ClusterStateTaskExecutor.ClusterTasksResult<JoinTaskExecutor.Task> result = joinTaskExecutor.execute(clusterState, List.of(new JoinTaskExecutor.Task(node_3_new_join, "test"), JoinTaskExecutor.newBecomeMasterTask(), JoinTaskExecutor.newFinishElectionTask()));
    final ClusterStateTaskExecutor.TaskResult taskResult = result.executionResults.values().iterator().next();
    assertTrue(taskResult.isSuccess());
    DiscoveryNodes resultNodes = result.resultingState.getNodes();
    assertEquals(Version.CURRENT, resultNodes.get(node_1).getVersion());
    assertEquals(Version.CURRENT, resultNodes.get(node_2).getVersion());
    // 7.10.2 in old state but sent new join and processed
    assertEquals(Version.CURRENT, resultNodes.get(node_3).getVersion());
    assertEquals(LegacyESVersion.V_7_10_2, resultNodes.get(node_4).getVersion());
    // 7.10.2 node without active channel will be removed and should rejoin
    assertFalse(resultNodes.nodeExists(node_5));
    assertEquals(LegacyESVersion.V_7_10_0, resultNodes.get(node_6).getVersion());
    // 7.9.0 node without active channel but shouldn't get removed
    assertEquals(LegacyESVersion.V_7_10_0, resultNodes.get(node_7).getVersion());
}
Also used : DiscoveryNodes(org.opensearch.cluster.node.DiscoveryNodes) Metadata(org.opensearch.cluster.metadata.Metadata) IndexMetadata(org.opensearch.cluster.metadata.IndexMetadata) VersionUtils.maxCompatibleVersion(org.opensearch.test.VersionUtils.maxCompatibleVersion) VersionUtils.randomCompatibleVersion(org.opensearch.test.VersionUtils.randomCompatibleVersion) AllocationService(org.opensearch.cluster.routing.allocation.AllocationService) Version(org.opensearch.Version) HashMap(java.util.HashMap) VersionUtils.randomVersionBetween(org.opensearch.test.VersionUtils.randomVersionBetween) HashSet(java.util.HashSet) ClusterState(org.opensearch.cluster.ClusterState) DiscoveryNode(org.opensearch.cluster.node.DiscoveryNode) LegacyESVersion(org.opensearch.LegacyESVersion) VersionUtils(org.opensearch.test.VersionUtils) Mockito.anyBoolean(org.mockito.Mockito.anyBoolean) Map(java.util.Map) RerouteService(org.opensearch.cluster.routing.RerouteService) Matchers.hasSize(org.hamcrest.Matchers.hasSize) UUIDs(org.opensearch.common.UUIDs) OpenSearchTestCase(org.opensearch.test.OpenSearchTestCase) VersionUtils.randomVersion(org.opensearch.test.VersionUtils.randomVersion) ClusterStateTaskExecutor(org.opensearch.cluster.ClusterStateTaskExecutor) Settings(org.opensearch.common.settings.Settings) Mockito.when(org.mockito.Mockito.when) TransportService(org.opensearch.transport.TransportService) List(org.opensearch.common.collect.List) Matchers.equalTo(org.hamcrest.Matchers.equalTo) ClusterName(org.opensearch.cluster.ClusterName) Mockito.any(org.mockito.Mockito.any) Mockito.mock(org.mockito.Mockito.mock) ClusterState(org.opensearch.cluster.ClusterState) DiscoveryNode(org.opensearch.cluster.node.DiscoveryNode) HashMap(java.util.HashMap) VersionUtils.maxCompatibleVersion(org.opensearch.test.VersionUtils.maxCompatibleVersion) VersionUtils.randomCompatibleVersion(org.opensearch.test.VersionUtils.randomCompatibleVersion) Version(org.opensearch.Version) LegacyESVersion(org.opensearch.LegacyESVersion) VersionUtils.randomVersion(org.opensearch.test.VersionUtils.randomVersion) TransportService(org.opensearch.transport.TransportService) ClusterStateTaskExecutor(org.opensearch.cluster.ClusterStateTaskExecutor) RerouteService(org.opensearch.cluster.routing.RerouteService) AllocationService(org.opensearch.cluster.routing.allocation.AllocationService) DiscoveryNodes(org.opensearch.cluster.node.DiscoveryNodes)

Example 5 with AllocationService

use of org.opensearch.cluster.routing.allocation.AllocationService in project OpenSearch by opensearch-project.

the class NodeRemovalClusterStateTaskExecutorTests method testRerouteAfterRemovingNodes.

public void testRerouteAfterRemovingNodes() throws Exception {
    final AllocationService allocationService = mock(AllocationService.class);
    when(allocationService.disassociateDeadNodes(any(ClusterState.class), eq(true), any(String.class))).thenAnswer(im -> im.getArguments()[0]);
    final AtomicReference<ClusterState> remainingNodesClusterState = new AtomicReference<>();
    final NodeRemovalClusterStateTaskExecutor executor = new NodeRemovalClusterStateTaskExecutor(allocationService, logger) {

        @Override
        protected ClusterState remainingNodesClusterState(ClusterState currentState, DiscoveryNodes.Builder remainingNodesBuilder) {
            remainingNodesClusterState.set(super.remainingNodesClusterState(currentState, remainingNodesBuilder));
            return remainingNodesClusterState.get();
        }
    };
    final DiscoveryNodes.Builder builder = DiscoveryNodes.builder();
    final int nodes = randomIntBetween(2, 16);
    final List<NodeRemovalClusterStateTaskExecutor.Task> tasks = new ArrayList<>();
    // to ensure that there is at least one removal
    boolean first = true;
    for (int i = 0; i < nodes; i++) {
        final DiscoveryNode node = node(i);
        builder.add(node);
        if (first || randomBoolean()) {
            tasks.add(new NodeRemovalClusterStateTaskExecutor.Task(node, randomBoolean() ? "left" : "failed"));
        }
        first = false;
    }
    final ClusterState clusterState = ClusterState.builder(new ClusterName("test")).nodes(builder).build();
    final ClusterStateTaskExecutor.ClusterTasksResult<NodeRemovalClusterStateTaskExecutor.Task> result = executor.execute(clusterState, tasks);
    verify(allocationService).disassociateDeadNodes(eq(remainingNodesClusterState.get()), eq(true), any(String.class));
    for (final NodeRemovalClusterStateTaskExecutor.Task task : tasks) {
        assertNull(result.resultingState.nodes().get(task.node().getId()));
    }
}
Also used : ClusterState(org.opensearch.cluster.ClusterState) DiscoveryNode(org.opensearch.cluster.node.DiscoveryNode) ArrayList(java.util.ArrayList) AtomicReference(java.util.concurrent.atomic.AtomicReference) ClusterStateTaskExecutor(org.opensearch.cluster.ClusterStateTaskExecutor) ClusterName(org.opensearch.cluster.ClusterName) AllocationService(org.opensearch.cluster.routing.allocation.AllocationService) DiscoveryNodes(org.opensearch.cluster.node.DiscoveryNodes)

Aggregations

AllocationService (org.opensearch.cluster.routing.allocation.AllocationService)59 ClusterState (org.opensearch.cluster.ClusterState)55 IndexMetadata (org.opensearch.cluster.metadata.IndexMetadata)39 RoutingTable (org.opensearch.cluster.routing.RoutingTable)38 Metadata (org.opensearch.cluster.metadata.Metadata)36 TestGatewayAllocator (org.opensearch.test.gateway.TestGatewayAllocator)23 BalancedShardsAllocator (org.opensearch.cluster.routing.allocation.allocator.BalancedShardsAllocator)21 Settings (org.opensearch.common.settings.Settings)19 ClusterSettings (org.opensearch.common.settings.ClusterSettings)17 Matchers.containsString (org.hamcrest.Matchers.containsString)13 DiscoveryNodes (org.opensearch.cluster.node.DiscoveryNodes)13 ShardRouting (org.opensearch.cluster.routing.ShardRouting)13 ClusterName (org.opensearch.cluster.ClusterName)11 DiscoveryNode (org.opensearch.cluster.node.DiscoveryNode)11 IndexShardRoutingTable (org.opensearch.cluster.routing.IndexShardRoutingTable)11 ClusterInfo (org.opensearch.cluster.ClusterInfo)10 AllocationDeciders (org.opensearch.cluster.routing.allocation.decider.AllocationDeciders)10 ImmutableOpenMap (org.opensearch.common.collect.ImmutableOpenMap)10 ClusterInfoService (org.opensearch.cluster.ClusterInfoService)9 DiskUsage (org.opensearch.cluster.DiskUsage)9