Search in sources :

Example 1 with EC2AutoScaler

use of org.apache.druid.indexing.overlord.autoscaling.ec2.EC2AutoScaler in project druid by druid-io.

the class WorkerBehaviorConfigTest method testSerde.

@Test
public void testSerde() throws Exception {
    WorkerBehaviorConfig config = new WorkerBehaviorConfig(new FillCapacityWithAffinityWorkerSelectStrategy(new FillCapacityWithAffinityConfig(ImmutableMap.of("foo", Arrays.asList("localhost")))), new EC2AutoScaler(7, 11, new EC2EnvironmentConfig("us-east-1a", new EC2NodeData("amiid", "instanceType", 3, 5, Arrays.asList("securityGroupIds"), "keyNames", "subnetId", null, null), new StringEC2UserData("availZone", "replace", "version")), null, null));
    final ObjectMapper mapper = new DefaultObjectMapper();
    mapper.setInjectableValues(new InjectableValues() {

        @Override
        public Object findInjectableValue(Object valueId, DeserializationContext ctxt, BeanProperty forProperty, Object beanInstance) {
            return null;
        }
    });
    Assert.assertEquals(config, mapper.readValue(mapper.writeValueAsBytes(config), WorkerBehaviorConfig.class));
}
Also used : EC2EnvironmentConfig(io.druid.indexing.overlord.autoscaling.ec2.EC2EnvironmentConfig) InjectableValues(com.fasterxml.jackson.databind.InjectableValues) EC2AutoScaler(io.druid.indexing.overlord.autoscaling.ec2.EC2AutoScaler) BeanProperty(com.fasterxml.jackson.databind.BeanProperty) StringEC2UserData(io.druid.indexing.overlord.autoscaling.ec2.StringEC2UserData) DeserializationContext(com.fasterxml.jackson.databind.DeserializationContext) DefaultObjectMapper(io.druid.jackson.DefaultObjectMapper) EC2NodeData(io.druid.indexing.overlord.autoscaling.ec2.EC2NodeData) ObjectMapper(com.fasterxml.jackson.databind.ObjectMapper) DefaultObjectMapper(io.druid.jackson.DefaultObjectMapper) Test(org.junit.Test)

Example 2 with EC2AutoScaler

use of org.apache.druid.indexing.overlord.autoscaling.ec2.EC2AutoScaler in project druid by druid-io.

the class EC2AutoScalerSerdeTest method testSerde.

@Test
public void testSerde() throws Exception {
    final ObjectMapper objectMapper = new DefaultObjectMapper();
    objectMapper.setInjectableValues(new InjectableValues() {

        @Override
        public Object findInjectableValue(Object o, DeserializationContext deserializationContext, BeanProperty beanProperty, Object o1) {
            return null;
        }
    });
    final EC2AutoScaler autoScaler = objectMapper.readValue(json, EC2AutoScaler.class);
    verifyAutoScaler(autoScaler);
    final EC2AutoScaler roundTripAutoScaler = objectMapper.readValue(objectMapper.writeValueAsBytes(autoScaler), EC2AutoScaler.class);
    verifyAutoScaler(roundTripAutoScaler);
    Assert.assertEquals("Round trip equals", autoScaler, roundTripAutoScaler);
}
Also used : DeserializationContext(com.fasterxml.jackson.databind.DeserializationContext) DefaultObjectMapper(io.druid.jackson.DefaultObjectMapper) InjectableValues(com.fasterxml.jackson.databind.InjectableValues) DefaultObjectMapper(io.druid.jackson.DefaultObjectMapper) ObjectMapper(com.fasterxml.jackson.databind.ObjectMapper) EC2AutoScaler(io.druid.indexing.overlord.autoscaling.ec2.EC2AutoScaler) BeanProperty(com.fasterxml.jackson.databind.BeanProperty) Test(org.junit.Test)

Example 3 with EC2AutoScaler

use of org.apache.druid.indexing.overlord.autoscaling.ec2.EC2AutoScaler in project druid by druid-io.

the class EC2AutoScalerTest method testScale.

@Test
public void testScale() {
    RunInstancesResult runInstancesResult = EasyMock.createMock(RunInstancesResult.class);
    EC2AutoScaler autoScaler = new EC2AutoScaler(0, 1, ENV_CONFIG, amazonEC2Client, managementConfig);
    EasyMock.expect(amazonEC2Client.runInstances(EasyMock.anyObject(RunInstancesRequest.class))).andReturn(runInstancesResult);
    EasyMock.expect(amazonEC2Client.describeInstances(EasyMock.anyObject(DescribeInstancesRequest.class))).andReturn(describeInstancesResult);
    EasyMock.expect(amazonEC2Client.terminateInstances(EasyMock.anyObject(TerminateInstancesRequest.class))).andReturn(null);
    EasyMock.replay(amazonEC2Client);
    EasyMock.expect(runInstancesResult.getReservation()).andReturn(reservation).atLeastOnce();
    EasyMock.replay(runInstancesResult);
    EasyMock.expect(describeInstancesResult.getReservations()).andReturn(Collections.singletonList(reservation)).atLeastOnce();
    EasyMock.replay(describeInstancesResult);
    EasyMock.expect(reservation.getInstances()).andReturn(Collections.singletonList(instance)).atLeastOnce();
    EasyMock.replay(reservation);
    AutoScalingData created = autoScaler.provision();
    Assert.assertEquals(created.getNodeIds().size(), 1);
    Assert.assertEquals("theInstance", created.getNodeIds().get(0));
    AutoScalingData deleted = autoScaler.terminate(Arrays.asList("dummyIP"));
    Assert.assertEquals(deleted.getNodeIds().size(), 1);
    Assert.assertEquals(INSTANCE_ID, deleted.getNodeIds().get(0));
    EasyMock.verify(runInstancesResult);
}
Also used : RunInstancesResult(com.amazonaws.services.ec2.model.RunInstancesResult) RunInstancesRequest(com.amazonaws.services.ec2.model.RunInstancesRequest) DescribeInstancesRequest(com.amazonaws.services.ec2.model.DescribeInstancesRequest) EC2AutoScaler(io.druid.indexing.overlord.autoscaling.ec2.EC2AutoScaler) TerminateInstancesRequest(com.amazonaws.services.ec2.model.TerminateInstancesRequest) Test(org.junit.Test)

Example 4 with EC2AutoScaler

use of org.apache.druid.indexing.overlord.autoscaling.ec2.EC2AutoScaler in project druid by druid-io.

the class EC2AutoScalerTest method testIptoIdLookup.

@Test
public void testIptoIdLookup() throws Exception {
    EC2AutoScaler autoScaler = new EC2AutoScaler(0, 1, ENV_CONFIG, amazonEC2Client, managementConfig);
    final int n = 150;
    Assert.assertTrue(n <= 2 * EC2AutoScaler.MAX_AWS_FILTER_VALUES);
    List<String> ips = Lists.transform(ContiguousSet.create(Range.closedOpen(0, n), DiscreteDomain.integers()).asList(), Functions.toStringFunction());
    EasyMock.expect(amazonEC2Client.describeInstances(new DescribeInstancesRequest().withFilters(new Filter("private-ip-address", ips.subList(0, EC2AutoScaler.MAX_AWS_FILTER_VALUES))))).andReturn(describeInstancesResult);
    EasyMock.expect(amazonEC2Client.describeInstances(new DescribeInstancesRequest().withFilters(new Filter("private-ip-address", ips.subList(EC2AutoScaler.MAX_AWS_FILTER_VALUES, n))))).andReturn(describeInstancesResult);
    EasyMock.replay(amazonEC2Client);
    final Reservation[] chunk1 = new Reservation[EC2AutoScaler.MAX_AWS_FILTER_VALUES];
    Arrays.fill(chunk1, reservation);
    final Reservation[] chunk2 = new Reservation[n - EC2AutoScaler.MAX_AWS_FILTER_VALUES];
    Arrays.fill(chunk2, reservation);
    EasyMock.expect(describeInstancesResult.getReservations()).andReturn(Lists.newArrayList(chunk1));
    EasyMock.expect(describeInstancesResult.getReservations()).andReturn(Lists.newArrayList(chunk2));
    EasyMock.replay(describeInstancesResult);
    EasyMock.expect(reservation.getInstances()).andReturn(Arrays.asList(instance)).times(n);
    EasyMock.replay(reservation);
    List<String> ids = autoScaler.ipToIdLookup(ips);
    Assert.assertEquals(n, ids.size());
}
Also used : Reservation(com.amazonaws.services.ec2.model.Reservation) Filter(com.amazonaws.services.ec2.model.Filter) DescribeInstancesRequest(com.amazonaws.services.ec2.model.DescribeInstancesRequest) EC2AutoScaler(io.druid.indexing.overlord.autoscaling.ec2.EC2AutoScaler) Test(org.junit.Test)

Example 5 with EC2AutoScaler

use of org.apache.druid.indexing.overlord.autoscaling.ec2.EC2AutoScaler in project druid by druid-io.

the class EC2AutoScalerTest method testIdToIpLookup.

@Test
public void testIdToIpLookup() throws Exception {
    EC2AutoScaler autoScaler = new EC2AutoScaler(0, 1, ENV_CONFIG, amazonEC2Client, managementConfig);
    final int n = 150;
    Assert.assertTrue(n <= 2 * EC2AutoScaler.MAX_AWS_FILTER_VALUES);
    List<String> ids = Lists.transform(ContiguousSet.create(Range.closedOpen(0, n), DiscreteDomain.integers()).asList(), Functions.toStringFunction());
    EasyMock.expect(amazonEC2Client.describeInstances(new DescribeInstancesRequest().withFilters(new Filter("instance-id", ids.subList(0, EC2AutoScaler.MAX_AWS_FILTER_VALUES))))).andReturn(describeInstancesResult);
    EasyMock.expect(amazonEC2Client.describeInstances(new DescribeInstancesRequest().withFilters(new Filter("instance-id", ids.subList(EC2AutoScaler.MAX_AWS_FILTER_VALUES, n))))).andReturn(describeInstancesResult);
    EasyMock.replay(amazonEC2Client);
    final Reservation[] chunk1 = new Reservation[EC2AutoScaler.MAX_AWS_FILTER_VALUES];
    Arrays.fill(chunk1, reservation);
    final Reservation[] chunk2 = new Reservation[n - EC2AutoScaler.MAX_AWS_FILTER_VALUES];
    Arrays.fill(chunk2, reservation);
    EasyMock.expect(describeInstancesResult.getReservations()).andReturn(Lists.newArrayList(chunk1));
    EasyMock.expect(describeInstancesResult.getReservations()).andReturn(Lists.newArrayList(chunk2));
    EasyMock.replay(describeInstancesResult);
    EasyMock.expect(reservation.getInstances()).andReturn(Collections.singletonList(instance)).times(n);
    EasyMock.replay(reservation);
    List<String> ips = autoScaler.idToIpLookup(ids);
    Assert.assertEquals(n, ips.size());
}
Also used : Reservation(com.amazonaws.services.ec2.model.Reservation) Filter(com.amazonaws.services.ec2.model.Filter) DescribeInstancesRequest(com.amazonaws.services.ec2.model.DescribeInstancesRequest) EC2AutoScaler(io.druid.indexing.overlord.autoscaling.ec2.EC2AutoScaler) Test(org.junit.Test)

Aggregations

Test (org.junit.Test)6 EC2AutoScaler (io.druid.indexing.overlord.autoscaling.ec2.EC2AutoScaler)5 DescribeInstancesRequest (com.amazonaws.services.ec2.model.DescribeInstancesRequest)3 BeanProperty (com.fasterxml.jackson.databind.BeanProperty)3 DeserializationContext (com.fasterxml.jackson.databind.DeserializationContext)3 InjectableValues (com.fasterxml.jackson.databind.InjectableValues)3 ObjectMapper (com.fasterxml.jackson.databind.ObjectMapper)3 Filter (com.amazonaws.services.ec2.model.Filter)2 Reservation (com.amazonaws.services.ec2.model.Reservation)2 DefaultObjectMapper (io.druid.jackson.DefaultObjectMapper)2 RunInstancesRequest (com.amazonaws.services.ec2.model.RunInstancesRequest)1 RunInstancesResult (com.amazonaws.services.ec2.model.RunInstancesResult)1 TerminateInstancesRequest (com.amazonaws.services.ec2.model.TerminateInstancesRequest)1 EC2EnvironmentConfig (io.druid.indexing.overlord.autoscaling.ec2.EC2EnvironmentConfig)1 EC2NodeData (io.druid.indexing.overlord.autoscaling.ec2.EC2NodeData)1 StringEC2UserData (io.druid.indexing.overlord.autoscaling.ec2.StringEC2UserData)1 EC2AutoScaler (org.apache.druid.indexing.overlord.autoscaling.ec2.EC2AutoScaler)1 EC2EnvironmentConfig (org.apache.druid.indexing.overlord.autoscaling.ec2.EC2EnvironmentConfig)1 EC2NodeData (org.apache.druid.indexing.overlord.autoscaling.ec2.EC2NodeData)1 StringEC2UserData (org.apache.druid.indexing.overlord.autoscaling.ec2.StringEC2UserData)1