Examples with TServerInstance org.apache.accumulo.server.master.state.TServerInstance used on opensource projects

Search in sources :

Example 46 with TServerInstance

use of org.apache.accumulo.server.master.state.TServerInstance in project accumulo by apache.

the class DefaultLoadBalancerTest method testUnevenAssignment2.

@Test
public void testUnevenAssignment2() {
    // make 26 servers
    for (char c : "abcdefghijklmnopqrstuvwxyz".toCharArray()) {
        String cString = Character.toString(c);
        HostAndPort fakeAddress = HostAndPort.fromParts("127.0.0.1", c);
        String fakeInstance = cString;
        TServerInstance tsi = new TServerInstance(fakeAddress, fakeInstance);
        FakeTServer fakeTServer = new FakeTServer();
        servers.put(tsi, fakeTServer);
    }
    // put 60 tablets on 25 of them
    List<Entry<TServerInstance, FakeTServer>> shortList = new ArrayList<>(servers.entrySet());
    Entry<TServerInstance, FakeTServer> shortServer = shortList.remove(0);
    int c = 0;
    for (int i = 0; i < 60; i++) {
        for (Entry<TServerInstance, FakeTServer> entry : shortList) {
            entry.getValue().extents.add(makeExtent("t" + c, null, null));
        }
    }
    // put 10 on the that short server:
    for (int i = 0; i < 10; i++) {
        shortServer.getValue().extents.add(makeExtent("s" + i, null, null));
    }
    TestDefaultLoadBalancer balancer = new TestDefaultLoadBalancer();
    Set<KeyExtent> migrations = Collections.emptySet();
    int moved = 0;
    // balance until we can't balance no more!
    while (true) {
        List<TabletMigration> migrationsOut = new ArrayList<>();
        balancer.balance(getAssignments(servers), migrations, migrationsOut);
        if (migrationsOut.size() == 0)
            break;
        for (TabletMigration migration : migrationsOut) {
            if (servers.get(migration.oldServer).extents.remove(migration.tablet))
                moved++;
            last.remove(migration.tablet);
            servers.get(migration.newServer).extents.add(migration.tablet);
            last.put(migration.tablet, migration.newServer);
        }
    }
    // average is 58, with 2 at 59: we need 48 more moved to the short server
    assertEquals(48, moved);
}
Also used : TabletMigration(org.apache.accumulo.server.master.state.TabletMigration) ArrayList(java.util.ArrayList) KeyExtent(org.apache.accumulo.core.data.impl.KeyExtent) TServerInstance(org.apache.accumulo.server.master.state.TServerInstance) HostAndPort(org.apache.accumulo.core.util.HostAndPort) Entry(java.util.Map.Entry) Test(org.junit.Test)

Example 47 with TServerInstance

use of org.apache.accumulo.server.master.state.TServerInstance in project accumulo by apache.

the class HostRegexTableLoadBalancerTest method testSplitCurrentByRegexUsingOverlappingPools.

@Test
public void testSplitCurrentByRegexUsingOverlappingPools() {
    init(new AccumuloServerContext(instance, new TestServerConfigurationFactory(instance) {

        @Override
        public TableConfiguration getTableConfiguration(Table.ID tableId) {
            NamespaceConfiguration defaultConf = new NamespaceConfiguration(Namespace.ID.DEFAULT, this.instance, DefaultConfiguration.getInstance());
            return new TableConfiguration(instance, tableId, defaultConf) {

                HashMap<String, String> tableProperties = new HashMap<>();

                {
                    tableProperties.put(HostRegexTableLoadBalancer.HOST_BALANCER_PREFIX + FOO.getTableName(), "r.*");
                    tableProperties.put(HostRegexTableLoadBalancer.HOST_BALANCER_PREFIX + BAR.getTableName(), "r01.*|r02.*");
                }

                @Override
                public String get(Property property) {
                    return tableProperties.get(property.name());
                }

                @Override
                public void getProperties(Map<String, String> props, Predicate<String> filter) {
                    for (Entry<String, String> e : tableProperties.entrySet()) {
                        if (filter.test(e.getKey())) {
                            props.put(e.getKey(), e.getValue());
                        }
                    }
                }

                @Override
                public long getUpdateCount() {
                    return 0;
                }
            };
        }
    }));
    Map<String, SortedMap<TServerInstance, TabletServerStatus>> groups = this.splitCurrentByRegex(createCurrent(15));
    Assert.assertEquals(2, groups.size());
    Assert.assertTrue(groups.containsKey(FOO.getTableName()));
    SortedMap<TServerInstance, TabletServerStatus> fooHosts = groups.get(FOO.getTableName());
    Assert.assertEquals(15, fooHosts.size());
    Assert.assertTrue(fooHosts.containsKey(new TServerInstance("192.168.0.1:9997", 1)));
    Assert.assertTrue(fooHosts.containsKey(new TServerInstance("192.168.0.2:9997", 1)));
    Assert.assertTrue(fooHosts.containsKey(new TServerInstance("192.168.0.3:9997", 1)));
    Assert.assertTrue(fooHosts.containsKey(new TServerInstance("192.168.0.4:9997", 1)));
    Assert.assertTrue(fooHosts.containsKey(new TServerInstance("192.168.0.5:9997", 1)));
    Assert.assertTrue(fooHosts.containsKey(new TServerInstance("192.168.0.6:9997", 1)));
    Assert.assertTrue(fooHosts.containsKey(new TServerInstance("192.168.0.7:9997", 1)));
    Assert.assertTrue(fooHosts.containsKey(new TServerInstance("192.168.0.8:9997", 1)));
    Assert.assertTrue(fooHosts.containsKey(new TServerInstance("192.168.0.9:9997", 1)));
    Assert.assertTrue(fooHosts.containsKey(new TServerInstance("192.168.0.10:9997", 1)));
    Assert.assertTrue(fooHosts.containsKey(new TServerInstance("192.168.0.11:9997", 1)));
    Assert.assertTrue(fooHosts.containsKey(new TServerInstance("192.168.0.12:9997", 1)));
    Assert.assertTrue(fooHosts.containsKey(new TServerInstance("192.168.0.13:9997", 1)));
    Assert.assertTrue(fooHosts.containsKey(new TServerInstance("192.168.0.14:9997", 1)));
    Assert.assertTrue(fooHosts.containsKey(new TServerInstance("192.168.0.15:9997", 1)));
    Assert.assertTrue(groups.containsKey(BAR.getTableName()));
    SortedMap<TServerInstance, TabletServerStatus> barHosts = groups.get(BAR.getTableName());
    Assert.assertEquals(10, barHosts.size());
    Assert.assertTrue(barHosts.containsKey(new TServerInstance("192.168.0.1:9997", 1)));
    Assert.assertTrue(barHosts.containsKey(new TServerInstance("192.168.0.2:9997", 1)));
    Assert.assertTrue(barHosts.containsKey(new TServerInstance("192.168.0.3:9997", 1)));
    Assert.assertTrue(barHosts.containsKey(new TServerInstance("192.168.0.4:9997", 1)));
    Assert.assertTrue(barHosts.containsKey(new TServerInstance("192.168.0.5:9997", 1)));
    Assert.assertTrue(barHosts.containsKey(new TServerInstance("192.168.0.6:9997", 1)));
    Assert.assertTrue(barHosts.containsKey(new TServerInstance("192.168.0.7:9997", 1)));
    Assert.assertTrue(barHosts.containsKey(new TServerInstance("192.168.0.8:9997", 1)));
    Assert.assertTrue(barHosts.containsKey(new TServerInstance("192.168.0.9:9997", 1)));
    Assert.assertTrue(barHosts.containsKey(new TServerInstance("192.168.0.10:9997", 1)));
}
Also used : AccumuloServerContext(org.apache.accumulo.server.AccumuloServerContext) HashMap(java.util.HashMap) TServerInstance(org.apache.accumulo.server.master.state.TServerInstance) Predicate(java.util.function.Predicate) SortedMap(java.util.SortedMap) NamespaceConfiguration(org.apache.accumulo.server.conf.NamespaceConfiguration) Property(org.apache.accumulo.core.conf.Property) HashMap(java.util.HashMap) Map(java.util.Map) SortedMap(java.util.SortedMap) TableConfiguration(org.apache.accumulo.server.conf.TableConfiguration) TabletServerStatus(org.apache.accumulo.core.master.thrift.TabletServerStatus) Test(org.junit.Test)

Example 48 with TServerInstance

use of org.apache.accumulo.server.master.state.TServerInstance in project accumulo by apache.

the class HostRegexTableLoadBalancerTest method testSplitCurrentByRegexUsingIP.

@Test
public void testSplitCurrentByRegexUsingIP() {
    init(new AccumuloServerContext(instance, new TestServerConfigurationFactory(instance) {

        @Override
        public synchronized AccumuloConfiguration getSystemConfiguration() {
            HashMap<String, String> props = new HashMap<>();
            props.put(HostRegexTableLoadBalancer.HOST_BALANCER_OOB_CHECK_KEY, "30s");
            props.put(HostRegexTableLoadBalancer.HOST_BALANCER_REGEX_USING_IPS_KEY, "true");
            return new ConfigurationCopy(props);
        }

        @Override
        public TableConfiguration getTableConfiguration(Table.ID tableId) {
            NamespaceConfiguration defaultConf = new NamespaceConfiguration(Namespace.ID.DEFAULT, this.instance, DefaultConfiguration.getInstance());
            return new TableConfiguration(instance, tableId, defaultConf) {

                HashMap<String, String> tableProperties = new HashMap<>();

                {
                    tableProperties.put(HostRegexTableLoadBalancer.HOST_BALANCER_PREFIX + FOO.getTableName(), "192\\.168\\.0\\.[1-5]");
                    tableProperties.put(HostRegexTableLoadBalancer.HOST_BALANCER_PREFIX + BAR.getTableName(), "192\\.168\\.0\\.[6-9]|192\\.168\\.0\\.10");
                }

                @Override
                public String get(Property property) {
                    return tableProperties.get(property.name());
                }

                @Override
                public void getProperties(Map<String, String> props, Predicate<String> filter) {
                    for (Entry<String, String> e : tableProperties.entrySet()) {
                        if (filter.test(e.getKey())) {
                            props.put(e.getKey(), e.getValue());
                        }
                    }
                }

                @Override
                public long getUpdateCount() {
                    return 0;
                }
            };
        }
    }));
    Assert.assertTrue(isIpBasedRegex());
    Map<String, SortedMap<TServerInstance, TabletServerStatus>> groups = this.splitCurrentByRegex(createCurrent(15));
    Assert.assertEquals(3, groups.size());
    Assert.assertTrue(groups.containsKey(FOO.getTableName()));
    SortedMap<TServerInstance, TabletServerStatus> fooHosts = groups.get(FOO.getTableName());
    Assert.assertEquals(5, fooHosts.size());
    Assert.assertTrue(fooHosts.containsKey(new TServerInstance("192.168.0.1:9997", 1)));
    Assert.assertTrue(fooHosts.containsKey(new TServerInstance("192.168.0.2:9997", 1)));
    Assert.assertTrue(fooHosts.containsKey(new TServerInstance("192.168.0.3:9997", 1)));
    Assert.assertTrue(fooHosts.containsKey(new TServerInstance("192.168.0.4:9997", 1)));
    Assert.assertTrue(fooHosts.containsKey(new TServerInstance("192.168.0.5:9997", 1)));
    Assert.assertTrue(groups.containsKey(BAR.getTableName()));
    SortedMap<TServerInstance, TabletServerStatus> barHosts = groups.get(BAR.getTableName());
    Assert.assertEquals(5, barHosts.size());
    Assert.assertTrue(barHosts.containsKey(new TServerInstance("192.168.0.6:9997", 1)));
    Assert.assertTrue(barHosts.containsKey(new TServerInstance("192.168.0.7:9997", 1)));
    Assert.assertTrue(barHosts.containsKey(new TServerInstance("192.168.0.8:9997", 1)));
    Assert.assertTrue(barHosts.containsKey(new TServerInstance("192.168.0.9:9997", 1)));
    Assert.assertTrue(barHosts.containsKey(new TServerInstance("192.168.0.10:9997", 1)));
    Assert.assertTrue(groups.containsKey(DEFAULT_POOL));
    SortedMap<TServerInstance, TabletServerStatus> defHosts = groups.get(DEFAULT_POOL);
    Assert.assertEquals(5, defHosts.size());
    Assert.assertTrue(defHosts.containsKey(new TServerInstance("192.168.0.11:9997", 1)));
    Assert.assertTrue(defHosts.containsKey(new TServerInstance("192.168.0.12:9997", 1)));
    Assert.assertTrue(defHosts.containsKey(new TServerInstance("192.168.0.13:9997", 1)));
    Assert.assertTrue(defHosts.containsKey(new TServerInstance("192.168.0.14:9997", 1)));
    Assert.assertTrue(defHosts.containsKey(new TServerInstance("192.168.0.15:9997", 1)));
}
Also used : AccumuloServerContext(org.apache.accumulo.server.AccumuloServerContext) ConfigurationCopy(org.apache.accumulo.core.conf.ConfigurationCopy) HashMap(java.util.HashMap) TServerInstance(org.apache.accumulo.server.master.state.TServerInstance) Predicate(java.util.function.Predicate) SortedMap(java.util.SortedMap) NamespaceConfiguration(org.apache.accumulo.server.conf.NamespaceConfiguration) Property(org.apache.accumulo.core.conf.Property) HashMap(java.util.HashMap) Map(java.util.Map) SortedMap(java.util.SortedMap) TableConfiguration(org.apache.accumulo.server.conf.TableConfiguration) TabletServerStatus(org.apache.accumulo.core.master.thrift.TabletServerStatus) Test(org.junit.Test)

Example 49 with TServerInstance

use of org.apache.accumulo.server.master.state.TServerInstance in project accumulo by apache.

the class HostRegexTableLoadBalancerTest method testSplitCurrentByRegexUsingHostname.

@Test
public void testSplitCurrentByRegexUsingHostname() {
    init(new AccumuloServerContext(instance, factory));
    Map<String, SortedMap<TServerInstance, TabletServerStatus>> groups = this.splitCurrentByRegex(createCurrent(15));
    Assert.assertEquals(3, groups.size());
    Assert.assertTrue(groups.containsKey(FOO.getTableName()));
    SortedMap<TServerInstance, TabletServerStatus> fooHosts = groups.get(FOO.getTableName());
    Assert.assertEquals(5, fooHosts.size());
    Assert.assertTrue(fooHosts.containsKey(new TServerInstance("192.168.0.1:9997", 1)));
    Assert.assertTrue(fooHosts.containsKey(new TServerInstance("192.168.0.2:9997", 1)));
    Assert.assertTrue(fooHosts.containsKey(new TServerInstance("192.168.0.3:9997", 1)));
    Assert.assertTrue(fooHosts.containsKey(new TServerInstance("192.168.0.4:9997", 1)));
    Assert.assertTrue(fooHosts.containsKey(new TServerInstance("192.168.0.5:9997", 1)));
    Assert.assertTrue(groups.containsKey(BAR.getTableName()));
    SortedMap<TServerInstance, TabletServerStatus> barHosts = groups.get(BAR.getTableName());
    Assert.assertEquals(5, barHosts.size());
    Assert.assertTrue(barHosts.containsKey(new TServerInstance("192.168.0.6:9997", 1)));
    Assert.assertTrue(barHosts.containsKey(new TServerInstance("192.168.0.7:9997", 1)));
    Assert.assertTrue(barHosts.containsKey(new TServerInstance("192.168.0.8:9997", 1)));
    Assert.assertTrue(barHosts.containsKey(new TServerInstance("192.168.0.9:9997", 1)));
    Assert.assertTrue(barHosts.containsKey(new TServerInstance("192.168.0.10:9997", 1)));
    Assert.assertTrue(groups.containsKey(DEFAULT_POOL));
    SortedMap<TServerInstance, TabletServerStatus> defHosts = groups.get(DEFAULT_POOL);
    Assert.assertEquals(5, defHosts.size());
    Assert.assertTrue(defHosts.containsKey(new TServerInstance("192.168.0.11:9997", 1)));
    Assert.assertTrue(defHosts.containsKey(new TServerInstance("192.168.0.12:9997", 1)));
    Assert.assertTrue(defHosts.containsKey(new TServerInstance("192.168.0.13:9997", 1)));
    Assert.assertTrue(defHosts.containsKey(new TServerInstance("192.168.0.14:9997", 1)));
    Assert.assertTrue(defHosts.containsKey(new TServerInstance("192.168.0.15:9997", 1)));
}
Also used : AccumuloServerContext(org.apache.accumulo.server.AccumuloServerContext) SortedMap(java.util.SortedMap) TServerInstance(org.apache.accumulo.server.master.state.TServerInstance) TabletServerStatus(org.apache.accumulo.core.master.thrift.TabletServerStatus) Test(org.junit.Test)

Example 50 with TServerInstance

use of org.apache.accumulo.server.master.state.TServerInstance in project accumulo by apache.

the class HostRegexTableLoadBalancerTest method testAllAssigned.

@Test
public void testAllAssigned() {
    init(new AccumuloServerContext(instance, factory));
    Map<KeyExtent, TServerInstance> assignments = new HashMap<>();
    Map<KeyExtent, TServerInstance> unassigned = new HashMap<>();
    this.getAssignments(Collections.unmodifiableSortedMap(allTabletServers), Collections.unmodifiableMap(unassigned), assignments);
    Assert.assertEquals(0, assignments.size());
}
Also used : AccumuloServerContext(org.apache.accumulo.server.AccumuloServerContext) HashMap(java.util.HashMap) TKeyExtent(org.apache.accumulo.core.data.thrift.TKeyExtent) KeyExtent(org.apache.accumulo.core.data.impl.KeyExtent) TServerInstance(org.apache.accumulo.server.master.state.TServerInstance) Test(org.junit.Test)

Aggregations

TServerInstance (org.apache.accumulo.server.master.state.TServerInstance)67 KeyExtent (org.apache.accumulo.core.data.impl.KeyExtent)35 HashMap (java.util.HashMap)22 Test (org.junit.Test)22 ArrayList (java.util.ArrayList)21 TabletServerStatus (org.apache.accumulo.core.master.thrift.TabletServerStatus)14 HashSet (java.util.HashSet)10 Value (org.apache.accumulo.core.data.Value)10 AccumuloServerContext (org.apache.accumulo.server.AccumuloServerContext)10 TServerConnection (org.apache.accumulo.server.master.LiveTServerSet.TServerConnection)9 TabletMigration (org.apache.accumulo.server.master.state.TabletMigration)9 TreeMap (java.util.TreeMap)8 Instance (org.apache.accumulo.core.client.Instance)8 Table (org.apache.accumulo.core.client.impl.Table)8 TKeyExtent (org.apache.accumulo.core.data.thrift.TKeyExtent)8 UUID (java.util.UUID)7 Key (org.apache.accumulo.core.data.Key)7 Text (org.apache.hadoop.io.Text)7 TException (org.apache.thrift.TException)7 List (java.util.List)6
About Me
UseOf

Java Tutorials :

Simple Java RabbitMQ Example with Spring
Simple Springboot JPA Eclipeslink Example
Simple SpringBoot JPA Hibernate Example
Jackson JSON @JsonIgnore and @JsonIgnoreProperties Examples
Java Infinite recursion (StackOverflowError) Jackson solutions
Simple Java Jackson Serialize Objects to JSON and convert them back To Object
ElasticSearch 5 - Upload files using Java API in binary field Example
Java JAXB marshall unmarshall Examples from String and Stream
Java 8 forEach List and Map examples
ElasticSearch 5 Java API SearchRequestBuilder examples
Java ElasticSearch 5 examples with node index and mapping - running local
Convert String to int or Integer Java 8
Java 9 in action - JShell - Ubuntu
Java - removing invalid characters from a file name
Hello world!? or Hello Tutorial