Search in sources :

Example 11 with ServicePropertiesJsonSerializer

use of com.linkedin.d2.balancer.properties.ServicePropertiesJsonSerializer in project rest.li by linkedin.

the class LoadBalancerClientCli method getLoadBalancer.

public static SimpleLoadBalancer getLoadBalancer(ZKConnection zkclient, String zkserver, String d2path, String service) throws IOException, IllegalStateException, URISyntaxException, PropertyStoreException, ExecutionException, TimeoutException, InterruptedException {
    // zk stores
    String clstoreString = zkserver + ZKFSUtil.clusterPath(d2path);
    String scstoreString = zkserver + ZKFSUtil.servicePath(d2path);
    String uristoreString = zkserver + ZKFSUtil.uriPath(d2path);
    ZooKeeperPermanentStore<ClusterProperties> zkClusterRegistry = (ZooKeeperPermanentStore<ClusterProperties>) getStore(zkclient, clstoreString, new ClusterPropertiesJsonSerializer());
    ZooKeeperPermanentStore<ServiceProperties> zkServiceRegistry = (ZooKeeperPermanentStore<ServiceProperties>) getStore(zkclient, scstoreString, new ServicePropertiesJsonSerializer());
    ZooKeeperEphemeralStore<UriProperties> zkUriRegistry = (ZooKeeperEphemeralStore<UriProperties>) getEphemeralStore(zkclient, uristoreString, new UriPropertiesJsonSerializer(), new UriPropertiesMerger());
    ScheduledThreadPoolExecutor executor = new ScheduledThreadPoolExecutor(1, new NamedThreadFactory("D2 PropertyEventExecutor"));
    PropertyEventBus<ServiceProperties> serviceBus = new PropertyEventBusImpl<ServiceProperties>(executor, zkServiceRegistry);
    PropertyEventBus<UriProperties> uriBus = new PropertyEventBusImpl<UriProperties>(executor, zkUriRegistry);
    PropertyEventBus<ClusterProperties> clusterBus = new PropertyEventBusImpl<ClusterProperties>(executor, zkClusterRegistry);
    Map<String, LoadBalancerStrategyFactory<? extends LoadBalancerStrategy>> loadBalancerStrategyFactories = new HashMap<String, LoadBalancerStrategyFactory<? extends LoadBalancerStrategy>>();
    loadBalancerStrategyFactories.put("random", new RandomLoadBalancerStrategyFactory());
    loadBalancerStrategyFactories.put("degrader", new DegraderLoadBalancerStrategyFactoryV2());
    loadBalancerStrategyFactories.put("degraderV2", new DegraderLoadBalancerStrategyFactoryV2());
    loadBalancerStrategyFactories.put("degraderV3", new DegraderLoadBalancerStrategyFactoryV3());
    loadBalancerStrategyFactories.put("degraderV2_1", new DegraderLoadBalancerStrategyFactoryV2_1());
    Map<String, TransportClientFactory> clientFactories = new HashMap<String, TransportClientFactory>();
    clientFactories.put("http", new HttpClientFactory());
    // create the state
    SimpleLoadBalancerState state = new SimpleLoadBalancerState(executor, uriBus, clusterBus, serviceBus, clientFactories, loadBalancerStrategyFactories, null, null, false);
    SimpleLoadBalancer balancer = new SimpleLoadBalancer(state, 5, TimeUnit.SECONDS);
    FutureCallback<None> callback = new FutureCallback<None>();
    balancer.start(callback);
    callback.get(5, TimeUnit.SECONDS);
    new JmxManager().registerLoadBalancer("balancer", balancer).registerLoadBalancerState("state", state).registerScheduledThreadPoolExecutor("executorService", executor).registerZooKeeperPermanentStore("zkClusterRegistry", zkClusterRegistry).registerZooKeeperPermanentStore("zkServiceRegistry", zkServiceRegistry).registerZooKeeperEphemeralStore("zkUriRegistry", zkUriRegistry);
    return balancer;
}
Also used : DegraderLoadBalancerStrategyFactoryV2(com.linkedin.d2.balancer.strategies.degrader.DegraderLoadBalancerStrategyFactoryV2) ServicePropertiesJsonSerializer(com.linkedin.d2.balancer.properties.ServicePropertiesJsonSerializer) HashMap(java.util.HashMap) DegraderLoadBalancerStrategyFactoryV3(com.linkedin.d2.balancer.strategies.degrader.DegraderLoadBalancerStrategyFactoryV3) ScheduledThreadPoolExecutor(java.util.concurrent.ScheduledThreadPoolExecutor) ZooKeeperPermanentStore(com.linkedin.d2.discovery.stores.zk.ZooKeeperPermanentStore) UriPropertiesMerger(com.linkedin.d2.balancer.properties.UriPropertiesMerger) UriPropertiesJsonSerializer(com.linkedin.d2.balancer.properties.UriPropertiesJsonSerializer) JmxManager(com.linkedin.d2.jmx.JmxManager) ClusterPropertiesJsonSerializer(com.linkedin.d2.balancer.properties.ClusterPropertiesJsonSerializer) UriProperties(com.linkedin.d2.balancer.properties.UriProperties) TransportClientFactory(com.linkedin.r2.transport.common.TransportClientFactory) HttpClientFactory(com.linkedin.r2.transport.http.client.HttpClientFactory) FutureCallback(com.linkedin.common.callback.FutureCallback) SimpleLoadBalancerState(com.linkedin.d2.balancer.simple.SimpleLoadBalancerState) SimpleLoadBalancer(com.linkedin.d2.balancer.simple.SimpleLoadBalancer) RandomLoadBalancerStrategyFactory(com.linkedin.d2.balancer.strategies.random.RandomLoadBalancerStrategyFactory) LoadBalancerStrategyFactory(com.linkedin.d2.balancer.strategies.LoadBalancerStrategyFactory) NamedThreadFactory(com.linkedin.r2.util.NamedThreadFactory) RandomLoadBalancerStrategyFactory(com.linkedin.d2.balancer.strategies.random.RandomLoadBalancerStrategyFactory) LoadBalancerStrategy(com.linkedin.d2.balancer.strategies.LoadBalancerStrategy) ZooKeeperEphemeralStore(com.linkedin.d2.discovery.stores.zk.ZooKeeperEphemeralStore) ServiceProperties(com.linkedin.d2.balancer.properties.ServiceProperties) DegraderLoadBalancerStrategyFactoryV2_1(com.linkedin.d2.balancer.strategies.degrader.DegraderLoadBalancerStrategyFactoryV2_1) PropertyEventBusImpl(com.linkedin.d2.discovery.event.PropertyEventBusImpl) ClusterProperties(com.linkedin.d2.balancer.properties.ClusterProperties) None(com.linkedin.common.util.None)

Example 12 with ServicePropertiesJsonSerializer

use of com.linkedin.d2.balancer.properties.ServicePropertiesJsonSerializer in project rest.li by linkedin.

the class LoadBalancerEchoClient method getLoadBalancer.

public static SimpleLoadBalancer getLoadBalancer(String hostPort) throws IOException, PropertyStoreException {
    // zk stores
    ZooKeeperPermanentStore<ClusterProperties> zkClusterRegistry = null;
    ZooKeeperPermanentStore<ServiceProperties> zkServiceRegistry = null;
    ZooKeeperEphemeralStore<UriProperties> zkUriRegistry = null;
    ZKConnection zkClient = new ZKConnection(hostPort, 10000);
    zkClusterRegistry = new ZooKeeperPermanentStore<ClusterProperties>(zkClient, new ClusterPropertiesJsonSerializer(), _basePath + "/clusters");
    zkServiceRegistry = new ZooKeeperPermanentStore<ServiceProperties>(zkClient, new ServicePropertiesJsonSerializer(), _basePath + "/services");
    zkUriRegistry = new ZooKeeperEphemeralStore<UriProperties>(zkClient, new UriPropertiesJsonSerializer(), new UriPropertiesMerger(), _basePath + "/uris", false, true);
    // fs stores
    File testDirectory = LoadBalancerUtil.createTempDirectory("lb-degrader-witih-file-store-large");
    testDirectory.deleteOnExit();
    new File(testDirectory + File.separator + "cluster").mkdir();
    new File(testDirectory + File.separator + "service").mkdir();
    new File(testDirectory + File.separator + "uri").mkdir();
    FileStore<ClusterProperties> fsClusterStore = new FileStore<ClusterProperties>(testDirectory + File.separator + "cluster", ".ini", new ClusterPropertiesJsonSerializer());
    FileStore<ServiceProperties> fsServiceStore = new FileStore<ServiceProperties>(testDirectory + File.separator + "service", ".ini", new ServicePropertiesJsonSerializer());
    FileStore<UriProperties> fsUriStore = new FileStore<UriProperties>(testDirectory + File.separator + "uri", ".ini", new UriPropertiesJsonSerializer());
    // chains
    PropertyEventThread thread = new PropertyEventThread("echo client event thread");
    ScheduledExecutorService executorService = Executors.newSingleThreadScheduledExecutor(new NamedThreadFactory("echo client event thread"));
    // start up the world
    thread.start();
    PropertyEventBus<ServiceProperties> serviceBus = new PropertyEventBusImpl<ServiceProperties>(executorService, zkServiceRegistry);
    serviceBus.register(fsServiceStore);
    new ZooKeeperTogglingStore<ServiceProperties>(zkServiceRegistry, fsServiceStore, serviceBus, true);
    PropertyEventBus<UriProperties> uriBus = new PropertyEventBusImpl<UriProperties>(executorService, zkUriRegistry);
    uriBus.register(fsUriStore);
    new ZooKeeperTogglingStore<UriProperties>(zkUriRegistry, fsUriStore, uriBus, true);
    PropertyEventBus<ClusterProperties> clusterBus = new PropertyEventBusImpl<ClusterProperties>(executorService, zkClusterRegistry);
    clusterBus.register(fsClusterStore);
    new ZooKeeperTogglingStore<ClusterProperties>(zkClusterRegistry, fsClusterStore, clusterBus, true);
    Map<String, LoadBalancerStrategyFactory<? extends LoadBalancerStrategy>> loadBalancerStrategyFactories = new HashMap<String, LoadBalancerStrategyFactory<? extends LoadBalancerStrategy>>();
    // strategy and scheme factories
    loadBalancerStrategyFactories.put("degrader", new DegraderLoadBalancerStrategyFactoryV3());
    Map<String, TransportClientFactory> clientFactories = new HashMap<String, TransportClientFactory>();
    clientFactories.put("http", new HttpClientFactory());
    // create the state
    SimpleLoadBalancerState state = new SimpleLoadBalancerState(executorService, uriBus, clusterBus, serviceBus, clientFactories, loadBalancerStrategyFactories, null, null, false);
    SimpleLoadBalancer balancer = new SimpleLoadBalancer(state, 5, TimeUnit.SECONDS);
    new JmxManager().registerLoadBalancer("balancer", balancer).registerLoadBalancerState("state", state);
    return balancer;
}
Also used : ServicePropertiesJsonSerializer(com.linkedin.d2.balancer.properties.ServicePropertiesJsonSerializer) HashMap(java.util.HashMap) DegraderLoadBalancerStrategyFactoryV3(com.linkedin.d2.balancer.strategies.degrader.DegraderLoadBalancerStrategyFactoryV3) UriPropertiesMerger(com.linkedin.d2.balancer.properties.UriPropertiesMerger) ZooKeeperTogglingStore(com.linkedin.d2.discovery.stores.zk.ZooKeeperTogglingStore) UriPropertiesJsonSerializer(com.linkedin.d2.balancer.properties.UriPropertiesJsonSerializer) JmxManager(com.linkedin.d2.jmx.JmxManager) ClusterPropertiesJsonSerializer(com.linkedin.d2.balancer.properties.ClusterPropertiesJsonSerializer) UriProperties(com.linkedin.d2.balancer.properties.UriProperties) TransportClientFactory(com.linkedin.r2.transport.common.TransportClientFactory) HttpClientFactory(com.linkedin.r2.transport.http.client.HttpClientFactory) ScheduledExecutorService(java.util.concurrent.ScheduledExecutorService) SimpleLoadBalancerState(com.linkedin.d2.balancer.simple.SimpleLoadBalancerState) SimpleLoadBalancer(com.linkedin.d2.balancer.simple.SimpleLoadBalancer) ZKConnection(com.linkedin.d2.discovery.stores.zk.ZKConnection) LoadBalancerStrategyFactory(com.linkedin.d2.balancer.strategies.LoadBalancerStrategyFactory) NamedThreadFactory(com.linkedin.r2.util.NamedThreadFactory) LoadBalancerStrategy(com.linkedin.d2.balancer.strategies.LoadBalancerStrategy) FileStore(com.linkedin.d2.discovery.stores.file.FileStore) ServiceProperties(com.linkedin.d2.balancer.properties.ServiceProperties) PropertyEventBusImpl(com.linkedin.d2.discovery.event.PropertyEventBusImpl) ClusterProperties(com.linkedin.d2.balancer.properties.ClusterProperties) File(java.io.File) PropertyEventThread(com.linkedin.d2.discovery.event.PropertyEventThread)

Example 13 with ServicePropertiesJsonSerializer

use of com.linkedin.d2.balancer.properties.ServicePropertiesJsonSerializer in project rest.li by linkedin.

the class ZKFSTest method testZKDown.

@Test
public void testZKDown() throws Exception {
    final String TEST_SERVICE_NAME = "testingService";
    final String TEST_CLUSTER_NAME = "someCluster";
    startServer();
    try {
        ZKFSLoadBalancer balancer = getBalancer();
        FutureCallback<None> callback = new FutureCallback<None>();
        balancer.start(callback);
        callback.get(30, TimeUnit.SECONDS);
        ZKConnection conn = new ZKConnection("localhost:" + PORT, 30000);
        conn.start();
        ZooKeeperPermanentStore<ServiceProperties> store = new ZooKeeperPermanentStore<ServiceProperties>(conn, new ServicePropertiesJsonSerializer(), ZKFSUtil.servicePath(BASE_PATH));
        callback = new FutureCallback<None>();
        store.start(callback);
        callback.get(30, TimeUnit.SECONDS);
        ServiceProperties props = new ServiceProperties(TEST_SERVICE_NAME, TEST_CLUSTER_NAME, "/somePath", Arrays.asList("degrader"), Collections.<String, Object>emptyMap(), null, null, Arrays.asList("http"), null);
        store.put(TEST_SERVICE_NAME, props);
        ZooKeeperPermanentStore<ClusterProperties> clusterStore = new ZooKeeperPermanentStore<ClusterProperties>(conn, new ClusterPropertiesJsonSerializer(), ZKFSUtil.clusterPath(BASE_PATH));
        callback = new FutureCallback<None>();
        clusterStore.start(callback);
        callback.get(30, TimeUnit.SECONDS);
        ClusterProperties clusterProps = new ClusterProperties("someCluster");
        clusterStore.put(TEST_CLUSTER_NAME, clusterProps);
        ZKConnection serverConn = new ZKConnection("localhost:" + PORT, 30000);
        serverConn.start();
        ZooKeeperEphemeralStore<UriProperties> uriStore = new ZooKeeperEphemeralStore<UriProperties>(serverConn, new UriPropertiesJsonSerializer(), new UriPropertiesMerger(), ZKFSUtil.uriPath(BASE_PATH));
        callback = new FutureCallback<None>();
        uriStore.start(callback);
        callback.get(30, TimeUnit.SECONDS);
        ZooKeeperServer server = new ZooKeeperServer(uriStore);
        callback = new FutureCallback<None>();
        Map<Integer, PartitionData> partitionDataMap = new HashMap<Integer, PartitionData>();
        partitionDataMap.put(DefaultPartitionAccessor.DEFAULT_PARTITION_ID, new PartitionData(1.0));
        server.markUp(TEST_CLUSTER_NAME, URI.create("http://test.uri"), partitionDataMap, callback);
        callback.get(30, TimeUnit.SECONDS);
        URIRequest request = new URIRequest("d2://" + TEST_SERVICE_NAME + "/foo");
        TransportClient client = balancer.getClient(request, new RequestContext());
        // Stop the server to cause a disconnect event
        stopServer();
        // Sleep to ensure the disconnect has propagated; ideally the Toggle should expose
        // some interface to allow detection that the toggle occurred
        Thread.sleep(1000);
        // Now see if it still works
        client = balancer.getClient(request, new RequestContext());
    } finally {
        stopServer();
    }
}
Also used : ServicePropertiesJsonSerializer(com.linkedin.d2.balancer.properties.ServicePropertiesJsonSerializer) HashMap(java.util.HashMap) ZooKeeperPermanentStore(com.linkedin.d2.discovery.stores.zk.ZooKeeperPermanentStore) UriPropertiesMerger(com.linkedin.d2.balancer.properties.UriPropertiesMerger) UriPropertiesJsonSerializer(com.linkedin.d2.balancer.properties.UriPropertiesJsonSerializer) ClusterPropertiesJsonSerializer(com.linkedin.d2.balancer.properties.ClusterPropertiesJsonSerializer) PartitionData(com.linkedin.d2.balancer.properties.PartitionData) UriProperties(com.linkedin.d2.balancer.properties.UriProperties) RequestContext(com.linkedin.r2.message.RequestContext) FutureCallback(com.linkedin.common.callback.FutureCallback) TransportClient(com.linkedin.r2.transport.common.bridge.client.TransportClient) ZKConnection(com.linkedin.d2.discovery.stores.zk.ZKConnection) URIRequest(com.linkedin.d2.balancer.util.URIRequest) ZooKeeperEphemeralStore(com.linkedin.d2.discovery.stores.zk.ZooKeeperEphemeralStore) ServiceProperties(com.linkedin.d2.balancer.properties.ServiceProperties) ClusterProperties(com.linkedin.d2.balancer.properties.ClusterProperties) None(com.linkedin.common.util.None) ZooKeeperServer(com.linkedin.d2.balancer.servers.ZooKeeperServer) Test(org.testng.annotations.Test)

Example 14 with ServicePropertiesJsonSerializer

use of com.linkedin.d2.balancer.properties.ServicePropertiesJsonSerializer in project rest.li by linkedin.

the class SimpleLoadBalancerTest method testLoadBalancerSimulationDegraderWithFileStore.

@Test(groups = { "medium", "back-end" })
public void testLoadBalancerSimulationDegraderWithFileStore() throws URISyntaxException, IOException, ServiceUnavailableException, InterruptedException {
    SimpleLoadBalancerSimulation simulator = new SimpleLoadBalancerSimulation(new DegraderLoadBalancerStrategyFactoryV3(), new FileStoreTestFactory<ClusterProperties>("cluster", new ClusterPropertiesJsonSerializer()), new FileStoreTestFactory<ServiceProperties>("service", new ServicePropertiesJsonSerializer()), new FileStoreTestFactory<UriProperties>("uri", new UriPropertiesJsonSerializer()));
    simulator.simulateMultithreaded(1, 1000, 20);
    simulator.reset();
    simulator.simulateMultithreaded(50, 10000, 20);
    simulator.reset();
}
Also used : UriPropertiesJsonSerializer(com.linkedin.d2.balancer.properties.UriPropertiesJsonSerializer) ServiceProperties(com.linkedin.d2.balancer.properties.ServiceProperties) DegraderLoadBalancerStrategyFactoryV3(com.linkedin.d2.balancer.strategies.degrader.DegraderLoadBalancerStrategyFactoryV3) ClusterPropertiesJsonSerializer(com.linkedin.d2.balancer.properties.ClusterPropertiesJsonSerializer) ServicePropertiesJsonSerializer(com.linkedin.d2.balancer.properties.ServicePropertiesJsonSerializer) UriProperties(com.linkedin.d2.balancer.properties.UriProperties) ClusterProperties(com.linkedin.d2.balancer.properties.ClusterProperties) SimpleLoadBalancerSimulation(com.linkedin.d2.balancer.simulator.SimpleLoadBalancerSimulation) Test(org.testng.annotations.Test)

Example 15 with ServicePropertiesJsonSerializer

use of com.linkedin.d2.balancer.properties.ServicePropertiesJsonSerializer in project rest.li by linkedin.

the class SimpleLoadBalancerTest method testLoadBalancerSimulationDegraderWithFileStoreLarge.

@Test(enabled = false, groups = { "large", "back-end" })
public void testLoadBalancerSimulationDegraderWithFileStoreLarge() throws URISyntaxException, IOException, ServiceUnavailableException, InterruptedException {
    SimpleLoadBalancerSimulation simulator = new SimpleLoadBalancerSimulation(new DegraderLoadBalancerStrategyFactoryV3(), new FileStoreTestFactory<ClusterProperties>("cluster", new ClusterPropertiesJsonSerializer()), new FileStoreTestFactory<ServiceProperties>("service", new ServicePropertiesJsonSerializer()), new FileStoreTestFactory<UriProperties>("uri", new UriPropertiesJsonSerializer()));
    simulator.simulateMultithreaded(1, 1000, 20);
    simulator.reset();
    simulator.simulateMultithreaded(1, 10000, 20);
    simulator.reset();
    simulator.simulateMultithreaded(8, 10000, 750);
    simulator.reset();
    simulator.simulateMultithreaded(50, 10000, 100);
    simulator.reset();
    simulator.simulateMultithreaded(50, 10000, 100);
    simulator.reset();
}
Also used : UriPropertiesJsonSerializer(com.linkedin.d2.balancer.properties.UriPropertiesJsonSerializer) ServiceProperties(com.linkedin.d2.balancer.properties.ServiceProperties) DegraderLoadBalancerStrategyFactoryV3(com.linkedin.d2.balancer.strategies.degrader.DegraderLoadBalancerStrategyFactoryV3) ClusterPropertiesJsonSerializer(com.linkedin.d2.balancer.properties.ClusterPropertiesJsonSerializer) ServicePropertiesJsonSerializer(com.linkedin.d2.balancer.properties.ServicePropertiesJsonSerializer) UriProperties(com.linkedin.d2.balancer.properties.UriProperties) ClusterProperties(com.linkedin.d2.balancer.properties.ClusterProperties) SimpleLoadBalancerSimulation(com.linkedin.d2.balancer.simulator.SimpleLoadBalancerSimulation) Test(org.testng.annotations.Test)

Aggregations

ServicePropertiesJsonSerializer (com.linkedin.d2.balancer.properties.ServicePropertiesJsonSerializer)15 ServiceProperties (com.linkedin.d2.balancer.properties.ServiceProperties)14 UriProperties (com.linkedin.d2.balancer.properties.UriProperties)11 UriPropertiesJsonSerializer (com.linkedin.d2.balancer.properties.UriPropertiesJsonSerializer)11 ClusterPropertiesJsonSerializer (com.linkedin.d2.balancer.properties.ClusterPropertiesJsonSerializer)10 ClusterProperties (com.linkedin.d2.balancer.properties.ClusterProperties)9 UriPropertiesMerger (com.linkedin.d2.balancer.properties.UriPropertiesMerger)9 ZooKeeperPermanentStore (com.linkedin.d2.discovery.stores.zk.ZooKeeperPermanentStore)8 HashMap (java.util.HashMap)7 ZooKeeperEphemeralStore (com.linkedin.d2.discovery.stores.zk.ZooKeeperEphemeralStore)6 None (com.linkedin.common.util.None)5 ZKConnection (com.linkedin.d2.discovery.stores.zk.ZKConnection)5 Test (org.testng.annotations.Test)5 FutureCallback (com.linkedin.common.callback.FutureCallback)4 DegraderLoadBalancerStrategyFactoryV3 (com.linkedin.d2.balancer.strategies.degrader.DegraderLoadBalancerStrategyFactoryV3)4 SimpleLoadBalancer (com.linkedin.d2.balancer.simple.SimpleLoadBalancer)3 SimpleLoadBalancerState (com.linkedin.d2.balancer.simple.SimpleLoadBalancerState)3 PropertyEventBusImpl (com.linkedin.d2.discovery.event.PropertyEventBusImpl)3 HashSet (java.util.HashSet)3 List (java.util.List)3