Search in sources :

Example 6 with UnableToStartException

use of org.apache.metron.integration.UnableToStartException in project metron by apache.

the class FluxTopologyComponent method start.

@Override
public void start() throws UnableToStartException {
    try {
        stormCluster = new LocalCluster();
        RetryPolicy retryPolicy = new ExponentialBackoffRetry(1000, 3);
        try (CuratorFramework client = CuratorFrameworkFactory.newClient(getZookeeperConnectString(), retryPolicy)) {
            client.start();
            String root = "/storm/leader-lock";
            Stat exists = client.checkExists().forPath(root);
            if (exists == null) {
                client.create().creatingParentsIfNeeded().forPath(root);
            }
        } catch (Exception e) {
            LOG.error("Unable to create leaderlock", e);
        } finally {
        }
    } catch (Exception e) {
        throw new UnableToStartException("Unable to start flux topology: " + getTopologyLocation(), e);
    }
}
Also used : LocalCluster(org.apache.storm.LocalCluster) CuratorFramework(org.apache.curator.framework.CuratorFramework) Stat(org.apache.zookeeper.data.Stat) ExponentialBackoffRetry(org.apache.curator.retry.ExponentialBackoffRetry) UnableToStartException(org.apache.metron.integration.UnableToStartException) RetryPolicy(org.apache.curator.RetryPolicy) UnableToStartException(org.apache.metron.integration.UnableToStartException) TException(org.apache.storm.thrift.TException) InvocationTargetException(java.lang.reflect.InvocationTargetException) TProtocolException(org.apache.storm.thrift.protocol.TProtocolException)

Example 7 with UnableToStartException

use of org.apache.metron.integration.UnableToStartException in project metron by apache.

the class YarnComponent method start.

@Override
public void start() throws UnableToStartException {
    conf = new YarnConfiguration();
    conf.setInt(YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB, 128);
    conf.set("yarn.log.dir", "target");
    conf.setBoolean(YarnConfiguration.TIMELINE_SERVICE_ENABLED, true);
    conf.set(YarnConfiguration.RM_SCHEDULER, CapacityScheduler.class.getName());
    conf.setBoolean(YarnConfiguration.NODE_LABELS_ENABLED, true);
    try {
        yarnCluster = new MiniYARNCluster(testName, 1, NUM_NMS, 1, 1, true);
        yarnCluster.init(conf);
        yarnCluster.start();
        waitForNMsToRegister();
        URL url = Thread.currentThread().getContextClassLoader().getResource("yarn-site.xml");
        if (url == null) {
            throw new RuntimeException("Could not find 'yarn-site.xml' dummy file in classpath");
        }
        Configuration yarnClusterConfig = yarnCluster.getConfig();
        yarnClusterConfig.set("yarn.application.classpath", new File(url.getPath()).getParent());
        // write the document to a buffer (not directly to the file, as that
        // can cause the file being written to get read -which will then fail.
        ByteArrayOutputStream bytesOut = new ByteArrayOutputStream();
        yarnClusterConfig.writeXml(bytesOut);
        bytesOut.close();
        // write the bytes to the file in the classpath
        OutputStream os = new FileOutputStream(new File(url.getPath()));
        os.write(bytesOut.toByteArray());
        os.close();
        FileContext fsContext = FileContext.getLocalFSFileContext();
        fsContext.delete(new Path(conf.get("yarn.timeline-service.leveldb-timeline-store.path")), true);
        try {
            Thread.sleep(2000);
        } catch (InterruptedException e) {
        }
    } catch (Exception e) {
        throw new UnableToStartException("Exception setting up yarn cluster", e);
    }
}
Also used : Path(org.apache.hadoop.fs.Path) YarnConfiguration(org.apache.hadoop.yarn.conf.YarnConfiguration) Configuration(org.apache.hadoop.conf.Configuration) UnableToStartException(org.apache.metron.integration.UnableToStartException) URL(java.net.URL) UnableToStartException(org.apache.metron.integration.UnableToStartException) YarnConfiguration(org.apache.hadoop.yarn.conf.YarnConfiguration) MiniYARNCluster(org.apache.hadoop.yarn.server.MiniYARNCluster) CapacityScheduler(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler) FileContext(org.apache.hadoop.fs.FileContext)

Aggregations

UnableToStartException (org.apache.metron.integration.UnableToStartException)7 IOException (java.io.IOException)3 File (java.io.File)2 LocalCluster (org.apache.storm.LocalCluster)2 InvocationTargetException (java.lang.reflect.InvocationTargetException)1 URL (java.net.URL)1 HashMap (java.util.HashMap)1 RetryPolicy (org.apache.curator.RetryPolicy)1 CuratorFramework (org.apache.curator.framework.CuratorFramework)1 ExponentialBackoffRetry (org.apache.curator.retry.ExponentialBackoffRetry)1 TestingServer (org.apache.curator.test.TestingServer)1 Configuration (org.apache.hadoop.conf.Configuration)1 FileContext (org.apache.hadoop.fs.FileContext)1 Path (org.apache.hadoop.fs.Path)1 YarnConfiguration (org.apache.hadoop.yarn.conf.YarnConfiguration)1 MiniYARNCluster (org.apache.hadoop.yarn.server.MiniYARNCluster)1 CapacityScheduler (org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler)1 ElasticsearchDao (org.apache.metron.elasticsearch.dao.ElasticsearchDao)1 ComponentRunner (org.apache.metron.integration.ComponentRunner)1 ParserTopologyBuilder (org.apache.metron.parsers.topology.ParserTopologyBuilder)1