Search in sources :

Example 1 with HamaConfiguration

use of org.apache.hama.HamaConfiguration in project whirr by apache.

the class HamaServiceController method startup.

public synchronized void startup() throws Exception {
    LOG.info("Starting up cluster...");
    CompositeConfiguration config = new CompositeConfiguration();
    if (System.getProperty("config") != null) {
        config.addConfiguration(new PropertiesConfiguration(System.getProperty("config")));
    }
    config.addConfiguration(new PropertiesConfiguration(HamaServiceController.class.getResource("/whirr-hama-test.properties")));
    clusterSpec = ClusterSpec.withTemporaryKeys(config);
    controller = new ClusterController();
    cluster = controller.launchCluster(clusterSpec);
    proxy = new HadoopProxy(clusterSpec, cluster);
    proxy.start();
    HamaConfiguration conf = getConfiguration();
    BSPJobClient client = new BSPJobClient(conf);
    waitForGroomServers(client);
    running = true;
}
Also used : ClusterController(org.apache.whirr.ClusterController) BSPJobClient(org.apache.hama.bsp.BSPJobClient) CompositeConfiguration(org.apache.commons.configuration.CompositeConfiguration) HadoopProxy(org.apache.whirr.service.hadoop.HadoopProxy) HamaConfiguration(org.apache.hama.HamaConfiguration) PropertiesConfiguration(org.apache.commons.configuration.PropertiesConfiguration)

Example 2 with HamaConfiguration

use of org.apache.hama.HamaConfiguration in project whirr by apache.

the class HamaServiceTest method test.

@Test(timeout = TestConstants.ITEST_TIMEOUT)
public void test() throws Exception {
    HamaConfiguration jobConf = controller.getConfiguration();
    jobConf.set("hadoop.rpc.socket.factory.class.default", "org.apache.hadoop.net.StandardSocketFactory");
    BSPJob bsp = new BSPJob(jobConf, new BSPJobID());
    LOG.info("Job conf: " + bsp.getConf().get("hadoop.rpc.socket.factory.class.default") + ", " + bsp.getJobID().toString());
    bsp.setJarByClass(MyEstimator.class);
    bsp.setBspClass(MyEstimator.class);
    bsp.setInputFormat(NullInputFormat.class);
    bsp.setOutputKeyClass(Text.class);
    bsp.setOutputValueClass(DoubleWritable.class);
    bsp.setOutputFormat(TextOutputFormat.class);
    bsp.set("bsp.working.dir", "/tmp");
    FileOutputFormat.setOutputPath(bsp, TMP_OUTPUT);
    LOG.info("Client configuration start ..");
    HamaConfiguration clientConf = controller.getConfiguration();
    BSPJobClient jobClient = new BSPJobClient(clientConf);
    ClusterStatus cluster = jobClient.getClusterStatus(true);
    assertNotNull(cluster);
    assertTrue(cluster.getGroomServers() > 0);
    assertTrue(cluster.getMaxTasks() > 1);
    bsp.setNumBspTask(cluster.getMaxTasks());
    LOG.info("Client conf: " + clientConf.get("hadoop.rpc.socket.factory.class.default"));
    RunningJob rJob = jobClient.submitJob(bsp);
    rJob.waitForCompletion();
    LOG.info("finished");
}
Also used : BSPJobClient(org.apache.hama.bsp.BSPJobClient) BSPJobID(org.apache.hama.bsp.BSPJobID) BSPJob(org.apache.hama.bsp.BSPJob) RunningJob(org.apache.hama.bsp.RunningJob) HamaConfiguration(org.apache.hama.HamaConfiguration) ClusterStatus(org.apache.hama.bsp.ClusterStatus) Test(org.junit.Test)

Aggregations

HamaConfiguration (org.apache.hama.HamaConfiguration)2 BSPJobClient (org.apache.hama.bsp.BSPJobClient)2 CompositeConfiguration (org.apache.commons.configuration.CompositeConfiguration)1 PropertiesConfiguration (org.apache.commons.configuration.PropertiesConfiguration)1 BSPJob (org.apache.hama.bsp.BSPJob)1 BSPJobID (org.apache.hama.bsp.BSPJobID)1 ClusterStatus (org.apache.hama.bsp.ClusterStatus)1 RunningJob (org.apache.hama.bsp.RunningJob)1 ClusterController (org.apache.whirr.ClusterController)1 HadoopProxy (org.apache.whirr.service.hadoop.HadoopProxy)1 Test (org.junit.Test)1