Search in sources :

Example 21 with MatrixContext

use of com.tencent.angel.ml.matrix.MatrixContext in project angel by Tencent.

the class PeriodHATest method setup.

@Before
public void setup() throws Exception {
    try {
        // set basic configuration keys
        Configuration conf = new Configuration();
        conf.setBoolean("mapred.mapper.new-api", true);
        conf.setBoolean(AngelConf.ANGEL_JOB_OUTPUT_PATH_DELETEONEXIST, true);
        conf.set(AngelConf.ANGEL_TASK_USER_TASKCLASS, DummyTask.class.getName());
        // use local deploy mode and dummy dataspliter
        conf.set(AngelConf.ANGEL_DEPLOY_MODE, "LOCAL");
        conf.setBoolean(AngelConf.ANGEL_AM_USE_DUMMY_DATASPLITER, true);
        conf.set(AngelConf.ANGEL_INPUTFORMAT_CLASS, CombineTextInputFormat.class.getName());
        conf.set(AngelConf.ANGEL_SAVE_MODEL_PATH, LOCAL_FS + TMP_PATH + "/out");
        conf.set(AngelConf.ANGEL_TRAIN_DATA_PATH, LOCAL_FS + TMP_PATH + "/in");
        conf.set(AngelConf.ANGEL_LOG_PATH, LOCAL_FS + TMP_PATH + "/log");
        conf.setInt(AngelConf.ANGEL_WORKERGROUP_NUMBER, 1);
        conf.setInt(AngelConf.ANGEL_PS_NUMBER, 2);
        conf.setInt(AngelConf.ANGEL_WORKER_TASK_NUMBER, 1);
        conf.setInt(AngelConf.ANGEL_PS_HA_REPLICATION_NUMBER, 2);
        conf.setInt(AngelConf.ANGEL_PS_HA_PUSH_INTERVAL_MS, 1000);
        // conf.setBoolean(AngelConf.ANGEL_PS_HA_USE_EVENT_PUSH, true);
        // conf.setBoolean(AngelConf.ANGEL_PS_HA_PUSH_SYNC, true);
        // get a angel client
        angelClient = AngelClientFactory.get(conf);
        // add matrix
        MatrixContext mMatrix = new MatrixContext();
        mMatrix.setName("w1");
        mMatrix.setRowNum(1);
        mMatrix.setColNum(dim);
        mMatrix.setMaxRowNumInBlock(1);
        mMatrix.setMaxColNumInBlock(dim / 2);
        mMatrix.setRowType(RowType.T_INT_DENSE);
        mMatrix.set(MatrixConf.MATRIX_HOGWILD, "true");
        mMatrix.set(MatrixConf.MATRIX_AVERAGE, "false");
        angelClient.addMatrix(mMatrix);
        angelClient.startPSServer();
        angelClient.run();
        Thread.sleep(5000);
        task0Id = new TaskId(0);
    } catch (Exception x) {
        LOG.error("setup failed ", x);
        throw x;
    }
}
Also used : CombineTextInputFormat(org.apache.hadoop.mapreduce.lib.input.CombineTextInputFormat) MatrixContext(com.tencent.angel.ml.matrix.MatrixContext) TaskId(com.tencent.angel.worker.task.TaskId) Configuration(org.apache.hadoop.conf.Configuration) Before(org.junit.Before)

Example 22 with MatrixContext

use of com.tencent.angel.ml.matrix.MatrixContext in project angel by Tencent.

the class AngelPSClient method startPS.

/**
 * Start Angel ps
 * @return Angel ps running context
 * @throws AngelException
 */
public AngelContext startPS() throws AngelException {
    int psNum = conf.getInt(AngelConf.ANGEL_PS_NUMBER, AngelConf.DEFAULT_ANGEL_PS_NUMBER);
    if (psNum <= 0) {
        throw new AngelException("Invalid parameter:Wrong ps number!");
    }
    conf.set(AngelConf.ANGEL_RUNNING_MODE, RunningMode.ANGEL_PS.toString());
    client.addMatrix(new MatrixContext("init", 1, psNum, 1, 1));
    client.startPSServer();
    client.run();
    return new AngelContext(client.getMasterLocation(), conf);
}
Also used : AngelException(com.tencent.angel.exception.AngelException) MatrixContext(com.tencent.angel.ml.matrix.MatrixContext)

Aggregations

MatrixContext (com.tencent.angel.ml.matrix.MatrixContext)22 Configuration (org.apache.hadoop.conf.Configuration)16 CombineTextInputFormat (org.apache.hadoop.mapreduce.lib.input.CombineTextInputFormat)15 TaskId (com.tencent.angel.worker.task.TaskId)13 PSAttemptId (com.tencent.angel.ps.PSAttemptId)10 ParameterServerId (com.tencent.angel.ps.ParameterServerId)10 Before (org.junit.Before)10 WorkerAttemptId (com.tencent.angel.worker.WorkerAttemptId)9 WorkerGroupId (com.tencent.angel.worker.WorkerGroupId)9 WorkerId (com.tencent.angel.worker.WorkerId)9 DummyTask (com.tencent.angel.master.DummyTask)5 AngelClient (com.tencent.angel.client.AngelClient)4 AngelException (com.tencent.angel.exception.AngelException)4 IOException (java.io.IOException)3 BeforeClass (org.junit.BeforeClass)3 ServiceException (com.google.protobuf.ServiceException)2 PartitionKey (com.tencent.angel.PartitionKey)1 PSModel (com.tencent.angel.ml.model.PSModel)1 DummyTask (com.tencent.angel.psagent.DummyTask)1 ByteBuf (io.netty.buffer.ByteBuf)1