Search in sources :

Example 1 with PSLocationManager

use of com.tencent.angel.ps.client.PSLocationManager in project angel by Tencent.

the class ParameterServer method initialize.

/**
 * Initialize.
 */
public void initialize() throws IOException, InstantiationException, IllegalAccessException {
    LOG.info("Initialize a parameter server");
    ServerRow.maxLockWaitTimeMs = conf.getInt(AngelConf.ANGEL_PS_MAX_LOCK_WAITTIME_MS, AngelConf.DEFAULT_ANGEL_PS_MAX_LOCK_WAITTIME_MS);
    ServerRow.useAdaptiveKey = conf.getBoolean(AngelConf.ANGEL_PS_USE_ADAPTIVE_KEY_ENABLE, AngelConf.DEFAULT_ANGEL_PS_USE_ADAPTIVE_KEY_ENABLE);
    ServerRow.useAdaptiveStorage = conf.getBoolean(AngelConf.ANGEL_PS_USE_ADAPTIVE_STORAGE_ENABLE, AngelConf.DEFAULT_ANGEL_PS_USE_ADAPTIVE_STORAGE_ENABLE);
    ServerRow.sparseToDenseFactor = conf.getFloat(AngelConf.ANGEL_PS_SPARSE_TO_DENSE_FACTOR, AngelConf.DEFAULT_ANGEL_PS_SPARSE_TO_DENSE_FACTOR);
    dataCollectionInterval = conf.getInt(AngelConf.ANGEL_PS_HEARTBEAT_DATA_COLLECTION_INTERVAL, AngelConf.DEFAULT_ANGEL_PS_HEARTBEAT_DATA_COLLECTION_INTERVAL);
    locationManager = new PSLocationManager(context);
    locationManager.setMasterLocation(masterLocation);
    runningContext = new RunningContext(context);
    workerPool = new WorkerPool(context, runningContext);
    workerPool.init();
    ioExecutor = new PSModelIOExecutor(context);
    ioExecutor.init();
    matrixStorageManager = new MatrixStorageManager(context);
    matrixMetaManager = new PSMatrixMetaManager(context);
    master = new MasterClient(context);
    master.init();
    psServerService = new ParameterServerService(context);
    psServerService.start();
    matrixTransportServer = new MatrixTransportServer(getPort() + 1, context);
    saver = new PSModelSaver(context);
    loader = new PSModelLoader(context);
    snapshotDumper = new SnapshotDumper(context);
}
Also used : PSMatrixMetaManager(com.tencent.angel.ps.meta.PSMatrixMetaManager) PSModelLoader(com.tencent.angel.ps.io.load.PSModelLoader) WorkerPool(com.tencent.angel.ps.server.data.WorkerPool) SnapshotDumper(com.tencent.angel.ps.io.save.SnapshotDumper) RunningContext(com.tencent.angel.ps.server.data.RunningContext) PSModelIOExecutor(com.tencent.angel.ps.io.PSModelIOExecutor) MasterClient(com.tencent.angel.ps.client.MasterClient) MatrixStorageManager(com.tencent.angel.ps.storage.MatrixStorageManager) MatrixTransportServer(com.tencent.angel.ps.server.data.MatrixTransportServer) PSLocationManager(com.tencent.angel.ps.client.PSLocationManager) ParameterServerService(com.tencent.angel.ps.server.control.ParameterServerService) PSModelSaver(com.tencent.angel.ps.io.save.PSModelSaver)

Example 2 with PSLocationManager

use of com.tencent.angel.ps.client.PSLocationManager in project angel by Tencent.

the class ParameterServer method initialize.

/**
 * Initialize.
 *
 * @throws IOException
 * @throws InstantiationException
 * @throws IllegalAccessException
 */
public void initialize() throws IOException, InstantiationException, IllegalAccessException {
    LOG.info("Initialize a parameter server");
    locationManager = new PSLocationManager(context);
    locationManager.setMasterLocation(masterLocation);
    workerPool = new WorkerPool(context);
    workerPool.init();
    ioExecutors = new IOExecutors(context);
    ioExecutors.init();
    matrixStorageManager = new MatrixStorageManager(context);
    int taskNum = conf.getInt(AngelConf.ANGEL_TASK_ACTUAL_NUM, 1);
    clockVectorManager = new ClockVectorManager(taskNum, context);
    clockVectorManager.init();
    matrixMetaManager = new PSMatrixMetaManager(context);
    master = new MasterClient(context);
    master.init();
    psServerService = new ParameterServerService(context);
    psServerService.start();
    matrixTransportServer = new MatrixTransportServer(getPort() + 1, context);
    int replicNum = conf.getInt(AngelConf.ANGEL_PS_HA_REPLICATION_NUMBER, AngelConf.DEFAULT_ANGEL_PS_HA_REPLICATION_NUMBER);
    if (replicNum > 1) {
        // conf.getBoolean(AngelConf.ANGEL_PS_HA_USE_EVENT_PUSH, AngelConf.DEFAULT_ANGEL_PS_HA_USE_EVENT_PUSH);
        boolean useEventPush = false;
        if (useEventPush) {
            boolean sync = conf.getBoolean(AngelConf.ANGEL_PS_HA_PUSH_SYNC, AngelConf.DEFAULT_ANGEL_PS_HA_PUSH_SYNC);
            if (sync) {
                ps2PSPusher = new SyncEventPusher(context);
            } else {
                ps2PSPusher = new AsyncEventPusher(context);
            }
        } else {
            ps2PSPusher = new PeriodPusher(context);
        }
        ps2PSPusher.init();
    } else {
        snapshotDumper = new SnapshotDumper(context);
    }
}
Also used : SnapshotDumper(com.tencent.angel.ps.backup.snapshot.SnapshotDumper) MasterClient(com.tencent.angel.ps.client.MasterClient) PSLocationManager(com.tencent.angel.ps.client.PSLocationManager) AsyncEventPusher(com.tencent.angel.ps.backup.ha.push.AsyncEventPusher) WorkerPool(com.tencent.angel.ps.matrix.transport.WorkerPool) SyncEventPusher(com.tencent.angel.ps.backup.ha.push.SyncEventPusher) IOExecutors(com.tencent.angel.ps.io.IOExecutors) PeriodPusher(com.tencent.angel.ps.backup.ha.push.PeriodPusher) MatrixTransportServer(com.tencent.angel.ps.matrix.transport.MatrixTransportServer)

Aggregations

MasterClient (com.tencent.angel.ps.client.MasterClient)2 PSLocationManager (com.tencent.angel.ps.client.PSLocationManager)2 AsyncEventPusher (com.tencent.angel.ps.backup.ha.push.AsyncEventPusher)1 PeriodPusher (com.tencent.angel.ps.backup.ha.push.PeriodPusher)1 SyncEventPusher (com.tencent.angel.ps.backup.ha.push.SyncEventPusher)1 SnapshotDumper (com.tencent.angel.ps.backup.snapshot.SnapshotDumper)1 IOExecutors (com.tencent.angel.ps.io.IOExecutors)1 PSModelIOExecutor (com.tencent.angel.ps.io.PSModelIOExecutor)1 PSModelLoader (com.tencent.angel.ps.io.load.PSModelLoader)1 PSModelSaver (com.tencent.angel.ps.io.save.PSModelSaver)1 SnapshotDumper (com.tencent.angel.ps.io.save.SnapshotDumper)1 MatrixTransportServer (com.tencent.angel.ps.matrix.transport.MatrixTransportServer)1 WorkerPool (com.tencent.angel.ps.matrix.transport.WorkerPool)1 PSMatrixMetaManager (com.tencent.angel.ps.meta.PSMatrixMetaManager)1 ParameterServerService (com.tencent.angel.ps.server.control.ParameterServerService)1 MatrixTransportServer (com.tencent.angel.ps.server.data.MatrixTransportServer)1 RunningContext (com.tencent.angel.ps.server.data.RunningContext)1 WorkerPool (com.tencent.angel.ps.server.data.WorkerPool)1 MatrixStorageManager (com.tencent.angel.ps.storage.MatrixStorageManager)1