Search in sources :

Example 11 with FutureResult

use of com.tencent.angel.psagent.matrix.transport.FutureResult in project angel by Tencent.

the class UserRequestAdapter method get.

private FutureResult<Vector[]> get(IndexGetRowsRequest request) {
    // Only support column-partitioned matrix now!!
    MatrixMeta matrixMeta = PSAgentContext.get().getMatrixMetaManager().getMatrixMeta(request.getMatrixId());
    PartitionKey[] matrixParts = matrixMeta.getPartitionKeys();
    for (PartitionKey part : matrixParts) {
        if (part.getStartRow() != 0 || part.getEndRow() != matrixMeta.getRowNum()) {
            throw new UnsupportedOperationException("Get rows by indices only support column-partitioned matrix now");
        }
    }
    // Split the user request to partition requests
    FutureResult<Vector[]> result = new FutureResult<>();
    long startTs = System.currentTimeMillis();
    KeyPart[] splits;
    if (request instanceof IntIndexGetRowsRequest) {
        splits = RouterUtils.split(matrixMeta, -1, ((IntIndexGetRowsRequest) request).getIndices());
    } else if (request instanceof LongIndexGetRowsRequest) {
        splits = RouterUtils.split(matrixMeta, -1, ((LongIndexGetRowsRequest) request).getIndices());
    } else {
        throw new UnsupportedOperationException("Unsupport index request type " + request.getClass().toString());
    }
    LOG.info("Get by indices split use time: " + (System.currentTimeMillis() - startTs));
    assert matrixParts.length == splits.length;
    // filter empty partition requests
    int needRequestPartNum = noEmptyPartNum(splits);
    if (needRequestPartNum == 0) {
        result.set(null);
        return result;
    }
    // Create partition results cache
    ResponseCache cache = new MapResponseCache(needRequestPartNum);
    int requestId = request.getRequestId();
    requestIdToResponseCache.put(requestId, cache);
    requestIdToResultMap.put(requestId, result);
    requests.put(requestId, request);
    MatrixTransportClient matrixClient = PSAgentContext.get().getMatrixTransportClient();
    for (int i = 0; i < splits.length; i++) {
        if (splits[i] != null && splits[i].size() > 0) {
            sendIndexGetRowsRequest(matrixClient, requestId, request.getMatrixId(), request.getRowIds(), matrixParts[i].getPartitionId(), splits[i], request.getFunc());
        }
    }
    return result;
}
Also used : MatrixMeta(com.tencent.angel.ml.matrix.MatrixMeta) KeyPart(com.tencent.angel.psagent.matrix.transport.router.KeyPart) MapResponseCache(com.tencent.angel.psagent.matrix.transport.response.MapResponseCache) MatrixTransportClient(com.tencent.angel.psagent.matrix.transport.MatrixTransportClient) FutureResult(com.tencent.angel.psagent.matrix.transport.FutureResult) PartitionKey(com.tencent.angel.PartitionKey) MapResponseCache(com.tencent.angel.psagent.matrix.transport.response.MapResponseCache) ResponseCache(com.tencent.angel.psagent.matrix.transport.response.ResponseCache)

Example 12 with FutureResult

use of com.tencent.angel.psagent.matrix.transport.FutureResult in project angel by Tencent.

the class UserRequestAdapter method clear.

public void clear(String errorLog) {
    for (Entry<Integer, FutureResult> resultEntry : requestIdToResultMap.entrySet()) {
        resultEntry.getValue().setExecuteException(new ExecutionException(new AngelException(errorLog)));
    }
    requestIdToResultMap.clear();
    requestIdToResponseCache.clear();
    requests.clear();
}
Also used : AngelException(com.tencent.angel.exception.AngelException) FutureResult(com.tencent.angel.psagent.matrix.transport.FutureResult) ExecutionException(java.util.concurrent.ExecutionException)

Example 13 with FutureResult

use of com.tencent.angel.psagent.matrix.transport.FutureResult in project angel by Tencent.

the class UserRequestAdapter method update.

public Future<VoidResult> update(int matrixId, int rowId, Vector delta, UpdateOp op) {
    checkParams(matrixId, rowId);
    delta.setMatrixId(matrixId);
    delta.setRowId(rowId);
    MatrixMeta matrixMeta = PSAgentContext.get().getMatrixMetaManager().getMatrixMeta(matrixId);
    PartitionKey[] parts = matrixMeta.getPartitionKeys();
    CompStreamKeyValuePart[] splits = RouterUtils.splitStream(matrixMeta, delta);
    FutureResult<VoidResult> result = new FutureResult<>();
    int needRequestPartNum = noEmptyPartNum(splits);
    if (needRequestPartNum == 0) {
        result.set(null);
        return result;
    }
    UpdateRowRequest request = new UpdateRowRequest(matrixId, rowId, op);
    ResponseCache cache = new MapResponseCache(needRequestPartNum);
    int requestId = request.getRequestId();
    requestIdToResponseCache.put(requestId, cache);
    requestIdToResultMap.put(requestId, result);
    requests.put(requestId, request);
    MatrixTransportClient matrixClient = PSAgentContext.get().getMatrixTransportClient();
    for (int i = 0; i < splits.length; i++) {
        if (splits[i] != null && splits[i].size() > 0) {
            sendUpdateRequest(matrixClient, requestId, matrixId, parts[i].getPartitionId(), splits[i], op);
        }
    }
    return result;
}
Also used : VoidResult(com.tencent.angel.ml.matrix.psf.update.base.VoidResult) MatrixMeta(com.tencent.angel.ml.matrix.MatrixMeta) MapResponseCache(com.tencent.angel.psagent.matrix.transport.response.MapResponseCache) MatrixTransportClient(com.tencent.angel.psagent.matrix.transport.MatrixTransportClient) FutureResult(com.tencent.angel.psagent.matrix.transport.FutureResult) PartitionKey(com.tencent.angel.PartitionKey) MapResponseCache(com.tencent.angel.psagent.matrix.transport.response.MapResponseCache) ResponseCache(com.tencent.angel.psagent.matrix.transport.response.ResponseCache) CompStreamKeyValuePart(com.tencent.angel.psagent.matrix.transport.router.CompStreamKeyValuePart)

Example 14 with FutureResult

use of com.tencent.angel.psagent.matrix.transport.FutureResult in project angel by Tencent.

the class UserRequestAdapter method update.

/**
 * Update matrix use a udf.
 *
 * @param updateFunc update udf function
 * @return Future<VoidResult> update future result
 */
public Future<VoidResult> update(UpdateFunc updateFunc) {
    MatrixTransportClient matrixClient = PSAgentContext.get().getMatrixTransportClient();
    UpdateParam param = updateFunc.getParam();
    // Split the param use matrix partitions
    List<PartitionUpdateParam> partParams = param.split();
    int size = partParams.size();
    UpdatePSFRequest request = new UpdatePSFRequest(updateFunc);
    ResponseCache cache = new MapResponseCache(size);
    FutureResult<VoidResult> result = new FutureResult<>();
    int requestId = request.getRequestId();
    requests.put(requestId, request);
    requestIdToResponseCache.put(requestId, cache);
    requestIdToResultMap.put(requestId, result);
    // Send request to PSS
    for (int i = 0; i < size; i++) {
        sendUpdateUDFRequest(matrixClient, requestId, partParams.get(i).getMatrixId(), partParams.get(i).getPartKey().getPartitionId(), updateFunc, partParams.get(i));
    }
    return result;
}
Also used : MatrixTransportClient(com.tencent.angel.psagent.matrix.transport.MatrixTransportClient) VoidResult(com.tencent.angel.ml.matrix.psf.update.base.VoidResult) FutureResult(com.tencent.angel.psagent.matrix.transport.FutureResult) PartitionUpdateParam(com.tencent.angel.ml.matrix.psf.update.base.PartitionUpdateParam) PartitionUpdateParam(com.tencent.angel.ml.matrix.psf.update.base.PartitionUpdateParam) UpdateParam(com.tencent.angel.ml.matrix.psf.update.base.UpdateParam) MapResponseCache(com.tencent.angel.psagent.matrix.transport.response.MapResponseCache) ResponseCache(com.tencent.angel.psagent.matrix.transport.response.ResponseCache) MapResponseCache(com.tencent.angel.psagent.matrix.transport.response.MapResponseCache)

Example 15 with FutureResult

use of com.tencent.angel.psagent.matrix.transport.FutureResult in project angel by Tencent.

the class UserRequestAdapter method update.

public Future<VoidResult> update(int matrixId, int[] rowIds, Vector[] rows, UpdateOp op) {
    assert rowIds.length == rows.length;
    checkParams(matrixId, rowIds);
    for (int i = 0; i < rowIds.length; i++) {
        rows[i].setRowId(rowIds[i]);
    }
    MatrixMeta matrixMeta = PSAgentContext.get().getMatrixMetaManager().getMatrixMeta(matrixId);
    PartitionKey[] parts = matrixMeta.getPartitionKeys();
    CompStreamKeyValuePart[] splits = RouterUtils.splitStream(matrixMeta, rows);
    int needRequestPartNum = noEmptyPartNum(splits);
    FutureResult<VoidResult> result = new FutureResult<>();
    if (needRequestPartNum == 0) {
        result.set(null);
        return result;
    }
    UpdateRowsRequest request = new UpdateRowsRequest(matrixId, op);
    ResponseCache cache = new MapResponseCache(needRequestPartNum);
    int requestId = request.getRequestId();
    requestIdToResponseCache.put(requestId, cache);
    requestIdToResultMap.put(requestId, result);
    requests.put(requestId, request);
    MatrixTransportClient matrixClient = PSAgentContext.get().getMatrixTransportClient();
    for (int i = 0; i < splits.length; i++) {
        if (splits[i] != null && splits[i].size() > 0) {
            sendUpdateRequest(matrixClient, requestId, matrixId, parts[i].getPartitionId(), splits[i], op);
        }
    }
    return result;
}
Also used : VoidResult(com.tencent.angel.ml.matrix.psf.update.base.VoidResult) MatrixMeta(com.tencent.angel.ml.matrix.MatrixMeta) MapResponseCache(com.tencent.angel.psagent.matrix.transport.response.MapResponseCache) MatrixTransportClient(com.tencent.angel.psagent.matrix.transport.MatrixTransportClient) FutureResult(com.tencent.angel.psagent.matrix.transport.FutureResult) PartitionKey(com.tencent.angel.PartitionKey) MapResponseCache(com.tencent.angel.psagent.matrix.transport.response.MapResponseCache) ResponseCache(com.tencent.angel.psagent.matrix.transport.response.ResponseCache) CompStreamKeyValuePart(com.tencent.angel.psagent.matrix.transport.router.CompStreamKeyValuePart)

Aggregations

FutureResult (com.tencent.angel.psagent.matrix.transport.FutureResult)29 PartitionKey (com.tencent.angel.PartitionKey)10 AngelException (com.tencent.angel.exception.AngelException)10 MatrixTransportClient (com.tencent.angel.psagent.matrix.transport.MatrixTransportClient)9 MapResponseCache (com.tencent.angel.psagent.matrix.transport.response.MapResponseCache)9 ResponseCache (com.tencent.angel.psagent.matrix.transport.response.ResponseCache)9 VoidResult (com.tencent.angel.ml.matrix.psf.update.base.VoidResult)8 Vector (com.tencent.angel.ml.math2.vector.Vector)6 MatrixMeta (com.tencent.angel.ml.matrix.MatrixMeta)5 Response (com.tencent.angel.ml.matrix.transport.Response)4 CompStreamKeyValuePart (com.tencent.angel.psagent.matrix.transport.router.CompStreamKeyValuePart)3 ServerPartition (com.tencent.angel.ps.impl.matrix.ServerPartition)2 KeyPart (com.tencent.angel.psagent.matrix.transport.router.KeyPart)2 ByteBuf (io.netty.buffer.ByteBuf)2 ArrayList (java.util.ArrayList)2 ExecutionException (java.util.concurrent.ExecutionException)2 GetParam (com.tencent.angel.ml.matrix.psf.get.base.GetParam)1 GetResult (com.tencent.angel.ml.matrix.psf.get.base.GetResult)1 PartitionGetParam (com.tencent.angel.ml.matrix.psf.get.base.PartitionGetParam)1 PartitionUpdateParam (com.tencent.angel.ml.matrix.psf.update.base.PartitionUpdateParam)1