Search in sources :

Example 16 with MatrixFilesMeta

use of com.tencent.angel.model.output.format.MatrixFilesMeta in project angel by Tencent.

the class ModelLoader method loadToFloatArrays.

/**
 * Load dense float model to a 2-dimension float array
 *
 * @param modelDir model save directory path
 * @return model data
 */
public static float[][] loadToFloatArrays(String modelDir, Configuration conf) throws IOException {
    // Load model meta
    MatrixFilesMeta meta = getMeta(modelDir, conf);
    RowType rowType = RowType.valueOf(meta.getRowType());
    // Check row type
    if (rowType != RowType.T_FLOAT_DENSE && rowType != RowType.T_FLOAT_DENSE_COMPONENT) {
        throw new IOException("model row type is not dense float, you should check it");
    }
    // Load model
    DenseFloatModel model = new DenseFloatModel(meta.getRow(), meta.getCol());
    loadModel(modelDir, model, meta, conf);
    return model.getModel();
}
Also used : MatrixFilesMeta(com.tencent.angel.model.output.format.MatrixFilesMeta) RowType(com.tencent.angel.ml.matrix.RowType) IOException(java.io.IOException)

Aggregations

MatrixFilesMeta (com.tencent.angel.model.output.format.MatrixFilesMeta)16 IOException (java.io.IOException)13 RowType (com.tencent.angel.ml.matrix.RowType)7 Path (org.apache.hadoop.fs.Path)5 FSDataInputStream (org.apache.hadoop.fs.FSDataInputStream)4 FileSystem (org.apache.hadoop.fs.FileSystem)4 MatrixMeta (com.tencent.angel.ml.matrix.MatrixMeta)2 ParameterServerId (com.tencent.angel.ps.ParameterServerId)2 ConcurrentHashMap (java.util.concurrent.ConcurrentHashMap)2 AMMatrixMetaManager (com.tencent.angel.master.matrixmeta.AMMatrixMetaManager)1 PartitionMeta (com.tencent.angel.ml.matrix.PartitionMeta)1 MatrixPartitionMeta (com.tencent.angel.model.output.format.MatrixPartitionMeta)1 PSMatrixFilesMeta (com.tencent.angel.model.output.format.PSMatrixFilesMeta)1 ArrayList (java.util.ArrayList)1 HashMap (java.util.HashMap)1 Map (java.util.Map)1 FSDataOutputStream (org.apache.hadoop.fs.FSDataOutputStream)1