Search in sources :

Example 31 with SparseVector

use of com.alibaba.alink.common.linalg.SparseVector in project Alink by alibaba.

the class ToVectorMapperTest method testVec.

@Test
public void testVec() throws Exception {
    final Mapper mapper = new ToVectorMapper(new TableSchema(new String[] { "vec" }, new TypeInformation<?>[] { VectorTypes.VECTOR }), new Params().set(ToVectorParams.SELECTED_COL, "vec"));
    String vecStr = "0.0 0.1 1.0 1.1 2.0 2.1";
    DenseVector vector = VectorUtil.parseDense(vecStr);
    DenseVector result = (DenseVector) mapper.map(Row.of(vector)).getField(0);
    Assert.assertEquals(vector, result);
    String sVecStr = "0:0.0 1:0.1 5:1.0 10:1.1 12:2.0 15:2.1";
    SparseVector sVector = VectorUtil.parseSparse(sVecStr);
    SparseVector sResult = (SparseVector) mapper.map(Row.of(sVector)).getField(0);
    Assert.assertEquals(sVector, sResult);
}
Also used : Mapper(com.alibaba.alink.common.mapper.Mapper) TableSchema(org.apache.flink.table.api.TableSchema) ToVectorParams(com.alibaba.alink.params.dataproc.ToVectorParams) Params(org.apache.flink.ml.api.misc.param.Params) SparseVector(com.alibaba.alink.common.linalg.SparseVector) TypeInformation(org.apache.flink.api.common.typeinfo.TypeInformation) DenseVector(com.alibaba.alink.common.linalg.DenseVector) Test(org.junit.Test)

Example 32 with SparseVector

use of com.alibaba.alink.common.linalg.SparseVector in project Alink by alibaba.

the class ToVectorMapperTest method test.

@Test
public void test() throws Exception {
    final Mapper mapper = new ToVectorMapper(new TableSchema(new String[] { "vec" }, new TypeInformation<?>[] { Types.STRING }), new Params().set(ToVectorParams.SELECTED_COL, "vec"));
    String vecStr = "0.0 0.1 1.0 1.1 2.0 2.1";
    DenseVector vector = VectorUtil.parseDense(vecStr);
    DenseVector result = (DenseVector) mapper.map(Row.of(vecStr)).getField(0);
    Assert.assertEquals(vector, result);
    String sVecStr = "0:0.0 1:0.1 5:1.0 10:1.1 12:2.0 15:2.1";
    SparseVector sVector = VectorUtil.parseSparse(sVecStr);
    SparseVector sResult = (SparseVector) mapper.map(Row.of(sVecStr)).getField(0);
    Assert.assertEquals(sVector, sResult);
}
Also used : Mapper(com.alibaba.alink.common.mapper.Mapper) TableSchema(org.apache.flink.table.api.TableSchema) ToVectorParams(com.alibaba.alink.params.dataproc.ToVectorParams) Params(org.apache.flink.ml.api.misc.param.Params) SparseVector(com.alibaba.alink.common.linalg.SparseVector) TypeInformation(org.apache.flink.api.common.typeinfo.TypeInformation) DenseVector(com.alibaba.alink.common.linalg.DenseVector) Test(org.junit.Test)

Example 33 with SparseVector

use of com.alibaba.alink.common.linalg.SparseVector in project Alink by alibaba.

the class ToVectorMapperTest method testOp.

@Test
public void testOp() throws Exception {
    final String vecStr = "1 0 3 4";
    final DenseVector expect = VectorUtil.parseDense("1 0 3 4");
    final SparseVector expect1 = VectorUtil.parseSparse("$4$0:1 2:3 3:4");
    Row[] rows = new Row[] { Row.of(vecStr) };
    MemSourceBatchOp memSourceBatchOp = new MemSourceBatchOp(rows, new String[] { "vec" });
    memSourceBatchOp.link(new ToVectorBatchOp().setSelectedCol("vec")).lazyCollect(rows1 -> Assert.assertEquals(expect, rows1.get(0).getField(0)));
    memSourceBatchOp.link(new ToVectorBatchOp().setVectorType(VectorType.SPARSE).setSelectedCol("vec")).lazyCollect(rows1 -> Assert.assertEquals(expect1, rows1.get(0).getField(0)));
    BatchOperator.execute();
}
Also used : MemSourceBatchOp(com.alibaba.alink.operator.batch.source.MemSourceBatchOp) Row(org.apache.flink.types.Row) SparseVector(com.alibaba.alink.common.linalg.SparseVector) ToVectorBatchOp(com.alibaba.alink.operator.batch.dataproc.ToVectorBatchOp) DenseVector(com.alibaba.alink.common.linalg.DenseVector) Test(org.junit.Test)

Example 34 with SparseVector

use of com.alibaba.alink.common.linalg.SparseVector in project Alink by alibaba.

the class VectorSliceMapperTest method testSparse.

@Test
public void testSparse() throws Exception {
    TableSchema schema = new TableSchema(new String[] { "vec" }, new TypeInformation<?>[] { Types.STRING });
    Params params = new Params().set(VectorSliceParams.SELECTED_COL, "vec").set(VectorSliceParams.OUTPUT_COL, "res").set(VectorSliceParams.RESERVED_COLS, new String[] {}).set(VectorSliceParams.INDICES, new int[] { 0, 2, 4 });
    VectorSliceMapper mapper = new VectorSliceMapper(schema, params);
    assertEquals(mapper.map(Row.of(new SparseVector(5, new int[] { 0, 2, 4 }, new double[] { 3.0, 4.0, 3.0 }))).getField(0), new SparseVector(3, new int[] { 0, 1, 2 }, new double[] { 3.0, 4.0, 3.0 }));
    assertEquals(mapper.getOutputSchema(), new TableSchema(new String[] { "res" }, new TypeInformation<?>[] { VectorTypes.VECTOR }));
}
Also used : TableSchema(org.apache.flink.table.api.TableSchema) VectorSliceParams(com.alibaba.alink.params.dataproc.vector.VectorSliceParams) Params(org.apache.flink.ml.api.misc.param.Params) SparseVector(com.alibaba.alink.common.linalg.SparseVector) TypeInformation(org.apache.flink.api.common.typeinfo.TypeInformation) Test(org.junit.Test)

Example 35 with SparseVector

use of com.alibaba.alink.common.linalg.SparseVector in project Alink by alibaba.

the class FastDistanceTest method testSparseVectorRowInput.

@Test
public void testSparseVectorRowInput() {
    Vector vec = new SparseVector(10, new int[] { 1, 2 }, new double[] { 1.0, 1.0 });
    EuclideanDistance distance = new EuclideanDistance();
    FastDistanceVectorData vectorData = distance.prepareVectorData(Row.of(vec, 0, "a"), 0, 1, 2);
    assertVectorInput(vectorData, vec, Row.of(0, "a"));
}
Also used : SparseVector(com.alibaba.alink.common.linalg.SparseVector) Vector(com.alibaba.alink.common.linalg.Vector) DenseVector(com.alibaba.alink.common.linalg.DenseVector) SparseVector(com.alibaba.alink.common.linalg.SparseVector) Test(org.junit.Test)

Aggregations

SparseVector (com.alibaba.alink.common.linalg.SparseVector)125 Test (org.junit.Test)63 DenseVector (com.alibaba.alink.common.linalg.DenseVector)60 Params (org.apache.flink.ml.api.misc.param.Params)45 Row (org.apache.flink.types.Row)45 Vector (com.alibaba.alink.common.linalg.Vector)40 TableSchema (org.apache.flink.table.api.TableSchema)27 ArrayList (java.util.ArrayList)21 TypeInformation (org.apache.flink.api.common.typeinfo.TypeInformation)15 HashMap (java.util.HashMap)12 Tuple2 (org.apache.flink.api.java.tuple.Tuple2)12 List (java.util.List)11 DenseMatrix (com.alibaba.alink.common.linalg.DenseMatrix)10 MTable (com.alibaba.alink.common.MTable)7 BaseVectorSummary (com.alibaba.alink.operator.common.statistics.basicstatistic.BaseVectorSummary)6 CollectSinkStreamOp (com.alibaba.alink.operator.stream.sink.CollectSinkStreamOp)6 Map (java.util.Map)6 MemSourceBatchOp (com.alibaba.alink.operator.batch.source.MemSourceBatchOp)5 VectorAssemblerParams (com.alibaba.alink.params.dataproc.vector.VectorAssemblerParams)5 OneHotPredictParams (com.alibaba.alink.params.feature.OneHotPredictParams)5