Examples with Source com.google.cloud.securitycenter.v1.Source used on opensource projects

Search in sources :

Example 26 with Source

use of com.google.cloud.securitycenter.v1.Source in project h2o-2 by h2oai.

the class GLMTest2 method testCars.

//------------ TEST on selected files form small data and compare to R results ------------------------------------
/**
   * Simple test for poisson, gamma and gaussian families (no regularization, test both lsm solvers).
   * Basically tries to predict horse power based on other parameters of the cars in the dataset.
   * Compare against the results from standard R glm implementation.
   * @throws ExecutionException
   * @throws InterruptedException
   */
@Test
public void testCars() throws InterruptedException, ExecutionException {
    Key parsed = Key.make("cars_parsed");
    Key modelKey = Key.make("cars_model");
    Frame fr = null;
    GLMModel model = null;
    try {
        String[] ignores = new String[] { "name" };
        String response = "power (hp)";
        fr = getFrameForFile(parsed, "smalldata/cars.csv", ignores, response);
        new GLM2("GLM test on cars.", Key.make(), modelKey, new Source(fr, fr.lastVec(), true), Family.poisson).setRegularization(new double[] { 0 }, new double[] { 0 }).doInit().fork().get();
        model = DKV.get(modelKey).get();
        testHTML(model);
        HashMap<String, Double> coefs = model.coefficients();
        String[] cfs1 = new String[] { "Intercept", "economy (mpg)", "cylinders", "displacement (cc)", "weight (lb)", "0-60 mph (s)", "year" };
        double[] vls1 = new double[] { 4.9504805, -0.0095859, -0.0063046, 0.0004392, 0.0001762, -0.0469810, 0.0002891 };
        for (int i = 0; i < cfs1.length; ++i) assertEquals(vls1[i], coefs.get(cfs1[i]), 1e-4);
        // test gamma
        double[] vls2 = new double[] { 8.992e-03, 1.818e-04, -1.125e-04, 1.505e-06, -1.284e-06, 4.510e-04, -7.254e-05 };
        model.delete();
        new GLM2("GLM test on cars.", Key.make(), modelKey, new Source(fr, fr.lastVec(), true), Family.gamma).setRegularization(new double[] { 0 }, new double[] { 0 }).doInit().fork().get();
        model = DKV.get(modelKey).get();
        testHTML(model);
        coefs = model.coefficients();
        for (int i = 0; i < cfs1.length; ++i) assertEquals(vls2[i], coefs.get(cfs1[i]), 1e-4);
        model.delete();
        // test gaussian
        double[] vls3 = new double[] { 166.95862, -0.00531, -2.46690, 0.12635, 0.02159, -4.66995, -0.85724 };
        new GLM2("GLM test on cars.", Key.make(), modelKey, new Source(fr, fr.lastVec(), true), Family.gaussian).setRegularization(new double[] { 0 }, new double[] { 0 }).doInit().fork().get();
        model = DKV.get(modelKey).get();
        testHTML(model);
        coefs = model.coefficients();
        for (int i = 0; i < cfs1.length; ++i) assertEquals(vls3[i], coefs.get(cfs1[i]), 1e-4);
    } catch (Throwable t) {
        t.printStackTrace();
    } finally {
        if (fr != null)
            fr.delete();
        if (model != null)
            model.delete();
    }
}
Also used : Source(hex.glm.GLM2.Source) Test(org.junit.Test)

Example 27 with Source

use of com.google.cloud.securitycenter.v1.Source in project h2o-2 by h2oai.

the class GLMTest2 method testBounds.

@Test
public void testBounds() {
    //    glmnet's result:
    //    res2 <- glmnet(x=M,y=D$CAPSULE,lower.limits=-.5,upper.limits=.5,family='binomial')
    //    res2$beta[,58]
    //    AGE        RACE          DPROS       PSA         VOL         GLEASON
    //    -0.00616326 -0.50000000  0.50000000  0.03628192 -0.01249324  0.50000000 //    res2$a0[100]
    //    res2$a0[58]
    //    s57
    //    -4.155864
    //    lambda = 0.001108, null dev =  512.2888, res dev = 379.7597
    Key parsed = Key.make("prostate_parsed");
    Key modelKey = Key.make("prostate_model");
    GLMModel model = null;
    Frame fr = getFrameForFile(parsed, "smalldata/logreg/prostate.csv", new String[] { "ID" }, "CAPSULE");
    Key k = Key.make("rebalanced");
    H2O.submitTask(new RebalanceDataSet(fr, k, 64)).join();
    fr.delete();
    fr = DKV.get(k).get();
    Key betaConsKey = Key.make("beta_constraints");
    FVecTest.makeByteVec(betaConsKey, "names, lower_bounds, upper_bounds\n RACE, -.5, .5\n DCAPS, -.4, .4\n DPROS, -.5, .5 \nPSA, -.5, .5\n VOL, -.5, .5\nGLEASON, -.5, .5\nAGE, -.5, .5");
    Frame betaConstraints = ParseDataset2.parse(parsed, new Key[] { betaConsKey });
    try {
        // H2O differs on intercept and race, same residual deviance though
        String[] cfs1 = new String[] { "AGE", "RACE", "DPROS", "DCAPS", "PSA", "VOL", "GLEASON", "Intercept" };
        double[] vals = new double[] { -0.006502588, -0.500000000, 0.500000000, 0.400000000, 0.034826559, -0.011661747, 0.500000000, -4.564024 };
        //.setHighAccuracy().doInit().fork().get();
        new GLM2("GLM offset test on prostate.", Key.make(), modelKey, new GLM2.Source((Frame) fr.clone(), fr.vec("CAPSULE"), true, true), Family.binomial).setNonNegative(false).setRegularization(new double[] { 1 }, new double[] { 0.001607 }).setBetaConstraints(betaConstraints).doInit().fork().get();
        model = DKV.get(modelKey).get();
        Assert.assertTrue(model.get_params().state == Job.JobState.DONE);
        testHTML(model);
        System.out.println(Arrays.toString(model.norm_beta(model.lambda())));
        System.out.println(model.coefficients().toString());
        System.out.println(model.validation().toString());
        HashMap<String, Double> coefs = model.coefficients();
        for (int i = 0; i < cfs1.length; ++i) assertEquals(vals[i], coefs.get(cfs1[i]), 1e-2);
        GLMValidation val = model.validation();
        assertEquals(512.2888, model.null_validation.residualDeviance(), 1e-1);
        assertEquals(388.4686, val.residualDeviance(), 1e-1);
        model.delete();
        betaConstraints.delete();
        // fix AGE at .5
        FVecTest.makeByteVec(betaConsKey, "names, lower_bounds, upper_bounds\n RACE, -.5, .5\n DCAPS, -.4, .4\n DPROS, -.5, .5 \nPSA, -.5, .5\n VOL, -.5, .5\nGLEASON, -.5, .5\nAGE, .5, .5");
        betaConstraints = ParseDataset2.parse(parsed, new Key[] { betaConsKey });
        //.setHighAccuracy().doInit().fork().get();
        new GLM2("GLM offset test on prostate.", Key.make(), modelKey, new GLM2.Source((Frame) fr.clone(), fr.vec("CAPSULE"), true, true), Family.binomial).setNonNegative(false).setRegularization(new double[] { 1 }, new double[] { 0.001607 }).setBetaConstraints(betaConstraints).doInit().fork().get();
        model = DKV.get(modelKey).get();
        System.out.println(model.coefficients());
        assertEquals(.5, model.coefficients().get("AGE"), 1e-16);
        // negative tests
        // a) upper bound > lower bound
        FVecTest.makeByteVec(betaConsKey, "names, lower_bounds, upper_bounds\n AGE, .51, .5\n RACE, -.5, .5\n DCAPS, -.4, .4\n DPROS, -.5, .5 \nPSA, -.5, .5\n VOL, -.5, .5\nGLEASON, -.5, .5");
        model.delete();
        betaConstraints = ParseDataset2.parse(parsed, new Key[] { betaConsKey });
        try {
            //.setHighAccuracy().doInit().fork().get();
            new GLM2("GLM offset test on prostate.", Key.make(), modelKey, new GLM2.Source((Frame) fr.clone(), fr.vec("CAPSULE"), true, true), Family.binomial).setNonNegative(false).setRegularization(new double[] { 1 }, new double[] { 0.001607 }).setBetaConstraints(betaConstraints).doInit().fork().get();
            assertTrue("should've thrown", false);
        } catch (IllegalArgumentException t) {
            assertTrue(t.getMessage().contains("Invalid upper/lower bounds"));
        }
        // b) duplicate coordinate
        FVecTest.makeByteVec(betaConsKey, "names, lower_bounds, upper_bounds\n AGE, .5, .5\n AGE, -.5, .5\n DCAPS, -.4, .4\n DPROS, -.5, .5 \nPSA, -.5, .5\n VOL, -.5, .5\nGLEASON, -.5, .5");
        model.delete();
        betaConstraints = ParseDataset2.parse(parsed, new Key[] { betaConsKey });
        try {
            //.setHighAccuracy().doInit().fork().get();
            new GLM2("GLM offset test on prostate.", Key.make(), modelKey, new GLM2.Source((Frame) fr.clone(), fr.vec("CAPSULE"), true, true), Family.binomial).setNonNegative(false).setRegularization(new double[] { 1 }, new double[] { 0.001607 }).setBetaConstraints(betaConstraints).doInit().fork().get();
            assertTrue("should've thrown", false);
        } catch (IllegalArgumentException t) {
            assertTrue(t.getMessage().contains("duplicate constraints for 'AGE'"));
        }
        FVecTest.makeByteVec(betaConsKey, "names, lower_bounds, upper_bounds\n AGE, .5, .5\n XXX, -.5, .5\n DCAPS, -.4, .4\n DPROS, -.5, .5 \nPSA, -.5, .5\n VOL, -.5, .5\nGLEASON, -.5, .5");
        betaConstraints = ParseDataset2.parse(parsed, new Key[] { betaConsKey });
        try {
            //.setHighAccuracy().doInit().fork().get();
            new GLM2("GLM offset test on prostate.", Key.make(), modelKey, new GLM2.Source((Frame) fr.clone(), fr.vec("CAPSULE"), true, true), Family.binomial).setNonNegative(false).setRegularization(new double[] { 1 }, new double[] { 0.001607 }).setBetaConstraints(betaConstraints).doInit().fork().get();
            assertTrue("should've thrown", false);
        } catch (IllegalArgumentException t) {
            assertTrue(t.getMessage().contains("unknown predictor name 'XXX'"));
        }
        betaConstraints.delete();
        FVecTest.makeByteVec(betaConsKey, "nms, lower_bounds, upper_bounds\n AGE, .5, .5\n XXX, -.5, .5\n DCAPS, -.4, .4\n DPROS, -.5, .5 \nPSA, -.5, .5\n VOL, -.5, .5\nGLEASON, -.5, .5");
        betaConstraints = ParseDataset2.parse(parsed, new Key[] { betaConsKey });
        try {
            //.setHighAccuracy().doInit().fork().get();
            new GLM2("GLM offset test on prostate.", Key.make(), modelKey, new GLM2.Source((Frame) fr.clone(), fr.vec("CAPSULE"), true, true), Family.binomial).setNonNegative(false).setRegularization(new double[] { 1 }, new double[] { 0.001607 }).setBetaConstraints(betaConstraints).doInit().fork().get();
            assertTrue("should've thrown", false);
        } catch (IllegalArgumentException t) {
            assertTrue(t.getMessage().contains("missing column with predictor names"));
        }
    } finally {
        if (betaConstraints != null)
            betaConstraints.delete();
        fr.delete();
        if (model != null)
            model.delete();
    }
}
Also used : Source(hex.glm.GLM2.Source) Test(org.junit.Test)

Example 28 with Source

use of com.google.cloud.securitycenter.v1.Source in project h2o-2 by h2oai.

the class GLMTest2 method testProximal.

@Test
public void testProximal() {
    //    glmnet's result:
    //    res2 <- glmnet(x=M,y=D$CAPSULE,lower.limits=-.5,upper.limits=.5,family='binomial')
    //    res2$beta[,58]
    //    AGE        RACE          DPROS       PSA         VOL         GLEASON
    //    -0.00616326 -0.50000000  0.50000000  0.03628192 -0.01249324  0.50000000 //    res2$a0[100]
    //    res2$a0[58]
    //    s57
    //    -4.155864
    //    lambda = 0.001108, null dev =  512.2888, res dev = 379.7597
    Key parsed = Key.make("prostate_parsed");
    Key modelKey = Key.make("prostate_model");
    GLMModel model = null;
    Frame fr = getFrameForFile(parsed, "smalldata/logreg/prostate.csv", new String[] { "ID" }, "CAPSULE");
    Key k = Key.make("rebalanced");
    H2O.submitTask(new RebalanceDataSet(fr, k, 64)).join();
    fr.delete();
    fr = DKV.get(k).get();
    fr.remove("ID");
    Key betaConsKey = Key.make("beta_constraints");
    //String[] cfs1 = new String[]{"RACE", "AGE", "DPROS", "DCAPS", "PSA", "VOL", "GLEASON","Intercept"};
    //double[] vals = new double[]{0, 0, 0.54788332,0.53816534, 0.02380097, 0, 0.98115670,-8.945984};
    // [AGE, RACE, DPROS, DCAPS, PSA, VOL, GLEASON, Intercept]
    FVecTest.makeByteVec(betaConsKey, "names, beta_given, rho\n AGE, 0.1, 1\nRACE, -0.1, 1 \n DPROS, 10, 1 \n DCAPS, -10, 1 \n PSA, 0, 1\n VOL, 0, 1\nGLEASON, 0, 1\n Intercept, 0, 0 \n");
    Frame betaConstraints = ParseDataset2.parse(parsed, new Key[] { betaConsKey });
    try {
        // H2O differs on intercept and race, same residual deviance though
        GLM2.Source src = new GLM2.Source((Frame) fr.clone(), fr.vec("CAPSULE"), false, true);
        //.setHighAccuracy().doInit().fork().get();
        new GLM2("GLM offset test on prostate.", Key.make(), modelKey, src, Family.binomial).setNonNegative(false).setRegularization(new double[] { 0 }, new double[] { 0.000 }).setBetaConstraints(betaConstraints).setHighAccuracy().doInit().fork().get();
        model = DKV.get(modelKey).get();
        fr.add("CAPSULE", fr.remove("CAPSULE"));
        DataInfo dinfo = new DataInfo(fr, 1, true, false, TransformType.NONE, DataInfo.TransformType.NONE);
        GLMIterationTask glmt = new GLMTask.GLMIterationTask(0, null, dinfo, new GLMParams(Family.binomial), false, true, true, model.beta(), 0, 1.0 / 380, ModelUtils.DEFAULT_THRESHOLDS, null).doAll(dinfo._adaptedFrame);
        double[] beta = model.beta();
        double[] grad = glmt.gradient(0, 0);
        for (int i = 0; i < beta.length; ++i) Assert.assertEquals(0, grad[i] + betaConstraints.vec("rho").at(i) * (beta[i] - betaConstraints.vec("beta_given").at(i)), 1e-8);
        // now standardized
        src = new GLM2.Source((Frame) fr.clone(), fr.vec("CAPSULE"), true, true);
        //.setHighAccuracy().doInit().fork().get();
        new GLM2("GLM offset test on prostate.", Key.make(), modelKey, src, Family.binomial).setNonNegative(false).setRegularization(new double[] { 0 }, new double[] { 0.000 }).setBetaConstraints(betaConstraints).setHighAccuracy().doInit().fork().get();
        model = DKV.get(modelKey).get();
        fr.add("CAPSULE", fr.remove("CAPSULE"));
        dinfo = new DataInfo(fr, 1, true, false, TransformType.STANDARDIZE, DataInfo.TransformType.NONE);
        glmt = new GLMTask.GLMIterationTask(0, null, dinfo, new GLMParams(Family.binomial), false, true, true, model.norm_beta(0), 0, 1.0 / 380, ModelUtils.DEFAULT_THRESHOLDS, null).doAll(dinfo._adaptedFrame);
        double[] beta2 = model.norm_beta(0);
        double[] grad2 = glmt.gradient(0, 0);
        for (int i = 0; i < beta.length - 1; ++i) Assert.assertEquals("grad[" + i + "] != 0", 0, grad2[i] + betaConstraints.vec("rho").at(i) * (beta2[i] - betaConstraints.vec("beta_given").at(i) * dinfo._adaptedFrame.vec(i).sigma()), 1e-8);
        Assert.assertEquals("grad[intercept] != 0", 0, grad2[grad2.length - 1], 1e-8);
    } finally {
        fr.delete();
        if (model != null)
            model.delete();
    }
}
Also used : Source(hex.glm.GLM2.Source) DataInfo(hex.FrameTask.DataInfo) GLMIterationTask(hex.glm.GLMTask.GLMIterationTask) Source(hex.glm.GLM2.Source) GLMIterationTask(hex.glm.GLMTask.GLMIterationTask) Test(org.junit.Test)

Example 29 with Source

use of com.google.cloud.securitycenter.v1.Source in project h2o-2 by h2oai.

the class GLMTest2 method testPoissonRegression.

/**
  * Test Poisson regression on simple and small synthetic dataset.
  * Equation is: y = exp(x+1);
  */
@Test
public void testPoissonRegression() throws InterruptedException, ExecutionException {
    Key raw = Key.make("poisson_test_data_raw");
    Key parsed = Key.make("poisson_test_data_parsed");
    Key modelKey = Key.make("poisson_test");
    GLMModel model = null;
    Frame fr = null;
    try {
        // make data so that the expected coefficients is icept = col[0] = 1.0
        FVecTest.makeByteVec(raw, "x,y\n0,2\n1,4\n2,8\n3,16\n4,32\n5,64\n6,128\n7,256");
        fr = ParseDataset2.parse(parsed, new Key[] { raw });
        new GLM2("GLM test of poisson regression.", Key.make(), modelKey, new Source(fr, fr.lastVec(), false), Family.poisson).setRegularization(new double[] { 0 }, new double[] { 0 }).doInit().fork().get();
        model = DKV.get(modelKey).get();
        for (double c : model.beta()) assertEquals(Math.log(2), c, 1e-4);
        // Test 2, example from http://www.biostat.umn.edu/~dipankar/bmtry711.11/lecture_13.pdf
        //new byte []{1,2,3,4,5,6,7,8, 9, 10,11,12,13,14},
        //     new byte []{0,1,2,3,1,4,9,18,23,31,20,25,37,45});
        model.delete();
        fr.delete();
        FVecTest.makeByteVec(raw, "x,y\n1,0\n2,1\n3,2\n4,3\n5,1\n6,4\n7,9\n8,18\n9,23\n10,31\n11,20\n12,25\n13,37\n14,45\n");
        fr = ParseDataset2.parse(parsed, new Key[] { raw });
        new GLM2("GLM test of poisson regression(2).", Key.make(), modelKey, new Source(fr, fr.lastVec(), false), Family.poisson).setRegularization(new double[] { 0 }, new double[] { 0 }).doInit().fork().get();
        model = DKV.get(modelKey).get();
        testHTML(model);
        assertEquals(0.3396, model.beta()[1], 5e-3);
        assertEquals(0.2565, model.beta()[0], 5e-3);
    } finally {
        if (fr != null)
            fr.delete();
        if (model != null)
            model.delete();
    }
}
Also used : Source(hex.glm.GLM2.Source) Test(org.junit.Test)

Example 30 with Source

use of com.google.cloud.securitycenter.v1.Source in project ORCID-Source by ORCID.

the class JpaJaxbNameAdapterTest method fromNameToRecordNameEntityTest.

@Test
public void fromNameToRecordNameEntityTest() throws JAXBException {
    Name name = new Name();
    name.setCreditName(new CreditName("Credit Name"));
    name.setFamilyName(new FamilyName("Family Name"));
    name.setGivenNames(new GivenNames("Given Names"));
    name.setPath("0000-0000-0000-0000");
    name.setVisibility(Visibility.PUBLIC);
    name.setSource(new Source("0000-0000-0000-0000"));
    RecordNameEntity entity = adapter.toRecordNameEntity(name);
    assertNotNull(entity);
    assertEquals("Credit Name", entity.getCreditName());
    assertEquals("Family Name", entity.getFamilyName());
    assertEquals("Given Names", entity.getGivenNames());
    assertEquals(Visibility.PUBLIC, entity.getVisibility());
    assertNotNull(entity.getProfile());
    assertEquals("0000-0000-0000-0000", entity.getProfile().getId());
}
Also used : FamilyName(org.orcid.jaxb.model.record_v2.FamilyName) GivenNames(org.orcid.jaxb.model.record_v2.GivenNames) CreditName(org.orcid.jaxb.model.common_v2.CreditName) RecordNameEntity(org.orcid.persistence.jpa.entities.RecordNameEntity) Source(org.orcid.jaxb.model.common_v2.Source) FamilyName(org.orcid.jaxb.model.record_v2.FamilyName) CreditName(org.orcid.jaxb.model.common_v2.CreditName) Name(org.orcid.jaxb.model.record_v2.Name) Test(org.junit.Test)

Aggregations

SecurityCenterClient (com.google.cloud.securitycenter.v1.SecurityCenterClient)20 IOException (java.io.IOException)20 Test (org.junit.Test)17 Source (hex.glm.GLM2.Source)11 Source (org.orcid.jaxb.model.common_v2.Source)10 Instant (org.threeten.bp.Instant)6 SourceClientId (org.orcid.jaxb.model.common_v2.SourceClientId)5 Finding (com.google.cloud.securitycenter.v1.Finding)4 GroupFindingsRequest (com.google.cloud.securitycenter.v1.GroupFindingsRequest)4 GroupResult (com.google.cloud.securitycenter.v1.GroupResult)4 ListFindingsRequest (com.google.cloud.securitycenter.v1.ListFindingsRequest)4 ListFindingsResult (com.google.cloud.securitycenter.v1.ListFindingsResponse.ListFindingsResult)4 GroupFindingsPagedResponse (com.google.cloud.securitycenter.v1.SecurityCenterClient.GroupFindingsPagedResponse)4 ListFindingsPagedResponse (com.google.cloud.securitycenter.v1.SecurityCenterClient.ListFindingsPagedResponse)4 Source (com.google.cloud.securitycenter.v1.Source)4 FieldMask (com.google.protobuf.FieldMask)3 SourceName (org.orcid.jaxb.model.common_v2.SourceName)3 Items (org.orcid.jaxb.model.notification.permission_v2.Items)3 NotificationPermission (org.orcid.jaxb.model.notification.permission_v2.NotificationPermission)3 Policy (com.google.iam.v1.Policy)2
About Me
UseOf

Java Tutorials :

Simple Java RabbitMQ Example with Spring
Simple Springboot JPA Eclipeslink Example
Simple SpringBoot JPA Hibernate Example
Jackson JSON @JsonIgnore and @JsonIgnoreProperties Examples
Java Infinite recursion (StackOverflowError) Jackson solutions
Simple Java Jackson Serialize Objects to JSON and convert them back To Object
ElasticSearch 5 - Upload files using Java API in binary field Example
Java JAXB marshall unmarshall Examples from String and Stream
Java 8 forEach List and Map examples
ElasticSearch 5 Java API SearchRequestBuilder examples
Java ElasticSearch 5 examples with node index and mapping - running local
Convert String to int or Integer Java 8
Java 9 in action - JShell - Ubuntu
Java - removing invalid characters from a file name
Hello world!? or Hello Tutorial