Search in sources :

Example 1 with FieldDetail

use of co.cask.cdap.dq.FieldDetail in project cdap by caskdata.

the class DataQualityAppTest method testTotals.

@Test
public void testTotals() throws Exception {
    Map<String, Set<String>> testMap = new HashMap<>();
    Set<String> testSet = new HashSet<>();
    testSet.add("DiscreteValuesHistogram");
    testMap.put("content_length", testSet);
    testMap.put("status", testSet);
    testMap.put("request_time", testSet);
    DataQualityApp.DataQualityConfig config = new DataQualityApp.DataQualityConfig(WORKFLOW_SCHEDULE_MINUTES, getStreamSource(), "histogram", testMap);
    ApplicationId appId = NamespaceId.DEFAULT.app("newApp3");
    AppRequest<DataQualityApp.DataQualityConfig> appRequest = new AppRequest<>(new ArtifactSummary(appArtifact.getArtifact(), appArtifact.getVersion()), config);
    ApplicationManager applicationManager = deployApplication(appId, appRequest);
    MapReduceManager mrManager = applicationManager.getMapReduceManager("FieldAggregator").start();
    mrManager.waitForRun(ProgramRunStatus.COMPLETED, 180, TimeUnit.SECONDS);
    Map<String, Integer> expectedMap = new HashMap<>();
    expectedMap.put("256", 3);
    /* Test for the aggregationsGetter handler */
    ServiceManager serviceManager = applicationManager.getServiceManager(DataQualityService.SERVICE_NAME).start();
    serviceManager.waitForStatus(true);
    URL url = new URL(serviceManager.getServiceURL(), "v1/sources/logStream/fields/content_length/aggregations/DiscreteValuesHistogram/totals");
    HttpResponse httpResponse = HttpRequests.execute(HttpRequest.get(url).build());
    Assert.assertEquals(HttpURLConnection.HTTP_OK, httpResponse.getResponseCode());
    String response = httpResponse.getResponseBodyAsString();
    Map<String, Integer> histogramMap = GSON.fromJson(response, TOKEN_TYPE_MAP_STRING_INTEGER);
    Assert.assertEquals(expectedMap, histogramMap);
    /* Test for the fieldsGetter handler */
    url = new URL(serviceManager.getServiceURL(), "v1/sources/logStream/fields");
    httpResponse = HttpRequests.execute(HttpRequest.get(url).build());
    Assert.assertEquals(HttpURLConnection.HTTP_OK, httpResponse.getResponseCode());
    response = httpResponse.getResponseBodyAsString();
    Set<FieldDetail> outputSet = GSON.fromJson(response, TOKEN_TYPE_SET_FIELD_DETAIL);
    Set<FieldDetail> expectedSet = new HashSet<>();
    AggregationTypeValue aggregationTypeValue = new AggregationTypeValue("DiscreteValuesHistogram", true);
    Set<AggregationTypeValue> aggregationTypeValuesList = Sets.newHashSet(aggregationTypeValue);
    expectedSet.add(new FieldDetail("content_length", aggregationTypeValuesList));
    expectedSet.add(new FieldDetail("request_time", aggregationTypeValuesList));
    expectedSet.add(new FieldDetail("status", aggregationTypeValuesList));
    Assert.assertEquals(expectedSet, outputSet);
    /* Test for the aggregationTypesGetter handler */
    url = new URL(serviceManager.getServiceURL(), "v1/sources/logStream/fields/content_length");
    httpResponse = HttpRequests.execute(HttpRequest.get(url).build());
    Assert.assertEquals(HttpURLConnection.HTTP_OK, httpResponse.getResponseCode());
    response = httpResponse.getResponseBodyAsString();
    List<AggregationTypeValue> expectedAggregationTypeValuesList = new ArrayList<>();
    List<AggregationTypeValue> outputAggregationTypeValuesList = GSON.fromJson(response, TOKEN_TYPE_LIST_AGGREGATION_TYPE_VALUES);
    expectedAggregationTypeValuesList.add(new AggregationTypeValue("DiscreteValuesHistogram", true));
    Assert.assertEquals(expectedAggregationTypeValuesList, outputAggregationTypeValuesList);
    serviceManager.stop();
    serviceManager.waitForRun(ProgramRunStatus.KILLED, 180, TimeUnit.SECONDS);
}
Also used : ApplicationManager(co.cask.cdap.test.ApplicationManager) Set(java.util.Set) HashSet(java.util.HashSet) MapReduceManager(co.cask.cdap.test.MapReduceManager) HashMap(java.util.HashMap) ArrayList(java.util.ArrayList) URL(java.net.URL) ServiceManager(co.cask.cdap.test.ServiceManager) AggregationTypeValue(co.cask.cdap.dq.AggregationTypeValue) HashSet(java.util.HashSet) HttpResponse(co.cask.common.http.HttpResponse) FieldDetail(co.cask.cdap.dq.FieldDetail) AppRequest(co.cask.cdap.proto.artifact.AppRequest) ArtifactSummary(co.cask.cdap.api.artifact.ArtifactSummary) ApplicationId(co.cask.cdap.proto.id.ApplicationId) DataQualityApp(co.cask.cdap.dq.DataQualityApp) Test(org.junit.Test)

Example 2 with FieldDetail

use of co.cask.cdap.dq.FieldDetail in project cdap by caskdata.

the class FieldDetailTest method test.

@Test
public void test() {
    Set<AggregationTypeValue> aggregationTypeValuesSet1 = Sets.newHashSet(new AggregationTypeValue("a", true), new AggregationTypeValue("b", true));
    Set<AggregationTypeValue> aggregationTypeValuesSet2 = Sets.newHashSet(new AggregationTypeValue("a", true), new AggregationTypeValue("c", true));
    Set<AggregationTypeValue> aggregationTypeValuesSetExpected = Sets.newHashSet(new AggregationTypeValue("a", true), new AggregationTypeValue("b", true), new AggregationTypeValue("c", true));
    FieldDetail fieldDetail1 = new FieldDetail("name", aggregationTypeValuesSet1);
    FieldDetail fieldDetail2 = new FieldDetail("name", aggregationTypeValuesSet2);
    fieldDetail1.addAggregations(fieldDetail2.getAggregationTypeSet());
    FieldDetail expected = new FieldDetail("name", aggregationTypeValuesSetExpected);
    Assert.assertEquals(expected, fieldDetail1);
}
Also used : AggregationTypeValue(co.cask.cdap.dq.AggregationTypeValue) FieldDetail(co.cask.cdap.dq.FieldDetail) Test(org.junit.Test)

Aggregations

AggregationTypeValue (co.cask.cdap.dq.AggregationTypeValue)2 FieldDetail (co.cask.cdap.dq.FieldDetail)2 Test (org.junit.Test)2 ArtifactSummary (co.cask.cdap.api.artifact.ArtifactSummary)1 DataQualityApp (co.cask.cdap.dq.DataQualityApp)1 AppRequest (co.cask.cdap.proto.artifact.AppRequest)1 ApplicationId (co.cask.cdap.proto.id.ApplicationId)1 ApplicationManager (co.cask.cdap.test.ApplicationManager)1 MapReduceManager (co.cask.cdap.test.MapReduceManager)1 ServiceManager (co.cask.cdap.test.ServiceManager)1 HttpResponse (co.cask.common.http.HttpResponse)1 URL (java.net.URL)1 ArrayList (java.util.ArrayList)1 HashMap (java.util.HashMap)1 HashSet (java.util.HashSet)1 Set (java.util.Set)1