Search in sources :

Example 46 with ApplicationId

use of co.cask.cdap.proto.id.ApplicationId in project cdap by caskdata.

the class ExistingEntitySystemMetadataWriter method writeSystemMetadataForApps.

private void writeSystemMetadataForApps(NamespaceId namespace) {
    for (ApplicationSpecification appSpec : store.getAllApplications(namespace)) {
        ApplicationId app = namespace.app(appSpec.getName());
        SystemMetadataWriter writer = new AppSystemMetadataWriter(metadataStore, app, appSpec);
        writer.write();
        writeSystemMetadataForPrograms(app, appSpec);
    }
}
Also used : ApplicationSpecification(co.cask.cdap.api.app.ApplicationSpecification) DatasetSystemMetadataWriter(co.cask.cdap.data2.metadata.system.DatasetSystemMetadataWriter) ProgramSystemMetadataWriter(co.cask.cdap.data2.metadata.system.ProgramSystemMetadataWriter) ViewSystemMetadataWriter(co.cask.cdap.data2.metadata.system.ViewSystemMetadataWriter) SystemMetadataWriter(co.cask.cdap.data2.metadata.system.SystemMetadataWriter) AppSystemMetadataWriter(co.cask.cdap.data2.metadata.system.AppSystemMetadataWriter) ArtifactSystemMetadataWriter(co.cask.cdap.data2.metadata.system.ArtifactSystemMetadataWriter) StreamSystemMetadataWriter(co.cask.cdap.data2.metadata.system.StreamSystemMetadataWriter) ApplicationId(co.cask.cdap.proto.id.ApplicationId) AppSystemMetadataWriter(co.cask.cdap.data2.metadata.system.AppSystemMetadataWriter)

Example 47 with ApplicationId

use of co.cask.cdap.proto.id.ApplicationId in project cdap by caskdata.

the class FlowVerificationTest method testFlowWithMoreOutputThanWhatInputCanHandle.

@Test
public void testFlowWithMoreOutputThanWhatInputCanHandle() throws Exception {
    ApplicationSpecification appSpec = Specifications.from(new WebCrawlApp());
    ApplicationSpecificationAdapter adapter = ApplicationSpecificationAdapter.create(new ReflectionSchemaGenerator());
    ApplicationSpecification newSpec = adapter.fromJson(adapter.toJson(appSpec));
    FlowVerification flowSpec = new FlowVerification();
    for (Map.Entry<String, FlowSpecification> entry : newSpec.getFlows().entrySet()) {
        VerifyResult result = flowSpec.verify(new ApplicationId("test", newSpec.getName()), entry.getValue());
        // that is not connected to any input to flowlet CountByField.
        if (entry.getValue().getName().equals("WordCountFlow")) {
            Assert.assertTrue(result.getStatus() == VerifyResult.Status.FAILED);
        } else {
            Assert.assertTrue(result.getStatus() == VerifyResult.Status.SUCCESS);
        }
    }
}
Also used : ApplicationSpecification(co.cask.cdap.api.app.ApplicationSpecification) WebCrawlApp(co.cask.cdap.WebCrawlApp) FlowSpecification(co.cask.cdap.api.flow.FlowSpecification) ApplicationSpecificationAdapter(co.cask.cdap.internal.app.ApplicationSpecificationAdapter) ReflectionSchemaGenerator(co.cask.cdap.internal.io.ReflectionSchemaGenerator) VerifyResult(co.cask.cdap.app.verification.VerifyResult) ApplicationId(co.cask.cdap.proto.id.ApplicationId) Map(java.util.Map) Test(org.junit.Test)

Example 48 with ApplicationId

use of co.cask.cdap.proto.id.ApplicationId in project cdap by caskdata.

the class FlowVerificationTest method testFlowMissingConnection.

/**
   * This test that verification of flow connections
   */
@Test
public void testFlowMissingConnection() throws Exception {
    ApplicationSpecification appSpec = Specifications.from(new NoConsumerApp());
    ApplicationSpecificationAdapter adapter = ApplicationSpecificationAdapter.create(new ReflectionSchemaGenerator());
    ApplicationSpecification newSpec = adapter.fromJson(adapter.toJson(appSpec));
    FlowVerification flowVerifier = new FlowVerification();
    for (FlowSpecification flowSpec : appSpec.getFlows().values()) {
        VerifyResult result = flowVerifier.verify(new ApplicationId("test", newSpec.getName()), flowSpec);
        Assert.assertTrue(result.getStatus() == VerifyResult.Status.FAILED);
    }
}
Also used : ApplicationSpecification(co.cask.cdap.api.app.ApplicationSpecification) FlowSpecification(co.cask.cdap.api.flow.FlowSpecification) ApplicationSpecificationAdapter(co.cask.cdap.internal.app.ApplicationSpecificationAdapter) ReflectionSchemaGenerator(co.cask.cdap.internal.io.ReflectionSchemaGenerator) VerifyResult(co.cask.cdap.app.verification.VerifyResult) ApplicationId(co.cask.cdap.proto.id.ApplicationId) Test(org.junit.Test)

Example 49 with ApplicationId

use of co.cask.cdap.proto.id.ApplicationId in project cdap by caskdata.

the class AppMetadataStoreTest method testScanRunningInRangeWithBatch.

@Test
public void testScanRunningInRangeWithBatch() throws Exception {
    DatasetId storeTable = NamespaceId.DEFAULT.dataset("testScanRunningInRange");
    datasetFramework.addInstance(Table.class.getName(), storeTable, DatasetProperties.EMPTY);
    Table table = datasetFramework.getDataset(storeTable, ImmutableMap.<String, String>of(), null);
    Assert.assertNotNull(table);
    final AppMetadataStore metadataStoreDataset = new AppMetadataStore(table, cConf, new AtomicBoolean(false));
    TransactionExecutor txnl = txExecutorFactory.createExecutor(Collections.singleton((TransactionAware) metadataStoreDataset));
    // Add some run records
    TreeSet<Long> expected = new TreeSet<>();
    for (int i = 0; i < 100; ++i) {
        ApplicationId application = NamespaceId.DEFAULT.app("app" + i);
        final ProgramId program = application.program(ProgramType.values()[i % ProgramType.values().length], "program" + i);
        final RunId runId = RunIds.generate((i + 1) * 10000);
        expected.add(RunIds.getTime(runId, TimeUnit.MILLISECONDS));
        // Start the program and stop it
        final int j = i;
        txnl.execute(new TransactionExecutor.Subroutine() {

            @Override
            public void apply() throws Exception {
                metadataStoreDataset.recordProgramStart(program, runId.getId(), RunIds.getTime(runId, TimeUnit.SECONDS), null, null, null);
                metadataStoreDataset.recordProgramStop(program, runId.getId(), RunIds.getTime(runId, TimeUnit.SECONDS), ProgramRunStatus.values()[j % ProgramRunStatus.values().length], null);
            }
        });
    }
    // Run full scan
    runScan(txnl, metadataStoreDataset, expected, 0, Long.MAX_VALUE);
    // In all below assertions, TreeSet and metadataStore both have start time inclusive and end time exclusive.
    // Run the scan with time limit
    runScan(txnl, metadataStoreDataset, expected.subSet(30 * 10000L, 90 * 10000L), TimeUnit.MILLISECONDS.toSeconds(30 * 10000), TimeUnit.MILLISECONDS.toSeconds(90 * 10000));
    runScan(txnl, metadataStoreDataset, expected.subSet(90 * 10000L, 101 * 10000L), TimeUnit.MILLISECONDS.toSeconds(90 * 10000), TimeUnit.MILLISECONDS.toSeconds(101 * 10000));
    // After range
    runScan(txnl, metadataStoreDataset, expected.subSet(101 * 10000L, 200 * 10000L), TimeUnit.MILLISECONDS.toSeconds(101 * 10000), TimeUnit.MILLISECONDS.toSeconds(200 * 10000));
    // Identical start and end time
    runScan(txnl, metadataStoreDataset, expected.subSet(31 * 10000L, 31 * 10000L), TimeUnit.MILLISECONDS.toSeconds(31 * 10000), TimeUnit.MILLISECONDS.toSeconds(31 * 10000));
    // One unit difference between start and end time
    runScan(txnl, metadataStoreDataset, expected.subSet(30 * 10000L, 31 * 10000L), TimeUnit.MILLISECONDS.toSeconds(30 * 10000), TimeUnit.MILLISECONDS.toSeconds(31 * 10000));
    // Before range
    runScan(txnl, metadataStoreDataset, expected.subSet(1000L, 10000L), TimeUnit.MILLISECONDS.toSeconds(1000), TimeUnit.MILLISECONDS.toSeconds(10000));
}
Also used : Table(co.cask.cdap.api.dataset.table.Table) TransactionExecutor(org.apache.tephra.TransactionExecutor) ProgramId(co.cask.cdap.proto.id.ProgramId) TransactionFailureException(org.apache.tephra.TransactionFailureException) DatasetId(co.cask.cdap.proto.id.DatasetId) AtomicBoolean(java.util.concurrent.atomic.AtomicBoolean) TransactionAware(org.apache.tephra.TransactionAware) TreeSet(java.util.TreeSet) ApplicationId(co.cask.cdap.proto.id.ApplicationId) RunId(org.apache.twill.api.RunId) ProgramRunId(co.cask.cdap.proto.id.ProgramRunId) Test(org.junit.Test)

Example 50 with ApplicationId

use of co.cask.cdap.proto.id.ApplicationId in project cdap by caskdata.

the class AppMetadataStoreTest method testgetRuns.

@Test
public void testgetRuns() throws Exception {
    DatasetId storeTable = NamespaceId.DEFAULT.dataset("testgetRuns");
    datasetFramework.addInstance(Table.class.getName(), storeTable, DatasetProperties.EMPTY);
    Table table = datasetFramework.getDataset(storeTable, ImmutableMap.<String, String>of(), null);
    Assert.assertNotNull(table);
    final AppMetadataStore metadataStoreDataset = new AppMetadataStore(table, cConf, new AtomicBoolean(false));
    TransactionExecutor txnl = txExecutorFactory.createExecutor(Collections.singleton((TransactionAware) metadataStoreDataset));
    // Add some run records
    final Set<String> expected = new TreeSet<>();
    final Set<String> expectedHalf = new TreeSet<>();
    final Set<ProgramRunId> programRunIdSet = new HashSet<>();
    final Set<ProgramRunId> programRunIdSetHalf = new HashSet<>();
    for (int i = 0; i < 100; ++i) {
        ApplicationId application = NamespaceId.DEFAULT.app("app");
        final ProgramId program = application.program(ProgramType.FLOW, "program");
        final RunId runId = RunIds.generate((i + 1) * 10000);
        expected.add(runId.toString());
        final int index = i;
        // Add every other runId
        if ((i % 2) == 0) {
            expectedHalf.add(runId.toString());
        }
        ProgramRunId programRunId = new ProgramRunId(program.getNamespace(), program.getApplication(), program.getType(), program.getProgram(), runId.toString());
        programRunIdSet.add(programRunId);
        //Add every other programRunId
        if ((i % 2) == 0) {
            programRunIdSetHalf.add(programRunId);
        }
        txnl.execute(new TransactionExecutor.Subroutine() {

            @Override
            public void apply() throws Exception {
                // Start the program and stop it
                metadataStoreDataset.recordProgramStart(program, runId.getId(), RunIds.getTime(runId, TimeUnit.SECONDS), null, null, null);
                metadataStoreDataset.recordProgramStop(program, runId.getId(), RunIds.getTime(runId, TimeUnit.SECONDS), ProgramRunStatus.values()[index % ProgramRunStatus.values().length], null);
            }
        });
    }
    txnl.execute(new TransactionExecutor.Subroutine() {

        @Override
        public void apply() throws Exception {
            Map<ProgramRunId, RunRecordMeta> runMap = metadataStoreDataset.getRuns(programRunIdSet);
            Set<String> actual = new TreeSet<>();
            for (Map.Entry<ProgramRunId, RunRecordMeta> entry : runMap.entrySet()) {
                actual.add(entry.getValue().getPid());
            }
            Assert.assertEquals(expected, actual);
            Map<ProgramRunId, RunRecordMeta> runMapHalf = metadataStoreDataset.getRuns(programRunIdSetHalf);
            Set<String> actualHalf = new TreeSet<>();
            for (Map.Entry<ProgramRunId, RunRecordMeta> entry : runMapHalf.entrySet()) {
                actualHalf.add(entry.getValue().getPid());
            }
            Assert.assertEquals(expectedHalf, actualHalf);
        }
    });
}
Also used : Table(co.cask.cdap.api.dataset.table.Table) TreeSet(java.util.TreeSet) HashSet(java.util.HashSet) Set(java.util.Set) TransactionExecutor(org.apache.tephra.TransactionExecutor) ProgramId(co.cask.cdap.proto.id.ProgramId) TransactionFailureException(org.apache.tephra.TransactionFailureException) DatasetId(co.cask.cdap.proto.id.DatasetId) AtomicBoolean(java.util.concurrent.atomic.AtomicBoolean) TransactionAware(org.apache.tephra.TransactionAware) TreeSet(java.util.TreeSet) ProgramRunId(co.cask.cdap.proto.id.ProgramRunId) ApplicationId(co.cask.cdap.proto.id.ApplicationId) RunId(org.apache.twill.api.RunId) ProgramRunId(co.cask.cdap.proto.id.ProgramRunId) Map(java.util.Map) ImmutableMap(com.google.common.collect.ImmutableMap) HashSet(java.util.HashSet) Test(org.junit.Test)

Aggregations

ApplicationId (co.cask.cdap.proto.id.ApplicationId)234 Test (org.junit.Test)123 ProgramId (co.cask.cdap.proto.id.ProgramId)73 AppRequest (co.cask.cdap.proto.artifact.AppRequest)64 ApplicationManager (co.cask.cdap.test.ApplicationManager)49 ApplicationSpecification (co.cask.cdap.api.app.ApplicationSpecification)45 ETLStage (co.cask.cdap.etl.proto.v2.ETLStage)44 NamespaceId (co.cask.cdap.proto.id.NamespaceId)43 Table (co.cask.cdap.api.dataset.table.Table)37 StructuredRecord (co.cask.cdap.api.data.format.StructuredRecord)36 Schema (co.cask.cdap.api.data.schema.Schema)34 ETLBatchConfig (co.cask.cdap.etl.proto.v2.ETLBatchConfig)32 WorkflowManager (co.cask.cdap.test.WorkflowManager)31 Path (javax.ws.rs.Path)31 ArtifactSummary (co.cask.cdap.api.artifact.ArtifactSummary)29 StreamId (co.cask.cdap.proto.id.StreamId)28 KeyValueTable (co.cask.cdap.api.dataset.lib.KeyValueTable)26 ArrayList (java.util.ArrayList)25 HashSet (java.util.HashSet)24 NotFoundException (co.cask.cdap.common.NotFoundException)23