Search in sources :

Example 1 with ObjectStore

use of io.cdap.cdap.api.dataset.lib.ObjectStore in project cdap by caskdata.

the class CoreDatasetsModule method register.

@Override
public void register(DatasetDefinitionRegistry registry) {
    DatasetDefinition<Table, DatasetAdmin> tableDef = registry.get("table");
    DatasetDefinition<KeyValueTable, DatasetAdmin> kvTableDef = new KeyValueTableDefinition(KeyValueTable.TYPE, tableDef);
    registry.add(kvTableDef);
    registry.add(new KeyValueTableDefinition(KeyValueTable.class.getName(), tableDef));
    DatasetDefinition<ObjectStore, DatasetAdmin> objectStoreDef = new ObjectStoreDefinition(ObjectStore.TYPE, kvTableDef);
    registry.add(new ObjectStoreDefinition(ObjectStore.TYPE, kvTableDef));
    registry.add(new ObjectStoreDefinition(ObjectStore.class.getName(), kvTableDef));
    registry.add(new IndexedObjectStoreDefinition(IndexedObjectStore.TYPE, tableDef, objectStoreDef));
    registry.add(new IndexedObjectStoreDefinition(IndexedObjectStore.class.getName(), tableDef, objectStoreDef));
    registry.add(new IndexedTableDefinition(IndexedTable.TYPE, tableDef));
    registry.add(new IndexedTableDefinition(IndexedTable.class.getName(), tableDef));
    registry.add(new TimeseriesTableDefinition(TimeseriesTable.TYPE, tableDef));
    registry.add(new TimeseriesTableDefinition(TimeseriesTable.class.getName(), tableDef));
    registry.add(new CounterTimeseriesTableDefinition(CounterTimeseriesTable.TYPE, tableDef));
    registry.add(new CounterTimeseriesTableDefinition(CounterTimeseriesTable.class.getName(), tableDef));
    // in-memory table
    registry.add(new InMemoryTableDefinition(InMemoryTable.TYPE));
}
Also used : ObjectStore(io.cdap.cdap.api.dataset.lib.ObjectStore) IndexedObjectStore(io.cdap.cdap.api.dataset.lib.IndexedObjectStore) Table(io.cdap.cdap.api.dataset.table.Table) InMemoryTable(io.cdap.cdap.data2.dataset2.lib.table.inmemory.InMemoryTable) CounterTimeseriesTable(io.cdap.cdap.api.dataset.lib.CounterTimeseriesTable) TimeseriesTable(io.cdap.cdap.api.dataset.lib.TimeseriesTable) IndexedTable(io.cdap.cdap.api.dataset.lib.IndexedTable) KeyValueTable(io.cdap.cdap.api.dataset.lib.KeyValueTable) DatasetAdmin(io.cdap.cdap.api.dataset.DatasetAdmin) InMemoryTableDefinition(io.cdap.cdap.data2.dataset2.lib.table.inmemory.InMemoryTableDefinition) CounterTimeseriesTableDefinition(io.cdap.cdap.api.dataset.lib.CounterTimeseriesTableDefinition) IndexedTableDefinition(io.cdap.cdap.api.dataset.lib.IndexedTableDefinition) KeyValueTable(io.cdap.cdap.api.dataset.lib.KeyValueTable) IndexedObjectStoreDefinition(io.cdap.cdap.api.dataset.lib.IndexedObjectStoreDefinition) KeyValueTableDefinition(io.cdap.cdap.api.dataset.lib.KeyValueTableDefinition) IndexedObjectStoreDefinition(io.cdap.cdap.api.dataset.lib.IndexedObjectStoreDefinition) TimeseriesTableDefinition(io.cdap.cdap.api.dataset.lib.TimeseriesTableDefinition) CounterTimeseriesTableDefinition(io.cdap.cdap.api.dataset.lib.CounterTimeseriesTableDefinition)

Example 2 with ObjectStore

use of io.cdap.cdap.api.dataset.lib.ObjectStore in project cdap by caskdata.

the class Spark2Test method testScalaSparkWithObjectStore.

@Test
public void testScalaSparkWithObjectStore() throws Exception {
    ApplicationManager applicationManager = deploy(NamespaceId.DEFAULT, SparkAppUsingObjectStore.class);
    DataSetManager<ObjectStore<String>> keysManager = getDataset("keys");
    prepareInputData(keysManager);
    SparkManager sparkManager = applicationManager.getSparkManager(ScalaCharCountProgram.class.getSimpleName()).start();
    sparkManager.waitForRun(ProgramRunStatus.RUNNING, 10, TimeUnit.SECONDS);
    sparkManager.waitForStopped(60, TimeUnit.SECONDS);
    DataSetManager<KeyValueTable> countManager = getDataset("count");
    checkOutputData(countManager);
}
Also used : ApplicationManager(io.cdap.cdap.test.ApplicationManager) ObjectStore(io.cdap.cdap.api.dataset.lib.ObjectStore) SparkAppUsingObjectStore(io.cdap.cdap.spark.app.SparkAppUsingObjectStore) SparkManager(io.cdap.cdap.test.SparkManager) KeyValueTable(io.cdap.cdap.api.dataset.lib.KeyValueTable) Test(org.junit.Test)

Example 3 with ObjectStore

use of io.cdap.cdap.api.dataset.lib.ObjectStore in project cdap by caskdata.

the class Spark2Test method testSparkWithObjectStore.

@Test
public void testSparkWithObjectStore() throws Exception {
    ApplicationManager applicationManager = deploy(NamespaceId.DEFAULT, SparkAppUsingObjectStore.class);
    DataSetManager<ObjectStore<String>> keysManager = getDataset("keys");
    prepareInputData(keysManager);
    SparkManager sparkManager = applicationManager.getSparkManager(CharCountProgram.class.getSimpleName()).start();
    sparkManager.waitForRun(ProgramRunStatus.RUNNING, 10, TimeUnit.SECONDS);
    sparkManager.waitForStopped(60, TimeUnit.SECONDS);
    DataSetManager<KeyValueTable> countManager = getDataset("count");
    checkOutputData(countManager);
    // validate that the table emitted metrics
    // one read + one write in beforeSubmit(), increment (= read + write) in main -> 4
    Tasks.waitFor(4L, new Callable<Long>() {

        @Override
        public Long call() throws Exception {
            Collection<MetricTimeSeries> metrics = getMetricsManager().query(new MetricDataQuery(0, System.currentTimeMillis() / 1000L, Integer.MAX_VALUE, "system." + Constants.Metrics.Name.Dataset.OP_COUNT, AggregationFunction.SUM, ImmutableMap.of(Constants.Metrics.Tag.NAMESPACE, DefaultId.NAMESPACE.getNamespace(), Constants.Metrics.Tag.APP, SparkAppUsingObjectStore.class.getSimpleName(), Constants.Metrics.Tag.SPARK, CharCountProgram.class.getSimpleName(), Constants.Metrics.Tag.DATASET, "totals"), Collections.<String>emptyList()));
            if (metrics.isEmpty()) {
                return 0L;
            }
            Assert.assertEquals(1, metrics.size());
            MetricTimeSeries ts = metrics.iterator().next();
            Assert.assertEquals(1, ts.getTimeValues().size());
            return ts.getTimeValues().get(0).getValue();
        }
    }, 10L, TimeUnit.SECONDS, 50L, TimeUnit.MILLISECONDS);
}
Also used : ApplicationManager(io.cdap.cdap.test.ApplicationManager) ObjectStore(io.cdap.cdap.api.dataset.lib.ObjectStore) SparkAppUsingObjectStore(io.cdap.cdap.spark.app.SparkAppUsingObjectStore) SparkManager(io.cdap.cdap.test.SparkManager) MetricTimeSeries(io.cdap.cdap.api.metrics.MetricTimeSeries) Assume.assumeNoException(org.junit.Assume.assumeNoException) IOException(java.io.IOException) KeyValueTable(io.cdap.cdap.api.dataset.lib.KeyValueTable) Collection(java.util.Collection) MetricDataQuery(io.cdap.cdap.api.metrics.MetricDataQuery) Test(org.junit.Test)

Example 4 with ObjectStore

use of io.cdap.cdap.api.dataset.lib.ObjectStore in project cdap by caskdata.

the class Spark2Test method testScalaSparkCrossNSDataset.

@Test
public void testScalaSparkCrossNSDataset() throws Exception {
    // Deploy and create a dataset in namespace datasetSpaceForSpark
    NamespaceMeta inputDSNSMeta = new NamespaceMeta.Builder().setName("datasetSpaceForSpark").build();
    getNamespaceAdmin().create(inputDSNSMeta);
    deploy(inputDSNSMeta.getNamespaceId(), SparkAppUsingObjectStore.class);
    DataSetManager<ObjectStore<String>> keysManager = getDataset(inputDSNSMeta.getNamespaceId().dataset("keys"));
    prepareInputData(keysManager);
    Map<String, String> args = ImmutableMap.of(ScalaCharCountProgram.INPUT_DATASET_NAMESPACE(), inputDSNSMeta.getNamespaceId().getNamespace(), ScalaCharCountProgram.INPUT_DATASET_NAME(), "keys");
    ApplicationManager applicationManager = deploy(NamespaceId.DEFAULT, SparkAppUsingObjectStore.class);
    SparkManager sparkManager = applicationManager.getSparkManager(ScalaCharCountProgram.class.getSimpleName()).start(args);
    sparkManager.waitForRun(ProgramRunStatus.RUNNING, 10, TimeUnit.SECONDS);
    sparkManager.waitForStopped(60, TimeUnit.SECONDS);
    DataSetManager<KeyValueTable> countManager = getDataset("count");
    checkOutputData(countManager);
}
Also used : ObjectStore(io.cdap.cdap.api.dataset.lib.ObjectStore) SparkAppUsingObjectStore(io.cdap.cdap.spark.app.SparkAppUsingObjectStore) ApplicationManager(io.cdap.cdap.test.ApplicationManager) SparkManager(io.cdap.cdap.test.SparkManager) NamespaceMeta(io.cdap.cdap.proto.NamespaceMeta) KeyValueTable(io.cdap.cdap.api.dataset.lib.KeyValueTable) Test(org.junit.Test)

Aggregations

KeyValueTable (io.cdap.cdap.api.dataset.lib.KeyValueTable)4 ObjectStore (io.cdap.cdap.api.dataset.lib.ObjectStore)4 SparkAppUsingObjectStore (io.cdap.cdap.spark.app.SparkAppUsingObjectStore)3 ApplicationManager (io.cdap.cdap.test.ApplicationManager)3 SparkManager (io.cdap.cdap.test.SparkManager)3 Test (org.junit.Test)3 DatasetAdmin (io.cdap.cdap.api.dataset.DatasetAdmin)1 CounterTimeseriesTable (io.cdap.cdap.api.dataset.lib.CounterTimeseriesTable)1 CounterTimeseriesTableDefinition (io.cdap.cdap.api.dataset.lib.CounterTimeseriesTableDefinition)1 IndexedObjectStore (io.cdap.cdap.api.dataset.lib.IndexedObjectStore)1 IndexedObjectStoreDefinition (io.cdap.cdap.api.dataset.lib.IndexedObjectStoreDefinition)1 IndexedTable (io.cdap.cdap.api.dataset.lib.IndexedTable)1 IndexedTableDefinition (io.cdap.cdap.api.dataset.lib.IndexedTableDefinition)1 KeyValueTableDefinition (io.cdap.cdap.api.dataset.lib.KeyValueTableDefinition)1 TimeseriesTable (io.cdap.cdap.api.dataset.lib.TimeseriesTable)1 TimeseriesTableDefinition (io.cdap.cdap.api.dataset.lib.TimeseriesTableDefinition)1 Table (io.cdap.cdap.api.dataset.table.Table)1 MetricDataQuery (io.cdap.cdap.api.metrics.MetricDataQuery)1 MetricTimeSeries (io.cdap.cdap.api.metrics.MetricTimeSeries)1 InMemoryTable (io.cdap.cdap.data2.dataset2.lib.table.inmemory.InMemoryTable)1