Search in sources :

Example 1 with TypeRepresentation

use of io.cdap.cdap.internal.io.TypeRepresentation in project cdap by caskdata.

the class ObjectStoreDefinition method getDataset.

@Override
public ObjectStoreDataset<?> getDataset(DatasetContext datasetContext, DatasetSpecification spec, Map<String, String> arguments, ClassLoader classLoader) throws IOException {
    DatasetSpecification kvTableSpec = spec.getSpecification("objects");
    KeyValueTable table = tableDef.getDataset(datasetContext, kvTableSpec, arguments, classLoader);
    TypeRepresentation typeRep = GSON.fromJson(spec.getProperty("type"), TypeRepresentation.class);
    Schema schema = GSON.fromJson(spec.getProperty("schema"), Schema.class);
    return new ObjectStoreDataset(spec.getName(), table, typeRep, schema, classLoader);
}
Also used : KeyValueTable(io.cdap.cdap.api.dataset.lib.KeyValueTable) TypeRepresentation(io.cdap.cdap.internal.io.TypeRepresentation) Schema(io.cdap.cdap.api.data.schema.Schema) DatasetSpecification(io.cdap.cdap.api.dataset.DatasetSpecification)

Example 2 with TypeRepresentation

use of io.cdap.cdap.internal.io.TypeRepresentation in project cdap by caskdata.

the class RecordWithString method testEmptyValue.

// this tests that the datum reader treats empty fields correctly. It reproduces the issue in ENG-2404.
@Test
public void testEmptyValue() throws UnsupportedTypeException, IOException {
    Schema schema = new ReflectionSchemaGenerator().generate(RecordWithString.class);
    TypeRepresentation typeRep = new TypeRepresentation(RecordWithString.class);
    DatumWriter<RecordWithString> datumWriter = new ReflectionDatumWriter<>(schema);
    @SuppressWarnings("unchecked") ReflectionDatumReader<RecordWithString> datumReader = new ReflectionDatumReader<>(schema, (TypeToken<RecordWithString>) TypeToken.of(typeRep.toType()));
    RecordWithString record = new RecordWithString();
    record.setA(42);
    record.setTheString("");
    ByteArrayOutputStream bos = new ByteArrayOutputStream();
    BinaryEncoder encoder = new BinaryEncoder(bos);
    datumWriter.encode(record, encoder);
    byte[] bytes = bos.toByteArray();
    ByteArrayInputStream bis = new ByteArrayInputStream(bytes);
    BinaryDecoder decoder = new BinaryDecoder(bis);
    RecordWithString rec = datumReader.read(decoder, schema);
    Assert.assertEquals(record.getA(), rec.getA());
    Assert.assertEquals(record.getTheString(), rec.getTheString());
}
Also used : Schema(io.cdap.cdap.api.data.schema.Schema) ReflectionDatumWriter(io.cdap.cdap.internal.io.ReflectionDatumWriter) ReflectionDatumReader(io.cdap.cdap.internal.io.ReflectionDatumReader) ByteArrayOutputStream(java.io.ByteArrayOutputStream) ReflectionSchemaGenerator(io.cdap.cdap.internal.io.ReflectionSchemaGenerator) BinaryDecoder(io.cdap.cdap.common.io.BinaryDecoder) BinaryEncoder(io.cdap.cdap.common.io.BinaryEncoder) ByteArrayInputStream(java.io.ByteArrayInputStream) TypeRepresentation(io.cdap.cdap.internal.io.TypeRepresentation) Test(org.junit.Test)

Example 3 with TypeRepresentation

use of io.cdap.cdap.internal.io.TypeRepresentation in project cdap by caskdata.

the class ObjectStoreDatasetTest method testWithCustomClassLoader.

@Test
public void testWithCustomClassLoader() throws Exception {
    DatasetId kv = DatasetFrameworkTestUtil.NAMESPACE_ID.dataset("kv");
    // create a dummy class loader that records the name of the class it loaded
    final AtomicReference<String> lastClassLoaded = new AtomicReference<>(null);
    ClassLoader loader = new ClassLoader() {

        @Override
        public Class<?> loadClass(String name) throws ClassNotFoundException {
            lastClassLoaded.set(name);
            return super.loadClass(name);
        }
    };
    dsFrameworkUtil.createInstance("keyValueTable", kv, DatasetProperties.EMPTY);
    KeyValueTable kvTable = dsFrameworkUtil.getInstance(kv);
    Type type = Custom.class;
    TypeRepresentation typeRep = new TypeRepresentation(type);
    Schema schema = new ReflectionSchemaGenerator().generate(type);
    final ObjectStoreDataset<Custom> objectStore = new ObjectStoreDataset<>("kv", kvTable, typeRep, schema, loader);
    TransactionExecutor txnl = dsFrameworkUtil.newInMemoryTransactionExecutor(objectStore);
    // need to call this to actually load the Custom class, because the Custom class is no longer used in the
    // ObjectStoreDataset's constructor, but rather lazily when its actually needed.
    objectStore.getRecordType();
    txnl.execute(new TransactionExecutor.Subroutine() {

        @Override
        public void apply() throws Exception {
            objectStore.write("dummy", new Custom(382, Lists.newArrayList("blah")));
        }
    });
    // verify the class name was recorded (the dummy class loader was used).
    txnl.execute(new TransactionExecutor.Subroutine() {

        @Override
        public void apply() throws Exception {
            Assert.assertEquals(Custom.class.getName(), lastClassLoaded.get());
        }
    });
    txnl.execute(new TransactionExecutor.Subroutine() {

        @Override
        public void apply() throws Exception {
            deleteAndVerify(objectStore, Bytes.toBytes("dummy"));
        }
    });
    dsFrameworkUtil.deleteInstance(kv);
}
Also used : Schema(io.cdap.cdap.api.data.schema.Schema) AtomicReference(java.util.concurrent.atomic.AtomicReference) TransactionExecutor(org.apache.tephra.TransactionExecutor) ReflectionSchemaGenerator(io.cdap.cdap.internal.io.ReflectionSchemaGenerator) TransactionFailureException(org.apache.tephra.TransactionFailureException) NoSuchElementException(java.util.NoSuchElementException) DatasetId(io.cdap.cdap.proto.id.DatasetId) Type(java.lang.reflect.Type) KeyValueTable(io.cdap.cdap.api.dataset.lib.KeyValueTable) TypeRepresentation(io.cdap.cdap.internal.io.TypeRepresentation) Test(org.junit.Test)

Example 4 with TypeRepresentation

use of io.cdap.cdap.internal.io.TypeRepresentation in project cdap by caskdata.

the class ObjectStores method objectStoreProperties.

/**
 * Creates properties for {@link ObjectStore} dataset instance.
 *
 * @param type type of objects to be stored in dataset
 * @return {@link DatasetProperties} for the dataset
 * @throws UnsupportedTypeException
 */
public static DatasetProperties objectStoreProperties(Type type, DatasetProperties props) throws UnsupportedTypeException {
    Schema schema = new ReflectionSchemaGenerator().generate(type);
    TypeRepresentation typeRep = new TypeRepresentation(type);
    return DatasetProperties.builder().add("schema", schema.toString()).add("type", new Gson().toJson(typeRep)).addAll(props.getProperties()).build();
}
Also used : TypeRepresentation(io.cdap.cdap.internal.io.TypeRepresentation) Schema(io.cdap.cdap.api.data.schema.Schema) Gson(com.google.gson.Gson) ReflectionSchemaGenerator(io.cdap.cdap.internal.io.ReflectionSchemaGenerator)

Example 5 with TypeRepresentation

use of io.cdap.cdap.internal.io.TypeRepresentation in project cdap by caskdata.

the class ObjectMappedTableDefinition method getDataset.

@Override
public ObjectMappedTableDataset<?> getDataset(DatasetContext datasetContext, DatasetSpecification spec, Map<String, String> arguments, ClassLoader classLoader) throws IOException {
    String keyName = ObjectMappedTableProperties.getRowKeyExploreName(spec.getProperties());
    DatasetSpecification tableSpec = spec.getSpecification(TABLE_NAME);
    // TODO: remove after CDAP-2122 is done
    if (!tableSpec.getProperties().containsKey(Table.PROPERTY_SCHEMA)) {
        tableSpec = DatasetSpecification.builder(tableSpec.getName(), tableSpec.getType()).properties(tableSpec.getProperties()).property(Table.PROPERTY_SCHEMA, spec.getProperty(Table.PROPERTY_SCHEMA)).property(Table.PROPERTY_SCHEMA_ROW_FIELD, keyName).datasets(tableSpec.getSpecifications().values()).build();
    }
    // reconstruct the table schema here because of backwards compatibility
    DatasetDefinition<Table, DatasetAdmin> tableDef = getDelegate(TABLE_NAME);
    Table table = tableDef.getDataset(datasetContext, tableSpec, arguments, classLoader);
    Map<String, String> properties = spec.getProperties();
    TypeRepresentation typeRep = GSON.fromJson(ObjectMappedTableProperties.getObjectTypeRepresentation(properties), TypeRepresentation.class);
    Schema objSchema = ObjectMappedTableProperties.getObjectSchema(properties);
    return new ObjectMappedTableDataset(spec.getName(), table, typeRep, objSchema, classLoader);
}
Also used : ObjectMappedTable(io.cdap.cdap.api.dataset.lib.ObjectMappedTable) Table(io.cdap.cdap.api.dataset.table.Table) TypeRepresentation(io.cdap.cdap.internal.io.TypeRepresentation) Schema(io.cdap.cdap.api.data.schema.Schema) DatasetSpecification(io.cdap.cdap.api.dataset.DatasetSpecification) DatasetAdmin(io.cdap.cdap.api.dataset.DatasetAdmin)

Aggregations

Schema (io.cdap.cdap.api.data.schema.Schema)5 TypeRepresentation (io.cdap.cdap.internal.io.TypeRepresentation)5 ReflectionSchemaGenerator (io.cdap.cdap.internal.io.ReflectionSchemaGenerator)3 DatasetSpecification (io.cdap.cdap.api.dataset.DatasetSpecification)2 KeyValueTable (io.cdap.cdap.api.dataset.lib.KeyValueTable)2 Test (org.junit.Test)2 Gson (com.google.gson.Gson)1 DatasetAdmin (io.cdap.cdap.api.dataset.DatasetAdmin)1 ObjectMappedTable (io.cdap.cdap.api.dataset.lib.ObjectMappedTable)1 Table (io.cdap.cdap.api.dataset.table.Table)1 BinaryDecoder (io.cdap.cdap.common.io.BinaryDecoder)1 BinaryEncoder (io.cdap.cdap.common.io.BinaryEncoder)1 ReflectionDatumReader (io.cdap.cdap.internal.io.ReflectionDatumReader)1 ReflectionDatumWriter (io.cdap.cdap.internal.io.ReflectionDatumWriter)1 DatasetId (io.cdap.cdap.proto.id.DatasetId)1 ByteArrayInputStream (java.io.ByteArrayInputStream)1 ByteArrayOutputStream (java.io.ByteArrayOutputStream)1 Type (java.lang.reflect.Type)1 NoSuchElementException (java.util.NoSuchElementException)1 AtomicReference (java.util.concurrent.atomic.AtomicReference)1