Search in sources :

Example 1 with DatasetProperties

use of org.talend.components.common.dataset.DatasetProperties in project components by Talend.

the class RuntimeControllerImpl method useDatasetRuntime.

private <// 
T> // 
T useDatasetRuntime(// 
final DatasetDefinition<DatasetProperties<DatastoreProperties>> datasetDefinition, // 
DatasetProperties datasetProperties, Function<DatasetRuntime<DatasetProperties<DatastoreProperties>>, T> consumer) {
    try (SandboxedInstance instance = RuntimeUtil.createRuntimeClass(datasetDefinition.getRuntimeInfo(datasetProperties), datasetProperties.getClass().getClassLoader())) {
        DatasetRuntime<DatasetProperties<DatastoreProperties>> datasetRuntimeInstance = (DatasetRuntime<DatasetProperties<DatastoreProperties>>) instance.getInstance();
        datasetRuntimeInstance.initialize(null, datasetProperties);
        return consumer.apply(datasetRuntimeInstance);
    }
}
Also used : SandboxedInstance(org.talend.daikon.sandbox.SandboxedInstance) DatastoreProperties(org.talend.components.common.datastore.DatastoreProperties) DatasetRuntime(org.talend.components.common.dataset.runtime.DatasetRuntime) DatasetProperties(org.talend.components.common.dataset.DatasetProperties)

Example 2 with DatasetProperties

use of org.talend.components.common.dataset.DatasetProperties in project components by Talend.

the class RuntimeControllerImpl method writeData.

@Override
public void writeData(InputStream rawPayload) throws IOException {
    // 1) Read payload (with data as a stream of course)
    DatasetWritePayload payload = DatasetWritePayload.readData(rawPayload, mapper);
    String definitionName = payload.getConfiguration().getDefinitionName();
    // 2) Create properties
    Properties properties = propertiesHelpers.propertiesFromDto(payload.getConfiguration());
    if (properties instanceof ComponentProperties) {
        ComponentProperties componentProperties = (ComponentProperties) properties;
        // 3) Retrieve component definition to be able to create the runtime
        final ComponentDefinition definition = propertiesHelpers.getDefinition(ComponentDefinition.class, definitionName);
        // 4) Get the execution engine
        ExecutionEngine executionEngine;
        if (definition.isSupportingExecutionEngines(DI)) {
            executionEngine = DI;
            // 5) Create the sandbox
            try (SandboxedInstance instance = RuntimeUtil.createRuntimeClass(definition.getRuntimeInfo(executionEngine, componentProperties, INCOMING), definition.getClass().getClassLoader())) {
                Sink datasetRuntimeInstance = (Sink) instance.getInstance();
                datasetRuntimeInstance.initialize(null, componentProperties);
                Iterator<IndexedRecord> data = payload.getData();
                WriteOperation writeOperation = datasetRuntimeInstance.createWriteOperation();
                // Supplier return null to signify end of data stream => see WriterDataSupplier.writeData
                WriterDataSupplier<?, IndexedRecord> stringWriterDataSupplier = new WriterDataSupplier<Object, IndexedRecord>(writeOperation, () -> data.hasNext() ? data.next() : null, null);
                stringWriterDataSupplier.writeData();
            }
        } else if (definition.isSupportingExecutionEngines(BEAM)) {
            throw new UnsupportedOperationException("Beam runtime is not available for dataset write through HTTP API.");
        } else {
            throw new TalendRuntimeException(CommonErrorCodes.UNREGISTERED_DEFINITION);
        }
    } else if (properties instanceof DatasetProperties) {
        throw new UnsupportedOperationException("HTTP API is only able to write using component implementations. Not " + properties.getClass());
    }
}
Also used : ComponentProperties(org.talend.components.api.properties.ComponentProperties) IndexedRecord(org.apache.avro.generic.IndexedRecord) DatasetProperties(org.talend.components.common.dataset.DatasetProperties) DatasetProperties(org.talend.components.common.dataset.DatasetProperties) DatastoreProperties(org.talend.components.common.datastore.DatastoreProperties) ComponentProperties(org.talend.components.api.properties.ComponentProperties) Properties(org.talend.daikon.properties.Properties) SandboxedInstance(org.talend.daikon.sandbox.SandboxedInstance) TalendRuntimeException(org.talend.daikon.exception.TalendRuntimeException) ExecutionEngine(org.talend.components.api.component.runtime.ExecutionEngine) Sink(org.talend.components.api.component.runtime.Sink) WriteOperation(org.talend.components.api.component.runtime.WriteOperation) WriterDataSupplier(org.talend.components.api.component.runtime.WriterDataSupplier) ComponentDefinition(org.talend.components.api.component.ComponentDefinition)

Example 3 with DatasetProperties

use of org.talend.components.common.dataset.DatasetProperties in project components by Talend.

the class GoogleDriveDatastoreDefinitionTest method testCreateDatasetProperties.

@Test
public void testCreateDatasetProperties() throws Exception {
    GoogleDriveDatastoreProperties dsp = new GoogleDriveDatastoreProperties("test");
    dsp.setupProperties();
    dsp.setupLayout();
    DatasetProperties ds = def.createDatasetProperties(dsp);
    assertNotNull(ds);
    assertEquals(dsp, ds.getDatastoreProperties());
}
Also used : DatasetProperties(org.talend.components.common.dataset.DatasetProperties) Test(org.junit.Test)

Example 4 with DatasetProperties

use of org.talend.components.common.dataset.DatasetProperties in project components by Talend.

the class JDBCDatastoreDefinitionTest method testCreateDatasetProperties.

/**
 * Run the DatasetProperties createDatasetProperties(JDBCDatastoreProperties) method test.
 *
 * @throws Exception
 *
 * @generatedBy CodePro at 17-6-23 PM2:14
 */
@Test
public void testCreateDatasetProperties() throws Exception {
    JDBCDatastoreDefinition fixture = new JDBCDatastoreDefinition();
    JDBCDatastoreProperties storeProp = new JDBCDatastoreProperties("datastore");
    DatasetProperties result = fixture.createDatasetProperties(storeProp);
    assertNotNull(result);
    assertEquals("properties.dataset.displayName", result.getDisplayName());
    assertEquals("dataset", result.getName());
    assertEquals("dataset", result.getTitle());
}
Also used : DatasetProperties(org.talend.components.common.dataset.DatasetProperties) Test(org.junit.Test)

Example 5 with DatasetProperties

use of org.talend.components.common.dataset.DatasetProperties in project components by Talend.

the class KafkaDatastoreDefinitionTest method createDatasetProperties.

@Test
public void createDatasetProperties() throws Exception {
    KafkaDatastoreProperties properties = new KafkaDatastoreProperties("");
    DatasetProperties datasetProperties = definition.createDatasetProperties(properties);
    assertNotNull(datasetProperties);
    assertEquals(properties, datasetProperties.getDatastoreProperties());
}
Also used : DatasetProperties(org.talend.components.common.dataset.DatasetProperties) Test(org.junit.Test)

Aggregations

DatasetProperties (org.talend.components.common.dataset.DatasetProperties)6 Test (org.junit.Test)4 DatastoreProperties (org.talend.components.common.datastore.DatastoreProperties)2 SandboxedInstance (org.talend.daikon.sandbox.SandboxedInstance)2 IndexedRecord (org.apache.avro.generic.IndexedRecord)1 ComponentDefinition (org.talend.components.api.component.ComponentDefinition)1 ExecutionEngine (org.talend.components.api.component.runtime.ExecutionEngine)1 Sink (org.talend.components.api.component.runtime.Sink)1 WriteOperation (org.talend.components.api.component.runtime.WriteOperation)1 WriterDataSupplier (org.talend.components.api.component.runtime.WriterDataSupplier)1 ComponentProperties (org.talend.components.api.properties.ComponentProperties)1 DatasetRuntime (org.talend.components.common.dataset.runtime.DatasetRuntime)1 SalesforceTestBase (org.talend.components.salesforce.SalesforceTestBase)1 SalesforceInputProperties (org.talend.components.salesforce.dataprep.SalesforceInputProperties)1 TalendRuntimeException (org.talend.daikon.exception.TalendRuntimeException)1 Properties (org.talend.daikon.properties.Properties)1 Form (org.talend.daikon.properties.presentation.Form)1