Search in sources :

Example 11 with ExportParameters

use of org.talend.dataprep.api.export.ExportParameters in project data-prep by Talend.

the class CachedExportStrategy method execute.

@Override
public StreamingResponseBody execute(ExportParameters parameters) {
    final TransformationCacheKey contentKey = getCacheKey(parameters);
    // 
    ExportUtils.setExportHeaders(// 
    parameters.getExportName(), // 
    parameters.getArguments().get(ExportFormat.PREFIX + CSVFormat.ParametersCSV.ENCODING), getFormat(parameters.getExportType()));
    return outputStream -> {
        try (InputStream cachedContent = contentCache.get(contentKey)) {
            IOUtils.copy(cachedContent, outputStream);
        }
    };
}
Also used : TransformationCacheKey(org.talend.dataprep.cache.TransformationCacheKey) ExportFormat(org.talend.dataprep.format.export.ExportFormat) StringUtils(org.apache.commons.lang.StringUtils) ExportParameters(org.talend.dataprep.api.export.ExportParameters) TDPException(org.talend.dataprep.exception.TDPException) BaseExportStrategy(org.talend.dataprep.transformation.service.BaseExportStrategy) IOUtils(org.apache.poi.util.IOUtils) Autowired(org.springframework.beans.factory.annotation.Autowired) StreamingResponseBody(org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBody) PreparationErrorCodes(org.talend.dataprep.exception.error.PreparationErrorCodes) PreparationMessage(org.talend.dataprep.api.preparation.PreparationMessage) CacheKeyGenerator(org.talend.dataprep.cache.CacheKeyGenerator) CSVFormat(org.talend.dataprep.transformation.format.CSVFormat) Component(org.springframework.stereotype.Component) TransformationCacheKey(org.talend.dataprep.cache.TransformationCacheKey) ExportUtils(org.talend.dataprep.transformation.service.ExportUtils) InputStream(java.io.InputStream) InputStream(java.io.InputStream)

Example 12 with ExportParameters

use of org.talend.dataprep.api.export.ExportParameters in project data-prep by Talend.

the class DataSetExportStrategy method execute.

@Override
public StreamingResponseBody execute(ExportParameters parameters) {
    final String formatName = parameters.getExportType();
    final ExportFormat format = getFormat(formatName);
    // 
    ExportUtils.setExportHeaders(// 
    parameters.getExportName(), // 
    parameters.getArguments().get(ExportFormat.PREFIX + CSVFormat.ParametersCSV.ENCODING), format);
    return outputStream -> {
        // get the dataset content (in an auto-closable block to make sure it is properly closed)
        final String datasetId = parameters.getDatasetId();
        final DataSetGet dataSetGet = applicationContext.getBean(DataSetGet.class, datasetId, false, true);
        final DataSetGetMetadata dataSetGetMetadata = applicationContext.getBean(DataSetGetMetadata.class, datasetId);
        try (InputStream datasetContent = dataSetGet.execute()) {
            try (JsonParser parser = mapper.getFactory().createParser(new InputStreamReader(datasetContent, UTF_8))) {
                // Create dataset
                final DataSet dataSet = mapper.readerFor(DataSet.class).readValue(parser);
                dataSet.setMetadata(dataSetGetMetadata.execute());
                // get the actions to apply (no preparation ==> dataset export ==> no actions)
                Configuration configuration = // 
                Configuration.builder().args(// 
                parameters.getArguments()).outFilter(// 
                rm -> filterService.build(parameters.getFilter(), rm)).format(// 
                format.getName()).volume(// 
                Configuration.Volume.SMALL).output(// 
                outputStream).limit(// 
                limit).build();
                factory.get(configuration).buildExecutable(dataSet, configuration).execute();
            }
        } catch (TDPException e) {
            throw e;
        } catch (Exception e) {
            throw new TDPException(TransformationErrorCodes.UNABLE_TO_TRANSFORM_DATASET, e);
        }
    };
}
Also used : ExportFormat(org.talend.dataprep.format.export.ExportFormat) StringUtils(org.apache.commons.lang.StringUtils) ExportParameters(org.talend.dataprep.api.export.ExportParameters) DataSetGet(org.talend.dataprep.command.dataset.DataSetGet) TDPException(org.talend.dataprep.exception.TDPException) TransformationErrorCodes(org.talend.dataprep.exception.error.TransformationErrorCodes) BaseExportStrategy(org.talend.dataprep.transformation.service.BaseExportStrategy) JsonParser(com.fasterxml.jackson.core.JsonParser) UTF_8(java.nio.charset.StandardCharsets.UTF_8) StreamingResponseBody(org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBody) Configuration(org.talend.dataprep.transformation.api.transformer.configuration.Configuration) InputStreamReader(java.io.InputStreamReader) CSVFormat(org.talend.dataprep.transformation.format.CSVFormat) Component(org.springframework.stereotype.Component) DataSetGetMetadata(org.talend.dataprep.command.dataset.DataSetGetMetadata) ExportUtils(org.talend.dataprep.transformation.service.ExportUtils) DataSet(org.talend.dataprep.api.dataset.DataSet) InputStream(java.io.InputStream) TDPException(org.talend.dataprep.exception.TDPException) DataSetGet(org.talend.dataprep.command.dataset.DataSetGet) InputStreamReader(java.io.InputStreamReader) Configuration(org.talend.dataprep.transformation.api.transformer.configuration.Configuration) DataSet(org.talend.dataprep.api.dataset.DataSet) InputStream(java.io.InputStream) ExportFormat(org.talend.dataprep.format.export.ExportFormat) DataSetGetMetadata(org.talend.dataprep.command.dataset.DataSetGetMetadata) TDPException(org.talend.dataprep.exception.TDPException) JsonParser(com.fasterxml.jackson.core.JsonParser)

Example 13 with ExportParameters

use of org.talend.dataprep.api.export.ExportParameters in project data-prep by Talend.

the class ApplyPreparationExportStrategyTest method shouldNotAcceptIfPreparationParameterNotSet.

@Test
public void shouldNotAcceptIfPreparationParameterNotSet() throws Exception {
    // Given
    final ExportParameters parameters = new ExportParameters();
    parameters.setDatasetId("1234");
    parameters.setPreparationId("");
    parameters.setStepId("0");
    parameters.setExportType("text");
    parameters.setFrom(ExportParameters.SourceType.HEAD);
    // Then
    assertFalse(applyPreparationExportStrategy.accept(parameters));
}
Also used : ExportParameters(org.talend.dataprep.api.export.ExportParameters) ServiceBaseTest(org.talend.ServiceBaseTest) Test(org.junit.Test)

Example 14 with ExportParameters

use of org.talend.dataprep.api.export.ExportParameters in project data-prep by Talend.

the class CachedExportStrategyTest method shouldNotAcceptNullParameterContent.

@Test
public void shouldNotAcceptNullParameterContent() throws Exception {
    // Then
    final ExportParameters parameters = new ExportParameters();
    parameters.setContent(null);
    assertFalse(cachedExportStrategy.accept(parameters));
}
Also used : ExportParameters(org.talend.dataprep.api.export.ExportParameters) Test(org.junit.Test) ServiceBaseTest(org.talend.ServiceBaseTest)

Example 15 with ExportParameters

use of org.talend.dataprep.api.export.ExportParameters in project data-prep by Talend.

the class OptimizedExportStrategyTest method testAcceptOK.

@Test
public void testAcceptOK() throws Exception {
    // Given
    final String datasetId = "1234";
    final String format = "";
    final String preparation = createEmptyPreparationFromDataset(datasetId, "test");
    applyAction(preparation, "[{}]");
    applyAction(preparation, "[{}]");
    final Preparation preparationDetails = getPreparation(preparation);
    for (Step step : preparationDetails.getSteps()) {
        try (OutputStream content = contentCache.put(cacheKeyGenerator.generateMetadataKey(preparation, step.id(), HEAD), ContentCache.TimeToLive.DEFAULT)) {
            content.write("{}".getBytes());
            content.flush();
        }
        final TransformationCacheKey key = // 
        cacheKeyGenerator.generateContentKey(// 
        datasetId, // 
        preparation, // 
        step.id(), // 
        format, // 
        HEAD, // 
        "");
        try (OutputStream content = contentCache.put(key, ContentCache.TimeToLive.DEFAULT)) {
            content.write("{}".getBytes());
            content.flush();
        }
    }
    ExportParameters exportParameters = new ExportParameters();
    exportParameters.setPreparationId(preparation);
    exportParameters.setDatasetId(datasetId);
    exportParameters.setExportType(format);
    exportParameters.setFrom(HEAD);
    // Then
    assertTrue(optimizedExportStrategy.accept(exportParameters));
}
Also used : TransformationCacheKey(org.talend.dataprep.cache.TransformationCacheKey) Preparation(org.talend.dataprep.api.preparation.Preparation) ExportParameters(org.talend.dataprep.api.export.ExportParameters) OutputStream(java.io.OutputStream) Step(org.talend.dataprep.api.preparation.Step) Test(org.junit.Test) TransformationServiceBaseTest(org.talend.dataprep.transformation.service.TransformationServiceBaseTest)

Aggregations

ExportParameters (org.talend.dataprep.api.export.ExportParameters)40 Test (org.junit.Test)23 StreamingResponseBody (org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBody)11 TDPException (org.talend.dataprep.exception.TDPException)11 Preparation (org.talend.dataprep.api.preparation.Preparation)10 DataSet (org.talend.dataprep.api.dataset.DataSet)9 TransformationCacheKey (org.talend.dataprep.cache.TransformationCacheKey)9 OutputStream (java.io.OutputStream)8 Configuration (org.talend.dataprep.transformation.api.transformer.configuration.Configuration)8 TransformationServiceBaseTest (org.talend.dataprep.transformation.service.TransformationServiceBaseTest)8 JsonParser (com.fasterxml.jackson.core.JsonParser)7 InputStream (java.io.InputStream)7 StringUtils (org.apache.commons.lang.StringUtils)7 Component (org.springframework.stereotype.Component)7 ServiceBaseTest (org.talend.ServiceBaseTest)7 ExportFormat (org.talend.dataprep.format.export.ExportFormat)7 CSVFormat (org.talend.dataprep.transformation.format.CSVFormat)7 BaseExportStrategy (org.talend.dataprep.transformation.service.BaseExportStrategy)7 ExportUtils (org.talend.dataprep.transformation.service.ExportUtils)7 InputStreamReader (java.io.InputStreamReader)6