Search in sources :

Example 6 with TransformationCacheKey

use of org.talend.dataprep.cache.TransformationCacheKey in project data-prep by Talend.

the class TransformationServiceTest method testCache.

@Test
public void testCache() throws Exception {
    // given
    String dsId = createDataset("input_dataset.csv", "uppercase", "text/csv");
    String prepId = createEmptyPreparationFromDataset(dsId, "uppercase prep");
    applyActionFromFile(prepId, "uppercase_action.json");
    final Preparation preparation = getPreparation(prepId);
    final String headId = preparation.getHeadId();
    final TransformationCacheKey key = // 
    cacheKeyGenerator.generateContentKey(// 
    dsId, // 
    preparation.getId(), // 
    headId, // 
    JSON, // 
    HEAD, // no filter
    "");
    assertFalse(contentCache.has(key));
    // when
    // 
    given().expect().statusCode(200).log().ifError().when().get("/apply/preparation/{prepId}/dataset/{datasetId}/{format}", prepId, dsId, // 
    "JSON").asString();
    // then
    assertTrue(contentCache.has(key));
    // just to pass through the cache
    final Response response = // 
    given().expect().statusCode(200).log().ifError().when().get("/apply/preparation/{prepId}/dataset/{datasetId}/{format}", prepId, dsId, "JSON");
    Assert.assertThat(response.getStatusCode(), is(200));
}
Also used : TransformationCacheKey(org.talend.dataprep.cache.TransformationCacheKey) Response(com.jayway.restassured.response.Response) Preparation(org.talend.dataprep.api.preparation.Preparation) Test(org.junit.Test)

Example 7 with TransformationCacheKey

use of org.talend.dataprep.cache.TransformationCacheKey in project data-prep by Talend.

the class OptimizedExportStrategyTest method testAcceptOK.

@Test
public void testAcceptOK() throws Exception {
    // Given
    final String datasetId = "1234";
    final String format = "";
    final String preparation = createEmptyPreparationFromDataset(datasetId, "test");
    applyAction(preparation, "[{}]");
    applyAction(preparation, "[{}]");
    final Preparation preparationDetails = getPreparation(preparation);
    for (Step step : preparationDetails.getSteps()) {
        try (OutputStream content = contentCache.put(cacheKeyGenerator.generateMetadataKey(preparation, step.id(), HEAD), ContentCache.TimeToLive.DEFAULT)) {
            content.write("{}".getBytes());
            content.flush();
        }
        final TransformationCacheKey key = // 
        cacheKeyGenerator.generateContentKey(// 
        datasetId, // 
        preparation, // 
        step.id(), // 
        format, // 
        HEAD, // 
        "");
        try (OutputStream content = contentCache.put(key, ContentCache.TimeToLive.DEFAULT)) {
            content.write("{}".getBytes());
            content.flush();
        }
    }
    ExportParameters exportParameters = new ExportParameters();
    exportParameters.setPreparationId(preparation);
    exportParameters.setDatasetId(datasetId);
    exportParameters.setExportType(format);
    exportParameters.setFrom(HEAD);
    // Then
    assertTrue(optimizedExportStrategy.accept(exportParameters));
}
Also used : TransformationCacheKey(org.talend.dataprep.cache.TransformationCacheKey) Preparation(org.talend.dataprep.api.preparation.Preparation) ExportParameters(org.talend.dataprep.api.export.ExportParameters) OutputStream(java.io.OutputStream) Step(org.talend.dataprep.api.preparation.Step) Test(org.junit.Test) TransformationServiceBaseTest(org.talend.dataprep.transformation.service.TransformationServiceBaseTest)

Example 8 with TransformationCacheKey

use of org.talend.dataprep.cache.TransformationCacheKey in project data-prep by Talend.

the class OptimizedExportStrategy method performOptimizedTransform.

private void performOptimizedTransform(ExportParameters parameters, OutputStream outputStream) throws IOException {
    // Initial check
    final OptimizedPreparationInput optimizedPreparationInput = new OptimizedPreparationInput(parameters).invoke();
    if (optimizedPreparationInput == null) {
        throw new IllegalStateException("Unable to use this strategy (call accept() before calling this).");
    }
    final String preparationId = parameters.getPreparationId();
    final String dataSetId = optimizedPreparationInput.getDataSetId();
    final TransformationCacheKey transformationCacheKey = optimizedPreparationInput.getTransformationCacheKey();
    final DataSetMetadata metadata = optimizedPreparationInput.getMetadata();
    final String previousVersion = optimizedPreparationInput.getPreviousVersion();
    final String version = optimizedPreparationInput.getVersion();
    final ExportFormat format = getFormat(parameters.getExportType());
    // Get content from previous step
    try (JsonParser parser = mapper.getFactory().createParser(new InputStreamReader(contentCache.get(transformationCacheKey), UTF_8))) {
        // Create dataset
        final DataSet dataSet = mapper.readerFor(DataSet.class).readValue(parser);
        dataSet.setMetadata(metadata);
        // get the actions to apply (no preparation ==> dataset export ==> no actions)
        final String actions = getActions(preparationId, previousVersion, version);
        final PreparationMessage preparation = getPreparation(preparationId);
        preparation.setSteps(getMatchingSteps(preparation.getSteps(), previousVersion, version));
        LOGGER.debug("Running optimized strategy for preparation {} @ step #{}", preparationId, version);
        // create tee to broadcast to cache + service output
        final TransformationCacheKey key = // 
        cacheKeyGenerator.generateContentKey(// 
        dataSetId, // 
        preparationId, // 
        version, // 
        parameters.getExportType(), // 
        parameters.getFrom(), // 
        parameters.getArguments(), // 
        parameters.getFilter());
        LOGGER.debug("Cache key: " + key.getKey());
        LOGGER.debug("Cache key details: " + key.toString());
        try (final TeeOutputStream tee = new TeeOutputStream(outputStream, contentCache.put(key, ContentCache.TimeToLive.DEFAULT))) {
            final Configuration configuration = // 
            Configuration.builder().args(// 
            parameters.getArguments()).outFilter(// 
            rm -> filterService.build(parameters.getFilter(), rm)).sourceType(parameters.getFrom()).format(// 
            format.getName()).actions(// 
            actions).preparation(// 
            preparation).stepId(// 
            version).volume(// 
            Configuration.Volume.SMALL).output(// 
            tee).limit(// 
            limit).build();
            factory.get(configuration).buildExecutable(dataSet, configuration).execute();
            tee.flush();
        } catch (Throwable e) {
            // NOSONAR
            contentCache.evict(key);
            throw e;
        }
    } catch (TDPException e) {
        throw e;
    } catch (Exception e) {
        throw new TDPException(TransformationErrorCodes.UNABLE_TO_TRANSFORM_DATASET, e);
    }
}
Also used : ExportFormat(org.talend.dataprep.format.export.ExportFormat) StringUtils(org.apache.commons.lang.StringUtils) TDPException(org.talend.dataprep.exception.TDPException) TransformationErrorCodes(org.talend.dataprep.exception.error.TransformationErrorCodes) LoggerFactory(org.slf4j.LoggerFactory) Autowired(org.springframework.beans.factory.annotation.Autowired) Configuration(org.talend.dataprep.transformation.api.transformer.configuration.Configuration) PreparationMessage(org.talend.dataprep.api.preparation.PreparationMessage) ArrayList(java.util.ArrayList) CacheKeyGenerator(org.talend.dataprep.cache.CacheKeyGenerator) TransformationMetadataCacheKey(org.talend.dataprep.cache.TransformationMetadataCacheKey) DataSetMetadata(org.talend.dataprep.api.dataset.DataSetMetadata) TeeOutputStream(org.apache.commons.io.output.TeeOutputStream) ExportUtils(org.talend.dataprep.transformation.service.ExportUtils) DataSet(org.talend.dataprep.api.dataset.DataSet) OutputStream(java.io.OutputStream) Preparation(org.talend.dataprep.api.preparation.Preparation) ExportParameters(org.talend.dataprep.api.export.ExportParameters) Logger(org.slf4j.Logger) BaseExportStrategy(org.talend.dataprep.transformation.service.BaseExportStrategy) JsonParser(com.fasterxml.jackson.core.JsonParser) UTF_8(java.nio.charset.StandardCharsets.UTF_8) StreamingResponseBody(org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBody) IOException(java.io.IOException) Step(org.talend.dataprep.api.preparation.Step) InputStreamReader(java.io.InputStreamReader) Collectors(java.util.stream.Collectors) ContentCache(org.talend.dataprep.cache.ContentCache) CSVFormat(org.talend.dataprep.transformation.format.CSVFormat) List(java.util.List) Component(org.springframework.stereotype.Component) TransformationCacheKey(org.talend.dataprep.cache.TransformationCacheKey) InputStream(java.io.InputStream) TeeOutputStream(org.apache.commons.io.output.TeeOutputStream) InputStreamReader(java.io.InputStreamReader) Configuration(org.talend.dataprep.transformation.api.transformer.configuration.Configuration) DataSet(org.talend.dataprep.api.dataset.DataSet) ExportFormat(org.talend.dataprep.format.export.ExportFormat) DataSetMetadata(org.talend.dataprep.api.dataset.DataSetMetadata) TDPException(org.talend.dataprep.exception.TDPException) IOException(java.io.IOException) TransformationCacheKey(org.talend.dataprep.cache.TransformationCacheKey) TDPException(org.talend.dataprep.exception.TDPException) PreparationMessage(org.talend.dataprep.api.preparation.PreparationMessage) JsonParser(com.fasterxml.jackson.core.JsonParser)

Example 9 with TransformationCacheKey

use of org.talend.dataprep.cache.TransformationCacheKey in project data-prep by Talend.

the class PreparationExportStrategyTest method setUp.

@Before
public void setUp() throws Exception {
    // Given
    mapper.registerModule(new Jdk8Module());
    strategy.setMapper(new ObjectMapper());
    when(formatRegistrationService.getByName(eq("JSON"))).thenReturn(new JsonFormat());
    final DataSetGetMetadata dataSetGetMetadata = mock(DataSetGetMetadata.class);
    when(applicationContext.getBean(eq(DataSetGetMetadata.class), anyVararg())).thenReturn(dataSetGetMetadata);
    DataSetGet dataSetGet = mock(DataSetGet.class);
    final StringWriter dataSetAsString = new StringWriter();
    DataSet dataSet = new DataSet();
    final DataSetMetadata dataSetMetadata = new DataSetMetadata("ds-1234", "", "", 0L, 0L, new RowMetadata(), "");
    final DataSetContent content = new DataSetContent();
    dataSetMetadata.setContent(content);
    dataSet.setMetadata(dataSetMetadata);
    dataSet.setRecords(Stream.empty());
    mapper.writerFor(DataSet.class).writeValue(dataSetAsString, dataSet);
    when(dataSetGet.execute()).thenReturn(new ByteArrayInputStream(dataSetAsString.toString().getBytes()));
    when(applicationContext.getBean(eq(DataSetGet.class), anyVararg())).thenReturn(dataSetGet);
    final PreparationGetActions preparationGetActions = mock(PreparationGetActions.class);
    when(preparationGetActions.execute()).thenReturn(new ByteArrayInputStream("{}".getBytes()));
    when(applicationContext.getBean(eq(PreparationGetActions.class), eq("prep-1234"), anyString())).thenReturn(preparationGetActions);
    final TransformationCacheKey cacheKey = mock(TransformationCacheKey.class);
    when(cacheKey.getKey()).thenReturn("cache-1234");
    when(cacheKeyGenerator.generateContentKey(anyString(), anyString(), anyString(), anyString(), any(), any(), anyString())).thenReturn(cacheKey);
    final ExecutableTransformer executableTransformer = mock(ExecutableTransformer.class);
    reset(transformer);
    when(transformer.buildExecutable(any(), any())).thenReturn(executableTransformer);
    when(factory.get(any())).thenReturn(transformer);
    when(contentCache.put(any(), any())).thenReturn(new NullOutputStream());
}
Also used : DataSetGet(org.talend.dataprep.command.dataset.DataSetGet) DataSet(org.talend.dataprep.api.dataset.DataSet) DataSetGetMetadata(org.talend.dataprep.command.dataset.DataSetGetMetadata) DataSetMetadata(org.talend.dataprep.api.dataset.DataSetMetadata) TransformationCacheKey(org.talend.dataprep.cache.TransformationCacheKey) Jdk8Module(com.fasterxml.jackson.datatype.jdk8.Jdk8Module) JsonFormat(org.talend.dataprep.transformation.format.JsonFormat) StringWriter(java.io.StringWriter) ByteArrayInputStream(java.io.ByteArrayInputStream) PreparationGetActions(org.talend.dataprep.command.preparation.PreparationGetActions) ExecutableTransformer(org.talend.dataprep.transformation.api.transformer.ExecutableTransformer) RowMetadata(org.talend.dataprep.api.dataset.RowMetadata) DataSetContent(org.talend.dataprep.api.dataset.DataSetContent) ObjectMapper(com.fasterxml.jackson.databind.ObjectMapper) NullOutputStream(org.apache.commons.io.output.NullOutputStream) Before(org.junit.Before)

Example 10 with TransformationCacheKey

use of org.talend.dataprep.cache.TransformationCacheKey in project data-prep by Talend.

the class CachedExportStrategyTest method setUp.

@Before
public void setUp() {
    super.setUp();
    final Preparation preparation = new Preparation("1234", "1.0");
    preparation.setDataSetId("1234");
    preparation.setHeadId("0");
    preparationRepository.add(preparation);
    final TransformationCacheKey cacheKey = cacheKeyGenerator.generateContentKey("1234", "1234", "0", "text", HEAD, "");
    try (OutputStream text = cache.put(cacheKey, ContentCache.TimeToLive.DEFAULT)) {
        text.write("{}".getBytes());
    } catch (IOException e) {
        e.printStackTrace();
    }
}
Also used : TransformationCacheKey(org.talend.dataprep.cache.TransformationCacheKey) Preparation(org.talend.dataprep.api.preparation.Preparation) OutputStream(java.io.OutputStream) IOException(java.io.IOException) Before(org.junit.Before)

Aggregations

TransformationCacheKey (org.talend.dataprep.cache.TransformationCacheKey)16 Test (org.junit.Test)9 Preparation (org.talend.dataprep.api.preparation.Preparation)7 ExportParameters (org.talend.dataprep.api.export.ExportParameters)6 OutputStream (java.io.OutputStream)5 InputStream (java.io.InputStream)4 DataSet (org.talend.dataprep.api.dataset.DataSet)4 TDPException (org.talend.dataprep.exception.TDPException)4 ExportFormat (org.talend.dataprep.format.export.ExportFormat)4 JsonParser (com.fasterxml.jackson.core.JsonParser)3 IOException (java.io.IOException)3 InputStreamReader (java.io.InputStreamReader)3 TeeOutputStream (org.apache.commons.io.output.TeeOutputStream)3 StringUtils (org.apache.commons.lang.StringUtils)3 Autowired (org.springframework.beans.factory.annotation.Autowired)3 Component (org.springframework.stereotype.Component)3 StreamingResponseBody (org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBody)3 PreparationMessage (org.talend.dataprep.api.preparation.PreparationMessage)3 Step (org.talend.dataprep.api.preparation.Step)3 CacheKeyGenerator (org.talend.dataprep.cache.CacheKeyGenerator)3