Search in sources :

Example 6 with PreparationMessage

use of org.talend.dataprep.api.preparation.PreparationMessage in project data-prep by Talend.

the class APIPreparationConversionsTest method shouldNotEnrichPreparationWithDataset.

@Test
public void shouldNotEnrichPreparationWithDataset() {
    // given
    final PreparationMessage preparation = getPreparationMessage(null);
    // when
    final EnrichedPreparation actual = conversionService.convert(preparation, EnrichedPreparation.class);
    // then
    assertNotNull(actual);
    assertNull(actual.getSummary());
    assertNull(actual.getFolder());
}
Also used : EnrichedPreparation(org.talend.dataprep.api.service.api.EnrichedPreparation) PreparationMessage(org.talend.dataprep.api.preparation.PreparationMessage) Test(org.junit.Test)

Example 7 with PreparationMessage

use of org.talend.dataprep.api.preparation.PreparationMessage in project data-prep by Talend.

the class APIPreparationConversionsTest method getPreparationMessage.

private PreparationMessage getPreparationMessage(String dataSetId) {
    final PreparationMessage preparation = new PreparationMessage();
    preparation.setDataSetId(dataSetId);
    List<Step> steps = new ArrayList<>(12);
    Step parentStep = preparation.getSteps().get(0);
    for (int i = 0; i < 12; i++) {
        final Step step = new Step(parentStep.id(), new PreparationActions().id(), "2.1");
        steps.add(step);
        parentStep = step;
    }
    preparation.setSteps(steps);
    return preparation;
}
Also used : ArrayList(java.util.ArrayList) PreparationMessage(org.talend.dataprep.api.preparation.PreparationMessage) PreparationActions(org.talend.dataprep.api.preparation.PreparationActions) Step(org.talend.dataprep.api.preparation.Step)

Example 8 with PreparationMessage

use of org.talend.dataprep.api.preparation.PreparationMessage in project data-prep by Talend.

the class APIPreparationConversionsTest method shouldDealWithRepeatedStepIds.

@Test
public void shouldDealWithRepeatedStepIds() {
    // given
    DataSetMetadata metadata = getDataSetMetadata("super dataset", 1001L);
    setupHystrixCommand(DataSetGetMetadata.class, metadata);
    final PreparationMessage preparation = getPreparationMessage(metadata.getId());
    preparation.setSteps(asList(Step.ROOT_STEP, Step.ROOT_STEP));
    Folder folder = getFolder("F-753854");
    setupHystrixCommand(LocatePreparation.class, folder);
    // when
    final EnrichedPreparation actual = conversionService.convert(preparation, EnrichedPreparation.class);
    // then
    assertEquals(1, actual.getSteps().size());
    assertEquals(Step.ROOT_STEP.id(), actual.getSteps().get(0));
}
Also used : EnrichedPreparation(org.talend.dataprep.api.service.api.EnrichedPreparation) PreparationMessage(org.talend.dataprep.api.preparation.PreparationMessage) Folder(org.talend.dataprep.api.folder.Folder) DataSetMetadata(org.talend.dataprep.api.dataset.DataSetMetadata) Test(org.junit.Test)

Example 9 with PreparationMessage

use of org.talend.dataprep.api.preparation.PreparationMessage in project data-prep by Talend.

the class PipelineTransformer method buildExecutable.

@Override
public ExecutableTransformer buildExecutable(DataSet input, Configuration configuration) {
    final RowMetadata rowMetadata = input.getMetadata().getRowMetadata();
    // prepare the fallback row metadata
    RowMetadata fallBackRowMetadata = transformationRowMetadataUtils.getMatchingEmptyRowMetadata(rowMetadata);
    final TransformerWriter writer = writerRegistrationService.getWriter(configuration.formatId(), configuration.output(), configuration.getArguments());
    final ConfiguredCacheWriter metadataWriter = new ConfiguredCacheWriter(contentCache, DEFAULT);
    final TransformationMetadataCacheKey metadataKey = cacheKeyGenerator.generateMetadataKey(configuration.getPreparationId(), configuration.stepId(), configuration.getSourceType());
    final PreparationMessage preparation = configuration.getPreparation();
    // function that from a step gives the rowMetadata associated to the previous/parent step
    final Function<Step, RowMetadata> previousStepRowMetadataSupplier = s -> // 
    Optional.ofNullable(s.getParent()).map(// 
    id -> preparationUpdater.get(id)).orElse(null);
    final Pipeline pipeline = // 
    Pipeline.Builder.builder().withAnalyzerService(// 
    analyzerService).withActionRegistry(// 
    actionRegistry).withPreparation(// 
    preparation).withActions(// 
    actionParser.parse(configuration.getActions())).withInitialMetadata(rowMetadata, // 
    configuration.volume() == SMALL).withMonitor(// 
    configuration.getMonitor()).withFilter(// 
    configuration.getFilter()).withLimit(// 
    configuration.getLimit()).withFilterOut(// 
    configuration.getOutFilter()).withOutput(// 
    () -> new WriterNode(writer, metadataWriter, metadataKey, fallBackRowMetadata)).withStatisticsAdapter(// 
    adapter).withStepMetadataSupplier(// 
    previousStepRowMetadataSupplier).withGlobalStatistics(// 
    configuration.isGlobalStatistics()).allowMetadataChange(// 
    configuration.isAllowMetadataChange()).build();
    // wrap this transformer into an executable transformer
    return new ExecutableTransformer() {

        @Override
        public void execute() {
            try {
                LOGGER.debug("Before transformation: {}", pipeline);
                pipeline.execute(input);
            } finally {
                LOGGER.debug("After transformation: {}", pipeline);
            }
            if (preparation != null) {
                final UpdatedStepVisitor visitor = new UpdatedStepVisitor(preparationUpdater);
                pipeline.accept(visitor);
            }
        }

        @Override
        public void signal(Signal signal) {
            pipeline.signal(signal);
        }
    };
}
Also used : WriterNode(org.talend.dataprep.transformation.pipeline.model.WriterNode) WriterRegistrationService(org.talend.dataprep.transformation.format.WriterRegistrationService) SMALL(org.talend.dataprep.transformation.api.transformer.configuration.Configuration.Volume.SMALL) StepMetadataRepository(org.talend.dataprep.transformation.service.StepMetadataRepository) TransformerWriter(org.talend.dataprep.transformation.api.transformer.TransformerWriter) LoggerFactory(org.slf4j.LoggerFactory) Autowired(org.springframework.beans.factory.annotation.Autowired) Configuration(org.talend.dataprep.transformation.api.transformer.configuration.Configuration) Signal(org.talend.dataprep.transformation.pipeline.Signal) DEFAULT(org.talend.dataprep.cache.ContentCache.TimeToLive.DEFAULT) PreparationMessage(org.talend.dataprep.api.preparation.PreparationMessage) Function(java.util.function.Function) AnalyzerService(org.talend.dataprep.quality.AnalyzerService) ActionParser(org.talend.dataprep.transformation.api.action.ActionParser) CacheKeyGenerator(org.talend.dataprep.cache.CacheKeyGenerator) TransformationMetadataCacheKey(org.talend.dataprep.cache.TransformationMetadataCacheKey) DataSet(org.talend.dataprep.api.dataset.DataSet) Logger(org.slf4j.Logger) ActionRegistry(org.talend.dataprep.transformation.pipeline.ActionRegistry) TransformationRowMetadataUtils(org.talend.dataprep.transformation.service.TransformationRowMetadataUtils) Step(org.talend.dataprep.api.preparation.Step) ConfiguredCacheWriter(org.talend.dataprep.transformation.api.transformer.ConfiguredCacheWriter) ContentCache(org.talend.dataprep.cache.ContentCache) ExecutableTransformer(org.talend.dataprep.transformation.api.transformer.ExecutableTransformer) Component(org.springframework.stereotype.Component) StatisticsAdapter(org.talend.dataprep.dataset.StatisticsAdapter) Optional(java.util.Optional) Pipeline(org.talend.dataprep.transformation.pipeline.Pipeline) Transformer(org.talend.dataprep.transformation.api.transformer.Transformer) RowMetadata(org.talend.dataprep.api.dataset.RowMetadata) TransformationMetadataCacheKey(org.talend.dataprep.cache.TransformationMetadataCacheKey) Step(org.talend.dataprep.api.preparation.Step) Pipeline(org.talend.dataprep.transformation.pipeline.Pipeline) Signal(org.talend.dataprep.transformation.pipeline.Signal) WriterNode(org.talend.dataprep.transformation.pipeline.model.WriterNode) ExecutableTransformer(org.talend.dataprep.transformation.api.transformer.ExecutableTransformer) RowMetadata(org.talend.dataprep.api.dataset.RowMetadata) PreparationMessage(org.talend.dataprep.api.preparation.PreparationMessage) TransformerWriter(org.talend.dataprep.transformation.api.transformer.TransformerWriter) ConfiguredCacheWriter(org.talend.dataprep.transformation.api.transformer.ConfiguredCacheWriter)

Aggregations

PreparationMessage (org.talend.dataprep.api.preparation.PreparationMessage)9 Test (org.junit.Test)4 EnrichedPreparation (org.talend.dataprep.api.service.api.EnrichedPreparation)4 TDPException (org.talend.dataprep.exception.TDPException)4 Logger (org.slf4j.Logger)3 LoggerFactory (org.slf4j.LoggerFactory)3 Autowired (org.springframework.beans.factory.annotation.Autowired)3 Component (org.springframework.stereotype.Component)3 DataSet (org.talend.dataprep.api.dataset.DataSet)3 DataSetMetadata (org.talend.dataprep.api.dataset.DataSetMetadata)3 Step (org.talend.dataprep.api.preparation.Step)3 CacheKeyGenerator (org.talend.dataprep.cache.CacheKeyGenerator)3 ContentCache (org.talend.dataprep.cache.ContentCache)3 Configuration (org.talend.dataprep.transformation.api.transformer.configuration.Configuration)3 JsonParser (com.fasterxml.jackson.core.JsonParser)2 IOException (java.io.IOException)2 InputStream (java.io.InputStream)2 InputStreamReader (java.io.InputStreamReader)2 OutputStream (java.io.OutputStream)2 UTF_8 (java.nio.charset.StandardCharsets.UTF_8)2