Search in sources :

Example 1 with StreamingResponseBody

use of org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBody in project data-prep by Talend.

the class CommandHelper method toStreaming.

public static ResponseEntity<StreamingResponseBody> toStreaming(final GenericCommand<InputStream> command) {
    final Observable<InputStream> stream = command.toObservable();
    return stream.map(is -> {
        // Content for the response entity
        final StreamingResponseBody body = outputStream -> {
            try {
                IOUtils.copyLarge(is, outputStream);
                outputStream.flush();
            } catch (IOException e) {
                try {
                    is.close();
                } catch (IOException closingException) {
                    LOGGER.warn("could not close command result, a http connection may be leaked !", closingException);
                }
                LOGGER.error("Unable to fully copy command result '{}'.", command.getClass(), e);
            }
        };
        // copy all headers from the command response so that the mime-type is correctly forwarded. Command has
        // the correct headers due to call to toBlocking() below.
        final MultiValueMap<String, String> headers = new HttpHeaders();
        final HttpStatus status = command.getStatus();
        for (Header header : command.getCommandResponseHeaders()) {
            headers.put(header.getName(), Collections.singletonList(header.getValue()));
        }
        return new ResponseEntity<>(body, headers, status == null ? HttpStatus.OK : status);
    }).toBlocking().first();
}
Also used : TDPException(org.talend.dataprep.exception.TDPException) Logger(org.slf4j.Logger) HttpHeaders(org.springframework.http.HttpHeaders) Publisher(org.reactivestreams.Publisher) FluxSink(reactor.core.publisher.FluxSink) LoggerFactory(org.slf4j.LoggerFactory) ObjectMapper(com.fasterxml.jackson.databind.ObjectMapper) MultiValueMap(org.springframework.util.MultiValueMap) StreamingResponseBody(org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBody) IOException(java.io.IOException) Header(org.apache.http.Header) Observable(rx.Observable) IOUtils(org.apache.commons.io.IOUtils) HttpStatus(org.springframework.http.HttpStatus) Flux(reactor.core.publisher.Flux) HystrixCommand(com.netflix.hystrix.HystrixCommand) Stream(java.util.stream.Stream) AtomicInteger(java.util.concurrent.atomic.AtomicInteger) CommonErrorCodes(org.talend.dataprep.exception.error.CommonErrorCodes) ResponseEntity(org.springframework.http.ResponseEntity) Collections(java.util.Collections) InputStream(java.io.InputStream) HttpHeaders(org.springframework.http.HttpHeaders) ResponseEntity(org.springframework.http.ResponseEntity) StreamingResponseBody(org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBody) Header(org.apache.http.Header) HttpStatus(org.springframework.http.HttpStatus) InputStream(java.io.InputStream) IOException(java.io.IOException)

Example 2 with StreamingResponseBody

use of org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBody in project data-prep by Talend.

the class CommandHelperTest method testCommandToStreamingWithHeader.

@Test
public void testCommandToStreamingWithHeader() throws Exception {
    GenericCommand<InputStream> command = new CommandHelperTestCommand();
    final ResponseEntity<StreamingResponseBody> responseEntity = CommandHelper.toStreaming(command);
    final ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
    responseEntity.getBody().writeTo(outputStream);
    assertEquals("test", new String(outputStream.toByteArray()));
    assertEquals(HttpStatus.NO_CONTENT, responseEntity.getStatusCode());
    assertEquals("custom value", responseEntity.getHeaders().get("custom").get(0));
}
Also used : StreamingResponseBody(org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBody) ByteArrayInputStream(java.io.ByteArrayInputStream) InputStream(java.io.InputStream) ByteArrayOutputStream(java.io.ByteArrayOutputStream) Test(org.junit.Test)

Example 3 with StreamingResponseBody

use of org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBody in project data-prep by Talend.

the class CommandHelperTest method testCommandToStreamingWithNoHeader.

@Test
public void testCommandToStreamingWithNoHeader() throws Exception {
    HystrixCommand<InputStream> command = new CommandHelperTestCommand();
    final StreamingResponseBody responseBody = CommandHelper.toStreaming(command);
    final ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
    responseBody.writeTo(outputStream);
    assertEquals("test", new String(outputStream.toByteArray()));
}
Also used : StreamingResponseBody(org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBody) ByteArrayInputStream(java.io.ByteArrayInputStream) InputStream(java.io.InputStream) ByteArrayOutputStream(java.io.ByteArrayOutputStream) Test(org.junit.Test)

Example 4 with StreamingResponseBody

use of org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBody in project data-prep by Talend.

the class TransformationService method aggregate.

/**
 * Compute the given aggregation.
 *
 * @param rawParams the aggregation rawParams as body rawParams.
 */
// @formatter:off
@RequestMapping(value = "/aggregate", method = POST, consumes = APPLICATION_JSON_VALUE)
@ApiOperation(value = "Compute the aggregation according to the request body rawParams", consumes = APPLICATION_JSON_VALUE)
@VolumeMetered
public AggregationResult aggregate(@ApiParam(value = "The aggregation rawParams in json") @RequestBody final String rawParams) {
    // @formatter:on
    // parse the aggregation parameters
    final AggregationParameters parameters;
    try {
        parameters = mapper.readerFor(AggregationParameters.class).readValue(rawParams);
        LOG.debug("Aggregation requested {}", parameters);
    } catch (IOException e) {
        throw new TDPException(CommonErrorCodes.BAD_AGGREGATION_PARAMETERS, e);
    }
    InputStream contentToAggregate;
    // get the content of the preparation (internal call with piped streams)
    if (StringUtils.isNotBlank(parameters.getPreparationId())) {
        try {
            PipedOutputStream temp = new PipedOutputStream();
            contentToAggregate = new PipedInputStream(temp);
            // because of piped streams, processing must be asynchronous
            Runnable r = () -> {
                try {
                    final ExportParameters exportParameters = new ExportParameters();
                    exportParameters.setPreparationId(parameters.getPreparationId());
                    exportParameters.setDatasetId(parameters.getDatasetId());
                    final String filter = parameters.getFilter();
                    if (filter != null) {
                        if (filter.isEmpty()) {
                            throw new TDPException(CommonErrorCodes.UNABLE_TO_AGGREGATE, new IllegalArgumentException("Source should not be empty"));
                        }
                        exportParameters.setFilter(mapper.readTree(filter));
                    }
                    exportParameters.setExportType(JSON);
                    exportParameters.setStepId(parameters.getStepId());
                    final StreamingResponseBody body = executeSampleExportStrategy(exportParameters);
                    body.writeTo(temp);
                } catch (IOException e) {
                    throw new TDPException(CommonErrorCodes.UNABLE_TO_AGGREGATE, e);
                }
            };
            executor.execute(r);
        } catch (IOException e) {
            throw new TDPException(CommonErrorCodes.UNABLE_TO_AGGREGATE, e);
        }
    } else {
        final DataSetGet dataSetGet = context.getBean(DataSetGet.class, parameters.getDatasetId(), false, true);
        contentToAggregate = dataSetGet.execute();
    }
    // apply the aggregation
    try (JsonParser parser = mapper.getFactory().createParser(new InputStreamReader(contentToAggregate, UTF_8))) {
        final DataSet dataSet = mapper.readerFor(DataSet.class).readValue(parser);
        return aggregationService.aggregate(parameters, dataSet);
    } catch (IOException e) {
        throw new TDPException(CommonErrorCodes.UNABLE_TO_PARSE_JSON, e);
    } finally {
        // don't forget to release the connection
        if (contentToAggregate != null) {
            try {
                contentToAggregate.close();
            } catch (IOException e) {
                LOG.warn("Could not close dataset input stream while aggregating", e);
            }
        }
    }
}
Also used : DataSetGet(org.talend.dataprep.command.dataset.DataSetGet) StreamingResponseBody(org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBody) DataSet(org.talend.dataprep.api.dataset.DataSet) AggregationParameters(org.talend.dataprep.transformation.aggregation.api.AggregationParameters) TDPException(org.talend.dataprep.exception.TDPException) ExportParameters(org.talend.dataprep.api.export.ExportParameters) JsonParser(com.fasterxml.jackson.core.JsonParser) VolumeMetered(org.talend.dataprep.metrics.VolumeMetered) ApiOperation(io.swagger.annotations.ApiOperation)

Example 5 with StreamingResponseBody

use of org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBody in project data-prep by Talend.

the class CachedExportStrategy method execute.

@Override
public StreamingResponseBody execute(ExportParameters parameters) {
    final TransformationCacheKey contentKey = getCacheKey(parameters);
    // 
    ExportUtils.setExportHeaders(// 
    parameters.getExportName(), // 
    parameters.getArguments().get(ExportFormat.PREFIX + CSVFormat.ParametersCSV.ENCODING), getFormat(parameters.getExportType()));
    return outputStream -> {
        try (InputStream cachedContent = contentCache.get(contentKey)) {
            IOUtils.copy(cachedContent, outputStream);
        }
    };
}
Also used : TransformationCacheKey(org.talend.dataprep.cache.TransformationCacheKey) ExportFormat(org.talend.dataprep.format.export.ExportFormat) StringUtils(org.apache.commons.lang.StringUtils) ExportParameters(org.talend.dataprep.api.export.ExportParameters) TDPException(org.talend.dataprep.exception.TDPException) BaseExportStrategy(org.talend.dataprep.transformation.service.BaseExportStrategy) IOUtils(org.apache.poi.util.IOUtils) Autowired(org.springframework.beans.factory.annotation.Autowired) StreamingResponseBody(org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBody) PreparationErrorCodes(org.talend.dataprep.exception.error.PreparationErrorCodes) PreparationMessage(org.talend.dataprep.api.preparation.PreparationMessage) CacheKeyGenerator(org.talend.dataprep.cache.CacheKeyGenerator) CSVFormat(org.talend.dataprep.transformation.format.CSVFormat) Component(org.springframework.stereotype.Component) TransformationCacheKey(org.talend.dataprep.cache.TransformationCacheKey) ExportUtils(org.talend.dataprep.transformation.service.ExportUtils) InputStream(java.io.InputStream) InputStream(java.io.InputStream)

Aggregations

StreamingResponseBody (org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBody)26 Test (org.junit.Test)16 Problem (org.zalando.problem.Problem)10 ExportParameters (org.talend.dataprep.api.export.ExportParameters)9 InputStream (java.io.InputStream)8 TDPException (org.talend.dataprep.exception.TDPException)8 ByteArrayOutputStream (java.io.ByteArrayOutputStream)6 OutputStream (java.io.OutputStream)6 Configuration (org.talend.dataprep.transformation.api.transformer.configuration.Configuration)6 JsonParser (com.fasterxml.jackson.core.JsonParser)5 StringUtils (org.apache.commons.lang.StringUtils)5 Logger (org.slf4j.Logger)5 LoggerFactory (org.slf4j.LoggerFactory)5 Autowired (org.springframework.beans.factory.annotation.Autowired)5 Component (org.springframework.stereotype.Component)5 DataSet (org.talend.dataprep.api.dataset.DataSet)5 ExportFormat (org.talend.dataprep.format.export.ExportFormat)5 CSVFormat (org.talend.dataprep.transformation.format.CSVFormat)5 BaseExportStrategy (org.talend.dataprep.transformation.service.BaseExportStrategy)5 ExportUtils (org.talend.dataprep.transformation.service.ExportUtils)5