Search in sources :

Example 1 with CsvMapper

use of com.fasterxml.jackson.dataformat.csv.CsvMapper in project SONG by overture-stack.

the class PcawgSampleSheetConfig method pcawgSampleSheetReader.

public static ObjectReader pcawgSampleSheetReader(boolean hasHeader, char separator) {
    val csvMapper = new CsvMapper();
    val csvSchema = CsvSchema.emptySchema().withHeader().withColumnSeparator(separator);
    return csvMapper.readerFor(PcawgSampleBean.class).with(csvSchema);
}
Also used : lombok.val(lombok.val) CsvMapper(com.fasterxml.jackson.dataformat.csv.CsvMapper) PcawgSampleBean(org.icgc.dcc.song.importer.model.PcawgSampleBean)

Example 2 with CsvMapper

use of com.fasterxml.jackson.dataformat.csv.CsvMapper in project flink by apache.

the class CsvBulkWriter method forPojo.

/**
 * Builds a writer based on a POJO class definition.
 *
 * @param pojoClass The class of the POJO.
 * @param stream The output stream.
 * @param <T> The type of the elements accepted by this writer.
 */
static <T> CsvBulkWriter<T, T, Void> forPojo(Class<T> pojoClass, FSDataOutputStream stream) {
    final Converter<T, T, Void> converter = (value, context) -> value;
    final CsvMapper csvMapper = new CsvMapper();
    final CsvSchema schema = csvMapper.schemaFor(pojoClass).withoutQuoteChar();
    return new CsvBulkWriter<>(csvMapper, schema, converter, null, stream);
}
Also used : Converter(org.apache.flink.formats.common.Converter) FSDataOutputStream(org.apache.flink.core.fs.FSDataOutputStream) ObjectWriter(org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectWriter) BulkWriter(org.apache.flink.api.common.serialization.BulkWriter) JsonGenerator(org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonGenerator) CsvSchema(org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema) IOException(java.io.IOException) Preconditions.checkNotNull(org.apache.flink.util.Preconditions.checkNotNull) Nullable(javax.annotation.Nullable) CsvMapper(org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper) CsvSchema(org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema) CsvMapper(org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper)

Example 3 with CsvMapper

use of com.fasterxml.jackson.dataformat.csv.CsvMapper in project flink by apache.

the class CsvFileFormatFactory method createEncodingFormat.

@Override
public EncodingFormat<Factory<RowData>> createEncodingFormat(DynamicTableFactory.Context context, ReadableConfig formatOptions) {
    return new EncodingFormat<BulkWriter.Factory<RowData>>() {

        @Override
        public BulkWriter.Factory<RowData> createRuntimeEncoder(DynamicTableSink.Context context, DataType physicalDataType) {
            final RowType rowType = (RowType) physicalDataType.getLogicalType();
            final CsvSchema schema = buildCsvSchema(rowType, formatOptions);
            final RowDataToCsvConverter converter = RowDataToCsvConverters.createRowConverter(rowType);
            final CsvMapper mapper = new CsvMapper();
            final ObjectNode container = mapper.createObjectNode();
            final RowDataToCsvConverter.RowDataToCsvFormatConverterContext converterContext = new RowDataToCsvConverter.RowDataToCsvFormatConverterContext(mapper, container);
            return out -> CsvBulkWriter.forSchema(mapper, schema, converter, converterContext, out);
        }

        @Override
        public ChangelogMode getChangelogMode() {
            return ChangelogMode.insertOnly();
        }
    };
}
Also used : Context(org.apache.flink.table.connector.source.DynamicTableSource.Context) DynamicTableFactory(org.apache.flink.table.factories.DynamicTableFactory) DataType(org.apache.flink.table.types.DataType) EncodingFormat(org.apache.flink.table.connector.format.EncodingFormat) ChangelogMode(org.apache.flink.table.connector.ChangelogMode) FIELD_DELIMITER(org.apache.flink.formats.csv.CsvFormatOptions.FIELD_DELIMITER) BulkWriterFormatFactory(org.apache.flink.connector.file.table.factories.BulkWriterFormatFactory) CsvSchema(org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema) Context(org.apache.flink.table.connector.source.DynamicTableSource.Context) JsonNode(org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode) RowType(org.apache.flink.table.types.logical.RowType) ALLOW_COMMENTS(org.apache.flink.formats.csv.CsvFormatOptions.ALLOW_COMMENTS) Factory(org.apache.flink.api.common.serialization.BulkWriter.Factory) ReadableConfig(org.apache.flink.configuration.ReadableConfig) FileSourceSplit(org.apache.flink.connector.file.src.FileSourceSplit) IGNORE_PARSE_ERRORS(org.apache.flink.formats.csv.CsvFormatOptions.IGNORE_PARSE_ERRORS) QUOTE_CHARACTER(org.apache.flink.formats.csv.CsvFormatOptions.QUOTE_CHARACTER) RowDataToCsvConverter(org.apache.flink.formats.csv.RowDataToCsvConverters.RowDataToCsvConverter) ESCAPE_CHARACTER(org.apache.flink.formats.csv.CsvFormatOptions.ESCAPE_CHARACTER) StreamFormatAdapter(org.apache.flink.connector.file.src.impl.StreamFormatAdapter) ConfigOption(org.apache.flink.configuration.ConfigOption) StringEscapeUtils(org.apache.commons.lang3.StringEscapeUtils) Preconditions.checkNotNull(org.apache.flink.util.Preconditions.checkNotNull) BulkDecodingFormat(org.apache.flink.connector.file.table.format.BulkDecodingFormat) Projection(org.apache.flink.table.connector.Projection) BulkReaderFormatFactory(org.apache.flink.connector.file.table.factories.BulkReaderFormatFactory) RowData(org.apache.flink.table.data.RowData) DynamicTableSink(org.apache.flink.table.connector.sink.DynamicTableSink) BulkWriter(org.apache.flink.api.common.serialization.BulkWriter) ObjectNode(org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.node.ObjectNode) Set(java.util.Set) ProjectableDecodingFormat(org.apache.flink.table.connector.format.ProjectableDecodingFormat) DISABLE_QUOTE_CHARACTER(org.apache.flink.formats.csv.CsvFormatOptions.DISABLE_QUOTE_CHARACTER) ARRAY_ELEMENT_DELIMITER(org.apache.flink.formats.csv.CsvFormatOptions.ARRAY_ELEMENT_DELIMITER) Converter(org.apache.flink.formats.common.Converter) NULL_LITERAL(org.apache.flink.formats.csv.CsvFormatOptions.NULL_LITERAL) Internal(org.apache.flink.annotation.Internal) BulkFormat(org.apache.flink.connector.file.src.reader.BulkFormat) Collections(java.util.Collections) CsvMapper(org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper) RowDataToCsvConverter(org.apache.flink.formats.csv.RowDataToCsvConverters.RowDataToCsvConverter) ObjectNode(org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.node.ObjectNode) CsvMapper(org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper) RowType(org.apache.flink.table.types.logical.RowType) EncodingFormat(org.apache.flink.table.connector.format.EncodingFormat) RowData(org.apache.flink.table.data.RowData) CsvSchema(org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema) BulkWriter(org.apache.flink.api.common.serialization.BulkWriter) DataType(org.apache.flink.table.types.DataType)

Example 4 with CsvMapper

use of com.fasterxml.jackson.dataformat.csv.CsvMapper in project goci by EBISPOT.

the class FileHandler method getStudyPatchRequests.

public static List<StudyPatchRequest> getStudyPatchRequests(FileUploadRequest fileUploadRequest) {
    CsvMapper mapper = new CsvMapper();
    CsvSchema schema = getSchemaFromMultiPartFile(fileUploadRequest.getMultipartFile());
    List<StudyPatchRequest> studyPatchRequests;
    try {
        InputStream inputStream = fileUploadRequest.getMultipartFile().getInputStream();
        MappingIterator<StudyPatchRequest> iterator = mapper.readerFor(StudyPatchRequest.class).with(schema).readValues(inputStream);
        studyPatchRequests = iterator.readAll();
    } catch (IOException e) {
        throw new FileUploadException("Could not read the file");
    }
    return studyPatchRequests;
}
Also used : CsvSchema(com.fasterxml.jackson.dataformat.csv.CsvSchema) InputStream(java.io.InputStream) CsvMapper(com.fasterxml.jackson.dataformat.csv.CsvMapper) IOException(java.io.IOException) FileUploadException(uk.ac.ebi.spot.goci.curation.exception.FileUploadException) StudyPatchRequest(uk.ac.ebi.spot.goci.curation.dto.StudyPatchRequest)

Example 5 with CsvMapper

use of com.fasterxml.jackson.dataformat.csv.CsvMapper in project goci by EBISPOT.

the class FileHandler method serializePojoToTsv.

public static String serializePojoToTsv(List<?> pojoList) {
    CsvMapper csvMapper = new CsvMapper();
    List<Map<String, Object>> dataList = csvMapper.convertValue(pojoList, new TypeReference<Object>() {
    });
    List<List<String>> csvData = new ArrayList<>();
    List<String> csvHead = new ArrayList<>();
    AtomicInteger counter = new AtomicInteger();
    dataList.forEach(row -> {
        List<String> rowData = new ArrayList<>();
        row.forEach((key, value) -> {
            rowData.add(String.valueOf(value));
            if (counter.get() == 0) {
                csvHead.add(key);
            }
        });
        csvData.add(rowData);
        counter.getAndIncrement();
    });
    CsvSchema.Builder builder = CsvSchema.builder();
    csvHead.forEach(builder::addColumn);
    CsvSchema schema = builder.build().withHeader().withLineSeparator("\n").withColumnSeparator('\t');
    String result = "";
    try {
        result = csvMapper.writer(schema).writeValueAsString(csvData);
    } catch (IOException e) {
        throw new FileUploadException("Could not read the file");
    }
    return result;
}
Also used : CsvMapper(com.fasterxml.jackson.dataformat.csv.CsvMapper) ArrayList(java.util.ArrayList) IOException(java.io.IOException) CsvSchema(com.fasterxml.jackson.dataformat.csv.CsvSchema) AtomicInteger(java.util.concurrent.atomic.AtomicInteger) ArrayList(java.util.ArrayList) List(java.util.List) Map(java.util.Map) FileUploadException(uk.ac.ebi.spot.goci.curation.exception.FileUploadException)

Aggregations

CsvMapper (com.fasterxml.jackson.dataformat.csv.CsvMapper)21 CsvSchema (com.fasterxml.jackson.dataformat.csv.CsvSchema)15 IOException (java.io.IOException)10 ArrayList (java.util.ArrayList)7 ObjectWriter (com.fasterxml.jackson.databind.ObjectWriter)5 InputStream (java.io.InputStream)5 List (java.util.List)5 CsvMapper (org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvMapper)5 ObjectReader (com.fasterxml.jackson.databind.ObjectReader)4 File (java.io.File)4 HashMap (java.util.HashMap)4 Map (java.util.Map)4 Converter (org.apache.flink.formats.common.Converter)4 CsvSchema (org.apache.flink.shaded.jackson2.com.fasterxml.jackson.dataformat.csv.CsvSchema)4 FileUploadException (uk.ac.ebi.spot.goci.curation.exception.FileUploadException)4 OutputStream (java.io.OutputStream)3 JsonProcessingException (com.fasterxml.jackson.core.JsonProcessingException)2 MappingIterator (com.fasterxml.jackson.databind.MappingIterator)2 JetException (com.hazelcast.jet.JetException)2 FileOutputStream (java.io.FileOutputStream)2