Search in sources :

Example 1 with TimeSeriesDataPoint

use of com.amazonaws.services.timestreamquery.model.TimeSeriesDataPoint in project aws-athena-query-federation by awslabs.

the class TimestreamRecordHandler method buildTimeSeriesExtractor.

private void buildTimeSeriesExtractor(GeneratedRowWriter.RowWriterBuilder builder, Field field, int curFieldNum) {
    builder.withFieldWriterFactory(field.getName(), (FieldVector vector, Extractor extractor, ConstraintProjector constraint) -> (Object context, int rowNum) -> {
        Row row = (Row) context;
        Datum datum = row.getData().get(curFieldNum);
        Field timeField = field.getChildren().get(0).getChildren().get(0);
        Field valueField = field.getChildren().get(0).getChildren().get(1);
        if (datum.getTimeSeriesValue() != null) {
            List<Map<String, Object>> values = new ArrayList<>();
            for (TimeSeriesDataPoint nextDatum : datum.getTimeSeriesValue()) {
                Map<String, Object> eventMap = new HashMap<>();
                eventMap.put(timeField.getName(), TIMESTAMP_FORMATTER.parse(nextDatum.getTime()).getTime());
                switch(Types.getMinorTypeForArrowType(valueField.getType())) {
                    case FLOAT8:
                        eventMap.put(valueField.getName(), Double.valueOf(nextDatum.getValue().getScalarValue()));
                        break;
                    case BIGINT:
                        eventMap.put(valueField.getName(), Long.valueOf(nextDatum.getValue().getScalarValue()));
                        break;
                    case INT:
                        eventMap.put(valueField.getName(), Integer.valueOf(nextDatum.getValue().getScalarValue()));
                        break;
                    case BIT:
                        eventMap.put(valueField.getName(), Boolean.valueOf(((Row) context).getData().get(curFieldNum).getScalarValue()) == false ? 0 : 1);
                        break;
                }
                values.add(eventMap);
            }
            BlockUtils.setComplexValue(vector, rowNum, FieldResolver.DEFAULT, values);
        } else {
            throw new RuntimeException("Only LISTs of type TimeSeries are presently supported.");
        }
        // we don't yet support predicate pushdown on complex types
        return true;
    });
}
Also used : TimeSeriesDataPoint(com.amazonaws.services.timestreamquery.model.TimeSeriesDataPoint) Datum(com.amazonaws.services.timestreamquery.model.Datum) HashMap(java.util.HashMap) ConstraintProjector(com.amazonaws.athena.connector.lambda.domain.predicate.ConstraintProjector) ArrayList(java.util.ArrayList) FieldVector(org.apache.arrow.vector.FieldVector) Field(org.apache.arrow.vector.types.pojo.Field) BigIntExtractor(com.amazonaws.athena.connector.lambda.data.writers.extractors.BigIntExtractor) BitExtractor(com.amazonaws.athena.connector.lambda.data.writers.extractors.BitExtractor) Float8Extractor(com.amazonaws.athena.connector.lambda.data.writers.extractors.Float8Extractor) VarCharExtractor(com.amazonaws.athena.connector.lambda.data.writers.extractors.VarCharExtractor) DateMilliExtractor(com.amazonaws.athena.connector.lambda.data.writers.extractors.DateMilliExtractor) Extractor(com.amazonaws.athena.connector.lambda.data.writers.extractors.Extractor) Row(com.amazonaws.services.timestreamquery.model.Row) HashMap(java.util.HashMap) Map(java.util.Map)

Example 2 with TimeSeriesDataPoint

use of com.amazonaws.services.timestreamquery.model.TimeSeriesDataPoint in project aws-athena-query-federation by awslabs.

the class TestUtils method buildTimeSeries.

private static void buildTimeSeries(Field field, Datum datum) {
    List<TimeSeriesDataPoint> dataPoints = new ArrayList<>();
    for (int i = 0; i < 10; i++) {
        TimeSeriesDataPoint dataPoint = new TimeSeriesDataPoint();
        Datum dataPointValue = new Datum();
        dataPoint.setTime(TIMESTAMP_FORMATTER.format(new Date(System.currentTimeMillis() - RAND.nextInt(1_000_000))));
        /**
         * Presently we only support TimeSeries as LIST<STRUCT<DATEMILLISECONDS, DOUBLE|INT|FLOAT8|BIT|BIGINT>>
         */
        Field struct = field.getChildren().get(0);
        assertEquals(Types.MinorType.STRUCT, Types.getMinorTypeForArrowType(struct.getType()));
        Field baseSeriesType = struct.getChildren().get(1);
        switch(Types.getMinorTypeForArrowType(baseSeriesType.getType())) {
            case FLOAT8:
                dataPointValue.setScalarValue(String.valueOf(RAND.nextFloat()));
                break;
            case BIT:
                dataPointValue.setScalarValue(String.valueOf(RAND.nextBoolean()));
                break;
            case INT:
                dataPointValue.setScalarValue(String.valueOf(RAND.nextInt()));
                break;
            case BIGINT:
                dataPointValue.setScalarValue(String.valueOf(RAND.nextLong()));
                break;
        }
        dataPoint.setValue(dataPointValue);
        dataPoints.add(dataPoint);
    }
    datum.setTimeSeriesValue(dataPoints);
}
Also used : TimeSeriesDataPoint(com.amazonaws.services.timestreamquery.model.TimeSeriesDataPoint) Field(org.apache.arrow.vector.types.pojo.Field) Datum(com.amazonaws.services.timestreamquery.model.Datum) ArrayList(java.util.ArrayList) TimeSeriesDataPoint(com.amazonaws.services.timestreamquery.model.TimeSeriesDataPoint) Date(java.util.Date)

Aggregations

Datum (com.amazonaws.services.timestreamquery.model.Datum)2 TimeSeriesDataPoint (com.amazonaws.services.timestreamquery.model.TimeSeriesDataPoint)2 ArrayList (java.util.ArrayList)2 Field (org.apache.arrow.vector.types.pojo.Field)2 BigIntExtractor (com.amazonaws.athena.connector.lambda.data.writers.extractors.BigIntExtractor)1 BitExtractor (com.amazonaws.athena.connector.lambda.data.writers.extractors.BitExtractor)1 DateMilliExtractor (com.amazonaws.athena.connector.lambda.data.writers.extractors.DateMilliExtractor)1 Extractor (com.amazonaws.athena.connector.lambda.data.writers.extractors.Extractor)1 Float8Extractor (com.amazonaws.athena.connector.lambda.data.writers.extractors.Float8Extractor)1 VarCharExtractor (com.amazonaws.athena.connector.lambda.data.writers.extractors.VarCharExtractor)1 ConstraintProjector (com.amazonaws.athena.connector.lambda.domain.predicate.ConstraintProjector)1 Row (com.amazonaws.services.timestreamquery.model.Row)1 Date (java.util.Date)1 HashMap (java.util.HashMap)1 Map (java.util.Map)1 FieldVector (org.apache.arrow.vector.FieldVector)1