Examples with RollupSpan net.opentsdb.rollup.RollupSpan used on opensource projects

Search in sources :

Example 1 with RollupSpan

use of net.opentsdb.rollup.RollupSpan in project opentsdb by OpenTSDB.

the class MultiGetQuery method mergeDataPoints.

private void mergeDataPoints() {
    for (List<KeyValue> kvs : kvsmap.values()) {
        if (kvs == null || kvs.isEmpty()) {
            LOG.error("Found a key value list that was null or empty");
            continue;
        }
        for (final KeyValue kv : kvs) {
            if (kv == null) {
                LOG.error("Found a key value item that was null");
                continue;
            }
            if (kv.key() == null) {
                LOG.error("A key for a kv was null");
                continue;
            }
            Span datapoints = null;
            try {
                datapoints = spans.get(kv.key());
            } catch (RuntimeException e) {
                LOG.error("Failed to fetch the span", e);
            }
            // non-compcated or out of order rows here
            if (datapoints == null) {
                datapoints = RollupQuery.isValidQuery(rollup_query) ? new RollupSpan(tsdb, rollup_query) : new Span(tsdb);
                spans.put(kv.key(), datapoints);
            }
            if (annotMap.containsKey(kv.key())) {
                for (Annotation note : annotMap.get(kv.key())) {
                    datapoints.getAnnotations().add(note);
                }
                annotMap.remove(kv.key());
            }
            try {
                datapoints.addRow(kv);
            } catch (RuntimeException e) {
                LOG.error("Exception adding row to span", e);
            }
        }
    }
    kvsmap.clear();
    for (byte[] key : annotMap.keySet()) {
        Span datapoints = (Span) spans.get(key);
        if (datapoints == null) {
            datapoints = new Span(tsdb);
            spans.put(key, datapoints);
        }
        for (Annotation note : annotMap.get(key)) {
            datapoints.getAnnotations().add(note);
        }
    }
    annotMap.clear();
}
Also used : KeyValue(org.hbase.async.KeyValue) RollupSpan(net.opentsdb.rollup.RollupSpan) RollupSpan(net.opentsdb.rollup.RollupSpan) Annotation(net.opentsdb.meta.Annotation)

Example 2 with RollupSpan

use of net.opentsdb.rollup.RollupSpan in project opentsdb by OpenTSDB.

the class SaltScanner method mergeDataPoints.

/**
 * Called once all of the scanners have reported back in to record our
 * latency and merge the results into the spans map. If there was an exception
 * stored then we'll return that instead.
 */
private void mergeDataPoints() {
    // Merge sorted spans together
    final long merge_start = DateTime.nanoTime();
    for (final List<KeyValue> kvs : kv_map.values()) {
        if (kvs == null || kvs.isEmpty()) {
            LOG.warn("Found a key value list that was null or empty");
            continue;
        }
        for (final KeyValue kv : kvs) {
            if (kv == null) {
                LOG.warn("Found a key value item that was null");
                continue;
            }
            if (kv.key() == null) {
                LOG.warn("A key for a kv was null");
                continue;
            }
            Span datapoints = spans.get(kv.key());
            if (datapoints == null) {
                datapoints = RollupQuery.isValidQuery(rollup_query) ? new RollupSpan(tsdb, this.rollup_query) : new Span(tsdb);
                spans.put(kv.key(), datapoints);
            }
            if (annotation_map.containsKey(kv.key())) {
                for (final Annotation note : annotation_map.get(kv.key())) {
                    datapoints.getAnnotations().add(note);
                }
                annotation_map.remove(kv.key());
            }
            try {
                datapoints.addRow(kv);
            } catch (RuntimeException e) {
                LOG.error("Exception adding row to span", e);
                throw e;
            }
        }
    }
    kv_map.clear();
    for (final byte[] key : annotation_map.keySet()) {
        Span datapoints = spans.get(key);
        if (datapoints == null) {
            datapoints = new Span(tsdb);
            spans.put(key, datapoints);
        }
        for (final Annotation note : annotation_map.get(key)) {
            datapoints.getAnnotations().add(note);
        }
    }
    annotation_map.clear();
}
Also used : KeyValue(org.hbase.async.KeyValue) RollupSpan(net.opentsdb.rollup.RollupSpan) RollupSpan(net.opentsdb.rollup.RollupSpan) Annotation(net.opentsdb.meta.Annotation)

Example 3 with RollupSpan

use of net.opentsdb.rollup.RollupSpan in project opentsdb by OpenTSDB.

the class TestRollupSpan method iterateNormalizedMS.

@Test
public void iterateNormalizedMS() throws Exception {
    final byte[] qual1 = { 0x73, 0x75, 0x6D, 0x3A, 0x00, 0x07 };
    final byte[] val1 = Bytes.fromLong(4L);
    final byte[] qual2 = { 0x73, 0x75, 0x6D, 0x3A, 0x00, 0x27 };
    final byte[] val2 = Bytes.fromLong(5L);
    final Span span = new RollupSpan(tsdb, rollup_query);
    span.addRow(new KeyValue(hour1, TSDB.FAMILY(), qual1, val1));
    span.addRow(new KeyValue(hour1, TSDB.FAMILY(), qual2, val2));
    span.addRow(new KeyValue(hour2, TSDB.FAMILY(), qual1, val1));
    span.addRow(new KeyValue(hour2, TSDB.FAMILY(), qual2, val2));
    span.addRow(new KeyValue(hour3, TSDB.FAMILY(), qual1, val1));
    span.addRow(new KeyValue(hour3, TSDB.FAMILY(), qual2, val2));
    assertEquals(6, span.size());
    final SeekableView it = span.iterator();
    DataPoint dp = it.next();
    assertEquals(1356998400000L, dp.timestamp());
    assertEquals(4, dp.longValue());
    dp = it.next();
    assertEquals(1356998402000L, dp.timestamp());
    assertEquals(5, dp.longValue());
    dp = it.next();
    assertEquals(1357002000000L, dp.timestamp());
    assertEquals(4, dp.longValue());
    dp = it.next();
    assertEquals(1357002002000L, dp.timestamp());
    assertEquals(5, dp.longValue());
    dp = it.next();
    assertEquals(1357005600000L, dp.timestamp());
    assertEquals(4, dp.longValue());
    dp = it.next();
    assertEquals(1357005602000L, dp.timestamp());
    assertEquals(5, dp.longValue());
    assertFalse(it.hasNext());
}
Also used : KeyValue(org.hbase.async.KeyValue) RollupSpan(net.opentsdb.rollup.RollupSpan) RollupSpan(net.opentsdb.rollup.RollupSpan) Test(org.junit.Test)

Example 4 with RollupSpan

use of net.opentsdb.rollup.RollupSpan in project opentsdb by OpenTSDB.

the class TestRollupSpan method timestampNormalized.

/*
  * TODO - fix up these tests 
  @Test (expected = IllegalArgumentException.class)
  public void addRowBadKeyLength() {
    final byte[] qual1 = {0x73, 0x75, 0x6D, 0x3A, 0x00, 0x07 };
    final byte[] val1 = Bytes.fromLong(4L);
    final byte[] qual2 = {0x73, 0x75, 0x6D, 0x3A, 0x00, 0x07 };
    final byte[] val2 = Bytes.fromLong(8L);
    
    final Span span = new RollupSpan(tsdb, rollup_query);
    span.addRow(new KeyValue(HOUR1, TSDB.FAMILY(), qual1, val1));
    
    final byte[] bad_key = 
      new byte[] { 0, 0, 0, 1, 0x50, (byte)0xE2, 0x43, 0x20, 0, 0, 0, 1 };
    span.addRow(new KeyValue(bad_key, TSDB.FAMILY(), qual2, 
        MockBase.concatByteArrays(val1, val2, ZERO)));
  }
  
  @Test (expected = IllegalArgumentException.class)
  public void addRowMissMatchedMetric() {
    final byte[] qual1 = {0x73, 0x75, 0x6D, 0x3A, 0x00, 0x07 };
    final byte[] val1 = Bytes.fromLong(4L);
    final byte[] qual2 = {0x73, 0x75, 0x6D, 0x3A, 0x00, 0x07 };
    final byte[] val2 = Bytes.fromLong(8L);
    
    final Span span = new RollupSpan(tsdb, rollup_query);
    span.addRow(new KeyValue(HOUR1, TSDB.FAMILY(), qual1,val1));
    
    final byte[] bad_key = 
      new byte[] { 0, 0, 0, 2, 0x50, (byte)0xE2, 0x35, 0x10, 0, 0, 0, 1, 0, 0, 0, 2 };
    span.addRow(new KeyValue(bad_key, TSDB.FAMILY(), qual2,val2));
  }
  
  @Test (expected = IllegalArgumentException.class)
  public void addRowMissMatchedTagk() {
    final byte[] qual1 = {0x73, 0x75, 0x6D, 0x3A, 0x00, 0x07 };
    final byte[] val1 = Bytes.fromLong(4L);
    final byte[] qual2 = {0x73, 0x75, 0x6D, 0x3A, 0x00, 0x07 };
    final byte[] val2 = Bytes.fromLong(8L);
    
    final Span span = new RollupSpan(tsdb, rollup_query);
    span.addRow(new KeyValue(HOUR1, TSDB.FAMILY(), qual1, val1));
    
    final byte[] bad_key = 
      new byte[] { 0, 0, 0, 1, 0x50, (byte)0xE2, 0x35, 0x10, 0, 0, 0, 2, 0, 0, 0, 2 };
    span.addRow(new KeyValue(bad_key, TSDB.FAMILY(), qual2, val2));
  }
  
  @Test (expected = IllegalArgumentException.class)
  public void addRowMissMatchedTagv() {
    final byte[] qual1 = {0x73, 0x75, 0x6D, 0x3A, 0x00, 0x07 };
    final byte[] val1 = Bytes.fromLong(4L);
    final byte[] qual2 = {0x73, 0x75, 0x6D, 0x3A, 0x00, 0x07 };
    final byte[] val2 = Bytes.fromLong(8L);
    
    final Span span = new RollupSpan(tsdb, rollup_query);
    span.addRow(new KeyValue(HOUR1, TSDB.FAMILY(), qual1, val1));
    
    final byte[] bad_key = 
      new byte[] { 0, 0, 0, 1, 0x50, (byte)0xE2, 0x35, 0x10, 0, 0, 0, 1, 0, 0, 0, 3 };
    span.addRow(new KeyValue(bad_key, TSDB.FAMILY(), qual2, val2));
  }
  
  @Test
  public void addRowOutOfOrder() {
    //2nd hour
    final byte[] qual1 = {0x73, 0x75, 0x6D, 0x3A, 0x00, 0x07 };
    final byte[] val1 = Bytes.fromLong(4L);
    //1st hour
    final byte[] qual2 = {0x73, 0x75, 0x6D, 0x3A, 0x00, 0x27 };
    final byte[] val2 = Bytes.fromLong(5L);
    
    final Span span = new RollupSpan(tsdb, rollup_query);
    span.addRow(new KeyValue(HOUR2, TSDB.FAMILY(), qual1, val1));
    span.addRow(new KeyValue(HOUR1, TSDB.FAMILY(), qual2, val2));
    assertEquals(2, span.size());
    
    assertEquals(1356998402000L, span.timestamp(0));
    assertEquals(5, span.longValue(0));
    assertEquals(1357002000000L, span.timestamp(1));
    assertEquals(4, span.longValue(1));
  }

  @Test
  public void addDifferentSalt() throws Exception {
    List<byte[]> qualifiers = new ArrayList<byte[]>();
    List<byte[]> values = new ArrayList<byte[]>();

    qualifiers.add(new byte[]{ 0x73, 0x75, 0x6D, 0x3A, 0x00, 0x07 });
    qualifiers.add(new byte[]{ 0x73, 0x75, 0x6D, 0x3A, 0x00, 0x27 });
    qualifiers.add(new byte[]{ 0x73, 0x75, 0x6D, 0x3A, 0x00, 0x37 });
    qualifiers.add(new byte[]{ 0x73, 0x75, 0x6D, 0x3A, 0x00, 0x47 });
    values.add(Bytes.fromLong(4L));
    values.add(Bytes.fromLong(5L));
    
    final Span span = new RollupSpan(tsdb, rollup_query);
    final byte[] hour1 = Arrays.copyOf(HOUR1, HOUR1.length);
    hour1[0] = 2;
    span.addRow(new KeyValue(HOUR1, TSDB.FAMILY(), qualifiers.get(0), values.get(0)));
    span.addRow(new KeyValue(HOUR1, TSDB.FAMILY(), qualifiers.get(1), values.get(1)));
    span.addRow(new KeyValue(hour1, TSDB.FAMILY(), qualifiers.get(2), values.get(0)));
    span.addRow(new KeyValue(hour1, TSDB.FAMILY(), qualifiers.get(3), values.get(1)));
    assertEquals(4, span.size());
    assertEquals(1356998400000L, span.timestamp(0));
    assertEquals(4, span.longValue(0));
    assertEquals(1356998402000L, span.timestamp(1));
    assertEquals(5, span.longValue(1));
    assertEquals(1356998403000L, span.timestamp(2));
    assertEquals(4, span.longValue(2));
    assertEquals(1356998404000L, span.timestamp(3));
    assertEquals(5, span.longValue(3));
  }

  @Test
  public void addDifferentSaltDiffHour() throws Exception {
    List<byte[]> qualifiers = new ArrayList<byte[]>();
    List<byte[]> values = new ArrayList<byte[]>();

    qualifiers.add(new byte[]{ 0x73, 0x75, 0x6D, 0x3A, 0x00, 0x07 });
    values.add(Bytes.fromLong(4L));
    qualifiers.add(new byte[]{ 0x73, 0x75, 0x6D, 0x3A, 0x00, 0x27 });
    values.add(Bytes.fromLong(5L));
    
    final Span span = new RollupSpan(tsdb, rollup_query);
    final byte[] hour2 = Arrays.copyOf(HOUR2, HOUR2.length);
    hour2[0] = 2;
    span.addRow(new KeyValue(HOUR1, TSDB.FAMILY(), qualifiers.get(0), values.get(0)));
    span.addRow(new KeyValue(HOUR1, TSDB.FAMILY(), qualifiers.get(1), values.get(1)));
    span.addRow(new KeyValue(hour2, TSDB.FAMILY(), qualifiers.get(0), values.get(0)));
    span.addRow(new KeyValue(hour2, TSDB.FAMILY(), qualifiers.get(1), values.get(1)));
    assertEquals(4, span.size());
    assertEquals(1356998400000L, span.timestamp(0));
    assertEquals(4, span.longValue(0));
    assertEquals(1356998402000L, span.timestamp(1));
    assertEquals(5, span.longValue(1));
    assertEquals(1357002000000L, span.timestamp(2));
    assertEquals(4, span.longValue(2));
    assertEquals(1357002002000L, span.timestamp(3));
    assertEquals(5, span.longValue(3));
  }
  
  @Test
  public void addDifferentSaltDiffHourOO() throws Exception {
    when(tsdb.followAppendRowLogic()).thenReturn(true);
    List<byte[]> qualifiers = new ArrayList<byte[]>();
    List<byte[]> values = new ArrayList<byte[]>();


    qualifiers.add(new byte[]{ 0x73, 0x75, 0x6D, 0x3A, 0x00, 0x07 });
    values.add(Bytes.fromLong(4L));
    qualifiers.add(new byte[]{ 0x73, 0x75, 0x6D, 0x3A, 0x00, 0x27 });
    values.add(Bytes.fromLong(5L));
    
    final Span span = new RollupSpan(tsdb, rollup_query);
    final byte[] hour2 = Arrays.copyOf(HOUR2, HOUR2.length);
    hour2[0] = 2;
    span.addRow(new KeyValue(hour2, TSDB.FAMILY(), qualifiers.get(0), values.get(0)));
    span.addRow(new KeyValue(hour2, TSDB.FAMILY(), qualifiers.get(1), values.get(1)));
    span.addRow(new KeyValue(HOUR1, TSDB.FAMILY(), qualifiers.get(0), values.get(0)));
    span.addRow(new KeyValue(HOUR1, TSDB.FAMILY(), qualifiers.get(1), values.get(1)));

    assertEquals(4, span.size());
    assertEquals(1356998400000L, span.timestamp(0));
    assertEquals(4, span.longValue(0));
    assertEquals(1356998402000L, span.timestamp(1));
    assertEquals(5, span.longValue(1));
    assertEquals(1357002000000L, span.timestamp(2));
    assertEquals(4, span.longValue(2));
    assertEquals(1357002002000L, span.timestamp(3));
    assertEquals(5, span.longValue(3));
  }
  
  @Test (expected = IllegalArgumentException.class)
  public void addDifferentKey() throws Exception {
    when(tsdb.followAppendRowLogic()).thenReturn(true);
    List<byte[]> qualifiers = new ArrayList<byte[]>();
    List<byte[]> values = new ArrayList<byte[]>();
  
    qualifiers.add(new byte[]{ 0x73, 0x75, 0x6D, 0x3A, 0x00, 0x07 });
    values.add(Bytes.fromLong(4L));
    qualifiers.add(new byte[]{ 0x73, 0x75, 0x6D, 0x3A, 0x00, 0x27 });
    values.add(Bytes.fromLong(5L));
    qualifiers.add(new byte[]{ 0x73, 0x75, 0x6D, 0x3A, 0x00, 0x37 });
    qualifiers.add(new byte[]{ 0x73, 0x75, 0x6D, 0x3A, 0x00, 0x47 });
    
    final Span span = new RollupSpan(tsdb, rollup_query);
    final byte[] hour1 = Arrays.copyOf(HOUR1, HOUR1.length);
    hour1[hour1.length - 1] = 3;
    span.addRow(new KeyValue(HOUR1, TSDB.FAMILY(), qualifiers.get(0), values.get(0)));
    span.addRow(new KeyValue(HOUR1, TSDB.FAMILY(), qualifiers.get(1), values.get(1)));
    span.addRow(new KeyValue(hour1, TSDB.FAMILY(), qualifiers.get(2), values.get(0)));
    span.addRow(new KeyValue(hour1, TSDB.FAMILY(), qualifiers.get(3), values.get(1)));
  }
  
  @Test (expected = IllegalArgumentException.class)
  public void addDifferentSaltAndKey() throws Exception {
    when(tsdb.followAppendRowLogic()).thenReturn(true);
    List<byte[]> qualifiers = new ArrayList<byte[]>();
    List<byte[]> values = new ArrayList<byte[]>();
  
  
    qualifiers.add(new byte[]{ 0x73, 0x75, 0x6D, 0x3A, 0x00, 0x07 });
    values.add(Bytes.fromLong(4L));
    qualifiers.add(new byte[]{ 0x73, 0x75, 0x6D, 0x3A, 0x00, 0x27 });
    values.add(Bytes.fromLong(5L));
    qualifiers.add(new byte[]{ 0x73, 0x75, 0x6D, 0x3A, 0x00, 0x37 });
    qualifiers.add(new byte[]{ 0x73, 0x75, 0x6D, 0x3A, 0x00, 0x47 });
    
    final Span span = new RollupSpan(tsdb, rollup_query);
    final byte[] hour1 = Arrays.copyOf(HOUR1, HOUR1.length);
    hour1[0] = 2;
    hour1[hour1.length - 1] = 3;
    span.addRow(new KeyValue(HOUR1, TSDB.FAMILY(), qualifiers.get(0), values.get(0)));
    span.addRow(new KeyValue(HOUR1, TSDB.FAMILY(), qualifiers.get(1), values.get(1)));
    span.addRow(new KeyValue(hour1, TSDB.FAMILY(), qualifiers.get(2), values.get(0)));
    span.addRow(new KeyValue(hour1, TSDB.FAMILY(), qualifiers.get(3), values.get(1)));
  }
  */
@Test
public void timestampNormalized() throws Exception {
    final byte[] qual1 = { 0x73, 0x75, 0x6D, 0x3A, 0x00, 0x07 };
    final byte[] val1 = Bytes.fromLong(4L);
    final byte[] qual2 = { 0x73, 0x75, 0x6D, 0x3A, 0x00, 0x27 };
    final byte[] val2 = Bytes.fromLong(5L);
    final Span span = new RollupSpan(tsdb, rollup_query);
    span.addRow(new KeyValue(hour1, TSDB.FAMILY(), qual1, val1));
    span.addRow(new KeyValue(hour1, TSDB.FAMILY(), qual2, val2));
    span.addRow(new KeyValue(hour2, TSDB.FAMILY(), qual1, val1));
    span.addRow(new KeyValue(hour2, TSDB.FAMILY(), qual2, val2));
    span.addRow(new KeyValue(hour3, TSDB.FAMILY(), qual1, val1));
    span.addRow(new KeyValue(hour3, TSDB.FAMILY(), qual2, val2));
    assertEquals(6, span.size());
    assertEquals(1356998400000L, span.timestamp(0));
    assertEquals(1356998402000L, span.timestamp(1));
    assertEquals(1357002000000L, span.timestamp(2));
    assertEquals(1357002002000L, span.timestamp(3));
    assertEquals(1357005600000L, span.timestamp(4));
    assertEquals(1357005602000L, span.timestamp(5));
}
Also used : KeyValue(org.hbase.async.KeyValue) RollupSpan(net.opentsdb.rollup.RollupSpan) RollupSpan(net.opentsdb.rollup.RollupSpan) Test(org.junit.Test)

Example 5 with RollupSpan

use of net.opentsdb.rollup.RollupSpan in project opentsdb by OpenTSDB.

the class TestRollupSpan method addRowNull.

@Test(expected = NullPointerException.class)
public void addRowNull() {
    final Span span = new RollupSpan(tsdb, rollup_query);
    span.addRow(null);
}
Also used : RollupSpan(net.opentsdb.rollup.RollupSpan) RollupSpan(net.opentsdb.rollup.RollupSpan) Test(org.junit.Test)

Aggregations

RollupSpan (net.opentsdb.rollup.RollupSpan)8 KeyValue (org.hbase.async.KeyValue)7 Test (org.junit.Test)5 Annotation (net.opentsdb.meta.Annotation)2
About Me
UseOf

Java Tutorials :

Simple Java RabbitMQ Example with Spring
Simple Springboot JPA Eclipeslink Example
Simple SpringBoot JPA Hibernate Example
Jackson JSON @JsonIgnore and @JsonIgnoreProperties Examples
Java Infinite recursion (StackOverflowError) Jackson solutions
Simple Java Jackson Serialize Objects to JSON and convert them back To Object
ElasticSearch 5 - Upload files using Java API in binary field Example
Java JAXB marshall unmarshall Examples from String and Stream
Java 8 forEach List and Map examples
ElasticSearch 5 Java API SearchRequestBuilder examples
Java ElasticSearch 5 examples with node index and mapping - running local
Convert String to int or Integer Java 8
Java 9 in action - JShell - Ubuntu
Java - removing invalid characters from a file name
Hello world!? or Hello Tutorial