Search in sources :

Example 6 with LinkedByteString

use of com.srotya.sidewinder.core.storage.LinkedByteString in project sidewinder by srotya.

the class DiskMalloc method appendBufferPointersToDisk.

protected LinkedByteString appendBufferPointersToDisk(LinkedByteString seriesId, ByteString filename, int curr, long offset, int size, Integer tsBucket) throws IOException {
    lock.lock();
    try {
        ByteString[] split = filename.split("/");
        LinkedByteString line = new LinkedByteString(BUF_PARTS_LENGTH);
        line.concat(seriesId).concat(SEPARATOR).concat(cache.get(split[split.length - 1])).concat(SEPARATOR).concat(String.valueOf(curr)).concat(SEPARATOR).concat(String.valueOf(offset)).concat(SEPARATOR).concat(String.valueOf(size)).concat(SEPARATOR).concat(Integer.toHexString(tsBucket)).concat("\n");
        logger.fine("Measurement(" + measurementName + ")Appending pointer information to ptr file:" + line);
        // resize
        String strLine = line.toString();
        byte[] bytes = strLine.getBytes();
        if (ptrBuf.remaining() < bytes.length + Short.BYTES) {
            logger.fine("Need to resize ptrbuf because ptrBufRem:" + ptrBuf.remaining() + " line:" + bytes.length);
            int pos = ptrBuf.position();
            int newSize = pos + ptrFileIncrement;
            ptrBuf.force();
            ptrBuf = rafPtr.getChannel().map(MapMode.READ_WRITE, 0, newSize);
            // BUGFIX: missing pointer reset caused PTR file corruption
            ptrBuf.position(pos);
            logger.info("Resizing ptr file:" + ptrBuf.getInt(0) + " ptrcount:" + ptrCounter + " inc:" + ptrFileIncrement + " position:" + pos);
        }
        MiscUtils.writeStringToBuffer(strLine, ptrBuf);
        ptrBuf.putInt(0, ++ptrCounter);
        logger.fine(() -> "Measurement(" + measurementName + ") appending pointer information to ptr file:" + line + " pos:" + ptrBuf.position());
        return line;
    } finally {
        lock.unlock();
    }
}
Also used : LinkedByteString(com.srotya.sidewinder.core.storage.LinkedByteString) ByteString(com.srotya.sidewinder.core.storage.ByteString) LinkedByteString(com.srotya.sidewinder.core.storage.LinkedByteString) LinkedByteString(com.srotya.sidewinder.core.storage.LinkedByteString) ByteString(com.srotya.sidewinder.core.storage.ByteString)

Example 7 with LinkedByteString

use of com.srotya.sidewinder.core.storage.LinkedByteString in project sidewinder by srotya.

the class TestDownSamplingIFunction method testAverageDownsampling.

@Test
public void testAverageDownsampling() throws IOException {
    Field tField = new TimeField(measurement, new LinkedByteString().concat(new ByteString("time")), 121213, new HashMap<>());
    long ts = 1546755994280L;
    for (int i = 0; i < 100; i++) {
        tField.addDataPoint(measurement, ts + i * 1000);
    }
    Field vField = new ValueField(measurement, new LinkedByteString().concat(new ByteString("field1")), 121213, new HashMap<>());
    for (int i = 0; i < 100; i++) {
        vField.addDataPoint(measurement, 1L);
    }
    DataPointIterator itr = new DataPointIterator(tField.queryReader(null, new NoLock()), vField.queryReader(null, new NoLock()));
    DownsampleFunction f = new DownsampleFunction(itr, 5, TimeUnit.SECONDS, ((x, y) -> (x + y)));
    int c = 0;
    while (f.hasNext()) {
        DataPoint next = f.next();
        if (c < 20) {
            assertEquals(ts + c * 1000 * 5, next.getTimestamp());
            assertEquals(5, next.getLongValue());
        }
        c++;
    }
    assertEquals(21, c);
    itr = new DataPointIterator(tField.queryReader(null, new NoLock()), vField.queryReader(null, new NoLock()));
    f = new DownsampleFunction(itr, 10, TimeUnit.SECONDS, ((x, y) -> (x + y) / 2));
    c = 0;
    while (f.hasNext()) {
        DataPoint next = f.next();
        if (c < 10) {
            assertEquals(ts + c * 1000 * 10, next.getTimestamp());
            assertEquals(1, next.getLongValue());
        }
        c++;
    }
    assertEquals(10, c, 1);
}
Also used : DownsampleFunction(com.srotya.sidewinder.core.functions.iterative.DownsampleFunction) NoLock(com.srotya.sidewinder.core.storage.NoLock) IOException(java.io.IOException) HashMap(java.util.HashMap) Test(org.junit.Test) DataPoint(com.srotya.sidewinder.core.storage.DataPoint) LinkedByteString(com.srotya.sidewinder.core.storage.LinkedByteString) TimeUnit(java.util.concurrent.TimeUnit) Field(com.srotya.sidewinder.core.storage.Field) ValueField(com.srotya.sidewinder.core.storage.ValueField) CompressionFactory(com.srotya.sidewinder.core.storage.compression.CompressionFactory) ByteString(com.srotya.sidewinder.core.storage.ByteString) MockMeasurement(com.srotya.sidewinder.core.storage.MockMeasurement) Assert.assertEquals(org.junit.Assert.assertEquals) DataPointIterator(com.srotya.sidewinder.core.storage.DataPointIterator) TimeField(com.srotya.sidewinder.core.storage.TimeField) Before(org.junit.Before) LinkedByteString(com.srotya.sidewinder.core.storage.LinkedByteString) ByteString(com.srotya.sidewinder.core.storage.ByteString) DataPoint(com.srotya.sidewinder.core.storage.DataPoint) Field(com.srotya.sidewinder.core.storage.Field) ValueField(com.srotya.sidewinder.core.storage.ValueField) TimeField(com.srotya.sidewinder.core.storage.TimeField) TimeField(com.srotya.sidewinder.core.storage.TimeField) DataPointIterator(com.srotya.sidewinder.core.storage.DataPointIterator) DataPoint(com.srotya.sidewinder.core.storage.DataPoint) NoLock(com.srotya.sidewinder.core.storage.NoLock) LinkedByteString(com.srotya.sidewinder.core.storage.LinkedByteString) ValueField(com.srotya.sidewinder.core.storage.ValueField) DownsampleFunction(com.srotya.sidewinder.core.functions.iterative.DownsampleFunction) Test(org.junit.Test)

Example 8 with LinkedByteString

use of com.srotya.sidewinder.core.storage.LinkedByteString in project sidewinder by srotya.

the class TestTumblingWindowFunction method testBasicMinDownSampling.

@Test
public void testBasicMinDownSampling() throws Exception {
    Field tField = new TimeField(measurement, new LinkedByteString().concat(new ByteString("time")), 121213, new HashMap<>());
    long ts = 1546755991280L;
    for (int i = 0; i < 100; i++) {
        tField.addDataPoint(measurement, ts + i * 1000);
    }
    Field vField = new ValueField(measurement, new LinkedByteString().concat(new ByteString("field1")), 121213, new HashMap<>());
    for (int i = 0; i < 100; i++) {
        vField.addDataPoint(measurement, i);
    }
    DataPointIterator itr = new DataPointIterator(tField.queryReader(null, new NoLock()), vField.queryReader(null, new NoLock()));
    TumblingWindowFunction tumblingWindowFunction = new TumblingWindowFunction(itr, false);
    tumblingWindowFunction.init(new Object[] { 10, "smin" });
    int c = 0;
    while (tumblingWindowFunction.hasNext()) {
        tumblingWindowFunction.next();
        c++;
    }
    assertEquals(11, c);
}
Also used : Field(com.srotya.sidewinder.core.storage.Field) ValueField(com.srotya.sidewinder.core.storage.ValueField) TimeField(com.srotya.sidewinder.core.storage.TimeField) TimeField(com.srotya.sidewinder.core.storage.TimeField) DataPointIterator(com.srotya.sidewinder.core.storage.DataPointIterator) LinkedByteString(com.srotya.sidewinder.core.storage.LinkedByteString) ByteString(com.srotya.sidewinder.core.storage.ByteString) NoLock(com.srotya.sidewinder.core.storage.NoLock) LinkedByteString(com.srotya.sidewinder.core.storage.LinkedByteString) ValueField(com.srotya.sidewinder.core.storage.ValueField) Test(org.junit.Test)

Example 9 with LinkedByteString

use of com.srotya.sidewinder.core.storage.LinkedByteString in project sidewinder by srotya.

the class TestTumblingWindowFunction method testBasicSumDownSampling.

@Test
public void testBasicSumDownSampling() throws Exception {
    Field tField = new TimeField(measurement, new LinkedByteString().concat(new ByteString("time")), 121213, new HashMap<>());
    long ts = 1546755991280L;
    for (int i = 0; i < 100; i++) {
        tField.addDataPoint(measurement, ts + i * 1000);
    }
    Field vField = new ValueField(measurement, new LinkedByteString().concat(new ByteString("field1")), 121213, new HashMap<>());
    for (int i = 0; i < 100; i++) {
        vField.addDataPoint(measurement, 1L);
    }
    DataPointIterator itr = new DataPointIterator(tField.queryReader(null, new NoLock()), vField.queryReader(null, new NoLock()));
    TumblingWindowFunction tumblingWindowFunction = new TumblingWindowFunction(itr, false);
    tumblingWindowFunction.init(new Object[] { 10, "ssum" });
    int c = 0;
    while (tumblingWindowFunction.hasNext()) {
        tumblingWindowFunction.next();
        c++;
    }
    assertEquals(11, c);
}
Also used : Field(com.srotya.sidewinder.core.storage.Field) ValueField(com.srotya.sidewinder.core.storage.ValueField) TimeField(com.srotya.sidewinder.core.storage.TimeField) TimeField(com.srotya.sidewinder.core.storage.TimeField) DataPointIterator(com.srotya.sidewinder.core.storage.DataPointIterator) LinkedByteString(com.srotya.sidewinder.core.storage.LinkedByteString) ByteString(com.srotya.sidewinder.core.storage.ByteString) NoLock(com.srotya.sidewinder.core.storage.NoLock) LinkedByteString(com.srotya.sidewinder.core.storage.LinkedByteString) ValueField(com.srotya.sidewinder.core.storage.ValueField) Test(org.junit.Test)

Aggregations

LinkedByteString (com.srotya.sidewinder.core.storage.LinkedByteString)9 ByteString (com.srotya.sidewinder.core.storage.ByteString)8 DataPointIterator (com.srotya.sidewinder.core.storage.DataPointIterator)5 Field (com.srotya.sidewinder.core.storage.Field)5 NoLock (com.srotya.sidewinder.core.storage.NoLock)5 TimeField (com.srotya.sidewinder.core.storage.TimeField)5 ValueField (com.srotya.sidewinder.core.storage.ValueField)5 Test (org.junit.Test)4 BufferObject (com.srotya.sidewinder.core.storage.BufferObject)3 DataPoint (com.srotya.sidewinder.core.storage.DataPoint)3 MockMeasurement (com.srotya.sidewinder.core.storage.MockMeasurement)3 ByteBuffer (java.nio.ByteBuffer)3 Before (org.junit.Before)3 DownsampleFunction (com.srotya.sidewinder.core.functions.iterative.DownsampleFunction)2 CompressionFactory (com.srotya.sidewinder.core.storage.compression.CompressionFactory)2 IOException (java.io.IOException)2 MappedByteBuffer (java.nio.MappedByteBuffer)2 HashMap (java.util.HashMap)2 TimeUnit (java.util.concurrent.TimeUnit)2 Assert.assertEquals (org.junit.Assert.assertEquals)2