Search in sources :

Example 61 with Instant

use of de.invesdwin.util.time.Instant in project invesdwin-context-persistence by subes.

the class ADelegateDailyDownloadRangeTableRequest method getIterator.

public ICloseableIterator<V> getIterator() {
    try {
        final Instant start = new Instant();
        log.info("Starting download [%s] ...", getDownloadFileName());
        final InputStream content = dailyDownloadCache.downloadStream(getDownloadFileName(), new Consumer<OutputStream>() {

            @Override
            public void accept(final OutputStream t) {
                try (InputStream in = download()) {
                    IOUtils.copy(in, t);
                } catch (final Exception e) {
                    throw new RuntimeException(e);
                }
            }
        }, getNow());
        final ICloseableIterator<V> reader = newReader(content);
        log.info("Finished download [%s] after: %s", getDownloadFileName(), start);
        return reader;
    } catch (final Exception e) {
        throw new RuntimeException(e);
    }
}
Also used : InputStream(java.io.InputStream) Instant(de.invesdwin.util.time.Instant) OutputStream(java.io.OutputStream) NoSuchElementException(java.util.NoSuchElementException)

Example 62 with Instant

use of de.invesdwin.util.time.Instant in project invesdwin-context-persistence by subes.

the class ADelegateDailyDownloadRangeTableRequest method maybeUpdate.

protected void maybeUpdate() {
    if (shouldUpdate()) {
        synchronized (this) {
            try {
                if (shouldUpdate()) {
                    beforeUpdate();
                    final ICloseableIterator<V> reader = getIterator();
                    final Instant start = new Instant();
                    log.info("Starting indexing [%s] ...", getDownloadFileName());
                    final LoopInterruptedCheck loopCheck = new LoopInterruptedCheck(Duration.ONE_SECOND);
                    try (Batch<K, V> batch = table.newBatch()) {
                        int count = 0;
                        try {
                            while (true) {
                                final V value = reader.next();
                                final K key = extractKey(value);
                                batch.put(key, value);
                                count++;
                                if (count >= ADelegateRangeTable.BATCH_FLUSH_INTERVAL) {
                                    batch.flush();
                                    if (loopCheck.check()) {
                                        printProgress("Indexing [" + getDownloadFileName() + "]", start, count);
                                    }
                                }
                            }
                        } catch (final NoSuchElementException e) {
                        // end reached
                        }
                        if (count > 0) {
                            batch.flush();
                        }
                    }
                    afterUpdate();
                    log.info("Finished indexing [%s] after: %s", getDownloadFileName(), start);
                }
            } catch (final Throwable t) {
                deleteDownloadedFile();
                table.deleteTable();
                throw Throwables.propagate(t);
            }
        }
    }
}
Also used : Instant(de.invesdwin.util.time.Instant) LoopInterruptedCheck(de.invesdwin.util.concurrent.loop.LoopInterruptedCheck) NoSuchElementException(java.util.NoSuchElementException)

Example 63 with Instant

use of de.invesdwin.util.time.Instant in project invesdwin-context-persistence by subes.

the class PersistentEzdbPerformanceTest method testPersistentEzdbPerformance.

@Test
public void testPersistentEzdbPerformance() throws InterruptedException {
    @SuppressWarnings("resource") final APersistentMap<FDate, FDate> table = new APersistentMap<FDate, FDate>("testPersistentEzdbPerformance") {

        @Override
        public File getBaseDirectory() {
            return ContextProperties.TEMP_DIRECTORY;
        }

        @Override
        public ISerde<FDate> newKeySerde() {
            return FDateSerde.GET;
        }

        @Override
        public ISerde<FDate> newValueSerde() {
            return FDateSerde.GET;
        }

        @Override
        protected IPersistentMapFactory<FDate, FDate> newFactory() {
            return new PersistentEzdbMapFactory<>();
        }
    };
    final LoopInterruptedCheck loopCheck = new LoopInterruptedCheck(Duration.ONE_SECOND);
    final Instant writesStart = new Instant();
    int i = 0;
    for (final FDate date : newValues()) {
        table.put(date, date);
        i++;
        if (i % FLUSH_INTERVAL == 0) {
            if (loopCheck.check()) {
                printProgress("Writes", writesStart, i, VALUES);
            }
        }
    }
    printProgress("WritesFinished", writesStart, VALUES, VALUES);
    readIterator(table);
    readGet(table);
    table.deleteTable();
}
Also used : APersistentMap(de.invesdwin.context.integration.persistentmap.APersistentMap) PersistentEzdbMapFactory(de.invesdwin.context.persistence.ezdb.PersistentEzdbMapFactory) Instant(de.invesdwin.util.time.Instant) LoopInterruptedCheck(de.invesdwin.util.concurrent.loop.LoopInterruptedCheck) FDate(de.invesdwin.util.time.date.FDate) Test(org.junit.jupiter.api.Test)

Example 64 with Instant

use of de.invesdwin.util.time.Instant in project invesdwin-context-persistence by subes.

the class PersistentEzdbPerformanceTest method readIterator.

private void readIterator(final APersistentMap<FDate, FDate> table) throws InterruptedException {
    final LoopInterruptedCheck loopCheck = new LoopInterruptedCheck(Duration.ONE_SECOND);
    final Instant readsStart = new Instant();
    for (int reads = 1; reads <= READS; reads++) {
        // FDate prevValue = null;
        final Iterator<FDate> range = table.values().iterator();
        int count = 0;
        while (true) {
            try {
                final FDate value = range.next();
                // if (prevValue != null) {
                // Assertions.checkTrue(prevValue.isBefore(value));
                // }
                // prevValue = value;
                count++;
            } catch (final NoSuchElementException e) {
                break;
            }
        }
        Assertions.checkEquals(count, VALUES);
        if (loopCheck.check()) {
            printProgress("Reads", readsStart, VALUES * reads, VALUES * READS);
        }
    }
    printProgress("ReadsFinished", readsStart, VALUES * READS, VALUES * READS);
}
Also used : Instant(de.invesdwin.util.time.Instant) LoopInterruptedCheck(de.invesdwin.util.concurrent.loop.LoopInterruptedCheck) FDate(de.invesdwin.util.time.date.FDate) NoSuchElementException(java.util.NoSuchElementException)

Example 65 with Instant

use of de.invesdwin.util.time.Instant in project invesdwin-context-persistence by subes.

the class BasicRecordFilePerformanceTest method testBasicRecordFilePerformance.

@Test
public void testBasicRecordFilePerformance() throws IOException, InterruptedException {
    final File directory = new File(ContextProperties.getCacheDirectory(), BasicRecordFilePerformanceTest.class.getSimpleName());
    Files.deleteNative(directory);
    Files.forceMkdir(directory);
    final File file = new File(directory, "testBasicRecordFilePerformance");
    final LoopInterruptedCheck loopCheck = new LoopInterruptedCheck(Duration.ONE_SECOND);
    final Instant writesStart = new Instant();
    try (BasicRecordFile.Writer<TimedDecimal> writer = new BasicRecordFile.Writer<>(file, IndeedSerializer.valueOf(TimedDecimalSerde.GET))) {
        int i = 0;
        for (final FDate date : newValues()) {
            writer.append(new TimedDecimal(date, date.millisValue()));
            i++;
            if (i % FLUSH_INTERVAL == 0) {
                if (loopCheck.check()) {
                    printProgress("Writes", writesStart, i, VALUES);
                }
            }
        }
        printProgress("WritesFinished", writesStart, VALUES, VALUES);
    }
    readIterator(file);
}
Also used : BasicRecordFile(com.indeed.lsmtree.recordlog.BasicRecordFile) Instant(de.invesdwin.util.time.Instant) LoopInterruptedCheck(de.invesdwin.util.concurrent.loop.LoopInterruptedCheck) BasicRecordFile(com.indeed.lsmtree.recordlog.BasicRecordFile) File(java.io.File) RecordFile(com.indeed.lsmtree.recordlog.RecordFile) TimedDecimal(de.invesdwin.util.math.decimal.TimedDecimal) FDate(de.invesdwin.util.time.date.FDate) Test(org.junit.jupiter.api.Test)

Aggregations

Instant (de.invesdwin.util.time.Instant)93 FDate (de.invesdwin.util.time.date.FDate)78 LoopInterruptedCheck (de.invesdwin.util.concurrent.loop.LoopInterruptedCheck)65 NoSuchElementException (java.util.NoSuchElementException)30 Test (org.junit.jupiter.api.Test)25 File (java.io.File)16 PreparedStatement (java.sql.PreparedStatement)16 ResultSet (java.sql.ResultSet)15 Statement (java.sql.Statement)12 ATest (de.invesdwin.context.test.ATest)8 RecordFile (com.indeed.lsmtree.recordlog.RecordFile)6 ProcessedEventsRateString (de.invesdwin.util.lang.ProcessedEventsRateString)6 TimedDecimal (de.invesdwin.util.math.decimal.TimedDecimal)5 FDate (de.invesdwin.util.time.fdate.FDate)5 IOException (java.io.IOException)5 Connection (java.sql.Connection)5 APersistentMap (de.invesdwin.context.integration.persistentmap.APersistentMap)4 ATimeSeriesUpdater (de.invesdwin.context.persistence.timeseriesdb.updater.ATimeSeriesUpdater)4 Test (org.junit.Test)4 SnappyCodec (com.indeed.util.compress.SnappyCodec)3