Search in sources :

Example 1 with DefaultNamingConvention

use of cz.o2.proxima.direct.bulk.DefaultNamingConvention in project proxima-platform by O2-Czech-Republic.

the class ComplexWriteTest method initWriter.

AbstractBulkFileSystemAttributeWriter initWriter() throws IOException {
    FileFormat format = FileFormatUtils.getFileFormatFromName("binary", false);
    NamingConvention naming = new DefaultNamingConvention(Duration.ofMillis(rollPeriod), "prefix", "suffix");
    FileSystem fs = FileSystem.local(new File(tempFolder.newFolder(), UUID.randomUUID().toString()), naming);
    return new AbstractBulkFileSystemAttributeWriter(entity, uri, fs, naming, format, direct.getContext(), rollPeriod, allowedLateness) {

        @Override
        public BulkAttributeWriter.Factory<?> asFactory() {
            return repo -> ExceptionUtils.uncheckedFactory(() -> initWriter());
        }

        @Override
        protected void flush(Bulk v) {
            try {
                List<StreamElement> elements = Lists.newArrayList(format.openReader(v.getPath(), entity));
                flushed.addAndGet(elements.size());
                log.info("Written {} elements to stamp {}", elements.size(), v.getMaxTs());
            } catch (IOException ex) {
                throw new RuntimeException(ex);
            }
        }
    };
}
Also used : IntStream(java.util.stream.IntStream) URISyntaxException(java.net.URISyntaxException) EntityDescriptor(cz.o2.proxima.repository.EntityDescriptor) Random(java.util.Random) ArrayList(java.util.ArrayList) ExceptionUtils(cz.o2.proxima.util.ExceptionUtils) Lists(com.google.common.collect.Lists) StreamElement(cz.o2.proxima.storage.StreamElement) AtomicInteger(java.util.concurrent.atomic.AtomicInteger) Duration(java.time.Duration) ConfigFactory(com.typesafe.config.ConfigFactory) URI(java.net.URI) Before(org.junit.Before) BulkAttributeWriter(cz.o2.proxima.direct.core.BulkAttributeWriter) Repository(cz.o2.proxima.repository.Repository) AttributeDescriptor(cz.o2.proxima.repository.AttributeDescriptor) IOException(java.io.IOException) Test(org.junit.Test) UUID(java.util.UUID) Collectors(java.util.stream.Collectors) File(java.io.File) Serializable(java.io.Serializable) TimeUnit(java.util.concurrent.TimeUnit) AtomicLong(java.util.concurrent.atomic.AtomicLong) List(java.util.List) Slf4j(lombok.extern.slf4j.Slf4j) Rule(org.junit.Rule) DirectDataOperator(cz.o2.proxima.direct.core.DirectDataOperator) Assert(org.junit.Assert) Collections(java.util.Collections) TemporaryFolder(org.junit.rules.TemporaryFolder) StreamElement(cz.o2.proxima.storage.StreamElement) IOException(java.io.IOException) BulkAttributeWriter(cz.o2.proxima.direct.core.BulkAttributeWriter) File(java.io.File)

Example 2 with DefaultNamingConvention

use of cz.o2.proxima.direct.bulk.DefaultNamingConvention in project proxima-platform by O2-Czech-Republic.

the class BulkBlobWriterTest method initWriter.

void initWriter(Map<String, Object> cfg) {
    accessor = new TestBlobStorageAccessor(TestUtils.createTestFamily(entity, URI.create("blob-test://bucket/path"), cfg), () -> Optional.ofNullable(latch.get()).ifPresent(CountDownLatch::countDown), onFlushToBlob) {

        @Override
        public NamingConvention getNamingConvention() {
            return new DefaultNamingConvention(Duration.ofMillis(getRollPeriod()), "prefix", "suffix", () -> "uuid");
        }
    };
    writer = accessor.new BlobWriter(direct.getContext());
}
Also used : BlobWriter(cz.o2.proxima.direct.blob.TestBlobStorageAccessor.BlobWriter) NamingConvention(cz.o2.proxima.direct.bulk.NamingConvention) DefaultNamingConvention(cz.o2.proxima.direct.bulk.DefaultNamingConvention) DefaultNamingConvention(cz.o2.proxima.direct.bulk.DefaultNamingConvention) CountDownLatch(java.util.concurrent.CountDownLatch)

Aggregations

Lists (com.google.common.collect.Lists)1 ConfigFactory (com.typesafe.config.ConfigFactory)1 BlobWriter (cz.o2.proxima.direct.blob.TestBlobStorageAccessor.BlobWriter)1 DefaultNamingConvention (cz.o2.proxima.direct.bulk.DefaultNamingConvention)1 NamingConvention (cz.o2.proxima.direct.bulk.NamingConvention)1 BulkAttributeWriter (cz.o2.proxima.direct.core.BulkAttributeWriter)1 DirectDataOperator (cz.o2.proxima.direct.core.DirectDataOperator)1 AttributeDescriptor (cz.o2.proxima.repository.AttributeDescriptor)1 EntityDescriptor (cz.o2.proxima.repository.EntityDescriptor)1 Repository (cz.o2.proxima.repository.Repository)1 StreamElement (cz.o2.proxima.storage.StreamElement)1 ExceptionUtils (cz.o2.proxima.util.ExceptionUtils)1 File (java.io.File)1 IOException (java.io.IOException)1 Serializable (java.io.Serializable)1 URI (java.net.URI)1 URISyntaxException (java.net.URISyntaxException)1 Duration (java.time.Duration)1 ArrayList (java.util.ArrayList)1 Collections (java.util.Collections)1