Examples with Path cz.o2.proxima.direct.bulk.Path used on opensource projects

Example 1 with Path

use of cz.o2.proxima.direct.bulk.Path in project proxima-platform by O2-Czech-Republic.

the class HadoopFileSystem method list.

@Override
public Stream<Path> list(long minTs, long maxTs) {
    URI remappedUri = HadoopStorage.remap(getUri());
    RemoteIterator<LocatedFileStatus> iterator = ExceptionUtils.uncheckedFactory(() -> fs().listFiles(new org.apache.hadoop.fs.Path(remappedUri), true));
    Spliterator<LocatedFileStatus> spliterator = asSpliterator(iterator);
    return StreamSupport.stream(spliterator, false).filter(LocatedFileStatus::isFile).map(f -> f.getPath().toUri().toString().substring(remappedUri.toString().length())).filter(name -> namingConvention.isInRange(name, minTs, maxTs)).map(name -> HadoopPath.of(this, remappedUri + name, accessor));
}

Example 2 with Path

use of cz.o2.proxima.direct.bulk.Path in project proxima-platform by O2-Czech-Republic.

the class HadoopStorageTest method testHashCodeAndEquals.

@Test
public void testHashCodeAndEquals() {
    TestUtils.assertHashCodeAndEquals(new HadoopStorage(), new HadoopStorage());
    EntityDescriptor entity = EntityDescriptor.newBuilder().setName("dummy").build();
    final AttributeFamilyDescriptor family = TestUtils.createTestFamily(entity, URI.create("hdfs://host:9000/path"));
    TestUtils.assertHashCodeAndEquals(new HadoopDataAccessor(family), new HadoopDataAccessor(family));
}

Example 3 with Path

use of cz.o2.proxima.direct.bulk.Path in project proxima-platform by O2-Czech-Republic.

the class HadoopBatchLogReader method processPath.

private boolean processPath(BatchLogObserver observer, long watermark, HadoopPartition partition, HadoopPath path, TerminationContext terminationContext) {
    try {
        try (Reader reader = accessor.getFormat().openReader(path, accessor.getEntityDesc())) {
            long elementIndex = 0;
            final Iterator<StreamElement> iterator = reader.iterator();
            while (iterator.hasNext()) {
                final StreamElement element = iterator.next();
                final Offset offset = Offset.of(partition, elementIndex++, !iterator.hasNext());
                if (terminationContext.isCancelled() || !observer.onNext(element, BatchLogObservers.withWatermark(partition, offset, watermark))) {
                    return false;
                }
            }
        }
    } catch (IOException ex) {
        throw new RuntimeException("Failed to read file " + partition, ex);
    }
    return true;
}

Example 4 with Path

use of cz.o2.proxima.direct.bulk.Path in project proxima-platform by O2-Czech-Republic.

the class S3AccessorTest method testNamingConventionWithBucket.

@Test
public void testNamingConventionWithBucket() {
    S3Accessor accessor = new S3Accessor(TestUtils.createTestFamily(entity, URI.create("s3://bucket/path")));
    NamingConvention convention = accessor.getNamingConvention();
    assertTrue(convention.nameOf(1500000000000L).startsWith("/2017/07/"));
}

Example 5 with Path

use of cz.o2.proxima.direct.bulk.Path in project proxima-platform by O2-Czech-Republic.

the class S3AccessorTest method testNamingConventionWithBucketAndNoPath.

@Test
public void testNamingConventionWithBucketAndNoPath() {
    S3Accessor accessor = new S3Accessor(TestUtils.createTestFamily(entity, URI.create("s3://bucket/path")));
    NamingConvention convention = accessor.getNamingConvention();
    assertTrue(convention.nameOf(1500000000000L).startsWith("/2017/07/"));
}