Search in sources :

Example 1 with OsmPbfParser

use of com.baremaps.osm.pbf.OsmPbfParser in project baremaps by baremaps.

the class OpenStreetMapTest method monacoOsmPbf.

@Test
void monacoOsmPbf() throws IOException, URISyntaxException {
    try (InputStream inputStream = MONACO_OSM_PBF.openStream()) {
        Stream<Entity> stream = new OsmPbfParser().entities(inputStream);
        process(stream, 1, 1, 25002, 4018, 243);
    }
}
Also used : Entity(com.baremaps.osm.domain.Entity) BZip2CompressorInputStream(org.apache.commons.compress.compressors.bzip2.BZip2CompressorInputStream) InputStream(java.io.InputStream) OsmPbfParser(com.baremaps.osm.pbf.OsmPbfParser) Test(org.junit.jupiter.api.Test)

Example 2 with OsmPbfParser

use of com.baremaps.osm.pbf.OsmPbfParser in project baremaps by baremaps.

the class OpenStreetMapBenchmark method entityStream.

@Benchmark
@BenchmarkMode(Mode.SingleShotTime)
@Warmup(iterations = 2)
@Measurement(iterations = 5)
public void entityStream() throws IOException {
    AtomicLong nodes = new AtomicLong(0);
    AtomicLong ways = new AtomicLong(0);
    AtomicLong relations = new AtomicLong(0);
    try (InputStream inputStream = new BufferedInputStream(Files.newInputStream(path))) {
        new OsmPbfParser().entities(inputStream).forEach(new EntityConsumerAdapter() {

            @Override
            public void match(Node node) {
                nodes.incrementAndGet();
            }

            @Override
            public void match(Way way) {
                ways.incrementAndGet();
            }

            @Override
            public void match(Relation relation) {
                relations.incrementAndGet();
            }
        });
    }
}
Also used : AtomicLong(java.util.concurrent.atomic.AtomicLong) Relation(com.baremaps.osm.domain.Relation) BufferedInputStream(java.io.BufferedInputStream) BufferedInputStream(java.io.BufferedInputStream) InputStream(java.io.InputStream) Node(com.baremaps.osm.domain.Node) EntityConsumerAdapter(com.baremaps.osm.function.EntityConsumerAdapter) Way(com.baremaps.osm.domain.Way) OsmPbfParser(com.baremaps.osm.pbf.OsmPbfParser) Measurement(org.openjdk.jmh.annotations.Measurement) Warmup(org.openjdk.jmh.annotations.Warmup) BenchmarkMode(org.openjdk.jmh.annotations.BenchmarkMode) Benchmark(org.openjdk.jmh.annotations.Benchmark)

Example 3 with OsmPbfParser

use of com.baremaps.osm.pbf.OsmPbfParser in project baremaps by baremaps.

the class ImportService method call.

@Override
public Void call() throws Exception {
    Consumer<Block> cacheBlock = new DataStoreConsumer(coordinates, references);
    Consumer<Entity> createGeometry = new CreateGeometryConsumer(coordinates, references);
    Consumer<Entity> reprojectGeometry = new ReprojectEntityConsumer(4326, srid);
    Consumer<Block> prepareGeometries = new BlockEntityConsumer(createGeometry.andThen(reprojectGeometry));
    Function<Block, Block> prepareBlock = consumeThenReturn(cacheBlock.andThen(prepareGeometries));
    Consumer<Block> saveBlock = new SaveBlockConsumer(headerRepository, nodeRepository, wayRepository, relationRepository);
    Blob blob = blobStore.get(uri);
    ProgressLogger progressLogger = new ProgressLogger(blob.getContentLength(), 5000);
    try (InputStream inputStream = new InputStreamProgress(blob.getInputStream(), progressLogger)) {
        batch(new OsmPbfParser().blocks(inputStream).map(prepareBlock)).forEach(saveBlock);
    }
    return null;
}
Also used : Entity(com.baremaps.osm.domain.Entity) CreateGeometryConsumer(com.baremaps.osm.geometry.CreateGeometryConsumer) Blob(com.baremaps.blob.Blob) InputStream(java.io.InputStream) BlockEntityConsumer(com.baremaps.osm.function.BlockEntityConsumer) InputStreamProgress(com.baremaps.osm.progress.InputStreamProgress) ProgressLogger(com.baremaps.osm.progress.ProgressLogger) ReprojectEntityConsumer(com.baremaps.osm.geometry.ReprojectEntityConsumer) Block(com.baremaps.osm.domain.Block) DataStoreConsumer(com.baremaps.osm.store.DataStoreConsumer) OsmPbfParser(com.baremaps.osm.pbf.OsmPbfParser)

Example 4 with OsmPbfParser

use of com.baremaps.osm.pbf.OsmPbfParser in project baremaps by baremaps.

the class SaveBlockConsumerTest method test.

@Test
@Tag("integration")
void test() throws BlobStoreException, RepositoryException, URISyntaxException {
    // Import data
    SaveBlockConsumer dataImporter = new SaveBlockConsumer(headerRepository, nodeRepository, tableRepository, relationRepository);
    InputStream inputStream = new ResourceBlobStore().get(new URI("res://simple/data.osm.pbf")).getInputStream();
    new OsmPbfParser().blocks(inputStream).forEach(dataImporter);
    // Check node importation
    assertNull(nodeRepository.get(0l));
    assertNotNull(nodeRepository.get(1l));
    assertNotNull(nodeRepository.get(2l));
    assertNotNull(nodeRepository.get(3l));
    assertNull(nodeRepository.get(4l));
    // Check way importation
    assertNull(tableRepository.get(0l));
    assertNotNull(tableRepository.get(1l));
    assertNull(tableRepository.get(2l));
    // Check relation importation
    assertNull(relationRepository.get(0l));
    assertNotNull(relationRepository.get(1l));
    assertNull(relationRepository.get(2l));
}
Also used : SaveBlockConsumer(com.baremaps.osm.repository.SaveBlockConsumer) ResourceBlobStore(com.baremaps.blob.ResourceBlobStore) InputStream(java.io.InputStream) URI(java.net.URI) OsmPbfParser(com.baremaps.osm.pbf.OsmPbfParser) Test(org.junit.jupiter.api.Test) Tag(org.junit.jupiter.api.Tag)

Example 5 with OsmPbfParser

use of com.baremaps.osm.pbf.OsmPbfParser in project baremaps by baremaps.

the class OpenStreetMapGeometriesBenchmark method store.

@Benchmark
@BenchmarkMode(Mode.SingleShotTime)
@Warmup(iterations = 0)
@Measurement(iterations = 1)
public void store() throws IOException {
    Path directory = Files.createTempDirectory(Paths.get("."), "benchmark_");
    LongDataMap<Coordinate> coordinates = new LongDataOpenHashMap<>(new DataStore<>(new CoordinateDataType(), new OnDiskMemory(directory)));
    LongDataMap<List<Long>> references = new LongDataOpenHashMap<>(new DataStore<>(new LongListDataType(), new OnHeapMemory()));
    AtomicLong nodes = new AtomicLong(0);
    AtomicLong ways = new AtomicLong(0);
    AtomicLong relations = new AtomicLong(0);
    try (InputStream inputStream = new BufferedInputStream(Files.newInputStream(path))) {
        new OsmPbfParser().coordinates(coordinates).references(references).projection(4326).entities(inputStream).forEach(new EntityConsumerAdapter() {

            @Override
            public void match(Node node) {
                nodes.incrementAndGet();
            }

            @Override
            public void match(Way way) {
                ways.incrementAndGet();
            }

            @Override
            public void match(Relation relation) {
                relations.incrementAndGet();
            }
        });
    }
}
Also used : Path(java.nio.file.Path) LongListDataType(com.baremaps.store.type.LongListDataType) BufferedInputStream(java.io.BufferedInputStream) InputStream(java.io.InputStream) Node(com.baremaps.osm.domain.Node) OnDiskMemory(com.baremaps.store.memory.OnDiskMemory) EntityConsumerAdapter(com.baremaps.osm.function.EntityConsumerAdapter) Way(com.baremaps.osm.domain.Way) AtomicLong(java.util.concurrent.atomic.AtomicLong) Relation(com.baremaps.osm.domain.Relation) Coordinate(org.locationtech.jts.geom.Coordinate) BufferedInputStream(java.io.BufferedInputStream) CoordinateDataType(com.baremaps.store.type.CoordinateDataType) List(java.util.List) LongDataOpenHashMap(com.baremaps.store.LongDataOpenHashMap) OnHeapMemory(com.baremaps.store.memory.OnHeapMemory) OsmPbfParser(com.baremaps.osm.pbf.OsmPbfParser) Measurement(org.openjdk.jmh.annotations.Measurement) Warmup(org.openjdk.jmh.annotations.Warmup) BenchmarkMode(org.openjdk.jmh.annotations.BenchmarkMode) Benchmark(org.openjdk.jmh.annotations.Benchmark)

Aggregations

OsmPbfParser (com.baremaps.osm.pbf.OsmPbfParser)5 InputStream (java.io.InputStream)5 Entity (com.baremaps.osm.domain.Entity)2 Node (com.baremaps.osm.domain.Node)2 Relation (com.baremaps.osm.domain.Relation)2 Way (com.baremaps.osm.domain.Way)2 EntityConsumerAdapter (com.baremaps.osm.function.EntityConsumerAdapter)2 BufferedInputStream (java.io.BufferedInputStream)2 AtomicLong (java.util.concurrent.atomic.AtomicLong)2 Test (org.junit.jupiter.api.Test)2 Benchmark (org.openjdk.jmh.annotations.Benchmark)2 BenchmarkMode (org.openjdk.jmh.annotations.BenchmarkMode)2 Measurement (org.openjdk.jmh.annotations.Measurement)2 Warmup (org.openjdk.jmh.annotations.Warmup)2 Blob (com.baremaps.blob.Blob)1 ResourceBlobStore (com.baremaps.blob.ResourceBlobStore)1 Block (com.baremaps.osm.domain.Block)1 BlockEntityConsumer (com.baremaps.osm.function.BlockEntityConsumer)1 CreateGeometryConsumer (com.baremaps.osm.geometry.CreateGeometryConsumer)1 ReprojectEntityConsumer (com.baremaps.osm.geometry.ReprojectEntityConsumer)1