OpenStreetMap PBF Perfomance Tricks
paulkernfeld.com
OpenStreetMap PBF Perfomance Tricks
1–6 of 6 posts
Re: OpenStreetMap PBF Perfomance Tricks
#2My favorite OpenStreetMap data science tool is "osmium"
* https://osmcode.org/osmium-tool/manual.html
And you can convert osm.pbf to osm.opl - and can reprocess with unix tools (grep,awk,cut,tr,...):
Re: OpenStreetMap PBF Perfomance Tricks
#3Yeah, the OSM PBF format is really efficient in size, but it can be quite a pain to work with those files. Every way consists of nodes, which are usually at the very beginning of the file (first nodes, then ways, then relations). So in order to work efficiently, you'll need a node cache (which at the moment is more than 40G). Seeking is not viable because there is no indication in which block the node will be. On-disk caching is an option, but it will slow you down substantially.
So unless you are doing meta-analysis of the raw OSM data, without assembling geometries, hand-rolling an OSMPBF reader is viable, otherwise I would suggest either using something pre-processed (extracts in real geodata formats), an established parser like osmium or rather import the data into e.g. postgres and do some querying there.
Re: OpenStreetMap PBF Perfomance Tricks
#4Yeah, the OSM PBF format is really efficient in size, but it can be quite a pain to work with those files. Every way consists of nodes, which are usually at the very beginning of the file (first nodes, then ways, then relations). So in order to work efficiently, you'll need a node cache (which at the moment is more than 40G). Seeking is not viable because there is no indication in which block the node will be. On-dis…
Are nodes in first fileblocks in "planet dumps"? And then fileblocks mostly with ways? With this dump configuration fileblocks are mostly useless.
Re: OpenStreetMap PBF Perfomance Tricks
#5Yeah, the OSM PBF format is really efficient in size, but it can be quite a pain to work with those files. Every way consists of nodes, which are usually at the very beginning of the file (first nodes, then ways, then relations). So in order to work efficiently, you'll need a node cache (which at the moment is more than 40G). Seeking is not viable because there is no indication in which block the node will be. On-dis…
Yeah, fair. I'm only working with Boston-area data so I can pretty easily load a Massachusetts OSM file from memory. However, memory would definitely be an issue for processing the entire world.
Re: OpenStreetMap PBF Perfomance Tricks
#6Yeah, the OSM PBF format is really efficient in size, but it can be quite a pain to work with those files. Every way consists of nodes, which are usually at the very beginning of the file (first nodes, then ways, then relations). So in order to work efficiently, you'll need a node cache (which at the moment is more than 40G). Seeking is not viable because there is no indication in which block the node will be. On-dis…
Are nodes in first fileblocks in "planet dumps"? And then fileblocks mostly with ways? With this dump configuration fileblocks are mostly useless.
Blocks are useful, because you can distribute the workload onto several workers. Most encoders nowadays write first blocks with just nodes, then blocks with just ways and then just relations.