Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A new way to make maps with OpenStreetMap

protomaps.com

Post reply on HN
Print view View original

71–80 of 102 posts

Re: A new way to make maps with OpenStreetMap

#71

Post by incanus77 »

TheTaytay wrote: post #69
bdon wrote: post #40

Earlier quoted context omitted.

Author of OP here, I use the tile unpacker as well! I didn't point this out in the OP, but I found the ergonomics of uploading tiles to S3 etc get bad once you're over a couple tens of thousands of tiles, as each upload has some overhead. I've tested PMTiles on S3 successfully with hundreds of millions of tiles. Deduplication is also important to save space on redundant ocean/earth tiles; MBTiles can deduplicate via…

When I was playing around with hosting rendered Minecraft slippy maps, I found the upload overhead to S3 to be pretty poor due to the “lots of tiny files” problem. (It renders as hundreds of thousands of rasterized image files for the various zoom levels). You could parallelize it, but it was still poor, and made me want to come up with a way to pack them into fewer files and fetch portions of them at render time so…

That was the impetus behind MBTiles as well. I was building out an iOS toolkit to render OSM-based maps on iPad, offline, and transferring 1000s of rasters (at the time) even over USB was way too slow and error-prone.

Re: A new way to make maps with OpenStreetMap

#72

Post by TheTaytay »

bdon wrote: post #70
TheTaytay wrote: post #69

Earlier quoted context omitted.

When I was playing around with hosting rendered Minecraft slippy maps, I found the upload overhead to S3 to be pretty poor due to the “lots of tiny files” problem. (It renders as hundreds of thousands of rasterized image files for the various zoom levels). You could parallelize it, but it was still poor, and made me want to come up with a way to pack them into fewer files and fetch portions of them at render time so…

That's exactly what PMTiles is for, and there's nothing in it specific to maps or geographic coordinates, just a Z/X/Y pyramid structure. I chose to support Leaflet for now because it is very popular; there's other viewers like OpenSeadragon for which a decoder plugin would be possible.

Fantastic. The renderer I was using was indeed leaflet based, so this sounds drop-in easy. I just read through https://github.com/protomaps/PMTiles Nice work! The only downside would be that these maps are often re-rendered periodically and only have a few tiles that actually get updated. In that case, it’s nice to have to update only a handful of files. But I think the benefits would far outweigh the costs. Upload one massive file repeatedly is still gonna be faster I would think. Thanks!

Re: A new way to make maps with OpenStreetMap

#73

Post by jamil7 »

tppiotrowski wrote: post #20

Super excited for your project. I have been using map tiles from Mapbox, MapTiler, OSM Buildings for my project and am coming to the point that I need to merge elevation data with building height data in order to cast realistic shadows across the earth. Unfortunately elevation data and building data come from two different tile sets from two different tile providers and it's a lot of download/processing overhead to m…

Edit: replied to the wrong comment

Re: A new way to make maps with OpenStreetMap

#74

Post by TheTaytay »

incanus77 wrote: post #71
TheTaytay wrote: post #69

Earlier quoted context omitted.

When I was playing around with hosting rendered Minecraft slippy maps, I found the upload overhead to S3 to be pretty poor due to the “lots of tiny files” problem. (It renders as hundreds of thousands of rasterized image files for the various zoom levels). You could parallelize it, but it was still poor, and made me want to come up with a way to pack them into fewer files and fetch portions of them at render time so…

That was the impetus behind MBTiles as well. I was building out an iOS toolkit to render OSM-based maps on iPad, offline, and transferring 1000s of rasters (at the time) even over USB was way too slow and error-prone.

Oh wow - very cool. Thanks for this! I love SQLite, and think this is an excellent use case. (File systems, especially networked ones, don’t handle “lots of tiny files” well). I think using SQLite as the primary, writeable storage mechanism makes sense because I could ship that around easily to rendering nodes and incrementally update it easily. Then, I could export to a static PMTile for upload to S3 for serving purposes.

Re: A new way to make maps with OpenStreetMap

#75

Post by loa_in_ »

wizzwizz4 wrote: post #18
loa_in_ wrote: post #14

Earlier quoted context omitted.

> It would be great if there were a solution that was as simple to get started with as My Maps but also had the flexibility to easily add custom data layers programmatically. I believe _with some trickery_ you can use Google Earth for that

The computer version, not the web version, right?

Yes

Re: A new way to make maps with OpenStreetMap

#76

Post by Demiurge »

hardwaresofton wrote: post #46

Does anyone have a really good (maybe illustrated?) guide/short reference to the world of GIS? Every time I see GIS stuff mentioned the problem space itself, solution jargon, and various movements in the ecosystem (projects being unsupported, stewards/companies changing etc) always throws me for a loop. It's certainly the case that I just haven't built enough mapping-related software to be comfortable but if there's…

Check out mapschool.io

Re: A new way to make maps with OpenStreetMap

#78

Post by nishnik »

hardwaresofton wrote: post #57
bdon wrote: post #55

Earlier quoted context omitted.

Author of the OP here. I don't have a GIS background in particular; I do have a decade of experience building web mapping applications, and here's some of the foundational software I use: http://leafletjs.com https://shapely.readthedocs.io/ https://github.com/libgeos/geos https://rasterio.readthedocs.io/ https://gdal.org http://www.angusj.com/delphi/clipper.php

Thanks for the pointers -- Leaflet I know as the visualization layer that is basically not-mapbox-GL. The rest are new to me, and will crawl through them. One of the nice out-of-the-box systems I've come across is Pelias[0], what are your opinions on it? It bundles and provides the API so it's one of the things I've thought to self-host if I'm going to. It looks like it doesn't do tiles so I'd need to find something…

Where do folks use tile38?
Post reply on HN