Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: mbtiles-s3-server – serve vector map tiles directly from mbtiles on S3

github.com

Post reply on HN
Print view View original

11–20 of 20 posts

Re: Show HN: mbtiles-s3-server – serve vector map tiles directly from mbtiles on S3

#11

Post by piperswe »

elithrar wrote: post #10
toomuchtodo wrote: post #8

Earlier quoted context omitted.

Do range requests against s3 work if you’re fronting with Cloudflare and aggressively caching? Could be more economical.

It should work directly against R2 - our S3 compatible object storage) w/ no egress fees: https://developers.cloudflare.com/r2/ Makes experimenting with things like this much cheaper. (I work for CF)

Though if you have enough users pinging the server you'll probably end up with a bunch of operations happening, which do have fees. It may be better cost-wise to just put this SQLite database on D1 (especially if it gets read-only replication at some point) and set up a Worker to return tile data in one HTTP request. That way the infrastructure is still serverless, but doesn't require N requests per tile, just 1.

(I also work for CF)

Re: Show HN: mbtiles-s3-server – serve vector map tiles directly from mbtiles on S3

#12

Post by simonw »

piperswe wrote: post #11
elithrar wrote: post #10

Earlier quoted context omitted.

It should work directly against R2 - our S3 compatible object storage) w/ no egress fees: https://developers.cloudflare.com/r2/ Makes experimenting with things like this much cheaper. (I work for CF)

Though if you have enough users pinging the server you'll probably end up with a bunch of operations happening, which do have fees. It may be better cost-wise to just put this SQLite database on D1 (especially if it gets read-only replication at some point) and set up a Worker to return tile data in one HTTP request. That way the infrastructure is still serverless, but doesn't require N requests per tile, just 1. (I…

Can D1 handle a 92GB SQLite database file?

Re: Show HN: mbtiles-s3-server – serve vector map tiles directly from mbtiles on S3

#13

Post by zylepe »

bdon wrote: post #9
jtmiclat wrote: post #2

This is pretty cool! Didn't know you can query SQLite databases from s3. I recently tried out PMTiles[1] to do something similar but it requires an extra processing step. Would anybody know the pros and cons of this vs PMTtiles? [1] https://github.com/protomaps/PMTiles

The end goal of PMTiles is similar to SQLite-over-HTTP: SQLite is designed for access on local disk, and might make many requests for a single query, each incurring a HTTP roundtrip (could take hundreds of milliseconds). PMTiles is designed from the ground up for HTTP, so attempts to minimize the # of requests to 3 or less for any tile. The extra conversion step is something I'm hoping to get rid of by building PMTil…

You can track progress for planetiler producing pmtiles output here: https://github.com/onthegomap/planetiler/issues/98.

Another bonus of writing pmtiles directly is that it will cut the time to generate vector tiles for the planet from ~40 to ~30 minutes on a 64-core ec2 instance since pmtiles format is much simpler and more efficient to write than SQLite. It should also cut the output file size a bit from the 80gb that mbtiles uses.

Re: Show HN: mbtiles-s3-server – serve vector map tiles directly from mbtiles on S3

#14

Post by xemoka »

jtmiclat wrote: post #2

This is pretty cool! Didn't know you can query SQLite databases from s3. I recently tried out PMTiles[1] to do something similar but it requires an extra processing step. Would anybody know the pros and cons of this vs PMTtiles? [1] https://github.com/protomaps/PMTiles

Com-tiles[1] also looks pretty good, a bit more optimized than pmtiles.

The whole cloud optimized vector tile/data space is next after COGs (cloud optimized geotiffs). Point-clouds are also getting the index-range-request treatment as COPC[2].

[1] https://github.com/mactrem/com-tiles.

[2] https://copc.io/

Re: Show HN: mbtiles-s3-server – serve vector map tiles directly from mbtiles on S3

#15

Post by zylepe »

elithrar wrote: post #10
toomuchtodo wrote: post #8

Earlier quoted context omitted.

Do range requests against s3 work if you’re fronting with Cloudflare and aggressively caching? Could be more economical.

It should work directly against R2 - our S3 compatible object storage) w/ no egress fees: https://developers.cloudflare.com/r2/ Makes experimenting with things like this much cheaper. (I work for CF)

We’ve been experimenting with R2 for serving tiles from an 80gb pmtiles archive, unfortunately the latency on up to 3 byte range requests from a worker is 5-10 seconds so it’s not practical for real usage yet. Can we expect that to improve as R2 gets closer to general availability?

Re: Show HN: mbtiles-s3-server – serve vector map tiles directly from mbtiles on S3

#16

Post by bdon »

xemoka wrote: post #14
jtmiclat wrote: post #2

This is pretty cool! Didn't know you can query SQLite databases from s3. I recently tried out PMTiles[1] to do something similar but it requires an extra processing step. Would anybody know the pros and cons of this vs PMTtiles? [1] https://github.com/protomaps/PMTiles

Com-tiles[1] also looks pretty good, a bit more optimized than pmtiles. The whole cloud optimized vector tile/data space is next after COGs (cloud optimized geotiffs). Point-clouds are also getting the index-range-request treatment as COPC[2]. [1] https://github.com/mactrem/com-tiles . [2] https://copc.io/

The COMTiles design has a very neat idea for batching requests directly on the client due to its ordered layout; this should also be possible in PMTiles spec v3 which introduces an optional Hilbert ordering on disk, as well as index compression and optimizations for run-length-encoding of repetitive tiles. I'm hoping to have this ready and implemented in clients by the FOSS4G conference (late August)

Re: Show HN: mbtiles-s3-server – serve vector map tiles directly from mbtiles on S3

#18

Post by piperswe »

simonw wrote: post #12
piperswe wrote: post #11

Earlier quoted context omitted.

Though if you have enough users pinging the server you'll probably end up with a bunch of operations happening, which do have fees. It may be better cost-wise to just put this SQLite database on D1 (especially if it gets read-only replication at some point) and set up a Worker to return tile data in one HTTP request. That way the infrastructure is still serverless, but doesn't require N requests per tile, just 1. (I…

Can D1 handle a 92GB SQLite database file?

I'm not sure! I would imagine it's planned though, scaling to that size seems important for a database product. I'm not on the D1 or R2 teams so I'm not intimately familiar with their products.

Re: Show HN: mbtiles-s3-server – serve vector map tiles directly from mbtiles on S3

#19

Post by toomuchtodo »

piperswe wrote: post #11
elithrar wrote: post #10

Earlier quoted context omitted.

It should work directly against R2 - our S3 compatible object storage) w/ no egress fees: https://developers.cloudflare.com/r2/ Makes experimenting with things like this much cheaper. (I work for CF)

Though if you have enough users pinging the server you'll probably end up with a bunch of operations happening, which do have fees. It may be better cost-wise to just put this SQLite database on D1 (especially if it gets read-only replication at some point) and set up a Worker to return tile data in one HTTP request. That way the infrastructure is still serverless, but doesn't require N requests per tile, just 1. (I…

Thank you Cloudflare folks for contributing, extremely helpful.

Re: Show HN: mbtiles-s3-server – serve vector map tiles directly from mbtiles on S3

#20

Post by dacort »

jtmiclat wrote: post #2

This is pretty cool! Didn't know you can query SQLite databases from s3. I recently tried out PMTiles[1] to do something similar but it requires an extra processing step. Would anybody know the pros and cons of this vs PMTtiles? [1] https://github.com/protomaps/PMTiles

A while ago I wrote an Athena connector for querying SQLite in S3. ;) https://github.com/dacort/athena-sqlite Uses a similar approach.
Post reply on HN