Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tile38 – Realtime geofencing and geospatial index, v1.7.0

github.com

Post reply on HN
Print view View original

1–10 of 29 posts

Re: Tile38 – Realtime geofencing and geospatial index, v1.7.0

#3

Post by Bedon292 »

I love it when geo stuff shows up here. Its always fun to see what people are doing.

I see that it is in Memory, but are there any practical limits? How much memory is recommended for what size datasets?

And it doesn't look like the documentation has an examples as to how to load data in? Do I need to convert to Web Mercator, or can I give it WGS84 and have it convert? Also, any plans to allow for pure WGS84? I never actually use Web Mercator, so would rather a system that uses WGS84 natively.

Re: Tile38 – Realtime geofencing and geospatial index, v1.7.0

#4

Post by tidwall »

sandking wrote: post #2

Looks pretty neat! Since OP is the author, can I ask: how many simultaneous concurrent (client) connections does this support, how many simultaneous moving objects and at what update frequency? Do these limits change if the shapes get complicated?

> how many simultaneous concurrent (client) connections does this support?

The client connections are very lightweight. The protocol library is based on https://github.com/tidwall/redcon. I don't have exact numbers but this link may be helpful (which compares Redcon to Redis): https://simongui.github.io/2016/10/24/benchmarking-go-redis-.... I've seen it handle 16K+ client connection without issue, but typically a client library that supports pooling should be used (like Redigo).

> how many simultaneous moving objects and at what update frequency?

Depends on the complexity of the objects, the number of nearby objects, and the server hardware. Each collection of objects consists of one btree and one rtree for key/id lookups and spatial queries respectively. So updating a single point is like O(log(n)+log(m)). An in-memory database like Tile38 can achieve 100K+ per second. Utilizing network pipelining can help out too. FYI, I use a 4-core server with 8GB ram for testing.

In the case of a roaming geofence (http://tile38.com/topics/roaming-geofences), following each update there an additional query of the rtree O(log(n)) to retrieve nearby neighbors. If there's a lot of nearby objects, this can result in a lot of data sent over the network. But in general, a collection of millions of points spread over a continent with 100K+ updates per second should work, while keeping with realtimelyness.

> Do these limits change if the shapes get complicated?

Yes. Complicated objects such are MultiPolygons may require additional calculations like raycasting.

Re: Tile38 – Realtime geofencing and geospatial index, v1.7.0

#5

Post by tidwall »

Bedon292 wrote: post #3

I love it when geo stuff shows up here. Its always fun to see what people are doing. I see that it is in Memory, but are there any practical limits? How much memory is recommended for what size datasets? And it doesn't look like the documentation has an examples as to how to load data in? Do I need to convert to Web Mercator, or can I give it WGS84 and have it convert? Also, any plans to allow for pure WGS84? I never…

> I see that it is in Memory, but are there any practical limits?

Yes. The in-memory database is basically a big btree + a big rtree. These trees contain many pointers to geo objects and depending on the complexity of the objects, it may take up a lot of memory. A 16GB machine can typically store 100 million+ points.

> How much memory is recommended for what size datasets?

Depends on the type of object and the length of the keys, but I would conservatively use 128-bytes per point as a general rule. So 16GB machine should support ~135 million points.

> And it doesn't look like the documentation has an examples as to how to load data in?

Please check out http://tile38.com for extra info and docs on commands. Tile38 uses the Redis protocol (https://redis.io/topics/protocol), so it's importing data is similar to https://redis.io/topics/mass-insert.

> Do I need to convert to Web Mercator, or can I give it WGS84 and have it convert? Also, any plans to allow for pure WGS84?

All coordinates are in WGS84 lat/lon, so you should be fine. Under-the-hood the calculations and such are in EPSG:3857 / WGS 84 Web Mercator.

Re: Tile38 – Realtime geofencing and geospatial index, v1.7.0

#6

Post by Bedon292 »

tidwall wrote: post #5
Bedon292 wrote: post #3

I love it when geo stuff shows up here. Its always fun to see what people are doing. I see that it is in Memory, but are there any practical limits? How much memory is recommended for what size datasets? And it doesn't look like the documentation has an examples as to how to load data in? Do I need to convert to Web Mercator, or can I give it WGS84 and have it convert? Also, any plans to allow for pure WGS84? I never…

> I see that it is in Memory, but are there any practical limits? Yes. The in-memory database is basically a big btree + a big rtree. These trees contain many pointers to geo objects and depending on the complexity of the objects, it may take up a lot of memory. A 16GB machine can typically store 100 million+ points. > How much memory is recommended for what size datasets? Depends on the type of object and the length…

Thanks for the response. I see what I was actually looking for was on http://tile38.com/topics/object-types/ Right there on the front page, and not in the docs tab where I was looking.

For the GeoJSON, does it support properties? Meaning can I store a a whole document in there? Or would it be best to just store an ID and lookup the ID in a DB or something?

Re: Tile38 – Realtime geofencing and geospatial index, v1.7.0

#8

Post by tidwall »

Bedon292 wrote: post #6
tidwall wrote: post #5

Earlier quoted context omitted.

> I see that it is in Memory, but are there any practical limits? Yes. The in-memory database is basically a big btree + a big rtree. These trees contain many pointers to geo objects and depending on the complexity of the objects, it may take up a lot of memory. A 16GB machine can typically store 100 million+ points. > How much memory is recommended for what size datasets? Depends on the type of object and the length…

Thanks for the response. I see what I was actually looking for was on http://tile38.com/topics/object-types/ Right there on the front page, and not in the docs tab where I was looking. For the GeoJSON, does it support properties? Meaning can I store a a whole document in there? Or would it be best to just store an ID and lookup the ID in a DB or something?

> For the GeoJSON, does it support properties?

Yes, Tile38 has full GeoJSON support including properties for Feature objects. Though there is one exception, the CRS key is ignored because WGS84 is assumed.

> Meaning can I store a a whole document in there?

Yes, as long as the document conforms to http://geojson.org/geojson-spec.html. Though JSON keys that are not defined in the spec will be stripped out of the document prior to storing in memory. For example, a "properties" key of a type that is not a "Feature" will be stripped, and requesting that object at later time will result in a JSON document that does not include the "properties".

Re: Tile38 – Realtime geofencing and geospatial index, v1.7.0

#9

Post by tidwall »

realrocker wrote: post #7

Neat. Seems like a good weekend is coming up. I was wondering how was the gifs/images done? I would really like to see the code if it was done programmatically.

Some programming, some design tools.

I programmatically simulated the movements as paths at 30fps into a series of data points, then imported into Animate CC and tweaked the tweening by hand, then exported to a series of PNGs, then imported PNGs into Photoshop as a movie sequence, finally exported as gif.

The superior tweening capabilities of Animate CC and the excellent gif compression of Photoshop makes tuning the animation super easy.

Perhaps next time I'll programmatically draw gif frames, for the fun of it.

Post reply on HN