Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I made a simple geolocation service

maxkostinevich.com

Post reply on HN
Print view View original

51–60 of 187 posts

Re: I made a simple geolocation service

#51

Post by hansitomani »

layoutIfNeeded wrote: post #47

Can you make a geolocation service from static files? 2^32 IP4 addresses with 2 floats per address would take only 34 gigabytes of storage. Put 256 addresses in a given file, and turn the other three octets into folders. E.g. https://example.com/192/168/0.txt would contain the location for addresses 192.168.0.0-192.168.0.255. Would this be cheaper than running these services?

You could also generate all those files, put that into a GCS Bucket and put an API Gateway in front.

easier would be and thats just a thought: You could build a go/c++/java app which contains that database pre compressed in some arbitrary better format than just 2 floats, get it down from your original 34gig and just keep it in memory.

Like if you know that 99% of all ips in a certain range are from US, then only store the 1% as a list.

Re: I made a simple geolocation service

#52

Post by hansitomani »

jacobush wrote: post #45
stereo wrote: post #37

That country list doesn't have Kosovo (2008) or South Sudan (2011).

Kosovo is not universally recognised though, to be fair. But yes, I'd include it. Even if you are against in principle, it's de facto a state and treated as such even by many countries not recognising it in name.

But that would mean that your API should return a country specific result to be accurate.

Re: I made a simple geolocation service

#53

Post by elorm »

jgrahamc wrote: post #49
skuhn wrote: post #41

Since most CDNs are already doing GeoIP lookups (for request headers or log entries), you can leverage that to provide the data back in the response body via origin, worker or even CDN edge config. Programmatically populating the response body, as in the Cloudflare worker example from the post, is better than going to the origin just to echo some headers back in the response. To me, something like Fastly's VCL config…

What does the code for this look like?

Here you go.

https://gist.github.com/simonkuhn/a380a6fa205db87a3625f26ad0...

Re: I made a simple geolocation service

#54

Post by Moru »

stormdennis wrote: post #42
gitgud wrote: post #40

Side note: 6 million requests per month is only 2.2 requests per second, a raspberry PI could do this (technically).

Would they all be spaced out evenly though?

If the processing time is much below a second per request, requests per second might be more interesting than to stretch it out on a month.

Re: I made a simple geolocation service

#55

Post by layoutIfNeeded »

hansitomani wrote: post #51
layoutIfNeeded wrote: post #47

Can you make a geolocation service from static files? 2^32 IP4 addresses with 2 floats per address would take only 34 gigabytes of storage. Put 256 addresses in a given file, and turn the other three octets into folders. E.g. https://example.com/192/168/0.txt would contain the location for addresses 192.168.0.0-192.168.0.255. Would this be cheaper than running these services?

You could also generate all those files, put that into a GCS Bucket and put an API Gateway in front. easier would be and thats just a thought: You could build a go/c++/java app which contains that database pre compressed in some arbitrary better format than just 2 floats, get it down from your original 34gig and just keep it in memory. Like if you know that 99% of all ips in a certain range are from US, then only sto…

Sure, but my point was that it can be done with zero compute cost. There's nothing cheaper than serving static files from cloud storage.

Re: I made a simple geolocation service

#56

Post by skuhn »

jgrahamc wrote: post #49
skuhn wrote: post #41

Since most CDNs are already doing GeoIP lookups (for request headers or log entries), you can leverage that to provide the data back in the response body via origin, worker or even CDN edge config. Programmatically populating the response body, as in the Cloudflare worker example from the post, is better than going to the origin just to echo some headers back in the response. To me, something like Fastly's VCL config…

What does the code for this look like?

The VCL for it is available at https://gist.github.com/simonkuhn/a380a6fa205db87a3625f26ad0...

vcl_recv and vcl_error are the important bits, the rest is VCL boilerplate from https://developer.fastly.com/learning/vcl/using/ for unused subroutines.

Re: I made a simple geolocation service

#57

Post by hansitomani »

layoutIfNeeded wrote: post #55
hansitomani wrote: post #51

Earlier quoted context omitted.

You could also generate all those files, put that into a GCS Bucket and put an API Gateway in front. easier would be and thats just a thought: You could build a go/c++/java app which contains that database pre compressed in some arbitrary better format than just 2 floats, get it down from your original 34gig and just keep it in memory. Like if you know that 99% of all ips in a certain range are from US, then only sto…

Sure, but my point was that it can be done with zero compute cost. There's nothing cheaper than serving static files from cloud storage.

That is just not true.

GCS for example and its the same with S3, costs space, egress and requests themself.

Data Storage 50 GB Standard Storage * $0.026 per GB $1.30 Network 10 GB egress * $0.12 per GB $1.20 Operations 10,000 Class A operations * $0.05 per 10,000 operations $0.05 Operations 50,000 Class B operations * $0.004 per 10,000 operations $0.02

Re: I made a simple geolocation service

#58

Post by harpratap »

Nextgrid wrote: post #24
speedgoose wrote: post #7

Well that's a bit more than 2 requests per second on average. Sure you may have some bursts once in a while, but nothing a 3€ VM can't manage IMHO.

I always find it funny when people brag about cost savings on cloud services while overlooking the insane premium they're paying by using the "cloud" to begin with. It's kinda like a construction firm using a supercar to haul materials and then bragging about their efforts to make cosmetic repairs cost-effective... maybe you could just not use an overpriced car to begin with and then damage won't be an issue?

Am I missing something? Isn't a $3 VM referring to a VM service in a cloud somewhere? If you try to do the same thing without cloud you'll have to end up renting space in a datacenter, install your own servers, connect to an ISP, power bills, etc etc. Cloud definitely seems cheaper in comparison

Re: I made a simple geolocation service

#59

Post by dmortin »

Nextgrid wrote: post #24
speedgoose wrote: post #7

Well that's a bit more than 2 requests per second on average. Sure you may have some bursts once in a while, but nothing a 3€ VM can't manage IMHO.

I always find it funny when people brag about cost savings on cloud services while overlooking the insane premium they're paying by using the "cloud" to begin with. It's kinda like a construction firm using a supercar to haul materials and then bragging about their efforts to make cosmetic repairs cost-effective... maybe you could just not use an overpriced car to begin with and then damage won't be an issue?

The main advantage of the cloud for me that it provides PaaS services. You don't have to administer a VM, etc, you just use the services and it's someone else's job to install security updates and stuff.

Re: I made a simple geolocation service

#60

Post by AdrianoKF »

hansitomani wrote: post #52
jacobush wrote: post #45

Earlier quoted context omitted.

Kosovo is not universally recognised though, to be fair. But yes, I'd include it. Even if you are against in principle, it's de facto a state and treated as such even by many countries not recognising it in name.

But that would mean that your API should return a country specific result to be accurate.

Which is precisely what for example Mapbox does - they allow consumers to specify a worldview, which adjusts borders and other geographic features based on the specified culture: https://docs.mapbox.com/help/glossary/worldview/
Post reply on HN