Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

From side project to 250M daily requests

medium.com

Post reply on HN
Print view View original

1–10 of 18 posts

Re: From side project to 250M daily requests

#4

Post by dougb5 »

coderholic wrote: post #3
psuter wrote: post #2

Good insight into the engineering that went into this. Functionally, is this "MaxMind-GeoIP-as-a-service"? Any other data sources? http://dev.maxmind.com/geoip/

Yeah, lots of custom data. See https://ipinfo.io/data for an overview

Since you are building off of GeoLite which has a ShareAlike license (http://creativecommons.org/licenses/by-sa/3.0/), does that mean you are distributing your modified database under the same terms?

Re: From side project to 250M daily requests

#5

Post by [deleted] »

dougb5 wrote: post #4
coderholic wrote: post #3

Earlier quoted context omitted.

Yeah, lots of custom data. See https://ipinfo.io/data for an overview

Since you are building off of GeoLite which has a ShareAlike license ( http://creativecommons.org/licenses/by-sa/3.0/ ), does that mean you are distributing your modified database under the same terms?

[deleted]

Re: From side project to 250M daily requests

#6

Post by pilif »

This is information I would never accept an external dependency on. There are not too many IP ranges around, so this information can easily be stored somewhere and used without the need to query an external service.

Having the data locally allows for potentially synchronous lookups, or at least for lookups with an availability guarantee which makes your code much simpler to reason about.

Plus, you are guaranteed to get the information, independent of the availability of a third party.

And finally for $100 per month I get access to a weekly updated locally available database (that's what Maxmind charges for the city-level database) no matter how many queries I'm going to issue locally.

Flat-fee access to a local database would even allow to bulk post-process web server log files (yeah - they still exist and don't come with the usual privacy issues surrounding third-party analytics providers) within a reasonable time- and cost frame.

Not everything that can be an external service has to be an external service and for geolocation I definitely cannot see any advantage to not having this data stored locally.

This isn't big-data and will easily fit any amount of RAM (if it even needs to), this doesn't require a costly sys-admin team, this doesn't require any hardware knowledge. This is about fetching a file and putting it somewhere on the server. You are doing this daily with your daily web-browsing.

Now, to OP, I'm very happy for you and I appreciate the service you are offering and I'm very happy that you are solving an issue some people are having. I don't want to belittle this at all.

I'm just saying that while I might personally err a bit too much on "doing it on my own", I absolutely cannot see any justification to do IP geolocation with an external dependency.

Re: From side project to 250M daily requests

#7

Post by lossolo »

coderholic wrote: post #3
psuter wrote: post #2

Good insight into the engineering that went into this. Functionally, is this "MaxMind-GeoIP-as-a-service"? Any other data sources? http://dev.maxmind.com/geoip/

Yeah, lots of custom data. See https://ipinfo.io/data for an overview

The "Network data" is just ripe, arin etc data that is available publicly to anyone.

Re: From side project to 250M daily requests

#8

Post by toomuchtodo »

pilif wrote: post #6

This is information I would never accept an external dependency on. There are not too many IP ranges around, so this information can easily be stored somewhere and used without the need to query an external service. Having the data locally allows for potentially synchronous lookups, or at least for lookups with an availability guarantee which makes your code much simpler to reason about. Plus, you are guaranteed to g…

I could see the value of having this as a docker container to run internally as a microservice, but agree I would never rely on an external service for this.

Re: From side project to 250M daily requests

#9

Post by coderholic »

lossolo wrote: post #7
coderholic wrote: post #3

Earlier quoted context omitted.

Yeah, lots of custom data. See https://ipinfo.io/data for an overview

The "Network data" is just ripe, arin etc data that is available publicly to anyone.

That's one of the raw sources that go into the network data. There's also the routing tables (without which you can't always match ASNs to netblocks), other datasets, and a bunch of custom scripts that post-process the data, performing cleanup, heuristic matching and more. And then putting it all into a data format that supports handling 250 million requests a day, and serving over 90% of them in less than 10ms.
Post reply on HN