From side project to 250M daily requests • Hacker News Board

Re: From side project to 250M daily requests

#11

Post by coderholic » Tue, Mar 29, 2016, 5:40 PM UTC

dougb5 wrote: post #4

coderholic wrote: post #3

Earlier quoted context omitted.

Yeah, lots of custom data. See https://ipinfo.io/data for an overview

Since you are building off of GeoLite which has a ShareAlike license ( http://creativecommons.org/licenses/by-sa/3.0/ ), does that mean you are distributing your modified database under the same terms?

It's a good question. I've been in touch with Maxmind's legal team before and they were happy with what we were doing, but that was before we had started building out our custom data.

My thoughts are:

1) We're not modifying the maxmind DB. We download their MMDB file and leave it completely untouched. Our code basically does something like this:

    if ip in custom_db
        return custom_db[ip]
    else
        return maxmind_db[ip]

2) Our database is openly available via our API. Anyone can access it without needing to signup etc.

They're the principles I've been operating under, but I'm not a lawyer, and I'll definitely go consult with one to get a definitive answer on where I stand with this. If it turns out that we're not meeting the license terms then we'll certainly make whatever changes necessary to ensure we do.

Re: From side project to 250M daily requests

#12

Post by rokhayakebe » Tue, Mar 29, 2016, 5:51 PM UTC

Sometimes the reason things succeed is simply because they are simple. Simple to understand. Simple to use. Simple to integrate. Simple to pay for. Just simple. One Feature, maybe. Three max.

Re: From side project to 250M daily requests

#13

Post by psuter » Tue, Mar 29, 2016, 6:08 PM UTC

One service you could consider offering is "resolve IP at past date". I don't know that the standard providers do it and it useful when, for instance, batch-processing a backlog of session data.

Re: From side project to 250M daily requests

#14

Post by azylman » Tue, Mar 29, 2016, 6:16 PM UTC

toomuchtodo wrote: post #8

pilif wrote: post #6

This is information I would never accept an external dependency on. There are not too many IP ranges around, so this information can easily be stored somewhere and used without the need to query an external service. Having the data locally allows for potentially synchronous lookups, or at least for lookups with an availability guarantee which makes your code much simpler to reason about. Plus, you are guaranteed to g…

I could see the value of having this as a docker container to run internally as a microservice, but agree I would never rely on an external service for this.

We've done this at Clever, if you're interested: https://github.com/Clever/resolve-ip

We use the same dataset that IP info started with (from MaxMind), although it sounds like they have many different sources now. It's worked fine for us, matches over 99% of the IP addresses we look up, and generally responds in less than 5 milliseconds - basically only limited by network latency.

Right now this is published to a private Docker hub repository, for no good reason, but if people would find it useful we can make it public.

Re: From side project to 250M daily requests

#15

Post by mmriis » Tue, Mar 29, 2016, 6:40 PM UTC

I'm curious... are you seeing 99% using your free plan or are you actually selling the $400 plans? What is the conversion ratio?

Re: From side project to 250M daily requests

#16

Post by swah » Tue, Mar 29, 2016, 7:04 PM UTC

This just placed me 1000km from where I actually am - the porn ads offering to meet local girls do much better :(

Re: From side project to 250M daily requests

#17

Post by lossolo » Tue, Mar 29, 2016, 9:39 PM UTC

coderholic wrote: post #9

lossolo wrote: post #7

Earlier quoted context omitted.

The "Network data" is just ripe, arin etc data that is available publicly to anyone.

That's one of the raw sources that go into the network data. There's also the routing tables (without which you can't always match ASNs to netblocks), other datasets, and a bunch of custom scripts that post-process the data, performing cleanup, heuristic matching and more. And then putting it all into a data format that supports handling 250 million requests a day, and serving over 90% of them in less than 10ms.

So you use your own BGP tables or use HE or some other service to do it for you ? 250 mil requests is just ~2800 req/s. Serving 90% of them in 10 ms using api is not really any achievement, considering what kind of data you are serving. I've worked on api with 1 bil requests a day and 97% of them are below 10 ms (And the best is that only 2 machines are needed for that).

Re: From side project to 250M daily requests

#18

Post by [deleted] » Wed, Mar 30, 2016, 2:03 PM UTC

pilif wrote: post #6

This is information I would never accept an external dependency on. There are not too many IP ranges around, so this information can easily be stored somewhere and used without the need to query an external service. Having the data locally allows for potentially synchronous lookups, or at least for lookups with an availability guarantee which makes your code much simpler to reason about. Plus, you are guaranteed to g…

[deleted]