Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to build a IP geolocation database from scratch?

ipapi.is

Post reply on HN
Print view View original

131–140 of 186 posts

Re: How to build a IP geolocation database from scratch?

#131

Post by freedomben »

reincoder wrote: post #97
freedomben wrote: post #81

Earlier quoted context omitted.

Just wanted to let you know, it's this transparency that turned me into a customer! I love your company and service, but I hate your pricing. I work with a lot of small clients/apps that paying for usage would be a no-brainer, but the defined monthly price buckets don't make any economical sense at their scale. If you added a "pay as you go" tier that a small app could reasonably start by using dollars worth of API c…

Thank you very much; I really appreciate your feedback. This is not the first time I have heard this. The solution is to try to take as much advantage as you can from the free tier. # Check out the free IP databases https://ipinfo.io/products/free-ip-database The free databases come with commercial usage permission, and because they are databases, you can make unlimited lookups from them. The databases provide full a…

Thank you, that's super useful info. I didn't realize you had an Erlang library! I'm definitely going to be putting that to use :-)

Re: How to build a IP geolocation database from scratch?

#133

Post by SnorkelTan »

theogravity wrote: post #130
reincoder wrote: post #31

First, I am big fan of your articles even before I joined IPinfo, where we provide IP geolocation data service. Our geolocation methodology expands on the methodology you described. We utilize some of the publicly available datasets that you are using. However, the core geolocation data comes from our ping-based operation. We ping an IP address from multiple servers across the world and identify the location of the I…

How does that work with edge servers that use anycast to assume the same IP across different regions?

Aren’t any cast addresses a specific subset of ips and thus knowable? Iirc, each autonomous system is allocated anycast ip space?

Re: How to build a IP geolocation database from scratch?

#134

Post by matwood »

banana_giraffe wrote: post #64
emadda wrote: post #61

Earlier quoted context omitted.

Does anyone know how accurate Cloudflare geolocation is (for workers requests)?

As accurate as MaxMind[1], since that's what they use [2]. In my experience, it's reasonably accurate for the US, less so for other countries. MaxMind publishes some accuracy data which might be an interesting starting point [3] That said, for any analytics use cases of this data, be aware that MaxMind will group a lot of what should be unknowns in the middle of a country. Or, in the case the US now, I think they all…

Yeah, MaxMind is the best I have used with caveats. You need to update it frequently, and you need to allow for overrides.

Re: How to build a IP geolocation database from scratch?

#135

Post by junto »

As someone that lives in a country where the national language is not my first language, I hate websites that use IP location to make assumptions about my choice of language and it being forced on me based on a lazy assumption, when my browser is sending language headers quite clearly, and they are ignored.

Re: How to build a IP geolocation database from scratch?

#136

Post by carstenhag »

louison11 wrote: post #11

If you don't want to do this yourself, you can actually just get Cloudflare to do it for you for free using a simple Worker since all Cloudflare requests contain approximate IP location information. You can also just send a request to my URL (Cloudflare Worker operated - so it should have global low latency): https://www.edenmaps.net/iplocation Use it for small applications, I don't mind. Just don't start sending me…

I'm in Munich. Cloudflare tells a position that is 730km to the north in a random forest.

Re: How to build a IP geolocation database from scratch?

#137

Post by sgjohnson »

SnorkelTan wrote: post #133
theogravity wrote: post #130

Earlier quoted context omitted.

How does that work with edge servers that use anycast to assume the same IP across different regions?

Aren’t any cast addresses a specific subset of ips and thus knowable? Iirc, each autonomous system is allocated anycast ip space?

No. Any prefix can be anycasted.

Re: How to build a IP geolocation database from scratch?

#138

Post by incolumitas »

notlukesky wrote: post #7
incolumitas wrote: post #5

Earlier quoted context omitted.

I have to scrape the whole IP address space since I offer location information as part of my API. Also I only need to scrape as many WHOIS records as there are different networks out there. So for example for the IPv4 address space, there are much less networks as there are IPv4 addresses (2^32). Also, most RIR's provide their WHOIS databases for download. Therefore, "scraping" is not really the correct word, it's an…

What was the easiest and the most frustrating part?

LACNIC is a fucking pain in the ass since they block after like 27 records

ARIN is awesome

Re: How to build a IP geolocation database from scratch?

#139

Post by ggm »

gsich wrote: post #129
TZubiri wrote: post #3

"how to scrape an ip geolocation database" You know you can just run a whois query per ip you want to analyze, no point in scraping the whole ipvN space.

whois has no sane format.

RDAP is run by all the RIR and is Json and has all the whois data except IRR.

And it does 302 redirect to best source.

Post reply on HN