Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to build a IP geolocation database from scratch?

ipapi.is

Post reply on HN
Print view View original

61–70 of 186 posts

Re: How to build a IP geolocation database from scratch?

#61

Post by emadda »

louison11 wrote: post #11

If you don't want to do this yourself, you can actually just get Cloudflare to do it for you for free using a simple Worker since all Cloudflare requests contain approximate IP location information. You can also just send a request to my URL (Cloudflare Worker operated - so it should have global low latency): https://www.edenmaps.net/iplocation Use it for small applications, I don't mind. Just don't start sending me…

Does anyone know how accurate Cloudflare geolocation is (for workers requests)?

Re: How to build a IP geolocation database from scratch?

#62

Post by reincoder »

chaps wrote: post #40
reincoder wrote: post #31

First, I am big fan of your articles even before I joined IPinfo, where we provide IP geolocation data service. Our geolocation methodology expands on the methodology you described. We utilize some of the publicly available datasets that you are using. However, the core geolocation data comes from our ping-based operation. We ping an IP address from multiple servers across the world and identify the location of the I…

Not gonna lie, this creeps the heck out of me.

Thousands of people live in a zip code, while hundreds and thousands of people live in a city. We are literally giving away that data for free through our API and database. The creepiness of IP geolocation is mostly a meme.

IP geolocation is mainly used in cybersecurity and marketing analytics. There are many ways to geolocate someone. I once came across a project that could estimate the country a user is from based on their writing style and grammar mistakes. For example, American people sometimes use "should of" instead of "should have". Knowing the geolocation of an IP address isn't super creepy. It's just how things work on the internet.

Re: How to build a IP geolocation database from scratch?

#63

Post by johnklos »

I think it's interesting that the one IP range I decided to check has correct information on the ipapi.is web site, but unambiguously incorrect information in the downloadable geolocationDatabaseIPv4.csv. Somehow Bedford, New Hampshire (which came straight from WHOIS) became Bedford, Texas.

How'd that happen?

Re: How to build a IP geolocation database from scratch?

#64

Post by banana_giraffe »

emadda wrote: post #61
louison11 wrote: post #11

If you don't want to do this yourself, you can actually just get Cloudflare to do it for you for free using a simple Worker since all Cloudflare requests contain approximate IP location information. You can also just send a request to my URL (Cloudflare Worker operated - so it should have global low latency): https://www.edenmaps.net/iplocation Use it for small applications, I don't mind. Just don't start sending me…

Does anyone know how accurate Cloudflare geolocation is (for workers requests)?

As accurate as MaxMind[1], since that's what they use [2]. In my experience, it's reasonably accurate for the US, less so for other countries. MaxMind publishes some accuracy data which might be an interesting starting point [3]

That said, for any analytics use cases of this data, be aware that MaxMind will group a lot of what should be unknowns in the middle of a country. Or, in the case the US now, I think they all end up in the middle of some lake, since some farm owners in Butler County, Kansas got tired of cops showing up and sued MaxMind. It can cause odd artifacts unless you filter the addresses out somehow.

1 https://developers.cloudflare.com/support/network/configurin...

2 https://www.maxmind.com/en/geoip-demo

3 https://www.maxmind.com/en/geoip2-city-accuracy-comparison

Re: How to build a IP geolocation database from scratch?

#65

Post by gniv »

hddqsb wrote: post #58

Somewhat relevant: Google Maps can learn the location of your IP based on which locations you browse in the map. If you browse a specific location enough times, it will use that as the default location when you open Google Maps, even if you clear all cookies. (I discovered this just from using Google Maps, and I'm a little concerned by the privacy implications, considering that multiple people may share an IP address…

I suspect it's the other way around. Google just has a very good IP geolocation db, so it uses that when you browse, absent any other info.

Re: How to build a IP geolocation database from scratch?

#66

Post by reincoder »

voltagex_ wrote: post #42
reincoder wrote: post #31

First, I am big fan of your articles even before I joined IPinfo, where we provide IP geolocation data service. Our geolocation methodology expands on the methodology you described. We utilize some of the publicly available datasets that you are using. However, the core geolocation data comes from our ping-based operation. We ping an IP address from multiple servers across the world and identify the location of the I…

Can your probes be identified and blocked?

It is just ping data. We ping an IP address, get the RTT, draw a radius on the globe, and say that the IP could be anywhere inside that radius. Then we do another ping and draw another radius, and at the cross-section of the two radii could be your IP address. Now, if we do it enough times, we can get an estimate of where the IP address is located.

The data is not derived from the IP address itself, but rather from the process itself. And it's just a ping. Moreover, the majority of the IP addresses are not pingable. So, we rely on other in house statistical and scientific models to estimate the location. The probe infrastructure is extremely complicated and there are billions and billions of IP addresses, which is why we do not have a robust range filter mechanism.

You can implement a dynamic ping blocking mechanism or use our data to find hosting ASNs and block ranges of those ASNs. You can download the database for free: https://ipinfo.io/developers/ip-to-country-asn-database

Re: How to build a IP geolocation database from scratch?

#67

Post by is_true »

hddqsb wrote: post #58

Somewhat relevant: Google Maps can learn the location of your IP based on which locations you browse in the map. If you browse a specific location enough times, it will use that as the default location when you open Google Maps, even if you clear all cookies. (I discovered this just from using Google Maps, and I'm a little concerned by the privacy implications, considering that multiple people may share an IP address…

That would explain why it sometimes it thinks I'm in a river I paddle often and other times where I have my summer house.

Re: How to build a IP geolocation database from scratch?

#68

Post by netsharc »

gniv wrote: post #65
hddqsb wrote: post #58

Somewhat relevant: Google Maps can learn the location of your IP based on which locations you browse in the map. If you browse a specific location enough times, it will use that as the default location when you open Google Maps, even if you clear all cookies. (I discovered this just from using Google Maps, and I'm a little concerned by the privacy implications, considering that multiple people may share an IP address…

I suspect it's the other way around. Google just has a very good IP geolocation db, so it uses that when you browse, absent any other info.

Well it has reporting beacons all over the world with GPS receivers, in the form of Android phones, and perhaps Google Maps users on iPhone too..

Re: How to build a IP geolocation database from scratch?

#69

Post by chankstein38 »

reincoder wrote: post #31

First, I am big fan of your articles even before I joined IPinfo, where we provide IP geolocation data service. Our geolocation methodology expands on the methodology you described. We utilize some of the publicly available datasets that you are using. However, the core geolocation data comes from our ping-based operation. We ping an IP address from multiple servers across the world and identify the location of the I…

That's pretty neat! You're basically using ping triangulation!

Re: How to build a IP geolocation database from scratch?

#70

Post by spacedcowboy »

So, at the risk of outing myself, I wrote http://www.hostip.info a long time ago* which used a community approach to get ip address location ("is this guess wrong ? Fix it please").

The last time I checked (maybe a decade ago [grin]) it worked pretty much perfectly for a country, imperfectly for a region, and better-than-a-coin-toss for city resolution. All the data is free.

I don't think they have it on the site any more, but I used to have a rotating 3D-cube thing (x,y,z were the first 3 octets of the address) for things like known-addresses, recent lookups, etc. I used different colours for different groups (country, continent,...) It was so old it was written as a Java applet. Yeah. I guess if I were to do it again, it'd be WebGL.

--

*: I sold it a long time ago, with the proviso that the data must always remain free. I actually didn't believe the offer at first (it came as an email, and looked like a scam) but it went through escrow.com just fine, and I think we both walked away happy. That was almost 2 decades ago now though.

Post reply on HN