Live data from Hacker News

How to build a IP geolocation database from scratch?

ipapi.is

121–130 of 186 posts

Re: How to build a IP geolocation database from scratch?

#121

Earlier quoted context omitted.

You don't need precise lat/lon to make a good database. Even a 1km circle would be more than enough. > but not ASN Why wasn't ASN allowed? That's what Netflix used to make endpoint routing decisions and worked really well.

You’re not wrong, but privacy concerns were paramount. ASNs were allowed but too vague. We needed more granularity. Corporate proxies, subdelegations, many providers aggregating announcements below /24, etc.

[deleted]

Re: How to build a IP geolocation database from scratch?

#122

Earlier quoted context omitted.

Even 10 years ago, Apple internal privacy policies prevented itself from collecting precise lat/long. We had to use HTTP session telemetry to determine which endpoints were best for a given IP (or subnet, but not ASN), which informed our own pseudo-geoIP database so we knew which endpoint to connect to based on real world conditions. Even still, it had to be as ephemeral as possible for the sake of privacy. We weren’…

You don't need precise lat/lon to make a good database. Even a 1km circle would be more than enough. > but not ASN Why wasn't ASN allowed? That's what Netflix used to make endpoint routing decisions and worked really well.

[deleted]

Re: How to build a IP geolocation database from scratch?

#123

If you don't want to do this yourself, you can actually just get Cloudflare to do it for you for free using a simple Worker since all Cloudflare requests contain approximate IP location information. You can also just send a request to my URL (Cloudflare Worker operated - so it should have global low latency): https://www.edenmaps.net/iplocation Use it for small applications, I don't mind. Just don't start sending me…

[deleted]

Re: How to build a IP geolocation database from scratch?

#124

First, I am big fan of your articles even before I joined IPinfo, where we provide IP geolocation data service. Our geolocation methodology expands on the methodology you described. We utilize some of the publicly available datasets that you are using. However, the core geolocation data comes from our ping-based operation. We ping an IP address from multiple servers across the world and identify the location of the I…

[dead]

Re: How to build a IP geolocation database from scratch?

#125
post #120

Earlier quoted context omitted.

Yes, I’m sure it is iOS anti-tracking and directly related to why firewall apps inside SIP my not know what is going on.

More likely to be just standard Mobile IP https://en.wikipedia.org/wiki/Mobile_IP . Fairly standard stuff, can cause some false positives around traveling (I've seen people get freaked out about stuff like "This person just logged in from their home state and then less then an hour later logged in from France!" when it was just mobile IP treating their phone as still in the US while they were in France on a trip, but…

this was a consistent ip address nothing to do with location and nobody was freaked out.

Re: How to build a IP geolocation database from scratch?

#126

Any suggestions for geolocating datacenter IPs, even very roughly? I'm analysing traceroute data, and while I have known start and end locations, it's the bit in the middle I'm interested in. I can infer certain details from airport codes in node hostnames, for example. It would also be possible - I guess - to infer locations based on average RTT times, presuming a given node's not having a bad day. Anyone have any o…

Those IPs are owned by Google and NTT, who both run large international networks and can redeploy their IPs around the world when they feel like it. So lookup based geolocation is going to be iffy, as you've seen.

Traceroute to those IPs certainly looks like the networking goes to London.

The google IP doesn't respond to ping, but the NTT/Verio ones do. I'd bet if you ping from London based hosting, you'll get single digit ms ping responses, which sets an upper bound on the distance from London. Ping from other hosting in the country and across the channel, and you can confirm the lowest ping you can get is from London hosting, and there you go. It could also be that its connectivity is through London, but it's elsewhere --- you can't really tell.

Check from other vantage points, just to make sure it's not anycast; if you ping 8.8.8.8 from most networks around the world, you'll get something nearby; but these IPs give traceroutes to london from the Seattle area, so probably not anycast (at least at the moment, things can change).

If you don't have hosting around the world, search for public looking glasses at well connected network that you can use for pings like this from time to time.

Re: How to build a IP geolocation database from scratch?

#127

First, I am big fan of your articles even before I joined IPinfo, where we provide IP geolocation data service. Our geolocation methodology expands on the methodology you described. We utilize some of the publicly available datasets that you are using. However, the core geolocation data comes from our ping-based operation. We ping an IP address from multiple servers across the world and identify the location of the I…

Big fan of what articles? On https://incolumitas.com/ or on https://ipapi.is/ ? Great idea with latency triangulation, I used latency information for a lot of things, especially VPN and Proxy detection. But I didn't assume you can obtain that accurate location. I am honestly impressed. But latency triangulation with 600 servers gives some very good approximation. Nice man! Some questions: - ICMP traffic is penalised/…

You can guess pretty well how IP's are related by BGP announcements, so as long as a few per block and if small, ASN. You can use that logic.

Re: How to build a IP geolocation database from scratch?

#128

First, I am big fan of your articles even before I joined IPinfo, where we provide IP geolocation data service. Our geolocation methodology expands on the methodology you described. We utilize some of the publicly available datasets that you are using. However, the core geolocation data comes from our ping-based operation. We ping an IP address from multiple servers across the world and identify the location of the I…

Have you considered making your database available for download as Parquet format so people could just copy the file to S3, Google Cloud, etc, and query it immediately with various tools?

I know it can be done with CSV but it's not as smooth.

Re: How to build a IP geolocation database from scratch?

#130

First, I am big fan of your articles even before I joined IPinfo, where we provide IP geolocation data service. Our geolocation methodology expands on the methodology you described. We utilize some of the publicly available datasets that you are using. However, the core geolocation data comes from our ping-based operation. We ping an IP address from multiple servers across the world and identify the location of the I…

How does that work with edge servers that use anycast to assume the same IP across different regions?
Post reply on HN