The easiest way to get a geolocation is to ask the user. Maybe they’ll just tell you, and if that’s good enough for your application there’s no need for such solutions.
How to build a IP geolocation database from scratch?
51–60 of 186 posts
Re: How to build a IP geolocation database from scratch?
#52Earlier quoted context omitted.
Can your probes be identified and blocked?
iptables -A INPUT -p icmp -j DROP
http://shouldiblockicmp.com/
(But the guy running the probes is making a good counter argument)
Re: How to build a IP geolocation database from scratch?
#53First, I am big fan of your articles even before I joined IPinfo, where we provide IP geolocation data service. Our geolocation methodology expands on the methodology you described. We utilize some of the publicly available datasets that you are using. However, the core geolocation data comes from our ping-based operation. We ping an IP address from multiple servers across the world and identify the location of the I…
Not gonna lie, this creeps the heck out of me.
You might want to unplug your router then. A conceit of being connected to a network is you're connected to the network. If you can see other nodes they can see you.
Re: How to build a IP geolocation database from scratch?
#54First, I am big fan of your articles even before I joined IPinfo, where we provide IP geolocation data service. Our geolocation methodology expands on the methodology you described. We utilize some of the publicly available datasets that you are using. However, the core geolocation data comes from our ping-based operation. We ping an IP address from multiple servers across the world and identify the location of the I…
Big fan of what articles? On https://incolumitas.com/ or on https://ipapi.is/?
Great idea with latency triangulation, I used latency information for a lot of things, especially VPN and Proxy detection.
But I didn't assume you can obtain that accurate location. I am honestly impressed. But latency triangulation with 600 servers gives some very good approximation. Nice man!
Some questions:
- ICMP traffic is penalised/degraded by some ISP's. How do you deal with that?
- In order to geolocate every IPv4 address, you need to constantly ping billions of IPv4's, how do you do that? You only ping an arbitrary IP of each allocated inetnum/NetRange?
- Most IP addresses do not respond to ICMP packets. Only some servers do. How do you deal with that? Do you find the router in front of the target IP and you geolocate the closest router to the target IP (traceroute)?
Re: How to build a IP geolocation database from scratch?
#55Interesting but this isn't actually how geolocation is done, right? The ARIN/RIPE data isn't sufficiently accurate to be useful beyond country. Commercial geolocation involves correlating client IP vs known physical location e.g. from WiFi AP or mailing a package to the user. At least that's what I have been told over the decades.
I work in adtech and this is how we do geolocation. There's also device geolocation but if the user doesn't consent to sharing their GPS data with us, we just use IP address for targeting. Common provider for this is Maxmind; they ship a database that you host locally and query
Even the free maxmind db is accurate enough for most applications.
Re: How to build a IP geolocation database from scratch?
#56First, I am big fan of your articles even before I joined IPinfo, where we provide IP geolocation data service. Our geolocation methodology expands on the methodology you described. We utilize some of the publicly available datasets that you are using. However, the core geolocation data comes from our ping-based operation. We ping an IP address from multiple servers across the world and identify the location of the I…
Not gonna lie, this creeps the heck out of me.
Your IP address is LEAKING!
Re: How to build a IP geolocation database from scratch?
#57Re: How to build a IP geolocation database from scratch?
#58Somewhat relevant: Google Maps can learn the location of your IP based on which locations you browse in the map. If you browse a specific location enough times, it will use that as the default location when you open Google Maps, even if you clear all cookies. (I discovered this just from using Google Maps, and I'm a little concerned by the privacy implications, considering that multiple people may share an IP address.)
Re: How to build a IP geolocation database from scratch?
#59First, I am big fan of your articles even before I joined IPinfo, where we provide IP geolocation data service. Our geolocation methodology expands on the methodology you described. We utilize some of the publicly available datasets that you are using. However, the core geolocation data comes from our ping-based operation. We ping an IP address from multiple servers across the world and identify the location of the I…
Big fan of what articles? On https://incolumitas.com/ or on https://ipapi.is/ ? Great idea with latency triangulation, I used latency information for a lot of things, especially VPN and Proxy detection. But I didn't assume you can obtain that accurate location. I am honestly impressed. But latency triangulation with 600 servers gives some very good approximation. Nice man! Some questions: - ICMP traffic is penalised/…
I'm very curious why you'd do VPN/proxy detection...
But at a previous company I worked at that ran a very large chunk of the internet, we did indexing of nearly the entire internet (even large portions of the dark web) approximately every two weeks. There were about 500 servers doing that non-stop. So, I think it is relatively reasonable if you have 600 servers to do that.
Re: How to build a IP geolocation database from scratch?
#60First, I am big fan of your articles even before I joined IPinfo, where we provide IP geolocation data service. Our geolocation methodology expands on the methodology you described. We utilize some of the publicly available datasets that you are using. However, the core geolocation data comes from our ping-based operation. We ping an IP address from multiple servers across the world and identify the location of the I…
Big fan of what articles? On https://incolumitas.com/ or on https://ipapi.is/ ? Great idea with latency triangulation, I used latency information for a lot of things, especially VPN and Proxy detection. But I didn't assume you can obtain that accurate location. I am honestly impressed. But latency triangulation with 600 servers gives some very good approximation. Nice man! Some questions: - ICMP traffic is penalised/…
https://incolumitas.com/
This is my all-time favorite article: https://incolumitas.com/2021/11/03/so-you-want-to-scrape-lik...
I used to do freelance web scraping, and that article felt like some kind of forbidden knowledge. After reading the article, I went down the rabbit hole and actually found a Discord server that provided carrier-grade traffic relay from a van which contained dozens of phones.
For the questions..... we have to kinda wait a bit, someone from our engineering team might come here and reply.
By the way, as I have you here have you considered converting the CSV files to MMDB format? I was planning to do that with our mmdbctl tool later today.