Step 1. Download Visual Basic
How to build a IP geolocation database from scratch?
101–110 of 186 posts
Re: How to build a IP geolocation database from scratch?
#102Earlier quoted context omitted.
I grok and understand your concern. I am not being defensive; I am just trying to provide an explanation. I really enjoy having conversations like this with developers as honestly and empathetically possible. I apologize if I was rude in any way by saying the word "meme". I saw a sister comment and thought you were being sarcastic. There is a popular meme about "I have your IP address", so I thought you were referenc…
Thank you for the well thought out response. I disagree with just about everything you say, but I understand where you're coming from and I appreciate the validation that the use of a VPN is more important than it's ever been. As a professional courtesy: calling yourself "essential" is an enormous red flag and you might want to consider different phrasing.
I should have used a different phrasing. :) I was reading an article about essential workers today, and that word popped up in my head when I wrote the comment.
It's good that you are using a VPN. I advocate for the usage of VPNs, and many VPN companies actually use our data to verify their server locations. In the VPN industry, VPN companies get their VPN servers from specialized hosting services that cater to dozens of VPN companies. You can check out the ASNs of the VPN IP addresses to find them.
VPN companies use our IP geolocation data to confirm the actual location of their servers. Let me tell you a fun story. One VPN company claimed to have a server in the Bahamas, but upon investigation, we discovered that the server was actually located in New York. It was a surprising find. Getting a server in the Bahamas is more challenging than getting one in NY. Just imagine users thinking their internet activity is immune to US jurisdiction because they are using a VPN service based in Bahamas but in fact it is actually located in NY. So, we might not be essential, but we are certainly very useful!
Thank you for the great conversation, dude. Appreciate it.
Re: How to build a IP geolocation database from scratch?
#103It all depends on what you want to use it for and how accurate it needs to be. The best way to build a geolocation service is to have a billion devices that report their location to you at the same time they report their IP to you. That's basically Apple and Google. They have by far the best geolocation databases in the world, because they get constant updates of IP and location. The trick is basically to make an app…
Even 10 years ago, Apple internal privacy policies prevented itself from collecting precise lat/long. We had to use HTTP session telemetry to determine which endpoints were best for a given IP (or subnet, but not ASN), which informed our own pseudo-geoIP database so we knew which endpoint to connect to based on real world conditions.
Even still, it had to be as ephemeral as possible for the sake of privacy. We weren’t allowed to use or record results from Apple Maps’ reverse geo service outside of the context of a live user request (finding nearby restaurants, etc).
Re: How to build a IP geolocation database from scratch?
#104Earlier quoted context omitted.
This is literally the most prominent thing on the https://ipinfo.io home page.
Huh, that's cool. It got my home IP about 15 miles from where I am, but still not bad. Wait - how does this work for cell IPs? A lot of cellphone v4 IPs are now shared between hundreds or thousands of devices, right?
I work there, and I am supposed to know these things, but I don't exactly :/
It probably has something to do with important routers. What tags do we show when you visit the IP data page? The IP data page can be accessed by visiting ipinfo.io/.
We use the generic term "data experts," but it actually consists of about 2 dozen engineers, including data engineers, data scientists, infrastructure engineers, backend engineers, and a great technical CEO working on all that. All those folks have gone on a boating trip off the coast of Spain for a retreat.....except for me.
I will ask them and try to circle back with some answers.
Re: How to build a IP geolocation database from scratch?
#105Earlier quoted context omitted.
In the business of media streaming, rightholder will require that you check for vpn and proxies in addition to countries when deciding if a given viewer will be able to stream a given media.
Does that actually work? That could explain an issue with a particular streaming service I use. There are currently some ongoing routing issues in BGP land and my ISP. When trying to stream, it says I’m using a proxy, so due to the incredible route my packets are taking, that might be it. What’s funny is that the only way to watch this service is to use a vpn right now.
of course it doesnt work but they gotta try clutching pearls and applying whatever pressure they can think of on these fronts
Re: How to build a IP geolocation database from scratch?
#106It all depends on what you want to use it for and how accurate it needs to be. The best way to build a geolocation service is to have a billion devices that report their location to you at the same time they report their IP to you. That's basically Apple and Google. They have by far the best geolocation databases in the world, because they get constant updates of IP and location. The trick is basically to make an app…
Even 10 years ago, Apple internal privacy policies prevented itself from collecting precise lat/long. We had to use HTTP session telemetry to determine which endpoints were best for a given IP (or subnet, but not ASN), which informed our own pseudo-geoIP database so we knew which endpoint to connect to based on real world conditions. Even still, it had to be as ephemeral as possible for the sake of privacy. We weren’…
You don't need precise lat/lon to make a good database. Even a 1km circle would be more than enough.
> but not ASN
Why wasn't ASN allowed? That's what Netflix used to make endpoint routing decisions and worked really well.
Re: How to build a IP geolocation database from scratch?
#107First, I am big fan of your articles even before I joined IPinfo, where we provide IP geolocation data service. Our geolocation methodology expands on the methodology you described. We utilize some of the publicly available datasets that you are using. However, the core geolocation data comes from our ping-based operation. We ping an IP address from multiple servers across the world and identify the location of the I…
Would you consider no-signup inspection of the data you hold on the requesters IP address? I would love to see what you have on MY IP address, and if sufficiency accurate it feels that it would be a good incentive to sign up to use commerically. It feels like it couldn't be abused by 'freeloaders', because i'd guess their use-case is viewing other peoples.
We have a very open approach to our data. In fact, our website is extremely accessible. It is quite useful for researching IP addresses and does not require signing up. The data is largely available to view on the website. Although we display all IP address meta data on the home page, if you intend to use our website frequently, I recommend utilizing the IP data pages.
You can enter IP addresses on the right side to look up information here: https://ipinfo.io/what-is-my-ip
Additionally, we offer some enjoyable tools that you can use here: https://ipinfo.io/tools
The CLI tool is particularly entertaining.
You can also use our API service without signing up, with a limit of 1000 requests per day.
If you do choose to sign up for a free account, you will receive 50,000 requests per month, free IP databases, a bulk lookup feature, and more.
Re: How to build a IP geolocation database from scratch?
#108Earlier quoted context omitted.
Even 10 years ago, Apple internal privacy policies prevented itself from collecting precise lat/long. We had to use HTTP session telemetry to determine which endpoints were best for a given IP (or subnet, but not ASN), which informed our own pseudo-geoIP database so we knew which endpoint to connect to based on real world conditions. Even still, it had to be as ephemeral as possible for the sake of privacy. We weren’…
You don't need precise lat/lon to make a good database. Even a 1km circle would be more than enough. > but not ASN Why wasn't ASN allowed? That's what Netflix used to make endpoint routing decisions and worked really well.
You’re not wrong, but privacy concerns were paramount.
ASNs were allowed but too vague. We needed more granularity. Corporate proxies, subdelegations, many providers aggregating announcements below /24, etc.
Re: How to build a IP geolocation database from scratch?
#109[dead]
Re: How to build a IP geolocation database from scratch?
#110Comments seem fairly dismissive but I actually found this really interesting. It reminds me of a task I had in my first position to add PostGIS to our database and a location based search. That was based off addresses and zipcodes.
That's relatively simple to do, even in mysql. One trick is to use a square instead of a circle, which avoids a lot of math.