Live data from Hacker News

How to build a IP geolocation database from scratch?

ipapi.is

151–160 of 186 posts

Re: How to build a IP geolocation database from scratch?

#151
post #143

First, I am big fan of your articles even before I joined IPinfo, where we provide IP geolocation data service. Our geolocation methodology expands on the methodology you described. We utilize some of the publicly available datasets that you are using. However, the core geolocation data comes from our ping-based operation. We ping an IP address from multiple servers across the world and identify the location of the I…

Are there any historical sources for geo ip info?

We don't have any free data for that. We have historical data that we sell as part of our custom enterprise deals. Historical data requests are rare, though.

A time series IP database requires a substantial amount of storage and computational cost to query, as I imagine. The city level geolocation data we have is ~1.5 gb in size. IP range data is complicated to query efficiently as you need to understand data platform settings and good amount of computer network math and computer science stuff. Adding a layer of time series complexities on top of that, makes this process quite difficult.

To give you some context of how IP metadata lookups work, you can check out this article

https://ipinfo.io/blog/ip-address-data-in-snowflake/

Even if you keep all your database in a binary format, the computational cost is still non-negligible.

Re: How to build a IP geolocation database from scratch?

#152

If you don't want to do this yourself, you can actually just get Cloudflare to do it for you for free using a simple Worker since all Cloudflare requests contain approximate IP location information. You can also just send a request to my URL (Cloudflare Worker operated - so it should have global low latency): https://www.edenmaps.net/iplocation Use it for small applications, I don't mind. Just don't start sending me…

I'm in Munich. Cloudflare tells a position that is 730km to the north in a random forest.

You've inverted lat, lon.

Re: How to build a IP geolocation database from scratch?

#153

Earlier quoted context omitted.

I work for IPinfo and we do ping based geolocation. The best thing you can do to verify geolocation accuracy is the following: - Download a few free IP databases - Generate a random list of IP addresses - Do the IP address lookups across all those databases - Identify the IP address that can be pinged - Visit a site that can ping an IP address from multiple server - Sort the results by lowest avg ping time Then check…

On one hand, I love that there’s some good alternatives in the geolocation space, but misleading geolocation precision can lead to very undesirable side effects[0]. [0] https://www.theguardian.com/technology/2016/aug/09/maxmind-m...

That is part of the reason why I am here in HN. I am pretty down to earth and willing to talk with anyone about IP metadata.

Our business approach is that we will come to you before you come to us. Whenever people complain about something related to IP data, I will just reach out to them and verify that we provide good data. Our support team is extremely responsive. We also have a community where we walk through users about IP metadata. I am super active on HN, Reddit and Twitter.

The challenge for me is usually that we will tell people, "We are providing good geolocation data for you, but the service that provides geolocation data to the service you are complaining about is not using our data. Our hands are tied." I have reached out to bigger corporations and streaming services to say, "You are providing bad data to your customers. I am not even trying to sell you something. Even using our free database could lead you to better results." But I have not heard from them yet. It is frustrating to me as it is for the person with bad IP data.

IP geolocation is never going to be absolutely accurate or precise. We understand that completely. Aside from continuously investing in improving our data from a product standpoint, we also take a very human approach of trying to fix every geolocation complaint one by one.

Re: How to build a IP geolocation database from scratch?

#154

First, I am big fan of your articles even before I joined IPinfo, where we provide IP geolocation data service. Our geolocation methodology expands on the methodology you described. We utilize some of the publicly available datasets that you are using. However, the core geolocation data comes from our ping-based operation. We ping an IP address from multiple servers across the world and identify the location of the I…

Have you considered making your database available for download as Parquet format so people could just copy the file to S3, Google Cloud, etc, and query it immediately with various tools? I know it can be done with CSV but it's not as smooth.

Thank you for the feature request.

We usually just send users the documentation of ingesting the data in CSV or NDJSON format (Newline Delimited JSON). We don't actually get many requests for data downloads in Parquet format. I think we have a few customers where we deliver the data in parquet format directly to their cloud storage bucket.

But keep an eye out for our emails if we announce the parquet data downloads. I will talk with the folks about this.

BUT, there are some good news.

At least for the free database, we deliver the data directly to data warehouse platforms. Not even storage buckets. And we supply a good amount of documentation.

We have the free database in Snowflake, GCP, Kaggle, and Splitgraph, and we are working on a few more deals. For the free database, atleast, we are working on better things than parquet. Like literally one-click solution to bring the IP data to your data warehouse.

Kaggle: https://www.kaggle.com/code/ipinfo/ipinfo-ip-to-country-asn-...

Snowflake: https://app.snowflake.com/marketplace/listing/GZSTZSHKQ4QY/i...

If you want to use our free IP database on Google Cloud or BigQuery, please send us an email (support@ipinfo.io) and mention that the DevRel sent you from HN. I can easily set you up with the free IP database in GCP/BQ.

Re: How to build a IP geolocation database from scratch?

#155
post #116

First, I am big fan of your articles even before I joined IPinfo, where we provide IP geolocation data service. Our geolocation methodology expands on the methodology you described. We utilize some of the publicly available datasets that you are using. However, the core geolocation data comes from our ping-based operation. We ping an IP address from multiple servers across the world and identify the location of the I…

Great comment. I'm a big fan and customer of IPinfo, using your API in our login notification emails to say "You just logged in from Berlin, Germany. If this wasn't you click here." To provide country data for customers in their audit logs. And for anti-spam and fraud detection.

I appreciate it, sir! If you have any questions or feedback, please let us know.

The challenge of being a data provider is that you can use our data in a million ways, and we don't have coverage of all. So, when you come up with questions or ideas, we can help you better.

As you mentioned, audit logs. I highly recommend you look into the ASN field.

The ASN identifies an organization that owns a block of IP addresses. In my experience, I have found that the combination of ASN+Country is the most valuable information you can use in spam and fraud detection. You can fake the IP geolocation information with a VPN. However, it is not as easy to fake the ASN information of the IP address. So, when you use a combination of country + ASN, you can have a robust cybersecurity system.

Re: How to build a IP geolocation database from scratch?

#157
post #116

Earlier quoted context omitted.

Great comment. I'm a big fan and customer of IPinfo, using your API in our login notification emails to say "You just logged in from Berlin, Germany. If this wasn't you click here." To provide country data for customers in their audit logs. And for anti-spam and fraud detection.

I appreciate it, sir! If you have any questions or feedback, please let us know. The challenge of being a data provider is that you can use our data in a million ways, and we don't have coverage of all. So, when you come up with questions or ideas, we can help you better. As you mentioned, audit logs. I highly recommend you look into the ASN field. The ASN identifies an organization that owns a block of IP addresses.…

Can you explain more how to use ASN to detect fraud and how it's different from the country detected for the IP? I thought ASN was derived from the IP, basically the route to that IP? Here's the ipinfo response for an IP used by a recent fraud signup attempt. The asn field matches country.

  {
    "city": "Mumbai",
    "connection": {
      "asn": 24560,
      "isp": "Bharti Airtel Ltd."
    },
    "continent_code": "AS",
    "continent_name": "Asia",
    "country_code": "IN",
    "country_name": "India",
    "currency": {
      "code": "INR",
      "name": "Indian Rupee",
      "plural": "Indian rupees",
      "symbol": "Rs",
      "symbol_native": "\u099f\u0995\u09be"
    },
    "ip": "2401:4900:1f38:7402:5569:2e45:3bb:9c0d",
    "latitude": 19.076000213623047,
    "location": {
      "calling_code": "91",
      "capital": "New Delhi",
      "country_flag": "https://assets.ipstack.com/flags/in.svg",
      "country_flag_emoji": "\ud83c\uddee\ud83c\uddf3",
      "country_flag_emoji_unicode": "U+1F1EE U+1F1F3",
      "geoname_id": 1275339,
      "is_eu": false,
      "languages": [
        {
          "code": "hi",
          "name": "Hindi",
          "native": "\u0939\u093f\u0928\u094d\u0926\u0940"
        },
        {
          "code": "en",
          "name": "English",
          "native": "English"
        }
      ]
    },
    "longitude": 72.87770080566406,
    "region_code": "MH",
    "region_name": "Maharashtra",
    "time_zone": {
      "code": "IST",
      "current_time": "2023-09-15T10:52:42+05:30",
      "gmt_offset": 19800,
      "id": "Asia/Kolkata",
      "is_daylight_saving": false
    },
    "type": "ipv6",
    "zip": "400203"
  }
Here's the response from ipinfo.io which includes privacy fields. It's technically a proxy but might be hard to detect because it's probably a crowdsourced/botnet proxy not a public one. We don't pay for

  {
    "ip": "2401:4900:1f38:7402:5569:2e45:3bb:9c0d",
    "city": "Najafgarh",
    "region": "Delhi",
    "country": "IN",
    "loc": "28.6114,77.2982",
    "org": "AS24560 Bharti Airtel Ltd., Telemedia Services",
    "postal": "110097",
    "timezone": "Asia/Kolkata",
    "asn": {
      "asn": "AS24560",
      "name": "Bharti Airtel Ltd., Telemedia Services",
      "domain": "airtel.com",
      "route": "2401:4900:1f38::/48",
      "type": "isp"
    },
    "company": {
      "name": "ABTS (Karnataka),",
      "domain": "airtel.com",
      "type": "isp"
    },
    "privacy": {
      "vpn": false,
      "proxy": false,
      "tor": false,
      "relay": false,
      "hosting": false,
      "service": ""
    },
    "abuse": {
      "address": "Bharti Airtel Ltd., ISP Division - Transport Network Group, 234 , Okhla Industrial Estate,, Phase III, New Delhi-110020, INDIA",
      "country": "IN",
      "email": "ip.misuse@airtel.com",
      "name": "ABUSE BHARTIIN",
      "network": "2401:4900:1f30::/44",
      "phone": "+000000000"
    }
  }
EDIT: Oops, I confused ipinfo with ipstack. I'm actually using ipstack. Their security field also doesn't detect this IP as a proxy, which is why we only pay for Professional (no security field).

  {
    "ip": "2401:4900:1f38:7402:5569:2e45:3bb:9c0d",
    "type": "ipv6",
    "continent_code": "AS",
    "continent_name": "Asia",
    "country_code": "IN",
    "country_name": "India",
    "region_code": "MH",
    "region_name": "Maharashtra",
    "city": "Mumbai",
    "zip": "400203",
    "latitude": 19.076000213623047,
    "longitude": 72.87770080566406,
    "location": {
      "geoname_id": 1275339,
      "capital": "New Delhi",
      "languages": [
        {
          "code": "hi",
          "name": "Hindi",
          "native": "\u0939\u093f\u0928\u094d\u0926\u0940"
        },
        {
          "code": "en",
          "name": "English",
          "native": "English"
        }
      ],
      "country_flag": "https://assets.ipstack.com/flags/in.svg",
      "country_flag_emoji": "\ud83c\uddee\ud83c\uddf3",
      "country_flag_emoji_unicode": "U+1F1EE U+1F1F3",
      "calling_code": "91",
      "is_eu": false
    },
    "time_zone": {
      "id": "Asia/Kolkata",
      "current_time": "2023-09-15T12:27:08+05:30",
      "gmt_offset": 19800,
      "code": "IST",
      "is_daylight_saving": false
    },
    "currency": {
      "code": "INR",
      "name": "Indian Rupee",
      "plural": "Indian rupees",
      "symbol": "Rs",
      "symbol_native": "\u099f\u0995\u09be"
    },
    "connection": {
      "asn": 24560,
      "isp": "Bharti Airtel Ltd."
    },
    "security": {
      "is_proxy": false,
      "proxy_type": null,
      "is_crawler": false,
      "crawler_name": null,
      "crawler_type": null,
      "is_tor": false,
      "threat_level": "low",
      "threat_types": null
    }
  }

Re: How to build a IP geolocation database from scratch?

#158

Earlier quoted context omitted.

That is a good point. I will try to add some additional points to your statement. In the context of a butcher and a vegan we are not either of them, we would be someone who sell salt. Salt as an ingredient is used by vegans, vegetarians, and meat eaters. Salt is useful, and it is a complimentary item that makes the food taste better. But it is not a main feature of any dish. Salt plays an important and somewhat unive…

Huh. Ignoring the somewhat strained metaphor, you seem to be suggesting that you don't have moral responsibility for aiding the creepiness of professional manipulators because you also enable other things. I don't think that's true. But even if it were, it would only be so if it were a minimal, possibly negligible part of your business. What percentage of revenue would you say comes from adtech?

I am happy to assist any business, organization or person to use our data. I enjoy talking with people and helping them with the technical and philosophical aspects of IP metadata.

If you progressively remove the layers of metadata associated with internet-connected devices one by one from GPS data (user-permitted action) → IP geolocation (IPinfo.io)→ IP address (ISP), and come to a system that is completely anonymous, you have the dark web.

The dark web represents an ecosystem that is completely devoid of ad tech because there is no identity, and the users are purely homogenous. The system itself is not widely adopted because internet participants need to feel secure from cyberattacks and harmful words. Traditional internet services do not exist there because they are getting bombarded with anonymous attacks, and they cannot provide any value to the customer because they don't even know who/what the user is.

IP metadata helps with running a business and serves as a way to protect yourself. There are organizations and users who actively utilize our services to check email headers, find the IP address, and lookup the IP metadata of those IP addresses. The data is utilized for recognizing spam and phishing emails. Conversely, our service is also used in adtech-driven cold emails and newsletter services.

IP metadata is required for a functioning and widely adopted internet.

Re: How to build a IP geolocation database from scratch?

#159

First, I am big fan of your articles even before I joined IPinfo, where we provide IP geolocation data service. Our geolocation methodology expands on the methodology you described. We utilize some of the publicly available datasets that you are using. However, the core geolocation data comes from our ping-based operation. We ping an IP address from multiple servers across the world and identify the location of the I…

Hi, cool idea with the geolocation via latency. But I encountered 2 things using ipinfo: Hetzner Server that are in Germany in a fixed location that never moved are sometimes located in another country, for me it was once s Server placed into Moscow and once in South America. How does this happen?

If you can give me some information about the IP address or the IP range, I can take a closer look.

I guess it is because of IPv4 trading or IP address shuffling.

As far as I know, Hertzer, like many hosting companies, is buying IPv4 addresses around the world. Here is an article on the IPv4 trades:

https://tech.marksblogg.com/ipinfo-free-ip-address-location-...

When a company buys an IP address block or relocates an IP block from one of its data centers to another, the location of those IP addresses changes.

If your IP address is static, but we have made an error in geolocation, I would love to take a closer look. You can email our support (support@ipinfo.io) and send a link to the comment. We can discuss it further from there.

Re: How to build a IP geolocation database from scratch?

#160

First, I am big fan of your articles even before I joined IPinfo, where we provide IP geolocation data service. Our geolocation methodology expands on the methodology you described. We utilize some of the publicly available datasets that you are using. However, the core geolocation data comes from our ping-based operation. We ping an IP address from multiple servers across the world and identify the location of the I…

hm, ipinfo.io tells me that I'm using a VPN even though I'm not...
Post reply on HN