Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Free full BGP feed. IPv4 and IPv6 (2020)

lukasz.bromirski.net

Post reply on HN
Print view View original

21–30 of 41 posts

Re: Free full BGP feed. IPv4 and IPv6 (2020)

#21

Post by toast0 »

Bender wrote: post #16
toast0 wrote: post #11

Earlier quoted context omitted.

> Just to have a compressed static file that contains all the routes one could download from time to time I think would be interesting to analyze. When I needed this for some work stuff, it was pretty easy to find table dumps and work with them? I don't remember where it was, but I'm sure you can find some. After an acquisition, we had our own ASN and I was able to get table dumps from our own infrastructure. > Same…

What do you think the root servers have that's not in the root.zone? Every glue record for every domain name. The glue is the valuable goo. Without the goo everyone goes back to /etc/hosts which I am not opposed to.

The root servers don't have the glue record for every domain name though. All they have is the glue records for the TLDs. (which is in the zone file you linked)

Re: Free full BGP feed. IPv4 and IPv6 (2020)

#22

Post by basilikum »

faangguyindia wrote: post #10

does anyone know where can i buy rDNS records? purpose is find all domains hosted on specific ip for a monthly price? I think crawling all my own is big environment waste of resources, i'd be happy to pay for a subscription or somethign

Do you mean rDNS records or a mapping of an IP address to all ([semi] publicly known) DNS records that point to an IP address? Cause these are two different things. Just because a.com points to 1.2.3.4 does not mean 1.2.3.4 has an rDNS record to a.com.

Re: Free full BGP feed. IPv4 and IPv6 (2020)

#23

Post by faangguyindia »

basilikum wrote: post #22
faangguyindia wrote: post #10

does anyone know where can i buy rDNS records? purpose is find all domains hosted on specific ip for a monthly price? I think crawling all my own is big environment waste of resources, i'd be happy to pay for a subscription or somethign

Do you mean rDNS records or a mapping of an IP address to all ([semi] publicly known) DNS records that point to an IP address? Cause these are two different things. Just because a.com points to 1.2.3.4 does not mean 1.2.3.4 has an rDNS record to a.com.

You are right, i'd need both actually. I can't find any service which provides weekly/monthly data download for a flat subscription price, so that i can perform local fast lookups.

Re: Free full BGP feed. IPv4 and IPv6 (2020)

#24

Post by faangguyindia »

zamadatix wrote: post #19
faangguyindia wrote: post #13

Earlier quoted context omitted.

i am talking about this data: https://host.io/lgyy.cc

They mean anyone will only ever be able to get some rather than all because of how DNS works. Even that site is unable to find all such mappings for every IP. I.e. the first option to build such a dataset is to use rDNS lookups, but it's optional to configure those and many don't bother (opting to just have the forward lookup). The other way about it is collecting domains and seeing which IPs they resolve to, but thi…

Sonar does seem to have data here: https://opendata.rapid7.com/

But it seems you've to go through their sales team and all for the data.

I can't find CommonCrawl making their DNS resolution available.

Re: Free full BGP feed. IPv4 and IPv6 (2020)

#25

Post by afshari »

satyamkapoor wrote: post #17
afshari wrote: post #7

[shameless plug] I made a tool to look into BGP data as well as other networky things at https://god.ad . Can be added to homescreen as a PWA app. https://god.ad/view/AS15169

Thanks, looks very llmish

Front-end was indeed done with Claude, with many iterations and tweaks though

Re: Free full BGP feed. IPv4 and IPv6 (2020)

#26

Post by basilikum »

faangguyindia wrote: post #23
basilikum wrote: post #22

Earlier quoted context omitted.

Do you mean rDNS records or a mapping of an IP address to all ([semi] publicly known) DNS records that point to an IP address? Cause these are two different things. Just because a.com points to 1.2.3.4 does not mean 1.2.3.4 has an rDNS record to a.com.

You are right, i'd need both actually. I can't find any service which provides weekly/monthly data download for a flat subscription price, so that i can perform local fast lookups.

How granular do you need this data? Domains are ultimately not publicly known, especially when it comes to subdomains. Do you just need domains that get a certain amount of public traffic or are you trying to get as much domains as possible? In the latter case scraping this data would require the logs of public DNS servers to even know of the existence of many domains (although certificate transparency logs could get one half the way).

Re: Free full BGP feed. IPv4 and IPv6 (2020)

#27

Post by zamadatix »

faangguyindia wrote: post #24
zamadatix wrote: post #19

Earlier quoted context omitted.

They mean anyone will only ever be able to get some rather than all because of how DNS works. Even that site is unable to find all such mappings for every IP. I.e. the first option to build such a dataset is to use rDNS lookups, but it's optional to configure those and many don't bother (opting to just have the forward lookup). The other way about it is collecting domains and seeing which IPs they resolve to, but thi…

Sonar does seem to have data here: https://opendata.rapid7.com/ But it seems you've to go through their sales team and all for the data. I can't find CommonCrawl making their DNS resolution available.

Common Crawl doesn't publish raw DNS separately, you have to pull the information out of the aggregate database. The WARC-IP-Address header should record the IP Common Crawl connected to for the site.

Re: Free full BGP feed. IPv4 and IPv6 (2020)

#28

Post by Bender »

toast0 wrote: post #21
Bender wrote: post #16

Earlier quoted context omitted.

What do you think the root servers have that's not in the root.zone? Every glue record for every domain name. The glue is the valuable goo. Without the goo everyone goes back to /etc/hosts which I am not opposed to.

The root servers don't have the glue record for every domain name though. All they have is the glue records for the TLDs. (which is in the zone file you linked)

Every time I stand up new name servers I have to add the glue records into the root servers or the name servers do not exist. In fairness to me I keep forgetting the trend is to shove everything into big centralized DNS servers as it is something I would never do at least not as a primary.

Re: Free full BGP feed. IPv4 and IPv6 (2020)

#29

Post by toast0 »

Bender wrote: post #28
toast0 wrote: post #21

Earlier quoted context omitted.

The root servers don't have the glue record for every domain name though. All they have is the glue records for the TLDs. (which is in the zone file you linked)

Every time I stand up new name servers I have to add the glue records into the root servers or the name servers do not exist. In fairness to me I keep forgetting the trend is to shove everything into big centralized DNS servers as it is something I would never do at least not as a primary.

Afaik, those glue records are held at your TLD's registry and served by the tld nameservers, not the root servers.

It might be nice to get a zone transfer for every tld, but that's not possible for the public. (I understand there's some way to get many of them, but $$$$)

Re: Free full BGP feed. IPv4 and IPv6 (2020)

#30

Post by johnhtodd »

Bender wrote: post #4

Just to have a compressed static file that contains all the routes one could download from time to time I think would be interesting to analyze. Same goes for the anycast root DNS servers. To have a full dump from them could be interesting. Not to be confused with the root.zone [1] I mean the whole kit and caboodle. [1] - https://www.internic.net/domain/root.zone

This is conflating BGP routes and DNS.

DNS data: Root server data is available via AXFR ("dig . AXFR @f.root-servers.net") but this isn't what you're referencing.

Second level TLD server data is available is available at CZDS. (https://czds.icann.org/home) but some TLDs don't participate, but this also isn't what you're referencing.

What I think you want: There is no canonical list of all zones that exist - there is no "central repository" once you pass the root zone downwards - that's a feature, not a bug. Some organizations have partial views based on large recursive resolver data (DomainTools, Google, Cisco, Cloudflare, Quad9) but access to that data is limited to vetted researchers or more typically only available at a cost (disclaimer: I work for Quad9.) Smaller versions of recursive data sets exist, but are usually significantly limited by geography and demographics of the user community that generates the data set.

BGP route data: This exists in many forms in realtime like the site referenced above, though historic data is difficult to track. No matter what the source or latency, there is bias in the data set because BGP pathing is unique to each ASN that collects it - no two views of the table are identical, and any data set is as "best guess" at state conditions at that time.

Here are some possible data sets for BGP:

Packet Clearing House (PCH) provides a set of snapshots going back 20+ years (though it seems to be offline at the moment): https://www.pch.net/resources/Routing_Data/

Cymru has a live version you can query via various APIs (including ironically via DNS): https://www.team-cymru.com/ip-asn-mapping

Routeviews from University of Oregon also has a data set that is widely used by researchers: https://www.routeviews.org/routeviews/

Post reply on HN