Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Free full BGP feed. IPv4 and IPv6 (2020)

lukasz.bromirski.net

Post reply on HN
Print view View original

11–20 of 41 posts

Re: Free full BGP feed. IPv4 and IPv6 (2020)

#11

Post by toast0 »

Bender wrote: post #4

Just to have a compressed static file that contains all the routes one could download from time to time I think would be interesting to analyze. Same goes for the anycast root DNS servers. To have a full dump from them could be interesting. Not to be confused with the root.zone [1] I mean the whole kit and caboodle. [1] - https://www.internic.net/domain/root.zone

> Just to have a compressed static file that contains all the routes one could download from time to time I think would be interesting to analyze.

When I needed this for some work stuff, it was pretty easy to find table dumps and work with them? I don't remember where it was, but I'm sure you can find some. After an acquisition, we had our own ASN and I was able to get table dumps from our own infrastructure.

> Same goes for the anycast root DNS servers. To have a full dump from them could be interesting. Not to be confused with the root.zone [1] I mean the whole kit and caboodle.

What do you think the root servers have that's not in the root.zone? I think you can AXFR from the root servers, too, but it should be the same thing as from the https site.

Re: Free full BGP feed. IPv4 and IPv6 (2020)

#12

Post by yjftsjthsd-h »

faangguyindia wrote: post #10

does anyone know where can i buy rDNS records? purpose is find all domains hosted on specific ip for a monthly price? I think crawling all my own is big environment waste of resources, i'd be happy to pay for a subscription or somethign

> does anyone know where can i buy rDNS records? purpose is find all domains hosted on specific ip for a monthly price

Are rdns records always populated? I thought they were optional and many sites didn't bother

Re: Free full BGP feed. IPv4 and IPv6 (2020)

#13

Post by faangguyindia »

yjftsjthsd-h wrote: post #12
faangguyindia wrote: post #10

does anyone know where can i buy rDNS records? purpose is find all domains hosted on specific ip for a monthly price? I think crawling all my own is big environment waste of resources, i'd be happy to pay for a subscription or somethign

> does anyone know where can i buy rDNS records? purpose is find all domains hosted on specific ip for a monthly price Are rdns records always populated? I thought they were optional and many sites didn't bother

i am talking about this data: https://host.io/lgyy.cc

Re: Free full BGP feed. IPv4 and IPv6 (2020)

#14

Post by [deleted] »

Bender wrote: post #4

Just to have a compressed static file that contains all the routes one could download from time to time I think would be interesting to analyze. Same goes for the anycast root DNS servers. To have a full dump from them could be interesting. Not to be confused with the root.zone [1] I mean the whole kit and caboodle. [1] - https://www.internic.net/domain/root.zone

[deleted]

Re: Free full BGP feed. IPv4 and IPv6 (2020)

#16

Post by Bender »

toast0 wrote: post #11
Bender wrote: post #4

Just to have a compressed static file that contains all the routes one could download from time to time I think would be interesting to analyze. Same goes for the anycast root DNS servers. To have a full dump from them could be interesting. Not to be confused with the root.zone [1] I mean the whole kit and caboodle. [1] - https://www.internic.net/domain/root.zone

> Just to have a compressed static file that contains all the routes one could download from time to time I think would be interesting to analyze. When I needed this for some work stuff, it was pretty easy to find table dumps and work with them? I don't remember where it was, but I'm sure you can find some. After an acquisition, we had our own ASN and I was able to get table dumps from our own infrastructure. > Same…

What do you think the root servers have that's not in the root.zone?

Every glue record for every domain name. The glue is the valuable goo. Without the goo everyone goes back to /etc/hosts which I am not opposed to.

Re: Free full BGP feed. IPv4 and IPv6 (2020)

#18

Post by satyamkapoor »

satyamkapoor wrote: post #17
afshari wrote: post #7

[shameless plug] I made a tool to look into BGP data as well as other networky things at https://god.ad . Can be added to homescreen as a PWA app. https://god.ad/view/AS15169

Thanks, looks very llmish

Doesn’t mean it’s bad. Thanks for the work

Re: Free full BGP feed. IPv4 and IPv6 (2020)

#19

Post by zamadatix »

faangguyindia wrote: post #13
yjftsjthsd-h wrote: post #12

Earlier quoted context omitted.

> does anyone know where can i buy rDNS records? purpose is find all domains hosted on specific ip for a monthly price Are rdns records always populated? I thought they were optional and many sites didn't bother

i am talking about this data: https://host.io/lgyy.cc

They mean anyone will only ever be able to get some rather than all because of how DNS works. Even that site is unable to find all such mappings for every IP. I.e. the first option to build such a dataset is to use rDNS lookups, but it's optional to configure those and many don't bother (opting to just have the forward lookup). The other way about it is collecting domains and seeing which IPs they resolve to, but this is only ever partial (there is no such thig as a conplete lists of domains in the world) and has additional problems with CDNs or things hosted in more than one place at once (you need to do a lookup from every possible location each domain could conceivably be hosted to be sure you get all of the reverse mappings for that domain).

As an example: Unless you know my domains ahead of time you'll never be able to come up with what domains are hosted on my IPs because I don't bother to configure rDNS. So those IPs will look like they host no servers (or only some if you only had a partial list of my domains) rather than all of them on those IPs.

Anyways, for free data sources trying to get a partial view of this you can check out Rapid7's Sonar or Common Crawl. Each should have the pieces needed to construct this kind of view from the data.

Post reply on HN