Live data from Hacker News

How to Download a List of All Registered Domain Names

jordan-wright.com

51–55 of 55 posts

Re: How to Download a List of All Registered Domain Names

#51
post #47

Earlier quoted context omitted.

just out of curiosity, what are you (and the other commenters) planning to use this for? I myself was trying to download all X509 server certificates on the Internet I could find to: - gain some insight into the dealings of CAs - see if private keys were reused across multiple sites - check the average expiry date of certificates etc.

This might be useful for that: https://scans.io/study/sonar.ssl

Yes I found that some time afterwards, so I decided not to continue with it ;) I did gather about 1.4M certificates in the process though.

Re: How to Download a List of All Registered Domain Names

#52
post #50
post #19

A warning about parsing zone files... the grammar is deceptively tricky. While TLD registries will probably provide you with files in a sane subset[0] of that specified in RFC 1035, there are a number of things that will NOT work in general: - Splitting the file in to lines (paren-blocks and quoted strings can span lines, strings can contain ';' etc). - Splitting the file on whitespace (it's significant in column 1 a…

Yep. I use the BIND tool named-compilezone to canonicalize zone files, which allows me to apply simple regex parsing, because I can assume one record per line, all fields present, and no abbreviated names. Main disadvantage is it is not very fast.

It's probably slow because by default it does a bunch of DNS queries to foreign zones to see if your NS records etc are good.

Re: How to Download a List of All Registered Domain Names

#53

I think its sad how closed this data is.

There are very good reasons for this data being closed, not least of which is that allowing zone transfer by arbitrary individuals is an excellent way of allowing your DNS server to be DOS'd.

How is that achieved with read only access to the list of registered domains?

Re: How to Download a List of All Registered Domain Names

#54
post #48

What about CCTLDs?

They available from a third party source such as http://domains-index.com/

Yeah. My point was, he wrote an article titled "How to Download a List of All Registered Domain Names", and then didn't even mention the existence of cctlds. Which is like writing an article titled "How to learn how to speak every language", and then pretending there are only 10 languages in existence.

Re: How to Download a List of All Registered Domain Names

#55

Earlier quoted context omitted.

There are very good reasons for this data being closed, not least of which is that allowing zone transfer by arbitrary individuals is an excellent way of allowing your DNS server to be DOS'd.

How is that achieved with read only access to the list of registered domains?

I'm not sure what you mean. Do you know what a zone transfer is? If you wanted to get a list of the domains and records published in a nameserver, you would perform a zone transfer. Because that can amount to quite a bit of information being transferred, if a nameserver allows unrestricted zone transfers, that's a vector for a denial of service attack against that nameserver.

If you're a domain registry, your zone files are huge. Allowing arbitrary zone transfers could potentially put massive sustained strain on their DNS infrastructure. And thus because only a very small number of nameservers really need to be able to perform zone transfers against their nameservers, they're better off locking down the ability.

If you're running your own nameservers, then it's still worth locking down zone transfers for similar reasons. At the very least, it gives you a degree of defence in depth as you're giving attackers less of an opportunity to gather information on the structure of your network. If they could simply do a zone transfer to find out all the names in a given zone, then they don't have to do more costly brute force enumeration to guess at the hosts in the zone.

Take a read of this for why, if you run your own nameservers, you shouldn't allow arbitrary zone transfers: http://www.iodigitalsec.com/dns-zone-transfer-axfr-vulnerabi...

Post reply on HN