Earlier quoted context omitted.
just out of curiosity, what are you (and the other commenters) planning to use this for? I myself was trying to download all X509 server certificates on the Internet I could find to: - gain some insight into the dealings of CAs - see if private keys were reused across multiple sites - check the average expiry date of certificates etc.
This might be useful for that: https://scans.io/study/sonar.ssl
How to Download a List of All Registered Domain Names
51–55 of 55 posts
Re: How to Download a List of All Registered Domain Names
#52A warning about parsing zone files... the grammar is deceptively tricky. While TLD registries will probably provide you with files in a sane subset[0] of that specified in RFC 1035, there are a number of things that will NOT work in general: - Splitting the file in to lines (paren-blocks and quoted strings can span lines, strings can contain ';' etc). - Splitting the file on whitespace (it's significant in column 1 a…
Yep. I use the BIND tool named-compilezone to canonicalize zone files, which allows me to apply simple regex parsing, because I can assume one record per line, all fields present, and no abbreviated names. Main disadvantage is it is not very fast.
Re: How to Download a List of All Registered Domain Names
#53I think its sad how closed this data is.
There are very good reasons for this data being closed, not least of which is that allowing zone transfer by arbitrary individuals is an excellent way of allowing your DNS server to be DOS'd.
Re: How to Download a List of All Registered Domain Names
#54What about CCTLDs?
They available from a third party source such as http://domains-index.com/
Re: How to Download a List of All Registered Domain Names
#55Earlier quoted context omitted.
There are very good reasons for this data being closed, not least of which is that allowing zone transfer by arbitrary individuals is an excellent way of allowing your DNS server to be DOS'd.
How is that achieved with read only access to the list of registered domains?
If you're a domain registry, your zone files are huge. Allowing arbitrary zone transfers could potentially put massive sustained strain on their DNS infrastructure. And thus because only a very small number of nameservers really need to be able to perform zone transfers against their nameservers, they're better off locking down the ability.
If you're running your own nameservers, then it's still worth locking down zone transfers for similar reasons. At the very least, it gives you a degree of defence in depth as you're giving attackers less of an opportunity to gather information on the structure of your network. If they could simply do a zone transfer to find out all the names in a given zone, then they don't have to do more costly brute force enumeration to guess at the hosts in the zone.
Take a read of this for why, if you run your own nameservers, you shouldn't allow arbitrary zone transfers: http://www.iodigitalsec.com/dns-zone-transfer-axfr-vulnerabi...