I think its sad how closed this data is.
There are very good reasons for this data being closed, not least of which is that allowing zone transfer by arbitrary individuals is an excellent way of allowing your DNS server to be DOS'd.
How to Download a List of All Registered Domain Names
41–50 of 55 posts
Re: How to Download a List of All Registered Domain Names
#42FWIW, a TLD zone file does not contain every registered domain name, just those with DNS records. There is typically a good amount of domain names registered but without records, for reasons such as reserved names, malicious content takedowns, etc.
Exactly. The title "How to Download a List of All Registered Domain Names" is not correct.
For example in .com .net if a registrar puts a domain "on hold" that pulls it from the zone file. As such it will not appear in any zone file download. This is in fact a way for a company to keep a domain under wraps. As such:
register domain name (in .com )
pull the name servers
The name won't be in any zone file. And will be off the radar until it has nameservers or until someone guesses to see if it's been registered.
(Note other registries work differently (or the same) the above is specific to .com and .net)
Re: How to Download a List of All Registered Domain Names
#43A couple of months ago I processed all metadata from the Common Crawl project for all indexed domain names. This was about 10TB of metadata and resulted in 26 million domain names. EC2 costs were only about 10$ to process this. If anyone is interested, let me know. edit: available as torrent here: https://all-certificates.s3.amazonaws.com/domainnames.gz?tor...
Any way to extract just country specific domains without processing all 10TB of it?
Re: How to Download a List of All Registered Domain Names
#44A couple of months ago I processed all metadata from the Common Crawl project for all indexed domain names. This was about 10TB of metadata and resulted in 26 million domain names. EC2 costs were only about 10$ to process this. If anyone is interested, let me know. edit: available as torrent here: https://all-certificates.s3.amazonaws.com/domainnames.gz?tor...
Any way to extract just country specific domains without processing all 10TB of it?
No, but my torrent is just the list of unique domain names and 147MB gzipped.
Re: How to Download a List of All Registered Domain Names
#45What about CCTLDs?
ccTLDs are the responsibility of "designated managers" for each sovereign country (see https://tools.ietf.org/html/rfc1591).
Re: How to Download a List of All Registered Domain Names
#46A couple of months ago I processed all metadata from the Common Crawl project for all indexed domain names. This was about 10TB of metadata and resulted in 26 million domain names. EC2 costs were only about 10$ to process this. If anyone is interested, let me know. edit: available as torrent here: https://all-certificates.s3.amazonaws.com/domainnames.gz?tor...
Synchronicity :-) Just this week I scraped and analyzed only the homepages from the Alexa top 1 million [1] and majestic million [2]. I used rqworker[3] and and a fleet of mini-servers from scaleway[4] to do the scraping. Some results:
- 1.8MM successful scrapes - 25GB total size (stored in postgres) - 1 server to host redis and postgres - 9 physical servers for workers (4x core ARM servers) = 36 total cores - Peak rate of ~ 100 reqs/second achieved across all workers (36 physical cores total) - I saw that I could oversubscribe workers to cores by a factor of 2x (72 total workers) to achieve ~75% utilization on each server
All in all it was a fun project that provided quite a bit of learning. There are a lot of levers to pull here to make things faster, robust, and more portable. Next steps are to optimize the database and wrap a simple django app around it for exploring data.
Or maybe push it futher and try my hand at these 26MM domains?
[1] - http://s3.amazonaws.com/alexa-static/top-1m.csv.zip
[2] - https://blog.majestic.com/development/majestic-million-csv-d...
[3] - http://python-rq.org/
[4] - http://scaleway.com
Re: How to Download a List of All Registered Domain Names
#47Earlier quoted context omitted.
thank you for this!
just out of curiosity, what are you (and the other commenters) planning to use this for? I myself was trying to download all X509 server certificates on the Internet I could find to: - gain some insight into the dealings of CAs - see if private keys were reused across multiple sites - check the average expiry date of certificates etc.
This might be useful for that: https://scans.io/study/sonar.ssl
Re: How to Download a List of All Registered Domain Names
#48What about CCTLDs?
They available from a third party source such as http://domains-index.com/
Re: How to Download a List of All Registered Domain Names
#49This list would be useful for my attempt for a list of parked/squatted domains..
Re: How to Download a List of All Registered Domain Names
#50A warning about parsing zone files... the grammar is deceptively tricky. While TLD registries will probably provide you with files in a sane subset[0] of that specified in RFC 1035, there are a number of things that will NOT work in general: - Splitting the file in to lines (paren-blocks and quoted strings can span lines, strings can contain ';' etc). - Splitting the file on whitespace (it's significant in column 1 a…
Yep.
I use the BIND tool named-compilezone to canonicalize zone files, which allows me to apply simple regex parsing, because I can assume one record per line, all fields present, and no abbreviated names. Main disadvantage is it is not very fast.