Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to Download a List of All Registered Domain Names

jordan-wright.com

Post reply on HN
Print view View original

11–20 of 55 posts

Re: How to Download a List of All Registered Domain Names

#11

Post by scandinavian »

vortico wrote: post #3

What if someone were to maintain an unofficial list with one domain per line, freely available as a daily torrent or served directly? Would there be a rights problem with mirroring and filtering ICANN data?

A lot of tlds don't provide zone files unless you are a registrar. They would probably not be happy if someone put those out to the pubic. For com and the likes they would probably not care as much.

The best public list of domains I have found is the Project Sonar DNS (ANY) scans. I don't know how they do it, but their scans are pretty complete, at least for.dk domains, which are the ones I use.

https://scans.io/study/sonar.fdns

Re: How to Download a List of All Registered Domain Names

#12

Post by ajcr »

vortico wrote: post #3

What if someone were to maintain an unofficial list with one domain per line, freely available as a daily torrent or served directly? Would there be a rights problem with mirroring and filtering ICANN data?

Unfortunately, as part of the application you are compelled to sign forms promising that you won't make "significant" parts of the zone file publicly available in any way (at least this was my experience when applying to Verisign for .com and .net zone file access).

Re: How to Download a List of All Registered Domain Names

#13

Post by ndr »

scandinavian wrote: post #11
vortico wrote: post #3

What if someone were to maintain an unofficial list with one domain per line, freely available as a daily torrent or served directly? Would there be a rights problem with mirroring and filtering ICANN data?

A lot of tlds don't provide zone files unless you are a registrar. They would probably not be happy if someone put those out to the pubic. For com and the likes they would probably not care as much. The best public list of domains I have found is the Project Sonar DNS (ANY) scans. I don't know how they do it, but their scans are pretty complete, at least for.dk domains, which are the ones I use. https://scans.io/stud…

It's quite slow to download from that site, I guess there are a lot of HN readers consuming their bandwidth :)

Re: How to Download a List of All Registered Domain Names

#14

Post by scandinavian »

ndr wrote: post #13
scandinavian wrote: post #11

Earlier quoted context omitted.

A lot of tlds don't provide zone files unless you are a registrar. They would probably not be happy if someone put those out to the pubic. For com and the likes they would probably not care as much. The best public list of domains I have found is the Project Sonar DNS (ANY) scans. I don't know how they do it, but their scans are pretty complete, at least for.dk domains, which are the ones I use. https://scans.io/stud…

It's quite slow to download from that site, I guess there are a lot of HN readers consuming their bandwidth :)

Their download speeds are often bad sadly, they really should provide a torrent. But it's free and they have some really interesting datasets, so it's worth the wait.

Re: How to Download a List of All Registered Domain Names

#15

Post by ech »

zamalek wrote: post #7

Now someone just needs to train an NN to recognize botnets and spam domains.

cough https://news.ycombinator.com/item?id=10352001 cough

we're in the process of getting all the zone files we can to reduce the amount of DNS requests we have to do, but the real kicker is whois databases, for example AFNIC asks for 10K€ to get access to a copy of the database...

Re: How to Download a List of All Registered Domain Names

#16

Post by jtwaleson »

A couple of months ago I processed all metadata from the Common Crawl project for all indexed domain names. This was about 10TB of metadata and resulted in 26 million domain names. EC2 costs were only about 10$ to process this. If anyone is interested, let me know.

edit: available as torrent here: https://all-certificates.s3.amazonaws.com/domainnames.gz?tor...

Re: How to Download a List of All Registered Domain Names

#17

Post by radicalbyte »

r3bl wrote: post #10
radicalbyte wrote: post #8

Earlier quoted context omitted.

Why not just upload it to Github? Then you only need to send the deltas for every update.

I was just thinking of creating a bot that will update the list(s) and push the changes back on GitHub every day. But first, I definitely have to see if it's against GitHub's terms of service.

I think that GitHub would love the publicity personally. They strike me as being very savvy regards this sort of thing ;)

Re: How to Download a List of All Registered Domain Names

#18

Post by zamalek »

ech wrote: post #15
zamalek wrote: post #7

Now someone just needs to train an NN to recognize botnets and spam domains.

cough https://news.ycombinator.com/item?id=10352001 cough we're in the process of getting all the zone files we can to reduce the amount of DNS requests we have to do, but the real kicker is whois databases, for example AFNIC asks for 10K€ to get access to a copy of the database...

Oh I missed that. You're doing good work there!

Re: How to Download a List of All Registered Domain Names

#19

Post by nly »

A warning about parsing zone files... the grammar is deceptively tricky.

While TLD registries will probably provide you with files in a sane subset[0] of that specified in RFC 1035, there are a number of things that will NOT work in general:

- Splitting the file in to lines (paren-blocks and quoted strings can span lines, strings can contain ';' etc).

- Splitting the file on whitespace (it's significant in column 1 and inside strings)

- Applying a regex (you'll need lookahead for conditional matching and it'll get ugly fast)

Don't go down the road of assuming it's a simple delimited file.

A few references:

https://www.nlnetlabs.nl/projects/nsd/documentation.html

http://www.verycomputer.com/96_5ad11cc47053d8b0_1.htm

[0] See page 9 of https://archive.icann.org/en/topics/new-gtlds/zfa-strategy-p...

Re: How to Download a List of All Registered Domain Names

#20

Post by mortenlarsen »

jtwaleson wrote: post #16

A couple of months ago I processed all metadata from the Common Crawl project for all indexed domain names. This was about 10TB of metadata and resulted in 26 million domain names. EC2 costs were only about 10$ to process this. If anyone is interested, let me know. edit: available as torrent here: https://all-certificates.s3.amazonaws.com/domainnames.gz?tor...

Please make a torrent.
Post reply on HN