Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to Download a List of All Registered Domain Names

jordan-wright.com

Post reply on HN
Print view View original

31–40 of 55 posts

Re: How to Download a List of All Registered Domain Names

#31

Post by flexd »

jtwaleson wrote: post #29
pibefision wrote: post #24

Earlier quoted context omitted.

thank you for this!

just out of curiosity, what are you (and the other commenters) planning to use this for? I myself was trying to download all X509 server certificates on the Internet I could find to: - gain some insight into the dealings of CAs - see if private keys were reused across multiple sites - check the average expiry date of certificates etc.

My intentions is that I want a complete list of .no (Norwegian) domains, and http://norid.no refuse to give out that list to anyone.

I would like to be able to continuously check the Norwegian IP-space for compromised sites, because it would be interesting to see. Of course doing this on a bigger scale would be cool as well.

Re: How to Download a List of All Registered Domain Names

#32

Post by irfan »

I had been downloading the zone file for .PK domains on daily bases until they blocked the zone transfers. Based on comparison of these daily zone files I managed to publish the statistics [1] and also broke the news about hacked .PK domains [2] which was picked up by all leading tech blogs and news agencies.

Currently, I cannot find a way to get the zone file even by officially requesting the registry manager.

[1]: https://www.i.com.pk/pknic-domain-registration-statistics/

[2]: https://www.i.com.pk/110-pk-domains-managed-by-markmonitor-g...

Re: How to Download a List of All Registered Domain Names

#34

Post by SmokyBorbon »

vortico wrote: post #3

What if someone were to maintain an unofficial list with one domain per line, freely available as a daily torrent or served directly? Would there be a rights problem with mirroring and filtering ICANN data?

It could be used to create a competing system. ICANN would never allow this. If someone tried to put this together, I think they would quickly find their access to the data revoked.

Re: How to Download a List of All Registered Domain Names

#35

Post by allannienhuis »

pyvpx wrote: post #23
radicalbyte wrote: post #17

Earlier quoted context omitted.

I think that GitHub would love the publicity personally. They strike me as being very savvy regards this sort of thing ;)

does GitHub need any publicity? honest question as I've found that anyone who would ever use the functionality GitHub provides is already very aware of git and GitHub.

people shift between services like github and bitbucket and alternatives all the time. Perhaps not often on an individual basis, but at any one time many people are deciding where to put their stuff.

Almost anything that gets the name of a particular service bumped up to the top of someone's consciousness for a little while will shift some of those decisions toward that service.

This is why even the world's most popular brands (Apple, Coke, etc) never stop spending money on marketing / PR :)

Re: How to Download a List of All Registered Domain Names

#38

Post by vortico »

ajcr wrote: post #12
vortico wrote: post #3

What if someone were to maintain an unofficial list with one domain per line, freely available as a daily torrent or served directly? Would there be a rights problem with mirroring and filtering ICANN data?

Unfortunately, as part of the application you are compelled to sign forms promising that you won't make "significant" parts of the zone file publicly available in any way (at least this was my experience when applying to Verisign for .com and .net zone file access).

I see. Oh well.

Re: How to Download a List of All Registered Domain Names

#39

Post by gerner »

jtwaleson wrote: post #27
jtwaleson wrote: post #16

A couple of months ago I processed all metadata from the Common Crawl project for all indexed domain names. This was about 10TB of metadata and resulted in 26 million domain names. EC2 costs were only about 10$ to process this. If anyone is interested, let me know. edit: available as torrent here: https://all-certificates.s3.amazonaws.com/domainnames.gz?tor...

This was a fun Saturday afternoon project to do actually. I spawned a single c4.8xlarge (10 Gbit) instance in US-EAST-1 (next to where the Common Crawl Public Data Set lives in S3) and downloaded 10TB spread over 33k files in +/- 30 simultaneous 'curl | ungzip | grep whatever-the-common-crawl-url-prefix-was' pipelines and got a solid 5Gbit transfer speed. The bottleneck was userland CPU, so probably the gzip processe…

In my experience, and I suppose depending on the data, I've found that grep is often the bottleneck for data pipeline tasks like you describe. The silver searcher (https://github.com/ggreer/the_silver_searcher) is, in my experience, about 10x faster than grep for tasks like pulling out fields from json files. It's changed my life.

pv (pipe viewer, http://www.ivarch.com/programs/pv.shtml) and top are pretty handy to measure this kind of thing. You should be able to see exactly which process is using how much CPU, and what your throughput is.

Re: How to Download a List of All Registered Domain Names

#40

Post by talideon »

canow wrote: post #36

would it be easier to download a list of available domains?

No. The list of available domains is the list of all possible domains less the list of registered domains. The list of registered domains is vastly smaller than the list of possible domains. The list of available domains would mostly consist of junk nobody would be interested in.
Post reply on HN