21–30 of 55 posts
A couple of months ago I processed all metadata from the Common Crawl project for all indexed domain names. This was about 10TB of metadata and resulted in 26 million domain names. EC2 costs were only about 10$ to process this. If anyone is interested, let me know. edit: available as torrent here: https://all-certificates.s3.amazonaws.com/domainnames.gz?tor...
A couple of months ago I processed all metadata from the Common Crawl project for all indexed domain names. This was about 10TB of metadata and resulted in 26 million domain names. EC2 costs were only about 10$ to process this. If anyone is interested, let me know. edit: available as torrent here: https://all-certificates.s3.amazonaws.com/domainnames.gz?tor...
Please make a torrent.
Earlier quoted context omitted.
I was just thinking of creating a bot that will update the list(s) and push the changes back on GitHub every day. But first, I definitely have to see if it's against GitHub's terms of service.
I think that GitHub would love the publicity personally. They strike me as being very savvy regards this sort of thing ;)
What if someone were to maintain an unofficial list with one domain per line, freely available as a daily torrent or served directly? Would there be a rights problem with mirroring and filtering ICANN data?
A lot of tlds don't provide zone files unless you are a registrar. They would probably not be happy if someone put those out to the pubic. For com and the likes they would probably not care as much. The best public list of domains I have found is the Project Sonar DNS (ANY) scans. I don't know how they do it, but their scans are pretty complete, at least for.dk domains, which are the ones I use. https://scans.io/stud…
A couple of months ago I processed all metadata from the Common Crawl project for all indexed domain names. This was about 10TB of metadata and resulted in 26 million domain names. EC2 costs were only about 10$ to process this. If anyone is interested, let me know. edit: available as torrent here: https://all-certificates.s3.amazonaws.com/domainnames.gz?tor...
The bottleneck was userland CPU, so probably the gzip processes. Took about 5 hours. Cutting of the path and piping through sort and uniq took another hour or so.
The only costs were the price/hour of running the EC2 instance. Network costs were 0 as you're only transferring data to your instance.
Earlier quoted context omitted.
https://all-certificates.s3.amazonaws.com/domainnames.gz?tor...
thank you for this!
I myself was trying to download all X509 server certificates on the Internet I could find to:
- gain some insight into the dealings of CAs
- see if private keys were reused across multiple sites
- check the average expiry date of certificates etc.
A couple of months ago I processed all metadata from the Common Crawl project for all indexed domain names. This was about 10TB of metadata and resulted in 26 million domain names. EC2 costs were only about 10$ to process this. If anyone is interested, let me know. edit: available as torrent here: https://all-certificates.s3.amazonaws.com/domainnames.gz?tor...
This was a fun Saturday afternoon project to do actually. I spawned a single c4.8xlarge (10 Gbit) instance in US-EAST-1 (next to where the Common Crawl Public Data Set lives in S3) and downloaded 10TB spread over 33k files in +/- 30 simultaneous 'curl | ungzip | grep whatever-the-common-crawl-url-prefix-was' pipelines and got a solid 5Gbit transfer speed. The bottleneck was userland CPU, so probably the gzip processe…