Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL Certs for $10 on AWS
1–10 of 56 posts
Re: Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL Certs for $10 on AWS
#2On a side note, I recently discovered https://scans.io/ where you can find pretty much all of the data that I collected as well. Might be interesting.
Re: Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL Certs for $10 on AWS
#3Can you comment on how many additional domains you mined - compared (for example) to the 1M domains from alexa top-1M
Re: Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL Certs for $10 on AWS
#4Great write-up; really interesting that the CPU ended up being the bottleneck in this experiment! Regarding the cost of sending this data out of AWS, did you run into any issues there using rsync? IIRC rsync copies the data over TCP, so wouldn't this end up being expensive as well? Generally, though, that was my favorite part of the experiment!
Re: Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL Certs for $10 on AWS
#5Can you comment on how many additional domains you mined - compared (for example) to the 1M domains from alexa top-1M
$ cat alexa myset myset | sort | uniq -u | wc -l
773733
0.77M of Alexa top 1M were not in my list.
$ cat alexa alexa myset | sort | uniq -u | wc -l
25842205
I mined 25,842,205 additional domain names.
Re: Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL Certs for $10 on AWS
#6Sort uses only a fixed amount of memory, you can sort files larger than memory, but for such situations where you have only a few tens of millions of distinct values you can just use a python dictionary and it works even faster. While sort would shuffle data around a lot, the memory dictionary would just hold a key and a count as it gobbles the logs. It works because it is a special case of sorting where there are relatively few different values relative to the count of the whole list.
Re: Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL Certs for $10 on AWS
#7The key to the low cost seems to be that he needed to process 10TB. You get 10TB "data in" free, per month. Had it been 10TB more, or if he needed to run more than once a month, or if he needed to get that 10TB back out, the bill is around $920.
Edit: Inbound might be unlimited free. The calculator did show me an inbound total a few times, but I can't reproduce it now.
Re: Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL Certs for $10 on AWS
#8Great write-up; really interesting that the CPU ended up being the bottleneck in this experiment! Regarding the cost of sending this data out of AWS, did you run into any issues there using rsync? IIRC rsync copies the data over TCP, so wouldn't this end up being expensive as well? Generally, though, that was my favorite part of the experiment!
My use case converted 10TB in only a couple of GB after processing. Downloading that was very cheap.
Re: Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL Certs for $10 on AWS
#9Can you comment on how many additional domains you mined - compared (for example) to the 1M domains from alexa top-1M
$ cat alexa myset myset | sort | uniq -u | wc -l 773733 0.77M of Alexa top 1M were not in my list. $ cat alexa alexa myset | sort | uniq -u | wc -l 25842205 I mined 25,842,205 additional domain names.
man.. the internet really is full of crappy domains...
(and yes.. now i see that you mentioned it in the article.. took me time to get there)
Re: Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL Certs for $10 on AWS
#10Sort uses only a fixed amount of memory, you can sort files larger than memory, but for such situations where you have only a few tens of millions of distinct values you can just use a python dictionary and it works even faster. While sort would shuffle data around a lot, the memory dictionary would just hold a key and a count as it gobbles the logs. It works because it is a special case of sorting where there are re…
Don't you mean a python set? But yes, for use cases containing many duplicates where the result easily fits in memory, that is probably the fastest.