Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL Certs for $10 on AWS • Hacker News Board

Re: Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL Certs for $10 on AWS

#41

Post by thebelal » Sun, Jan 17, 2016, 8:10 PM UTC

visarga wrote: post #6

Sort uses only a fixed amount of memory, you can sort files larger than memory, but for such situations where you have only a few tens of millions of distinct values you can just use a python dictionary and it works even faster. While sort would shuffle data around a lot, the memory dictionary would just hold a key and a count as it gobbles the logs. It works because it is a special case of sorting where there are re…

This is even easier to do with awk.

    awk -e '!a[$0]++'

This also preserves the original input order, which is a nice property.

Re: Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL Certs for $10 on AWS

#42

Post by howaboutit » Sun, Jan 17, 2016, 8:56 PM UTC

jtwaleson wrote: post #5

yazriel wrote: post #3

Can you comment on how many additional domains you mined - compared (for example) to the 1M domains from alexa top-1M

$ cat alexa myset myset | sort | uniq -u | wc -l 773733 0.77M of Alexa top 1M were not in my list. $ cat alexa alexa myset | sort | uniq -u | wc -l 25842205 I mined 25,842,205 additional domain names.

Did you consider using the gTLD zone files (from the respective registries) and the ccTLD zone files found @ http://viewdns.info/data/? A much bigger initial dataset than 25M domains right there?

Re: Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL Certs for $10 on AWS

#43

Post by jtwaleson » Sun, Jan 17, 2016, 9:06 PM UTC

howaboutit wrote: post #42

jtwaleson wrote: post #5

Earlier quoted context omitted.

$ cat alexa myset myset | sort | uniq -u | wc -l 773733 0.77M of Alexa top 1M were not in my list. $ cat alexa alexa myset | sort | uniq -u | wc -l 25842205 I mined 25,842,205 additional domain names.

Did you consider using the gTLD zone files (from the respective registries) and the ccTLD zone files found @ http://viewdns.info/data/ ? A much bigger initial dataset than 25M domains right there?

No, getting access will probably take a couple of days (or in case of viewdns, more than 100$) and thereby all the fun out of the project. If you know of any other way to get the list I'd be happy to hear it though!

Re: Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL Certs for $10 on AWS

#44

Post by adamseabrook » Sun, Jan 17, 2016, 9:39 PM UTC

jtwaleson wrote: post #43

howaboutit wrote: post #42

Earlier quoted context omitted.

Did you consider using the gTLD zone files (from the respective registries) and the ccTLD zone files found @ http://viewdns.info/data/ ? A much bigger initial dataset than 25M domains right there?

No, getting access will probably take a couple of days (or in case of viewdns, more than 100$) and thereby all the fun out of the project. If you know of any other way to get the list I'd be happy to hear it though!

http://meanpath.com/freedirectory.html

Feel free to grab a copy of our domain list. The "All domains with NS records" is the one you want. Has 191 million in it.

Re: Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL Certs for $10 on AWS

#45

Post by jvehent » Sun, Jan 17, 2016, 9:58 PM UTC

15155 wrote: post #40

jvehent wrote: post #28

Earlier quoted context omitted.

Exporting is one possibility, but eventually I'd like to provide a read-only sql access to the database we host. We have a few ideas on how to do this [1], but it's not implemented yet. [1] https://github.com/mozilla/tls-observatory/issues/92

Perhaps something like a modified PostgREST could work? https://github.com/begriffs/postgrest

The problem isn't so much exposing the data as a rest api, as it is allowing for complex queries that may contain various table joins, subqueries or recursive conditions. I only skimmed through the documentation of postgrest, but it doesn't make mention of joining tables, which is a deal breaker for our use case.

Re: Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL Certs for $10 on AWS

#46

Post by Animats » Sun, Jan 17, 2016, 10:10 PM UTC

jvehent wrote: post #22

I have about the same amount of data in a Postgres database as part of the TLS Observatory project [1]. observatory=> select count(distinct(sha256_fingerprint)) from certificates; count --------- 1239943 observatory=> select count(distinct(target)) from scans; count --------- 6483386 The scanner evaluates both certificate and ciphersuites and stores the results in DB, so we can run complex analysis [2,3]. There is al…

I have something similar, running on a Linux desktop, to analyze SSL certs. The front end is in Go, which picks certs of interest (I wanted all certs with more than one domain) and loads them into a MariaDB database. Here's the code.[1] It's amazing how much work you can do on a modern computer when you actually use it for computing.

[1] https://github.com/John-Nagle/certscan

Re: Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL Certs for $10 on AWS

#47

Post by hdmoore » Sun, Jan 17, 2016, 10:31 PM UTC

jtwaleson wrote: post #2

On a side note, I recently discovered https://scans.io/ where you can find pretty much all of the data that I collected as well. Might be interesting.

You might find the processing tips on the Project Sonar wiki useful:

https://github.com/rapid7/sonar/wiki/Analyzing-Datasets

Project Sonar is one of the primary contributors to scans.io. The DAP utility is handy for parsing raw x509 certificates and generating JSON output.

Re: Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL Certs for $10 on AWS

#48

Post by hdmoore » Sun, Jan 17, 2016, 10:35 PM UTC

yazriel wrote: post #3

Can you comment on how many additional domains you mined - compared (for example) to the 1M domains from alexa top-1M

The Sonar FDNS set contains about 1.4 billion host names (50m+ domains). The FDNS set is seeded from TLD zones, CZDAP, PTR lookups (RDNS), SSL/TLS scans, and HTTP link extraction. It updates every two weeks: https://github.com/rapid7/sonar/wiki/Forward-DNS

Re: Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL Certs for $10 on AWS

#49

Post by hdmoore » Sun, Jan 17, 2016, 10:39 PM UTC

jtwaleson wrote: post #43

howaboutit wrote: post #42

Earlier quoted context omitted.

Did you consider using the gTLD zone files (from the respective registries) and the ccTLD zone files found @ http://viewdns.info/data/ ? A much bigger initial dataset than 25M domains right there?

No, getting access will probably take a couple of days (or in case of viewdns, more than 100$) and thereby all the fun out of the project. If you know of any other way to get the list I'd be happy to hear it though!

A shortcut to getting com, net, info, org, us, sk, and biz is to give premiumdrops.com $24.95/mo. You can get these for free from the TLD operators, but it takes a few weeks of snail mail (last I checked). The gTLD access via CZDAP is free, but takes a few days for approvals to process.

Re: Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL Certs for $10 on AWS

#50

Post by adamseabrook » Sun, Jan 17, 2016, 10:53 PM UTC

hdmoore wrote: post #49

jtwaleson wrote: post #43

Earlier quoted context omitted.

No, getting access will probably take a couple of days (or in case of viewdns, more than 100$) and thereby all the fun out of the project. If you know of any other way to get the list I'd be happy to hear it though!

A shortcut to getting com, net, info, org, us, sk, and biz is to give premiumdrops.com $24.95/mo. You can get these for free from the TLD operators, but it takes a few weeks of snail mail (last I checked). The gTLD access via CZDAP is free, but takes a few days for approvals to process.

https://czds.icann.org/en It has been largely automated now so you can request access to the files with one click vs having to sign and email hundreds of forms. Approval seems to be automatic for most of them.