Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL Certs for $10 on AWS

Post by jvehent » Sun, Jan 17, 2016, 11:29 PM UTC

Animats wrote: post #46

jvehent wrote: post #22

I have about the same amount of data in a Postgres database as part of the TLS Observatory project [1]. observatory=> select count(distinct(sha256_fingerprint)) from certificates; count --------- 1239943 observatory=> select count(distinct(target)) from scans; count --------- 6483386 The scanner evaluates both certificate and ciphersuites and stores the results in DB, so we can run complex analysis [2,3]. There is al…

I have something similar, running on a Linux desktop, to analyze SSL certs. The front end is in Go, which picks certs of interest (I wanted all certs with more than one domain) and loads them into a MariaDB database. Here's the code.[1] It's amazing how much work you can do on a modern computer when you actually use it for computing. [1] https://github.com/John-Nagle/certscan

Post by i336_ » Mon, Jan 18, 2016, 2:55 AM UTC

jvehent wrote: post #45

15155 wrote: post #40

Earlier quoted context omitted.

Perhaps something like a modified PostgREST could work? https://github.com/begriffs/postgrest

The problem isn't so much exposing the data as a rest api, as it is allowing for complex queries that may contain various table joins, subqueries or recursive conditions. I only skimmed through the documentation of postgrest, but it doesn't make mention of joining tables, which is a deal breaker for our use case.

Post by fitzwatermellow » Tue, Jan 19, 2016, 1:45 AM UTC

fitzwatermellow wrote: post #33

Sorry, but is this golang concurrent networking pattern correct: func main() { ch := make(chan string) for i := 0; i All goroutines receive on the same channel! Instead a new goroutine should be launched for each net conn. One should be able to spawn 1000s (or 1Ms) of conns and avoid ulimits using buffered chans, waitgroups, timeouts, or counters...