it will not help for data transfer pricing but for cpu/vm time spot instances can be amazing value for these short lived projects. typically 1/8 of the price of the on demand. always take care to not set your bid higher than on demand price as wild fluctuations can happen. also if you are afraid of losing your work, there is an api you can query from within the vm that tells you 2 minutes ahead that its going to get killed. also, price is per ZONE, so there are zones in the same region that people use less.
Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL Certs for $10 on AWS
21–30 of 56 posts
Re: Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL Certs for $10 on AWS
#22I have about the same amount of data in a Postgres database as part of the TLS Observatory project [1].
observatory=> select count(distinct(sha256_fingerprint)) from certificates;
count
---------
1239943
observatory=> select count(distinct(target)) from scans;
count
---------
6483386
I don't have a good way to provide direct access to the database yet, but if you're a researcher, ping me directly and we can figure something out.
[1] https://github.com/mozilla/tls-observatory
[2] https://twitter.com/jvehent/status/684127067005390848
Re: Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL Certs for $10 on AWS
#23I have about the same amount of data in a Postgres database as part of the TLS Observatory project [1]. observatory=> select count(distinct(sha256_fingerprint)) from certificates; count --------- 1239943 observatory=> select count(distinct(target)) from scans; count --------- 6483386 The scanner evaluates both certificate and ciphersuites and stores the results in DB, so we can run complex analysis [2,3]. There is al…
> I have about the same amount of data in a Postgres database ...
I'm curious, how fast can one load data into Postgres? Is it possible to import data directly from CSV files?
Re: Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL Certs for $10 on AWS
#24Sort uses only a fixed amount of memory, you can sort files larger than memory, but for such situations where you have only a few tens of millions of distinct values you can just use a python dictionary and it works even faster. While sort would shuffle data around a lot, the memory dictionary would just hold a key and a count as it gobbles the logs. It works because it is a special case of sorting where there are re…
'sort | uniq' is another special case of this, and it is much better to replace that with 'sort -u' the 'sort' in 'sort | uniq' doesn't know you are going to be throwing away all the duplicate data. If anyone is wondering, here is an implementation of the python approach i have lying around: #!/usr/bin/env python2 import sys from collections import defaultdict c = defaultdict(int) for line in sys.stdin: c[line] += 1…
Just for fun, here's a version using `Counter` from the same `collections` module which makes that blissfully simple:
#!/usr/bin/env python2
import sys
from collections import Counter
for pair in Counter(sys.stdin).most_common():
print pairRe: Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL Certs for $10 on AWS
#25I have about the same amount of data in a Postgres database as part of the TLS Observatory project [1]. observatory=> select count(distinct(sha256_fingerprint)) from certificates; count --------- 1239943 observatory=> select count(distinct(target)) from scans; count --------- 6483386 The scanner evaluates both certificate and ciphersuites and stores the results in DB, so we can run complex analysis [2,3]. There is al…
> I have about the same amount of data in a Postgres database ... I'm curious, how fast can one load data into Postgres? Is it possible to import data directly from CSV files?
Our dataset is not loaded from an external source, it is generated by scanners.
But to answer your question: yes, postgres can load data from csv files: http://stackoverflow.com/questions/2987433/how-to-import-csv...
Re: Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL Certs for $10 on AWS
#26I have about the same amount of data in a Postgres database as part of the TLS Observatory project [1]. observatory=> select count(distinct(sha256_fingerprint)) from certificates; count --------- 1239943 observatory=> select count(distinct(target)) from scans; count --------- 6483386 The scanner evaluates both certificate and ciphersuites and stores the results in DB, so we can run complex analysis [2,3]. There is al…
> I have about the same amount of data in a Postgres database ... I'm curious, how fast can one load data into Postgres? Is it possible to import data directly from CSV files?
>Is it possible to import data directly from CSV files?
Yup! http://www.postgresql.org/docs/current/static/sql-copy.html
Re: Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL Certs for $10 on AWS
#27I have about the same amount of data in a Postgres database as part of the TLS Observatory project [1]. observatory=> select count(distinct(sha256_fingerprint)) from certificates; count --------- 1239943 observatory=> select count(distinct(target)) from scans; count --------- 6483386 The scanner evaluates both certificate and ciphersuites and stores the results in DB, so we can run complex analysis [2,3]. There is al…
That's awesome. If you are not already doing so, you can download my set from the torrent and include it in your database.
https://all-certificates.s3.amazonaws.com/certificates.tar.g...
For exporting, pg_dump -F c greatly compresses the data so cost-wise you might be able to put on S3 and publish as a torrent.
Re: Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL Certs for $10 on AWS
#28I have about the same amount of data in a Postgres database as part of the TLS Observatory project [1]. observatory=> select count(distinct(sha256_fingerprint)) from certificates; count --------- 1239943 observatory=> select count(distinct(target)) from scans; count --------- 6483386 The scanner evaluates both certificate and ciphersuites and stores the results in DB, so we can run complex analysis [2,3]. There is al…
That's awesome. If you are not already doing so, you can download my set from the torrent and include it in your database. https://all-certificates.s3.amazonaws.com/certificates.tar.g... For exporting, pg_dump -F c greatly compresses the data so cost-wise you might be able to put on S3 and publish as a torrent.
Exporting is one possibility, but eventually I'd like to provide a read-only sql access to the database we host. We have a few ideas on how to do this [1], but it's not implemented yet.
Re: Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL Certs for $10 on AWS
#29I have about the same amount of data in a Postgres database as part of the TLS Observatory project [1]. observatory=> select count(distinct(sha256_fingerprint)) from certificates; count --------- 1239943 observatory=> select count(distinct(target)) from scans; count --------- 6483386 The scanner evaluates both certificate and ciphersuites and stores the results in DB, so we can run complex analysis [2,3]. There is al…
> I have about the same amount of data in a Postgres database ... I'm curious, how fast can one load data into Postgres? Is it possible to import data directly from CSV files?
> I'm curious, how fast can one load data into Postgres?
Hard to answer considering the number of variables impacting. pg_bulkload[0] quotes 18MB/s for parallel loading on DBT-2 (221s to load 4GB), and 12MB/s for the built-in COPY (with post-indexing, that is first import all the data then enable and build the indexes)
> Is it possible to import data directly from CSV files?
Yes, the COPY command[1] can probably be configured to support whatever your *SV format is. There's also pg_bulkload (which should be faster but works offline).
[0] http://ossc-db.github.io/pg_bulkload/index.html
[1] http://www.postgresql.org/docs/current/interactive/sql-copy....
Re: Parsing 10TB of Metadata, 26M Domain Names and 1.4M SSL Certs for $10 on AWS
#30Earlier quoted context omitted.
Don't you mean a python set? But yes, for use cases containing many duplicates where the result easily fits in memory, that is probably the fastest.
Fun fact: they are nearly the same implementation. See: http://markmail.org/message/ktzomp4uwrmnzao6
As one would generally expect, the backing store of most hashsets is little more than a hashmap with zero-sized/no values.
In fact, that's exactly how Rust's standard library hashset is implemented since rust supports zero-sized types "in userland" (and unit `()` is a ZST):
pub struct HashSet {
map: HashMap
}