Ask HN: All of you working with Big Data, what is your Data?

Post by jaredbroad » Sat, Jan 24, 2015, 3:32 PM UTC

noname123 wrote: post #34

dmichulke wrote: post #32

Financial data (tick to EOD), network traffic data (TCP packet level sends / receives) and farm data (sensor + farm ERP data) All of them are basically time series with some master data, none of them is more than a few dozen GB So in any case, I think time series data is worth a look.

I run backtesting for options trading as a hobby and storing EOD tick data, querying it and extracting it is a pain. I dl my source data currently from a retail historical data provider, then store it in MongoDB in AWS. What would you recommend tech stackwise to do backtesting on time seriea data?

Post by vijayr » Sat, Jan 24, 2015, 3:33 PM UTC

alanctgardner3 wrote: post #16

If you want to work in a "big data"-type role as a developer, I wouldn't worry about finding huge data sets. There's a dearth of candidates, especially ones who actually have hands-on experience, and having deep knowledge of (and a little experience with) a broad range of tools will make you a pretty good candidate: Fire up a VM with a single-node install on it [1] and just grab any old CSVs. Load them into HDFS, que…

Post by billyhoffman » Sat, Jan 24, 2015, 3:37 PM UTC

me_bx wrote: post #8

The twitter social graph (follow connections between people) is my data source, I extract it from the API and cache it in a database. The mariadb table storing this information currently takes a bit more than 500GB, it has about 4 billion rows (based on the statistics, I don't run SELECT count(*) on it anymore). I usually don't use the term "big data" because the buzzword is so popular that it doesn't mean anything a…

Post by crzrcn » Sat, Jan 24, 2015, 3:56 PM UTC

Post by IndianAstronaut » Sat, Jan 24, 2015, 4:17 PM UTC

bbrazil wrote: post #49

gaius wrote: post #27

Earlier quoted context omitted.

I've worked on 50Tb relational databases, I don't consider myself to be a "big data" guy.

I've worked on relational databases of similar size. There's two challenges. The first is maintaining the relational model at that scale is quite tricky, tradeoffs need to be made. The second is the systems-level management of that large a deployment requires a bit more than standard configuration management. These days Amazon have Multi-AZ RDS, which should handle the 2nd item.