Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: All of you working with Big Data, what is your Data?

news.ycombinator.com

Post reply on HN
Print view View original

41–50 of 103 posts

Re: Ask HN: All of you working with Big Data, what is your Data?

#41

Post by calinet6 »

gaius wrote: post #28
benihana wrote: post #20

Earlier quoted context omitted.

> NOTE: there are people in the world who would laugh at my definition and say that big data starts at 1Pb. I commend them for having a larger penis ^H^H^H^H^H^H data stack than you. I thought big data was less about the actual size of the data store and more about where it comes from (typically passive collection from user activity) and how it's accessed (through some kind of large map-reduce style framework) and us…

So some other definition of "big" than umm "big" then?

Yes, absolutely. When people talk about big data, more often than not it's a measure of complexity and difficulty, not size.

Re: Ask HN: All of you working with Big Data, what is your Data?

#42

Post by leoplct »

noname123 wrote: post #34
dmichulke wrote: post #32

Financial data (tick to EOD), network traffic data (TCP packet level sends / receives) and farm data (sensor + farm ERP data) All of them are basically time series with some master data, none of them is more than a few dozen GB So in any case, I think time series data is worth a look.

I run backtesting for options trading as a hobby and storing EOD tick data, querying it and extracting it is a pain. I dl my source data currently from a retail historical data provider, then store it in MongoDB in AWS. What would you recommend tech stackwise to do backtesting on time seriea data?

Have you tried quantopian.com ?

Re: Ask HN: All of you working with Big Data, what is your Data?

#45

Post by [deleted] »

gaius wrote: post #30
learnstats2 wrote: post #21

Earlier quoted context omitted.

Those numbers make me think of "store all the things" rather than useful statistical data. 1PB is arguably enough data to store genetic variation across all human beings .

I'm referring to guys I know who have worked or presently work at CERN.

[deleted]

Re: Ask HN: All of you working with Big Data, what is your Data?

#46

Post by saalweachter »

calinet6 wrote: post #41
gaius wrote: post #28

Earlier quoted context omitted.

So some other definition of "big" than umm "big" then?

Yes, absolutely. When people talk about big data, more often than not it's a measure of complexity and difficulty, not size.

Yeah, for everyone but physicists it's really "big enough" data: it's a big enough data set that you've started recording things you didn't even try to record.

An excellent example was on HN the other day, using the NYC taxi data to determine which drivers are observant Muslims. It's not something anyone set out to record, but the data set has gotten so large that if you turn it sideways and shake, random facts like that fall out.

Re: Ask HN: All of you working with Big Data, what is your Data?

#47

Post by malux85 »

Here's some stuff I have done in the past year, I work for a small company, but run a personal computing cluster of 167 servers that I pay for out of my own pocket. I really enjoy loading "big" datasets into them and working on improving algorithms or gaining insight into the data.

I (try and) network around London and offer my services for free to people who have interesting problems.

- Very high resolution FMRI data. A single scan can be 10-20GB

- Infringing URLs for a piracy company, 4 billion rows

- DNA sequences and Protein Data, lots of variation in sizes, from a few hundred MB's of string data, to hundreds of GBs

- RAW radio data for a military skunkworks project (10's of GB / min)

I would really like to find an investor who could take me off my full time job, I have 3 quite large projects I would like to build, one I have almost finished.

Re: Ask HN: All of you working with Big Data, what is your Data?

#49

Post by bbrazil »

gaius wrote: post #27
bbrazil wrote: post #15

Earlier quoted context omitted.

In practice it's more a marketing term, and how big is big depends on what the nature of the data is and what you're doing with it. If it fits in RAM on your laptop, it isn't big data. If you can't process/handle it in a reasonable time on a single machine and your methods need to explicitly worry about how to scale to handle the data volumes it probably is "Big Data". Problems that are embarrassingly parallel need f…

I've worked on 50Tb relational databases, I don't consider myself to be a "big data" guy.

I've worked on relational databases of similar size. There's two challenges. The first is maintaining the relational model at that scale is quite tricky, tradeoffs need to be made. The second is the systems-level management of that large a deployment requires a bit more than standard configuration management.

These days Amazon have Multi-AZ RDS, which should handle the 2nd item.

Re: Ask HN: All of you working with Big Data, what is your Data?

#50

Post by wglb »

dmichulke wrote: post #32

Financial data (tick to EOD), network traffic data (TCP packet level sends / receives) and farm data (sensor + farm ERP data) All of them are basically time series with some master data, none of them is more than a few dozen GB So in any case, I think time series data is worth a look.

You can get vast amounts of stock option data from http://www.orats.com/.

I don't think of the options stream as big data. It is more of a "fast" data problem, where at the open you need to handle 100,000 ticks per second, doing a implied volatility calculation on each tick. It is pretty demanding, when, for example, you consider that when the price of IBM moves by one penny, you get 400 options quote right then. You end up with a billion quotes in a typical day, if memory serves correctly.

I didn't think of this as Big Data, somehow. It seems to me that big data is more complex than that. You most often just process the data, say for backtesting, in a strict series, from the start to the end, or for an interval. I think of Big Data having more structure.

Post reply on HN