Ask HN: All of you working with Big Data, what is your Data? • Hacker News Board

Re: Ask HN: All of you working with Big Data, what is your Data?

#31

Post by sp332 » Sat, Jan 24, 2015, 2:33 PM UTC

benihana wrote: post #20

gaius wrote: post #9

I think I would consider anything 100Tb and up to be big data. There is no big data that is "easily accessible"; that's why it's "big" because it requires extremely powerful hardware and advanced techniques to work on. Otherwise its just "data". NOTE: there are people in the world who would laugh at my definition and say that big data starts at 1Pb.

> NOTE: there are people in the world who would laugh at my definition and say that big data starts at 1Pb. I commend them for having a larger penis ^H^H^H^H^H^H data stack than you. I thought big data was less about the actual size of the data store and more about where it comes from (typically passive collection from user activity) and how it's accessed (through some kind of large map-reduce style framework) and us…

I have seen several news articles using it that way. I think that's going to become an alternate definition soon.

Re: Ask HN: All of you working with Big Data, what is your Data?

#32

Post by dmichulke » Sat, Jan 24, 2015, 2:33 PM UTC

Financial data (tick to EOD), network traffic data (TCP packet level sends / receives) and farm data (sensor + farm ERP data)

All of them are basically time series with some master data, none of them is more than a few dozen GB

So in any case, I think time series data is worth a look.

Re: Ask HN: All of you working with Big Data, what is your Data?

#33

Post by gesman » Sat, Jan 24, 2015, 2:36 PM UTC

Banking and brokerage portal access data.

Utilizing Splunk as analytics and alerting platform to correlate real time financial activity events with multiple threat intelligence feeds.

Re: Ask HN: All of you working with Big Data, what is your Data?

#34

Post by noname123 » Sat, Jan 24, 2015, 2:43 PM UTC

dmichulke wrote: post #32

Financial data (tick to EOD), network traffic data (TCP packet level sends / receives) and farm data (sensor + farm ERP data) All of them are basically time series with some master data, none of them is more than a few dozen GB So in any case, I think time series data is worth a look.

I run backtesting for options trading as a hobby and storing EOD tick data, querying it and extracting it is a pain. I dl my source data currently from a retail historical data provider, then store it in MongoDB in AWS. What would you recommend tech stackwise to do backtesting on time seriea data?

Re: Ask HN: All of you working with Big Data, what is your Data?

#35

Post by nevinera » Sat, Jan 24, 2015, 2:49 PM UTC

Retail transaction/loyalty, network traffic, financial, and health data.

To be clear 'big data' is poorly defined, and I mostly do not work with terabyte+ data sets, but rather with highly dimensional data in moderate volume. Data is only 'big' relative to the algorithms you try to use on it.

Re: Ask HN: All of you working with Big Data, what is your Data?

#36

Post by alexvay » Sat, Jan 24, 2015, 2:49 PM UTC

Working on reducing Big Data to help network security engineers investigate threats faster and respond more accurately.

The data, currently, is mainly from various Network Security Monitoring appliances & SIEMs.

Re: Ask HN: All of you working with Big Data, what is your Data?

#37

Post by alexvay » Sat, Jan 24, 2015, 2:49 PM UTC

Working on reducing Big Data to help network security engineers investigate threats faster and respond more accurately.

The data, currently, is mainly from various Network Security Monitoring appliances & SIEMs.

Re: Ask HN: All of you working with Big Data, what is your Data?

#38

Post by Maro » Sat, Jan 24, 2015, 2:51 PM UTC

I work at Prezi. We have about a petabyte of data. It's usage data coming from the product and the website. Clicks in the editor and such. Then we have a data warehouse with cleaned and accurate datasets, that's much less. We are on AWS, we use S3, EMR for Hadoop, Pig, Redshift for SQL, chartio, etc. We have our own hourly ETL written in Go which we will opensource this year.

I recently talked at Strata, here's the Prezi:

https://prezi.com/d1889jmlziks/strata-2014/

Re: Ask HN: All of you working with Big Data, what is your Data?

#39

Post by daemonk » Sat, Jan 24, 2015, 2:51 PM UTC

I work with genomics data. The data is more complex than big.

Re: Ask HN: All of you working with Big Data, what is your Data?

#40

Post by remyp » Sat, Jan 24, 2015, 2:55 PM UTC

noname123 wrote: post #34

dmichulke wrote: post #32

Financial data (tick to EOD), network traffic data (TCP packet level sends / receives) and farm data (sensor + farm ERP data) All of them are basically time series with some master data, none of them is more than a few dozen GB So in any case, I think time series data is worth a look.

I run backtesting for options trading as a hobby and storing EOD tick data, querying it and extracting it is a pain. I dl my source data currently from a retail historical data provider, then store it in MongoDB in AWS. What would you recommend tech stackwise to do backtesting on time seriea data?

They don't have options data, but you might check out QuantConnect or Quantopian if you're interested in stocks and FX.