Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: All of you working with Big Data, what is your Data?

news.ycombinator.com

Post reply on HN
Print view View original

11–20 of 103 posts

Re: Ask HN: All of you working with Big Data, what is your Data?

#11

Post by hdra »

me_bx wrote: post #8

The twitter social graph (follow connections between people) is my data source, I extract it from the API and cache it in a database. The mariadb table storing this information currently takes a bit more than 500GB, it has about 4 billion rows (based on the statistics, I don't run SELECT count(*) on it anymore). I usually don't use the term "big data" because the buzzword is so popular that it doesn't mean anything a…

I'm looking into doing something similar as well, may I know how long have you been collecting the data, and how do you decide which data to collect?

I have been collecting data from the twitter API for a few days as well. I wanted to get an idea of average tweeting pattern, but without access to the firehose API, I got a feeling that the sample I have isn't very "neutral" as I have been mostly pulling from the popular and local tweets endpoints.

Any advice on how should I approach these kind of data collection?

Re: Ask HN: All of you working with Big Data, what is your Data?

#12

Post by bsmartt »

Working with information about attacks all the way down the killchain. Everything from IDS sigs, english descriptions, attribution, ip/host reputation.

AlienVault is hiring security researchers.

edit: we have some limited data sets that we make public, incase you're interested, hence the name 'open threat exchange'.

Re: Ask HN: All of you working with Big Data, what is your Data?

#14

Post by jleahy »

gaius wrote: post #9

I think I would consider anything 100Tb and up to be big data. There is no big data that is "easily accessible"; that's why it's "big" because it requires extremely powerful hardware and advanced techniques to work on. Otherwise its just "data". NOTE: there are people in the world who would laugh at my definition and say that big data starts at 1Pb.

Actually you can normally get away without resorting to "big data" techniques at 1PB scale, it's probably around 10PB that you're really forced to start thinking about things like hadoop style map-reduce/rack-locality.

Re: Ask HN: All of you working with Big Data, what is your Data?

#15

Post by bbrazil »

gaius wrote: post #9

I think I would consider anything 100Tb and up to be big data. There is no big data that is "easily accessible"; that's why it's "big" because it requires extremely powerful hardware and advanced techniques to work on. Otherwise its just "data". NOTE: there are people in the world who would laugh at my definition and say that big data starts at 1Pb.

In practice it's more a marketing term, and how big is big depends on what the nature of the data is and what you're doing with it.

If it fits in RAM on your laptop, it isn't big data.

If you can't process/handle it in a reasonable time on a single machine and your methods need to explicitly worry about how to scale to handle the data volumes it probably is "Big Data".

Problems that are embarrassingly parallel need far more data before I'd consider them big (I'd be in the >10PB camp), whereas for relational data I'd say >1TB.

Re: Ask HN: All of you working with Big Data, what is your Data?

#16

Post by alanctgardner3 »

If you want to work in a "big data"-type role as a developer, I wouldn't worry about finding huge data sets. There's a dearth of candidates, especially ones who actually have hands-on experience, and having deep knowledge of (and a little experience with) a broad range of tools will make you a pretty good candidate:

Fire up a VM with a single-node install on it [1] and just grab any old CSVs. Load them into HDFS, query them with Hive, query them with Impala (Drill, SparkQL, etc.). Rinse and repeat for any size of syslog data, then JSON data. Write a MapReduce job to transform the files in some way. Move on to some Spark exercises [2]. Read up on Kafka, understand how it works and think about ways to get exactly-once message delivery. Hook Kafka up to HDFS, or HBase, or a complex event processing pipeline. You'll probably need to know about serialization formats too, so study up on Avro, protobuf and Parquet (or ORCfile, as long as you understand columnar storage).

If you can talk intelligently about the whole grab bag of stuff these teams use, that'll get you in the door. Understanding RDBMSes, data warehousing concepts, and ETL is a big plus for people doing infrastructure work. If you're focused on analytics you can get away with less of the above, but knowing some of it, plus stats and BI tools (or D3 if you want to roll your own visualization) is a plus.

[1] http://www.cloudera.com/content/cloudera/en/downloads/quicks... [2] http://ampcamp.berkeley.edu/5/

Re: Ask HN: All of you working with Big Data, what is your Data?

#18

Post by Demiurge »

jleahy wrote: post #14
gaius wrote: post #9

I think I would consider anything 100Tb and up to be big data. There is no big data that is "easily accessible"; that's why it's "big" because it requires extremely powerful hardware and advanced techniques to work on. Otherwise its just "data". NOTE: there are people in the world who would laugh at my definition and say that big data starts at 1Pb.

Actually you can normally get away without resorting to "big data" techniques at 1PB scale, it's probably around 10PB that you're really forced to start thinking about things like hadoop style map-reduce/rack-locality.

Probably depends on what you're doing with the data?

Re: Ask HN: All of you working with Big Data, what is your Data?

#19

Post by mgkimsal »

OT but I've never liked the term 'big data' precisely because it's so ill-defined. Most people I speak with on this think they have 'big data'. Anything they can't comprehend is "big data". Anything that makes their Excel 97 crash is "big data". It's pervasive a term enough that people have heard it, and use it wrongly.

Colleague of mine is at a company that's advertising for someone with "big data" experience. Collectively, for more than 10 years in business, they have maybe 100g of data. They just do not know how to organize the data sanely in a relational database, and actively refuse to consider normal data structures.

Re: Ask HN: All of you working with Big Data, what is your Data?

#20

Post by benihana »

gaius wrote: post #9

I think I would consider anything 100Tb and up to be big data. There is no big data that is "easily accessible"; that's why it's "big" because it requires extremely powerful hardware and advanced techniques to work on. Otherwise its just "data". NOTE: there are people in the world who would laugh at my definition and say that big data starts at 1Pb.

>NOTE: there are people in the world who would laugh at my definition and say that big data starts at 1Pb.

I commend them for having a larger penis ^H^H^H^H^H^H data stack than you.

I thought big data was less about the actual size of the data store and more about where it comes from (typically passive collection from user activity) and how it's accessed (through some kind of large map-reduce style framework) and used (to inform product decisions or learn more about human behavior)?

Post reply on HN