Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: All of you working with Big Data, what is your Data?

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 103 posts

Re: Ask HN: All of you working with Big Data, what is your Data?

#21

Post by learnstats2 »

gaius wrote: post #9

I think I would consider anything 100Tb and up to be big data. There is no big data that is "easily accessible"; that's why it's "big" because it requires extremely powerful hardware and advanced techniques to work on. Otherwise its just "data". NOTE: there are people in the world who would laugh at my definition and say that big data starts at 1Pb.

Those numbers make me think of "store all the things" rather than useful statistical data.

1PB is arguably enough data to store genetic variation across all human beings.

Re: Ask HN: All of you working with Big Data, what is your Data?

#22

Post by jayshahtx »

I think the most interesting datasets are within reach but require curation yourself. For example there are extremely powerful scraping libraries in just about every popular language today, not to mention APIs such as Twitter's.

If you're looking for a cool dataset to play with, I think it is more productive to ask yourself what questions you want to answer and then find/curate the data VS find a dataset and then ask "what questions can I answer?". The former approach will also keep motivations high if you're driven by curiosity.

Re: Ask HN: All of you working with Big Data, what is your Data?

#23

Post by mattbrewsbytes »

Data collected from devices and it is large, but not big. Around 40-60TB and very repetitive data. Find some open set of data that interests you and just do something to get familiar with the tools.

I think most data sets could be handled via RDBMS and Big Data is just another choice. The more interesting thing to me is what you accomplish and if a new tech can get you there faster or cheaper, etc.

Re: Ask HN: All of you working with Big Data, what is your Data?

#25

Post by jongos »

Job: Data Science Consultant, Governments and NGOs

For me it's primarily population data. It's not exactly 'big' data in the raw form, but what makes it bigger are the variations analyzing, applications of predictive models, and new metadata values extracted from it.

The data grows faster than we're collecting it exponentially because of all the analysis.

Re: Ask HN: All of you working with Big Data, what is your Data?

#26

Post by serhanbaker »

You can actually think an interesting application and generate your own data. For example, we were developing a product for processing network events in real-time. There were 6-10K events per second, and we were creating alerts for several different scenarios. For testing purposes, we actually wrote a program to simulate those events, with 20K events per second. It was generating fake (but realistic) data with the right format.

Application idea from top of my head: Generate turnstile data for different subway stations (enter/exit, time) and wrote an application to show the density of those stations with times. You can create a scenario where a certain station is more dense than others, and this could be your test. And this application could be your proof of concept

Re: Ask HN: All of you working with Big Data, what is your Data?

#27

Post by gaius »

bbrazil wrote: post #15
gaius wrote: post #9

I think I would consider anything 100Tb and up to be big data. There is no big data that is "easily accessible"; that's why it's "big" because it requires extremely powerful hardware and advanced techniques to work on. Otherwise its just "data". NOTE: there are people in the world who would laugh at my definition and say that big data starts at 1Pb.

In practice it's more a marketing term, and how big is big depends on what the nature of the data is and what you're doing with it. If it fits in RAM on your laptop, it isn't big data. If you can't process/handle it in a reasonable time on a single machine and your methods need to explicitly worry about how to scale to handle the data volumes it probably is "Big Data". Problems that are embarrassingly parallel need f…

I've worked on 50Tb relational databases, I don't consider myself to be a "big data" guy.

Re: Ask HN: All of you working with Big Data, what is your Data?

#28

Post by gaius »

benihana wrote: post #20
gaius wrote: post #9

I think I would consider anything 100Tb and up to be big data. There is no big data that is "easily accessible"; that's why it's "big" because it requires extremely powerful hardware and advanced techniques to work on. Otherwise its just "data". NOTE: there are people in the world who would laugh at my definition and say that big data starts at 1Pb.

> NOTE: there are people in the world who would laugh at my definition and say that big data starts at 1Pb. I commend them for having a larger penis ^H^H^H^H^H^H data stack than you. I thought big data was less about the actual size of the data store and more about where it comes from (typically passive collection from user activity) and how it's accessed (through some kind of large map-reduce style framework) and us…

So some other definition of "big" than umm "big" then?

Re: Ask HN: All of you working with Big Data, what is your Data?

#30

Post by gaius »

learnstats2 wrote: post #21
gaius wrote: post #9

I think I would consider anything 100Tb and up to be big data. There is no big data that is "easily accessible"; that's why it's "big" because it requires extremely powerful hardware and advanced techniques to work on. Otherwise its just "data". NOTE: there are people in the world who would laugh at my definition and say that big data starts at 1Pb.

Those numbers make me think of "store all the things" rather than useful statistical data. 1PB is arguably enough data to store genetic variation across all human beings .

I'm referring to guys I know who have worked or presently work at CERN.
Post reply on HN