Ask HN: All of you working with Big Data, what is your Data?

#1

Post by sysk » Sat, Jan 24, 2015, 10:50 AM UTC

Big data is a trending topic these days and I'd like to get my hands dirty both out of curiosity and to make myself more relevant on the marketplace. That being said, I'm not sure which data sets are both interesting to play with and easily accessible. My question is:

For those of you already working with big data, what kind of data do you work with?

Re: Ask HN: All of you working with Big Data, what is your Data?

#2

Post by [deleted] » Sat, Jan 24, 2015, 12:23 PM UTC

[deleted]

Re: Ask HN: All of you working with Big Data, what is your Data?

#3

Post by valevk » Sat, Jan 24, 2015, 12:27 PM UTC

Mostly logfiles, and other machine generated data (of which 99% can be thrown away, but that's what "big data" does for me, filter out what's important).

Re: Ask HN: All of you working with Big Data, what is your Data?

#4

Post by robinho364 » Sat, Jan 24, 2015, 12:39 PM UTC

In search engine companies, we sort out cookies every day.

Re: Ask HN: All of you working with Big Data, what is your Data?

#5

Post by jawns » Sat, Jan 24, 2015, 12:40 PM UTC

Day job: Web site user sessions and offline retail sales data.

Side project: Poll responses on http://www.correlated.org

Re: Ask HN: All of you working with Big Data, what is your Data?

#6

Post by sjwhitworth » Sat, Jan 24, 2015, 12:43 PM UTC

GPS and transport data.

Re: Ask HN: All of you working with Big Data, what is your Data?

#7

Post by quentindemetz » Sat, Jan 24, 2015, 12:56 PM UTC

Hotel reservations, prices, and numerous market indicators. for thousands of hotels.

PriceMatch is hiring in Paris!

Re: Ask HN: All of you working with Big Data, what is your Data?

#8

Post by me_bx » Sat, Jan 24, 2015, 12:59 PM UTC

The twitter social graph (follow connections between people) is my data source, I extract it from the API and cache it in a database.

The mariadb table storing this information currently takes a bit more than 500GB, it has about 4 billion rows (based on the statistics, I don't run SELECT count(*) on it anymore).

I usually don't use the term "big data" because the buzzword is so popular that it doesn't mean anything anymore.

Re: Ask HN: All of you working with Big Data, what is your Data?

#9

Post by gaius » Sat, Jan 24, 2015, 1:02 PM UTC

I think I would consider anything 100Tb and up to be big data. There is no big data that is "easily accessible"; that's why it's "big" because it requires extremely powerful hardware and advanced techniques to work on. Otherwise its just "data".

NOTE: there are people in the world who would laugh at my definition and say that big data starts at 1Pb.

Re: Ask HN: All of you working with Big Data, what is your Data?

#10

Post by lolwhat » Sat, Jan 24, 2015, 1:03 PM UTC

IMDB and Boxoffice mojo data. Thinking of moving to mongodb.