Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: All of you working with Big Data, what is your Data?

news.ycombinator.com

Post reply on HN
Print view View original

61–70 of 103 posts

Re: Ask HN: All of you working with Big Data, what is your Data?

#61

Post by sixdimensional »

mgkimsal wrote: post #19

OT but I've never liked the term 'big data' precisely because it's so ill-defined. Most people I speak with on this think they have 'big data'. Anything they can't comprehend is "big data". Anything that makes their Excel 97 crash is "big data". It's pervasive a term enough that people have heard it, and use it wrongly. Colleague of mine is at a company that's advertising for someone with "big data" experience. Colle…

There is a definition for "very large database" (VLDB) on Wikipedia- the precursor to the term "big data".. although it is somewhat outdated.

Re: Ask HN: All of you working with Big Data, what is your Data?

#62

Post by icebraining »

billyhoffman wrote: post #55
me_bx wrote: post #8

The twitter social graph (follow connections between people) is my data source, I extract it from the API and cache it in a database. The mariadb table storing this information currently takes a bit more than 500GB, it has about 4 billion rows (based on the statistics, I don't run SELECT count(*) on it anymore). I usually don't use the term "big data" because the buzzword is so popular that it doesn't mean anything a…

Won't SELECT count(*) be super slow? Isn't SELECT Count(some_primary_key) a better idea?

It should be the same. count(*) only needs to return the number of columns (regardless of its value), so it can use only indexes, while count(column) must only count non-null values. But since the primary is non-null, it should end up taking the same time.

Re: Ask HN: All of you working with Big Data, what is your Data?

#63

Post by sysk »

malux85 wrote: post #47

Here's some stuff I have done in the past year, I work for a small company, but run a personal computing cluster of 167 servers that I pay for out of my own pocket. I really enjoy loading "big" datasets into them and working on improving algorithms or gaining insight into the data. I (try and) network around London and offer my services for free to people who have interesting problems. - Very high resolution FMRI dat…

Do you actually have 167 servers running in your home or do you rent them on e.g. EC2 when you need them? If it's the former case, is it because it makes financial sense (I'd be surprised) or is it for the experience/fun?

Re: Ask HN: All of you working with Big Data, what is your Data?

#64

Post by me_bx »

hdra wrote: post #11
me_bx wrote: post #8

The twitter social graph (follow connections between people) is my data source, I extract it from the API and cache it in a database. The mariadb table storing this information currently takes a bit more than 500GB, it has about 4 billion rows (based on the statistics, I don't run SELECT count(*) on it anymore). I usually don't use the term "big data" because the buzzword is so popular that it doesn't mean anything a…

I'm looking into doing something similar as well, may I know how long have you been collecting the data, and how do you decide which data to collect? I have been collecting data from the twitter API for a few days as well. I wanted to get an idea of average tweeting pattern, but without access to the firehose API, I got a feeling that the sample I have isn't very "neutral" as I have been mostly pulling from the popul…

It all depends on your use case.

If user streams are not releavant to you, you may use the `filter` endpoint instead of `sample`, and focus on keywords describing a relevant niche for your analysis.

In case you want to limit yourself to tweets geo-located in a certain location, you have to be aware that the bounding box filter of twitter is buggy: it will return you tweets geo-located outside of the desired area, and you are not sure to be getting all tweets.

Re: Ask HN: All of you working with Big Data, what is your Data?

#65

Post by yankoff »

malux85 wrote: post #47

Here's some stuff I have done in the past year, I work for a small company, but run a personal computing cluster of 167 servers that I pay for out of my own pocket. I really enjoy loading "big" datasets into them and working on improving algorithms or gaining insight into the data. I (try and) network around London and offer my services for free to people who have interesting problems. - Very high resolution FMRI dat…

What do you pay for such cluster if I may ask? I assume it's not in EC2? Have you thought of adding a web-crawling service on the top of that?

Re: Ask HN: All of you working with Big Data, what is your Data?

#66

Post by angrybits »

IndianAstronaut wrote: post #59
bbrazil wrote: post #49

Earlier quoted context omitted.

I've worked on relational databases of similar size. There's two challenges. The first is maintaining the relational model at that scale is quite tricky, tradeoffs need to be made. The second is the systems-level management of that large a deployment requires a bit more than standard configuration management. These days Amazon have Multi-AZ RDS, which should handle the 2nd item.

The problem with databases that are 50tb or more is that you soon run into limits with the relational model. I have been reading up on different modeling techniques for converting relational models into cassandra's column family stores.

No, you run into limits with some relational engines. I work in a Teradata shop and we handle relational models of this size just fine.
Post reply on HN