Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don't use Hadoop when your data isn't that big

chrisstucchio.com

Post reply on HN
Print view View original

161–170 of 235 posts

Re: Don't use Hadoop when your data isn't that big

#161

Post by sgt101 »

hackula1 wrote: post #59

99% of the cases I have seen where people have been working with tables that are in the 5+ TB range for analysis, there is some obvious way to compress the data that they have overlooked. Most analysts find some way to aggregate a dataset once, then do actual work on that aggregated dataset, rather than the raw data. In geospatial analytics, for example, a trillion records can be aggregated down to census blocks/bloc…

I've found compression to be one of the big benefits of Hive / Impala; we are able to load and manipulate compressed data very quickly compared with our friends using a RDBMS who have to uncompress and then import the data. This shortens the cycle time on some analytics challenges by days (typically we get data in the 3-8TB range) where we can answer simple questions in a few hours (and senior management loose focus immediately afterwards).

Re: Don't use Hadoop when your data isn't that big

#162

Post by cdavid »

t1m wrote: post #77
mrcactu5 wrote: post #38

I remember starting to grasp what "big data" meant when I had a phone interview with Twitter. @ Imagine you have some numbers spread over some computers -- too many to fit in one computer find the median. ▪ Uhh, sort them? @ Can you find the median on a single computer without sorting them. ▪ :-( @ We'll call you back tomorrow. I was promptly rejected, but it set the tone for my later studies. The criterion for Big D…

Twitter often gets singled out for it's big dataness, though the latest numbers I've seen are only about 400 million tweets per day. Even allowing 1K/tweet this is a rather manageable 400GB uncompressed. 15 minutes of US tweets would fit on your phone :-)

I am pretty sure a tweet is more than 1kb: the message itself is 140 characters, not 140 bytes, and you have lots of metadata around a tweet. I would expect easily one order of magnitude more / weet.

Re: Don't use Hadoop when your data isn't that big

#163

Post by toomuchtodo »

dekhn wrote: post #154
toomuchtodo wrote: post #144

Earlier quoted context omitted.

Fermilab in Batavia takes data from CERN over a 40Gb/s optical link (CMS Tier1, CERN CMS is Tier0); when I was there several years ago, we were staging the data to ~500TB of spinning disk in Nexans and cold storage were 2-3 Storagetek tape libraries the size of school busses. Computing and particle physics is where awesome meet.

CERN and its associated teams (CMS and ATLAS) are awesome. They are the only outfit outside of Google that seems to be handling data management and computing properly.

Not to disappoint you, but I could tell you horror stories about my time working on the CMS team at Fermilab. Horrible, horrible stories.

I hope the CMS and ATLAS teams onsite at CERN were much better.

Re: Don't use Hadoop when your data isn't that big

#165

Post by sgt101 »

mindcrime wrote: post #101

On a related note, I find it amusing that people seem to think that "Big Data == Hadoop". In actuality, there are plenty of other approaches to scaling out clusters to handle large jobs, including MPI and OpenMP, as well as BSP (Bulk Synchronous Parallel)[3] frameworks. [1]: http://en.wikipedia.org/wiki/Message_Passing_Interface [2]: http://en.wikipedia.org/wiki/OpenMP [3]: http://en.wikipedia.org/wiki/Bulk_synchrono…

Isn't YARN / Giraffe BSP ?

Re: Don't use Hadoop when your data isn't that big

#166

Post by pwang »

vtuulos wrote: post #104
hackula1 wrote: post #59

99% of the cases I have seen where people have been working with tables that are in the 5+ TB range for analysis, there is some obvious way to compress the data that they have overlooked. Most analysts find some way to aggregate a dataset once, then do actual work on that aggregated dataset, rather than the raw data. In geospatial analytics, for example, a trillion records can be aggregated down to census blocks/bloc…

Exactly - encoding is the key. I just gave a presentation last week at the SF Python Meetup about how AdRoll uses a single server to query terabytes of compressed data with sub-minute latencies, using a system that is implemented in Python: http://tuulos.github.io/sf-python-meetup-sep-2013 This is probably a thousand times less resource intensive than using Hadoop for the same queries.

Compression and out-of-core are not nearly discussed enough in the "big data" circles. The key challenge is efficiently using bandwidth-limited channels, whether it's a network link between nodes, the SATA bus between disk and RAM, or the memory channel between RAM and L3.

It's also why we are both building compression into the native storage format for Blaze, and why Blaze is designed to run out-of-core from the start.

https://github.com/ContinuumIO/blaze/blob/master/docs/source...

Re: Don't use Hadoop when your data isn't that big

#167

Post by cshimmin »

dekhn wrote: post #154
toomuchtodo wrote: post #144

Earlier quoted context omitted.

Fermilab in Batavia takes data from CERN over a 40Gb/s optical link (CMS Tier1, CERN CMS is Tier0); when I was there several years ago, we were staging the data to ~500TB of spinning disk in Nexans and cold storage were 2-3 Storagetek tape libraries the size of school busses. Computing and particle physics is where awesome meet.

CERN and its associated teams (CMS and ATLAS) are awesome. They are the only outfit outside of Google that seems to be handling data management and computing properly.

ATLAS guy here. I agree with @toomuchtodo, we're stuck in the stone age. It's basically just batch processing and lots of copying data around for no reason.

Re: Don't use Hadoop when your data isn't that big

#168

Post by pwang »

deathflute wrote: post #111

Well written article. I think most people who do not have a background in data are unaware of the various options out there and fall for the marketing behind hadoop like tools. I would urge people doing analytics to take a look at kdb+ from kx. Unless you have ridiculously large amounts of data(> 200 TB), I can bet that you would be better off with kdb. The only downside is that it costs a lot of money which is a pit…

You'd have to hire a team of people that can write good K. Since they're so highly in demand in the niche area of finance, you'll be paying a very pretty penny for them.

Re: Don't use Hadoop when your data isn't that big

#169

Post by [deleted] »

dekhn wrote: post #153
w_t_payne wrote: post #147

Earlier quoted context omitted.

1.5 PB is pretty huge for most people outside of supercomputing. The biggest dataset I ever worked with fell some way short of that (It was approaching 1PB, I think, although we were not too sure how much data we had exactly, and it might have been slightly over, depending on how you measured). Even handling small(er) working sets could be a challenge in organisation, since we only had a small compute cluster (& asso…

"Supercomputing" has never really been about big data. I used to work in supercomputing (NERSC) and have used most of the major machines in the field in the past. The supercomputer centers claim they're moving lots of data around, but it you look closely, it's almost entirely message passing during a calculation. And in most cases, the calculations they are doing are just wasted resources- simulating a protein over o…

[deleted]
Post reply on HN