Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: To everybody who uses MapReduce: what problems do you solve?

news.ycombinator.com

Post reply on HN
Print view View original

101–110 of 122 posts

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#101

Post by smoyer »

DenisM wrote: post #60

It's worth noting that CouchDB is using map-reduce to define materialized views. Whereas normally MR parallelization is used to scale out, in this case it's used instead to allow incremental updates for the materialized views, which is to say incremental updates for arbitrarily defined indexes ! By contrast SQL databases allow incremental updates only for indexes whose definition is well understood by the database en…

I've been using CouchDB (and now BigCouch) for about four years and it's both clever and useful. We're storing engineering documents (as attachments) and using map/reduce (CouchDB views) to segment the documents by the metadata stored in the fields. The only downside is that adding a view with trillions of rows can take quite a while.

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#102

Post by nl »

AsymetricCom wrote: post #69
alecco wrote: post #15

A large telco has a 600 node cluster of powerful hardware. They barely use it. Moving Big Data around is hard. Managing is harder. A lot of people fail to understand the overheads and limitations of this kind of architecture. Or how hard it is to program, especially considering salaries for this skyrocketed. More often than not a couple of large 1TB SSD PCIe and a lot of RAM can handle your "big" data problem. Before…

> Moving Big Data around is hard. I never had any issues with Hadoop. Took about 2 days for me to familiarize myself with it and adhoc a script to do the staging and setup the local functions processing the data. I really would like to understand what you consider "hard" about Hadoop or managing a cluster. It's pretty straight forward idea, architecture is dead simple, requiring no specialized hardware at any level.…

Then again, I come from the /. crowd, so YC isn't really my kind of people, generally.

W T F does this even mean? I genuinely do not understand what point you are trying to make?

I have used Slashdot longer than you have (ok, possibly not.. but username registered in 1998 here...).

I find HN has generally much more experienced people on it, who understand more about the tradeoffs different solutions provide.

The old Slashdot hidden forums like wahiscool etc were good like this too, but I don't think they exist anymore do they?

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#103

Post by tumanian »

lclarkmichalek wrote: post #28

This system isn't in production just yet, but should be shortly. We're parsing Dota2 replays and generating statistics and visualisation data from them, which can then be used by casters and analysts for tournaments, and players. The replay file format breaks the game down into 1 min chunks, which are the natural thing to iterate over. Before someone comes along and says "this isn't big data!", I know. It's medium da…

Wait, it takes more then a few minutes to spin up the hadoop mapreduce job. Are you using mapreduce for preprocessing or did you optimize hadoop for this scenario?

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#104

Post by baudehlo »

lclarkmichalek wrote: post #58
baudehlo wrote: post #44

Earlier quoted context omitted.

This sounds like something you could just do in SQL and have it all done in milliseconds.

For our highly unstructured, untyped, non relational artefacts? It'd be fairly impossible to use it as a datastore in this particular case, and regardless, it would provide little to no speed increase over our current application, as the limiting factor is the CPU cost of the map function.

I have extremely strong doubts about that. Everything can be modelled in modern databases, and SQL is probably much more powerful than you understand here.

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#105

Post by StavrosK »

lclarkmichalek wrote: post #28

This system isn't in production just yet, but should be shortly. We're parsing Dota2 replays and generating statistics and visualisation data from them, which can then be used by casters and analysts for tournaments, and players. The replay file format breaks the game down into 1 min chunks, which are the natural thing to iterate over. Before someone comes along and says "this isn't big data!", I know. It's medium da…

Which service is this for? It sounds very interesting.

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#106

Post by kamakazizuru »

_random_ wrote: post #87
kamakazizuru wrote: post #45

Earlier quoted context omitted.

sure - but that's not the original question. Maybe the OP is just curious to understand some in production use cases (and the approach to their implementation) - and isn't that interested to know the edge cases where it doesn't work.

Maybe his case is an edge case.

right - but then he would've framed the question as "what are the cases where mapreduce really doesn't make sense".

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#107

Post by conradev »

Airbnb uses distributed computing to analyze the huge amount of data that it generates every day[1]. The results are used for all sorts of things: assessing how Airbnb affects the local economy (for government relations), optimizing user growth, etc.

[1] http://nerds.airbnb.com/distributed-computing-at-airbnb/

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#108

Post by memracom »

collyw wrote: post #94
msellout wrote: post #85

Earlier quoted context omitted.

You can imagine cases where map-reduce is useful without any starting data. If you are analyzing combinations or permutations, you can create a massive amount of data in an intermediate step, even if the initial and final data sets are small.

Have you got any links on how to do that, as it sounds very like a problem I am trying to sole just now - combinations of DNA sequences that work together on a sequencing machine. At the moment I am self joining a table of the sequences to itself in MySQL, but after a certain number of self joins the table gets massive. Time to compute is more the problem rather than storage space though, as I am only storing the one…

If I had your problem, the first thing that I would do is try PostgreSQL to see if it does the joins fast enough. Second thing that I would try is to put the data in a SOLR db and translate the queries to a SOLR base query (q=) plus filter queries (fq=) on top.

Only if both of these fail to provide sufficient performance, would I look at a map reduce solution based on the Hadoop ecosystem. Actually, I wouldn't necessarily use the Hadoop ecosystem. It has a lot of parts/layers and the newer and generally better parts are not as well known so it is a bit more leading edge than lots of folks like. I'd also look at somethink like Riak http://docs.basho.com/riak/latest/dev/using/mapreduce/ because then you have your data storage and clustering issues solved in a bulletproof way (unlike Mongo) but you can do MapReduce as well.

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#109

Post by memracom »

mileswu wrote: post #80
batbomb wrote: post #31

I'll tell you where it's not used: High Energy Physics. We use a workflow engine/scheduler to run jobs over a few thousand nodes at several different locations/batch systems in the world. If processing latencies don't matter much, it's an easier more flexible system to use.

At least on the experimental LHC side, we process/analyse each event independently from every other event, so it's an embarrassingly parallel workload. All we do is split our input dataset up into N files, run N jobs, combine the N outputs. Because we have so much data (of the order of 25+ PB of raw data per year; it actually balloons to much more than this due to copies in many slightly different formats) and so man…

I remember when HEP invented that WWW technology stuff which turned out to be rather popular outside of HEP as well.

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#110

Post by DenisM »

smoyer wrote: post #101
DenisM wrote: post #60

It's worth noting that CouchDB is using map-reduce to define materialized views. Whereas normally MR parallelization is used to scale out, in this case it's used instead to allow incremental updates for the materialized views, which is to say incremental updates for arbitrarily defined indexes ! By contrast SQL databases allow incremental updates only for indexes whose definition is well understood by the database en…

I've been using CouchDB (and now BigCouch) for about four years and it's both clever and useful. We're storing engineering documents (as attachments) and using map/reduce (CouchDB views) to segment the documents by the metadata stored in the fields. The only downside is that adding a view with trillions of rows can take quite a while.

Strangely, there aren't many discussions of Couch* family on hacker news. Do you know why would that be?

I'm thinking about basing a new product around couchbase lite, but lack of popular acceptance is one of the things holding me back.

Post reply on HN