Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: To everybody who uses MapReduce: what problems do you solve?

news.ycombinator.com

Post reply on HN
Print view View original

91–100 of 122 posts

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#91

Post by pwang »

srl wrote: post #16
willvarfar wrote: post #10

We had been using hadoop+hive+mr to run targetting expressions over billions of time series events from users. But we have recently moved a lot back to mysql+tokudb+sql which can compress the data well and keep it to just a few terrabytes. Seems we weren't big data enough and we were tired of the execution times, although impala and fb's newly released presto might also have fitted. Add: down voters can explain their…

This doesn't answer the question - you described /how/ you solved some problem, not what problem you're actually solving. (Mind you, at my writing this is the top comment, so I don't think you're getting many downvotes. But your comment irked me, so there you go.)

> run targetting expressions over billions of time series events from users.

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#92

Post by alecco »

lclarkmichalek wrote: post #58
baudehlo wrote: post #44

Earlier quoted context omitted.

This sounds like something you could just do in SQL and have it all done in milliseconds.

For our highly unstructured, untyped, non relational artefacts? It'd be fairly impossible to use it as a datastore in this particular case, and regardless, it would provide little to no speed increase over our current application, as the limiting factor is the CPU cost of the map function.

Maybe you should consider transforming the data to structure it a bit. Have a unique identifier per object and arrays for each seen characteristic with (value, id), and it's reverse index. Then decompose the processing of each object to sub-problems matching n-way the characteristics. It doesn't have to be in SQL, though. I'd try a columnar RDBMS. YMMV

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#93

Post by oceanplexian »

alecco wrote: post #82
AsymetricCom wrote: post #78

Earlier quoted context omitted.

It was a pretty easy problem, parsing logs for performance statistics. But moving the data is the easy part and that's why I was incredulous of the OP's statement. I'm starting to wonder if this is really "Hacker News" or if it's "we want free advice and comments from engineers on our startups so lets start a forum with technical articles"

Big Data should be on the Peta+ level. Even with 10G Ethernet it takes a lot of bandwidth and time to move things around (and it's very hard to keep 10G ethernet full at a constant rate from storage). This is hard even for telcos. Note Terabyte+ level today fits on SSD.

Not really, "Big Data" has nothing to do with how many bytes you're pushing around.

Some types of data analytics are CPU heavy and require distributed resources. Your comment about 10G isn't true. You can move around a Tb every 10 minutes or so. SSDs or a medium sized SAN could easily keep up with the bandwidth.

If your data isn't latency sensitive and run in batches, building a Hadoop cluster is a great solution to a lot of problems.

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#94

Post by collyw »

msellout wrote: post #85
collyw wrote: post #26

Earlier quoted context omitted.

I completely agree as well, but I don't consider myself much of an expert in NoSQL technologies (which is why I read up on threads like this to find out). Does anyone have a use case where data is on a single machine and map reduce is still relevant? (I am involved in a project at work where the other guys seem to have enthusiastically jumped on MongoDB without great reasons in my opinion).

You can imagine cases where map-reduce is useful without any starting data. If you are analyzing combinations or permutations, you can create a massive amount of data in an intermediate step, even if the initial and final data sets are small.

Have you got any links on how to do that, as it sounds very like a problem I am trying to sole just now - combinations of DNA sequences that work together on a sequencing machine.

At the moment I am self joining a table of the sequences to itself in MySQL, but after a certain number of self joins the table gets massive. Time to compute is more the problem rather than storage space though, as I am only storing the ones that work (> 2 mismatches in the sequence). Would Map Reduce help in this scenario?

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#95

Post by twic »

thrownaway2424 wrote: post #19
alecco wrote: post #15

A large telco has a 600 node cluster of powerful hardware. They barely use it. Moving Big Data around is hard. Managing is harder. A lot of people fail to understand the overheads and limitations of this kind of architecture. Or how hard it is to program, especially considering salaries for this skyrocketed. More often than not a couple of large 1TB SSD PCIe and a lot of RAM can handle your "big" data problem. Before…

I strongly agree. Although there are clearly uses for map/reduce at large scale there is also a tendency to use it for small problems where the overhead is objectionable. At work I've taken multiple Mao/reduce systems and converted them to run on my desktop, in one case taking a job that used to take 5 minutes just to startup down to a few seconds total. Right tool for the job and all that. If you need to process a 5…

> Mao/reduce systems

Well, that certainly sounds like ...

puts on sunglasses

... a Great Leap Forward.

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#96

Post by grogenaut »

bcoughlan wrote: post #65

A lot of people in this thread are saying that most data is not big enough for MapReduce. I use Hadoop on a single node for ~20GB of data because it is an excellent utility for sorting and grouping data, not because of its size. What should I be using instead?

Obviously you should throw out the solution that worked for you and start over. 20GB just isn't cool enough to use M/R.

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#98

Post by apw »

virtuabhi wrote: post #79
batbomb wrote: post #31

I'll tell you where it's not used: High Energy Physics. We use a workflow engine/scheduler to run jobs over a few thousand nodes at several different locations/batch systems in the world. If processing latencies don't matter much, it's an easier more flexible system to use.

Yes, people in sciences write distributed code (e.g., using MPI) and employ job queueing systems. I think they work well for embarrassingly parallel tasks. But what if you need to do a group-by operation? MapReduce makes these operations easy. Also the Hadoop system takes care of failing nodes, data partitioning, communication, etc. I do hope physics people consider MR systems in addition to the existing ones they us…

MPI has a reduce operator. On some supercomputers, the operator is implemented in an ASIC.

It is quite easy to implement map/reduce style programs in MPI. It is quite difficult or impossible to implement the parallel algorithms that win the Gordon Bell prize, almost all of which are implemented in MPI, using Hadoop.

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#99

Post by seanmcdirmid »

apw wrote: post #98
virtuabhi wrote: post #79

Earlier quoted context omitted.

Yes, people in sciences write distributed code (e.g., using MPI) and employ job queueing systems. I think they work well for embarrassingly parallel tasks. But what if you need to do a group-by operation? MapReduce makes these operations easy. Also the Hadoop system takes care of failing nodes, data partitioning, communication, etc. I do hope physics people consider MR systems in addition to the existing ones they us…

MPI has a reduce operator. On some supercomputers, the operator is implemented in an ASIC. It is quite easy to implement map/reduce style programs in MPI. It is quite difficult or impossible to implement the parallel algorithms that win the Gordon Bell prize, almost all of which are implemented in MPI, using Hadoop.

MPI can't schedule to optimize IO/data shuffling like MapReduce, this is the price for being more general. The biggest competition to MPI in the HPC space is actually CUDA and GPUs. In either case, the data pipeline is exposed to the compiler or run time for optimization, where as MPI everything is just messaging.

Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?

#100

Post by schrodinger »

oceanplexian wrote: post #93
alecco wrote: post #82

Earlier quoted context omitted.

Big Data should be on the Peta+ level. Even with 10G Ethernet it takes a lot of bandwidth and time to move things around (and it's very hard to keep 10G ethernet full at a constant rate from storage). This is hard even for telcos. Note Terabyte+ level today fits on SSD.

Not really, "Big Data" has nothing to do with how many bytes you're pushing around. Some types of data analytics are CPU heavy and require distributed resources. Your comment about 10G isn't true. You can move around a Tb every 10 minutes or so. SSDs or a medium sized SAN could easily keep up with the bandwidth. If your data isn't latency sensitive and run in batches, building a Hadoop cluster is a great solution to…

Of course big data is about number of bytes. That's what something like map reduce helps with. It depends on breaking down your input into smaller chunks, and the number of chunks is certainly related to the number of bytes.
Post reply on HN