How can I make myself feel better about what I do by trying to diminish the work other people do using the same technologies?
Ask HN: To everybody who uses MapReduce: what problems do you solve?
11–20 of 122 posts
Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?
#12How strongly does a RNA binding protein bind to each possible sequence of RNA?
Interesting, I'm using it for protein localization prediction, PTMs, and NGS data.
Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?
#13Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?
#14Taking dumps of analytics logs and pulling out relevant info for our customers on app usage
Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?
#15A large telco has a 600 node cluster of powerful hardware. They barely use it. Moving Big Data around is hard. Managing is harder.
A lot of people fail to understand the overheads and limitations of this kind of architecture. Or how hard it is to program, especially considering salaries for this skyrocketed. More often than not a couple of large 1TB SSD PCIe and a lot of RAM can handle your "big" data problem.
Before doing any Map/Reduce (or equivalent), please I beg you to check out Introduction to Data Science at Coursera https://www.coursera.org/course/datasci
Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?
#16We had been using hadoop+hive+mr to run targetting expressions over billions of time series events from users. But we have recently moved a lot back to mysql+tokudb+sql which can compress the data well and keep it to just a few terrabytes. Seems we weren't big data enough and we were tired of the execution times, although impala and fb's newly released presto might also have fitted. Add: down voters can explain their…
This doesn't answer the question - you described /how/ you solved some problem, not what problem you're actually solving.
(Mind you, at my writing this is the top comment, so I don't think you're getting many downvotes. But your comment irked me, so there you go.)
Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?
#17I was using it on a D3.JS chart to aggregate data flow though our custom real-time analytic pipeline.
Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?
#18MongoDB: where you need Map/Reduce to do any aggregation at all.
Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?
#19A large telco has a 600 node cluster of powerful hardware. They barely use it. Moving Big Data around is hard. Managing is harder. A lot of people fail to understand the overheads and limitations of this kind of architecture. Or how hard it is to program, especially considering salaries for this skyrocketed. More often than not a couple of large 1TB SSD PCIe and a lot of RAM can handle your "big" data problem. Before…
I strongly agree. Although there are clearly uses for map/reduce at large scale there is also a tendency to use it for small problems where the overhead is objectionable. At work I've taken multiple Mao/reduce systems and converted them to run on my desktop, in one case taking a job that used to take 5 minutes just to startup down to a few seconds total.
Right tool for the job and all that. If you need to process a 50PB input though, map/reduce is the way to go.
Re: Ask HN: To everybody who uses MapReduce: what problems do you solve?
#20We had been using hadoop+hive+mr to run targetting expressions over billions of time series events from users. But we have recently moved a lot back to mysql+tokudb+sql which can compress the data well and keep it to just a few terrabytes. Seems we weren't big data enough and we were tired of the execution times, although impala and fb's newly released presto might also have fitted. Add: down voters can explain their…
This doesn't answer the question - you described /how/ you solved some problem, not what problem you're actually solving. (Mind you, at my writing this is the top comment, so I don't think you're getting many downvotes. But your comment irked me, so there you go.)
> not what problem you're actually solving.
> targetting expressions over billions of time series events from users.