31–40 of 122 posts
If processing latencies don't matter much, it's an easier more flexible system to use.
Earlier quoted context omitted.
I strongly agree. Although there are clearly uses for map/reduce at large scale there is also a tendency to use it for small problems where the overhead is objectionable. At work I've taken multiple Mao/reduce systems and converted them to run on my desktop, in one case taking a job that used to take 5 minutes just to startup down to a few seconds total. Right tool for the job and all that. If you need to process a 5…
I completely agree as well, but I don't consider myself much of an expert in NoSQL technologies (which is why I read up on threads like this to find out). Does anyone have a use case where data is on a single machine and map reduce is still relevant? (I am involved in a project at work where the other guys seem to have enthusiastically jumped on MongoDB without great reasons in my opinion).
I'll tell you where it's not used: High Energy Physics. We use a workflow engine/scheduler to run jobs over a few thousand nodes at several different locations/batch systems in the world. If processing latencies don't matter much, it's an easier more flexible system to use.
Not for gauge configuration generation, which has to be done sequentially, but inversions and measurements can be mapped / reduced.
Earlier quoted context omitted.
I completely agree as well, but I don't consider myself much of an expert in NoSQL technologies (which is why I read up on threads like this to find out). Does anyone have a use case where data is on a single machine and map reduce is still relevant? (I am involved in a project at work where the other guys seem to have enthusiastically jumped on MongoDB without great reasons in my opinion).
> Does anyone have a use case where data is on a single machine and map reduce is still relevant? What matters is the running data structure. For example, you can have Petabytes of logs but you need a map/table of some kind to do aggregations/transformations. Or a sparse-matrix based model. There are types of problems that can partition the data structure and work in parallel in the RAM of many servers. Related: it's…
That doesn't imply a NoSQL solution to me. Just parallel processing on different parts of the data. If I am wrong can you point me to a clearer example?
Earlier quoted context omitted.
I completely agree as well, but I don't consider myself much of an expert in NoSQL technologies (which is why I read up on threads like this to find out). Does anyone have a use case where data is on a single machine and map reduce is still relevant? (I am involved in a project at work where the other guys seem to have enthusiastically jumped on MongoDB without great reasons in my opinion).
On modern hardware with many cpu cores you can use a similar process of fork and join to maximise throughput of large datasets.
Why is that any better than all of your data on one database server, and each cluster node querying for part of the data to process it? Obviously there will be a bottleneck if all nodes try to access the database at the same time, but I see no benefit otherwise, and depending on the data organisation, I don't even see NoSQL solving that problem (you are going to have to separate the data to different servers for the NoSQL solution, why is that any better than cached query from a central server?).
Earlier quoted context omitted.
How big is your data? Yours is the first example where I a decent knowledge of the field, so can understand the needs accurately. In most cases I see people using NoSQL in places where MySQL could handle it easily. Maybe I am just too set in my relational database ways of thinking (having used them for 13 years), but there are few cases where I see NoSQL solutions being beneficial, other than for ease of use(most peo…
First, I should note that my needs are fairly specific, and not typical of the rest of the NGS world. The datasets are essentially the same though. The rate at which we are acquiring new data has been accelerating, but each of our Illumina datasets is only 30GB or so. The total accumulated data is still just a few TB. The real imperative for using MR is more about the processing of that data. Integrating HMMER, for i…
MongoDB: where you need Map/Reduce to do any aggregation at all.
[1] http://docs.mongodb.org/manual/aggregation/
[2] http://stackoverflow.com/questions/13908438/is-mongodb-aggre...
Earlier quoted context omitted.
First, I should note that my needs are fairly specific, and not typical of the rest of the NGS world. The datasets are essentially the same though. The rate at which we are acquiring new data has been accelerating, but each of our Illumina datasets is only 30GB or so. The total accumulated data is still just a few TB. The real imperative for using MR is more about the processing of that data. Integrating HMMER, for i…
So is it fair to say it is an "ease of use" use case?
Edit: I should say 'currently impossible' since as I noted, I can imagine being able to build SQL queries around PSSM comparisons and the like. I just can't build a system to last 5+ years around something that might be available at some point.
Since I can't reply directly- agreed :)
We had been using hadoop+hive+mr to run targetting expressions over billions of time series events from users. But we have recently moved a lot back to mysql+tokudb+sql which can compress the data well and keep it to just a few terrabytes. Seems we weren't big data enough and we were tired of the execution times, although impala and fb's newly released presto might also have fitted. Add: down voters can explain their…
This doesn't answer the question - you described /how/ you solved some problem, not what problem you're actually solving. (Mind you, at my writing this is the top comment, so I don't think you're getting many downvotes. But your comment irked me, so there you go.)