Earlier quoted context omitted.
Libraries like Weka and Mahout are no more toys than any other library that implements standard and widely applicable algorithms. Yes, you need to do a lot of extra work to properly model your problems, choose features, and combine different algorithms into a final product. But it's not often that you need to tweak the core algorithms that these libraries provide. If you really understand enough to implement new clas…
You're correct to identify the point of libraries like Weka and Mahout, which are both written in Java, as providing a solid framework for interaction between and among your program and other algorithms. However, Java isn't the right solution for everyone. Moreover, in Weka's case, the GPL licensing may not comport with everyone's requirements. Mahout's license is more friendly to proprietary software, so it's admitt…
Apache Mahout: Scalable machine learning for everyone
21–26 of 26 posts
Re: Apache Mahout: Scalable machine learning for everyone
#22Earlier quoted context omitted.
Libraries like Weka and Mahout are no more toys than any other library that implements standard and widely applicable algorithms. Yes, you need to do a lot of extra work to properly model your problems, choose features, and combine different algorithms into a final product. But it's not often that you need to tweak the core algorithms that these libraries provide. If you really understand enough to implement new clas…
Dunno about weka but my last experience ( 5 months back) with Mahout was not good. There still are quite a few bugs and the fact that entire code base is in Java makes it extremely unpleasant for someone who wants to hack and modify the code to jump right in and start tweaking stuff. However, in its defense, it is open source is probably the only hadoopified ml library out there and has given me a ton of good ideas o…
I would much rather have library and framework code that someone else has written, debugged, and supports to be written in Java: easy to browse in a good IDE, statically typed, lots of unit tests so you can hack away with some protection, etc.
Re: Apache Mahout: Scalable machine learning for everyone
#23Re: Apache Mahout: Scalable machine learning for everyone
#24I think Mahout has one key problem, and that's its purported scope. The committers' attitude for a long while, which I didn't like myself, was to ingest as many different algorithms that had anything to do with large-scale machine learning.
The result is an impressive-looking array of algorithms. It creates a certain level of expectation about coverage. If there were no clustering algorithms, you wouldn't notice the lack of algorithm X or Y. But there are a few, so, people complain it's not supporting what they're looking for.
But there's also large variation in quality. Some pieces of the project are quite literally a code dump from someone 2 years ago. Now, some is quite excellent. But because there's a certain level of interest and hype and usage, finding anything a bit stale or buggy leaves a negative impression.
I do think Mahout is much, much better than nothing, at least. There is really only one game in town for "mainstream" distributed ML. If it is only a source of good ideas, and a framework to build on, then it's added a lot of value.
I also think that some corners of the project are quite excellent. The recommender portions are more mature as they predate Mahout and have more active support. Naive Bayes, for example, in contrast, I don't think has been touched in a while.
And I can tell you that Mahout is certainly really used by real companies to do real work! I doubt it solves everyone's problems, but it sure solves some problems better than they'd have solved them from scratch.
I strongly agree with here is that you're never likely to find an ML system that works well out-of-the-box. It's always a matter of tuning, customizing for your domain, preparing input, etc. properly. If that's true, then something like Mahout is never going to be satisfying, because any one system is going to be suboptimal as-is for any given system.
And for the specialist, no system, including Mahout, is ever going to look as smart or sophisticated as what you know and have done. There are infinite variations, specializations, optimizations possible for any algorithm.
So I do see a lot of feedback from smart people that, hmm, I don't think this all that great, and it's valid. For example, I wrote the recommender bits (mostly) and I think the ML implemented there is quite basic. But you see there's somehow a lot of enthusiasm for it, if only because it's managed to roughly bring together, simplify, and make practical the basic ML that people here take for granted. That's good!
Re: Apache Mahout: Scalable machine learning for everyone
#25Earlier quoted context omitted.
There aren't very many statisticians/MLers who suggest (or practice) reimplementing your own algorithms, except for quite simple things, because the risk of getting something wrong is pretty high, and the work to make things efficient is non-trivial. If anything, the current push is in the other direction, towards encouraging more people to share their code, and more people to use well-tested code, through initiative…
It's a fine line to walk. On the one hand, community-vetted code is a spectacular idea for the core algorithms, but on the other, overly-restrictive licenses (like [L]GPL) effectively preclude the maximum utility being derived from them.
Re: Apache Mahout: Scalable machine learning for everyone
#26Earlier quoted context omitted.
Dunno about weka but my last experience ( 5 months back) with Mahout was not good. There still are quite a few bugs and the fact that entire code base is in Java makes it extremely unpleasant for someone who wants to hack and modify the code to jump right in and start tweaking stuff. However, in its defense, it is open source is probably the only hadoopified ml library out there and has given me a ton of good ideas o…
Wow, we disagree. As much as I like to do my own development in dynamic languages like Clojure and JRuby, for me: I would much rather have library and framework code that someone else has written, debugged, and supports to be written in Java: easy to browse in a good IDE, statically typed, lots of unit tests so you can hack away with some protection, etc.
2. Their unit test coverage was not good enough which incidentally is how I found that there were bugs. The problem in trying to contribute back to the community by trying to rectify these bugs? When I read the source code, I get the feeling that each algorithm is owned to a great extent by one developer who brings in their own idiosyncrasies which means that you need to really study the code to make sure you don't accidentally add more bugs. The other disadvantage of this approach is that questions regarding potential bugs and puzzling issues can go unanswered or answered in an unsatisfactory manner (mainly because of the one developer writing most of the code issue).
Having said all this, I want to be charitable and chart these to growing pains. But if I were building something critical and big dataish, I would either use Python (dumbo) or Scala which are much more concise languages where it is easier to express math without introducing bugs.