1> You can find many analysts and many more engineers with Hadoop skills.
2> Hardware & power costs - are not huge (assuming hadoop is slower) - at least till you reach a massive scale.
LexisNexis open sources Hadoop challenger
11–15 of 15 posts
Re: LexisNexis open sources Hadoop challenger
#12Earlier quoted context omitted.
Anything there yet?
No source yet, but there's a demonstration VM. Unfortunately, it looks like they decided to choose to license the code under the AGPL[1], so it will probably be of little use to most of us. [1] http://hpccsystems.com/print/518
I'm not really sure what AGPL means for a subcomponent like that. MongoDB is similar. Does that just mean you have to share modifications to the component itself?
Re: LexisNexis open sources Hadoop challenger
#13"We are four faster than Hadoop on the Thor side. If Hadoop needs 1,000 nodes we can do it with 250 – that means less cooling and data center space." Many (most?) Hadoop jobs are IO-bound. It's unlikely that switching from Java to C++ could speed those up 4x. We should reserve judgement until these things are actually open-sourced.
I wont be surprised at all by 4x over Hadoop, rather I would be a little underwhelmed. What follows is an anecdotal data point. I have run similar code on similarly sized input on an Yahoo configuration of Hadoop (which is written in Java) and Google's mapreduce (which is written in C++). Google's implementation clearly ran in 4x to 5x less time. The difference would be between my job finishing in 2.5 hrs vs I have t…
You say similar input and similar code... but don't mention similar cores/RAM/disks. Do you know if those were similar, as well?
Re: LexisNexis open sources Hadoop challenger
#14Earlier quoted context omitted.
No source yet, but there's a demonstration VM. Unfortunately, it looks like they decided to choose to license the code under the AGPL[1], so it will probably be of little use to most of us. [1] http://hpccsystems.com/print/518
I'm not really sure what AGPL means for a subcomponent like that. MongoDB is similar. Does that just mean you have to share modifications to the component itself?
That's how the MongoDB guys interpret it[1]:
To say this another way: if you modify the core database source code, the goal is that you have to contribute those modifications back to the community.
Note however that it is NOT required that applications using mongo be published.
Re: LexisNexis open sources Hadoop challenger
#15"The risk business is worth $10m a year" - typo or British m? Seems like it would be much bigger than this. Or maybe this is just what LexisNexis makes on their old product?
Seems like a typo, though (confusingly) in some financial circles, MM is millions (1000 * 1000) and MMM is billions (1000 * 1000 * 1000).