MapR may shut down as investor pulls out after ‘extremely poor results’ • Hacker News Board

Re: MapR may shut down as investor pulls out after ‘extremely poor results’

#41

Post by threeseed » Thu, May 30, 2019, 9:05 PM UTC

dmix wrote: post #35

capkutay wrote: post #28

Earlier quoted context omitted.

Hadoop's whole appeal was it was cheap and scalable. Did it actually work without serious engineering teams maintaining each distribution? absolutely not. the hype was purely VC funded. fast forward 10 years and hadoop has basically been killed off by hosted storage services that are more expensive but 10000x easier to manage.

I'm not really familiar with this industry. Are these Hadoop clusters hosted locally by these businesses? And MapR et all were providing the software/consulting to help manage them? Then I'm guessing better cloud hosted options came out offering similar capabilities. If that's the case was it really a big surprise that "cloud" hosting would eat any self-hosted platform's lunch?

So in the very recent past (i.e. a few years) businesses wanting to do Data Science ran a distro of Hadoop e.g. MapR that they bought from the vendor. They charged an exorbitant charge per node (e.g. $10k) because they figured they would get people to switch from Teradata or Oracle.

Now what the cloud offered was so much more compelling. You had per hour pricing on the order of $20 for a minimal cluster. You had unlimited autoscaling so you didn't have to do capacity planning and go through procurement processes to pre-order hardware/licenses. And of course you had unlimited, ultra-cheap storage courtesy of S3. And it also allowed each team to have their own mini-cluster instead of everyone relying on some giant one.

I don't think the recent explosion in Data Science would've happened without the cloud.

Re: MapR may shut down as investor pulls out after ‘extremely poor results’

#42

Post by notyourday » Thu, May 30, 2019, 9:07 PM UTC

dmix wrote: post #35

capkutay wrote: post #28

Earlier quoted context omitted.

Hadoop's whole appeal was it was cheap and scalable. Did it actually work without serious engineering teams maintaining each distribution? absolutely not. the hype was purely VC funded. fast forward 10 years and hadoop has basically been killed off by hosted storage services that are more expensive but 10000x easier to manage.

I'm not really familiar with this industry. Are these Hadoop clusters hosted locally by these businesses? And MapR et all were providing the software/consulting to help manage them? Then I'm guessing better cloud hosted options came out offering similar capabilities. If that's the case was it really a big surprise that "cloud" hosting would eat any self-hosted platform's lunch?

Yes, MapR was local hosting.

Re: MapR may shut down as investor pulls out after ‘extremely poor results’

#43

Post by notyourday » Thu, May 30, 2019, 9:08 PM UTC

brokensegue wrote: post #25

notyourday wrote: post #13

Earlier quoted context omitted.

I'm not someone very much in the "big data" [0] but from what I see streaming analytics especially Spark-based solutions are eating the world that used to be dominated by Hadoop. I also think cloud based data lakes and tooling around them seriously decreased the appeal of Hadoop. [0] If I can fit your big data into my memory, it is not "big data".

i mean spark is often deployed on top of yarn where it's still in the "hadoop" sphere.

But I'm not loading up on my SuperMicro, Dell or HP servers. So there's no need to buy a decent Hadoop implementation from MapR.

Re: MapR may shut down as investor pulls out after ‘extremely poor results’

#44

Post by threeseed » Thu, May 30, 2019, 9:10 PM UTC

notyourday wrote: post #13

heyyyouu wrote: post #10

Earlier quoted context omitted.

Which of the shinier, newer solutions do you think people have found more attractive in particular? Genuinely curious, thanks!

I'm not someone very much in the "big data" [0] but from what I see streaming analytics especially Spark-based solutions are eating the world that used to be dominated by Hadoop. I also think cloud based data lakes and tooling around them seriously decreased the appeal of Hadoop. [0] If I can fit your big data into my memory, it is not "big data".

Spark is very much part of Hadoop. It uses Hadoop libraries throughout it including for the core part of reading/writing data

And that adage of big data can't fit in memory is nonsense these days. We run clusters with hundreds of terabytes of RAM which is very much big data. It's pretty easy and affordable with the cloud.

Re: MapR may shut down as investor pulls out after ‘extremely poor results’

#45

Post by threeseed » Thu, May 30, 2019, 9:15 PM UTC

capkutay wrote: post #37

dmix wrote: post #35

Earlier quoted context omitted.

I'm not really familiar with this industry. Are these Hadoop clusters hosted locally by these businesses? And MapR et all were providing the software/consulting to help manage them? Then I'm guessing better cloud hosted options came out offering similar capabilities. If that's the case was it really a big surprise that "cloud" hosting would eat any self-hosted platform's lunch?

yes mostly locally hosted back then. the idea was you'd have this massive distributed file system to hold all your data, and VC backed comopanies like Cloudera, MapR, Hortonworks promised everyone that they'd build the SQL layer, the data warehousing, BI, and all the other enterprise features you'd need to basically replace your expensive Teradata, Oracle Exadata, and other data warehousing systems. But of course it…

Actually it did play out that way.

Most enterprises these days will have a massive distributed file system (S3 or equivalent) that has most of their data. And most will be running a decent percentage of data transformation jobs using Spark or some SQL layer e.g. Presto and then running BI tools like Tableau using Athena/Redshift Spectrum (or equivalent) as their SQL layer.

It's just that this is all playing out on the cloud instead of on-premise with some vendor. But you have definitely been seeing the decline of the core enterprise data warehouse.

Re: MapR may shut down as investor pulls out after ‘extremely poor results’

#46

Post by stormbeard » Thu, May 30, 2019, 9:21 PM UTC

noir-york wrote: post #29

notyourday wrote: post #14

Earlier quoted context omitted.

That's why I think MapR missed the mark. They had a software only solution one could deploy on a top of generic hardware in a data center to solve hundreds of TB to small number of PB storage problem that was real for enterprises and since those nodes would also be able to do compute jobs that was the way to get enterprises to adapt that technology. Going to object store was possible but it required dev time and in t…

Interesting post. Out of curiosity: aren't there others who have developed cluster storage exposed as NFS / FUSE?

Nutanix and VMWare have been doing that pretty successfully since ~2012.

Re: MapR may shut down as investor pulls out after ‘extremely poor results’

#47

Post by moonjoAWS » Thu, May 30, 2019, 9:25 PM UTC

Any impacted engineers looking for a new home in the Bay Area or Seattle, feel free to reach out moonjo@amazon.com to learn more about teams with Redshift.

Re: MapR may shut down as investor pulls out after ‘extremely poor results’

#48

Post by Communitivity » Thu, May 30, 2019, 9:40 PM UTC

This is sad. I remember the MapR demo at the NVidia GTC a couple years ago, and their demo was amazing. I hope they pull a rabbit out of the hat and are able to continue, but if they don't then I hope they Open Source as much as they can.

Re: MapR may shut down as investor pulls out after ‘extremely poor results’

#49

Post by noir-york » Thu, May 30, 2019, 9:41 PM UTC

notacoward wrote: post #39

noir-york wrote: post #29

Earlier quoted context omitted.

Interesting post. Out of curiosity: aren't there others who have developed cluster storage exposed as NFS / FUSE?

Absolutely. In fact, that was my last project, which I worked on for almost a decade. Unfortunately, it's a tough market. At the low end, everyone thinks they can do it themselves. At the high end, you're up against either the "POSIX is dead let's use an object store" crowd or the enterprise EMC/NetApp/Isilon crowd. On the other side of the ledger, development costs are high. Developers are harder to find, more equip…

Thank you very much for taking the time!

Re: MapR may shut down as investor pulls out after ‘extremely poor results’

#50

Post by jamesblonde » Thu, May 30, 2019, 10:03 PM UTC

joehandzik wrote: post #15

We're seeing a lot of regret around sprawling Hadoop deployments, so this doesn't surprise me. Other Hadoop vendors (vendor?) pivoting to machine learning is a bandaid as the compute capabilities scale beyond HDFS's performance limitations. Look towards new-gen startups around NVME/NVMEOF (WekaIO, Excelero, E8, etc etc) to fill the void. The question is going to be: will anyone provide an intelligent way to maintain…

We build a next-gen version of HDFS (HopsFS) that has distributed, consistent, transactional metadata where small files (https://www.logicalclocks.com/millions-and-millions-of-files...

Our business model is to build a data science platform, Hopsworks, around our distributed metadata layer. And yes we use YARN (training models) and also Kubernetes (serving models). The choice of resource mgr is really just an implementation detail, as the platform is backed by a REST API.