Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudera and Hortonworks merge

cnbc.com

Post reply on HN
Print view View original

41–50 of 60 posts

Re: Cloudera and Hortonworks merge

#41

Post by batoure »

Tsarbomb wrote: post #27

Can someone explain to me what the big draw was for Hortonworks or Cloudera? Working as a lead in a small team that deals with a colossal amount of data (human genomics), it was always easier for us to hand roll deployments with terraform/ansible in either baremetal or OpenStack environments. In the public clouds like AWS we are using the managed services like EMR. The whole sales pitch I've always got from either ho…

Both companies make most of their money providing professional services. I was going to touch on EMR but I think the “small team” comment is a better differentiator. When you have a large cluster with a large volume of workloads distributed computing becomes much less forgiving. The algorithm that was so fast and so valueable when you first deployed suddenly becomes a bottle neck for the rest of your operations because not enough time was spent thinking about partition strategies... Horton and Cloudera have made their money giving big companies advice when they get in over their heads. It’s rarely in the beginning but usually several years after companies have made a hard commitment to Hadoop and distributed computing.

Re: Cloudera and Hortonworks merge

#42

Post by batoure »

capkutay wrote: post #8

Good move for both companies. The surplus of 'enterprise hadoop' companies was created by a mixture of hype and a peak in VC investment in open source. The fact that Cloudera, Hortonworks, MapR were all founded and raised $100m+ around the same time was a bit superfluous for the whole market. Hard to say where this leaves MapR now. They seem to be the odd man out in terms of growth and adoption.

Don’t count MapR down, their commitment to developing ports and tools for the ecosystem outside the jvm was very forward thinking. Their internal engineering talent either makes them a prime aquaition target for CDH in future or a fierce competitor as systems like kudu become more common and production ready.

Re: Cloudera and Hortonworks merge

#43

Post by jamesblonde »

thinkmassive wrote: post #28

> The two companies are committed to supporting existing offerings from the two companies for at least three years but will work on a "unity release" of software, drawing on technologies from both companies' portfolios, Reilly said. I’m really intrigued to see whether Ambari or Cloudera Manager wins out in the long run. This is unexpected but interesting. (I worked at Hortonworks 2014-2016, no current affiliation)

This is, of course, an anti open-source move. So, Cloudera Manager will win, there. The stack will go largely proprietary, as this is a defensive move - AWS are just killing them in the cloud. They need to prevent AWS just repackaging the Hadoop core and reselling it - what better way than by making your new platform AGPL-v3 or something, so you're still open-source, but AWS can't just resell it as EMR.

Re: Cloudera and Hortonworks merge

#44

Post by rawoke083600 »

ewencp wrote: post #31
Tsarbomb wrote: post #27

Can someone explain to me what the big draw was for Hortonworks or Cloudera? Working as a lead in a small team that deals with a colossal amount of data (human genomics), it was always easier for us to hand roll deployments with terraform/ansible in either baremetal or OpenStack environments. In the public clouds like AWS we are using the managed services like EMR. The whole sales pitch I've always got from either ho…

terraform: 2014 ansible: 2012 openstack: 2010 cloudera: 2008 hortonworks: 2011 emr: 2011 (those were quickly googled, apologies if i got any dates wrong) timing is important. cloudera had good, early timing and looked promising because of it. you are right that EMR definitely hurt all the other hadoop vendors, though i think people overestimate how comfortable big enterprises are with moving to public cloud. way more…

Plain-Old-Bash-Scripts 2018 ~

Re: Cloudera and Hortonworks merge

#45

Post by colin_mccabe »

alexnewman wrote: post #12

I was an early employee at Cloudera, am a Hadoop contributor and think this entire market is garbage. Basically the big data field went off the rails and this is Cloudera's way of trying to remain relevant. It's hard to point to a product that came after my tenure (I was on the team that original made the POS called Cloudera Manager) that's really used by anyone at scale. Cloud is displacing all of these tools and ne…

> It's hard to point to a product that came after my tenure (I was on the team that original made the POS called Cloudera Manager) that's really used by anyone at scale.

There are tons of them. Spark, Hue, Sentry, Kafka, the list goes on.

Re: Cloudera and Hortonworks merge

#46

Post by jamesblonde »

HenryR wrote: post #26
jamesblonde wrote: post #9

Earlier quoted context omitted.

We have a startup, Logical Clocks (www.logicalclocks.com), that just raised money to sell our data-science next-generation Hadoop stack. It has a new verison of HDFS called HopsFS (Nvme storage, distributed metadata) and support for GPUs in YARN. And distributed tensorflow, Spark, Airflow, Flink. So, what does that mean for us? I am shellshocked. I expect prices to increase (good for us). What else should we expect?

I am surprised that anyone is trying to differentiate on storage at this time, precisely when that's the part of the stack that's being cannibalized by the cloud vendors (look at the rate of innovation in HDFS over time; the effort is going elsewhere). Are you just targeting on-premise clusters, or is there some differentiation planned for the cloud as well?

We think that there is a niche for a higher performance dist FS than S3. We have integrated NVMe hardware with our HDFS implementation (HopsFS) and made its metadata layer distributed. NVMe means you can, for example, work with datasets with millions of files for deep learning - instead of having to munge them into parquet files because your FS is slowing down your machine learning pipeline.

Reference: https://www.logicalclocks.com/millions-and-millions-of-files...

We have also redesigned the stack around our distributed metadata layer.

We are primarily targeting on-prem right now, but HopsFS would be the fastest DFS in the cloud if you ran it there today.

Re: Cloudera and Hortonworks merge

#47

Post by softliving »

Hmmm, merger slowdown market, employees beware of the trickery! this happens and then the mgmt thinks of redundancy to show markets how efficient they are ... is there a humane company which maybe does not pay much but secure enough and challenges us to rise together ... seems like a distant dream!

Re: Cloudera and Hortonworks merge

#48

Post by tumanian »

Thaxll wrote: post #6

Hadoop very "fancy" 5-6 years ago, what's the trend now? I guess with managed services from AWS / Google it makes Hadoop less useful?

Its still there - storage is still mainly HDFS, but computation layer morphed into Spark on Yarn (Mesos as a scheduler is barely used). Barely anyone runs classic Mapreduce jobs, Spark all around, written in Scala(or java or python).

Re: Cloudera and Hortonworks merge

#49

Post by tumanian »

batoure wrote: post #42
capkutay wrote: post #8

Good move for both companies. The surplus of 'enterprise hadoop' companies was created by a mixture of hype and a peak in VC investment in open source. The fact that Cloudera, Hortonworks, MapR were all founded and raised $100m+ around the same time was a bit superfluous for the whole market. Hard to say where this leaves MapR now. They seem to be the odd man out in terms of growth and adoption.

Don’t count MapR down, their commitment to developing ports and tools for the ecosystem outside the jvm was very forward thinking. Their internal engineering talent either makes them a prime aquaition target for CDH in future or a fierce competitor as systems like kudu become more common and production ready.

MapR is great while you are in their ecosystem - once you step outside and try to bring in technology thats not supported by them, you run into little showstopper bugs and complete lack of documentation and community support.
Post reply on HN