Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MapR may shut down as investor pulls out after ‘extremely poor results’

siliconangle.com

Post reply on HN
Print view View original

61–70 of 84 posts

Re: MapR may shut down as investor pulls out after ‘extremely poor results’

#61

Post by jamesblonde »

joehandzik wrote: post #38
threeseed wrote: post #30

Earlier quoted context omitted.

So I deal with hundreds of Data Scientists and none of this is true for us. Hadoop vendors are simply getting killed by the cloud. AWS EMR, Azure ML, Google Dataproc. And many people are simply swapping HDFS for S3 or similar object stores. Never seen anyone switching to very expensive, more HPC style NVME setups. That seems strange given the data sizes we are working with. And 95% of what Data Scientists do is ETL,…

It really depends on what the workload is and if performance or scale are priorities. Our team and the products we build come from the HPC side of the fence and we've been engaged with all sorts of large businesses that know their performance requirements (think 100s of GB/s read bandwidth), and it leads them down the path of either HPC filesystems or newer NVMe-oriented filesystems. Does 'AI' or 'data science' alway…

The problem you have with Weko and those newer filesystems is that the tooling hasn't caught up. They are trying to get their APIs upstreamed into TensorFlow. But what about Spark, Pandas, Arrow, etc? It's not enough to have just the training part of the pipeline - you need to be the filesyste for the whole pipeline. That's what we provide with HopsFS.

Re: MapR may shut down as investor pulls out after ‘extremely poor results’

#62

Post by xkgt »

As many others pointed out, these product vendors are getting killed by the cloud (I am not saying whether it is good or bad). On-demand compute and storage scalability of cloud makes perfect sense for Big data infrastructure.

These companies are failing because they can't establish a use case of running their product on cloud. It is ironic because when AWS started EMR, MapR was one of the distributions they offered via EMR[1]. Over period of time, they weened off MapR and stopped offering them as deployment option. So it is another case of AWS cannibalizing its own third party ecosystem.

Once this was gone, MapR got reduced just another marketplace partner [2] and their additional licensing cost didn't make sense when native EMR was sufficient for most use case.

[1] https://aws.amazon.com/emr/mapr/pricing/ [2] https://mapr.com/partners/partner/amazon-elastic-mapreduce-a...

Re: MapR may shut down as investor pulls out after ‘extremely poor results’

#63

Post by atombender »

Lots of interesting discussion here about billions of files and what not. I don't really work with big data -- the largest amounts of data I work with are in the range of tens of millions at most, and my modest "ETL" is about reading from APIs and cleaning up incoming data into consistent schemas, refining data (e.g. geocoding addresses, or correlating against public demographic records/statistics), a process that is super fast and doesn't even require multiple nodes for the most part -- so I never encounter anything I can't solve with some Postgres or Redis. What I'd love to know is what everyone is actually doing with their huge clusters. I can understand the need for big data in hard sciences (CERN, or genetics, or similar), and of course machine learning (e.g. image feature extraction), and then there are real time auction use cases like ad servers that probably have some big data component. What else are people doing out there?

Re: MapR may shut down as investor pulls out after ‘extremely poor results’

#64

Post by jcims »

xkgt wrote: post #62

As many others pointed out, these product vendors are getting killed by the cloud (I am not saying whether it is good or bad). On-demand compute and storage scalability of cloud makes perfect sense for Big data infrastructure. These companies are failing because they can't establish a use case of running their product on cloud. It is ironic because when AWS started EMR, MapR was one of the distributions they offered…

Which has to be a gut punch because EMR feels like a bag of bolts rather than any kind of polished system.

Re: MapR may shut down as investor pulls out after ‘extremely poor results’

#65

Post by mbreese »

notyourday wrote: post #51
noir-york wrote: post #29

Earlier quoted context omitted.

Interesting post. Out of curiosity: aren't there others who have developed cluster storage exposed as NFS / FUSE?

They all sucked. MapR sucked the least. Considering a comparable EMC solution was ~20x more expensive and less flexible, MapR happened to be in the right place at the right time with the right software but blew it because they decided they knew better what the customers needed to buy vs what the customers wanted to buy. Fundamentally, file system clusters are a very difficult business to get into and make enough mone…

I remember being in a pitch meeting with MapR a few years ago. It was for a large academic institution’s HPC group. We were all very interested in their NFS/HDFS tools, which would have met a very critical need. They just kept talking about the Hadoop workflows, which wouldn’t have flown at all for this group. Most HPC workflows/tools don’t map very well to Hadoop or at the minimum would have required tools to have been rewritten.

We really wanted to spend the money, but as far as I remember, we didn’t end up doing anything more than a test install.

I’d still love a middle option that could handle petabyte scale storage without the Isilon price tag.

Re: MapR may shut down as investor pulls out after ‘extremely poor results’

#66

Post by dsl »

notyourday wrote: post #14
notacoward wrote: post #12

Earlier quoted context omitted.

The increasing adoption of object stores and other not-quite-filesystems, even when a real filesystem would be the more appropriate choice, is definitely part of this. For good or ill, people would rather work on or with one of the simplified alternatives. I'm in that camp myself now, working on a system with lame HDFS-like semantics after two decades of proving the "POSIX can't scale" folks wrong. It's just not a ba…

That's why I think MapR missed the mark. They had a software only solution one could deploy on a top of generic hardware in a data center to solve hundreds of TB to small number of PB storage problem that was real for enterprises and since those nodes would also be able to do compute jobs that was the way to get enterprises to adapt that technology. Going to object store was possible but it required dev time and in t…

This was exactly my experience with MapR. They refused to sell you the products you actually wanted.

My first response to this news was "good riddance", my second thought was "I hope they open source the storage stuff."

Re: MapR may shut down as investor pulls out after ‘extremely poor results’

#67

Post by kristjansson »

parasubvert wrote: post #59
capkutay wrote: post #52

Earlier quoted context omitted.

> Because Hadoop has not been killed off. > HDFS being replaced with more scalable solutions like S3/EMRFS. Thank you for re-affirming my point. I'm not going to go into the typical HN "I'm going to nitpick your point down to the bone and argue you over the pointless details", but 2012 Hadoop is not the same thing as the tools you're describing.

Except you’re not being just a bit inaccurate, you’re flat wrong. Simply put, Hadoop is not HDFS, it is a much larger Apache ecosystem that includes Spark, Hive, MR, NiFi, etc. All of which are doing fine.

I’m not sure how common that usage is . ‘Hadoop Ecosystem’ surely encompasses Spark and the rest, but I’d argue ‘Hadoop’ most commonly refers to MR, HDFS, and maybe YARN. Which is only to say I’d be surprised to hear an application described as ‘using Hadoop’ and find it using Spark on data in S3

Re: MapR may shut down as investor pulls out after ‘extremely poor results’

#68

Post by saganus »

joehandzik wrote: post #38
threeseed wrote: post #30

Earlier quoted context omitted.

So I deal with hundreds of Data Scientists and none of this is true for us. Hadoop vendors are simply getting killed by the cloud. AWS EMR, Azure ML, Google Dataproc. And many people are simply swapping HDFS for S3 or similar object stores. Never seen anyone switching to very expensive, more HPC style NVME setups. That seems strange given the data sizes we are working with. And 95% of what Data Scientists do is ETL,…

It really depends on what the workload is and if performance or scale are priorities. Our team and the products we build come from the HPC side of the fence and we've been engaged with all sorts of large businesses that know their performance requirements (think 100s of GB/s read bandwidth), and it leads them down the path of either HPC filesystems or newer NVMe-oriented filesystems. Does 'AI' or 'data science' alway…

Wow... how do you achieve petabytes of NVMe? like, what is the biggest drive that you can get today?

And then you just plug a ton of them into what kind of "motherboard" to interconnect them? I guess Infiniband based bus? or I am totally off-base here?

Any more details are appreciated, if only to satisfy my curiosity :)

Re: MapR may shut down as investor pulls out after ‘extremely poor results’

#69

Post by opportune »

nwsm wrote: post #16
Joeri wrote: post #2

So, that would mean the hadoop distro market would be down to just cloudera? A single vendor market is not a market, it's a legacy product.

Azure's HDInsight all-in-one suite uses it.

In addition to HDInsight being cloudera, it's also basically a legacy product. Azure technical solutions people are now all pushing databricks. And personally, when designing a new project, I don't see why you would use HDInsight instead of a bespoke distributed vm architecture with one (azure storage) or two-tier storage (one local to the vm, one in Azure storage) unless you really need to use some hadoop jar you found lying around.

Re: MapR may shut down as investor pulls out after ‘extremely poor results’

#70

Post by capkutay »

parasubvert wrote: post #59
capkutay wrote: post #52

Earlier quoted context omitted.

> Because Hadoop has not been killed off. > HDFS being replaced with more scalable solutions like S3/EMRFS. Thank you for re-affirming my point. I'm not going to go into the typical HN "I'm going to nitpick your point down to the bone and argue you over the pointless details", but 2012 Hadoop is not the same thing as the tools you're describing.

Except you’re not being just a bit inaccurate, you’re flat wrong. Simply put, Hadoop is not HDFS, it is a much larger Apache ecosystem that includes Spark, Hive, MR, NiFi, etc. All of which are doing fine.

Spark is NOT hadoop. You don't even need hadoop for spark anymore. Spark is more or less completely independent from the hadoop ecosystem.

Just because they're compatible doesn't mean they're synonymous. You can use spark with mesos or kubernetes instead of YARN.

Post reply on HN