31–40 of 84 posts
MapR has been reduced. Hadoop was the floppy disk of big data. Ubiquitous, but always beat by other solutions.
All that is happening is that the vendors are being killed by the cloud.
So, that would mean the hadoop distro market would be down to just cloudera? A single vendor market is not a market, it's a legacy product.
AFAIK, AWS's EMR and Glue leverage Hadoop under the covers. Does that count towards that market? and/or at least keeping Hadoop development active?
DataPipeline is their managed Hadoop for ETL.
Glue ETL is their managed Spark.
And it all absolutely counts towards the market.
Weren't they killed the moment Spark appeared, having heavily invested in accelerating Hadoop? I am surprised that Cloudera is still around...
Why spend $10k per node in an upfront licensing deal when you can buy it from AWS for a few dollars an hour.
Weren't they killed the moment Spark appeared, having heavily invested in accelerating Hadoop? I am surprised that Cloudera is still around...
And spark was sort of killed by Storm / Heron as even Spark Streaming does microbatches and isn't truly realtime.
Spark Streaming is just one part of Spark and the far less used part as well.
Spark hasn't been killed by anything and is still heavily used for ETL and ML.
We're seeing a lot of regret around sprawling Hadoop deployments, so this doesn't surprise me. Other Hadoop vendors (vendor?) pivoting to machine learning is a bandaid as the compute capabilities scale beyond HDFS's performance limitations. Look towards new-gen startups around NVME/NVMEOF (WekaIO, Excelero, E8, etc etc) to fill the void. The question is going to be: will anyone provide an intelligent way to maintain…
Hadoop's whole appeal was it was cheap and scalable. Did it actually work without serious engineering teams maintaining each distribution? absolutely not. the hype was purely VC funded. fast forward 10 years and hadoop has basically been killed off by hosted storage services that are more expensive but 10000x easier to manage.
Then I'm guessing better cloud hosted options came out offering similar capabilities.
If that's the case was it really a big surprise that "cloud" hosting would eat any self-hosted platform's lunch?
We're seeing a lot of regret around sprawling Hadoop deployments, so this doesn't surprise me. Other Hadoop vendors (vendor?) pivoting to machine learning is a bandaid as the compute capabilities scale beyond HDFS's performance limitations. Look towards new-gen startups around NVME/NVMEOF (WekaIO, Excelero, E8, etc etc) to fill the void. The question is going to be: will anyone provide an intelligent way to maintain…
It's at least 4.
Earlier quoted context omitted.
Hadoop's whole appeal was it was cheap and scalable. Did it actually work without serious engineering teams maintaining each distribution? absolutely not. the hype was purely VC funded. fast forward 10 years and hadoop has basically been killed off by hosted storage services that are more expensive but 10000x easier to manage.
I'm not really familiar with this industry. Are these Hadoop clusters hosted locally by these businesses? And MapR et all were providing the software/consulting to help manage them? Then I'm guessing better cloud hosted options came out offering similar capabilities. If that's the case was it really a big surprise that "cloud" hosting would eat any self-hosted platform's lunch?
But of course it didn't play out that way.
We're seeing a lot of regret around sprawling Hadoop deployments, so this doesn't surprise me. Other Hadoop vendors (vendor?) pivoting to machine learning is a bandaid as the compute capabilities scale beyond HDFS's performance limitations. Look towards new-gen startups around NVME/NVMEOF (WekaIO, Excelero, E8, etc etc) to fill the void. The question is going to be: will anyone provide an intelligent way to maintain…
So I deal with hundreds of Data Scientists and none of this is true for us. Hadoop vendors are simply getting killed by the cloud. AWS EMR, Azure ML, Google Dataproc. And many people are simply swapping HDFS for S3 or similar object stores. Never seen anyone switching to very expensive, more HPC style NVME setups. That seems strange given the data sizes we are working with. And 95% of what Data Scientists do is ETL,…
Everyone's data volume is different as well...working a proposal for several petabytes of NVMe storage right now. It's not everyone, but use cases are out there for large volumes of high performance storage.
Fair point on the ability to swap the hdfs interface trivially, I hope it's that simple everywhere. Another issue we run into is that these companies have invested heavily in these Hadoop clusters, and would prefer to continue to get some use out of the mountains of HDDs that are captive in these environments. So tiering/HSM functionality is another facet of the issue that these environments will anchor for quite some time.
Earlier quoted context omitted.
That's why I think MapR missed the mark. They had a software only solution one could deploy on a top of generic hardware in a data center to solve hundreds of TB to small number of PB storage problem that was real for enterprises and since those nodes would also be able to do compute jobs that was the way to get enterprises to adapt that technology. Going to object store was possible but it required dev time and in t…
Interesting post. Out of curiosity: aren't there others who have developed cluster storage exposed as NFS / FUSE?
We're seeing a lot of regret around sprawling Hadoop deployments, so this doesn't surprise me. Other Hadoop vendors (vendor?) pivoting to machine learning is a bandaid as the compute capabilities scale beyond HDFS's performance limitations. Look towards new-gen startups around NVME/NVMEOF (WekaIO, Excelero, E8, etc etc) to fill the void. The question is going to be: will anyone provide an intelligent way to maintain…
Hadoop's whole appeal was it was cheap and scalable. Did it actually work without serious engineering teams maintaining each distribution? absolutely not. the hype was purely VC funded. fast forward 10 years and hadoop has basically been killed off by hosted storage services that are more expensive but 10000x easier to manage.
Because Hadoop has not been killed off. In fact it is far bigger than it has ever been and growing each year. It's just that it is all transitioning to the cloud with parts like HDFS being replaced with more scalable solutions like S3/EMRFS.
But you can't say it's been killed off when on AWS you have managed Hadoop (EMR), managed Hadoop pipelines (DataPipeline) managed Spark (Glue ETL), managed Hive Metastore (Glue Catalog) etc. And similar on Azure or GCP.