21–30 of 84 posts
Weren't they killed the moment Spark appeared, having heavily invested in accelerating Hadoop? I am surprised that Cloudera is still around...
Weren't they killed the moment Spark appeared, having heavily invested in accelerating Hadoop? I am surprised that Cloudera is still around...
And spark was sort of killed by Storm / Heron as even Spark Streaming does microbatches and isn't truly realtime.
Weren't they killed the moment Spark appeared, having heavily invested in accelerating Hadoop? I am surprised that Cloudera is still around...
And spark was sort of killed by Storm / Heron as even Spark Streaming does microbatches and isn't truly realtime.
We're seeing a lot of regret around sprawling Hadoop deployments, so this doesn't surprise me. Other Hadoop vendors (vendor?) pivoting to machine learning is a bandaid as the compute capabilities scale beyond HDFS's performance limitations. Look towards new-gen startups around NVME/NVMEOF (WekaIO, Excelero, E8, etc etc) to fill the void. The question is going to be: will anyone provide an intelligent way to maintain…
Earlier quoted context omitted.
Which of the shinier, newer solutions do you think people have found more attractive in particular? Genuinely curious, thanks!
I'm not someone very much in the "big data" [0] but from what I see streaming analytics especially Spark-based solutions are eating the world that used to be dominated by Hadoop. I also think cloud based data lakes and tooling around them seriously decreased the appeal of Hadoop. [0] If I can fit your big data into my memory, it is not "big data".
Earlier quoted context omitted.
And spark was sort of killed by Storm / Heron as even Spark Streaming does microbatches and isn't truly realtime.
Wasn't it the other way around? Wasn't Storm killed by Spark? At least once Spark was out and gained traction, nobody in my bubble talked about Storm anymore. But I guess it depends on the bubble you're in.
Weren't they killed the moment Spark appeared, having heavily invested in accelerating Hadoop? I am surprised that Cloudera is still around...
And spark was sort of killed by Storm / Heron as even Spark Streaming does microbatches and isn't truly realtime.
We're seeing a lot of regret around sprawling Hadoop deployments, so this doesn't surprise me. Other Hadoop vendors (vendor?) pivoting to machine learning is a bandaid as the compute capabilities scale beyond HDFS's performance limitations. Look towards new-gen startups around NVME/NVMEOF (WekaIO, Excelero, E8, etc etc) to fill the void. The question is going to be: will anyone provide an intelligent way to maintain…
fast forward 10 years and hadoop has basically been killed off by hosted storage services that are more expensive but 10000x easier to manage.
Earlier quoted context omitted.
The increasing adoption of object stores and other not-quite-filesystems, even when a real filesystem would be the more appropriate choice, is definitely part of this. For good or ill, people would rather work on or with one of the simplified alternatives. I'm in that camp myself now, working on a system with lame HDFS-like semantics after two decades of proving the "POSIX can't scale" folks wrong. It's just not a ba…
That's why I think MapR missed the mark. They had a software only solution one could deploy on a top of generic hardware in a data center to solve hundreds of TB to small number of PB storage problem that was real for enterprises and since those nodes would also be able to do compute jobs that was the way to get enterprises to adapt that technology. Going to object store was possible but it required dev time and in t…
We're seeing a lot of regret around sprawling Hadoop deployments, so this doesn't surprise me. Other Hadoop vendors (vendor?) pivoting to machine learning is a bandaid as the compute capabilities scale beyond HDFS's performance limitations. Look towards new-gen startups around NVME/NVMEOF (WekaIO, Excelero, E8, etc etc) to fill the void. The question is going to be: will anyone provide an intelligent way to maintain…
Hadoop vendors are simply getting killed by the cloud. AWS EMR, Azure ML, Google Dataproc. And many people are simply swapping HDFS for S3 or similar object stores. Never seen anyone switching to very expensive, more HPC style NVME setups. That seems strange given the data sizes we are working with.
And 95% of what Data Scientists do is ETL, Data Curation, Feature Engineering etc and which Spark is still the overwhelming dominant player. And Spark uses Hadoop filesystem API underneath which is trivial to replace hdfs:// with s3://