Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Billions wasted on Hadoop startups, the same will eventually be true of Docker

smashcompany.com

Post reply on HN
Print view View original

181–190 of 216 posts

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#181

Post by geodel »

threeseed wrote: post #26

Cloudera earnt $145m last quarter and grew by 37% over the previous quarter. Other Hadoop startups like Databricks are doing well and Docker has gone from a 2 digit revenue to a 3 digit revenue company from 2017-2018. How has billions been wasted when we have successful companies doing well against the toughest competitors ever i.e. Google, Microsoft and Amazon.

Well that include revenues from Hortonworks also. With revenue rise losses are also rising to $85.5M. I feel in couple of year they will fold up or brought by some big cloud vendor.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#182

Post by geezerjay »

stingraycharles wrote: post #57
geezerjay wrote: post #38

Earlier quoted context omitted.

> Hadoop solved a hard problem that nobody had. This baseless assertion is patently wrong on so many levels. Building computing clusters on COTS hardware is a very mundane problem. Running processing jobs on data shards is a very mundane problem. Scaling COTS clusters transparently is a very mundane problem.

Insert "almost" into parent's sentence and it becomes correct. Many people use/used Hadoop for problems that did not warrant the overhead and complexity that comes with Hadoop. I've seen it countless times with my own eyes that people pre-emptively use tools like Hadoop and Spark because of a chance that they will hit a massive scale in the future. This happens in both startups and enterprises alike: people like to t…

> Insert "almost" into parent's sentence and it becomes correct.

No, it still remains astonishingly wrong. Even container orchestration platforms are being adapted to provide the same service that Hadoop has been providing for years, and no one in their right mind would claim that running processing jobs on the cloud is a problem that almost no one has.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#183

Post by hardwaresofton »

panpanna wrote: post #175
hardwaresofton wrote: post #158

Earlier quoted context omitted.

This is precisely why I said the fact that someone would gloss over this is a red flag. The point is super critical . VMs are literally so hard to do correctly and in a performant fashion that parts of CPU instruction sets[0], and kernel subsystems (KVM[1]) were created to make them easier to run. Containers, in contrast are literally a few flags and a bunch of in-kernel antics. A few people, notably Liz Rice and Jes…

Containers are "easy" because they are backed by tons of kennel code (cgroups, namespaces and basically a small part of many other subsystems). You can actually create containers from the shell! VMs are "hard" because you start with nothing save some very low-level help from the hardware.

Also, I want to be clear that I'm using "simple" and "easy" in the rick hickey sense of the words, as in "simple" has more to do with what a thing is made of, and "easy" has more to do with ease of use, taking available tooling/familiarity and context into mind.

What I should have made clearer was that I think containers are both easier and simpler than VMs. There are both less moving parts (I haven't looked, but I assume less code), and containers are easier to get started with than VMs (set some flags on some syscalls versus make sure you buy the right CPU).

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#184

Post by hardwaresofton »

panpanna wrote: post #171
hardwaresofton wrote: post #154

Earlier quoted context omitted.

> It’s a black box that uses root to do everything and is a pain to debug. This is less true these days (and on macs that had to use the docker machine "hack" it was barely ever true, per say) -- rootless containers are on the way thanks to user namespaces. For example LXC can run fully rootless containers that act more like VMs themselves (as in they will have systemd as pid 1 inside) -- kernel support, user namespa…

Lxc/lxd is a fantastic piece of software that I wish more people would try. Especially if people already on Ubuntu, Debian and Arch. You absolutely don't need to be a cloud person to use containers. For example I use lxc on my laptop to keep different projects separate and test new software.

I agree -- it also offered more isolation (user namespaces) from very early on and is way more featureful (live migration has been a thing in LXC for a while IIRC).

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#185

Post by vkazanov »

notacoward wrote: post #108
vkazanov wrote: post #31

Earlier quoted context omitted.

I'll correct you: Hadoop solves a hard problem that lots of big enterprises have. A distributed file system is a big deal, and everything that on top also is. Now making (big)money with its ecosystem is another question.

...except that HDFS has never been a distributed filesystem, despite the name. It's an object store with just enough hackery so that a particular set of apps with very limited needs could be modified in a short time to use it instead of a filesystem. Arguably that's still a big deal, but nowhere near as impressive or generally useful as a distributed filesystem. To me it's a bit amazing that anyone involved in its de…

...ok, then what are the alternatives? And why is it so bad?

Something that can handle hundreds of terabytes on hundreds of machines and provides useful tools on top of the whole thing (Spark, Hive, etc)?

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#186

Post by panpanna »

hardwaresofton wrote: post #183
panpanna wrote: post #175

Earlier quoted context omitted.

Containers are "easy" because they are backed by tons of kennel code (cgroups, namespaces and basically a small part of many other subsystems). You can actually create containers from the shell! VMs are "hard" because you start with nothing save some very low-level help from the hardware.

Also, I want to be clear that I'm using "simple" and "easy" in the rick hickey sense of the words, as in "simple" has more to do with what a thing is made of, and "easy" has more to do with ease of use, taking available tooling/familiarity and context into mind. What I should have made clearer was that I think containers are both easier and simpler than VMs. There are both less moving parts (I haven't looked, but I a…

Fair enough, but don't forget there are some extremely simple hypervisor implementations out there too.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#187

Post by StreamBright »

yoz-y wrote: post #173
StreamBright wrote: post #96

Earlier quoted context omitted.

Exactly. The said reality that solving this problem helps many developers who would otherwise not capable of deploying to servers.

Much like accessibility, even if you are capable of doing everything by hand, it is usually nicer to have most of the gruntwork handled for you.

Well, not necessarily by hand. I never gave up deploying our Java application with Ansible. We could have used Docker but the team decided to use fat jars and Ansible instead. Nowadays with Java 11 you can make those fat jars even slimmer. There was no value proposition for us to change.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#188

Post by gerbilly »

hardwaresofton wrote: post #180
gerbilly wrote: post #150

Earlier quoted context omitted.

> if you can take the time to understand VMs then you can (and should, IMO) take the time to understand how containers work as well I don't see it as VMs vs containers. We have a good devops process to deploy onto our instances, so we rarely have resource clashes you mention (ports/directories) because none of that is ever configured manually. All our infrastructure is derived from 'scripts', so it hasn't been a prob…

> I don't see it as VMs vs containers. It's not? I didn't mean to pit them against each other in competition, I'm saying that if VMs are worth learning about and taking the time time to understand, so are containers. It doesn't have to be zero sum. > We have a good devops process to deploy onto our instances, so we rarely have resource clashes you mention (ports/directories) because none of that is ever configured ma…

> It seems like it was a class of problems that you have fixed with "good devops process". I'd argue that it probably was a problem at once point, and you improved your devops process to make sure it wasn't.

It certainly was but we fixed it and it's not a problem anymore.

> Sounds like you could use some more of that "good devops process" you had when you set up the deploy machinery.

Yeah, there are people within my group that want to 'modernize' things and put them into containers willy nilly for no real reason.

We have already solved all the difficult problems that containers are supposed to 'save' us from. Many of the proposed containers would just be a single statically linked binary with a config file.

Why?

FYI, our stuff is hosted internally, so security considerations are not such a big deal.

To hear these container advocates, you'd think that till they came around no one ever managed to use linux.

I'm fully expecting linux userland tools to go away, to be replaced by custom 'distributions' with only a kernel and a docker API soon.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#190

Post by notacoward »

vkazanov wrote: post #185
notacoward wrote: post #108

Earlier quoted context omitted.

...except that HDFS has never been a distributed filesystem, despite the name. It's an object store with just enough hackery so that a particular set of apps with very limited needs could be modified in a short time to use it instead of a filesystem. Arguably that's still a big deal, but nowhere near as impressive or generally useful as a distributed filesystem. To me it's a bit amazing that anyone involved in its de…

...ok, then what are the alternatives? And why is it so bad? Something that can handle hundreds of terabytes on hundreds of machines and provides useful tools on top of the whole thing (Spark, Hive, etc)?

Well, rather topically, there's MapR. Depending on the desired scale and level of fidelity to real-filesystem semantics, one might also count Gluster, Ceph, and Lustre. Don't know if PVFS is still around. There are more proprietary offerings from IBM (GPFS/SpectrumScale), EMC, NetApp, etc. plus a plethora of startups. There are other object stores that don't make a pretense of being filesystems. Alternatives abound.

BTW, "hundreds of terabytes on hundreds of machines" isn't interesting territory any more. Most people's needs are far smaller, so HDFS isn't much help. Those who need more generally need much more, so HDFS isn't much help again. Richer semantics are nice either way. Imagine thousands of machines with dozens of terabytes each and you might start to see the problems with HDFS's design (though you'll still be far short of the domain I work in).

Post reply on HN