Live data from Hacker News

Billions wasted on Hadoop startups, the same will eventually be true of Docker

smashcompany.com

151–160 of 216 posts

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#151
post #139

Earlier quoted context omitted.

Docker famously doesn't isolate very well, as known in infosec circles for years now. If you're unaware, search 'containers don't contain'. MicroVMs start faster and provide better isolation. Meanwhile, none of this is relevant unless you're building your own cloud platform, which is a huge waste of time for most companies. MicroVMs, containers, VMs, zones and bare metal are places to execute code. Serverless makes a…

And given recent revelations, VMs don't contain either! Docker is far from betting on the wrong horse. I can build a docker container and deploy/orchestrate it however I wish - via a docker runtime, kubernetes (which can vary in its underlying implementation), microVMs, VMs, or bare metal. But then the containers don't need to be docker either. Theres many options available there too. Either way, there are many optio…

Sure, having a lot of VM knowledge is also irrelevant unless you're building your own cloud platform too. *

Again, most people don't need to build their own cloud platform.

Being able to reuse Docker files is nice, you've reinvented CFengine for the eighth time. That's great.

The point is Docker is yet another standard way to build boxes and contain them. Serverless / FaaS platforms obviate that need.

* obviously an isolated kernel is better than a shared one, but the point of this post (and this comment) is that Docker vs VM vs MicroVMs vs zones etc is irrelevant for companies who are not cloud providers

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#152

Earlier quoted context omitted.

Relational databases work just fine for terabytes of data. Aparently there are postgres deployments with petabytes of data. There aren't all that many companies with datasets that require anything else. Google and Twitter have the problems Hadoop solves, but it's fair to say that approximately nobody else has those problems.

Whether Postgres (or similar) can handle the data depends not only on the data size but also on what you want to do with the data . I.e., if you are generating reports, running aggregations over a large amount of data you definitely need some parallelism and Postgres isn't designed to handle these loads (certainly not petabytes). Even aggregating 100's of GB probably requires (or at least is more cost effective using…

> I.e., if you are generating reports, running aggregations over a large amount of data you definitely need some parallelism and Postgres isn't designed to handle these loads (certainly not petabytes). Even aggregating 100's of GB probably requires (or at least is more cost effective using) multiple machines.

Aggregating 100s of GB isn't much of a problem for PG these days. Yes, you can be faster - obviously - but it works quite well. And the price for separate systems (duplicated infrastructure, duplicated data, out-of-sync systems, ...) is noticable as well.

But yea, for many petabytes of data you either have to go to an entirely different system, or use something like Citus.

Disclaimer: I work on PG, and I used to work for Citus. So I'm definitely biased.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#153
post #139

I am very much a fan of hot-takes, but this one is trash -- > The money was wasted on hype. The same will eventually be said of Docker. I’ve yet to hear a single benefit attributed to Docker that isn’t also true of other VMs, but standard VMs allow the use of standard operating systems that solved all the hard problems decades ago, whereas Docker is struggling to solve those problems today. Linux containerization (us…

Docker famously doesn't isolate very well, as known in infosec circles for years now. If you're unaware, search 'containers don't contain'. MicroVMs start faster and provide better isolation. Meanwhile, none of this is relevant unless you're building your own cloud platform, which is a huge waste of time for most companies. MicroVMs, containers, VMs, zones and bare metal are places to execute code. Serverless makes a…

Your point is actually the point I'm trying to make -- you shouldn't be using docker to try and contain possibly malicious code, it's not for that.

It does isolate a process's view of the filesystem, it does isolate a process's view of the PID namespace, and that is valuable. Let's say it's a spectrum:

raw processes -------- processes with a certain user ---------- namespace+cgroup isolated processes ---------------- VMs

Docker is certainly an improvement for processes that you don't want running completely contained but want somewhat isolated. For isolation in every sense, you want a VM (a lightweight one if you can get it, i.e. some stripped down qemu).

> MicroVMs start faster and provide better isolation.

Agreed on isolation -- VMs are hands down better at isolating, but I'm a bit skeptical about starting faster.

> Meanwhile, none of this is relevant unless you're building your own cloud platform, which is a huge waste of time for most companies.

Again, this is exactly why it's relevant, 99% of people aren't building their own cloud platform, so they don't need the full isolation of VMs -- most of the time they're just trying to prevent program A that their devs wrote from clobbering program B that their devs also wrote when they both run on the same machine, and making it easier to deploy the dependencies that come with each.

> MicroVMs, containers, VMs, zones and bare metal are places to execute code. Serverless makes all those distinctions irrelevant.

OK I don't even really know what this means, you know serverless runs on MicroVMs right? and most of the time it's actually containers in MicroVMs? MicroVMs are just stripped down versions of regular VMs, and no one is in a zone unless they're running Solaris.

The distinction between these things is still very important, unless you mean that the future is everyone just deploying functions for their applications? But even if you mean that, cold start is basically the first stumbling block you see and it literally exists because of the distinction between how these technologies work (and how fast they can be started on demand, with how much isolation).

> Sorry if you spent 2015 getting really into Docker. You bet on the wrong horse. It's OK, this happens in tech.

Is there anyone that spent 2015 really getting into Docker and isn't better for it now? The technologies that have sprung out of this part of computing are very valuable to know and are getting more valuable, not less. You don't have to install a VM to run an isolated postgres instance on your dev machine because containerization exists -- if you're still doing this you should probably look into updating your tooling.

Also, don't forget that containerization is how some of the richest and supposedly best (due to their ability to spend money on engineers) companies in the world have been handling deployment for nearly a decade -- 2015 is late for realizing containerization is a good thing, not early.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#154

I am very much a fan of hot-takes, but this one is trash -- > The money was wasted on hype. The same will eventually be said of Docker. I’ve yet to hear a single benefit attributed to Docker that isn’t also true of other VMs, but standard VMs allow the use of standard operating systems that solved all the hard problems decades ago, whereas Docker is struggling to solve those problems today. Linux containerization (us…

It’s very easy to underestimate how helpful it can be when you first start working with it. It’s a black box that uses root to do everything and is a pain to debug. Because of this it becomes easy to hand wave it away. Once you have a properly setup project going and your entire build process is mostly repeatable the benefits start becoming more obvious. Yes, you can do all the same things to a certain extent in a vm…

> It’s a black box that uses root to do everything and is a pain to debug.

This is less true these days (and on macs that had to use the docker machine "hack" it was barely ever true, per say) -- rootless containers are on the way thanks to user namespaces. For example LXC can run fully rootless containers that act more like VMs themselves (as in they will have systemd as pid 1 inside) -- kernel support, user namespaces, and File systems in Userspace (FUSE) make this possible.

IMO One of the biggest benefits to running containers is actually E2E tests -- I don't see it done as often as I should, but it has become drastically easier to run an entire postgres instance for a single local E2E test runs -- I do this on almost every project I start now, I set up E2E tests that spin up the actual world I expect in production (so all the backing services, at the version they will run at), and interact with my application -- this is a huge step forward compared to a huge wiki that has "how to set up the local test VM" -- you can spin up and shutdown these services so fast that you can use them ephemerally (I do) without much worry.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#155

Earlier quoted context omitted.

Packaging, in the way it is used in Linux, just needs to be taken out back and shot in the head. If it were any good at all, you wouldn't need so many package managers, package maintainers, repos, etc just to get software. You wouldn't have to worry about updates to your system conflicting with what's already installed, and you wouldn't have to compile things from source or juggle PPAs to get software in a timely man…

You don’t need so many package managers. they exist, but they are virtually equivalent these days. Docker is (almost) alone as a result of timeline. And the ppa juggling is somewhat accidental historical complexity, and some trust management issues. Non-official dockers in widespread use will, I believe, explode as a security nightmare sooner than later.

And nobody cares. Because actually being able to use software, even if it is insecure, trumps not being able to use it.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#156

I am very much a fan of hot-takes, but this one is trash -- > The money was wasted on hype. The same will eventually be said of Docker. I’ve yet to hear a single benefit attributed to Docker that isn’t also true of other VMs, but standard VMs allow the use of standard operating systems that solved all the hard problems decades ago, whereas Docker is struggling to solve those problems today. Linux containerization (us…

So, last time I had a web dev job was back in 2016, so the whole container thing kind of passed me by. So I don't know the details- but, on the other hand, I also don't have any baggage about it being "a VM". But, starting from that kind of "clean slate" state I have to say that if it takes exasperated internet posts, like your helpful comment, to explain why containers are not like VMs and how they are not like VMs.…

Well the thing is, they aren't a big new thing! But the hype train needed it to be (and to be fair, the ecosystem has benefited greatly from the attention).

You rarely see hypetrains for old, established things :)

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#157

Earlier quoted context omitted.

90% of the time Docker is used to solve the problem of "how do I upload this bucket of Python crud to a production server?" (Replace 'Python' with any other language to taste.) A slightly smarter .tar.gz would have solved the problem just as well.

You are confusing Docker = the deployment artifact with Docker = self-contained runtime perfect for CI/CD Longer answer here https://thenewstack.io/docker-based-dynamic-tooling-a-freque...

I'm not.

My point was that Docker purports to solve the sandboxing and security problems.

In reality, this is something that 90% of people who use Docker don't give a shit about. For the vast majority Docker is just a nice and easy-to-use packaging format.

The sad part is that

a) Docker failed at security.

b) In trying to solve the security problem Docker ended up with a pretty crufty (from a technical point of view) packaging format.

Maybe we need to start from scratch, listen to the devs this time and build something they actually want.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#158
post #130

Earlier quoted context omitted.

I'm not going to support the general thesis of this article, but I want to address something you said. You're right that containers are not VMs, but that's only really relevant as pedantry of technical details. I think that what the author was trying to say (without really understanding it) was a comparison of containers to VMs as units of software deployment. I don't think anyone is credibly using containers as a se…

> You're right that containers are not VMs, but that's only really relevant as pedantry of technical details. That isn't pedantry, it is an extremely critical point and the one most people miss when figuring out Docker. Both from a security context (docker doesn't provide vm level promises about isolation) and from a resource management side (docker is really close to no overhead). It is not uncommon to deploy contai…

This is precisely why I said the fact that someone would gloss over this is a red flag. The point is super critical.

VMs are literally so hard to do correctly and in a performant fashion that parts of CPU instruction sets[0], and kernel subsystems (KVM[1]) were created to make them easier to run. Containers, in contrast are literally a few flags and a bunch of in-kernel antics.

A few people, notably Liz Rice and Jessie Frazelle have given talks on how to make containers from scratch that are very illuminating for those that are interested:

https://www.youtube.com/watch?v=HPuvDm8IC-4

https://www.youtube.com/watch?v=cYsVvV1aVss

[0]: https://en.wikipedia.org/wiki/X86_virtualization

[1]: https://en.wikipedia.org/wiki/Kernel-based_Virtual_Machine

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#159
post #139

Earlier quoted context omitted.

Docker famously doesn't isolate very well, as known in infosec circles for years now. If you're unaware, search 'containers don't contain'. MicroVMs start faster and provide better isolation. Meanwhile, none of this is relevant unless you're building your own cloud platform, which is a huge waste of time for most companies. MicroVMs, containers, VMs, zones and bare metal are places to execute code. Serverless makes a…

Your point is actually the point I'm trying to make -- you shouldn't be using docker to try and contain possibly malicious code , it's not for that. It does isolate a process's view of the filesystem, it does isolate a process's view of the PID namespace, and that is valuable. Let's say it's a spectrum: raw processes -------- processes with a certain user ---------- namespace+cgroup isolated processes ---------------…

> you shouldn't be using docker to try and contain possibly malicious code

Indeed. And since we're not building our own cloud environment, because that's a waste of resources for most companies, we will share an environment with possibly malicious code so therefore need isolation.

> unless you mean that the future is everyone just deploying functions for their applications?

Yes, that is exactly what I mean.

> You know serverless runs on MicroVMs right?

Yes, that's why I mentioned them

> and most of the time it's actually containers in MicroVMs?

I doubt this - AWSs performance documentation focuses on MicroVMs as an alternative to containers, not an addition. Which makes sense as containers do less than MicroVMs.

But hey, even if the AWS MicroVM documentation is wrong, it doesn't matter. I am not building a cloud platform. I do not care.

You're right about spin up time. Open a socket and let your apps terminate with it open for a greater chance of reuse. As a FaaS user that's your entire concern with your execution environment.

> 99% of people aren't building their own cloud platform, so they don't need the full isolation of VMs

All cloud environments must provide isolation between customers, hence VMs / MicroVMs. Customers adding docker on top of that add a huge administrative overhead that duplicates the features of their cloud provider for littl benefit,

> Is there anyone that spent 2015 really getting into Docker and isn't better for it now?

Every single person whose product is not Internet infrastructure, that wrote or configured their own unnecessary custom LXC or Docker and VM environment because 'Docker changes everything'.

If a company's product is machine learning for detecting cancer and their ops person has a custom Docker/kubernetes environment they're misusing their employer for their own technical interest

>2015 is late for realizing containerization is a good thing, not early.

It is indeed. It's just that then Docker hype was at its maximum.

Post reply on HN