Billions wasted on Hadoop startups, the same will eventually be true of Docker • Hacker News Board

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#141

Post by hardwaresofton » Fri, May 31, 2019, 1:14 PM UTC

gerbilly wrote: post #127

hardwaresofton wrote: post #81

I am very much a fan of hot-takes, but this one is trash -- > The money was wasted on hype. The same will eventually be said of Docker. I’ve yet to hear a single benefit attributed to Docker that isn’t also true of other VMs, but standard VMs allow the use of standard operating systems that solved all the hard problems decades ago, whereas Docker is struggling to solve those problems today. Linux containerization (us…

> do you want your process to be isolated, or not. No, not always. Why?. At work I have a few coworkers pushing hard to dockerize (isolate?) everything. This makes debugging when things go wrong a lot harder. I see isolation as one of several qualities a process could have, that sometimes is valuable enough to be worth the sacrifice. Isolation is not some absolute quality that is without significant tradeoffs.

I avoided saying that processes should always be isolated because there are sometimes very good reasons to not isolate a processes with the containerization approach we're talking about, performance being one that came to mind quickly.

Containerization of processes definitely increases complexity but if you can take the time to understand VMs then you can (and should, IMO) take the time to understand how containers work as well, they are lighter and simpler (for example, you don't need to build a kernel or make an initrd). I would argue that people who think VMs are simpler are actually being fooled by huge advancements in tooling over the years and the fact that it's become "easy", not that it was ever simple.

I also want to point out that containers should actually make tracking down some bugs easier, but it does so in a counter-intuitive way -- it removes whole classes of bugs from ever occurring. You'll never have two programs clobber some shared folder or resource, you'll never have programs fight over dependencies, or struggle for locally-bound ports if you're running them in containers.

Containerization definitely represents an increase in complexity, but it is well worth the effort, most of the time, granted you understand the tooling.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#142

Post by kazinator » Fri, May 31, 2019, 1:15 PM UTC

hardwaresofton wrote: post #81

I am very much a fan of hot-takes, but this one is trash -- > The money was wasted on hype. The same will eventually be said of Docker. I’ve yet to hear a single benefit attributed to Docker that isn’t also true of other VMs, but standard VMs allow the use of standard operating systems that solved all the hard problems decades ago, whereas Docker is struggling to solve those problems today. Linux containerization (us…

> they call it a VM (and don't qualify/note that they're being fast and loose with terminology).

It's only confusing to people who are familiar just with the popularized forms of computer science terms.

Isolation and sandboxing is virtualization. In a container, the applications seem to have an operating system and machine to themselves.

A single Unix process and its address space is also a kind of virtual machine, creating the illusion that the process has a machine all to itself. Thanks to virtual memory, other processes are not even visible; they are in a different name space of pointers. That concept breaks for multi-process applications: processes are aware of each other through manipulations of shared resources like files. Or effects like not being able to bind a networking port because some other process is tying it up. The next level of virtualization is to have namespaces for resource-related namespaces in the system beyond the address space. As far as just the filesystem space goes, we can virtualize with tools like chroot. A group of applications can have their own global /etc configuration, their own version of the C library in /lib and so on. That's the beginning of "containerization".

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#143

Post by treis » Fri, May 31, 2019, 1:16 PM UTC

archi42 wrote: post #86

manigandham wrote: post #68

Earlier quoted context omitted.

No it wouldn't have. What's unzipping and running that code? What's monitoring it and restarting it? How do you mount volumes and env variables? How do you open ports and maintain isolation? A container is vastly more powerful for running an application than a tar file.

No offense, but you're aware you make it sound as if we used to use punch cards until the arrival of docker? You can often run daemons as different users and set appropriate file permissions. You can add ENV variables to your start up scripts or configuration files. Volumes are mounted by the system (and you set appropriate access rights again). Monitoring and restarting services is managed by your init system (and p…

>You can often run daemons as different users and set appropriate file permissions. You can add ENV variables to your start up scripts or configuration files. Volumes are mounted by the system (and you set appropriate access rights again).

That sounds exactly like creating a Dockerfile. The difference is that your script has to work any number of times on an endless number of system configurations. The Dockerfile has to work once on one system which is a much easier target to hit. The "any number of times on an endless number of system configurations" is a problem taken care of by the Docker team.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#144

Post by dijksterhuis » Fri, May 31, 2019, 1:18 PM UTC

bdavis__ wrote: post #95

eeZah7Ux wrote: post #52

Earlier quoted context omitted.

> A slightly smarter .tar.gz would have solved the problem just as well. It's called "OS package" ;) and can provide more strict sandboxing using a systemd unit file: unit files provide seccomp, cgroups and more.

docker solves 2 problems. first is you have no control over your devs and allow them to install any software from anywhere. and second is you want to sell cpu time from the cloud in an efficient way (for the seller).

Disagree with both statements.

1) is not a containerisation problem. It’s a team problem. I can jam in a load of npm and pip installs in to a shell install script. Maybe even delete /usr/ for the hell of it. Because the script isn’t isolated from the OS I can cause more damage.

This problem is actually solved by doing code reviews properly and team discussions.

2) errr no. Containers != infrastructure. If you want to deploy on bare metal, you can.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#145

Post by supermatt » Fri, May 31, 2019, 1:20 PM UTC

nailer wrote: post #139

hardwaresofton wrote: post #81

I am very much a fan of hot-takes, but this one is trash -- > The money was wasted on hype. The same will eventually be said of Docker. I’ve yet to hear a single benefit attributed to Docker that isn’t also true of other VMs, but standard VMs allow the use of standard operating systems that solved all the hard problems decades ago, whereas Docker is struggling to solve those problems today. Linux containerization (us…

Docker famously doesn't isolate very well, as known in infosec circles for years now. If you're unaware, search 'containers don't contain'. MicroVMs start faster and provide better isolation. Meanwhile, none of this is relevant unless you're building your own cloud platform, which is a huge waste of time for most companies. MicroVMs, containers, VMs, zones and bare metal are places to execute code. Serverless makes a…

And given recent revelations, VMs don't contain either!

Docker is far from betting on the wrong horse. I can build a docker container and deploy/orchestrate it however I wish - via a docker runtime, kubernetes (which can vary in its underlying implementation), microVMs, VMs, or bare metal.

But then the containers don't need to be docker either. Theres many options available there too.

Either way, there are many options available. And the horse(s) are still in the running.

As for "serverless" vs containers/other, if you can provision accordingly, containers/other can be a better option at scale. Horses for courses, I guess.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#146

Post by yourbandsucks » Fri, May 31, 2019, 1:24 PM UTC

mrep wrote: post #137

yourbandsucks wrote: post #121

Earlier quoted context omitted.

The same concepts are still there, it's just a nicer API now. Think Hadoop -> spark. Map -> map, filter, flatmap, etc Reduce -> reduce, joins, folds, group by, etc Those other concepts were always expressible as map and reduce, of course, just with a bunch of annoying repetitive work

Dataflow looks like it is just their public version of flumejava (both mention pcollections) [0] which indeed is basically a just a bunch of mapreduces pipelined into a fancy directed acyclic graph. Here is the abstract: "MapReduce and similar systems significantly ease the task of writing data-parallel code. However, many real-world computations require a pipeline of MapReduces, and programming and managing such pip…

Yeah, reduce should have really been named groupBy or something. The shiffle-sort to group by key is the important part.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#147

Post by pixelrevision » Fri, May 31, 2019, 1:26 PM UTC

hardwaresofton wrote: post #81

I am very much a fan of hot-takes, but this one is trash -- > The money was wasted on hype. The same will eventually be said of Docker. I’ve yet to hear a single benefit attributed to Docker that isn’t also true of other VMs, but standard VMs allow the use of standard operating systems that solved all the hard problems decades ago, whereas Docker is struggling to solve those problems today. Linux containerization (us…

It’s very easy to underestimate how helpful it can be when you first start working with it. It’s a black box that uses root to do everything and is a pain to debug. Because of this it becomes easy to hand wave it away.

Once you have a properly setup project going and your entire build process is mostly repeatable the benefits start becoming more obvious. Yes, you can do all the same things to a certain extent in a vm but it’s really hard to keep that streamlined and up to date. Having a script that sets up your stack in a vm on both windows and Mac then runs on Linux is a pretty big maintenance nightmare. A dockerfile works with a few commands and can be added to your repo.

It’s not without trade offs but I think if they can solve the issue of debugging in a better way then we’ll really see things solidify on this concept.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#148

Post by dijksterhuis » Fri, May 31, 2019, 1:27 PM UTC

harimau777 wrote: post #135

What is a "Hadoop business"? Hadoop is a tool not an industry or product. Can someone explain what the author might be trying to say?

Businesses that based their SaaS data analysis products on Hadoop.

Preconfigured clusters, integration with your existing AWS deployments. All that sort of jazz.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#149

Post by some_random » Fri, May 31, 2019, 1:28 PM UTC

Seriously, who reads this kind of garbage? It's painfully clear to anyone with any experience with Docker that the author hasn't even skimmed the wikipedia page. Reminds me of the idiots blasting out blog posts about how BITCOIN IS THE FUTURE one month, then BITCOIN IS A SCAM the next.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#150

Post by gerbilly » Fri, May 31, 2019, 1:28 PM UTC

hardwaresofton wrote: post #141

gerbilly wrote: post #127

Earlier quoted context omitted.

> do you want your process to be isolated, or not. No, not always. Why?. At work I have a few coworkers pushing hard to dockerize (isolate?) everything. This makes debugging when things go wrong a lot harder. I see isolation as one of several qualities a process could have, that sometimes is valuable enough to be worth the sacrifice. Isolation is not some absolute quality that is without significant tradeoffs.

I avoided saying that processes should always be isolated because there are sometimes very good reasons to not isolate a processes with the containerization approach we're talking about, performance being one that came to mind quickly. Containerization of processes definitely increases complexity but if you can take the time to understand VMs then you can (and should, IMO) take the time to understand how containers w…

> if you can take the time to understand VMs then you can (and should, IMO) take the time to understand how containers work as well

I don't see it as VMs vs containers.

We have a good devops process to deploy onto our instances, so we rarely have resource clashes you mention (ports/directories) because none of that is ever configured manually. All our infrastructure is derived from 'scripts', so it hasn't been a problem at all.

Aside from python, I see no advantage in containerizing any of our processes at all.

As for debugging, I always forget how infuriating it is, till in the heat of the moment I have to open up a shell into someone's badly made docker image and try to use common tools to help diagnose a problem (ps, nslookup, dig, all) all missing from the wonderful little container.

It's like being on a big navy ship, stranded in the ocean because the engines broke down, but everyone left all the tools back at the base. Yay!