Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Billions wasted on Hadoop startups, the same will eventually be true of Docker

smashcompany.com

Post reply on HN
Print view View original

121–130 of 216 posts

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#121

Post by yourbandsucks »

StreamBright wrote: post #94
misterman0 wrote: post #91

Earlier quoted context omitted.

>> Google stopped using MapReduce How are they then querying over big data these days? We don't know, do we? Or did they open-source their search engine? By using Hadoop people are trying to not reinvent the big data wheel, partly because it's a motherfucker of a problem to have to solve and party because they want to solve the business problem, not the technical one. I don't see how that is in any way worthy of bein…

Maybe you and the downvoter crew do not know it. https://www.datacenterknowledge.com/archives/2014/06/25/goog... https://www.quora.com/Why-did-Google-stop-using-MapReduce-an...

The same concepts are still there, it's just a nicer API now. Think Hadoop -> spark.

Map -> map, filter, flatmap, etc

Reduce -> reduce, joins, folds, group by, etc

Those other concepts were always expressible as map and reduce, of course, just with a bunch of annoying repetitive work

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#123

Post by ransom1538 »

hardwaresofton wrote: post #81

I am very much a fan of hot-takes, but this one is trash -- > The money was wasted on hype. The same will eventually be said of Docker. I’ve yet to hear a single benefit attributed to Docker that isn’t also true of other VMs, but standard VMs allow the use of standard operating systems that solved all the hard problems decades ago, whereas Docker is struggling to solve those problems today. Linux containerization (us…

"All this said -- I do think Docker will die"

Harsh. Docker is a nice way to specify how you want your container to work. Sure release Bocker (A better Docker). But Bocker will just basically be Docker: a simple set of instructions to get a machine running.

That is all we want. A few instructions to get a system up. Devs are sick of setting up machines. Demand is there.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#124

Post by beagle3 »

AnIdiotOnTheNet wrote: post #106
beagle3 wrote: post #16

Earlier quoted context omitted.

Properly packaged Debian/raspbian apps are still an “apt install” away. Your use case, which is common, tells me that packaging / distributing may need some love, not that there’s a fundamental difference. And the convenience does not come free - a random docker is almost as bad as a random executable.

Packaging, in the way it is used in Linux, just needs to be taken out back and shot in the head. If it were any good at all, you wouldn't need so many package managers, package maintainers, repos, etc just to get software. You wouldn't have to worry about updates to your system conflicting with what's already installed, and you wouldn't have to compile things from source or juggle PPAs to get software in a timely man…

You don’t need so many package managers. they exist, but they are virtually equivalent these days. Docker is (almost) alone as a result of timeline.

And the ppa juggling is somewhat accidental historical complexity, and some trust management issues.

Non-official dockers in widespread use will, I believe, explode as a security nightmare sooner than later.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#125

Post by stult »

pytester wrote: post #39
sz4kerto wrote: post #29

Earlier quoted context omitted.

The difference is standardization. Postgres can be installed in subdirectories nicely, as you say -- if you know how. Same is true for JBoss, CI build agent, whatever. Now if you have dozens of these apps then e.g. onboarding people in a remote team suddenly becomes non-trivial (a nightmare, to be precise). With Docker, they can get a complex system running in an hour. The two primary use cases for Docker is, as far…

>The difference is standardization. Postgres can be installed in subdirectories nicely, as you say -- if you know how. Or if you have, y'know, a really simple script. ...the kind which also runs inside most semi-complex Docker containers anyway.

I've very rarely seen setup scripts that don't have implicit dependencies on the details of the host environment. Whether it's assumptions about the OS or file system or what related software may be installed or whatever. Often because the original developer can't predict every permutation of the possible interactions because of a combinatorial explosion of possible system setups, sometimes because the script is poorly written, and very often because the language or tooling itself is poorly isolated (eg, pip and npm). Docker is a lightweight way of guaranteeing deterministic setup script execution. Obviously you can still screw it up by pulling from unversioned base images etc, but the numbers of failure modes are limited, typically easy to locate (ie within a single dockerfile rather than across an entire OS), and relatively easy to prevent with some best practices. And best of all, that's true across languages. I can achieve the same effects with pipenv, yarn, or other tooling specific techniques like folder level postgres installs. But then as a developer I have to know the thousands of idiosyncratic pitfalls that occur across the plethora of tools I have to deal with every day. And realistically, I have to deal with a ton of poorly written applications and scripts that I want to execute with some degree of isolation without a ton of overhead. More than I could possibly ever fix.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#126

Post by beagle3 »

vetinari wrote: post #23
beagle3 wrote: post #16

Earlier quoted context omitted.

Properly packaged Debian/raspbian apps are still an “apt install” away. Your use case, which is common, tells me that packaging / distributing may need some love, not that there’s a fundamental difference. And the convenience does not come free - a random docker is almost as bad as a random executable.

Packaged and configured are two different things. The fun begins, when you need several applications, which each have different ideas about configuring services, that would be otherwise shared. Then, as a matter of convenience, you get packages, that either bundle everything preconfigured to their liking (i.e. gitlab omnibus) or packages, that configure shared services how they need it, and running them shared is not…

I mostly Ubuntu these days, have been for 15 years, and I cannot recall a time installing a package like Freeipa disabled a website installed by other packages. Obviously YMMv and I might have just been lucky.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#127

Post by gerbilly »

hardwaresofton wrote: post #81

I am very much a fan of hot-takes, but this one is trash -- > The money was wasted on hype. The same will eventually be said of Docker. I’ve yet to hear a single benefit attributed to Docker that isn’t also true of other VMs, but standard VMs allow the use of standard operating systems that solved all the hard problems decades ago, whereas Docker is struggling to solve those problems today. Linux containerization (us…

> do you want your process to be isolated, or not.

No, not always. Why?.

At work I have a few coworkers pushing hard to dockerize (isolate?) everything.

This makes debugging when things go wrong a lot harder.

I see isolation as one of several qualities a process could have, that sometimes is valuable enough to be worth the sacrifice.

Isolation is not some absolute quality that is without significant tradeoffs.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#128

Post by beagle3 »

heavenlyblue wrote: post #19
beagle3 wrote: post #16

Earlier quoted context omitted.

Properly packaged Debian/raspbian apps are still an “apt install” away. Your use case, which is common, tells me that packaging / distributing may need some love, not that there’s a fundamental difference. And the convenience does not come free - a random docker is almost as bad as a random executable.

>> Properly packaged >> packaging / distributing may need some love I am yet to see a single project which is relatively new and is packaged on Debian (let alone Raspbian). And lets be honest - nobody is genuinely using Raspberry Pi to run an outdated media server.

Debian unstable (and even testing) has many. Ubuntu PPAs do too.

Raspbian might not, I don’t follow it.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#129

Post by hardwaresofton »

ransom1538 wrote: post #123
hardwaresofton wrote: post #81

I am very much a fan of hot-takes, but this one is trash -- > The money was wasted on hype. The same will eventually be said of Docker. I’ve yet to hear a single benefit attributed to Docker that isn’t also true of other VMs, but standard VMs allow the use of standard operating systems that solved all the hard problems decades ago, whereas Docker is struggling to solve those problems today. Linux containerization (us…

"All this said -- I do think Docker will die" Harsh. Docker is a nice way to specify how you want your container to work. Sure release Bocker (A better Docker). But Bocker will just basically be Docker: a simple set of instructions to get a machine running. That is all we want. A few instructions to get a system up. Devs are sick of setting up machines. Demand is there.

I agree the way I said it was harsh -- I don't mean Docker the company should die or Docker the project should die or anything of that nature, I mean literally people referring to everything in containerization as "docker" + .

Also, Docker itself is very much trying to be more than what it is right now, Docker swarm, compose, and basically everything Docker as a company does to improve their product offering moves them farther away from the way people are using "docker" in conversation right now.

Basically, when people say "docker image" they should be saying "container image" (for lack of a better term, at least).

> That is all we want. A few instructions to get a system up. Devs are sick of setting up machines. Demand is there.

This is pretty vague -- "setting up machines" can mean a lot of things. I want to make it clear that it's not the container runtime's job to set up a single machine, it's to enable a process to pretend that the machine it has access to is it's own. It's a subtle difference but it's worth noting IMO.

If you want to fix setting up your machines, you should be looking at tools like Packer, Container Linux distributions (formerly CoreOS, now Atomic I believe), Nix, Guix, and Linuxkit. Docker does not set up machines, it sets up processes, but happens to bring along a filesystem (amongst other things).

A linux container is not a machine (virtualized or otherwise), it is an isolated process, I agree that devs are sick of worrying about machine configuration when setting up the required dependencies to run their processes, though, containerization is here to stay -- it was around before docker and will be here after.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#130

Post by cmsj »

hardwaresofton wrote: post #81

I am very much a fan of hot-takes, but this one is trash -- > The money was wasted on hype. The same will eventually be said of Docker. I’ve yet to hear a single benefit attributed to Docker that isn’t also true of other VMs, but standard VMs allow the use of standard operating systems that solved all the hard problems decades ago, whereas Docker is struggling to solve those problems today. Linux containerization (us…

I'm not going to support the general thesis of this article, but I want to address something you said.

You're right that containers are not VMs, but that's only really relevant as pedantry of technical details.

I think that what the author was trying to say (without really understanding it) was a comparison of containers to VMs as units of software deployment.

I don't think anyone is credibly using containers as a security measure on Linux, because if they think they are, they are in for several large surprises.

Rather, we're seeing the unbundling of software - it used to be that you deployed software to a physical machine with a full OS, then you could deploy it to a virtual machine with a full OS, then you could deploy the process, its dependencies and a minimal OS into a container.

I agree that Docker doesn't have a huge and profitable future ahead of it, because it's providing commodity infrastructure. Rather I think it's interesting to think about what the next level of software deployment decomposition will be, and I'd wager that it's FaaS (ie serverless).

Post reply on HN