Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Billions wasted on Hadoop startups, the same will eventually be true of Docker

smashcompany.com

Post reply on HN
Print view View original

171–180 of 216 posts

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#171

Post by panpanna »

hardwaresofton wrote: post #154
pixelrevision wrote: post #147

Earlier quoted context omitted.

It’s very easy to underestimate how helpful it can be when you first start working with it. It’s a black box that uses root to do everything and is a pain to debug. Because of this it becomes easy to hand wave it away. Once you have a properly setup project going and your entire build process is mostly repeatable the benefits start becoming more obvious. Yes, you can do all the same things to a certain extent in a vm…

> It’s a black box that uses root to do everything and is a pain to debug. This is less true these days (and on macs that had to use the docker machine "hack" it was barely ever true, per say) -- rootless containers are on the way thanks to user namespaces. For example LXC can run fully rootless containers that act more like VMs themselves (as in they will have systemd as pid 1 inside) -- kernel support, user namespa…

Lxc/lxd is a fantastic piece of software that I wish more people would try. Especially if people already on Ubuntu, Debian and Arch.

You absolutely don't need to be a cloud person to use containers. For example I use lxc on my laptop to keep different projects separate and test new software.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#173

Post by yoz-y »

StreamBright wrote: post #96
otabdeveloper1 wrote: post #50

Earlier quoted context omitted.

90% of the time Docker is used to solve the problem of "how do I upload this bucket of Python crud to a production server?" (Replace 'Python' with any other language to taste.) A slightly smarter .tar.gz would have solved the problem just as well.

Exactly. The said reality that solving this problem helps many developers who would otherwise not capable of deploying to servers.

Much like accessibility, even if you are capable of doing everything by hand, it is usually nicer to have most of the gruntwork handled for you.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#174

Post by nailer »

hardwaresofton wrote: post #167
nailer wrote: post #159

Earlier quoted context omitted.

> you shouldn't be using docker to try and contain possibly malicious code Indeed. And since we're not building our own cloud environment, because that's a waste of resources for most companies, we will share an environment with possibly malicious code so therefore need isolation. > unless you mean that the future is everyone just deploying functions for their applications? Yes, that is exactly what I mean. > You kno…

> Yes, that is exactly what I mean. OK, I'd like to note that it's also the past -- CGI was (and in some dark corners still is) a thing. > Every single person whose product is not Internet infrastructure, that wrote or configured their own unnecessary custom LXC or Docker and VM environment because 'Docker changes everything'. Yeah but those people now have way easier to run local environments? > If a company's produ…

> CGI was (and in some dark corners still is) a thing.

Yep. Tech does that - think about centralisation / distribution every few years, maybe it'll cycle back to people caring about their own containment tech in future. But not right now.

> Yeah but those people now have way easier to run local environments?

OK, so they wasted their time on their awful custom Docker/k8s thing that runs on top of EC2 anyway, and they have a slightly better way to spin up dev environments?

The rest of the conversation is about dev environments, bare metal, VMs and containers all have their place and I mostly agree with you (obviously containers are only useful when there's a Linux kernel on that desktop, Windows and Mac are bare metal or virtualising for the most part).

> What you're saying is that overzealous ops people who are looking to pad their own resumes should not be allowed to run amuck, and I agree with that, but docker is not the poster boy for engineering largess -- and I'd argue it never was.

You understand my point perfectly. I believe that docker is precisely the poster boy for engineering largess, but this is based on my own experiences (talking to a lot of young engineers in the startup world who love wasting investor money on ops) and it seems reasonable that you have had different experiences.

I think we have a good understanding of where we each come from and can end it here. Thanks for being civil.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#175

Post by panpanna »

hardwaresofton wrote: post #158
vorpalhex wrote: post #140

Earlier quoted context omitted.

> You're right that containers are not VMs, but that's only really relevant as pedantry of technical details. That isn't pedantry, it is an extremely critical point and the one most people miss when figuring out Docker. Both from a security context (docker doesn't provide vm level promises about isolation) and from a resource management side (docker is really close to no overhead). It is not uncommon to deploy contai…

This is precisely why I said the fact that someone would gloss over this is a red flag. The point is super critical . VMs are literally so hard to do correctly and in a performant fashion that parts of CPU instruction sets[0], and kernel subsystems (KVM[1]) were created to make them easier to run. Containers, in contrast are literally a few flags and a bunch of in-kernel antics. A few people, notably Liz Rice and Jes…

Containers are "easy" because they are backed by tons of kennel code (cgroups, namespaces and basically a small part of many other subsystems). You can actually create containers from the shell!

VMs are "hard" because you start with nothing save some very low-level help from the hardware.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#176

Post by merb »

archi42 wrote: post #86
manigandham wrote: post #68

Earlier quoted context omitted.

No it wouldn't have. What's unzipping and running that code? What's monitoring it and restarting it? How do you mount volumes and env variables? How do you open ports and maintain isolation? A container is vastly more powerful for running an application than a tar file.

No offense, but you're aware you make it sound as if we used to use punch cards until the arrival of docker? You can often run daemons as different users and set appropriate file permissions. You can add ENV variables to your start up scripts or configuration files. Volumes are mounted by the system (and you set appropriate access rights again). Monitoring and restarting services is managed by your init system (and p…

basically everything you described basically came with systemd. the thing that later made docker possible.

before it was just a mess. and it also isn't that much older than docker.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#177

Post by anarazel »

throwaway082729 wrote: post #170
anarazel wrote: post #152

Earlier quoted context omitted.

> I.e., if you are generating reports, running aggregations over a large amount of data you definitely need some parallelism and Postgres isn't designed to handle these loads (certainly not petabytes). Even aggregating 100's of GB probably requires (or at least is more cost effective using) multiple machines. Aggregating 100s of GB isn't much of a problem for PG these days. Yes, you can be faster - obviously - but it…

I'm genuinely curious. Can PG handle hundreds of users querying 100s of GB, sometimes the same set of tables, at the same time?

Well. You're going to run out of CPU and memory bandwidth pretty quickly. So you'd need replicas to share processing load.

But I honestly don't think hundreds of users each querying 100s of GBs is all that common.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#178

Post by vetinari »

beagle3 wrote: post #126
vetinari wrote: post #23

Earlier quoted context omitted.

Packaged and configured are two different things. The fun begins, when you need several applications, which each have different ideas about configuring services, that would be otherwise shared. Then, as a matter of convenience, you get packages, that either bundle everything preconfigured to their liking (i.e. gitlab omnibus) or packages, that configure shared services how they need it, and running them shared is not…

I mostly Ubuntu these days, have been for 15 years, and I cannot recall a time installing a package like Freeipa disabled a website installed by other packages. Obviously YMMv and I might have just been lucky.

With freeipa I meant server, not client. Installing is not enough, configuring it into usable state is needed too (i.e. running freeipa-server-install or freeipa-replica-install).

It is possible to run other services of such configured httpd, but you need to be careful. When freeipa wanted mod_nss, you had to use mod_nss, not mod_ssl (though they switched it); when freeipa wants to use gssproxy, you are going to use gssproxy too. These changes can happen during upgrades, and it is up to you to fix everything after such change.

The project doesn't recommend to run anything else with the same server; you are free to try though.

The point was, that with docker or another container system, any such problems are irrelevant, and it allows you to have separate service instances without having to run separate VMs.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#179

Post by linuxftw »

YeGoblynQueenne wrote: post #138
hardwaresofton wrote: post #81

I am very much a fan of hot-takes, but this one is trash -- > The money was wasted on hype. The same will eventually be said of Docker. I’ve yet to hear a single benefit attributed to Docker that isn’t also true of other VMs, but standard VMs allow the use of standard operating systems that solved all the hard problems decades ago, whereas Docker is struggling to solve those problems today. Linux containerization (us…

So, last time I had a web dev job was back in 2016, so the whole container thing kind of passed me by. So I don't know the details- but, on the other hand, I also don't have any baggage about it being "a VM". But, starting from that kind of "clean slate" state I have to say that if it takes exasperated internet posts, like your helpful comment, to explain why containers are not like VMs and how they are not like VMs.…

Beach volleyball is played typically with 2's or 4's. The rules are slightly different, court size is smaller. There are no positions in 2's, there are no position faults.

Feel free to make analogies between two other similar sports, such as Competition Karate and Taekwon-Do.

Re: Billions wasted on Hadoop startups, the same will eventually be true of Docker

#180

Post by hardwaresofton »

gerbilly wrote: post #150
hardwaresofton wrote: post #141

Earlier quoted context omitted.

I avoided saying that processes should always be isolated because there are sometimes very good reasons to not isolate a processes with the containerization approach we're talking about, performance being one that came to mind quickly. Containerization of processes definitely increases complexity but if you can take the time to understand VMs then you can (and should, IMO) take the time to understand how containers w…

> if you can take the time to understand VMs then you can (and should, IMO) take the time to understand how containers work as well I don't see it as VMs vs containers. We have a good devops process to deploy onto our instances, so we rarely have resource clashes you mention (ports/directories) because none of that is ever configured manually. All our infrastructure is derived from 'scripts', so it hasn't been a prob…

> I don't see it as VMs vs containers.

It's not? I didn't mean to pit them against each other in competition, I'm saying that if VMs are worth learning about and taking the time time to understand, so are containers. It doesn't have to be zero sum.

> We have a good devops process to deploy onto our instances, so we rarely have resource clashes you mention (ports/directories) because none of that is ever configured manually. All our infrastructure is derived from 'scripts', so it hasn't been a problem at all.

It seems like it was a class of problems that you have fixed with "good devops process". I'd argue that it probably was a problem at once point, and you improved your devops process to make sure it wasn't.

> Aside from python, I see no advantage in containerizing any of our processes at all.

Well I don't know your infrastructure so I'm can't comment on that. I doubt that python is the only thing you run that could benefit from containerization (which again, means limiting access to system resources through namespaces and cgroups), but if you say so then I have no choice but to believe that it's the case.

> As for debugging, I always forget how infuriating it is, till in the heat of the moment I have to open up a shell into someone's badly made docker image and try to use common tools to help diagnose a problem (ps, nslookup, dig, all) all missing from the wonderful little container.

Sounds like you could use some more of that "good devops process" you had when you set up the deploy machinery.

Also, the fact that all of that stuff is missing from the container is actually beneficial from a security point of view -- the same inconvenience you're experiencing is the same inconvenience an intruder would experience first before breaking out of the container (assuming they had the skill to do that). This means that you have another chance to catch them downloading and/or running `ps`/`nslookup`/`bash` or whatever tooling and flag the suspicious behavior. Whether you're in a VM or not, containers are another line of defense, and that's almost certainly a good thing.

Post reply on HN