Interview with Eric Brewer • Hacker News Board

Re: Interview with Eric Brewer

#51

Post by jacquesm » Tue, May 19, 2015, 1:06 AM UTC

bcantrill wrote: post #42

jacquesm wrote: post #9

Earlier quoted context omitted.

I'd like a real explanation for why containers and unikernels are better than regular run-of-the-mill applications running on dedicated servers. It's almost as if the wild west of the web isn't quite enough and we now need to add another explosion of layers-of-abstraction but this time on the server in order to pretend we have infinite hardware which then becomes it's own reason for existence rather than to simply ru…

Containers have actually been around for a long, long time -- and have well-known operational efficiencies. So this isn't a "shiny new layer-of-abstraction", it's a tried-and-true abstraction that has been operating in production and at scale for a the better part of the last decade.[1] That said, the developer fascination with containers (which is to say, Docker) is new, and there is a bit of a wild west of abstract…

How do you see this delivering on the security component of the isolation? (Not that VMs are perfect in this respect but it seems to me that containers are a lot less solid)

Re: Interview with Eric Brewer

#52

Post by jacquesm » Tue, May 19, 2015, 1:14 AM UTC

mbesto wrote: post #50

jacquesm wrote: post #9

Earlier quoted context omitted.

I'd like a real explanation for why containers and unikernels are better than regular run-of-the-mill applications running on dedicated servers. It's almost as if the wild west of the web isn't quite enough and we now need to add another explosion of layers-of-abstraction but this time on the server in order to pretend we have infinite hardware which then becomes it's own reason for existence rather than to simply ru…

Do you have experience using containers? Here's my answer for "why": DRY. Once you've deployed hundreds of servers using the same exact Ubuntu 12.04 LTS kernel base, why not just completely abstract the OS away and focus the attention on scaling the OS services that matter? Why is that when I decide that I need to scale out, I need to copy every library of the OS and every line of code for the kernel and redeploy it…

> why not just completely abstract the OS away and focus the attention on scaling the OS services that matter?

Because it adds a layer that makes no sense unless you have very specific use cases. Though I see the point regarding people efficiency, that one makes good sense (see other comment in this sub-thread)

> Why is that when I decide that I need to scale out, I need to copy every library of the OS and every line of code for the kernel and redeploy it every time I add a node?

If you're doing it that way then you are simply doing it wrong. See: chef, configuration management and various deployment services (of which you could argue containers are one off-shoot, but they focus (imo) on the wrong level for all but the largest companies). Containers are like sandboxes with significant overhead for applications that focus on ease of deployment (but that's strange to me because I see that as a one-time cost for most of my own use cases, though I can see how that equation would change if you deploy lots of things configured by lots of different people to a single set of servers, especially if there are conflicting requirements between those deployments).

> Care to elaborate on the downside of such a promise?

That's my personal view of hell, if you don't see any downside there please ignore my vision and continue as if nothing was said.

From open, text based standards to shipping arbitrary binaries in a couple of decades. And I thought GKS was about as bad as it got ;)

Re: Interview with Eric Brewer

#53

Post by btilly » Tue, May 19, 2015, 1:14 AM UTC

menage wrote: post #31

jacquesm wrote: post #17

Earlier quoted context omitted.

> Primarily because dedicated servers are a lot less efficient Assuming you're operating at scale I don't see why that would be the case. And if you're not, what's the point? > The more different things you can pack on a machine while still ensuring that the high-priority/low-latency jobs get prompt access to the resources that they've reserved, the higher overall utilization you can achieve (and hence bring costs do…

Yes, I think it's safe to say that Google operates at scale. A lot of user-facing services at Google have to be over-provisioned in order to handle the cyclical usage patterns (the daily query peak is far higher than the average for most services) and to be able to survive the loss of a datacenter or two. This results in a lot of under-utilized servers for a big fraction of the time. So by packing lots of medium and…

Borg containers consisted generally of a single process...

Really?

My impression was that typically you'd have a process for the service, a borgmon process for monitoring, and maybe another process to ship logs off in the background.

Developers would only think about the service process (which itself typically was a fairly thin shim in front of other services), but a borg container would have more than that going on in it.

Re: Interview with Eric Brewer

#54

Post by bostonpete » Tue, May 19, 2015, 1:21 AM UTC

jedberg wrote: post #29

A funny and somewhat off topic story -- back in 2003, before the Google IPO, Google was doing a recruiting event at Berkeley. They brought a few of their folks with them: their founder Larry, one of their female engineers, Marissa, and some others. They did a little talk, and during the Q&A, professor Brewer told Larry that there was an opening in the PhD program and he was welcome to it. Larry politely declined. Aft…

In 2003, Google had over a billion dollars in revenue. I suspect that answer was tongue in cheek.

Re: Interview with Eric Brewer

#55

Post by DannoHung » Tue, May 19, 2015, 1:26 AM UTC

jacquesm wrote: post #37

DannoHung wrote: post #32

Earlier quoted context omitted.

Isn't this only true if your container build process pulls in the same version of a library in multiple different virutal filesystems? That is, if you are using the same base image for a number of applications and the libraries are installed in the base image rather than the image the application resides, in the kernel should recognize the shared libraries being used as coming from the same place and be able to perfo…

Presumably you could arrange things in such a way that several container images shared libraries and such but that would likely interfere with the (desirable) isolation properties and versioning will play havoc with that anyway (since all dependencies are part-and-parcel of a container and nothing stops multiple containers from shipping different versions of the same package). Where regular virtualization runs multip…

Seems like you'd have to construct a pretty weird situation in order to blow up your cache, particularly if your load is enough that you can max out a server. Like, if you're running a lot of instances of the same app, deduplication should work fine, right? It only would show up if you've built a bunch of different applications that use the same libraries and consume roughly similar amounts of CPU; if you're running multiple copies of the same application, the deduplication should work just fine.

And that's assuming that it'd work exactly the way you're thinking.

I feel like the win over running VMs (which incur something like a 12% overhead compared to both Docker and running right on the machine for a single application), plus flexibility, plus ease of deployment is worthwhile. I mean, the current situation is running VM images anyway, right? This is a step in the right direction over that, even you must admit.

Re: Interview with Eric Brewer

#56

Post by btilly » Tue, May 19, 2015, 1:29 AM UTC

jacquesm wrote: post #9

MCRed wrote: post #5

I'd like a real explanation for why containers are better than unikernels. Yes, unikernals are still early, and containers are convenient, because you have all of linux there... but it seems that running several linuxes on a linux machine is a bit much. One operating system plus XEN plus several applications in unikernels seems more efficient, and more exciting. But it's the less common choice. I am guessing convenie…

I'd like a real explanation for why containers and unikernels are better than regular run-of-the-mill applications running on dedicated servers. It's almost as if the wild west of the web isn't quite enough and we now need to add another explosion of layers-of-abstraction but this time on the server in order to pretend we have infinite hardware which then becomes it's own reason for existence rather than to simply ru…

Do you want to design for failover of a complex system? Then containers are your friend.

Every dedicated server that you set up has an opportunity to not be duplicated perfectly. Every system that knows about your dedicated server has an opportunity to hard code what it shouldn't. Which makes these a potential point of failure. Add enough of those, and you're statistically guaranteed that the careful architecture that you have for failover is a pipe dream.

If everything is deployed with containers and discovery and the correct provisioning, then dealing with the fact that containers move around forces you to solve all of your other problems. And containers provide an abstraction layer that makes the rest of it straightforward.

Let me illustrate with an example.

When I worked at Google in 2010, I remember reading an article from eBay about how they finally manage to transition everything off of a running data center, without interrupting live traffic, and how much planning it took them. And they were congratulating themselves on what a heroic feat they had managed. Most companies today would still consider that a pretty amazing feat, and would find that challenging.

At the same point time I was learning how things were set up at Google so that you could drop any data center at random with barely any interruption of live traffic, and with no manual intervention required. And Google occasionally does this without warning to important data centers just to be sure that it works.

Re: Interview with Eric Brewer

#57

Post by zobzu » Tue, May 19, 2015, 1:29 AM UTC

nickpsecurity wrote: post #49

One thing that bothers me about the article is that it shows a recurring problem: IT not knowing what it knows. The NoSQL movement didn't notice that NonStop Architecture scaled linearly to thousands of cores with strong-consistency, five 9's, and SQL support. In the mid-80's. Instead of making a low-cost knockoff, like cluster movement did for NUMA's, they ditched consistency altogether and launched NoSQL movement.…

thats a pretty good comment actually. there's quite a bit of similarity.

we generally tend to jump into these as "omg awesome new tech" with a very narrow view. But it also helps boosting more though-out techs (even thus it feels less efficient to go through that route first, its perhaps the only route that works with human: try, fail, try again, etc.)

Re: Interview with Eric Brewer

#58

Post by zobzu » Tue, May 19, 2015, 1:31 AM UTC

sp332 wrote: post #34

MCRed wrote: post #5

I'd like a real explanation for why containers are better than unikernels. Yes, unikernals are still early, and containers are convenient, because you have all of linux there... but it seems that running several linuxes on a linux machine is a bit much. One operating system plus XEN plus several applications in unikernels seems more efficient, and more exciting. But it's the less common choice. I am guessing convenie…

If you have a statically-linked binary you can just run it in a container. You don't need a whole OS userland in there.

[...] since the whole userland that you need is included in the binary (worth noting)

you just dont get the debugging stuff (which is okay as long as you can choose)

Re: Interview with Eric Brewer

#59

Post by alextgordon » Tue, May 19, 2015, 1:32 AM UTC

jacquesm wrote: post #52

mbesto wrote: post #50

Earlier quoted context omitted.

Do you have experience using containers? Here's my answer for "why": DRY. Once you've deployed hundreds of servers using the same exact Ubuntu 12.04 LTS kernel base, why not just completely abstract the OS away and focus the attention on scaling the OS services that matter? Why is that when I decide that I need to scale out, I need to copy every library of the OS and every line of code for the kernel and redeploy it…

> why not just completely abstract the OS away and focus the attention on scaling the OS services that matter? Because it adds a layer that makes no sense unless you have very specific use cases. Though I see the point regarding people efficiency, that one makes good sense (see other comment in this sub-thread) > Why is that when I decide that I need to scale out, I need to copy every library of the OS and every line…

Containers are simply a generalisation of python virtualenvs or ruby rbenvs, to encapsulate the entire environment. If you asked a Python programmer "well, why don't you just buy a new laptop for each project?" you might get strange looks.

Re: Interview with Eric Brewer

#60

Post by jacquesm » Tue, May 19, 2015, 1:34 AM UTC

DannoHung wrote: post #55

jacquesm wrote: post #37

Earlier quoted context omitted.

Presumably you could arrange things in such a way that several container images shared libraries and such but that would likely interfere with the (desirable) isolation properties and versioning will play havoc with that anyway (since all dependencies are part-and-parcel of a container and nothing stops multiple containers from shipping different versions of the same package). Where regular virtualization runs multip…

Seems like you'd have to construct a pretty weird situation in order to blow up your cache, particularly if your load is enough that you can max out a server. Like, if you're running a lot of instances of the same app, deduplication should work fine, right? It only would show up if you've built a bunch of different applications that use the same libraries and consume roughly similar amounts of CPU; if you're running…

I wished VMs would only incur a 12% overhead (that's assuming absolutely optimal configuration and a fairly static load), it can be substantially more than that, especially when people go 'enterprise' on you for what would otherwise be a relatively simple setup.

But you've made me curious enough that I'll do some benchmarks to see how virtualization compares to present day containers for practical use cases faced by mid-size and small companies, my fooling around with this about a year ago led to nothing but frustration, it's always a risk to argue from data older than a few months in a field moving this fast and more measurements are the preferred way to settle stuff like this anyway.