Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Interview with Eric Brewer

medium.com

Post reply on HN
Print view View original

31–40 of 142 posts

Re: Interview with Eric Brewer

#31

Post by menage »

jacquesm wrote: post #17
menage wrote: post #12

Earlier quoted context omitted.

Primarily because dedicated servers are a lot less efficient. The more different things you can pack on a machine while still ensuring that the high-priority/low-latency jobs get prompt access to the resources that they've reserved, the higher overall utilization you can achieve (and hence bring costs down), At least with the Borg containers (I'm not familiar with how Omega and Kubernetes do things), there wasn't any…

> Primarily because dedicated servers are a lot less efficient Assuming you're operating at scale I don't see why that would be the case. And if you're not, what's the point? > The more different things you can pack on a machine while still ensuring that the high-priority/low-latency jobs get prompt access to the resources that they've reserved, the higher overall utilization you can achieve (and hence bring costs do…

Yes, I think it's safe to say that Google operates at scale.

A lot of user-facing services at Google have to be over-provisioned in order to handle the cyclical usage patterns (the daily query peak is far higher than the average for most services) and to be able to survive the loss of a datacenter or two. This results in a lot of under-utilized servers for a big fraction of the time. So by packing lots of medium and low priority jobs on those same servers (and over-committing the resources on the server), you can soak up the slack resources; in the event that the resources are needed by the user-facing service the kernel containers ensure that the all the less latency-sensitive jobs on the machine don't compete for resources with the user-facing services.

It's true that the performance isolation when there are tens of jobs running on the same machine isn't going to completely match the performance isolation of running a service on a dedicated server, even with kernel resource isolation via containers, but you have to make cost trade-offs somewhere. The number of Borg services that could justify requesting dedicated machines was very small.

And to address your other concern about the OS not having so much insight into what's going on - Borg containers consisted generally of a single process, running on the machine's normal kernel. The containerization was just for in-kernel resource accounting/isolation. (Using Linux control groups, rather than anything fancier like LXC or Xen)

Re: Interview with Eric Brewer

#32

Post by DannoHung »

jacquesm wrote: post #27
DannoHung wrote: post #20

Earlier quoted context omitted.

Do you know where I can read more about this? This is the first I have heard that processes running inside containers interfere with the kernel's scheduler. Or, sorry again, but maybe I'm not following your meaning in some way?

Each 'container' is in effect an application plus all its dependencies. So if one or more active containers could share resources then they won't, which leads to inefficiencies because you'll be running a much larger number of processes than you would otherwise (because of duplication) requiring a larger memory footprint and probably less efficient cache and/or IO utilization. The deployment of the apps will be easie…

Isn't this only true if your container build process pulls in the same version of a library in multiple different virutal filesystems? That is, if you are using the same base image for a number of applications and the libraries are installed in the base image rather than the image the application resides, in the kernel should recognize the shared libraries being used as coming from the same place and be able to perform deduplication as normal?

Re: Interview with Eric Brewer

#33

Post by menage »

jacquesm wrote: post #27
DannoHung wrote: post #20

Earlier quoted context omitted.

Do you know where I can read more about this? This is the first I have heard that processes running inside containers interfere with the kernel's scheduler. Or, sorry again, but maybe I'm not following your meaning in some way?

Each 'container' is in effect an application plus all its dependencies. So if one or more active containers could share resources then they won't, which leads to inefficiencies because you'll be running a much larger number of processes than you would otherwise (because of duplication) requiring a larger memory footprint and probably less efficient cache and/or IO utilization. The deployment of the apps will be easie…

Most Borg containers were in fact single processes (or more correctly, a shell wrapper around a single process)

Re: Interview with Eric Brewer

#34

Post by sp332 »

MCRed wrote: post #5

I'd like a real explanation for why containers are better than unikernels. Yes, unikernals are still early, and containers are convenient, because you have all of linux there... but it seems that running several linuxes on a linux machine is a bit much. One operating system plus XEN plus several applications in unikernels seems more efficient, and more exciting. But it's the less common choice. I am guessing convenie…

If you have a statically-linked binary you can just run it in a container. You don't need a whole OS userland in there.

Re: Interview with Eric Brewer

#35

Post by necubi »

jacquesm wrote: post #9
MCRed wrote: post #5

I'd like a real explanation for why containers are better than unikernels. Yes, unikernals are still early, and containers are convenient, because you have all of linux there... but it seems that running several linuxes on a linux machine is a bit much. One operating system plus XEN plus several applications in unikernels seems more efficient, and more exciting. But it's the less common choice. I am guessing convenie…

I'd like a real explanation for why containers and unikernels are better than regular run-of-the-mill applications running on dedicated servers. It's almost as if the wild west of the web isn't quite enough and we now need to add another explosion of layers-of-abstraction but this time on the server in order to pretend we have infinite hardware which then becomes it's own reason for existence rather than to simply ru…

I'd strongly recommend reading the Borg paper (http://research.google.com/pubs/pub43438.html). It outlines in clear and persuasive detail why Google started using containers. Their reasons may or may not be convincing for smaller companies, but I'd expect the advantages to start accruing above ~10 developers.

Re: Interview with Eric Brewer

#36

Post by vectorjohn »

jacquesm wrote: post #18
DannoHung wrote: post #10

Earlier quoted context omitted.

Sorry, but what costs do containers incur? From my understanding, the resource overhead should be exceedingly minimal (disk space would ostensibly be the largest drawback, if you don't spend time cutting out the fat. Personally, I see this as a tooling issue since fat containers are completely orthogonal to how a container really executes). I understand some of the situation with IO isn't perfect yet, but I haven't h…

An OS has some pretty extensive insight into the processes that it executes, a container is an OS with a single application, so containers (assuming they take basic precautions for isolation) are not going to be able to schedule with anywhere near the efficiency that multiple processes on a single OS will. I can see some (mostly potential at this point) security advantages but that's about it (and maybe those advanta…

I'm not sure what this means, but at least in the case of Docker containers, you can see the running processes from the host OS. I.e., top shows running node.js processes, etc. If that's the case, maybe it does indeed still have some control.

And I just verified, you can kill a running process from outside a docker container. So the OS does see it and probably can do all its scheduling magic.

Re: Interview with Eric Brewer

#37

Post by jacquesm »

DannoHung wrote: post #32
jacquesm wrote: post #27

Earlier quoted context omitted.

Each 'container' is in effect an application plus all its dependencies. So if one or more active containers could share resources then they won't, which leads to inefficiencies because you'll be running a much larger number of processes than you would otherwise (because of duplication) requiring a larger memory footprint and probably less efficient cache and/or IO utilization. The deployment of the apps will be easie…

Isn't this only true if your container build process pulls in the same version of a library in multiple different virutal filesystems? That is, if you are using the same base image for a number of applications and the libraries are installed in the base image rather than the image the application resides, in the kernel should recognize the shared libraries being used as coming from the same place and be able to perfo…

Presumably you could arrange things in such a way that several container images shared libraries and such but that would likely interfere with the (desirable) isolation properties and versioning will play havoc with that anyway (since all dependencies are part-and-parcel of a container and nothing stops multiple containers from shipping different versions of the same package).

Where regular virtualization runs multiple kernels (which in turn will run whatever applications you assign to them) containers appear (to me, feel free to correct me) as a way to 'share a single kernel' across multiple applications dividing each into domains that are as isolated as possible with respect to CPU, memory, namespaces and IO (including network) provisioning and allowing multiple version of the same software to present at the time without interference.

The CPU, memory and IO provisioning can be thought of as a kind of 'virtualization light' and the namespaces partitioning should (in theory) help to make things a bit harder to mess up during deployment.

Leakage from one container to another will probably put a dent in any security advantages but should (again, theoretically) be a bit more robust than multiple processes on a single kernel with shared namespaces.

So I see them as a 'gain' for deployment but a definite detriment for performance because it appears to me we have all (or at least most) of the downsides of virtualization but of course you can expect both virtualization and containers to be used simultaneously in a single installation with predictable (messy) results.

I'm really curious if there is an objective way to measure the overhead of a setup of a bunch of applications on a single machine installed 'as usual' and the same setup using containers on that same machine. That would be a very interesting benchmark, especially when machine utilization in the container-less setup nears the saturation point for either CPU, memory or IO.

Re: Interview with Eric Brewer

#38

Post by vectorjohn »

jacquesm wrote: post #27
DannoHung wrote: post #20

Earlier quoted context omitted.

Do you know where I can read more about this? This is the first I have heard that processes running inside containers interfere with the kernel's scheduler. Or, sorry again, but maybe I'm not following your meaning in some way?

Each 'container' is in effect an application plus all its dependencies. So if one or more active containers could share resources then they won't, which leads to inefficiencies because you'll be running a much larger number of processes than you would otherwise (because of duplication) requiring a larger memory footprint and probably less efficient cache and/or IO utilization. The deployment of the apps will be easie…

Are you talking about shared libraries, that kind of deduplication?

Although true, that probably isn't really very significant compared to the vast wasted resources of idle dedicated machines. Which is hard to avoid without the vast wasted resources of a highly paid somebod(y|ies)

Re: Interview with Eric Brewer

#39

Post by jacquesm »

vectorjohn wrote: post #36
jacquesm wrote: post #18

Earlier quoted context omitted.

An OS has some pretty extensive insight into the processes that it executes, a container is an OS with a single application, so containers (assuming they take basic precautions for isolation) are not going to be able to schedule with anywhere near the efficiency that multiple processes on a single OS will. I can see some (mostly potential at this point) security advantages but that's about it (and maybe those advanta…

I'm not sure what this means, but at least in the case of Docker containers, you can see the running processes from the host OS. I.e., top shows running node.js processes, etc. If that's the case, maybe it does indeed still have some control. And I just verified, you can kill a running process from outside a docker container. So the OS does see it and probably can do all its scheduling magic.

What happens when you install two containers each of which contain the same application, does that install only a single instance of the application binaries and are those still executed with the same efficiency (shared binary) as normally or will that trash the cache? (I understand that if multiple application install different versions of some dependency that that will lead to trouble but in a way that's something 'unfixable').

How does this perform in practice when they start talking to the outside world at or near capacity? How does it perform when they start talking to each other using some defined interface? (But presumably, no longer regular IPC).

Re: Interview with Eric Brewer

#40

Post by jacquesm »

vectorjohn wrote: post #38
jacquesm wrote: post #27

Earlier quoted context omitted.

Each 'container' is in effect an application plus all its dependencies. So if one or more active containers could share resources then they won't, which leads to inefficiencies because you'll be running a much larger number of processes than you would otherwise (because of duplication) requiring a larger memory footprint and probably less efficient cache and/or IO utilization. The deployment of the apps will be easie…

Are you talking about shared libraries, that kind of deduplication? Although true, that probably isn't really very significant compared to the vast wasted resources of idle dedicated machines. Which is hard to avoid without the vast wasted resources of a highly paid somebod(y|ies)

Yes, but then rather than shared libraries the kind of de-duplication the kernel will do when it runs multiple instances of the same binary. This is (normally) very cache and IO efficient since it is done at the VM page level.

I also don't quite understand how one can reserve CPU cycles, memory and deliver IO guarantees without the same over-provisioning that you'd have to do using regular virtualization. After all, as soon as you make a guarantee nobody else can use that which is left over, so in that respect I see little difference between virtualizing the entire OS+app versus re-using the kernel (ok, that does save you the overhead of the kernel itself but that's not a huge difference unless you run a very large number of VMs on a single machine).

Post reply on HN