I love the idea of using containers. Due to linux popularity and google's backing, containers will be next.
But FREEBSD had jails since back in the day. What's the benefit of containers over bsd jails?
Interview with Eric Brewer
61–70 of 142 posts
Re: Interview with Eric Brewer
#62Earlier quoted context omitted.
I'd like a real explanation for why containers and unikernels are better than regular run-of-the-mill applications running on dedicated servers. It's almost as if the wild west of the web isn't quite enough and we now need to add another explosion of layers-of-abstraction but this time on the server in order to pretend we have infinite hardware which then becomes it's own reason for existence rather than to simply ru…
Do you want to design for failover of a complex system? Then containers are your friend. Every dedicated server that you set up has an opportunity to not be duplicated perfectly. Every system that knows about your dedicated server has an opportunity to hard code what it shouldn't. Which makes these a potential point of failure. Add enough of those, and you're statistically guaranteed that the careful architecture tha…
I'll definitely concede the 'mistakes are made' point because I recently spent a month untangling 8 supposedly identical VMs which really weren't but it took a lot of sleuthing to spot the differences. Oh, and to make life more interesting, not all those differences were made to the configuration files (making the systems not quite reboot safe), quite a few of them were only made to certain kernel settings in /proc and /sys (who thought that we should have two of those systems anyway...?).
I like the test-the-panic-button attitude that google brings to these things, I've yet to get someone to accept my challenge to power down their supposedly automated fail-over solution, it's supposed to work but they usually can't be sure or management would surely not allow such a rash thing as a live test and it's bad form for me to then walk up to the switchboard and trip the breaker. Verry tempting...
Re: Interview with Eric Brewer
#63One thing that bothers me about the article is that it shows a recurring problem: IT not knowing what it knows. The NoSQL movement didn't notice that NonStop Architecture scaled linearly to thousands of cores with strong-consistency, five 9's, and SQL support. In the mid-80's. Instead of making a low-cost knockoff, like cluster movement did for NUMA's, they ditched consistency altogether and launched NoSQL movement.…
thats a pretty good comment actually. there's quite a bit of similarity. we generally tend to jump into these as "omg awesome new tech" with a very narrow view. But it also helps boosting more though-out techs (even thus it feels less efficient to go through that route first, its perhaps the only route that works with human: try, fail, try again, etc.)
Yeah we do. My only guess is it's two things: (a) our industry is horrendous at communicating previous generation's wisdom in a usable way; (b) a social phenomenon. Quick example of the first are industry pro's locking up their good advice in obscure, expensive books and cutting edge research silo'd into ACM, IEEE, etc.
The other is a social thing that leads to the "network" effect. People flock to something for whatever reason. This builds a community (or network) that entices others to join. That also tend to forget about other things and reinvent the wheel. Example: much of current work in Web applications aims to solve problems already solved in client-server apps with better efficiency, security, reliability, and portability. Even Facebook went back to that model for mobile IIRC. Good luck convincing most Web technologists to switch to client-server, though.
Whoever solves both these problems will create ripple effects that grow innovation at a heightened, maybe exponentially better, pace. The reason will be a combination of avoiding wasted effort plus visibility into best efforts. I got ideas on Problem 1 but the best minds need to get on Problem 2: it's a gold mine if it's solved.
Re: Interview with Eric Brewer
#64One thing that bothers me about the article is that it shows a recurring problem: IT not knowing what it knows. The NoSQL movement didn't notice that NonStop Architecture scaled linearly to thousands of cores with strong-consistency, five 9's, and SQL support. In the mid-80's. Instead of making a low-cost knockoff, like cluster movement did for NUMA's, they ditched consistency altogether and launched NoSQL movement.…
You have some interesting points, however Eric Brewer isn't just 'some Google guy discussing CAP'. He actually invented the theorem... It's also known as 'Brewer's Theorem'.
FWIW, CAP is not about NoSQL or the 'NoSQL movement', it's about distributed systems and distributed shared memory, which applies to a whole range of computing problems.
Re: Interview with Eric Brewer
#65Earlier quoted context omitted.
Yes, I think it's safe to say that Google operates at scale. A lot of user-facing services at Google have to be over-provisioned in order to handle the cyclical usage patterns (the daily query peak is far higher than the average for most services) and to be able to survive the loss of a datacenter or two. This results in a lot of under-utilized servers for a big fraction of the time. So by packing lots of medium and…
Borg containers consisted generally of a single process... Really? My impression was that typically you'd have a process for the service, a borgmon process for monitoring, and maybe another process to ship logs off in the background. Developers would only think about the service process (which itself typically was a fairly thin shim in front of other services), but a borg container would have more than that going on…
The borgmon process would be a completely separate job on separate machines (generally with a lot fewer instances).
The logsaver would also be a separate job, although typically running co-located 1:1 with instances of the actual service job. The service and the logsaver would have access to the same chunk of disk (where the logs were generated) but otherwise they were separate as far as the kernel was concerned. (As far as Borg was concerned they were very much related, but that was at a much higher level than the kernel).
Re: Interview with Eric Brewer
#66I'd like a real explanation for why containers are better than unikernels. Yes, unikernals are still early, and containers are convenient, because you have all of linux there... but it seems that running several linuxes on a linux machine is a bit much. One operating system plus XEN plus several applications in unikernels seems more efficient, and more exciting. But it's the less common choice. I am guessing convenie…
I'd like a real explanation for why containers and unikernels are better than regular run-of-the-mill applications running on dedicated servers. It's almost as if the wild west of the web isn't quite enough and we now need to add another explosion of layers-of-abstraction but this time on the server in order to pretend we have infinite hardware which then becomes it's own reason for existence rather than to simply ru…
Because rails app 1 needs version x.y.z and rails app 2 needs version a.b.c and RVM sucks and managing gems sucks.
This is, literally, the reason.
You can replace rails with any similarly bad technology. I got this explanation a few weeks ago at my job:
the java build process (I'm not kidding) has such a complex dependency graph that it must spin up full containers to do each build.
----
If dynamic languages supported better packaging/isolation this entire thing would be research projects.
Re: Interview with Eric Brewer
#67Earlier quoted context omitted.
Do you have experience using containers? Here's my answer for "why": DRY. Once you've deployed hundreds of servers using the same exact Ubuntu 12.04 LTS kernel base, why not just completely abstract the OS away and focus the attention on scaling the OS services that matter? Why is that when I decide that I need to scale out, I need to copy every library of the OS and every line of code for the kernel and redeploy it…
> why not just completely abstract the OS away and focus the attention on scaling the OS services that matter? Because it adds a layer that makes no sense unless you have very specific use cases. Though I see the point regarding people efficiency, that one makes good sense (see other comment in this sub-thread) > Why is that when I decide that I need to scale out, I need to copy every library of the OS and every line…
> See: chef, configuration management and various deployment services
A chef script is basically the automation of "I need to copy every library of the OS and every line of code for the kernel and redeploy it every time I add a node?" I'm sorry if you didn't pick up on my implied remark. Two problems are then introduced when automating those actions: (1) it doesn't negate the fact that I need to store and deploy a 700M sized OS layer every time I want to add a node (which takes minutes, not seconds with non-containerized configs) and (2) maintaining config scripts can (not always) be painful (version control, rollbacks, etc)
> unless you have very specific use cases.
> Containers are like sandboxes with significant overhead for applications
Again, do you have experience using containers? You seem awfully dismissive ("you're doing it wrong!") in a way that suggests that you might not entirely understand how they actually work...
Re: Interview with Eric Brewer
#68Great Interview! I worked as a contractor at Google in 2013 and loved their infrastructure. It was amazing to fire off a Borg job that used hundreds to thousands of servers, and the web based tools for tracking the job, fantastic logging to drill into problems, etc. And, Borg was two generations ago! Even though I am very happy doing what I am now, sometimes I literally wake up in the morning thinking about Google's…
Eric Brewer is still on the faculty at Berkeley, just taking an extended leave at Google. He's scheduled to teach the undergrad database class next fall: http://www.eecs.berkeley.edu/Scheduling/CS/schedule-next.htm...
That is good to hear, thanks.
Re: Interview with Eric Brewer
#69Earlier quoted context omitted.
I'm not sure there will be enough duplication to make a difference most of the time. However, it will be less than with dedicated hardware. There is also ksmd that is useful with VMs, where memory is at a premium, though I'm not certain it is compatible with lxc yet.
I can see how there will be less duplication than with virtualization (because you share the kernel, rather than running multiple instances of the kernel) but I can't see how duplication will be less than with dedicated hardware.
It won't - for any single given service, running on a dedicated system will probably be able to squeeze out a tiny bit more performance than running on a shared system with containers, for whatever fraction of the time that you're close to maxing out that server's performance.
But in practice (at Google-scale, anyway), that's dwarfed by the efficiency gains you can get by squeezing lots of things on to the same machine and increasing the overall utilization of the machine. Prior to adding kernel containers to Borg to allow proper resource isolation between the different jobs on a machine, the per-machine utilization was really embarrassingly low.
Another point to consider is that not all jobs are shaped the same as the machines - some jobs need more memory (so if you put them on a number of dedicated machines adding up to the total amount of memory needed, there will be lots of wasted CPU), and other jobs use a lot more CPU and less memory (so if you put them on a number of dedicated machines adding up to the total amount of CPU needed, there will be lots of wasted memory).
By breaking each job up into a greater number of smaller instances and bin-packing on to each machine, you could take advantage of the different resource shapes of different jobs to get better overall utilization.
Re: Interview with Eric Brewer
#70Earlier quoted context omitted.
I'd like a real explanation for why containers and unikernels are better than regular run-of-the-mill applications running on dedicated servers. It's almost as if the wild west of the web isn't quite enough and we now need to add another explosion of layers-of-abstraction but this time on the server in order to pretend we have infinite hardware which then becomes it's own reason for existence rather than to simply ru…
Do you have experience using containers? Here's my answer for "why": DRY. Once you've deployed hundreds of servers using the same exact Ubuntu 12.04 LTS kernel base, why not just completely abstract the OS away and focus the attention on scaling the OS services that matter? Why is that when I decide that I need to scale out, I need to copy every library of the OS and every line of code for the kernel and redeploy it…
Why do you bother with an OS or a container if you want to abstract the OS away? Just PXE boot all the machines and have the application set to run as init. No need to worry about copying every library for the OS to every machine and you only require the minimal kernel that you tell it to boot with.