Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Interview with Eric Brewer

medium.com

Post reply on HN
Print view View original

71–80 of 142 posts

Re: Interview with Eric Brewer

#71

Post by JustSomeNobody »

nickpsecurity wrote: post #63
zobzu wrote: post #57

Earlier quoted context omitted.

thats a pretty good comment actually. there's quite a bit of similarity. we generally tend to jump into these as "omg awesome new tech" with a very narrow view. But it also helps boosting more though-out techs (even thus it feels less efficient to go through that route first, its perhaps the only route that works with human: try, fail, try again, etc.)

Yeah we do. My only guess is it's two things: (a) our industry is horrendous at communicating previous generation's wisdom in a usable way; (b) a social phenomenon. Quick example of the first are industry pro's locking up their good advice in obscure, expensive books and cutting edge research silo'd into ACM, IEEE, etc. The other is a social thing that leads to the "network" effect. People flock to something for what…

The "whatever reason" tends to be one's resume. Some people think (and I'm not necessarily disagreeing here) the only way to stay employable is to have the absolute "blogging"edge tech on your resume.

Re: Interview with Eric Brewer

#72

Post by mattmcknight »

jacquesm wrote: post #9
MCRed wrote: post #5

I'd like a real explanation for why containers are better than unikernels. Yes, unikernals are still early, and containers are convenient, because you have all of linux there... but it seems that running several linuxes on a linux machine is a bit much. One operating system plus XEN plus several applications in unikernels seems more efficient, and more exciting. But it's the less common choice. I am guessing convenie…

I'd like a real explanation for why containers and unikernels are better than regular run-of-the-mill applications running on dedicated servers. It's almost as if the wild west of the web isn't quite enough and we now need to add another explosion of layers-of-abstraction but this time on the server in order to pretend we have infinite hardware which then becomes it's own reason for existence rather than to simply ru…

I would say containers are primarily an improvement over hypervisors / VMs in performance, not over dedicated hardware. However, they still allow for some of the capabilities of VMs, in terms of elastic reshaping of a cluster. So, if you have a static environment, it's probably not useful to you. If you have a dynamic environment, where you need to frequently repurpose particular machines, but want to do it more efficiently than with a VM, it can be very useful.

In another view, it's another approach to what many look to Chef, Ansible, and Puppet to do. Combined with something like Mesos or Kubernetes, you can quickly deploy to a heterogenous cluster, without a lot of install scripts running.

Some of the other uses cases, such as running multiple containers simultaneously on the same hardware, make less to me.

Re: Interview with Eric Brewer

#73

Post by menage »

jacquesm wrote: post #40
vectorjohn wrote: post #38

Earlier quoted context omitted.

Are you talking about shared libraries, that kind of deduplication? Although true, that probably isn't really very significant compared to the vast wasted resources of idle dedicated machines. Which is hard to avoid without the vast wasted resources of a highly paid somebod(y|ies)

Yes, but then rather than shared libraries the kind of de-duplication the kernel will do when it runs multiple instances of the same binary. This is (normally) very cache and IO efficient since it is done at the VM page level. I also don't quite understand how one can reserve CPU cycles, memory and deliver IO guarantees without the same over-provisioning that you'd have to do using regular virtualization. After all,…

But you can make guarantees to the critical jobs (up to the total size of the machine) and then let batch jobs (with less time-sensitive requirements) run best-effort in the slack.

In the event that there ends up being no best-effort resources available on a machine for a significant period of time (because all the user-facing jobs are busy and using their guaranteed resources) Borg will shift the starving batch jobs to other machines that aren't so busy.

Re: Interview with Eric Brewer

#74

Post by vacri »

falcolas wrote: post #46
DannoHung wrote: post #10

Earlier quoted context omitted.

Sorry, but what costs do containers incur? From my understanding, the resource overhead should be exceedingly minimal (disk space would ostensibly be the largest drawback, if you don't spend time cutting out the fat. Personally, I see this as a tooling issue since fat containers are completely orthogonal to how a container really executes). I understand some of the situation with IO isn't perfect yet, but I haven't h…

Containers, by themselves? Very little. Containers, as implemented by Docker, Rkt and Systemd? Quite a bit. Disk - the technologies used for disk isolation (save chroots) are very poor performance, and in some cases can cause resource contention between what would otherwise appear to be unrelated containers. As an exmaple, using AUFS with Node creates a situation where any containers running on the same file system c…

I'm having much the same experience with Docker, and talking to other ops folks who get to actually put it into production, they usually have similar experiences.

Docker is fun and great when it's running on your workstation and coddled by your fingers at the terminal, but there's a lot of gotchas and missing parts when it comes to putting things into production, to be taken care of in a hands-off manner. There still isn't an easy way to centralise logs from a container app's STDOUT. Yes, there are other containers you can install to ship logs (which work for the author's use-case, not necessarily yours) or you can hack together something horrible. If you want to look at container logs, you have to have root rights. You can be in the docker group and have full control over the daemon, but the container log location is root only, and is made afresh with every container. (and don't forget to rotate those logs!)

My latest fun with docker is that one of my docker servers, built from the same source image and running on the same configuration plan in ansible as my other docker servers, fails to start docker on boot. Some sort of race condition, I assume. Basically it fails to apply its iptables rules and dies. People talk about making problems go away with docker, but it's a trope in my team that any day I'm working with docker, I'll be spamming chat with problems I'm finding in it from an ops point of view. And I'm just a midrange sysadmin :) But the point is that adding Docker adds an extra layer of debugging. The app stack still needs to be debugged, and now there's an extra abstraction layer that needs debugging.

Plus, in my particular case, there's the irony of using single-function VMs to run a docker container, which is running the same OS version as the VM :) (my devs bought into docker before I arrived...)

Re: Interview with Eric Brewer

#75

Post by jacquesm »

mbesto wrote: post #67
jacquesm wrote: post #52

Earlier quoted context omitted.

> why not just completely abstract the OS away and focus the attention on scaling the OS services that matter? Because it adds a layer that makes no sense unless you have very specific use cases. Though I see the point regarding people efficiency, that one makes good sense (see other comment in this sub-thread) > Why is that when I decide that I need to scale out, I need to copy every library of the OS and every line…

> See: chef, configuration management and various deployment services A chef script is basically the automation of "I need to copy every library of the OS and every line of code for the kernel and redeploy it every time I add a node?" I'm sorry if you didn't pick up on my implied remark. Two problems are then introduced when automating those actions: (1) it doesn't negate the fact that I need to store and deploy a 70…

I'm not dismissive enough to have already committed elsewhere in this thread to re-do a bit of experimentation I did about a year ago on using containers and at the time the performance overhead was such that I failed to see the use case, but since this is a fast moving field it won't hurt me one bit to update my knowledge.

In a nutshell: running a 'standard' combo of apache and a DB server as well as some auxiliary bits and pieces inside 'containers' a year ago gave significant overhead compared to running those without the containers. I'll re-do this and I'll probably do a write-up because the subject is interesting. This comment and follow up (https://news.ycombinator.com/item?id=9567623) are by people using this tech in production right now and their experience echos mine (but they're very far down the line compared to where I stopped).

Besides that particular use case (where performance and isolation are the key components to be looked at) some interesting points have been made in this thread which has shifted my stance on container use depending on what the situation is. So I don't think it is valid to classify me as 'awfully dismissive'.

FWIW I have not used containers in production (yet) but I'll be more than happy to if I can figure out where and how they can bring me an advantage, which is pretty much how I approach all tools.

Re: Interview with Eric Brewer

#76

Post by voronoff »

bostonpete wrote: post #54
jedberg wrote: post #29

A funny and somewhat off topic story -- back in 2003, before the Google IPO, Google was doing a recruiting event at Berkeley. They brought a few of their folks with them: their founder Larry, one of their female engineers, Marissa, and some others. They did a little talk, and during the Q&A, professor Brewer told Larry that there was an opening in the PhD program and he was welcome to it. Larry politely declined. Aft…

In 2003, Google had over a billion dollars in revenue. I suspect that answer was tongue in cheek.

I suspect the question was as well.

Re: Interview with Eric Brewer

#77

Post by btilly »

menage wrote: post #65
btilly wrote: post #53

Earlier quoted context omitted.

Borg containers consisted generally of a single process... Really? My impression was that typically you'd have a process for the service, a borgmon process for monitoring, and maybe another process to ship logs off in the background. Developers would only think about the service process (which itself typically was a fairly thin shim in front of other services), but a borg container would have more than that going on…

The borgmon process would be a completely separate job on separate machines (generally with a lot fewer instances). The logsaver would also be a separate job, although typically running co-located 1:1 with instances of the actual service job. The service and the logsaver would have access to the same chunk of disk (where the logs were generated) but otherwise they were separate as far as the kernel was concerned. (As…

You are right. I was remembering that I'd see those three together in the borg file, and was thinking about them being co-located because of that.

Re: Interview with Eric Brewer

#78

Post by nickpsecurity »

bitcrusher wrote: post #64
nickpsecurity wrote: post #49

One thing that bothers me about the article is that it shows a recurring problem: IT not knowing what it knows. The NoSQL movement didn't notice that NonStop Architecture scaled linearly to thousands of cores with strong-consistency, five 9's, and SQL support. In the mid-80's. Instead of making a low-cost knockoff, like cluster movement did for NUMA's, they ditched consistency altogether and launched NoSQL movement.…

You have some interesting points, however Eric Brewer isn't just 'some Google guy discussing CAP'. He actually invented the theorem... It's also known as 'Brewer's Theorem'. FWIW, CAP is not about NoSQL or the 'NoSQL movement', it's about distributed systems and distributed shared memory, which applies to a whole range of computing problems.

I've edited my comment to give him that credit. I understand the CAP theorem applies to many things. The reason I tied NoSQL in is that it's often cited as the reason people traded away strong consistency. Yet, there were strongly-consistent setups with the desirable properties in production and in academia. He and others rarely mention them in such discussions. It's why I tied them together.

Re: Interview with Eric Brewer

#79

Post by otterley »

nickpsecurity wrote: post #49

One thing that bothers me about the article is that it shows a recurring problem: IT not knowing what it knows. The NoSQL movement didn't notice that NonStop Architecture scaled linearly to thousands of cores with strong-consistency, five 9's, and SQL support. In the mid-80's. Instead of making a low-cost knockoff, like cluster movement did for NUMA's, they ditched consistency altogether and launched NoSQL movement.…

> NonStop Architecture scaled linearly to thousands of cores with strong-consistency, five 9's, and SQL support.

Admittedly so, but at a very, very high price. Similarly, Sun and SGI had amazing technology in the server and workstation space (after Solaris 2.3, anyway), but over time Linux became "good enough" and we became willing to sacrifice Sun's niceties to save millions per data center.

The mere existence of technology isn't enough; it has to be affordable - and rational managers will have to make cost/benefit decisions that suit their goals.

Re: Interview with Eric Brewer

#80

Post by otterley »

kayman wrote: post #61

I love the idea of using containers. Due to linux popularity and google's backing, containers will be next. But FREEBSD had jails since back in the day. What's the benefit of containers over bsd jails?

Linux has copy-on-write block devices that make it possible to efficiently layer container filesystems. FreeBSD has no such thing as far as I know; the best you can do involves hard links (correct me if I'm wrong).
Post reply on HN