Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Interview with Eric Brewer

medium.com

Post reply on HN
Print view View original

131–140 of 142 posts

Re: Interview with Eric Brewer

#131

Post by menage »

xorcist wrote: post #106
menage wrote: post #69

Earlier quoted context omitted.

It won't - for any single given service, running on a dedicated system will probably be able to squeeze out a tiny bit more performance than running on a shared system with containers, for whatever fraction of the time that you're close to maxing out that server's performance. But in practice (at Google-scale, anyway), that's dwarfed by the efficiency gains you can get by squeezing lots of things on to the same machi…

Hang on there, are you saying you can squeeze in more applications on any given server using containers, rather than just running them in the regular filesystem? That doesn't make sense. No, you use containers despite the fact that your hardware utilization goes down (mainly because no shared pages between applications), because your huge sprawling environment is too hard to change with flag days.

There are multiple definitions of the word 'container'. In the case of Borg, 'container' referred to the kernel resource isolation component, which is completely orthogonal to how the apps were packaged. Borg aggressively shared packages between jobs on the machine where possible.

Being able to strictly apportion resources between the different jobs on a machine (and decide who gets starved in the event that the scheduler has overcommitted the machine) means you can squeeze more out of a given server (by safely getting its utilization closer to 100%)

There are other definitions of the word 'container' that are closer to 'virtual machine' and include things like a disk image which is much harder to share, but that's not what's being discussed in the context of Borg. (Not sure about Kubernetes, that's after my time)

Re: Interview with Eric Brewer

#132

Post by nickpsecurity »

jeswin wrote: post #81
nickpsecurity wrote: post #49

One thing that bothers me about the article is that it shows a recurring problem: IT not knowing what it knows. The NoSQL movement didn't notice that NonStop Architecture scaled linearly to thousands of cores with strong-consistency, five 9's, and SQL support. In the mid-80's. Instead of making a low-cost knockoff, like cluster movement did for NUMA's, they ditched consistency altogether and launched NoSQL movement.…

> The NoSQL movement didn't notice that NonStop Architecture scaled linearly to thousands of cores with strong-consistency, five 9's, and SQL support. Also incredibly expensive. Take this case, Google took off because they were able to scale-out with off-the-shelf hardware, compared to the millions banks were pouring in for scale-up configurations which handled much less load. Scale-up can quickly hit hard limits, be…

In startup mode, I expect companies to do whatever works. Google's initial decisions were correct. Later, they dedicated expensive teams to inventing brand new solutions for about every aspect of their operation. That's when my critique apples. For instance, they might have looked at expensive NonStop, identified what they could copy with COTS tech, and made a cheap knockoff. They could've done the same with prior MPP or cluster tech which already scaled to thousands of nodes with open source management tools and microsecond latency. Much more efficient than web stacks, too. The one example I saw of them copying and improving past methods was Google File System. As I predicted, that good choice laid foundation for awesome stuff such as BigTable, Spanner, and F1.

NoSQL movement's origins are highly debatable. Here's what I saw in its beginning: an explosion of articles on the subject after a few success stories about big companies doing massive scale on cheap servers. Zealots argued that mature, strong-consistency solutions had problems. So, instead of fixing those, we should just ditch them and strong consistency for The New, Great Thing. The only time I even saw a real analysis of cost-benefits was a few articles which focused on a narrow class of applications where data consistency didn't really matter. So, my conclusion was that the movement was two things: 95% a social phenomenon that happens with each IT fad; 5% a result of weaknesses of relational model and tools. That last part makes sense and is why I've always opposed RDBMS's.

On your last point, do you have a citation that shows weak consistency databases offer enough integrity for "majority of use-cases?" I thought the majority of use cases for databases were protecting data that's important to a business: orders, billing, HR, inventory, customer information, and so on. I usually just tell them to use PostgresSQL and that covers it with high reliability. If you really can back your claim, though, I'll be glad to start transitioning my R&D toward designs that trade against integrity of data.

Re: Interview with Eric Brewer

#133

Post by nickpsecurity »

rdtsc wrote: post #85
nickpsecurity wrote: post #49

One thing that bothers me about the article is that it shows a recurring problem: IT not knowing what it knows. The NoSQL movement didn't notice that NonStop Architecture scaled linearly to thousands of cores with strong-consistency, five 9's, and SQL support. In the mid-80's. Instead of making a low-cost knockoff, like cluster movement did for NUMA's, they ditched consistency altogether and launched NoSQL movement.…

> There is hope: Google went from a famous, NoSQL player to inventing an amazing, strong-consistency RDBMS (F1). Let's hope more follow. The problem is at a large distributed scale forcing consistency is basically fighting with the laws of physics. Is something in Australia always consistent with something in US? Well "same time" is a funny thing in physics because it doesn't actually exist. So we have to work really…

Fighting the laws of physics is a great comparison. Every chip designer trying to create the illusion of a sequential (or multicore) machine with a specific memory model fights the laws of physics. So do people building bridges, designing planes, and trying to maintain space stations. The greatest weapon we have after human ingenuity is using good engineering principles built on what we've tried and learned.

So, back to this current example. Synchronizing time was a huge problem for consistency. They had expensive, highly-custom datacenters throughout the world. Yet, there wasn't even a solution worth buying that wouldn't cost a fortune or need new infrastructure. An engineer noticed that a time-source existed which all datacenters could sync with using affordable, COTS equipment. One among others. So, they used that to solve the critical problem, solved other problems with other technologies, and integrated the resulting components into a solution to their real problem (F1).

What I've just described is not a hack: it's Engineering 101. Identify the problem(s), look for known solutions to it, adapt them to your needs, and deliver the solution. Their use of GPS to solve a problem that otherwise would cost millions of dollars to solve is exemplary engineering. Given datacenter costs, using this at each one would barely be a blip on the budget sheet and will get easier to deploy as adoption increases (network effect).

Re: Interview with Eric Brewer

#134

Post by nickpsecurity »

otterley wrote: post #130
nickpsecurity wrote: post #129

Earlier quoted context omitted.

True. I always expect this. Yet, Google doesn't fit that profile because they weren't just buying something that works: they were investing countless sums into geniuses and their projects trying to invent new things. It's at that point I expect companies to learn from the past and do better.

NonStop and its ilk involve complex hardware engineering, not merely software. But software is Google's forte, not hardware. I wouldn't expect Google to reinvent legacy HA systems, if only because engineering HA systems like those isn't their business.

Software is their main skill. Yet, Google and Facebook have both been doing custom hardware for datacenters for years now. They're also funding academic R&D on chip design and such. Such companies needs also created an ecosystem offering things OpenFlow switches that are custom hardware implementations of radically different software. Intel and AMD are also both doing custom work for unnamed, datacenter companies. It's hearsay but I'd expect top players to be involved. So, they're already all well into the hardware business.

The simple route, as I indicated, would be to copy, contract, or even buy a MPP vendor. In academia, MIT Alewife showed one custom chip was all that was necessary to build a NUMA-style, 512 node machine with COTS hardware. Existing, shared-nothing clusters already scaled higher than Google using custom chips for interconnect. One can buy cards or license the I.P for FPGAs, Structured-ASIC's, etc. Much software for compute and management was open source thanks to groups such as Sandia. And so on. Plenty to build on that's largely been ignored except in academia.

Instead, they've largely invested in developing hardware and software to better support the inherently-inefficient, legacy software. So, they're doing the hardware stuff but just not imitating the best-of-breed solutions of the past or present. The only exception is OpenFlow: a great alternative to standard Internet tech that major players funded in academia and are putting in datacenters. Another success story is Microsoft testing a mid-90's approach of partitioning workloads between CPU's and FPGA's. So, they're... slowly... learning.

Re: Interview with Eric Brewer

#135

Post by nickpsecurity »

marktangotango wrote: post #120
nickpsecurity wrote: post #63

Earlier quoted context omitted.

Yeah we do. My only guess is it's two things: (a) our industry is horrendous at communicating previous generation's wisdom in a usable way; (b) a social phenomenon. Quick example of the first are industry pro's locking up their good advice in obscure, expensive books and cutting edge research silo'd into ACM, IEEE, etc. The other is a social thing that leads to the "network" effect. People flock to something for what…

I pimp this quote from Alan Kay regularly: "Computing spread out much, much faster than educating unsophisticated people can happen. In the last 25 years or so, we actually got something like a pop culture, similar to what happened when television came on the scene and some of its inventors thought it would be a way of getting Shakespeare to the masses. But they forgot that you have to be more sophisticated and have…

Thanks for that awesome article! Interestingly, he used the same example I've been using in INFOSEC discussions online: Burroughs B5000. It was simply brilliant and still better at the core than anything I use today. Let me explain.

There have been countless reliability and security problems that occur due to buffer, pointer, data becomes code, and interface errors. These are about 99% of worst problems. They happen because underlying Intel/IBM/RISC architecture treats all data the same... mostly. Plus, the systems languages (C/C++) are fundamentally broken far as preventing errors. The Burroughs team saw this [in 1961] and solved the problems at their source: CPU protection of pointers; CPU could tell code & data apart for security purposes; hardware-managed stack; CPU bounds-checked arrays; high-level language (Algol) for system code; interface types checked at compile & function-call time; hardware & software isolation of apps from OS. Good luck crashing or hacking that!

So, I've read thousands of hardware, firmware, and software solutions to these problems. Yet, very few will straight up fix the problem at its source. That's despite the existence of a proven solution since 1961 that costs a mere two bits of tagging. I'll give up a single-digit percentage of memory with single-digit performance hit to stop 99% of attacks. I'll do it today. Yet, industry's latest solutions are detecting this little tactic, hardware extensions for that, and no solution to the actual problem.

The failure of modern industry to do what Burrough's did, fix the underlying problem, is the source of most of our IT headaches. Aside from social reasons, backward compatibility with legacy is a big contributor. It's why heuristic-driven, software transformation systems such as Semantic Design's toolkit or Racket need a huge boost in R&D. Such tech is seeming like our only hope to getting legacy software onto better underlying platforms as nobody will pay for a human to understand and rewrite each codebase line for line, bug for bug.

Re: Interview with Eric Brewer

#136

Post by nickpsecurity »

andyidsinga wrote: post #86
nickpsecurity wrote: post #63

Earlier quoted context omitted.

Yeah we do. My only guess is it's two things: (a) our industry is horrendous at communicating previous generation's wisdom in a usable way; (b) a social phenomenon. Quick example of the first are industry pro's locking up their good advice in obscure, expensive books and cutting edge research silo'd into ACM, IEEE, etc. The other is a social thing that leads to the "network" effect. People flock to something for what…

re (a) : is not just a communication problem. a lot of solutions from previous generation are dramatically less easy than newer offerings. IMHO nosql and cloud compute offerings we're like this. the problem with folks hating on new technology "x" is often failing to see that the new thing might only offer one significant advantage over the prior tech, or they see it, but discount it too heavily to be motivated to try…

For (a), it's hard to say where the problem is: is it developers of new solutions who threw the baby out with the bathwater instead of fixing problems of proven methods? Or is it with demand side that's quick to jump on those new thing? Or is it demand side in the sense that they rarely fund solutions to problems with their almost-good-enough software?

Hard for me to say.

Re: Interview with Eric Brewer

#137

Post by inquisitiveio »

dunkelheit wrote: post #117

Can someone explain or provide an educated guess about what is the google's strategy with kubernetes here? Surely containers are hot now and it is nice to have a stake in the game but borg has been one of their key competitive advantages. What is the profit in making an open-source alternative?

I have been wondering the same thing. I suppose it's partly because it was only going to be a competitive advantage as long as as containers where not commoditized which seems to be the direction its now heading.

Re: Interview with Eric Brewer

#138

Post by andyidsinga »

TheIronYuppie wrote: post #128
andyidsinga wrote: post #90

imo, if containers are the future ( very plausible ), then, things like Aws lambda are just as plausible if only just a bit further out. I think this is the case due to granularity of workloads and what apears to be a continuum in the workload container from metal > vm > containers > lambdas (as first class workloads). fun stuff

Disclaimer: I work at Google on Kubernetes True, lambda is DEFINITELY an advance, but it's more like the salmon at a buffet that includes steak and chicken. Some applications will need just the ability to run code (lambda), some will need defined environments (containers) and some will need total isolation (VMs/bare metal). You'll see a mix of all of these in every mature environment - some things do not fit. For exa…

good points but I'm not sure anything about the fundamental lambda architecture prevents it from being applied to high bandwidth or low latency uses. especially if lambda functions are orchest rated to run in the metal ...think: kernel module implements lambda, then add orchestration glue up in user land

Re: Interview with Eric Brewer

#139

Post by andyidsinga »

nickpsecurity wrote: post #136
andyidsinga wrote: post #86

Earlier quoted context omitted.

re (a) : is not just a communication problem. a lot of solutions from previous generation are dramatically less easy than newer offerings. IMHO nosql and cloud compute offerings we're like this. the problem with folks hating on new technology "x" is often failing to see that the new thing might only offer one significant advantage over the prior tech, or they see it, but discount it too heavily to be motivated to try…

For (a), it's hard to say where the problem is: is it developers of new solutions who threw the baby out with the bathwater instead of fixing problems of proven methods? Or is it with demand side that's quick to jump on those new thing? Or is it demand side in the sense that they rarely fund solutions to problems with their almost-good-enough software? Hard for me to say.

ha! probably all of the above :)

Re: Interview with Eric Brewer

#140

Post by otterley »

TheIronYuppie wrote: post #127
kayman wrote: post #61

I love the idea of using containers. Due to linux popularity and google's backing, containers will be next. But FREEBSD had jails since back in the day. What's the benefit of containers over bsd jails?

Disclaimer: I work at Google on Kubernetes & Containers I mentioned earlier - https://news.ycombinator.com/item?id=9570639 - but the biggest thing is not isolation, it's portability. You do not care what you're running on, just that you have a certain amount of disk, CPU and network available to you, and the rest you handle yourself. That gives your organization a crazy amount of flexibility in moving that ball of st…

> You do not care what you're running on, just that you have a certain amount of disk, CPU and network available to you

Does Kubernetes provide those guarantees to the containers? And, if so, what does it do once those guarantees can no longer be met, say, due to neighbor growth?

Post reply on HN