Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why the data center needs an operating system

radar.oreilly.com

Post reply on HN
Print view View original

31–40 of 102 posts

Re: Why the data center needs an operating system

#31

Post by perlgeek »

presspot wrote: post #24
perlgeek wrote: post #18

Yes, we need new, standardizing APIs. However this made me cringe: > Exposing machines as the abstraction to developers unnecessarily complicates the engineering, causing developers to build software constrained by machine-specific characteristics, like IP addresses and local storage. It reminded me of the old RPC approach of making potentially any method call a remote method call. It didn't work out, simply because…

One thing worth noting is that the indirection Ben describes is, basically, the indirection that is already in the Apache Mesos kernel. In real world environments (e.g., Twitter), there is negligible overhead. Also, locality and latency can both be expressed in terms of placement rules and schedulers on Mesos can use those rules to guarantee or express preference for task placement that optimizes around reduced laten…

> The point that I take away is that the ability to express your needs in a declarative way (e.g., "place these two tasks such that they have such-and-such latency) is much more scalable, flexible and resilient than coding to machine-specific internals. The latter is easier to update and supports delegation of responsibilities.

That makes total sense to me, thanks.

Re: Why the data center needs an operating system

#32

Post by 23david »

I definitely wouldn't consider Mesos to be anything like an operating system for the datacenter. That's just marketing language and confuses things.

Mesos is basically an application scheduler. It doesn't manage the base operating systems or machine provisioning. Mesos is concerned with ensuring that one or multiple applications are launched and running on a cluster of machines.

The Saltstack framework is the only thing I know of that would provide all the primitives and control necessary for an operator to truly control a full 5-300,000 node datacenter from one station. It's the only thing out there that will allow for super low latency response to commands across the cluster. It also can do configuration management etc if you want, or a lot of people use it to just trigger existing chef/puppet jobs.

I've been using Saltstack in conjunction with Mesos to build out this full datacenter-as-OS stack. Works... :-)

Re: Why the data center needs an operating system

#33

Post by gtaylor »

23david wrote: post #32

I definitely wouldn't consider Mesos to be anything like an operating system for the datacenter. That's just marketing language and confuses things. Mesos is basically an application scheduler. It doesn't manage the base operating systems or machine provisioning. Mesos is concerned with ensuring that one or multiple applications are launched and running on a cluster of machines. The Saltstack framework is the only th…

> It's the only thing out there that will allow for super low latency response to commands across the cluster. It also can do configuration management etc if you want, or a lot of people use it to just trigger existing chef/puppet jobs.

FWIW, Ansible also does low-latency communications through pipelining/accelerated mode. Ditto for the config management and triggering.

Re: Why the data center needs an operating system

#34

Post by davidgerard »

preillyme wrote: post #30
davidgerard wrote: post #28

Earlier quoted context omitted.

Oh, we're completely cross-platform these days! That means it runs on both RHEL and Ubuntu.

ha ha ha

ha ha only serious http://www.catb.org/jargon/html/H/ha-ha-only-serious.html

It's VAXocentrism for the 21st century. http://www.catb.org/jargon/html/V/vaxocentrism.html

Re: Why the data center needs an operating system

#35

Post by nqnielsen »

23david wrote: post #32

I definitely wouldn't consider Mesos to be anything like an operating system for the datacenter. That's just marketing language and confuses things. Mesos is basically an application scheduler. It doesn't manage the base operating systems or machine provisioning. Mesos is concerned with ensuring that one or multiple applications are launched and running on a cluster of machines. The Saltstack framework is the only th…

Mesos itself is only considered as the kernel of a operating system for the datacenter (should also be the message of the article) - not the entirety of what _will_ be. It is Mesos in conjunction with distributed storage, app schedulers, interactive and batch systems for data processing that will make the OS. Just like Mac OS X isn't the Mach kernel alone, but a wide palette of applications that makes the entire OS experience.

Re: Why the data center needs an operating system

#36

Post by res0nat0r »

gtaylor wrote: post #33
23david wrote: post #32

I definitely wouldn't consider Mesos to be anything like an operating system for the datacenter. That's just marketing language and confuses things. Mesos is basically an application scheduler. It doesn't manage the base operating systems or machine provisioning. Mesos is concerned with ensuring that one or multiple applications are launched and running on a cluster of machines. The Saltstack framework is the only th…

> It's the only thing out there that will allow for super low latency response to commands across the cluster. It also can do configuration management etc if you want, or a lot of people use it to just trigger existing chef/puppet jobs. FWIW, Ansible also does low-latency communications through pipelining/accelerated mode. Ditto for the config management and triggering.

I don't believe even with Ansible's accelerated mode it can come anywhere close to Salt for large deployments. It is still agentless and has to run from a single source and using ssh which is going to be slower than 0MQ. Also using something like salt-syndic to create topologies can reliably scale to thousands of machines and still cope with the load.

Re: Why the data center needs an operating system

#37

Post by kaeloby »

Imagine if we ran applications on our laptops the same way we run applications in our data centers. Each time we launched a web browser or text editor, we’d have to specify which CPU to use, which memory modules are addressable, which caches are available, and so on. Thankfully, our laptops have an operating system that abstracts us away from the complexities of manual resource management.

Re: Why the data center needs an operating system

#38

Post by ldng »

Mesos is just one piece of the puzzle. I think the Hadoop project proper will be the first to get there. Maybe v3 or v4. That said, given the diversity under the Hadoop umbrela I expect distribution to appear, like RHEL is a GNU/Linux OS there might be a (Cloudera ?) Hadoop OS.

Re: Why the data center needs an operating system

#39

Post by cdnsteve »

A singular OS that acts on well defined abstractions and interfaces making the number of resources the massive machine uses transparent. Software won't be the only thing changing to support this shift in large scale computing but hardware as well. Would this type of machine be autonomous with little user management? AI could assist the machine in optimizing configuration. Pretty soon it could interface with drones that repair hardware and even add new nodes. Zero configuration environment. The operator simply tells it what it should be running, until Skynet is unleashed.

Re: Why the data center needs an operating system

#40

Post by oaktowner »

ep103 wrote: post #19
preillyme wrote: post #9

Earlier quoted context omitted.

Distributed computing is now the norm, not the exception, and we need a data center operating system that delivers a layer of abstraction and a portable API for distributed applications.

And then we'll need an OS for handling multiple data centers. Its gonna be a crazy day the first time someone realizes they accidentally rebooted 5% of all servers in the western hemisphere.

Maybe it's already happened!
Post reply on HN