Why the data center needs an operating system • Hacker News Board

Re: Why the data center needs an operating system

#61

Post by srean » Thu, Dec 04, 2014, 3:36 AM UTC

mattbee wrote: post #58

The article's premise is a poor introduction to the project. Sure, reinvent MOSIX if you want :) but don't pretend it'll serve more than a niche of a niche. Firstly, "distributed computing is the norm"? It's just not. Most businesses & app authors will never need to care about ultra-distributed computing, with all its problems and trade-offs. You can move faster with "local-only" computing and scale vertically very c…

> Secondly, who cares about "wasted" resources of a whole underutilised server when reliable dedicated servers are so cheap, and in such plentiful supply?

The people bankrolling Google / Facebook / Twitter's electricity bills seem to care quite a bit.

There is another trope that gets repeated often, (and this is not even remotely directed at you, just a digression hopefully somewhat on topic) "performant languages runtimes are an anachronism, a bog slow language in which a programmer can code fast is way more useful than any of the performance bull crap". Typically the person repeating that would a be a webdev. However, in these large scale scenarios core infrastructural code can save orders of magnitude more in money in running costs than saving days in software development. So yeah at the interesting places algorithms and efficiency continue to matter. A reason that Google always managed to be ahead is partly due to how successful it was in minimizing running costs.

Re: Why the data center needs an operating system

#62

Post by mattbee » Thu, Dec 04, 2014, 3:44 AM UTC

srean wrote: post #61

mattbee wrote: post #58

The article's premise is a poor introduction to the project. Sure, reinvent MOSIX if you want :) but don't pretend it'll serve more than a niche of a niche. Firstly, "distributed computing is the norm"? It's just not. Most businesses & app authors will never need to care about ultra-distributed computing, with all its problems and trade-offs. You can move faster with "local-only" computing and scale vertically very c…

> Secondly, who cares about "wasted" resources of a whole underutilised server when reliable dedicated servers are so cheap, and in such plentiful supply? The people bankrolling Google / Facebook / Twitter's electricity bills seem to care quite a bit. There is another trope that gets repeated often, (and this is not even remotely directed at you, just a digression hopefully somewhat on topic) "performant languages ru…

Totally agree on that front. Efficient software = fewer servers. You can't know when to invest in new capacity if you can't tell the difference between hitting the limit of the hardware, and a fixable performance problem. Seemded like Twitter wasn't sure what was going on there for a few years :)

We do a lot of finding ENORMOUS performance problems with customer servers - simple stuff like a thundering herd, a vital missing index or a filesystem that's being overtaxed That's the kind of scale we work at. But those sorts of insights can make the difference between "help we might need a new server" and "oh thank god it's all working again".

Re: Why the data center needs an operating system

#63

Post by sciurus » Thu, Dec 04, 2014, 3:53 AM UTC

vertex-four wrote: post #29

florianleibert wrote: post #2

In light of all the excitement around Docker & containers, it's important to note that Mesos has been using cgroups / LXC (for isolation) since its inception. Mesos paper: http://people.csail.mit.edu/matei/papers/2011/nsdi_mesos.pdf

It also supports Docker as of recently.

For reference

https://mesosphere.com/2014/11/10/docker-on-mesos-with-marat...

https://mesosphere.com/2014/12/03/docker-on-mesos-with-chron...

Re: Why the data center needs an operating system

#64

Post by sciurus » Thu, Dec 04, 2014, 4:01 AM UTC

SEJeff wrote: post #46

ldng wrote: post #38

Mesos is just one piece of the puzzle. I think the Hadoop project proper will be the first to get there. Maybe v3 or v4. That said, given the diversity under the Hadoop umbrela I expect distribution to appear, like RHEL is a GNU/Linux OS there might be a (Cloudera ?) Hadoop OS.

I wouldn't be betting my marbles on Hadoop, or anything Map Reduce now adays. Spark beats it pretty handily for data processing although HBase isn't too bad.

With YARN, Hadoop has built a more generic scheduler comparable to Mesos.

https://www.quora.com/How-does-YARN-compare-to-Mesos

Re: Why the data center needs an operating system

#65

Post by tacticus » Thu, Dec 04, 2014, 4:06 AM UTC

toast0 wrote: post #60

presspot wrote: post #25

Earlier quoted context omitted.

Datacenter and back-end apps simply don't fit on a single machine anymore. Every app of reasonable scale is probably a distributed system of some sort. That, and there are a new class of "apps" (or, more precisely, datacenter services) that were built to operate across fleets of machines from day one, such as Spark, Hadoop, Cassandra, Kafka, Elasticsearch, and so on

On the other hand, a single machine can get pretty big today, for example http://www.supermicro.com/products/system/2U/6028/SYS-6028U-... is 2u, has 12 drive bays and can have 1.5 tb ram. You could do 24 bays if you are using 2.5" drives (most ssds). A lot of complexity can be avoided by getting many fewer, but bigger machines (doesn't help much if you need massive CPU though, but intels architectures are getting bet…

The problem with this is that singular systems require a lot of engineering effort and produce interesting side effects when you attempt to make them reliable.

Any singular system is going to fail.

Re: Why the data center needs an operating system

#66

Post by devonkim » Thu, Dec 04, 2014, 4:06 AM UTC

mattzito wrote: post #47

chuhnk wrote: post #26

It's interesting and great to see this stuff take center stage more and more. Those lucky enough to work at places like Twitter, Google, Facebook and other large tech companies will have already seen how this kind of thing dominates the datacenter's there and has been at the core of their systems for many years. People on the outside though are rarely exposed to this concept of datacenter scale computing aside from t…

> Google SREs by last count were 1 engineer to 1000 machines That number does not seem particularly impressive, if it is accurate. Even "traditional" well-run enterprise IT organizations are often in the 1 admin/SRE to 600-ish machines, so I have a hard time seeing that Google can only do ~2x as good at their scale and with their level of focus. 1 SRE to 5k machines, 10k machines, that makes more sense to me.

I don't know which kinds of enterprise orgs you're envisioning but when I think about "traditional" enterprise IT orgs, I'm picturing companies in healthcare, insurance, certain finance business units, non-profits, public sector, and defense, not any place ever mentioned a lot here on HN in a technical discussion. I've worked with a LOT of them and I'd be surprised if the ratio of ops engineers to servers was anything better than 1:10 as a ball-park number. Network guy, storage guy, DB guy, VM guy, automation guy, net-sec guy, system-sec guy... that's pretty typical for maybe a 20-server rollout and grows shy of sub-linearly with number of users (wild guess of O(n^1/2) ). If you mean specific to SRE-type of roles or auto-scaled groups in EC2 / GCE to count as servers perhaps that may be the case, but my idea of a traditional enterprise IT org is that it views automation with fear and hesitation preferring to add more people before trying to "disrupt" their business with automation tools and so will be stuck somewhere around 1:30 at best.

Admittedly, I probably have a bad impression of enterprise from consulting because who would pay for automation consulting at $$$ / hr when you do it pretty well with existing resources in the first place?

Re: Why the data center needs an operating system

#67

Post by toast0 » Thu, Dec 04, 2014, 4:14 AM UTC

tacticus wrote: post #65

toast0 wrote: post #60

Earlier quoted context omitted.

On the other hand, a single machine can get pretty big today, for example http://www.supermicro.com/products/system/2U/6028/SYS-6028U-... is 2u, has 12 drive bays and can have 1.5 tb ram. You could do 24 bays if you are using 2.5" drives (most ssds). A lot of complexity can be avoided by getting many fewer, but bigger machines (doesn't help much if you need massive CPU though, but intels architectures are getting bet…

The problem with this is that singular systems require a lot of engineering effort and produce interesting side effects when you attempt to make them reliable. Any singular system is going to fail.

Ok, get two huge systems instead of a datacenter of small systems. You still have made great strides in reducing complexity.

Re: Why the data center needs an operating system

#68

Post by superbaconman » Thu, Dec 04, 2014, 5:05 AM UTC

I feel like this is on the right track, but sadly fails to address the underlying problem. Until the network is virtualized everything else will suffer. If the cloud is to become the default platform for services (I think it will), then the user must be able to define their own virtual data center. This includes their own virtual nets. What the author writes about will be built upon these virtual nets, but until the network itself is able to be programmed we'll likely be drifting between optimal states of computing.

Re: Why the data center needs an operating system

#69

Post by florianleibert » Thu, Dec 04, 2014, 5:26 AM UTC

mattbee wrote: post #58

The article's premise is a poor introduction to the project. Sure, reinvent MOSIX if you want :) but don't pretend it'll serve more than a niche of a niche. Firstly, "distributed computing is the norm"? It's just not. Most businesses & app authors will never need to care about ultra-distributed computing, with all its problems and trade-offs. You can move faster with "local-only" computing and scale vertically very c…

Arguably, most systems are distributed systems today. Even if you're just connecting a Rails app with a MySQL DB.

If we had a fabric (which is called Mesos ;) which allowed you to write elastic, distributed systems without the need for worrying about interconnecting hosts and segmenting hosts into static partitions, etc. wouldn't that be a big win?

Spark is also a great example for an app that was built directly on top of Mesos - the authors could focus on implementing the actual logic rather than worrying about interconnection. Same is true by the way for systems like Chronos and Marathon - all of which run on top of Mesos.

Re: Why the data center needs an operating system

#70

Post by florianleibert » Thu, Dec 04, 2014, 5:28 AM UTC

mattzito wrote: post #47

chuhnk wrote: post #26

It's interesting and great to see this stuff take center stage more and more. Those lucky enough to work at places like Twitter, Google, Facebook and other large tech companies will have already seen how this kind of thing dominates the datacenter's there and has been at the core of their systems for many years. People on the outside though are rarely exposed to this concept of datacenter scale computing aside from t…

> Google SREs by last count were 1 engineer to 1000 machines That number does not seem particularly impressive, if it is accurate. Even "traditional" well-run enterprise IT organizations are often in the 1 admin/SRE to 600-ish machines, so I have a hard time seeing that Google can only do ~2x as good at their scale and with their level of focus. 1 SRE to 5k machines, 10k machines, that makes more sense to me.

Twitter recently stated they had roughly 1 SRE / 10K nodes.