Rolling your own servers with Kubernetes • Hacker News Board

Re: Rolling your own servers with Kubernetes

#91

Post by gambler » Mon, May 06, 2019, 6:37 PM UTC

jasode wrote: post #80

gambler wrote: post #53

>Should you roll your own servers? If you are not certain, the answer is most likely “no”. The staggering growth of AWS happened for a reason. Funny how for many decades companies and people were running their own servers. The hardware was getting cheaper each year. More software became available via open source. Then several ubecorporations entered the hosting/cloud business, and suddenly no one seems to be able to…

>The hardware was getting cheaper each year. More software became available via open source. Then several ubecorporations entered the hosting/cloud business, and suddenly no one seems to be able to afford their own infrastructure. The decision framework the executives use isn't just the "hardware+software" -- it's the whole "IT organization" . In other words, it's not "in-house cpu" vs "Amazon's cpu". It's in-house I…

>Most companies are not like Facebook or Dropbox that can maintain an internal IT organization at a high level equivalent to AWS.

Let's try this with different phrasing. In 200X a lot of companies were able to maintain their own infrastructure, just like Facebook and Amazon did at the time. Forward 10-13 years. We have cheaper hardware. We have extra 10+ years of development in open-source software. And yet that list of self-hosting companies shrunk by a huge degree. Doesn't that seem interesting?

Re: Rolling your own servers with Kubernetes

#92

Post by jononor » Mon, May 06, 2019, 6:39 PM UTC

diehunde wrote: post #88

So before we had an IT team that would maintain the bare metal servers. Now we need "Cloud Engineers" to maintain the cloud infrastructure working properly. I don't know if the argument of externalize the maintenance of the servers is valid since the complexity of the cloud services is just increasing everyday.

True for those that go for Kubernetes etc straight away. If you just have a few apps to run and put it on a PaaS like Heroku, things are pretty manageable as a side-thing for the regular software developers. No dedicated team, or even person, needed.

Re: Rolling your own servers with Kubernetes

#93

Post by acdha » Mon, May 06, 2019, 6:42 PM UTC

kortilla wrote: post #65

acdha wrote: post #61

Earlier quoted context omitted.

> The hardware was getting cheaper each year. More software became available via open source. Then several ubecorporations entered the hosting/cloud business, and suddenly no one seems to be able to afford their own infrastructure. That conspiracy theory depends on not asking about the major costs areas you left out (e.g. ops, security, reliability) which are the most significant until you're at a fairly large scale.…

Ops, security, and reliability don’t go away when using EC2 either, which is what led the massive cloud explosion. Places switched because they didn’t like having to forecast capacity so hourly billed resources turned on and off at the drop of a hat were super attractive.

They don't go away but you are generally focused a lot more on your apps and you have better tools for many problems, especially because you're getting out of the lower-level stuff which is harder to work with and full of pitfalls. If you're running your own datacenter, you're going to pay for the team of people supporting and debugging basic things — power & cooling, network infrastructure, storage, routing & load-balancing, server hardware & firmware, etc. creating APIs for all of that for your application teams to use, performance and security testing all of the above, etc. in multiple locations. If you haven't done that before, it's really easy to underestimate how many little things will soak up expensive staff time, especially because so much enterprise hardware was designed on the theory that you'd have humans doing all of the work with automation either an afterthought or a perceived chance to sell $$$ add-ons.

Beyond a certain level of scale you can find efficiencies which pay for all of that but many places aren't at the level where it's unquestioned win, and doing it yourself means you have to start paying the full cost immediately in the hopes that it'll become a win in the future.

Re: Rolling your own servers with Kubernetes

#94

Post by technological » Mon, May 06, 2019, 6:42 PM UTC

gambler wrote: post #53

>Should you roll your own servers? If you are not certain, the answer is most likely “no”. The staggering growth of AWS happened for a reason. Funny how for many decades companies and people were running their own servers. The hardware was getting cheaper each year. More software became available via open source. Then several ubecorporations entered the hosting/cloud business, and suddenly no one seems to be able to…

I think one of the issue is availability or distributed data centers. Its indeed cheaper to buy hardware and setup server but to ensure you have datacenters in many locations is the problem.

Re: Rolling your own servers with Kubernetes

#95

Post by lixtra » Mon, May 06, 2019, 6:50 PM UTC

gambler wrote: post #91

jasode wrote: post #80

Earlier quoted context omitted.

>The hardware was getting cheaper each year. More software became available via open source. Then several ubecorporations entered the hosting/cloud business, and suddenly no one seems to be able to afford their own infrastructure. The decision framework the executives use isn't just the "hardware+software" -- it's the whole "IT organization" . In other words, it's not "in-house cpu" vs "Amazon's cpu". It's in-house I…

>Most companies are not like Facebook or Dropbox that can maintain an internal IT organization at a high level equivalent to AWS. Let's try this with different phrasing. In 200X a lot of companies were able to maintain their own infrastructure, just like Facebook and Amazon did at the time. Forward 10-13 years. We have cheaper hardware. We have extra 10+ years of development in open-source software. And yet that list…

Ok, I will try to counter that. The stack grows higher. More specialized employees become more effective. In 200X you just needed a Linux admin. Now on top of being good a good Linux admin you also need to know docker, kubernetes (the admin part, not the user part) etc. If you don’t need that daily, because you also need to maintain some apps, you will be less professional than a cloud provider.

Of course it’s a trade-off.

If you’re large, with constant high resource needs and benefit from custom optimization, you may be better off with your own team/infrastructure.

If you’re small (2 it guys), fast changing, normal speed requirements / easy to scale out then the cloud should be interesting to you.

Re: Rolling your own servers with Kubernetes

#96

Post by jasode » Mon, May 06, 2019, 6:50 PM UTC

gambler wrote: post #91

jasode wrote: post #80

Earlier quoted context omitted.

>The hardware was getting cheaper each year. More software became available via open source. Then several ubecorporations entered the hosting/cloud business, and suddenly no one seems to be able to afford their own infrastructure. The decision framework the executives use isn't just the "hardware+software" -- it's the whole "IT organization" . In other words, it's not "in-house cpu" vs "Amazon's cpu". It's in-house I…

>Most companies are not like Facebook or Dropbox that can maintain an internal IT organization at a high level equivalent to AWS. Let's try this with different phrasing. In 200X a lot of companies were able to maintain their own infrastructure, just like Facebook and Amazon did at the time. Forward 10-13 years. We have cheaper hardware. We have extra 10+ years of development in open-source software. And yet that list…

>In 200X lots of companies were able to maintain their own infrastructure, just like Facebook and Amazon did at the time.

But my point is that companies' IT departments did not maintain infrastructure just like Amazon did.

In ~2005 when companies were first experimenting with AWS cloud, they might start with dev & test servers. They click a few buttons and are amazed when new servers get spun up in minutes and their programmers are productive immediately. The natural question that company execs ask is, "why can't our own internal IT department spin up servers for us in 10 minutes?!? Why does it take them so damn long?!?"

They wouldn't have asked those hard questions if their internal IT capability was equivalent to AWS. Eventually, their improved experiences with AWS on Dev&Test&QA convinced them to migrate mission-critical Production workloads to AWS as well.

>We have cheaper hardware. We have (supposedly) better software.

You're still focusing on on hardware+software and not considering the IT employees' speed of execution in how company executives compare the situation.

Even Netflix as a tech company maintained their own datacenters for over 10 years but ended up migrating to AWS. Their "Guardian" moment was a big database corruption in 2008. They also had ambitious plans to expand into countries outside of USA. Those were some of the factors that convinced them they didn't want to invest anymore in their own datacenters and preferred AWS take care of it. Amazon employees iterated on datacenter capabilities faster than Netflix engineers could do it.

Re: Rolling your own servers with Kubernetes

#97

Post by dwild » Mon, May 06, 2019, 6:54 PM UTC

gambler wrote: post #91

jasode wrote: post #80

Earlier quoted context omitted.

>The hardware was getting cheaper each year. More software became available via open source. Then several ubecorporations entered the hosting/cloud business, and suddenly no one seems to be able to afford their own infrastructure. The decision framework the executives use isn't just the "hardware+software" -- it's the whole "IT organization" . In other words, it's not "in-house cpu" vs "Amazon's cpu". It's in-house I…

>Most companies are not like Facebook or Dropbox that can maintain an internal IT organization at a high level equivalent to AWS. Let's try this with different phrasing. In 200X a lot of companies were able to maintain their own infrastructure, just like Facebook and Amazon did at the time. Forward 10-13 years. We have cheaper hardware. We have extra 10+ years of development in open-source software. And yet that list…

His whole comment was about how it's not about hardware cost, but staff cost.

> It's in-house IT employees' speed of tech innovation vs Amazon's engineers'

The cloud shrunk the cost of IT team. Sure hardware is cheaper, but that cost is marginal versus employee cost.

Re: Rolling your own servers with Kubernetes

#98

Post by lima » Mon, May 06, 2019, 6:55 PM UTC

humbleMouse wrote: post #74

windexh8er wrote: post #37

Earlier quoted context omitted.

> An extra abstraction layer like k8s makes it a lot easier, which is exciting. It's also precisely why IBM bought Red Hat - a well-built k8s distribution like OpenShift is one of the very few real alternative to public clouds for many companies. OpenShift is built on a prescriptive mentality. They choose everything for you from OS through pipeline and many deviations are unsupported. As you mentioned there are other…

Openshift doesn't choose the pipeline. You can deploy to it however you want. Openshift is literally the same exact thing as kubernetes, with a simpler api and a nice ui. Openshift is extremely flexible and you can use it however you want. Don't know why you think it's so restrictive.

It has the same API as k8s. It's literally the same thing, OpenShift just adds some PaaS feature on top of k8s.

Re: Rolling your own servers with Kubernetes

#99

Post by acdha » Mon, May 06, 2019, 6:55 PM UTC

gambler wrote: post #91

jasode wrote: post #80

Earlier quoted context omitted.

>The hardware was getting cheaper each year. More software became available via open source. Then several ubecorporations entered the hosting/cloud business, and suddenly no one seems to be able to afford their own infrastructure. The decision framework the executives use isn't just the "hardware+software" -- it's the whole "IT organization" . In other words, it's not "in-house cpu" vs "Amazon's cpu". It's in-house I…

>Most companies are not like Facebook or Dropbox that can maintain an internal IT organization at a high level equivalent to AWS. Let's try this with different phrasing. In 200X a lot of companies were able to maintain their own infrastructure, just like Facebook and Amazon did at the time. Forward 10-13 years. We have cheaper hardware. We have extra 10+ years of development in open-source software. And yet that list…

> In 200X a lot of companies were able to maintain their own infrastructure, just like Facebook and Amazon did at the time.

I would question that equivalence. Speaking as someone who was trying to get what we now call devops going at the time, there were a _few_ companies starting big wins from automation and a ton of places which were content pouring huge amounts of human time into doing it the hard way and slow metrics for time to patch, provision new services, recover after failures, etc. When they had automation, it was known-bad practices like cloning VMs and had accordingly greater cost and lower benefits.

The companies in the latter groups were the ones who were faced with either spending a huge amount of time and money catching up or cutting a cloud contract and not having do a large percentage of that remediation work at all, while being able to deliver results immediately.

Re: Rolling your own servers with Kubernetes

#100

Post by hotsauceror » Mon, May 06, 2019, 6:56 PM UTC

rhinoceraptor wrote: post #69

gambler wrote: post #53

>Should you roll your own servers? If you are not certain, the answer is most likely “no”. The staggering growth of AWS happened for a reason. Funny how for many decades companies and people were running their own servers. The hardware was getting cheaper each year. More software became available via open source. Then several ubecorporations entered the hosting/cloud business, and suddenly no one seems to be able to…

The hardware isn’t the expensive part, it’s the people to run it that is.

Using publicly available costing data from last November, I see that an m4.xlarge RDS reserved instance of SQL Server Enterprise Edition with multi-AZ failover is $39,000 / year. If I have 4 DBAs at a cost of $150K / yr salary + benefits, that's the equivalent of around 15 smallish, managed SQL Server instances. My current SQL footprint is on the order of 600 instances, which works out to be a bit over $23M a year. It is true that I've saved the $600K on the expensive personnel, though...