Live data from Hacker News

Rolling your own servers with Kubernetes

gravitational.com

151–160 of 209 posts

Re: Rolling your own servers with Kubernetes

#151
AWS employee here--thoughts and opinions are my own.

Prior to AWS, I was in IT Operations at a large financial services company. I saw the writing on the wall that over time, companies would not want to manage this part of their IT infrastructure themselves. Keep in mind, I was someone who was responsible for keeping the lights on for a decent number of Linux severs.

For an individual company, there really isn't much value in having to maintain firmware levels on all your hardware, patch hypervisors (and try to coordinate all of the maintenance around a fixed pool of hardware), perform months-long evaluation of new hardware before purchasing, test and validate configurations on new hardware, etc. I used to do all of this. I don't miss it either.

Yes, the items above are important, but doing them right is really table-stakes for any reliable IT Operations department. You can choose to spend time getting these right, or delegate that responsibility to a service provider whose main job is to get that stuff right (and recoups that cost across a much larger customer base).

Re: Rolling your own servers with Kubernetes

#152

Earlier quoted context omitted.

As your operation gets bigger, the scale generally starts to tip in favor of running your own hardware. But 600 SQL Server instances is fairly large—when that number is in the single digits, the cost of DBAs/DevOps dwarfs that of the hardware. So the scale of your business matters a lot, and it's not useful to generalize this to all companies. Edit: Your math seems off as well. Don't forget that you'd have to pay the…

One last thing to consider, even with a small team, is that without at least one DBA you may end up burning money by deploying non-optimized, non-performant database workloads, and using Amazon's very lucrative "throw money at the problem" service to buy your way out.

This is very true, but my point still stands: For a smaller team, the overhead of maintaining your own infrastructure is very significant compared to the resources you have. If you have, say, 10 engineers, having 3 of them doing DevOps fulltime means 30% of your engineering spending is going towards your infrastructure, even before we factor in capital costs and all of the other overhead—not to mention room for error—that comes with it (security, etc). And if you're small enough to only be able to afford 10 engineers, you're most likely not going to spend 3 engineers' salaries on the equivalent AWS services. So now you only need maybe a single DevOps person at most, and you get another engineer or maybe two for "free" that can work on your core product. I'm not saying it doesn't come with its own disadvantages, but there are a lot of benefits to going "all in" on cloud services.

And obviously there are no absolutes—there are certain workloads that really do benefit more from dedicated hardware, even at smaller scales, and there are huge companies that are better off with cloud services (ex: Netflix).

Re: Rolling your own servers with Kubernetes

#153
post #12

I am curious to see how they recommend handling storage on bare metal k8s.

I highly recommend Rook [1], which is based on Ceph, provides PersistentVolumes to k8s workloads and is also running on k8s itself. [1] - https://rook.io/

Big fan of rook myself. We took several nodes loaded with 16 disk drives and used Rook to make a Ceph and Minio cluster out of it. Great storage solution.

Re: Rolling your own servers with Kubernetes

#154

Earlier quoted context omitted.

Which is a valid argument, isn't it? Engineers are expensive! Whether they're more expensive than AWS depends entirely on what you're doing, but the answer is almost always "yes" at a small enough scale.

See my other comment.

At least link it

Re: Rolling your own servers with Kubernetes

#155
post #102
post #47

Earlier quoted context omitted.

But StatefulSets require storage...someplace. We're dealing with that exact issue right now. Our K8s cluster is not playing nice with a NetApp NAS. We are exploring other options such as local iscsi storage volumes ($$$) or an external state store db (slow).

The point is that with native HA, it doesn't have to be distributed storage. You can use locally attached SSDs or NVMe drives, which are straight forward to manage and fast.

Importantly, they also have independent failure modes. I’ve never known a SAN to be within an order of magnitude of the advertised uptime — and some interesting failures have been caused by HA features. Local storage avoids whole classes of problem and has much greater aggregate performance if you can make it work for your app.

Re: Rolling your own servers with Kubernetes

#156
I’ve done this for small projects using AWS and Scaleway.

The setup is simple and it just works, but the maintenance and security aspects were more worrying. One time I found out that one of my nodes failed to automatically apply security patches from my distro. I couldn’t detect it sooner because my monitoring was lacking. The overhead escalates fast.

If I start an effort like this again today, I would make sure to get monitoring right from day 1. Possibly a combination of Prometheus with Grafana, or something along the lines of ElasticSearch APM.

Re: Rolling your own servers with Kubernetes

#158
post #96
post #91

Earlier quoted context omitted.

>Most companies are not like Facebook or Dropbox that can maintain an internal IT organization at a high level equivalent to AWS. Let's try this with different phrasing. In 200X a lot of companies were able to maintain their own infrastructure, just like Facebook and Amazon did at the time. Forward 10-13 years. We have cheaper hardware. We have extra 10+ years of development in open-source software. And yet that list…

>In 200X lots of companies were able to maintain their own infrastructure, just like Facebook and Amazon did at the time. But my point is that companies' IT departments did not maintain infrastructure just like Amazon did. In ~2005 when companies were first experimenting with AWS cloud, they might start with dev & test servers. They click a few buttons and are amazed when new servers get spun up in minutes and their…

The Netflix example is a poor one. They have a friggin fleet of CDN-boxes around the world that are as locally close as possible to eye-balls. To say they are "100%" on AWS is disingenuous - they still have their own hardware deployed all around the world - just closer to the edge.

AWS is used for control-plane / management-plane tasking - but the bits are pushed from locally-peered CDN boxes at all kinds of ISPs.

Re: Rolling your own servers with Kubernetes

#159
post #68
post #53

>Should you roll your own servers? If you are not certain, the answer is most likely “no”. The staggering growth of AWS happened for a reason. Funny how for many decades companies and people were running their own servers. The hardware was getting cheaper each year. More software became available via open source. Then several ubecorporations entered the hosting/cloud business, and suddenly no one seems to be able to…

I worked in a shop that had dedicated lab equipment and resources. Suddenly buying new equipment that we would use for years became like pulling teeth, no money they said. They couldn't explain what changed. Then just as suddenly they were willing to approve spending more than our one time equipment purchase... per month in monthly cloud costs / some poorly thought out cloud deployments.... Result: -1 Dedicated perso…

Big company politics my friend.

I was just told to find a way to move 1M of capex to opex in the next 6 months for my project. Dumping shit into Cloud Provider X instead of buying our own stuff just because? Yeah, I agree, short-sighted. But then again everything in life is technically short-sighted on a long enough timeline.

Re: Rolling your own servers with Kubernetes

#160
post #2

I used to specialize in 2 - 10 rack build outs, but its been nearly a decade since those gigs evaporated. I wonder if k8s creates a new niche for it.

These kind of gigs are definitely nowhere near as common as they once were, but they haven't evaporated completely - I work in emergency services infrastructure in the process of a 10 rack build and that is something I don't see changing in the next 10 years, where we will do a refresh possibly once, or twice more.

The complexity in our case is in the backend transmission network and unless we control everything (hardware, network) end-to-end as much as possible, we can't guarantee to getting close meet our SLA's for our customer. They wouldn't allow us to run our system in the cloud even if we wanted to.

Post reply on HN