Live data from Hacker News

Rolling your own servers with Kubernetes

gravitational.com

121–130 of 209 posts

Re: Rolling your own servers with Kubernetes

#121
post #91

Earlier quoted context omitted.

>Most companies are not like Facebook or Dropbox that can maintain an internal IT organization at a high level equivalent to AWS. Let's try this with different phrasing. In 200X a lot of companies were able to maintain their own infrastructure, just like Facebook and Amazon did at the time. Forward 10-13 years. We have cheaper hardware. We have extra 10+ years of development in open-source software. And yet that list…

I worked at a couple of those self-hosted companies in the 200Xs. For me, as a developer, it was painful and way, way, way worse than AWS or GCP. Planning for big expected spikes of traffic was extremely painful, and once the cloud vendors started adding new services we felt continually behind the times. I can only think it was "interesting" if you've forgotten how difficult and expensive managing your own infrastruc…

It's a false dichotomy (or at least a false equivalence).

So, there were a lot of, and still there a lot of corporations running their own stuff. Some of these are still running their own stuff well. Some got big, some stayed small in the past decade (some downsized).

It's not necessarily hard to run your own stuff either.

Of course, of course, managed platforms are easy (or easier) from a lot of aspects. (That's their value offering after all.) And the IT expertise market was never exceptionally great in a lot of places. Meaning it was hard to find good sysadmins, good ops people, good devops folks, good programmers, leads, tech PMs, POs, etc. who were able to work together and effectively run their shit, build their product/service, run with the flow (of open source or whatever vendor they used), and so on.

AWS/GCP/Azure is a big paradigm change. No longer do you need to argue about what and how, at best you pick one of the three, and that's it. No longer you need to think about hardware, colo, uplink, switches, peering, multi-homing, picking the right dedicated server provider, and so on. (Sure, you need network guys, who at least have some basic understanding of VPC - overlay networking, but the basics are easy, and the rest is YAGNI anyway.) And this is due to their size. They are big enough, that their price premium - compared even to whatever random small hosting provider you might find on the 'net - is irrelevant, because they are usually more efficient, more secure, more geo-available than the small ones.

But, ... of course, while generalizing all self-hosted companies (meaning that all self-hosted is a pain) leads to a falsehood, on the margin, especially during the IT consolidation of the past years, most companies moved away from self-hosting. And developers usually rejoiced. And similarly, generalizing the problem of self-hosting from your experiences again leads to a flawed conclusion (that it's necessarily hard and expensive to manage your infra), on average, outsourcing a cost-center in an ever increasing complexity world (that is potentially unbounded costs) and focusing on profit-centers is a sane decision.

Re: Rolling your own servers with Kubernetes

#122

Earlier quoted context omitted.

Using publicly available costing data from last November, I see that an m4.xlarge RDS reserved instance of SQL Server Enterprise Edition with multi-AZ failover is $39,000 / year. If I have 4 DBAs at a cost of $150K / yr salary + benefits, that's the equivalent of around 15 smallish, managed SQL Server instances. My current SQL footprint is on the order of 600 instances, which works out to be a bit over $23M a year. I…

As your operation gets bigger, the scale generally starts to tip in favor of running your own hardware. But 600 SQL Server instances is fairly large—when that number is in the single digits, the cost of DBAs/DevOps dwarfs that of the hardware. So the scale of your business matters a lot, and it's not useful to generalize this to all companies. Edit: Your math seems off as well. Don't forget that you'd have to pay the…

One last thing to consider, even with a small team, is that without at least one DBA you may end up burning money by deploying non-optimized, non-performant database workloads, and using Amazon's very lucrative "throw money at the problem" service to buy your way out.

Re: Rolling your own servers with Kubernetes

#123
post #96
post #91

Earlier quoted context omitted.

>Most companies are not like Facebook or Dropbox that can maintain an internal IT organization at a high level equivalent to AWS. Let's try this with different phrasing. In 200X a lot of companies were able to maintain their own infrastructure, just like Facebook and Amazon did at the time. Forward 10-13 years. We have cheaper hardware. We have extra 10+ years of development in open-source software. And yet that list…

>In 200X lots of companies were able to maintain their own infrastructure, just like Facebook and Amazon did at the time. But my point is that companies' IT departments did not maintain infrastructure just like Amazon did. In ~2005 when companies were first experimenting with AWS cloud, they might start with dev & test servers. They click a few buttons and are amazed when new servers get spun up in minutes and their…

netflix does not depend on aws to deliver their video bits. they do it themselves based on a big network of core and edge pni and caches. this infra, that netflix has built, in datacenters, is a big competitive advantage. it would be worse performance and much higher cost to do this over something like aws cloudfront.

Re: Rolling your own servers with Kubernetes

#124
post #11

Earlier quoted context omitted.

Anecdotal evidence. I've also seen colleagues spend weeks arguing with AWS support while debugging a weird performance degradation issue, that would have been straight-forward to investigate in a bare metal deployment with full control over everything. It's not like the cloud is a magical place where no unexpected issues ever happen. Cloud providers can be surprisingly buggy, especially AWS, particularly at scale whe…

> spend weeks arguing with AWS support Yup. We've run into that repeatedly. The "we didn't notice anything on our side, please send more screenshots and logs" gets really old when working with "managed services". Network packet losses/truncations, EKS control plane failures, cloudformation stacks getting stuck in really wierd states, inconsistent cloudformation implementations for new and existing services, ENI weird…

My experience with AWS support for performance issues is similar. My ticket for an Aurora RDS performance issue ended up in a kind of purgatory where they continued to ask for more information and more tests on my side, continued, to infinity. Once one support engineer was satisfied with my results, the ticket would go dark for months, and when I would follow up, another (new) engineer would ask for a whole new round of tests, logs, etc., typically for the latest release.

After a few rounds of back-and-forth, I eventually gave up.

(I even gave them permission to copy test snapshot data, at their request, and sample queries to reproduce the issue. They could have done all the testing they wanted in-house. But that might have actually required them to spend time on their side to diagnose and fix something, I guess.)

It wasn't a critical issue, exactly, and we worked around it in code. I mostly just wanted to report a problem.

Re: Rolling your own servers with Kubernetes

#125

Earlier quoted context omitted.

Is switching to Terraform an option?

Based on our own research - probably. But it requires tooling to integrate into a CI/CD pipeline, conversion of existing CF stack definitions to Terraform definitions, and a lossy/downtime laden migration. For a team with the goals of IAC, hands-off, and high-availability infrastructure, migrating is a pain.

With Terraform's import functionality, you might be able to migrate into Terraform without any downtime. I believe there are even utilities to translate CloudFormation code straight into Terraform, though I've never tried them.

It would definitely be an annoying migration, even if import works for your use-case, but honestly, dealing with CloudFormation is so irritating on a day-to-day basis for me that I'd consider it worth it.

Re: Rolling your own servers with Kubernetes

#126

Earlier quoted context omitted.

Ops, security, and reliability don’t go away when using EC2 either, which is what led the massive cloud explosion. Places switched because they didn’t like having to forecast capacity so hourly billed resources turned on and off at the drop of a hat were super attractive.

> Ops, security, and reliability don’t go away when using EC2 either They don't go away, but you have to spend considerably fewer resources on it because Amazon does a lot of it for you. Edit: This effect is even more pronounced when you use Amazon's more specialized managed services, like S3, RDS, Lambda, etc. Then the things you mentioned almost completely go away, as you're letting Amazon completely maintain your…

What used to be behind firewalls oftentimes became accidentally public through the many, many blunders of publicly available S3 buckets that people started to use because they were frustrated with AWS IAM. This is ironic because IAM was supposed to help you secure your data and such, not make things worse for your security profile. Even with all the tooling that exists to help people write better IAM policies and figure things out, cross-account IAM permissions and roles, S3 bucket policies, etc. are a nightmare for most people beyond fairly seasoned engineers in AWS and that's a problem. Secure-by-default is not quite 100% true with most AWS services in practice unfortunately.

Re: Rolling your own servers with Kubernetes

#127

Earlier quoted context omitted.

Meanwhile the market and mindshare has moved on to the next shiny new tech.

Kubernetes has been here for a while. I'm sure we can agree that it ceased to be a candidate for a meteoric fad for a while now.

Openstack had similar hype (though more on the systems side than the developers), and it seems to finally be fizzling out.

Re: Rolling your own servers with Kubernetes

#129
post #98

Earlier quoted context omitted.

Openshift doesn't choose the pipeline. You can deploy to it however you want. Openshift is literally the same exact thing as kubernetes, with a simpler api and a nice ui. Openshift is extremely flexible and you can use it however you want. Don't know why you think it's so restrictive.

It has the same API as k8s. It's literally the same thing, OpenShift just adds some PaaS feature on top of k8s.

Which is precisely why I've run it for over 2 years now. The devs on my team love throwing a git URL into OpenShift and have a project go live in mere minutes, previously a more senior developer would need to setup a CI pipeline and deployment automation and even with experience it was a chore.

It's not like I'm personally afraid of k8s or building docker images myself either, but anything to help my team be more productive without needing to hold hands or do gruntwork on their behalf improves the lives of everyone.

Re: Rolling your own servers with Kubernetes

#130
post #26
post #17

Earlier quoted context omitted.

I’ve had performance issues with GCP, however I can switch cloud providers in that case without too much effort. With physical hardware that’s much harder.

If you have a well-built and automated bare-metal deployment, it's no harder than switching cloud providers. At the scale where it makes sense to go bare metal, either would be a lot of effort. An extra abstraction layer like k8s makes it a lot easier, which is exciting. It's also precisely why IBM bought Red Hat - a well-built k8s distribution like OpenShift is one of the very few real alternative to public clouds f…

You’re forgetting the network needs to be replaced too, there’s lead times on getting hardware, there the whole racking and cabling, and even at that if the this is due to a more complex networking issue then I can’t swap out the network since there are too many components and they’re shared by other teams. In the cloud I can do this quickly tho.
Post reply on HN