Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rolling your own servers with Kubernetes

gravitational.com

Post reply on HN
Print view View original

171–180 of 209 posts

Re: Rolling your own servers with Kubernetes

#171

Post by johngalt »

jasode wrote: post #80
gambler wrote: post #53

>Should you roll your own servers? If you are not certain, the answer is most likely “no”. The staggering growth of AWS happened for a reason. Funny how for many decades companies and people were running their own servers. The hardware was getting cheaper each year. More software became available via open source. Then several ubecorporations entered the hosting/cloud business, and suddenly no one seems to be able to…

>The hardware was getting cheaper each year. More software became available via open source. Then several ubecorporations entered the hosting/cloud business, and suddenly no one seems to be able to afford their own infrastructure. The decision framework the executives use isn't just the "hardware+software" -- it's the whole "IT organization" . In other words, it's not "in-house cpu" vs "Amazon's cpu". It's in-house I…

> For many non-tech companies where IT computing is a cost center, their employees won't be able to match the iteration speed of Google's engineers constantly improving on GCP or Amazon's employees enhancing the features of AWS.

Exactly this. As an ops person, this is exactly how I explain it. Sure we can build something fairly competitive on-prem or in a colo, but we won't have an entire team of top-flight experts bent on improving it as fast as possible. That per-GB cost is buying a lot more than just bandwidth and drive space.

> Over time, the internal IT dept treats the other departments as adversaries instead of customers. Executives get fed up with slow IT departments and get excited when a few clicks on AWS dashboard gets them servers spun up in 10 minutes.

This tends to be more of a problem with organizational mandates & processes. If those processes aren't addressed, then you'll end up with all the same problems. Perhaps they will have different labels on them, but underneath it will be the same issues, delays, outages, and recriminations

Re: Rolling your own servers with Kubernetes

#172

Post by yebyen »

jasode wrote: post #149
pathseeker wrote: post #148

Earlier quoted context omitted.

That's still in datacenters that aren't Amazon's, Google's, etc. It's the most critical component of netflix (the actual video delivery) and it's not "in the cloud".

>That's still in datacenters The point isn't that they are still _in_ datacenters. Yes, of course, they are. Even the "cloud" ultimately resolves down to somebody's datacenter somewhere. The point is that Reed Hastings & Netflix wanted to get out of managing their own datacenters. Putting their Netflix appliances inside of ISP owned datacenters still lets them avoid managing their own datacenters. Their critical user…

The distinction makes important fault domains and whose data centers.

If you are a Comcast customer and your internet goes down, and Netflix is unavailable, who do you complain at? If you're smart enough to notice that both are down, the answer is almost certainly not Netflix. That does not make the Netflix workloads in Comcast data center any less critical, they are core business functions. But they are well aligned with Comcast, who also depends on the proper functioning of those datacenters.

It makes sense for Netflix appliances to be in Comcast datacenters then, especially given that Comcast cannot outsource their data centers any more than the Pentagon can reasonably do so.

Joe Company from off the street can outsource their data centers and derives no competitive advantage from maintaining their own private data centers. Netflix in that sense is closer to Joe Company than they are to Comcast, I guess. I'm not sure what all we can learn from this, but it's interesting.

Re: Rolling your own servers with Kubernetes

#173

Post by jrumbut »

jononor wrote: post #92
diehunde wrote: post #88

So before we had an IT team that would maintain the bare metal servers. Now we need "Cloud Engineers" to maintain the cloud infrastructure working properly. I don't know if the argument of externalize the maintenance of the servers is valid since the complexity of the cloud services is just increasing everyday.

True for those that go for Kubernetes etc straight away. If you just have a few apps to run and put it on a PaaS like Heroku, things are pretty manageable as a side-thing for the regular software developers. No dedicated team, or even person, needed.

When your infrastructure is too simple, you have simple regrets. When your infrastructure is too exotic, you get exciting, bleeding edge regrets.

This is my view based on the work I've done. I like the ability to dispense with capacity planning or dealing with power supplies and fire suppression you get from cloud hosting, but when in any doubt I set up what I need using VMs (be those droplets or EC2 instances or whatever).

I like to imagine that I've avoided wasting weeks by wasting a few hours here and there.

Re: Rolling your own servers with Kubernetes

#174

Post by radium3d »

gambler wrote: post #53

>Should you roll your own servers? If you are not certain, the answer is most likely “no”. The staggering growth of AWS happened for a reason. Funny how for many decades companies and people were running their own servers. The hardware was getting cheaper each year. More software became available via open source. Then several ubecorporations entered the hosting/cloud business, and suddenly no one seems to be able to…

I dunno, you can get a lot of dedicated power for a small amount of money from great hosting companies such as Linode.com.

I have been working with them for the last few years and they've been great. They even roll out new hardware to existing customers for no additional cost.

Blows bad hosting companies out of the water. For example Media Temple who never upgrade their hardware for existing customers at no additional cost.

Re: Rolling your own servers with Kubernetes

#175

Post by linuxdude314 »

ericd wrote: post #109
linuxdude314 wrote: post #39

Earlier quoted context omitted.

Untrue. I purchased a 48 port 10GbE Arista 7148 the other month on eBay for $320.

Nice! OT, but have you seen any reasonably priced RJ45 10GBe out there? Preferably without screaming fans :-)

I haven't. I've also not heard that great of things about it, but we run it in our DCs so I assume it works well enough.

Re: Rolling your own servers with Kubernetes

#176

Post by latch »

ranrotx wrote: post #151

AWS employee here--thoughts and opinions are my own. Prior to AWS, I was in IT Operations at a large financial services company. I saw the writing on the wall that over time, companies would not want to manage this part of their IT infrastructure themselves. Keep in mind, I was someone who was responsible for keeping the lights on for a decent number of Linux severs. For an individual company, there really isn't much…

What seems to often go unsaid in these discussions is that the choice isn't between cloud and colo. There's a third, hugely popular and mature option: dedicated providers - which address all of your issues.

It's convenient for cloud vendors to have people believe the choice is between them or having to deal with hardware.

Re: Rolling your own servers with Kubernetes

#177

Post by ericd »

linuxdude314 wrote: post #175
ericd wrote: post #109

Earlier quoted context omitted.

Nice! OT, but have you seen any reasonably priced RJ45 10GBe out there? Preferably without screaming fans :-)

I haven't. I've also not heard that great of things about it, but we run it in our DCs so I assume it works well enough.

Ah, yeah, looking for something for my home office (hence the preference for no screaming 1U fans), so I wouldn't have very long runs. But I have heard that 10 GbE RJ45 can be finicky.

Re: Rolling your own servers with Kubernetes

#178

Post by scarejunba »

imtringued wrote: post #105
geezerjay wrote: post #51

Earlier quoted context omitted.

Kubernetes has been here for a while. I'm sure we can agree that it ceased to be a candidate for a meteoric fad for a while now.

I've probably wasted more than three weeks installing kubernetes on 3 nodes. I've tried Racher, RKE, kube-admin, kubespray and doing everything myself. I always failed and the way it failed was completely opaque and I didn't feel like I could even understand what was going wrong.

Don't install k8s. Use it managed. It's best that way. Too many moving parts for a small op to manage.

Re: Rolling your own servers with Kubernetes

#179

Post by luckylion »

Daishiman wrote: post #85
segmondy wrote: post #59

I use to enjoy having a cheap desktop under my table or in a closet serving traffic to people across the internet. Computers have gotten faster, software has gotten better, network has gotten faster, things have gotten cheaper. Sadly instead of seeing more of these, most of these are now being outsourced to cloud providers, we have bought and drank the kool-aid that they can do it better and cheap. Which is not true,…

As an ex big iron commercial UNIX sysadmin, let me tell you; they can do it way better than most sysadmins. Developing IT policies for physical security, server security, update policies, purchasing, wiring and so on takes a huge amount of knowledge, and the possibility that a middle-of-the-road sysadmin is not only competent at, but excels, in all those areas is literally zero. Cloud providers have world-class exper…

> As an ex big iron commercial UNIX sysadmin, let me tell you; they can do it way better than most sysadmins.

I'm pretty sure that is true. However, the parent said "they can do it better and cheap", and cheap it isn't.

Re: Rolling your own servers with Kubernetes

#180

Post by porker »

pas wrote: post #121
hn_throwaway_99 wrote: post #107

Earlier quoted context omitted.

I worked at a couple of those self-hosted companies in the 200Xs. For me, as a developer, it was painful and way, way, way worse than AWS or GCP. Planning for big expected spikes of traffic was extremely painful, and once the cloud vendors started adding new services we felt continually behind the times. I can only think it was "interesting" if you've forgotten how difficult and expensive managing your own infrastruc…

It's a false dichotomy (or at least a false equivalence). So, there were a lot of, and still there a lot of corporations running their own stuff. Some of these are still running their own stuff well. Some got big, some stayed small in the past decade (some downsized). It's not necessarily hard to run your own stuff either. Of course, of course, managed platforms are easy (or easier) from a lot of aspects. (That's the…

> it was hard to find good [key job roles] who were able to work together and effectively run their shit

Having had to interact with sysadmins and others, first at at a university and then in business, "work together" is too frequently overlooked.

The crusty, angry and rude stereotype proved right. I've never met a bigger group of downright bastards. Those who were nice people usually couldn't get the servers/software to work reliably. So you were forced back to group 1.

Post reply on HN