Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rolling your own servers with Kubernetes

gravitational.com

Post reply on HN
Print view View original

21–30 of 209 posts

Re: Rolling your own servers with Kubernetes

#21

Post by alexnewman »

lima wrote: post #11
argd678 wrote: post #7

Last physical datacenter my team at the time ran our app in we spent weeks troubleshooting some hardware network driver issues that caused the network to drop. Was an enormous distraction and Dell and VMware support were useless at resolving it for us. I’m glad to have others deal with those low level “oh it must just be your setup” issues.

Anecdotal evidence. I've also seen colleagues spend weeks arguing with AWS support while debugging a weird performance degradation issue, that would have been straight-forward to investigate in a bare metal deployment with full control over everything. It's not like the cloud is a magical place where no unexpected issues ever happen. Cloud providers can be surprisingly buggy, especially AWS, particularly at scale whe…

I've absurd bugs at AWS,GCP and azure. I love dynamic capacity but clouds are super flakey, especially AWS. Azure networks are the worst. GCP I haven't used in over a year so I wont talk ...

Re: Rolling your own servers with Kubernetes

#22

Post by nemonemo »

lima wrote: post #18
nemonemo wrote: post #14

Earlier quoted context omitted.

In my last team, our team suffered from a CPU bug and a SSD bug that caused painful debugging nights. We used multithreading heavily, and it could have been library/kernel issues or some bugs in our code, which cover massive amount of potential target code that needed to sort through initially. I don't think small companies can afford such gigantic debugging effort, so either they should rely on their luck (or flaky…

How is that related to bare metal vs. clouds? That could have happened on either.

Big cloud providers or some big VC backed companies have the resource to figure out the issues by wrestling with the gigantic and unyielding vendors, or by working around it with their engineering resource. Not many teams have the time for even wanting to understand the issues. This is not an inherent problem of bare metal, but one aspect to consider if you were to choose the path. But you are right, cloud providers have their own quirks.

Re: Rolling your own servers with Kubernetes

#23

Post by dilyevsky »

argd678 wrote: post #7

Last physical datacenter my team at the time ran our app in we spent weeks troubleshooting some hardware network driver issues that caused the network to drop. Was an enormous distraction and Dell and VMware support were useless at resolving it for us. I’m glad to have others deal with those low level “oh it must just be your setup” issues.

You wait until you have to debug something like that with a VPC. We had to spend couple days to convince our cloud support the issue was on their end during full site outage.

Re: Rolling your own servers with Kubernetes

#24

Post by argd678 »

closeparen wrote: post #8
argd678 wrote: post #7

Last physical datacenter my team at the time ran our app in we spent weeks troubleshooting some hardware network driver issues that caused the network to drop. Was an enormous distraction and Dell and VMware support were useless at resolving it for us. I’m glad to have others deal with those low level “oh it must just be your setup” issues.

At the scale where owning infrastructure makes sense, that's not a distraction but some team's full-time mission.

We did actually have a team who handles hardware and a networking department, and they couldn’t figure it out. So lots of people had to get involved to do a binary search for the problem. I should mention this is a company with considerable resources.

Re: Rolling your own servers with Kubernetes

#26

Post by lima »

argd678 wrote: post #17
lima wrote: post #11

Earlier quoted context omitted.

Anecdotal evidence. I've also seen colleagues spend weeks arguing with AWS support while debugging a weird performance degradation issue, that would have been straight-forward to investigate in a bare metal deployment with full control over everything. It's not like the cloud is a magical place where no unexpected issues ever happen. Cloud providers can be surprisingly buggy, especially AWS, particularly at scale whe…

I’ve had performance issues with GCP, however I can switch cloud providers in that case without too much effort. With physical hardware that’s much harder.

If you have a well-built and automated bare-metal deployment, it's no harder than switching cloud providers. At the scale where it makes sense to go bare metal, either would be a lot of effort.

An extra abstraction layer like k8s makes it a lot easier, which is exciting. It's also precisely why IBM bought Red Hat - a well-built k8s distribution like OpenShift is one of the very few real alternative to public clouds for many companies.

Re: Rolling your own servers with Kubernetes

#27

Post by alexk »

huangbong wrote: post #3

So what is Gravity? kops/kubespray without an MIT license?

Without MIT.... but with Apache2 ! :)

https://github.com/gravitational/gravity

In all seriousness though, gravity is an open core toolkit to package and deliver complicated sets of micro services in air gapped and restricted environments as a virtual installable appliance.

It takes care of both application and Kubernetes lifecycle, software distribution and licensing workflows.

Re: Rolling your own servers with Kubernetes

#28

Post by ori_b »

argd678 wrote: post #7

Last physical datacenter my team at the time ran our app in we spent weeks troubleshooting some hardware network driver issues that caused the network to drop. Was an enormous distraction and Dell and VMware support were useless at resolving it for us. I’m glad to have others deal with those low level “oh it must just be your setup” issues.

Counterpoint: These issues still exist, you just lose visibility into them by putting someone else in between. And usually, the someone else is big enough that you're not a priority.

Re: Rolling your own servers with Kubernetes

#29

Post by wazoox »

lima wrote: post #11
argd678 wrote: post #7

Last physical datacenter my team at the time ran our app in we spent weeks troubleshooting some hardware network driver issues that caused the network to drop. Was an enormous distraction and Dell and VMware support were useless at resolving it for us. I’m glad to have others deal with those low level “oh it must just be your setup” issues.

Anecdotal evidence. I've also seen colleagues spend weeks arguing with AWS support while debugging a weird performance degradation issue, that would have been straight-forward to investigate in a bare metal deployment with full control over everything. It's not like the cloud is a magical place where no unexpected issues ever happen. Cloud providers can be surprisingly buggy, especially AWS, particularly at scale whe…

Yup, the famous "leaky abstractions"[0]. You can make some reliable assumptions of the data rate of a 40Gbps link in your DC between this and that physical server, and how to get the best out of it. On AWS? Not so much.

0: https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-a...

Re: Rolling your own servers with Kubernetes

#30

Post by webwanderings »

This is basically the "on-prem/hybrid cloud" business which everyone's after (IBM, Google, AWS, MS). The market seems to be pushing towards this model. The old businesses were naturally not moving to "cloud" so the cloud wants to come to their house (which makes sense).
Post reply on HN