Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The full-time job of keeping up with Kubernetes

gravitational.com

Post reply on HN
Print view View original

71–80 of 237 posts

Re: The full-time job of keeping up with Kubernetes

#71

Post by macNchz »

> The absolute safest place to run Kubernetes application is still Google’s GKE.

Interesting to read given I recently had a GKE cluster auto-upgrade its master version from 1.6.x to 1.7.x at 8pm one night (however foolish it was to not be subscribed to the release notes RSS Feed[1]), which somehow caused a cascading nightmare of things breaking.

Logs stopped appearing in the GCP logging interface and in our own log parsing pipeline, a bug related to a change in the format of the yaml specifications meant that all the containers got stuck in a broken limbo state as we tried to upgrade the nodes (yay for googling the error and finding open github issues!), and then once we'd manually fixed all of our deployments and were finally able to get our nodes rolled over to the new k8s version, all of our load balancers started intermittently timing out until they were deleted and recreated.

Surely we need to keep a closer eye on the release cycle, and we're guilty of bandwagonning onto this cool new tech, but boy does it suck to get auto-upgraded at night only to discover several breaking changes while in emergency mode trying to fix things.

(1) https://cloud.google.com/kubernetes-engine/release-notes

Re: The full-time job of keeping up with Kubernetes

#72

Post by ionforce »

flukus wrote: post #66
enraged_camel wrote: post #25

Earlier quoted context omitted.

Sure, NIH syndrome can be rational for technology that is crucial/central to your system. The problem is it is often used to justify re-inventing even mundane stuff. I once worked with a client who wanted to implement their own bug tracking system. The client’s main product was something totally unrelated.

Bug trackers are one of those areas where a lot of companies should build there own. Everyone has there own workflows and information to capture and end up either conforming their process to the bug tracker or spending more time configuring the bug tracker than they would to build a new system from scratch. Those uber configurable systems always suck to use. Ones like JIRA can takes weeks to setup for your org and in…

> Bug trackers are one of those areas where a lot of companies should build there own.

This sounds insane. I have not worked at a place where the workflow was so holy and important that it couldn't be captured in a near default JIRA install.

Most of the customization asks I've seen with JIRA come from dysfunctional organizations that demand new swimlanes like "QA" and "product approval" and "spec design".

Re: The full-time job of keeping up with Kubernetes

#73

Post by emmelaich »

hueving wrote: post #7
Analemma_ wrote: post #4

Earlier quoted context omitted.

I think a lot of this is that once a tool reaches a certain level of size and complexity, it’s impossible to know all the technical details unless you’re actually a developer on the project (and after another threshold, not even then). So at that point you have to use news and social information to find out what’s going on, and just trust that people know what they’re doing. It can be frustrating, but it’s also unavo…

The important distinction here is the pace. I can use a Ubuntu server LTS version and not have to stay on top of weekly development release announcements from upstream. With k8s lacking LTS, it forces you to drink from the firehose.

In a sense LTS is provided by vendors such as Gravitational and Red Hat. They introduce hysteresis and smooth out the bumps on the upgrade cycle.

Re: The full-time job of keeping up with Kubernetes

#74

Post by bigiain »

jrochkind1 wrote: post #11
hueving wrote: post #5

This reads like a giant ad for GKE. It emphasizes several times to just use GKE for pretty lame reasons (Google has good SREs and Google started the project). The people that work on upstream k8s in Google (Tim et al) have a pretty limited overlap with the Google Cloud people that run GKE. Upstream k8s is a full time job so they are most certainly not spending their time also writing internal GKE code. I don't have a…

I dunno, if it's an ad for GKE, then "you typically have only about nine months until the latest version goes out of support" is indeed pretty persuasive marketing to me to let someone else deal with it.

This is a showstopper for me.

All our best clients are on a 12 or 24 month upgrade cycles. I need to be confident I can deliver them the latest version of their project knowing there's only regular security updates required over then next 1-2 years.

While I'd love to be doing continuous deployment and multiple production feature deliveries a day - that's not how the business I work at makes their money.

I can live with Ubuntu's 5 year LTS policy, I can't work with a "you might have to do significant rework after just 9 months" platform.

Re: The full-time job of keeping up with Kubernetes

#75

Post by jweir »

brudgers wrote: post #19
tw1010 wrote: post #2

There aught to be a name to the tendency that as tools get better and better, the more your time goes from having your mind in technical-space to social and news-space. It's like the authority to create goes from the individual first-principles (by necessity) maker, to the control over development being in the hands of an external group, and then all your time is spent keeping up with what they're doing. A similar th…

A description of -- not a name for -- is corporate control of an open source project. Trying to keep up with Angular or Kubernetes or Go is not a problem at the Google from whence they come because the technologies are primarily used by and hence primarily designed for use by teams. Not individual developers. A team's brain can schedule a time where part of it is training or studying while the rest of it is making pr…

I disagree on Go.

Go is well designed and takes upgrading into consideration. Upgrades very rarely break anything. You should just be able to recompile that 1.3 code on 1.8, etc.

Re: The full-time job of keeping up with Kubernetes

#76

Post by rpbo »

hueving wrote: post #5

This reads like a giant ad for GKE. It emphasizes several times to just use GKE for pretty lame reasons (Google has good SREs and Google started the project). The people that work on upstream k8s in Google (Tim et al) have a pretty limited overlap with the Google Cloud people that run GKE. Upstream k8s is a full time job so they are most certainly not spending their time also writing internal GKE code. I don't have a…

Full disclosure: I do a lot of work with cloud providers and GCloud is one of my partners.

I think the article covers a very real issue with the pace of k8s. GKE is particularly good at staying updated. There are other providers that are good like Stackpoint Cloud.

If I were going to make an ad for GKE I'd point out that they probably have the most robust internal fiber network of any cloud. AWS tries to push packets onto the public internet as fast as possible.

The load balancer you get with GKE is seriously overpowered and under priced (which makes me love it).

Maybe EKS will be amazing (but we don't have evidence yet). Azure is getting pretty good but they're a bit late to the party. IBM isn't bad but they've struggled the most with staying up-to-date.

Re: The full-time job of keeping up with Kubernetes

#77

Post by emmelaich »

crankylinuxuser wrote: post #22
lomnakkus wrote: post #18

Earlier quoted context omitted.

(I haven't read the rest of your post, but I think I can still answer this little snippet...) > Why unavoidable? It's not that one is forced to use some tech... Even though the hyped crowd chants the names so loudly. Peer pressure. Consultant-hiring-pressure, for example: Are you full-stack? "Full stack", now there's a phrase... (EDIT: I honestly do feel that it's a sort of an attack by the sheer meta on the hiring s…

"Yeah, he's a few PDF's shy of a full stack, I tell ya!" side story: Yeah, I went through 3 stages of hiring with $company. Interviews went great, and then I hear back that I didn't show up to the 3rd (much to my surprise). When the HR person checked back in, they said it was an error and instead they were no longer interested in me because I lacked Kubernetes experience specifically. And my resume includes Openstack…

I've also been interviewee recently in this space. I'm lucky in that in each one the interviewer showed a healthy scepticism or lower enthusiasm for K8s and similar products.

Maybe I'm doing my self a dis-service by not shooting for the moon. But I don't think so. I'd rather do real work than spend days reading upgrade notes. On-premises K8s time will come but for most that time is not now.

Re: The full-time job of keeping up with Kubernetes

#78

Post by deckard1 »

halayli wrote: post #14

This is a problem I faced using ansible, webpack + js modules, and more. it's a moving ground and you always need to keep up to date with the latest changes often which are breaking. I tend to design my systems so that they work even if I haven't touched a line in a year but when using such tools it's always a pain. I wish things were as stable as a bourne shell and unix environment in general. Not that they achieve…

Github is already littered with the dead carcasses of software using shiny new obsolete tools like gulp, grunt, and browserify to name just a tiny few.

You have to have a strong nose for bitrot and ruthlessly filter garbage out of the firehose, to work in web development today. If that package or tool hasn't been updated in 8 months, do you really want to learn it (and force your team to learn it), since it's practically abandonware already?

It's a self-reinforcing cycle. You don't want to put your organization at existential risk by betting on the wrong horse. But there is no choice. Unless you want to get stuck having to hire Perl guys in an ocean of node ninja rockstars. Your company will be the ugly girl that didn't get asked to prom night. The pariah of Silicon Valley.

Re: The full-time job of keeping up with Kubernetes

#79

Post by emmelaich »

munin wrote: post #15
tw1010 wrote: post #2

There aught to be a name to the tendency that as tools get better and better, the more your time goes from having your mind in technical-space to social and news-space. It's like the authority to create goes from the individual first-principles (by necessity) maker, to the control over development being in the hands of an external group, and then all your time is spent keeping up with what they're doing. A similar th…

The short story version of this was written by Arthur C Clarke in 1951, "Superiority" http://www.mayofamily.com/RLM/txt_Clarke_Superiority.html

Fantastic, thanks.

Re: The full-time job of keeping up with Kubernetes

#80

Post by emmelaich »

iooi wrote: post #50
aberoham wrote: post #12

Earlier quoted context omitted.

Author here. I do have a bias toward GKE but that is after trying to use everything else -- DIY, kubespray, OpenShift, Juju, Tectonic, Azure, Fargate, kops, Rancher v2 preview, kubeadm, etc. In my experience nothing (yet) has been as nice and clean as Google Cloud.

Have you tried EKS?

EKS = Amazon Kubernetes.

(I had to look it up)

Post reply on HN