Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Container technologies at Coinbase: Why Kubernetes is not part of our stack

blog.coinbase.com

Post reply on HN
Print view View original

391–400 of 414 posts

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#391

Post by folmar »

vandalatyou wrote: post #385
DevKoala wrote: post #154

I work for a mid size company with 30-40 engineers managing 20-30 very diverse apps in terms of scale requirements and architectural complexity. It took our devops team (4-5 people) probably 18 months to learn and fully migrate all our apps to Kubernetes. The upfront cost was massive, but nowadays the app teams own their deployments, configurations, SLA's, monitors, and cost metrics. Introducing Kubernetes into our o…

I'm a fan of cloud native kubernetes. But: I've also been put in the terrible position of supporting a platform sold to run on kubernetes in an air gapped, on prem, bare metal environment. But: I've also been put into the terrible position of fighting vendors armed with agile, k8s and microservices selling that combined mayhem as a replacement for an openmp,openacc based massively parallel, on prem, bare metal HPC sy…

> I've also been put in the terrible position of supporting a platform sold to run on kubernetes in an air gapped, on prem, bare metal environment.

Would you share your experience?

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#392

Post by folmar »

Shicholas wrote: post #361
speedgoose wrote: post #209

Earlier quoted context omitted.

You don't see random network errors, often visible with DNS, on your Azure managed kubernetes clusters?

I haven't yet our ingresses have passed routine ping checks (we use New Relic synthetics for this) for a while now. Fingers crossed.

Ingress on AKS is easy, the outgress will be the pain if you need anything from it.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#393

Post by folmar »

CraigJPerry wrote: post #191
daze42 wrote: post #133

Earlier quoted context omitted.

It's impossible to make essential complexity disappear but it is certainly possible to reduce incidental complexity. Most software is much more complex than it needs to be and Kubernetes is no exception.

Before k8s every serious shop automated the crap out of their infra. Jump/kickstart recipes, rolling cluster patching that split RAID mirrors before applying, blue/green deployment scripts to tickle the loadbalancer, cron jobs to purge old releases ... That stack is super complex and utterly bespoke to the company. With k8s it’s standardised and usually better quality.

It's on a path to standardised, but not there yet: etcd vs. others, different ingress controllers, providers replacing most of network parts, storage is bumpy/not so standard, deploy may be kubectl apply/helm/operator.

I would really appreciate a more mature ecosystem.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#394

Post by ethbro »

manigandham wrote: post #327
spottybanana wrote: post #284

Earlier quoted context omitted.

"Why piece all of that functionality together yourself into some fragile framework instead of using a industry standard?" Quite recently developed "industry standard". Many tools mentioned have been used for tens of years, they work robustly, are well documented and there is lots of people who can use them. I personally would use the word "industry standard" a little bit differently.

You still have to put them all together into some custom solution just for your setup which adds overhead and fragility. New employees will have to learn that instead of using K8S APIs. Deploying new components can’t take advantage of the wide and fast growing ecosystem. There’s really nothing like the full suite that kubernetes provides.

> New employees will have to learn that instead of using K8S APIs

You know tech has made it when people offer argumentum ad Java.

'Sure, X might not be the best solution, but you can always buy someone to do it.'

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#395

Post by dvcrn »

rawoke083600 wrote: post #221
dvcrn wrote: post #131

Earlier quoted context omitted.

I work on a 2-person project and decided to go with kubernetes (through digitalocean) for the cluster. I am managing everything with terraform and I don't have any big problems. I like that I can write everything as terraform manifests, have it diffed on git push and applied to prod if I want to. Sure it had a learning curve but now I just describe my deployments and k8s does the rest, which then reflects back on dig…

if you can run everything on a $10 vm...do you really need k8 ?

The point is to keep cost down while a project is in development, then scale depending on needs without having to worry about container distribution and resource utilization.

On dev I don't need a 10 node cluster if I had 10 containers running. One $10 vm is fine.

On prod I can start with a 3-node cluster for these 10 containers, then scale it up depending on traffic and needs while controlling my spending.

Not everyone has thousands of dollars of VC money to throw at hosting

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#396

Post by jmb12686 »

tylermenezes wrote: post #344
patricjansson wrote: post #261

Earlier quoted context omitted.

"conversations have led us to the conclusion that our customers want continued support of Swarm without an implied end date." https://www.mirantis.com/blog/mirantis-will-continue-to-supp...

No matter what they claim, it's really not supported in the sense most commercial oss projects are. We finally switched off after a minor version introduced a segfault when adding nodes in certain conditions, and the issue was unfixed after 5 months.

This. Docker Swarm, and by extension Docker EE / UCP, is barely in maintenance mode. Go compare the Moby and docker projects on GitHub vs kubernetes.

To be clear, I use Docker Swarm in my home cluster due to simplicity and ease of use. Unfortunately that pattern hasn't scaled to the Enterprise.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#397

Post by lsofzz »

andrenth wrote: post #348
lsofzz wrote: post #314

Earlier quoted context omitted.

> I think this is a big failing of the DevOps movement as a whole (at least what DevOps became in practice — devs doing ops) which results in things like passwordless mongodb exposed to the internet... Hardly. If anything at all, it tells about the _team_ and/or the culture of the organisation. In any DevOps/SRE/Opsec culture worth the salt, an immediate blameless postmortem analysis would be performed to help with p…

I'm not your son, nor am I talking about what DevOps "is about", but about what is became in practice, which you would have understood hadn't you rushed to reply in the most condescending tone you managed to invoke.

> I'm not your son, nor am I talking about what DevOps "is about", but about what is became in practice, which you would have understood hadn't you rushed to reply in the most condescending tone you managed to invoke.

Fair enough. I am letting you know though the part where you got mixed up - that is not because _What DevOps has become in practice_. That is precisely because of failings and shortcomings in team culture and/or the organisation that practice DevOps.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#398

Post by lsofzz »

vegai_ wrote: post #366
lsofzz wrote: post #317

Earlier quoted context omitted.

> Nomad has been brought forth in a lot of comments, and it has a feature that nobody brought up yet, I think. It's multi-platform. It currently has official task runners for Docker, Isolated/Raw Fork/Exec, Java, and Qemu. It has several community-based task drivers, including Windows IIS and FreeBSD Jails. I think `rkt` and `lxc` are missing from your list.

LXC yes, but RKT is marked as deprecated on Nomad's documentation.

Oh bummer

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#399

Post by ricksharp »

dvt wrote: post #11

> We would need to build/staff a full-time Compute team This actually was a very real problem at my current job. The data pipeline was migrated to k8s and I was one of the engineers that worked to do that. Unfortunately, neither myself (nor the other data engineer) was a Kubernetes guy, so we kept running into dev-ops walls while also trying to build features and maintain the current codebase. It was a nightmare. If…

I don’t understand the value of container tech (unless dealing with legacy code).

Serverless seems better and cheaper (lambda, GC Functions, Azure Functions). Pay as you go with no overhead costs. In many cases usage may never exceed the free credits.

For the database and persistent storage, you need a cloud native service like S3 and Postgres RDS. (Running a database in a container is asking for complete data loss from what I understand.) This is the primary cost for my tech stack and for lite loads $15 is about as minimal as possible.

So I don’t see how using containers could be cheaper unless you are keeping the data in the container, which is a bad idea since containers are supposed to be stateless.

So serverless cloud native tech seems better by far from everything I’ve experienced.

Also when using a tool like serverless framework, deploying an entire stack of serverless resources is dead simple.

I’ve tried a few times to see the value of containers, but I just don’t get it.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#400

Post by TurboHaskal »

vegai_ wrote: post #387
TurboHaskal wrote: post #228

Earlier quoted context omitted.

I am extremely happy with Nomad, it is one of the best decisions we have ever made in our organisation. We had a thin abstraction layer over Consul + Docker Swarm that we could port to Nomad in a matter of a few hours and it's been rock solid so far. But sometimes I wonder whether we are hurting our own careers not going down the k8s route. It really seems it has a bright future.

>But sometimes I wonder whether we are hurting our own careers not going down the k8s route. It really seems it has a bright future. I believe the world will always have a demand for simple technology.

True that, but I may be in the job market soon, and all I see is Kubernetes and a plethora of tools related to its ecosystem I have no exposure to.
Post reply on HN