Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Container technologies at Coinbase: Why Kubernetes is not part of our stack

blog.coinbase.com

Post reply on HN
Print view View original

311–320 of 414 posts

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#311

Post by StreamBright »

whatsmyusername wrote: post #58
dvt wrote: post #11

> We would need to build/staff a full-time Compute team This actually was a very real problem at my current job. The data pipeline was migrated to k8s and I was one of the engineers that worked to do that. Unfortunately, neither myself (nor the other data engineer) was a Kubernetes guy, so we kept running into dev-ops walls while also trying to build features and maintain the current codebase. It was a nightmare. If…

The 'needing a team' aspect of Kubernetes sounds remarkably similar to conversations I had like 8 years ago when Openstack was the new hotness. We went with ECS and have been happy with it. It plays well with all of AWS's other products and features. For the few things we have to run On-Prem we use Docker Swarm in single node mode and it works well (albeit missing a few features like crons from Kubernetes).

Yep. I remember the Openstack consultants swarming to office trying to sell it as a solution and not a giant overhead / problem. Luckily they failed the POC so we did not need to waste our life on a non-issue trying to solve it with a non-solution. Now it is k8s' time to do the same. We will see how far this buzzword train gonna go.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#312

Post by kissgyorgy »

snuxoll wrote: post #140
threeseed wrote: post #95

Earlier quoted context omitted.

Or use Dex: https://github.com/dexidp/dex Which has the advantage of not needing any external databases.

That works too depending on your requirements. Either way, authentication is not a hard problem to solve.

LOL. You clearly not worked with SSO or anything a bit more complex. It's a pretty hard problem, there are even companies whose whole portfolio is around authentication only!

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#313

Post by bschwindHN »

p_l wrote: post #258
bschwindHN wrote: post #251

Earlier quoted context omitted.

It can be worth it to piece things together yourself. A complex tool can also be fragile if you don't take the time to learn and understand every facet of it. If you only need certain parts of what k8s offers, building those parts yourself can offer you more stability, control, and insight into what your application is doing. As with anything else it varies case-by-case.

Building it yourself is also a case of "resume-driven development". Not that there aren't good reasons or good outcomes from having done a deep dive, but just putting it there.

I think using something like k8s prematurely is more an example of "resume driven development" than the other way around.

Building it yourself doesn't mean building all of it. For example, it's quite easy to get zero downtime deployments with a tiny bit of systemd configuration and the SO_REUSEPORT socket option. That seems easier for a team to understand than "here is kubernetes and everything that comes with it"

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#314

Post by lsofzz »

andrenth wrote: post #128
ramraj07 wrote: post #109

Earlier quoted context omitted.

Seems like you are a thoughtful engineer who can also make sure you don't make any fundamental design flaws while building these systems well. Whenever I have seen kubernetes fail it's often also because the engineer(s) who built it were not thoughtful at all and often didn't fully understand what they were doing. Perhaps k8s's failing is it makes people who dont know enough think that they do.

I think this is a big failing of the DevOps movement as a whole (at least what DevOps became in practice — devs doing ops) which results in things like passwordless mongodb exposed to the internet...

> I think this is a big failing of the DevOps movement as a whole (at least what DevOps became in practice — devs doing ops) which results in things like passwordless mongodb exposed to the internet...

Hardly. If anything at all, it tells about the _team_ and/or the culture of the organisation. In any DevOps/SRE/Opsec culture worth the salt, an immediate blameless postmortem analysis would be performed to help with premortem analysis in future.

DevOps is not about exposing unsecured endpoints. You've got it all wrong son.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#315

Post by lsofzz »

jatins wrote: post #168
nemothekid wrote: post #3

One thing that is regrettable about K8s winning the orchestration wars so remarkably, is that it pretty much killed all other solutions. Swarm is dead, Nomad doesn't seem like it has much community support and Mesos feels like it's on life support. Mesos still has a lot of people working on it however, but the perception feels different. Personally I've found Mesos much easier to manage, secure, and operate than k8s.…

I tried using Nomad once after being a little worn out by Kubernetes' complexity. For some reason, the Nomad abstractions didn't click on the first couple attempts. In comparison, Kubernetes' abstractions mapped 1:1 to my understanding of the service oriented architecture. I'd have probably gotten used to it had I spent more time using it, but it'd have taken some rewiring of thinking process.

> I tried using Nomad once after being a little worn out by Kubernetes' complexity. For some reason, the Nomad abstractions didn't click on the first couple attempts. In comparison, Kubernetes' abstractions mapped 1:1 to my understanding of the service oriented architecture.

<3 Nomad. However, Nomad only satisfies a really tiny part of Kubernetes ecosystem which is the ability to pack containers and schedule them efficiently in a cluster; Plus, it scales really well. Kubernetes provides a bit more than that.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#316

Post by vertex-four »

cameronbrown wrote: post #289
spottybanana wrote: post #284

Earlier quoted context omitted.

"Why piece all of that functionality together yourself into some fragile framework instead of using a industry standard?" Quite recently developed "industry standard". Many tools mentioned have been used for tens of years, they work robustly, are well documented and there is lots of people who can use them. I personally would use the word "industry standard" a little bit differently.

> Quite recently developed "industry standard". Many tools mentioned have been used for tens of years, they work robustly, are well documented and there is lots of people who can use them. k8s is based off of Borg, which has existed for far longer. https://research.google/pubs/pub43438/

"What Google does" is not an industry standard.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#317

Post by lsofzz »

vegai_ wrote: post #206

Nomad has been brought forth in a lot of comments, and it has a feature that nobody brought up yet, I think. It's multi-platform. It currently has official task runners for Docker, Isolated/Raw Fork/Exec, Java, and Qemu. It has several community-based task drivers, including Windows IIS and FreeBSD Jails. Kubernetes supports mostly Linux, although has recently gained Windows node support.

> Nomad has been brought forth in a lot of comments, and it has a feature that nobody brought up yet, I think. It's multi-platform. It currently has official task runners for Docker, Isolated/Raw Fork/Exec, Java, and Qemu. It has several community-based task drivers, including Windows IIS and FreeBSD Jails.

I think `rkt` and `lxc` are missing from your list.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#318

Post by donedealomg »

rawoke083600 wrote: post #221
dvcrn wrote: post #131

Earlier quoted context omitted.

I work on a 2-person project and decided to go with kubernetes (through digitalocean) for the cluster. I am managing everything with terraform and I don't have any big problems. I like that I can write everything as terraform manifests, have it diffed on git push and applied to prod if I want to. Sure it had a learning curve but now I just describe my deployments and k8s does the rest, which then reflects back on dig…

if you can run everything on a $10 vm...do you really need k8 ?

No, but that's also why he doesn't have any problems.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#319

Post by richardARPANET »

The CTO at a client of mine has spent over 1 year trying to deploy ~7 web services to k8s, hired 5 contractors to help and they still don't have a stable deployment.

I informed them my team could do it all in under two days using Terraform + AWS + EBS. Unfortunately, they didn't take us up on the offer.

Sunk cost, etc.

I've never seen a company use k8s and not end up with a giant expensive team maintaining it all.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#320

Post by gnud »

GOTERTHe wrote: post #207
Lukas_Skywalker wrote: post #201

Earlier quoted context omitted.

For those missing Heroku: there exists Dokku [1], a small Heroku-like implementation for container management. It uses the same underlying buildpacks and you get the same comfort as with Heroku. And it's free to use. You can't deploy to multiple host machines though. But for small projects that fit on a single host, it's very nice to use. [1] http://dokku.viewdocs.io/dokku/

Tried Dokku, but found CapRover [1] to be a much better / easier option [1] https://caprover.com/

Looks interesting - but the installation instructions put me off a bit. Open a port on your server, and don't change the default password `captain42` - then run a cli tool from your dev machine.

I'll look more into it, but it didn't really inspire confidence.

Post reply on HN