Live data from Hacker News

Container technologies at Coinbase: Why Kubernetes is not part of our stack

blog.coinbase.com

381–390 of 414 posts

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#381

One thing people forget is containers don’t necessarily not to be created with Docker. Lately I’ve been creating containers using NixOS and couldn’t be more happy with the ability to have everything in the container configured by a configuration.nix file. https://nixos.org/nixos/manual/#ch-containers The idea of a Docker Ubuntu, Arch, Alpine, etc base image is kind of silly when you think about it. The idea that we h…

How is a configuration.nix any different than a Dockerfile?

Here is a decent example of a docker image being created from nixPkgs https://github.com/input-output-hk/plutus#how-to-run-a-local...

Another advantage is a unified development environment as I can run the server in the container or directly on host machine in an isolated nix-shell.

Yet another advantage, is unlike the layer of a docker container, isolated dependencies are isolated and linked so packages link together based on a hash-address/path of a dependency in the nix store so there is no need to worry about order of steps but rather what depends on what making the cache misses in a nix-store vs a docker layer more efficient. For example if the first step in a Dockerfile needs to be changed all proceeding steps need to be rebuilt where as if a dependency in a nix file say paython-2 to python-3, every package that depends on python just get re-linked and only those packages need to be updated and you already have python-3 used in a nix-shell or in another container odds are it's already cached and ready to go!

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#382
post #286

Having used Kubernetes extensively at my last position, adoption of k8s strikes me as similar to the adoption of Linux as your desktop OS at the Turn of the century. Some people are doing amazing things with it! Other never figure out how to get their WiFi to consistently work and are bitter people keep talking about it. Eventually something akin to Ubuntu will grow up in the k8 ecosystem and people will stop complai…

Interesting analogy! I was very much looking forward to OpenShift/OKD to be "the Ubuntu" of K8s, but their targeted scope just keeps getting bigger (or it always has been big, not so sure anymore). On the other end, K3s from Rancher is a minimalist distribution, and I think it'll limit its adoption in sophisticated environments with larger teams. Does any other K8s distro look like a good candidate to standardize on…

https://jenkins-x.io/ has brought together a number of open source projects in theoretically a good way but it's still pretty rough around the edges (at leas last time I worked with it in March) and still falls short of an "Ubuntu" experience.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#383

Earlier quoted context omitted.

Then have your Dockerfile build from source. Many of the public layers that people build their containers upon do that for that exact reason. If NixOS is only "generally speaking" correct because of the order it runs its scripts, then put it in a Docker container, where the whole point is to perfectly reproduce the order you run your scripts in order to minimize the errors that arise from doing otherwise. Still worri…

In practice, building Dockerfiles is unreliable and can produce errors because they inherit all of the deficiencies of aptitude, python packaging, javascript packaging, et al. In practice, building nix dependencies is reliable and reproducible because of the militant isolation of build environments, hash integrity checks on _all_ inputs, explicit dependencies, and the long tail of bug fixes related to non-determinism…

That's a great a vote of confidence. I would've thought that nix would only manage packages on the OS-level. How does it re-implement pip to be reliable? I currently have to stage a couple of `apt-gets` and `pip installs` before my `pip install requirements`, and it took a couple of tries to lock in the right sequence that worked. How does nix solve those problems?

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#384
It's all about scale. Large scale requires an orchestrated application management system with progressively integrated bells and whistles and minimal fiddling with userspace. Small scale doesn't factor in. Do small scale the old way if you want: Binary artifacts, amis, home grown integrations and playbooks/scripts. Having done it both ways I continue to do it both ways. Run a monolith the old way and run your web tier, stateless and customer facing, highly available services in k8s. If it's not broken don't fix it (but for some use cases (large scale) the old way _was_ broken).

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#385

I work for a mid size company with 30-40 engineers managing 20-30 very diverse apps in terms of scale requirements and architectural complexity. It took our devops team (4-5 people) probably 18 months to learn and fully migrate all our apps to Kubernetes. The upfront cost was massive, but nowadays the app teams own their deployments, configurations, SLA's, monitors, and cost metrics. Introducing Kubernetes into our o…

I'm a fan of cloud native kubernetes.

But: I've also been put in the terrible position of supporting a platform sold to run on kubernetes in an air gapped, on prem, bare metal environment.

But: I've also been put into the terrible position of fighting vendors armed with agile, k8s and microservices selling that combined mayhem as a replacement for an openmp,openacc based massively parallel, on prem, bare metal HPC system with a strongly conservative development and operational tradition.

I understand loving efficiency and time savings but not everything simply works better with k8s.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#386

Earlier quoted context omitted.

AWS's version is pretty half-baked. You can't provision or use persistent volumes (so no stateful apps), and you have to use their load balancer which terminates TLS (preventing your software from being able to do ALPN, using Let's Encrypt, supporting HTTP/3, etc.). Cloud Run just seems like standard "serverless" stuff, nothing to do with Kubernetes. (The downsides involve not being able to run applications that are…

I use Cloud Run and nothing is specially developed for it, other than just being stateless. I can take my container and run it on a generic Linux box with zero changes (in fact, I run it in WSL2 on my Windows machine all the time). And I just install the normal Node.js, imagemagick, etc in my Dockerfile, no special builds or flags.

That sounds pretty compelling then.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#387
post #206

Nomad has been brought forth in a lot of comments, and it has a feature that nobody brought up yet, I think. It's multi-platform. It currently has official task runners for Docker, Isolated/Raw Fork/Exec, Java, and Qemu. It has several community-based task drivers, including Windows IIS and FreeBSD Jails. Kubernetes supports mostly Linux, although has recently gained Windows node support.

I am extremely happy with Nomad, it is one of the best decisions we have ever made in our organisation. We had a thin abstraction layer over Consul + Docker Swarm that we could port to Nomad in a matter of a few hours and it's been rock solid so far. But sometimes I wonder whether we are hurting our own careers not going down the k8s route. It really seems it has a bright future.

>But sometimes I wonder whether we are hurting our own careers not going down the k8s route. It really seems it has a bright future.

I believe the world will always have a demand for simple technology.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#388
post #270

Earlier quoted context omitted.

Many tutorials are unfortunately bad, they actually got worse over time. Oooold presentations (think 2016 and older) tended to talk more about basic building blocks and how they interacted, especially the design involving resources and their controllers working in a loop of "check requested state, check real state, do changes to implement requested state", and how those loops went up from single pods, through Replica…

This is the exact problem I ran into on rxjs. Every tutorial was badly out of date. Even ones written six months earlier. You had to become a master of figuring out the direction of where community was going and stay on bleeding edge. Some people found this fun. I have many times enjoyed this. When I want to get a boring feature built and hand off to the jr develops so I can spend some time with my kids. It’s frustra…

The most annoying thing, to me, is that Kubernetes doesn't even move fast enough!

The old presentations? Ones that remember K8s 1.0? There's one major change (move from Replication Controller to Deployment+ReplicaSet) that doesn't invalidate ~90% of the material, because the core stuff is about how controllers work!

Yet it seems more and more common to me that people don't learn the core mechanism of kubernetes unless per chance they got there writing CRDs :|

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#389
post #271

Earlier quoted context omitted.

“Plain old Linux” really isn’t an alternative to K8S though. You would need a load balancer, service discovery, a distributed scheduler, a configuration management system (K8S is a very strong alternative to building things around Ansible IMO). You can do all of those things without K8S, of course, but not with “plain old Linux” (what would that be anyway? GNU coreutils and the kernel? Vanilla Debian stable?)

Maybe I’m way off, but aren’t all of those things required for k8s? Ingress controllers, etcd cluster, terraform modules, storage configuration, etc... I guess if you pay for a hosted service a lot of the control plane is taken care of. I’ve used k8s in orgs where it’s a great fit and really fills a need, but it is considerably more complex than running a web service balanced across a couple of machines, and it defin…

At least in Azure Kubernetes ingress and etcd is yours, the other things are taken care of. And the terraform you get is quite nice.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#390
post #64

Earlier quoted context omitted.

I'm in the same boat opting for docker-compose instead. docker-compose is much simpler to manage. Obviously it doesn't have feature parity with k8s but docker-compose does the basics well. An inexpensive VPS runs compose well with more resources at a lower cost than the managed costs of k8s.

I don't understand this comment at all. Kubernetes is designed for a cluster of nodes not a single one. Of course a single VPS is lower cost and easier to use but what happens if you want a second or third. Or any form of redundancy.

A lot of people use it for deployment standard and don't care about multinode.
Post reply on HN