Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Container technologies at Coinbase: Why Kubernetes is not part of our stack

blog.coinbase.com

Post reply on HN
Print view View original

111–120 of 414 posts

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#111

Post by danenania »

sk0g wrote: post #103
danenania wrote: post #86

"For example, most folks that run large-scale container orchestration platforms cannot utilize their built-in secret or configuration management. These primitives are generally not meant, designed, or built for hundreds of engineers on tens of teams and generally do not include the necessary controls to be able to sanely manage, own, and operate their applications. It is extremely common for folks to separate their s…

Unfortunately I live in not-US (Aus), but I have to say, envkey looks very good! I'm actually floating this and HashiCorp Vault by management at the moment, since even with 4 devs, keeping environment variables in sync is a PAIN! If you don't mind me asking, how does the Go integration work? I initially thought it was actually some sort of alternate `os.Getenv()` that you imported, but that doesn't seem to be the cas…

The go libary (along with our other language-specific libs) is a thin wrapper around envkey-fetch -- https://github.com/envkey/envkey-fetch

When a process starts, envkeygo makes a request to our config service (via envkey-fetch) to fetch the encrypted config, then decrypts the config and sets them on the environment so they can be retrieved with `os.Getenv`. Both the lookup id (for fetching the encrypted config) and the encryption passphrase for decryption are initially passed in via an ENVKEY=... environment variable--you can think of it as a single environment variable that "expands" into all the others that you need.

Latency on a request is generally in the 150-300ms range from the US (our primary servers are in us-east-1).

EnvKey is strongly consistent and transactional, so once you make a change to your config, it will be available immediately for any subsequent requests. For now, it's still up to you to restart servers/services yourself after a change. With the v2, this will be scriptable.

We'll eventually hire outside the US too, but for now I'm trying to keep the timezone spread and administrative burden low :)

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#112

Post by AaronM »

whatsmyusername wrote: post #59
theptip wrote: post #56

Anecdata: Series B startup. I've found GKE to be almost completely painless, and I've been using it in production for more than 4 years now. I don't think the article gave a fair representation on this count; sharing a link to a single GKE incident that (according to the writeup) spanned multiple days and only affected a small segment of users doesn't (for me) substantiate the claim that "it isn’t uncommon for them t…

EKS is a feature parity product. The pricing makes that painfully obvious.

We've been using EKS since it went GA. We haven't had a single control plane outage that I am aware of.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#113

Post by [deleted] »

Ataraxy wrote: post #89
thrwn_frthr_awy wrote: post #22

Earlier quoted context omitted.

> This actually was a very real problem at my current job. The data pipeline was migrated to k8s and I was one of the engineers that worked to do that. Unfortunately, neither myself (nor the other data engineer) was a Kubernetes guy, so we kept running into dev-ops walls while also trying to build features and maintain the current codebase. I've experienced this as well. At the last large company I worked at we had a…

Isn't that the appeal for google's cloud run? A sort of go between from cloud functions/docker image.

[deleted]

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#114

Post by AaronM »

theptip wrote: post #66
whatsmyusername wrote: post #59

Earlier quoted context omitted.

EKS is a feature parity product. The pricing makes that painfully obvious.

Perhaps, in the box-ticking sense. I'd be surprised if they have the same operational maturity, which translates to uptime, which was what I was talking about. A quick search turns up these SLAs: vs https://cloud.google.com/blog/products/containers-kubernetes... EKS: 3 nines GKE: 3.5 nines So Google is committing to 1/2 the downtime. (Note the GKE SLA is for regional clusters, which is what you should be doing if you…

In the region we are running it, its not had an outage that has affected us since they launched it.

We haven't had any issues with k8's upgrades either.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#115

Post by chillfox »

d0m wrote: post #50

Quick shout-out toAptible (Heroku for HIPAA) which have been amazing for our healthcare startup. Still, I honestly feel that we've taken a step back in the industry where we went from ec2 --> heroku --> k8s. But I know there are many people working hard to create the next infrastructure so we don't have to deal with containers and all that re-inventing dev-ops non-sense for every project.

Heroku is missing the ability to get a dedicated IP for a project.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#116

Post by verroq »

flowerlad wrote: post #100
zaptheimpaler wrote: post #75

Earlier quoted context omitted.

Plain old linux is the alternative, which is also "learn once, use everywhere" whether its AWS EC2 or GCP Instances or nearly any machine under the sun. I don't see how k8s avoids the need to learn about cloud vendor specific tech. e.g searching "aws RDS k8s" gives me a beta github packages and a bunch of blog posts on how to configure it right. It doesn't sound like much less work than learning how to use RDS withou…

> Plain old linux is the alternative How do you run an application on a cluster of plain old linux machines? How do you do load balancing? How do you scale up and down? How do you update your app without downtime? How do you roll back easily if something goes wrong? How do you ensure all your servers are running the same version of dependencies? How do you update those dependencies? How do you replicate your environm…

The easy answer is that you don’t need to run your service on distributed across 1000 VPS instances.

A handful of dedicated machines is enough. For example: stackoverflow.

None of these are difficult to answer. You can set up automated deployment/rollback however you want. You don’t care about the dependencies your code is compiled and linux has binary compat. You don’t need to split your app into 1000 unmanageable interdependent microservices. You have enough disk space on your dedicated machine or use NFS. You set up your systemd service file to auto restart your service. Etc etc.

When you have k8 as a hammer everything looks like a nail.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#117

Post by tso »

theptip wrote: post #63
dvt wrote: post #11

> We would need to build/staff a full-time Compute team This actually was a very real problem at my current job. The data pipeline was migrated to k8s and I was one of the engineers that worked to do that. Unfortunately, neither myself (nor the other data engineer) was a Kubernetes guy, so we kept running into dev-ops walls while also trying to build features and maintain the current codebase. It was a nightmare. If…

> > We would need to build/staff a full-time Compute team I'm not sure I get this objection. Presumably in a company like Coinbase there is already an infrastructure team that runs the AWS instances, helps build the AMIs, etc. This team could re-tool and hire some k8s experts to help them make the shift. The promise of k8s (at least one of them) is that you can do more with less ops resource, since the system is so p…

> Presumably in a company like Coinbase there is already an infrastructure team that runs the AWS instances, helps build the AMIs, etc. This team could re-tool and hire some k8s experts to help them make the shift.

The key is that there is a lot of additional services and interface points to handle. As the Coinbase article noted, you need extra pieces on top of k8s (storage, service mesh, config/secrets, etc) that need care and feeding. Even if the company moved 100% of their services into k8s there's now more work to be done for the same level of service.

: The control points that k8s exposes are not simple "drop in your provider here" bits of integration. You would likely still have the same core providers (ex: EBS for storage) but there is now more code running to orchestrate them, and more access control to implement and verify.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#118

Post by ditonal »

flowerlad wrote: post #100
zaptheimpaler wrote: post #75

Earlier quoted context omitted.

Plain old linux is the alternative, which is also "learn once, use everywhere" whether its AWS EC2 or GCP Instances or nearly any machine under the sun. I don't see how k8s avoids the need to learn about cloud vendor specific tech. e.g searching "aws RDS k8s" gives me a beta github packages and a bunch of blog posts on how to configure it right. It doesn't sound like much less work than learning how to use RDS withou…

> Plain old linux is the alternative How do you run an application on a cluster of plain old linux machines? How do you do load balancing? How do you scale up and down? How do you update your app without downtime? How do you roll back easily if something goes wrong? How do you ensure all your servers are running the same version of dependencies? How do you update those dependencies? How do you replicate your environm…

The difficulty of all that stuff on plain old Linux is overstated and the difficulty of doing that on K8s is understated. And I agree with grandfathers point that if you don’t understand how to do that on plain old Linux you will struggle on K8s. K8s is ok for huge enterprises with tons of engineers but somehow K8s advocates make it seem like learning and operating nginx on ubuntu is this huge challenge when it’s usually not.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#119

Post by kryptk »

nihil75 wrote: post #78
kureikain wrote: post #77

Looks at what they have built: - container orchestration platform is Odin + AWS ASGs (auto-scaling groups). - Codeflow (our internal UI for deployments) - Odin kicks off a step function and begins to deploy your application. New VMs are stood up in AWS and loaded into a new ASG, your software is fetched from various internal locations, a load balancer starts health-checking these new instances, and eventually traffic…

Odin is just Step Function flow that deploys Auto Scaling groups. It's not that complicated, and not so different from your setup.. main difference is they deploy docker images to ec2 rather than run ansible and configure a running machine.

What about resource allocation? With Kubernetes I can specify both requests and limits of cpu and memory and it will fill my nodes to match. A node autoscaler gets me more nodes if needed. I can define vertical pod autoscaler that can dynamically modify those requests and get me an emptier or bigger node if needed. I can define a horizontal pod autoscaler to keep aggregate cpu at a set target by auto-spawning more containers for me, and k8s handles load balancing via dns. Do typical production setups not require some/most of these features? Mine do.

Re: Container technologies at Coinbase: Why Kubernetes is not part of our stack

#120

Post by [deleted] »

threeseed wrote: post #88
zaptheimpaler wrote: post #75

Earlier quoted context omitted.

Plain old linux is the alternative, which is also "learn once, use everywhere" whether its AWS EC2 or GCP Instances or nearly any machine under the sun. I don't see how k8s avoids the need to learn about cloud vendor specific tech. e.g searching "aws RDS k8s" gives me a beta github packages and a bunch of blog posts on how to configure it right. It doesn't sound like much less work than learning how to use RDS withou…

> I don't see how k8s avoids the need to learn about cloud vendor specific tech e.g searching "aws RDS k8s" Not sure what you mean here. Kubernetes is designed to be cloud and vendor agnostic. And RDS is a hosted database that you connect to from your application. Whether that applications runs standalone, in a container or VM is irrelevant.

[deleted]
Post reply on HN