Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS Controllers for Kubernetes

github.com

Post reply on HN
Print view View original

21–30 of 54 posts

Re: AWS Controllers for Kubernetes

#21

Post by nrmitchi »

etxm wrote: post #14
rektide wrote: post #5

Earlier quoted context omitted.

Every Kubernetes shop I've seen uses Helm to deploy their workloads, & now you can set up all your S3 Buckets, SQS, SNS stuff in that same Helm chart as your app. This is hella ideal. I also just generally love the thought of being able to manage any cloud resources at all via standard open-protocols & systems. Compounding the investment, rather than having to invest in a bunch of specific not-interconnected areas is…

It’s all fun and games until a deployment is acting funny and somebody destroys the chart and re-creates it and takes the bucket out with it. Does it have functionality like Kube DB that makes a “dormant” version of the state store?

This is, in my opinion, the most dangerous thing about CRDs, especially for resources that are not scoped the the cluster.

It is way to easy to accidently delete something you need. PV's were my first experience dealing with this (a chart upgrade recreated a PVC, the old one unbound, and was immediately cleaned up), and it's not a risk that I want to see extended to buckets, RDS instances, etc.

The other side of this is that CRDs can lead to abandoned resources; if you find your cluster borked, or shut down improperly, any resources which existed as CRDs (or in cloud Kubernetes land, include LoadBalancers) probably did not get cleaned up, and will be abandoned (but left running).

It's not clear that there is actually a good solution here that fits neatly in with existing CRD behaviour.

Re: AWS Controllers for Kubernetes

#22

Post by freedomben »

digitallogic wrote: post #16
mfer wrote: post #8

One consequence is the accidental deletion of AWS things... If a CRD is deleted the CRs described it are also deleted. So, deleting a CRD (even accidentally) could end up deleting resources in AWS (e.g., backups). So, be careful. Some things being managed by Kubernetes would be really cool. Other things being managed by k8s could break things if something goes wrong. I would plan accordingly.

Also, not all AWS follow the same deletion semantics. Example: S3 buckets. The report as being deleted somewhat quickly, but their name may not be available again for hour or so. In this case the delete will appear to succeed, but the recreation, if done with the same name, may fail.

Great example. I worry about adding a layer of abstraction over provisioning resources this way.

Of course we have to try this because it's a badass (tho obvious in hindsight) idea, but in practice it might have some downsides.

Re: AWS Controllers for Kubernetes

#25

Post by uHuge »

mey wrote: post #10
mfer wrote: post #8

One consequence is the accidental deletion of AWS things... If a CRD is deleted the CRs described it are also deleted. So, deleting a CRD (even accidentally) could end up deleting resources in AWS (e.g., backups). So, be careful. Some things being managed by Kubernetes would be really cool. Other things being managed by k8s could break things if something goes wrong. I would plan accordingly.

There are certain things that freak me out in IAC, like loosing a primary DB or secrets. I want IAC to create them, wire things together and index them, but the idea of an accidental mis-configuration deleting a production database gives me heartburn.

That's what you should have QA pipeline ready to check before you'll be able to apply the config, right? Secrets should be encrypted in git to my best awareness.

Re: AWS Controllers for Kubernetes

#26

Post by takeda »

sytse wrote: post #3

Kelsey Hightower had a good take in https://twitter.com/kelseyhightower/status/12963119515307048... "AWS Controllers for Kubernetes is pretty dope. You can leverage Kubernetes to manage AWS resources such as API gateways and S3 buckets. Think Terraform but backed by Kubernetes style APIs and "realtime" control loops." An in the thread he mentions Crossplane as the cross-cloud way to do this https://twitter.com/kelsey…

I thought whole point of running K8S in AWS was to have infrastructure not tied to their offerings.

If you use something like that, why use K8S and not just use AWS services natively?

Re: AWS Controllers for Kubernetes

#27

Post by DanielDent »

jrockway wrote: post #19
etxm wrote: post #14

Earlier quoted context omitted.

It’s all fun and games until a deployment is acting funny and somebody destroys the chart and re-creates it and takes the bucket out with it. Does it have functionality like Kube DB that makes a “dormant” version of the state store?

Deletion is typically specified by an immutable finalizer on the resource. For example, you can create a StorageClass that saves your PersistentVolume when you delete the PersistentVolumeClaim, and you can't change the class of a claim, so even if your templating goes haywire and deletes your volume claim, the actual data is safe and can be recovered. I have to imagine that any API that lets you create things like RD…

What this describes is a mechanism for k8s to delete underlying resources prior to removing them from k8s.

E.g. before completely removing the CRD representing an S3 bucket, this provides a mechanism for that S3 bucket to be deleted from AWS systems.

Which I think is the opposite of what you were hoping.

Re: AWS Controllers for Kubernetes

#29

Post by dilyevsky »

jrockway wrote: post #19
etxm wrote: post #14

Earlier quoted context omitted.

It’s all fun and games until a deployment is acting funny and somebody destroys the chart and re-creates it and takes the bucket out with it. Does it have functionality like Kube DB that makes a “dormant” version of the state store?

Deletion is typically specified by an immutable finalizer on the resource. For example, you can create a StorageClass that saves your PersistentVolume when you delete the PersistentVolumeClaim, and you can't change the class of a claim, so even if your templating goes haywire and deletes your volume claim, the actual data is safe and can be recovered. I have to imagine that any API that lets you create things like RD…

Finalizers dont do much for safety. They are simply there to ensure controller (in this case ACK) won’t miss the deletion event and leave the resource dangling. To actually prevent object from being deleted you need a validation webhook

Re: AWS Controllers for Kubernetes

#30

Post by brnkrygs »

NathanKP wrote: post #12
mfer wrote: post #8

One consequence is the accidental deletion of AWS things... If a CRD is deleted the CRs described it are also deleted. So, deleting a CRD (even accidentally) could end up deleting resources in AWS (e.g., backups). So, be careful. Some things being managed by Kubernetes would be really cool. Other things being managed by k8s could break things if something goes wrong. I would plan accordingly.

Hi I'm a developer advocate in the AWS Container organization. This question of how to handle resource destruction is an active issue on the project, and we'd welcome your comments (or those of anyone else) on this Github issue: https://github.com/aws/aws-controllers-k8s/issues/82 Our goal is to make this project have "no surprises" and therefore no unexpected destruction of resources. The specifics of how we mark re…

One way our team has dealt with this question in a highly serverless-oriented and infrastructure-as-code-driven (how's THAT for buzzword soup) environment is to explicitly separate stateful resources from stateless, while exposing reference hooks in a configuration store to cross from one to the other. We've found that doing so _greatly_ reduces the blast radius of mistakes and lets us move more quickly and confidently.

The stateless stacks generally have a lot of development activity going on, and rapidly iterate. This is where most of our code and logic lives. This is where the vast majority of our deployment (and related cloud configuration) activity happens.

All of that thrash is kept away from the stateful stacks - think S3 buckets or DynamoDB tables - where, if THOSE thrash, we potentially get an outage at best, or lose data at worst (backups notwithstanding).

We DO NOT WANT stateless oriented stacks to own the lifecycle for stateful stacks. They inherently need to be treated differently. Or, at least the impact of mistakes is different.

The trick comes when you need to tie them together. To do this, we've added CloudFormation hooks and other deployment time logic that publish ARN and other connectivity info to our configuration store. The stateless services look up config values either during deployment or at runtime and are able to find the details they need to reference the state resources they need access to.

We've poked at toolsets like Amplify that lump everything together and have already been bitten numerous times. We've found that the difference between stateful and stateless resources should not be papered over, but instead emphasized and supported explicitly by tooling.

... all of this being one team's experience over the years, of course.

Very curious to see how this paradigm evolves here!

[edit]… Riffing on this just a little bit further… as I’m thinking about it here, it comes down to abstraction level. In a deployment or resource management domain, a generic “this is a cloud resource” isn’t very useful. What’s way _more_ useful is something like “this is a stateful resource” or “this is a stateless resource”, because that level describes resource behavior more clearly, AND how to interface with or manage those resources.

There are echos of code development principles here intentionally - robust cloud infrastructure management is mirrors software dev practices as much as infrastructure management ones!

Post reply on HN