Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS CloudFormation now supports blue/green deployments for Amazon ECS

aws.amazon.com

Post reply on HN
Print view View original

51–60 of 80 posts

Re: AWS CloudFormation now supports blue/green deployments for Amazon ECS

#51

Post by devonkim »

gchamonlive wrote: post #49
devonkim wrote: post #44

Earlier quoted context omitted.

Terraform doesn’t support rollbacks (handy for application roll-outs) and if your shop is heavily invested in AWS CF is a perfectly fine tool. I’ve maintained and started up tens of thousands of lines of both TF and CF and they both have their strengths and weaknesses.

are you not mixing infrastructure deployment and application deployment? terraform is not the tool for the later, an ansible stack would be better suited for it

Once upon a time people would put applications delivered baked into AMIs as a deployment approach. Using a combination of CloudFormation metadata, Cloudwatch Alarms, cfn-init, and other tools applications could be deployed end to end with a single tool immutably. Many people are stuck with this rather coupled approach because trying to pull it apart would take more effort than is worth to the business.

In a lot of situations infrastructure deployment and configuration management is deeply coupled in the AWS ecosystem (DB changes, Serverless technically is changing your infrastructure definition as application changes).

Re: AWS CloudFormation now supports blue/green deployments for Amazon ECS

#52

Post by devonkim »

tbrock wrote: post #48
devonkim wrote: post #44

Earlier quoted context omitted.

Terraform doesn’t support rollbacks (handy for application roll-outs) and if your shop is heavily invested in AWS CF is a perfectly fine tool. I’ve maintained and started up tens of thousands of lines of both TF and CF and they both have their strengths and weaknesses.

Sure it does? Revert your code and apply. It’s not atomic but I’m guessing neither is CF. I’m not sure I’d use either of these tools to roll out new code though.

That’s more of a GitOps style intervention requiring a source change and the workflow to revert a change could certainly be done but is by design not a first class construct in Terraform providers (not every provider supports every feature such as importing of resources). To respond to a Cloudwatch (heck, Prometheus, Grafana, ELK, etc) alarm saying your error rate went up because of a Route53 change it’s not out of the box with Terraform (would be a custom providers or null resources probably). And as a CLI application (granted, it’s run more like an RPC style architecture) there’s no obvious way to signal different failure levels and to respond to different failure modes of different resources.

CF roll-backs are on by default and will revert changes. Sometimes it can fail and be a real pain, but it’s overall been more of a help for myself than harm.

Re: AWS CloudFormation now supports blue/green deployments for Amazon ECS

#53

Post by nowahe »

gtsteve wrote: post #27
grubbypaw wrote: post #9

I've never understood CloudFormation's lag time in supporting new service functionality. This launched nearly 3 years ago. https://aws.amazon.com/blogs/compute/bluegreen-deployments-w...

I understand the logic of not waiting for the CloudFormation team to do their work because it could introduce delays but it makes it a considerably less useful tool. When considering Terraform vs CloudFormation I certainly took that into account and that's why I don't use CloudFormation for anything. I wonder if there are any killer features of CloudFormation I missed when I looked into this (years ago now).

Not so much a killer feature, but I find that some actions in newer services are only possible to automate through Cloudformation (looking at you Workspaces), because of the time they take to open up the API.

For Workspaces it took them something like 2-3 years to release to the public API what you could do with Cloudformation

Re: AWS CloudFormation now supports blue/green deployments for Amazon ECS

#54

Post by nerdjon »

tbrock wrote: post #40

I don’t think I’ve ever met someone that actually uses cloudformation unless a template was provided them to setup something like a lambda. Why would you when terraform exists?

I use CloudFormation currently, and it gets the job done just fine.

It has some nice things that terraform just doesn't have out of the box:

- Its a managed service, I don't need to setup a state bucket (or similar), I upload my template and it handles it. CI itself is not running CloudFormation, it just tells CF to do its thing. I could achieve this with other services with TF, but its not something I have seen out of the box.

- Drift Detection

- Stack Sets, the ability to deploy easily deploy a cloudformation stack to multiple AWS Accounts

- How sharing outputs works in CF between deployed stacks. There are protections in place if you have a stack that relies on a value your stack creates, you cannot modify or delete it without first changing your other stacks. I wish AWS could instead trigger a downstream change in any of those stacks, but it does not. (This is not for nested stacks, but for 2 completely separate stacks that just happen to need to share something)

However even without these, I struggle to see the point of going with Terraform if we are all in for AWS anyways. Especially when there are templates that we may want to use, so why not keep everything in the same place.

Re: AWS CloudFormation now supports blue/green deployments for Amazon ECS

#55

Post by drfritznunkie »

grubbypaw wrote: post #9

I've never understood CloudFormation's lag time in supporting new service functionality. This launched nearly 3 years ago. https://aws.amazon.com/blogs/compute/bluegreen-deployments-w...

Having spoken with core teams like IAM and Cloudformation teams at length, this appears to an internal AWS organizational issue. Those teams are not responsible for the services integration with them and so they're at the mercy of those teams priorities.

But honestly, I think the reason that Cloudformation support isn't as widespread or a top level priority is that it simply exposes the poor architecture and behavior of many of AWSs second tier services and teams. There are many services that simply do not behave well when managed by Cloudformation, but are also completely janky on their own and I'm betting it's far easier to cover up for poor architecture in the console than expose all the services dirty laundry with a Cloudformation integration.

Additionally, there are a lot of service teams that probably don't have a lot of customers using Cloudformation, so don't prioritize it or half-ass it completely. I'm looking at you DMS, and your terrible turd of a Cloudformation integration.

I'd say nearly the same thing about IAM and service teams inability to implement it well. I still do not understand why AWS has not mandated all services need to support both tag and resource based policies and predictable IAM semantics (looking at you Glue with your little fu of love called the write action "glue:GetMapping").

Cloudformation and IAM are, to me, the two of the most killer services from AWS, neither of which I've seen replicated at other providers.

Re: AWS CloudFormation now supports blue/green deployments for Amazon ECS

#56

Post by yandie »

drfritznunkie wrote: post #55
grubbypaw wrote: post #9

I've never understood CloudFormation's lag time in supporting new service functionality. This launched nearly 3 years ago. https://aws.amazon.com/blogs/compute/bluegreen-deployments-w...

Having spoken with core teams like IAM and Cloudformation teams at length, this appears to an internal AWS organizational issue. Those teams are not responsible for the services integration with them and so they're at the mercy of those teams priorities. But honestly, I think the reason that Cloudformation support isn't as widespread or a top level priority is that it simply exposes the poor architecture and behavior…

Ex AWS here. I had the fun of digging into the rabbit hole of IAM and its convoluted logic. It's definitely possible to do what your said, but it's super easy to make mistake and the internal documentation is lacking. It took me multiple trips talking to people to deliver the integration we wanted.

It's also very old with some odd decisions in there - I can't go into the specifics. And it's practically impossible for the IAM team to deprecate those impossible corners

Re: AWS CloudFormation now supports blue/green deployments for Amazon ECS

#57

Post by drfritznunkie »

yandie wrote: post #56
drfritznunkie wrote: post #55

Earlier quoted context omitted.

Having spoken with core teams like IAM and Cloudformation teams at length, this appears to an internal AWS organizational issue. Those teams are not responsible for the services integration with them and so they're at the mercy of those teams priorities. But honestly, I think the reason that Cloudformation support isn't as widespread or a top level priority is that it simply exposes the poor architecture and behavior…

Ex AWS here. I had the fun of digging into the rabbit hole of IAM and its convoluted logic. It's definitely possible to do what your said, but it's super easy to make mistake and the internal documentation is lacking. It took me multiple trips talking to people to deliver the integration we wanted. It's also very old with some odd decisions in there - I can't go into the specifics. And it's practically impossible for…

I am not surprised, being that it's one of the oldest AWS services? What I do love about IAM is that with the work that the Automated Reasoning Group is doing with Zelkova, it's really a dream to be able test IAM policies before deploying them. I really hope their work trickles back to the service teams so that they too can leverage it to see their way out of those dark corners in IAM :)

Re: AWS CloudFormation now supports blue/green deployments for Amazon ECS

#58

Post by gchamonlive »

devonkim wrote: post #51
gchamonlive wrote: post #49

Earlier quoted context omitted.

are you not mixing infrastructure deployment and application deployment? terraform is not the tool for the later, an ansible stack would be better suited for it

Once upon a time people would put applications delivered baked into AMIs as a deployment approach. Using a combination of CloudFormation metadata, Cloudwatch Alarms, cfn-init, and other tools applications could be deployed end to end with a single tool immutably. Many people are stuck with this rather coupled approach because trying to pull it apart would take more effort than is worth to the business. In a lot of si…

indeed

then terraform is probably not the right tool and cloudformation is a better suited solution

Re: AWS CloudFormation now supports blue/green deployments for Amazon ECS

#59

Post by ganstyles »

scarface74 wrote: post #43
vazamb wrote: post #4

Earlier quoted context omitted.

I would love to know what the problem is. We do dozen of deployments every week with a ALB + ECS + Fargate setup. We upload a new container image, create a new task and launch as many tasks as desired (so if we want 2 containers running we launch 2, for a total of 4). ALB calls the /health endpoints on the new containers and if they pass the healthchecks it drains connections to the old containers and stops the tasks…

That’s not a Blue Green deployment.....

I don't think they are saying it was, unless I'm misreading. They're talking about standard ECS deploys. Yo add my anecdata, I do lots of ECS deploys via terraform into production and it works pretty seamlessly.

Re: AWS CloudFormation now supports blue/green deployments for Amazon ECS

#60

Post by ReidZB »

nerdjon wrote: post #54
tbrock wrote: post #40

I don’t think I’ve ever met someone that actually uses cloudformation unless a template was provided them to setup something like a lambda. Why would you when terraform exists?

I use CloudFormation currently, and it gets the job done just fine. It has some nice things that terraform just doesn't have out of the box: - Its a managed service, I don't need to setup a state bucket (or similar), I upload my template and it handles it. CI itself is not running CloudFormation, it just tells CF to do its thing. I could achieve this with other services with TF, but its not something I have seen out…

Stack Sets are a killer feature, and so much easier than trying to do the same thing in Terraform.

CloudFormation Drift Detection is very limited currently: it only supports a small subset of resources that you can create with CloudFormation. If your needs are covered by it, great, but it doesn't take much to go beyond its bounds. Also, it detects drift, but won't correct it.

Terraform both detects and corrects drift on almost every resource, on every application. Sometimes there are limitations. This does result in extra work, as you can't just ignore drift without capturing that in code (via ignore_changes), but to me this is absolutely a desirable thing.

We use CloudFormation to actually create the state resources to store Terraform state.

Terraform can share outputs between different states using the terraform_remote_state data source, though there isn't any restriction on what can be done there (no requirement to update other stacks/states).

Despite being an AWS-only shop, we've found the multi-provider support in Terraform to be really useful. For example, as part of our environment configuration, we store resources in Consul, create Pingdom checks, etc all from the same set of Terraform code.

Post reply on HN