Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Setup AWS Cloudwatch Monitoring and Alerts Using Bash Scripts

themythicalengineer.com

Post reply on HN
Print view View original

21–30 of 34 posts

Re: Setup AWS Cloudwatch Monitoring and Alerts Using Bash Scripts

#21

Post by hughrr »

gizdan wrote: post #20
hughrr wrote: post #3

Earlier quoted context omitted.

As someone who spends two hours a day dealing with buggered terraform state and upgrading terraform and dealing with terraform bugs I can see it. It’s one of those things that really works pretty well but there are enough edge cases to make it slightly soul sucking.

This sounds like a lack of understanding of terraform. We use Terraform pretty heavily and I've rarely seen bad states across our whole org, and the few that I do see are usually people who don't know the core concepts (often non-devops engineers). Terraform has its faults, but it is the best in its class, especially when you need to manage infrastructure beyond a single cloud provider (e.g. we manage our datadog mon…

I completely agree with your points there and that is probably the issue.

Re: Setup AWS Cloudwatch Monitoring and Alerts Using Bash Scripts

#22

Post by coredog64 »

void_mint wrote: post #19
rantwasp wrote: post #7

Earlier quoted context omitted.

SAM is cloudformation. cloudformation is the thing to use if you’re on the AWS cloud

CloudFormation is without a doubt the worst cloud technology I have ever used.

It's at least second or third worst. Worst would be writing your own deployment tool that does what CloudFormation (or TF or Pulumi) do. Second worst would be writing a tool that uses a templating language to generate CloudFormation and only using that.

Re: Setup AWS Cloudwatch Monitoring and Alerts Using Bash Scripts

#23

Post by coredog64 »

High CPU alerts are terrible alerts. If I'm paying per instance, I want CPU utilization to be high. If it's low, I'm wasting money. So now what I need is an alert where it's not high, but somewhere between "high and too high". You know, like when there's an arbitrary spike because the Java is doing some GC. Or you have a one minute spike of traffic that fires an Ops Genie alert at 2am but auto-clears between when the on-call engineer wakes up and when they log in to check.

For the love of $DIETY, if you're going to set up CloudWatch monitoring, create custom metrics that map to your business outcomes and alert when those go off the rails.

Re: Setup AWS Cloudwatch Monitoring and Alerts Using Bash Scripts

#24

Post by vergessenmir »

coredog64 wrote: post #23

High CPU alerts are terrible alerts. If I'm paying per instance, I want CPU utilization to be high. If it's low, I'm wasting money. So now what I need is an alert where it's not high, but somewhere between "high and too high". You know, like when there's an arbitrary spike because the Java is doing some GC. Or you have a one minute spike of traffic that fires an Ops Genie alert at 2am but auto-clears between when the…

Custom Cloudwatch metrics are expensive to write to making them useful for coarse grained high level service metrics. If you can afford it go ahead but setting up some other cloud native monitoring service may be the way to go.

Re: Setup AWS Cloudwatch Monitoring and Alerts Using Bash Scripts

#25

Post by dimitar »

coredog64 wrote: post #23

High CPU alerts are terrible alerts. If I'm paying per instance, I want CPU utilization to be high. If it's low, I'm wasting money. So now what I need is an alert where it's not high, but somewhere between "high and too high". You know, like when there's an arbitrary spike because the Java is doing some GC. Or you have a one minute spike of traffic that fires an Ops Genie alert at 2am but auto-clears between when the…

If you are running some software that requires an instance, but is in not expected to create load you can put it in a burstable, and setup such an alert, so you know when it is time to upgrade.

Re: Setup AWS Cloudwatch Monitoring and Alerts Using Bash Scripts

#26

Post by void_mint »

coredog64 wrote: post #22
void_mint wrote: post #19

Earlier quoted context omitted.

CloudFormation is without a doubt the worst cloud technology I have ever used.

It's at least second or third worst. Worst would be writing your own deployment tool that does what CloudFormation (or TF or Pulumi) do. Second worst would be writing a tool that uses a templating language to generate CloudFormation and only using that.

Yeah I wasn't really considering home rolled stuff. Officially supported tech.

Re: Setup AWS Cloudwatch Monitoring and Alerts Using Bash Scripts

#27

Post by rantwasp »

void_mint wrote: post #19
rantwasp wrote: post #7

Earlier quoted context omitted.

SAM is cloudformation. cloudformation is the thing to use if you’re on the AWS cloud

CloudFormation is without a doubt the worst cloud technology I have ever used.

lol. what’s next? you used terraform and it was awesome?

Re: Setup AWS Cloudwatch Monitoring and Alerts Using Bash Scripts

#28

Post by throwawaaarrgh »

gizdan wrote: post #20
hughrr wrote: post #3

Earlier quoted context omitted.

As someone who spends two hours a day dealing with buggered terraform state and upgrading terraform and dealing with terraform bugs I can see it. It’s one of those things that really works pretty well but there are enough edge cases to make it slightly soul sucking.

This sounds like a lack of understanding of terraform. We use Terraform pretty heavily and I've rarely seen bad states across our whole org, and the few that I do see are usually people who don't know the core concepts (often non-devops engineers). Terraform has its faults, but it is the best in its class, especially when you need to manage infrastructure beyond a single cloud provider (e.g. we manage our datadog mon…

I work with a large group of engineers that manage a very large array of infrastructure. We see weird Terraform issues all the time. There's a multitude of ways that Terraform gets into a bad state and has to be fixed manually (in production). Even a Terraform expert runs into them, because it's not necessarily an issue "with Terraform", but with a buggy provider, or some feature of Terraform which wasn't tested well in certain scenarios, etc.

Terraform allows for too much complex configuration/operation, the codebases change too frequently, there's not enough testing, and even extremely simple operations fail in a way that can't be reverted automatically. In practice the tool is clunky, complicated, difficult, and unreliable. Whenever I run "terraform apply" I know I am rolling dice, and plan for how I'm going to recover everything if I need to (which was what Terraform was supposed to prevent!)

But at the same time, if lots of people need to manage the same infra, you really have to use some common tool. Bash scripts are a great fix for small isolated problems, but they don't scale.

Re: Setup AWS Cloudwatch Monitoring and Alerts Using Bash Scripts

#29

Post by jeppesen-io »

coredog64 wrote: post #23

High CPU alerts are terrible alerts. If I'm paying per instance, I want CPU utilization to be high. If it's low, I'm wasting money. So now what I need is an alert where it's not high, but somewhere between "high and too high". You know, like when there's an arbitrary spike because the Java is doing some GC. Or you have a one minute spike of traffic that fires an Ops Genie alert at 2am but auto-clears between when the…

Certainly not perfect, but I've had very good success alerting load avg over 120 to 150 percent of core count

What's nice, is it catches A variety disk issues as well

I'm sure not perfect for all cases but for me, most of them

Post reply on HN