Why would one need to check Datadog every morning? Wouldn't alerts fire if there was something to do?
Well, the industry standard solution is correct monitoring and alerting. This doesn’t sound like “the right way”.
I'm Too Lazy to Check Datadog Every Morning, So I Made AI Do It
11–17 of 17 posts
Re: I'm Too Lazy to Check Datadog Every Morning, So I Made AI Do It
#12Why would one need to check Datadog every morning? Wouldn't alerts fire if there was something to do?
Almost no one actually knows how to set up their monitoring. Like, they know the words but not the full picture or how the pieces should actually fit together. Then they do shit like this to try and make up for that fact.
the ones that know do not check anything every morning
Re: I'm Too Lazy to Check Datadog Every Morning, So I Made AI Do It
#13> Total alerts/errors found: 7 Apps written in an exceptions language (Java, JavaScript, PHP, etc..) are really annoying to monitor as everything that isn't the happy path triggers an 'error'/'fatal' log/metric. Yes, you can technically work around it with (near) Go-level error verbosity (try/catches everywhere on every call) but I've never seen a team actually do that. Modern languages that don't throw exceptions fo…
> On this note, a login failure is not an error Login failure is like the most important error you'll track. A login failure isn't necessarily actionable but a spike of thousands of them for sure is. No single system has been more responsible for causing outages in my career than auth. And I get that it's annoying when they appear in your Rollbar but sometimes Login Failed is the only signal you get that something is…
Rather than login failures I would monitor login successes. A sharp decrease of successes likely points to some issue, but an increase in login failures might easily be someone trying tons of random credentials on your website (still not ideal, but much harder to act on)
Re: I'm Too Lazy to Check Datadog Every Morning, So I Made AI Do It
#14Earlier quoted context omitted.
> On this note, a login failure is not an error Login failure is like the most important error you'll track. A login failure isn't necessarily actionable but a spike of thousands of them for sure is. No single system has been more responsible for causing outages in my career than auth. And I get that it's annoying when they appear in your Rollbar but sometimes Login Failed is the only signal you get that something is…
Rather than login failures I would monitor login successes. A sharp decrease of successes likely points to some issue, but an increase in login failures might easily be someone trying tons of random credentials on your website (still not ideal, but much harder to act on)
Creating this metric/alert is practically a rite of passage for junior ops people who then get paged around 5pm.
Re: I'm Too Lazy to Check Datadog Every Morning, So I Made AI Do It
#15Why would one need to check Datadog every morning? Wouldn't alerts fire if there was something to do?
I'm not sure if this is what the writer was getting at, but I tend to check telemetry for my production applications regularly not because I'm looking for things that would fire alerts, but to keep a sense of what production looks like. Things like request rate, average latency, top request paths etc. It's not about knowing something is broken, it's about knowing what healthy looks like. Understanding what your code…
This is a good answer, and I agree that having a good production intuition like this is important. You're probably also right that having AI do it probably doesn't get that value.
I'm not sure I'd do this once a day. I tend to take note of things to build that intuition when I have other reasons to go and look at dashboards, and we have a weekly SLO review as a team, but perhaps there's a place for this in some way.
Re: I'm Too Lazy to Check Datadog Every Morning, So I Made AI Do It
#16Earlier quoted context omitted.
I'm not sure if this is what the writer was getting at, but I tend to check telemetry for my production applications regularly not because I'm looking for things that would fire alerts, but to keep a sense of what production looks like. Things like request rate, average latency, top request paths etc. It's not about knowing something is broken, it's about knowing what healthy looks like. Understanding what your code…
This is a good answer, and I agree that having a good production intuition like this is important. You're probably also right that having AI do it probably doesn't get that value. I'm not sure I'd do this once a day. I tend to take note of things to build that intuition when I have other reasons to go and look at dashboards, and we have a weekly SLO review as a team, but perhaps there's a place for this in some way.
Yeah, agreed. Daily isn't really necessary outside of initial launch and maybe a busy season. It's really just often enough to build a good sense of production use, and keep it up to date.
Re: I'm Too Lazy to Check Datadog Every Morning, So I Made AI Do It
#17> Total alerts/errors found: 7 Apps written in an exceptions language (Java, JavaScript, PHP, etc..) are really annoying to monitor as everything that isn't the happy path triggers an 'error'/'fatal' log/metric. Yes, you can technically work around it with (near) Go-level error verbosity (try/catches everywhere on every call) but I've never seen a team actually do that. Modern languages that don't throw exceptions fo…
> On this note, a login failure is not an error Login failure is like the most important error you'll track. A login failure isn't necessarily actionable but a spike of thousands of them for sure is. No single system has been more responsible for causing outages in my career than auth. And I get that it's annoying when they appear in your Rollbar but sometimes Login Failed is the only signal you get that something is…
> Login failure is like the most important error you'll track. A login failure isn't necessarily actionable but a spike of thousands of them for sure is.
Sounds like you agree with me. Re-read my comment. Errors are actionable individually. Warnings are actionable in aggregate.
You don't have to treat logs and metrics as separate, you can have rules on log counts without emitting a metric.