Earlier quoted context omitted.
I wrote the parent post. It was a rhetorical question.
But it did not get the answer you expected. So a failed rhetorical question.
Exactly.
Alert-driven monitoring
41–48 of 48 posts
Re: Alert-driven monitoring
#42I work writing analytics and monitoring for industrial equipment. We have hundreds of sensors sending back realtime data. There was a period of time where people were writing alerts for the sake of it (i.e we have this sensor, when should we alert on it). Nowadays we're strictly failure mode driven, this has meant lots of sensors aren't used in the analytics. They are however available to the experts to plot them for…
We went through the same switch. Half our alerts had been firing for a while and nobody ever acted on them.
Re: Alert-driven monitoring
#43I certainly agree in spirit that the alerts are important, and should be actionable. But I wouldn't start at just "looking at the service" and then trying to define the first set of alerts. Instead I would move up a level and start with a SLO for the various "business level" metrics you might care about. Things like "request latency", "successful requests", etc. Then use the longer lookahead "error budget" burndowns…
SLO timelines are usually over 7d, 30d etc no? and also often don't work that great for backend services in my experience ... they can't give you the level of reactivity that defining alerts about things you care about give you. I'd argue that moving from that direction upwards to figure out what alerts to aggregate and define SLOs around, rather than the other way around in those cases.
Would love to hear more, since I largely used SLOs on backend services (which in turn called other services that also had their own SLOs).
As far as timespans for the error budget consumption, I’ve seen 1 hour -> 1 day -> 1 week. The 1 hour error budget rate would be a page and the others would be low priority.
So you could either keep that as the alerting and/or use the error budget “look ahead” to see if there are more specific alerts you need.
Re: Alert-driven monitoring
#44I have some thoughts here. I work for a startup; we have what I think is a fairly typical setup: metrics ingested from a variety of sources, fed into industry-standard metrics/dashboard solutions, triggering escalations to humans. It's fine and I'm happy we have it, but... The highest value source of alerting right now is one of our growth marketers who pays close attention to our CRM and product analytics tool and n…
you are defining the art of setting up SLOs for end user workflows. This is typically achieved with contract monitoring (top down). This article is focused on bottom up approach of fine tuning and setting up alerts
Re: Alert-driven monitoring
#45Good metrics and alerting systems are designed, from the top down. Not bottom up. Lots of metrics are typically available, but almost all of them are noise. Start with the business: what is important to the business ? What kind of failures are existential threats ? Then work your way down and design your metrics and alerts, instead of just throwing stuff at the wall. I’ve had to push back so many times with teams who…
This! Again, like all things in engineering, requires systematic approach of having distributed traces chart out your upstream and downstream dependencies.
you then carve out essential user flows from these system diagrams and only then, why you look at an alert in isolation can you tie the entire story, whether is this alert important, which user workflow does it break, what is the SLO on it.
Re: Alert-driven monitoring
#46I see the installation command is as follows:
curl -fsSL https://simpleobservability.com/install.sh | sudo bash -s --
I dont feel comfortable running this in prod.
Re: Alert-driven monitoring
#47I have some thoughts here. I work for a startup; we have what I think is a fairly typical setup: metrics ingested from a variety of sources, fed into industry-standard metrics/dashboard solutions, triggering escalations to humans. It's fine and I'm happy we have it, but... The highest value source of alerting right now is one of our growth marketers who pays close attention to our CRM and product analytics tool and n…
you are defining the art of setting up SLOs for end user workflows. This is typically achieved with contract monitoring (top down). This article is focused on bottom up approach of fine tuning and setting up alerts
Thanks for the pointer to SLOs; I've read more than a few posts about them but it never clicked, will look more closely.
My point, I think, is still that the overwhelming focus of the tools I've seen focus on the kind of fine-tuning/setup you are describing and not the things that I find most valuable. And I think that part of the problem is that it's easy to build technology around mechanics than judgement.
Re: Alert-driven monitoring
#48> The real core of infrastructure monitoring isn’t dashboards. It’s the alerts. “it’s not X it’s Y” at this point when I see this pattern in writing I assume most if not all of it is AI generated - same with em-dashes. This is not to discount the idea that alerts are more important than dashboards (I work directly in observability) - but just to say that I personally shut off reading anything else with these patterns…
THANK YOU, 1000% agree. I call this style “contrastive language”, and find it viscerally unpleasant. It is a pithy attempt to be overly attention-grabbing and “punchy”. Everything that is (was?) wrong with social media (sic LinkedIn) posts, even in the immediate lead up to when it became 100% LLM slop.