Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Alert-driven monitoring

simpleobservability.com

Post reply on HN
Print view View original

21–30 of 48 posts

Re: Alert-driven monitoring

#21

Post by tuetuopay »

thunky wrote: post #19
kylemaxwell wrote: post #18

I like the ideas, but either it’s entirely LLM written or the writer has internalized “LLM voice”. At this point that is more distracting than helpful.

Do we really need this comment for every article? Who cares if AI wrote it if you like the ideas?

The issue is, there’s not a lot of meat in this article. Anyone who’s done any amount of SRE can perfectly articulate alert fatigue in way less words.

Yet the article doesn’t tackle at all the hard part: making alerts that are actually meaningful. They handwave it instead of giving actual advice. This post is a good intro, but I didn’t "walk away" with anything useful.

This is why, in this case, AI is important. Someone puts in an effort to write a short article (if a bit wordy) that can be used by e.g. beginners or managers? Good! I’m not the target audience. But if it’s the output of AI, what’s the intent?

Re: Alert-driven monitoring

#22

Post by loloquwowndueo »

thunky wrote: post #19
kylemaxwell wrote: post #18

I like the ideas, but either it’s entirely LLM written or the writer has internalized “LLM voice”. At this point that is more distracting than helpful.

Do we really need this comment for every article? Who cares if AI wrote it if you like the ideas?

I care.

Re: Alert-driven monitoring

#23

Post by adrian_b »

prpl wrote: post #16

> The real core of infrastructure monitoring isn’t dashboards. It’s the alerts. “it’s not X it’s Y” at this point when I see this pattern in writing I assume most if not all of it is AI generated - same with em-dashes. This is not to discount the idea that alerts are more important than dashboards (I work directly in observability) - but just to say that I personally shut off reading anything else with these patterns…

I think that this is sad, because it is a useful pattern.

It is very frequent to find things about which a majority of the people wrongly believe that they are X, but in fact they are Y.

In such cases, you must point to them that "it's not X it's Y".

There are a few alternative ways to formulate this, but the alternatives are typically longer and more complex.

The same happens with em-dashes, which have valid uses and one should not care that there exist some people who are not familiar with the classic ways of using punctuation.

I do not believe that the right solution is to attempt to use more convoluted expressions or inappropriate punctuation in order to avoid to be accused of being a clanker.

Re: Alert-driven monitoring

#24

Post by thunky »

tuetuopay wrote: post #21
thunky wrote: post #19

Earlier quoted context omitted.

Do we really need this comment for every article? Who cares if AI wrote it if you like the ideas?

The issue is, there’s not a lot of meat in this article. Anyone who’s done any amount of SRE can perfectly articulate alert fatigue in way less words. Yet the article doesn’t tackle at all the hard part: making alerts that are actually meaningful. They handwave it instead of giving actual advice. This post is a good intro, but I didn’t "walk away" with anything useful. This is why, in this case, AI is important. Some…

The aricle is a marketing page under a "Winning with us" section right next to a "CEO Page" that describes a CEO pitch. I really don't think this article is very different from thousands of others like it that were published before AI.

Re: Alert-driven monitoring

#25

Post by trustfundbaby »

ecoffey wrote: post #17

I certainly agree in spirit that the alerts are important, and should be actionable. But I wouldn't start at just "looking at the service" and then trying to define the first set of alerts. Instead I would move up a level and start with a SLO for the various "business level" metrics you might care about. Things like "request latency", "successful requests", etc. Then use the longer lookahead "error budget" burndowns…

SLO timelines are usually over 7d, 30d etc no? and also often don't work that great for backend services in my experience ... they can't give you the level of reactivity that defining alerts about things you care about give you. I'd argue that moving from that direction upwards to figure out what alerts to aggregate and define SLOs around, rather than the other way around in those cases.

Re: Alert-driven monitoring

#27

Post by tuetuopay »

thunky wrote: post #24
tuetuopay wrote: post #21

Earlier quoted context omitted.

The issue is, there’s not a lot of meat in this article. Anyone who’s done any amount of SRE can perfectly articulate alert fatigue in way less words. Yet the article doesn’t tackle at all the hard part: making alerts that are actually meaningful. They handwave it instead of giving actual advice. This post is a good intro, but I didn’t "walk away" with anything useful. This is why, in this case, AI is important. Some…

The aricle is a marketing page under a "Winning with us" section right next to a "CEO Page" that describes a CEO pitch. I really don't think this article is very different from thousands of others like it that were published before AI.

Huh, thanks, I failed to see it on my phone. Down the trash the article goes then.

Re: Alert-driven monitoring

#28

Post by spike021 »

prpl wrote: post #16

> The real core of infrastructure monitoring isn’t dashboards. It’s the alerts. “it’s not X it’s Y” at this point when I see this pattern in writing I assume most if not all of it is AI generated - same with em-dashes. This is not to discount the idea that alerts are more important than dashboards (I work directly in observability) - but just to say that I personally shut off reading anything else with these patterns…

I'm not sure in this case it's AI per se so much as a change over time.

At the first role I ever had 10+ years ago, we had a TV in our team's office space constantly showing our dashboard for our critical services and health. We still had alerting monitors but it felt like those alarms were for important issues (like sev-2 or worse).

the last couple roles I've had we don't constantly look at our dashboards unless our monitors keep ringing us with alerts. We have also had more monitors in general than the first role I mentioned. Occasionally if another team asks us if we're affected by something we'll look at the dashboards we have to make sure we don't have a monitoring gap.

Re: Alert-driven monitoring

#29

Post by baalimago »

My solution to this is to have leveled alerts. Some are... recommendations, the ones which you look at with a glance to get a heads up about something being wrong. These are the ones which OP would claim cause alert fatigue, most likely.

Then I have a second level of this, the superpanic. Here is the "true" alert, which means "drop all things, fix this now". On every superpanic, there are stricter routines which intentionally cause friction, such as creating tickets about said superpanic, potentially hosting post mortems etc. This additional manual labour encourages tweaking the levels of the superpanic so that they sometimes are more lack, sometimes stricter, depending on the quality of the deployed services + the current load.

What signals a superpanic? Key valuable functionality being offline. Off-site uptime-checkers assuring that all primary domains resolve + serve traffic, mostly. Also crontime integration tests of core functionality. Stuff like that.

Re: Alert-driven monitoring

#30

Post by hyperadvanced »

I used to believe in alert fatigue, because you’re frequently told to repeat the line: if you have too many alerts, eventually everyone will stop paying attention to them.

I have tons of alerts at work. They go to specialized slack channels that I can look at if I need. We have on call escalation paths for critical ones and housekeeping duties for the ones that require engineers to perform a maintenance task. We have the hell channels that are 99.99% flapping, if you ever need that.

I find that observability in general has an extremely linear marginal reward curve, it basically always justifies the effort you put into setting it up.

Post reply on HN