Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Alert-driven monitoring

simpleobservability.com

Post reply on HN
Print view View original

31–40 of 48 posts

Re: Alert-driven monitoring

#31

Post by solatic »

Not all alerts are created equal. You should generally have three levels of alerts: critical (which pages somebody, time-to-fix should be ASAP), warning (creates a ticket, time-to-fix should be within a few days), and suspicious (does not notify, appear only on an alert dashboard). The suspicious alerts are there to help guide your investigation on a critical or warning alert.

Each critical and warning alert should link to an "interactive runbook" - a dashboard that combines text instructions along with graphs showing real-time data.

Doing this at scale, correctly, requires both alerts-as-code and dashboards-as-code, which almost nobody does because nobody treats higher-level configuration languages (jsonnet, CUE...) with the attention and respect they deserve /cries-in-yaml

Re: Alert-driven monitoring

#32

Post by beachy »

kylemaxwell wrote: post #18

I like the ideas, but either it’s entirely LLM written or the writer has internalized “LLM voice”. At this point that is more distracting than helpful.

The prompt: "take existing decades-old knowledge about best practices in setting up alerts and spin it into a multi-page article presenting it as somehow novel, to assist our submarine marketing efforts".

Re: Alert-driven monitoring

#34

Post by mystifyingpoi »

baalimago wrote: post #29

My solution to this is to have leveled alerts. Some are... recommendations, the ones which you look at with a glance to get a heads up about something being wrong. These are the ones which OP would claim cause alert fatigue, most likely. Then I have a second level of this, the superpanic. Here is the "true" alert, which means "drop all things, fix this now". On every superpanic, there are stricter routines which inte…

> there are stricter routines which intentionally cause friction, such as creating tickets

While this sounds sensible, in my experience it often becomes just a convoluted punishment for people involved in the alert firing. In general, people are lazy (sorry), and if alert makes them fill up post-mortem forms and attend mandatory late meetings with management why something got triggered for any reason - 99% of people will push to remove the alert altogether, or at least lower the priority. I haven't found a solution that doesn't include a complete overhaul of organization in the enterprise.

Re: Alert-driven monitoring

#35

Post by muvlon »

analogpixel wrote: post #5

> Alerts should be actionable. If no action can or should be taken, then the alert is not needed. Also, the best alerts come from looking at actual failures you had and not trying to make up "good alerts" from thin air. After you have an outage, figure out what alerts would have caught it, and implement those.

This is one category of good alerts, but not everything.

I think alerts are to ops what tests are to dev. You have "unit alerts" for some small thing like the disk usage on a single host, "integration alerts" like literally "does the page load?" and then what you describe are "regression alerts", trying to prevent something that went wrong once from going wrong again. These are great but just like you wouldn't have 100% regression tests, I think it's also smart to try to get ahead of failures and have some common sense alerts defined.

Re: Alert-driven monitoring

#36

Post by smetj »

Depends on what you are monitoring but let's assume an API endpoint. Collect and monitor the RED metrics with detailed dimensions in combination with blackbox monitoring simulating client transactions as realistic possible and alert only on those 2 types.

When that happens, fire off a battery of diagnostic checks which you have collected over time to pinpoint the cause.

What if the diagnostics checks don't reveal the issue? There is still value since you know these are not the reason so no time is wasted re-evaluating them. Where to get these diagnostic checks from? Well, what's the first thing responding engineers do? Open the CLI and troubleshoot. Those are your diagnostic checks. Collect, automate, capture the domain specific knowlegde and democratize it.

Re: Alert-driven monitoring

#37

Post by thunky »

loloquwowndueo wrote: post #33
thunky wrote: post #26

Earlier quoted context omitted.

Nobody cares that you care, because you're not adding anything to the conversation.

I’m answering a direct question posed by the parent post. What’s your excuse?

I wrote the parent post. It was a rhetorical question.

Re: Alert-driven monitoring

#38

Post by ajanuary »

stackskipton wrote: post #13
esafak wrote: post #12

Earlier quoted context omitted.

I know something is going to happen if disk space runs out; I don't need to experience it first.

Sure, but for every alert, there is exception. ElasticSearch for example can be configured using ILM policies to fill up the disk then start deleting old records. I don't need to be woken up for disk filling up on those nodes. Even worse is CPU/RAM alerts.

The number of times I've had to explain how the JVM heap works...

Re: Alert-driven monitoring

#39

Post by radiator »

thunky wrote: post #37
loloquwowndueo wrote: post #33

Earlier quoted context omitted.

I’m answering a direct question posed by the parent post. What’s your excuse?

I wrote the parent post. It was a rhetorical question.

But it did not get the answer you expected. So a failed rhetorical question.
Post reply on HN