Show HN: Zebrium – ML that catches software incidents and shows you root cause

Yeah, it's an interesting problem, bootstrapping such a thing.

When we started out, we took a stab at a model, and then collected about 50 incidents from about a dozen stacks. These were actually not of our creation, but from real-world application stacks, where the owners gave us the data and permission to use it. It was painstaking, but we were able to gather from them what comprised a valid root-cause indicator from their perspective, for each incident, and what did not.

So we collected these datasets and put them in what we call "the dojo". Then, we ran our software against it. We achieved about a 2/3 recall rate on detection+root-cause... meaning, a detection did not count as a true positive unless we also caught a root-cause indicator and put it into the incident report.

At that point, we reviewed what appeared to be systematic ways we were missing things, and improved the software substantially. This improvement process took us well north of 90% of recall on this original dataset. But of course we had not set aside a validation set and refused to look at it, so I don't know what the real recall should have been at that point - and I don't care. The data set was small enough that it was more important to learn everything possible and get it implemented, so we could get happy users, so we could get MORE data.

Getting such a well-curated dataset again is prohibitive in terms of effort at our size, and the kind of feedback we now collect automatically has huge noise built in. Some users hit "Like" when they're happy with an incident, and some complain when they're not, but most times no feedback is given at the incident level. Sometimes the feedback you get is they sign up, or they keep using you, or they pay you, or they leave. I think at scale we will have enough Like/Mute/Spam feedback at the incident level to get meaningful systematic answers from it; but we're not there yet, because this sort of feedback is generally sparse.

Regarding false positives - here, I think, what matters is this: each user has a finite amount of bandwidth / tolerance for noise. The important thing is that we not exceed that, while missing as little as possible. If they have a small environment and get three false positives and one true positive in a week, they may be happy. If they have a larger environment and get 30 false positives and 10 true positives in a week, they may be less happy; they might have been able to tolerate 10 false positives only before exceeding their bandwidth.

Thanks for the question, I could go on all day, and you've made me realize that this might make for a really interesting blog post, where I go into more detail still. I think there's a continuum in this sort of bootstrapping where you start with an art and end with a science, and do a little bit of each as you go through this awkward transition. Hit me up by email if you'd like to continue the conversation, and maybe let me pick your brain: larry@zebrium.com

There are three components to Ze that involve what you'd think of as ML. We use a few different "classical" techniques together; our focus is on keeping costs down while providing useful results on day one on even a newly-deployed custom application.

First is the structuring of logs: we have a four stage pipeline for structuring and each stage has greater importance depending on how many ground truth instances there are of a given event type in the dataset (these are unlabeled, of course). These stages include heuristics, reachability clustering, a naive Bayes classifier with global fitness function, and a modified LCS. When the data comes in it is laid down in tables with typed columns directly, without post-processing; later, table merges are considered asynchronously. This lets you start doing anomaly detection really well right away.

Next is anomaly detection. There are lots of dimensions we consider for AD; at the end of the day most of them boil down to some reflection of either “badness" or "rareness". We'd rather catch too many anomalies than miss one. This AD is run on both logs and metrics.

Finally, there's incident detection: here we look primarily at how much independence is there between different streams w.r.t. their anomalies, and when we see an unusually high correlation across channels, we raise an incident. Here, a naive Bayes model is used to set cutoffs, with streams of anomalies considered as point processes. You can provide feedback here for training, but it is optional.

Cool! How does your software compare to other similar tools like BigPanda, Moogsoft, Splunk ITSI?

This is a game changer. I've met the team and they've got something special here.

You can see one of their talks and a great discussion at a BayLISA.org meeting.

https://www.youtube.com/watch?v=gNiWtoxJ9iM

Hi paridso, we tackle the problem at a more foundational level. AIops tools are designed to speed up resolution and reduce noise, but they act on a feed of alerts/incidents. So in a sense they are dependant on the quality of alerts generated by monitoring tools. And typically a human will end up drilling down into data like logs and metrics to determine root cause. Our ML acts on the raw data, and has better coverage than typical log/metrics monitoring tools (including previously unknown failure modes). It also cuts time to root cause by generating complete incident summaries.