Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Do you have problems finding stuff across logging/monitoring systems?

news.ycombinator.com

Post reply on HN
Print view View original

1–10 of 10 posts

Ask HN: Do you have problems finding stuff across logging/monitoring systems?

#1

Post by El_Mo »

I constantly find myself struggling to answer questions that require me to correlate data across multiple logging and monitoring products (Splunk, Pingdom, Elasticsearch, Icinga, Prometheus, etc., etc.).

For example, I’m often asked to get NFR figures, do SLA reporting or trace events across stacks for root cause analysis. It is not easy in a big shop. More and more I’m noticing that no matter how well thought out these tools are, they can act as data silos pretty darn quickly.

Do you face this issue? Do you easily solve it? If so, what tools do you use?

I’m asking ‘cause I have an idea for a new side project, but I don’t want to build something that only me will find useful (sadly, I’ve done THAT before).

I would be VERY grateful for any comments or PMs on this topic.

Re: Ask HN: Do you have problems finding stuff across logging/monitoring systems?

#2

Post by itamarst »

I've designed a logging system designed to address some of these issues (https://eliot.readthedocs.io/en/1.2.0/) and I know people have e.g. used the trace ID it provides to correlate Sentry errors with logging traces. Not a full solution, obviously, but if you do go this route do borrow some ideas if it's helpful.

As far as a business goes: it's a real problem. Whether people will pay for it, that's something else. You may wish to spend some time reading up on questions real people ask about these systems (e.g. on relevant forums) to see what patterns you notice. https://stackingthebricks.com/vintage-sales-safari-in-action... and other articles there have some examples of doing this sort of research.

Re: Ask HN: Do you have problems finding stuff across logging/monitoring systems?

#4

Post by brudgers »

The solution space for a cross logging/monitoring systems tool has a non-polynomial number of permutations. The permutations of (Splunk, Pingdom, Elasticsearch, Icinga, Prometheus, etc., etc.) is at least 6! or 720.

Each is a snowflake. Each will change independently over time as each product changes. Each user will have different configurations and use cases. There is a combinatorial explosion.

It is an interesting problem. But there isn't likely to be a general solution.

Re: Ask HN: Do you have problems finding stuff across logging/monitoring systems?

#5

Post by El_Mo »

brudgers wrote: post #4

The solution space for a cross logging/monitoring systems tool has a non-polynomial number of permutations. The permutations of (Splunk, Pingdom, Elasticsearch, Icinga, Prometheus, etc., etc.) is at least 6! or 720. Each is a snowflake. Each will change independently over time as each product changes. Each user will have different configurations and use cases. There is a combinatorial explosion. It is an interesting…

Thanks for the comment. It is true there is the possibility of too many permutations. I suppose the other way of approaching this is to let the tools do what they do, but export or forward on specific data that such a cross-logging/monitoring system could deal with for very specific use cases, like those I mentioned. It is these use cases I'm looking to validate.

Re: Ask HN: Do you have problems finding stuff across logging/monitoring systems?

#7

Post by El_Mo »

itamarst wrote: post #2

I've designed a logging system designed to address some of these issues ( https://eliot.readthedocs.io/en/1.2.0/ ) and I know people have e.g. used the trace ID it provides to correlate Sentry errors with logging traces. Not a full solution, obviously, but if you do go this route do borrow some ideas if it's helpful. As far as a business goes: it's a real problem. Whether people will pay for it, that's something else…

Thanks alot for the pointer; you are right, some very interesting ideas in that.

Having already tried a few products in the whole logging and monitoring space, I do find it is pretty challenging to operate here. I realise having paid solution can be a barrier ... unless there are compelling benefits, which is what I'm trying to define.

Re: Ask HN: Do you have problems finding stuff across logging/monitoring systems?

#9

Post by stympy »

El_Mo wrote: post #5
brudgers wrote: post #4

The solution space for a cross logging/monitoring systems tool has a non-polynomial number of permutations. The permutations of (Splunk, Pingdom, Elasticsearch, Icinga, Prometheus, etc., etc.) is at least 6! or 720. Each is a snowflake. Each will change independently over time as each product changes. Each user will have different configurations and use cases. There is a combinatorial explosion. It is an interesting…

Thanks for the comment. It is true there is the possibility of too many permutations. I suppose the other way of approaching this is to let the tools do what they do, but export or forward on specific data that such a cross-logging/monitoring system could deal with for very specific use cases, like those I mentioned. It is these use cases I'm looking to validate.

Check out https://bigpanda.io -- it might be in line with what you are thinking.

Re: Ask HN: Do you have problems finding stuff across logging/monitoring systems?

#10

Post by SirLJ »

I am building stock trading robots and regularly go trough the logs to make sure they are working as designed and in the begging it was a lot of wasted time, because I was logging everything I could think off... Begin lazy, I soon decided to log only the critical stuff, so now it takes me only couple of minutes per week, so the moral of the story is simply KISS :-)
Post reply on HN