Of course, it's not! It is a search engine and who knows its history will know it was not intended to be a logging system.
I don't think Elasticsearch is a good logging system
71–80 of 117 posts
Re: I don't think Elasticsearch is a good logging system
#72What's worse (and I've seen this trend a lot) is to use Elasticsearch for metrics. My god.
Why do you consider this a bad thing?
Because you're suddenly putting metrics (that should belong to a time-series database) into a full-text search database.
The result is a dashboard that takes ages to load just to show a trend in values.
I think it's using the wrong tool for the job, but maybe it's just me.
Re: I don't think Elasticsearch is a good logging system
#73Earlier quoted context omitted.
Grafana?
Haven’t played with the logs part of Grafana recently, but would it work on top of say Clickhouse? I thought it was more tuned for the Loki use case… is it not?
I'd agree, it isnt good for exploratory queries.
But if you have some predefined ES queries for correlating log messages to metrics it can be useful to have it all in one dashboard.
Re: I don't think Elasticsearch is a good logging system
#74This is spot on based on our experience. I would add that the default ELK settings aren't terribly log-friendly, and having to janitor index policies, sharding, lifecycle policies, VM resources, etc. etc., _which you have to do even with the managed Elastic Cloud offer_, is way too much effort just to find and aggregate your TimeoutExceptions. We moved to NewRelic and while its dashboards are not _quite_ as fancy or…
How are diagnostic logs not business critical? If you have an outage while your non-mission critical logs are offline what are you going to do?
Re: I don't think Elasticsearch is a good logging system
#75Just use Scalyr.
A million times this. Crazy fast flexible searching. I'm nervous they'll lose focus on the logging product after the SentinelOne acquisition but so much easier than anything else.
Re: I don't think Elasticsearch is a good logging system
#76If I remember correctly, the core Splunk patents are from around 2006. Once the patents expire, could an open source project duplicate the Splunk indexing structures? As much as Elasticsearch has tried, Splunk is better for logs.
Re: I don't think Elasticsearch is a good logging system
#77Kibana and Loki both load full messages in their list page. You end up loading megabytes (sometimes hundreds of megabytes) of data but it only displays a few kilobytes. I don't know when we forgot the basic paradigm of list -> click -> details where loading the details is a separate http call. This is what datadog does, and the difference is staggering. Almost enough to make me abandon Kibana/Elastic and pay datadog.…
Looks like Vector has a Loki outputter.
Re: I don't think Elasticsearch is a good logging system
#78Earlier quoted context omitted.
My point was that by the time you filter on keyword fields (and other exact matching fields), the number of logs is small enough that an efficient full text search isn't necessary. That doesn't mean that full text search itself isn't useful, just that maintaining an inverted index is overkill in the logging case
Completely agree. My gripe with ES is that it won't let you do post-pass filtering at all. If you create an index with a few keyword fields indexed and then some unindexed fields, you can't query the unindexed fields. Grafana's Loki seems to be exactly what we are looking for, although I haven't played with it.
Will runtime fields help you with post-pass filtering? https://www.elastic.co/blog/introducing-elasticsearch-runtim...
Re: I don't think Elasticsearch is a good logging system
#79Earlier quoted context omitted.
Hi. I have a question. I have put some time trying to learn the Grafana stack. Heaviness of elastic both self hosted and cloud version has led me to seek for alternatives. Is there a way to get logs into Loki that can be run offline, when jobs have finished and hosts have been shutdown? It seems Loki recommends Promtail agent which looks a bit heavy-handed ( https://grafana.com/docs/loki/latest/getting-started/get-lo…
One of the engineers has suggested trying: ```cat | promtail -stdin``` Though we like the idea as a feature request... "Have promtail support reading a file or directory and exiting once complete". So we've put that on the plan.
Ah, my thanks. I had somehow understood Promtail wrong. Its name, mention of Prometheus, scraping and such first tells me it's closer to something like node exporter. I had watched Loki release presentation and few others, but managed to still misunderstand. Oh well. Now, things make more sense. Moving forward with the installation... :)
Re: I don't think Elasticsearch is a good logging system
#80Earlier quoted context omitted.
Completely agree. My gripe with ES is that it won't let you do post-pass filtering at all. If you create an index with a few keyword fields indexed and then some unindexed fields, you can't query the unindexed fields. Grafana's Loki seems to be exactly what we are looking for, although I haven't played with it.
You can update the index with the new field specification and reindex your content. Your complaint really doesn't make sense, how would you query an unindexed field? Elasticsearch is a _search_ engine, which means it needs to index content that is to be discoverable. What you're saying with unindexed fields is you're completely fine with those not being included in any search or filtering.
Your response is, "Why can't you perfectly predict which columns/keywords will be necessary later on, or otherwise re-index the whole system at the drop of a hat for one query? And why would you think a search engine would be able to perform an unindexed, ad hoc search?"
Compared to my experience, you have a foundational difference of understanding with how systems are actually used.