Live data from Hacker News

I don't think Elasticsearch is a good logging system

blog.sinkingpoint.com

41–50 of 117 posts

Re: I don't think Elasticsearch is a good logging system

#41
post #26

Earlier quoted context omitted.

I have looked into ElasticSearch + Kibana as a solution to aggregate logs. There may be plenty of choice to replace ElasticSearch(ClickHouse, even Postgres, heck even journald), but a nice UI where you can simply search for that random piece of text you need to sift through the logs is the red herring. Until now, I have not seen a web interface to log as powerful as Kibana that can work with anything other than Elast…

Grafana?

Haven’t played with the logs part of Grafana recently, but would it work on top of say Clickhouse? I thought it was more tuned for the Loki use case… is it not?

Re: I don't think Elasticsearch is a good logging system

#42
post #35

What's worse (and I've seen this trend a lot) is to use Elasticsearch for metrics. My god.

> (and I've seen this trend a lot) any idea why? Seems like an odd approach to me.

Because that's what Elastic is positioning Elasticsearch for. If you look at the features being added these last few years, so much is about time series stuff — aggregation, time-based partitioning, warm/cold storage, etc. Relatively little focus on structured search.

Re: I don't think Elasticsearch is a good logging system

#43
post #18

I believe the complaints here are a case of 'not using it correctly'. The 'Reverse index' (Lucene's inverted index) is a fundamental data structure used to enable very fast search. Other data structures, like KD trees, are used for non-text data types. If you're not doing full text search, don't use `text` fields. If you're not querying the data, why store it in the first place? Full text search for logs is incredibl…

My point was that by the time you filter on keyword fields (and other exact matching fields), the number of logs is small enough that an efficient full text search isn't necessary. That doesn't mean that full text search itself isn't useful, just that maintaining an inverted index is overkill in the logging case

This has been my experience. Obviously different people use logs for different things, but in my case I'm usually looking for information about something bad that already happened, within a very specific window of time, and within a specific section of the application. 99% of the time, that means I am filtering until there are only a handful of entries that match, at which point I don't need full text search at all.

Re: I don't think Elasticsearch is a good logging system

#44
post #18

I believe the complaints here are a case of 'not using it correctly'. The 'Reverse index' (Lucene's inverted index) is a fundamental data structure used to enable very fast search. Other data structures, like KD trees, are used for non-text data types. If you're not doing full text search, don't use `text` fields. If you're not querying the data, why store it in the first place? Full text search for logs is incredibl…

My point was that by the time you filter on keyword fields (and other exact matching fields), the number of logs is small enough that an efficient full text search isn't necessary. That doesn't mean that full text search itself isn't useful, just that maintaining an inverted index is overkill in the logging case

Completely agree.

My gripe with ES is that it won't let you do post-pass filtering at all. If you create an index with a few keyword fields indexed and then some unindexed fields, you can't query the unindexed fields.

Grafana's Loki seems to be exactly what we are looking for, although I haven't played with it.

Re: I don't think Elasticsearch is a good logging system

#45

I can't say I agree with this article. Elasticsearch is a bit of a Swiss army knife and definitely needs some setup and configuring. But if you set it up properly, it does logs extremely well. And it does actually come with useful defaults and well integrated tools. For example, if you use the various Beats agents to push data to Elasticsearch, they come with a default schema and out of the box dashboards that they c…

Anyone that's used Splunk for any amount of time will feel like they are in a straightjacket with ES.

Anyone stuck paying for Splunk will think they probably should be.

Re: I don't think Elasticsearch is a good logging system

#46
post #35

What's worse (and I've seen this trend a lot) is to use Elasticsearch for metrics. My god.

> (and I've seen this trend a lot) any idea why? Seems like an odd approach to me.

I was interviewing earlier this year and saw at least three companies actively planning to migrate from their existing relational databases into Elasticsearch.

It seems to commonly be perceived as a panacea of databases.

Re: I don't think Elasticsearch is a good logging system

#47
About the field limit, an application shouldn't have more than 1000 fields in their structured logging.

Usually when this happens is because the source uses what could be considered user input as keys, for example in access logs with stuff such as headers o query parameter, and you make a new field for each header or query parameter. In the latter case all you need to exhaust your available fields in the index is some nasty bot trying random query strings on your site. This can be easily solved in the ingest/logstash pipeline

It is true that you have to spend time to carefully map and process the documents you're ingesting in elasticsearch, but once you do I don't think there's any other solution with the same performance and features for logs, especially considering the licensed features (like alerting from anomaly detection machine learning jobs)

The elastic stack is complex and might be hard to grasp (their own training courses are a great help though), requires significant computing resources, and managing a big deployment can be a full time job, but once implemented correctly you can do awesome stuff with it.

Loki is a simpler and slower solution which does less things, so if you don't need what elastic provides, it could be a better fit as it's probably easier to manage

Re: I don't think Elasticsearch is a good logging system

#49
post #18

Earlier quoted context omitted.

My point was that by the time you filter on keyword fields (and other exact matching fields), the number of logs is small enough that an efficient full text search isn't necessary. That doesn't mean that full text search itself isn't useful, just that maintaining an inverted index is overkill in the logging case

Completely agree. My gripe with ES is that it won't let you do post-pass filtering at all. If you create an index with a few keyword fields indexed and then some unindexed fields, you can't query the unindexed fields. Grafana's Loki seems to be exactly what we are looking for, although I haven't played with it.

You can update the index with the new field specification and reindex your content.

Your complaint really doesn't make sense, how would you query an unindexed field? Elasticsearch is a _search_ engine, which means it needs to index content that is to be discoverable. What you're saying with unindexed fields is you're completely fine with those not being included in any search or filtering.

Re: I don't think Elasticsearch is a good logging system

#50
post #18

Earlier quoted context omitted.

My point was that by the time you filter on keyword fields (and other exact matching fields), the number of logs is small enough that an efficient full text search isn't necessary. That doesn't mean that full text search itself isn't useful, just that maintaining an inverted index is overkill in the logging case

Completely agree. My gripe with ES is that it won't let you do post-pass filtering at all. If you create an index with a few keyword fields indexed and then some unindexed fields, you can't query the unindexed fields. Grafana's Loki seems to be exactly what we are looking for, although I haven't played with it.

I guess what they want is to use the elasticsearch query language but let it optionally do “expensive” non indexed filtering like a SQL database would let you do.

Without knowing for sure I imagine they originally expected the application side to handle this but many of the current solutions don’t do that. And they expose and overload the elastic search query language as the primary search interface with no additional app logic. The elastic search query “is” the search application.

Making some assumptions but might reconcile the different viewpoints on why it does or doesn’t make sense.

Post reply on HN