Live data from Hacker News

I don't think Elasticsearch is a good logging system

blog.sinkingpoint.com

11–20 of 117 posts

Re: I don't think Elasticsearch is a good logging system

#11
I believe the complaints here are a case of 'not using it correctly'.

The 'Reverse index' (Lucene's inverted index) is a fundamental data structure used to enable very fast search. Other data structures, like KD trees, are used for non-text data types. If you're not doing full text search, don't use `text` fields. If you're not querying the data, why store it in the first place?

Full text search for logs is incredibly useful for log files when combined with alerting. If you get log entries indicating that a disk is full, a service has stopped, or a user account is blocked, Elasticsearch can (with the right license) send emails or post on Slack.

Static mappings can be a pain but if you're constantly increasing the maximum field count for an index, use different indices for different log sources. Come up with an index pattern or alias that allows querying all those indices at the same time.

The main task here is reconciling the different logs so the index mappings are easily searchable, effectively as a union. Elastic Common Schema helps a lot with this. Elasticsearch mappings are easier to build when you first consider the queries you're going to be running on the data. You can then design the mapping with the right structure, field types, and settings.

Re: I don't think Elasticsearch is a good logging system

#12
post #5

Oh man, https://github.com/elastic/elasticsearch-py/issues/1734 is a disappointing read. I know ES wants to save their business, but alienating users isn't exactly the path to success.

FWIW, the ES team implemented a change that I requested via GitHub, and I’m just some guy who’s not even a paying customer. The change involved the behavior of force merge, and probably didn’t affect too many people. It’ll allow my employer (and hopefully some other companies) to make more efficient use of our disk space.

I’m really appreciative of the way they considered and then implemented my request.

Re: I don't think Elasticsearch is a good logging system

#13
post #8

This is spot on based on our experience. I would add that the default ELK settings aren't terribly log-friendly, and having to janitor index policies, sharding, lifecycle policies, VM resources, etc. etc., _which you have to do even with the managed Elastic Cloud offer_, is way too much effort just to find and aggregate your TimeoutExceptions. We moved to NewRelic and while its dashboards are not _quite_ as fancy or…

[deleted]

Re: I don't think Elasticsearch is a good logging system

#14
I can't say I agree with this article. Elasticsearch is a bit of a Swiss army knife and definitely needs some setup and configuring. But if you set it up properly, it does logs extremely well.

And it does actually come with useful defaults and well integrated tools. For example, if you use the various Beats agents to push data to Elasticsearch, they come with a default schema and out of the box dashboards that they can create for you. These are actually useful. I don't know that many tools in this space that have out of the box dashboards pre-configured, with matching schemas, etc.

Its also not that hard to create your own schemas and dashboards. Dynamic mapping is a rookie mistake that you might get away with if you don't have a lot of structured data. The classic mistake here would be mapping hundreds/thousands of fields. Not a great idea when you have billions of documents. The per document overhead adds up pretty quickly (disk, memory, indexing speed, querying overhead, etc). Much better is to only map those fields that matter to you and disable dynamic mapping for everything else.

And then there are features like rollups, data streams, index lifecycle management, etc. that make a lot of sense if you are storing logs at large scale. But you have to of course know to use them. As these were x-pack features, you won't be getting that with Opensearch.

Now there are of course many other solutions. But frankly it's a bit of a messy field of half integrated tools and a high DIY degree of how to set this up. Log analytics is only as good as the dashboards you get with them. Usually that means Grafana these days. How the data is stored is only part of the problem.

Grafana is basically an old fork of Kibana (v3) that has had lots of love and attention since then. It's a bit less feature rich perhaps but very capable. Incidentally, they went down the same re-licensing path as Elasticsearch: https://www.infoq.com/news/2021/04/grafana-licence-agpl/. So, you might want to think twice about putting that on a server unless you are completely fine having AGPLv3 licensed code close to where our own code is deployed (hint, if you are uncertain, you probably should not be doing that). LOKI is licensed similarly.

You can definitely gobble together some dashboards pretty easily using e.g. Grafana and any of its backends. But the out of the box experience for that is a lot of work figuring out what to use, how top operate it properly, etc. Elastic Cloud is expensive at scale but kind of plug and play on this front.

Probably the next best thing would be Opensearch. It's a fairly recent fork of Elasticsearch and Kibana. And unlike Elasticsearch/Kibana or Grafana/Loki, it remains under the Apache 2.0 license. Most of what you would need for a decent log analytics system is in there.

What you get by default from Google Cloud, AWS, etc. is pretty bare-bones in comparison in their cloud logging.

Re: I don't think Elasticsearch is a good logging system

#15
Kibana and Loki both load full messages in their list page. You end up loading megabytes (sometimes hundreds of megabytes) of data but it only displays a few kilobytes.

I don't know when we forgot the basic paradigm of list -> click -> details where loading the details is a separate http call. This is what datadog does, and the difference is staggering. Almost enough to make me abandon Kibana/Elastic and pay datadog.

I can't let a ELK discussion pass without mentioning vector.dev (https://vector.dev) which I'm not affiliated with aside from being a very happy user (for log ingestion).

Re: I don't think Elasticsearch is a good logging system

#17

I believe the complaints here are a case of 'not using it correctly'. The 'Reverse index' (Lucene's inverted index) is a fundamental data structure used to enable very fast search. Other data structures, like KD trees, are used for non-text data types. If you're not doing full text search, don't use `text` fields. If you're not querying the data, why store it in the first place? Full text search for logs is incredibl…

I have little knowledge of the log aggregation domain, but generally indices are great for read mostly loads. It seems to me that for log aggregation writes are more frequent than searches; cheap writes and the occasional brute force search.

For alerting you might better off running each new line against a set of filters/watchers. It seems wasteful to run it after indexing.

Again, no experience or knowledge on the domain, so I might be completely off.

Re: I don't think Elasticsearch is a good logging system

#18

I believe the complaints here are a case of 'not using it correctly'. The 'Reverse index' (Lucene's inverted index) is a fundamental data structure used to enable very fast search. Other data structures, like KD trees, are used for non-text data types. If you're not doing full text search, don't use `text` fields. If you're not querying the data, why store it in the first place? Full text search for logs is incredibl…

My point was that by the time you filter on keyword fields (and other exact matching fields), the number of logs is small enough that an efficient full text search isn't necessary. That doesn't mean that full text search itself isn't useful, just that maintaining an inverted index is overkill in the logging case

Re: I don't think Elasticsearch is a good logging system

#19
post #7

I often find myself wondering if all this is really that much better than the old rsyslog/syslog-ng stuff we used to do. These days its all some timeseries on top of journald, has anyone tested the remote journald options? Also at what point do we break logs out from metrics? Trying to do everything in one tool is so un-*nix.

What we do internally is use syslog-ng ( https://github.com/syslog-ng/syslog-ng ) to read the journald socket and push to a remote and into Kafka. I think journald works well as a structured logging tool, but it's certainly deficient in other ways

I have looked into ElasticSearch + Kibana as a solution to aggregate logs. There may be plenty of choice to replace ElasticSearch(ClickHouse, even Postgres, heck even journald), but a nice UI where you can simply search for that random piece of text you need to sift through the logs is the red herring.

Until now, I have not seen a web interface to log as powerful as Kibana that can work with anything other than ElasticSearch.

This is why I chose to stop my search and pay for Datadog to do this correctly, and simply allow me to search for that keyword on logs when I need it the most(and not worry about whether I indexed stuff correctly, or balanced some whatever in ElasticSearch, or remembered to setup something far too technical for a log system). Datadog allows you to keep a short periods worth of data in the index and "expire" old content into archives while retaining the ability to add them back to index if needed for any investigation.

Re: I don't think Elasticsearch is a good logging system

#20
post #15

Kibana and Loki both load full messages in their list page. You end up loading megabytes (sometimes hundreds of megabytes) of data but it only displays a few kilobytes. I don't know when we forgot the basic paradigm of list -> click -> details where loading the details is a separate http call. This is what datadog does, and the difference is staggering. Almost enough to make me abandon Kibana/Elastic and pay datadog.…

+1 for vector. Very stable, performing and feature-rich.
Post reply on HN