Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I don't think Elasticsearch is a good logging system

blog.sinkingpoint.com

Post reply on HN
Print view View original

81–90 of 117 posts

Re: I don't think Elasticsearch is a good logging system

#81

Post by CSDude »

denysvitali wrote: post #34

What's worse (and I've seen this trend a lot) is to use Elasticsearch for metrics. My god.

Elasticsearch, with a mapping that does not full text search indices disabled, and a relatively larger time, works very well for timeseries data. It does filtering & metric and bucket aggregations crazy fast. And it's very easy to add a new node. Just run on spot instances. We push event per request, which might have 100-500 unique keys, 100 millions of records per day, and no solution we've tried comes at the easiness of Elastic.

Try to replicate the freedom (no-schema, cardinality, adding a new node to cluster) of Elasticsearch with InfluxDB or others and you will hit cardinality problems real fast.

Re: I don't think Elasticsearch is a good logging system

#82

Post by piaste »

Sevii wrote: post #74
piaste wrote: post #8

This is spot on based on our experience. I would add that the default ELK settings aren't terribly log-friendly, and having to janitor index policies, sharding, lifecycle policies, VM resources, etc. etc., _which you have to do even with the managed Elastic Cloud offer_, is way too much effort just to find and aggregate your TimeoutExceptions. We moved to NewRelic and while its dashboards are not _quite_ as fancy or…

How are diagnostic logs not business critical? If you have an outage while your non-mission critical logs are offline what are you going to do?

If it's a live outage, we can still SSH into a machine and grep the console output / local rotating log files. (For that matter, I still prefer to do that when I'm just testing new stuff in dev/staging environments)

NewRelic "only" stores the logs for 30 days and displays them in a nice web UI with searching, alerting, sharing, and a bunch of other stuff. It's not like they cease to exist without it.

Re: I don't think Elasticsearch is a good logging system

#83

Post by hodgesrm »

willvarfar wrote: post #44
vimda wrote: post #18

Earlier quoted context omitted.

My point was that by the time you filter on keyword fields (and other exact matching fields), the number of logs is small enough that an efficient full text search isn't necessary. That doesn't mean that full text search itself isn't useful, just that maintaining an inverted index is overkill in the logging case

Completely agree. My gripe with ES is that it won't let you do post-pass filtering at all. If you create an index with a few keyword fields indexed and then some unindexed fields, you can't query the unindexed fields. Grafana's Loki seems to be exactly what we are looking for, although I haven't played with it.

There's also cLoki. It's a new project that puts a Loki gateway over a ClickHouse backend store. We're looking at it and plan a presentation from the author(s) at the next ClickHouse SF Bay Area Meetup.

https://github.com/lmangani/cLoki

Re: I don't think Elasticsearch is a good logging system

#84

Post by hintymad »

> Grafana Labs' Loki is very exciting. Instead of storing a costly Inverted Index, Loki only indexes on fields (the equivalent of keyword fields in ElasticSearch)

One can configure Elasticsearch to index only on fields too, so I'm not sure "only indexes on fields" is a differentiating factor. The real advantage of Elasticsearch, or any search engine in general, is arbitrary boolean filter, as many log aggregation systems have started to use inverted index too. In addition, Elasticsearch has its own column-oriented data structure specifically for aggregation. Static sharding is a problem, but is not necessarily a big one, as many companies do not have enough scale to reach the problem yet.

BTW, we should really take a grain of salt on what Uber claims and what they do use. Case in point, they internally used Elasticsearch for years to aggregate all the logs in their marketplace for both real-time use cases and historical data that spanned months. Their Pinot-based solution and the promotion-oriented GPU-db didn't go anywhere.

Re: I don't think Elasticsearch is a good logging system

#85

Post by annexrichmond »

I moved to a company that uses ElasticSearch as an observability stack. It has been such an awful experience. It’s just painful creating new dashboards and alerts, such that people avoid having to deal with it at all. One example is that alerts are completely independent from dashboards. It’s so difficult to get a full graph related to the alert condition to gather more context. The reason it’s like this, imo, is that they are trying to do too much with the same product and so each feature feels compromised.

At my last company I was very enthusiastic about monitoring where we were using Splunk and SignalFX. It was just fast, seamless, and reliable.

Re: I don't think Elasticsearch is a good logging system

#86

Post by justinsaccount »

> Uber has not open sourced this work so we are unable to benchmark it and see how it performs

I implemented their design here, specifically for importing zeek logs:

https://github.com/JustinAzoff/zeek-clickhouse

I don't have the elastic compatible query api though, or the smarts that auto materialize popular columns.

It works though, does a good job at soaking up any sort of log type and handling fields being added or removed.

Re: I don't think Elasticsearch is a good logging system

#87

Post by buro9 »

jarpineh wrote: post #79
buro9 wrote: post #64

Earlier quoted context omitted.

One of the engineers has suggested trying: ```cat | promtail -stdin``` Though we like the idea as a feature request... "Have promtail support reading a file or directory and exiting once complete". So we've put that on the plan.

Ah, my thanks. I had somehow understood Promtail wrong. Its name, mention of Prometheus, scraping and such first tells me it's closer to something like node exporter. I had watched Loki release presentation and few others, but managed to still misunderstand. Oh well. Now, things make more sense. Moving forward with the installation... :)

Promtail is a process that watches a directory and effectively tails the files and sends them to Loki.

Agree that the name isn't ideal as it's not clear that this is a "tail and send text files" process.

But yup, Promtail is what you want.

Re: I don't think Elasticsearch is a good logging system

#88

Post by beardface »

denysvitali wrote: post #72
beardface wrote: post #39

Earlier quoted context omitted.

Why do you consider this a bad thing?

Because you're suddenly putting metrics (that should belong to a time-series database) into a full-text search database. The result is a dashboard that takes ages to load just to show a trend in values. I think it's using the wrong tool for the job, but maybe it's just me.

Elasticsearch is fine for time series data. A lot of tasks are actually easier with time series data. You add a field called `@timestamp` to your documents and a lot of analysis becomes possible, like date histograms, date range queries, ML jobs, etc.

Immutable time series data like logs and metrics are a great fit for Elasticsearch due to the way Lucene stores data. Documents in Lucene are immutable so an update in Elasticsearch is creates a new document and places a tombstone marker on the old one. Immutable data means you don't have to tolerate those inefficiencies.

Dashboards don't load the entire dataset by default. I can't remember what the exact default time range is but I think it's ~15 minutes or so. They're fairly quick to render in Kibana.

Elasticsearch is a great tool for observability data (logs, metrics, and APM data). Elastic's tooling makes a lot of this really easy in most cases.

Re: I don't think Elasticsearch is a good logging system

#89

Post by nemo44x »

denysvitali wrote: post #72
beardface wrote: post #39

Earlier quoted context omitted.

Why do you consider this a bad thing?

Because you're suddenly putting metrics (that should belong to a time-series database) into a full-text search database. The result is a dashboard that takes ages to load just to show a trend in values. I think it's using the wrong tool for the job, but maybe it's just me.

I think you may want to check your mappings/templates. There are a lot of data types for this kind of data and they don't rely on the inverted index that you would use for searching fields. Lucene, which Elasticsearch is built on, has a feature called "doc values" that stores data as column-oriented fields. This makes for the fast aggregations, sorting, and grouping for numeric and text fields.

One of the main strengths of Elasticsearch is that you can use it for searching and aggregating in a single query. But you need to ensure you are searching on fields that are indexed for search and sorting/aggregating on fields that are indexed for that.

Re: I don't think Elasticsearch is a good logging system

#90

Post by siddharthgoel88 »

Elasticsearch apart from being an overkill for logging, I feel has a high cost of maintenance as the system starts to scale. Somehow that cost is not very visible in the start. One of the Singapore based successful start-up (Grab) had blogged about some other pain-points a couple of years back ( https://engineering.grab.com/how-built-logging-stack ) as well.
Post reply on HN