Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I don't think Elasticsearch is a good logging system

blog.sinkingpoint.com

Post reply on HN
Print view View original

21–30 of 117 posts

Re: I don't think Elasticsearch is a good logging system

#21

Post by rio517 »

vosper wrote: post #12
rio517 wrote: post #5

Oh man, https://github.com/elastic/elasticsearch-py/issues/1734 is a disappointing read. I know ES wants to save their business, but alienating users isn't exactly the path to success.

FWIW, the ES team implemented a change that I requested via GitHub, and I’m just some guy who’s not even a paying customer. The change involved the behavior of force merge, and probably didn’t affect too many people. It’ll allow my employer (and hopefully some other companies) to make more efficient use of our disk space. I’m really appreciative of the way they considered and then implemented my request.

That's good to hear.

Re: I don't think Elasticsearch is a good logging system

#22

Post by OldOneEye »

piaste wrote: post #8

This is spot on based on our experience. I would add that the default ELK settings aren't terribly log-friendly, and having to janitor index policies, sharding, lifecycle policies, VM resources, etc. etc., _which you have to do even with the managed Elastic Cloud offer_, is way too much effort just to find and aggregate your TimeoutExceptions. We moved to NewRelic and while its dashboards are not _quite_ as fancy or…

At high volumes, at my job, we have yet to find a good third-party log SaaS that performs not only better than self-managed Elastic but actually to perform good enough to be used.

New Relic could not handle the query aspect of having at around 5TB+ terabytes of logs (I know, a ludicrous amount of logs, but that's what it is) per day. Their architecture does not really allow for that. For small volumes I guess it would be enough. Not to ingest, that it did fine, but to query them under a reasonable time frame without a timeout, that's where it couldn't handle the high volume.

Also, their support service, while trying to win us over, that is, in their best moment, was nothing really stellar. Favouring sending us sales/presales people to solve technical problems.

It didn't leave us a good aftertaste.

Re: I don't think Elasticsearch is a good logging system

#23

Post by devin »

vimda wrote: post #9
piaste wrote: post #8

This is spot on based on our experience. I would add that the default ELK settings aren't terribly log-friendly, and having to janitor index policies, sharding, lifecycle policies, VM resources, etc. etc., _which you have to do even with the managed Elastic Cloud offer_, is way too much effort just to find and aggregate your TimeoutExceptions. We moved to NewRelic and while its dashboards are not _quite_ as fancy or…

At an old job we used NewRelic right up until they tried to 3x our annual bill :/

You are not alone.

Re: I don't think Elasticsearch is a good logging system

#25

Post by thinkharderdev »

jillesvangurp wrote: post #14

I can't say I agree with this article. Elasticsearch is a bit of a Swiss army knife and definitely needs some setup and configuring. But if you set it up properly, it does logs extremely well. And it does actually come with useful defaults and well integrated tools. For example, if you use the various Beats agents to push data to Elasticsearch, they come with a default schema and out of the box dashboards that they c…

My experience with ElasticSearch is that it is amazing for log aggregation until you reach a certain volume and then the operational cost (both $ and time/complexity) of running a large ES cluster starts to really bite hard.

To take one recent(ish) example from my last job. We built a bunch of streaming pipelines to enrich log data during ingest so we could use the ES ML jobs to do some proactive alerting on our application logs. It was all really nice but then we had to set up a regular patching cycle for our ES cluster. Generally everything is ES is stored in an index so doing a blue/green deployment works pretty well. Except that the ML jobs have some sort of in-memory state that can't be migrated, so the ML jobs would get migrated but in a weird, half-working state. So we ended up having to manually delete and recreate the "half-working" ML jobs whenever we rolled out the new patched cluster.

Having an "all in one" solution like ES is great until you hit a certain scale and the inherent complexity of such a system starts to really make itself apparent.

Re: I don't think Elasticsearch is a good logging system

#26

Post by sofixa »

reacharavindh wrote: post #19
vimda wrote: post #7

Earlier quoted context omitted.

What we do internally is use syslog-ng ( https://github.com/syslog-ng/syslog-ng ) to read the journald socket and push to a remote and into Kafka. I think journald works well as a structured logging tool, but it's certainly deficient in other ways

I have looked into ElasticSearch + Kibana as a solution to aggregate logs. There may be plenty of choice to replace ElasticSearch(ClickHouse, even Postgres, heck even journald), but a nice UI where you can simply search for that random piece of text you need to sift through the logs is the red herring. Until now, I have not seen a web interface to log as powerful as Kibana that can work with anything other than Elast…

Grafana?

Re: I don't think Elasticsearch is a good logging system

#28

Post by pas »

reacharavindh wrote: post #19
vimda wrote: post #7

Earlier quoted context omitted.

What we do internally is use syslog-ng ( https://github.com/syslog-ng/syslog-ng ) to read the journald socket and push to a remote and into Kafka. I think journald works well as a structured logging tool, but it's certainly deficient in other ways

I have looked into ElasticSearch + Kibana as a solution to aggregate logs. There may be plenty of choice to replace ElasticSearch(ClickHouse, even Postgres, heck even journald), but a nice UI where you can simply search for that random piece of text you need to sift through the logs is the red herring. Until now, I have not seen a web interface to log as powerful as Kibana that can work with anything other than Elast…

journald is not good at handling a lot of data, nor is it good at managing imported data. (It could be improved, probably "easily", but it's main feature is that it's an "always on" not terribly dumb log target, it's not a long term log management system.)

Re: I don't think Elasticsearch is a good logging system

#29

Post by OldOneEye »

thinkharderdev wrote: post #25
jillesvangurp wrote: post #14

I can't say I agree with this article. Elasticsearch is a bit of a Swiss army knife and definitely needs some setup and configuring. But if you set it up properly, it does logs extremely well. And it does actually come with useful defaults and well integrated tools. For example, if you use the various Beats agents to push data to Elasticsearch, they come with a default schema and out of the box dashboards that they c…

My experience with ElasticSearch is that it is amazing for log aggregation until you reach a certain volume and then the operational cost (both $ and time/complexity) of running a large ES cluster starts to really bite hard. To take one recent(ish) example from my last job. We built a bunch of streaming pipelines to enrich log data during ingest so we could use the ES ML jobs to do some proactive alerting on our appl…

Thanks for sharing your experience with ML.

We're experienced with managing a big scale elastic cluster, but never managed the ML jobs, so knowing about this limitation is definitely useful!

Re: I don't think Elasticsearch is a good logging system

#30

Post by gertrunde »

rio517 wrote: post #5

Oh man, https://github.com/elastic/elasticsearch-py/issues/1734 is a disappointing read. I know ES wants to save their business, but alienating users isn't exactly the path to success.

I'm not sure I regard that particular example as actually bad.

The response of "use the client version that matches the server version" seems perfectly fair.

Post reply on HN