Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I don't think Elasticsearch is a good logging system

blog.sinkingpoint.com

Post reply on HN
Print view View original

61–70 of 117 posts

Re: I don't think Elasticsearch is a good logging system

#61

Post by fn1 »

sethammons wrote: post #53
baq wrote: post #36

Earlier quoted context omitted.

there's expensive and then there's splunk. but you get what you pay for. splunk will handle your load unless you're google.

I love splunk. Our clusters process 10s of billions of structured log events daily. We have search, reports, PagerDuty integration, dashboards, etc. It is crazy expensive but is the best system I've used in this space. We are having to save costs with so much data, so we are lowering retention time and moving the data to snowflake for data older than a week. More and more, we are leveraging Looker for reporting out o…

I second that. I love splunk as well.

Costs can also be reduced by spending some development-effort into abbreviating logs and being smart about deciding what to log and where.

Re: I don't think Elasticsearch is a good logging system

#62

Post by closeparen »

detaro wrote: post #35
denysvitali wrote: post #34

What's worse (and I've seen this trend a lot) is to use Elasticsearch for metrics. My god.

> (and I've seen this trend a lot) any idea why? Seems like an odd approach to me.

When I see a nonzero value in a metric about “things that shouldn’t happen,” the very next thing I’m going to ask for is examples.

It’s generally considered too expensive to derive metrics from logs in this way, but it’d be a killer debugging experience.

Re: I don't think Elasticsearch is a good logging system

#63

Post by mrweasel »

baq wrote: post #36
pfarrell wrote: post #33

Earlier quoted context omitted.

Not advocating for this decision, but did you investigate Splunk? In my experience, that’s the paid logging service that competes with ELK. It will be expensive, so you have to consider the total cost of ownership (e.g., ELK requires some experienced people to run it at your volumes) but it works AFAIK.

there's expensive and then there's splunk. but you get what you pay for. splunk will handle your load unless you're google.

Primary issue with using Splunk is that pretty much all other solution will seem inferiour. Great product, terrible business partner.

Re: I don't think Elasticsearch is a good logging system

#64

Post by buro9 »

jarpineh wrote: post #54
buro9 wrote: post #6

Earlier quoted context omitted.

You can do a surprisingly lot with Loki. To get the performance run more queriers (horizontally scaled read path that is in front of the object storage). We're improving performance constantly, and we already run Loki at a very large scale (multi-region, multi-tenant, etc) with aggressive internal SLOs. We see customers doing network analytics, log analytics, line-of-business data analytics all on Loki and it works r…

Hi. I have a question. I have put some time trying to learn the Grafana stack. Heaviness of elastic both self hosted and cloud version has led me to seek for alternatives. Is there a way to get logs into Loki that can be run offline, when jobs have finished and hosts have been shutdown? It seems Loki recommends Promtail agent which looks a bit heavy-handed ( https://grafana.com/docs/loki/latest/getting-started/get-lo…

One of the engineers has suggested trying:

```cat | promtail -stdin```

Though we like the idea as a feature request... "Have promtail support reading a file or directory and exiting once complete".

So we've put that on the plan.

Re: I don't think Elasticsearch is a good logging system

#65

Post by alkz »

moduspol wrote: post #55
alkz wrote: post #52

Earlier quoted context omitted.

in most cases elastic is easier to scale than a relational or timeseries db

I mean, kind of? It depends on what your data looks like and how you're querying it. If you're not doing free-text search, and your data will fit in memory in a big relational database VM for the foreseeable future, why Elasticsearch?

sure relational db is ok if fits in memory, but if we're talking about TBs of metrics i'd rather look for another way

Re: I don't think Elasticsearch is a good logging system

#66

Post by 101011 »

beardface wrote: post #39
denysvitali wrote: post #34

What's worse (and I've seen this trend a lot) is to use Elasticsearch for metrics. My god.

Why do you consider this a bad thing?

I'm not convinced it is a bad thing. Good scale, relatively low latency, keeps your infrastructure costs down, and many of the other alerting options (looking at you Datadog and NewRelic) are crazy expensive.

Open to hearing other opinions though.

Re: I don't think Elasticsearch is a good logging system

#68

Post by OrvalWintermute »

> I’ve really pushed ElasticSearch to its limits, with hundreds of terrabytes of data across dozens of machines and tens of thousands of shards and in all that time I’ve found that it really only works well for one of those situations.

This is a non sequitur

Re: I don't think Elasticsearch is a good logging system

#70

Post by pachico »

Elasticsearch is good because it just ingests whatever you sent to it, which allows you to deliver solutions rather quickly.

Having said this, I agree there are better solutions. (Also, Elasticsearch shines because of its full text search capability, which is not often exploited in case of logs.)

Loki is fine (or better said, it will be fine once they finally release a version without write-out-of-order constrain) but I find its lack of high-availability solution a bit frustrating.

ClickHouse, on the other side, is just magnificent. I use it in combination with Vector as message pipeline solution (it's an alternative to Fluentd, let's say).

So, yes, Elasticsearch is just not great and not only for logs, but for everything else that doesn't require full text search, in my opinion.

Post reply on HN