Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I don't think Elasticsearch is a good logging system

blog.sinkingpoint.com

Post reply on HN
Print view View original

51–60 of 117 posts

Re: I don't think Elasticsearch is a good logging system

#51

Post by cmarschner »

gpderetta wrote: post #17
beardface wrote: post #11

I believe the complaints here are a case of 'not using it correctly'. The 'Reverse index' (Lucene's inverted index) is a fundamental data structure used to enable very fast search. Other data structures, like KD trees, are used for non-text data types. If you're not doing full text search, don't use `text` fields. If you're not querying the data, why store it in the first place? Full text search for logs is incredibl…

I have little knowledge of the log aggregation domain, but generally indices are great for read mostly loads. It seems to me that for log aggregation writes are more frequent than searches; cheap writes and the occasional brute force search. For alerting you might better off running each new line against a set of filters/watchers. It seems wasteful to run it after indexing. Again, no experience or knowledge on the do…

At the core of Lucene, as you index a document, it creates first an index containing a single document, and everything else is merge operations (operating in log N - merging larger and larger chunks). So the nice thing is that you can use the same query language, in fact the exact same implementation, to run a search query in alerting mode. You would create this single document index (which you'd do anyway to make it searchable) and run the query against it before adding it to the other documents.

Re: I don't think Elasticsearch is a good logging system

#52

Post by alkz »

moduspol wrote: post #46
detaro wrote: post #35

Earlier quoted context omitted.

> (and I've seen this trend a lot) any idea why? Seems like an odd approach to me.

I was interviewing earlier this year and saw at least three companies actively planning to migrate from their existing relational databases into Elasticsearch. It seems to commonly be perceived as a panacea of databases.

in most cases elastic is easier to scale than a relational or timeseries db

Re: I don't think Elasticsearch is a good logging system

#53

Post by sethammons »

baq wrote: post #36
pfarrell wrote: post #33

Earlier quoted context omitted.

Not advocating for this decision, but did you investigate Splunk? In my experience, that’s the paid logging service that competes with ELK. It will be expensive, so you have to consider the total cost of ownership (e.g., ELK requires some experienced people to run it at your volumes) but it works AFAIK.

there's expensive and then there's splunk. but you get what you pay for. splunk will handle your load unless you're google.

I love splunk. Our clusters process 10s of billions of structured log events daily. We have search, reports, PagerDuty integration, dashboards, etc. It is crazy expensive but is the best system I've used in this space. We are having to save costs with so much data, so we are lowering retention time and moving the data to snowflake for data older than a week. More and more, we are leveraging Looker for reporting out of Snowflake and relying more on Prometheus monitoring for alerting. But Splunk would still be my ideal service if we had less total data.

Re: I don't think Elasticsearch is a good logging system

#54

Post by jarpineh »

buro9 wrote: post #6
airocker wrote: post #2

You cannot do much analytics on Loki, there is no index so each query is costly

You can do a surprisingly lot with Loki. To get the performance run more queriers (horizontally scaled read path that is in front of the object storage). We're improving performance constantly, and we already run Loki at a very large scale (multi-region, multi-tenant, etc) with aggressive internal SLOs. We see customers doing network analytics, log analytics, line-of-business data analytics all on Loki and it works r…

Hi.

I have a question. I have put some time trying to learn the Grafana stack. Heaviness of elastic both self hosted and cloud version has led me to seek for alternatives. Is there a way to get logs into Loki that can be run offline, when jobs have finished and hosts have been shutdown? It seems Loki recommends Promtail agent which looks a bit heavy-handed (https://grafana.com/docs/loki/latest/getting-started/get-log... )

I have relaxed requirements on gathering logs for analytics. I’d much rather just run a tool that goes through log files and ingests them into Loki. Results of several jobs would be compared as a whole.

From what I’ve seen Loki with Grafana can do a lot. For example this video: https://m.youtube.com/watch?v=7zmRhHd-ohk

Thank you!

Re: I don't think Elasticsearch is a good logging system

#55

Post by moduspol »

alkz wrote: post #52
moduspol wrote: post #46

Earlier quoted context omitted.

I was interviewing earlier this year and saw at least three companies actively planning to migrate from their existing relational databases into Elasticsearch. It seems to commonly be perceived as a panacea of databases.

in most cases elastic is easier to scale than a relational or timeseries db

I mean, kind of? It depends on what your data looks like and how you're querying it.

If you're not doing free-text search, and your data will fit in memory in a big relational database VM for the foreseeable future, why Elasticsearch?

Re: I don't think Elasticsearch is a good logging system

#56

Post by [deleted] »

alkz wrote: post #31
rio517 wrote: post #5

Oh man, https://github.com/elastic/elasticsearch-py/issues/1734 is a disappointing read. I know ES wants to save their business, but alienating users isn't exactly the path to success.

elastic 5.x is ANCIENT and no one in their right mind should still use that for any production workload

[deleted]

Re: I don't think Elasticsearch is a good logging system

#57

Post by dvdkon »

jillesvangurp wrote: post #14

I can't say I agree with this article. Elasticsearch is a bit of a Swiss army knife and definitely needs some setup and configuring. But if you set it up properly, it does logs extremely well. And it does actually come with useful defaults and well integrated tools. For example, if you use the various Beats agents to push data to Elasticsearch, they come with a default schema and out of the box dashboards that they c…

AGPL is just GPL where usage over a network requires giving users the source code of the AGPL codebase. I'll take it over a custom licence where I have to worry if I need to release the source code of everything around it (SSPL) or if I'm providing it as a managed service (EL).

Re: I don't think Elasticsearch is a good logging system

#58

Post by dvdkon »

dd82 wrote: post #49
willvarfar wrote: post #44

Earlier quoted context omitted.

Completely agree. My gripe with ES is that it won't let you do post-pass filtering at all. If you create an index with a few keyword fields indexed and then some unindexed fields, you can't query the unindexed fields. Grafana's Loki seems to be exactly what we are looking for, although I haven't played with it.

You can update the index with the new field specification and reindex your content. Your complaint really doesn't make sense, how would you query an unindexed field? Elasticsearch is a _search_ engine, which means it needs to index content that is to be discoverable. What you're saying with unindexed fields is you're completely fine with those not being included in any search or filtering.

Exactly, it's a search engine. It probably doesn't make sense to use it as a storage engine for logs unless you need to search all of them efficiently.

Re: I don't think Elasticsearch is a good logging system

#59

Post by barrkel »

dd82 wrote: post #49
willvarfar wrote: post #44

Earlier quoted context omitted.

Completely agree. My gripe with ES is that it won't let you do post-pass filtering at all. If you create an index with a few keyword fields indexed and then some unindexed fields, you can't query the unindexed fields. Grafana's Loki seems to be exactly what we are looking for, although I haven't played with it.

You can update the index with the new field specification and reindex your content. Your complaint really doesn't make sense, how would you query an unindexed field? Elasticsearch is a _search_ engine, which means it needs to index content that is to be discoverable. What you're saying with unindexed fields is you're completely fine with those not being included in any search or filtering.

You use the index to identify a subset of records and scan those for unindexed criteria.

It's ok to fail if the indexed criteria are not selective enough. In fact it's usually preferable to a long timeout.

Post reply on HN