Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I don't think Elasticsearch is a good logging system

blog.sinkingpoint.com

Post reply on HN
Print view View original

101–110 of 117 posts

Re: I don't think Elasticsearch is a good logging system

#101

Post by nerdponx »

patrakov wrote: post #92
nerdponx wrote: post #43

Earlier quoted context omitted.

This has been my experience. Obviously different people use logs for different things, but in my case I'm usually looking for information about something bad that already happened, within a very specific window of time, and within a specific section of the application. 99% of the time, that means I am filtering until there are only a handful of entries that match, at which point I don't need full text search at all.

I am not really sure about this. A few days ago, a colleague asked me why a certain Google cloud instance does not exist. I did not know either, so I searched for this name in the Google audit log, and found when and by whom it was decommissioned. But it was a full-text search, given the instance name. I probably could do it (in theory) as a field match, if I knew which field it was, and which format it was in (I am…

This sounds like the 1% of my experiences not served by filtering.

Naturally your experiences will be different from mine!

Re: I don't think Elasticsearch is a good logging system

#102

Post by larkost »

lathiat wrote: post #50
willvarfar wrote: post #44

Earlier quoted context omitted.

Completely agree. My gripe with ES is that it won't let you do post-pass filtering at all. If you create an index with a few keyword fields indexed and then some unindexed fields, you can't query the unindexed fields. Grafana's Loki seems to be exactly what we are looking for, although I haven't played with it.

I guess what they want is to use the elasticsearch query language but let it optionally do “expensive” non indexed filtering like a SQL database would let you do. Without knowing for sure I imagine they originally expected the application side to handle this but many of the current solutions don’t do that. And they expose and overload the elastic search query language as the primary search interface with no additiona…

The problem with this thinking is that in most cases having the server send all of the data back to the client to do their own search is going to be far more expensive than running the search (even of unindexed data) on the server.

And I am only talking about server-side costs here as moving data between server and client has costs both in serialization and transmission. Yes, I can make up regexes that wind up throwing this cost comparison out the window (e.g.: lookbacks), but the fast majority of cases this is true.

I think the main reason that ElasticSeach does not do this is that they would either have to provide grep-like or regex support, and those two would provided different answers than the lexical search system they provide otherwise. That would be a nightmare to try and explain the differences to clients.

Note: in most places I wind up using ElasticSearch I absolutely hate that it is lexical search rather than grep or regex... especially when I am looking for exact text. This is particularly a problem in Jira where I have to be very careful about word boundaries.

Re: I don't think Elasticsearch is a good logging system

#103

Post by t-writescode »

About 5 or 6 years ago, my previous job introduced ElasticSearch to our infrastructure.

Our existing codebase had a pattern where all logs for a transaction were stored in a single, big log and then that log was uploaded to a server to be stored.

We moved this large log to ElasticSearch, formatted it differently, labeled a bunch of columns, used Logstash to standardize variable names, etc.

We did this for our main services and kept those services separated as different indexes.

Each log had what user did the web request, response codes, we had detailed logging and general logging stored in 2 separate places and threw out the detailed logs after a very short while. We had fields for all common detail work.

It. Was. Perfect. Everyone could use it. Our in-house customer support team used those logs to help diagnose customer issues, our tech team used it to track issues. Our NOC used it to investigate issues.

We had tens of dashboards that were on a rotating view. We had at-a-glance server health tracking through it, all from active traffic that was being formatted and used.

-

And then the next company I worked with that used ElasticSearch tried to use it in a world where each logger.info() was its own row in ElasticSearch. That just seemed like a horrible, horrible idea.

It really does seem to come down to how you use it.

Re: I don't think Elasticsearch is a good logging system

#104

Post by rad_gruchalski »

toddpersen wrote: post #96

Former co-founder and CTO of InfluxData here, currently building a new company in this space. My strongly-opinionated view on this is that Elasticsearch is not a time-series database and asking it to handle large volumes of logs (fundamentally a time-series use case) is always going to be painful and expensive. We've built a product called EraSearch that mimics the Elasticsearch APIs for ease of integration but is bu…

I’d love to try your product out in combination with Jaeger. Is it possible?

Re: I don't think Elasticsearch is a good logging system

#105

Post by toddpersen »

rad_gruchalski wrote: post #104
toddpersen wrote: post #96

Former co-founder and CTO of InfluxData here, currently building a new company in this space. My strongly-opinionated view on this is that Elasticsearch is not a time-series database and asking it to handle large volumes of logs (fundamentally a time-series use case) is always going to be painful and expensive. We've built a product called EraSearch that mimics the Elasticsearch APIs for ease of integration but is bu…

I’d love to try your product out in combination with Jaeger. Is it possible?

We've actually been doing some internal work with OpenTelemetry and Jaeger - we would love some feedback on it. Drop me a note at todd@era.co and we can get you set up with a demo instance.

Re: I don't think Elasticsearch is a good logging system

#106

Post by therealdrag0 »

pfarrell wrote: post #33
OldOneEye wrote: post #22

Earlier quoted context omitted.

At high volumes, at my job, we have yet to find a good third-party log SaaS that performs not only better than self-managed Elastic but actually to perform good enough to be used. New Relic could not handle the query aspect of having at around 5TB+ terabytes of logs (I know, a ludicrous amount of logs, but that's what it is) per day. Their architecture does not really allow for that. For small volumes I guess it woul…

Not advocating for this decision, but did you investigate Splunk? In my experience, that’s the paid logging service that competes with ELK. It will be expensive, so you have to consider the total cost of ownership (e.g., ELK requires some experienced people to run it at your volumes) but it works AFAIK.

Yep. We’re using a Splunk with TBs of logs a day and it’s been great.

Re: I don't think Elasticsearch is a good logging system

#107

Post by lazyant »

ES and the ELK stack is the worst pile of shit I've had to deal with. Everybody I know who has worked with it has had problems. Obscure messages that nobody can figure out what they mean. Dropping logs silently, sometimes because it can't reconcile two index mappings. Constant baby-sitting and needing to add more hardware even if there are no indications that it needs it, super hard to tune or understand. A proper db doesn't need this much work and shipping logs (syslog) has been solved since the 80s. Move to Loki.

Re: I don't think Elasticsearch is a good logging system

#108

Post by waste_monk »

vosper wrote: post #12
rio517 wrote: post #5

Oh man, https://github.com/elastic/elasticsearch-py/issues/1734 is a disappointing read. I know ES wants to save their business, but alienating users isn't exactly the path to success.

FWIW, the ES team implemented a change that I requested via GitHub, and I’m just some guy who’s not even a paying customer. The change involved the behavior of force merge, and probably didn’t affect too many people. It’ll allow my employer (and hopefully some other companies) to make more efficient use of our disk space. I’m really appreciative of the way they considered and then implemented my request.

I know someone who works at Elastic and it sounds like a really nice workplace (in terms of culture, average competency level across all employees, etc.), and that seems to be reflected in their products.

Re: I don't think Elasticsearch is a good logging system

#109

Post by airocker »

buro9 wrote: post #6
airocker wrote: post #2

You cannot do much analytics on Loki, there is no index so each query is costly

You can do a surprisingly lot with Loki. To get the performance run more queriers (horizontally scaled read path that is in front of the object storage). We're improving performance constantly, and we already run Loki at a very large scale (multi-region, multi-tenant, etc) with aggressive internal SLOs. We see customers doing network analytics, log analytics, line-of-business data analytics all on Loki and it works r…

I wonder the Loki designers thought about the features that ES provides before rewriting the whole thing in a new way? Search and Analyze is pretty cool. Almost all of Loki can be an analzer plugin. Automatic schema detection is cool. ITs not that crazy to administer.

Re: I don't think Elasticsearch is a good logging system

#110

Post by jakozaur »

(sameless plug of Sumo Logic employee)

The SaaS vendors are much better solution for logs, such as Sumo Logic. They have multiple tier of data (continuous, frequent, infrequent) which different performance and cost trade-offs. Most companies would be better, dividing logs into different category than running and tuning Elastic Search.

Post reply on HN