91–100 of 117 posts
Other solutions like Loki, which only index non-full-text fields, are suggested.
Why not RDBMS? I have over 10 years come full circle from SQL/RDBMS -> NoSQL of various types -> SQL/RDBMS.
edit: Is it really that other databases types are better suited to high ingest rate / write-heavy light-read workloads?
Earlier quoted context omitted.
My point was that by the time you filter on keyword fields (and other exact matching fields), the number of logs is small enough that an efficient full text search isn't necessary. That doesn't mean that full text search itself isn't useful, just that maintaining an inverted index is overkill in the logging case
This has been my experience. Obviously different people use logs for different things, but in my case I'm usually looking for information about something bad that already happened, within a very specific window of time, and within a specific section of the application. 99% of the time, that means I am filtering until there are only a handful of entries that match, at which point I don't need full text search at all.
A few days ago, a colleague asked me why a certain Google cloud instance does not exist. I did not know either, so I searched for this name in the Google audit log, and found when and by whom it was decommissioned.
But it was a full-text search, given the instance name. I probably could do it (in theory) as a field match, if I knew which field it was, and which format it was in (I am talking about project/abc/location/xyz type of junk that precedes the actual instance name).
And yes it was slow (this instance was deleted months ago, and Google tries to search the most recent logs first).
What's worse (and I've seen this trend a lot) is to use Elasticsearch for metrics. My god.
Elasticsearch is good because it just ingests whatever you sent to it, which allows you to deliver solutions rather quickly. Having said this, I agree there are better solutions. (Also, Elasticsearch shines because of its full text search capability, which is not often exploited in case of logs.) Loki is fine (or better said, it will be fine once they finally release a version without write-out-of-order constrain) bu…
High availability in Loki is supported in distributed mode. Helm chart here: https://github.com/grafana/helm-charts/tree/main/charts/loki...
I believe the complaints here are a case of 'not using it correctly'. The 'Reverse index' (Lucene's inverted index) is a fundamental data structure used to enable very fast search. Other data structures, like KD trees, are used for non-text data types. If you're not doing full text search, don't use `text` fields. If you're not querying the data, why store it in the first place? Full text search for logs is incredibl…
Former co-founder and CTO of InfluxData here, currently building a new company in this space. My strongly-opinionated view on this is that Elasticsearch is not a time-series database and asking it to handle large volumes of logs (fundamentally a time-series use case) is always going to be painful and expensive.
We've built a product called EraSearch that mimics the Elasticsearch APIs for ease of integration but is built with a significantly more efficient (read time-series) architecture. We can handle ingest volumes with about 1/10th of the hardware required for Elasticsearch while still offering comparable (or faster) query performance. If you are generating large amounts of logs (~1TB per day or more), my guess is that this will resonate with you.
If any of this sounds interesting, drop me a note at todd@era.co - I'd love to hear more about your use case. Or even if you just want to talk about time-series data, I'm game. ;)
What's worse (and I've seen this trend a lot) is to use Elasticsearch for metrics. My god.
Elasticsearch is good because it just ingests whatever you sent to it, which allows you to deliver solutions rather quickly. Having said this, I agree there are better solutions. (Also, Elasticsearch shines because of its full text search capability, which is not often exploited in case of logs.) Loki is fine (or better said, it will be fine once they finally release a version without write-out-of-order constrain) bu…
Out of Order support is available in Loki's main branch and included in the next release. It's already live in production on Grafana Cloud. https://grafana.com/blog/2021/09/16/avoid-dropped-logs-due-t... High availability in Loki is supported in distributed mode. Helm chart here: https://github.com/grafana/helm-charts/tree/main/charts/loki...
Regarding HA, I meant something beyond a k8s deployment.
This is spot on based on our experience. I would add that the default ELK settings aren't terribly log-friendly, and having to janitor index policies, sharding, lifecycle policies, VM resources, etc. etc., _which you have to do even with the managed Elastic Cloud offer_, is way too much effort just to find and aggregate your TimeoutExceptions. We moved to NewRelic and while its dashboards are not _quite_ as fancy or…
At high volumes, at my job, we have yet to find a good third-party log SaaS that performs not only better than self-managed Elastic but actually to perform good enough to be used. New Relic could not handle the query aspect of having at around 5TB+ terabytes of logs (I know, a ludicrous amount of logs, but that's what it is) per day. Their architecture does not really allow for that. For small volumes I guess it woul…
Earlier quoted context omitted.
How are diagnostic logs not business critical? If you have an outage while your non-mission critical logs are offline what are you going to do?
If it's a live outage, we can still SSH into a machine and grep the console output / local rotating log files. (For that matter, I still prefer to do that when I'm just testing new stuff in dev/staging environments) NewRelic "only" stores the logs for 30 days and displays them in a nice web UI with searching, alerting, sharing, and a bunch of other stuff. It's not like they cease to exist without it.