31–40 of 117 posts
Oh man, https://github.com/elastic/elasticsearch-py/issues/1734 is a disappointing read. I know ES wants to save their business, but alienating users isn't exactly the path to success.
This is spot on based on our experience. I would add that the default ELK settings aren't terribly log-friendly, and having to janitor index policies, sharding, lifecycle policies, VM resources, etc. etc., _which you have to do even with the managed Elastic Cloud offer_, is way too much effort just to find and aggregate your TimeoutExceptions. We moved to NewRelic and while its dashboards are not _quite_ as fancy or…
At high volumes, at my job, we have yet to find a good third-party log SaaS that performs not only better than self-managed Elastic but actually to perform good enough to be used. New Relic could not handle the query aspect of having at around 5TB+ terabytes of logs (I know, a ludicrous amount of logs, but that's what it is) per day. Their architecture does not really allow for that. For small volumes I guess it woul…
What's worse (and I've seen this trend a lot) is to use Elasticsearch for metrics. My god.
any idea why? Seems like an odd approach to me.
Earlier quoted context omitted.
At high volumes, at my job, we have yet to find a good third-party log SaaS that performs not only better than self-managed Elastic but actually to perform good enough to be used. New Relic could not handle the query aspect of having at around 5TB+ terabytes of logs (I know, a ludicrous amount of logs, but that's what it is) per day. Their architecture does not really allow for that. For small volumes I guess it woul…
Not advocating for this decision, but did you investigate Splunk? In my experience, that’s the paid logging service that competes with ELK. It will be expensive, so you have to consider the total cost of ownership (e.g., ELK requires some experienced people to run it at your volumes) but it works AFAIK.
but you get what you pay for. splunk will handle your load unless you're google.
I believe the complaints here are a case of 'not using it correctly'. The 'Reverse index' (Lucene's inverted index) is a fundamental data structure used to enable very fast search. Other data structures, like KD trees, are used for non-text data types. If you're not doing full text search, don't use `text` fields. If you're not querying the data, why store it in the first place? Full text search for logs is incredibl…
I have little knowledge of the log aggregation domain, but generally indices are great for read mostly loads. It seems to me that for log aggregation writes are more frequent than searches; cheap writes and the occasional brute force search. For alerting you might better off running each new line against a set of filters/watchers. It seems wasteful to run it after indexing. Again, no experience or knowledge on the do…
Generally, you write and read to/from the same index in Elasticsearch. Where this falls apart is that you'll often want to change the configuration for an index based on whether it's write or read heavy. The main thing that changes in this scenario is the number of primary and replica shards (Lucene indices) for the Elasticsearch index.
Indices with a high write, low search workload will generally require more primary shards and less replicas. Low write, high search workloads require the opposite; lower primaries and more replicas.
The problem comes when you need high write and high search rates. Using a single cluster with lots of primaries and lots of replicas will overwhelm the hosts and you end up with terrible performance. The general pattern with Elasticsearch is to run two clusters. Index into one cluster, then use cross-cluster-replication (CCR) into a different cluster you run queries against.
There's an incredible amount of nuance to all of this. I've worked with many clusters and they all have different usage and configuration requirements. There's no magic formula for calculating configuration values; it all comes down to experience, monitoring, and experimentation.
What's worse (and I've seen this trend a lot) is to use Elasticsearch for metrics. My god.
> (and I've seen this trend a lot) any idea why? Seems like an odd approach to me.
What's worse (and I've seen this trend a lot) is to use Elasticsearch for metrics. My god.
Earlier quoted context omitted.
I have looked into ElasticSearch + Kibana as a solution to aggregate logs. There may be plenty of choice to replace ElasticSearch(ClickHouse, even Postgres, heck even journald), but a nice UI where you can simply search for that random piece of text you need to sift through the logs is the red herring. Until now, I have not seen a web interface to log as powerful as Kibana that can work with anything other than Elast…
journald is not good at handling a lot of data, nor is it good at managing imported data. (It could be improved, probably "easily", but it's main feature is that it's an "always on" not terribly dumb log target, it's not a long term log management system.)
To be honest I wasn’t looking for a long term log management system and that is why Journald even came up in mind. If it could aggregate logs from several servers and retain them for a week while expiring older logs to an archive source, it’s sufficient for my needs.