ClickHouse is written in C++ and is open source: https://github.com/ClickHouse/ClickHouse
Where can I find resources on tools that integrate with Clickhouse, ex. are there any tools for gathering server metrics and sending them to Clickhouse?
I don't think Elasticsearch is a good logging system
111–117 of 117 posts
Re: I don't think Elasticsearch is a good logging system
#112Earlier quoted context omitted.
If it's a live outage, we can still SSH into a machine and grep the console output / local rotating log files. (For that matter, I still prefer to do that when I'm just testing new stuff in dev/staging environments) NewRelic "only" stores the logs for 30 days and displays them in a nice web UI with searching, alerting, sharing, and a bunch of other stuff. It's not like they cease to exist without it.
SSH to machine and grepping through is much harder when the number of machine is > ~5 (say). SSH to host doesn't scale as you keep adding the hosts.
We use Docker Swarm so we can SSH into a manager and run `docker service logs` regardless of which hosts are actually running the services. I assume other orchestration systems have equivalent features.
Re: I don't think Elasticsearch is a good logging system
#113Earlier quoted context omitted.
Not advocating for this decision, but did you investigate Splunk? In my experience, that’s the paid logging service that competes with ELK. It will be expensive, so you have to consider the total cost of ownership (e.g., ELK requires some experienced people to run it at your volumes) but it works AFAIK.
there's expensive and then there's splunk. but you get what you pay for. splunk will handle your load unless you're google.
> there's expensive and then there's splunk.
This got me curious, so OK, Splunk's pricing pages are very obtuse and they are really pushy about getting you to contact sales directly to get bleeded, but I managed to get to this "actually has a number in it" page for their Log Observer services[0], and... it looks cheaper than NewRelic, especially at scale?
NR charges $0.25 per ingested GB after the first 100 free GB; Splunk apparently only charges a flat $0.10, if you choose ingest pricing.
I guess that NR includes (a free tier of) a bunch of alerts, monitoring etc. features in their package, while they're separate packages for Splunk. Still, that doesn't seem wildly expensive at a glance. Where's the catch?
[0] https://www.splunk.com/en_us/software/pricing/faqs/devops.ht...
Re: I don't think Elasticsearch is a good logging system
#114Earlier quoted context omitted.
journald is not good at handling a lot of data, nor is it good at managing imported data. (It could be improved, probably "easily", but it's main feature is that it's an "always on" not terribly dumb log target, it's not a long term log management system.)
Hmm. Never tried to use journald at any reasonable scale beyond tens of servers. Good to know its characteristics. To be honest I wasn’t looking for a long term log management system and that is why Journald even came up in mind. If it could aggregate logs from several servers and retain them for a week while expiring older logs to an archive source, it’s sufficient for my needs.
Exactly why I wrote my comment. :) Because it seems it's able to do that, but not really. And it seems easy to fix, but of course patches are welcome. (Hopefully.)
https://github.com/systemd/systemd/issues/5242
Sure, it's not terribly hard to work around it with a cron (or systemd-timer) script, but why go uphill, when there are better tools.
Re: I don't think Elasticsearch is a good logging system
#115This is spot on based on our experience. I would add that the default ELK settings aren't terribly log-friendly, and having to janitor index policies, sharding, lifecycle policies, VM resources, etc. etc., _which you have to do even with the managed Elastic Cloud offer_, is way too much effort just to find and aggregate your TimeoutExceptions. We moved to NewRelic and while its dashboards are not _quite_ as fancy or…
At high volumes, at my job, we have yet to find a good third-party log SaaS that performs not only better than self-managed Elastic but actually to perform good enough to be used. New Relic could not handle the query aspect of having at around 5TB+ terabytes of logs (I know, a ludicrous amount of logs, but that's what it is) per day. Their architecture does not really allow for that. For small volumes I guess it woul…
>At high volumes, at my job, we have yet to find a good third-party log SaaS that performs
Not even just performs but the costs are always astronomically higher.
Re: I don't think Elasticsearch is a good logging system
#116Earlier quoted context omitted.
there's expensive and then there's splunk. but you get what you pay for. splunk will handle your load unless you're google.
I love splunk. Our clusters process 10s of billions of structured log events daily. We have search, reports, PagerDuty integration, dashboards, etc. It is crazy expensive but is the best system I've used in this space. We are having to save costs with so much data, so we are lowering retention time and moving the data to snowflake for data older than a week. More and more, we are leveraging Looker for reporting out o…
> Our clusters process 10s of billions of structured log events daily.
Whats that run you?
Re: I don't think Elasticsearch is a good logging system
#117Earlier quoted context omitted.
If it's a live outage, we can still SSH into a machine and grep the console output / local rotating log files. (For that matter, I still prefer to do that when I'm just testing new stuff in dev/staging environments) NewRelic "only" stores the logs for 30 days and displays them in a nice web UI with searching, alerting, sharing, and a bunch of other stuff. It's not like they cease to exist without it.
SSH to machine and grepping through is much harder when the number of machine is > ~5 (say). SSH to host doesn't scale as you keep adding the hosts.
When someone says "we ssh and grep" you know pretty much how small their operation is.