We run on GCP and I have to say that Stackdriver Logging with the google fluentd agent is actually pretty good and relatively cheap. I don't like Stackdriver's metrics at all, but Logging feels more like part of GCP. Fluentd has given me far fewer problems than logstash or filebeat did when we were running an ELK setup. The search UI is obviously nowhere near as nice as something like Kibana, but it gets the job done. If you aren't on GCP, it's not worth it, but if you are, the whole setup is "good enough" that you might not need to set something more sophisticated up (I'm still looking at Loki though because I can't help myself).
Ask HN: How do you manage logs for your backend services?
21–30 of 142 posts
Re: Ask HN: How do you manage logs for your backend services?
#22I have used sumologic, your servers send the logs to sumo and you can setup your alerts/monitoring/dashboards etc on their side.
Re: Ask HN: How do you manage logs for your backend services?
#23Presumably you're running your service as a container, and presumably you only run one master process per container. Print your logs to stdout/stderr, and then you can use any log-capturing mechanism (fluent bit?) to stream them to any log-indexing system.
Tweak your system so you can selectively turn on log collection, and collect useful metrics 24/7. Logs almost never matter until you're diagnosing an active problem, anyway, and then only a few logs will be necessary. Differentiating the type and quality of particular log messages is also very useful.
Re: Ask HN: How do you manage logs for your backend services?
#24I don't have a lot of experience with it yet, but Loki looks promising for small projects. You'd still use it as a centralized logging server, but it's not as resource-expensive as something like self-hosting ELK. I've only been using it for my homelab, and haven't even moved everything to it yet - but I like it so far. I already use Grafana+influxdb for metrics so having logs in the same interface is nice. https://g…
I've been benching it in production. On a 1core 2gb vps it can do around 1200 logs/sec. Compared to ELK, we're saving several hundred $ per month.
If you can't do 1200 logs/sec with that hardware + elasticsearch, you've done something horrendously wrong.
You can do 1200 logs/sec with a container limited to 1 core and 256mb of memory running elasticsearch.
Re: Ask HN: How do you manage logs for your backend services?
#25For POSIX systems, syslog is the standard way. For systemd-based ones, journald may be preferable because of its additional features; both support sending of logs to a remote server. I'd suggest to avoid custom logging facilities (including just writing into a file) whenever possible, since maintaining many services with each using custom logging becomes quite painful.
Re: Ask HN: How do you manage logs for your backend services?
#26Also:
- Exist a tool that allow to navigate structured logs easily, without bring a heavy machinery like ELF stack? and work in terminal?
- That also filter only ERRORS and few lines above/below?
Re: Ask HN: How do you manage logs for your backend services?
#27People jumped to recommending things before asking:
What's the volume of logs?
What's the expected usage? (debugging? audit?)
Are you using text, or structured logs? (or can you switch to structured easily?)
How far back do you want to keep the logs? How far back do you want to query them easily?
Are you in an environment with an existing simple solution? (GCP, AWS)
Re: Ask HN: How do you manage logs for your backend services?
#28Nobody mentioning Splunk? Too obvious or am I not understanding the ask?
Re: Ask HN: How do you manage logs for your backend services?
#29for the endpoint itself, consider switching to syslog so you get a bunch of stuff for free (auto-rotations, docker logging, ...) and more easy to change decisions later (pipe to splunk/elk/...). it's thought out and pretty easy!
Expanding a bit here: Splunk is free for We work with a ton of log tools as company bringing graph analytics+vis to investogations largely log & event data, and I've increasingly shifted to that when I need easy basics.
Re: Ask HN: How do you manage logs for your backend services?
#30I don't have a lot of experience with it yet, but Loki looks promising for small projects. You'd still use it as a centralized logging server, but it's not as resource-expensive as something like self-hosting ELK. I've only been using it for my homelab, and haven't even moved everything to it yet - but I like it so far. I already use Grafana+influxdb for metrics so having logs in the same interface is nice. https://g…
We use Loki in production. It's pretty good! There are still some issues with memory usage (particularly the log-cli tool) and query performance, but it's a great start.