Ask HN: How do you manage logs for your backend services?

#1

Post by tusharsoni » Fri, Jan 03, 2020, 8:31 PM UTC

I have been working on a few side projects that usually have Go backend services. Right now, they log straight to a file and it works just fine since I only have a couple of instances running.

If I were to scale this to more instances/servers, I would need a centralized logging service. I've considered ELK but it's fairly expensive to both self-host or buy a managed subscription.

Any advice?

Re: Ask HN: How do you manage logs for your backend services?

#2

Post by [deleted] » Fri, Jan 03, 2020, 9:12 PM UTC

[deleted]

Re: Ask HN: How do you manage logs for your backend services?

#3

Post by deepersprout » Fri, Jan 03, 2020, 9:13 PM UTC

Keep logs local, and send only errors and noteworthy events to a central server like sentry.io, rollbar, or something else.

If you log everything to a single server, as you noticed yourself, it will become very expensive and difficult to filter out the stuff that you really need when looking for errors or when you try to hunt down a specific problem.

Re: Ask HN: How do you manage logs for your backend services?

#4

Post by bradstewart » Fri, Jan 03, 2020, 9:14 PM UTC

When running on AWS, I generally just use CloudWatch. For non-AWS hosts, and/or when I need something more feature rich: DataDog is a solid hosted service with reasonable pricing.

Re: Ask HN: How do you manage logs for your backend services?

#5

Post by tusharsoni » Fri, Jan 03, 2020, 9:17 PM UTC

deepersprout wrote: post #3

Keep logs local, and send only errors and noteworthy events to a central server like sentry.io, rollbar, or something else. If you log everything to a single server, as you noticed yourself, it will become very expensive and difficult to filter out the stuff that you really need when looking for errors or when you try to hunt down a specific problem.

I used sentry.io for a project and it worked well for what it is - error tracking. But when users reach out with issues, it is useful to look at logs and having them local means I would have to look at each log file per instance which could be on multiple servers.

Re: Ask HN: How do you manage logs for your backend services?

#6

Post by lmeyerov » Fri, Jan 03, 2020, 9:22 PM UTC

for the endpoint itself, consider switching to syslog so you get a bunch of stuff for free (auto-rotations, docker logging, ...) and more easy to change decisions later (pipe to splunk/elk/...). it's thought out and pretty easy!

Re: Ask HN: How do you manage logs for your backend services?

#7

Post by reacharavindh » Fri, Jan 03, 2020, 9:31 PM UTC

I have been having a TODO on my list to explore using Toshi + Tantivvy( Rust projects as replacement for ELastic Search) and using it to supplement a simpler (ripgrep + AGrind) file based search on logs centralized using rsyslog. Haven’t gotten around to play with them yet. Hopefully sometime this year.

I could not find an equivalent to Kibana though :-(

Re: Ask HN: How do you manage logs for your backend services?

#8

Post by johntash » Fri, Jan 03, 2020, 9:41 PM UTC

I don't have a lot of experience with it yet, but Loki looks promising for small projects. You'd still use it as a centralized logging server, but it's not as resource-expensive as something like self-hosting ELK.

I've only been using it for my homelab, and haven't even moved everything to it yet - but I like it so far. I already use Grafana+influxdb for metrics so having logs in the same interface is nice.

https://grafana.com/oss/loki/

Re: Ask HN: How do you manage logs for your backend services?

#9

Post by zMiller » Fri, Jan 03, 2020, 9:50 PM UTC

fluentd is great. You can setup forwarding nodes, that relay logs to one or mutiple masters that then persists into whatever layer(s) you want. Tolerance and failover baked in. Tons of connectors and best of all docker logs driver is built ships with docker so almost zero setup to get your container logs to fluentd. Also works nicely with kubernetes too!

Re: Ask HN: How do you manage logs for your backend services?

#10

Post by toomuchtodo » Fri, Jan 03, 2020, 9:51 PM UTC

Graylog