Live data from Hacker News

Ask HN: How do you manage logs for your backend services?

news.ycombinator.com

141–142 of 142 posts

Re: Ask HN: How do you manage logs for your backend services?

#141
At trivago, we rely on the ELK stack to support our logging pipeline. We use a mix of filebeat and Gollum (github.com/trivago/gollum) to write logs into Kafka (usually we encode them using Protocol Buffers) these are later on read by Logstash and written into our ELK cluster.

For us, ELK has scaled quite well both in terms of queries/s and writes/s and we ingest ~1.5TB of logs daily just in our on-premise infrastructure. The performance/monitoring team (where I work at) consists of only 4 people and although we take care of our ELK cluster it is not our only job (nor a component that requires constant attention).

Re: Ask HN: How do you manage logs for your backend services?

#142

I'm pretty late to this party, but I've been using rsyslog and "EK" (skip the L - it's way too slow and resource hungry). rsyslog / syslog-ng handles shipping logs to a central server and it's dead simple to keep local logs and a central log at the same time. Every language can spit logs to syslog vey quickly. And then you can use plugins to inject your life from rsyslog directly into elastisearch, which is incredibl…

Yes please give some configuration and performance details.

I don't have anything regarding performance detail, but I wrote this detailed post about configuration.

https://www.reddit.com/r/devops/comments/9g1nts/rsyslog_elas...

Post reply on HN