What do you guys do with docker logs? They seem to just accumulate over time.
Ask HN: How do you manage logs for your backend services?
61–70 of 142 posts
Re: Ask HN: How do you manage logs for your backend services?
#62I've recently spent a few days adding proper instrumentation to a .NET Core side project using CloudWatch Logs.
I don't know anything about the Go ecosystem, but if there is a nice solution for structured logging then CloudWatch Logs is very easy to implement, very cheap, can easily make decent dashboards with it and if needed in the future you can forward the logs onto Elastic Search.
I'm using a library called Serilog in my project to log everything in a consistent structured log that gets all kinds of metadata automatically appended to it and the json payload winds up in CloudWatch Logs. Then I've got a couple of Custom metrics to measure throughput as well as some log filter metrics to track latency of the service and it's downstream dependencies.
It works very well and was surprisingly quick to put together. I believe cost wise my usage level is covered pretty much for free. Can't complain!
Re: Ask HN: How do you manage logs for your backend services?
#63Earlier quoted context omitted.
I don't recommend Splunk because I don't want someone recommending we run Splunk onsite ever again. Running your own log infrastructure is the absolute worst. As in, we had to put a staff devops engineer on just that for 2 months the last time the company I was at needed an upgrade.
Correct, it's not uncommon to have one engineer full time on an internal Splunk infrastructure at a mediumish org. Not everything has to be Cloud. This is what people are paid to do.
Yeah we're not putting a person on logging when we can outsource it for a fraction of the cost.
Re: Ask HN: How do you manage logs for your backend services?
#64I was just about to start looking into doing this myself and for the foreseeable future, I'll probably just use `dsh`... since I'm a cheapskate, have been trying to reduce my usage on cloud tools, and I just found out about it today: https://www.netfort.gr.jp/~dancer/software/dsh.html.en Once installed, change the default from rsh to ssh where it's installed e.g. `/usr/local/Cellar/dsh/0.25.10/etc/dsh.conf` Then setu…
Maybe give this a read, https://www.usenix.org/short-topics/building-logging-infrast...
Re: Ask HN: How do you manage logs for your backend services?
#65We use and like https://www.scalyr.com/ .
Second this. Love scalyr. Very fast/efficient setup and fast/efficient interface. Great API which allows config to be stored in version control.
Re: Ask HN: How do you manage logs for your backend services?
#66My main advice is avoid ELK. I have no clue how Elastic managed to convince the world that Elasticsearch should be the default log database when it is _terrible_ for logs. If you're logging structured JSON, then you'll hit a ton of issues - Elasticsearch can't handle, say, one record with {foo: 123} and another with {foo: "abc"} - it'll choke on the different types and 400 error on ingest. Even if you try to coerce v…
I came to the same conclusion about ELK. But you know, enterprisey + consultants = uptake.
Try Scalyr.
Re: Ask HN: How do you manage logs for your backend services?
#67We use flume forwarding to s3 and then athena to query the logs. Flume processes each logfile with morphline (which is akin to logstash) and parses each rawlog into json before pushing to s3. We used to run an elk stack but hit a bottleneck crunching logs with logstash. We found flume's morphline to be performant enough and the nice property of flume is that you can fanout and write to multiple datasources. It's iron…
I've always wondered why companies for whom logs are important but not their core focus / product bother implementing stuff like this themselves. Surely a saas service that does just logs can do it cheaper and with more features? Is it compliance?
Re: Ask HN: How do you manage logs for your backend services?
#68We use DataDog. We run their logging agent in a container which tails the log file and syncs with their cloud service.
Re: Ask HN: How do you manage logs for your backend services?
#69I used Papertrail at my last job, search function works well, and it was easy to use.
Seconding Papertrail. I don't love it but it works and it's cheap enough and I don't need to love every tool I use. I do constantly wonder if there's a logging/searching solution small and lightweight enough to fit on the cheapest DO droplet. $7/gb/mo on Papertrail vs $5/mo with a 25GB SSD? That'd be a no-brainer, but ELK and Graylog won't run with that low of hardware.
Depends on how risky you're willing to be with your log data. Something like what you're describing could lose all of your data with a single instance or hard drive failure while a saas like Paper trail has your data replicated across data centers.
Re: Ask HN: How do you manage logs for your backend services?
#70syslog / stderr -> Kafka -> centralized ELK
Files / log rotation is completely the wrong approach because log entries are mostly innately structured, rich data that occurs at a specific time. Serializing and then parsing log lines again is wasted effort. Messaging is a better fit than lines in files which create log-management headaches like not rotating, losing messaging on rotation, compressing/decompressing and a lengthy soup of destructured data that fills up local disks.
Logging to files on local disks is wrong and often creates privacy problems. Logging to cloud services is also expensive, a legal quagmire and raise data portability concerns.