What do you guys do with docker logs? They seem to just accumulate over time.
You can setup log rotation & log pruning in daemon.json. Works flawlessly.
Ask HN: How do you manage logs for your backend services?
81–90 of 142 posts
Re: Ask HN: How do you manage logs for your backend services?
#82If you haven't read the chapter of 12factor on logging I highly recommend it https://www.12factor.net/logs This is coming from an ops person, do that and I'll be happy. Essentially the goal is to externalize all your log routing to stdout, then wrap tooling around your application to route it wherever you want it to go. It's geared toward heroku but same rules apply in docker land and more traditional VM environments…
I'm interested in these webserver configs you got for logs, please
Re: Ask HN: How do you manage logs for your backend services?
#83AWS CloudWatch Logs has come a long way. The new Insights UI is great. No need for us to manage ELK for logs anymore.
Looking into it for the current ELK replacement, looks great so far (especially simple enough query language). Offtopic: what do you use for load balancing logs in S3? I’m seeing basically two in-AWS options: Athena (but it lacks built-in fancy UI) or ship LB logs to the CloudWatch too.
Re: Ask HN: How do you manage logs for your backend services?
#84Cloudwatch logs + insights or Fluentd - kinesis firehose - elasticsearch
Re: Ask HN: How do you manage logs for your backend services?
#85If it's small, text files that you rotate per day and delete after 1-3 month(s). If it's big, Graylog is great. If it's too big, /dev/null, best logs gathering since 1971.
does /dev/null support sharding?
Re: Ask HN: How do you manage logs for your backend services?
#86Google's Stackdriver. I've been using Google App Engine for some ten years now and I'm still dumbfounded that this is still an ongoing struggle for other platforms. It collates logs from a variety of sources, presents requests as a single unit, has sophisticated searching capabilities, and the UI doesn't suck. Best of all, it just works... there's zero configuration on GAE. Such a time saver.
I find Stackdriver ugly compared to SumoLogic or Datadog. Also it has ingestion limits, we're losing logs when load becomes considerable.
From what I've seen at demo booths in conferences, Datadog's logging is impressive but also incredibly expensive. At the rate that we produce logs we'd be paying over $30k/mo. They claim that we can use log rehydration and not ingest all logs but then we can't really have alerts on them so what's the point in having them. Yes, I understand that you can look at the logs when things are going wrong but you can also know _from_ the logs when things are going wrong.
Re: Ask HN: How do you manage logs for your backend services?
#87My main advice is avoid ELK. I have no clue how Elastic managed to convince the world that Elasticsearch should be the default log database when it is _terrible_ for logs. If you're logging structured JSON, then you'll hit a ton of issues - Elasticsearch can't handle, say, one record with {foo: 123} and another with {foo: "abc"} - it'll choke on the different types and 400 error on ingest. Even if you try to coerce v…
Second every line of this. If you can find someone else to handle all this stuff with quality that you can accept - do that and focus on your app instead. All that field types and cardinality mess are “easily” prevented with a bit of self-discipline, which you’re going to develop anyway if you really going to use your logs for incident response or statistics.
Re: Ask HN: How do you manage logs for your backend services?
#88I highly recommend Datadog's logging platform. One important lesson I've learned in my career is never to run your own observability platform if you can afford for someone else (whose entire product is observability) do it for you.
I've used ELK (managed and hosted), Splunk, NewRelic, Loki, and home grown local/cloud file logs and nothing has been as cheap, easy, and powerful as Datadog. They charge per million log events indexed but also allow you to exclude log events by patterns/source/etc and they ingest but ignore those rows (you pay $0.10/gb for those ignored logs).
The 12 factor way to do logging is very easy with Datadog, as you can tell the agent to injest from stdout, containers, or file sources, then the application is agnostic to the log aggregator as the agent will collect and send logs to the platform.
Not only is it cheap and easy to set up, it also gives you the option to take advantage of the other features of Datadog that can be built on your log data. Metrics based on log parsing, alerts, configuration via terraform, etc become possible when you ship your logs to their platform.
I've seen our production apps log 10-20k messages per second without Datadog breaking a sweat but I'm not sure if they have any limits.
Re: Ask HN: How do you manage logs for your backend services?
#89Try Graylog!!!!!
Re: Ask HN: How do you manage logs for your backend services?
#90Earlier quoted context omitted.
I find Stackdriver ugly compared to SumoLogic or Datadog. Also it has ingestion limits, we're losing logs when load becomes considerable.
From what I've seen at demo booths in conferences, Datadog's logging is impressive but also incredibly expensive. At the rate that we produce logs we'd be paying over $30k/mo. They claim that we can use log rehydration and not ingest all logs but then we can't really have alerts on them so what's the point in having them. Yes, I understand that you can look at the logs when things are going wrong but you can also kno…
You can create metrics and alerts from filtered logs in Datadog.
The process would be: log data -> add index filters -> go to live tail and create a metric on a filtered log event -> create monitor on metric.
edit: also, you log 24 billion messages a month? I think that's what it would be to cost $30k for their platform per month