Ask HN: How do you manage logs for your backend services?

Somehow I came across scalyr and it's just phenomenally fast - and cost less than our ELK cluster. Definitely worth trying if it provides the features you need.

This is coming from an ops person, do that and I'll be happy. Essentially the goal is to externalize all your log routing to stdout, then wrap tooling around your application to route it wherever you want it to go. It's geared toward heroku but same rules apply in docker land and more traditional VM environments.

We either send logs ECS -> Cloudwatch for our AWS stuff or docker swarm -> fluentd -> s3 for an onsite appliance (also anything syslog ends up in Sumologic through a collector we host in the environment). From there the logs get consumed by our SIEM (in our case Sumologic, which is a great service). We keep logs hot for 90 days and ship archives back to s3 for a year. Set up the lifecycle management stuff, keeping log files forever is not only a waste but can actively hurt you if they ever get exposed in a breach.

I highly recommend formatting your logs JSON (even if you have to do it by hand like in Apache). If you do that and go to Splunk, Sumologic, or ELK all your fields will be populated either automatically or with a single filter. Saves writing or buying your own and if you add a field there's no action for you to take.

nginx/apache default logging is complete trash. Look at the variables they expose, there's a lot of stuff in there you'll want to add to the log format to make your life loads easier. I have a format I use I'd be willing to send you if you want it.

I don't recommend logstash (the L in ELK), ever (except maybe if you're java across the board). It's way too damn heavy to run on a workload host, fluentd is much lighter (and not java, why would I deploy java for a system tool ever?). Maybe as a network collector you throw syslog at but that would be it.

For your use case Sumologics free service would be great. You can get I think up to 200m a day with a weeks retention for free and you'll get exposed to what an SIEM can do for you (ingestion rate and retention period are typically how hosted solutions are billed, you'll need email with any non-free email domain to get a free account from them). IMO you have to get to some fairly insane log rates for me to ever recommend running ELK stack yourself, it has way too much care and feeding if you want to run it correctly with good security.

I had a good experience with a local decentralized logging system, essentially daemontools-style service logs, that then fed in to ELK. ELK provided the bulk storage and analysis; and the local daemontools logs provided the immediate on-machine per-individual-service recent log access, and decoupled logging from the network connection to logstash.

* http://jdebp.uk./Softwares/nosh/guide/commands/export-to-rsy...

* http://jdebp.uk./Softwares/nosh/guide/commands/follow-log-di...

One of the advantages of this approach is that one can do the daemontools-style logging first, very simply, without centralization, and with comparatively minor expense; and then tack on ELK later, when volume or number of services gets large enough, without having to alter the daemontools-style logging when doing so.

Of course, it can be something else other than ELK, fed from the daemontools-style logs.

One thing that I recommend against, ELK or no, is laboriously re-treading the path of the late 1970s and the 1980s, starting from that "logging straight to a file". Skip straight through to the 1990s. (-:

* http://jdebp.uk./FGA/do-not-use-logrotate.html