Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do you manage logs for your backend services?

news.ycombinator.com

Post reply on HN
Print view View original

91–100 of 142 posts

Re: Ask HN: How do you manage logs for your backend services?

#91

Post by firstbabylonian »

jjjensen90 wrote: post #88

I highly recommend Datadog's logging platform. One important lesson I've learned in my career is never to run your own observability platform if you can afford for someone else (whose entire product is observability) do it for you. I've used ELK (managed and hosted), Splunk, NewRelic, Loki, and home grown local/cloud file logs and nothing has been as cheap, easy, and powerful as Datadog. They charge per million log e…

> One important lesson I've learned in my career is never to run your own observability platform if you can afford for someone else (whose entire product is observability) do it for you.

Why is that?

Re: Ask HN: How do you manage logs for your backend services?

#93

Post by jjjensen90 »

firstbabylonian wrote: post #91
jjjensen90 wrote: post #88

I highly recommend Datadog's logging platform. One important lesson I've learned in my career is never to run your own observability platform if you can afford for someone else (whose entire product is observability) do it for you. I've used ELK (managed and hosted), Splunk, NewRelic, Loki, and home grown local/cloud file logs and nothing has been as cheap, easy, and powerful as Datadog. They charge per million log e…

> One important lesson I've learned in my career is never to run your own observability platform if you can afford for someone else (whose entire product is observability) do it for you. Why is that?

1. You have to keep your logging, monitoring, alerting infrastructure up yourself

2. You have to monitor, log, and alert on your that infrastructure yourself with something else

3. You usually have to spend more money, both in hosting and in developer/ops time, on getting something mediocre compared to a provider that exclusively does observability as a product

4. Logging etc are a commodity and you should have a really good reason to build or run something yourself if it is not your core competency

5. Observability is much harder than it sounds and providing a cohesive platform that ties log events to traces to metrics to alerts, and keeping it up and highly available is a hard problem that you shouldn't do if observability isn't your core competency/product

Re: Ask HN: How do you manage logs for your backend services?

#94

Post by tmikaeld »

chucky_z wrote: post #24
tmikaeld wrote: post #16

Earlier quoted context omitted.

I've been benching it in production. On a 1core 2gb vps it can do around 1200 logs/sec. Compared to ELK, we're saving several hundred $ per month.

If you can't do 1200 logs/sec with that hardware + elasticsearch, you've done something horrendously wrong. You can do 1200 logs/sec with a container limited to 1 core and 256mb of memory running elasticsearch.

The savings are in RAM costs due to ELK stack basically not running under 4GB. And larger logs with stack trace are much slower compared to Loki. There's a reason there is no cheap ELK services.

Re: Ask HN: How do you manage logs for your backend services?

#95

Post by geewee »

I've had the misfortune of setting up Application Insights for logging across a distributed system.

It's awful. The integration with most (C#) logging frameworks is horribly, the adaptive sampling, which is hard to turn off, means that Application Insights randomly drops logs, which makes any sort of distributed tracing of events really difficult.

To top that, there's a delay of 5-10 minutes from the logs are written until they're queryable, which is a huge pain when debugging your setup.

Re: Ask HN: How do you manage logs for your backend services?

#98

Post by particlesplus »

Papertrail/Timber.io - cheapest way to aggregate logs and has simple search functionality.

Scalyr - my personal favorite. Just a little more costly than Papertrail, but can do as much as any full service SaaS - powerful queries, dashboards, and alerts. Takes some practice to learn, but their support is very helpful.

Sumologic - Fully featured log agregator. It works pretty good but their UI is super duper annoying. You have to use their tabbing system on their page, you can't open multiple dashboards/logs in multiple browser tabs. For the money, I personally prefered Scalyr, but this is a reasonable option.

Splunk - a great place to $plunk down your $$$$. I think their cheapest plan was $60k/yr, but I will admit that it was easy to get going and use and also had the most features. It's not a bad bang for your buck as long as you have lots of bucks to spend.

Re: Ask HN: How do you manage logs for your backend services?

#100

Post by particlesplus »

ekimekim wrote: post #52

My main advice is avoid ELK. I have no clue how Elastic managed to convince the world that Elasticsearch should be the default log database when it is _terrible_ for logs. If you're logging structured JSON, then you'll hit a ton of issues - Elasticsearch can't handle, say, one record with {foo: 123} and another with {foo: "abc"} - it'll choke on the different types and 400 error on ingest. Even if you try to coerce v…

don't forget that the forward slash "/" is a special character in elastic search. so try searching service logs by http route and you are completely f&^$ed.
Post reply on HN