Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do you manage logs for your backend services?

news.ycombinator.com

Post reply on HN
Print view View original

101–110 of 142 posts

Re: Ask HN: How do you manage logs for your backend services?

#101

Post by mrweasel »

sethammons wrote: post #55

It is expensive. We forward logs to spkunk (we run our own instances). Splunk is really solid. All the logs are json and require certain fields. We use it for tend analysis, alerting, graphs, reports, and digging into production issues. It digs through terabytes of data relatively quickly.

I think Splunk is the reason I can’t recommend ELK, Splunk simply makes ELK look almost non-functional. This is years ago but we tried Splunk for a year, but the switched to ELK due to pricing. Our number of searches dropped to almost zero after that, because usability was so poor, in comparison.

As a result we didn’t utilize the data we had, or in many cases reverted to using grep.

If you want a cheaper alternative, Humio has become rather good and is relatively easy to use.

Re: Ask HN: How do you manage logs for your backend services?

#102

Post by Too »

james_s_tayler wrote: post #85
keyle wrote: post #32

If it's small, text files that you rotate per day and delete after 1-3 month(s). If it's big, Graylog is great. If it's too big, /dev/null, best logs gathering since 1971.

does /dev/null support sharding?

Absolutely https://devnull-as-a-service.com/

Re: Ask HN: How do you manage logs for your backend services?

#103

Post by Havoc »

whatsmyusername wrote: post #43

If you haven't read the chapter of 12factor on logging I highly recommend it https://www.12factor.net/logs This is coming from an ops person, do that and I'll be happy. Essentially the goal is to externalize all your log routing to stdout, then wrap tooling around your application to route it wherever you want it to go. It's geared toward heroku but same rules apply in docker land and more traditional VM environments…

>I have a format I use I'd be willing to send you if you want it.

Throw the nginx one on a paste in please

Re: Ask HN: How do you manage logs for your backend services?

#106

Post by rvanmil »

solatic wrote: post #79

Allow me to rep the tool I help build: Coralogix, which is a managed log analytics service. You haven't said what your budget is, but our pricing starts at $15/month to handle 5 GB/month of logs - certainly cheaper than running ELK yourself. https://coralogix.com

Very happy with Coralogix. We use the service on many of our Heroku dynos.

Re: Ask HN: How do you manage logs for your backend services?

#107

Post by boulos »

jjjensen90 wrote: post #90
dmlittle wrote: post #86

Earlier quoted context omitted.

From what I've seen at demo booths in conferences, Datadog's logging is impressive but also incredibly expensive. At the rate that we produce logs we'd be paying over $30k/mo. They claim that we can use log rehydration and not ingest all logs but then we can't really have alerts on them so what's the point in having them. Yes, I understand that you can look at the logs when things are going wrong but you can also kno…

You can create metrics and alerts from filtered logs in Datadog. The process would be: log data -> add index filters -> go to live tail and create a metric on a filtered log event -> create monitor on metric. edit: also, you log 24 billion messages a month? I think that's what it would be to cost $30k for their platform per month

24B/month is less than 10k/second.

10k qps is certainly not a dev/test instance, but if you were to log all RPCs including database lookups, you could easily get there with even just 100-ish user-facing requests per second.

Re: Ask HN: How do you manage logs for your backend services?

#108

Post by WinonaRyder »

We love JSON logs and previously just sent most of it to systemd's journald and use a custom tool to view them. But maybe a year ago Grafana released https://github.com/grafana/loki and we've been using it on https://oya.to/ ever since.

IIRC, the recommended way to integrate it with Grafana is via promtail but we weren't too keen on the added complexity of yet-another service in the middle so we developed a custom client library in Go to just send the logs straight to Loki (which we should probably open source at some point).

I don't think there's any fancy graph integration yet, but the Grafana explore tab with log level/severity and label filtering works well enough esp. since they introduced support for pretty printed JSON log payloads.

Re: Ask HN: How do you manage logs for your backend services?

#110

Post by golanggeek »

dmlittle wrote: post #86
batter wrote: post #58

Earlier quoted context omitted.

I find Stackdriver ugly compared to SumoLogic or Datadog. Also it has ingestion limits, we're losing logs when load becomes considerable.

From what I've seen at demo booths in conferences, Datadog's logging is impressive but also incredibly expensive. At the rate that we produce logs we'd be paying over $30k/mo. They claim that we can use log rehydration and not ingest all logs but then we can't really have alerts on them so what's the point in having them. Yes, I understand that you can look at the logs when things are going wrong but you can also kno…

In which conferences do you see Datadog booths
Post reply on HN