Ask HN: How do you manage logs for your backend services? • Hacker News Board

Re: Ask HN: How do you manage logs for your backend services?

#71

Post by natebutler » Sat, Jan 04, 2020, 4:33 AM UTC

RhodesianHunter wrote: post #67

natebutler wrote: post #19

We use flume forwarding to s3 and then athena to query the logs. Flume processes each logfile with morphline (which is akin to logstash) and parses each rawlog into json before pushing to s3. We used to run an elk stack but hit a bottleneck crunching logs with logstash. We found flume's morphline to be performant enough and the nice property of flume is that you can fanout and write to multiple datasources. It's iron…

I've always wondered why companies for whom logs are important but not their core focus / product bother implementing stuff like this themselves. Surely a saas service that does just logs can do it cheaper and with more features? Is it compliance?

We are in the unfortunate circumstance where we have high traffic but are budget conscious. S3 is cheap and Athena is as well. We significantly reduced cost moving from a fairly large elk cluster on ec2 to a handful of flume machines running morphline. We’ve looked at datadog and scalyr and even went so far as to implement a flume sink to scalyr but the scalyr quote was way too high.

Re: Ask HN: How do you manage logs for your backend services?

#72

Post by jdc0589 » Sat, Jan 04, 2020, 4:57 AM UTC

zygy wrote: post #15

We use and like https://www.scalyr.com/ .

ditto. we've got maybe 4,000 logs/second going to scalyr. it's fast, has good features, and is easily searchable (with structured logs).

Re: Ask HN: How do you manage logs for your backend services?

#73

Post by koreth1 » Sat, Jan 04, 2020, 4:59 AM UTC

zygy wrote: post #15

We use and like https://www.scalyr.com/ .

Same for us, though they have outages more often than I'd prefer. Half the time when we get a PagerDuty alert, it's because Scalyr is screwed up and not ingesting our logs, not anything wrong with our systems.

But when it's not having an outage, it's lightning fast and easy to use. I especially like that you can do basically 100% of the configuration (aside from stuff like billing info) by uploading JSON files to them, so you can keep all your parsing and alerting configuration in your own version control system.

Re: Ask HN: How do you manage logs for your backend services?

#74

Post by Xorlev » Sat, Jan 04, 2020, 5:11 AM UTC

chucky_z wrote: post #24

tmikaeld wrote: post #16

Earlier quoted context omitted.

I've been benching it in production. On a 1core 2gb vps it can do around 1200 logs/sec. Compared to ELK, we're saving several hundred $ per month.

If you can't do 1200 logs/sec with that hardware + elasticsearch, you've done something horrendously wrong. You can do 1200 logs/sec with a container limited to 1 core and 256mb of memory running elasticsearch.

Really depends on the size of each log and the complexity of the tokenizers. With 1 core you have time budget of less than a millisecond per log statement for processing and that doesn't include the relevant ES/Lucene operational overheads.

This is extremely doable for some workloads, but not others. Really depends on what you're stuffing in.

Re: Ask HN: How do you manage logs for your backend services?

#75

Post by snisarenko » Sat, Jan 04, 2020, 5:29 AM UTC

weitzj wrote: post #40

I used logdna in a previous company. This is a really nice hosted, cheap logging service (like papertrail)

I am currently trying out logdna. Their charting is underwhelming.

Re: Ask HN: How do you manage logs for your backend services?

#76

Post by cmclaughlin » Sat, Jan 04, 2020, 5:30 AM UTC

AWS CloudWatch Logs has come a long way. The new Insights UI is great. No need for us to manage ELK for logs anymore.

Re: Ask HN: How do you manage logs for your backend services?

#77

Post by kabdib » Sat, Jan 04, 2020, 5:40 AM UTC

sethammons wrote: post #55

It is expensive. We forward logs to spkunk (we run our own instances). Splunk is really solid. All the logs are json and require certain fields. We use it for tend analysis, alerting, graphs, reports, and digging into production issues. It digs through terabytes of data relatively quickly.

Splunk can get really expensive. And there's something about renting your own data that rubs me the wrong way.

Re: Ask HN: How do you manage logs for your backend services?

#78

Post by edoceo » Sat, Jan 04, 2020, 5:42 AM UTC

rubyn00bie wrote: post #50

I was just about to start looking into doing this myself and for the foreseeable future, I'll probably just use `dsh`... since I'm a cheapskate, have been trying to reduce my usage on cloud tools, and I just found out about it today: https://www.netfort.gr.jp/~dancer/software/dsh.html.en Once installed, change the default from rsh to ssh where it's installed e.g. `/usr/local/Cellar/dsh/0.25.10/etc/dsh.conf` Then setu…

Wow. Maybe just use syslog for this case? It's designed for centralized logging

Re: Ask HN: How do you manage logs for your backend services?

#79

Post by solatic » Sat, Jan 04, 2020, 5:50 AM UTC

Allow me to rep the tool I help build: Coralogix, which is a managed log analytics service. You haven't said what your budget is, but our pricing starts at $15/month to handle 5 GB/month of logs - certainly cheaper than running ELK yourself.

https://coralogix.com

Re: Ask HN: How do you manage logs for your backend services?

#80

Post by gingerlime » Sat, Jan 04, 2020, 5:52 AM UTC

I used to use Logentries for a while. It's essentially a hosted ELK. I think they got bought-out or something, and the service went downhill. Don't remember exactly, but it was slow and clunky, and support wasn't great either.

Then discovered Scalyr. It was awesome. The UI isn't pretty, but it's super-powerful. It's fast. You have to format your logs to make the most of it, but it's worth it.

Unfortunately their alerting wasn't as flexible at the time (for example, it wouldn't include contextual info from the matching logs that triggered the alert). Besides that, we decided to consolidate things and move to Datadog.

Datadog is pretty great. The monitoring and alerting features are solid. They then added logging, APM and keep adding more. It's not that cheap, but overall works great for us.