Clickhouse + Grafana or Prometheus + Victoriametrics
Ask HN: How do you manage logs for your backend services?
111–120 of 142 posts
Re: Ask HN: How do you manage logs for your backend services?
#112Earlier quoted context omitted.
> One important lesson I've learned in my career is never to run your own observability platform if you can afford for someone else (whose entire product is observability) do it for you. Why is that?
1. You have to keep your logging, monitoring, alerting infrastructure up yourself 2. You have to monitor, log, and alert on your that infrastructure yourself with something else 3. You usually have to spend more money, both in hosting and in developer/ops time, on getting something mediocre compared to a provider that exclusively does observability as a product 4. Logging etc are a commodity and you should have a rea…
Yep, if you can afford it make it someone else's problem. No team should need to have dedicated OPs people that are responsible for keeping the logging platform running. This is a solved problem and costs the company more in engineering time (paying a person to do the job) than the Saas solution.
E.g. 30K/month isn't much when you compare it to paying a sole engineer 15K/year + cloud costs of hosting your own logging solution + any licenses. There's also a slim chance the home grown logging platform will be as resilient as a Saas product.
I for one am very grateful to DataDog's support team as they've been very helpful in debugging logging problems we've had in the past.
Re: Ask HN: How do you manage logs for your backend services?
#113AWS CloudWatch Logs has come a long way. The new Insights UI is great. No need for us to manage ELK for logs anymore.
Do all developers have AWS console access to check cloudwatch logs?
Re: Ask HN: How do you manage logs for your backend services?
#114We're using Seq as log server which you must host yourself. This fine and free product is from the same guys as serilog which is known in the dotnet world.
Re: Ask HN: How do you manage logs for your backend services?
#115Earlier quoted context omitted.
From what I've seen at demo booths in conferences, Datadog's logging is impressive but also incredibly expensive. At the rate that we produce logs we'd be paying over $30k/mo. They claim that we can use log rehydration and not ingest all logs but then we can't really have alerts on them so what's the point in having them. Yes, I understand that you can look at the logs when things are going wrong but you can also kno…
In which conferences do you see Datadog booths
AWS Re:Invent had a DataDog booth.
Re: Ask HN: How do you manage logs for your backend services?
#116Earlier quoted context omitted.
I find Stackdriver ugly compared to SumoLogic or Datadog. Also it has ingestion limits, we're losing logs when load becomes considerable.
From what I've seen at demo booths in conferences, Datadog's logging is impressive but also incredibly expensive. At the rate that we produce logs we'd be paying over $30k/mo. They claim that we can use log rehydration and not ingest all logs but then we can't really have alerts on them so what's the point in having them. Yes, I understand that you can look at the logs when things are going wrong but you can also kno…
Imo stack driver on GCP is too expensive too
Re: Ask HN: How do you manage logs for your backend services?
#117I was just about to start looking into doing this myself and for the foreseeable future, I'll probably just use `dsh`... since I'm a cheapskate, have been trying to reduce my usage on cloud tools, and I just found out about it today: https://www.netfort.gr.jp/~dancer/software/dsh.html.en Once installed, change the default from rsh to ssh where it's installed e.g. `/usr/local/Cellar/dsh/0.25.10/etc/dsh.conf` Then setu…
Wow. Maybe just use syslog for this case? It's designed for centralized logging
Yes... Uhh.. `dsh` works with tailing syslog too. I don't know why it wouldn't?
Sorry for my example not being general enough?
Re: Ask HN: How do you manage logs for your backend services?
#118I'm pretty late to this party, but I've been using rsyslog and "EK" (skip the L - it's way too slow and resource hungry).
rsyslog / syslog-ng handles shipping logs to a central server and it's dead simple to keep local logs and a central log at the same time. Every language can spit logs to syslog vey quickly. And then you can use plugins to inject your life from rsyslog directly into elastisearch, which is incredibly fast.
Other critiques about ES still apply especially when it comes to managing conflicting keys in structured logs, but most complaints about fragility and scaling are because of Logstash, which I agree, is Terrible for logging.
I've written this up in detail if anyone is interested.
Re: Ask HN: How do you manage logs for your backend services?
#119Re: Ask HN: How do you manage logs for your backend services?
#120I highly recommend Datadog's logging platform. One important lesson I've learned in my career is never to run your own observability platform if you can afford for someone else (whose entire product is observability) do it for you. I've used ELK (managed and hosted), Splunk, NewRelic, Loki, and home grown local/cloud file logs and nothing has been as cheap, easy, and powerful as Datadog. They charge per million log e…
> One important lesson I've learned in my career is never to run your own observability platform if you can afford for someone else (whose entire product is observability) do it for you. Why is that?
A think a more generic advice would be to focus on what brings value to your customers and makes your product better.
> I thought using loops was cheating, so I programmed my own using samples. I then thought using samples was cheating, so I recorded real drums. I then thought that programming it was cheating, so I learned to play drums for real. I then thought using bought drums was cheating, so I learned to make my own. I then thought using premade skins was cheating, so I killed a goat and skinned it. I then thought that that was cheating too, so I grew my own goat from a baby goat. I also think that is cheating, but I’m not sure where to go from here. I haven’t made any music lately, what with the goat farming and all.