Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Preview of Explore Logs, a new way to browse your logs without writing LogQL

grafana.com

Post reply on HN
Print view View original

71–80 of 95 posts

Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL

#71

Post by zikohh »

Spiwux wrote: post #61

After having used Datadog for several years, going back to Grafana / Loki / Prometheus felt like regressing by two decades. As much as I appreciate free solutions, I feel like Grafana has really fallen behind when it comes to developer experience

Grafana cloud is better for querying logs. Grafana cloud is probably a bit better for querying metrics. Grafana cloud is terrible at finding traces or even loading them. Datadog is lightyears ahead. For alerting I feel datadog has better features but is overwhelming with all the different options.

grafana is very quirky for searching for traces. And has a huge learning curve.

Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL

#72

Post by mason55 »

ta1243 wrote: post #64
mason55 wrote: post #28

Earlier quoted context omitted.

Yup and the reason no one markets something like "tail the logs for server X" is because, if you're talking in the context of an individual server, you're too small for anyone to care about.

I've got logs from hundreds of servers that I use standard tools to look at, and that's a small system. Centralising logs has been a thing for decades.

Which is fine, I'm just saying you're not the target market for the big observability vendors.

The current generation of observability tools is built for distributed systems that are basically too complex to reason about, and so you have other ways of monitoring and debugging them. When you have 10's of k's of ephemeral containers running hundreds of services, you can't just look at some logs for a server to understand what's going on (ignoring the fact that servers aren't even a primitive in this system).

10's of GBs of logs a day just doesn't move the needle on pricing. They want the customers that are going to generate 7 figures in revenue and those customers aren't talking about aggregating logs from a few hundred servers.

Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL

#73

Post by remram »

Matthias247 wrote: post #60
remram wrote: post #49

Earlier quoted context omitted.

That's definitely false

Why? If the pod is defined to spawn multiple containers, and each container runs the same application, then this seems true to me? Unless you would add an additional filter on the container name.

Well yes obviously you have to filter on container if you want a single container (just like kubectl logs -l ). The parent comment was phrased as a limitation of Loki, of course if you request all logs for an application you'll get all containers, or if you request all logs for an applications or a namespace you will get that.

Not being able to filter between multiple processes or multiple restart of a container was a genuine issue, not being able to filter between pods of a deployment is not.

Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL

#74

Post by adra »

School-Cotton wrote: post #69
dzikimarian wrote: post #67

Earlier quoted context omitted.

Well, honestly I don't understand what's missing - you just pick time window, instance and have logs displayed line-by-line as they happened. Best to configure view for this to limit columns and maybe pre-configure some filters. Plus annotate your logs with timestamp, so you rely on time of event and not time of ingestion. But these both are one-time configuration thing and then you can simply scroll.

> Well, honestly I don't understand what's missing - you just pick time window, instance and have logs displayed line-by-line as they happened. What's missing is that I don't want to learn and use some clunky web UI in order to do this. I want the UI to be "download this text file" and then use the tools I already know and understand (local text processing utilities and text editors)

This seems like a solution for pets. If you have a lot of pets, this sounds totally reasonable, but it isn't some universal truism. People are moving away from pets as they're often harder to work with cattle. That also means you need an observability aggregation which can make sense of what's happening everywhere, not just one instance of on machine.

Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL

#75

Post by Matthias247 »

remram wrote: post #73
Matthias247 wrote: post #60

Earlier quoted context omitted.

Why? If the pod is defined to spawn multiple containers, and each container runs the same application, then this seems true to me? Unless you would add an additional filter on the container name.

Well yes obviously you have to filter on container if you want a single container (just like kubectl logs -l ). The parent comment was phrased as a limitation of Loki, of course if you request all logs for an application you'll get all containers, or if you request all logs for an applications or a namespace you will get that. Not being able to filter between multiple processes or multiple restart of a container was…

I actually didn't understand it being phrased as a limitation. It could also be a feature - maybe one would prefer to look at logs for multiple services within a single query?

Anyhow, the nice thing about the system is that one can get anything that is preferable as long as the logs are annotated correctly (with pod and container id).

Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL

#76

Post by School-Cotton »

sofixa wrote: post #34
School-Cotton wrote: post #27

Earlier quoted context omitted.

That will, if I understand correctly, get the logs for one pod, not for one process. For example if the pod restarted 10 times you will not get 10 separate files from that query.

You'd have the label shown in the output that indicates the log line in question is from a different process/pod/container/host/whatever.

How so? The pod, container, and host labels should be the same for a process that crashes and is automatically restarted, no?

Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL

#77

Post by pas »

remus wrote: post #58
fellerts wrote: post #17

Earlier quoted context omitted.

Just want to chip in and say that I wholeheartedly agree with you. I'm not a cloud developer either, but I'm regularly forced into what's apparently called "Google Cloud's operations suite" to grovel through logs. Compared to working with Linux journals using the tried and true text manipulation tools, it feels like looking through a straw with oven mitts on. I'd happily download a 500 MB text file instead, but there…

> but I'm regularly forced into what's apparently called "Google Cloud's operations suite" to grovel through logs Is this google cloud logging? If so, personally I quite like it, especially for looking through logs from multiple sources at the same time. Being able to put all your logs through there, and then search them with a simple query language, feels very convenient.

it's ridiculously slow. and compared to how expensive it is ... it's robbery in daylight.

Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL

#78

Post by remram »

nicoritschel wrote: post #7

While this is a step in the right direction, just let me write something closer to SQL. Influx did this correctly.

Influx used SQL, then deprecated it and made everyone use Flux, then deprecated that and moved back to SQL. They are definitely not "doing it correctly" when it comes to query language.

Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL

#79

Post by remram »

Matthias247 wrote: post #75
remram wrote: post #73

Earlier quoted context omitted.

Well yes obviously you have to filter on container if you want a single container (just like kubectl logs -l ). The parent comment was phrased as a limitation of Loki, of course if you request all logs for an application you'll get all containers, or if you request all logs for an applications or a namespace you will get that. Not being able to filter between multiple processes or multiple restart of a container was…

I actually didn't understand it being phrased as a limitation. It could also be a feature - maybe one would prefer to look at logs for multiple services within a single query? Anyhow, the nice thing about the system is that one can get anything that is preferable as long as the logs are annotated correctly (with pod and container id).

No, once again, the trouble is that you can't get the logs for a specific execution. If a container in your pod restarts, that is invisible to Loki, you have to look for whatever the container writes on startup and cut there manually. If you want a specific process in your container, it's mixed with the rest.

Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL

#80

Post by [deleted] »

nine_k wrote: post #11
School-Cotton wrote: post #6

I'm not really a cloud expert so maybe I'm fundamentally missing something about how I'm "supposed to work", but honestly all I have ever wanted to do, when looking at logs, is see the log from one process, from beginning to end, as a text file. You can of course do this using kubectl but only for the most recent two instances of a given pod which isn't helpful when investigating an incident that happened a while ago…

Could LogQL do.something like select * from stdout, stderr where session_id = 123456 ? If not, why?

[deleted]
Post reply on HN