Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: LogsQL – opinionated query language for logs

docs.victoriametrics.com

Post reply on HN
Print view View original

21–30 of 31 posts

Re: Show HN: LogsQL – opinionated query language for logs

#21

Post by valyala »

dan-robertson wrote: post #19
gbuk2013 wrote: post #18

Earlier quoted context omitted.

Genuine question: does your job involve troubleshooting from logs on a regular basis? Because if it does, I would be surprised that you feel the way you do. My experience is with ELK but at least Kibana interface is pretty decent for applying filter combinations to find the needle in a haystack of logs. And in terms of ingestion, if you are in a container environment you can just configure stdout from the container t…

I used to spend a lot of time looking at logs from a complex state machine. I would pull up a half day of logs in less (maybe a few GB), and search for something I was interested in like an id from an error message. This could be slow (tricks were disabling line numbers and searching backwards from the end) and then answer questions of the form ‘how long from this line until the next line matching x?’ or ‘what events…

VictoriaLogs could fit your use case:

- It natively supports 'stream' concept [1] - this is basically logs received from a single application instance.

- It allows efficiently querying all the logs, which belong to a single stream, on the given time range, via 'curl', and passing them to 'less' or to any other Unix command for further processing in streaming manner [2].

[1] https://docs.victoriametrics.com/victorialogs/keyconcepts/#s...

[2] https://docs.victoriametrics.com/victorialogs/querying/#comm...

Re: Show HN: LogsQL – opinionated query language for logs

#22

Post by valyala »

karmarepellent wrote: post #20
rixrax wrote: post #7

Am I the only one that feels that EVERYTHING is wrong in this ELK, Splunk, etc. Grafana world? The user interfaces that these monstrosities present us with are barely useable, everyone has their own query language, they force us to install their own agents own our hosts and servers, when I upload logs, many can't even take random JSON logs and input them in a structured way without defining pipeline rules or what now…

As other commenters already suggested I think it just comes down to what your actual day-to-day job is. In some companies you have dedicated data engineers whose job it is to understand these complex logging systems. But despite the complexity they may still derive value from it since they are deeply involved in writing SomeQL queries pretty much all day. At my place of work we do not derive much value from our ELK i…

I actively use grep, cat, uniq, sort, less and other Unix commands every day. That's why I added good integration with Unix pipes into VictoriaLogs, since it may cover missing functionality [1].

[1] https://docs.victoriametrics.com/victorialogs/querying/#comm...

Re: Show HN: LogsQL – opinionated query language for logs

#23

Post by valyala »

BillFranklin wrote: post #3

Interesting to see a new approach! You wrote that you don't like Loki's LogQL, but it looks quite similar (Victoria's LogQL first): log.level:error _stream:{app!~"buggy_app|foobar"} {app!~"buggy_app|foobar"} | "log.level:error" The pipes are arguably a bit noisy in Loki queries (compared to spaces in Victoria's), but I find they do make the queries a bit more readable, and it's easier to understand under the hood how…

- Loki doesn't allow queries without stream filters. This may be very inconvenient. For example, try selecting all the logs with the 'error' word in Loki. In LogsQL you just type 'error' and that's it!

- As I know, Loki doesn't allow selecting all the logs on the given time range. For example, try selecting all the logs for the last 5 minutes in Loki query language. In LogsQL this is just '_time:5m'.

- Loki has unreadable syntax for calculating analytics over the selected logs. For example, counting the number of logs with the 'error' word over the last 5 minutes in Loki looks like:

count_over_time({required="stream_filter"} | "error" [5m])

Compare this to LogsQL:

_time:5m error | stats count() as errors

- Loki doesn't allow calculating multiple stats in a single query. For example, try calculating the number of logs with the 'error' word, plus the total number of logs, over the last 5 minutes. In LogsQL this is easy:

_time:5m | stats count() if(error) as errors, count() as total

- As I know, Loki doesn't provide functionality for sorting of the returned logs.

- Loki can return only up to 5000 logs from a single query by default. VictoriaLogs allows returning billions of logs from a single query, without any performance and resource usage issues [1].

[1] https://docs.victoriametrics.com/victorialogs/querying/#comm...

Re: Show HN: LogsQL – opinionated query language for logs

#24

Post by valyala »

deepsun wrote: post #17
BillFranklin wrote: post #3

Interesting to see a new approach! You wrote that you don't like Loki's LogQL, but it looks quite similar (Victoria's LogQL first): log.level:error _stream:{app!~"buggy_app|foobar"} {app!~"buggy_app|foobar"} | "log.level:error" The pipes are arguably a bit noisy in Loki queries (compared to spaces in Victoria's), but I find they do make the queries a bit more readable, and it's easier to understand under the hood how…

I believe pipes for logs were invented by SumoLogic 10+ years ago. Or maybe someone before that.

Canonical pipes were invented ~50 years ago by Unix creators. They are successfully used for logs' analysis to this day.

Unix pipes have a simple idea - connecting the output of one program to the input of another program. This allows building arbitrary complex data processing pipelines by combining simple programs like grep, cat, awk, cut, head, etc., via pipes. These pipelines have nice properties:

- The data is processed in a streaming manner. This allows processing unlimited data streams without excess resource usage.

- The data processing speed is limited by the slowest program in the pipeline. This allows pausing and resuming data processing at any time. For example, by putting 'less' to the end of the pipeline.

- The data processing pipeline is terminated instantly as soon as some program in the pipeline terminates. For example, if you put 'tail -100' to the end of pipeline, then only the first 100 lines of data will be processed.

VictoriaLogs adapts these properties into logs processing with LogsQL pipes. The difference is that the data, which is flown between pipes, represents structured logs [1].

[1] https://docs.victoriametrics.com/victorialogs/querying/#comm...

Re: Show HN: LogsQL – opinionated query language for logs

#25

Post by valyala »

ople wrote: post #6

Recently there seems to be an bunch of SPL (Splunk) -like query languages popping up: PQL, PRQL, Grafana Explore Logs syntax, Kusto.. Probably others as well. Does yet another similar but slightly different language make sense? Why not leverage an existing one?

Diversity is great! Let's see which query language for logs will survive.

Re: Show HN: LogsQL – opinionated query language for logs

#26

Post by valyala »

bdcravens wrote: post #2

You have to feed your logs into the VictoriaLogs database in order to use LogsQL, right? "LogsQL is a simple yet powerful query language for VictoriaLogs."

Correct for now. I'm sure LogsQL support will be added into other systems for logs' management over time.

Re: Show HN: LogsQL – opinionated query language for logs

#27

Post by valyala »

piterrro wrote: post #13

I'm a https://logdy.dev (logs to UI interface) author and been recently thinking about how to enable users use a query language to search throught logs beyond usual filter. I was looking at LogsQL but then I felt that is just another QL a user will need to learn. My next though was on SQL, but it was not designed for this purpose. Any ideas? I would appreciate any recommendation (peter at logd.dev) [1] https://github…

How about allowing users creating pipelines with good old grep, awk, cut, etc., directly in the web UI?

Re: Show HN: LogsQL – opinionated query language for logs

#28

Post by dan-robertson »

valyala wrote: post #21
dan-robertson wrote: post #19

Earlier quoted context omitted.

I used to spend a lot of time looking at logs from a complex state machine. I would pull up a half day of logs in less (maybe a few GB), and search for something I was interested in like an id from an error message. This could be slow (tricks were disabling line numbers and searching backwards from the end) and then answer questions of the form ‘how long from this line until the next line matching x?’ or ‘what events…

VictoriaLogs could fit your use case: - It natively supports 'stream' concept [1] - this is basically logs received from a single application instance. - It allows efficiently querying all the logs, which belong to a single stream, on the given time range, via 'curl', and passing them to 'less' or to any other Unix command for further processing in streaming manner [2]. [1] https://docs.victoriametrics.com/victorialo…

I am indeed excited about VictoriaLogs.

Re: Show HN: LogsQL – opinionated query language for logs

#29

Post by ople »

valyala wrote: post #25
ople wrote: post #6

Recently there seems to be an bunch of SPL (Splunk) -like query languages popping up: PQL, PRQL, Grafana Explore Logs syntax, Kusto.. Probably others as well. Does yet another similar but slightly different language make sense? Why not leverage an existing one?

Diversity is great! Let's see which query language for logs will survive.

True. My bet is that arbitrary text queries converted to SQL via LLM will become an increasingly popular alternative. Not sure if it will completely replace the custom query languages though..

Re: Show HN: LogsQL – opinionated query language for logs

#30

Post by gbuk2013 »

dan-robertson wrote: post #19
gbuk2013 wrote: post #18

Earlier quoted context omitted.

Genuine question: does your job involve troubleshooting from logs on a regular basis? Because if it does, I would be surprised that you feel the way you do. My experience is with ELK but at least Kibana interface is pretty decent for applying filter combinations to find the needle in a haystack of logs. And in terms of ingestion, if you are in a container environment you can just configure stdout from the container t…

I used to spend a lot of time looking at logs from a complex state machine. I would pull up a half day of logs in less (maybe a few GB), and search for something I was interested in like an id from an error message. This could be slow (tricks were disabling line numbers and searching backwards from the end) and then answer questions of the form ‘how long from this line until the next line matching x?’ or ‘what events…

In the past I spent a lot of time cutting up logs with grep, less, jq and Perl. It was amazing UX that Kibana can't beat in terms of performance, assuming you already know the time-window you are interested in (although I never learned enough gnuplot to be able to do visualisations so Kibana wins there). However, all that went out of the window when I moved into a world of milti-instance micro-services in the cloud and SOC2 compliance. No more downloading of logs to my local machine and cutting them up with sensible tools. :(

That said, nothing that you outlined above is particularly difficult in Kibana, the main annoyance being the response time of the query API (somewhat mitigated by indexing). Based on your vague description my vague workflow would be:

  - filter for type x
  - limit time range to between occurrences of x
  - change filter to type y
  - an any point pick out the interesting fields of the log message to reduce noise in UI
  - save and reuse this query if it is something I do regularly
  - if your state machine has a concept of a flow then filter by a relevant correlation ID
Not sure what you mean by "finding the log file" since Elasticsearch is a document database where each log line is a document.
Post reply on HN