I used to spend a lot of time looking at logs from a complex state machine. I would pull up a half day of logs in less (maybe a few GB), and search for something I was interested in like an id from an error message. This could be slow (tricks were disabling line numbers and searching backwards from the end) and then answer questions of the form ‘how long from this line until the next line matching x?’ or ‘what events of type y happened after the first event of type x after this log line’ or ‘what events of type x containing y happened between these log lines’ and suchlike. This is annoying to do with less; useful tricks were learning the shortcuts, taking advantages of regexes to highlight things or advance long wrapped lines with /^ and copying interesting messages into an editor and writing notes.
ELK/grafana don’t give great solutions here. Elastic/kibana already struggles with the first part because it doesn’t make it ergonomic to separate the part of your query that is ‘finding the log file’ and the part that is ‘searching within the file’. For the rest the UI tends to be more clunky and less information-dense than less, though if your data is sufficiently structured the kibana tables help. In particular, you’re still copying notes somewhere but next before/after queries aren’t easy/quick/ergonomic to express (I think), and changing the query and waiting is pretty slow. The typical search described above would be fast because the result wouldn’t be far away, and pressing n or N brings the next result exactly where you expect it on the screen, so you don’t need to try to find it on a page.
I think sql-like queries aren’t great here either because they are verbose to write and require either very complicated self-joins/ctes trying to write queries that find rows to search between/after or copying a bunch of timestamps back and forth.
Something people sometimes talk about is ltl (linear temporal logic) for log analysis. I think it can be useful for certain kinds of analysis but is less useful for interactive exploratory log querying. I don’t know of how to do ltl queries against data in Loki or Elasticsearch.
To be clear, most of the reasons that elk/grafana don’t work for the use case I described vaguely above are problems with the frontend ui rather than fundamental issues with the backend. It may just be the kind of logs too – if your logs all look similar to logs of incoming http requests, the existing ui may work great.