Preview of Explore Logs, a new way to browse your logs without writing LogQL

This is still a valid use case but pretend for a minute you have thousands or millions of log lines to inspect. Even after filtering for ERROR level only, you still have too many "those are normal" errors, devs swear (but do not fix). And maybe the data you need to diagnose isn't even in ERROR!

The solution? Use log queries to compare a normal and abnormal process or cluster, group them by some kind of fingerprint, then apply some Laplace smoothing or other bayesian techniques to score fingerprints by strength of association with abnormal. This lets me rapidly identify problems at scale that would otherwise take hours of pouring through logs to exclude stuff by hand.

This works any time you can divide logs into "good" and "bad." Example scenarios:

- canary analysis, comparing canary and baseline

- single faulty pod in a deploy, comparing the bad container to the n good ones

- one AZ or region in a multi-region deploy

- now versus yesterday, or versus an hour ago, etc

- Android versus iPhone

ldpreloadhook has an example of hooking write() with LD_PRELOAD=, which e.g. golang programs built without libc don't support.

When systemd is /sbin/init, it owns all subprocess' file handles already, so there's no need to close(0), time, open(0) with gdb.

Without having to logship (copy buffers that are flushed and/or have newline characters in the stream) to a network or local Arrow database files and or SQLite vtables,

journalctl (journald) supports pattern matching with: -t syslogidentifier, -u unit; and -g grepexpr of the MESSAGE= field:

  journalctl -u 
  journalctl -u init.scope --reverse
  journalctl -u unit.scope -g "Reached target"  # and then "/sleep" to search and highlight with less
  
  journalctl -u auditd.service

  # this is slow because it's a full table scan, because
  # journald does not index the logfiles;
  # and -g/--grep is case insensitive if the query is all lowercase:
  journalctl -g avc --reverse
  journalctl -g AVC --reverse

  # this is faster:
  journalctl -t audit -g AVC -r

  # this is still faster,
  # because it only searches the current boot:
  journalctl -b 0 -t audit -g AVC

  # these are equivalent:
  journalctl -b 0 --dmesg -t kernel
  journalctl -k

  # 
  journalctl -b 0 --user | grep -i -C "xyz123"

There is a Grafana Loki Docker Driver that logships from all containers visible on that DOCKER_HOST docker socket to Grafana for querying with Loki: https://grafana.com/docs/loki/latest/send-data/docker-driver...

Podman with Systemd doesn't need the Grafana Docker Driver (or other logshippers like logstash, loggly, or fluentd) because systemd spawns containers and optionally pipes their stdout/stderr logs to journald.

Influx has Telegraf, InfluxDB, Chronograf, and Kapacitor. Chronograf is their WebUI which provides a query interface for configurable chart dashboards and InfluxQL.

Grafana supports SQL, PromQL, InfluxQL, and LogQL.

Graylog2 also indexes logfiles.

But you can't query stdout and stderr you or /sbin/init haven't logged to a file.

I want to query my logs like a SQL table, not a time series database.

Is this google cloud logging? If so, personally I quite like it, especially for looking through logs from multiple sources at the same time. Being able to put all your logs through there, and then search them with a simple query language, feels very convenient.