Ask HN: How do you manage logs for your backend services?

If you're logging structured JSON, then you'll hit a ton of issues - Elasticsearch can't handle, say, one record with {foo: 123} and another with {foo: "abc"} - it'll choke on the different types and 400 error on ingest.

Even if you try to coerce values to string, you'll hit problems with nested values like {foo: "abc"} vs {foo: {bar: "baz"}}. So now you have to coerce to something like {"foo.bar": "baz"}, and then you have to escape the dot when querying...

Finally, if you solve all the above, you'll hit problems with high cardinality of unique fields. Especially if you are accepting arbitrary nested values, at some point someone is going to log something like {users_by_email: {: }} and now you have one field for every unique email...

These problems are tractable but a massive hassle to get right without majorly locking down what applications are allowed to log in terms of structured data.

As a seperate issue, Elasticsearch does fuzzy keyword matching by default. eg. if you search for "foo" you'll get results for "fooing" and "fooed" but not nessecarily for "foobar" (because it's splitting by word - and the characters it considers part of one "word" aren't obvious). This is great if you want to search a product catalog, but horrible when you're trying to find an exact string in a technical context. Yes you can rephrase your query to avoid it, but that's not the default and most people won't know how to structure their query perfectly to avoid all the footguns.

Finally, as others are saying here, Elasticsearch is just painful and heavy to manage.

As for what to use instead...I don't have good answers. I haven't exhaustively checked out all the other products being mentioned, but in my experience a lot of them will have similar issues around field cardinality, which means it'll always be possible to cripple your database with bad data. This is less of an issue if you're just running a few services, but in larger orgs it can be nigh impossible to keep ahead of.

For smaller scale deployments, don't underestimate just shipping everything to timestamp+service named files as newline-delimited JSON, and using jq and grep for search and a cronjob to delete/archive old files.

When it comes to the "read from local source and ship elsewhere" component, I've had the best luck with filebeat (specifically for files -> kafka). Most others tend to read as fast as they can then buffer in memory or disk if they can't write immediately, whereas filebeat will only read the source file as fast as it can write downstream.

Note however that all such components are awful to configure, as they seek to provide a (often turing-complete) configuration file for transforming your logs before shipping them, and like most turing-complete configuration scripts, they're less readable and more buggy than the equivalent would've been in any real programming language.

Ok, rant over. Sorry, a good logging system is kind of my white whale.