From recent experience, I'm thinking logs need to be written for AI. Over the last few months, I've had a couple of issues where I took a bunch of logs from a bunch of interacting programs, pointed the AI at the logs and the source code and it's been really effective and finding the problems, often seeing patterns that would have been really hard for me to spot in all the noise.
Log messages are mostly for the people operating your software
31–40 of 83 posts
Re: Log messages are mostly for the people operating your software
#32From recent experience, I'm thinking logs need to be written for AI. Over the last few months, I've had a couple of issues where I took a bunch of logs from a bunch of interacting programs, pointed the AI at the logs and the source code and it's been really effective and finding the problems, often seeing patterns that would have been really hard for me to spot in all the noise.
The log needs to document, at least in broad steps and critical details, what the next operation is and what key parameters were provided to it.
A human, or an 'agent' can use those to figure out why said next step might have gone wrong.
Re: Log messages are mostly for the people operating your software
#33All software should provide something meaningful for anybody to diagnose, if they’re inclined to. It’s particularly bad in the (Apple) mobile ecosystem, including AppleTV. I have AdGuard Home but one of my spouse’s streaming services wouldn’t work. “There was a problem.” Gee thanks. Eventually figured out that I had to unblock a few hosts so it would work. Only found which ones by googling and finding some other poor…
They don't want tinkering or tinkerers. Apple is all about walled-off, locked-down, black box, just-works (when it does) etc. It's supposed to seem like magic. You're not supposed to tinker with magic, it makes it pedestrian. Apple as a brand is a lifestyle, a feeling. The slick, polished brand. Remember "I'm a Mac, and I'm a PC"? PC is where you tinker, and there is screws and nuts and bolts and jargon and troublesh…
Okay, but then their stuff needs to be perfect as designed. Because the moment there's a bug, we're back to needing diagnostic tools.
Re: Log messages are mostly for the people operating your software
#34Earlier quoted context omitted.
They don't want tinkering or tinkerers. Apple is all about walled-off, locked-down, black box, just-works (when it does) etc. It's supposed to seem like magic. You're not supposed to tinker with magic, it makes it pedestrian. Apple as a brand is a lifestyle, a feeling. The slick, polished brand. Remember "I'm a Mac, and I'm a PC"? PC is where you tinker, and there is screws and nuts and bolts and jargon and troublesh…
Okay, but then their stuff needs to be perfect as designed. Because the moment there's a bug, we're back to needing diagnostic tools.
There is a self-regulating loop that Apple users quickly learn not to "draw outside the lines" and just use the thing as designed and intended by Apple. If you use stuff like AdGuard, custom DNS etc, that's tinkerer tier stuff. A good Apple user either watches the ads or pays not to see them.
Re: Log messages are mostly for the people operating your software
#35Earlier quoted context omitted.
Okay, but then their stuff needs to be perfect as designed. Because the moment there's a bug, we're back to needing diagnostic tools.
There is a self-regulating loop that Apple users quickly learn not to "draw outside the lines" and just use the thing as designed and intended by Apple. If you use stuff like AdGuard, custom DNS etc, that's tinkerer tier stuff. A good Apple user either watches the ads or pays not to see them.
My point is that even inside the lines there are still bugs.
Re: Log messages are mostly for the people operating your software
#36I recently went all-in on the systemd ecosystem as much as I could on some recent hardware installs, and my biggest pet peeve is the double timestamps and double logs I find in journalctl... it's like they never intended you to read the logs...
Re: Log messages are mostly for the people operating your software
#37Well the useful ones are. The rest are screaming into the void, or rather, the operator’s ear.
Re: Log messages are mostly for the people operating your software
#38For years now I’ve been pushing for moving of all non actionable error messages and all aggregate-actionable error messages into telemetry data instead.
Not the least of which because log processing SaaS companies seem to be overcharging for their services even versus hosted Grafana services, and really many of us could do away with the rent seeking entirely.
The computational complexity of finding meaning in log files versus telemetry data leans toward this always being the case. It will never change except in brief cases of VC money subsidizing your subscription.
If an error shouldn’t trigger operator actions, but 1000 should, that’s a telemetry alert not a data dog or Splunk problem.
Re: Log messages are mostly for the people operating your software
#39All software should provide something meaningful for anybody to diagnose, if they’re inclined to. It’s particularly bad in the (Apple) mobile ecosystem, including AppleTV. I have AdGuard Home but one of my spouse’s streaming services wouldn’t work. “There was a problem.” Gee thanks. Eventually figured out that I had to unblock a few hosts so it would work. Only found which ones by googling and finding some other poor…
They don't want tinkering or tinkerers. Apple is all about walled-off, locked-down, black box, just-works (when it does) etc. It's supposed to seem like magic. You're not supposed to tinker with magic, it makes it pedestrian. Apple as a brand is a lifestyle, a feeling. The slick, polished brand. Remember "I'm a Mac, and I'm a PC"? PC is where you tinker, and there is screws and nuts and bolts and jargon and troublesh…
People make decisions based on their own value system. I’m glad to have choices. I can get everything done with the tools we call computers.
When I view the logs on my Apple systems they make sense to me. One does have to understand the logs which implies understanding the system under diagnosis.
Re: Log messages are mostly for the people operating your software
#40From recent experience, I'm thinking logs need to be written for AI. Over the last few months, I've had a couple of issues where I took a bunch of logs from a bunch of interacting programs, pointed the AI at the logs and the source code and it's been really effective and finding the problems, often seeing patterns that would have been really hard for me to spot in all the noise.
The log needs to document, at least in broad steps and critical details, what the next operation is and what key parameters were provided to it. A human, or an 'agent' can use those to figure out why said next step might have gone wrong.
This is the way I like to do it. I know bloating the logs too much can be a problem, but it's even worse if you're lacking information to reconstruct what happened when there ends up being a problem. And only providing that detail when there's an error isn't enough. What if the issue never triggered an error in the application and it was only caught later on either by a person seeing something was off or by an error a downstream system?
Also it's helpful to log before operations rather than after because if a step gets stuck it's possible to know what it's stuck on.